I am working to do some testing using Ceph in a RED and non-RED environment. That might be a long way off though.
Can you tell me if the OSD to OSD flows are via long-lived TCP sessions for writes or is the session set-up/torn-down once the block is ACK'ed as durable back to the client? On Sun, Apr 13, 2014 at 7:00 PM, Adam Clark <[email protected]> wrote: > Most definitely, I actually had the same thoughts after I replied. > > The many more flows behaviour is more likely to be kept healthy with RED > rather than pause frames. > > Pause frames would halt all flows at time of congestion (while keeping TCP > windowing unaware of the congestion issues), > > RED would impact a fraction of the flows and keep the overall throughput > of the system high using TCP as the primary method for individual flow > rates. > > You are most likely to have congestion into an OSD node during multiple > clients writes, or into a client node for reads from multiple OSDs. Using > DCB-X may inadvertently impact other Ceph IOPS that are not directly > experiencing congestion. > > Would be interesting to do some tests though. > > Regards > > Adam > > > > On Mon, Apr 14, 2014 at 12:46 AM, N. Richard Solis <[email protected]>wrote: > >> Adam, >> >> Thank you so much for replying. >> >> The links you posted are excellent discussions for me to reference. >> >> What's interesting about the difference between iSCSI and Ceph/RBD type >> storage is the fact that under Ceph the client will make use of lots of >> individual TCP sessions to access any particular virtual block device, >> whereas iSCSI will use a single TCP session for every block. That's how >> you get into the elephant-flow vs mice-flow problem in a converged network. >> The (elephant) iSCSI device would compete for limited bandwidth with all >> of the other TCP flows and you get crappy storage performance as the TCP >> loss mechanisms kick in and drop the iSCSI TCP frames. >> >> If I'm thinking correctly, then Ceph wouldn't have this problem since any >> individual block device would be spread across multiple OSDs and have >> multiple TCP sessions connected to any particular client. So the problem >> of a single elephant flow really doesn't develop because you don't have >> those large single-session TCP flows. Am I thinking correctly? >> >> >> >> >> On Sun, Apr 13, 2014 at 5:00 AM, Adam Clark <[email protected]>wrote: >> >>> Heya, >>> I have read a bunch of stuff regarding PFC/DCB-X and iSCSI, so it >>> might be a similar argument with Ceph given the underlying transport is TCP. >>> DCB-X was needed as FC has no tolerance for loss and deals with >>> congestion by instructing senders to slow down using pause frames. >>> >>> From what I read, it can help, but you really need to know what you are >>> doing, as TCP has it's own congestion avoidance mechanisms it works under >>> most use cases. It it probably better to randomly introduce drops in >>> congestions rather than use a Pause Frame. >>> >>> Check these out: >>> http://blog.ipspace.net/2013/07/iscsi-with-pfc.html >>> https://blogs.cisco.com/datacenter/the-napkin-dialogues-lossless-iscsi/ >>> >>> Cisco centric, but but applicable to other vendors also. >>> >>> If you are planning on running your ceph traffic alongside other traffic >>> types, I would definitely be putting it in it's own class and guaranteeing >>> bandwidth under times of congestion, short queues (<50ms max delay) with >>> somewhat aggressive RED thresholds would probably suffice. >>> >>> Cheers >>> >>> Adam >>> >>> >>> >>> On Fri, Apr 11, 2014 at 11:42 PM, N. Richard Solis <[email protected]>wrote: >>> >>>> Guys, >>>> >>>> I'm new to ceph in general but I'm wondering if anyone out there is >>>> using Ceph with any of the Data Center Bridging (DCB) technologies? I'm >>>> specifically thinking of DCB-X support provided by the open-lldp package. >>>> >>>> I'm wondering if there is any benefit to be gained by making use of the >>>> lossless Ethernet support in a properly configured network for the OSD to >>>> OSD traffic that would normally occur on the "cluster" network. >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> [email protected] >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>>> >>> >> >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
