Hmmm, that's interesting, because in the past I didn't see this behavior. It might be worth having someone look into because it seems to have a 2x impact on sustained ingest.
Regards. -Jeremy On Jan 2, 2013, at 2:23 PM, Keith Turner wrote: > On Wed, Jan 2, 2013 at 2:11 PM, Jeremy Kepner <[email protected]> wrote: >> So what mechanism causes the number of Xceivers to increase? > > Its been a while since I looked at the data node source code. When I > last look at it an Xceiver was just a thread created to handle a > datanode request. The thread went away after the request was > processed. So major and minor compactions running would cause more > Xceivers to be created to read and write data. > > Newer datanode code may use a thread pool instead of creating a > thread/xceiver for each request. I am not sure. > >> I am carefully controlling the number of ingestors and the data isn't >> varying too much. >> I would expect the number of Xceivers to remain consant. >> >> Regards. -Jeremy >> >> On Tue, Jan 01, 2013 at 09:45:20PM -0500, Eric Newton wrote: >>> Hey Jeremy, >>> >>> Can you compare the ingest rate to the number of tablets, too? >>> >>> I've found, that if I have 20-80 tablets per server (on similar hardware) I >>> get the best performance. >>> >>> # of Xceivers == number of writers when ingest is the primary target. >>> >>> Also, is this 1.4 or trunk? >>> >>> -Eric >>> >>> >>> >>> On Tue, Jan 1, 2013 at 9:19 PM, Kepner, Jeremy - 1010 - MITLL < >>> [email protected]> wrote: >>> >>>> Accumulo Colleagues, >>>> I am trying to optimize my ingest into a single node Accumulo instance >>>> running on a 32 core node with 96 GB of RAM. I am seeing the follow ingest >>>> variations as a I change the number of ingest processes (see attached): >>>> >>>> ------------------------------------- >>>> Ingestors, Ingest rate >>>> ------------------------------------- >>>> 1, 60K inserts/sec (stable) >>>> 2, 120K inserts/sec (stable) >>>> 3, 60K to 180K inserts/sec >>>> 4, 90K to 220K inserts/sec >>>> 8, 80K to 280K inserts/sec >>>> 12, 80K to 280K inserts/sec >>>> ------------------------------------- >>>> >>>> The only thing I can see that correlates with the ingest rate is the >>>> number of Xceivers. When the ingest rate is high the number of Xceivers is >>>> usually low. Likewise, when the ingest rate drops, the number of Xceivers >>>> usually increases significantly. >>>> >>>> Question: What role to Xceivers play in ingest? >>>> >>>> Request: It would be great to add a plot showing the number of Xceivers >>>> over time to the diagnostics. >>>> >>>> Regards. -Jeremy >>>> >>>>
