You may be seeing some impact due to ACCUMULO-893: http://issues.apache.org/jira/browse/ACCUMULO-893
Are you seeing 2-minute hold times popping up? -Eric On Wed, Jan 2, 2013 at 7:25 PM, Kepner, Jeremy - 0553 - MITLL < [email protected]> wrote: > Hmmm, that's interesting, because in the past I didn't see this behavior. > It might be worth having someone look into because it seems to have a 2x > impact on sustained ingest. > > Regards. -Jeremy > > On Jan 2, 2013, at 2:23 PM, Keith Turner wrote: > > > On Wed, Jan 2, 2013 at 2:11 PM, Jeremy Kepner <[email protected]> wrote: > >> So what mechanism causes the number of Xceivers to increase? > > > > Its been a while since I looked at the data node source code. When I > > last look at it an Xceiver was just a thread created to handle a > > datanode request. The thread went away after the request was > > processed. So major and minor compactions running would cause more > > Xceivers to be created to read and write data. > > > > Newer datanode code may use a thread pool instead of creating a > > thread/xceiver for each request. I am not sure. > > > >> I am carefully controlling the number of ingestors and the data isn't > varying too much. > >> I would expect the number of Xceivers to remain consant. > >> > >> Regards. -Jeremy > >> > >> On Tue, Jan 01, 2013 at 09:45:20PM -0500, Eric Newton wrote: > >>> Hey Jeremy, > >>> > >>> Can you compare the ingest rate to the number of tablets, too? > >>> > >>> I've found, that if I have 20-80 tablets per server (on similar > hardware) I > >>> get the best performance. > >>> > >>> # of Xceivers == number of writers when ingest is the primary target. > >>> > >>> Also, is this 1.4 or trunk? > >>> > >>> -Eric > >>> > >>> > >>> > >>> On Tue, Jan 1, 2013 at 9:19 PM, Kepner, Jeremy - 1010 - MITLL < > >>> [email protected]> wrote: > >>> > >>>> Accumulo Colleagues, > >>>> I am trying to optimize my ingest into a single node Accumulo > instance > >>>> running on a 32 core node with 96 GB of RAM. I am seeing the follow > ingest > >>>> variations as a I change the number of ingest processes (see > attached): > >>>> > >>>> ------------------------------------- > >>>> Ingestors, Ingest rate > >>>> ------------------------------------- > >>>> 1, 60K inserts/sec (stable) > >>>> 2, 120K inserts/sec (stable) > >>>> 3, 60K to 180K inserts/sec > >>>> 4, 90K to 220K inserts/sec > >>>> 8, 80K to 280K inserts/sec > >>>> 12, 80K to 280K inserts/sec > >>>> ------------------------------------- > >>>> > >>>> The only thing I can see that correlates with the ingest rate is the > >>>> number of Xceivers. When the ingest rate is high the number of > Xceivers is > >>>> usually low. Likewise, when the ingest rate drops, the number of > Xceivers > >>>> usually increases significantly. > >>>> > >>>> Question: What role to Xceivers play in ingest? > >>>> > >>>> Request: It would be great to add a plot showing the number of > Xceivers > >>>> over time to the diagnostics. > >>>> > >>>> Regards. -Jeremy > >>>> > >>>> > >
