No correlation with compactions. No queries.
On Thu, Jan 03, 2013 at 11:24:17AM -0500, William Slacum wrote: > Have you also been tracking compactions? Did you have a query load? > > > On Wed, Jan 2, 2013 at 7:25 PM, Kepner, Jeremy - 0553 - MITLL < > [email protected]> wrote: > > > Hmmm, that's interesting, because in the past I didn't see this behavior. > > It might be worth having someone look into because it seems to have a 2x > > impact on sustained ingest. > > > > Regards. -Jeremy > > > > On Jan 2, 2013, at 2:23 PM, Keith Turner wrote: > > > > > On Wed, Jan 2, 2013 at 2:11 PM, Jeremy Kepner <[email protected]> wrote: > > >> So what mechanism causes the number of Xceivers to increase? > > > > > > Its been a while since I looked at the data node source code. When I > > > last look at it an Xceiver was just a thread created to handle a > > > datanode request. The thread went away after the request was > > > processed. So major and minor compactions running would cause more > > > Xceivers to be created to read and write data. > > > > > > Newer datanode code may use a thread pool instead of creating a > > > thread/xceiver for each request. I am not sure. > > > > > >> I am carefully controlling the number of ingestors and the data isn't > > varying too much. > > >> I would expect the number of Xceivers to remain consant. > > >> > > >> Regards. -Jeremy > > >> > > >> On Tue, Jan 01, 2013 at 09:45:20PM -0500, Eric Newton wrote: > > >>> Hey Jeremy, > > >>> > > >>> Can you compare the ingest rate to the number of tablets, too? > > >>> > > >>> I've found, that if I have 20-80 tablets per server (on similar > > hardware) I > > >>> get the best performance. > > >>> > > >>> # of Xceivers == number of writers when ingest is the primary target. > > >>> > > >>> Also, is this 1.4 or trunk? > > >>> > > >>> -Eric > > >>> > > >>> > > >>> > > >>> On Tue, Jan 1, 2013 at 9:19 PM, Kepner, Jeremy - 1010 - MITLL < > > >>> [email protected]> wrote: > > >>> > > >>>> Accumulo Colleagues, > > >>>> I am trying to optimize my ingest into a single node Accumulo > > instance > > >>>> running on a 32 core node with 96 GB of RAM. I am seeing the follow > > ingest > > >>>> variations as a I change the number of ingest processes (see > > attached): > > >>>> > > >>>> ------------------------------------- > > >>>> Ingestors, Ingest rate > > >>>> ------------------------------------- > > >>>> 1, 60K inserts/sec (stable) > > >>>> 2, 120K inserts/sec (stable) > > >>>> 3, 60K to 180K inserts/sec > > >>>> 4, 90K to 220K inserts/sec > > >>>> 8, 80K to 280K inserts/sec > > >>>> 12, 80K to 280K inserts/sec > > >>>> ------------------------------------- > > >>>> > > >>>> The only thing I can see that correlates with the ingest rate is the > > >>>> number of Xceivers. When the ingest rate is high the number of > > Xceivers is > > >>>> usually low. Likewise, when the ingest rate drops, the number of > > Xceivers > > >>>> usually increases significantly. > > >>>> > > >>>> Question: What role to Xceivers play in ingest? > > >>>> > > >>>> Request: It would be great to add a plot showing the number of > > Xceivers > > >>>> over time to the diagnostics. > > >>>> > > >>>> Regards. -Jeremy > > >>>> > > >>>> > > > >
