Hence the new TimedBuffer didn't change drastically the latencies? I've got some graph after the stress test with a maxed IO, if you're curious :)
2017-05-12 22:19 GMT+02:00 Michael André Pearce [via ActiveMQ] < [email protected]>: > Yup deleting the data on version change removes the issue. > > Sent from my iPhone > > > On 12 May 2017, at 20:54, Clebert Suconic <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=4726097&i=0>> wrote: > > > > Can you cleanup your data (as a test only?) rm -rf data. > > > > > > > > > > perhaps the spikes you have is the compacting running? > > > > On Fri, May 12, 2017 at 3:51 PM, Michael André Pearce > > <[hidden email] <http:///user/SendEmail.jtp?type=node&node=4726097&i=1>> > wrote: > >> All I'm doing is changing Artemis.profile of the instance to the > different versions. > >> > >> > >> Sent from my iPhone > >> > >>> On 12 May 2017, at 20:35, nigro_franz <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=4726097&i=2>> wrote: > >>> > >>> Too strange...have you tried with NIO too?It uses always the same > buffer to > >>> batch writes AFAIK.... > >>> > >>> 2017-05-12 21:30 GMT+02:00 Michael André Pearce [via ActiveMQ] < > >>> [hidden email] <http:///user/SendEmail.jtp?type=node&node=4726097&i=3>>: > > >>> > >>>> Ok so it isn't the times buffer changes. Just deployed 2.1.0 with and > >>>> without the timed buffer changes. Running the same tests no > difference. > >>>> > >>>> Must be something else. Causing the behaviour change. > >>>> > >>>> Sent from my iPhone > >>>> > >>>>> On 12 May 2017, at 18:10, Michael André Pearce <[hidden email] > >>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=0>> wrote: > >>>>> > >>>>> I've left for the day now. If I get time over the weekend I'll try > see > >>>> if I can make a build of 2.1.0 without that change and see if it > makes any > >>>> difference. > >>>>> > >>>>> Can someone point me to the PR for that change, so I know what I'm > >>>> unpicking locally? > >>>>> > >>>>> Sent from my iPhone > >>>>> > >>>>>> On 12 May 2017, at 17:58, Clebert Suconic <[hidden email] > >>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=1>> wrote: > >>>>>> > >>>>>> As the only thing that could affect this is the Change on timed > buffer. > >>>>>> Afaik > >>>>>> > >>>>>> > >>>>>> On Fri, May 12, 2017 at 12:57 PM Clebert Suconic <[hidden email] > >>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=2>> > >>>>>> wrote: > >>>>>> > >>>>>>> I'm considering only keeping the pooled buffer part and switch > back to > >>>> the > >>>>>>> ole sleep or an improved sleep we had. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Fri, May 12, 2017 at 12:49 PM Michael André Pearce < > >>>>>>> [hidden email] <http:///user/SendEmail.jtp? > type=node&node=4726091&i=3>> > >>>> wrote: > >>>>>>> > >>>>>>>> As it seems I can't send images to mail list, just sent to you > both > >>>> via > >>>>>>>> email. Some graphs we have comparing versions. > >>>>>>>> > >>>>>>>> Not sure what changes might cause it. > >>>>>>>> > >>>>>>>>> On 12 May 2017, at 17:37, Clebert Suconic <[hidden email] > >>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=4>> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> There is a class we use on producer. TokenLimiter. Perhaps you > >>>> could > >>>>>>>>> reuse that one ? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> On Fri, May 12, 2017 at 11:00 AM nigro_franz <[hidden email] > >>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=5>> > >>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> I was thinking of a similar solution but I've discovered that > >>>> couldn't > >>>>>>>> work > >>>>>>>>>> (in the old or the new TimedBuffer too), because of the > >>>>>>>>>> TimedBuffer::checkSize method that could force a flush if the > batch > >>>>>>>> buffer > >>>>>>>>>> if not big enough to receive new data, going IOPS. > >>>>>>>>>> Sadly TimedBuffer::checkSize is outside any timeout, but > depends on > >>>> the > >>>>>>>>>> writers. > >>>>>>>>>> > >>>>>>>>>> That's why I've implemented the "compensation" right after any > >>>> flush, > >>>>>>>> in > >>>>>>>>>> order to work with forced flushes too: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>> https://github.com/franz1981/activemq-artemis/blob/ > >>>> 4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/ > >>>> src/main/java/org/apache/activemq/artemis/core/io/ > >>>> buffer/TimedBuffer.java#L338 > >>>>>>>>>> > >>>>>>>>>> Doing it on TimedBuffer::flush all the flushes on disk will be > >>>>>>>> compensated > >>>>>>>>>> (half of the story: ASYNCIO is async so depends on libAIO > >>>> partially!) > >>>>>>>>>> Regarding the IOPS computation I've built this, as you've > >>>> suggested: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>> https://github.com/franz1981/activemq-artemis/blob/ > >>>> 4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/ > >>>> src/main/java/org/apache/activemq/artemis/core/io/ > >>>> buffer/TimedBuffer.java#L119 > >>>>>>>>>> > >>>>>>>>>> The performance seems pretty good, it compensates well but it > is > >>>> faster > >>>>>>>>>> than > >>>>>>>>>> the original version, limiting IOPS too! > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Franz > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> View this message in context: > >>>>>>>>>> > >>>>>>>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis- > >>>> IOPS-Limiter-strategy-tp4725875p4726057.html > >>>>>>>>>> Sent from the ActiveMQ - Dev mailing list archive at > Nabble.com. > >>>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Clebert Suconic > >>>>>>>> > >>>>>>> -- > >>>>>>> Clebert Suconic > >>>>>>> > >>>>>> -- > >>>>>> Clebert Suconic > >>>> > >>>> > >>>> ------------------------------ > >>>> If you reply to this email, your message will be added to the > discussion > >>>> below: > >>>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis- > >>>> IOPS-Limiter-strategy-tp4725875p4726091.html > >>>> To unsubscribe from [DISCUSS] Artemis IOPS Limiter strategy, click > here > >>>> < > > >>>> . > >>>> NAML > >>>> <http://activemq.2283324.n4.nabble.com/template/ > NamlServlet.jtp?macro=macro_viewer&id=instant_html% > 21nabble%3Aemail.naml&base=nabble.naml.namespaces. > BasicNamespace-nabble.view.web.template.NabbleNamespace- > nabble.view.web.template.NodeNamespace&breadcrumbs= > notify_subscribers%21nabble%3Aemail.naml-instant_emails% > 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > >>>> > >>> > >>> > >>> > >>> > >>> -- > >>> View this message in context: http://activemq.2283324.n4. > nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726093.html > >>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com. > > > > > > > > -- > > Clebert Suconic > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis- > IOPS-Limiter-strategy-tp4725875p4726097.html > To unsubscribe from [DISCUSS] Artemis IOPS Limiter strategy, click here > <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4725875&code=bmlncm8uZnJhQGdtYWlsLmNvbXw0NzI1ODc1fC0zNzI3NjE2NQ==> > . > NAML > <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726099.html Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
