And the most weird thing is that I'm having better results with NIO then ASYNCIO (with small sized messages) and I'm not getting why....
Time to go, 'm off; in Italy is 23:30 now..have fun guys and thanks: always great stuff :) 2017-05-12 23:02 GMT+02:00 nigro_franz <[email protected]>: > ahah np Michael!!! > > Anyway I was really expecting some more spike compared to the old > TimedBuffer: Is one of the reasons why I wanted to put the IO compensation. > The new one seems to work well (with ASYNCIO) in this regard by the tests > I've done: I'm hoping it will help to remove that little spikes too! > > > 2017-05-12 22:54 GMT+02:00 Michael André Pearce [via ActiveMQ] < > [email protected]>: > > > Correct, :) seems like tester error. Apologies > > > > Sent from my iPhone > > > > > On 12 May 2017, at 21:24, nigro_franz <[hidden email] > > <http:///user/SendEmail.jtp?type=node&node=4726101&i=0>> wrote: > > > > > > Hence the new TimedBuffer didn't change drastically the latencies? > > > I've got some graph after the stress test with a maxed IO, if you're > > > curious :) > > > > > > 2017-05-12 22:19 GMT+02:00 Michael André Pearce [via ActiveMQ] < > > > [hidden email] <http:///user/SendEmail.jtp? > type=node&node=4726101&i=1>>: > > > > > > > >> Yup deleting the data on version change removes the issue. > > >> > > >> Sent from my iPhone > > >> > > >>> On 12 May 2017, at 20:54, Clebert Suconic <[hidden email] > > >> <http:///user/SendEmail.jtp?type=node&node=4726097&i=0>> wrote: > > >>> > > >>> Can you cleanup your data (as a test only?) rm -rf data. > > >>> > > >>> > > >>> > > >>> > > >>> perhaps the spikes you have is the compacting running? > > >>> > > >>> On Fri, May 12, 2017 at 3:51 PM, Michael André Pearce > > >>> <[hidden email] <http:///user/SendEmail.jtp? > type=node&node=4726097&i=1>> > > > > >> wrote: > > >>>> All I'm doing is changing Artemis.profile of the instance to the > > >> different versions. > > >>>> > > >>>> > > >>>> Sent from my iPhone > > >>>> > > >>>>> On 12 May 2017, at 20:35, nigro_franz <[hidden email] > > >> <http:///user/SendEmail.jtp?type=node&node=4726097&i=2>> wrote: > > >>>>> > > >>>>> Too strange...have you tried with NIO too?It uses always the same > > >> buffer to > > >>>>> batch writes AFAIK.... > > >>>>> > > >>>>> 2017-05-12 21:30 GMT+02:00 Michael André Pearce [via ActiveMQ] < > > >>>>> [hidden email] <http:///user/SendEmail.jtp? > > type=node&node=4726097&i=3>>: > > >> > > >>>>> > > >>>>>> Ok so it isn't the times buffer changes. Just deployed 2.1.0 with > > and > > >>>>>> without the timed buffer changes. Running the same tests no > > >> difference. > > >>>>>> > > >>>>>> Must be something else. Causing the behaviour change. > > >>>>>> > > >>>>>> Sent from my iPhone > > >>>>>> > > >>>>>>> On 12 May 2017, at 18:10, Michael André Pearce <[hidden email] > > >>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=0>> wrote: > > >>>>>>> > > >>>>>>> I've left for the day now. If I get time over the weekend I'll > try > > >> see > > >>>>>> if I can make a build of 2.1.0 without that change and see if it > > >> makes any > > >>>>>> difference. > > >>>>>>> > > >>>>>>> Can someone point me to the PR for that change, so I know what > I'm > > >>>>>> unpicking locally? > > >>>>>>> > > >>>>>>> Sent from my iPhone > > >>>>>>> > > >>>>>>>> On 12 May 2017, at 17:58, Clebert Suconic <[hidden email] > > >>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=1>> wrote: > > >>>>>>>> > > >>>>>>>> As the only thing that could affect this is the Change on timed > > >> buffer. > > >>>>>>>> Afaik > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Fri, May 12, 2017 at 12:57 PM Clebert Suconic <[hidden email] > > >>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=2>> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> I'm considering only keeping the pooled buffer part and switch > > >> back to > > >>>>>> the > > >>>>>>>>> ole sleep or an improved sleep we had. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> On Fri, May 12, 2017 at 12:49 PM Michael André Pearce < > > >>>>>>>>> [hidden email] <http:///user/SendEmail.jtp? > > >> type=node&node=4726091&i=3>> > > >>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> As it seems I can't send images to mail list, just sent to > > you > > >> both > > >>>>>> via > > >>>>>>>>>> email. Some graphs we have comparing versions. > > >>>>>>>>>> > > >>>>>>>>>> Not sure what changes might cause it. > > >>>>>>>>>> > > >>>>>>>>>>> On 12 May 2017, at 17:37, Clebert Suconic <[hidden email] > > >>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=4>> > > >>>>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>> There is a class we use on producer. TokenLimiter. Perhaps > > you > > >>>>>> could > > >>>>>>>>>>> reuse that one ? > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>>> On Fri, May 12, 2017 at 11:00 AM nigro_franz <[hidden email] > > >>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=5>> > > >>>>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>> I was thinking of a similar solution but I've discovered > that > > >>>>>> couldn't > > >>>>>>>>>> work > > >>>>>>>>>>>> (in the old or the new TimedBuffer too), because of the > > >>>>>>>>>>>> TimedBuffer::checkSize method that could force a flush if > the > > >> batch > > >>>>>>>>>> buffer > > >>>>>>>>>>>> if not big enough to receive new data, going IOPS. > > >>>>>>>>>>>> Sadly TimedBuffer::checkSize is outside any timeout, but > > >> depends on > > >>>>>> the > > >>>>>>>>>>>> writers. > > >>>>>>>>>>>> > > >>>>>>>>>>>> That's why I've implemented the "compensation" right after > > any > > >>>>>> flush, > > >>>>>>>>>> in > > >>>>>>>>>>>> order to work with forced flushes too: > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>> https://github.com/franz1981/activemq-artemis/blob/ > > >>>>>> 4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/ > > >>>>>> src/main/java/org/apache/activemq/artemis/core/io/ > > >>>>>> buffer/TimedBuffer.java#L338 > > >>>>>>>>>>>> > > >>>>>>>>>>>> Doing it on TimedBuffer::flush all the flushes on disk will > > be > > >>>>>>>>>> compensated > > >>>>>>>>>>>> (half of the story: ASYNCIO is async so depends on libAIO > > >>>>>> partially!) > > >>>>>>>>>>>> Regarding the IOPS computation I've built this, as you've > > >>>>>> suggested: > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>> https://github.com/franz1981/activemq-artemis/blob/ > > >>>>>> 4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/ > > >>>>>> src/main/java/org/apache/activemq/artemis/core/io/ > > >>>>>> buffer/TimedBuffer.java#L119 > > >>>>>>>>>>>> > > >>>>>>>>>>>> The performance seems pretty good, it compensates well but > it > > >> is > > >>>>>> faster > > >>>>>>>>>>>> than > > >>>>>>>>>>>> the original version, limiting IOPS too! > > >>>>>>>>>>>> > > >>>>>>>>>>>> Thanks, > > >>>>>>>>>>>> Franz > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> -- > > >>>>>>>>>>>> View this message in context: > > >>>>>>>>>>>> > > >>>>>>>>>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis- > > >>>>>> IOPS-Limiter-strategy-tp4725875p4726057.html > > >>>>>>>>>>>> Sent from the ActiveMQ - Dev mailing list archive at > > >> Nabble.com. > > >>>>>>>>>>>> > > >>>>>>>>>>> -- > > >>>>>>>>>>> Clebert Suconic > > >>>>>>>>>> > > >>>>>>>>> -- > > >>>>>>>>> Clebert Suconic > > >>>>>>>>> > > >>>>>>>> -- > > >>>>>>>> Clebert Suconic > > >>>>>> > > >>>>>> > > >>>>>> ------------------------------ > > >>>>>> If you reply to this email, your message will be added to the > > >> discussion > > >>>>>> below: > > >>>>>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis- > > >>>>>> IOPS-Limiter-strategy-tp4725875p4726091.html > > >>>>>> To unsubscribe from [DISCUSS] Artemis IOPS Limiter strategy, click > > >> here > > >>>>>> < > > >> > > >>>>>> . > > >>>>>> NAML > > >>>>>> <http://activemq.2283324.n4.nabble.com/template/ > > >> NamlServlet.jtp?macro=macro_viewer&id=instant_html% > > >> 21nabble%3Aemail.naml&base=nabble.naml.namespaces. > > >> BasicNamespace-nabble.view.web.template.NabbleNamespace- > > >> nabble.view.web.template.NodeNamespace&breadcrumbs= > > >> notify_subscribers%21nabble%3Aemail.naml-instant_emails% > > >> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > >>>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> View this message in context: http://activemq.2283324.n4. > > >> nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy- > tp4725875p4726093.html > > >>>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com. > > >>> > > >>> > > >>> > > >>> -- > > >>> Clebert Suconic > > >> > > >> > > >> ------------------------------ > > >> If you reply to this email, your message will be added to the > > discussion > > >> below: > > >> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis- > > >> IOPS-Limiter-strategy-tp4725875p4726097.html > > >> To unsubscribe from [DISCUSS] Artemis IOPS Limiter strategy, click > here > > >> < > > >> . > > >> NAML > > >> <http://activemq.2283324.n4.nabble.com/template/ > > NamlServlet.jtp?macro=macro_viewer&id=instant_html% > > 21nabble%3Aemail.naml&base=nabble.naml.namespaces. > > BasicNamespace-nabble.view.web.template.NabbleNamespace- > > nabble.view.web.template.NodeNamespace&breadcrumbs= > > notify_subscribers%21nabble%3Aemail.naml-instant_emails% > > 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > >> > > > > > > > > > > > > > > > -- > > > View this message in context: http://activemq.2283324.n4. > > nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726099.html > > > Sent from the ActiveMQ - Dev mailing list archive at Nabble.com. > > > > > > ------------------------------ > > If you reply to this email, your message will be added to the discussion > > below: > > http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis- > > IOPS-Limiter-strategy-tp4725875p4726101.html > > To unsubscribe from [DISCUSS] Artemis IOPS Limiter strategy, click here > > <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro= > unsubscribe_by_code&node=4725875&code=bmlncm8uZnJhQGdtYWlsLmNvbXw0Nz > I1ODc1fC0zNzI3NjE2NQ==> > > . > > NAML > > <http://activemq.2283324.n4.nabble.com/template/ > NamlServlet.jtp?macro=macro_viewer&id=instant_html% > 21nabble%3Aemail.naml&base=nabble.naml.namespaces. > BasicNamespace-nabble.view.web.template.NabbleNamespace- > nabble.view.web.template.NodeNamespace&breadcrumbs= > notify_subscribers%21nabble%3Aemail.naml-instant_emails% > 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > > > > > > -- > View this message in context: http://activemq.2283324.n4. > nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726102.html > Sent from the ActiveMQ - Dev mailing list archive at Nabble.com. >
