[GitHub] [activemq-artemis] wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers
wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are
a lot of subscribers
URL: https://github.com/apache/activemq-artemis/pull/2750#issuecomment-521938303
@michaelandrepearce squashed commits and pushed. BTW, I added some code to
avoid file leak in race conditions in PageSubscriptionImpl::internalGetNext() :
` serverMessage = ((PageReader) cache).getMessage(retPos, false,
true);
PageCache previousPageCache =
pageReaders.putIfAbsent(retPos.getPageNr(), (PageReader) cache);
if (previousPageCache != null && previousPageCache != cache) {
// Maybe other cursor iterators have added page reader, we
have to close this one to avoid file leak
cache.close();
}`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [activemq-artemis] wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers
wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers URL: https://github.com/apache/activemq-artemis/pull/2750#issuecomment-521921421 pushed commit to close page reader when cursor iterator is closed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [activemq-artemis] wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers
wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers URL: https://github.com/apache/activemq-artemis/pull/2750#issuecomment-521883667 @michaelandrepearce Can you wait a moment? When the browse cursor iterators(like the one created by QueueControl::browse, browse only consumer, etc) are closed, the page opened by them is not closed. I would push another commit to fix this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [activemq-artemis] wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers
wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers URL: https://github.com/apache/activemq-artemis/pull/2750#issuecomment-518556136 @franz1981 > And while catching up, the producers are not seriously decreasing their rate? This PR has helped in your tests? This PR helped to improve consumer throughput while entering into page mode with lots of subscribers. I didn't see the producers rate seriously decreased, although producers slowed down bcs consumers used extra io. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [activemq-artemis] wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers
wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers URL: https://github.com/apache/activemq-artemis/pull/2750#issuecomment-518139732 @franz1981 In our test consumers were falling behind producers all the time. I saw ~80GB page files left after test lasting 1 hour. I'm not sure whether it's related to this pr. Considering 10GB page data produced, it's possible the pages are constantly dropping in an out while consumers are cathing up, resulting in performance drop. Could you try this pr to check it? :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [activemq-artemis] wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers
wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers URL: https://github.com/apache/activemq-artemis/pull/2750#issuecomment-516807021 Pushed to change PageReader per queue. The perf result is below: 1. Running in 51MB size page and 1 page cache in the case of 100 multicast queues. 19000 msg/s received and 9500 msg/s sent vs. 16000 msg/s received and 11000 msg/s sent with previous PageReader cache. consumer tps:  producer tps:  2. Running in 5MB size page and 100 page cache in the case of 100 multicast queues. 16000msg/s received and 8200 msg/s sent vs. 15000 msg/s received and 8500 msg/s sent with previous PageReader cache. consumer tps:  producer tps:  3. Running in 51MB size page and 1 page cache in the case of 1 queue. 16000 msg/s received and 29000 msg/s sent vs. 13500 msg/s received and 3 msg/s sent with previous PageReader cache. consumer tps:  producer tps:  Generally speaking, the consumer tps increase a little and producer tps reduce less, with total tps increasing by about 1000. It's to be noted that there was frequent file opening/closing in second test scenario(5MB size page and 100 page cache). I added some logging and saw two subsequent deliver was scheduled a few seconds after depage. In the case that message of PagedReference was cleared, calling getPriority/checkExpired in deliver() would trigger reading message. Due to the small page file, reading messages were more likely to hit page before the current cursor. Given PageReader before current cursor was closed, this would cause constantly page file open/close(log showed approximately 6000 times per second in average). In first test scenario(51MB size page and 1 page cache), approximately 700 times per second in average of file open/closing occurred. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [activemq-artemis] wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers
wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers URL: https://github.com/apache/activemq-artemis/pull/2750#issuecomment-515812679 I've submitted a new pr #2769 to fix it. After reviewed and merged, I would rebase and push the new code with PageReader per queue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [activemq-artemis] wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers
wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers URL: https://github.com/apache/activemq-artemis/pull/2750#issuecomment-514875375 > I will need some time to review this last week. > > Can you rebase again after I merged your change? I'm working on PageReader per queue these days. Unluckily some problem was encountered. After investigation, I found it was related to #2740 which would sync files while closing. Given we put PageReader in PageSubscription meaning 100 times of file open/close for each page in our case(100 queues), although page file is only opened for reading, the sync operation does cause great performance loss - ~5000 msg/s received and 13000 msg/s sent now compared with 16000 msg/s received and 11000 msg/s sent of pr. Should I file a new jira and pr to fix the problem introuduced by #2740 or just fix it in this one? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [activemq-artemis] wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers
wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers URL: https://github.com/apache/activemq-artemis/pull/2750#issuecomment-513725688 Rebased and fixed to add deps and configuration This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [activemq-artemis] wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers
wy96f commented on issue #2750: ARTEMIS-2399 Improve performance when there are a lot of subscribers URL: https://github.com/apache/activemq-artemis/pull/2750#issuecomment-511398767 @michaelandrepearce I deployed the test without MAX_DEPAGE_NUM limit during depage. The performance result as follows: 1. Running in 51MB size page and 1 page cache in the case of 100 multicast queues. The consumer tps dropped to ~11000 compared to ~16000 in pr.   2. Running in 5MB size page and 100 page cache in the case of 100 multicast queues. The consumer tps dropped to ~1 compared to ~15000 in pr.   3. Running in 51MB size page and 1 page cache in the case of 1 queue. Performance results similar.   Seems like paged message removed and re reading file resulted in the performance drop particularly in the case of heavy usage of memory(test #1,#2). Even if there was enough free memory and messages were not gced(test #3), the performance didn't make a big difference - instead memory footprint might be increased. Considering we've totally relied on the kernel page cache, it seems not to need spare messages in memory. I personally assume this is an import feature. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
