[jira] [Commented] (ARTEMIS-2678) Incomplete records for pages under high load
[
https://issues.apache.org/jira/browse/ARTEMIS-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065774#comment-17065774
]
Francesco Nigro commented on ARTEMIS-2678:
--
{quote}1) The broker is running on all our stages with the same version. I
guess that using the latest upstream is nothing I would want to run on any
setup but a local one.{quote}
We have had some changes on OpenWire and paging so it's worth to check IMO
{quote}4) So what do you suggest for max-size-bytes and page-size-bytes? The
default is using 50% of max heap and 10Mb for page.{quote}
Depends on your requirements in term of number of addresses and your
global-max-size (that's by default is half of your max heap size).
> Incomplete records for pages under high load
>
>
> Key: ARTEMIS-2678
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2678
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Environment: Linux
>Reporter: Ansgar J. Sachs
>Priority: Major
> Attachments: Bildschirmfoto 2020-03-23 um 17.35.41.png
>
>
> {quote}As developer, I expect paging to be resource saving and resilient to
> high load{quote}
> h3. Current Situation
> During a performance test with a throughput of ~25.000 messages per second
> that run mulitple hours, some consumers were too slow to consume and messages
> piled up on the broker. For this reason, the broker started to page the
> messages of growing queues. When we reduced the load from the broker, some
> queues stopped consuming due to the following logs:
> {code}
> AMQ222033: Page file 7.page had incomplete records at position
> 39,795,399 at record number 13,952
> target message no.16146 not found from start offset 46032883 and start
> message number 16146: java.lang.RuntimeException: target message no.16146 not
> found from start offset 46032883 and start message number 16146
> {code}
> It wasnt possible to recover from this state but deleting the paging
> directory.
> h3. Expected Situation
> I would expect that the paging mechanism is resilient to any errors.
> h3. Scenario Setup
> Master configuration:
> {code:xml}
>
>
>
> true
>
>
>
>
>
> 256Mb
> 64Mb
>
> 10
> PAGE
>
> {code}
> An extract of the monitoring of the Performance Test is attached to this
> issue.
> h3. Workaround
> Right now we disabled paging at all and only use our Heap. However, the heap
> is exhausted at 5 million messages which is in our use case better than
> loosing any of them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARTEMIS-2678) Incomplete records for pages under high load
[
https://issues.apache.org/jira/browse/ARTEMIS-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065475#comment-17065475
]
Ansgar J. Sachs commented on ARTEMIS-2678:
--
1) The broker is running on all our stages with the same version. I guess that
using the latest upstream is nothing I would want to run on any setup but a
local one.
2) The OpenWire protocol is something caused by our old Jboss EAP which
prevents us from upgrading to a more up-to-date RA (like Artemis)
3) I will checkout the read-whole-page configuration and let you know about the
results
4) So what do you suggest for max-size-bytes and page-size-bytes? The default
is using 50% of max heap and 20Mb for page. However, at least this 50% boundary
is way to high if you have more than 1 queue or do I misunderstand this setting?
> Incomplete records for pages under high load
>
>
> Key: ARTEMIS-2678
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2678
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Environment: Linux
>Reporter: Ansgar J. Sachs
>Priority: Major
> Attachments: Bildschirmfoto 2020-03-23 um 17.35.41.png
>
>
> {quote}As developer, I expect paging to be resource saving and resilient to
> high load{quote}
> h3. Current Situation
> During a performance test with a throughput of ~25.000 messages per second
> that run mulitple hours, some consumers were too slow to consume and messages
> piled up on the broker. For this reason, the broker started to page the
> messages of growing queues. When we reduced the load from the broker, some
> queues stopped consuming due to the following logs:
> {code}
> AMQ222033: Page file 7.page had incomplete records at position
> 39,795,399 at record number 13,952
> target message no.16146 not found from start offset 46032883 and start
> message number 16146: java.lang.RuntimeException: target message no.16146 not
> found from start offset 46032883 and start message number 16146
> {code}
> It wasnt possible to recover from this state but deleting the paging
> directory.
> h3. Expected Situation
> I would expect that the paging mechanism is resilient to any errors.
> h3. Scenario Setup
> Master configuration:
> {code:xml}
>
>
>
> true
>
>
>
>
>
> 256Mb
> 64Mb
>
> 10
> PAGE
>
> {code}
> An extract of the monitoring of the Performance Test is attached to this
> issue.
> h3. Workaround
> Right now we disabled paging at all and only use our Heap. However, the heap
> is exhausted at 5 million messages which is in our use case better than
> loosing any of them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARTEMIS-2678) Incomplete records for pages under high load
[
https://issues.apache.org/jira/browse/ARTEMIS-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065440#comment-17065440
]
Francesco Nigro commented on ARTEMIS-2678:
--
I suggests several things to do:
* try with the latest master upstream
* try switch to core protocol (if you can do it)
* try configure read-whole-page true (default is false)
Other (separate) questions: why you've chosen 64Mb of page size and just 256Mb
of max size bytes?
> Incomplete records for pages under high load
>
>
> Key: ARTEMIS-2678
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2678
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Environment: Linux
>Reporter: Ansgar J. Sachs
>Priority: Major
> Attachments: Bildschirmfoto 2020-03-23 um 17.35.41.png
>
>
> {quote}As developer, I expect paging to be resource saving and resilient to
> high load{quote}
> h3. Current Situation
> During a performance test with a throughput of ~25.000 messages per second
> that run mulitple hours, some consumers were too slow to consume and messages
> piled up on the broker. For this reason, the broker started to page the
> messages of growing queues. When we reduced the load from the broker, some
> queues stopped consuming due to the following logs:
> {code}
> AMQ222033: Page file 7.page had incomplete records at position
> 39,795,399 at record number 13,952
> target message no.16146 not found from start offset 46032883 and start
> message number 16146: java.lang.RuntimeException: target message no.16146 not
> found from start offset 46032883 and start message number 16146
> {code}
> It wasnt possible to recover from this state but deleting the paging
> directory.
> h3. Expected Situation
> I would expect that the paging mechanism is resilient to any errors.
> h3. Scenario Setup
> Master configuration:
> {code:xml}
>
>
>
> true
>
>
>
>
>
> 256Mb
> 64Mb
>
> 10
> PAGE
>
> {code}
> An extract of the monitoring of the Performance Test is attached to this
> issue.
> h3. Workaround
> Right now we disabled paging at all and only use our Heap. However, the heap
> is exhausted at 5 million messages which is in our use case better than
> loosing any of them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARTEMIS-2678) Incomplete records for pages under high load
[
https://issues.apache.org/jira/browse/ARTEMIS-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064970#comment-17064970
]
Ansgar J. Sachs commented on ARTEMIS-2678:
--
- I am using version 2.11.0 of Artemis with OpenWire as protocol.
- We are using a shared-store after we had some troubles with Replication and
XA-Transactions.
- For our test setup we had two JVMs running on one node - so just separated by
process
> Incomplete records for pages under high load
>
>
> Key: ARTEMIS-2678
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2678
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Environment: Linux
>Reporter: Ansgar J. Sachs
>Priority: Major
> Attachments: Bildschirmfoto 2020-03-23 um 17.35.41.png
>
>
> {quote}As developer, I expect paging to be resource saving and resilient to
> high load{quote}
> h3. Current Situation
> During a performance test with a throughput of ~25.000 messages per second
> that run mulitple hours, some consumers were too slow to consume and messages
> piled up on the broker. For this reason, the broker started to page the
> messages of growing queues. When we reduced the load from the broker, some
> queues stopped consuming due to the following logs:
> {code}
> AMQ222033: Page file 7.page had incomplete records at position
> 39,795,399 at record number 13,952
> target message no.16146 not found from start offset 46032883 and start
> message number 16146: java.lang.RuntimeException: target message no.16146 not
> found from start offset 46032883 and start message number 16146
> {code}
> It wasnt possible to recover from this state but deleting the paging
> directory.
> h3. Expected Situation
> I would expect that the paging mechanism is resilient to any errors.
> h3. Scenario Setup
> Master configuration:
> {code:xml}
>
>
>
> true
>
>
>
>
>
> 256Mb
> 64Mb
>
> 10
> PAGE
>
> {code}
> An extract of the monitoring of the Performance Test is attached to this
> issue.
> h3. Workaround
> Right now we disabled paging at all and only use our Heap. However, the heap
> is exhausted at 5 million messages which is in our use case better than
> loosing any of them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARTEMIS-2678) Incomplete records for pages under high load
[
https://issues.apache.org/jira/browse/ARTEMIS-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064946#comment-17064946
]
Francesco Nigro commented on ARTEMIS-2678:
--
* Which version of artemis?
* I see you're using shared-store, are you sure is the right choice if high
throughput are needed?
* Which protocol of messages is being used? Core, AMQP, etc etc
> Incomplete records for pages under high load
>
>
> Key: ARTEMIS-2678
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2678
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Environment: Linux
>Reporter: Ansgar J. Sachs
>Priority: Major
> Attachments: Bildschirmfoto 2020-03-23 um 17.35.41.png
>
>
> {quote}As developer, I expect paging to be resource saving and resilient to
> high load{quote}
> h3. Current Situation
> During a performance test with a throughput of ~25.000 messages per second
> that run mulitple hours, some consumers were too slow to consume and messages
> piled up on the broker. For this reason, the broker started to page the
> messages of growing queues. When we reduced the load from the broker, some
> queues stopped consuming due to the following logs:
> {code}
> AMQ222033: Page file 7.page had incomplete records at position
> 39,795,399 at record number 13,952
> target message no.16146 not found from start offset 46032883 and start
> message number 16146: java.lang.RuntimeException: target message no.16146 not
> found from start offset 46032883 and start message number 16146
> {code}
> It wasnt possible to recover from this state but deleting the paging
> directory.
> h3. Expected Situation
> I would expect that the paging mechanism is resilient to any errors.
> h3. Scenario Setup
> Master configuration:
> {code:xml}
>
>
>
> true
>
>
>
>
>
> 256Mb
> 64Mb
>
> 10
> PAGE
>
> {code}
> An extract of the monitoring of the Performance Test is attached to this
> issue.
> h3. Workaround
> Right now we disabled paging at all and only use our Heap. However, the heap
> is exhausted at 5 million messages which is in our use case better than
> loosing any of them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
