[jira] [Commented] (CAMEL-12727) java.util.ConcurrentModificationException at org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) in 2.20.1
[ https://issues.apache.org/jira/browse/CAMEL-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593359#comment-16593359 ] Claus Ibsen commented on CAMEL-12727: - Yeah the hack would be needed as the locking is optimistick and therefore concurrent access on the copy method. > java.util.ConcurrentModificationException at > org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) > in 2.20.1 > --- > > Key: CAMEL-12727 > URL: https://issues.apache.org/jira/browse/CAMEL-12727 > Project: Camel > Issue Type: Bug > Components: camel-core >Affects Versions: 2.20.0 > Environment: Camel 2.20.1. > The exception occurs within a aggregation processor using a custom > aggregation strategy and optimistic locking. >Reporter: Philippe Bouteleux >Priority: Major > Fix For: 2.23.0 > > Attachments: dih-error.queue_2 (1).txt > > > A concurrent access exception occurs during our load tests while consuming > messages from a rabbitmq queue. > The call stack embedded below shows that the issue is related to an > "optimization" which was introduced in > https://issues.apache.org/jira/browse/CAMEL-11330 to replace a > ConcurrentHashMap with a regular one from version 2.19.5 to 2.20.0 onward. > The faulty code is still present in latest 2.22.0. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CAMEL-12727) java.util.ConcurrentModificationException at org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) in 2.20.1
[ https://issues.apache.org/jira/browse/CAMEL-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588948#comment-16588948 ] Philippe Bouteleux commented on CAMEL-12727: No definitive results yet from our full perf platform. However, from our CI smaller stress test, after having integrated our own aggregation repository bypassing the hack and therefore the copy, no more exception occurred for now. Which sounds good. Now, IMHO, re-introducing the ConcurrentHashMap is the proper and safer solution. However, adding the lock within the hack only has less impacts on performances. Q: is this hack really still useful ? > java.util.ConcurrentModificationException at > org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) > in 2.20.1 > --- > > Key: CAMEL-12727 > URL: https://issues.apache.org/jira/browse/CAMEL-12727 > Project: Camel > Issue Type: Bug > Components: camel-core >Affects Versions: 2.20.0 > Environment: Camel 2.20.1. > The exception occurs within a aggregation processor using a custom > aggregation strategy and optimistic locking. >Reporter: Philippe Bouteleux >Priority: Major > Fix For: 2.23.0 > > Attachments: dih-error.queue_2 (1).txt > > > A concurrent access exception occurs during our load tests while consuming > messages from a rabbitmq queue. > The call stack embedded below shows that the issue is related to an > "optimization" which was introduced in > https://issues.apache.org/jira/browse/CAMEL-11330 to replace a > ConcurrentHashMap with a regular one from version 2.19.5 to 2.20.0 onward. > The faulty code is still present in latest 2.22.0. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CAMEL-12727) java.util.ConcurrentModificationException at org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) in 2.20.1
[ https://issues.apache.org/jira/browse/CAMEL-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588894#comment-16588894 ] Claus Ibsen commented on CAMEL-12727: - Any feedback [~boutblock] > java.util.ConcurrentModificationException at > org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) > in 2.20.1 > --- > > Key: CAMEL-12727 > URL: https://issues.apache.org/jira/browse/CAMEL-12727 > Project: Camel > Issue Type: Bug > Components: camel-core >Affects Versions: 2.20.0 > Environment: Camel 2.20.1. > The exception occurs within a aggregation processor using a custom > aggregation strategy and optimistic locking. >Reporter: Philippe Bouteleux >Priority: Major > Fix For: 2.23.0 > > Attachments: dih-error.queue_2 (1).txt > > > A concurrent access exception occurs during our load tests while consuming > messages from a rabbitmq queue. > The call stack embedded below shows that the issue is related to an > "optimization" which was introduced in > https://issues.apache.org/jira/browse/CAMEL-11330 to replace a > ConcurrentHashMap with a regular one from version 2.19.5 to 2.20.0 onward. > The faulty code is still present in latest 2.22.0. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CAMEL-12727) java.util.ConcurrentModificationException at org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) in 2.20.1
[
https://issues.apache.org/jira/browse/CAMEL-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581023#comment-16581023
]
Claus Ibsen commented on CAMEL-12727:
-
We could synchronize that hack
{code}
// need to synchronize this to avoid concurrency issue
synchronized (this) {
oldExchange = originalExchange.copy();
}
{code}
We could also consider going back to ConcurrentHashMap. The implementation in
newer JVMs are much better now.
> java.util.ConcurrentModificationException at
> org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550)
> in 2.20.1
> ---
>
> Key: CAMEL-12727
> URL: https://issues.apache.org/jira/browse/CAMEL-12727
> Project: Camel
> Issue Type: Bug
> Components: camel-core
>Affects Versions: 2.20.0
> Environment: Camel 2.20.1.
> The exception occurs within a aggregation processor using a custom
> aggregation strategy and optimistic locking.
>Reporter: Philippe Bouteleux
>Priority: Major
> Fix For: 2.23.0
>
> Attachments: dih-error.queue_2 (1).txt
>
>
> A concurrent access exception occurs during our load tests while consuming
> messages from a rabbitmq queue.
> The call stack embedded below shows that the issue is related to an
> "optimization" which was introduced in
> https://issues.apache.org/jira/browse/CAMEL-11330 to replace a
> ConcurrentHashMap with a regular one from version 2.19.5 to 2.20.0 onward.
> The faulty code is still present in latest 2.22.0.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CAMEL-12727) java.util.ConcurrentModificationException at org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) in 2.20.1
[ https://issues.apache.org/jira/browse/CAMEL-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581003#comment-16581003 ] Claus Ibsen commented on CAMEL-12727: - Thanks, when you have feedback then provide it here for us > java.util.ConcurrentModificationException at > org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) > in 2.20.1 > --- > > Key: CAMEL-12727 > URL: https://issues.apache.org/jira/browse/CAMEL-12727 > Project: Camel > Issue Type: Bug > Components: camel-core >Affects Versions: 2.20.0 > Environment: Camel 2.20.1. > The exception occurs within a aggregation processor using a custom > aggregation strategy and optimistic locking. >Reporter: Philippe Bouteleux >Priority: Major > Attachments: dih-error.queue_2 (1).txt > > > A concurrent access exception occurs during our load tests while consuming > messages from a rabbitmq queue. > The call stack embedded below shows that the issue is related to an > "optimization" which was introduced in > https://issues.apache.org/jira/browse/CAMEL-11330 to replace a > ConcurrentHashMap with a regular one from version 2.19.5 to 2.20.0 onward. > The faulty code is still present in latest 2.22.0. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CAMEL-12727) java.util.ConcurrentModificationException at org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) in 2.20.1
[ https://issues.apache.org/jira/browse/CAMEL-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579559#comment-16579559 ] Philippe Bouteleux commented on CAMEL-12727: So. I added a custom 'NullAggregationRepository' to my route which is a copy of the original MemoryAggregationRepository to avoid the internal copy of the exchange. I will unfortunately only have a result when our perf platform is ready to upgrade. I'll give back more information ASAP. Note that I could avoid the in-memory cache of the exchanges without breaking the internal logic of the aggregator. > java.util.ConcurrentModificationException at > org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) > in 2.20.1 > --- > > Key: CAMEL-12727 > URL: https://issues.apache.org/jira/browse/CAMEL-12727 > Project: Camel > Issue Type: Bug > Components: camel-core >Affects Versions: 2.20.0 > Environment: Camel 2.20.1. > The exception occurs within a aggregation processor using a custom > aggregation strategy and optimistic locking. >Reporter: Philippe Bouteleux >Priority: Major > Attachments: dih-error.queue_2 (1).txt > > > A concurrent access exception occurs during our load tests while consuming > messages from a rabbitmq queue. > The call stack embedded below shows that the issue is related to an > "optimization" which was introduced in > https://issues.apache.org/jira/browse/CAMEL-11330 to replace a > ConcurrentHashMap with a regular one from version 2.19.5 to 2.20.0 onward. > The faulty code is still present in latest 2.22.0. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CAMEL-12727) java.util.ConcurrentModificationException at org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) in 2.20.1
[
https://issues.apache.org/jira/browse/CAMEL-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579480#comment-16579480
]
Philippe Bouteleux commented on CAMEL-12727:
You are certainly right as the lines of code where the exception occurs depends
on tests on optimisticLocking and aggregationRepository being in-memory
(AggregateProcessor.java line 398 in 2.20.1):
// hack to support legacy AggregationStrategy's that modify and return the
oldExchange, these will not
// working when using an identify based approach for optimistic locking like
the MemoryAggregationRepository.
if (optimisticLocking && aggregationRepository instanceof
MemoryAggregationRepository) {
oldExchange = originalExchange.copy();
}
Please note that we did not choose to follow the repository pattern to save
each individual message to disk but we chose to directly save them in the
aggregation strategy class we provided.
Actually, the default MemoryAggregationRepository is of no use in our use case
and shall may be replaced by our own 'null repository' to not put pressure on
the garbage collector for nothing. This may also workaround the issue as the
conditional test would be false and the exchange would not be copied and would
not crash.
> java.util.ConcurrentModificationException at
> org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550)
> in 2.20.1
> ---
>
> Key: CAMEL-12727
> URL: https://issues.apache.org/jira/browse/CAMEL-12727
> Project: Camel
> Issue Type: Bug
> Components: camel-core
>Affects Versions: 2.20.0
> Environment: Camel 2.20.1.
> The exception occurs within a aggregation processor using a custom
> aggregation strategy and optimistic locking.
>Reporter: Philippe Bouteleux
>Priority: Major
> Attachments: dih-error.queue_2 (1).txt
>
>
> A concurrent access exception occurs during our load tests while consuming
> messages from a rabbitmq queue.
> The call stack embedded below shows that the issue is related to an
> "optimization" which was introduced in
> https://issues.apache.org/jira/browse/CAMEL-11330 to replace a
> ConcurrentHashMap with a regular one from version 2.19.5 to 2.20.0 onward.
> The faulty code is still present in latest 2.22.0.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CAMEL-12727) java.util.ConcurrentModificationException at org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) in 2.20.1
[
https://issues.apache.org/jira/browse/CAMEL-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579457#comment-16579457
]
Philippe Bouteleux commented on CAMEL-12727:
Hi,
I'm sorry this cannot be easily reproduced with a simple unit test as it only
occurs under heavy workload on our perf platform of test.
The route looks like this:
from(auditQueueProperties.getAuditQueueUri()) // rabbitmq queue
.routeId(ROUTE_NAME)
.autoStartup(true)
.split().method(batchUnfoldPlugin) // extract individual xml audit messages
from the input batch
.aggregate(simple("${header." + AUDIT_TYPE + "}"), aggregationStrategy) //
aggregate the audit messages by appending them directly into files on disk,
spread by type
.optimisticLocking() // as exclusive pessimistic locking is directly managed
by the file system for each individual file instead of a less efficient global
lock.
.completionSize(stagingAreaProperties.getCompletionSize())
.completionInterval(stagingAreaProperties.getCompletionInterval())
// copy each completed file to another location
.process(ex ->
stagingArea.push(ex.getIn().getHeader(TRANSACTION_TYPE).toString(),
ex.getIn().getHeader(AUDIT_DATA_PATH).toString(), ex.getIn().getBody()))
.end();
The custom aggregation strategy implements CompletionAwareAggregationStrategy
and TimeoutAwareAggregationStrategy to save each incoming message by appending
it to the correct file by type, date etc... protected by a FileLock and retries
on OverlappingFileLockException.
We are currently in the process to downgrade from our current version 2.20.1 to
version 2.19.5 to validate our assumption on the issue but we are facing other
incompatibility problems doing so that needs to be fixed/circumvented.
> java.util.ConcurrentModificationException at
> org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550)
> in 2.20.1
> ---
>
> Key: CAMEL-12727
> URL: https://issues.apache.org/jira/browse/CAMEL-12727
> Project: Camel
> Issue Type: Bug
> Components: camel-core
>Affects Versions: 2.20.0
> Environment: Camel 2.20.1.
> The exception occurs within a aggregation processor using a custom
> aggregation strategy and optimistic locking.
>Reporter: Philippe Bouteleux
>Priority: Major
> Attachments: dih-error.queue_2 (1).txt
>
>
> A concurrent access exception occurs during our load tests while consuming
> messages from a rabbitmq queue.
> The call stack embedded below shows that the issue is related to an
> "optimization" which was introduced in
> https://issues.apache.org/jira/browse/CAMEL-11330 to replace a
> ConcurrentHashMap with a regular one from version 2.19.5 to 2.20.0 onward.
> The faulty code is still present in latest 2.22.0.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CAMEL-12727) java.util.ConcurrentModificationException at org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) in 2.20.1
[ https://issues.apache.org/jira/browse/CAMEL-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579451#comment-16579451 ] Claus Ibsen commented on CAMEL-12727: - Optimistick locking implies concurrent access and is not always safe to be used. We may need to do a synchronization point in the aggregator at a certain point > java.util.ConcurrentModificationException at > org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) > in 2.20.1 > --- > > Key: CAMEL-12727 > URL: https://issues.apache.org/jira/browse/CAMEL-12727 > Project: Camel > Issue Type: Bug > Components: camel-core >Affects Versions: 2.20.0 > Environment: Camel 2.20.1. > The exception occurs within a aggregation processor using a custom > aggregation strategy and optimistic locking. >Reporter: Philippe Bouteleux >Priority: Major > Attachments: dih-error.queue_2 (1).txt > > > A concurrent access exception occurs during our load tests while consuming > messages from a rabbitmq queue. > The call stack embedded below shows that the issue is related to an > "optimization" which was introduced in > https://issues.apache.org/jira/browse/CAMEL-11330 to replace a > ConcurrentHashMap with a regular one from version 2.19.5 to 2.20.0 onward. > The faulty code is still present in latest 2.22.0. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CAMEL-12727) java.util.ConcurrentModificationException at org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) in 2.20.1
[ https://issues.apache.org/jira/browse/CAMEL-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579417#comment-16579417 ] Dmitry Volodin commented on CAMEL-12727: [~boutblock], could you please provide a reproducer as simple as possible. > java.util.ConcurrentModificationException at > org.apache.camel.impl.DefaultExchange.createProperties(DefaultExchange.java:550) > in 2.20.1 > --- > > Key: CAMEL-12727 > URL: https://issues.apache.org/jira/browse/CAMEL-12727 > Project: Camel > Issue Type: Bug > Components: camel-core >Affects Versions: 2.20.0, 2.22.0 > Environment: Camel 2.20.1. > The exception occurs within a aggregation processor using a custom > aggregation strategy and optimistic locking. >Reporter: Philippe Bouteleux >Priority: Critical > Fix For: 2.19.5 > > Attachments: dih-error.queue_2 (1).txt > > > A concurrent access exception occurs during our load tests while consuming > messages from a rabbitmq queue. > The call stack embedded below shows that the issue is related to an > "optimization" which was introduced in > https://issues.apache.org/jira/browse/CAMEL-11330 to replace a > ConcurrentHashMap with a regular one from version 2.19.5 to 2.20.0 onward. > The faulty code is still present in latest 2.22.0. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
