Re: Camel AWS-S3 prefixes do not accept wildcards
There is no method for sync from CLI in SDK. I'll add a note in the prefix description about wildcards and regexes missing support from SDK. Thanks for the suggestion! Il giorno mer 13 mar 2024 alle ore 21:13 Andrea Cosentino ha scritto: > I'll investigate if there is a way of using sync somewhat through the sdk. > I'll keep you posted > > Il mer 13 mar 2024, 20:47 Mitch Trachtenberg ha > scritto: > >> My mistake. What I'd thought was just aws cli ls was actually output >> piped >> through a grep. >> >> Again, though, for many people used to non-AWS file listings this may come >> as a surprise, so it would IMO be helpful to tell people up front that >> wildcards, regexes, etc... won't work. And thanks for letting me know >> that >> the asterisk is a legal character in an AWS S3 object name. >> >> My situation is I expect only objects whose names match a particular >> pattern, but I want to prevent the download of any surprise objects that >> might get deposited. It looks like list, eliminate, and download the ones >> you want is going to have to be the approach to accomplish that. >> >> On Wed, Mar 13, 2024 at 12:13 PM Andrea Cosentino >> wrote: >> >> > Hi, >> > In the CLI it was escalated but it doesn't seem to be supported: >> > https://github.com/aws/aws-cli/issues/3784 >> > >> > Usually the CLI/SDK/Rest are aligned, so if it was possible natively >> > through CLI it would have been possible through SDK. >> > >> > The prefix is only a String in v1 and v2 and most importantly * is valid >> > character in the key name for an S3 object. >> > >> > My suggestion is to list the objects by prefix and then check through >> > regular expression if the key name responds to your regex or not. >> > >> > As far as I know and see in the documentation, list objects through >> > wildcards is not supported with AWS S3 ls, it works with AWS S3 sync but >> > sync is a different command than list objects and I guess is a different >> > Java library too. >> > >> > Cheers >> > >> > Il giorno mer 13 mar 2024 alle ore 19:37 Mitch Trachtenberg < >> > mjt...@gmail.com> ha scritto: >> > >> > > Thanks, Claus. I'm sure you're correct and I've looked at the >> > > documentation for ListObjectsRequest. >> > > >> > > >> > >> https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/ListObjectsRequest.html >> > > >> > > But it seems like something that is such a common need that it would >> be >> > > helpful IMO to include a note to that effect in the docs. >> > > >> > > On Wed, Mar 13, 2024 at 11:30 AM Claus Ibsen >> > > wrote: >> > > >> > > > Hi >> > > > >> > > > Camel uses the AWS Java SDK and it set the prefix on this SDK so its >> > > > depends on if this SDK has any kind of support for wildcards. >> > > > As you write its likely it does not have that. >> > > > >> > > > You can try to dive into the AWS SDK and see more deeper. >> > > > >> > > > On Wed, Mar 13, 2024 at 7:19 PM Mitch Trachtenberg < >> mjt...@gmail.com> >> > > > wrote: >> > > > >> > > > > I cannot specifically find this stated in either AWS S3 or Camel >> > > > > documentation, but it appears that I can give a prefix 1234- to >> > > restrict >> > > > > Camel's (4.3.0) downloads to files starting 1234-, but cannot >> give a >> > > > prefix >> > > > > - to restrict Camel's downloads to files beginning with four >> > > > characters >> > > > > and a hyphen. >> > > > > >> > > > > I believe AWS CLI does provide a way to do this. If I'm correct, >> it >> > > > would >> > > > > be helpful if this could be added to the documentation. >> > > > > >> > > > > If someone were to want to instruct me in where/how to do that, >> I'd >> > be >> > > > > happy to generate a pull request, but it might be easier to just >> add: >> > > > > "Note: due to AWS S3 limitations, the prefix parameter does not >> work >> > > with >> > > > > wildcards." >> > > > > >> > > > > It would be even more helpful, of course, if the component were >> > > modified >> > > > in >> > > > > some way to allow for wildcards. >> > > > > >> > > > >> > > > >> > > > -- >> > > > Claus Ibsen >> > > > - >> > > > @davsclaus >> > > > Camel in Action 2: https://www.manning.com/ibsen2 >> > > > >> > > >> > >> >
Re: Camel AWS-S3 prefixes do not accept wildcards
I'll investigate if there is a way of using sync somewhat through the sdk. I'll keep you posted Il mer 13 mar 2024, 20:47 Mitch Trachtenberg ha scritto: > My mistake. What I'd thought was just aws cli ls was actually output piped > through a grep. > > Again, though, for many people used to non-AWS file listings this may come > as a surprise, so it would IMO be helpful to tell people up front that > wildcards, regexes, etc... won't work. And thanks for letting me know that > the asterisk is a legal character in an AWS S3 object name. > > My situation is I expect only objects whose names match a particular > pattern, but I want to prevent the download of any surprise objects that > might get deposited. It looks like list, eliminate, and download the ones > you want is going to have to be the approach to accomplish that. > > On Wed, Mar 13, 2024 at 12:13 PM Andrea Cosentino > wrote: > > > Hi, > > In the CLI it was escalated but it doesn't seem to be supported: > > https://github.com/aws/aws-cli/issues/3784 > > > > Usually the CLI/SDK/Rest are aligned, so if it was possible natively > > through CLI it would have been possible through SDK. > > > > The prefix is only a String in v1 and v2 and most importantly * is valid > > character in the key name for an S3 object. > > > > My suggestion is to list the objects by prefix and then check through > > regular expression if the key name responds to your regex or not. > > > > As far as I know and see in the documentation, list objects through > > wildcards is not supported with AWS S3 ls, it works with AWS S3 sync but > > sync is a different command than list objects and I guess is a different > > Java library too. > > > > Cheers > > > > Il giorno mer 13 mar 2024 alle ore 19:37 Mitch Trachtenberg < > > mjt...@gmail.com> ha scritto: > > > > > Thanks, Claus. I'm sure you're correct and I've looked at the > > > documentation for ListObjectsRequest. > > > > > > > > > https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/ListObjectsRequest.html > > > > > > But it seems like something that is such a common need that it would be > > > helpful IMO to include a note to that effect in the docs. > > > > > > On Wed, Mar 13, 2024 at 11:30 AM Claus Ibsen > > > wrote: > > > > > > > Hi > > > > > > > > Camel uses the AWS Java SDK and it set the prefix on this SDK so its > > > > depends on if this SDK has any kind of support for wildcards. > > > > As you write its likely it does not have that. > > > > > > > > You can try to dive into the AWS SDK and see more deeper. > > > > > > > > On Wed, Mar 13, 2024 at 7:19 PM Mitch Trachtenberg > > > > > wrote: > > > > > > > > > I cannot specifically find this stated in either AWS S3 or Camel > > > > > documentation, but it appears that I can give a prefix 1234- to > > > restrict > > > > > Camel's (4.3.0) downloads to files starting 1234-, but cannot give > a > > > > prefix > > > > > - to restrict Camel's downloads to files beginning with four > > > > characters > > > > > and a hyphen. > > > > > > > > > > I believe AWS CLI does provide a way to do this. If I'm correct, > it > > > > would > > > > > be helpful if this could be added to the documentation. > > > > > > > > > > If someone were to want to instruct me in where/how to do that, I'd > > be > > > > > happy to generate a pull request, but it might be easier to just > add: > > > > > "Note: due to AWS S3 limitations, the prefix parameter does not > work > > > with > > > > > wildcards." > > > > > > > > > > It would be even more helpful, of course, if the component were > > > modified > > > > in > > > > > some way to allow for wildcards. > > > > > > > > > > > > > > > > > -- > > > > Claus Ibsen > > > > - > > > > @davsclaus > > > > Camel in Action 2: https://www.manning.com/ibsen2 > > > > > > > > > >
Re: Camel AWS-S3 prefixes do not accept wildcards
My mistake. What I'd thought was just aws cli ls was actually output piped through a grep. Again, though, for many people used to non-AWS file listings this may come as a surprise, so it would IMO be helpful to tell people up front that wildcards, regexes, etc... won't work. And thanks for letting me know that the asterisk is a legal character in an AWS S3 object name. My situation is I expect only objects whose names match a particular pattern, but I want to prevent the download of any surprise objects that might get deposited. It looks like list, eliminate, and download the ones you want is going to have to be the approach to accomplish that. On Wed, Mar 13, 2024 at 12:13 PM Andrea Cosentino wrote: > Hi, > In the CLI it was escalated but it doesn't seem to be supported: > https://github.com/aws/aws-cli/issues/3784 > > Usually the CLI/SDK/Rest are aligned, so if it was possible natively > through CLI it would have been possible through SDK. > > The prefix is only a String in v1 and v2 and most importantly * is valid > character in the key name for an S3 object. > > My suggestion is to list the objects by prefix and then check through > regular expression if the key name responds to your regex or not. > > As far as I know and see in the documentation, list objects through > wildcards is not supported with AWS S3 ls, it works with AWS S3 sync but > sync is a different command than list objects and I guess is a different > Java library too. > > Cheers > > Il giorno mer 13 mar 2024 alle ore 19:37 Mitch Trachtenberg < > mjt...@gmail.com> ha scritto: > > > Thanks, Claus. I'm sure you're correct and I've looked at the > > documentation for ListObjectsRequest. > > > > > https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/ListObjectsRequest.html > > > > But it seems like something that is such a common need that it would be > > helpful IMO to include a note to that effect in the docs. > > > > On Wed, Mar 13, 2024 at 11:30 AM Claus Ibsen > > wrote: > > > > > Hi > > > > > > Camel uses the AWS Java SDK and it set the prefix on this SDK so its > > > depends on if this SDK has any kind of support for wildcards. > > > As you write its likely it does not have that. > > > > > > You can try to dive into the AWS SDK and see more deeper. > > > > > > On Wed, Mar 13, 2024 at 7:19 PM Mitch Trachtenberg > > > wrote: > > > > > > > I cannot specifically find this stated in either AWS S3 or Camel > > > > documentation, but it appears that I can give a prefix 1234- to > > restrict > > > > Camel's (4.3.0) downloads to files starting 1234-, but cannot give a > > > prefix > > > > - to restrict Camel's downloads to files beginning with four > > > characters > > > > and a hyphen. > > > > > > > > I believe AWS CLI does provide a way to do this. If I'm correct, it > > > would > > > > be helpful if this could be added to the documentation. > > > > > > > > If someone were to want to instruct me in where/how to do that, I'd > be > > > > happy to generate a pull request, but it might be easier to just add: > > > > "Note: due to AWS S3 limitations, the prefix parameter does not work > > with > > > > wildcards." > > > > > > > > It would be even more helpful, of course, if the component were > > modified > > > in > > > > some way to allow for wildcards. > > > > > > > > > > > > > -- > > > Claus Ibsen > > > - > > > @davsclaus > > > Camel in Action 2: https://www.manning.com/ibsen2 > > > > > >
Re: Camel AWS-S3 prefixes do not accept wildcards
Hi, In the CLI it was escalated but it doesn't seem to be supported: https://github.com/aws/aws-cli/issues/3784 Usually the CLI/SDK/Rest are aligned, so if it was possible natively through CLI it would have been possible through SDK. The prefix is only a String in v1 and v2 and most importantly * is valid character in the key name for an S3 object. My suggestion is to list the objects by prefix and then check through regular expression if the key name responds to your regex or not. As far as I know and see in the documentation, list objects through wildcards is not supported with AWS S3 ls, it works with AWS S3 sync but sync is a different command than list objects and I guess is a different Java library too. Cheers Il giorno mer 13 mar 2024 alle ore 19:37 Mitch Trachtenberg < mjt...@gmail.com> ha scritto: > Thanks, Claus. I'm sure you're correct and I've looked at the > documentation for ListObjectsRequest. > > https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/ListObjectsRequest.html > > But it seems like something that is such a common need that it would be > helpful IMO to include a note to that effect in the docs. > > On Wed, Mar 13, 2024 at 11:30 AM Claus Ibsen > wrote: > > > Hi > > > > Camel uses the AWS Java SDK and it set the prefix on this SDK so its > > depends on if this SDK has any kind of support for wildcards. > > As you write its likely it does not have that. > > > > You can try to dive into the AWS SDK and see more deeper. > > > > On Wed, Mar 13, 2024 at 7:19 PM Mitch Trachtenberg > > wrote: > > > > > I cannot specifically find this stated in either AWS S3 or Camel > > > documentation, but it appears that I can give a prefix 1234- to > restrict > > > Camel's (4.3.0) downloads to files starting 1234-, but cannot give a > > prefix > > > - to restrict Camel's downloads to files beginning with four > > characters > > > and a hyphen. > > > > > > I believe AWS CLI does provide a way to do this. If I'm correct, it > > would > > > be helpful if this could be added to the documentation. > > > > > > If someone were to want to instruct me in where/how to do that, I'd be > > > happy to generate a pull request, but it might be easier to just add: > > > "Note: due to AWS S3 limitations, the prefix parameter does not work > with > > > wildcards." > > > > > > It would be even more helpful, of course, if the component were > modified > > in > > > some way to allow for wildcards. > > > > > > > > > -- > > Claus Ibsen > > - > > @davsclaus > > Camel in Action 2: https://www.manning.com/ibsen2 > > >
Re: Camel AWS-S3 prefixes do not accept wildcards
Thanks, Claus. I'm sure you're correct and I've looked at the documentation for ListObjectsRequest. https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/ListObjectsRequest.html But it seems like something that is such a common need that it would be helpful IMO to include a note to that effect in the docs. On Wed, Mar 13, 2024 at 11:30 AM Claus Ibsen wrote: > Hi > > Camel uses the AWS Java SDK and it set the prefix on this SDK so its > depends on if this SDK has any kind of support for wildcards. > As you write its likely it does not have that. > > You can try to dive into the AWS SDK and see more deeper. > > On Wed, Mar 13, 2024 at 7:19 PM Mitch Trachtenberg > wrote: > > > I cannot specifically find this stated in either AWS S3 or Camel > > documentation, but it appears that I can give a prefix 1234- to restrict > > Camel's (4.3.0) downloads to files starting 1234-, but cannot give a > prefix > > - to restrict Camel's downloads to files beginning with four > characters > > and a hyphen. > > > > I believe AWS CLI does provide a way to do this. If I'm correct, it > would > > be helpful if this could be added to the documentation. > > > > If someone were to want to instruct me in where/how to do that, I'd be > > happy to generate a pull request, but it might be easier to just add: > > "Note: due to AWS S3 limitations, the prefix parameter does not work with > > wildcards." > > > > It would be even more helpful, of course, if the component were modified > in > > some way to allow for wildcards. > > > > > -- > Claus Ibsen > - > @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 >
Re: Camel AWS-S3 prefixes do not accept wildcards
Hi Camel uses the AWS Java SDK and it set the prefix on this SDK so its depends on if this SDK has any kind of support for wildcards. As you write its likely it does not have that. You can try to dive into the AWS SDK and see more deeper. On Wed, Mar 13, 2024 at 7:19 PM Mitch Trachtenberg wrote: > I cannot specifically find this stated in either AWS S3 or Camel > documentation, but it appears that I can give a prefix 1234- to restrict > Camel's (4.3.0) downloads to files starting 1234-, but cannot give a prefix > - to restrict Camel's downloads to files beginning with four characters > and a hyphen. > > I believe AWS CLI does provide a way to do this. If I'm correct, it would > be helpful if this could be added to the documentation. > > If someone were to want to instruct me in where/how to do that, I'd be > happy to generate a pull request, but it might be easier to just add: > "Note: due to AWS S3 limitations, the prefix parameter does not work with > wildcards." > > It would be even more helpful, of course, if the component were modified in > some way to allow for wildcards. > -- Claus Ibsen - @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
Camel AWS-S3 prefixes do not accept wildcards
I cannot specifically find this stated in either AWS S3 or Camel documentation, but it appears that I can give a prefix 1234- to restrict Camel's (4.3.0) downloads to files starting 1234-, but cannot give a prefix - to restrict Camel's downloads to files beginning with four characters and a hyphen. I believe AWS CLI does provide a way to do this. If I'm correct, it would be helpful if this could be added to the documentation. If someone were to want to instruct me in where/how to do that, I'd be happy to generate a pull request, but it might be easier to just add: "Note: due to AWS S3 limitations, the prefix parameter does not work with wildcards." It would be even more helpful, of course, if the component were modified in some way to allow for wildcards.
Re: Error executing reactive work due to Java heap space. This exception is ignored.
Yeah that makes sense. Just wanted to make sure.
So we will have to implement a health monitoring on route level!
Thanks for the clarification!
Regards Christian
Am 13.03.24, 12:50 schrieb "Claus Ibsen" :
Hi
Okay as its a very serious error then its not caught by Camel.
From the javadoc you can read
An Error is a subclass of Throwable that indicates serious problems that a
reasonable application should not try to catch. Most such errors are
abnormal conditions. The ThreadDeath error, though a "normal" condition, is
also a subclass of Error because most applications should not try to catch
it.
On Wed, Mar 13, 2024 at 11:10 AM MÜLLER Christian (ICS460-BAI)
wrote:
> Hi,
>
> thanks for the repsonse!
>
> Camel 3.22.0
> the thrown exception class is:
> java.lang.OutOfMemoryError: Java heap space
>
> Regards Christian
>
> Am 13.03.24, 10:16 schrieb "Claus Ibsen" :
>
> What Camel version, and can you tell the exact exception class that was
> throw
>
> On Tue, Mar 12, 2024 at 2:17 PM MÜLLER Christian (ICS460-BAI)
> wrote:
>
> Hello,
>
> we are currently trying to understand the behaviour of camel if a thread
> “crashes” due to heap space.
>
> We observe the following error message in our logs:
> Error executing reactive work due to Java heap space. This exception is
> ignored.
> Camel (camel-1) thread #11 -
> sql://select%20*%20from%20LVS_REP.EXC_XMLFROMCOS%20where%20STATUS='25'%20and%20EXC_MSGTYPE%20like%20'LOAD00%25'%20and%20EXC_XML100%20in%20('AT0100','AT0730')
> After this error the route goes silent and no further “selects” are
> executed.
> We understand that this is intentional in order to protect other routes…
> but for us this behaviour is a serious problem as we are unable to alarm
> that there is an issue with the route.
>
> Is there any way to force camel not to ignore this exception and “rethrow”
> it?
>
> Thanks in Advance!
> Regards Christian
>
>
> [cid:ii_18e3718573ea4b0cb8a1]
>
> SPAR Business Services GmbH
> Information & Communication Services
> Europastrasse 3, 5015 Salzburg, Austria
>
> [cid:ii_18e3718573e81c1b3703]
>
> Sollten Sie diese E-Mail unbeabsichtigt bzw. irrtümlich erhalten haben, so
> weisen wir Sie darauf hin, dass gemäß § 161 Abs 4 TKG 2021 der Inhalt sowie
> die Tatsache des Empfangs dieser E-Mail weder aufgezeichnet noch verwertet
> oder Unbefugten mitgeteilt werden dürfen. Wir ersuchen Sie, die Nachricht
> von Ihrem System zu löschen und sich mit uns in Verbindung zu setzen.
> Unter diesem Link finden Sie die Informationen über die Verarbeitung Ihrer
> personenbezogenen Daten sowie zu Ihren Rechten als betroffene Person:
> https://www.spar.at/service/datenschutzhinweise
>
> If you have received this email accidentally or in error, we point out
> that, in accordance with § 161 para. 4 TKG 2021 (Telecommunications Act),
> the contents of this email and the fact of its receipt must not be
> recorded, exploited or communicated to unauthorized persons. We ask you to
> delete the message from your system and to contact us.
> Under this link you will find information about the processing of your
> personal data and about your rights as a data subject
> https://www.spar.at/service/datenschutzhinweise
>
>
>
>
> --
> Claus Ibsen
> -
> @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
--
Claus Ibsen
-
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Camel-Kafka parameters sanitization
And what about my question?
Riccardo
From: Modanese, Riccardo
Date: Monday, 11 March 2024 at 14:14
To: users@camel.apache.org
Subject: Re: Camel-Kafka parameters sanitization
Thanks!
From: Claus Ibsen
Date: Monday, 11 March 2024 at 14:12
To: users@camel.apache.org
Subject: Re: Camel-Kafka parameters sanitization
Hi
I created a ticket
https://issues.apache.org/jira/browse/CAMEL-20549
On Mon, Mar 11, 2024 at 1:20 PM Claus Ibsen wrote:
> Hi
>
>
> Thanks for reporting. You are welcome to create a JIRA so we can fix this
> little bug.
> We need to make Camel pass in that value as upper case so it can match the
> PEM equals
>
>
>
> On Mon, Mar 11, 2024 at 12:13 PM Modanese, Riccardo
> wrote:
>
>> Hi guys,
>>
>> I have a question about Kafka routes and security.
>> I’m worried about injection (like what can be done in sql or http I mean)
>> through “url options” parameters value (especially for those enclosed
>> between RAW() such as truststore/keystore password or jaas config).
>>
>> Is my concern valid or is completely without any fundament or there is
>> anyway a check and sanitization done by Camel (I saw while debugging that
>> Camel-Kafka endpoint parses the parameters to create an option object to
>> pass down to the Kafka-client so I’m wondering if this parsing will
>> sanitize values also).
>>
>> Thanks!
>>
>> P.S. I think I discovered a possible issue while trying to set truststore
>> in PEM format.
>> I’m using Camel 3.21 (Kafka-client is 3.4.1).
>> I generated keys and certificates in PEM format and they worked fine if
>> used to setup a Kafka-client (same version imported by Camel-Kafka endpoint)
>> Since wasn’t working from the route I debugged the route startup and I
>> discovered that, even if the truststoreType was set as PEM (uppercase) the
>> value that come to the lower level (the Kakfa-client ssl initialization I
>> mean) was lowercase (pem) and then failed to match this check
>> (DefaultSslEngineFactory):
>>
>>
>> private static SecurityStore createTruststore(String type, String path,
>> Password password, Password trustStoreCerts) {
>>
>> if (trustStoreCerts != null) {
>>
>> if (!PEM_TYPE.equals(type))
>>
>> throw new InvalidConfigurationException("SSL trust store
>> certs can be specified only for PEM, but trust store type is " + type +
>> ".");
>>
>> else if (path != null)
>>
>> throw new InvalidConfigurationException("Both SSL trust
>> store location and separate trust certificates are specified.");
>>
>> else if (password != null)
>>
>> throw new InvalidConfigurationException("SSL trust store
>> password cannot be specified for PEM format.");
>>
>> else
>>
>> return new PemStore(trustStoreCerts);
>>
>> } else if (PEM_TYPE.equals(type) && path != null) {
>>
>> if (password != null)
>>
>> throw new InvalidConfigurationException("SSL trust store
>> password cannot be specified for PEM format.");
>>
>> else
>>
>> return new FileBasedPemStore(path, null, false);
>>
>> } else if (path == null && password != null) {
>>
>> throw new InvalidConfigurationException("SSL trust store is
>> not specified, but trust store password is specified.");
>>
>> } else if (path != null) {
>>
>> return new FileBasedStore(type, path, password, null, false);
>>
>> } else
>>
>> return null;
>>
>> }
>>
>>
>> As result the truststore was not set and my Kafka-client wasn’t able to
>> connect to the Kafka server.
>> I tried to change the at debug time, on the fly while entering this
>> method, the type from “pem” to “PEM” and it worked fine.
>>
>> Am doing something wrong?
>>
>>
>> Regards,
>> Riccardo Modanese
>>
>
>
> --
> Claus Ibsen
> -
> @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
--
Claus Ibsen
-
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Error executing reactive work due to Java heap space. This exception is ignored.
Hi
Okay as its a very serious error then its not caught by Camel.
>From the javadoc you can read
An Error is a subclass of Throwable that indicates serious problems that a
reasonable application should not try to catch. Most such errors are
abnormal conditions. The ThreadDeath error, though a "normal" condition, is
also a subclass of Error because most applications should not try to catch
it.
On Wed, Mar 13, 2024 at 11:10 AM MÜLLER Christian (ICS460-BAI)
wrote:
> Hi,
>
> thanks for the repsonse!
>
> Camel 3.22.0
> the thrown exception class is:
> java.lang.OutOfMemoryError: Java heap space
>
> Regards Christian
>
> Am 13.03.24, 10:16 schrieb "Claus Ibsen" :
>
> What Camel version, and can you tell the exact exception class that was
> throw
>
> On Tue, Mar 12, 2024 at 2:17 PM MÜLLER Christian (ICS460-BAI)
> wrote:
>
> Hello,
>
> we are currently trying to understand the behaviour of camel if a thread
> “crashes” due to heap space.
>
> We observe the following error message in our logs:
> Error executing reactive work due to Java heap space. This exception is
> ignored.
> Camel (camel-1) thread #11 -
> sql://select%20*%20from%20LVS_REP.EXC_XMLFROMCOS%20where%20STATUS='25'%20and%20EXC_MSGTYPE%20like%20'LOAD00%25'%20and%20EXC_XML100%20in%20('AT0100','AT0730')
> After this error the route goes silent and no further “selects” are
> executed.
> We understand that this is intentional in order to protect other routes…
> but for us this behaviour is a serious problem as we are unable to alarm
> that there is an issue with the route.
>
> Is there any way to force camel not to ignore this exception and “rethrow”
> it?
>
> Thanks in Advance!
> Regards Christian
>
>
> [cid:ii_18e3718573ea4b0cb8a1]
>
> SPAR Business Services GmbH
> Information & Communication Services
> Europastrasse 3, 5015 Salzburg, Austria
>
> [cid:ii_18e3718573e81c1b3703]
>
> Sollten Sie diese E-Mail unbeabsichtigt bzw. irrtümlich erhalten haben, so
> weisen wir Sie darauf hin, dass gemäß § 161 Abs 4 TKG 2021 der Inhalt sowie
> die Tatsache des Empfangs dieser E-Mail weder aufgezeichnet noch verwertet
> oder Unbefugten mitgeteilt werden dürfen. Wir ersuchen Sie, die Nachricht
> von Ihrem System zu löschen und sich mit uns in Verbindung zu setzen.
> Unter diesem Link finden Sie die Informationen über die Verarbeitung Ihrer
> personenbezogenen Daten sowie zu Ihren Rechten als betroffene Person:
> https://www.spar.at/service/datenschutzhinweise
>
> If you have received this email accidentally or in error, we point out
> that, in accordance with § 161 para. 4 TKG 2021 (Telecommunications Act),
> the contents of this email and the fact of its receipt must not be
> recorded, exploited or communicated to unauthorized persons. We ask you to
> delete the message from your system and to contact us.
> Under this link you will find information about the processing of your
> personal data and about your rights as a data subject
> https://www.spar.at/service/datenschutzhinweise
>
>
>
>
> --
> Claus Ibsen
> -
> @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
--
Claus Ibsen
-
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Error executing reactive work due to Java heap space. This exception is ignored.
Hi,
thanks for the repsonse!
Camel 3.22.0
the thrown exception class is:
java.lang.OutOfMemoryError: Java heap space
Regards Christian
Am 13.03.24, 10:16 schrieb "Claus Ibsen" :
What Camel version, and can you tell the exact exception class that was throw
On Tue, Mar 12, 2024 at 2:17 PM MÜLLER Christian (ICS460-BAI)
wrote:
Hello,
we are currently trying to understand the behaviour of camel if a thread
“crashes” due to heap space.
We observe the following error message in our logs:
Error executing reactive work due to Java heap space. This exception is ignored.
Camel (camel-1) thread #11 -
sql://select%20*%20from%20LVS_REP.EXC_XMLFROMCOS%20where%20STATUS='25'%20and%20EXC_MSGTYPE%20like%20'LOAD00%25'%20and%20EXC_XML100%20in%20('AT0100','AT0730')
After this error the route goes silent and no further “selects” are executed.
We understand that this is intentional in order to protect other routes…
but for us this behaviour is a serious problem as we are unable to alarm that
there is an issue with the route.
Is there any way to force camel not to ignore this exception and “rethrow” it?
Thanks in Advance!
Regards Christian
[cid:ii_18e3718573ea4b0cb8a1]
SPAR Business Services GmbH
Information & Communication Services
Europastrasse 3, 5015 Salzburg, Austria
[cid:ii_18e3718573e81c1b3703]
Sollten Sie diese E-Mail unbeabsichtigt bzw. irrtümlich erhalten haben, so
weisen wir Sie darauf hin, dass gemäß § 161 Abs 4 TKG 2021 der Inhalt sowie die
Tatsache des Empfangs dieser E-Mail weder aufgezeichnet noch verwertet oder
Unbefugten mitgeteilt werden dürfen. Wir ersuchen Sie, die Nachricht von Ihrem
System zu löschen und sich mit uns in Verbindung zu setzen.
Unter diesem Link finden Sie die Informationen über die Verarbeitung Ihrer
personenbezogenen Daten sowie zu Ihren Rechten als betroffene Person:
https://www.spar.at/service/datenschutzhinweise
If you have received this email accidentally or in error, we point out that, in
accordance with § 161 para. 4 TKG 2021 (Telecommunications Act), the contents
of this email and the fact of its receipt must not be recorded, exploited or
communicated to unauthorized persons. We ask you to delete the message from
your system and to contact us.
Under this link you will find information about the processing of your personal
data and about your rights as a data subject
https://www.spar.at/service/datenschutzhinweise
--
Claus Ibsen
-
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Error executing reactive work due to Java heap space. This exception is ignored.
What Camel version, and can you tell the exact exception class that was
throw
On Tue, Mar 12, 2024 at 2:17 PM MÜLLER Christian (ICS460-BAI)
wrote:
> Hello,
>
> we are currently trying to understand the behaviour of camel if a thread
> “crashes” due to heap space.
>
> We observe the following error message in our logs:
>
> *Error executing reactive work due to Java heap space. This exception is
> ignored. Camel (camel-1) thread #11 -
> sql://select%20*%20from%20LVS_REP.EXC_XMLFROMCOS%20where%20STATUS='25'%20and%20EXC_MSGTYPE%20like%20'LOAD00%25'%20and%20EXC_XML100%20in%20('AT0100','AT0730')*
>
> After this error the route goes silent and no further “selects” are
> executed.
> We understand that this is intentional in order to protect other routes…
> but for us this behaviour is a serious problem as we are unable to alarm
> that there is an issue with the route.
>
> Is there any way to force camel not to ignore this exception and “rethrow”
> it?
>
> Thanks in Advance!
> Regards Christian
>
>
>
Understanding backpressure
I try to understand backpressure on Camel routes better. For this purpose,
I set up a load test. This test contains 5 routes that are connected
through direct or seda endpoints. At the end of all routes, I put a delay
or throttle. The routes are getting more messages than the throttle/delay
can handle.
The reason of the test is that I like to notify users when backpressure is
applied, and what the impact on the resources.
In the test I measure the inflight messages on each route. I noticed that
the maximum inflight exchanges on each route is 10. Thus total inflight is
maximum of 50. I try to change the maximum inflight exchanges by setting a
route policy:
ThrottlingInflightRoutePolicy routePolicy = new ThrottlingInflightRoutePolicy();
routePolicy.setScope(ThrottlingInflightRoutePolicy.ThrottlingScope.Context);
routePolicy.setMaxInflightExchanges(1);
routePolicy.setCamelContext(context);
Bind it to the registry:
context.getRegistry().bind("myPolicy", routePolicy);
And then apply it to a route with
routePolicyRef
I can see that policy is applied (route.getRoutePolicyList()), but it
doesn't seem to have any effect. The max inflight exchanges on a route
remain 10.
My questions:
1. Where does this number "10" of max inflight exchanges per route come
from?
2. How can the max inflight exchanges be changed/configured?
3. What is the impact of changing this value? Is it just more memory that
is used by caching exchanges?
4. Is backpressure only related to routePolicy or there other settings
(threadpool, streamcache etc) that has influence?
5. Is there documentation available that has a detailed explanation on how
backpressure works, what the impact is and how to tune it?
Raymond
