Re: Camel plugin no longer shows up in hawtio
Might be a version issue. I had the same problem and that is what it was. Also, debugging is not there unless I drop back camel versions. On Thu, Jun 4, 2020, 7:38 PM Jeremy Ross wrote: > Hey camel folks, > > At some point the Camel plugin stopped showing up in hawtio. I can access > hawtio just fine, but Camel is not in the menu navigation. I don't recall > seeing anything in the upgrade guides about it. Did I miss something? > > Thanks, > > Jeremy >
Camel plugin no longer shows up in hawtio
Hey camel folks, At some point the Camel plugin stopped showing up in hawtio. I can access hawtio just fine, but Camel is not in the menu navigation. I don't recall seeing anything in the upgrade guides about it. Did I miss something? Thanks, Jeremy
Re: Passwords in Camel endpoint URIs and limitations of RAW syntax
Hi Florian,
if you're concerned about logging sensitive data, I'd recommend to
configure your logging framework to filter such sensitive information in
first place as the sensitive information might otherwise leak through
other means not in control of Camel itself, i.e. as logged directly from
within a bean.
I.e. in logback you could implement a CompositeConverter
(https://github.com/RovoMe/camel-rest-dsl-with-spring-security/blob/master/src/main/java/at/rovo/awsxray/utils/MaskingConverter.java)
where you perform masking of sensitive data yourself. In the logback
configuration you then need to register the custom implementation like
this
https://github.com/RovoMe/camel-rest-dsl-with-spring-security/blob/master/src/main/resources/logback.xml#L4
and use the conversionWord within your log pattern, i.e. %mask(%msg).
As this might be a costly operation on each log, it is probably
beneficial to define when to mask sensitive data by using markers. I.e.
in the linked implementation a CONFIDENTIAL marker is created which can
also be used in a Camel log
(https://github.com/RovoMe/camel-rest-dsl-with-spring-security/blob/master/src/main/java/at/rovo/awsxray/routes/HttpInvokerRoute.java#L61)
as well.
HTH,
Roman
On 04.06.2020 10:49, Florian Patzl wrote:
Hello Ralf,
thanks for your response. No, I didn't mention that in my description. :-)
URL encoding would be my preferred solution, too, but unfortunately that does not seem to
prevent the problems with passwords containing ")&". Unless something about my
encoding is wrong.
For example, given a password "pwd)=b", both with and without RAW(...) the
result is wrong:
TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw uri:
pop3://localhost:3110/?username=test2=RAW%28pwd%29%26a%3Db%29, normalized uri:
pop3://localhost:3110/?a=b%29=RAW(pwd)=test2
TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw uri:
pop3://localhost:3110/?username=test2=pwd%29%26a%3Db, normalized uri:
pop3://localhost:3110/?a=b=pwd%29=test2
I'm currently testing on 3.3.0.
Best Regards,
Florian
From: Claussnitzer, Ralf
Sent: Thursday, June 4, 2020 08:13
To: users@camel.apache.org
Subject: Re: Passwords in Camel endpoint URIs and limitations of RAW syntax
Hi Florian,
I may have missed the answer to my questions in your detailed problem
description. But how is this not solved by URL-Encoding?
There was once a bug with URL encodings in Camel. Does this bug still exist?
What version of Camel are you using?
-Ralf
From: Florian Patzl
Sent: Wednesday, June 3, 2020 2:49 PM
To: users@camel.apache.org
Subject: Passwords in Camel endpoint URIs and limitations of RAW syntax
Hello,
I'm trying to figure out the best way to handle passwords in Camel endpoint
URIs in my application.
I know the topic has been cause for Stack Overflow posts, JIRA entries and
mails but I'm still not sure I've got everything right.
Sorry for the big wall of text, but I think I should explain what exactly I've
tried and found out on the topic.
The main problem is that the reserved URI characters '+' and '&' (plus and
ampersand) cause parsing problems in Camel endpoint URIs.
'+' is replaced by a blank, and '&' is treated as the delimiter to the next
parameter.
An example URI for the password "pwd2+2":
pop3://localhost:3110/?username=test2=pwd2%2B2
A relevant post is here:
https://stackoverflow.com/questions/11018987/camel-how-to-include-an-ampersand-as-data-in-a-uri-not-as-a-delimiter/34926623#34926623
Now, the solution in documentation is using the RAW(...) syntax:
https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues
So for example:
pop3://localhost:3110/?username=test2=RAW(pwd2+2)
Using that feature means we can no longer treat Camel URIs as pure URIs in our
application, because the '+' of the password must not be escaped for this to
work.
But if there's no way around that, we can deal with that.
However, when trying the limits of the RAW(...) syntax, we noticed that it can not
parse passwords that contain ')&'.
This was covered in the following JIRA issue, and since then there is support
for an alternative syntax using curly braces: RAW{...}, that I didn't find in
documentation:
https://issues.apache.org/jira/browse/CAMEL-12982
The last comment provides a pretty detailed summary of the options and limits.
The alternative RAW{...} syntax works fine, except for a minor flaw: It breaks
URI sanitizing.
For example, Camel leaks the '&2' portion of the password 'pwd2&2' in log
entries like:
pop3://localhost:3110/?password=xx&2%7D=test2
The same problem existed for the RAW(...) syntax:
https://issues.apache.org/jira/browse/CAMEL-11269
So the fix should be rather simple, I will check whether there's already an
issue for that and might even be able to submit a PR for that.
But, more importantly: By
Re: camel-mail does not fetch attachments in Karaf.
Hi JB:
I am not exactly sure about the internals.
https://camel.apache.org/components/latest/others/attachments.html
>From the point of view of my code, it is:
...
Map attachments = exchange.getIn().getAttachments();
...
for (String name : attachments.keySet()) {
DataHandler dh = attachments.get(name);
// get the file name
String filename = dh.getName();
...
}
which basically gets the javax.activation.DataHandler instance of the
attachment
Do you know which particular package I need to import?
In my bundle, I had the following configuration as well
org.apache.felix
maven-bundle-plugin
true
javax.activation;version=1.1,
*;resolution:=optional
But, this also doesn't seem to help either.
Thank you.
Regards,
Cooshal.
On Thu, Jun 4, 2020 at 3:05 PM Jean-Baptiste Onofre wrote:
> Hi,
>
> How are you looking for the attachment ? In the class loader resource or
> using path ?
>
> I guess your attachement files are not found (either because it’s not
> private package of your bundle, or not imported correctly).
>
> Regards
> JB
>
> > Le 4 juin 2020 à 15:03, Kushal Gautam a écrit
> :
> >
> > Camel Version: 2.20.3
> > Java: Open JDK 1.8.0_242
> > Karaf: 4.2.0
> >
> > Hi:
> >
> > Currently, I am using camel-mail to fetch mails via IMAP.
> >
> > The route is pretty simple and looks like:
> >
> > from("imaps://{{IMAP_SERVER_URL}}"
> >+ "?username={{IMAP_EMAIL_USER}}"
> >+ "={{IMAP_EMAIL_PASS}}"
> >+ "=true"
> >+ "=false"
> >+ "=100"
> >+ "={{IMAP_POLL_DURATION}}")
> > .
> >
> > My custom processor looks something like below (most of the stuffs taken
> > from the attachments example):
> >
> > ...
> > @Override
> >public void process(Exchange exchange) throws Exception {
> >
> >exchange.getIn().setHeader("HAS_ATTACHMENTS", false);
> >
> >Map attachments =
> > exchange.getIn().getAttachments();
> >
> >if (attachments.size() > 0) {
> >for (String name : attachments.keySet()) {
> >DataHandler dh = attachments.get(name);
> >// get the file name
> >String filename = dh.getName();
> >
> >System.out.println(filename);
> >
> >// check if the attachment is an xml file
> >// if not continue to another attachment
> >if(!filename.endsWith(".xml")) {
> >continue;
> >}
> >
> >System.out.println("email has an xml attachment");
> >
> >// get the content and convert it to byte[]
> >byte[] data = exchange
> >.getContext()
> >.getTypeConverter()
> >.convertTo(byte[].class, dh.getInputStream());
> >
> >exchange.getIn().setHeader("FILE_NAME", filename);
> >exchange.getIn().setHeader("HAS_ATTACHMENTS", true);
> >
> >exchange.getIn().setBody(data);
> >break;
> >}
> >}
> > ...
> >
> > If I send an email with some attachments, this code works perfectly fine
> > when I execute it via Netbeans. But, attachments.size() returns 0 for the
> > same code and same email inside Karaf.
> >
> > Do I need to configure something specific for this?
> >
> > Any inputs on this would be helpful.
> >
> > Thanks,
> > Cooshal.
>
>
Re: camel-mail does not fetch attachments in Karaf.
Hi,
How are you looking for the attachment ? In the class loader resource or using
path ?
I guess your attachement files are not found (either because it’s not private
package of your bundle, or not imported correctly).
Regards
JB
> Le 4 juin 2020 à 15:03, Kushal Gautam a écrit :
>
> Camel Version: 2.20.3
> Java: Open JDK 1.8.0_242
> Karaf: 4.2.0
>
> Hi:
>
> Currently, I am using camel-mail to fetch mails via IMAP.
>
> The route is pretty simple and looks like:
>
> from("imaps://{{IMAP_SERVER_URL}}"
>+ "?username={{IMAP_EMAIL_USER}}"
>+ "={{IMAP_EMAIL_PASS}}"
>+ "=true"
>+ "=false"
>+ "=100"
>+ "={{IMAP_POLL_DURATION}}")
> .
>
> My custom processor looks something like below (most of the stuffs taken
> from the attachments example):
>
> ...
> @Override
>public void process(Exchange exchange) throws Exception {
>
>exchange.getIn().setHeader("HAS_ATTACHMENTS", false);
>
>Map attachments =
> exchange.getIn().getAttachments();
>
>if (attachments.size() > 0) {
>for (String name : attachments.keySet()) {
>DataHandler dh = attachments.get(name);
>// get the file name
>String filename = dh.getName();
>
>System.out.println(filename);
>
>// check if the attachment is an xml file
>// if not continue to another attachment
>if(!filename.endsWith(".xml")) {
>continue;
>}
>
>System.out.println("email has an xml attachment");
>
>// get the content and convert it to byte[]
>byte[] data = exchange
>.getContext()
>.getTypeConverter()
>.convertTo(byte[].class, dh.getInputStream());
>
>exchange.getIn().setHeader("FILE_NAME", filename);
>exchange.getIn().setHeader("HAS_ATTACHMENTS", true);
>
>exchange.getIn().setBody(data);
>break;
>}
>}
> ...
>
> If I send an email with some attachments, this code works perfectly fine
> when I execute it via Netbeans. But, attachments.size() returns 0 for the
> same code and same email inside Karaf.
>
> Do I need to configure something specific for this?
>
> Any inputs on this would be helpful.
>
> Thanks,
> Cooshal.
camel-mail does not fetch attachments in Karaf.
Camel Version: 2.20.3
Java: Open JDK 1.8.0_242
Karaf: 4.2.0
Hi:
Currently, I am using camel-mail to fetch mails via IMAP.
The route is pretty simple and looks like:
from("imaps://{{IMAP_SERVER_URL}}"
+ "?username={{IMAP_EMAIL_USER}}"
+ "={{IMAP_EMAIL_PASS}}"
+ "=true"
+ "=false"
+ "=100"
+ "={{IMAP_POLL_DURATION}}")
.
My custom processor looks something like below (most of the stuffs taken
from the attachments example):
...
@Override
public void process(Exchange exchange) throws Exception {
exchange.getIn().setHeader("HAS_ATTACHMENTS", false);
Map attachments =
exchange.getIn().getAttachments();
if (attachments.size() > 0) {
for (String name : attachments.keySet()) {
DataHandler dh = attachments.get(name);
// get the file name
String filename = dh.getName();
System.out.println(filename);
// check if the attachment is an xml file
// if not continue to another attachment
if(!filename.endsWith(".xml")) {
continue;
}
System.out.println("email has an xml attachment");
// get the content and convert it to byte[]
byte[] data = exchange
.getContext()
.getTypeConverter()
.convertTo(byte[].class, dh.getInputStream());
exchange.getIn().setHeader("FILE_NAME", filename);
exchange.getIn().setHeader("HAS_ATTACHMENTS", true);
exchange.getIn().setBody(data);
break;
}
}
...
If I send an email with some attachments, this code works perfectly fine
when I execute it via Netbeans. But, attachments.size() returns 0 for the
same code and same email inside Karaf.
Do I need to configure something specific for this?
Any inputs on this would be helpful.
Thanks,
Cooshal.
Re: Passwords in Camel endpoint URIs and limitations of RAW syntax
Hi,
thanks a lot, Fabry and Omar! Storing the password in a Properties file helps,
I just tested with a really contrived example "++pwd2&)more)&}&}" and it worked.
I hadn't tried that before because the explanations for using RAW in properties
files put me off:
> Notice we still define the RAW(value) style to ensure the password is used as
> is[...]
https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues
We haven't used the properties mechanism before and storing them in a file
might be problematic, but I think I've read that there are other options for
providing properties as well.
I'll read up on that; your suggestions definitely helped, thanks!
Best regards,
Florian
From: Omar Al-Safi
Sent: Thursday, June 4, 2020 10:59
To: users@camel.apache.org
Subject: Re: Passwords in Camel endpoint URIs and limitations of RAW syntax
Hi Florian,
As Fabry mentioned, it would be worth checking to use config file to
achieve this. Here is an example:
https://github.com/apache/camel-examples/tree/master/examples/camel-example-debezium
Regards,
Omar
On Thu, Jun 4, 2020 at 10:56 AM FabryProg wrote:
> Hello Florian,
>
> Did you try to save the password into a variable / parameter / config file
> and lookup it into the URI?
>
> Kind regards!
>
> Il giorno gio 4 giu 2020 alle ore 10:50 Florian Patzl <
> florian.pa...@evolit.com> ha scritto:
>
> > Hello Ralf,
> > thanks for your response. No, I didn't mention that in my description.
> :-)
> > URL encoding would be my preferred solution, too, but unfortunately that
> > does not seem to prevent the problems with passwords containing ")&".
> > Unless something about my encoding is wrong.
> >
> > For example, given a password "pwd)=b", both with and without RAW(...)
> > the result is wrong:
> >
> > TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw
> uri:
> > pop3://localhost:3110/?username=test2=RAW%28pwd%29%26a%3Db%29,
> > normalized uri:
> > pop3://localhost:3110/?a=b%29=RAW(pwd)=test2
> >
> > TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw
> uri:
> > pop3://localhost:3110/?username=test2=pwd%29%26a%3Db, normalized
> > uri: pop3://localhost:3110/?a=b=pwd%29=test2
> >
> > I'm currently testing on 3.3.0.
> >
> > Best Regards,
> > Florian
> >
> > From: Claussnitzer, Ralf
> > Sent: Thursday, June 4, 2020 08:13
> > To: users@camel.apache.org
> > Subject: Re: Passwords in Camel endpoint URIs and limitations of RAW
> syntax
> >
> > Hi Florian,
> >
> > I may have missed the answer to my questions in your detailed problem
> > description. But how is this not solved by URL-Encoding?
> > There was once a bug with URL encodings in Camel. Does this bug still
> > exist? What version of Camel are you using?
> >
> > -Ralf
> >
> > From: Florian Patzl
> > Sent: Wednesday, June 3, 2020 2:49 PM
> > To: users@camel.apache.org
> > Subject: Passwords in Camel endpoint URIs and limitations of RAW syntax
> >
> > Hello,
> > I'm trying to figure out the best way to handle passwords in Camel
> > endpoint URIs in my application.
> > I know the topic has been cause for Stack Overflow posts, JIRA entries
> and
> > mails but I'm still not sure I've got everything right.
> > Sorry for the big wall of text, but I think I should explain what exactly
> > I've tried and found out on the topic.
> >
> > The main problem is that the reserved URI characters '+' and '&' (plus
> and
> > ampersand) cause parsing problems in Camel endpoint URIs.
> > '+' is replaced by a blank, and '&' is treated as the delimiter to the
> > next parameter.
> > An example URI for the password "pwd2+2":
> > pop3://localhost:3110/?username=test2=pwd2%2B2
> >
> > A relevant post is here:
> >
> >
> https://stackoverflow.com/questions/11018987/camel-how-to-include-an-ampersand-as-data-in-a-uri-not-as-a-delimiter/34926623#34926623
> >
> >
> > Now, the solution in documentation is using the RAW(...) syntax:
> >
> >
> https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues
> > So for example:
> > pop3://localhost:3110/?username=test2=RAW(pwd2+2)
> >
> > Using that feature means we can no longer treat Camel URIs as pure URIs
> in
> > our application, because the '+' of the password must not be escaped for
> > this to work.
> > But if there's no way around that, we can deal with that.
> >
> > However, when trying the limits of the RAW(...) syntax, we noticed that
> it
> > can not parse passwords that contain ')&'.
> > This was covered in the following JIRA issue, and since then there is
> > support for an alternative syntax using curly braces: RAW{...}, that I
> > didn't find in documentation:
> > https://issues.apache.org/jira/browse/CAMEL-12982
> > The last comment provides a pretty detailed summary of the options
Re: Passwords in Camel endpoint URIs and limitations of RAW syntax
Hi Florian,
As Fabry mentioned, it would be worth checking to use config file to
achieve this. Here is an example:
https://github.com/apache/camel-examples/tree/master/examples/camel-example-debezium
Regards,
Omar
On Thu, Jun 4, 2020 at 10:56 AM FabryProg wrote:
> Hello Florian,
>
> Did you try to save the password into a variable / parameter / config file
> and lookup it into the URI?
>
> Kind regards!
>
> Il giorno gio 4 giu 2020 alle ore 10:50 Florian Patzl <
> florian.pa...@evolit.com> ha scritto:
>
> > Hello Ralf,
> > thanks for your response. No, I didn't mention that in my description.
> :-)
> > URL encoding would be my preferred solution, too, but unfortunately that
> > does not seem to prevent the problems with passwords containing ")&".
> > Unless something about my encoding is wrong.
> >
> > For example, given a password "pwd)=b", both with and without RAW(...)
> > the result is wrong:
> >
> > TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw
> uri:
> > pop3://localhost:3110/?username=test2=RAW%28pwd%29%26a%3Db%29,
> > normalized uri:
> > pop3://localhost:3110/?a=b%29=RAW(pwd)=test2
> >
> > TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw
> uri:
> > pop3://localhost:3110/?username=test2=pwd%29%26a%3Db, normalized
> > uri: pop3://localhost:3110/?a=b=pwd%29=test2
> >
> > I'm currently testing on 3.3.0.
> >
> > Best Regards,
> > Florian
> >
> > From: Claussnitzer, Ralf
> > Sent: Thursday, June 4, 2020 08:13
> > To: users@camel.apache.org
> > Subject: Re: Passwords in Camel endpoint URIs and limitations of RAW
> syntax
> >
> > Hi Florian,
> >
> > I may have missed the answer to my questions in your detailed problem
> > description. But how is this not solved by URL-Encoding?
> > There was once a bug with URL encodings in Camel. Does this bug still
> > exist? What version of Camel are you using?
> >
> > -Ralf
> >
> > From: Florian Patzl
> > Sent: Wednesday, June 3, 2020 2:49 PM
> > To: users@camel.apache.org
> > Subject: Passwords in Camel endpoint URIs and limitations of RAW syntax
> >
> > Hello,
> > I'm trying to figure out the best way to handle passwords in Camel
> > endpoint URIs in my application.
> > I know the topic has been cause for Stack Overflow posts, JIRA entries
> and
> > mails but I'm still not sure I've got everything right.
> > Sorry for the big wall of text, but I think I should explain what exactly
> > I've tried and found out on the topic.
> >
> > The main problem is that the reserved URI characters '+' and '&' (plus
> and
> > ampersand) cause parsing problems in Camel endpoint URIs.
> > '+' is replaced by a blank, and '&' is treated as the delimiter to the
> > next parameter.
> > An example URI for the password "pwd2+2":
> > pop3://localhost:3110/?username=test2=pwd2%2B2
> >
> > A relevant post is here:
> >
> >
> https://stackoverflow.com/questions/11018987/camel-how-to-include-an-ampersand-as-data-in-a-uri-not-as-a-delimiter/34926623#34926623
> >
> >
> > Now, the solution in documentation is using the RAW(...) syntax:
> >
> >
> https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues
> > So for example:
> > pop3://localhost:3110/?username=test2=RAW(pwd2+2)
> >
> > Using that feature means we can no longer treat Camel URIs as pure URIs
> in
> > our application, because the '+' of the password must not be escaped for
> > this to work.
> > But if there's no way around that, we can deal with that.
> >
> > However, when trying the limits of the RAW(...) syntax, we noticed that
> it
> > can not parse passwords that contain ')&'.
> > This was covered in the following JIRA issue, and since then there is
> > support for an alternative syntax using curly braces: RAW{...}, that I
> > didn't find in documentation:
> > https://issues.apache.org/jira/browse/CAMEL-12982
> > The last comment provides a pretty detailed summary of the options and
> > limits.
> >
> >
> > The alternative RAW{...} syntax works fine, except for a minor flaw: It
> > breaks URI sanitizing.
> > For example, Camel leaks the '&2' portion of the password 'pwd2&2' in log
> > entries like:
> > pop3://localhost:3110/?password=xx&2%7D=test2
> >
> > The same problem existed for the RAW(...) syntax:
> > https://issues.apache.org/jira/browse/CAMEL-11269
> > So the fix should be rather simple, I will check whether there's already
> > an issue for that and might even be able to submit a PR for that.
> >
> > But, more importantly: By checking the passwords for ')&' and '}&' and
> > dynamically deciding the RAW syntax to use, we should be able to support
> > any password *except* if they contain both ')&' and '}&'.
> > That is a weird constraint for passwords, but should be justifiable as
> > technical limitation.
> >
> >
> > As an alternative to all of this, I sometimes saw the suggestion to
> > configure the component
Re: Passwords in Camel endpoint URIs and limitations of RAW syntax
Hello Florian,
Did you try to save the password into a variable / parameter / config file
and lookup it into the URI?
Kind regards!
Il giorno gio 4 giu 2020 alle ore 10:50 Florian Patzl <
florian.pa...@evolit.com> ha scritto:
> Hello Ralf,
> thanks for your response. No, I didn't mention that in my description. :-)
> URL encoding would be my preferred solution, too, but unfortunately that
> does not seem to prevent the problems with passwords containing ")&".
> Unless something about my encoding is wrong.
>
> For example, given a password "pwd)=b", both with and without RAW(...)
> the result is wrong:
>
> TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw uri:
> pop3://localhost:3110/?username=test2=RAW%28pwd%29%26a%3Db%29,
> normalized uri:
> pop3://localhost:3110/?a=b%29=RAW(pwd)=test2
>
> TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw uri:
> pop3://localhost:3110/?username=test2=pwd%29%26a%3Db, normalized
> uri: pop3://localhost:3110/?a=b=pwd%29=test2
>
> I'm currently testing on 3.3.0.
>
> Best Regards,
> Florian
>
> From: Claussnitzer, Ralf
> Sent: Thursday, June 4, 2020 08:13
> To: users@camel.apache.org
> Subject: Re: Passwords in Camel endpoint URIs and limitations of RAW syntax
>
> Hi Florian,
>
> I may have missed the answer to my questions in your detailed problem
> description. But how is this not solved by URL-Encoding?
> There was once a bug with URL encodings in Camel. Does this bug still
> exist? What version of Camel are you using?
>
> -Ralf
>
> From: Florian Patzl
> Sent: Wednesday, June 3, 2020 2:49 PM
> To: users@camel.apache.org
> Subject: Passwords in Camel endpoint URIs and limitations of RAW syntax
>
> Hello,
> I'm trying to figure out the best way to handle passwords in Camel
> endpoint URIs in my application.
> I know the topic has been cause for Stack Overflow posts, JIRA entries and
> mails but I'm still not sure I've got everything right.
> Sorry for the big wall of text, but I think I should explain what exactly
> I've tried and found out on the topic.
>
> The main problem is that the reserved URI characters '+' and '&' (plus and
> ampersand) cause parsing problems in Camel endpoint URIs.
> '+' is replaced by a blank, and '&' is treated as the delimiter to the
> next parameter.
> An example URI for the password "pwd2+2":
> pop3://localhost:3110/?username=test2=pwd2%2B2
>
> A relevant post is here:
>
> https://stackoverflow.com/questions/11018987/camel-how-to-include-an-ampersand-as-data-in-a-uri-not-as-a-delimiter/34926623#34926623
>
>
> Now, the solution in documentation is using the RAW(...) syntax:
>
> https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues
> So for example:
> pop3://localhost:3110/?username=test2=RAW(pwd2+2)
>
> Using that feature means we can no longer treat Camel URIs as pure URIs in
> our application, because the '+' of the password must not be escaped for
> this to work.
> But if there's no way around that, we can deal with that.
>
> However, when trying the limits of the RAW(...) syntax, we noticed that it
> can not parse passwords that contain ')&'.
> This was covered in the following JIRA issue, and since then there is
> support for an alternative syntax using curly braces: RAW{...}, that I
> didn't find in documentation:
> https://issues.apache.org/jira/browse/CAMEL-12982
> The last comment provides a pretty detailed summary of the options and
> limits.
>
>
> The alternative RAW{...} syntax works fine, except for a minor flaw: It
> breaks URI sanitizing.
> For example, Camel leaks the '&2' portion of the password 'pwd2&2' in log
> entries like:
> pop3://localhost:3110/?password=xx&2%7D=test2
>
> The same problem existed for the RAW(...) syntax:
> https://issues.apache.org/jira/browse/CAMEL-11269
> So the fix should be rather simple, I will check whether there's already
> an issue for that and might even be able to submit a PR for that.
>
> But, more importantly: By checking the passwords for ')&' and '}&' and
> dynamically deciding the RAW syntax to use, we should be able to support
> any password *except* if they contain both ')&' and '}&'.
> That is a weird constraint for passwords, but should be justifiable as
> technical limitation.
>
>
> As an alternative to all of this, I sometimes saw the suggestion to
> configure the component with 'useRawUri':
>
> * In DefaultComponent, useRawUri() is hardcoded to false. That means
> for applying that to built-in components (e.g. Mail, FTP) we'd have to
> subclass the components to override the method?
> * Setting useRawUri on endpoint level does not seem to be supported:
> https://issues.apache.org/jira/browse/CAMEL-6230
> I tried that for the Mail component and got an error for unknown parameter
> useRawUri.
>
> So, is my conclusion correct that escaping passwords using RAW(...) or
>
Re: Passwords in Camel endpoint URIs and limitations of RAW syntax
Hello Ralf,
thanks for your response. No, I didn't mention that in my description. :-)
URL encoding would be my preferred solution, too, but unfortunately that does
not seem to prevent the problems with passwords containing ")&". Unless
something about my encoding is wrong.
For example, given a password "pwd)=b", both with and without RAW(...) the
result is wrong:
TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw uri:
pop3://localhost:3110/?username=test2=RAW%28pwd%29%26a%3Db%29,
normalized uri: pop3://localhost:3110/?a=b%29=RAW(pwd)=test2
TRACE o.a.c.i.engine.AbstractCamelContext - Getting endpoint with raw uri:
pop3://localhost:3110/?username=test2=pwd%29%26a%3Db, normalized uri:
pop3://localhost:3110/?a=b=pwd%29=test2
I'm currently testing on 3.3.0.
Best Regards,
Florian
From: Claussnitzer, Ralf
Sent: Thursday, June 4, 2020 08:13
To: users@camel.apache.org
Subject: Re: Passwords in Camel endpoint URIs and limitations of RAW syntax
Hi Florian,
I may have missed the answer to my questions in your detailed problem
description. But how is this not solved by URL-Encoding?
There was once a bug with URL encodings in Camel. Does this bug still exist?
What version of Camel are you using?
-Ralf
From: Florian Patzl
Sent: Wednesday, June 3, 2020 2:49 PM
To: users@camel.apache.org
Subject: Passwords in Camel endpoint URIs and limitations of RAW syntax
Hello,
I'm trying to figure out the best way to handle passwords in Camel endpoint
URIs in my application.
I know the topic has been cause for Stack Overflow posts, JIRA entries and
mails but I'm still not sure I've got everything right.
Sorry for the big wall of text, but I think I should explain what exactly I've
tried and found out on the topic.
The main problem is that the reserved URI characters '+' and '&' (plus and
ampersand) cause parsing problems in Camel endpoint URIs.
'+' is replaced by a blank, and '&' is treated as the delimiter to the next
parameter.
An example URI for the password "pwd2+2":
pop3://localhost:3110/?username=test2=pwd2%2B2
A relevant post is here:
https://stackoverflow.com/questions/11018987/camel-how-to-include-an-ampersand-as-data-in-a-uri-not-as-a-delimiter/34926623#34926623
Now, the solution in documentation is using the RAW(...) syntax:
https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues
So for example:
pop3://localhost:3110/?username=test2=RAW(pwd2+2)
Using that feature means we can no longer treat Camel URIs as pure URIs in our
application, because the '+' of the password must not be escaped for this to
work.
But if there's no way around that, we can deal with that.
However, when trying the limits of the RAW(...) syntax, we noticed that it can
not parse passwords that contain ')&'.
This was covered in the following JIRA issue, and since then there is support
for an alternative syntax using curly braces: RAW{...}, that I didn't find in
documentation:
https://issues.apache.org/jira/browse/CAMEL-12982
The last comment provides a pretty detailed summary of the options and limits.
The alternative RAW{...} syntax works fine, except for a minor flaw: It breaks
URI sanitizing.
For example, Camel leaks the '&2' portion of the password 'pwd2&2' in log
entries like:
pop3://localhost:3110/?password=xx&2%7D=test2
The same problem existed for the RAW(...) syntax:
https://issues.apache.org/jira/browse/CAMEL-11269
So the fix should be rather simple, I will check whether there's already an
issue for that and might even be able to submit a PR for that.
But, more importantly: By checking the passwords for ')&' and '}&' and
dynamically deciding the RAW syntax to use, we should be able to support any
password *except* if they contain both ')&' and '}&'.
That is a weird constraint for passwords, but should be justifiable as
technical limitation.
As an alternative to all of this, I sometimes saw the suggestion to configure
the component with 'useRawUri':
* In DefaultComponent, useRawUri() is hardcoded to false. That means for
applying that to built-in components (e.g. Mail, FTP) we'd have to subclass the
components to override the method?
* Setting useRawUri on endpoint level does not seem to be supported:
https://issues.apache.org/jira/browse/CAMEL-6230
I tried that for the Mail component and got an error for unknown parameter
useRawUri.
So, is my conclusion correct that escaping passwords using RAW(...) or RAW{...}
- depending on the input - is the best approach for handling passwords in
endpoint URIs?
Or am I missing a different approach to configure the passwords on endpoints?
I've briefly read up on using property placeholders in URIs and saw that we'd
still have to use RAW(...) there. So I don't think that helps.
I _think_ completely moving away from endpoint URIs and instantiating endpoints
in
Re: Passwords in Camel endpoint URIs and limitations of RAW syntax
Hi Florian,
I may have missed the answer to my questions in your detailed problem
description. But how is this not solved by URL-Encoding?
There was once a bug with URL encodings in Camel. Does this bug still exist?
What version of Camel are you using?
-Ralf
From: Florian Patzl
Sent: Wednesday, June 3, 2020 2:49 PM
To: users@camel.apache.org
Subject: Passwords in Camel endpoint URIs and limitations of RAW syntax
Hello,
I'm trying to figure out the best way to handle passwords in Camel endpoint
URIs in my application.
I know the topic has been cause for Stack Overflow posts, JIRA entries and
mails but I'm still not sure I've got everything right.
Sorry for the big wall of text, but I think I should explain what exactly I've
tried and found out on the topic.
The main problem is that the reserved URI characters '+' and '&' (plus and
ampersand) cause parsing problems in Camel endpoint URIs.
'+' is replaced by a blank, and '&' is treated as the delimiter to the next
parameter.
An example URI for the password "pwd2+2":
pop3://localhost:3110/?username=test2=pwd2%2B2
A relevant post is here:
https://stackoverflow.com/questions/11018987/camel-how-to-include-an-ampersand-as-data-in-a-uri-not-as-a-delimiter/34926623#34926623
Now, the solution in documentation is using the RAW(...) syntax:
https://camel.apache.org/manual/latest/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues
So for example:
pop3://localhost:3110/?username=test2=RAW(pwd2+2)
Using that feature means we can no longer treat Camel URIs as pure URIs in our
application, because the '+' of the password must not be escaped for this to
work.
But if there's no way around that, we can deal with that.
However, when trying the limits of the RAW(...) syntax, we noticed that it can
not parse passwords that contain ')&'.
This was covered in the following JIRA issue, and since then there is support
for an alternative syntax using curly braces: RAW{...}, that I didn't find in
documentation:
https://issues.apache.org/jira/browse/CAMEL-12982
The last comment provides a pretty detailed summary of the options and limits.
The alternative RAW{...} syntax works fine, except for a minor flaw: It breaks
URI sanitizing.
For example, Camel leaks the '&2' portion of the password 'pwd2&2' in log
entries like:
pop3://localhost:3110/?password=xx&2%7D=test2
The same problem existed for the RAW(...) syntax:
https://issues.apache.org/jira/browse/CAMEL-11269
So the fix should be rather simple, I will check whether there's already an
issue for that and might even be able to submit a PR for that.
But, more importantly: By checking the passwords for ')&' and '}&' and
dynamically deciding the RAW syntax to use, we should be able to support any
password *except* if they contain both ')&' and '}&'.
That is a weird constraint for passwords, but should be justifiable as
technical limitation.
As an alternative to all of this, I sometimes saw the suggestion to configure
the component with 'useRawUri':
* In DefaultComponent, useRawUri() is hardcoded to false. That means for
applying that to built-in components (e.g. Mail, FTP) we'd have to subclass the
components to override the method?
* Setting useRawUri on endpoint level does not seem to be supported:
https://issues.apache.org/jira/browse/CAMEL-6230
I tried that for the Mail component and got an error for unknown parameter
useRawUri.
So, is my conclusion correct that escaping passwords using RAW(...) or RAW{...}
- depending on the input - is the best approach for handling passwords in
endpoint URIs?
Or am I missing a different approach to configure the passwords on endpoints?
I've briefly read up on using property placeholders in URIs and saw that we'd
still have to use RAW(...) there. So I don't think that helps.
I _think_ completely moving away from endpoint URIs and instantiating endpoints
in plain Java code would also solve the issue, but that would require a couple
of major changes in our application.
Best regards,
Florian
