RE: [PossibleSpam] Re: Tomcat Rewrite Valve

[Joe Aldrich]

>Am 10.02.2016 um 15:23 schrieb Rémy Maucherat:
>> 2016-02-10 15:06 GMT+01:00 Joe Aldrich :
>>
 Am 29.01.2016 15:34, schrieb Joe Aldrich:
> Hello,
>
> I am using Tomcat 8.0.28 on Windows 10 and am having a problem with 
> the Rewrite Value. I must include the escaped form of an ampersand 
> '%26' in the output URL.
>
> My rewrite.config has the following:
>
> RewriteCond %{QUERY_STRING} ^(.*&)?SCID=8(&.*)?$ RewriteRule 
> ^/(product|specs|avail-options|avail-category)\.php$
> /Product.action?select=Model+4+\%26+4C [R=301,L,NE]
>
> I am escaping the percent sign with a backslash, and I have tried 
> using the NE flag. However, Tomcat always is treating the percent 
> symbol as a back reference to the above RewriteCond. If I don't 
> have a second capture group, then I get a 500 error from a 
> NullPointerException.
 The current tomcat code does not allow escaping of percent or dollar sign.

 The parser just looks for percent (or dollar) and applies it either 
 as a
>>> backreference (when it is followed by a digit), or a map.
 I have not found any indication, that escaping is possible with httpd.
 Could you provide a link to the doc, that states it is possible?
>>> In Apache mod_rewrite it is possible per this documentation:
>>> https://httpd.apache.org/docs/2.2/mod/mod_rewrite.html#quoting
>>
>> Ok, I added an item for that since the mod_rewrite behavior should be
>> implemented:
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=58988
>The fix will be included in 9.0.0.M4 and 8.0.33. The syntax will be the same 
>as with httpd 2.2 using a backslash to quota a percent sign.
>
>@Joe, could you test the current trunk for 8 or 9?

I tested the truck for Tomcat 8. Escaping '%' or '$' with '\' now works as 
expected according to the documentation.

Thanks for all the help on this,
 Joe

>Regards,
>  Felix
>> Rémy
>>
>>>
 If you are willing to build tomcat yourself, you could try the 
 attached
>>> patch, which will allow escaping of percent signs by specifying them as %%.
 Your example would thus look like
 "/Product.action?select=Model+4+%%26+4C".

 Regards,
   Felix
>>> I will look into applying the patch as I need to be able to redirect 
>>> to URLs that contain %26 in the query string.
>>> Much thanks,
>>> Joe
>>>
> I was working with the documentation on this page:
>
> http://tomcat.apache.org/tomcat-8.0-doc/rewrite.html
>
> The desired output URL would be:
>
> http://www.domain.com/Product.html?select=Model+4+%26+4C
>
> In the example given for the NE flag on the page reference above, 
> the percent sign is escaped by a backslash to prevent it from being 
> treated as a back-reference. This is not working for me. Instead I
> get:
>
> http://www.domain.com/Product.action?select=Model+4+\null6+4C
>
> Where the "null" is due to an empty second back-reference.  I 
> believe this is a bug in that it is not escaping the percent sign 
> (making it impossible to create the %26 in the redirect URL). Or am 
> I misunderstanding something here?
>
> As a side question, shouldn't an empty back-reference be blank 
> instead of adding 'null' to the URL?
>
> Joseph B Aldrich
>
>
> ---
> -- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: [PossibleSpam] Re: Tomcat Rewrite Valve

[Joe Aldrich]

>Am 10.02.2016 um 15:23 schrieb Rémy Maucherat:
>> 2016-02-10 15:06 GMT+01:00 Joe Aldrich :
>>
 Am 29.01.2016 15:34, schrieb Joe Aldrich:
> Hello,
>
> I am using Tomcat 8.0.28 on Windows 10 and am having a problem with 
> the Rewrite Value. I must include the escaped form of an ampersand 
> '%26' in the output URL.
>
> My rewrite.config has the following:
>
> RewriteCond %{QUERY_STRING} ^(.*&)?SCID=8(&.*)?$ RewriteRule 
> ^/(product|specs|avail-options|avail-category)\.php$
> /Product.action?select=Model+4+\%26+4C [R=301,L,NE]
>
> I am escaping the percent sign with a backslash, and I have tried 
> using the NE flag. However, Tomcat always is treating the percent 
> symbol as a back reference to the above RewriteCond. If I don't 
> have a second capture group, then I get a 500 error from a 
> NullPointerException.
 The current tomcat code does not allow escaping of percent or dollar sign.

 The parser just looks for percent (or dollar) and applies it either 
 as a
>>> backreference (when it is followed by a digit), or a map.
 I have not found any indication, that escaping is possible with httpd.
 Could you provide a link to the doc, that states it is possible?
>>> In Apache mod_rewrite it is possible per this documentation:
>>> https://httpd.apache.org/docs/2.2/mod/mod_rewrite.html#quoting
>>
>> Ok, I added an item for that since the mod_rewrite behavior should be
>> implemented:
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=58988
>The fix will be included in 9.0.0.M4 and 8.0.33. The syntax will be the same 
>as with httpd 2.2 using a backslash to quota a percent sign.
>
>@Joe, could you test the current trunk for 8 or 9?

Yes, I will test it out as soon as I get the chance.

Thanks again,
Joe

>Regards,
>  Felix
>> Rémy
>>
>>>
 If you are willing to build tomcat yourself, you could try the 
 attached
>>> patch, which will allow escaping of percent signs by specifying them as %%.
 Your example would thus look like
 "/Product.action?select=Model+4+%%26+4C".

 Regards,
   Felix
>>> I will look into applying the patch as I need to be able to redirect 
>>> to URLs that contain %26 in the query string.
>>> Much thanks,
>>> Joe
>>>
> I was working with the documentation on this page:
>
> http://tomcat.apache.org/tomcat-8.0-doc/rewrite.html
>
> The desired output URL would be:
>
> http://www.domain.com/Product.html?select=Model+4+%26+4C
>
> In the example given for the NE flag on the page reference above, 
> the percent sign is escaped by a backslash to prevent it from being 
> treated as a back-reference. This is not working for me. Instead I
> get:
>
> http://www.domain.com/Product.action?select=Model+4+\null6+4C
>
> Where the "null" is due to an empty second back-reference.  I 
> believe this is a bug in that it is not escaping the percent sign 
> (making it impossible to create the %26 in the redirect URL). Or am 
> I misunderstanding something here?
>
> As a side question, shouldn't an empty back-reference be blank 
> instead of adding 'null' to the URL?
>
> Joseph B Aldrich
>
>
> ---
> -- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [PossibleSpam] Re: Tomcat Rewrite Valve

[Felix Schumacher]

Am 10.02.2016 um 15:23 schrieb Rémy Maucherat:

2016-02-10 15:06 GMT+01:00 Joe Aldrich :


Am 29.01.2016 15:34, schrieb Joe Aldrich:

Hello,

I am using Tomcat 8.0.28 on Windows 10 and am having a problem with
the Rewrite Value. I must include the escaped form of an ampersand
'%26' in the output URL.

My rewrite.config has the following:

RewriteCond %{QUERY_STRING} ^(.*&)?SCID=8(&.*)?$ RewriteRule
^/(product|specs|avail-options|avail-category)\.php$
/Product.action?select=Model+4+\%26+4C [R=301,L,NE]

I am escaping the percent sign with a backslash, and I have tried
using the NE flag. However, Tomcat always is treating the percent
symbol as a back reference to the above RewriteCond. If I don't have a
second capture group, then I get a 500 error from a
NullPointerException.

The current tomcat code does not allow escaping of percent or dollar sign.

The parser just looks for percent (or dollar) and applies it either as a

backreference (when it is followed by a digit), or a map.

I have not found any indication, that escaping is possible with httpd.
Could you provide a link to the doc, that states it is possible?

In Apache mod_rewrite it is possible per this documentation:
https://httpd.apache.org/docs/2.2/mod/mod_rewrite.html#quoting


Ok, I added an item for that since the mod_rewrite behavior should be
implemented:
https://bz.apache.org/bugzilla/show_bug.cgi?id=58988
The fix will be included in 9.0.0.M4 and 8.0.33. The syntax will be the 
same as with httpd 2.2 using a backslash to quota a percent sign.


@Joe, could you test the current trunk for 8 or 9?

Regards,
 Felix

Rémy




If you are willing to build tomcat yourself, you could try the attached

patch, which will allow escaping of percent signs by specifying them as %%.

Your example would thus look like
"/Product.action?select=Model+4+%%26+4C".

Regards,
  Felix

I will look into applying the patch as I need to be able to redirect to
URLs that contain %26 in the query string.
Much thanks,
Joe


I was working with the documentation on this page:

http://tomcat.apache.org/tomcat-8.0-doc/rewrite.html

The desired output URL would be:

http://www.domain.com/Product.html?select=Model+4+%26+4C

In the example given for the NE flag on the page reference above, the
percent sign is escaped by a backslash to prevent it from being
treated as a back-reference. This is not working for me. Instead I
get:

http://www.domain.com/Product.action?select=Model+4+\null6+4C

Where the "null" is due to an empty second back-reference.  I believe
this is a bug in that it is not escaping the percent sign (making it
impossible to create the %26 in the redirect URL). Or am I
misunderstanding something here?

As a side question, shouldn't an empty back-reference be blank instead
of adding 'null' to the URL?

Joseph B Aldrich


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: [PossibleSpam] Re: Tomcat Rewrite Valve

[Joe Aldrich]

>Am 29.01.2016 15:34, schrieb Joe Aldrich:
>> Hello,
>> 
>> I am using Tomcat 8.0.28 on Windows 10 and am having a problem with 
>> the Rewrite Value. I must include the escaped form of an ampersand 
>> '%26' in the output URL.
>> 
>> My rewrite.config has the following:
>> 
>> RewriteCond %{QUERY_STRING} ^(.*&)?SCID=8(&.*)?$ RewriteRule 
>> ^/(product|specs|avail-options|avail-category)\.php$
>> /Product.action?select=Model+4+\%26+4C [R=301,L,NE]
>> 
>> I am escaping the percent sign with a backslash, and I have tried 
>> using the NE flag. However, Tomcat always is treating the percent 
>> symbol as a back reference to the above RewriteCond. If I don't have a 
>> second capture group, then I get a 500 error from a 
>> NullPointerException.

>The current tomcat code does not allow escaping of percent or dollar sign.
>
>The parser just looks for percent (or dollar) and applies it either as a 
>backreference (when it is followed by a digit), or a map.
>
>I have not found any indication, that escaping is possible with httpd. 
>Could you provide a link to the doc, that states it is possible?

In Apache mod_rewrite it is possible per this documentation:
https://httpd.apache.org/docs/2.2/mod/mod_rewrite.html#quoting

>If you are willing to build tomcat yourself, you could try the attached patch, 
>which will allow escaping of percent signs by specifying them as %%.
>
>Your example would thus look like
>"/Product.action?select=Model+4+%%26+4C".
>
>Regards,
>  Felix

I will look into applying the patch as I need to be able to redirect to URLs 
that contain %26 in the query string. 
Much thanks,
Joe

>> 
>> I was working with the documentation on this page:
>> 
>> http://tomcat.apache.org/tomcat-8.0-doc/rewrite.html
>> 
>> The desired output URL would be:
>> 
>> http://www.domain.com/Product.html?select=Model+4+%26+4C
>> 
>> In the example given for the NE flag on the page reference above, the 
>> percent sign is escaped by a backslash to prevent it from being 
>> treated as a back-reference. This is not working for me. Instead I
>> get:
>> 
>> http://www.domain.com/Product.action?select=Model+4+\null6+4C
>> 
>> Where the "null" is due to an empty second back-reference.  I believe 
>> this is a bug in that it is not escaping the percent sign (making it 
>> impossible to create the %26 in the redirect URL). Or am I 
>> misunderstanding something here?
>> 
>> As a side question, shouldn't an empty back-reference be blank instead 
>> of adding 'null' to the URL?
>> 
>> Joseph B Aldrich
>> 
>> 
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [PossibleSpam] Re: Tomcat Rewrite Valve

[Rémy Maucherat]

2016-02-10 15:06 GMT+01:00 Joe Aldrich :

>
> >Am 29.01.2016 15:34, schrieb Joe Aldrich:
> >> Hello,
> >>
> >> I am using Tomcat 8.0.28 on Windows 10 and am having a problem with
> >> the Rewrite Value. I must include the escaped form of an ampersand
> >> '%26' in the output URL.
> >>
> >> My rewrite.config has the following:
> >>
> >> RewriteCond %{QUERY_STRING} ^(.*&)?SCID=8(&.*)?$ RewriteRule
> >> ^/(product|specs|avail-options|avail-category)\.php$
> >> /Product.action?select=Model+4+\%26+4C [R=301,L,NE]
> >>
> >> I am escaping the percent sign with a backslash, and I have tried
> >> using the NE flag. However, Tomcat always is treating the percent
> >> symbol as a back reference to the above RewriteCond. If I don't have a
> >> second capture group, then I get a 500 error from a
> >> NullPointerException.
>
> >The current tomcat code does not allow escaping of percent or dollar sign.
> >
> >The parser just looks for percent (or dollar) and applies it either as a
> backreference (when it is followed by a digit), or a map.
> >
> >I have not found any indication, that escaping is possible with httpd.
> >Could you provide a link to the doc, that states it is possible?
>
> In Apache mod_rewrite it is possible per this documentation:
> https://httpd.apache.org/docs/2.2/mod/mod_rewrite.html#quoting


Ok, I added an item for that since the mod_rewrite behavior should be
implemented:
https://bz.apache.org/bugzilla/show_bug.cgi?id=58988

Rémy

>
>
> >If you are willing to build tomcat yourself, you could try the attached
> patch, which will allow escaping of percent signs by specifying them as %%.
> >
> >Your example would thus look like
> >"/Product.action?select=Model+4+%%26+4C".
> >
> >Regards,
> >  Felix
>
> I will look into applying the patch as I need to be able to redirect to
> URLs that contain %26 in the query string.
> Much thanks,
> Joe
>
> >>
> >> I was working with the documentation on this page:
> >>
> >> http://tomcat.apache.org/tomcat-8.0-doc/rewrite.html
> >>
> >> The desired output URL would be:
> >>
> >> http://www.domain.com/Product.html?select=Model+4+%26+4C
> >>
> >> In the example given for the NE flag on the page reference above, the
> >> percent sign is escaped by a backslash to prevent it from being
> >> treated as a back-reference. This is not working for me. Instead I
> >> get:
> >>
> >> http://www.domain.com/Product.action?select=Model+4+\null6+4C
> >>
> >> Where the "null" is due to an empty second back-reference.  I believe
> >> this is a bug in that it is not escaping the percent sign (making it
> >> impossible to create the %26 in the redirect URL). Or am I
> >> misunderstanding something here?
> >>
> >> As a side question, shouldn't an empty back-reference be blank instead
> >> of adding 'null' to the URL?
> >>
> >> Joseph B Aldrich
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
>

RE: [PossibleSpam] Re: Tomcat Rewrite Valve

[Joe Aldrich]

Hello,

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Joe,

>On 1/29/16 9:34 AM, Joe Aldrich wrote:
>> I am using Tomcat 8.0.28 on Windows 10 and am having a problem with 
>> the Rewrite Value. I must include the escaped form of an ampersand 
>> '%26' in the output URL.
>> 
>> My rewrite.config has the following:
>> 
>> RewriteCond %{QUERY_STRING} ^(.*&)?SCID=8(&.*)?$ RewriteRule 
>> ^/(product|specs|avail-options|avail-category)\.php$
>> /Product.action?select=Model+4+\%26+4C [R=301,L,NE]
>> 
>> I am escaping the percent sign with a backslash, and I have tried 
>> using the NE flag. However, Tomcat always is treating the percent 
>> symbol as a back reference to the above RewriteCond. If I don't have a 
>> second capture group, then I get a 500 error from a 
>> NullPointerException.

>Can you please post the stack trace from that?

Here is what I get if I don’t specify a second capture group:

HTTP Status 500 - No group 2

type Exception report

message No group 2

description The server encountered an internal error that prevented it from 
fulfilling this request.

exception

java.lang.IndexOutOfBoundsException: No group 2
java.util.regex.Matcher.group(Unknown Source)

org.apache.catalina.valves.rewrite.Substitution$RewriteCondBackReferenceElement.evaluate(Substitution.java:51)

org.apache.catalina.valves.rewrite.Substitution.evaluate(Substitution.java:238)

org.apache.catalina.valves.rewrite.RewriteRule.evaluate(RewriteRule.java:133)

org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:292)

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)

org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)

org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)

org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:673)

org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)

org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.lang.Thread.run(Unknown Source)
note The full stack trace of the root cause is available in the Apache 
Tomcat/8.0.28 logs.

Apache Tomcat/8.0.28

>> I was working with the documentation on this page:
>> 
>> http://tomcat.apache.org/tomcat-8.0-doc/rewrite.html
>> 
>> The desired output URL would be:
>> 
>> http://www.domain.com/Product.html?select=Model+4+%26+4C

>Presumably, if you don't escape it at all, you get:
>
>http://www.domain.com/Product.html?select=Model+4+%2526+4C
>
>?

If I do not use the backslash to escape the percent sign, then (with or without 
the [NE] flag) I get a back-reference resulting in a 500 error if there isn't a 
second capture group. If there is a second capture group I get:

http://www.domain.com/Product.html?select=Model+4+null26+4C

(where again, null represents there was nothing specified after the SCID=8 in 
the query string).

If I omit the [NE] flag and keep the backslash to escape the percent sign, the 
escaping of the percent sign fails and I get similar results except for the 
presence of the backslash in the output URL as:

http://www.domain.com/Product.html?select=Model+4+\null26+4C


>> In the example given for the NE flag on the page reference above, the 
>> percent sign is escaped by a backslash to prevent it from being 
>> treated as a back-reference. This is not working for me. Instead I
>> get:
>> 
>> http://www.domain.com/Product.action?select=Model+4+\null6+4C
>> 
>> Where the "null" is due to an empty second back-reference.  I believe 
>> this is a bug in that it is not escaping the percent sign (making it 
>> impossible to create the %26 in the redirect URL). Or am I 
>> misunderstanding something here?
>> 
>> As a side question, shouldn't an empty back-reference be blank instead 
>> of adding 'null' to the URL?

>I agree that the "null" is incorrect. That is almost certainly a bug.
>
>[NE] should be preventing escaping of the resulting URL, but that might break 
>if you had user-specified input being re-written, but then not escaped.
>
>I'm not entirely sure if backslash-escaping is expected to work for 
>back-references. It's certainly a reasonable expectation, especially if that's 
>the way that mod_rewrite >works (and I don't know if that's the case). The 
>"escaping" section is only mentioned in the "regular expressions" section, and 
>not in the "backreferences" section, which is >why I think there may be some 
>room for alternative interpretations, here.
>
>I'm curious if \$25 works (as opposed to