Re: Apache SSI breaks with tomcat-connectors-1.2.43 or newer

[Rainer Jung]

Thanks, eactly what we need. I will look into it, but it might take two 
or three days. Anyone who can do it more quickly is welcome. I suspect 
these log lines point to the root cause:


> [Mon Sep 30 15:04:03 2019][3906:140148297873536] [debug]
> jk_servlet_normalize::jk_util.c (2184): URI on entering
> jk_servlet_normalize: []
> [Mon Sep 30 15:04:03 2019][3906:140148297873536] [emerg]
> jk_servlet_normalize::jk_util.c (2188): [] does not start with '/'.
> [Mon Sep 30 15:04:03 2019][3906:140148297873536] [debug]

but we'll need to investigate whether it can be fixed by configuration 
or needs a code change. Our configs are pretty flexible but we need to 
understand a bit better how we can let mod_jk keep out of the SSI 
handling of the internal requests.


As far as I have understoff, you do not actually want these SSI requests 
to be forwarded via mod_jk/AJP and instead want all of them being 
handled directly by Apache?


regards,

Rainer

Am 30.09.2019 um 17:17 schrieb Ezsra McDonald:


Apologies as this is a long e-mail. I tried to include all the 
information to help troubleshoot the issue.


I have a lab setup with  a vagrant box to test.: CentOS Linux release 
7.7.1908 (Core)


httpd-2.4.6-90.el7
tomcat-connectors-1.2.44-src

The only changes I made to the config from default install are as follows

#*
FILE: /etc/httpd/conf/httpd.conf

#test-ssi

    Options +Includes
    AddHandler server-parsed .shtml


#*
FILE: /etc/httpd/conf/workers.properties

This file is currently blank.

#*
FILE: /etc/httpd/conf.d/mod_jk.conf

# Load mod_jk module
# Specify the filename of the mod_jk lib
#LoadModule jk_module modules/mod_jk.so



     # Where to find workers.properties
     JkWorkersFile conf/workers.properties

     # Where to put jk logs
     JkLogFile logs/mod_jk.log

     # Set the jk log level [debug/error/info]
     #JkLogLevel info
     JkLogLevel trace

     # Select the log format
     JkLogStampFormat  "[%a %b %d %H:%M:%S %Y]"

     # JkOptions indicates to send SSK KEY SIZE
     JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories

     # JkRequestLogFormat
     JkRequestLogFormat "%w %V %T"

     # Mount your applications
     # The default setting only sends Java application data to mod_jk.
     # Use the commented-out line to send all URLs through mod_jk.
     # JkMount /* loadbalancer
     # JkMount /application/* loadbalancer
     #JkMountFile conf.d/uriworkermap.properties

     # Add shared memory.
     # This directive is present with 1.2.10 and
     # later versions of mod_jk, and is needed for
     # for load balancing to work properly
     JkShmFile logs/jk.shm

     # Should mod_jk send SSL information (default is On)
     JkExtractSSL On
     # What is the indicator for SSL (default is HTTPS)
     JkHTTPSIndicator HTTPS
     # What is the indicator for SSL session (default is SSL_SESSION_ID)
     JkSESSIONIndicator SSL_SESSION_ID
     # What is the indicator for client SSL cipher suit (default is 
SSL_CIPHER)

     JkCIPHERIndicator SSL_CIPHER
     # What is the indicator for the client SSL certificated? (default 
is SSL_CLIENT_CERT)

     JkCERTSIndicator SSL_CLIENT_CERT

     # Add jkstatus for managing runtime data
     
         Order deny,allow
         Deny from all
         Allow from localhost 192.168.19.54 192.168.19.52
     



#*
FILE: /etc/httpd/conf.modules.d/02-mod_jk.conf

# This file configures the mod_jk modules:
LoadModule jk_module modules/mod_jk.so

#*
SSI Structure modeled after a previously working customer site.

[vagrant@WEBSRV test-ssi]$ tree /var/www/html/test-ssi
/var/www/html/test-ssi
├── includes
│   ├── footer.html
│   └── header.html
└── index.shtml

#*
FILE: /var/www/html/test-ssi/index.shtml


   
       SSI Test stuff
   
   
       
         
       
         
       
   


#*
When I load index.shtml in the browser I get the following page:

[an error occurred while processing this directive]
[an error occurred while processing this directive]

#*
error_log

[Mon Sep 30 15:04:03.634861 2019] [include:error] [pid 3906] [client 
10.211.55.42:37042 ] unable to include 
"includes/header.html" in parsed file 
/var/www/html/test-ssi/index.shtml, subrequest setup returned 404
[Mon Sep 30 15:04:03.634907 2019] [include:error] [pid 3906] [client 
10.211.55.42:37042 ] unable to include 
"includes/footer.html" in parsed file 
/var/www/html/test-ssi/index.shtml, subrequest setup returned 404


#*
If I reconfigure the SSI structure as follows the SSI works:

Re: Bug with Tomcat-8.5 and Apache Commons FileUpload

[i...@flyingfischer.ch]

Thanks! Setting overheadDataThreshold="0" does fix the issue.

Without overheadDataThreshold the issue is very inconsistent:

Firefox 69.0.1 (64-Bit) on LNX and on other OS does result in the
described error below.

Very strange: Chrome on LNX mostly works, which means I got
intermittently once got "connection closed" with Chrome. On the second
retry the upload worked.
Even stranger: after a successful upload by Chrome it even works with FF
with the same file.

Markus

Am 30.09.19 um 20:49 schrieb Mark Thomas:
> That sounds like the client has tripped the overhead threshold protection.
>
> As a short term fix you probably want to see a lower value for:
> overheadDataThreshold
>
> see: http://tomcat.apache.org/tomcat-8.5-doc/config/http2.html
>
> possibly as low as zero.
>
> Longer term you should ideally look into why the client is sending
> small, non-final DATA frames since that as inefficient and has been
> identified as a potential DoS vector in some servers (not Tomcat but we
> block it as abusive behaviour anyway).
>
> Depending on what the client is doing, you may need to adjust the other
> over head thresholds as well. If you have a reproducible test case,
> enabling debug for http2 in logging.properties should shed some light on
> exactly what is going on.
>
> Mark
>
>
> On 30/09/2019 17:48, i...@flyingfischer.ch wrote:
>> I stumbled over a new problem which very likely appeared after
>> apache-tomcat-8.5.43 and between apache-tomcat-8.5.46
>>
>> Using Apache Commons FileUpload gives for some kind of PDF files:
>>
>> [https-openssl-apr-443-exec-15]
>> org.apache.struts.upload.CommonsMultipartRequestHandler.handleRequest
>> Failed to parse multipart request
>>
>> org.apache.commons.fileupload.FileUploadBase$IOFileUploadException:
>> Processing of multipart/form-data request failed. java.io.IOException:
>> The socket associated with this connection has been closed.
>>
>> This only happens when using HTTP/2. Upload works when downgrading
>> tomcat to HTTP/1.
>>
>> There has been posted a similiar error report on Stackoverflow, which
>> indicates that the sizes of the files may be the reason:
>>
>> https://stackoverflow.com/questions/58118776/apache-commons-fileupload-problems-uploading-files-greater-than-100kb-using-htt
>>
>> Thanks for considering
>> Markus
>>
>>
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Bug with Tomcat-8.5 and Apache Commons FileUpload

[Mark Thomas]

That sounds like the client has tripped the overhead threshold protection.

As a short term fix you probably want to see a lower value for:
overheadDataThreshold

see: http://tomcat.apache.org/tomcat-8.5-doc/config/http2.html

possibly as low as zero.

Longer term you should ideally look into why the client is sending
small, non-final DATA frames since that as inefficient and has been
identified as a potential DoS vector in some servers (not Tomcat but we
block it as abusive behaviour anyway).

Depending on what the client is doing, you may need to adjust the other
over head thresholds as well. If you have a reproducible test case,
enabling debug for http2 in logging.properties should shed some light on
exactly what is going on.

Mark


On 30/09/2019 17:48, i...@flyingfischer.ch wrote:
> I stumbled over a new problem which very likely appeared after
> apache-tomcat-8.5.43 and between apache-tomcat-8.5.46
> 
> Using Apache Commons FileUpload gives for some kind of PDF files:
> 
> [https-openssl-apr-443-exec-15]
> org.apache.struts.upload.CommonsMultipartRequestHandler.handleRequest
> Failed to parse multipart request
> 
> org.apache.commons.fileupload.FileUploadBase$IOFileUploadException:
> Processing of multipart/form-data request failed. java.io.IOException:
> The socket associated with this connection has been closed.
> 
> This only happens when using HTTP/2. Upload works when downgrading
> tomcat to HTTP/1.
> 
> There has been posted a similiar error report on Stackoverflow, which
> indicates that the sizes of the files may be the reason:
> 
> https://stackoverflow.com/questions/58118776/apache-commons-fileupload-problems-uploading-files-greater-than-100kb-using-htt
> 
> Thanks for considering
> Markus
> 
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Bug with Tomcat-8.5 and Apache Commons FileUpload

[i...@flyingfischer.ch]

I stumbled over a new problem which very likely appeared after
apache-tomcat-8.5.43 and between apache-tomcat-8.5.46

Using Apache Commons FileUpload gives for some kind of PDF files:

[https-openssl-apr-443-exec-15]
org.apache.struts.upload.CommonsMultipartRequestHandler.handleRequest
Failed to parse multipart request

org.apache.commons.fileupload.FileUploadBase$IOFileUploadException:
Processing of multipart/form-data request failed. java.io.IOException:
The socket associated with this connection has been closed.

This only happens when using HTTP/2. Upload works when downgrading
tomcat to HTTP/1.

There has been posted a similiar error report on Stackoverflow, which
indicates that the sizes of the files may be the reason:

https://stackoverflow.com/questions/58118776/apache-commons-fileupload-problems-uploading-files-greater-than-100kb-using-htt

Thanks for considering
Markus



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Apache SSI breaks with tomcat-connectors-1.2.43 or newer

[Ezsra McDonald]

Apologies as this is a long e-mail. I tried to include all the information
to help troubleshoot the issue.

I have a lab setup with  a vagrant box to test.: CentOS Linux release
7.7.1908 (Core)

httpd-2.4.6-90.el7
tomcat-connectors-1.2.44-src

The only changes I made to the config from default install are as follows

#*
FILE: /etc/httpd/conf/httpd.conf

#test-ssi

   Options +Includes
   AddHandler server-parsed .shtml


#*
FILE: /etc/httpd/conf/workers.properties

This file is currently blank.

#*
FILE: /etc/httpd/conf.d/mod_jk.conf

# Load mod_jk module
# Specify the filename of the mod_jk lib
#LoadModule jk_module modules/mod_jk.so



# Where to find workers.properties
JkWorkersFile conf/workers.properties

# Where to put jk logs
JkLogFile logs/mod_jk.log

# Set the jk log level [debug/error/info]
#JkLogLevel info
JkLogLevel trace

# Select the log format
JkLogStampFormat  "[%a %b %d %H:%M:%S %Y]"

# JkOptions indicates to send SSK KEY SIZE
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories

# JkRequestLogFormat
JkRequestLogFormat "%w %V %T"

# Mount your applications
# The default setting only sends Java application data to mod_jk.
# Use the commented-out line to send all URLs through mod_jk.
# JkMount /* loadbalancer
# JkMount /application/* loadbalancer
#JkMountFile conf.d/uriworkermap.properties

# Add shared memory.
# This directive is present with 1.2.10 and
# later versions of mod_jk, and is needed for
# for load balancing to work properly
JkShmFile logs/jk.shm

# Should mod_jk send SSL information (default is On)
JkExtractSSL On
# What is the indicator for SSL (default is HTTPS)
JkHTTPSIndicator HTTPS
# What is the indicator for SSL session (default is SSL_SESSION_ID)
JkSESSIONIndicator SSL_SESSION_ID
# What is the indicator for client SSL cipher suit (default is
SSL_CIPHER)
JkCIPHERIndicator SSL_CIPHER
# What is the indicator for the client SSL certificated? (default is
SSL_CLIENT_CERT)
JkCERTSIndicator SSL_CLIENT_CERT

# Add jkstatus for managing runtime data

Order deny,allow
Deny from all
Allow from localhost 192.168.19.54 192.168.19.52




#*
FILE: /etc/httpd/conf.modules.d/02-mod_jk.conf

# This file configures the mod_jk modules:
LoadModule jk_module modules/mod_jk.so

#*
SSI Structure modeled after a previously working customer site.

[vagrant@WEBSRV test-ssi]$ tree /var/www/html/test-ssi
/var/www/html/test-ssi
├── includes
│   ├── footer.html
│   └── header.html
└── index.shtml

#*
FILE: /var/www/html/test-ssi/index.shtml


  
  SSI Test stuff
  
  
  

  

  
  


#*
When I load index.shtml in the browser I get the following page:

[an error occurred while processing this directive]
[an error occurred while processing this directive]

#*
error_log

[Mon Sep 30 15:04:03.634861 2019] [include:error] [pid 3906] [client
10.211.55.42:37042] unable to include "includes/header.html" in parsed file
/var/www/html/test-ssi/index.shtml, subrequest setup returned 404
[Mon Sep 30 15:04:03.634907 2019] [include:error] [pid 3906] [client
10.211.55.42:37042] unable to include "includes/footer.html" in parsed file
/var/www/html/test-ssi/index.shtml, subrequest setup returned 404

#*
If I reconfigure the SSI structure as follows the SSI works:

/var/www/html/test-ssi
├── footer.html
├── header.html
└── index.shtml

FILE: /var/www/html/test-ssi/index.shtml


  
  SSI Test stuff
  
  
  

  

  
  


It seems like mod_include can not find resources in sub directories


#*
FILE: /var/log/httpd/mod_jk.log. (257 lines. happy reading)

[Mon Sep 30 15:01:43 2019][3905:140148297873536] [debug]
jk_set_time_fmt::jk_util.c (487): Pre-processed log time stamp format is
'[%a %b %d %H:%M:%S %Y]'
[Mon Sep 30 15:01:43 2019][3905:140148297873536] [debug]
dump_options::mod_jk.c (665): JkOption 'ForwardURICompat' set in server
'127.0.0.1'
[Mon Sep 30 15:01:43 2019][3905:140148297873536] [debug]
dump_options::mod_jk.c (673): JkOption 'ForwardURIEscaped' set in server
'127.0.0.1'
[Mon Sep 30 15:01:43 2019][3905:140148297873536] [debug]
dump_options::mod_jk.c (697): JkOption 'ForwardKeySize' set in server
'127.0.0.1' (default)
[Mon Sep 30 15:01:43 2019][3905:140148297873536] [trace]
uri_worker_map_alloc::jk_uri_worker_map.c (216): enter
[Mon Sep 30 15:01:43 2019][3905:140148297873536] [trace]

Re: Tomcat 9.0.24/9.0.26 suspected memory leak

[Rémy Maucherat]

On Sat, Sep 28, 2019 at 9:05 PM Mark Thomas  wrote:

> On 27/09/2019 22:39, Chen Levy wrote:
> > -Original Message-
> > From: Mark Thomas 
> > Sent: Friday, September 27, 2019 15:34
> > To: users@tomcat.apache.org
> > Subject: Re: Tomcat 9.0.24/9.0.26 suspected memory leak
> >
> > On 27/09/2019 16:34, Chen Levy wrote:
> >> On 26/09/2019 18:22, Chen Levy wrote:
> >
> > 
> >
> >>> The HashMap referenced in the report appears to be "waitingProcessors"
> inside AbstractProtocol which contain 262K entries.
> >>
> >> OK. Those are asynchronous Servlets that are still in async mode.
> >
> > 
> >
> >> * I do not employ async servlets in my application
> >
> > OK. Do you use WebSocket? There is a code path to add Processors to the
> waitingProcessors Map for WebSocket as well.
> >
> > Mark
> >
> >
> > No, no WebSocket either; just plain old Servlets, Filters and the
> occasional JSP
>
> OK. That narrows down where/how this might be happening.
>
> What about if you disable HTTP/2. Do you still see the issue then?
>

I added debug code in
AbstractProtocol.ConnectionHandler.release(SocketWrapperBase) to check
if the processor considered was present in the waitingProcessors map. The
result is the following:
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@77b16580
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@1d902704
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@610c4fc8
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@1a3a3cb6
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@336f552d
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@3cd94f25
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@66e24762
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@7c7a1c3c
TEST-org.apache.coyote.http11.TestHttp11Processor.NIO.txt:CHECK PROCESSOR
FAILED org.apache.coyote.http11.Http11Processor@55a44822
TEST-org.apache.coyote.http11.upgrade.TestUpgradeInternalHandler.NIO.txt:CHECK
PROCESSOR FAILED
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal@6e55ff60
TEST-org.apache.coyote.http11.upgrade.TestUpgrade.NIO.txt:CHECK PROCESSOR
FAILED org.apache.coyote.http11.upgrade.UpgradeProcessorExternal@37d98b7f
TEST-org.apache.tomcat.websocket.server.TestShutdown.NIO.txt:CHECK
PROCESSOR FAILED
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal@6be9bd85
TEST-org.apache.tomcat.websocket.TestWsRemoteEndpoint.NIO.txt:CHECK
PROCESSOR FAILED
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal@3bd4e02f
TEST-org.apache.tomcat.websocket.TestWsRemoteEndpoint.NIO.txt:CHECK
PROCESSOR FAILED
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal@4bb23a77
TEST-org.apache.tomcat.websocket.TestWsRemoteEndpoint.NIO.txt:CHECK
PROCESSOR FAILED
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal@32e20d65
TEST-org.apache.tomcat.websocket.TestWsRemoteEndpoint.NIO.txt:CHECK
PROCESSOR FAILED
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal@16abf52f

All instances of not removed processors are either from async or upgraded
processors (the internal kind), as expected. I have verified the processor
instances above are never removed so it might be more robust to simply call
proto.removeWaitingProcessor(processor); in
AbstractProtocol.ConnectionHandler.release(SocketWrapperBase) (after all
the socket is closed and done after that point). There could be a more fine
grained solution of course.

However, this does not match the leak scenario described by the user, this
doesn't happen without async or websockets being used.

Rémy