Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
On Thu, Mar 14, 2019 at 2:58 PM Santhosh Kumar wrote: > I have tried the same set of testcases using h2load with tomcat 9.0.17 > from github releases > (https://github.com/apache/tomcat/releases/tag/9.0.17) . > > h2load -n 100 -d /tmp/1k https://localhost:8443/ > requests: 100 total, 64 started, 63 done, 63 succeeded, 37 failed, 37 > errored, 0 timeout > > h2load -n 200 -d /tmp/512b https://localhost:8443/ > requests: 200 total, 128 started, 127 done, 127 succeeded, 73 failed, > 73 errored, 0 timeout > > The issue still occurs in latest version too. > This is not related to the issue that you reported (you gave a stacktrace for it, and it was helpful). Please use a real test with a real servlet consuming the data, Rémy
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
I have tried the same set of testcases using h2load with tomcat 9.0.17 from github releases (https://github.com/apache/tomcat/releases/tag/9.0.17) . h2load -n 100 -d /tmp/1k https://localhost:8443/ requests: 100 total, 64 started, 63 done, 63 succeeded, 37 failed, 37 errored, 0 timeout h2load -n 200 -d /tmp/512b https://localhost:8443/ requests: 200 total, 128 started, 127 done, 127 succeeded, 73 failed, 73 errored, 0 timeout The issue still occurs in latest version too. On Fri, Mar 8, 2019 at 1:23 AM Rémy Maucherat wrote: > On Thu, Mar 7, 2019 at 5:22 PM Rémy Maucherat wrote: > > > On Thu, Mar 7, 2019 at 2:54 PM Rémy Maucherat wrote: > > > >> On Thu, Mar 7, 2019 at 1:47 PM Mark Thomas wrote: > >> > >>> On 07/03/2019 07:40, Santhosh Kumar wrote: > >>> > From some of the test cases I can safely say that tomcat is hitting > >>> some > >>> > limits, I have two test cases ran with two diff size of payload and > >>> without > >>> > any queryParams. The servlet is a empty servlet just returns after > >>> > receiving without doing any business side logic > >>> > >>> Can you repeat those tests with the NIO connector? It would be helpful > >>> to know if we should be looking at the HTTP/2 code or the low-level > >>> connector I/O code. > >>> > >> > >> I was planning to investigate since I'm hunting NIO2 additional issues > >> after the fix for BZ63182. This one looks simpler to reproduce at least > >> [assuming there's an issue]. > >> > > > > Ok, so it's a buffer size issue with vectored IO and SSL, the sizes used > > are too optimized. > > > > The "fix" will be in Tomcat 9.0.17, the read buffer used simply needs to be > larger with SSL. Both the JSSE and OpenSSL engines exhibit the same > behavior, so no possible workaround elsewhere. > > Rémy > -- *With Regards,* *Santhosh Kumar J*
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
On Thu, Mar 7, 2019 at 5:22 PM Rémy Maucherat wrote: > On Thu, Mar 7, 2019 at 2:54 PM Rémy Maucherat wrote: > >> On Thu, Mar 7, 2019 at 1:47 PM Mark Thomas wrote: >> >>> On 07/03/2019 07:40, Santhosh Kumar wrote: >>> > From some of the test cases I can safely say that tomcat is hitting >>> some >>> > limits, I have two test cases ran with two diff size of payload and >>> without >>> > any queryParams. The servlet is a empty servlet just returns after >>> > receiving without doing any business side logic >>> >>> Can you repeat those tests with the NIO connector? It would be helpful >>> to know if we should be looking at the HTTP/2 code or the low-level >>> connector I/O code. >>> >> >> I was planning to investigate since I'm hunting NIO2 additional issues >> after the fix for BZ63182. This one looks simpler to reproduce at least >> [assuming there's an issue]. >> > > Ok, so it's a buffer size issue with vectored IO and SSL, the sizes used > are too optimized. > The "fix" will be in Tomcat 9.0.17, the read buffer used simply needs to be larger with SSL. Both the JSSE and OpenSSL engines exhibit the same behavior, so no possible workaround elsewhere. Rémy
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
On Thu, Mar 7, 2019 at 2:54 PM Rémy Maucherat wrote: > On Thu, Mar 7, 2019 at 1:47 PM Mark Thomas wrote: > >> On 07/03/2019 07:40, Santhosh Kumar wrote: >> > From some of the test cases I can safely say that tomcat is hitting some >> > limits, I have two test cases ran with two diff size of payload and >> without >> > any queryParams. The servlet is a empty servlet just returns after >> > receiving without doing any business side logic >> >> Can you repeat those tests with the NIO connector? It would be helpful >> to know if we should be looking at the HTTP/2 code or the low-level >> connector I/O code. >> > > I was planning to investigate since I'm hunting NIO2 additional issues > after the fix for BZ63182. This one looks simpler to reproduce at least > [assuming there's an issue]. > Ok, so it's a buffer size issue with vectored IO and SSL, the sizes used are too optimized. Rémy > > Rémy > > >> >> Thanks, >> >> Mark >> >> >> > >> > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d >> > /home/local/santhosh/A-Test/nghttp2/agentRequest.txt >> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet >> > starting benchmark... >> > spawning thread #0: 1 total client(s). 100 total requests >> > TLS Protocol: TLSv1.3 >> > Cipher: TLS_AES_256_GCM_SHA384 >> > Server Temp Key: X25519 253 bits >> > Application protocol: h2 >> > progress: 10% done >> > progress: 20% done >> > progress: 30% done >> > progress: 40% done >> > progress: 50% done >> > >> > finished in 5.16s, 10.48 req/s, 552B/s >> > requests: 100 total, 55 started, 54 done, 54 succeeded, 46 failed, 46 >> > errored, 0 timeout >> > status codes: 55 2xx, 0 3xx, 0 4xx, 0 5xx >> > traffic: 2.78KB (2846) total, 1.77KB (1815) headers (space savings >> 43.10%), >> > 0B (0) data >> > min max mean sd+/- >> sd >> > time for request: 1.57ms 9.43ms 2.24ms 1.17ms >> 94.44% >> > time for connect: 4.69ms 4.69ms 4.69ms 0us >> 100.00% >> > time to 1st byte:0us 0us 0us 0us >> 0.00% >> > req/s : 10.48 10.48 10.480.00 >> 100.00% >> > >> > This above configuration always returns 54 succeeded, payload size is >> 1200B >> > (1200x54=64800) >> > >> -- >> > Now reduce the payload and trying the same test, >> > >> > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d >> > /home/local/santhosh/A-Test/nghttp2/agentRequest2.txt >> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet >> > starting benchmark... >> > spawning thread #0: 1 total client(s). 100 total requests >> > TLS Protocol: TLSv1.3 >> > Cipher: TLS_AES_256_GCM_SHA384 >> > Server Temp Key: X25519 253 bits >> > Application protocol: h2 >> > progress: 10% done >> > progress: 20% done >> > progress: 30% done >> > progress: 40% done >> > progress: 50% done >> > progress: 60% done >> > progress: 70% done >> > progress: 80% done >> > >> > finished in 5.21s, 16.11 req/s, 839B/s >> > requests: 100 total, 85 started, 84 done, 84 succeeded, 16 failed, 16 >> > errored, 0 timeout >> > status codes: 85 2xx, 0 3xx, 0 4xx, 0 5xx >> > traffic: 4.27KB (4376) total, 2.74KB (2805) headers (space savings >> 43.10%), >> > 0B (0) data >> > min max mean sd+/- >> sd >> > time for request: 1.43ms 5.80ms 2.04ms 760us >> 89.29% >> > time for connect: 5.02ms 5.02ms 5.02ms 0us >> 100.00% >> > time to 1st byte:0us 0us 0us 0us >> 0.00% >> > req/s : 16.11 16.11 16.110.00 >> 100.00% >> > >> > This above configuration always returns 84 succeeded, payload size is >> 775B >> > (775x84=65100) >> > >> -- >> > Reducing the payload much smaller, >> > >> > h2load -n200 -c1 -m1 --header="Content-Type:application/json" -d >> > /home/local/santhosh/A-Test/nghttp2/agentRequest3.txt >> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet >> > starting benchmark... >> > spawning thread #0: 1 total client(s). 200 total requests >> > TLS Protocol: TLSv1.3 >> > Cipher: TLS_AES_256_GCM_SHA384 >> > Server Temp Key: X25519 253 bits >> > Application protocol: h2 >> > progress: 10% done >> > progress: 20% done >> > progress: 30% done >> > progress: 40% done >> > progress: 50% done >> > progress: 60% done >> > progress: 70% done >> > progress: 80% done >> > progress: 90% done >> > >> > finished in 5.41s, 34.40 req/s, 1.73KB/s >> > requests: 200 total, 187 started, 186 done, 186 succeeded, 14 failed, 14 >> > errored, 0 timeout >> > status codes: 187 2xx, 0 3xx, 0 4xx, 0 5xx >> > traffic: 9.35KB (9578) total, 6.03KB (6171) headers (space savings >> 43.10%), >> > 0B (0) data
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
I have executed few test cases in both NIO and NIO2 as Mark has requested for testing NIO connector. --Below is NIO h2load -n50 -c1 -m1 --header="Content-Type:application/json" -d /home/local/santhosh/A-Test/nghttp2/agentRequest.txt https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet starting benchmark... spawning thread #0: 1 total client(s). 50 total requests TLS Protocol: TLSv1.3 Cipher: TLS_AES_256_GCM_SHA384 Server Temp Key: X25519 253 bits Application protocol: h2 progress: 10% done progress: 20% done progress: 30% done progress: 40% done progress: 50% done progress: 60% done progress: 70% done progress: 80% done progress: 90% done progress: 100% done finished in 133.78ms, 373.74 req/s, 18.91KB/s requests: 50 total, 50 started, 50 done, 50 succeeded, 0 failed, 0 errored, 0 timeout status codes: 50 2xx, 0 3xx, 0 4xx, 0 5xx traffic: 2.53KB (2591) total, 1.61KB (1650) headers (space savings 43.10%), 0B (0) data min max mean sd+/- sd time for request: 1.30ms 6.49ms 2.53ms 1.37ms80.00% time for connect: 3.86ms 3.86ms 3.86ms 0us 100.00% time to 1st byte:0us 0us 0us 0us 0.00% req/s : 375.11 375.11 375.110.00 100.00% This is for 1200B of data --Below is NIO h2load -n80 -c1 -m1 --header="Content-Type:application/json" -d /home/local/santhosh/A-Test/nghttp2/agentRequest2.txt https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet starting benchmark... spawning thread #0: 1 total client(s). 80 total requests TLS Protocol: TLSv1.3 Cipher: TLS_AES_256_GCM_SHA384 Server Temp Key: X25519 253 bits Application protocol: h2 progress: 10% done progress: 20% done progress: 30% done progress: 40% done progress: 50% done progress: 60% done progress: 70% done progress: 80% done progress: 90% done progress: 100% done finished in 153.28ms, 521.93 req/s, 26.26KB/s requests: 80 total, 80 started, 80 done, 80 succeeded, 0 failed, 0 errored, 0 timeout status codes: 80 2xx, 0 3xx, 0 4xx, 0 5xx traffic: 4.02KB (4121) total, 2.58KB (2640) headers (space savings 43.10%), 0B (0) data min max mean sd+/- sd time for request: 1.12ms 5.17ms 1.83ms 657us86.25% time for connect: 4.84ms 4.84ms 4.84ms 0us 100.00% time to 1st byte:0us 0us 0us 0us 0.00% req/s : 523.06 523.06 523.060.00 100.00% This is for 775B of data --Below is NIO h2load -n180 -c1 -m1 --header="Content-Type:application/json" -d /home/local/santhosh/A-Test/nghttp2/agentRequest3.txt https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet starting benchmark... spawning thread #0: 1 total client(s). 180 total requests TLS Protocol: TLSv1.3 Cipher: TLS_AES_256_GCM_SHA384 Server Temp Key: X25519 253 bits Application protocol: h2 progress: 10% done progress: 20% done progress: 30% done progress: 40% done progress: 50% done progress: 60% done progress: 70% done progress: 80% done progress: 90% done progress: 100% done finished in 280.89ms, 640.82 req/s, 32.06KB/s requests: 180 total, 180 started, 180 done, 180 succeeded, 0 failed, 0 errored, 0 timeout status codes: 180 2xx, 0 3xx, 0 4xx, 0 5xx traffic: 9.00KB (9221) total, 5.80KB (5940) headers (space savings 43.10%), 0B (0) data min max mean sd+/- sd time for request: 1.05ms 6.15ms 1.50ms 637us91.11% time for connect: 7.24ms 7.24ms 7.24ms 0us 100.00% time to 1st byte:0us 0us 0us 0us 0.00% req/s : 641.64 641.64 641.640.00 100.00% This is for 352B of data --Below is NIO Above are the Ideal cases it works for NIO beyond which it breaks. For ex: take the case 1 and increase the request count from 50 to 60 it hangs and its applicable for all the 3 cases , tweak the request count beyond the configured 50(1200B), 80(775B) or 180(352B) the test will hang forever h2load -n60 -c1 -m1 --header="Content-Type:application/json" -d /home/local/santhosh/A-Test/nghttp2/agentRequest.txt https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet starting benchmark... spawning thread #0: 1 total client(s). 60 total requests TLS Protocol: TLSv1.3 Cipher: TLS_AES_256_GCM_SHA384 Server Temp Key: X25519 253 bits Application protocol: h2 progress: 10% done progress: 20% done progress: 30% done progress: 40% done progress: 50% done progress: 60% done progress: 70% done progress: 80% done progress: 90% done
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
On Thu, Mar 7, 2019 at 1:47 PM Mark Thomas wrote: > On 07/03/2019 07:40, Santhosh Kumar wrote: > > From some of the test cases I can safely say that tomcat is hitting some > > limits, I have two test cases ran with two diff size of payload and > without > > any queryParams. The servlet is a empty servlet just returns after > > receiving without doing any business side logic > > Can you repeat those tests with the NIO connector? It would be helpful > to know if we should be looking at the HTTP/2 code or the low-level > connector I/O code. > I was planning to investigate since I'm hunting NIO2 additional issues after the fix for BZ63182. This one looks simpler to reproduce at least [assuming there's an issue]. Rémy > > Thanks, > > Mark > > > > > > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d > > /home/local/santhosh/A-Test/nghttp2/agentRequest.txt > > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet > > starting benchmark... > > spawning thread #0: 1 total client(s). 100 total requests > > TLS Protocol: TLSv1.3 > > Cipher: TLS_AES_256_GCM_SHA384 > > Server Temp Key: X25519 253 bits > > Application protocol: h2 > > progress: 10% done > > progress: 20% done > > progress: 30% done > > progress: 40% done > > progress: 50% done > > > > finished in 5.16s, 10.48 req/s, 552B/s > > requests: 100 total, 55 started, 54 done, 54 succeeded, 46 failed, 46 > > errored, 0 timeout > > status codes: 55 2xx, 0 3xx, 0 4xx, 0 5xx > > traffic: 2.78KB (2846) total, 1.77KB (1815) headers (space savings > 43.10%), > > 0B (0) data > > min max mean sd+/- > sd > > time for request: 1.57ms 9.43ms 2.24ms 1.17ms > 94.44% > > time for connect: 4.69ms 4.69ms 4.69ms 0us > 100.00% > > time to 1st byte:0us 0us 0us 0us > 0.00% > > req/s : 10.48 10.48 10.480.00 > 100.00% > > > > This above configuration always returns 54 succeeded, payload size is > 1200B > > (1200x54=64800) > > > -- > > Now reduce the payload and trying the same test, > > > > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d > > /home/local/santhosh/A-Test/nghttp2/agentRequest2.txt > > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet > > starting benchmark... > > spawning thread #0: 1 total client(s). 100 total requests > > TLS Protocol: TLSv1.3 > > Cipher: TLS_AES_256_GCM_SHA384 > > Server Temp Key: X25519 253 bits > > Application protocol: h2 > > progress: 10% done > > progress: 20% done > > progress: 30% done > > progress: 40% done > > progress: 50% done > > progress: 60% done > > progress: 70% done > > progress: 80% done > > > > finished in 5.21s, 16.11 req/s, 839B/s > > requests: 100 total, 85 started, 84 done, 84 succeeded, 16 failed, 16 > > errored, 0 timeout > > status codes: 85 2xx, 0 3xx, 0 4xx, 0 5xx > > traffic: 4.27KB (4376) total, 2.74KB (2805) headers (space savings > 43.10%), > > 0B (0) data > > min max mean sd+/- > sd > > time for request: 1.43ms 5.80ms 2.04ms 760us > 89.29% > > time for connect: 5.02ms 5.02ms 5.02ms 0us > 100.00% > > time to 1st byte:0us 0us 0us 0us > 0.00% > > req/s : 16.11 16.11 16.110.00 > 100.00% > > > > This above configuration always returns 84 succeeded, payload size is > 775B > > (775x84=65100) > > > -- > > Reducing the payload much smaller, > > > > h2load -n200 -c1 -m1 --header="Content-Type:application/json" -d > > /home/local/santhosh/A-Test/nghttp2/agentRequest3.txt > > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet > > starting benchmark... > > spawning thread #0: 1 total client(s). 200 total requests > > TLS Protocol: TLSv1.3 > > Cipher: TLS_AES_256_GCM_SHA384 > > Server Temp Key: X25519 253 bits > > Application protocol: h2 > > progress: 10% done > > progress: 20% done > > progress: 30% done > > progress: 40% done > > progress: 50% done > > progress: 60% done > > progress: 70% done > > progress: 80% done > > progress: 90% done > > > > finished in 5.41s, 34.40 req/s, 1.73KB/s > > requests: 200 total, 187 started, 186 done, 186 succeeded, 14 failed, 14 > > errored, 0 timeout > > status codes: 187 2xx, 0 3xx, 0 4xx, 0 5xx > > traffic: 9.35KB (9578) total, 6.03KB (6171) headers (space savings > 43.10%), > > 0B (0) data > > min max mean sd+/- > sd > > time for request: 1.18ms 13.49ms 1.91ms 1.13ms > 95.16% > > time for connect: 5.93ms 5.93ms 5.93ms 0us > 100.00% > > time to 1st byte:0us 0us 0us
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
On 07/03/2019 07:40, Santhosh Kumar wrote: > From some of the test cases I can safely say that tomcat is hitting some > limits, I have two test cases ran with two diff size of payload and without > any queryParams. The servlet is a empty servlet just returns after > receiving without doing any business side logic Can you repeat those tests with the NIO connector? It would be helpful to know if we should be looking at the HTTP/2 code or the low-level connector I/O code. Thanks, Mark > > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d > /home/local/santhosh/A-Test/nghttp2/agentRequest.txt > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet > starting benchmark... > spawning thread #0: 1 total client(s). 100 total requests > TLS Protocol: TLSv1.3 > Cipher: TLS_AES_256_GCM_SHA384 > Server Temp Key: X25519 253 bits > Application protocol: h2 > progress: 10% done > progress: 20% done > progress: 30% done > progress: 40% done > progress: 50% done > > finished in 5.16s, 10.48 req/s, 552B/s > requests: 100 total, 55 started, 54 done, 54 succeeded, 46 failed, 46 > errored, 0 timeout > status codes: 55 2xx, 0 3xx, 0 4xx, 0 5xx > traffic: 2.78KB (2846) total, 1.77KB (1815) headers (space savings 43.10%), > 0B (0) data > min max mean sd+/- sd > time for request: 1.57ms 9.43ms 2.24ms 1.17ms94.44% > time for connect: 4.69ms 4.69ms 4.69ms 0us 100.00% > time to 1st byte:0us 0us 0us 0us 0.00% > req/s : 10.48 10.48 10.480.00 100.00% > > This above configuration always returns 54 succeeded, payload size is 1200B > (1200x54=64800) > -- > Now reduce the payload and trying the same test, > > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d > /home/local/santhosh/A-Test/nghttp2/agentRequest2.txt > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet > starting benchmark... > spawning thread #0: 1 total client(s). 100 total requests > TLS Protocol: TLSv1.3 > Cipher: TLS_AES_256_GCM_SHA384 > Server Temp Key: X25519 253 bits > Application protocol: h2 > progress: 10% done > progress: 20% done > progress: 30% done > progress: 40% done > progress: 50% done > progress: 60% done > progress: 70% done > progress: 80% done > > finished in 5.21s, 16.11 req/s, 839B/s > requests: 100 total, 85 started, 84 done, 84 succeeded, 16 failed, 16 > errored, 0 timeout > status codes: 85 2xx, 0 3xx, 0 4xx, 0 5xx > traffic: 4.27KB (4376) total, 2.74KB (2805) headers (space savings 43.10%), > 0B (0) data > min max mean sd+/- sd > time for request: 1.43ms 5.80ms 2.04ms 760us89.29% > time for connect: 5.02ms 5.02ms 5.02ms 0us 100.00% > time to 1st byte:0us 0us 0us 0us 0.00% > req/s : 16.11 16.11 16.110.00 100.00% > > This above configuration always returns 84 succeeded, payload size is 775B > (775x84=65100) > -- > Reducing the payload much smaller, > > h2load -n200 -c1 -m1 --header="Content-Type:application/json" -d > /home/local/santhosh/A-Test/nghttp2/agentRequest3.txt > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet > starting benchmark... > spawning thread #0: 1 total client(s). 200 total requests > TLS Protocol: TLSv1.3 > Cipher: TLS_AES_256_GCM_SHA384 > Server Temp Key: X25519 253 bits > Application protocol: h2 > progress: 10% done > progress: 20% done > progress: 30% done > progress: 40% done > progress: 50% done > progress: 60% done > progress: 70% done > progress: 80% done > progress: 90% done > > finished in 5.41s, 34.40 req/s, 1.73KB/s > requests: 200 total, 187 started, 186 done, 186 succeeded, 14 failed, 14 > errored, 0 timeout > status codes: 187 2xx, 0 3xx, 0 4xx, 0 5xx > traffic: 9.35KB (9578) total, 6.03KB (6171) headers (space savings 43.10%), > 0B (0) data > min max mean sd+/- sd > time for request: 1.18ms 13.49ms 1.91ms 1.13ms95.16% > time for connect: 5.93ms 5.93ms 5.93ms 0us 100.00% > time to 1st byte:0us 0us 0us 0us 0.00% > req/s : 34.41 34.41 34.410.00 100.00% > > This above configuration always returns 186 succeeded, payload size is > 356 (356x186=66216) > > On Wed, Mar 6, 2019 at 9:15 PM John Dale wrote: > >> When you run your test(s), does it fail after a certain period of >> time, or just keep on going under a certain number of requests? >> >> Also, to confirm: you're sending 1000 Byte + query strings? >> >> Are you
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
>From some of the test cases I can safely say that tomcat is hitting some limits, I have two test cases ran with two diff size of payload and without any queryParams. The servlet is a empty servlet just returns after receiving without doing any business side logic h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d /home/local/santhosh/A-Test/nghttp2/agentRequest.txt https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet starting benchmark... spawning thread #0: 1 total client(s). 100 total requests TLS Protocol: TLSv1.3 Cipher: TLS_AES_256_GCM_SHA384 Server Temp Key: X25519 253 bits Application protocol: h2 progress: 10% done progress: 20% done progress: 30% done progress: 40% done progress: 50% done finished in 5.16s, 10.48 req/s, 552B/s requests: 100 total, 55 started, 54 done, 54 succeeded, 46 failed, 46 errored, 0 timeout status codes: 55 2xx, 0 3xx, 0 4xx, 0 5xx traffic: 2.78KB (2846) total, 1.77KB (1815) headers (space savings 43.10%), 0B (0) data min max mean sd+/- sd time for request: 1.57ms 9.43ms 2.24ms 1.17ms94.44% time for connect: 4.69ms 4.69ms 4.69ms 0us 100.00% time to 1st byte:0us 0us 0us 0us 0.00% req/s : 10.48 10.48 10.480.00 100.00% This above configuration always returns 54 succeeded, payload size is 1200B (1200x54=64800) -- Now reduce the payload and trying the same test, h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d /home/local/santhosh/A-Test/nghttp2/agentRequest2.txt https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet starting benchmark... spawning thread #0: 1 total client(s). 100 total requests TLS Protocol: TLSv1.3 Cipher: TLS_AES_256_GCM_SHA384 Server Temp Key: X25519 253 bits Application protocol: h2 progress: 10% done progress: 20% done progress: 30% done progress: 40% done progress: 50% done progress: 60% done progress: 70% done progress: 80% done finished in 5.21s, 16.11 req/s, 839B/s requests: 100 total, 85 started, 84 done, 84 succeeded, 16 failed, 16 errored, 0 timeout status codes: 85 2xx, 0 3xx, 0 4xx, 0 5xx traffic: 4.27KB (4376) total, 2.74KB (2805) headers (space savings 43.10%), 0B (0) data min max mean sd+/- sd time for request: 1.43ms 5.80ms 2.04ms 760us89.29% time for connect: 5.02ms 5.02ms 5.02ms 0us 100.00% time to 1st byte:0us 0us 0us 0us 0.00% req/s : 16.11 16.11 16.110.00 100.00% This above configuration always returns 84 succeeded, payload size is 775B (775x84=65100) -- Reducing the payload much smaller, h2load -n200 -c1 -m1 --header="Content-Type:application/json" -d /home/local/santhosh/A-Test/nghttp2/agentRequest3.txt https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet starting benchmark... spawning thread #0: 1 total client(s). 200 total requests TLS Protocol: TLSv1.3 Cipher: TLS_AES_256_GCM_SHA384 Server Temp Key: X25519 253 bits Application protocol: h2 progress: 10% done progress: 20% done progress: 30% done progress: 40% done progress: 50% done progress: 60% done progress: 70% done progress: 80% done progress: 90% done finished in 5.41s, 34.40 req/s, 1.73KB/s requests: 200 total, 187 started, 186 done, 186 succeeded, 14 failed, 14 errored, 0 timeout status codes: 187 2xx, 0 3xx, 0 4xx, 0 5xx traffic: 9.35KB (9578) total, 6.03KB (6171) headers (space savings 43.10%), 0B (0) data min max mean sd+/- sd time for request: 1.18ms 13.49ms 1.91ms 1.13ms95.16% time for connect: 5.93ms 5.93ms 5.93ms 0us 100.00% time to 1st byte:0us 0us 0us 0us 0.00% req/s : 34.41 34.41 34.410.00 100.00% This above configuration always returns 186 succeeded, payload size is 356 (356x186=66216) On Wed, Mar 6, 2019 at 9:15 PM John Dale wrote: > When you run your test(s), does it fail after a certain period of > time, or just keep on going under a certain number of requests? > > Also, to confirm: you're sending 1000 Byte + query strings? > > Are you doing anything in the server side component to verify that > your parameters have been received successfully? > > I seems very possible that there is increased overhead parsing the > request (POST) body. That's why I was wondering about the dynamics of > your test case. If you can achieve a steady load state, either some > optimization of the POST request parser could be done, or you could > accept that overhead if it is comparable to
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
When you run your test(s), does it fail after a certain period of time, or just keep on going under a certain number of requests? Also, to confirm: you're sending 1000 Byte + query strings? Are you doing anything in the server side component to verify that your parameters have been received successfully? I seems very possible that there is increased overhead parsing the request (POST) body. That's why I was wondering about the dynamics of your test case. If you can achieve a steady load state, either some optimization of the POST request parser could be done, or you could accept that overhead if it is comparable to other solutions. On 3/6/19, Santhosh Kumar wrote: > I hope so, I used updated packages/components at the time of development. > few may be outdated like tomcat native as I was using 1.2.18 while > developing but 1.2.21 got released recently. > > On Wed, Mar 6, 2019 at 6:18 PM John Dale wrote: > >> Have you upgraded to the most recent release of your major version? >> >> If so, and if this issue still persists, it is something that the core >> development team might want to look at assuming they can replicate the >> issue. >> >> On 3/5/19, Santhosh Kumar wrote: >> > Sometimes more than 10x >> > >> > On Tue, Mar 5, 2019 at 10:00 PM John Dale wrote: >> > >> >> How many orders of magnitude slower are the post requests? >> >> >> >> On 3/5/19, Santhosh Kumar wrote: >> >> > I was testing in the localhost >> >> > >> >> > On Tue, Mar 5, 2019 at 9:32 PM John Dale wrote: >> >> > >> >> >> Are you running your test client (h2load) on the same machine, same >> >> >> network, or is it over the net (so, like 20ms latency on each >> >> >> request)? The reason I ask is that if you are local (especially), >> >> >> it >> >> >> may queue up too many requests for tomcat to handle in the testing >> >> >> period with its thread pool. Will let you know if I have any other >> >> >> ideas. >> >> >> >> >> >> On 3/5/19, Santhosh Kumar wrote: >> >> >> > Bytes >> >> >> > >> >> >> > On Tue, Mar 5, 2019 at 9:28 PM John Dale >> wrote: >> >> >> > >> >> >> >> 1000-1500 MB or KB? >> >> >> >> >> >> >> >> On 3/4/19, Santhosh Kumar wrote: >> >> >> >> > As per the documentation, >> >> >> >> > >> >> >> >> >> >> >> >> >> >> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig >> >> >> >> > >> >> >> >> > this connector supports maxPostSize, by default the limit is >> >> >> >> > set >> >> >> >> > to >> >> >> 2MB >> >> >> >> > >> >> >> >> > On Tue, Mar 5, 2019 at 5:09 AM John Dale >> >> wrote: >> >> >> >> > >> >> >> >> >> Does anyone know if this connector supports maxPostSize >> >> >> >> >> parameter? >> >> >> >> >> >> >> >> >> >> On 3/4/19, Santhosh Kumar wrote: >> >> >> >> >> > Hi, >> >> >> >> >> > >> >> >> >> >> > We have a tomcat instance which is http2 enabled and it >> >> >> >> >> > needs >> >> >> >> >> > to >> >> >> >> >> > serve >> >> >> >> >> > large number of requests using multiplexing, so we have >> >> >> >> >> > configured >> >> >> >> >> > our >> >> >> >> >> > instance as follows, >> >> >> >> >> > >> >> >> >> >> > > >> >> >> >> > sslImplementationName="org.apache.tomcat.util.net >> >> >> >> >> .openssl.OpenSSLImplementation" >> >> >> >> >> > protocol="org.apache.coyote.http11.Http11Nio2Protocol" >> >> >> >> >> > maxThreads="5" >> >> >> >> >> > SSLEnabled="true" >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" >> >> >> >> >> > compression="on" >> minSpareThreads="25" >> >> >> >> >> > noCompressionUserAgents="gozilla, traviata" scheme="https" >> >> >> >> >> > secure="true" >> >> >> >> >> > keystoreFile="conf/myfile.keystore" keystorePass="password" >> >> >> >> >> > socket.appReadBufSize="81920" >> >> >> >> >> > socket.appWriteBufSize="81920" socket.rxBufSize="251880" >> >> >> >> >> > socket.txBufSize="438000"> >> >> >> >> >> > > >> >> >> >> > >> >> maxConcurrentStreamExecution="200" >> >> >> >> >> > maxConcurrentStreams="200" >> >> >> >> >> > className="org.apache.coyote.http2.Http2Protocol"/> >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > This instance mainly serves concurrent POST request which >> will >> >> >> >> >> > have >> >> >> >> >> payload >> >> >> >> >> > of size, approx 1000-1500, which can be verified by tomcat >> >> >> >> >> > logs >> >> >> >> >> > >> >> >> >> >> > org.apache.coyote.http2.Http2Parser.validateFrame >> >> >> >> >> > Connection >> >> [0], >> >> >> >> >> > Stream >> >> >> >> >> > [19], Frame type [DATA], Flags [1], Payload size [*1195*] >> >> >> >> >> > >> >> >> >> >> > We tested our server with the help of h2load as follows, >> >> >> >> >> > >> >> >> >> >> > h2load -n100 -c1 -m100 https://localhost:9191/ -d >> >> >> >> >> > '/agentRequest.txt' >> >> >> >> >> > >> >> >> >> >> > We are getting
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
I hope so, I used updated packages/components at the time of development. few may be outdated like tomcat native as I was using 1.2.18 while developing but 1.2.21 got released recently. On Wed, Mar 6, 2019 at 6:18 PM John Dale wrote: > Have you upgraded to the most recent release of your major version? > > If so, and if this issue still persists, it is something that the core > development team might want to look at assuming they can replicate the > issue. > > On 3/5/19, Santhosh Kumar wrote: > > Sometimes more than 10x > > > > On Tue, Mar 5, 2019 at 10:00 PM John Dale wrote: > > > >> How many orders of magnitude slower are the post requests? > >> > >> On 3/5/19, Santhosh Kumar wrote: > >> > I was testing in the localhost > >> > > >> > On Tue, Mar 5, 2019 at 9:32 PM John Dale wrote: > >> > > >> >> Are you running your test client (h2load) on the same machine, same > >> >> network, or is it over the net (so, like 20ms latency on each > >> >> request)? The reason I ask is that if you are local (especially), it > >> >> may queue up too many requests for tomcat to handle in the testing > >> >> period with its thread pool. Will let you know if I have any other > >> >> ideas. > >> >> > >> >> On 3/5/19, Santhosh Kumar wrote: > >> >> > Bytes > >> >> > > >> >> > On Tue, Mar 5, 2019 at 9:28 PM John Dale > wrote: > >> >> > > >> >> >> 1000-1500 MB or KB? > >> >> >> > >> >> >> On 3/4/19, Santhosh Kumar wrote: > >> >> >> > As per the documentation, > >> >> >> > > >> >> >> > >> >> > >> > https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig > >> >> >> > > >> >> >> > this connector supports maxPostSize, by default the limit is set > >> >> >> > to > >> >> 2MB > >> >> >> > > >> >> >> > On Tue, Mar 5, 2019 at 5:09 AM John Dale > >> wrote: > >> >> >> > > >> >> >> >> Does anyone know if this connector supports maxPostSize > >> >> >> >> parameter? > >> >> >> >> > >> >> >> >> On 3/4/19, Santhosh Kumar wrote: > >> >> >> >> > Hi, > >> >> >> >> > > >> >> >> >> > We have a tomcat instance which is http2 enabled and it needs > >> >> >> >> > to > >> >> >> >> > serve > >> >> >> >> > large number of requests using multiplexing, so we have > >> >> >> >> > configured > >> >> >> >> > our > >> >> >> >> > instance as follows, > >> >> >> >> > > >> >> >> >> > >> >> >> >> > sslImplementationName="org.apache.tomcat.util.net > >> >> >> >> .openssl.OpenSSLImplementation" > >> >> >> >> > protocol="org.apache.coyote.http11.Http11Nio2Protocol" > >> >> >> >> > maxThreads="5" SSLEnabled="true" > >> >> >> >> > > >> >> >> >> > >> >> >> > >> >> > >> > compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" > >> >> >> >> > compression="on" > minSpareThreads="25" > >> >> >> >> > noCompressionUserAgents="gozilla, traviata" scheme="https" > >> >> >> >> > secure="true" > >> >> >> >> > keystoreFile="conf/myfile.keystore" keystorePass="password" > >> >> >> >> > socket.appReadBufSize="81920" > >> >> >> >> > socket.appWriteBufSize="81920" socket.rxBufSize="251880" > >> >> >> >> > socket.txBufSize="438000"> > >> >> >> >> > >> >> >> >> > > >> maxConcurrentStreamExecution="200" > >> >> >> >> > maxConcurrentStreams="200" > >> >> >> >> > className="org.apache.coyote.http2.Http2Protocol"/> > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > This instance mainly serves concurrent POST request which > will > >> >> >> >> > have > >> >> >> >> payload > >> >> >> >> > of size, approx 1000-1500, which can be verified by tomcat > >> >> >> >> > logs > >> >> >> >> > > >> >> >> >> > org.apache.coyote.http2.Http2Parser.validateFrame Connection > >> [0], > >> >> >> >> > Stream > >> >> >> >> > [19], Frame type [DATA], Flags [1], Payload size [*1195*] > >> >> >> >> > > >> >> >> >> > We tested our server with the help of h2load as follows, > >> >> >> >> > > >> >> >> >> > h2load -n100 -c1 -m100 https://localhost:9191/ -d > >> >> >> >> > '/agentRequest.txt' > >> >> >> >> > > >> >> >> >> > We are getting this error as follows, > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch > >> >> >> >> > Connection > >> >> >> >> [0] > >> >> >> >> > java.io.IOException: Unable to unwrap data, invalid status > >> >> >> >> > [BUFFER_OVERFLOW] > >> >> >> >> > at > >> >> >> >> > org.apache.tomcat.util.net > >> >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) > >> >> >> >> > at > >> >> >> >> > org.apache.tomcat.util.net > >> >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > >> >> >> >> > at > >> >> >> >> > > java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > >> >> >> >> > at > >> >> >> >> > java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) > >> >> >> >> > at > >> >> >> >> > java.base/sun.nio.ch > >> >> >> >> > >> >>
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
Have you upgraded to the most recent release of your major version? If so, and if this issue still persists, it is something that the core development team might want to look at assuming they can replicate the issue. On 3/5/19, Santhosh Kumar wrote: > Sometimes more than 10x > > On Tue, Mar 5, 2019 at 10:00 PM John Dale wrote: > >> How many orders of magnitude slower are the post requests? >> >> On 3/5/19, Santhosh Kumar wrote: >> > I was testing in the localhost >> > >> > On Tue, Mar 5, 2019 at 9:32 PM John Dale wrote: >> > >> >> Are you running your test client (h2load) on the same machine, same >> >> network, or is it over the net (so, like 20ms latency on each >> >> request)? The reason I ask is that if you are local (especially), it >> >> may queue up too many requests for tomcat to handle in the testing >> >> period with its thread pool. Will let you know if I have any other >> >> ideas. >> >> >> >> On 3/5/19, Santhosh Kumar wrote: >> >> > Bytes >> >> > >> >> > On Tue, Mar 5, 2019 at 9:28 PM John Dale wrote: >> >> > >> >> >> 1000-1500 MB or KB? >> >> >> >> >> >> On 3/4/19, Santhosh Kumar wrote: >> >> >> > As per the documentation, >> >> >> > >> >> >> >> >> >> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig >> >> >> > >> >> >> > this connector supports maxPostSize, by default the limit is set >> >> >> > to >> >> 2MB >> >> >> > >> >> >> > On Tue, Mar 5, 2019 at 5:09 AM John Dale >> wrote: >> >> >> > >> >> >> >> Does anyone know if this connector supports maxPostSize >> >> >> >> parameter? >> >> >> >> >> >> >> >> On 3/4/19, Santhosh Kumar wrote: >> >> >> >> > Hi, >> >> >> >> > >> >> >> >> > We have a tomcat instance which is http2 enabled and it needs >> >> >> >> > to >> >> >> >> > serve >> >> >> >> > large number of requests using multiplexing, so we have >> >> >> >> > configured >> >> >> >> > our >> >> >> >> > instance as follows, >> >> >> >> > >> >> >> >> > > >> >> >> > sslImplementationName="org.apache.tomcat.util.net >> >> >> >> .openssl.OpenSSLImplementation" >> >> >> >> > protocol="org.apache.coyote.http11.Http11Nio2Protocol" >> >> >> >> > maxThreads="5" SSLEnabled="true" >> >> >> >> > >> >> >> >> >> >> >> >> >> >> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" >> >> >> >> > compression="on" minSpareThreads="25" >> >> >> >> > noCompressionUserAgents="gozilla, traviata" scheme="https" >> >> >> >> > secure="true" >> >> >> >> > keystoreFile="conf/myfile.keystore" keystorePass="password" >> >> >> >> > socket.appReadBufSize="81920" >> >> >> >> > socket.appWriteBufSize="81920" socket.rxBufSize="251880" >> >> >> >> > socket.txBufSize="438000"> >> >> >> >> > > >> >> >> > >> maxConcurrentStreamExecution="200" >> >> >> >> > maxConcurrentStreams="200" >> >> >> >> > className="org.apache.coyote.http2.Http2Protocol"/> >> >> >> >> > >> >> >> >> > >> >> >> >> > This instance mainly serves concurrent POST request which will >> >> >> >> > have >> >> >> >> payload >> >> >> >> > of size, approx 1000-1500, which can be verified by tomcat >> >> >> >> > logs >> >> >> >> > >> >> >> >> > org.apache.coyote.http2.Http2Parser.validateFrame Connection >> [0], >> >> >> >> > Stream >> >> >> >> > [19], Frame type [DATA], Flags [1], Payload size [*1195*] >> >> >> >> > >> >> >> >> > We tested our server with the help of h2load as follows, >> >> >> >> > >> >> >> >> > h2load -n100 -c1 -m100 https://localhost:9191/ -d >> >> >> >> > '/agentRequest.txt' >> >> >> >> > >> >> >> >> > We are getting this error as follows, >> >> >> >> > >> >> >> >> > >> >> >> >> > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch >> >> >> >> > Connection >> >> >> >> [0] >> >> >> >> > java.io.IOException: Unable to unwrap data, invalid status >> >> >> >> > [BUFFER_OVERFLOW] >> >> >> >> > at >> >> >> >> > org.apache.tomcat.util.net >> >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) >> >> >> >> > at >> >> >> >> > org.apache.tomcat.util.net >> >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) >> >> >> >> > at >> >> >> >> > java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) >> >> >> >> > at >> >> >> >> > java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) >> >> >> >> > at >> >> >> >> > java.base/sun.nio.ch >> >> >> >> >> >> >> >> >> >> .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552) >> >> >> >> > at >> >> >> >> > java.base/sun.nio.ch >> >> >> >> >> >> >> >> >> >> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) >> >> >> >> > at >> >> >> >> > java.base/sun.nio.ch >> >> >> >> >> >> >> >> >> >> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) >> >> >> >> > at >> >> >> >> >
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
Sometimes more than 10x On Tue, Mar 5, 2019 at 10:00 PM John Dale wrote: > How many orders of magnitude slower are the post requests? > > On 3/5/19, Santhosh Kumar wrote: > > I was testing in the localhost > > > > On Tue, Mar 5, 2019 at 9:32 PM John Dale wrote: > > > >> Are you running your test client (h2load) on the same machine, same > >> network, or is it over the net (so, like 20ms latency on each > >> request)? The reason I ask is that if you are local (especially), it > >> may queue up too many requests for tomcat to handle in the testing > >> period with its thread pool. Will let you know if I have any other > >> ideas. > >> > >> On 3/5/19, Santhosh Kumar wrote: > >> > Bytes > >> > > >> > On Tue, Mar 5, 2019 at 9:28 PM John Dale wrote: > >> > > >> >> 1000-1500 MB or KB? > >> >> > >> >> On 3/4/19, Santhosh Kumar wrote: > >> >> > As per the documentation, > >> >> > > >> >> > >> > https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig > >> >> > > >> >> > this connector supports maxPostSize, by default the limit is set to > >> 2MB > >> >> > > >> >> > On Tue, Mar 5, 2019 at 5:09 AM John Dale > wrote: > >> >> > > >> >> >> Does anyone know if this connector supports maxPostSize parameter? > >> >> >> > >> >> >> On 3/4/19, Santhosh Kumar wrote: > >> >> >> > Hi, > >> >> >> > > >> >> >> > We have a tomcat instance which is http2 enabled and it needs to > >> >> >> > serve > >> >> >> > large number of requests using multiplexing, so we have > >> >> >> > configured > >> >> >> > our > >> >> >> > instance as follows, > >> >> >> > > >> >> >> > >> >> >> > sslImplementationName="org.apache.tomcat.util.net > >> >> >> .openssl.OpenSSLImplementation" > >> >> >> > protocol="org.apache.coyote.http11.Http11Nio2Protocol" > >> >> >> > maxThreads="5" SSLEnabled="true" > >> >> >> > > >> >> >> > >> >> > >> > compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" > >> >> >> > compression="on" minSpareThreads="25" > >> >> >> > noCompressionUserAgents="gozilla, traviata" scheme="https" > >> >> >> > secure="true" > >> >> >> > keystoreFile="conf/myfile.keystore" keystorePass="password" > >> >> >> > socket.appReadBufSize="81920" > >> >> >> > socket.appWriteBufSize="81920" socket.rxBufSize="251880" > >> >> >> > socket.txBufSize="438000"> > >> >> >> > >> >> >> > > maxConcurrentStreamExecution="200" > >> >> >> > maxConcurrentStreams="200" > >> >> >> > className="org.apache.coyote.http2.Http2Protocol"/> > >> >> >> > > >> >> >> > > >> >> >> > This instance mainly serves concurrent POST request which will > >> >> >> > have > >> >> >> payload > >> >> >> > of size, approx 1000-1500, which can be verified by tomcat logs > >> >> >> > > >> >> >> > org.apache.coyote.http2.Http2Parser.validateFrame Connection > [0], > >> >> >> > Stream > >> >> >> > [19], Frame type [DATA], Flags [1], Payload size [*1195*] > >> >> >> > > >> >> >> > We tested our server with the help of h2load as follows, > >> >> >> > > >> >> >> > h2load -n100 -c1 -m100 https://localhost:9191/ -d > >> >> >> > '/agentRequest.txt' > >> >> >> > > >> >> >> > We are getting this error as follows, > >> >> >> > > >> >> >> > > >> >> >> > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch > >> >> >> > Connection > >> >> >> [0] > >> >> >> > java.io.IOException: Unable to unwrap data, invalid status > >> >> >> > [BUFFER_OVERFLOW] > >> >> >> > at > >> >> >> > org.apache.tomcat.util.net > >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) > >> >> >> > at > >> >> >> > org.apache.tomcat.util.net > >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > >> >> >> > at > >> >> >> > java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > >> >> >> > at > >> >> >> > java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) > >> >> >> > at > >> >> >> > java.base/sun.nio.ch > >> >> >> > >> >> > >> > .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552) > >> >> >> > at > >> >> >> > java.base/sun.nio.ch > >> >> >> > >> >> > >> > .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) > >> >> >> > at > >> >> >> > java.base/sun.nio.ch > >> >> >> > >> >> > >> > .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) > >> >> >> > at > >> >> >> > org.apache.tomcat.util.net > >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027) > >> >> >> > at > >> >> >> > org.apache.tomcat.util.net > >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > >> >> >> > at > >> >> >> > org.apache.tomcat.util.net > >> >> >> .SecureNio2Channel.read(SecureNio2Channel.java:1067) > >> >> >> > at > >> >> >> > org.apache.tomcat.util.net > >> >> >> > >> >> > >> >
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
How many orders of magnitude slower are the post requests? On 3/5/19, Santhosh Kumar wrote: > I was testing in the localhost > > On Tue, Mar 5, 2019 at 9:32 PM John Dale wrote: > >> Are you running your test client (h2load) on the same machine, same >> network, or is it over the net (so, like 20ms latency on each >> request)? The reason I ask is that if you are local (especially), it >> may queue up too many requests for tomcat to handle in the testing >> period with its thread pool. Will let you know if I have any other >> ideas. >> >> On 3/5/19, Santhosh Kumar wrote: >> > Bytes >> > >> > On Tue, Mar 5, 2019 at 9:28 PM John Dale wrote: >> > >> >> 1000-1500 MB or KB? >> >> >> >> On 3/4/19, Santhosh Kumar wrote: >> >> > As per the documentation, >> >> > >> >> >> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig >> >> > >> >> > this connector supports maxPostSize, by default the limit is set to >> 2MB >> >> > >> >> > On Tue, Mar 5, 2019 at 5:09 AM John Dale wrote: >> >> > >> >> >> Does anyone know if this connector supports maxPostSize parameter? >> >> >> >> >> >> On 3/4/19, Santhosh Kumar wrote: >> >> >> > Hi, >> >> >> > >> >> >> > We have a tomcat instance which is http2 enabled and it needs to >> >> >> > serve >> >> >> > large number of requests using multiplexing, so we have >> >> >> > configured >> >> >> > our >> >> >> > instance as follows, >> >> >> > >> >> >> > > >> >> > sslImplementationName="org.apache.tomcat.util.net >> >> >> .openssl.OpenSSLImplementation" >> >> >> > protocol="org.apache.coyote.http11.Http11Nio2Protocol" >> >> >> > maxThreads="5" SSLEnabled="true" >> >> >> > >> >> >> >> >> >> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" >> >> >> > compression="on" minSpareThreads="25" >> >> >> > noCompressionUserAgents="gozilla, traviata" scheme="https" >> >> >> > secure="true" >> >> >> > keystoreFile="conf/myfile.keystore" keystorePass="password" >> >> >> > socket.appReadBufSize="81920" >> >> >> > socket.appWriteBufSize="81920" socket.rxBufSize="251880" >> >> >> > socket.txBufSize="438000"> >> >> >> > > >> >> >maxConcurrentStreamExecution="200" >> >> >> > maxConcurrentStreams="200" >> >> >> > className="org.apache.coyote.http2.Http2Protocol"/> >> >> >> > >> >> >> > >> >> >> > This instance mainly serves concurrent POST request which will >> >> >> > have >> >> >> payload >> >> >> > of size, approx 1000-1500, which can be verified by tomcat logs >> >> >> > >> >> >> > org.apache.coyote.http2.Http2Parser.validateFrame Connection [0], >> >> >> > Stream >> >> >> > [19], Frame type [DATA], Flags [1], Payload size [*1195*] >> >> >> > >> >> >> > We tested our server with the help of h2load as follows, >> >> >> > >> >> >> > h2load -n100 -c1 -m100 https://localhost:9191/ -d >> >> >> > '/agentRequest.txt' >> >> >> > >> >> >> > We are getting this error as follows, >> >> >> > >> >> >> > >> >> >> > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch >> >> >> > Connection >> >> >> [0] >> >> >> > java.io.IOException: Unable to unwrap data, invalid status >> >> >> > [BUFFER_OVERFLOW] >> >> >> > at >> >> >> > org.apache.tomcat.util.net >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) >> >> >> > at >> >> >> > org.apache.tomcat.util.net >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) >> >> >> > at >> >> >> > java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) >> >> >> > at >> >> >> > java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) >> >> >> > at >> >> >> > java.base/sun.nio.ch >> >> >> >> >> >> .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552) >> >> >> > at >> >> >> > java.base/sun.nio.ch >> >> >> >> >> >> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) >> >> >> > at >> >> >> > java.base/sun.nio.ch >> >> >> >> >> >> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) >> >> >> > at >> >> >> > org.apache.tomcat.util.net >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027) >> >> >> > at >> >> >> > org.apache.tomcat.util.net >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) >> >> >> > at >> >> >> > org.apache.tomcat.util.net >> >> >> .SecureNio2Channel.read(SecureNio2Channel.java:1067) >> >> >> > at >> >> >> > org.apache.tomcat.util.net >> >> >> >> >> >> .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153) >> >> >> > at >> >> >> > org.apache.tomcat.util.net >> >> >> .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026) >> >> >> > at >> >> >> > org.apache.tomcat.util.net >> >> >>
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
I was testing in the localhost On Tue, Mar 5, 2019 at 9:32 PM John Dale wrote: > Are you running your test client (h2load) on the same machine, same > network, or is it over the net (so, like 20ms latency on each > request)? The reason I ask is that if you are local (especially), it > may queue up too many requests for tomcat to handle in the testing > period with its thread pool. Will let you know if I have any other > ideas. > > On 3/5/19, Santhosh Kumar wrote: > > Bytes > > > > On Tue, Mar 5, 2019 at 9:28 PM John Dale wrote: > > > >> 1000-1500 MB or KB? > >> > >> On 3/4/19, Santhosh Kumar wrote: > >> > As per the documentation, > >> > > >> > https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig > >> > > >> > this connector supports maxPostSize, by default the limit is set to > 2MB > >> > > >> > On Tue, Mar 5, 2019 at 5:09 AM John Dale wrote: > >> > > >> >> Does anyone know if this connector supports maxPostSize parameter? > >> >> > >> >> On 3/4/19, Santhosh Kumar wrote: > >> >> > Hi, > >> >> > > >> >> > We have a tomcat instance which is http2 enabled and it needs to > >> >> > serve > >> >> > large number of requests using multiplexing, so we have configured > >> >> > our > >> >> > instance as follows, > >> >> > > >> >> > >> >> > sslImplementationName="org.apache.tomcat.util.net > >> >> .openssl.OpenSSLImplementation" > >> >> > protocol="org.apache.coyote.http11.Http11Nio2Protocol" > >> >> > maxThreads="5" SSLEnabled="true" > >> >> > > >> >> > >> > compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" > >> >> > compression="on" minSpareThreads="25" > >> >> > noCompressionUserAgents="gozilla, traviata" scheme="https" > >> >> > secure="true" > >> >> > keystoreFile="conf/myfile.keystore" keystorePass="password" > >> >> > socket.appReadBufSize="81920" > >> >> > socket.appWriteBufSize="81920" socket.rxBufSize="251880" > >> >> > socket.txBufSize="438000"> > >> >> > >> >> >maxConcurrentStreamExecution="200" > >> >> > maxConcurrentStreams="200" > >> >> > className="org.apache.coyote.http2.Http2Protocol"/> > >> >> > > >> >> > > >> >> > This instance mainly serves concurrent POST request which will have > >> >> payload > >> >> > of size, approx 1000-1500, which can be verified by tomcat logs > >> >> > > >> >> > org.apache.coyote.http2.Http2Parser.validateFrame Connection [0], > >> >> > Stream > >> >> > [19], Frame type [DATA], Flags [1], Payload size [*1195*] > >> >> > > >> >> > We tested our server with the help of h2load as follows, > >> >> > > >> >> > h2load -n100 -c1 -m100 https://localhost:9191/ -d > >> >> > '/agentRequest.txt' > >> >> > > >> >> > We are getting this error as follows, > >> >> > > >> >> > > >> >> > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch > >> >> > Connection > >> >> [0] > >> >> > java.io.IOException: Unable to unwrap data, invalid status > >> >> > [BUFFER_OVERFLOW] > >> >> > at > >> >> > org.apache.tomcat.util.net > >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) > >> >> > at > >> >> > org.apache.tomcat.util.net > >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > >> >> > at > >> >> > java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > >> >> > at > >> >> > java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) > >> >> > at > >> >> > java.base/sun.nio.ch > >> >> > >> > .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552) > >> >> > at > >> >> > java.base/sun.nio.ch > >> >> > >> > .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) > >> >> > at > >> >> > java.base/sun.nio.ch > >> >> > >> > .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) > >> >> > at > >> >> > org.apache.tomcat.util.net > >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027) > >> >> > at > >> >> > org.apache.tomcat.util.net > >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > >> >> > at > >> >> > org.apache.tomcat.util.net > >> >> .SecureNio2Channel.read(SecureNio2Channel.java:1067) > >> >> > at > >> >> > org.apache.tomcat.util.net > >> >> > >> > .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153) > >> >> > at > >> >> > org.apache.tomcat.util.net > >> >> .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026) > >> >> > at > >> >> > org.apache.tomcat.util.net > >> >> .SocketWrapperBase.read(SocketWrapperBase.java:1012) > >> >> > at > >> >> > > >> >> > >> > org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61) > >> >> > at > >> >> >
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
Are you running your test client (h2load) on the same machine, same network, or is it over the net (so, like 20ms latency on each request)? The reason I ask is that if you are local (especially), it may queue up too many requests for tomcat to handle in the testing period with its thread pool. Will let you know if I have any other ideas. On 3/5/19, Santhosh Kumar wrote: > Bytes > > On Tue, Mar 5, 2019 at 9:28 PM John Dale wrote: > >> 1000-1500 MB or KB? >> >> On 3/4/19, Santhosh Kumar wrote: >> > As per the documentation, >> > >> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig >> > >> > this connector supports maxPostSize, by default the limit is set to 2MB >> > >> > On Tue, Mar 5, 2019 at 5:09 AM John Dale wrote: >> > >> >> Does anyone know if this connector supports maxPostSize parameter? >> >> >> >> On 3/4/19, Santhosh Kumar wrote: >> >> > Hi, >> >> > >> >> > We have a tomcat instance which is http2 enabled and it needs to >> >> > serve >> >> > large number of requests using multiplexing, so we have configured >> >> > our >> >> > instance as follows, >> >> > >> >> > > >> > sslImplementationName="org.apache.tomcat.util.net >> >> .openssl.OpenSSLImplementation" >> >> > protocol="org.apache.coyote.http11.Http11Nio2Protocol" >> >> > maxThreads="5" SSLEnabled="true" >> >> > >> >> >> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" >> >> > compression="on" minSpareThreads="25" >> >> > noCompressionUserAgents="gozilla, traviata" scheme="https" >> >> > secure="true" >> >> > keystoreFile="conf/myfile.keystore" keystorePass="password" >> >> > socket.appReadBufSize="81920" >> >> > socket.appWriteBufSize="81920" socket.rxBufSize="251880" >> >> > socket.txBufSize="438000"> >> >> > > >> >maxConcurrentStreamExecution="200" >> >> > maxConcurrentStreams="200" >> >> > className="org.apache.coyote.http2.Http2Protocol"/> >> >> > >> >> > >> >> > This instance mainly serves concurrent POST request which will have >> >> payload >> >> > of size, approx 1000-1500, which can be verified by tomcat logs >> >> > >> >> > org.apache.coyote.http2.Http2Parser.validateFrame Connection [0], >> >> > Stream >> >> > [19], Frame type [DATA], Flags [1], Payload size [*1195*] >> >> > >> >> > We tested our server with the help of h2load as follows, >> >> > >> >> > h2load -n100 -c1 -m100 https://localhost:9191/ -d >> >> > '/agentRequest.txt' >> >> > >> >> > We are getting this error as follows, >> >> > >> >> > >> >> > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch >> >> > Connection >> >> [0] >> >> > java.io.IOException: Unable to unwrap data, invalid status >> >> > [BUFFER_OVERFLOW] >> >> > at >> >> > org.apache.tomcat.util.net >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) >> >> > at >> >> > org.apache.tomcat.util.net >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) >> >> > at >> >> > java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) >> >> > at >> >> > java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) >> >> > at >> >> > java.base/sun.nio.ch >> >> >> .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552) >> >> > at >> >> > java.base/sun.nio.ch >> >> >> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) >> >> > at >> >> > java.base/sun.nio.ch >> >> >> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) >> >> > at >> >> > org.apache.tomcat.util.net >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027) >> >> > at >> >> > org.apache.tomcat.util.net >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) >> >> > at >> >> > org.apache.tomcat.util.net >> >> .SecureNio2Channel.read(SecureNio2Channel.java:1067) >> >> > at >> >> > org.apache.tomcat.util.net >> >> >> .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153) >> >> > at >> >> > org.apache.tomcat.util.net >> >> .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026) >> >> > at >> >> > org.apache.tomcat.util.net >> >> .SocketWrapperBase.read(SocketWrapperBase.java:1012) >> >> > at >> >> > >> >> >> org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61) >> >> > at >> >> > org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69) >> >> > at >> >> > >> >> >> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322) >> >> > at >> >> > >> >> >> org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37) >> >> > at >> >> > >> >> >>
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
Bytes On Tue, Mar 5, 2019 at 9:28 PM John Dale wrote: > 1000-1500 MB or KB? > > On 3/4/19, Santhosh Kumar wrote: > > As per the documentation, > > > https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig > > > > this connector supports maxPostSize, by default the limit is set to 2MB > > > > On Tue, Mar 5, 2019 at 5:09 AM John Dale wrote: > > > >> Does anyone know if this connector supports maxPostSize parameter? > >> > >> On 3/4/19, Santhosh Kumar wrote: > >> > Hi, > >> > > >> > We have a tomcat instance which is http2 enabled and it needs to serve > >> > large number of requests using multiplexing, so we have configured our > >> > instance as follows, > >> > > >> > >> > sslImplementationName="org.apache.tomcat.util.net > >> .openssl.OpenSSLImplementation" > >> > protocol="org.apache.coyote.http11.Http11Nio2Protocol" > >> > maxThreads="5" SSLEnabled="true" > >> > > >> > compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" > >> > compression="on" minSpareThreads="25" > >> > noCompressionUserAgents="gozilla, traviata" scheme="https" > >> > secure="true" > >> > keystoreFile="conf/myfile.keystore" keystorePass="password" > >> > socket.appReadBufSize="81920" > >> > socket.appWriteBufSize="81920" socket.rxBufSize="251880" > >> > socket.txBufSize="438000"> > >> > >> >maxConcurrentStreamExecution="200" > >> > maxConcurrentStreams="200" > >> > className="org.apache.coyote.http2.Http2Protocol"/> > >> > > >> > > >> > This instance mainly serves concurrent POST request which will have > >> payload > >> > of size, approx 1000-1500, which can be verified by tomcat logs > >> > > >> > org.apache.coyote.http2.Http2Parser.validateFrame Connection [0], > >> > Stream > >> > [19], Frame type [DATA], Flags [1], Payload size [*1195*] > >> > > >> > We tested our server with the help of h2load as follows, > >> > > >> > h2load -n100 -c1 -m100 https://localhost:9191/ -d '/agentRequest.txt' > >> > > >> > We are getting this error as follows, > >> > > >> > > >> > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch Connection > >> [0] > >> > java.io.IOException: Unable to unwrap data, invalid status > >> > [BUFFER_OVERFLOW] > >> > at > >> > org.apache.tomcat.util.net > >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) > >> > at > >> > org.apache.tomcat.util.net > >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > >> > at > >> > java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > >> > at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) > >> > at > >> > java.base/sun.nio.ch > >> > .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552) > >> > at > >> > java.base/sun.nio.ch > >> > .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) > >> > at > >> > java.base/sun.nio.ch > >> > .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) > >> > at > >> > org.apache.tomcat.util.net > >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027) > >> > at > >> > org.apache.tomcat.util.net > >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > >> > at > >> > org.apache.tomcat.util.net > >> .SecureNio2Channel.read(SecureNio2Channel.java:1067) > >> > at > >> > org.apache.tomcat.util.net > >> > .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153) > >> > at > >> > org.apache.tomcat.util.net > >> .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026) > >> > at > >> > org.apache.tomcat.util.net > >> .SocketWrapperBase.read(SocketWrapperBase.java:1012) > >> > at > >> > > >> > org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61) > >> > at > >> > org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69) > >> > at > >> > > >> > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322) > >> > at > >> > > >> > org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37) > >> > at > >> > > >> > org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) > >> > at > >> > > >> > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53) > >> > at > >> > > >> > org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) > >> > at > >> > org.apache.tomcat.util.net > >> .Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769) > >> > at > >> > org.apache.tomcat.util.net > >> .SocketProcessorBase.run(SocketProcessorBase.java:49) > >> > at > >>
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
1000-1500 MB or KB? On 3/4/19, Santhosh Kumar wrote: > As per the documentation, > https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig > > this connector supports maxPostSize, by default the limit is set to 2MB > > On Tue, Mar 5, 2019 at 5:09 AM John Dale wrote: > >> Does anyone know if this connector supports maxPostSize parameter? >> >> On 3/4/19, Santhosh Kumar wrote: >> > Hi, >> > >> > We have a tomcat instance which is http2 enabled and it needs to serve >> > large number of requests using multiplexing, so we have configured our >> > instance as follows, >> > >> > > > sslImplementationName="org.apache.tomcat.util.net >> .openssl.OpenSSLImplementation" >> > protocol="org.apache.coyote.http11.Http11Nio2Protocol" >> > maxThreads="5" SSLEnabled="true" >> > >> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" >> > compression="on" minSpareThreads="25" >> > noCompressionUserAgents="gozilla, traviata" scheme="https" >> > secure="true" >> > keystoreFile="conf/myfile.keystore" keystorePass="password" >> > socket.appReadBufSize="81920" >> > socket.appWriteBufSize="81920" socket.rxBufSize="251880" >> > socket.txBufSize="438000"> >> > > >maxConcurrentStreamExecution="200" >> > maxConcurrentStreams="200" >> > className="org.apache.coyote.http2.Http2Protocol"/> >> > >> > >> > This instance mainly serves concurrent POST request which will have >> payload >> > of size, approx 1000-1500, which can be verified by tomcat logs >> > >> > org.apache.coyote.http2.Http2Parser.validateFrame Connection [0], >> > Stream >> > [19], Frame type [DATA], Flags [1], Payload size [*1195*] >> > >> > We tested our server with the help of h2load as follows, >> > >> > h2load -n100 -c1 -m100 https://localhost:9191/ -d '/agentRequest.txt' >> > >> > We are getting this error as follows, >> > >> > >> > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch Connection >> [0] >> > java.io.IOException: Unable to unwrap data, invalid status >> > [BUFFER_OVERFLOW] >> > at >> > org.apache.tomcat.util.net >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) >> > at >> > org.apache.tomcat.util.net >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) >> > at >> > java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) >> > at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) >> > at >> > java.base/sun.nio.ch >> .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552) >> > at >> > java.base/sun.nio.ch >> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) >> > at >> > java.base/sun.nio.ch >> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) >> > at >> > org.apache.tomcat.util.net >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027) >> > at >> > org.apache.tomcat.util.net >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) >> > at >> > org.apache.tomcat.util.net >> .SecureNio2Channel.read(SecureNio2Channel.java:1067) >> > at >> > org.apache.tomcat.util.net >> .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153) >> > at >> > org.apache.tomcat.util.net >> .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026) >> > at >> > org.apache.tomcat.util.net >> .SocketWrapperBase.read(SocketWrapperBase.java:1012) >> > at >> > >> org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61) >> > at >> > org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69) >> > at >> > >> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322) >> > at >> > >> org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37) >> > at >> > >> org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) >> > at >> > >> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53) >> > at >> > >> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) >> > at >> > org.apache.tomcat.util.net >> .Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769) >> > at >> > org.apache.tomcat.util.net >> .SocketProcessorBase.run(SocketProcessorBase.java:49) >> > at >> > org.apache.tomcat.util.net >> .AbstractEndpoint.processSocket(AbstractEndpoint.java:1048) >> > at >> > org.apache.tomcat.util.net >> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:116) >> > at >> > org.apache.tomcat.util.net >>
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
As per the documentation, https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig this connector supports maxPostSize, by default the limit is set to 2MB On Tue, Mar 5, 2019 at 5:09 AM John Dale wrote: > Does anyone know if this connector supports maxPostSize parameter? > > On 3/4/19, Santhosh Kumar wrote: > > Hi, > > > > We have a tomcat instance which is http2 enabled and it needs to serve > > large number of requests using multiplexing, so we have configured our > > instance as follows, > > > > > sslImplementationName="org.apache.tomcat.util.net > .openssl.OpenSSLImplementation" > > protocol="org.apache.coyote.http11.Http11Nio2Protocol" > > maxThreads="5" SSLEnabled="true" > > > compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" > > compression="on" minSpareThreads="25" > > noCompressionUserAgents="gozilla, traviata" scheme="https" secure="true" > > keystoreFile="conf/myfile.keystore" keystorePass="password" > > socket.appReadBufSize="81920" > > socket.appWriteBufSize="81920" socket.rxBufSize="251880" > > socket.txBufSize="438000"> > > >maxConcurrentStreamExecution="200" > > maxConcurrentStreams="200" > > className="org.apache.coyote.http2.Http2Protocol"/> > > > > > > This instance mainly serves concurrent POST request which will have > payload > > of size, approx 1000-1500, which can be verified by tomcat logs > > > > org.apache.coyote.http2.Http2Parser.validateFrame Connection [0], Stream > > [19], Frame type [DATA], Flags [1], Payload size [*1195*] > > > > We tested our server with the help of h2load as follows, > > > > h2load -n100 -c1 -m100 https://localhost:9191/ -d '/agentRequest.txt' > > > > We are getting this error as follows, > > > > > > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch Connection > [0] > > java.io.IOException: Unable to unwrap data, invalid status > > [BUFFER_OVERFLOW] > > at > > org.apache.tomcat.util.net > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) > > at > > org.apache.tomcat.util.net > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > > at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > > at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) > > at > > java.base/sun.nio.ch > .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552) > > at > > java.base/sun.nio.ch > .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) > > at > > java.base/sun.nio.ch > .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) > > at > > org.apache.tomcat.util.net > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027) > > at > > org.apache.tomcat.util.net > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > > at > > org.apache.tomcat.util.net > .SecureNio2Channel.read(SecureNio2Channel.java:1067) > > at > > org.apache.tomcat.util.net > .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153) > > at > > org.apache.tomcat.util.net > .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026) > > at > > org.apache.tomcat.util.net > .SocketWrapperBase.read(SocketWrapperBase.java:1012) > > at > > > org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61) > > at > > org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69) > > at > > > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322) > > at > > > org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37) > > at > > > org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) > > at > > > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53) > > at > > > org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) > > at > > org.apache.tomcat.util.net > .Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769) > > at > > org.apache.tomcat.util.net > .SocketProcessorBase.run(SocketProcessorBase.java:49) > > at > > org.apache.tomcat.util.net > .AbstractEndpoint.processSocket(AbstractEndpoint.java:1048) > > at > > org.apache.tomcat.util.net > .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:116) > > at > > org.apache.tomcat.util.net > .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:109) > > at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > > at
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
Does anyone know if this connector supports maxPostSize parameter? On 3/4/19, Santhosh Kumar wrote: > Hi, > > We have a tomcat instance which is http2 enabled and it needs to serve > large number of requests using multiplexing, so we have configured our > instance as follows, > > sslImplementationName="org.apache.tomcat.util.net.openssl.OpenSSLImplementation" > protocol="org.apache.coyote.http11.Http11Nio2Protocol" > maxThreads="5" SSLEnabled="true" > compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" > compression="on" minSpareThreads="25" > noCompressionUserAgents="gozilla, traviata" scheme="https" secure="true" > keystoreFile="conf/myfile.keystore" keystorePass="password" > socket.appReadBufSize="81920" > socket.appWriteBufSize="81920" socket.rxBufSize="251880" > socket.txBufSize="438000"> > maxConcurrentStreamExecution="200" > maxConcurrentStreams="200" > className="org.apache.coyote.http2.Http2Protocol"/> > > > This instance mainly serves concurrent POST request which will have payload > of size, approx 1000-1500, which can be verified by tomcat logs > > org.apache.coyote.http2.Http2Parser.validateFrame Connection [0], Stream > [19], Frame type [DATA], Flags [1], Payload size [*1195*] > > We tested our server with the help of h2load as follows, > > h2load -n100 -c1 -m100 https://localhost:9191/ -d '/agentRequest.txt' > > We are getting this error as follows, > > > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch Connection [0] > java.io.IOException: Unable to unwrap data, invalid status > [BUFFER_OVERFLOW] > at > org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) > at > org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) > at > java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552) > at > java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) > at > java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) > at > org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1027) > at > org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > at > org.apache.tomcat.util.net.SecureNio2Channel.read(SecureNio2Channel.java:1067) > at > org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153) > at > org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026) > at > org.apache.tomcat.util.net.SocketWrapperBase.read(SocketWrapperBase.java:1012) > at > org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61) > at > org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69) > at > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322) > at > org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37) > at > org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) > at > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53) > at > org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) > at > org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769) > at > org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) > at > org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1048) > at > org.apache.tomcat.util.net.SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:116) > at > org.apache.tomcat.util.net.SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:109) > at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) > > Why is this error is thrown? How can I configure tomcat to handle > concurrent POST requests which have a decent payload? > > > We have tried with various java clients like http-client-5-beta, jetty or > okhttp3 and spam requests to our tomcat using http2 multiplexing and we > found the time taken to process a requests increases(sometimes even 10x) > when load is increased. > We have tweaked all common configuration related to http2 on both client > and server side with no
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
Nice test case. Is the JVM able to garbage collect fast enough? Are you releasing the resources properly in your components as early as possible? Java VM's achieve a steady-state after a certain period of time, but I've found in some cases garbage collection doesn't keep-up with my load tests before steady state is reached. Just an idea. Again - nice test. :) On 3/4/19, Santhosh Kumar wrote: >> Do you actually need all those values and where do they come from in the > first place ? Do you understand what they do ? > > I was just experimenting with all available attributes related to socket > and multiplexing. I have followed the documentations, > https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig > >> The buffer sizes seem way >> too large, and maxConcurrentStreamExecution is as well (multiplexing 200 >> threads output over a single socket is not a very good idea usually, >> which >> is why the setting was added). >> The attributes are from java tcp socket attributes, you can check here > > I have given the 10x of default values to handle high load, and > maxConcurrentStreamExecution is also increased assuming it will allow large > data to be exchanged. I ll set back to default if it doesn't help > > From h2load, the payload of 1000b approx sent with 100 requests with > multiplexing as any, it fails after 65 requests approximately. > > I don't know whether tomcat have any limit set on buffer to restrict data > pushed in a same connection. > > On Mon, Mar 4, 2019 at 7:27 PM Rémy Maucherat wrote: > >> On Mon, Mar 4, 2019 at 10:40 AM Santhosh Kumar >> wrote: >> >> > Hi, >> > >> > We have a tomcat instance which is http2 enabled and it needs to serve >> > large number of requests using multiplexing, so we have configured our >> > instance as follows, >> > >> > > > sslImplementationName="org.apache.tomcat.util.net >> > .openssl.OpenSSLImplementation" >> > protocol="org.apache.coyote.http11.Http11Nio2Protocol" >> > maxThreads="5" SSLEnabled="true" >> > >> > >> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" >> > compression="on" minSpareThreads="25" >> > noCompressionUserAgents="gozilla, traviata" scheme="https" >> > secure="true" >> > keystoreFile="conf/myfile.keystore" keystorePass="password" >> > socket.appReadBufSize="81920" >> > socket.appWriteBufSize="81920" socket.rxBufSize="251880" >> > socket.txBufSize="438000"> >> > > >maxConcurrentStreamExecution="200" >> > maxConcurrentStreams="200" >> > className="org.apache.coyote.http2.Http2Protocol"/> >> > >> > >> >> Do you actually need all those values and where do they come from in the >> first place ? Do you understand what they do ? The buffer sizes seem way >> too large, and maxConcurrentStreamExecution is as well (multiplexing 200 >> threads output over a single socket is not a very good idea usually, >> which >> is why the setting was added). >> >> Rémy >> >> >> > >> > This instance mainly serves concurrent POST request which will have >> payload >> > of size, approx 1000-1500, which can be verified by tomcat logs >> > >> > org.apache.coyote.http2.Http2Parser.validateFrame Connection [0], >> > Stream >> > [19], Frame type [DATA], Flags [1], Payload size [*1195*] >> > >> > We tested our server with the help of h2load as follows, >> > >> > h2load -n100 -c1 -m100 https://localhost:9191/ -d '/agentRequest.txt' >> > >> > We are getting this error as follows, >> > >> > >> > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch Connection >> [0] >> > java.io.IOException: Unable to unwrap data, invalid status >> > [BUFFER_OVERFLOW] >> > at >> > org.apache.tomcat.util.net >> > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) >> > at >> > org.apache.tomcat.util.net >> > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) >> > at >> > java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) >> > at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) >> > at >> > java.base/sun.nio.ch >> > >> .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552) >> > at >> > java.base/sun.nio.ch >> > >> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) >> > at >> > java.base/sun.nio.ch >> > >> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) >> > at >> > org.apache.tomcat.util.net >> > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027) >> > at >> > org.apache.tomcat.util.net >> > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) >> > at >> > org.apache.tomcat.util.net >> > .SecureNio2Channel.read(SecureNio2Channel.java:1067) >> > at >> > org.apache.tomcat.util.net >> > >>
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
> Do you actually need all those values and where do they come from in the first place ? Do you understand what they do ? I was just experimenting with all available attributes related to socket and multiplexing. I have followed the documentations, https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig > The buffer sizes seem way > too large, and maxConcurrentStreamExecution is as well (multiplexing 200 > threads output over a single socket is not a very good idea usually, which > is why the setting was added). > The attributes are from java tcp socket attributes, you can check here I have given the 10x of default values to handle high load, and maxConcurrentStreamExecution is also increased assuming it will allow large data to be exchanged. I ll set back to default if it doesn't help >From h2load, the payload of 1000b approx sent with 100 requests with multiplexing as any, it fails after 65 requests approximately. I don't know whether tomcat have any limit set on buffer to restrict data pushed in a same connection. On Mon, Mar 4, 2019 at 7:27 PM Rémy Maucherat wrote: > On Mon, Mar 4, 2019 at 10:40 AM Santhosh Kumar > wrote: > > > Hi, > > > > We have a tomcat instance which is http2 enabled and it needs to serve > > large number of requests using multiplexing, so we have configured our > > instance as follows, > > > > > sslImplementationName="org.apache.tomcat.util.net > > .openssl.OpenSSLImplementation" > > protocol="org.apache.coyote.http11.Http11Nio2Protocol" > > maxThreads="5" SSLEnabled="true" > > > > > compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" > > compression="on" minSpareThreads="25" > > noCompressionUserAgents="gozilla, traviata" scheme="https" secure="true" > > keystoreFile="conf/myfile.keystore" keystorePass="password" > > socket.appReadBufSize="81920" > > socket.appWriteBufSize="81920" socket.rxBufSize="251880" > > socket.txBufSize="438000"> > > >maxConcurrentStreamExecution="200" > > maxConcurrentStreams="200" > > className="org.apache.coyote.http2.Http2Protocol"/> > > > > > > Do you actually need all those values and where do they come from in the > first place ? Do you understand what they do ? The buffer sizes seem way > too large, and maxConcurrentStreamExecution is as well (multiplexing 200 > threads output over a single socket is not a very good idea usually, which > is why the setting was added). > > Rémy > > > > > > This instance mainly serves concurrent POST request which will have > payload > > of size, approx 1000-1500, which can be verified by tomcat logs > > > > org.apache.coyote.http2.Http2Parser.validateFrame Connection [0], Stream > > [19], Frame type [DATA], Flags [1], Payload size [*1195*] > > > > We tested our server with the help of h2load as follows, > > > > h2load -n100 -c1 -m100 https://localhost:9191/ -d '/agentRequest.txt' > > > > We are getting this error as follows, > > > > > > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch Connection > [0] > > java.io.IOException: Unable to unwrap data, invalid status > > [BUFFER_OVERFLOW] > > at > > org.apache.tomcat.util.net > > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) > > at > > org.apache.tomcat.util.net > > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > > at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > > at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) > > at > > java.base/sun.nio.ch > > > .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552) > > at > > java.base/sun.nio.ch > > > .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) > > at > > java.base/sun.nio.ch > > > .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) > > at > > org.apache.tomcat.util.net > > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027) > > at > > org.apache.tomcat.util.net > > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > > at > > org.apache.tomcat.util.net > > .SecureNio2Channel.read(SecureNio2Channel.java:1067) > > at > > org.apache.tomcat.util.net > > > .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153) > > at > > org.apache.tomcat.util.net > > .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026) > > at > > org.apache.tomcat.util.net > > .SocketWrapperBase.read(SocketWrapperBase.java:1012) > > at > > > > > org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61) > > at > > org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69) > > at > > > > >
Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
On Mon, Mar 4, 2019 at 10:40 AM Santhosh Kumar wrote: > Hi, > > We have a tomcat instance which is http2 enabled and it needs to serve > large number of requests using multiplexing, so we have configured our > instance as follows, > > sslImplementationName="org.apache.tomcat.util.net > .openssl.OpenSSLImplementation" > protocol="org.apache.coyote.http11.Http11Nio2Protocol" > maxThreads="5" SSLEnabled="true" > > compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" > compression="on" minSpareThreads="25" > noCompressionUserAgents="gozilla, traviata" scheme="https" secure="true" > keystoreFile="conf/myfile.keystore" keystorePass="password" > socket.appReadBufSize="81920" > socket.appWriteBufSize="81920" socket.rxBufSize="251880" > socket.txBufSize="438000"> > maxConcurrentStreamExecution="200" > maxConcurrentStreams="200" > className="org.apache.coyote.http2.Http2Protocol"/> > > Do you actually need all those values and where do they come from in the first place ? Do you understand what they do ? The buffer sizes seem way too large, and maxConcurrentStreamExecution is as well (multiplexing 200 threads output over a single socket is not a very good idea usually, which is why the setting was added). Rémy > > This instance mainly serves concurrent POST request which will have payload > of size, approx 1000-1500, which can be verified by tomcat logs > > org.apache.coyote.http2.Http2Parser.validateFrame Connection [0], Stream > [19], Frame type [DATA], Flags [1], Payload size [*1195*] > > We tested our server with the help of h2load as follows, > > h2load -n100 -c1 -m100 https://localhost:9191/ -d '/agentRequest.txt' > > We are getting this error as follows, > > > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch Connection [0] > java.io.IOException: Unable to unwrap data, invalid status > [BUFFER_OVERFLOW] > at > org.apache.tomcat.util.net > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) > at > org.apache.tomcat.util.net > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) > at > java.base/sun.nio.ch > .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552) > at > java.base/sun.nio.ch > .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) > at > java.base/sun.nio.ch > .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) > at > org.apache.tomcat.util.net > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027) > at > org.apache.tomcat.util.net > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > at > org.apache.tomcat.util.net > .SecureNio2Channel.read(SecureNio2Channel.java:1067) > at > org.apache.tomcat.util.net > .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153) > at > org.apache.tomcat.util.net > .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026) > at > org.apache.tomcat.util.net > .SocketWrapperBase.read(SocketWrapperBase.java:1012) > at > > org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61) > at > org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69) > at > > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322) > at > > org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37) > at > > org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) > at > > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53) > at > > org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) > at > org.apache.tomcat.util.net > .Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769) > at > org.apache.tomcat.util.net > .SocketProcessorBase.run(SocketProcessorBase.java:49) > at > org.apache.tomcat.util.net > .AbstractEndpoint.processSocket(AbstractEndpoint.java:1048) > at > org.apache.tomcat.util.net > .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:116) > at > org.apache.tomcat.util.net > .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:109) > at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) > > Why is this error is thrown? How can I configure tomcat to handle > concurrent POST requests which have a decent payload? > >
