-----Original Message-----
From: Mark Thomas <ma...@apache.org>
Sent: Wednesday, June 16, 2021 9:29 PM
To: users@tomcat.apache.org
Subject: Re: Trouble with HTTP/2 during concurrent bulk data transfer (server
-> client)
On 16/06/2021 15:05, Deshmukh, Kedar wrote:
> Dear Tomcat users/dev team,
>
> We are understanding the impact of HTTP/2 in our application as HTTP/2
> provides better throughput and performance.
I'd be wary of making such sweeping statements. HTTP/2 has some advantages and
some disadvantages. Generally, the advantages will outweigh the disadvantages
but that will not always be the case.
[Kedar] - Okay.
> Before directly tuning
> HTTP/2 in application, we thought of analyzing certain use cases which
> our application demands in standalone environment.
>
> Our use case is very simple. Java based standalone client is making
> simple POST request to the server and server read the file name from
> the request and push the requested file to client. Here, client can
> request multiple files same time by sending multiple requests
> concurrently, so server should be able to send multiple files
> concurrently depending on configured concurrency level.
>
> Currently, in this test only single client is making requests to the
> server with concurrency 5. Server is not overloaded and not performing
> any other tasks. Machine has more than 500GB empty space and not
> running any heavy applications.
>
> Test:
>
> We used different set of files for this test. Files with sizes between
> 1GB - 5GB and concurrency > 5. We are using traditional connector
> protocol HTTP11NIOProtocol with HTTP/2 is turned on.
>
> Observations:
>
> HTTP/1.1 - With HTTP/1.1 given sample code works fine. Only drawback
> here is it opens multiple TCP connections to satisfy HTTP/1.1
>
> HTTP/2 - With HTTP/2, it is expected to be only one TCP connection and
> multiple streams to handle the traffic. Tomcat HTTP/2 debug logs
> suggest that only one connection being used and multiple streams are
> spawned as expected. So far everything is fine. But sample code does
> not work consistently with higher concurrency (> 3). We captured the
> stack trace of tomcat process which is attached here. Couple of tomcat
> threads are waiting to acquire semaphore for socket write operation.
> When write operation is stuck servlet is not able to push any data to
> client and client is also stuck waiting for more data. I don't see any
> error/exception at the client/server.
That looks / sounds like there is a code path - probably an error condition ? -
where the semaphore isn't released.
[Kedar] Semaphore never released. As mentioned in sample code concurrency level
is 5 so If you observe attached tomcat stack trace, you would find following
very similar 4 threads which are waiting on semaphore which was acquired by
other thread "Thread-9".
"Thread-7" #58 daemon prio=5 os_prio=0 cpu=218.75ms elapsed=105.35s
tid=0x000001844997a800 nid=0x4bc0 waiting on condition [0x00000027cc7fe000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.10/Native Method)
- parking to wait for <0x00000006158518b0> (a
java.util.concurrent.Semaphore$NonfairSync)
at
java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.10/LockSupport.java:234)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(java.base@11.0.10/AbstractQueuedSynchronizer.java:1079)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(java.base@11.0.10/AbstractQueuedSynchronizer.java:1369)
at
java.util.concurrent.Semaphore.tryAcquire(java.base@11.0.10/Semaphore.java:415)
at
org.apache.tomcat.util.net.SocketWrapperBase.vectoredOperation(SocketWrapperBase.java:1426)
"Thread-9" #59 daemon prio=5 os_prio=0 cpu=187.50ms elapsed=105.35s
tid=0x000001844997b800 nid=0x4b48 in Object.wait() [0x00000027cc9fe000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.10/Native Method)
- waiting on <0x000000061ed36de0> (a
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState)
at
org.apache.tomcat.util.net.SocketWrapperBase.vectoredOperation(SocketWrapperBase.java:1457)
The other possibility is related to the HTTP/2 flow control windows. If
something goes wrong with the management of the flow control window for the
connection it would block everything.
[Kedar] - Okay. But I did not see anything in the logs or WINDOWS_UPDATE frame
related errors.
> streamReadTimeout and streamWriteTimeout are configured as -1 so they
> are infinitely waiting for the write semaphore.
That is generally a bad idea. By all means set it high but an infinite timeout
is going to cause problems - particularly if clients just drop off the network.
[Kedar] Okay. I will look into this, however I don't think this change would
make impact on the outcome. We might see sockettimeout exception and transfer
would fail.
> Outcome of this is client is able to receive only partial data from
> server and at some point server stuck to send any more data.
>
> We also tried IOUtils file transfer related APIs still it didn't help.
> I have also tried with Async non-blocking IO but the observations are same.
Generally, the simpler you keep the test case, the easier it is for us to work
with. Non-async and no external IO libraries is better.
[Kedar] IOUtils - I was referring from
org.apache.tomcat.util.http.fileupload.IOUtils which is part of tomcat. Async
is requirement for us.
> Our actual requirement is very similar where java based http client
> would request bulk data concurrently from server and server should
> push that without any trouble. But, it is not limited to files only.
> Server can push serialized java bulk objects over the stream concurrently.
The content type should not make any difference to Tomcat. Static files vs
dynamic content would make a difference.
[Kedar] - Okay.
> Note that sample code works fine most of the time if I enable HTTP/2
> logs either in client or tomcat. So I would suggest not to turn on
> HTTP/2 debug logs to conclude anything.
That suggests a timing issue of some sort.
[Kedar] - Yes, I was also thinking the same, somehow I did not find any
particular reason.
HTTP/2 is significantly more complex than HTTP/1.1 because you have multiple
independent application threads all trying to write to the same socket and
Tomcat has to track and allocate flow control window allocations both for
individual streams and the overall connection.
[Kedar] - Agreed
> Following components are used in sample code for the test
>
> 1. Client - Java 11.0.10 httpclient - (client\Client.java)
>
> 2. Server - Tomcat 9.0.46
>
> 3. Servlet - AsyncServlet - (server\Server.java)
>
> 4. Operating system - Windows 10
>
> 5. Machine specifications - 32GB RAM and 500GB open space.
>
> 5. Latency - None, client and server are running on same machine
>
> 6. Set of files - You can use any random files whose sizes are between
> 1GB-5GB to reproduce the issue.
>
> Refer attachment for
>
> 1. Client side code
>
> 2. Server side servlet
>
> 3. server.xml
>
> 4. Tomcat Stacktrace
>
> 5. Tomcat server logs
Thanks. That is all useful information.
> Could you please go through sample code along with server.xml. Here
> are my few questions
>
> 1. Why HTTP/2 is failing for such use case where large files are
> concurrently pushing to the client. I believe this is a very common
> use case must have be assessed earlier.
At this point, no idea. We'll need to see if we can recreate the problem you
describe first. Then we can try and identify root causes.
[Kedar] - It is very much reproducible with attached sample.
> 2. Connector configuration in server.xml is correct for HTTP/2 ?
See previous comments regarding timeouts.
Disabling the overhead protection shouldn't be necessary but often is and looks
to have been done correctly. The Tomcat releases scheduled for July have some
significant improvements in this area.
[Kedar] - Thanks for the update.
> 3. Does AsyncServlet is written properly for such type of use case ?
> If not, then please suggest correct way to manage it.
Looks OK but is entirely pointless. It adds unneeded complexity. All you are
doing is adding a context switch from original thread used for
doPost() to a new thread.
[Kedar] - Okay
> 4. If you are already aware of similar problem, can you point out any
> alternatives or recommendations.
Nothing comes to mind and a quick scan of the changelog since 9.0.46 doesn't
highlight anything that might be relevant.
I have one additional question at this point. How easy is this issue to
reproduce? Does it happen every time? In 10% of requests? 1% ?
[Kedar] It is reproducible 9/10 times in my environment. So 90% time it is
reproducible when concurrency is 5 or more and file sizes are between 1GB-5GB.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
