[
https://issues.apache.org/jira/browse/HDDS-16121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-16121:
-------------------------------
Description:
Currently, S3G use the blocking thread-per-request threading model. This has
the advantage of being simple to implement (as opposed to the asynchronous
event loop model).
However, we encountered performance issues where S3G HTTP threads are blocked
which causes the threads to hit hadoop.http.max.threads and therefore causes
the subsequent requests to be waiting in the waitingTaskCount which increases
the latenncy. This issue can happen even if the downstream services are fine
(i.e. Ozone cluster is not slow or stuck).
For example, if an HTTP client stalled while downloading a key from S3G (e.g.
client machine is slow, etc), the S3G thread will be blocked serving this read
request until hadoop.http.idle_timeout.ms which triggers the "Idle timeout
expired". Check the Jetty IdleTimeout.java for the logic for idle, the idea is
if no data is written in the socket for a while (e.g. notIdle is not invoked),
a scheduled task will fail this request to prevent request hogging the thread
for too long.
{code:java}
SEVERE: An I/O error has occurred while writing a response message entity to
the container output stream.
org.glassfish.jersey.server.internal.process.MappableException:
java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
expired: 60000/60000 ms
at
org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:67)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
at
org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
at
org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:649)
at
org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:380)
at
org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:370)
{code}
Meaning that a single thread can block a single thread although there are still
a lot of CPU capacity left.
One idea is to migrate the Jetty threading model to Servlet 3.1 Asynchronous IO
which implements WriteListener and ReadListener, this would prevent a single
request holding the thread for too long. However, this requires changes in the
S3G implementation which is tedious and risky.
A better solution is to use virtual threads. Since virtual threads are not real
threads, a single OS thread can be mapped to multiple virtual threads and
blocked virtual threads should not cause OS thread to be blocked since OS
thread will pick another virtual thread which can do useful work. Additionally,
since the threading model of virtual thread is similar to OS thread, we don't
need to change threading model so the risk is lower. Of course, there are some
risks in virtual threads (i.e. thread pinning, overloading the downstream
services, etc), so we should still evaluate it or we can make a flag to whether
enable or disable virtual threads and do canary testing.
This requires Jetty upgrade, but Jetty upgrade is overdo (we are still using a
very old 9.x Jetty version) and we should do it asap.
was:
Currently, S3G use the blocking thread-per-request threading model. This has
the advantage of being simple to implement (as opposed to the asynchronous
event loop model).
However, we encountered performance issues where S3G HTTP threads are blocked
which causes the threads to hit hadoop.http.max.threads and therefore causes
the subsequent requests to be waiting in the waitingTaskCount which increases
the latenncy. This issue can happen even if the downstream services are fine
(i.e. Ozone cluster is not slow or stuck).
For example, if an HTTP client stalled while downloading a key from S3G (e.g.
client machine is slow, etc), the S3G thread will be blocked serving this read
request until hadoop.http.idle_timeout.ms which triggers the "Idle timeout
expired". Check the Jetty IdleTimeout.java for the logic for idle, the idea is
if no data is written in the socket for a while (e.g. notIdle is not invoked),
a scheduled task will fail this request to prevent request hogging the thread
for too long.
{code:java}
SEVERE: An I/O error has occurred while writing a response message entity to
the container output stream.
org.glassfish.jersey.server.internal.process.MappableException:
java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
expired: 60000/60000 ms
at
org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:67)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
at
org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
at
org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:649)
at
org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:380)
at
org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:370)
{code}
Meaning that a single thread can block a single thread although there are still
a lot of CPU capacity left.
One idea is to migrate the Jetty threading model to Servlet 3.1 Asynchronous IO
which implements WriteListener and ReadListener, this would prevent a single
request holding the thread for too long. However, this requires changes in the
S3G implementation which is tedious and risky.
A better solution is to use virtual threads. Since virtual threads are not real
threads, a single OS thread can be mapped to multiple virtual threads and
blocked virtual threads should not cause OS thread to be blocked since OS
thread will pick another virtual thread which can do useful work. Additionally,
since the threading model of virtual thread is similar to OS thread, we don't
need to change threading model so the risk is lower. Of course, there are some
risks in virtual threads (i.e. thread pinning, overloading the downstream
services, etc), so we should still evaluate it or we can make a flag to whether
enable or disable virtual threads and do canary testing.
This requires Jetty upgrade, but Jetty upgrade is overdo (we are still using a
very old 9.x Jetty version) and we should do it asap.
> Use virtual threads for S3G
> ---------------------------
>
> Key: HDDS-16121
> URL: https://issues.apache.org/jira/browse/HDDS-16121
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> Currently, S3G use the blocking thread-per-request threading model. This has
> the advantage of being simple to implement (as opposed to the asynchronous
> event loop model).
> However, we encountered performance issues where S3G HTTP threads are blocked
> which causes the threads to hit hadoop.http.max.threads and therefore causes
> the subsequent requests to be waiting in the waitingTaskCount which increases
> the latenncy. This issue can happen even if the downstream services are fine
> (i.e. Ozone cluster is not slow or stuck).
> For example, if an HTTP client stalled while downloading a key from S3G (e.g.
> client machine is slow, etc), the S3G thread will be blocked serving this
> read request until hadoop.http.idle_timeout.ms which triggers the "Idle
> timeout expired". Check the Jetty IdleTimeout.java for the logic for idle,
> the idea is if no data is written in the socket for a while (e.g. notIdle is
> not invoked), a scheduled task will fail this request to prevent request
> hogging the thread for too long.
> {code:java}
> SEVERE: An I/O error has occurred while writing a response message entity to
> the container output stream.
> org.glassfish.jersey.server.internal.process.MappableException:
> java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
> expired: 60000/60000 ms
> at
> org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:67)
> at
> org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
> at
> org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
> at
> org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:649)
> at
> org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:380)
> at
> org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:370)
> {code}
> Meaning that a single thread can block a single thread although there are
> still a lot of CPU capacity left.
> One idea is to migrate the Jetty threading model to Servlet 3.1 Asynchronous
> IO which implements WriteListener and ReadListener, this would prevent a
> single request holding the thread for too long. However, this requires
> changes in the S3G implementation which is tedious and risky.
> A better solution is to use virtual threads. Since virtual threads are not
> real threads, a single OS thread can be mapped to multiple virtual threads
> and blocked virtual threads should not cause OS thread to be blocked since OS
> thread will pick another virtual thread which can do useful work.
> Additionally, since the threading model of virtual thread is similar to OS
> thread, we don't need to change threading model so the risk is lower. Of
> course, there are some risks in virtual threads (i.e. thread pinning,
> overloading the downstream services, etc), so we should still evaluate it or
> we can make a flag to whether enable or disable virtual threads and do canary
> testing.
> This requires Jetty upgrade, but Jetty upgrade is overdo (we are still using
> a very old 9.x Jetty version) and we should do it asap.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
