[
https://issues.apache.org/jira/browse/HDDS-16121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18103576#comment-18103576
]
Ivan Andika commented on HDDS-16121:
------------------------------------
[~weichiu] Thanks for checking this,
> IMO, Jetty 12 + virtual threads is a long term roadmap, not immediately
> actionable
> Also note: Jetty 12 requires JDK17. But Ozone is built with JDK8 now. We need
> community consensus to move up to JDK17.
There is already a plan to make JDK17 for server component (HDDS-14439)
including S3G, we can then upgrade to Jetty to 12+. cc: [~adoroszlai]
> Servlet 3.1 Asynchronous IO is fine, but we don't have expertise in this
> framework and not as popular as Jetty
Agreed, that's why Servlet 3.1 Asynchronous IO is not the way to go since I
don't think the community has any experience.
> Let's say we upgrade Jetty 9 to Jetty12. Does using Virtual Threads require
> building Ozone with JDK21+? Or does it only require JDK21+ runtime? (assuming
> Jetty uses Virtual Threads and no direct invocation of Virtual Threads API
> from Ozone)
The hope is whether we can detect it in runtime, similar to how Netty detect
whether a particular runtime can use Epoll or not. So JDK17 runtime will
fallback to the non-virtual threads path.
> Use virtual threads for S3G
> ---------------------------
>
> Key: HDDS-16121
> URL: https://issues.apache.org/jira/browse/HDDS-16121
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> Currently, S3G use the blocking thread-per-request threading model. This has
> the advantage of being simple to implement (as opposed to the asynchronous IO
> event loop model).
> However, we encountered performance issues where S3G HTTP threads are blocked
> which causes the threads to hit hadoop.http.max.threads and therefore causes
> the subsequent requests to be waiting in the waitingTaskCount which increases
> the latenncy. This issue can happen even if the downstream services are fine
> (i.e. Ozone cluster is not slow or stuck).
> For example, if an HTTP client stalled while downloading a key from S3G (e.g.
> client machine is slow, etc), the S3G thread will be blocked serving this
> read request until hadoop.http.idle_timeout.ms which triggers the "Idle
> timeout expired". Check the Jetty IdleTimeout.java for the logic for idle,
> the idea is if no data is written in the socket for a while (e.g. notIdle is
> not invoked), a scheduled task will fail this request to prevent request
> hogging the thread for too long.
> {code:java}
> SEVERE: An I/O error has occurred while writing a response message entity to
> the container output stream.
> org.glassfish.jersey.server.internal.process.MappableException:
> java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
> expired: 60000/60000 ms
> at
> org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:67)
> at
> org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
> at
> org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
> at
> org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:649)
> at
> org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:380)
> at
> org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:370)
> {code}
> Meaning that a single thread can block a single thread although there are
> still a lot of CPU capacity left.
> One idea is to migrate the Jetty threading model to Servlet 3.1 Asynchronous
> IO which implements WriteListener and ReadListener, this would prevent a
> single request holding the thread for too long. However, this requires
> changes in the S3G implementation which is tedious and risky.
> A better solution is to use virtual threads since we already support JDK21+
> version. Since virtual threads are not real threads, a single OS thread can
> be mapped to multiple virtual threads and blocked virtual threads should not
> cause OS thread to be blocked since OS thread will pick another virtual
> thread which can do useful work. Additionally, since the threading model of
> virtual thread is similar to OS thread, we don't need to change threading
> model so the risk is lower. Of course, there are some risks in virtual
> threads (i.e. thread pinning, overloading the downstream services, etc), so
> we should still evaluate it or we can make a flag to whether enable or
> disable virtual threads and do canary testing.
> This requires Jetty upgrade, but Jetty upgrade is overdo (we are still using
> a very old 9.x Jetty version) and we should do it asap.
> Note: I would like to hear from people who have worked with virtual threads
> before.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
