Chris Nauroth created HIVE-26677:
------------------------------------
Summary: Constrain available processors to Jetty during test runs
to prevent thread exhaustion.
Key: HIVE-26677
URL: https://issues.apache.org/jira/browse/HIVE-26677
Project: Hive
Issue Type: Test
Components: Test
Reporter: Chris Nauroth
As described during a [release candidate
vote|https://lists.apache.org/thread/8qjf7x9t9v09d79hlzh712ls4zthdwrh]:
HIVE-24484 introduced a change to limit {{hive.server2.webui.max.threads}} to
4. Jetty enforces thread leasing to warn or abort if there aren't enough
threads available [1]. During startup, it attempts to lease a thread per NIO
selector [2]. By default, the number of NIO selectors to use is determined
based on available CPUs [3]. This is mostly a passthrough to
{{Runtime.availableProcessors()}} [4]. In my case, running on a machine with 16
CPUs, this ended up creating more than 4 selectors, therefore requiring more
than 4 threads and violating the lease check. I was able to work around this by
passing the {{JETTY_AVAILABLE_PROCESSORS}} system property to constrain the
number of CPUs available to Jetty.
Since we are intentionally constraining the pool to 4 threads during itests,
let's also limit {{JETTY_AVAILABLE_PROCESSORS}} in {{maven.test.jvm.args}} of
the root pom.xml, so that others don't run into this problem later.
[1]
https://github.com/eclipse/jetty.project/blob/jetty-9.4.40.v20210413/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ThreadPoolBudget.java#L165
[2]
https://github.com/eclipse/jetty.project/blob/jetty-9.4.40.v20210413/jetty-io/src/main/java/org/eclipse/jetty/io/SelectorManager.java#L255
[3]
https://github.com/eclipse/jetty.project/blob/jetty-9.4.40.v20210413/jetty-io/src/main/java/org/eclipse/jetty/io/SelectorManager.java#L79
[4]
https://github.com/eclipse/jetty.project/blob/jetty-9.4.40.v20210413/jetty-util/src/main/java/org/eclipse/jetty/util/ProcessorUtils.java#L45
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
