[
https://issues.apache.org/jira/browse/HTTPCORE-530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16539031#comment-16539031
]
Michael Leith commented on HTTPCORE-530:
----------------------------------------
Hi Oleg,
I completely agree with looking into the application code first. I've attached
a stack trace below & the code given above is close to verbatim what we're
running. If you see something I'd be delighted to know, but nothing jumps out
to me.
Somehow the HeaderGroup is being accessed from multiple threads. On our side
the only place we touch the headers directly is in the post method (as can be
seen in the code snippet). I'll take a look again, but is there anywhere you
can point me to where the headers could be accessed outside of the posting
thread?
Caused by: java.lang.NullPointerException
at
org.apache.http.message.BasicListHeaderIterator.filterHeader(BasicListHeaderIterator.java:130)
~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
at
org.apache.http.message.BasicListHeaderIterator.findNext(BasicListHeaderIterator.java:110)
~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
at
org.apache.http.message.BasicListHeaderIterator.<init>(BasicListHeaderIterator.java:86)
~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
at org.apache.http.message.HeaderGroup.iterator(HeaderGroup.java:287)
~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
at
org.apache.http.message.AbstractHttpMessage.headerIterator(AbstractHttpMessage.java:155)
~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
at
org.apache.http.impl.DefaultConnectionReuseStrategy.keepAlive(DefaultConnectionReuseStrategy.java:86)
~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:274)
~[Apache-HttpComponents-HttpClient-4.4.x.jar:?]
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
~[Apache-HttpComponents-HttpClient-4.4.x.jar:?]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
~[Apache-HttpComponents-HttpClient-4.4.x.jar:?]
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
~[Apache-HttpComponents-HttpClient-4.4.x.jar:?]
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
~[Apache-HttpComponents-HttpClient-4.4.x.jar:?]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
~[Apache-HttpComponents-HttpClient-4.4.x.jar:?]
at <our post function>
> NullPointerException in HttpCore >= 4.4.5
> -----------------------------------------
>
> Key: HTTPCORE-530
> URL: https://issues.apache.org/jira/browse/HTTPCORE-530
> Project: HttpComponents HttpCore
> Issue Type: Bug
> Components: HttpCore
> Affects Versions: 4.4.9
> Reporter: Michael Leith
> Priority: Major
> Labels: bugfix
>
> h2. Details
> We're getting a NullPointerException exception in versions >= 4.4.5. The
> exception occurs ~20 times per 10m requests in DefaultConnectionReuseStrategy
> when calling request.headerIterator. The change that introduced it was
> swapping to checking the request for keepAlive instead of just the response
> [1] in version 4.4.5. From looking at the source in github, this change is
> still in version 5.0, so the NullPointerException may be in versions > 4.4.9,
> but that's the latest version I have tested on so far.
> From testing the source of the exception is multiple threads accessing the
> HeaderGroup, with one thread calling setHeader & the other calling
> iterator(name). The HeaderGroup is backed by a thread-unsafe ArrayList, which
> is the source of the exception. I have a fix for this, is there somewhere I
> can submit a pull request?
> The fix is to swap to a thread safe CopyOnWriteArrayList. This will have a
> performance impact, but it should be minor since the list is small (16
> indices) and headers don't appear to be modified frequently.
> [1]
> [https://github.com/apache/httpcomponents-core/blob/ff381e315a165047e6c2db9b1e749b47cad31405/httpcore5/src/main/java/org/apache/hc/core5/http/impl/DefaultConnectionReuseStrategy.java#L84]
> h2. Client details
> Here's a simplified version of our client. From comparing it to the examples
> provided in the docs there isn't anything that appears to be thread-unsafe
> aside from the shared HttpClientContext, which I've confirmed is not the root
> cause by making it thread local.
> // single threaded
> // Isn't necessarily thread safe, I've tried moving it into the "post"
> method to see if it is the cause, but the exceptions persist.
> private final HttpClientContext context = HttpClientContext.adapt(new
> BasicHttpContext());
> final RegistryBuilder<ConnectionSocketFactory> registryBuilder =
> RegistryBuilder.<ConnectionSocketFactory>create().register("https",
> sslSocketFactory.get());.
> final PoolingHttpClientConnectionManager connectionManager = new
> PoolingHttpClientConnectionManager(registryBuilder.build())
> connectionManager.setMaxTotal(connectionPoolSize);
> connectionManager.setDefaultMaxPerRoute(maxConnectionsPerRoute);
> connectionManager.setValidateAfterInactivity(duration);
> final CloseableHttpClient client = HttpClients.custom()
> .setConnectionManager(connectionManager)
> .setKeepAliveStrategy((response, context) -> keepAliveDuration.toMillis())
> .setUserAgent(userAgent)
> .build();
> // removing this does not fix the NullPointerException
> executorService.scheduleAtFixedRate(
> () ->
> connectionManager.closeIdleConnections(idleConnectionExpiry.toMillis(),
> TimeUnit.MILLISECONDS),
> Duration.ofSeconds(30).toMillis(),
> Duration.ofSeconds(30).toMillis(),
> TimeUnit.MILLISECONDS);
> // Called from multiple threads
> Result post(uri, config, entity, hostName) {
> final HttpPost req = new HttpPost(uri);
> req.setConfig(config);
> req.setEntity(entity);
> req.addHeader(HttpHeaders.HOST, hostName);
> try (CloseableHttpResponse response = client.execute(req, context))
> { // handle response }
> catch(...)
> { // handle exception }
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
