[
https://issues.apache.org/jira/browse/NIFI-9448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-9448:
-----------------------------------
Description:
NiFi Site-to-Site communication over HTTP relies on {{SiteToSiteRestApiClient}}
to handle requests and responses using the Apache HttpComponents library. The
HttpComponents library maintains a connection pool for processing HTTP
transactions.
In the course of sending or receiving files, NiFi starts a background thread to
make periodic requests to the remote NiFi system in order to extend the current
transaction. The {{SiteToSiteRestApiClient}} stops the background thread after
completing a transaction. In some cases, the request to extend the transaction
can occur after the HttpComponents connection pool is shutdown, resulting in an
{{{}IllegalStateException{}}}. The background thread treats all exceptions as
failure conditions and then attempts to close the S2S HTTP client itself. This
causes subsequent retry requests using the same {{SiteToSiteRestApiClient}} to
fail with the same {{IllegalStateException}} indicating that the connection
pool is shutdown.
The behavior of the extend transaction command should be changed to avoid
closing the S2S HTTP client when encountering an {{{}IllegalStateException{}}}.
This approach will support the potential for subsequent retries to work or fail
based on existing timeout configuration settings.
The following log messages provide a stack trace of the
{{IllegalStateException}} in the extend transaction command and subsequent
exception in the Remote Process Group Port connection.
{noformat}
WARN org.apache.nifi.remote.util.SiteToSiteRestApiClient: Failed to extend
transaction ttl
java.lang.IllegalStateException: Connection pool shut down
at org.apache.http.util.Asserts.check(Asserts.java:34)
at
org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:191)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:267)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at
org.apache.nifi.remote.util.SiteToSiteRestApiClient.extendTransaction(SiteToSiteRestApiClient.java:1028)
at
org.apache.nifi.remote.util.SiteToSiteRestApiClient.lambda$startExtendingTtl$0(SiteToSiteRestApiClient.java:990)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
INFO org.apache.http.impl.execchain.RetryExec: I/O exception
(org.apache.http.NoHttpResponseException) caught when processing request: The
target server failed to respond
INFO org.apache.http.impl.execchain.RetryExec: Retrying request
ERROR org.apache.nifi.remote.StandardRemoteGroupPort:
RemoteGroupPort[name=REMOTE_PORT,targets=URL] failed to communicate with remote
NiFi instance due to java.lang.IllegalStateException: Connection pool shut down
{noformat}
was:
NiFi Site-to-Site communication over HTTP relies on {{SiteToSiteRestApiClient}}
to handle requests and responses using the Apache HttpComponents library. The
HttpComponents library maintains a connection pool for processing HTTP
transactions.
In the course of sending or receiving files, NiFi starts a background thread to
make periodic requests to the remote NiFi system in order to extend the current
transaction. The {{SiteToSiteRestApiClient}} stops the background thread after
completing a transaction. In some cases, the request to extend the transaction
can occur after the HttpComponents connection pool is shutdown, resulting in an
{{{}IllegalStateException{}}}. The background thread treats all exceptions as
failure conditions and then attempts to close the S2S HTTP client itself. This
causes subsequent retry requests using the same {{SiteToSiteRestApiClient}} to
fail with the same {{IllegalStateException}} indicating that the connection
pool is shutdown.
The behavior of the extend transaction command should be changed to avoid
closing the S2S HTTP client when encountering an {{IllegalStateException}}.
This approach will support the potential for subsequent retries to work or fail
based on existing timeout configuration settings.
The following log messages provide a stack trace of the
{{IllegalStateException}} in the extend transaction command and subsequent
exception in the Remote Process Group Port connection.
{noformat}
WARN org.apache.nifi.remote.util.SiteToSiteRestApiClient: Failed to extend
transaction ttl
java.lang.IllegalStateException: Connection pool shut down
at org.apache.http.util.Asserts.check(Asserts.java:34)
at
org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:191)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:267)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at
org.apache.nifi.remote.util.SiteToSiteRestApiClient.extendTransaction(SiteToSiteRestApiClient.java:1028)
at
org.apache.nifi.remote.util.SiteToSiteRestApiClient.lambda$startExtendingTtl$0(SiteToSiteRestApiClient.java:990)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-11-10 08:17:10,352 INFO org.apache.http.impl.execchain.RetryExec: I/O
exception (org.apache.http.NoHttpResponseException) caught when processing
request: The target server failed to respond
INFO org.apache.http.impl.execchain.RetryExec: Retrying request
ERROR org.apache.nifi.remote.StandardRemoteGroupPort:
RemoteGroupPort[name=REMOTE_PORT,targets=URL] failed to communicate with remote
NiFi instance due to java.lang.IllegalStateException: Connection pool shut down
{noformat}
> Potential IllegalStateException when S2S HTTP Client Shutdown
> -------------------------------------------------------------
>
> Key: NIFI-9448
> URL: https://issues.apache.org/jira/browse/NIFI-9448
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.15.0
> Reporter: David Handermann
> Assignee: David Handermann
> Priority: Major
>
> NiFi Site-to-Site communication over HTTP relies on
> {{SiteToSiteRestApiClient}} to handle requests and responses using the Apache
> HttpComponents library. The HttpComponents library maintains a connection
> pool for processing HTTP transactions.
> In the course of sending or receiving files, NiFi starts a background thread
> to make periodic requests to the remote NiFi system in order to extend the
> current transaction. The {{SiteToSiteRestApiClient}} stops the background
> thread after completing a transaction. In some cases, the request to extend
> the transaction can occur after the HttpComponents connection pool is
> shutdown, resulting in an {{{}IllegalStateException{}}}. The background
> thread treats all exceptions as failure conditions and then attempts to close
> the S2S HTTP client itself. This causes subsequent retry requests using the
> same {{SiteToSiteRestApiClient}} to fail with the same
> {{IllegalStateException}} indicating that the connection pool is shutdown.
> The behavior of the extend transaction command should be changed to avoid
> closing the S2S HTTP client when encountering an
> {{{}IllegalStateException{}}}. This approach will support the potential for
> subsequent retries to work or fail based on existing timeout configuration
> settings.
> The following log messages provide a stack trace of the
> {{IllegalStateException}} in the extend transaction command and subsequent
> exception in the Remote Process Group Port connection.
> {noformat}
> WARN org.apache.nifi.remote.util.SiteToSiteRestApiClient: Failed to extend
> transaction ttl
> java.lang.IllegalStateException: Connection pool shut down
> at org.apache.http.util.Asserts.check(Asserts.java:34)
> at
> org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:191)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:267)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
> at
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.extendTransaction(SiteToSiteRestApiClient.java:1028)
> at
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.lambda$startExtendingTtl$0(SiteToSiteRestApiClient.java:990)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> INFO org.apache.http.impl.execchain.RetryExec: I/O exception
> (org.apache.http.NoHttpResponseException) caught when processing request: The
> target server failed to respond
> INFO org.apache.http.impl.execchain.RetryExec: Retrying request
> ERROR org.apache.nifi.remote.StandardRemoteGroupPort:
> RemoteGroupPort[name=REMOTE_PORT,targets=URL] failed to communicate with
> remote NiFi instance due to java.lang.IllegalStateException: Connection pool
> shut down
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
