[jira] [Commented] (SOLR-17707) HttpJdkSolrClient threads stuck on exceptions
[
https://issues.apache.org/jira/browse/SOLR-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100061#comment-18100061
]
ASF subversion and git services commented on SOLR-17707:
Commit 39238d9de71f74942344cd08fb04c56994ebd471 in solr's branch
refs/heads/branch_9x from Serhiy Bzhezytskyy
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=39238d9de71 ]
SOLR-17707: HttpJdkSolrClient fix thread leak when async request fails (#4639)
Fixed HttpJdkSolrClient leaking an executor thread when an async request's
connection failed while its body was still being written; enough such failures
could exhaust the client's thread pool.
(cherry picked from commit aa9e3f1a7841e765d3806951e2d6f0460f6a057d)
(cherry picked from commit ca51b1b33c2eee6a69cd1eeb79fe29f81fb9ab3e)
> HttpJdkSolrClient threads stuck on exceptions
> -
>
> Key: SOLR-17707
> URL: https://issues.apache.org/jira/browse/SOLR-17707
> Project: Solr
> Issue Type: Bug
> Components: SolrJ
>Affects Versions: 9.6.1
>Reporter: Artem Abeleshev
>Assignee: David Smiley
>Priority: Major
> Labels: SolrJ, pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> There is a problem with _HttpJdkSolrClient_ used by {_}SolrJ{_}. In case of
> exception occurs on sending data (in my case it was {_}connection reset by
> peer{_}) the thread from the http client executor pool get stuck. Internally,
> _HttpJdkSolrClient_ using _PipedOutputStream_ and _PipedInputStream_ and in
> case of exception it ends up waiting forever. Seems like _PipedOutputStream_
> buffer remains full and _PipedInputStream_ stuck in a waiting state waiting
> for the avaialble space to write. This will consume one thread from the pool
> making it blocked. By default, _HttpJdkSolrClient_ created with an executor
> of a _4_ threads, if all of them are blocked the _SolrJ_ will be unable to
> work.
> There seems to be no problem with that if switching to use _Http2SolrClient_
> instead of _HttpJdkSolrClient_ as _Http2SolrClient_ doesn't use
> _PipedOutputStream_ and {_}PipedInputStream{_} facility.
> Here is a part of threaddump showing the http client threads get stuck:
> {code:java}
> "HttpJdkSolrClient-1-thread-1-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=52 TIMED_WAITING on
> java.io.PipedInputStream@67168d76
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@67168d76
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@bdec8d6
> "HttpJdkSolrClient-1-thread-2-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=53 TIMED_WAITING on
> java.io.PipedInputStream@571e7f87
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@571e7f87
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@7e6cf6f2
> "HttpJdkSolrClient-1-thread-3-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=54 TIMED_WAITING on
> java.io.PipedInputStream@497c4c19
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@497c4c19
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(
[jira] [Commented] (SOLR-17707) HttpJdkSolrClient threads stuck on exceptions
[
https://issues.apache.org/jira/browse/SOLR-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18099878#comment-18099878
]
Jan Høydahl commented on SOLR-17707:
[~dsmiley] I assigned to you as you merged the PR, please resove this JIRA as
well...
> HttpJdkSolrClient threads stuck on exceptions
> -
>
> Key: SOLR-17707
> URL: https://issues.apache.org/jira/browse/SOLR-17707
> Project: Solr
> Issue Type: Bug
> Components: SolrJ
>Affects Versions: 9.6.1
>Reporter: Artem Abeleshev
>Assignee: David Smiley
>Priority: Major
> Labels: SolrJ, pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> There is a problem with _HttpJdkSolrClient_ used by {_}SolrJ{_}. In case of
> exception occurs on sending data (in my case it was {_}connection reset by
> peer{_}) the thread from the http client executor pool get stuck. Internally,
> _HttpJdkSolrClient_ using _PipedOutputStream_ and _PipedInputStream_ and in
> case of exception it ends up waiting forever. Seems like _PipedOutputStream_
> buffer remains full and _PipedInputStream_ stuck in a waiting state waiting
> for the avaialble space to write. This will consume one thread from the pool
> making it blocked. By default, _HttpJdkSolrClient_ created with an executor
> of a _4_ threads, if all of them are blocked the _SolrJ_ will be unable to
> work.
> There seems to be no problem with that if switching to use _Http2SolrClient_
> instead of _HttpJdkSolrClient_ as _Http2SolrClient_ doesn't use
> _PipedOutputStream_ and {_}PipedInputStream{_} facility.
> Here is a part of threaddump showing the http client threads get stuck:
> {code:java}
> "HttpJdkSolrClient-1-thread-1-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=52 TIMED_WAITING on
> java.io.PipedInputStream@67168d76
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@67168d76
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@bdec8d6
> "HttpJdkSolrClient-1-thread-2-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=53 TIMED_WAITING on
> java.io.PipedInputStream@571e7f87
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@571e7f87
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@7e6cf6f2
> "HttpJdkSolrClient-1-thread-3-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=54 TIMED_WAITING on
> java.io.PipedInputStream@497c4c19
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@497c4c19
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@7a2b5f30
> "HttpJdkSo
[jira] [Commented] (SOLR-17707) HttpJdkSolrClient threads stuck on exceptions
[
https://issues.apache.org/jira/browse/SOLR-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098318#comment-18098318
]
ASF subversion and git services commented on SOLR-17707:
Commit ca51b1b33c2eee6a69cd1eeb79fe29f81fb9ab3e in solr's branch
refs/heads/branch_10x from Serhiy Bzhezytskyy
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=ca51b1b33c2 ]
SOLR-17707: HttpJdkSolrClient fix thread leak when async request fails (#4639)
Fixed HttpJdkSolrClient leaking an executor thread when an async request's
connection failed while its body was still being written; enough such failures
could exhaust the client's thread pool.
(cherry picked from commit aa9e3f1a7841e765d3806951e2d6f0460f6a057d)
> HttpJdkSolrClient threads stuck on exceptions
> -
>
> Key: SOLR-17707
> URL: https://issues.apache.org/jira/browse/SOLR-17707
> Project: Solr
> Issue Type: Bug
> Components: SolrJ
>Affects Versions: 9.6.1
>Reporter: Artem Abeleshev
>Priority: Major
> Labels: SolrJ, pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> There is a problem with _HttpJdkSolrClient_ used by {_}SolrJ{_}. In case of
> exception occurs on sending data (in my case it was {_}connection reset by
> peer{_}) the thread from the http client executor pool get stuck. Internally,
> _HttpJdkSolrClient_ using _PipedOutputStream_ and _PipedInputStream_ and in
> case of exception it ends up waiting forever. Seems like _PipedOutputStream_
> buffer remains full and _PipedInputStream_ stuck in a waiting state waiting
> for the avaialble space to write. This will consume one thread from the pool
> making it blocked. By default, _HttpJdkSolrClient_ created with an executor
> of a _4_ threads, if all of them are blocked the _SolrJ_ will be unable to
> work.
> There seems to be no problem with that if switching to use _Http2SolrClient_
> instead of _HttpJdkSolrClient_ as _Http2SolrClient_ doesn't use
> _PipedOutputStream_ and {_}PipedInputStream{_} facility.
> Here is a part of threaddump showing the http client threads get stuck:
> {code:java}
> "HttpJdkSolrClient-1-thread-1-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=52 TIMED_WAITING on
> java.io.PipedInputStream@67168d76
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@67168d76
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@bdec8d6
> "HttpJdkSolrClient-1-thread-2-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=53 TIMED_WAITING on
> java.io.PipedInputStream@571e7f87
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@571e7f87
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@7e6cf6f2
> "HttpJdkSolrClient-1-thread-3-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=54 TIMED_WAITING on
> java.io.PipedInputStream@497c4c19
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@497c4c19
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:2
[jira] [Commented] (SOLR-17707) HttpJdkSolrClient threads stuck on exceptions
[
https://issues.apache.org/jira/browse/SOLR-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096880#comment-18096880
]
ASF subversion and git services commented on SOLR-17707:
Commit aa9e3f1a7841e765d3806951e2d6f0460f6a057d in solr's branch
refs/heads/main from Serhiy Bzhezytskyy
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=aa9e3f1a784 ]
SOLR-17707: HttpJdkSolrClient fix thread leak when async request fails (#4639)
Fixed HttpJdkSolrClient leaking an executor thread when an async request's
connection failed while its body was still being written; enough such failures
could exhaust the client's thread pool.
> HttpJdkSolrClient threads stuck on exceptions
> -
>
> Key: SOLR-17707
> URL: https://issues.apache.org/jira/browse/SOLR-17707
> Project: Solr
> Issue Type: Bug
> Components: SolrJ
>Affects Versions: 9.6.1
>Reporter: Artem Abeleshev
>Priority: Major
> Labels: SolrJ, pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> There is a problem with _HttpJdkSolrClient_ used by {_}SolrJ{_}. In case of
> exception occurs on sending data (in my case it was {_}connection reset by
> peer{_}) the thread from the http client executor pool get stuck. Internally,
> _HttpJdkSolrClient_ using _PipedOutputStream_ and _PipedInputStream_ and in
> case of exception it ends up waiting forever. Seems like _PipedOutputStream_
> buffer remains full and _PipedInputStream_ stuck in a waiting state waiting
> for the avaialble space to write. This will consume one thread from the pool
> making it blocked. By default, _HttpJdkSolrClient_ created with an executor
> of a _4_ threads, if all of them are blocked the _SolrJ_ will be unable to
> work.
> There seems to be no problem with that if switching to use _Http2SolrClient_
> instead of _HttpJdkSolrClient_ as _Http2SolrClient_ doesn't use
> _PipedOutputStream_ and {_}PipedInputStream{_} facility.
> Here is a part of threaddump showing the http client threads get stuck:
> {code:java}
> "HttpJdkSolrClient-1-thread-1-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=52 TIMED_WAITING on
> java.io.PipedInputStream@67168d76
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@67168d76
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@bdec8d6
> "HttpJdkSolrClient-1-thread-2-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=53 TIMED_WAITING on
> java.io.PipedInputStream@571e7f87
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@571e7f87
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@7e6cf6f2
> "HttpJdkSolrClient-1-thread-3-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=54 TIMED_WAITING on
> java.io.PipedInputStream@497c4c19
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@497c4c19
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(Fa
[jira] [Commented] (SOLR-17707) HttpJdkSolrClient threads stuck on exceptions
[
https://issues.apache.org/jira/browse/SOLR-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096458#comment-18096458
]
Serhiy Bzhezytskyy commented on SOLR-17707:
---
Picked this up — PR: https://github.com/apache/solr/pull/4639 (supersedes the
POC in https://github.com/apache/solr/pull/3273; [~janhoy], feel free to close
it — credited in the description).
The test reproduces the stuck thread deterministically: a request body larger
than the pipe buffer + a connection failure mid-write leaves the writer blocked
in PipedInputStream.awaitSpace(). One finding worth flagging: cancelling the
writer's Future alone does not fix it — cancel(true) succeeds but the interrupt
doesn't reliably reach the pipe-blocked thread. Closing the pipe's sink is what
releases it; the fix does both.
[~jdyer] — to your 2025 question: no CallerRunsPolicy needed; it reproduces on
the client's own default executor once the body exceeds the pipe buffer and the
connection dies mid-write. That's likely why it was hard to trigger with small
test payloads.
> HttpJdkSolrClient threads stuck on exceptions
> -
>
> Key: SOLR-17707
> URL: https://issues.apache.org/jira/browse/SOLR-17707
> Project: Solr
> Issue Type: Bug
> Components: SolrJ
>Affects Versions: 9.6.1
>Reporter: Artem Abeleshev
>Priority: Major
> Labels: SolrJ, pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> There is a problem with _HttpJdkSolrClient_ used by {_}SolrJ{_}. In case of
> exception occurs on sending data (in my case it was {_}connection reset by
> peer{_}) the thread from the http client executor pool get stuck. Internally,
> _HttpJdkSolrClient_ using _PipedOutputStream_ and _PipedInputStream_ and in
> case of exception it ends up waiting forever. Seems like _PipedOutputStream_
> buffer remains full and _PipedInputStream_ stuck in a waiting state waiting
> for the avaialble space to write. This will consume one thread from the pool
> making it blocked. By default, _HttpJdkSolrClient_ created with an executor
> of a _4_ threads, if all of them are blocked the _SolrJ_ will be unable to
> work.
> There seems to be no problem with that if switching to use _Http2SolrClient_
> instead of _HttpJdkSolrClient_ as _Http2SolrClient_ doesn't use
> _PipedOutputStream_ and {_}PipedInputStream{_} facility.
> Here is a part of threaddump showing the http client threads get stuck:
> {code:java}
> "HttpJdkSolrClient-1-thread-1-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=52 TIMED_WAITING on
> java.io.PipedInputStream@67168d76
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@67168d76
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@bdec8d6
> "HttpJdkSolrClient-1-thread-2-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=53 TIMED_WAITING on
> java.io.PipedInputStream@571e7f87
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@571e7f87
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@7e6cf6f2
> "HttpJdkSolrClient-1-thread-3-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=54 TIMED_WAITING on
> java.io.PipedInputStream@497c4c19
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@497c4
[jira] [Commented] (SOLR-17707) HttpJdkSolrClient threads stuck on exceptions
[
https://issues.apache.org/jira/browse/SOLR-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18082709#comment-18082709
]
Jan Høydahl commented on SOLR-17707:
I don't remember what I did last year. I think it was a quick AI prompt or
something.
But lately I committed the test AsyncTrackerSemaphoreLeakTest which has a
FakeTcpServer that injects some delay and then drops connection. Perhaps that
FakeTcpServer could be used in a test for SolrJdkClient to provoke something?
I'll not do more work on the open PR, so feel free to close it as a POC or run
further with it.
> HttpJdkSolrClient threads stuck on exceptions
> -
>
> Key: SOLR-17707
> URL: https://issues.apache.org/jira/browse/SOLR-17707
> Project: Solr
> Issue Type: Bug
> Components: SolrJ
>Affects Versions: 9.6.1
>Reporter: Artem Abeleshev
>Priority: Major
> Labels: SolrJ, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> There is a problem with _HttpJdkSolrClient_ used by {_}SolrJ{_}. In case of
> exception occurs on sending data (in my case it was {_}connection reset by
> peer{_}) the thread from the http client executor pool get stuck. Internally,
> _HttpJdkSolrClient_ using _PipedOutputStream_ and _PipedInputStream_ and in
> case of exception it ends up waiting forever. Seems like _PipedOutputStream_
> buffer remains full and _PipedInputStream_ stuck in a waiting state waiting
> for the avaialble space to write. This will consume one thread from the pool
> making it blocked. By default, _HttpJdkSolrClient_ created with an executor
> of a _4_ threads, if all of them are blocked the _SolrJ_ will be unable to
> work.
> There seems to be no problem with that if switching to use _Http2SolrClient_
> instead of _HttpJdkSolrClient_ as _Http2SolrClient_ doesn't use
> _PipedOutputStream_ and {_}PipedInputStream{_} facility.
> Here is a part of threaddump showing the http client threads get stuck:
> {code:java}
> "HttpJdkSolrClient-1-thread-1-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=52 TIMED_WAITING on
> java.io.PipedInputStream@67168d76
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@67168d76
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@bdec8d6
> "HttpJdkSolrClient-1-thread-2-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=53 TIMED_WAITING on
> java.io.PipedInputStream@571e7f87
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@571e7f87
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@7e6cf6f2
> "HttpJdkSolrClient-1-thread-3-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=54 TIMED_WAITING on
> java.io.PipedInputStream@497c4c19
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@497c4c19
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaB
[jira] [Commented] (SOLR-17707) HttpJdkSolrClient threads stuck on exceptions
[
https://issues.apache.org/jira/browse/SOLR-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18082693#comment-18082693
]
David Smiley commented on SOLR-17707:
-
{quote}It's a tricky test to write. I gave it a shot, terminating embedded
Jetty mid-flight and could reproduce hanging client thread, however It's hard
to make such a test reliable and consistent. Perhaps mocking is better? I'll
pass on this for now.
{quote}
That's adequate. It's better than nothing – just hopes/prayers. If you can
elaborate further on what you did, then it'd be appreciated.
> HttpJdkSolrClient threads stuck on exceptions
> -
>
> Key: SOLR-17707
> URL: https://issues.apache.org/jira/browse/SOLR-17707
> Project: Solr
> Issue Type: Bug
> Components: SolrJ
>Affects Versions: 9.6.1
>Reporter: Artem Abeleshev
>Priority: Major
> Labels: SolrJ, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> There is a problem with _HttpJdkSolrClient_ used by {_}SolrJ{_}. In case of
> exception occurs on sending data (in my case it was {_}connection reset by
> peer{_}) the thread from the http client executor pool get stuck. Internally,
> _HttpJdkSolrClient_ using _PipedOutputStream_ and _PipedInputStream_ and in
> case of exception it ends up waiting forever. Seems like _PipedOutputStream_
> buffer remains full and _PipedInputStream_ stuck in a waiting state waiting
> for the avaialble space to write. This will consume one thread from the pool
> making it blocked. By default, _HttpJdkSolrClient_ created with an executor
> of a _4_ threads, if all of them are blocked the _SolrJ_ will be unable to
> work.
> There seems to be no problem with that if switching to use _Http2SolrClient_
> instead of _HttpJdkSolrClient_ as _Http2SolrClient_ doesn't use
> _PipedOutputStream_ and {_}PipedInputStream{_} facility.
> Here is a part of threaddump showing the http client threads get stuck:
> {code:java}
> "HttpJdkSolrClient-1-thread-1-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=52 TIMED_WAITING on
> java.io.PipedInputStream@67168d76
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@67168d76
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@bdec8d6
> "HttpJdkSolrClient-1-thread-2-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=53 TIMED_WAITING on
> java.io.PipedInputStream@571e7f87
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@571e7f87
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@7e6cf6f2
> "HttpJdkSolrClient-1-thread-3-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=54 TIMED_WAITING on
> java.io.PipedInputStream@497c4c19
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@497c4c19
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marsha
[jira] [Commented] (SOLR-17707) HttpJdkSolrClient threads stuck on exceptions
[
https://issues.apache.org/jira/browse/SOLR-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17938944#comment-17938944
]
James Dyer commented on SOLR-17707:
---
Hi there, [~abeleshev]. I've been trying to reproduce this bug in a
unit/integration test, but so far have not been able to trigger it. I was
wonder if you are passing in your own ThreadPoolExecutor, using the
"CallerRunsPolicy"? Possibly, under load, we can run into a situation where
both the PipedInputStream and PipedOutputStream are executing in the same
thread?
> HttpJdkSolrClient threads stuck on exceptions
> -
>
> Key: SOLR-17707
> URL: https://issues.apache.org/jira/browse/SOLR-17707
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 9.6.1
>Reporter: Artem Abeleshev
>Priority: Major
> Labels: SolrJ, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> There is a problem with _HttpJdkSolrClient_ used by {_}SolrJ{_}. In case of
> exception occurs on sending data (in my case it was {_}connection reset by
> peer{_}) the thread from the http client executor pool get stuck. Internally,
> _HttpJdkSolrClient_ using _PipedOutputStream_ and _PipedInputStream_ and in
> case of exception it ends up waiting forever. Seems like _PipedOutputStream_
> buffer remains full and _PipedInputStream_ stuck in a waiting state waiting
> for the avaialble space to write. This will consume one thread from the pool
> making it blocked. By default, _HttpJdkSolrClient_ created with an executor
> of a _4_ threads, if all of them are blocked the _SolrJ_ will be unable to
> work.
> There seems to be no problem with that if switching to use _Http2SolrClient_
> instead of _HttpJdkSolrClient_ as _Http2SolrClient_ doesn't use
> _PipedOutputStream_ and {_}PipedInputStream{_} facility.
> Here is a part of threaddump showing the http client threads get stuck:
> {code:java}
> "HttpJdkSolrClient-1-thread-1-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=52 TIMED_WAITING on
> java.io.PipedInputStream@67168d76
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@67168d76
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@bdec8d6
> "HttpJdkSolrClient-1-thread-2-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=53 TIMED_WAITING on
> java.io.PipedInputStream@571e7f87
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@571e7f87
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@7e6cf6f2
> "HttpJdkSolrClient-1-thread-3-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=54 TIMED_WAITING on
> java.io.PipedInputStream@497c4c19
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@497c4c19
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> o
[jira] [Commented] (SOLR-17707) HttpJdkSolrClient threads stuck on exceptions
[
https://issues.apache.org/jira/browse/SOLR-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17936750#comment-17936750
]
Jan Høydahl commented on SOLR-17707:
It's a tricky test to write. I gave it a shot, terminating embedded Jetty
mid-flight and could reproduce hanging client thread, however It's hard to make
such a test reliable and consistent. Perhaps mocking is better? I'll pass on
this for now.
> HttpJdkSolrClient threads stuck on exceptions
> -
>
> Key: SOLR-17707
> URL: https://issues.apache.org/jira/browse/SOLR-17707
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 9.6.1
>Reporter: Artem Abeleshev
>Priority: Major
> Labels: SolrJ, pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is a problem with _HttpJdkSolrClient_ used by {_}SolrJ{_}. In case of
> exception occurs on sending data (in my case it was {_}connection reset by
> peer{_}) the thread from the http client executor pool get stuck. Internally,
> _HttpJdkSolrClient_ using _PipedOutputStream_ and _PipedInputStream_ and in
> case of exception it ends up waiting forever. Seems like _PipedOutputStream_
> buffer remains full and _PipedInputStream_ stuck in a waiting state waiting
> for the avaialble space to write. This will consume one thread from the pool
> making it blocked. By default, _HttpJdkSolrClient_ created with an executor
> of a _4_ threads, if all of them are blocked the _SolrJ_ will be unable to
> work.
> There seems to be no problem with that if switching to use _Http2SolrClient_
> instead of _HttpJdkSolrClient_ as _Http2SolrClient_ doesn't use
> _PipedOutputStream_ and {_}PipedInputStream{_} facility.
> Here is a part of threaddump showing the http client threads get stuck:
> {code:java}
> "HttpJdkSolrClient-1-thread-1-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=52 TIMED_WAITING on
> java.io.PipedInputStream@67168d76
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@67168d76
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@bdec8d6
> "HttpJdkSolrClient-1-thread-2-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=53 TIMED_WAITING on
> java.io.PipedInputStream@571e7f87
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@571e7f87
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@7e6cf6f2
> "HttpJdkSolrClient-1-thread-3-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=54 TIMED_WAITING on
> java.io.PipedInputStream@497c4c19
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@497c4c19
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBin
[jira] [Commented] (SOLR-17707) HttpJdkSolrClient threads stuck on exceptions
[
https://issues.apache.org/jira/browse/SOLR-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17936606#comment-17936606
]
David Smiley commented on SOLR-17707:
-
Definitely want to see a test for this, as I'll have little faith we've fixed
it without one.
> HttpJdkSolrClient threads stuck on exceptions
> -
>
> Key: SOLR-17707
> URL: https://issues.apache.org/jira/browse/SOLR-17707
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 9.6.1
>Reporter: Artem Abeleshev
>Priority: Major
> Labels: SolrJ, pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is a problem with _HttpJdkSolrClient_ used by {_}SolrJ{_}. In case of
> exception occurs on sending data (in my case it was {_}connection reset by
> peer{_}) the thread from the http client executor pool get stuck. Internally,
> _HttpJdkSolrClient_ using _PipedOutputStream_ and _PipedInputStream_ and in
> case of exception it ends up waiting forever. Seems like _PipedOutputStream_
> buffer remains full and _PipedInputStream_ stuck in a waiting state waiting
> for the avaialble space to write. This will consume one thread from the pool
> making it blocked. By default, _HttpJdkSolrClient_ created with an executor
> of a _4_ threads, if all of them are blocked the _SolrJ_ will be unable to
> work.
> There seems to be no problem with that if switching to use _Http2SolrClient_
> instead of _HttpJdkSolrClient_ as _Http2SolrClient_ doesn't use
> _PipedOutputStream_ and {_}PipedInputStream{_} facility.
> Here is a part of threaddump showing the http client threads get stuck:
> {code:java}
> "HttpJdkSolrClient-1-thread-1-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=52 TIMED_WAITING on
> java.io.PipedInputStream@67168d76
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@67168d76
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@bdec8d6
> "HttpJdkSolrClient-1-thread-2-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=53 TIMED_WAITING on
> java.io.PipedInputStream@571e7f87
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@571e7f87
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@7e6cf6f2
> "HttpJdkSolrClient-1-thread-3-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=54 TIMED_WAITING on
> java.io.PipedInputStream@497c4c19
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@497c4c19
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.Th
[jira] [Commented] (SOLR-17707) HttpJdkSolrClient threads stuck on exceptions
[
https://issues.apache.org/jira/browse/SOLR-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17936414#comment-17936414
]
Jan Høydahl commented on SOLR-17707:
I asked an AI to fix the problem and posted the proposed fix. Perhaps a
grown-up can have a look too :)
> HttpJdkSolrClient threads stuck on exceptions
> -
>
> Key: SOLR-17707
> URL: https://issues.apache.org/jira/browse/SOLR-17707
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 9.6.1
>Reporter: Artem Abeleshev
>Priority: Major
> Labels: SolrJ, pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is a problem with _HttpJdkSolrClient_ used by {_}SolrJ{_}. In case of
> exception occurs on sending data (in my case it was {_}connection reset by
> peer{_}) the thread from the http client executor pool get stuck. Internally,
> _HttpJdkSolrClient_ using _PipedOutputStream_ and _PipedInputStream_ and in
> case of exception it ends up waiting forever. Seems like _PipedOutputStream_
> buffer remains full and _PipedInputStream_ stuck in a waiting state waiting
> for the avaialble space to write. This will consume one thread from the pool
> making it blocked. By default, _HttpJdkSolrClient_ created with an executor
> of a _4_ threads, if all of them are blocked the _SolrJ_ will be unable to
> work.
> There seems to be no problem with that if switching to use _Http2SolrClient_
> instead of _HttpJdkSolrClient_ as _Http2SolrClient_ doesn't use
> _PipedOutputStream_ and {_}PipedInputStream{_} facility.
> Here is a part of threaddump showing the http client threads get stuck:
> {code:java}
> "HttpJdkSolrClient-1-thread-1-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=52 TIMED_WAITING on
> java.io.PipedInputStream@67168d76
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@67168d76
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@bdec8d6
> "HttpJdkSolrClient-1-thread-2-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=53 TIMED_WAITING on
> java.io.PipedInputStream@571e7f87
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@571e7f87
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@7e6cf6f2
> "HttpJdkSolrClient-1-thread-3-processing-system
> ad571f96-762d-4e9a-8f9c-f5580413397d" prio=5 Id=54 TIMED_WAITING on
> java.io.PipedInputStream@497c4c19
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.io.PipedInputStream@497c4c19
> at [email protected]/java.io.PipedInputStream.awaitSpace(Unknown Source)
> at [email protected]/java.io.PipedInputStream.receive(Unknown Source)
> at [email protected]/java.io.PipedOutputStream.write(Unknown Source)
> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:207)
> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:200)
> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:100)
> ...
> Number of locked synchronizers = 1
> - java.util.concu
