tomnewton opened a new issue, #39127:
URL: https://github.com/apache/beam/issues/39127
### What happened?
What happened:
After the client-v2 migration (Beam 2.72.0), ClickHouseIO.WriteFn builds
com.clickhouse.client.api.Client without a socket timeout, connect timeout, or
TCP keepalive, and calls client.insert(...).get() (blocking, no timeout) in
flush(). When a connection is silently dropped (LB idle timeout, server
restart, half-open TCP), the worker blocks forever in SSLSocketInputRecord.read
while awaiting the response header.
On Dataflow this manifests as Operation ongoing in bundle indefinitely
without outputting or completing, stack ending at ClickHouseIO.java:603 (flush)
→ :562 (finishBundle). The worker never recovers.
Critically, withMaxRetries/withInitialBackoff provide no protection: the
retry loop only catches exceptions, and a hung socket never throws.
`Operation ongoing in bundle process_bundle-3299-5293 for
PTransform{id=Write [REDACTED] to
ClickHouse/ParDo(AutoValue_ClickHouseIO_Write)/ParMultiDo(AutoValue_ClickHouseIO_Write)-ptransform-92,
name=Write [REDACTED] to
ClickHouse/ParDo(AutoValue_ClickHouseIO_Write)/ParMultiDo(AutoValue_ClickHouseIO_Write)-ptransform-92,
state=finish} for at least 4h06m18s without outputting or completing:\nat
java.base/sun.nio.ch.Net.poll(Native Method)\n at
java.base/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:191)\n at
java.base/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:201)\n at
java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309)\n at
java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)\n at
java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)\n at
java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099)\n at
java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)\n
at java.base/sun.security.ssl.S
SLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483)\n at
java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)\n
at
java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1461)\n
at
java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1066)\n
at
com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:149)\n
at
com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)\n
at
com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:250)\n
at
com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:56)\n
at
com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:33
1)\n at
com.clickhouse.shaded.org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:196)\n
at
com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.InternalExecRuntime.lambda$execute$0(InternalExecRuntime.java:236)\n
at
com.clickhouse.shaded.org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:791)\n
at
com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:233)\n
at
com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainClientExec.java:120)\n
at
com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)\n
at
com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:198)\n
at
com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execut
e(ExecChainElement.java:51)\n at
com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)\n
at
com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)\n
at
com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113)\n
at
com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)\n
at
com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:110)\n
at
com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)\n
at
com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:183)\n
at
com.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87)\n
at com
.clickhouse.shaded.org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:55)\n
at
com.clickhouse.shaded.org.apache.hc.client5.http.classic.HttpClient.executeOpen(HttpClient.java:183)\n
at
com.clickhouse.client.api.internal.HttpAPIClientHelper.executeRequest(HttpAPIClientHelper.java:453)\n
at com.clickhouse.client.api.Client.lambda$insert$3(Client.java:1474)\n at
com.clickhouse.client.api.Client.runAsyncOperation(Client.java:2012)\n at
com.clickhouse.client.api.Client.insert(Client.java:1518)\n at
com.clickhouse.client.api.Client.insert(Client.java:1376)\n at
com.clickhouse.client.api.Client.insert(Client.java:1320)\n at
org.apache.beam.sdk.io.clickhouse.ClickHouseIO$WriteFn.flush(ClickHouseIO.java:603)\n
at
org.apache.beam.sdk.io.clickhouse.ClickHouseIO$WriteFn.finishBundle(ClickHouseIO.java:562)\n
at
org.apache.beam.sdk.io.clickhouse.AutoValue_ClickHouseIO_WriteFn$DoFnInvoker$a1edf190.invokeFinishBundle(Unknown
Source)\n at org.apac
he.beam.fn.harness.FnApiDoFnRunner.finishBundle(FnApiDoFnRunner.java:1303)\n
at
org.apache.beam.fn.harness.data.PTransformFunctionRegistry.lambda$register$0(PTransformFunctionRegistry.java:115)\n
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:561)\n
at
org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:150)\n
at
org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:115)\n
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)\n
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)\n at
org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run(UnboundedScheduledExecutorService.java:163)\n
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n
at java.base/java.util.concurrent.ThreadPoolExecutor$Wo
rker.run(ThreadPoolExecutor.java:642)\n at
java.base/java.lang.Thread.run(Thread.java:1583)`
Expected: A stalled connection should fail with a (retryable) timeout, not
hang indefinitely.
Versions: Beam 2.72.0, ClickHouse client-v2, DataflowRunner, Java 21.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [x] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
