aprochko commented on issue #39353:
URL: https://github.com/apache/beam/issues/39353#issuecomment-5668994596
## Follow-up: traced this to a Beam POM change between 2.74.0 → 2.75.0
We dug into this a bit more on our side. Our pipeline: KafkaIO → aggregation
functions with state/timers, calling BigTable/BigQuery clients directly (not
via IO transforms) → Flatten PCollections → aggregation with state/timers →
BigTableIO + io.requestresponse.
**What changed between 2.74.0 and 2.75.0**
Comparing `beam-sdks-java-io-google-cloud-platform` POMs on Maven Central:
In 2.74.0, the `google-cloud-bigtable` dependency is explicitly pinned:
```xml
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-bigtable</artifactId>
<version>2.73.1</version>
...
```
In 2.75.0, the same dependency has **no explicit version** anymore. It now
resolves transitively through:
```
libraries-bom:26.83.0 → google-cloud-bom:0.264.0 →
gapic-libraries-bom:1.86.2 → google-cloud-bigtable-bom:2.78.1
```
(confirmed directly in `gapic-libraries-bom-1.86.2.pom`). For Beam 2.76.0,
the same chain resolves to bigtable 2.80.0.
**Why this matters**
Starting with `google-cloud-bigtable` 2.76.0, DirectPath is enabled by
default (bypasses the Cloud Front End, direct gRPC), and 2.77.0 introduced a
new session-based ("VRpc") client stack. The versions Beam now pulls in
transitively (2.78.1 for 2.75.0, 2.80.0 for 2.76.0) both sit inside this new
stack, and the client's own changelog shows a sequence of concurrency fixes
across 2.78.0–2.81.0 (session pool leaks, deadlocks, and in 2.81.0 specifically
"Bound SessionPoolImpl lock to prevent pod-wide wedge") — which lines up
closely with the "job gets stuck, 100% CPU, no progress" symptom.
**Short stack trace we saw** (appeared once, after ~2 hours of the job
running):
```
com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException:
INTERNAL:
Abrupt GOAWAY closed sent stream. HTTP/2 error code: INTERNAL_ERROR,
debug data: Cancelling all calls
at com.google.cloud.bigtable.data.v2.stub.BigtableChannelPool...
```
Is there anything that can realistically be improved on Beam's side here —
re-pinning explicitly, or at least surfacing this as a known-risky transitive
upgrade?
For context on our side, we use both `BigtableIO` (for writes) and our own
custom repository classes (for reads). We initially had the read-side client
created lazily and never explicitly closed, and suspected that pattern as a
contributing factor, so we changed it to create the `BigtableDataClient`
explicitly in each DoFn's `@Setup` and close it in `@Teardown`. That didn't
resolve it — we're still seeing the same freeze on 2.83.0 with the client
properly scoped and closed, which points more toward the client/DirectPath
negotiation itself rather than our lifecycle management around it.
We also run in a fairly locked-down private network (restrictive firewall
rules), and suspect DirectPath's connection negotiation may not be degrading
cleanly in that environment — which could explain why we see a silent freeze
instead of a clean connection error. We're not sure how much of this is on us
vs. the client/Beam interaction, but I guess it's important to share it there :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
