fatmanverse opened a new issue, #12762: URL: https://github.com/apache/seatunnel/issues/12762
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. (Searched for `CassandraIT` and `Direct buffer memory`. #12732 targets a TRUNCATE visibility race in the same test, which is a different failure mode; see below.) ### What happened `CassandraIT.testCassandra` and `CassandraIT.testCassandraMultiTableSource` fail on every run of the `all-connectors-it-2` job on current `dev`, on both JDK 11 and JDK 17, and only for the Flink 1.20.1 test container (`{TestContainer}[4]`). All other engines in the same class pass. Because `all-connectors-it-2` is part of the required `Build` check, every pull request whose change set schedules that job currently cannot reach a green `Build`. **The assertion is on the job exit code, not on the row count.** Line 134 / 144 is `Assertions.assertEquals(0, execResult.getExitCode())`. The Flink job itself fails: the Flink 1.20.1 TaskManager runs out of JVM direct memory while the Cassandra sink writer opens its driver session. ```text WARN com.datastax.oss.driver.internal.core.pool.ChannelPool - [s0|cassandra/172.18.0.2:9042] Error while opening new channel (ConnectionInitException: [s0|connecting...] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, ...}): failed to send request (io.netty.handler.codec.EncoderException: java.lang.OutOfMemoryError: Direct buffer memory)) ``` The Flink 1.20.1 TaskManager is started with `-XX:MaxDirectMemorySize=402653188` (384 MB, derived from the memory settings in `Flink20Container`). The Flink 1.13 / 1.15 / 1.18 containers in the same job get 261 MB but do not hit the error. **CI evidence on `dev` (`all-connectors-it-2`, both JDKs failed in every run):** | Run | Head | Result | |---|---|---| | [37927712604](https://github.com/apache/seatunnel/actions/runs/37927712604) | `0278a6a74` (2026-10-09) | passed | | [38142199131](https://github.com/apache/seatunnel/actions/runs/38142199131) | `83603e19b` (2026-10-11) | failed | | [38211579427](https://github.com/apache/seatunnel/actions/runs/38211579427) | `ccd85d2f5` (2026-10-12) | failed | | [38253368133](https://github.com/apache/seatunnel/actions/runs/38253368133) | `b3898b62d` (2026-10-12) | failed | | [38340962420](https://github.com/apache/seatunnel/actions/runs/38340962420) | `b0f6feddf` (2026-10-13) | failed | It also fails the same way on unrelated PR fork runs, for example #11516 ([run 38337320453](https://github.com/fatmanverse/seatunnel/actions/runs/38337320453)). **Bisect.** I reproduced it outside GitHub Actions on a Linux host (Docker 20.10, JDK 17 Maven image) by running `CassandraIT` alone, so it does not depend on other modules sharing the job: | Commit | CassandraIT | `Direct buffer memory` in log | |---|---|---| | `0278a6a74` | 15/15 passed | 0 | | `824136378` | 15/15 passed | 0 | | `136b12d32` | 15/15 passed | 0 | | `48160ec29` [Improve] Update snapshot version to 3.0.1 (#12682) | 2 failures, reproduced twice | 42 / 2 | | `ccd85d2f5` (current `dev` at the time) | 2 failures | 2 | `48160ec29` is the first bad commit, but it only bumps `<revision>` from `3.0.0-SNAPSHOT` to `3.0.1-SNAPSHOT` (plus docs and scripts). I compared the artifacts on both sides: the shaded `connector-cassandra` jar bundles the same dependencies (sizes differ by one byte), the `seatunnel-flink-20-starter` jar has the same content, the JDK in the engine image is the same (`11.0.32.1+1`), and the TaskManager JVM options are identical. So I do not think #12682 is wrong; it looks like the Flink 1.20.1 TaskManager's direct memory budget was already at the edge for the Cassandra driver's Netty allocations, and an otherwise neutral change pushed it over. I have not pinned down the exact mechanism. #12732 adds polling after `TRUNCATE`, which does not help here because the failing assertion is the job exit code and the job never writes. ### SeaTunnel Version `dev` @ `ccd85d2f5` (3.0.1-SNAPSHOT); first bad commit `48160ec29`. ### SeaTunnel Config ```conf Not applicable - this is an e2e test failure. The jobs are the ones built into CassandraIT: cassandra_to_cassandra.conf and cassandra_multitable_source.conf (parallelism = 1, BATCH). ``` ### Running Command ```shell ./mvnw -B install -DskipTests -Dskip.ui=true -Dlicense.skipAddThirdParty=true ./mvnw -B verify -DskipUT=true -DskipIT=false -Dskip.ui=true -Dlicense.skipAddThirdParty=true -Pci \ -pl :connector-cassandra-e2e -am -Dit.test=CassandraIT -DfailIfNoTests=false ``` ### Error Exception ```log [ERROR] Tests run: 15, Failures: 2, Errors: 0, Skipped: 0 <<< FAILURE! - in org.apache.seatunnel.connectors.seatunnel.cassandra.CassandraIT [ERROR] testCassandraMultiTableSource{TestContainer}[4] <<< FAILURE! [ERROR] testCassandra{TestContainer}[4] <<< FAILURE! [ERROR] CassandraIT.testCassandraMultiTableSource:144 expected: <0> but was: <1> [ERROR] CassandraIT.testCassandra:134 expected: <0> but was: <1> # Flink 1.20.1 TaskManager com.datastax.oss.driver.api.core.AllNodesFailedException: All 1 node(s) tried for the query failed ... at com.datastax.oss.driver.api.core.cql.SyncCqlSession.prepare(SyncCqlSession.java:224) at org.apache.seatunnel.connectors.seatunnel.cassandra.sink.CassandraSinkWriter.<init>(CassandraSinkWriter.java:77) at org.apache.seatunnel.connectors.seatunnel.cassandra.sink.CassandraSink.createWriter(CassandraSink.java:102) at org.apache.seatunnel.translation.flink.sink.FlinkSink.createWriter(FlinkSink.java:80) at org.apache.flink.streaming.runtime.operators.sink.StatefulSinkWriterStateHandler.createWriter(StatefulSinkWriterStateHandler.java:122) ... Caused by: io.netty.handler.codec.EncoderException: java.lang.OutOfMemoryError: Direct buffer memory ``` ### Zeta or Flink or Spark Version Flink 1.20.1 E2E container (`tyrantlucifer/flink:1.20.1-scala_2.12_hadoop27`, upgraded to JDK 11 by `EngineImageJdkUpgrader`). Zeta, Flink 1.13.6 / 1.15.3 / 1.18.0 and Spark containers pass. ### Java or Scala Version JDK 11 and JDK 17 (Maven side); JDK 11.0.32.1 inside the engine container. ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
