SEPURI-SAI-KRISHNA commented on PR #13126:
URL: https://github.com/apache/seatunnel/pull/13126#issuecomment-6664723074
Thanks, @DanielLeens, and no apology needed. I ran this to check the cause
you inferred, and it holds. The new commit disables
`testNullPrimaryKeyFailsTheJob` on Spark, as you suggested.
**What I ran.** The same job conf with `spark-submit --master local` on real
Spark 2.4.6 and 3.3.0 (JDK 11, no containers) against Cassandra 4.1.1, one run
each. I killed the two jobs that did not exit:
```
Spark 2.4.6 Spark 3.3.0
NULL key job no exit after 90 s no exit after 60
s
same job, abort() closes the writer exit 1 after 15 s exit 1 after 18 s
```
The second row is the only code change: `sinkWriter.close()` at the end of
`abort()` in a scratch copy, which is not in this PR. The job printed the NULL
key error in both rows. In the thread dumps of the first row `main` has ended,
and the only non-daemon Java threads left are the driver's `s1-timer-0`,
`s1-admin-*` and `s1-io-*`.
Spark 3.3.0 hangs too, so `type = {EngineType.SPARK}` is the right scope,
not only 2.4. It is also not caused by this PR's key check: with `dev`'s
`CassandraSinkWriter` and `TypeConvertUtil` the job fails with the old
NullPointerException and stays up the same way on both versions.
**The tests CI has not reached.** I also ran `cassandra_to_cassandra.conf`
and `cassandra_multitable_source.conf` on both Spark versions:
- Both exit 0 by themselves.
- The first copied 3 rows with no column differences, and the all-NULL row
stayed NULL in every column. These were my own 3 rows, not the IT's dataset.
- The second wrote the expected 10 ids.
That runs the jobs, not the test class, so it does not replace CI.
**Not verified yet.** `testCassandra` and `testCassandraMultiTableSource`
have still not run in CI on any engine, and I have not run them on Flink
myself. The disable itself compiles and passes `spotless:check`, but only CI
will show it taking effect. I will report back when the run ends, and I will
open the follow-up issue for `abort()` as you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
