SEPURI-SAI-KRISHNA opened a new pull request, #13156:
URL: https://github.com/apache/seatunnel/pull/13156

   Yes, glad to contribute to Apache SeaTunnel.
   
   ### Purpose of this pull request
   
   Closes #13155.
   
   On a failed task Spark calls `abort()`, which does not close the sink 
writer. With `--master local` the tasks run in the driver JVM, so a writer that 
holds non-daemon threads keeps `spark-submit` running after the job has failed, 
and a script waiting on it never gets the exit code. The Cassandra sink does 
this through its `CqlSession`, and on #13126 it used up the 5 hour CI limit.
   
   `SeaTunnelSpark.main` (2.4 and 3.3) now catches the failure and, if 
`spark.master` starts with `local`, prints the stack trace and calls 
`System.exit(1)`. `SeaTunnel.run` has already logged the failure, and the 
exception is printed because the JVM would print it only if it left `main`. 
`SeaTunnelApplication` handles a failure from `SeaTunnel.run` the same way.
   
   **Why not close the writer in `abort()`.** That also ends the hang, but 
`close()` is not resource-only, as the reviews on #10060 explain: a failed 
attempt can write its buffered rows. I measured it on a branch that did so: 
JDBC went from 0 to 2 rows, and from 0 to 6 with `local[1,3]`. This PR leaves 
the writer lifecycle as it is.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, a failed job now exits by itself in that local mode case. Exit codes 
and rows are otherwise unchanged. The stack trace no longer starts with 
`Exception in thread "main"`, because the exception does not leave `main`. No 
E2E test looks for that text. Other masters are not affected.
   
   ### How was this patch tested?
   
   **Real jobs**, one run each, JDK 11, on real Spark built from this branch. 
The Cassandra jobs ran against Cassandra 4.1.1 with `dev`'s connector classes. 
A SQL transform fails on row 3, or a NULL `bigint` fails in the sink where 
noted:
   
   ```
                                             dev                 Spark 2.4.6    
      Spark 3.3.0
   Cassandra, async_write = false            no exit, 0 rows     exit 1, 13 s, 
0      exit 1, 15 s, 0
   Cassandra, default async_write            no exit, 2 rows     exit 1, 13 s, 
2      exit 1, 15 s, 2
   Cassandra, NULL bigint, sync              no exit, 0 rows     exit 1, 13 s, 
0      exit 1, 16 s, 0
   Cassandra, NULL bigint, async             no exit, 2 rows     exit 1, 12 s, 
2      exit 1, 15 s, 2
   JDBC (SQLite)                             exit 1, 0 rows      exit 1, 0      
      exit 1, 0
   JDBC, local[*]                            not run             exit 1, 0      
      exit 1, 0
   JDBC, local[1,3]                          exit 1, 0 rows      exit 1, 0      
      exit 1, 0
   JDBC, local[1,3], batch_size = 1          exit 1, 6 rows      exit 1, 6      
      exit 1, 6
   LocalFile                                 exit 1, 0 files     exit 1, 0 
files      exit 1, 0 files
   LocalFile, local[1,3]                     not run             exit 1, 0 
files      exit 1, 0 files
   Console, Assert                           exit 1              exit 1         
      exit 1
   ```
   
   The `dev` column for the NULL `bigint` async row is from Spark 3.3.0 only, 
and the others that ran are from both versions.
   
   Also with this PR, on both versions:
   
   - Jobs that succeed still exit 0 and close the writer: Console, Assert, 
LocalFile and JDBC with 3 rows, and with the #13126 connector a Cassandra to 
Cassandra job, a multi-table job and a one-row job.
   - With the #13126 connector, its NULL key job exits 1 in 15 s on both 
versions, leaves 0 rows, and prints the key error.
   - A job that fails in `main` before any task runs exits 1. Compared with 
`dev` on 3.3.0, stderr differs only in the `Exception in thread "main"` prefix 
and one line number. On 2.4.6 the same two differences remain once values that 
change per run, such as paths and ports, are ignored.
   
   **Standalone cluster**, Spark 3.3.0, one worker: the failing Cassandra job 
exits 1 in 21 s with 0 rows on both `dev` and this PR, so the hang is specific 
to local mode and the new exit does not run. A job that succeeds exits 0.
   
   **IT:** `CassandraIT.testFailedJobEndsWhileTheSinkWriterIsOpen` runs a job 
whose SQL transform fails on row 3 while the sink writer holds its CQL session. 
It asserts a non-zero exit and the transform's error code. It has a 5 minute 
`@Timeout`, so if the hang returns it should fail the test instead of using up 
the CI limit. I did not trigger that timeout. In [this fork 
run](https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/42158589006) 
it passes on JDK 11 and 17 on all 7 containers (4 Flink, Zeta, Spark 2.4 and 
3.3), and each Spark case takes 16 to 20 s.
   
   #13126 disables `testNullPrimaryKeyFailsTheJob` on Spark because of this 
hang. Once both are merged that disable can go, and I will do that in a 
follow-up. Both PRs add to `CassandraIT`, so the second to merge needs a small 
conflict fix.
   
   **Unit tests:** no new unit test, because the change is the JVM exit itself, 
which the IT and the runs above cover. Existing tests in 
`seatunnel-spark-starter-common` pass on JDK 11 and 17, 11 of 11. The tests of 
the 2.4 and 3.3 starter modules do not start on my machine with 
`NoClassDefFoundError: LauncherFactory`, and the same command fails the same 
way on `dev`, so I am relying on CI for those. They pass in the fork run above.
   
   **Not tested:** YARN and Kubernetes, where the check does not apply because 
the master does not start with `local`, and streaming jobs.
   
   **Limit of this fix:** the writers of failed attempts stay open until the 
JVM exits, as they do today, so `local[1,3]` holds them across its retries. 
Closing them safely needs an abort-safe cleanup hook on the sink writer API, as 
suggested on #10060, which is a larger change for maintainers to decide on.
   
   #11174 adds a Spark 4.1 starter with its own `SeaTunnelSpark`, which would 
need the same change if it merges after this.
   
   Also run: `spotless:check`, and `verify -DskipTests` on the Spark starter 
and Cassandra E2E modules.
   
   ### Check list
   
   * [x] If any new Jar binary package adding in your PR, please add License 
Notice according
     [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
 (no new jar)
   * [x] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs (no option changes)
   * [x] If necessary, please update `incompatible-changes.md` to describe the 
incompatibility caused by this PR. (not needed, a job that failed still fails 
with the same exit code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to