SEPURI-SAI-KRISHNA opened a new issue, #13155: URL: https://github.com/apache/seatunnel/issues/13155
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Bug%22) and found no similar issues. #10059 and #10060 are related, see below. ### What happened With `--master local`, `spark-submit` may never exit after a job has failed, if a sink writer holds non-daemon threads. The job has failed and the error is printed, but a script or scheduler waiting on that command never gets the exit code. On a failed task Spark calls `abort()`, which does not close the sink writer. In local mode the tasks run in the driver JVM, so the writer's threads keep that JVM alive after `main` has ended. The Cassandra sink does this through its `CqlSession`. This was found through #13126, where a test that fails on purpose used up the 5 hour CI limit on Spark. Measured with FakeSource, a SQL transform that fails on row 3, and Cassandra 4.1.1 as the sink, on real Spark with `dev` (9f741b9c64), one run each: | Run | `async_write = false` | default `async_write` | |---|---|---| | Spark 2.4.6, `--master local` | no exit in 45 s, 0 rows | no exit in 45 s, 2 rows | | Spark 3.3.0, `--master local` | no exit in 45 s, 0 rows | no exit in 45 s, 2 rows | | Spark 3.3.0, standalone cluster | exit 1 in 22 s, 0 rows | not run | Thread dumps of the local runs show that `DestroyJavaVM` is waiting and that the only non-daemon application threads left are `s1-admin`, `s1-io` and `s1-timer`. **Why I am not proposing to close the writer in `abort()`.** It ends the hang, and I measured that too, but the reviews on #10060 explain the problem with it: `close()` is not resource-only, so a failed attempt can write its buffered rows. I can confirm this. With JDBC (SQLite) the target goes from 0 to 2 rows, and from 0 to 6 with `local[1,3]`. **Proposal.** Leave the writer lifecycle as it is, and end the JVM when the job has failed in local mode. `SeaTunnelSpark.main` would catch the failure, which `SeaTunnel.run` has already logged, print its stack trace as the JVM would, and call `System.exit(1)` if `spark.master` starts with `local`. `SeaTunnelApplication` already does this on Zeta: it catches an `Exception` from `SeaTunnel.run` and calls `System.exit(1)`. I built this from `dev` (6191598c29, no change to these modules since the runs above), so only `SeaTunnelSpark` differs. One run each on 2.4.6 and 3.3.0 unless noted: | Job | `dev` | With the proposal | |---|---|---| | Cassandra, `async_write = false` | no exit, 0 rows | exit 1 in 13 s and 15 s, 0 rows | | Cassandra, default `async_write` | no exit, 2 rows | exit 1 in 13 s and 15 s, 2 rows | | JDBC (SQLite) | exit 1, 0 rows | exit 1, 0 rows | | JDBC, `local[1,3]` | exit 1, 0 rows | exit 1, 0 rows | | JDBC, `local[1,3]`, `batch_size = 1` | exit 1, 6 rows | exit 1, 6 rows | | LocalFile | exit 1, 0 files | exit 1, 0 files | | Console, Assert | exit 1 | exit 1 | | Jobs that succeed | exit 0 | exit 0, same rows | | Cassandra on a standalone cluster (3.3.0) | exit 1, 0 rows | same, the new exit does not run | The original error stays in the output and `SparkContext` stops through its shutdown hook. On a job that fails in `main`, stderr matches `dev` apart from the `Exception in thread "main"` prefix and one line number. A writer is not closed on failure in these runs, as on `dev`. **What this does not fix.** Sessions of failed attempts stay open until the JVM exits, as they do today, so `local[1,3]` holds them across its retries. Not tested: YARN and Kubernetes, where the new check does not apply because the master does not start with `local`. I would like to submit a PR for this. It also adds a `CassandraIT` test for a job that fails while the sink writer is open. If maintainers prefer an abort-safe cleanup hook on the sink writer API, as suggested on #10060, I am happy to discuss that here instead. ### SeaTunnel Version dev (9f741b9c64 for the `dev` runs, 6191598c29 for the proposal). The Cassandra connector jar was built from the #13126 branch with `CassandraSinkWriter` and `TypeConvertUtil` from `dev` put back. ### SeaTunnel Config ```conf env { parallelism = 1 job.mode = "BATCH" } source { FakeSource { plugin_output = "fake" schema = { fields { id = bigint, name = string } } rows = [ { kind = INSERT, fields = [1, "1"] } { kind = INSERT, fields = [2, "2"] } { kind = INSERT, fields = [3, "boom"] } ] } } transform { Sql { plugin_input = "fake" plugin_output = "bad" query = "select cast(name as bigint) as id from dual" } } sink { Cassandra { plugin_input = "bad" host = "127.0.0.1:19042" username = "" password = "" datacenter = "datacenter1" keyspace = "test" table = "sink_table" fields = ["id"] async_write = false } } ``` ### Running Command ```shell bin/start-seatunnel-spark-3-connector-v2.sh --config fail.conf --master local --deploy-mode client ``` ### Error Exception ```log org.apache.seatunnel.transform.exception.TransformException: ErrorCode:[TRANSFORM_COMMON-06], ErrorDescription:[The expression 'cast(name AS bigint)' of SQL transform execute failed] Caused by: java.lang.NumberFormatException: For input string: "boom" Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: Run SeaTunnel on spark failed ``` The process does not exit after this. ### Zeta or Flink or Spark Version Spark 2.4.6 and 3.3.0 ### Java or Scala Version JDK 11 ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
