SEPURI-SAI-KRISHNA opened a new issue, #13155:
URL: https://github.com/apache/seatunnel/issues/13155

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Bug%22)
 and found no similar issues. #10059 and #10060 are related, see below.
   
   ### What happened
   
   With `--master local`, `spark-submit` may never exit after a job has failed, 
if a sink writer holds non-daemon threads. The job has failed and the error is 
printed, but a script or scheduler waiting on that command never gets the exit 
code.
   
   On a failed task Spark calls `abort()`, which does not close the sink 
writer. In local mode the tasks run in the driver JVM, so the writer's threads 
keep that JVM alive after `main` has ended. The Cassandra sink does this 
through its `CqlSession`. This was found through #13126, where a test that 
fails on purpose used up the 5 hour CI limit on Spark.
   
   Measured with FakeSource, a SQL transform that fails on row 3, and Cassandra 
4.1.1 as the sink, on real Spark with `dev` (9f741b9c64), one run each:
   
   | Run | `async_write = false` | default `async_write` |
   |---|---|---|
   | Spark 2.4.6, `--master local` | no exit in 45 s, 0 rows | no exit in 45 s, 
2 rows |
   | Spark 3.3.0, `--master local` | no exit in 45 s, 0 rows | no exit in 45 s, 
2 rows |
   | Spark 3.3.0, standalone cluster | exit 1 in 22 s, 0 rows | not run |
   
   Thread dumps of the local runs show that `DestroyJavaVM` is waiting and that 
the only non-daemon application threads left are `s1-admin`, `s1-io` and 
`s1-timer`.
   
   **Why I am not proposing to close the writer in `abort()`.** It ends the 
hang, and I measured that too, but the reviews on #10060 explain the problem 
with it: `close()` is not resource-only, so a failed attempt can write its 
buffered rows. I can confirm this. With JDBC (SQLite) the target goes from 0 to 
2 rows, and from 0 to 6 with `local[1,3]`.
   
   **Proposal.** Leave the writer lifecycle as it is, and end the JVM when the 
job has failed in local mode. `SeaTunnelSpark.main` would catch the failure, 
which `SeaTunnel.run` has already logged, print its stack trace as the JVM 
would, and call `System.exit(1)` if `spark.master` starts with `local`. 
`SeaTunnelApplication` already does this on Zeta: it catches an `Exception` 
from `SeaTunnel.run` and calls `System.exit(1)`.
   
   I built this from `dev` (6191598c29, no change to these modules since the 
runs above), so only `SeaTunnelSpark` differs. One run each on 2.4.6 and 3.3.0 
unless noted:
   
   | Job | `dev` | With the proposal |
   |---|---|---|
   | Cassandra, `async_write = false` | no exit, 0 rows | exit 1 in 13 s and 15 
s, 0 rows |
   | Cassandra, default `async_write` | no exit, 2 rows | exit 1 in 13 s and 15 
s, 2 rows |
   | JDBC (SQLite) | exit 1, 0 rows | exit 1, 0 rows |
   | JDBC, `local[1,3]` | exit 1, 0 rows | exit 1, 0 rows |
   | JDBC, `local[1,3]`, `batch_size = 1` | exit 1, 6 rows | exit 1, 6 rows |
   | LocalFile | exit 1, 0 files | exit 1, 0 files |
   | Console, Assert | exit 1 | exit 1 |
   | Jobs that succeed | exit 0 | exit 0, same rows |
   | Cassandra on a standalone cluster (3.3.0) | exit 1, 0 rows | same, the new 
exit does not run |
   
   The original error stays in the output and `SparkContext` stops through its 
shutdown hook. On a job that fails in `main`, stderr matches `dev` apart from 
the `Exception in thread "main"` prefix and one line number. A writer is not 
closed on failure in these runs, as on `dev`.
   
   **What this does not fix.** Sessions of failed attempts stay open until the 
JVM exits, as they do today, so `local[1,3]` holds them across its retries. Not 
tested: YARN and Kubernetes, where the new check does not apply because the 
master does not start with `local`.
   
   I would like to submit a PR for this. It also adds a `CassandraIT` test for 
a job that fails while the sink writer is open. If maintainers prefer an 
abort-safe cleanup hook on the sink writer API, as suggested on #10060, I am 
happy to discuss that here instead.
   
   ### SeaTunnel Version
   
   dev (9f741b9c64 for the `dev` runs, 6191598c29 for the proposal). The 
Cassandra connector jar was built from the #13126 branch with 
`CassandraSinkWriter` and `TypeConvertUtil` from `dev` put back.
   
   ### SeaTunnel Config
   
   ```conf
   env {
     parallelism = 1
     job.mode = "BATCH"
   }
   source {
     FakeSource {
       plugin_output = "fake"
       schema = { fields { id = bigint, name = string } }
       rows = [
         { kind = INSERT, fields = [1, "1"] }
         { kind = INSERT, fields = [2, "2"] }
         { kind = INSERT, fields = [3, "boom"] }
       ]
     }
   }
   transform {
     Sql {
       plugin_input = "fake"
       plugin_output = "bad"
       query = "select cast(name as bigint) as id from dual"
     }
   }
   sink {
     Cassandra {
       plugin_input = "bad"
       host = "127.0.0.1:19042"
       username = ""
       password = ""
       datacenter = "datacenter1"
       keyspace = "test"
       table = "sink_table"
       fields = ["id"]
       async_write = false
     }
   }
   ```
   
   ### Running Command
   
   ```shell
   bin/start-seatunnel-spark-3-connector-v2.sh --config fail.conf --master 
local --deploy-mode client
   ```
   
   ### Error Exception
   
   ```log
   org.apache.seatunnel.transform.exception.TransformException: 
ErrorCode:[TRANSFORM_COMMON-06], ErrorDescription:[The expression 'cast(name AS 
bigint)' of SQL transform execute failed]
   Caused by: java.lang.NumberFormatException: For input string: "boom"
   Exception in thread "main" 
org.apache.seatunnel.core.starter.exception.CommandExecuteException: Run 
SeaTunnel on spark failed
   ```
   
   The process does not exit after this.
   
   ### Zeta or Flink or Spark Version
   
   Spark 2.4.6 and 3.3.0
   
   ### Java or Scala Version
   
   JDK 11
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to