nsivabalan opened a new issue, #20370:
URL: https://github.com/apache/hudi/issues/20370
### Bug Description
When a micro-batch write fails
`hoodie.datasource.write.streaming.retry.count` times (default 3) and
`hoodie.datasource.write.streaming.ignore.failed.batch=false` (default),
`HoodieStreamingSink.addBatch` logs *"aborting streaming app to avoid data
loss"* and calls `System.exit(1)` (`HoodieStreamingSink.scala:~210`, comment:
*"will require redeployment / supervise mode to restart the streaming"*).
Terminating the JVM from inside a `Sink` is not the sink's decision to make:
- In a **Spark Connect server** (the execution model for Spark 4.1
Declarative Pipelines, and increasingly for notebooks), the JVM is shared by
every session and every query. Observed: one failing Hudi flow killed the
Connect server; the `spark-pipelines` client then hung indefinitely because the
gRPC stream never completed.
- In a driver running several streaming queries, an unrelated query is
killed.
- Structured Streaming already has a well-defined failure path: throwing
from `addBatch` fails the query, surfaces a `StreamingQueryException`, and lets
the orchestrator (SDP retries, Airflow, `awaitTermination` loops) decide.
Expected: throw a `HoodieException` (wrapping the last failure) after
retries are exhausted; never `System.exit`. If a hard-exit is wanted for legacy
deployments, it must be opt-in.
### Environment
Hudi 1.2.1 (`hudi-spark4.1-bundle_2.13`, Maven Central), Spark 4.1.1 (Spark
Connect server),
`spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog`,
`HoodieSparkSessionExtension`, local filesystem, macOS. Behaviour
cross-checked against current `master` source.
### Steps to reproduce
Start any `writeStream.format("hudi")…` query whose first micro-batch cannot
be written (e.g. table path without `hoodie.table.name` resolvable, or a bad
`hoodie.datasource.write.recordkey.field`), with default retry settings. After
3 attempts the JVM exits with code 1.
### Logs and Stack Trace
```
ERROR HoodieStreamingSink: Micro batch id=0 threw following exception:
org.apache.hudi.exception.HoodieException: 'hoodie.table.name' must be set.
… (×3)
ERROR HoodieStreamingSink: Micro batch id=0 threw following
expections,aborting streaming app to avoid data loss:
INFO SparkContext: Invoking stop() from shutdown hook
INFO SparkContext: Successfully stopped SparkContext
```
Code pointer:
`hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieStreamingSink.scala`
(`addBatch`, `Failure` branch; also `reset(true)` + per-sink JVM shutdown
hooks for async table services, which have the same multi-query problem).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
