Sasaki Toru created SPARK-20052:
-----------------------------------
Summary: Some InputDStream needs closing processing after all
batches processed when graceful shutdown
Key: SPARK-20052
URL: https://issues.apache.org/jira/browse/SPARK-20052
Project: Spark
Issue Type: Improvement
Components: DStreams
Affects Versions: 2.2.0
Reporter: Sasaki Toru
Some class extend InputDStream needs closing processing after all batches
processed when graceful shutdown enabled.
(e.g. When using Kafka as data source, need to commit processed offsets to
Kafka Broker)
InputDStream has method 'stop' to stop receiving data, but this method will be
called before processing last batches generated for graceful shutdown.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
