[
https://issues.apache.org/jira/browse/SPARK-20052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sasaki Toru updated SPARK-20052:
--------------------------------
Description:
Some class extend InputDStream needs closing processing after processing all
batches when graceful shutdown enabled.
(e.g. When using Kafka as data source, need to commit processed offsets to
Kafka Broker)
InputDStream has method 'stop' to stop receiving data, but this method will be
called before processing last batches generated for graceful shutdown.
was:
Some class extend InputDStream needs closing processing after all batches
processed when graceful shutdown enabled.
(e.g. When using Kafka as data source, need to commit processed offsets to
Kafka Broker)
InputDStream has method 'stop' to stop receiving data, but this method will be
called before processing last batches generated for graceful shutdown.
> Some InputDStream needs closing processing after all batches processed when
> graceful shutdown
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-20052
> URL: https://issues.apache.org/jira/browse/SPARK-20052
> Project: Spark
> Issue Type: Improvement
> Components: DStreams
> Affects Versions: 2.2.0
> Reporter: Sasaki Toru
>
> Some class extend InputDStream needs closing processing after processing all
> batches when graceful shutdown enabled.
> (e.g. When using Kafka as data source, need to commit processed offsets to
> Kafka Broker)
> InputDStream has method 'stop' to stop receiving data, but this method will
> be called before processing last batches generated for graceful shutdown.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
