Burak Yavuz created SPARK-11324:
-----------------------------------
Summary: Flag to close Write Ahead Log after writing
Key: SPARK-11324
URL: https://issues.apache.org/jira/browse/SPARK-11324
Project: Spark
Issue Type: Improvement
Components: Streaming
Reporter: Burak Yavuz
Currently the Write Ahead Log in Spark Streaming flushes data as writes need to
be made. S3 does not support flushing of data, data is written once the stream
is actually closed.
In case of failure, the data for the last minute (default rolling interval)
will not be properly written. Therefore we need a flag to close the stream
after the write, so that we achieve read after write consistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
