vanzin commented on a change in pull request #22952: [SPARK-20568][SS] Provide
option to clean up completed files in streaming query
URL: https://github.com/apache/spark/pull/22952#discussion_r334710234
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -546,6 +546,13 @@ Here are the details of all the sources in Spark.
"s3://a/dataset.txt"<br/>
"s3n://a/b/dataset.txt"<br/>
"s3a://a/b/c/dataset.txt"<br/>
+ <code>cleanSource</code>: option to clean up completed files after
processing.<br/>
+ Available options are "archive", "delete", "no_op". If the option is
not provided, the default value is "no_op".<br/>
+ When "archive" is provided, additional option
<code>sourceArchiveDir</code> must be provided as well. The value of
"sourceArchiveDir" must be outside of source path, to ensure archived files are
never included to new source files again.<br/>
+ Spark will move source files respecting its own path. For example, if
the path of source file is "/a/b/dataset.txt" and the path of archive directory
is "/archived/here", file will be moved to "/archived/here/a/b/dataset.txt"<br/>
+ NOTE: Both archiving (via moving) or deleting completed files would
introduce overhead (slow down) in each micro-batch, so you need to understand
the cost for each operation in your file system before enabling this option. On
the other hand, enabling this option would reduce the cost to list source files
which is considered as a heavy operation.<br/>
Review comment:
s/would/will
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
