GitHub user jkff opened a pull request:
https://github.com/apache/beam/pull/4103
Adds logging at INFO for all creation, deletion and copying of files in
WriteFiles
This will help with debugging issues such as
https://stackoverflow.com/questions/47113773/dataflow-2-1-0-streaming-application-is-not-cleaning-temp-folders/47142671
The amount of logging, I believe, should be reasonable: it's several
messages per output file (not per element or anything like that): when the temp
file is created, successfully closed, deleted on error, copied to final
location, and deleted after copying. This should allow tracing everything that
happens to suspicious files.
R: @chamikaramj
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jkff/incubator-beam write-files-logging
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/4103.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4103
----
commit 2ee2417154321c78595480ec1973f7c9e74579a5
Author: Eugene Kirpichov <[email protected]>
Date: 2017-11-09T00:13:25Z
Adds logging at INFO for all creation, deletion and copying of files in
WriteFiles
----
---
