lostluck opened a new issue, #27569: URL: https://github.com/apache/beam/issues/27569
### What needs to happen? It's common to have a canonical location for outputs, logs, files, and other miscelanea that may be needed by the SDK or for the prism runner, that needs to be accessible during the job lifetime. eg. For loopback mode jobs for Go, where the binary is unused, it's simpler/and faster to have an empty "binary" file that is uploaded to the runner if needed than to compile and upload an unused binary. So it's useful for the SDK to have a specific place to put this empty file, along with other files for the job. In the same vein, a canonical place to compile that binary to would avoid cluttering user code directories with binaries, vs something for the specific purpose of the job, and copying any manually specified binary as well. Other uses would be to write the JSON version of the Pipeline proto, and other request information. Then printing out this directory path. This avoids unnecessary cruft being logged on job startup, and provides a cannon folder with artifacts to upload for debugging purposes. ------ Further, in prism, it would be good to have a canonical directory when it begins to have artifacts, for restarts and similar, inside the same folder tree. ------- I'm aware that Python and the Go xlang code probably already has this, so I've filed this largely so I can remember to hunt for it, and make it obviously canonical with a common package. ### Issue Priority Priority: 3 (nice-to-have improvement) ### Issue Components - [ ] Component: Python SDK - [ ] Component: Java SDK - [X] Component: Go SDK - [ ] Component: Typescript SDK - [ ] Component: IO connector - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Samza Runner - [ ] Component: Twister2 Runner - [ ] Component: Hazelcast Jet Runner - [ ] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
