tvalentyn commented on code in PR #40699:
URL: https://github.com/apache/beam/pull/40699#discussion_r4365014083
##########
sdks/python/container/boot.go:
##########
Review Comment:
we may want to clean up the staged dir here to prevent accumulation for
runners that reuse the container environment for multiple pipelines.
##########
sdks/python/container/boot.go:
##########
@@ -257,7 +262,9 @@ func launchSDKProcess() error {
logger.Fatalf(ctx, "Failed to load pipeline options to worker:
%v", err)
}
- os.Setenv("SEMI_PERSISTENT_DIRECTORY", *semiPersistDir)
+ // Point the SDK worker at the per-environment root so that it keeps
+ // resolving <SEMI_PERSISTENT_DIRECTORY>/staged/... (e.g.
pickled_main_session).
Review Comment:
My main concern is: some users might have made assumptions where files are
staged, now the path becomes dynamic, so we need to evaluate it on the worker
and not at pipeline construction. We need to communicate that this is a
breaking change for Python in CHANGES.md and guide users how to navigate this
change.
I think any code that hardcodes /tmp/staged/... (non-dataflow) or smth like
/var/opt/google/staged/... will fail to find files in the right location.
We can point users to a code snippet
https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#accessing-staged-files-on-the-workers
(sent you a PR correcting it slightly).
We might need to highlight that this evaluation needs to run on the workers,
and not at pipeline submission.
We should check how would users use --files_to_stage to supply a model, and
then point RunInference transform to the model that was staged?
We can start with checking if these tests will fail
https://github.com/apache/beam/blob/eb8f207402d446ca1632438a1a179164af3667bc/sdks/go/test/integration/transforms/xlang/inference/inference.go#L42
https://github.com/apache/beam/blob/eb8f207402d446ca1632438a1a179164af3667bc/sdks/java/extensions/python/src/test/java/org/apache/beam/sdk/extensions/python/transforms/RunInferenceTransformTest.java#L49
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
