zendesk-kjaanson commented on issue #32743: URL: https://github.com/apache/beam/issues/32743#issuecomment-2714781666
I had a small issue within my setup that semi_persistent_dir (/tmp/staged) was not correctly shared between flink-main-container and python harness container using Apache Flink Operator. However, even when correcting this, artefacts are not shared to workers unless the `/tmp/beam-artefact-staging` directory is shared between jobmanager and taskmanagers. I was also confused as to why I was not able to change location of that directory using pipeline options, but it turns out that while using `FlinkPortableClientEntryPoint` this parameter needs to be given to to the entrypoint itself (not in the pipeline options passed to `--driver-cmd`). I also tried using remote storage (`gs://`) for the artefact staging and it did not seem to work. I am a bit unclear about this from docs, should the `artefact_dir` option accept remote paths or not? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
