[
https://issues.apache.org/jira/browse/BEAM-12792?focusedWorklogId=754829&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-754829
]
ASF GitHub Bot logged work on BEAM-12792:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Apr/22 22:00
Start Date: 08/Apr/22 22:00
Worklog Time Spent: 10m
Work Description: tvalentyn commented on PR #16658:
URL: https://github.com/apache/beam/pull/16658#issuecomment-1093395167
I spent some time debugging this today and got this far:
Dataflow container pod for SDK container has the following mount (from
container manifest):
```
"volumeMounts": [ {
"mountPath": "/var/opt/google",
"name": "persist"
},
...
"volumes": [ {
"hostPath": {
"path": "/var/opt/google/dataflow"
},
"name": "persist"
}
```
The above maps a persistent directory `/var/opt/google/dataflow` on the host
VM into a directory `/var/opt/google` in the running containers. This directory
is passed to sdk harness container in `--semi_persist_dir=/var/opt/google`
param.
We can see this directory referenced in the errors:
```
"2022/04/08 20:40:15 Failed to install required packages: failed to install
SDK: fork/exec /var/opt/google/beam-venv/beam-venv-worker-sdk-0-0/bin/pip:
permission denied
```
I can reproduce this behavior if I manually SSH into a VM created by
Dataflow, and manually start a container via:
docker run -it --entrypoint=/bin/bash -v
/var/opt/google/dataflow:/var/opt/google apache/beam_python3.7_sdk
```
root@beamapp-valentyn-04082037-04081337-w2xj-harness-zq0w:/# python -m venv
/var/opt/google/env
root@beamapp-valentyn-04082037-04081337-w2xj-harness-zq0w:/#
/var/opt/google/env/bin/pip
bash: /var/opt/google/env/bin/pip: Permission denied
```
On the other hand:
```
root@beamapp-valentyn-04082037-04081337-w2xj-harness-zq0w:/# python -m venv
/var/opt/env
root@beamapp-valentyn-04082037-04081337-w2xj-harness-zq0w:/#
/var/opt/env/bin/pip
Usage:
pip <command> [options]
```
Permissions on the files are identical:
```
root@beamapp-valentyn-04082037-04081337-w2xj-harness-zq0w:/# ls -al
/var/opt/env/bin/pip
-rwxr-xr-x 1 root root 228 Apr 8 21:29 /var/opt/env/bin/pip
root@beamapp-valentyn-04082037-04081337-w2xj-harness-zq0w:/# ls -al
/var/opt/google/env/bin/pip
-rwxr-xr-x 1 root root 235 Apr 8 21:29 /var/opt/google/env/bin/pip
```
but for some reasons, when we create a venv in the mounted directory, the
`pip` script cannot launch. I am not sure what is going on.
Issue Time Tracking
-------------------
Worklog Id: (was: 754829)
Time Spent: 16.5h (was: 16h 20m)
> Multiple jobs running on Flink session cluster reuse the persistent Python
> environment.
> ---------------------------------------------------------------------------------------
>
> Key: BEAM-12792
> URL: https://issues.apache.org/jira/browse/BEAM-12792
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
> Affects Versions: 2.27.0, 2.28.0, 2.29.0, 2.30.0, 2.31.0
> Environment: Kubernetes 1.20 on Ubuntu 18.04.
> Reporter: Jens Wiren
> Priority: P1
> Labels: FlinkRunner, beam
> Time Spent: 16.5h
> Remaining Estimate: 0h
>
> I'm running TFX pipelines on a Flink cluster using Beam in k8s. However,
> extra python packages passed to the Flink runner (or rather beam worker
> side-car) are only installed once per deployment cycle. Example:
> # Flink is deployed and is up and running
> # A TFX pipeline starts, submits a job to Flink along with a python whl of
> custom code and beam ops.
> # The beam worker installs the package and the pipeline finishes succesfully.
> # A new TFX pipeline is build where a new beam fn is introduced, the pipline
> is started and the new whl is submitted as in step 2).
> # This time, the new package is not being installed in the beam worker
> causing the job to fail due to a reference which does not exist in the beam
> worker, since it didn't install the new package.
>
> I started using Flink from beam version 2.27 and it has been an issue all the
> time.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
