tvalentyn commented on PR #16658:
URL: https://github.com/apache/beam/pull/16658#issuecomment-1093395167
I spent some time debugging this today and got this far:
Dataflow container pod for SDK container has the following mount (from
container manifest):
```
"volumeMounts": [ {
"mountPath": "/var/opt/google",
"name": "persist"
},
...
"volumes": [ {
"hostPath": {
"path": "/var/opt/google/dataflow"
},
"name": "persist"
}
```
The above maps a persistent directory `/var/opt/google/dataflow` on the host
VM into a directory `/var/opt/google` in the running containers. This directory
is passed to sdk harness container in `--semi_persist_dir=/var/opt/google`
param.
We can see this directory referenced in the errors:
```
"2022/04/08 20:40:15 Failed to install required packages: failed to install
SDK: fork/exec /var/opt/google/beam-venv/beam-venv-worker-sdk-0-0/bin/pip:
permission denied
```
I can reproduce this behavior if I manually SSH into a VM created by
Dataflow, and manually start a container via:
docker run -it --entrypoint=/bin/bash -v
/var/opt/google/dataflow:/var/opt/google apache/beam_python3.7_sdk
```
root@beamapp-valentyn-04082037-04081337-w2xj-harness-zq0w:/# python -m venv
/var/opt/google/env
root@beamapp-valentyn-04082037-04081337-w2xj-harness-zq0w:/#
/var/opt/google/env/bin/pip
bash: /var/opt/google/env/bin/pip: Permission denied
```
On the other hand:
```
root@beamapp-valentyn-04082037-04081337-w2xj-harness-zq0w:/# python -m venv
/var/opt/env
root@beamapp-valentyn-04082037-04081337-w2xj-harness-zq0w:/#
/var/opt/env/bin/pip
Usage:
pip <command> [options]
```
Permissions on the files are identical:
```
root@beamapp-valentyn-04082037-04081337-w2xj-harness-zq0w:/# ls -al
/var/opt/env/bin/pip
-rwxr-xr-x 1 root root 228 Apr 8 21:29 /var/opt/env/bin/pip
root@beamapp-valentyn-04082037-04081337-w2xj-harness-zq0w:/# ls -al
/var/opt/google/env/bin/pip
-rwxr-xr-x 1 root root 235 Apr 8 21:29 /var/opt/google/env/bin/pip
```
but for some reasons, when we create a venv in the mounted directory, the
`pip` script cannot launch. I am not sure what is going on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
