Abacn commented on code in PR #26913:
URL: https://github.com/apache/beam/pull/26913#discussion_r1207375171
##########
sdks/python/container/piputil.go:
##########
@@ -148,8 +148,11 @@ func findBeamSdkWhl(files []string, acceptableWhlSpecs
[]string) string {
// SDK from source tarball provided in sdkSrcFile.
func installSdk(files []string, workDir string, sdkSrcFile string,
acceptableWhlSpecs []string, required bool) error {
sdkWhlFile := findBeamSdkWhl(files, acceptableWhlSpecs)
+
if sdkWhlFile != "" {
- err := pipInstallPackage(files, workDir, sdkWhlFile, false,
false, []string{"gcp"})
+ // by default, pip rejects to install wheel if same version
already installed
+ isDev := strings.Contains(sdkWhlFile, ".dev")
Review Comment:
tarball is not affected. checked the log if passing a tarball (same pipeline
options above expect sdk_location)
```
DEBUG 2023-05-26T21:15:34.103230283Z Found artifact: dataflow_python_sdk.tar
INFO 2023-05-26T21:15:34.103325167Z 2023/05/26 21:15:34 Installing setup
packages ...
INFO 2023-05-26T21:15:34.645905671Z Processing
/var/opt/google/staged/dataflow_python_sdk.tar
INFO 2023-05-26T21:15:35.031355177Z Preparing metadata (setup.py): started
...
INFO 2023-05-26T21:16:30.151314981Z Successfully built apache-beam
INFO 2023-05-26T21:16:31.978811453Z Installing collected packages:
apache-beam
INFO 2023-05-26T21:16:31.979188478Z Attempting uninstall: apache-beam
INFO 2023-05-26T21:16:31.989685770Z Found existing installation: apache-beam
2.49.0.dev0
INFO 2023-05-26T21:16:31.989707826Z Not uninstalling apache-beam at
/usr/local/lib/python3.11/site-packages, outside environment
/opt/apache/beam-venv/beam-venv-worker-sdk-0-0
INFO 2023-05-26T21:16:31.989723246Z Can't uninstall 'apache-beam'. No files
were found to uninstall.
...
INFO 2023-05-26T21:16:37.762490639Z Successfully installed
apache-beam-2.49.0.dev0
DEBUG 2023-05-26T21:16:38.099652894Z Executing Python (worker
sdk-0-0_sibling_1): python -m apache_beam.runners.worker.sdk_worker_main
DEBUG 2023-05-26T21:16:38.268671515Z Executing Python (worker sdk-0-0):
python -m apache_beam.runners.worker.sdk_worker_main
```
btw now install from tarball costs 1 minute with sibling worker enabled
(previously took 3.5 min when every cpu core run a container)
though it seems PostCommit test still not run with sibling worker
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
