jrmccluskey commented on code in PR #28317:
URL: https://github.com/apache/beam/pull/28317#discussion_r1317757307
##########
sdks/python/container/boot.go:
##########
@@ -450,7 +450,7 @@ func processArtifactsInSetupOnlyMode() {
}
files[i] = filePayload.GetPath()
}
- if setupErr := installSetupPackages(files, workDir,
[]string{requirementsFile}); setupErr != nil {
+ if setupErr := installSetupPackages(context.Background(), nil, files,
workDir, []string{requirementsFile}); setupErr != nil {
Review Comment:
This is what that output looks like in cloud build from the validates
container suite:
```
starting build "e376d4cb-9bb0-4dbd-9055-d8a16fa50f2b"
FETCHSOURCE
Fetching storage object:
gs://temp-storage-for-end-to-end-tests/temp-validatesrunner-test/source-2cd52080-9790-46bb-90c3-7a82263d4907.tgz#1694026811357948
Copying
gs://temp-storage-for-end-to-end-tests/temp-validatesrunner-test/source-2cd52080-9790-46bb-90c3-7a82263d4907.tgz#1694026811357948...
/ [0 files][ 0.0 B/ 2.2 MiB]
/ [1 files][ 2.2 MiB/ 2.2 MiB]
Operation completed over 1 objects/2.2 MiB.
BUILD
Pulling image: gcr.io/kaniko-project/executor:latest
latest: Pulling from kaniko-project/executor
24b9c0f43324: Already exists
0d0cf3a75faa: Already exists
a13620305055: Already exists
3c02b8d0f8d0: Pulling fs layer
f2ed186425e5: Pulling fs layer
c113a25f71a0: Pulling fs layer
e600f5fdfda3: Pulling fs layer
41ac909a10bf: Pulling fs layer
3eb8d55ab910: Pulling fs layer
e600f5fdfda3: Waiting
41ac909a10bf: Waiting
3eb8d55ab910: Waiting
3c02b8d0f8d0: Verifying Checksum
3c02b8d0f8d0: Download complete
f2ed186425e5: Verifying Checksum
f2ed186425e5: Download complete
c113a25f71a0: Verifying Checksum
c113a25f71a0: Download complete
e600f5fdfda3: Verifying Checksum
e600f5fdfda3: Download complete
41ac909a10bf: Download complete
3c02b8d0f8d0: Pull complete
3eb8d55ab910: Verifying Checksum
3eb8d55ab910: Download complete
f2ed186425e5: Pull complete
c113a25f71a0: Pull complete
e600f5fdfda3: Pull complete
41ac909a10bf: Pull complete
3eb8d55ab910: Pull complete
Digest:
sha256:6aace7210ad8bf0da4b1aa707a1c0ea962d0e871bb1921504c1a5e690d5f6a2a
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest
gcr.io/kaniko-project/executor:latest
[36mINFO[0m[0000] Retrieving image manifest
us.gcr.io/apache-beam-testing/jrmccluskey/beam_python3.8_sdk:20230906-184728906838371
[36mINFO[0m[0000] Retrieving image
us.gcr.io/apache-beam-testing/jrmccluskey/beam_python3.8_sdk:20230906-184728906838371
from registry us.gcr.io
[36mINFO[0m[0001] Retrieving image manifest
us.gcr.io/apache-beam-testing/jrmccluskey/beam_python3.8_sdk:20230906-184728906838371
[36mINFO[0m[0001] Returning cached image manifest
[36mINFO[0m[0001] Built cross stage deps: map[]
[36mINFO[0m[0001] Retrieving image manifest
us.gcr.io/apache-beam-testing/jrmccluskey/beam_python3.8_sdk:20230906-184728906838371
[36mINFO[0m[0001] Returning cached image manifest
[36mINFO[0m[0001] Retrieving image manifest
us.gcr.io/apache-beam-testing/jrmccluskey/beam_python3.8_sdk:20230906-184728906838371
[36mINFO[0m[0001] Returning cached image manifest
[36mINFO[0m[0001] Executing 0 build triggers
[36mINFO[0m[0001] Building stage
'us.gcr.io/apache-beam-testing/jrmccluskey/beam_python3.8_sdk:20230906-184728906838371'
[idx: '0', base-idx: '-1']
[36mINFO[0m[0001] Checking for cached layer
us.gcr.io/apache-beam-testing/jrmccluskey/prebuild_python38_sdk/beam_python_prebuilt_sdk/cache:6292b3861d9f8df4a93fcc61e9419eab0788e341c0f37901a2379e7cf63f4836...
[36mINFO[0m[0001] No cached layer found for cmd RUN mkdir -p
/opt/apache/beam/artifacts
[36mINFO[0m[0001] Resolving srcs [*]...
[36mINFO[0m[0001] Unpacking rootfs as cmd RUN mkdir -p
/opt/apache/beam/artifacts requires it.
[36mINFO[0m[0095] RUN mkdir -p /opt/apache/beam/artifacts
[36mINFO[0m[0095] Initializing snapshotter ...
[36mINFO[0m[0095] Taking snapshot of full filesystem...
[36mINFO[0m[0114] Cmd: /bin/sh
[36mINFO[0m[0114] Args: [-c mkdir -p /opt/apache/beam/artifacts]
[36mINFO[0m[0114] Running: [/bin/sh -c mkdir -p
/opt/apache/beam/artifacts]
[36mINFO[0m[0114] Taking snapshot of full filesystem...
[36mINFO[0m[0120] Resolving srcs [*]...
[36mINFO[0m[0120] COPY ./* /opt/apache/beam/artifacts/
[36mINFO[0m[0120] Resolving srcs [*]...
[36mINFO[0m[0120] Pushing layer
us.gcr.io/apache-beam-testing/jrmccluskey/prebuild_python38_sdk/beam_python_prebuilt_sdk/cache:6292b3861d9f8df4a93fcc61e9419eab0788e341c0f37901a2379e7cf63f4836
to cache now
[36mINFO[0m[0120] Pushing image to
us.gcr.io/apache-beam-testing/jrmccluskey/prebuild_python38_sdk/beam_python_prebuilt_sdk/cache:6292b3861d9f8df4a93fcc61e9419eab0788e341c0f37901a2379e7cf63f4836
[36mINFO[0m[0120] Taking snapshot of files...
[36mINFO[0m[0120] RUN /opt/apache/beam/boot --setup_only --artifacts
/opt/apache/beam/artifacts/artifacts_info.json
[36mINFO[0m[0120] Cmd: /bin/sh
[36mINFO[0m[0120] Args: [-c /opt/apache/beam/boot --setup_only --artifacts
/opt/apache/beam/artifacts/artifacts_info.json]
[36mINFO[0m[0120] Running: [/bin/sh -c /opt/apache/beam/boot --setup_only
--artifacts /opt/apache/beam/artifacts/artifacts_info.json]
2023/09/06 19:02:30 Installing setup packages ...
Processing /opt/apache/beam/artifacts/dataflow_python_sdk.tar
Preparing metadata (setup.py): started
[36mINFO[0m[0123] Pushed
us.gcr.io/apache-beam-testing/jrmccluskey/prebuild_python38_sdk/beam_python_prebuilt_sdk/cache@sha256:ad8e29f349e899060c6dca81b839244c046d735808cd9b97ce12a9dd8a9f1de7
Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: crcmod<2.0,>=1.7 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (1.7)
Requirement already satisfied: orjson<4.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (3.9.4)
Requirement already satisfied: dill<0.3.2,>=0.3.1.1 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (0.3.1.1)
Requirement already satisfied: cloudpickle~=2.2.1 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (2.2.1)
Requirement already satisfied: fastavro<2,>=0.23.6 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (1.8.2)
Requirement already satisfied: fasteners<1.0,>=0.3 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (0.18)
Requirement already satisfied: grpcio!=1.48.0,<2,>=1.33.1 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (1.56.2)
Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (2.7.2)
Requirement already satisfied: httplib2<0.23.0,>=0.8 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (0.22.0)
Requirement already satisfied: numpy<1.25.0,>=1.14.3 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (1.24.3)
Requirement already satisfied: objsize<0.7.0,>=0.6.1 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (0.6.1)
Requirement already satisfied: packaging>=22.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (23.1)
Requirement already satisfied: pymongo<5.0.0,>=3.8.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (4.4.1)
Requirement already satisfied: proto-plus<2,>=1.7.1 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (1.22.3)
Requirement already satisfied: protobuf<4.24.0,>=3.20.3 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (4.23.4)
Requirement already satisfied: pydot<2,>=1.2.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (1.4.2)
Requirement already satisfied: python-dateutil<3,>=2.8.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (2.8.2)
Requirement already satisfied: pytz>=2018.3 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (2023.3)
Requirement already satisfied: regex>=2020.6.8 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0)
(2023.8.8)
Requirement already satisfied: requests<3.0.0,>=2.24.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (2.31.0)
Requirement already satisfied: typing-extensions>=3.7.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (4.5.0)
Requirement already satisfied: zstandard<1,>=0.18.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (0.21.0)
Requirement already satisfied: pyarrow<12.0.0,>=3.0.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (11.0.0)
Requirement already satisfied: cachetools<6,>=3.1.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (5.3.1)
Requirement already satisfied: google-api-core<3,>=2.0.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (2.11.1)
Requirement already satisfied: google-apitools<0.5.32,>=0.5.31 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (0.5.31)
Requirement already satisfied: google-auth<3,>=1.18.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (2.22.0)
Requirement already satisfied: google-auth-httplib2<0.2.0,>=0.1.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (0.1.0)
Requirement already satisfied: google-cloud-datastore<3,>=2.0.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (2.17.0)
Requirement already satisfied: google-cloud-pubsub<3,>=2.1.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (2.18.2)
Requirement already satisfied: google-cloud-pubsublite<2,>=1.2.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (1.8.3)
Requirement already satisfied: google-cloud-bigquery<4,>=2.0.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (3.11.4)
Requirement already satisfied: google-cloud-bigquery-storage<3,>=2.6.3 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (2.22.0)
Requirement already satisfied: google-cloud-core<3,>=2.0.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (2.3.3)
Requirement already satisfied: google-cloud-bigtable<3,>=2.19.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (2.21.0)
Requirement already satisfied: google-cloud-spanner<4,>=3.0.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (3.40.0)
Requirement already satisfied: google-cloud-dlp<4,>=3.0.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (3.12.2)
Requirement already satisfied: google-cloud-language<3,>=2.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (2.10.1)
Requirement already satisfied: google-cloud-videointelligence<3,>=2.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (2.11.3)
Requirement already satisfied: google-cloud-vision<4,>=2 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (3.4.4)
Requirement already satisfied:
google-cloud-recommendations-ai<0.11.0,>=0.1.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (0.10.4)
Requirement already satisfied: google-cloud-aiplatform<2.0,>=1.26.0 in
/usr/local/lib/python3.8/site-packages (from apache-beam==2.51.0.dev0) (1.29.0)
Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in
/usr/local/lib/python3.8/site-packages (from
google-api-core<3,>=2.0.0->apache-beam==2.51.0.dev0) (1.60.0)
Requirement already satisfied: six>=1.12.0 in
/usr/local/lib/python3.8/site-packages (from
google-apitools<0.5.32,>=0.5.31->apache-beam==2.51.0.dev0) (1.16.0)
Requirement already satisfied: oauth2client>=1.4.12 in
/usr/local/lib/python3.8/site-packages (from
google-apitools<0.5.32,>=0.5.31->apache-beam==2.51.0.dev0) (4.1.3)
Requirement already satisfied: pyasn1-modules>=0.2.1 in
/usr/local/lib/python3.8/site-packages (from
google-auth<3,>=1.18.0->apache-beam==2.51.0.dev0) (0.3.0)
Requirement already satisfied: urllib3<2.0 in
/usr/local/lib/python3.8/site-packages (from
google-auth<3,>=1.18.0->apache-beam==2.51.0.dev0) (1.26.16)
Requirement already satisfied: rsa<5,>=3.1.4 in
/usr/local/lib/python3.8/site-packages (from
google-auth<3,>=1.18.0->apache-beam==2.51.0.dev0) (4.9)
Requirement already satisfied:
google-cloud-resource-manager<3.0.0dev,>=1.3.3 in
/usr/local/lib/python3.8/site-packages (from
google-cloud-aiplatform<2.0,>=1.26.0->apache-beam==2.51.0.dev0) (1.10.3)
Requirement already satisfied: google-cloud-storage<3.0.0dev,>=1.32.0 in
/usr/local/lib/python3.8/site-packages (from
google-cloud-aiplatform<2.0,>=1.26.0->apache-beam==2.51.0.dev0) (2.10.0)
Requirement already satisfied: shapely<2.0.0 in
/usr/local/lib/python3.8/site-packages (from
google-cloud-aiplatform<2.0,>=1.26.0->apache-beam==2.51.0.dev0) (1.8.5.post1)
Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in
/usr/local/lib/python3.8/site-packages (from
google-cloud-bigquery<4,>=2.0.0->apache-beam==2.51.0.dev0) (2.5.0)
Requirement already satisfied: grpc-google-iam-v1<1.0.0dev,>=0.12.4 in
/usr/local/lib/python3.8/site-packages (from
google-cloud-bigtable<3,>=2.19.0->apache-beam==2.51.0.dev0) (0.12.6)
Requirement already satisfied: grpcio-status>=1.33.2 in
/usr/local/lib/python3.8/site-packages (from
google-cloud-pubsub<3,>=2.1.0->apache-beam==2.51.0.dev0) (1.56.2)
Requirement already satisfied: overrides<7.0.0,>=6.0.1 in
/usr/local/lib/python3.8/site-packages (from
google-cloud-pubsublite<2,>=1.2.0->apache-beam==2.51.0.dev0) (6.5.0)
Requirement already satisfied: sqlparse>=0.4.4 in
/usr/local/lib/python3.8/site-packages (from
google-cloud-spanner<4,>=3.0.0->apache-beam==2.51.0.dev0) (0.4.4)
Requirement already satisfied: docopt in
/usr/local/lib/python3.8/site-packages (from
hdfs<3.0.0,>=2.1.0->apache-beam==2.51.0.dev0) (0.6.2)
Requirement already satisfied:
pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in
/usr/local/lib/python3.8/site-packages (from
httplib2<0.23.0,>=0.8->apache-beam==2.51.0.dev0) (3.1.1)
Requirement already satisfied: dnspython<3.0.0,>=1.16.0 in
/usr/local/lib/python3.8/site-packages (from
pymongo<5.0.0,>=3.8.0->apache-beam==2.51.0.dev0) (2.4.2)
Requirement already satisfied: certifi>=2017.4.17 in
/usr/local/lib/python3.8/site-packages (from
requests<3.0.0,>=2.24.0->apache-beam==2.51.0.dev0) (2023.7.22)
Requirement already satisfied: idna<4,>=2.5 in
/usr/local/lib/python3.8/site-packages (from
requests<3.0.0,>=2.24.0->apache-beam==2.51.0.dev0) (3.4)
Requirement already satisfied: charset-normalizer<4,>=2 in
/usr/local/lib/python3.8/site-packages (from
requests<3.0.0,>=2.24.0->apache-beam==2.51.0.dev0) (3.2.0)
Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in
/usr/local/lib/python3.8/site-packages (from
google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<4,>=2.0.0->apache-beam==2.51.0.dev0)
(1.5.0)
Requirement already satisfied: pyasn1>=0.1.7 in
/usr/local/lib/python3.8/site-packages (from
oauth2client>=1.4.12->google-apitools<0.5.32,>=0.5.31->apache-beam==2.51.0.dev0)
(0.5.0)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam:
filename=apache_beam-2.51.0.dev0-cp38-cp38-linux_x86_64.whl size=15380624
sha256=41328e7e19a40e56bf4c4fe73746c3c05eef1491460d08f832be503d7971d028
Stored in directory:
/tmp/pip-ephem-wheel-cache-0qlxuiv1/wheels/de/36/40/7ee9d01abc53edf83ff29da2929769e8f314a1ac82af2171e8
Successfully built apache-beam
Installing collected packages: apache-beam
Attempting uninstall: apache-beam
Found existing installation: apache-beam 2.51.0.dev0
Uninstalling apache-beam-2.51.0.dev0:
Successfully uninstalled apache-beam-2.51.0.dev0
Successfully installed apache-beam-2.51.0.dev0
WARNING: Running pip as the 'root' user can result in broken permissions and
conflicting behaviour with the system package manager. It is recommended to use
a virtual environment instead: https://pip.pypa.io/warnings/venv
[36mINFO[0m[0152] Taking snapshot of full filesystem...
[36mINFO[0m[0160] Pushing layer
us.gcr.io/apache-beam-testing/jrmccluskey/prebuild_python38_sdk/beam_python_prebuilt_sdk/cache:4a91c7876a24f171c950805853e56721015e9a4edb22d2b9ee98f303b196ccc4
to cache now
[36mINFO[0m[0160] Pushing image to
us.gcr.io/apache-beam-testing/jrmccluskey/prebuild_python38_sdk/beam_python_prebuilt_sdk/cache:4a91c7876a24f171c950805853e56721015e9a4edb22d2b9ee98f303b196ccc4
[36mINFO[0m[0165] Pushed
us.gcr.io/apache-beam-testing/jrmccluskey/prebuild_python38_sdk/beam_python_prebuilt_sdk/cache@sha256:1b58305a81264f133ab511f3dfc16897e37cb48f45f293e86656853f7536aa1f
[36mINFO[0m[0165] Pushing image to
us.gcr.io/apache-beam-testing/jrmccluskey/prebuild_python38_sdk/beam_python_prebuilt_sdk:2cd52080-9790-46bb-90c3-7a82263d4907
[36mINFO[0m[0168] Pushed
us.gcr.io/apache-beam-testing/jrmccluskey/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:f107da60eca0482c98fd574b477fefa8efa6d5acb3a32b1d0a95444f98ddd07b
PUSH
DONE
```
If this isn't what you're looking for, I'm afraid I'm not quite following
what your hangup is here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
