See <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont_with_RC/121/display/redirect?page=changes>
Changes: [noreply] Support configuration of maxCapacityPerShard when reading from Kinesis ------------------------------------------ [...truncated 2.52 MB...] #16 1.036 objsize==0.6.1 #16 1.036 opt-einsum==3.3.0 #16 1.036 orjson==3.8.12 #16 1.036 overrides==6.5.0 #16 1.036 packaging==23.1 #16 1.036 pandas==1.5.3 #16 1.036 parameterized==0.9.0 #16 1.036 pip==23.1.2 #16 1.036 pluggy==1.0.0 #16 1.036 proto-plus==1.22.2 #16 1.036 protobuf==4.22.5 #16 1.036 psycopg2-binary==2.9.6 #16 1.036 pyarrow==11.0.0 #16 1.036 pyasn1==0.5.0 #16 1.036 pyasn1-modules==0.3.0 #16 1.036 pycparser==2.21 #16 1.036 pydot==1.4.2 #16 1.036 PyHamcrest==2.0.4 #16 1.036 pymongo==4.4.0b0 #16 1.036 PyMySQL==1.0.3 #16 1.036 pyparsing==3.1.0b1 #16 1.036 pytest==7.3.1 #16 1.036 pytest-timeout==2.1.0 #16 1.036 pytest-xdist==3.3.0 #16 1.036 python-dateutil==2.8.2 #16 1.036 python-snappy==0.6.1 #16 1.036 pytz==2023.3 #16 1.036 PyYAML==6.0 #16 1.036 regex==2023.5.5 #16 1.036 requests==2.30.0 #16 1.036 requests-mock==1.10.0 #16 1.036 requests-oauthlib==1.3.1 #16 1.036 rsa==4.9 #16 1.036 scikit-learn==1.2.2 #16 1.036 scipy==1.10.1 #16 1.036 setuptools==67.7.2 #16 1.036 six==1.16.0 #16 1.036 sortedcontainers==2.4.0 #16 1.036 soupsieve==2.4.1 #16 1.036 SQLAlchemy==1.4.48 #16 1.036 sqlparse==0.4.4 #16 1.036 tenacity==8.2.2 #16 1.036 tensorboard==2.13.0 #16 1.036 tensorboard-data-server==0.7.0 #16 1.036 tensorflow==2.13.0rc0 #16 1.036 tensorflow-estimator==2.13.0rc0 #16 1.036 tensorflow-io-gcs-filesystem==0.32.0 #16 1.036 termcolor==2.3.0 #16 1.036 testcontainers==3.7.1 #16 1.036 threadpoolctl==3.1.0 #16 1.036 tomli==2.0.1 #16 1.036 tqdm==4.65.0 #16 1.036 typing_extensions==4.5.0 #16 1.036 urllib3==1.26.15 #16 1.036 websocket-client==1.5.1 #16 1.036 Werkzeug==2.3.4 #16 1.036 wheel==0.40.0 #16 1.036 wrapt==1.14.1 #16 1.036 zipp==3.15.0 #16 1.036 zstandard==0.21.0 #16 DONE 1.1s #17 [beam 13/16] COPY target/LICENSE /opt/apache/beam/ #17 DONE 0.0s #18 [beam 14/16] COPY target/LICENSE.python /opt/apache/beam/ #18 DONE 0.0s #19 [beam 15/16] COPY target/NOTICE /opt/apache/beam/ #19 DONE 0.0s #20 [beam 16/16] COPY target/launcher/linux_amd64/boot /opt/apache/beam/ #20 DONE 0.2s #21 [third_party_licenses 1/4] COPY target/license_scripts /tmp/license_scripts/ #21 DONE 0.1s #22 [third_party_licenses 2/4] COPY target/go-licenses/* /opt/apache/beam/third_party_licenses/golang/ #22 DONE 0.1s #23 [third_party_licenses 3/4] COPY target/license_scripts /tmp/license_scripts/ #23 DONE 0.1s #24 [third_party_licenses 4/4] RUN if [ "true" = "true" ] ; then pip install 'pip-licenses<5' pyyaml tenacity && python /tmp/license_scripts/pull_licenses_py.py ; fi #24 1.171 Collecting pip-licenses<5 #24 1.247 Downloading pip_licenses-4.3.1-py3-none-any.whl (19 kB) #24 1.256 Requirement already satisfied: pyyaml in /usr/local/lib/python3.9/site-packages (6.0) #24 1.257 Requirement already satisfied: tenacity in /usr/local/lib/python3.9/site-packages (8.2.2) #24 1.320 Collecting prettytable>=2.3.0 (from pip-licenses<5) #24 1.337 Downloading prettytable-3.7.0-py3-none-any.whl (27 kB) #24 1.400 Collecting wcwidth (from prettytable>=2.3.0->pip-licenses<5) #24 1.418 Downloading wcwidth-0.2.6-py2.py3-none-any.whl (29 kB) #24 2.639 Installing collected packages: wcwidth, prettytable, pip-licenses #24 2.814 Successfully installed pip-licenses-4.3.1 prettytable-3.7.0 wcwidth-0.2.6 #24 2.814 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv #24 9.781 Skip pulling license for bs4 #24 9.781 Traceback (most recent call last): #24 9.781 File "/tmp/license_scripts/pull_licenses_py.py", line 190, in <module> #24 9.782 raise RuntimeError( #24 9.782 RuntimeError: Could not retrieve licences for packages ['keras'] in Python3.9 environment. #24 9.782 These licenses were not able to be pulled automatically. Please search code source of the dependencies on the internet and add urls to RAW license file at sdks/python/container/license_scripts/dep_urls_py.yaml for each missing license and rerun the test. If no such urls can be found, you need to manually add LICENSE and NOTICE (if available) files at sdks/python/container/license_scripts/manual_licenses/{dep}/ and add entries to sdks/python/container/license_scripts/dep_urls_py.yaml. #24 ERROR: executor failed running [/bin/sh -c if [ "$pull_licenses" = "true" ] ; then pip install 'pip-licenses<5' pyyaml tenacity && python /tmp/license_scripts/pull_licenses_py.py ; fi]: exit code: 1 ------ > [third_party_licenses 4/4] RUN if [ "true" = "true" ] ; then pip > install 'pip-licenses<5' pyyaml tenacity && python > /tmp/license_scripts/pull_licenses_py.py ; fi: #24 1.418 Downloading wcwidth-0.2.6-py2.py3-none-any.whl (29 kB) #24 2.639 Installing collected packages: wcwidth, prettytable, pip-licenses #24 2.814 Successfully installed pip-licenses-4.3.1 prettytable-3.7.0 wcwidth-0.2.6 #24 2.814 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv #24 9.781 Skip pulling license for bs4 #24 9.781 Traceback (most recent call last): #24 9.781 File "/tmp/license_scripts/pull_licenses_py.py", line 190, in <module> #24 9.782 raise RuntimeError( #24 9.782 RuntimeError: Could not retrieve licences for packages ['keras'] in Python3.9 environment. #24 9.782 These licenses were not able to be pulled automatically. Please search code source of the dependencies on the internet and add urls to RAW license file at sdks/python/container/license_scripts/dep_urls_py.yaml for each missing license and rerun the test. If no such urls can be found, you need to manually add LICENSE and NOTICE (if available) files at sdks/python/container/license_scripts/manual_licenses/{dep}/ and add entries to sdks/python/container/license_scripts/dep_urls_py.yaml. ------ ERROR: failed to solve: executor failed running [/bin/sh -c if [ "$pull_licenses" = "true" ] ; then pip install 'pip-licenses<5' pyyaml tenacity && python /tmp/license_scripts/pull_licenses_py.py ; fi]: exit code: 1 > Task :sdks:python:container:py39:docker FAILED > Task :sdks:python:test-suites:dataflow:py37:validatesContainer function cleanup_container { # Delete the container locally and remotely docker rmi $CONTAINER:$TAG || echo "Failed to remove container image" for image in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH) do docker rmi $image || echo "Failed to remove prebuilt sdk container image" done gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to delete container" for digest in $(gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk --format="get(digest)") do gcloud container images delete $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk@$digest --force-delete-tags --quiet || echo "Failed to remove prebuilt sdk container image" done echo "Removed the container" } trap cleanup_container EXIT echo ">>> Successfully built and push container $CONTAINER" cd sdks/python SDK_LOCATION=$2 # Run ValidatesRunner tests on Google Cloud Dataflow service echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST" pytest -o junit_suite_name=$IMAGE_NAME \ -m="it_validatescontainer" \ --show-capture=no \ --numprocesses=1 \ --timeout=1800 \ --junitxml=$XUNIT_FILE \ --ignore-glob '.*py3\d?\.py$' \ --log-cli-level=INFO \ --test-pipeline-options=" \ --runner=TestDataflowRunner \ --project=$PROJECT \ --region=$REGION \ --sdk_container_image=$CONTAINER:$TAG \ --staging_location=$GCS_LOCATION/staging-validatesrunner-test \ --temp_location=$GCS_LOCATION/temp-validatesrunner-test \ --output=$GCS_LOCATION/output \ --sdk_location=$SDK_LOCATION \ --num_workers=1 \ --docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH" echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST" cleanup_container WARNING: Successfully resolved tag to sha256, but it is recommended to use sha256 directly. Digests: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk@sha256:ec3439608883e2559bdbdeed9102d7f24f9f0459b3a2ee362e619c706b628710 Associated tags: - 20230515-082721439798829 Tags: - us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk:20230515-082721439798829 Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk:20230515-082721439798829]. Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk@sha256:ec3439608883e2559bdbdeed9102d7f24f9f0459b3a2ee362e619c706b628710]. Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:26331803bd7df5b42b5723527f172ee7baa7f0cbfa762cd8ec192c3fb8de7998 Associated tags: - e0d9be7e-7396-4c09-972d-54682650669b Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk:e0d9be7e-7396-4c09-972d-54682650669b]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:26331803bd7df5b42b5723527f172ee7baa7f0cbfa762cd8ec192c3fb8de7998]. Digests: - us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:046ae02f76ad5e79e007d307ec8b63e895b68380d4880b46098df12fdcabbc22 Associated tags: - 30635ee8-c38d-47ad-877c-8ff14ba2d5ff Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk:30635ee8-c38d-47ad-877c-8ff14ba2d5ff]. Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:046ae02f76ad5e79e007d307ec8b63e895b68380d4880b46098df12fdcabbc22]. FAILURE: Build completed with 4 failures. 1: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py311:docker'. > Process 'command 'docker'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py310:docker'. > Process 'command 'docker'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. ============================================================================== 3: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py38:docker'. > Process 'command 'docker'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. ============================================================================== 4: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py39:docker'. > Process 'command 'docker'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. ============================================================================== * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 41m 20s 74 actionable tasks: 68 executed, 6 up-to-date Publishing build scan... https://ge.apache.org/s/qno3txmybw3ns Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
