See <https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Spark/1585/display/redirect?page=changes>
Changes: [jozo.vilcek] [BEAM-10284] Add option to pass configuration into ParquetIO.Sink [jozo.vilcek] [BEAM-10284] Remove hadoop from ParquetIO.Sink public API [ajamato] [BEAM-10381] Increase retries in pull_licenses_java.py from 3 to 9. [noreply] [BEAM-10220] Add support for REQUIRE_MISSING in RowJsonDeserializer, [daniel.o.programmer] [BEAM-10312] Sub-element progress accounted for during splitting. ------------------------------------------ [...truncated 64.67 KB...] # Verify container has already been built docker images --format "{{.Repository}}:{{.Tag}}" | grep $PYTHON_CONTAINER_IMAGE apache/beam_python3.7_sdk:2.23.0.dev # Set up Python environment virtualenv -p python$PYTHON_VERSION $ENV_DIR Using base prefix '/usr' New python executable in <https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Spark/ws/src/build/gradleenv/-2047437407/bin/python3.7> Also creating executable in <https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Spark/ws/src/build/gradleenv/-2047437407/bin/python> Installing setuptools, pip, wheel... done. Running virtualenv with interpreter /usr/bin/python3.7 . $ENV_DIR/bin/activate # This file must be used with "source bin/activate" *from bash* # you cannot run it directly deactivate () { unset -f pydoc >/dev/null 2>&1 # reset old environment variables # ! [ -z ${VAR+_} ] returns true if VAR is declared at all if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then PATH="$_OLD_VIRTUAL_PATH" export PATH unset _OLD_VIRTUAL_PATH fi if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME" export PYTHONHOME unset _OLD_VIRTUAL_PYTHONHOME fi # This should detect bash and zsh, which have a hash command that must # be called to get it to forget past commands. Without forgetting # past commands the $PATH changes we made may not be respected if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then hash -r 2>/dev/null fi if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then PS1="$_OLD_VIRTUAL_PS1" export PS1 unset _OLD_VIRTUAL_PS1 fi unset VIRTUAL_ENV if [ ! "${1-}" = "nondestructive" ] ; then # Self destruct! unset -f deactivate fi } # unset irrelevant variables deactivate nondestructive VIRTUAL_ENV="<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Spark/ws/src/build/gradleenv/-2047437407"> export VIRTUAL_ENV _OLD_VIRTUAL_PATH="$PATH" PATH="$VIRTUAL_ENV/bin:$PATH" export PATH # unset PYTHONHOME if set if ! [ -z "${PYTHONHOME+_}" ] ; then _OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME" unset PYTHONHOME fi if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then _OLD_VIRTUAL_PS1="${PS1-}" if [ "x" != x ] ; then PS1="${PS1-}" else PS1="(`basename \"$VIRTUAL_ENV\"`) ${PS1-}" fi export PS1 fi basename "$VIRTUAL_ENV" # Make sure to unalias pydoc if it's already there alias pydoc 2>/dev/null >/dev/null && unalias pydoc || true pydoc () { python -m pydoc "$@" } # This should detect bash and zsh, which have a hash command that must # be called to get it to forget past commands. Without forgetting # past commands the $PATH changes we made may not be respected if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then hash -r 2>/dev/null fi pip install --retries 10 -e $PYTHON_ROOT_DIR Obtaining file://<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Spark/ws/src/sdks/python> Processing /home/jenkins/.cache/pip/wheels/dc/9a/e9/49e627353476cec8484343c4ab656f1e0d783ee77b9dde2d1f/crcmod-1.7-cp37-cp37m-linux_x86_64.whl Processing /home/jenkins/.cache/pip/wheels/a4/61/fd/c57e374e580aa78a45ed78d5859b3a44436af17e22ca53284f/dill-0.3.1.1-py3-none-any.whl Collecting fastavro<0.24,>=0.21.4 Using cached fastavro-0.23.5-cp37-cp37m-manylinux2010_x86_64.whl (1.4 MB) Processing /home/jenkins/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0/future-0.18.2-py3-none-any.whl Collecting grpcio<2,>=1.29.0 Using cached grpcio-1.30.0-cp37-cp37m-manylinux2010_x86_64.whl (3.0 MB) Processing /home/jenkins/.cache/pip/wheels/0a/7d/38/ea4eaf831518e6cd867b515b88919a9785eb66f11def5ab859/hdfs-2.5.8-py3-none-any.whl Collecting httplib2<0.18.0,>=0.8 Using cached httplib2-0.17.4-py3-none-any.whl (95 kB) Collecting mock<3.0.0,>=1.0.1 Using cached mock-2.0.0-py2.py3-none-any.whl (56 kB) Collecting numpy<2,>=1.14.3 Using cached numpy-1.19.0-cp37-cp37m-manylinux2010_x86_64.whl (14.6 MB) Collecting pymongo<4.0.0,>=3.8.0 Using cached pymongo-3.10.1-cp37-cp37m-manylinux2014_x86_64.whl (462 kB) Processing /home/jenkins/.cache/pip/wheels/86/73/7a/3b3f76a2142176605ff38fbca574327962c71e25a43197a4c1/oauth2client-3.0.0-py3-none-any.whl Collecting protobuf<4,>=3.5.0.post1 Using cached protobuf-3.12.2-cp37-cp37m-manylinux1_x86_64.whl (1.3 MB) Collecting pydot<2,>=1.2.0 Using cached pydot-1.4.1-py2.py3-none-any.whl (19 kB) Collecting python-dateutil<3,>=2.8.0 Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB) Collecting pytz>=2018.3 Using cached pytz-2020.1-py2.py3-none-any.whl (510 kB) Collecting typing-extensions<3.8.0,>=3.7.0 Using cached typing_extensions-3.7.4.2-py3-none-any.whl (22 kB) Processing /home/jenkins/.cache/pip/wheels/bc/49/5f/fdb5b9d85055c478213e0158ac122b596816149a02d82e0ab1/avro_python3-1.9.2.1-py3-none-any.whl Collecting pyarrow<0.18.0,>=0.15.1 Using cached pyarrow-0.17.1-cp37-cp37m-manylinux2014_x86_64.whl (63.8 MB) Collecting six>=1.5.2 Using cached six-1.15.0-py2.py3-none-any.whl (10 kB) Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl Collecting requests>=2.7.0 Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB) Collecting pbr>=0.11 Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB) Collecting rsa>=3.1.4 Using cached rsa-4.6-py3-none-any.whl (47 kB) Collecting pyasn1-modules>=0.0.5 Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB) Collecting pyasn1>=0.1.7 Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB) Requirement already satisfied: setuptools in ./build/gradleenv/-2047437407/lib/python3.7/site-packages (from protobuf<4,>=3.5.0.post1->apache-beam==2.23.0.dev0) (47.3.1) Collecting pyparsing>=2.1.4 Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB) Collecting chardet<4,>=3.0.2 Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB) Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 Using cached urllib3-1.25.9-py2.py3-none-any.whl (126 kB) Collecting idna<3,>=2.5 Using cached idna-2.10-py2.py3-none-any.whl (58 kB) Collecting certifi>=2017.4.17 Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB) Installing collected packages: crcmod, dill, pytz, fastavro, future, six, grpcio, docopt, chardet, urllib3, idna, certifi, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, rsa, pyasn1-modules, oauth2client, protobuf, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, apache-beam Running setup.py develop for apache-beam Successfully installed apache-beam avro-python3-1.9.2.1 certifi-2020.6.20 chardet-3.0.4 crcmod-1.7 dill-0.3.1.1 docopt-0.6.2 fastavro-0.23.5 future-0.18.2 grpcio-1.30.0 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 mock-2.0.0 numpy-1.19.0 oauth2client-3.0.0 pbr-5.4.5 protobuf-3.12.2 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pydot-1.4.1 pymongo-3.10.1 pyparsing-2.4.7 python-dateutil-2.8.1 pytz-2020.1 requests-2.24.0 rsa-4.6 six-1.15.0 typing-extensions-3.7.4.2 urllib3-1.25.9 PIPELINE_PY=" import apache_beam as beam from apache_beam.options.pipeline_options import PipelineOptions from apache_beam.options.pipeline_options import SetupOptions from apache_beam.testing.util import assert_that from apache_beam.testing.util import equal_to from apache_beam.transforms import Create from apache_beam.transforms import Map # To test that our main session is getting plumbed through artifact staging # correctly, create a global variable. If the main session is not plumbed # through properly, global_var will be undefined and the pipeline will fail. global_var = 1 pipeline_options = PipelineOptions() pipeline_options.view_as(SetupOptions).save_main_session = True pipeline = beam.Pipeline(options=pipeline_options) pcoll = (pipeline | Create([0, 1, 2]) | Map(lambda x: x + global_var)) assert_that(pcoll, equal_to([1, 2, 3])) result = pipeline.run() result.wait_until_finish() " if [[ "$RUNNER" = "FlinkRunner" ]]; then INPUT_JAR_ARG="flink_job_server_jar" else INPUT_JAR_ARG="spark_job_server_jar" fi # Create the jar OUTPUT_JAR=flink-test-$(date +%Y%m%d-%H%M%S).jar date +%Y%m%d-%H%M%S (python -c "$PIPELINE_PY" \ --runner "$RUNNER" \ --"$INPUT_JAR_ARG" "$JOB_SERVER_JAR" \ --output_executable_path $OUTPUT_JAR \ --parallelism 1 \ --sdk_worker_parallelism 1 \ --environment_type DOCKER \ --environment_config=$PYTHON_CONTAINER_IMAGE \ ) || TEST_EXIT_CODE=$? # don't fail fast here; clean up before exiting WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--parallelism', '1'] ERROR:root:java.lang.IllegalArgumentException: Expected exactly one jar on sun.misc.Launcher$AppClassLoader@1b6d3586 Traceback (most recent call last): File "<string>", line 24, in <module> File "<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Spark/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py",> line 546, in wait_until_finish raise self._runtime_exception RuntimeError: Pipeline BeamApp-jenkins-0701001121-757bb2a9_b1464b24-4bcf-40ea-b47e-856a9b97d3dc failed in state FAILED: java.lang.IllegalArgumentException: Expected exactly one jar on sun.misc.Launcher$AppClassLoader@1b6d3586 if [[ "$TEST_EXIT_CODE" -eq 0 ]]; then # Execute the jar java -jar $OUTPUT_JAR || TEST_EXIT_CODE=$? fi rm -rf $ENV_DIR rm -f $OUTPUT_JAR if [[ "$TEST_EXIT_CODE" -eq 0 ]]; then echo ">>> SUCCESS" else echo ">>> FAILURE" fi exit $TEST_EXIT_CODE >>> FAILURE > Task :runners:spark:job-server:testJavaJarCreatorPy37 FAILED FAILURE: Build failed with an exception. * Where: Build file '<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Spark/ws/src/runners/spark/job-server/build.gradle'> line: 166 * What went wrong: Execution failed for task ':runners:spark:job-server:testJavaJarCreatorPy37'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 11m 2s 67 actionable tasks: 51 executed, 15 from cache, 1 up-to-date Publishing build scan... https://gradle.com/s/5kn2z3oo5t4zk Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: builds-unsubscr...@beam.apache.org For additional commands, e-mail: builds-h...@beam.apache.org