See <https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Spark/2752/display/redirect>
Changes: ------------------------------------------ [...truncated 62.20 KB...] Resolving ./github.com/apache/beam/sdks/go@<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Spark/ws/src/sdks/go> > Task :sdks:python:container:installDependencies > Task :sdks:python:container:buildDarwinAmd64 > Task :sdks:python:container:buildLinuxAmd64 > Task :sdks:python:container:goBuild > Task :sdks:python:container:py37:copyLauncherDependencies > Task :sdks:python:container:py37:dockerPrepare > Task :sdks:python:container:py37:docker > Task :sdks:python:test-suites:portable:py37:testJavaJarCreatorSparkRunner while [[ $# -gt 0 ]] do key="$1" case $key in --job_server_jar) JOB_SERVER_JAR="$2" shift # past argument shift # past value ;; --runner) RUNNER="$2" shift # past argument shift # past value ;; --env_dir) ENV_DIR="$2" shift # past argument shift # past value ;; --python_root_dir) PYTHON_ROOT_DIR="$2" shift # past argument shift # past value ;; --python_version) PYTHON_VERSION="$2" shift # past argument shift # past value ;; --python_container_image) PYTHON_CONTAINER_IMAGE="$2" shift # past argument shift # past value ;; *) # unknown option echo "Unknown option: $1" exit 1 ;; esac done # Go to the root of the repository cd $(git rev-parse --show-toplevel) git rev-parse --show-toplevel # Verify docker command exists command -v docker docker -v # Verify container has already been built echo "Checking for Docker image ${PYTHON_CONTAINER_IMAGE}" docker images --format "{{.Repository}}:{{.Tag}}" | grep $PYTHON_CONTAINER_IMAGE # Set up Python environment virtualenv -p python$PYTHON_VERSION $ENV_DIR . $ENV_DIR/bin/activate # This file must be used with "source bin/activate" *from bash* # you cannot run it directly deactivate () { unset -f pydoc >/dev/null 2>&1 # reset old environment variables # ! [ -z ${VAR+_} ] returns true if VAR is declared at all if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then PATH="$_OLD_VIRTUAL_PATH" export PATH unset _OLD_VIRTUAL_PATH fi if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME" export PYTHONHOME unset _OLD_VIRTUAL_PYTHONHOME fi # This should detect bash and zsh, which have a hash command that must # be called to get it to forget past commands. Without forgetting # past commands the $PATH changes we made may not be respected if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then hash -r 2>/dev/null fi if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then PS1="$_OLD_VIRTUAL_PS1" export PS1 unset _OLD_VIRTUAL_PS1 fi unset VIRTUAL_ENV if [ ! "${1-}" = "nondestructive" ] ; then # Self destruct! unset -f deactivate fi } # unset irrelevant variables deactivate nondestructive VIRTUAL_ENV="<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Spark/ws/src/build/gradleenv/2022703441";> export VIRTUAL_ENV _OLD_VIRTUAL_PATH="$PATH" PATH="$VIRTUAL_ENV/bin:$PATH" export PATH # unset PYTHONHOME if set if ! [ -z "${PYTHONHOME+_}" ] ; then _OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME" unset PYTHONHOME fi if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then _OLD_VIRTUAL_PS1="${PS1-}" if [ "x" != x ] ; then PS1="${PS1-}" else PS1="(`basename \"$VIRTUAL_ENV\"`) ${PS1-}" fi export PS1 fi basename "$VIRTUAL_ENV" # Make sure to unalias pydoc if it's already there alias pydoc 2>/dev/null >/dev/null && unalias pydoc || true pydoc () { python -m pydoc "$@" } # This should detect bash and zsh, which have a hash command that must # be called to get it to forget past commands. Without forgetting # past commands the $PATH changes we made may not be respected if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then hash -r 2>/dev/null fi pip install --retries 10 -e $PYTHON_ROOT_DIR PIPELINE_PY=" import apache_beam as beam from apache_beam.options.pipeline_options import PipelineOptions from apache_beam.options.pipeline_options import SetupOptions from apache_beam.testing.util import assert_that from apache_beam.testing.util import equal_to from apache_beam.transforms import Create from apache_beam.transforms import Map # To test that our main session is getting plumbed through artifact staging # correctly, create a global variable. If the main session is not plumbed # through properly, global_var will be undefined and the pipeline will fail. global_var = 1 pipeline_options = PipelineOptions() pipeline_options.view_as(SetupOptions).save_main_session = True pipeline = beam.Pipeline(options=pipeline_options) pcoll = (pipeline | Create([0, 1, 2]) | Map(lambda x: x + global_var)) assert_that(pcoll, equal_to([1, 2, 3])) result = pipeline.run() result.wait_until_finish() " if [[ "$RUNNER" = "FlinkRunner" ]]; then INPUT_JAR_ARG="flink_job_server_jar" else INPUT_JAR_ARG="spark_job_server_jar" fi # Create the jar OUTPUT_JAR="test-pipeline-${RUNNER}-$(date +%Y%m%d-%H%M%S).jar" date +%Y%m%d-%H%M%S (python -c "$PIPELINE_PY" \ --runner "$RUNNER" \ --"$INPUT_JAR_ARG" "$JOB_SERVER_JAR" \ --output_executable_path $OUTPUT_JAR \ --parallelism 1 \ --sdk_worker_parallelism 1 \ --environment_type DOCKER \ --environment_options "docker_container_image=$PYTHON_CONTAINER_IMAGE" \ ) || TEST_EXIT_CODE=$? # don't fail fast here; clean up before exiting WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. WARNING:root:Waiting for grpc channel to be ready at localhost:52881. WARNING:root:Waiting for grpc channel to be ready at localhost:52881. WARNING:root:Waiting for grpc channel to be ready at localhost:52881. WARNING:root:Waiting for grpc channel to be ready at localhost:52881. WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--parallelism', '1'] if [[ "$TEST_EXIT_CODE" -eq 0 ]]; then # Execute the jar java -jar $OUTPUT_JAR || TEST_EXIT_CODE=$? fi Exception in thread "main" java.lang.RuntimeException: Job BeamApp-jenkins-0419002255-282dfed1_ad386867-13e5-44d4-a344-89c2ab821617 failed. at org.apache.beam.runners.spark.SparkPipelineRunner.main(SparkPipelineRunner.java:270) Caused by: java.lang.NullPointerException at org.apache.beam.runners.spark.SparkPipelineRunner.run(SparkPipelineRunner.java:120) at org.apache.beam.runners.spark.SparkPipelineRunner.main(SparkPipelineRunner.java:268) rm -rf $ENV_DIR rm -f $OUTPUT_JAR if [[ "$TEST_EXIT_CODE" -eq 0 ]]; then echo ">>> SUCCESS" else echo ">>> FAILURE" fi exit $TEST_EXIT_CODE > Task :sdks:python:test-suites:portable:py37:testJavaJarCreatorSparkRunner > FAILED FAILURE: Build failed with an exception. * Where: Script '<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Spark/ws/src/sdks/python/test-suites/portable/common.gradle'> line: 211 * What went wrong: Execution failed for task ':sdks:python:test-suites:portable:py37:testJavaJarCreatorSparkRunner'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 23m 22s 89 actionable tasks: 63 executed, 24 from cache, 2 up-to-date Gradle was unable to watch the file system for changes. The inotify watches limit is too low. Publishing build scan... https://gradle.com/s/zw4cbumssg6p4 Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
