See <https://builds.apache.org/job/beam_PerformanceTests_Python/1193/display/redirect?page=changes>
Changes: [sidhom] Add in-process ManagedChannelFactory and update tests to use it [sidhom] Add common pipeline options for portable runners [sidhom] Set artifact names explicitly while staging to service [sidhom] Add CloseableResource to wrap non-closeable resources that must be [sidhom] [BEAM-4071] Add Portable Runner Job API shim [aaltay] [BEAM-3255] Enabling gradle-based release process (#5187) [ekirpichov] Moves standard URNs into .proto files [ekirpichov] Refactors common_urns.py [ekirpichov] Fixes pylint errors [ekirpichov] Fixes compile errors [ekirpichov] Workaround for protobuf compiler bug [ekirpichov] Fixes more references to common_urns ------------------------------------------ [...truncated 816.84 KB...] [INFO] Apache Beam :: Model ............................... SUCCESS [ 0.029 s] [INFO] Apache Beam :: Model :: Pipeline ................... SUCCESS [ 5.413 s] [INFO] Apache Beam :: Model :: Job Management ............. SUCCESS [ 2.890 s] [INFO] Apache Beam :: Model :: Fn Execution ............... SUCCESS [ 3.534 s] [INFO] Apache Beam :: SDKs ................................ SUCCESS [ 0.145 s] [INFO] Apache Beam :: SDKs :: Go .......................... SUCCESS [ 22.851 s] [INFO] Apache Beam :: SDKs :: Go :: Container ............. SUCCESS [ 19.565 s] [INFO] Apache Beam :: SDKs :: Java ........................ SUCCESS [ 0.133 s] [INFO] Apache Beam :: SDKs :: Java :: Core ................ SUCCESS [ 21.859 s] [INFO] Apache Beam :: SDKs :: Java :: Fn Execution ........ SUCCESS [ 2.874 s] [INFO] Apache Beam :: SDKs :: Java :: Extensions .......... SUCCESS [ 0.047 s] [INFO] Apache Beam :: SDKs :: Java :: Extensions :: Google Cloud Platform Core SUCCESS [ 3.290 s] [INFO] Apache Beam :: Runners ............................. SUCCESS [ 0.061 s] [INFO] Apache Beam :: Runners :: Core Construction Java ... SUCCESS [ 4.354 s] [INFO] Apache Beam :: Runners :: Core Java ................ SUCCESS [ 4.227 s] [INFO] Apache Beam :: SDKs :: Java :: Harness ............. SUCCESS [ 8.126 s] [INFO] Apache Beam :: SDKs :: Java :: Container ........... SUCCESS [ 7.438 s] [INFO] Apache Beam :: SDKs :: Java :: IO .................. SUCCESS [ 0.060 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Amazon Web Services SUCCESS [ 2.670 s] [INFO] Apache Beam :: Runners :: Local Java Core .......... SUCCESS [ 1.578 s] [INFO] Apache Beam :: Runners :: Direct Java .............. SUCCESS [ 10.542 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: AMQP .......... SUCCESS [ 2.410 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Common ........ SUCCESS [ 1.776 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Cassandra ..... SUCCESS [ 2.721 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Elasticsearch . SUCCESS [ 1.900 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Elasticsearch-Tests SUCCESS [ 0.525 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Elasticsearch-Tests :: Common SUCCESS [ 0.589 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Elasticsearch-Tests :: 2.x SUCCESS [ 1.819 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Elasticsearch-Tests :: 5.x SUCCESS [ 1.877 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: XML ........... SUCCESS [ 2.270 s] [INFO] Apache Beam :: SDKs :: Java :: Extensions :: Protobuf SUCCESS [ 2.497 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform SUCCESS [ 4.969 s] [INFO] Apache Beam :: Runners :: Google Cloud Dataflow .... SUCCESS [ 8.706 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: File-based-io-tests SUCCESS [ 2.182 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Hadoop Common . SUCCESS [ 3.026 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Hadoop File System SUCCESS [ 3.657 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: JDBC .......... SUCCESS [ 3.051 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Hadoop Input Format SUCCESS [ 8.544 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: HBase ......... SUCCESS [ 5.564 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: HCatalog ...... SUCCESS [ 9.062 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: JMS ........... SUCCESS [ 2.212 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Kafka ......... SUCCESS [ 2.666 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Kinesis ....... SUCCESS [ 1.835 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: MongoDB ....... SUCCESS [ 3.124 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: MQTT .......... SUCCESS [ 2.194 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Redis ......... SUCCESS [ 2.542 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Solr .......... SUCCESS [ 4.551 s] [INFO] Apache Beam :: SDKs :: Java :: IO :: Tika .......... SUCCESS [ 3.131 s] [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes .... SUCCESS [ 0.021 s] [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Starter SUCCESS [ 7.019 s] [INFO] Apache Beam :: Examples ............................ SUCCESS [ 0.029 s] [INFO] Apache Beam :: Examples :: Java .................... SUCCESS [ 3.666 s] [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Examples SUCCESS [ 21.453 s] [INFO] Apache Beam :: SDKs :: Java :: Extensions :: Jackson SUCCESS [ 2.214 s] [INFO] Apache Beam :: SDKs :: Java :: Extensions :: Join library SUCCESS [ 1.875 s] [INFO] Apache Beam :: SDKs :: Java :: Extensions :: Sketching SUCCESS [ 2.332 s] [INFO] Apache Beam :: SDKs :: Java :: Extensions :: Sorter SUCCESS [ 2.450 s] [INFO] Apache Beam :: SDKs :: Java :: Extensions :: SQL ... SUCCESS [ 13.326 s] [INFO] Apache Beam :: SDKs :: Java :: Nexmark ............. SUCCESS [ 16.418 s] [INFO] Apache Beam :: SDKs :: Python ...................... SUCCESS [ 12.394 s] [INFO] Apache Beam :: SDKs :: Python :: Container ......... SUCCESS [ 7.195 s] [INFO] Apache Beam :: Runners :: Java Fn Execution ........ SUCCESS [ 3.208 s] [INFO] Apache Beam :: Runners :: Java Local Artifact Service SUCCESS [ 2.607 s] [INFO] Apache Beam :: Runners :: Reference ................ SUCCESS [ 0.023 s] [INFO] Apache Beam :: Runners :: Reference :: Java ........ FAILURE [ 0.578 s] [INFO] Apache Beam :: Runners :: Reference :: Job Orchestrator SKIPPED [INFO] Apache Beam :: Runners :: Flink .................... SKIPPED [INFO] Apache Beam :: Runners :: Gearpump ................. SKIPPED [INFO] Apache Beam :: Runners :: Spark .................... SKIPPED [INFO] Apache Beam :: Runners :: Apex ..................... SKIPPED [INFO] Apache Beam :: Runners :: Google Cloud Platform .... SKIPPED [INFO] Apache Beam :: Runners :: Google Cloud Platform :: GCE metadata provisioning SKIPPED [INFO] Apache Beam :: Runners :: Google Cloud Platform :: GCS artifact proxy SKIPPED [INFO] Apache Beam :: Runners :: Extensions Java .......... SKIPPED [INFO] Apache Beam :: Runners :: Extensions Java :: Metrics SKIPPED [INFO] Apache Beam :: SDKs :: Java :: Aggregated Javadoc .. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 05:13 min [INFO] Finished at: 2018-04-26T00:14:02Z [INFO] Final Memory: 532M/11154M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project beam-runners-reference-java: Compilation failure: Compilation failure: [ERROR] <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/runs/34dc0d65/beam/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/InProcessManagedChannelFactory.java>:[23,38] package org.apache.beam.sdk.fn.channel does not exist [ERROR] <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/runs/34dc0d65/beam/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/InProcessManagedChannelFactory.java>:[30,53] cannot find symbol [ERROR] symbol: class ManagedChannelFactory [ERROR] <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/runs/34dc0d65/beam/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java>:[49,38] package org.apache.beam.sdk.fn.channel does not exist [ERROR] <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/runs/34dc0d65/beam/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java>:[69,17] cannot find symbol [ERROR] symbol: class ManagedChannelFactory [ERROR] location: class org.apache.beam.runners.reference.PortableRunner [ERROR] <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/runs/34dc0d65/beam/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java>:[82,57] cannot find symbol [ERROR] symbol: class ManagedChannelFactory [ERROR] location: class org.apache.beam.runners.reference.PortableRunner [ERROR] <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/runs/34dc0d65/beam/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java>:[128,7] cannot find symbol [ERROR] symbol: class ManagedChannelFactory [ERROR] location: class org.apache.beam.runners.reference.PortableRunner [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :beam-runners-reference-java STDERR: log4j:WARN No appenders could be found for logger (freemarker.cache). log4j:WARN Please initialize the log4j system properly. Warning: Lookahead adequacy checking not being performed since option LOOKAHEAD is more than 1. Set option FORCE_LA_CHECK to true to force checking. /usr/local/lib/python2.7/dist-packages/setuptools/dist.py:397: UserWarning: Normalizing '2.5.0.dev' to '2.5.0.dev0' normalized_version, /usr/local/lib/python2.7/dist-packages/setuptools/dist.py:397: UserWarning: Normalizing '2.5.0.dev' to '2.5.0.dev0' normalized_version, <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/runs/34dc0d65/beam/sdks/python/gen_protos.py>:49: UserWarning: Installing grpcio-tools is recommended for development. warnings.warn('Installing grpcio-tools is recommended for development.') WARNING:root:Installing grpcio-tools into <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/runs/34dc0d65/beam/sdks/python/.eggs/grpcio-wheels> WARNING:root:Installing grpcio-tools took 3.72 seconds. The script virtualenv is installed in '<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/runs/34dc0d65/beam/sdks/python/target/python/bin'> which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. The scripts tox and tox-quickstart are installed in '<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/runs/34dc0d65/beam/sdks/python/target/python/bin'> which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/runs/34dc0d65/beam/sdks/python/target/python/lib/python2.7/site-packages/setuptools/dist.py>:397: UserWarning: Normalizing '2.5.0.dev' to '2.5.0.dev0' normalized_version, 2018-04-26 00:14:04,049 34dc0d65 MainThread beam_integration_benchmark(1/1) INFO Running benchmark beam_integration_benchmark 2018-04-26 00:14:04,068 34dc0d65 MainThread beam_integration_benchmark(1/1) INFO Running: python setup.py nosetests --tests=apache_beam.examples.wordcount_it_test --attr=IT --test-pipeline-options="--project=apache-beam-testing" "--staging_location=gs://temp-storage-for-end-to-end-tests/staging-it" "--temp_location=gs://temp-storage-for-end-to-end-tests/temp-it" "--output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output" --runner=TestDataflowRunner --sdk_location=<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/runs/34dc0d65/beam/sdks/python/target/apache-beam-2.5.0.dev0.tar.gz> 2018-04-26 00:14:04,352 34dc0d65 MainThread beam_integration_benchmark(1/1) INFO Ran: {python setup.py nosetests --tests=apache_beam.examples.wordcount_it_test --attr=IT --test-pipeline-options="--project=apache-beam-testing" "--staging_location=gs://temp-storage-for-end-to-end-tests/staging-it" "--temp_location=gs://temp-storage-for-end-to-end-tests/temp-it" "--output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output" --runner=TestDataflowRunner --sdk_location=<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/runs/34dc0d65/beam/sdks/python/target/apache-beam-2.5.0.dev0.tar.gz}> ReturnCode:1 STDOUT: STDERR: /usr/local/lib/python2.7/dist-packages/setuptools/dist.py:397: UserWarning: Normalizing '2.5.0.dev' to '2.5.0.dev0' normalized_version, usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help error: invalid command 'nosetests' 2018-04-26 00:14:04,353 34dc0d65 MainThread beam_integration_benchmark(1/1) ERROR Error during benchmark beam_integration_benchmark Traceback (most recent call last): File "<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,> line 646, in RunBenchmark DoRunPhase(spec, collector, detailed_timer) File "<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,> line 526, in DoRunPhase samples = spec.BenchmarkRun(spec) File "<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py";,> line 159, in Run job_type=job_type) File "<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py";,> line 90, in SubmitJob assert retcode == 0, "Integration Test Failed." AssertionError: Integration Test Failed. 2018-04-26 00:14:04,354 34dc0d65 MainThread beam_integration_benchmark(1/1) INFO Cleaning up benchmark beam_integration_benchmark 2018-04-26 00:14:04,356 34dc0d65 MainThread beam_integration_benchmark(1/1) ERROR Exception running benchmark Traceback (most recent call last): File "<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,> line 780, in RunBenchmarkTask RunBenchmark(spec, collector) File "<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,> line 646, in RunBenchmark DoRunPhase(spec, collector, detailed_timer) File "<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,> line 526, in DoRunPhase samples = spec.BenchmarkRun(spec) File "<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py";,> line 159, in Run job_type=job_type) File "<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py";,> line 90, in SubmitJob assert retcode == 0, "Integration Test Failed." AssertionError: Integration Test Failed. 2018-04-26 00:14:04,356 34dc0d65 MainThread beam_integration_benchmark(1/1) ERROR Benchmark 1/1 beam_integration_benchmark (UID: beam_integration_benchmark0) failed. Execution will continue. 2018-04-26 00:14:04,392 34dc0d65 MainThread INFO Benchmark run statuses: --------------------------------------------------------------------------------- Name UID Status Failed Substatus --------------------------------------------------------------------------------- beam_integration_benchmark beam_integration_benchmark0 FAILED --------------------------------------------------------------------------------- Success rate: 0.00% (0/1) 2018-04-26 00:14:04,393 34dc0d65 MainThread INFO Complete logs can be found at: <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/runs/34dc0d65/pkb.log> 2018-04-26 00:14:04,393 34dc0d65 MainThread INFO Completion statuses can be found at: <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/runs/34dc0d65/completion_statuses.json> Build step 'Execute shell' marked build as failure Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected]
