See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/4195/display/redirect?page=changes>
Changes: [joey.baruch] Add javadoc to ConsoleIO [rmannibucau] better error message when there are conflicting anonymous names [rmannibucau] adding comments to make the code more obvious and extracting the [rmannibucau] adding tostring in ParDo.*Output and removing unstableNames variable [batbat] Moved floating point equality findbugs annotation from generic xml file [melissapa] [BEAM-3351] Fix Javadoc formatting issues [melissapa] Update Context references to links [github] Fix code comment to match code [dariusz.aniszewski] Allow to specify timeout for FileBasedIOIT ran via PerfKit with default [holdenkarau] Apply futurize on gen_protos [holdenkarau] Run futurize regardless [holdenkarau] Only futurize for sys version geq 3 [holdenkarau] Long form command line options [kedin] [Nexmark][SQL] Implement Query1 [github] NotImplementedErrror --> NotImplementedError [tgroh] Add RemoteGrpcPort[Read/Write] abstractions [rmannibucau] BEAM-682 try to define generated classes in the correct classloader [kenn] Shade protobuf-java-util alongside protobuf-java [kenn] Remove Dataflow hack from shared translation code [lcwik] Add the ability to create an offline repository using [dariusz.aniszewski] split one job into several jobs, one for each IO. [lcwik] [BEAM-2863] Create an EncodedBoundedWindow allowing runners to pass [lcwik] [BEAM-3257] Move sdist execution to execution phase instead of [swegner] Increment Dataflow worker container version [robertwb] Remove legacy windowfn translation. [chamikara] Updates the error message when trying to execute an export job on an [ryan] Enable multiple stopping conditions for KafkaIO [iemejia] [BEAM-3405] Make maxNumRecords a long and validate if stream exists for [swegner] Revert "Increment Dataflow worker container version" [tgroh] Fix sdks/java/fn-execution dep declarations [tgroh] Move BeamFnDataGrpcMultiplexer to java-fn-execution [tgroh] Mark BeamFnDataGrpcMultiplexer ApiServiceDescriptor as Nullable [samuelw] Update NexmarkLauncher to not cancel jobs that have completed their [batbat] Fixed a bug that prevented processing time timers to be reset [batbat] Added ParDoTest for resetting timers. [chamikara] [BEAM-3060] HDFS cluster configuration, kubernetes scripts, filebased io [github] Update BUILD.md [iemejia] [BEAM-3467] Remove Java 7 from the docker development images [thw] [BEAM-3240] Fix quickstart archetypes to include expected Hadoop [tgroh] Handle a populated Error field in FnApiControlClient [tgroh] Enable Java8 for fn execution modules [sidhom] Expose side inputs in CombineTranslation and refactor portable [sidhom] Use portable constructs in Flink batch translator [lcwik] [BEAM-2863] Add the ability to length prefix unknown coders using the [daniel.o.programmer] [BEAM-3126] Creating flatten operation for bundle processor. [klk] Temporarily disable permared Dataflow integration tests [github] [BEAM-3436] Fix RetryHttpRequestInitializerTest to use fake [github] [BEAM-3336] Fix thread safety issues of MqttIOTest (#4406) [lcwik] [BEAM-3478] Fix checkstyle issue in runners/flink [iemejia] Fix unused dependency error on Flink runner [rmannibucau] adressing comments [iemejia] Add missing protobuf-java dependency on runners/java-fn-execution module [ryan] [BEAM-3454] Use multiple stopping conditions on JmsIO, AmqpIO, MqttIO. [dariusz.aniszewski] use VirtualEnv for PerfKit jobs [shashank] fix serialization error in BigQueryIO's DynamicDestinations [jbonofre] [BEAM-3438] Disable it on Kinesis [shashank] correct side input check in BigQueryIO DynamicDestination for pull [tgroh] Update GRPC Multiplexer and Associated to Java8 [lcwik] [BEAM-3125] Creating flatten runner, unfinished. [lcwik] [BEAM-3125] Creating flatten runner [lcwik] [BEAM-3125] Fixing little nitpicks and typos. [lcwik] [BEAM-3126] Fixes for flatten runner. [niemo] Update BigTableSource to use multiple key ranges [niemo] Add tests for supporting multiple key ranges. [niemo] Use multiple key ranges in BigtableServiceImpl [tgroh] Implement a GRPC Provision Service [lcwik] [BEAM-3461] Drop redundant beam-YYY-parent and use directory names for [ankurgoenka] Fixing Work Progress reporting [klk] Move DoFnAndMainOutput to top level [klk] Move DoFnSignature helpers to DoFnSignatures and rename [klk] Clarify names and visibility of ParDoTranslation helpers [klk] Make default ParDo output tag explicit [klk] Fix coders in Java SDK harness [klk] More logging and error checking in ProcessBundleHandler [klk] Pass portable transform IDs in Dataflow runner [klk] Correct getAdditionalInputs for Dataflow single-output ParDo override [tgroh] Add wire_coder_id to RemoteGrpcPort [tgroh] Add joda-time dependency to java/fn-execution [tgroh] Add InboundDataClient [kedin] [Nexmark][SQL] Implement sql query 2 [kenn] Add unshaded configurations in the default profile [tgroh] Update Dataflow Container Version [zoy] Increasing BatchElements's max_batch_size to 10K [tgroh] Revert "Temporarily disable permared Dataflow integration tests" [sidhom] Identify unconsumed reads by URN [sidhom] Use portable constructs in Flink streaming translator [robertwb] Fix reshuffle test on non-direct runners. [ekirpichov] Removes maxBundleSize option accidentally introduced in unrelated PR [robertwb] Curry CombineFn arguments into runner API protos. [batbat] Added an example pipeline that uses stateful processing to output team [robertwb] Enable counters from sources on workers. [robertwb] lint [chamikara] [BEAM-3456] Enable jenkins and large scale scenario in JDBC (#4392) [lcwik] [BEAM-2863] Fix translation of side inputs due to missing method. [chamikara] [BEAM-3217] add HadoopInputFormatIO integration test using DBInputFormat [lcwik] [BEAM-2863] Migrate away from deprecated methods. [tgroh] Give UnknownPTransformRunnerFactory a real name [robertwb] [BEAM-2732] Starting refactor of state tracking in Python (#4375) [lcwik] Add a more complete FnHarness#main method [chamikara] [BEAM-3496] Add missing dependencies in hadoopInputFormats build.gradle [markliu] [BEAM-2762] Python code coverage report in Postcommit [markliu] fixup! Exclude auto-generated files in portability/api [markliu] fixup! Clean up configuration and use better way to exclude cover in mvn [markliu] fixup! Add more comments [wcn] Update generated version of Fn API code. [iemejia] [BEAM-3275] Update KafkaIO to support Kafka version 1.0.0 [lcwik] Update generated version of Fn API code. [Pablo] Tracking of time spent reading side inputs, and bytes read in Dataflow. [Pablo] Fix lint issues [Pablo] Fixing counter names [Pablo] Using comments to improve changes [Pablo] Placing feature behind an experiment flag. [Pablo] Fixing lint issue [Pablo] Rebasing [Pablo] Added cythonization and test [Pablo] Fixing lint issue [Pablo] Modifying the experiment flag check to reduce performance impact. [Pablo] Addressing comments. Profile pending. [lcwik] Use platformThreadFactory for default thread pool. [robertwb] [BEAM-3490] Wrap DistributionData in a DistributionResult for [Pablo] Reducing impact - Tracking only Bytes if experiment is active. [tgroh] Add a BundleProcessor to SdkHarnessClient [robertwb] [BEAM-3490] Picklable side inputs for FnApi Workers. [robertwb] Manually specify direct runner for global-state modifying tests. [robertwb] FakeSource requires direct runner. [robertwb] Explicitly use DirectRunner in DirectRunner tests. [markliu] fixup! Improve comments [altay] Disable combiner lifting optimization in DataflowRunner for fnapi [lcwik] [BEAM-3502] Remove usage of proto.Builder.clone() in DatastoreIO (#4449) [jb] [BEAM-3507] Add a way to specify the batch size in JdbcIO Write [jbonofre] [BEAM-3507] Add DEFAULT_BATCH_SIZE and use batchSize in default [iemejia] [BEAM-3492] Fix spark 2.10 dependency leak on hadoop tests [iemejia] [BEAM-3492] Force netty version when running with the spark runner [lcwik] [BEAM-3427] Update build to Java 8 (before clean out) (#4424) [sduskis] [BEAM-3412] Upgrade Cloud Bigtable to 1.0.0 The 1.0.0 Cloud Bigtable [sduskis] BigtableServiceImplTest now uses a List of ranges. The PR did not [sduskis] Updating Cloud Bigtable dependnecy to 1.0.0 in build.gradle. [ekirpichov] Introduces the Wait transform [github] [BEAM-2963] Remove layer of indirection in output name mapping in [ehudm] Pass PipelineOptions to FileSystem constructor. [altay] Disable combiner lifting when only the streaming flag is set. [Pablo] Addressing comments [Pablo] Improved IO Target documentation [tgroh] Register Environments in SdkComponents [Pablo] Improving documentation [tgroh] Move BeamFnDataInboundObserver to java-fn-execution [tgroh] Add an implementation of the Fn API Data Service [robertwb] Document DirectRunnerOnly tests. [robert] Add CONTRIBUTING.md [lcwik] [BEAM-3008] Adds parameters templatization for Bigtable (#4357) [lcwik] Migrate Flink ValidatesRunner to Gradle [lcwik] Increment the Dataflow runner major version to 7. [zhouhai02] Update copyright date in NOTICE [sduskis] Using 1.0.0-pre3 for bigtable-proto. This should allow Cloud Bigtable's [sduskis] Updating build.gradle with a bigtable_proto_version of 1.0.0-pre3 [robertwb] [BEAM-3126] Adding a new Flatten test to Python SDK. (#4463) [rober] Update printed gcloud cancel commands to include the job's region. [sduskis] Fixing a bad merge in BigtableServiceImpl. [kirpichov] Code compiles after auto-transition to lambda [kirpichov] google-java-format [kirpichov] Removes unnecessary explicit type arguments [kirpichov] google-java-format [kirpichov] checkstyle fixups [kirpichov] More removal of explicit type arguments [kirpichov] google-java-format [kirpichov] Manually fixed a couple cases of bad comment formatting [kirpichov] Manual fixup of some call sites where lambdas mess up coder inference [kirpichov] A couple of final example fixups [lcwik] Upgrade Jenkins jobs to use Maven 3.5.2 [iemejia] [BEAM-3432] Remove hadoop/jdk1.8-tests module [lcwik] [BEAM-2728] Add Count-Min Sketch in sketching extension [lcwik] Small fix in SketchCoder [lcwik] Make Sketch AutoValue + Javadoc update [lcwik] Optimize coder memory use [lcwik] [BEAM-3160] Fix issue where we would choose which coder to use [lcwik] Initial post-release snapshot test [lcwik] Make the snapshot and url parameters to the job to allow installing a [lcwik] Make the snapshot and url parameters to the job to allow installing a [lcwik] Rename TestHelper to TestScripts, it() to intent() [chamikara] [BEAM-3060] Support for Perfkit execution of file-based-io-tests on HDFS [Pablo] Addressing comments. [jbonofre] [BEAM-3428] Merge Java8 examples in "main" Java examples [jbonofre] [BEAM-3428] Replace MinimalWordCount with Java8 one [echauchot] [BEAM-3534] Add a spark validates runner test for metrics sink in [coheigea] BEAM-3533 - Replace hard-coded UTF-8 Strings [tgroh] Retrieve Environments from PTransforms [jbonofre] [BEAM-3466] Remove JDK 7 references in Jenkins [kirpichov] [BEAM-3083] Do not call getSchema() and getTable() on every element [altay] Use non-deprecated version of Futures.transform [aromanenko.dev] [BEAM-3539] BigtableIO.Write javadoc of some methods is incorrect [iemejia] Remove unneeded profile for javadoc on Java 8 [iemejia] Remove unneeded explicit Java 8 references on maven-compiler-plugin [iemejia] Fix doc error on hadoop-input-format ITs after move to Java 8 only tests [iemejia] Remove references to non-existent examples:java8 module in gradle [iemejia] Remove references to java 7/8 only examples from the README [iemejia] Remove some comments on Java 7/8 only stuff that don't make sense [kenn] Add a test for an event time timer loop in ParDo [lcwik] [BEAM-2273] Cleanup examples Maven Archetype to copy in a clean state [XuMingmin] [BEAM-3526] KakfaIO support for finalizeCheckpoint() (#4481) [kedin] Add Avro dependency to KafkaIO [rangadi] Remove an unused test dependency in KafkaIO. [lcwik] [Beam-2500] Add S3FileSystem to SDKs/Java/IO [lcwik] implement serializing AWS credentials provider [lcwik] fixup! Clarify error message is received from SDK harness [iemejia] Fix modules that were activated only on Java 8 profile [iemejia] [BEAM-3275] Fix ValidatesRunner Spark runner after the Kafka update [iemejia] Refactor code into idiomatic Java 8 style [iemejia] Fix missing gearpump module activated only on Java 8 profile [iemejia] Add missing amazon-web-services module from javadoc [jbonofre] [BEAM-2271] Exclude files not intended to be in the releases source [jbonofre] Typo fix in MinimalWordCount example [Mottyg1] [BEAM-675] Introduce message mapper in JmsIO [iemejia] Update maven-shade-plugin version to 3.1.0 [iemejia] Update maven-compiler-plugin version to 3.7.0 [iemejia] Update maven-dependency-plugin version to 3.0.2 [iemejia] Update maven-surefire-plugin version to 2.20.1 [iemejia] Update maven-failsafe-plugin version to 2.20.1 [iemejia] Update maven-assembly-plugin version to 3.1.0 [iemejia] Update versions-maven-plugin version to 2.5 [iemejia] Update findbugs-maven-plugin version to 3.0.5 [iemejia] Update license-maven-plugin version to 1.14 [iemejia] Update jacoco-maven-plugin version to 0.7.9 [iemejia] Update dockerfile-maven-plugin version to 1.3.6 [iemejia] Add maven-enforcer-plugin version to pluginManagement [iemejia] Fix warning on using directly parent.version on Nexmark [iemejia] Remove warnings on repeated maven-jar-plugin invocation / deps [iemejia] Remove warning on defining <prerequisites> for non-maven plugin project [iemejia] Update parent POM to version 19 [iemejia] Remove repeated dependency in hadoop-input-format module [mariagh] Support argparse-style choices for ValueProvider [jbonofre] [BEAM-3551] Add -parameters flag to the compiler [jbonofre] [BEAM-3551] Align gradle java compile task options with the [coheigea] BEAM-3560 - Switch to use BigInteger/BigDecimal.ZERO/ONE/TEN [lcwik] [BEAM-2500, BEAM-3249] Add amazon-web-services gradle build rules. [jbonofre] [maven-release-plugin] prepare branch release-2.3.0 [jbonofre] [maven-release-plugin] prepare for next development iteration [jbonofre] Update Python SDK version post release [iemejia] Fix dependency order error, harness must be built before container [robertwb] [BEAM-3537] Allow more general eager in-process pipeline execution [lcwik] [BEAM-3550] Add --awsServiceEndpoint option and use with S3 filesystem. [lcwik] Fixes for sdks/java/core for the eclipse compiler [lcwik] Add some m2e lifecycle things to get the various maven plugins running [lcwik] Set the proper package for the Snippets example so eclipse won't raise [lcwik] Add some template args and direct casts to help the eclipse compiler [lcwik] Incorporate reviews and rebase on latest master [lcwik] Return no environment more often [Pablo] Logging deviation from sampling expectation. This will allow to track [kedin] [SQL] Refactor Variance [robertwb] [BEAM-3490] Make runtime type checking code runner agnostic. [kedin] [Nexmark][SQL] Implement sql query 3 [github] [BEAM-3557] Sets parent pointer of AppliedPTransform objects correctly [pawel.pk.kaczmarczyk] [BEAM-2469] Handling Kinesis shards splits and merges [Pablo] Adding a static getter for RuntimeValueProvider. [tgroh] Add CoderTranslatorRegistrar [tgroh] Add slf4j_simple to the top level Gradle build [tgroh] Implement FnService in FnApiControlClientPoolService [tgroh] Add a Timeout to GrpcDataService#send [tgroh] Use a Data Service in SdkHarnessClient [github] get the query from configuration not options [XuMingmin] [BEAM-3525] Fix KafkaIO metric (#4524) [chamikara] Updates PTransform overriding to create a new AppliedPTransform object [dariusz.aniszewski] use build $WORKSPACE as pkb temp_dir and update pip and setuptools in [iemejia] [BEAM-3578] SQL module build breaks because of missing dependency [Pablo] Renaming the ZERO element of DistributionResult to be IDENTITY_ELEMENT. [kenn] google-java-format [kenn] Fix Distinct null pointer error with speculative triggers [kenn] Move TestCountingSource to appropriate location [robertwb] Direct runner fixes. [lcwik] [BEAM-2926] Add support for side inputs to the runner harness. [kenn] Sickbay ApexRunner gradle WordCountIT [kenn] Sickbay flakey KinesisReaderTest [Pablo] Addressing comments. [ehudm] Split out buffered read and write code from gcsio. [github] Removing unnecessary code. [lcwik] [BEAM-3249] Make sure that all java projects package tests. Also package [lcwik] [BEAM-3249] Do not assume build directory is within build/, use the [github] Fix undefined names: exc_info --> self.exc_info [github] import logging for line 1163 [iemejia] [BEAM-3592] Fix spark-runner profile for Nexmark after move to Spark 2.x [dkulp] [BEAM-3562] Update to Checkstyle 8.7 [klk] Encourage a good description in a good spot on a PR description. [lcwik] Change info to debug statement [cclauss] global INT64_MAX, INT64_MIN to placate linters [tgroh] Add QueryablePipeline [gene] Changing FileNaming to public to allow for usage in lambdas/inheritance [robertwb] [BEAM-3207] Create a standard location to enumerate and document URNs. [robertwb] Reduce the flakiness of the state sampler progress metrics. [robertwb] Revert URNs that are currently hard-coded in the Dataflow worker. [kedin] [SQL] Inherit windowing strategy from the input in Aggregate operation [jbonofre] [BEAM-3551] Define compiler -parameters flag in the default options [tgroh] Add SdkHarnessClientControlService [tgroh] Update Synchronization in FnApiControlClient [coheigea] BEAM-3593 - Remove methods that just call super() [lcwik] Move off of deprecated method in Guava. [tgroh] Add a LocalArtifactStagingLocation [tgroh] Add LocalArtifactStagingLocation#forExisting [tgroh] Add an ArtifactRetrievalService interface [tgroh] Implement a Local ArtifactRetrievalService [chamikara] Adds a ReadAll transform to tfrecordio. [rangadi] KafkaIO : move source and sink implemenations into own files. [rangadi] minor [kedin] [SQL] Add SqlTypeCoder, replace java.sql.Types [Pablo] Moving User metrics to be in the PTransform proto for Fn API. [mairbek] Update cloud spanner library to 0.29.0 [mairbek] Fix test [mairbek] More google-cloud-platform whitelisting [mairbek] pom updates to make maven happy [mairbek] Update netty deps [ccy] [BEAM-3566] Replace apply_* hooks in DirectRunner with [ccy] Address reviewer comments [davidyan] Correct typo in SpannerIO.Write.withHost [klk] google-java-format [klk] Fix empty window assignments in Nexmark [klk] Fix empty window assignment in FlattenEvaluatorFactoryTest [klk] Switch DataflowRunner to its own private ValueInEmptyWindows [klk] Remove deprecated valueInEmptyWindows [jiangkai] Covariance Functions [aljoscha.krettek] Remove erroneous cast in FlinkStreamingTransformTranslators [aljoscha.krettek] [BEAM-3186] Correctly use deserialized timerService in Flink Runner [lcwik] Adjust gradle build dirs and hints to help IntelliJ (#4583) [coheigea] BEAM-3618 - Remove extraneous "return" statement [robertwb] [BEAM-3183] Allow a callable as input to runner.run(). [sidhom] Fix gradle java sdk image build [kenn] Add MoreFutures utility [kenn] Switch runners/java-fn-execution from Guava futures to Java 8 futures [kenn] Switch DataflowRunner from Guava futures to Java 8 futures [kenn] Switch gcp-core from Guava futures to Java 8 futures [kenn] Switch runners/core-construction-java from Guava futures to Java 8 [kenn] Switch AWS IO from Guava futures to Java 8 futures [kenn] Switch BigTableIO from Guava futures to Java 8 futures [kenn] Deprecate DoFnTester [mairbek] Fixed broken test [ekirpichov] Adds PositionT and claim callback to RestrictionTracker [ekirpichov] Changes OutputAndTimeBounded invoker to start checkpoint timer after [ekirpichov] Compresses encoded GrowthState with Snappy - about 2x-3x more compact [ekirpichov] InMemoryStateInternals.copy clones the values using the coder [ekirpichov] Final fixups [ekirpichov] Bump worker to 20180205 [klk] Sickbay flaky KinesisIO tests [klk] Remove DoFnTester from core SDK tests [aljoscha.krettek] [BEAM-2806] Fix pipeline translation mode recognition in Flink Runner [github] Consistantly show Python and pip versions in tests [jbonofre] Revert "Reinstate proto round trip in Java DirectRunner" [tgroh] Update Assign Window URN Constant Name [coheigea] BEAM-3624 - Remove collapsible if statements [kenn] Sickbay ApexRunner ParDoTranslatorTest.testAssertionFailure [kenn] Switch FullWindowedValueCoder to bypass validation [kedin] Refactor BeamRecordType and BeamRecord [sidhom] Allow docker tag root to be specified as in Maven image build [herohde] [BEAM-3457] Upgrade gogradle and fix thrift resolution issue [herohde] [BEAM-3457] Add Go Gradle precommit [ccy] [BEAM-3635] Infer type hints on PTransformOverrides [robertwb] [BEAM-3625] Enable DoFn params in Map, Filter, etc. [kedin] Rename BeamRecord -> Row, BeamRecordType -> RowType [iemejia] Add missing gradle build config for sdks/java/extensions/sketching [iemejia] Fix type on shadowTest when it should be testShadow [ccy] Update snippets to fix pickling and clarify encoding issues [wcn] Modify BufferedElementCountingOutputStream to use a buffer pool for its [klk] Fix stable name errors in HBaseIOTest [aromanenko.dev] [BEAM-3291] Add Kinesis write transform [coheigea] Remove unused private variables. [ehudm] Add and migrate to HdfsCLI library for Python SDK. [klk] Fix typo in gradle idea hints [dkulp] [BEAM-3639] Update to gradle 4.5.1 [github] Revert "Update cloud spanner library to 0.29.0" [kedin] [Schema Generation] Generate BeamRecordTypes based on pojos. [alan] [BEAM-3524] Automate testing using python sdk container built at head [coheigea] Replace boolean ternary operator + simplify some Boolean expressions [apilloud] [BEAM-410] Sort PriorityQueue<QuantileBuffer> with explicit comparator [robertwb] Disable verbose typecoder warnings. [fjetumale] [BEAM-2817] BigQuery queries are allowed to run in either BATCH or [kenn] Increase gradle logging to --info [ccy] Add switchable DirectRunner which uses the fast FnApiRunner when [ehudm] Integration test for Python HDFS implementation. ------------------------------------------ [...truncated 1.12 MB...] [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 562, in process_element [32mtest_1 |[0m self.runner.process(element) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 474, in process [32mtest_1 |[0m self._reraise_augmented(exn) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 472, in process [32mtest_1 |[0m self.do_fn_invoker.invoke_process(windowed_value) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 377, in invoke_process [32mtest_1 |[0m windowed_value, additional_kwargs, output_processor) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 411, in _invoke_per_window [32mtest_1 |[0m windowed_value, self.process_method(*args_for_process)) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/transforms/core.py", line 968, in <lambda> [32mtest_1 |[0m wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)] [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/io/iobase.py", line 922, in <lambda> [32mtest_1 |[0m lambda _, sink: sink.initialize_write(), self.sink) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/options/value_provider.py", line 121, in _f [32mtest_1 |[0m return fnc(self, *args, **kwargs) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/io/filebasedsink.py", line 151, in initialize_write [32mtest_1 |[0m FileSystems.mkdirs(tmp_dir) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/io/filesystems.py", line 124, in mkdirs [32mtest_1 |[0m return filesystem.mkdirs(path) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/io/hadoopfilesystem.py", line 169, in mkdirs [32mtest_1 |[0m if self._exists(path): [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/io/hadoopfilesystem.py", line 349, in _exists [32mtest_1 |[0m return self._hdfs_client.status(path, strict=False) is not None [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/hdfs/client.py", line 309, in status [32mtest_1 |[0m res = self._get_file_status(hdfs_path, strict=strict) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/hdfs/client.py", line 118, in api_handler [32mtest_1 |[0m raise err [32mtest_1 |[0m RuntimeError: ConnectionError: HTTPConnectionPool(host='namenode', port=50070): Max retries exceeded with url: /webhdfs/v1/tmp/beam-temp-py-wordcount-direct-f420b3440e1911e894bb0242ac120004?user.name=root&op=GETFILESTATUS (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1d1a0462d0>: Failed to establish a new connection: [Errno 111] Connection refused',)) [while running 'write/Write/WriteImpl/InitializeWrite'] [32mtest_1 |[0m [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO blockmanagement.BlockManager: The block deletion will start around 2018 Feb 10 04:22:06 [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO util.GSet: Computing capacity for map BlocksMap [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO util.GSet: VM type = 64-bit [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO util.GSet: capacity = 2^21 = 2097152 entries [32mtest_1 |[0m INFO:hdfs.client:Instantiated <InsecureClient(url='http://namenode:50070')>. [32mtest_1 |[0m INFO:hdfs.client:Instantiated <InsecureClient(url='http://namenode:50070')>. [32mtest_1 |[0m INFO:hdfs.client:Instantiated <InsecureClient(url='http://namenode:50070')>. [32mtest_1 |[0m INFO:hdfs.client:Fetching status for '/tmp/beam-temp-py-wordcount-direct-f42646b00e1911e894bb0242ac120004'. [32mtest_1 |[0m ERROR:root:Exception at bundle <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7f1d1c35c890>, due to an exception. [32mtest_1 |[0m Traceback (most recent call last): [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 305, in call [32mtest_1 |[0m side_input_values) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 338, in attempt_call [32mtest_1 |[0m evaluator.process_element(value) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 562, in process_element [32mtest_1 |[0m self.runner.process(element) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 474, in process [32mtest_1 |[0m self._reraise_augmented(exn) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 472, in process [32mtest_1 |[0m self.do_fn_invoker.invoke_process(windowed_value) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 377, in invoke_process [32mtest_1 |[0m windowed_value, additional_kwargs, output_processor) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 411, in _invoke_per_window [32mtest_1 |[0m windowed_value, self.process_method(*args_for_process)) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/transforms/core.py", line 968, in <lambda> [32mtest_1 |[0m wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)] [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/io/iobase.py", line 922, in <lambda> [32mtest_1 |[0m lambda _, sink: sink.initialize_write(), self.sink) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/options/value_provider.py", line 121, in _f [32mtest_1 |[0m return fnc(self, *args, **kwargs) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/io/filebasedsink.py", line 151, in initialize_write [32mtest_1 |[0m FileSystems.mkdirs(tmp_dir) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/io/filesystems.py", line 124, in mkdirs [32mtest_1 |[0m return filesystem.mkdirs(path) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/io/hadoopfilesystem.py", line 169, in mkdirs [32mtest_1 |[0m if self._exists(path): [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/io/hadoopfilesystem.py", line 349, in _exists [32mtest_1 |[0m return self._hdfs_client.status(path, strict=False) is not None [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/hdfs/client.py", line 309, in status [32mtest_1 |[0m res = self._get_file_status(hdfs_path, strict=strict) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/hdfs/client.py", line 118, in api_handler [32mtest_1 |[0m raise err [32mtest_1 |[0m RuntimeError: ConnectionError: HTTPConnectionPool(host='namenode', port=50070): Max retries exceeded with url: /webhdfs/v1/tmp/beam-temp-py-wordcount-direct-f42646b00e1911e894bb0242ac120004?user.name=root&op=GETFILESTATUS (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1d1a046d10>: Failed to establish a new connection: [Errno 111] Connection refused',)) [while running 'write/Write/WriteImpl/InitializeWrite'] [32mtest_1 |[0m [32mtest_1 |[0m ERROR:root:Giving up after 4 attempts. [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO blockmanagement.BlockManager: defaultReplication = 3 [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO blockmanagement.BlockManager: maxReplication = 512 [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO blockmanagement.BlockManager: minReplication = 1 [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO blockmanagement.BlockManager: encryptDataTransfer = false [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE) [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO namenode.FSNamesystem: supergroup = supergroup [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO namenode.FSNamesystem: isPermissionEnabled = true [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO namenode.FSNamesystem: HA Enabled: false [36mnamenode_1 |[0m 18/02/10 04:22:06 INFO namenode.FSNamesystem: Append Enabled: true [33mdatanode_1 |[0m 18/02/10 04:22:06 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO util.GSet: Computing capacity for map INodeMap [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO util.GSet: VM type = 64-bit [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO util.GSet: capacity = 2^20 = 1048576 entries [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO namenode.FSDirectory: ACLs enabled? false [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO namenode.FSDirectory: XAttrs enabled? true [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO namenode.NameNode: Caching file names occurring more than 10 times [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO util.GSet: Computing capacity for map cachedBlocks [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO util.GSet: VM type = 64-bit [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO util.GSet: capacity = 2^18 = 262144 entries [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10 [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10 [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25 [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO impl.MetricsSystemImpl: DataNode metrics system started [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO namenode.FSNamesystem: Retry cache on namenode is enabled [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO datanode.BlockScanner: Initialized block scanner with targetBytesPerSec 1048576 [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO util.GSet: Computing capacity for map NameNodeRetryCache [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO util.GSet: VM type = 64-bit [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO util.GSet: capacity = 2^15 = 32768 entries [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO datanode.DataNode: Configured hostname is datanode [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO datanode.DataNode: Starting DataNode with maxLockedMemory = 0 [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO namenode.FSImage: Allocated new BlockPoolId: BP-2004264375-172.18.0.2-1518236527227 [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO datanode.DataNode: Opened streaming server at /0.0.0.0:50010 [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO datanode.DataNode: Balancing bandwith is 10485760 bytes/s [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO datanode.DataNode: Number threads for balancing is 50 [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO common.Storage: Storage directory /hadoop/dfs/name has been successfully formatted. [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO namenode.FSImageFormatProtobuf: Saving image file /hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO http.HttpRequestLog: Http request log for http.requests.datanode is not defined [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO http.HttpServer2: Jetty bound to port 34358 [33mdatanode_1 |[0m 18/02/10 04:22:07 INFO mortbay.log: jetty-6.1.26 [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO namenode.FSImageFormatProtobuf: Image file /hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 321 bytes saved in 0 seconds. [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO util.ExitUtil: Exiting with status 0 [36mnamenode_1 |[0m 18/02/10 04:22:07 INFO namenode.NameNode: SHUTDOWN_MSG: [36mnamenode_1 |[0m /************************************************************ [36mnamenode_1 |[0m SHUTDOWN_MSG: Shutting down NameNode at namenode/172.18.0.2 [36mnamenode_1 |[0m ************************************************************/ [33mdatanode_1 |[0m 18/02/10 04:22:08 INFO mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34358 [32mtest_1 |[0m WARNING:root:A task failed with exception: ConnectionError: HTTPConnectionPool(host='namenode', port=50070): Max retries exceeded with url: /webhdfs/v1/tmp/beam-temp-py-wordcount-direct-f42646b00e1911e894bb0242ac120004?user.name=root&op=GETFILESTATUS (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1d1a046d10>: Failed to establish a new connection: [Errno 111] Connection refused',)) [while running 'write/Write/WriteImpl/InitializeWrite'] [32mtest_1 |[0m Traceback (most recent call last): [32mtest_1 |[0m File "/usr/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main [32mtest_1 |[0m "__main__", fname, loader, pkg_name) [32mtest_1 |[0m File "/usr/local/lib/python2.7/runpy.py", line 72, in _run_code [32mtest_1 |[0m exec code in run_globals [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/examples/wordcount.py", line 134, in <module> [32mtest_1 |[0m run() [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/examples/wordcount.py", line 114, in run [32mtest_1 |[0m result.wait_until_finish() [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py", line 404, in wait_until_finish [32mtest_1 |[0m self._executor.await_completion() [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 358, in await_completion [32mtest_1 |[0m self._executor.await_completion() [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 305, in call [32mtest_1 |[0m side_input_values) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 338, in attempt_call [32mtest_1 |[0m evaluator.process_element(value) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 562, in process_element [32mtest_1 |[0m self.runner.process(element) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 474, in process [32mtest_1 |[0m self._reraise_augmented(exn) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 472, in process [32mtest_1 |[0m self.do_fn_invoker.invoke_process(windowed_value) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 377, in invoke_process [32mtest_1 |[0m windowed_value, additional_kwargs, output_processor) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 411, in _invoke_per_window [32mtest_1 |[0m windowed_value, self.process_method(*args_for_process)) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/transforms/core.py", line 968, in <lambda> [32mtest_1 |[0m wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)] [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/io/iobase.py", line 922, in <lambda> [32mtest_1 |[0m lambda _, sink: sink.initialize_write(), self.sink) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/options/value_provider.py", line 121, in _f [32mtest_1 |[0m return fnc(self, *args, **kwargs) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/io/filebasedsink.py", line 151, in initialize_write [32mtest_1 |[0m FileSystems.mkdirs(tmp_dir) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/io/filesystems.py", line 124, in mkdirs [32mtest_1 |[0m return filesystem.mkdirs(path) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/io/hadoopfilesystem.py", line 169, in mkdirs [32mtest_1 |[0m if self._exists(path): [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/apache_beam/io/hadoopfilesystem.py", line 349, in _exists [32mtest_1 |[0m return self._hdfs_client.status(path, strict=False) is not None [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/hdfs/client.py", line 309, in status [32mtest_1 |[0m res = self._get_file_status(hdfs_path, strict=strict) [32mtest_1 |[0m File "/usr/local/lib/python2.7/site-packages/hdfs/client.py", line 118, in api_handler [32mtest_1 |[0m raise err [32mtest_1 |[0m RuntimeError: ConnectionError: HTTPConnectionPool(host='namenode', port=50070): Max retries exceeded with url: /webhdfs/v1/tmp/beam-temp-py-wordcount-direct-f42646b00e1911e894bb0242ac120004?user.name=root&op=GETFILESTATUS (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1d1a046d10>: Failed to establish a new connection: [Errno 111] Connection refused',)) [while running 'write/Write/WriteImpl/InitializeWrite'] [33mdatanode_1 |[0m 18/02/10 04:22:09 INFO web.DatanodeHttpServer: Listening HTTP traffic on /0.0.0.0:50075 [33mdatanode_1 |[0m 18/02/10 04:22:09 INFO util.JvmPauseMonitor: Starting JVM pause monitor [33mdatanode_1 |[0m 18/02/10 04:22:09 INFO datanode.DataNode: dnUserName = root [33mdatanode_1 |[0m 18/02/10 04:22:09 INFO datanode.DataNode: supergroup = supergroup [36mnamenode_1 |[0m 18/02/10 04:22:09 INFO namenode.NameNode: STARTUP_MSG: [36mnamenode_1 |[0m /************************************************************ [36mnamenode_1 |[0m STARTUP_MSG: Starting NameNode [36mnamenode_1 |[0m STARTUP_MSG: user = root [36mnamenode_1 |[0m STARTUP_MSG: host = namenode/172.18.0.2 [36mnamenode_1 |[0m STARTUP_MSG: args = [] [36mnamenode_1 |[0m STARTUP_MSG: version = 2.8.1 [36mnamenode_1 |[0m STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop-2.8.1/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/commons-net-3.1.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/json-smart-1.1.1.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jsch-0.1.51.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/activation-1.1.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jcip-annotations-1.0.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/gson-2.2.4.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/zookeeper-3.4.6.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/asm-3.2.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/htrace-core4-4.0.1-incubating.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/junit-4.11.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jettison-1.1.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/commons-io-2.4.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/paranamer-2.3.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/hadoop-annotations-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/hadoop-auth-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/httpcore-4.4.4.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/httpclient-4.5.2.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/xz-1.0.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/guava-11.0.2.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/avro-1.7.4.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.8.1/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/hadoop-2.8.1/share/hadoop/common/hadoop-common-2.8.1-tests.jar:/opt/hadoop-2.8.1/share/hadoop/common/hadoop-nfs-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/common/hadoop-common-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/okhttp-2.4.0.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/okio-1.4.0.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/htrace-core4-4.0.1-incubating.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.1-tests.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-2.8.1-tests.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-nfs-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-client-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-client-2.8.1-tests.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/objenesis-2.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/activation-1.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/guice-3.0.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/curator-test-2.7.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/asm-3.2.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/commons-math-2.2.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/xz-1.0.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/guava-11.0.2.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/fst-2.24.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-registry-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-tests-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-api-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-common-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-client-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-common-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/javax.inject-1.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/junit-4.11.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/hadoop-annotations-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.1-tests.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.8.1.jar:/opt/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.8.1.jar:/contrib/capacity-scheduler/*.jar [36mnamenode_1 |[0m STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 20fe5304904fc2f5a18053c389e43cd26f7a70fe; compiled by 'vinodkv' on 2017-06-02T06:14Z [36mnamenode_1 |[0m STARTUP_MSG: java = 1.8.0_131 [36mnamenode_1 |[0m ************************************************************/ [36mnamenode_1 |[0m 18/02/10 04:22:09 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] [36mnamenode_1 |[0m 18/02/10 04:22:09 INFO namenode.NameNode: createNameNode [] [32mhdfsitjenkinsbeampostcommitpythonverify4195_test_1 exited with code 1 [0mStopping hdfsitjenkinsbeampostcommitpythonverify4195_datanode_1 ... Stopping hdfsitjenkinsbeampostcommitpythonverify4195_namenode_1 ... [2A[2KStopping hdfsitjenkinsbeampostcommitpythonverify4195_datanode_1 ... [32mdone[0m[2B[1A[2KStopping hdfsitjenkinsbeampostcommitpythonverify4195_namenode_1 ... [32mdone[0m[1BAborting on container exit... Build step 'Execute shell' marked build as failure Not sending mail to unregistered user daniel.o.program...@gmail.com Not sending mail to unregistered user bat...@batbat-linuxworkstation.sea.corp.google.com Not sending mail to unregistered user rob...@frantil.com Not sending mail to unregistered user jiang...@gmail.com Not sending mail to unregistered user sid...@google.com Not sending mail to unregistered user aljoscha.kret...@gmail.com Not sending mail to unregistered user eh...@google.com Not sending mail to unregistered user dariusz.aniszew...@polidea.com Not sending mail to unregistered user ekirpic...@gmail.com Not sending mail to unregistered user ccla...@bluewin.ch Not sending mail to unregistered user aromanenko....@gmail.com Not sending mail to unregistered user mott...@gmail.com Not sending mail to unregistered user c...@google.com Not sending mail to unregistered user w...@google.com Not sending mail to unregistered user z...@giggles.nyc.corp.google.com Not sending mail to unregistered user ni...@google.com Not sending mail to unregistered user ro...@frantil.com Not sending mail to unregistered user xuming...@users.noreply.github.com Not sending mail to unregistered user meliss...@google.com Not sending mail to unregistered user pawel.pk.kaczmarc...@gmail.com Not sending mail to unregistered user zhouha...@meituan.com Not sending mail to unregistered user hero...@google.com Not sending mail to unregistered user kirpic...@google.com Not sending mail to unregistered user mark...@google.com Not sending mail to unregistered user sdus...@google.com Not sending mail to unregistered user apill...@google.com Not sending mail to unregistered user j...@nanthrax.net Not sending mail to unregistered user g...@telligent-data.com Not sending mail to unregistered user fjetum...@gmail.com Not sending mail to unregistered user sweg...@google.com Not sending mail to unregistered user samu...@google.com Not sending mail to unregistered user mari...@mariagh.svl.corp.google.com Not sending mail to unregistered user mair...@google.com Not sending mail to unregistered user holdenka...@google.com Not sending mail to unregistered user git...@alasdairhodge.co.uk Not sending mail to unregistered user ke...@google.com Not sending mail to unregistered user ankurgoe...@gmail.com Not sending mail to unregistered user k...@google.com Not sending mail to unregistered user joey.bar...@gmail.com