See <https://builds.apache.org/job/beam_PerformanceTests_Python/1255/display/redirect?page=changes>
Changes: [kedin] [SQL] Remove explicit timestamp parsing from PubsubIOJsonTable [kedin] [SQL] Wrap Pubsub payload into a message schema [pablo] Change reading sideinput experimental flag into new pattern. [mariagh] Call assert_that per window via reify_windows flag [github] [BEAM-4268] Improving the separation between Metrics API and Execution [aromanenko.dev] [BEAM-4048] Refactor COMBINE mode for reading/writing from/to Pub/Sub [matthias] Futurize metrics subpackage [pablo] Adding context switch to operations missing it. [sidhom] [BEAM-2597] Flink batch ExecutableStage operator [thw] Fix compile error in RemoteExecutionTest. [klk] Move Release Guide to committers section [klk] Move committer section to the bottom (we can find it :-) [tgroh] Suppress more flaky tests. [klk] Streamline the contribution section [tgroh] Add a FlattenRunnerTest for a multi-consumed input [tgroh] Use ListMultimap when creating Transform Runners [tgroh] Migrate ImpulseEvaluatorFactory [tgroh] Migrate FlattenEvaluatorFactoryTest [tgroh] Migrate GroupByKeyOnly to the RR [tgroh] Use ModelCoders in LengthPrefixUnknownCoders [ekirpichov] [BEAM-3515] Portable translation of SplittableProcessKeyed [ekirpichov] Fixes compile error in FlinkExecutableStageFunctionTest [ekirpichov] Sickbays flaky test [robbe.sneyders] Add py27-lint3 test to gradle.build [klk] Google has scrapped research, now only AI [iemejia] Fix error-prone warnings for HCatalogIO [iemejia] Fix error-prone warnings and multiple cleanups based on static analysis [lcwik] [BEAM-4280] Prevent DirectStreamObserver from blocking indefinitely if [tgroh] Enable MapFnRunner to provide full WindowedValues [tgroh] Implement AssignWindows in the Java SDK Harness [tgroh] Add a static constructor to GrpcStateService [tgroh] Log when Handler Delegation Throws [kirpichov] fromQuery().withTemplateCompatibility() should do the same thing as [klk] Temporarily suppress validation of ai.google links [iemejia] Fix error-prone warnings for HBaseIO [timrobertson100] Fix error-prone warnings for SolrIO [timrobertson100] Fix error-prone warnings for ElasticsearchIO [timrobertson100] Fix error-prone and some static analysis warnings in [mergebot] Add link to portability support sheet [timrobertson100] Fix error-prone and some static analysis warnings in XmlIO [pablo] Fix error-prone warnings for file-based-io-tests [kedin] [SQL] Add integration test for PubsubIO JSON -> SQL [kedin] [SQL] Fix typos, add rewindowing and retriggering to TestPubsubSignal [apilloud] [BEAM-3983] Add utils for converting to BigQuery types [apilloud] [BEAM-3983][SQL] Add BigQuery table provider [apilloud] [BEAM-4248] Copy enums from com.google.cloud [kedin] Add SQL postCommit to main javaPostCommit [wcn] Fix log statement to mark sink as generator of info. [timrobertson100] Fix error-prone warnings for io/hadoop-common [mairbek] Make Cloud Spanner transaction id public. Allows to build non-library [daniel.o.programmer] [BEAM-4277] Adding unshaded jars used by Dataflow to poms. [tgroh] Revert "Log when Handler Delegation Throws" [tgroh] Log when Handler Delegation Throws [tgroh] squash! Revert "Log when Handler Delegation Throws" [pablo] [BEAM-4249] Fix failing validates container test [valentyn] Add a Python 3 entry to the list of ongoing projects. [lcwik] [BEAM-4127] Flink streaming translation for portable pipeline. (#5358) [herohde] [BEAM-3826] Add missing package comments [kedin] [SQL] Add support for DQL to PubsubJsonTable [mergebot] Add Go SDK quickstart and minimal documentation page [mergebot] Add Go to the Wordcount example [mergebot] Fix mailing list links and docker image [pablo] Add a dependency on future. [timrobertson100] Fix error-prone and some static analysis warnings in [lukasz.gajowy] [BEAM-4140] Utilize beam_location perfkit flag [github] Update index.md [kpeterson] Adding a ValueProvider interface. [klk] Reorganize SQL documentation into sections [swegner] Upgrade Byte Buddy and FindBugs annotations dependencies. [swegner] Fix ErrorProne warnings [swegner] Remove isTriggerSpecified from WindowingStategy equality checks. [kpeterson] Fixing checkstyle error. [lcwik] Add window mapping transform. [lcwik] Use the coder from the side input PCollection definition by removing the [lcwik] Remove garbage collection from PValueCache [pablo] Fixing incorrect assert in dependency_test.py [kedin] [SQL] Add integration test for Pubsub Json Table DLQ [github] [BEAM-4300] Fix ValidatesRunner tests in Python: run with same mechanism [tgroh] Add an IdGenerator class to java/fn-execution [ringwalt] Improve the TaggedOutput error when a tuple is passed as a tag. [lukasz.gajowy] [BEAM-214] Add ParquetIO [lukasz.gajowy] [BEAM-214] Add gradle build in ParquetIO [lukasz.gajowy] [BEAM-214] Refactore the reads to leverage FileIO [lukasz.gajowy] [BEAM-214] Use Parquet 1.10.0 version in ParquetIO.Sink [lukasz.gajowy] [BEAM-214] Add ParquetIOIT [lukasz.gajowy] [BEAM-214] Update ReadFn to an non deprecated, newest Parquet API [lukasz.gajowy] [BEAM-214] Simplify the IO by removing redundant PTransforms [lukasz.gajowy] [BEAM-214] Implement writing slices of bytes in BeamOutputStream [apilloud] [BEAM-3733] [SQL] Use saffron.properties for charset [pablo] [BEAM-4331] Disable integrationTest task caching [tgroh] Add a new DockerEnvironmentFactory Constructor [pablo] Fixing Seed Job [samuelw] [BEAM-3776] Fix issue with merging late windows where a watermark hold [pablo] Add future dependency to the Dockerfile [apilloud] [SQL] Add jdbc:beam: [apilloud] [SQL] Use CalciteConnection as context [axelmagn] Create jobsubmission package for job management. [axelmagn] Add Job Invocation Interfaces [axelmagn] Add in-memory implementation of JobService [pablo] Remove stale TODO comments for adding archetype to java precommit [pablo] Upgrade Gradle shadowJar plugin to latest. [github] Mention adding a reviewer in PR template. (#5397) [kedin] Fix Maven build [rober] Update generated protos in Go SDK [herohde] Invoke Go Dataflow integration tests from post-commit [herohde] Add the Go SDK to the README [timrobertson100] [BEAM-4342] Enforce ErrorProne analysis in hadoop IO, and static [timrobertson100] [BEAM-4340] Enforce ErrorProne analysis in file-based-io-tests [timrobertson100] [BEAM-4341] Enforce ErrorProne analysis in google-cloud-platform IO [timrobertson100] [BEAM-4335] Enforce ErrorProne analysis in amazon-web-services IO [timrobertson100] [BEAM-4339] Enforce ErrorProne analysis in elasticsearch IO [timrobertson100] [BEAM-4355] Enforce ErrorProne analysis in XML IO [timrobertson100] [BEAM-4338] Enforce ErrorProne analysis in common IO [timrobertson100] [BEAM-4337] Enforce ErrorProne analysis in cassandra IO [timrobertson100] [BEAM-4355] Reduces scope of findbugs annotations to build time only [timrobertson100] [BEAM-4353] Enforce ErrorProne analysis in solr IO [timrobertson100] [BEAM-4345] Enforce ErrorProne analysis in JDBC IO [timrobertson100] [BEAM-4336] Enforce ErrorProne analysis in AMQP IO [timrobertson100] [BEAM-4347] Enforce ErrorProne analysis in kafka IO [dariusz.aniszewski] [BEAM-4041] Increase timeout for getting K8s LoadBalancer external IP [timrobertson100] [BEAM-4346] Enforce ErrorProne analysis in JMS IO [timrobertson100] [BEAM-4352] Enforce ErrorProne analysis in Redis IO [lcwik] [BEAM-4271] Support side inputs for ExecutableStage and provide runner [daniel.o.programmer] [BEAM-2937] Add new Combine URNs. [tgroh] Update worker_id Documentation [robbe.sneyders] Futurize internal subpackage [dariusz.aniszewski] fixed typo [mariagh] Add validation for streaming wordcount (streaming_wordcount_debugging) [pablo] Increasing the concurrent test execution count (#5408) [tgroh] Add a Default RemoteEnvironment Implementation [sidhom] Require explicit IdGenerator for all DockerEnvironmentFactory [sidhom] [BEAM-4148] Add pluggable UrlFactory to ServerFactory [sidhom] [BEAM-4267] Docker-based JobBundleFactory [tgroh] Add a Window-only State Context [lcwik] Address Ben's comments. [github] Make hash function in Coder base class more conservative. [github] updated right documentation for HCatalogIO.write [sidhom] Fix DataStreamsTest checkstyle error [ankurgoenka] Updating python container version to beam-master-20180518 [wcn] Fix the test to properly run. [ankurgoenka] Reverting fnapi container version [ankurgoenka] Copyting dependency.py and dependency_test.py [ankurgoenka] reformatting code [ankurgoenka] Introducing classes [ankurgoenka] gruping public and non public methods [ankurgoenka] Renaming method and updating reference for file_copy and file_download [ankurgoenka] Removing Google specific code and unused code in stager. [ankurgoenka] Removing reference to Google/GCS/Dataflow and restructuring a bit of [ankurgoenka] Fixing Stager Tests [ankurgoenka] Making dependency.py use stager.py and fixing dependency_test.py test [ankurgoenka] Moving stager to portability [ankurgoenka] Removing stage_job_resources from dependencies.py and other minor [ankurgoenka] Merging filehandler in stager [ankurgoenka] renaming GCSStager to _ParameterizedStager [ankurgoenka] Making a few methods static and moving code around [ankurgoenka] Applying changes from rebase [szewinho] HDFS large cluster configuration. Jenkins job updated to use large [ankurgoenka] Fixing lint [iemejia] [BEAM-4343] Enforce ErrorProne analysis in HBaseIO [iemejia] [BEAM-4344] Enforce ErrorProne analysis in the HCatalogIO [tgroh] Add a BundleFactoryOutputReceiverFactory [tgroh] Add a DirectJobBundleFactory [tgroh] Fix Copyright Headers in sdks/java/fn-execution [yifanzou] rollback BEAM-3256 add archetype testing to gradle build PreCommits [ankurgoenka] Adding ArtifactStagingFileHandler to push files to staging server [ankurgoenka] Adding filesystem based artifact service to ULR job service [ankurgoenka] Adding context manager to ArtifactStagingFileHandler [ankurgoenka] Converting ArtifactServiceFileHandler to PortableStager based on [ankurgoenka] Simplifying numeric constants [ekirpichov] [BEAM-4375] Replaces usage of JUnit runners that do not work well with [matthias] Futurize options subpackage [szewinho] [BEAM-4065] Basic performance tests analysis added. [szewinho] Fixed typo in env name, added unit tests for create report logic, slack [szewinho] Jenkins job will run tests before executing script. Extended testing of [timrobertson100] [BEAM-4347] Improve ErrorProne analysis in kafka IO [timrobertson100] [BEAM-4348] Enforce ErrorProne analysis in kinesis IO [timrobertson100] [BEAM-4349] Enforce ErrorProne analysis in mongodb IO [timrobertson100] [BEAM-4351] Enforce ErrorProne analysis in mqtt IO [lcwik] [BEAM-4060] Add the pom.xml and pom.properties file to the artifact when [github] [BEAM-1755] Add a directory with build-specific scripts to clear up the [ankurgoenka] Logging and naming changes [ankurgoenka] Enhancing test cases [tgroh] Add an InProcessEnvironmentFactory [timrobertson100] [BEAM-4348] Removes double-checked locking to aid readability [ankurgoenka] Updating the dostring [tgroh] Update InProcessManagedChannelFactory [tgroh] Add a RemoteStageEvaluatorFactory [timrobertson100] [BEAM-4354] Enforce ErrorProne analysis in tika IO [iemejia] Fix build with Maven (Again!) [lukasz.gajowy] [BEAM-214] Apply reviewer's suggestions [timrobertson100] [BEAM-4347] Minor changes following PR review [timrobertson100] [BEAM-4347] Correction of typo following PR review [tgroh] Use WindowOnlyContext in ReduceFnContextFactory [tgroh] Update GroupAlsoByWindowEvaluatorFactory [tgroh] Move InProcessEnvironmentFactory to java-fn-execution [klk] Autoformat sdks/java/extensions/sql [klk] Add spotless config to global gradle file [klk] Activate spotless for SQL module [boyuanz] Solve BEAM-4393: Missing copyright notices for shaded packages [iemejia] Remove repeated GCPIO dependency in SQL module (Again!) [iemejia] Enforce error-prone in ParquetIO, fix missing javadoc and refine deps [katarzyna.kucharczyk] Replaced created datetime to jenkins build id in kubernetes namespace. [schroederc] Use ElementEncoders for beam.Create [boyuanz] Change bulid rules of sdks/java/harness to make harness as a uber-jar [swegner] Remove DoFnTester from testing guide. [robertwb] Slowly ramp up wait time in state sampler. [tgroh] Move the Local Artifact Services to the DirectRunner [tgroh] Migrate the ReferenceRunnerJobServer to runners-direct [tgroh] Fix License Headers [schroederc] Add test case for proto oneof fields in create_test.go [tgroh] Add a test to show Flatten Execution [apilloud] [SQL] Fix transitive dependencies [apilloud] [SQL] Run tests against actual jar [boyuanz] [BEAM-4393]: Copy root repo's LICENSE & NOTICE into shadowJar [herohde] Tweak Go integration test driver [tgroh] Update Docker Command [iemejia] [BEAM-4306] Enforce ErrorProne analysis in apex runner [daniel.o.programmer] [BEAM-4406] Updating Java portable environment major version. [daniel.o.programmer] [BEAM-4406] Updating Python portable environment major version. [ehudm] Add a custom _url_dirname for local filesystems. [tgroh] Add a Portable TransformEvaluatorRegistry [tgroh] Consistent Parameter Order in Evaluator Factories [tgroh] Make ImmutableExecutableStage Public [mergebot] Fix typo in becoming-a-committer docs [iemejia] Fix broken build (again third time this week yay!) [timrobertson100] [BEAM-4361] Document usage of HBase TableSnapshotInputFormat [jbonofre] Fix Maven parent version in ParquetIO [iemejia] [BEAM-4312] Enforce ErrorProne analysis in gearpump runner [mergebot] Add Apache Kudu IO to the planned IOs table [mergebot] Add next Java LTS to the WIP section of the contribution guide [mergebot] Add PubSubIO to Python IO table row [iemejia] [BEAM-4307] Enforce ErrorProne analysis in [mergebot] Change ParquetIO status in built-in I/O section [iemejia] [BEAM-4308] Enforce ErrorProne analysis in runners-core-java [ammons] Say that merging windows are not supported in Distinct.java. [lcwik] [BEAM-4411] Update Jackson version to latest to pick up security fixes. [kenn] Implement UNNEST operator for SQL [iemejia] [BEAM-4317] Enforce ErrorProne analysis in runners/reference/java [iemejia] Refine dependencies, make new ones explicit and minor maven plugins [iemejia] [BEAM-4323] Enforce ErrorProne analysis in sketching extensions [iemejia] [BEAM-4324] Enforce ErrorProne analysis in sorter extensions [iemejia] [BEAM-4321] Enforce ErrorProne analysis in join-library extensions [kenn] Add equals, hashCode, and structuralValue to Nexmark model objects [thw] [BEAM-4297] Streaming executable stage translation and operator for [thw] [BEAM-4297] Add serialization test and make output mapping stable. [lcwik] Add link to design docs and template to the contribution guide. [timrobertson100] [BEAM-4389] Enable partial updates in ElasticsearchIO [kenn] Force load of Calcite JDBC driver (avoids internal error) [kenn] Explicitly load Beam JDBC driver before test [iemejia] [BEAM-3813] Support (de)serialization of S3 encryption options via JSON [pablo] [BEAM-4426]: Addressed DataflowDistributionAccumulatorTest failure on [aaltay] Support installing Beam SDK from a wheel distribution in SDK containers. [cademarkegard] fix errorprone annotations and enable failOnWarning [lukasz.gajowy] Add Performance Testing to "Work in Progress" section [iemejia] Update postgres version to 42.2.2 (Java 8) [coheigea] Update README to include virtualenv in the build instructions [iemejia] [BEAM-4356] Enforce ErrorProne analysis in nexmark [kenn] Finish RowType -> Schema rename [kenn] Add convenience aliases for nullary type constructors [kenn] Add MAP field support to Schema.Builder [kenn] Add static factories for array, row, map types [kenn] Schema fields are non-null by default [kenn] TestUtils.RowsBuilder to use FieldType instead of TypeName [kenn] Clean up CalciteUtils conversions a bit [pablo] enable failOnWarning for build-tools [apilloud] Run everything with shadowJar [apilloud] guava is on kinesis public API [apilloud] Fix direct runner shadow config [kenn] Add more field conveniences to Schema.Builder [kenn] Remove extraneous RowSqlTypes [pablo] [BEAM-4322] Enforce ErrorProne analysis in protobuf extensions project [apilloud] Run test on shadowJar on a flag [kenn] Remove Schema.TypeName.type() [kenn] Remove unsafe builder methods from FieldType [apilloud] [SQL] Build mega jar for JDBC [apilloud] [SQL] JDBC Class Loader IT [ankurgoenka] proto changes to support artifact_staging_id [ankurgoenka] Doc changes [ankurgoenka] Renaming artifact_staging_id to staging_session_token [ankurgoenka] Fixing python code for staging_session_token proto changes [marek.simunek] [BEAM-4368] add euphoria java 8 dsl documentation section [coheigea] Avoid fully qualified class names, when there is an existing (static) [coheigea] Fixing ErrorProne MissingOverrides warnings [kedin] [SQL] Fix PubsubsJsonIT [apilloud] [SQL] Init compiler factory with JDBC class loader [pablo] clean generated files prior to lint [ankurgoenka] Fixing java code for staging_session_token proto changes [ankurgoenka] Go proto update [amaliujia] Add LIKE operator to Beam SQL. [kedin] [SQL] Replace planner.compilePipeline() with sqlEnv.parseQuery() [kedin] [SQL] Add sqlEnv.executeDdl() [kedin] [SQL] Make planner package-private [kedin] [SQL] Add factory methods to BeamSqlEnv [kedin] [SQL] Rename ReadOnlyTableProvider [coheigea] Fixing some ErrorProne warnings [github] print() is a function in Python 3 [cademarkegard] [BEAM-4303] Enforce ErrorProne analysis in examples project [boyuanz] Address BEAM-4328: :beam-sdks-java-io-google-cloud-platform:test failure [kedin] [SQL] Wrap SQL parsing exceptions in ParseException [tgroh] Add an abstraction for State and Timers [tgroh] DirectRunner Cleanups [tgroh] Link up the Portable DirectRunner [tgroh] Reuse ID Generators across Environments [tgroh] Use the Current Users Container as the Environment [aromanenko.dev] [BEAM-4421] Fix for issue with reading s3 files using ParquetIO [github] Fix flaky comparison in log_handler_test.py [tgroh] Use the Current Users Container as the Environment [tgroh] Enable Docker in the PortableDirectRunner [tgroh] Move DirectJobBundleFactory to fn-execution [tgroh] Cleanups to the PortableDirectRunner [tgroh] Cleanups and Wiring for the ReferenceRunnerJobService [amyrvold] [BEAM-4253] Update dataflow worker to fix ParDoTest fails in Dataflow [kedin] [SQL] Add ParseException to sqlEnv.explain() [kedin] [SQL] Add integration tests for BigQuery writes [kedin] Add TestBigQuery rule [tgroh] Use an UnsupportedArtifactRetrievalService [altay] Move assert_that's new keyword argument to the end [jhsueh] [BEAM-4320] Enforce ErrorProne analysis in jackson extensions project [jiangkai] introduce optimized logic plan [jiangkai] update [jiangkai] fix unit tests failed [iemejia] [BEAM-4310] Enforce ErrorProne analysis in [jiangkai] address comments [relax] Now that Dataflow is updated, we can finish updating tests and usages to [szewinho] [BEAM-3214] Add integration test for HBaseIO. [iemejia] [BEAM-4137] Remove MongoDB specific options from [ankurgoenka] Code review comments fixes [ajamato] Add new metrics protos based on s.apache.org/beam-fn-api-metrics [jiangkai] separate rules into sets [amaliujia] Address comments: 1. Use SqlFunction.like to implement like operator. 2. [apilloud] [SQL] Use AutoService for jdbc [apilloud] [SQL] AutoService for TableProvider [apilloud] [SQL] Don't cache IT results [apilloud] [SQL] Test that TextIO and Direct Runner work [apilloud] [SQL] Add context class loader hack [rober] [BEAM-4276] Add combiner lifting support to Go SDK [rober] fixup! Address comments. [herohde] [BEAM-4474] Ensure unbounded Dataflow jobs are submitted as streaming [herohde] Fix build break due to artifact staging change [relax] Add a test with ReferenceRunner [aromanenko.dev] Added design docs page [aromanenko.dev] Added first intro sentence. [aromanenko.dev] Moved one doc to other section and added several new docs [aromanenko.dev] Added more docs [github] [BEAM-4480] Fixed deprecated method invoking [lcwik] [BEAM-4481] Remove duplicate definitions of dependencies. [herohde] [BEAM-4475] Add container boot code to Go precommit [herohde] [BEAM-4475] Add Go submodule build to Go precommit [axelmagn] Fix ProvisionInfo in DockerJobBundleFactory [github] Improve default value for experiments set in RuntimeValueProvider [cademarkegard] [BEAM-4326] Enforce ErrorProne analysis in the fn-execution project [alan] [BEAM-4423] Mark pull requests stale after 60 days; close 7 days after [jbonofre] Moving to 2.6.0-SNAPSHOT on master branch [lukasz.gajowy] [BEAM-4415] Add jenkins job for HDFS ParquetIOIT [lukasz.gajowy] [BEAM-4137] Extract file-related options from IOTestPipelineOptions [lukasz.gajowy] [BEAM-4137] Extract postgres related options from IOTestPipelineOptions [timrobertson100] [BEAM-4311] Enforce ErrorProne analysis in Flink runner [github] [BEAM-4517] Add maven status in README [jiangkai] add expression boolean casting [coheigea] Removing extraneous whitespace around the equals operator [katarzyna.kucharczyk] Changed a namespace to parsed job name with current job number. [jiangkai] add boolean type for agg function [jiangkai] support EXISTS operator [github] Update Environments.java [github] Remove unneeded collection import. [github] Combine immutability type fixes. (#3) [github] fixup! [lcwik] BEAM-3876 avoid NPE if checkpoint is null in an unbounded source [lcwik] testing npe fix and exception rethrow in unit tests [lcwik] Update https://github.com/apache/beam/pull/4894 to correspond with [github] [BEAM-4276] Add missing scope line [jiangkai] explain plan unit tests [lcwik] [BEAM-4517] Add PyPI status badge. [aljoscha.krettek] [BEAM-3905] Update Flink Runner to Flink 1.5.0 [kedin] [SQL] Manually read rows in TestBigQuery [robertwb] [BEAM-3981] Cleanup of coders futurization. [robertwb] Use ByteCountingOutputStream for estimate_size. [kedin] Add polling assertion support to TestBigQuery [kedin] [SQL] Add Pubsub to BigQuery E2E integration test [sidhom] Add reference to ArtifactSourcePool implementation [sidhom] Clean up FlinkExecutableStageFunction and tests [sidhom] [BEAM-4523] Implement batch flink executable stage context [kedin] [SQL] Delete BeamSql [kedin] [SQL] Rename QueryTransform to SqlTransform [mergebot] Add SQL area to ongoing projects [robertwb] Reject unknown arguments in WindowInto. [mergebot] [BEAM-4488] How to install Python SDK from source [BEAM-4489] Document [mergebot] More changes [mergebot] updates after review [mergebot] Updating release guide to use Gradle commands. [mergebot] Addressing comments. Improving commands. [mergebot] Addressing comments. Improving commands. [mergebot] Update release guide for publishing source zip from GitHub [mergebot] Addressing TODOs [mergebot] Address comments. [mergebot] Improving. [mergebot] Add more updates [mergebot] More updates [mergebot] Addressed Alan's comments [mergebot] Add instructions about how to stage python binaries [mergebot] copy renamed zi[ file to sdks/python/staging subdir [mergebot] Update gradle publish with --no-parallel --no-daemon [kenn] Suppress immutability warning caused by LoadingCache in enum instance [kenn] Make BeamSqlExpression for operators, use it for string operators [ekirpichov] Add retrieval_token to GetManifestRequest [ekirpichov] Refactor code that accesses staging_token [ekirpichov] Regenerate go protobufs [kenn] Replace project and filter with calc [kenn] Disable CalcRemoveRule [szewinho] Improve consistency of hadoop and hbase scripts, fix shellcheck issues [iemejia] Fix build (missed auto service for SQL) and model-fn-execution on Flink [iemejia] [BEAM-4318] Enforce ErrorProne analysis in Spark runner [wcn] Encode position ids. [robertwb] Explain why module namespace is modified in Cython codepath. [apilloud] [SQL] Make getString more generic [ankurgoenka] Beam File System based Artifact Staging Service [ankurgoenka] BeamFileSystemArtifactStagingService Tests [ankurgoenka] Enhancing test case and error handling. [ankurgoenka] Enhancing testcases and incorporating review comments. [ankurgoenka] Review comments Fix [ankurgoenka] Rebasing [apilloud] [SQL] Support unquoted table types [herohde] Remove new Go precommit dependencies due to failures [jiangkai] address conflicts and comments [jiangkai] remove volcano plan function [klk] [SQL] Separate assertions for each expression in ExpressionChecker [klk] Remove StringOperatorsTest in favor of SQL syntax tests [mergebot] Add Python Kafka to list of in-progress I/O transforms. [pablo] Upgrade Gradle to version 4.8 [ehudm] Remove with_attributes keyword from ReadFromPubSub. [xiliu] [BEAM-3079] Update samza-runner with more features and improvements [xiliu] Add missing apache headers [swegner] Use new DSL methods for generated POM metadata [lcwik] [BEAM-4540] Migrate junit/hamcrest to provided scope. [swegner] [BEAM-925] Fix FindBugs violations in Dataflow Runner and re-enable [swegner] [BEAM-4313] Fix ErrorProne warnings and enable validation in Dataflow [ankurgoenka] Followup PR for #5591 [swegner] Add support for pre-commit trigger paths [swegner] Set path triggers for existing pre-commit test jobs [rober] Add graph.Fn encoding functions. [robinyq] Defer calling formatTimestamp() to achieve better performance [ccy] [BEAM-4549] Use per-pipeline unique ids for side inputs in [altay] Explain experimental/deprecated from users perspective. [jiangkai] call SqlFunctions in numeric type conversion [github] Update Dockerfile for new setup.py changes. [robbe.sneyders] Futurize utils subpackage [iemejia] [BEAM-4551] Update spark runner to Spark version 2.3.1 [iemejia] Fix maven build error on sdks/java/io/google-cloud-platform module [robinyq] Throws exception directly instead of calling checkState() [github] Remove GPL findbugs dependency (#5609) [lcwik] Clean-up BeamFnDataBufferingOutboundObserver since Dataflow has migrated [jiangkai] unit test for BuildExpression literal casting [aaltay] Futurize portability subpackage (#5385) [altay] Futurize unpackaged files [altay] resolved six.string_types equivalency [altay] Futurize testing subpackage [altay] Futurize tools subpackage [altay] Remove old_div [swegner] Revert "Merge pull request #5611: [BEAM-4445] Filter pre-commit [robertwb] [BEAM-4546] Implement hot key fanout for combiners. [robertwb] Use discaring mode for first level of combine. [echauchot] [BEAM-4283] Export nexmark query execution times to BigQuery [echauchot] [BEAM-4283] Make BigQueryIO Services classes public and experimental and [echauchot] [BEAM-4283] Fix PerfsToBigQueryTest and serialization issue in [echauchot] Add Etienne Chauchot's ongoing work to the contribute page. [fvoichick] Fixed link to nonexistent IsBounded.apply method [fvoichick] Fixed documentation of default window size in example code [fvoichick] Fixed inappropriate Javadoc comments [aaltay] [BEAM-4333] Add integration tests for python mobile games (#5630) [jiangkai] ignore explain unit test [kedin] [SQL] Add INSERT tests to JdbcDriverTest [kedin] [SQL] Add complex type test to JdbcDriverTest [kedin] [SQL] Make field names case-sensitive [robertwb] Suppress warning. [robertwb] Implement BeamFileSystemArtifactSource [robertwb] Add job submission capabilities to Flink runner. [robertwb] Update to new JobInvoker/Invocation interface. [robertwb] Remove references to Flink distributed file cache. [robertwb] A bit more documentation. [robertwb] Address comments on Flink code. [kirpichov] Removed some dead code, fixed some warnings [kirpichov] Adds tracing of watermarks in direct runner [kirpichov] Fixes typo in class name [kirpichov] Simplifies GBK expansion code a bit [kirpichov] Fixes bug: Processes values arriving for a GBK key sequentially [kirpichov] Introduces BundleProgressHandler. [swegner] Fix ErrorProne violations in direct runner and enable failOnWarnings [robertwb] [BEAM-4565] Fix hot key fanout in the face of combiner lifting. [robertwb] Remove trailing whitespace. [kirpichov] [BEAM-4473,BEAM-4281] Makes ReferenceRunner use a Direct stream observer [ccy] [BEAM-4549] Revert #5623 [kirpichov] [BEAM-4291] Implements distributed artifact retrieval [kirpichov] Deletes ArtifactSource [thw] Fix Flink runner Maven build. [ccy] [BEAM-4549] Fix side inputs for streaming pipelines on Dataflow [mergebot] [BEAM-2852] Update Nexmark documentation with Kafka support [mergebot] [BEAM-4479] Fixed document for Coder [swegner] Fix ErrorProne violations in fn harness and enable failOnWarnings [swegner] fixup! Fix ErrorProne violations in fn harness and enable failOnWarnings [pablo] Improving state sampler Cython annotations. [pablo] Adding msecs tracking to side inputs iterator [pablo] Adding the sampling thread to side input microbench. [pablo] Adding flag to mask change [pablo] Fixing all lint raising-format-tuple warnings. [github] Make build_rules.gradle version of shadow match build.gradle [swegner] Temporarily exclude website sources from RAT [swegner] fixup! Fix ErrorProne violations in fn harness and enable failOnWarnings [ehudm] Increase Jenkins build retention from 14 to 30 days. [ankurgoenka] Fixing concurrency bug in BeamFileSystemArtifactServicesTest [iemejia] Fix Maven build 7 [iemejia] [BEAM-4570] Use pagination in [thw] Fix ValidatesRunner_Apex #5604 [iemejia] [BEAM-4399] Get Cassandra options out of IOTestPipelineOptions [iemejia] [BEAM-4399] Refactor CassandraIOIT to be writeThenRead [iemejia] [BEAM-4399] Update Cassandra driver to version 3.5.0 [kenn] Fix name of heuristic JAVA_CLASS_TO_TYPENAME [kenn] [SQL] Allow explicit return type in ExpressionChecker [kenn] DSL-level tests for standard operators [kenn] Add test that all supported operators are tested [kenn] [SQL] Make parse failure slightly more actionable [pablo] Upgrading python dependency on httplib2 [kenn] Fix case sensitivity in JdbcDriverTest [ehudm] Jupyter notebook for monitoring precommit job times [lcwik] [BEAM-4576] Add hamcrest dependencies to this scope that was moved to [mergebot] Add link to @RequiresStableInput design doc [matthias] Futurize examples subpackage [apilloud] [SQL] Add parse code for SET [apilloud] [SQL] Plumb PipelineOptions through BeamRelNode [apilloud] [SQL] Plumb PipelineOptions through BeamSqlCli [apilloud] [SQL] Remove unused BeamIOType [ccy] [BEAM-4582] Fix streaming Create transform on Dataflow [kirpichov] Propagates artifact retrieval token in Flink runner and to the Java [ankurgoenka] load beam fileSystems in FnHarness [axelmagn] Implement new staging token contract in java SDK [robertwb] Add Impulse primitive to Python SDK [robertwb] Expand Read to Impulse | ParDo(Split) | ParDo(Read) in Python SDK [robertwb] Fix default docker image name. [kirpichov] Add an AddHarnessIdInterceptor [kirpichov] Remove the runners/reference InProcessManagedChannelFactory [kirpichov] Allow ManagedChannelFactories to attach Interceptors [kirpichov] Populate and Use Harness IDs in the Java Harness [apilloud] [SQL] Set application name to BeamSql [apilloud] [SQL] Correctly represent pipeline [apilloud] [SQL] Cleanup table IO functions [apilloud] [SQL] Simplify overly verbose names [apilloud] [SQL] Add each Calcite node to Beam graph once [apilloud] [SQL] Seperate parse from pipeline construction [swegner] fixup! Fix ErrorProne violations in fn harness and enable failOnWarnings [kirpichov] Fixes checkstyle [chamikara] Fixes broken links for downloading source releases. [lcwik] [BEAM-4592] Make Dataflow understand kind:varint as a well known since [chamikara] Fixes null pointer exception on empty pubsub message data (#5016) [kirpichov] A couple more trivial changes extracted from #5662 [lcwik] Update Dataflow container images [katarzyna.kucharczyk] Created static retry method which accepts retriable method, amount of [robertwb] Update impulse-based read. [kenn] Make BeamRelNode take a PCollectionList<Row> always [rober] Fix build break from PR 5676, which modified method signatures. [migryz] Implemented working prototype for PostCommit jobs [migryz] Update post commit jobs to utilize new helper methods [ankurgoenka] Fix cache references in portable flink runner [robertwb] [BEAM-4150] Extract IO coders from the RemoteGrpcPort. [pablo] Adding ignore parameters for gitattributes [swegner] Remove submodule [robertwb] Fix and test global combine with fanout. [migryz] Fix typo in _ghprb suffix [migryz] Update suffix to be PullRequest [ccy] Revert #5689 to fix build [ankurgoenka] Remove unused import [github] Make suffix shorter [kirpichov] Make ImmutableExecutableStage constructors public [kirpichov] [BEAM-4285] Implement Flink batch side input handler [ankurgoenka] Fixing key lookup in FlinkBatchSideInputHandlerFactory.SideInputHandler [ekirpichov] Introduces PipelineValidator that checks the well-formedness of a [cademarkegard] [BEAM-4325] Enforce ErrorProne analysis in the SQL project [echauchot] [BEAM-4283] Fix naming of the BigQuery fields [coheigea] Removing some null checks, where we already know that the variable in [iemejia] [BEAM-3314] Set correctly host and port on RedisIO [iemejia] [BEAM-3314] Fix error-prone warnings and add extra test for Read [matthias] Futurize direct runner [matthias] Futurize dataflow,experimental,job,portability and test subpackages of [matthias] Futurize complete runners subpackage [matthias] Cleanup futurize runners [lcwik] Revert "Revert #5689 to fix build" [github] Add migration notice to apache/beam website README.me [robertwb] Always consider Read to be a primitive for dataflow pipelines. [migryz] Point gearpump to master [robertwb] [BEAM-4605] Remove subtransform of GBK. [robbe.sneyders] Futurize typehints subpackage [kenn] Migrate build_rules.gradle to buildSrc plugin [amaliujia] support unbounded limit. [aaltay] Fix backlog bytes reported by CountingSource. (#5719) [mairbek] Fixed utf8 encoding in Cloud Spanner mutation encoder [robertwb] Don't require a staged SDK for the Python portable worker. [pablo] Enabling gradle build to receive extra buildscript repositories [kenn] Remove build_rules.gradle [daniel.o.programmer] [BEAM-2937] Comment Combine Grouped Values URN. [daniel.o.programmer] [BEAM-2973] Moving CombineValues URN to different enum. [robertwb] Add back old code path for dataflow. [klk] Fix errors in BeamModulePlugin perf test section [mergebot] Adds a dependencies guide to Beam Website [iemejia] Fix maven compile for the 8th time (yay!) [iemejia] Fix errorprone warnings on extensions/google-cloud-platform-core module [iemejia] Enable errorprone globally [klk] Fix Gearpump validatesRunner gradle config [robertwb] Stage empty manifest if a staging session is given. [migryz] [BEAM-4559] Add latest post-commit tests status to GH PR [sidhom] Add registrars for the portable runner [sidhom] Add endpoint logging for PortableRunner [github] dataflow_runner.py: use the pbegin's pipeline [github] materialize_test.go: delint: non-idiomatic block [klk] Fix docker gated code paths [altay] Add instructions about how to perform release validations [ehudm] Parallelize Java precommits [scott] Spread jenkins load by randomizing daily cron schedule [amaliujia] logical operator tests [kirpichov] Adds support for SDF in ULR and the Java SDK. [klk] Allow specifying delimiter in TextIO [robbe.sneyders] Address PR comments [klk] Migrate some date expressions to operators [klk] Use label "stale" when closing stale PRs [kenn] Add support for "lines" format text tables [kenn] Beam SQL shell prompt is now 'BeamSQL>' [kenn] Override SqlLine.properties to customize welcome and help messages [pablo] Improving documentation for Metrics results [xiliu] Finalize Samza Runner with gradle and findbugs issues [amaliujia] add comparison operator test [ekirpichov] Address easy review comments [xiliu] Fix a unit test timeout and minor issues in pom.xml [kirpichov] Handles memoization of encoded key/window in StateAccessor [kirpichov] Simplifies API of WireCoders and LengthPrefixUnknownCoders. [sidhom] [BEAM-4615] Flink job server wrapper and shadow jar [apilloud] [SQL] Don't wait on Unbounded PCollections [kirpichov] Address comments [xiliu] [BEAM-4640] Samza runner postcommit ValidatesRunner job [aaltay] [BEAM-4333] Add integration tests for python mobile games (#5747) [jiangkai] [BEAM-4602][BEAM-4598] Date type comparison [github] Cythonizing a few hot calls (#5733) [daniel.o.programmer] [BEAM-3708] Adding Combine component implementations to Java SDK [iemejia] Fix maven build part 9 [xiliu] Remove unused previous names [daniel.o.programmer] Quick fix, removing commented out code. [altay] Publish 2.5.0 release [altay] Change download link for 2.4.0 version to use the archive link [altay] improvements to fanout [yifanzou] [BEAM-4302] Add Jenkins job to check Beam dependencies and generate [ekirpichov] Fixes Flink ValidatesRunner and a compile error [scott] Split pre-commit tests into separate jobs based on trigger condition. [scott] Filter pre-commit triggering based on touched files [scott] Rename setPostCommit to setAutoJob [amyrvold] [BEAM-4645] Cleanup docker images on failure [amaliujia] replace byte[] with String as Map key. [scott] Fix ErrorProne warnings in sdks-java-core project [scott] Enable ErrorProne analysis for all Java projects. [scott] Update Findbugs annotations dependency to version with [scott] Enable failOnWarnings for all projects. [pablo] Removing debug string from UserScore.java [scott] Add checkstyle validation to ban usage of @SuppressFBWarnings. [amyrvold] [BEAM-3261] Fix pipeline failures by waiting for logging server to be [github] Revert "improvements to fanout" [klk] Apply rename of JobBuilder to Samza VR job [xiliu] [BEAM-4641] Samza runner postcommit status in PR template [kenn] Add bundle-ready SQL shell using application plugin [relax] Use ByteBuddy to generate a coder class for a specific Schema. [relax] Subclass proper generic type, removing need for extra typecast. [relax] Address code-review comments. [scott] Rename refactored Precommit source files to remove GradleBuild suffix. [herohde] Run gofmt [kenn] Reduce stdout spam from BeamModulePlugin [scott] Update Pre-commit job names tracked in precommit_job_times.ipynb [lcwik] [BEAM-4661] Define well known timer URN. [herohde] [BEAM-4635] Avoid cross-compiling worker binary if linux-amd64 [klk] Increase website job timeout to 100 by default [scott] Add Apache license headers [ankurgoenka] Adding flinkMarterUrl as an option [robertwb] Give the root transform in Python a non-empty name. [herohde] [BEAM-4635] Do not remove worker binary on Go Dataflow staging [amaliujia] Improve BigQuery write integration tests and support TIMESTAMP SQL type [ryan.blake.williams] [BEAM-2810] use fastavro-blocks in Avro IO [klk] Fix accidental license header javadoc comments [klk] Fix HTML error in ApproximateQuantiles [klk] Turn on spotless java everywhere [aaltay] [BEAM-4606] Pin pytz version (#5751) [scott] Update pre-commit filter to include migrated build rules [klk] Do not email individuals about failures in Jenkins [klk] Pipeline options in JDBC URI; default userAgent=BeamSQL [klk] Spotless Java everywhere! [klk] Simplify checkstyle now that spotless is in play [klk] Temporarily disable JavadocParagraph and LineLength checkstyle [scott] Remove blanket website from rat and add appropriate exclusions. [github] improvements to fanout (#5800) [herohde] Do not launch containers with transient semi_persistent_dir. Fails on [herohde] Go-on-Flink: treat artifact staging EOF error as ok [herohde] Go-on-Flink: make transform UniqueNames _globally_ unique [herohde] Go-on-Flink: remove internal java runner class hack [herohde] Go-on-Flink: fix swapped arguments in artifact staging [herohde] Go-on-Flink: use clearer error if the artifact retrieval token or [herohde] Go-on-Flink: fix Go to actually send worker_id gRPC header [lukasz.gajowy] Remove mvn leftovers from jenkins IOIT job definitions [klk] Rearrange checkArgument to help findbugs [chamikara] Reduces the size of pretty string of BigQuery jobs [mariagh] Add Python Nexmark suite [herohde] CR: Don't enforce non-empty control worker_id yet [xiliu] Make Samza runner in lexicographical order [robertwb] [BEAM-4285] Extend side input handlers to handle multiple access [relax] Rename RowType -> Schema and move to new schemas package. [relax] Add classes for SchemaRegistry and SchemaProvider. [relax] Add SchemaCoder, some more detail in SchemaProvider/Registery, and [relax] Update to comment. [relax] Integrate Schemas into PCollection Coder inference. [relax] Add setSchema convenience method. [relax] Fix compilation errors. [relax] Refactor Schema: Schema now is a fully-specified based on primitive [relax] Implement SchemaCoder and fix up schema classes. [relax] Flesh out FieldAccessDescriptor. [relax] Start plumbing through DoFn. [robertwb] Rename several MultimapSideInput* classes to more generic SideInput*. [relax] Start editing signatures [relax] Finish plumbing schemas through. [relax] Add test for schema PCollection. [relax] Add withSchema to Create.java [relax] Plumb schemas through OutputReceiver. [relax] Add an implementation of SchemaProvider. [relax] Add SchemaRegistry and a test for schema inference. Change schema [relax] Add unit test for SchemaRegistry. [relax] Add FieldAccessDescriptorTest [relax] Add test for Create changes. [relax] Plumb FieldAccess through. [relax] Make sure everything is marked experimental with Kind.SCHEMAS [relax] Add graph-construction time verification of schemas. [relax] Fix build and style failures. [relax] Address code-review comments. [relax] Fixup after merge. [relax] Apply spotless autoformat. [lcwik] [BEAM-4659] Add well known timer coder to Java SDK. [kirpichov] Makes fuser materialize PCollections used as side inputs [relax] Fix compilation error introduced by merge. [ankurgoenka] Uploading job server jar when running on remote flink environment [ankurgoenka] Passing files to stage on flink explicitly while creating the [ankurgoenka] Spotless [robertwb] Remove obsolete comment about multimap fitting into memory. [relax] Fix CheckStyle errors. [ankurgoenka] Job service additionally need google cloud dependency to use GCS [relax] Fix Apex runner. [relax] Fix more breakages. [relax] run spotless. [kirpichov] Makes Python PortableRunner usable via PipelineOptions [amaliujia] Remove unused system.out.println [ankurgoenka] TODO to add more file systems [kirpichov] Brings back hack for embedded FnApiRunner [robertwb] [BEAM-3883] Stage files in the portability runner. [robertwb] Fix tests due to api change. [kirpichov] Changes portableWordCount gradle task to be runnable at master [ankurgoenka] Adding non standard side input urn to flink [relax] Fix issues. [relax] Fix Spark issue. [kirpichov] Fixes spotless errors at master [mariagh] Allow streaming mobile examples to read from either a topic or a [relax] Address code-review comments and fix a failing test. [relax] Fix Apex runner. [echauchot] [BEAM-2850] Add Nexmark PostCommit runs for spark, flink and direct [relax] Change ParDoSchemaTest to ValidatesRunner, and exclude unsupported [relax] Make worker changes backwards compatible with DataflowRunner. Once [mariagh] Address review comments [amaliujia] Support BigQuery read from SQL. [relax] Add another backwards-compatible method. [ekirpichov] Makes FileIO.match watermark advance even without new files [pablo] Test case for providing experiments to pipeline [kirpichov] [BEAM-4689] Reverts change of SDF key type [daniel.o.programmer] [BEAM-3708] Adding grouping table to Precombine step. [daniel.o.programmer] [BEAM-3708] Simplifying precombine grouping tables. [klk] Add integrationTest task for GCP IO module [klk] PubsubIO: Always use GCP project from PipelineOptions for subscriptions [klk] Use BagState in PubsubSignal since it doesn't use SetState methods [klk] Replace println with logging [klk] More logging when PubsubJsonIT fails [klk] Add start signal to TestPubsubSignal [klk] Use futures in PubsubJsonIT LIMIT test [klk] Integration test of Pubsub public dataset [klk] Use test PipelineOptions in PubsubJsonIT LIMIT test [klk] Sickbay PubsubReadIT on Dataflow [klk] Fix import order [mariagh] Add query 1 & 2 [mariagh] Clean up [mariagh] Cleaup [mariagh] Fix linter errors [mariagh] Address review commnets; Reuse topic for queries [github] Return timeout value to 180 [relax] Fix build. [relax] Add SchemaProviderRegistrar class. [klk] Revert accidental clobber of MinimalWordCount.java [relax] Exclude schema tests from Samza runner. [klk] Update BigQueryAvroUtils to support BigQuery's NUMERIC type. [relax] Add DefaultSchema and fix failures turned up by plugins. [klk] Convert Beam Row to Avatica Row in BeamEnumerableCollector [kenn] sqlline dep to 1.4.0 [kenn] Fix broken javadoc [kenn] Remove more extraneous printlns from build [kenn] Remove aliased tables in Nexmark SQL query 5 [lcwik] Revert "Enabling gradle build to receive extra buildscript repositories" [lcwik] Don't enable plugins which require you to be online when run in [kenn] fixup! Remove aliased tables in Nexmark SQL query 5 [lcwik] [BEAM-4126] Delete Maven build files. [kenn] Simplify/fix windowing check in BeamSortRel [robertwb] Fix global hot key fanout setting. [scott] Increase concurrency in ValidateRunner execution [lcwik] [BEAM-3971, BEAM-4284] Remove fromProto for Pipeline and PTransform [robinyqiu] Support @RequiresStableInput on Dataflow runner in Java SDK [kenn] Run SQL operator tests through JDBC as well [kenn] Test retrieval of TIMESTAMP via JDBC [kenn] Instantiate $SUM0 as a SUM operator [ankurgoenka] Read worker harness image name from pipeline option for portable runners [ankurgoenka] Spotless [kenn] Instantiate "+" as DATETIME_PLUS [ankurgoenka] Support only a single environment in SdkComponents [ankurgoenka] Renaming createEnvironment to createOrGetDefaultEnvironment. [lcwik] [BEAM-4716] Remove findbugs declarations in build.gradle files since it [relax] Address review commemnts. [ccy] [BEAM-4007] Fix TODO style in typehints.py [relax] Add schema inference from POJO. [relax] Add BYTES type to schema, and POJO inference. [relax] Add automatic getter/setter inference from POJOS. * Recursive POJOs [relax] Handle setter for ByteBuffer fields. Setter will be called with a byte[] [relax] Split Row into separate subclass for RowWithStorage and RowWithGetters. [relax] Introduce FieldValueGetterFactory and FieldValueSetterFactory. [relax] Add support for nested, array, and map types. [relax] Refactor getter-based SchemaProvider into a common base class. [relax] Refactor getter/setter conversion utilities into a library. This cleans [relax] Refactor Schema inference from POJOs so much of the code can be reused [relax] Add encoding byte BYTES type. [relax] Make FieldValueGetter and FieldValueSetter generic types again. [relax] Start adding JavaBean getters and setters. [relax] Make sure that field order is stable, and make tests robust to [relax] Simplify convertArray. [amaliujia] support decimal type in reading from PubSub JSON [ankurgoenka] Fix artifact retrival file system [jiangkai] [BEAM-4723] Support date type plus interval [ankurgoenka] Create SDKComponents with pipeline options [echauchot] [BEAM-4707] Add timestamp field to Nexmark tables and use explicit mode [lukasz.gajowy] [BEAM-4137] Divide separate PipelineOptions for elasticsearch tests [lukasz.gajowy] [BEAM-4137] Remove unused options from IOTestPipelineOptions [katarzyna.kucharczyk] Added integration test configuration to beam-examples gradle. Fixed [markliu] Move gradle-spotless plugin to buildscript [rober] Add a simple benchmark to the ParDo execution node. [robertwb] Use main output for cold keys. [pablo] Unpin GCP test requirement [lcwik] [BEAM-4654] Translate timers as PCollections within the Java SDK. [robertwb] Use local GCP credentials, if any, in local docker environment. [cclauss] [BEAM-1251] Modernize Python 2 code to get ready for Python 3 [robertwb] Split bounded sources in larger than 1k chunks. [cclauss] [BEAM-1251] Upgrade from buffer to memoryview (again) [thw] Use the beam:option:value:v1 as the portable pipeline options [thw] [BEAM-4733] Pass pipeline options from Python portable runner to job [matthias] removed unnecessary install_aliases, import io and unicode imports [thw] Fix Flink portable streaming translation executable stage output [echauchot] [BEAM-4707] insert timestamp to bigQuery as seconds as expected by [echauchot] [BEAM-4671] Skip MetricsPusherTest validates runner test for runners [matthias] Futurize io subpackage [matthias] incorporated all feedback for futurize io subpackage [robertwb] object() takes no parameters [amaliujia] remove decimal support in BigQuery SQL read. [iemejia] [BEAM-4016] Invoke Setup and TearDown on SplitRestrictionFn and [ankurgoenka] Initialization BeamFileSystem in FlinkExecutableStageFunction [ankurgoenka] [BEAM-4644] Use SLF4J in ExecutableStageDoFnOperator [ccy] Revert #5878 to unbreak Python postcommit [ankurgoenka] Pull the docker image before starting docker container. [iemejia] Rename COMBINE_TRANSFORM_URN to the more proper [ccy] [BEAM-4003] Fix missing iteritems import [relax] Address code-review comments. [relax] Address code-review comments. [github] Avoid silent error when calling unimplemented function. [github] Avoid silent error when calling unimplemented function. [relax] Add better validation to isSetter isGetter [ekirpichov] Infer boundedness for SDF application at application time [ekirpichov] [BEAM-4745] Revert "[BEAM-4016] Invoke Setup and TearDown on [markliu] Remove :buildSrc from settings.gradle since it's not a subproject [pablo] Logging relies on StateSampler for context [ccy] [BEAM-4594] Beam Python state and timers user-facing API [ccy] [BEAM-4593] Remove refcounts from the Python SDK [ankurgoenka] FileSystems should throw exception if filesystem for schema is not found [ankurgoenka] Make schema pattern stricter [ccy] [BEAM-1251] Revert #5887 to unbreak Python PostCommit [pablo] Improving signing of all published artifacts and adding test publication [ccy] [BEAM-1251] Upgrade snappy and use a memoryview [pablo] Removing extra publication [cclauss] [BEAM-3761] Define cmp() in Python 3 [cclauss] [BEAM-3959] Add Python 3 undefined names to flake8 [amyrvold] [BEAM-3457] Upgrade version of gogradle and add examples and containers [apilloud] [SQL] Inject JDBC rules through Hook. [lcwik] [BEAM-4481, BEAM-4484] Start vendoring portability dependencies to not [apilloud] [SQL] Plumb pipelineOptions through IOSinkRel [lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers. [apilloud] [SQL] Default timezone is UTC [lcwik] [BEAM-2915] Add support for handling bag user state to the [amaliujia] add testArithmeticOperator [altay] Wait on connection channel to be ready [ekirpichov] Bumps Dataflow worker container version [ekirpichov] [BEAM-4745, BEAM-4016] Reintroduces Setup and TearDown on [amaliujia] simplify unbounded limit [iemejia] [BEAM-4020] Add an HBaseIO read implementation based on SDF [iemejia] [BEAM-4020] Add HBaseIO.readAll() based on SDF [lukasz.gajowy] [BEAM-4691] Rename common_job_properties [aromanenko.dev] [BEAM-4622] Makes required to call Beam SQL expressions validation [aromanenko.dev] Check number of arguments at first [lcwik] [BEAM-4764] Fix missing reference to project to be able to refer to the [lcwik] [BEAM-4764] Fix java precommit to ensure that any checks in model/* are [apilloud] [SQL] Apply JDBC rules globally [rober] Add a small benchmark for metrics [ankurgoenka] Warn on docker image pull fail [robinyqiu] Fix bug in java mobile gaming example [lcwik] [BEAM-4744] Enable parallel publishing/release [qinyeli] Interactive runner and corresponding tests [github] [BEAM-4648] Remove experimental Python RPC DirectRunner (#5777) [s.lebedev] [BEAM-1251] Replace NameError-driven dispatch with ``past`` [altay] Install fastavro only in linux [pablo] [BEAM-4718]Run gradle build before publish [pablo] Add CODEOWNERS file (#5909) [pablo] Full tracking of Batch side inputs. Performance tests show no regression [kirpichov] Adds license header to CODEOWNERS [kirpichov] Adds a naive implementation of bounded SDFs [echauchot] Move echauchot from runner/core to runner/core/metrics in CODEOWNERS [github] [BEAM-4752] Add support for newer dill dependency (#5931) [elliottb] Fix the expected encoding of BigQuery's NUMERIC type when reading from [ankurgoenka] Adding tet cases for matchResouces [kirpichov] Makes SplittableDoFnTest exercise both bounded and unbounded SDFs. [github] [BEAM-4752] Add dill compatibility for older versions of dill (#5949) [ankurgoenka] Ignore generated vendor files in python container build [kirpichov] Supports bounded SDF in all runners. [relax] Add new Convert transform for schemas. [relax] Fix two bugs: * proper handling of nullable fields. * Row stores [relax] Add tests. [relax] Fix broken test. [relax] Add @Nullable annotation. [github] [BEAM-4432] Adding Sources to produce Synthetic output for Batch [ryan.blake.williams] send PutArtifactResponse in BeamFileSystemArtifactStagingService [apilloud] [BEAM-4774] Add Nexmark SQL to postcommits [aaltay] [BEAM-2810] fastavro integration test (#5862) [kirpichov] Improve error messages on Spark and Flink SDF translation [qinyeli] Interactive Beam -- yields Read Trans after apply() [apilloud] [SQL] Actually run JDBC Jar Test [apilloud] [SQL] Wrap PipelineOptions with correct class loader [kirpichov] Converts BoundedReadFromUnboundedSource to a DoFn [boyuanz] Automate release branch cut process [yifanzou] [BEAM-4801] add dependency owners file [lcwik] [BEAM-4769] Fix issue when performing coder inference for wildcard [lcwik] [BEAM-4744] Fix runners/google-cloud-dataflow-java/examples-streaming [ankurgoenka] Fixing Test case for beam Artifact Staging Service [boyuanz] Fix javadoc broken [daniel.o.programmer] [BEAM-3711] Enabling combiner lifting in Dataflow Runner. [ankurgoenka] Fix for Flaky test BeamFileSystemArtifactServicesTest [lcwik] [BEAM-4418] Fix duplicate source set declaration conflict. [pablo] Add coders microbenchmark. (#5565) [melissapa] [BEAM-3769] Fix broken JIRA links in README [pablo] Moving to 2.7.0-SNAPSHOT on master branch. [thw] [BEAM-4791] Portable Flink runner integration test. [thw] Use shadowTest configuration since we don't plan to shade away the SDK [thw] [BEAM-4791] Move executor to Before/After and use [github] Increase the complete time out [github] Minor javadoc fix up. [amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub messages [github] Adding @angoenka to CODEOWNERS [boyuanz] Fix javadoc build broken in synthetic for release branch [garrettjonesgoogle] Bumping google cloud versions [thw] Add JIRA reference. [lcwik] Address PR comments. [pablo] Fix bugs in 'cut release branch step' [lcwik] Address PR comments. [amaliujia] [BEAM-4806] improve unit tests for SqlLineMain [qinyeli] Added a round trip through proto in Interactive runner. [github] Add version bound to numpy. [echauchot] CodeOwners: Narrow down extensions-java to metrics and assign echauchot [jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem [jxlewis] Updated documentation, added BEAM-4828 to In-Progress [github] Fix indentation. [github] SQS -> Amazon SQS [boyuanz] Added '--continue' switches into nightly build [supercclank] Set BatchPath on google.api.services.storage.Storage Storage Builder [kirpichov] Converts SolrIO away from BoundedSource [supercclank] Change to using java.nio.file.Paths over string concatenation for batch [apilloud] [BEAM-4761] Run Nexmark on Dataflow [github] [BEAM-4832] Fix concurrent race in datamgr.go [github] Update datamgr.go [carlos.alonso] Adds the BigQueryInsertError model class [carlos.alonso] Adds BigQueryInsertErrorCoder and tests [carlos.alonso] Adds extended errors info to WriteResult [carlos.alonso] Makes streaming methods generic but limited to the implemented error [carlos.alonso] Removes ErrorContainers auxiliary class. Adds BigQueryServicesImpl test [carlos.alonso] Makes BigQueryIO.Write to be configurable to decide which kind of errors [carlos.alonso] Removes unneeded coder registration [carlos.alonso] Simplifies BigQueryInsertError codification [carlos.alonso] A few style improvements [katarzyna.kucharczyk] Changed order of installing perfkit requirements and beam requirements [thw] Remove dead link from README. [amyrvold] [BEAM-4831] Ignore failures during :beam-sdks-go:vet to allow ./gradlew [aaltay] Add automation script for verifying release build (#5973) [ekirpichov] Avoids use of discouraged Duration(long) ctor [qinyeli] Interactive Runner -- keep_alive [thw] [BEAM-4842] Update Flink Runner to Flink 1.5.1 [daniel.o.programmer] [BEAM-3709] Implementing new combine URNs in python. [aaltay] Add bash script to automate "Preparation for GPG" (#6015) [github] Fixing log message [lukasz.gajowy] [BEAM-4845] Make BigQueryServices and FakeDatasetService public to fix [amyrvold] Fix [BEAM-4847] by manually setting max workers and jvm memory and add [relax] Enable schemas for Apex runner. [relax] Enable Schemas on samza runner. [relax] Enable schemas for Flink runner. [relax] Enable schemas for Spark. [relax] Fix bugs. [relax] Fix Apex breakage. [ehudm] Remove CODEOWNERs. [kedin] [SQL] Enable running BeamSqlLine from gradle [thw] [BEAM-4847] Reduce Gradle JVM Xmx to 4g to fix Jenkins build failures. [aaltay] Automate 'Start a snapshot build' step in beam release guide (#6042) [lukasz.gajowy] [BEAM-4838] Add dockerfile for standalone Jenkins. Plugins included. [dariusz.aniszewski] separate virtualenvs for Beam and PerfKit [lcwik] [BEAM-4866] Fix missing licenses. [pablo] Removing scoped metrics container [pablo] Remove old style metrics context management [garrettjonesgoogle] Bumping versions that were missed in #5988 [altay] Remove reference to dataflow-distribution.properties [lcwik] [BEAM-4176] Initial implementation for running portable runner tests [pablo] Fix Java Nightly Snapshot Failures [lcwik] [BEAM-4629] Output the names of the failing licenses as part of the [aaltay] [BEAM-4859] Enable Python VR tests in streaming in postcommit task ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on beam7 (beam) in workspace <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/apache/beam.git # timeout=10 Fetching upstream changes from https://github.com/apache/beam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/apache/beam.git > +refs/heads/*:refs/remotes/origin/* > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/* > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision aa2b585a5356ce3b607dd3d17a3c6ab2ce65353b (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f aa2b585a5356ce3b607dd3d17a3c6ab2ce65353b Commit message: "Merge pull request #6056: [BEAM-4836] separate virtualenvs for Beam and PerfKit" > git rev-list --no-walk 0892e23940a584a170e2668d86b7bdc50c03e024 # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 [EnvInject] - Executing scripts and injecting environment variables after the SCM step. [EnvInject] - Injecting as environment variables the properties content SPARK_LOCAL_IP=127.0.0.1 [EnvInject] - Variables injected successfully. [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins266031542697616565.sh + rm -rf PerfKitBenchmarker [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins2118418907451248094.sh + rm -rf .beam_env [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins1013385759992617214.sh + rm -rf .perfkit_env [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins1440131868588494871.sh + virtualenv .beam_env --system-site-packages New python executable in <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/.beam_env/bin/python2> Also creating executable in <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/.beam_env/bin/python> Installing setuptools, pkg_resources, pip, wheel...done. Running virtualenv with interpreter /usr/bin/python2 [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins3203404953492079456.sh + virtualenv .perfkit_env --system-site-packages New python executable in <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/.perfkit_env/bin/python2> Also creating executable in <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/.perfkit_env/bin/python> Installing setuptools, pkg_resources, pip, wheel...done. Running virtualenv with interpreter /usr/bin/python2 [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins3388230821306939726.sh + .beam_env/bin/pip install --upgrade setuptools pip Requirement already up-to-date: setuptools in ./.beam_env/lib/python2.7/site-packages (40.0.0) Requirement already up-to-date: pip in ./.beam_env/lib/python2.7/site-packages (18.0) [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins8681682376553589136.sh + .perfkit_env/bin/pip install --upgrade setuptools pip Requirement already up-to-date: setuptools in ./.perfkit_env/lib/python2.7/site-packages (40.0.0) Requirement already up-to-date: pip in ./.perfkit_env/lib/python2.7/site-packages (18.0) [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins1014785923945432727.sh + git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git Cloning into 'PerfKitBenchmarker'... [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins7488451273964902903.sh + .beam_env/bin/pip install -e 'src/sdks/python/[gcp,test]' Obtaining file://<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/src/sdks/python> Requirement already satisfied: avro<2.0.0,>=1.8.1 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (1.8.2) Requirement already satisfied: crcmod<2.0,>=1.7 in /usr/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (1.7) Requirement already satisfied: dill<=0.2.8.2,>=0.2.6 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.2.8.2) Requirement already satisfied: grpcio<2,>=1.8 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (1.13.0) Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (2.1.0) Requirement already satisfied: httplib2<=0.11.3,>=0.8 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.11.3) Requirement already satisfied: mock<3.0.0,>=1.0.1 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (2.0.0) Requirement already satisfied: oauth2client<5,>=2.0.1 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (4.1.2) Requirement already satisfied: protobuf<4,>=3.5.0.post1 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (3.6.0) Requirement already satisfied: pydot<1.3,>=1.2.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (1.2.4) Requirement already satisfied: pytz<=2018.4,>=2018.3 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (2018.4) Requirement already satisfied: pyyaml<4.0.0,>=3.12 in /usr/local/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (3.12) Requirement already satisfied: pyvcf<0.7.0,>=0.6.8 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.6.8) Requirement already satisfied: six<1.12,>=1.9 in /usr/local/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (1.11.0) Requirement already satisfied: typing<3.7.0,>=3.6.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (3.6.4) Requirement already satisfied: futures<4.0.0,>=3.1.1 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (3.2.0) Requirement already satisfied: future<1.0.0,>=0.16.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.16.0) Requirement already satisfied: fastavro==0.19.7 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.19.7) Requirement already satisfied: google-apitools<=0.5.20,>=0.5.18 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.5.20) Requirement already satisfied: proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.90.4) Requirement already satisfied: googledatastore==7.0.1 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (7.0.1) Requirement already satisfied: google-cloud-pubsub==0.26.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.26.0) Requirement already satisfied: proto-google-cloud-pubsub-v1==0.15.4 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.15.4) Requirement already satisfied: google-cloud-bigquery==0.25.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (0.25.0) Requirement already satisfied: nose>=1.3.7 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (1.3.7) Collecting numpy<2,>=1.14.3 (from apache-beam==2.7.0.dev0) Using cached https://files.pythonhosted.org/packages/85/51/ba4564ded90e093dbb6adfc3e21f99ae953d9ad56477e1b0d4a93bacf7d3/numpy-1.15.0-cp27-cp27mu-manylinux1_x86_64.whl Requirement already satisfied: pyhamcrest<2.0,>=1.9 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) (1.9.0) Requirement already satisfied: enum34>=1.0.4 in /usr/local/lib/python2.7/dist-packages (from grpcio<2,>=1.8->apache-beam==2.7.0.dev0) (1.1.6) Requirement already satisfied: docopt in /usr/local/lib/python2.7/dist-packages (from hdfs<3.0.0,>=2.1.0->apache-beam==2.7.0.dev0) (0.6.2) Requirement already satisfied: requests>=2.7.0 in /usr/local/lib/python2.7/dist-packages (from hdfs<3.0.0,>=2.1.0->apache-beam==2.7.0.dev0) (2.18.4) Requirement already satisfied: pbr>=0.11 in /usr/lib/python2.7/dist-packages (from mock<3.0.0,>=1.0.1->apache-beam==2.7.0.dev0) (1.8.0) Requirement already satisfied: funcsigs>=1; python_version < "3.3" in /home/jenkins/.local/lib/python2.7/site-packages (from mock<3.0.0,>=1.0.1->apache-beam==2.7.0.dev0) (1.0.2) Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python2.7/dist-packages (from oauth2client<5,>=2.0.1->apache-beam==2.7.0.dev0) (0.4.2) Requirement already satisfied: pyasn1-modules>=0.0.5 in /home/jenkins/.local/lib/python2.7/site-packages (from oauth2client<5,>=2.0.1->apache-beam==2.7.0.dev0) (0.2.2) Requirement already satisfied: rsa>=3.1.4 in /home/jenkins/.local/lib/python2.7/site-packages (from oauth2client<5,>=2.0.1->apache-beam==2.7.0.dev0) (3.4.2) Requirement already satisfied: setuptools in ./.beam_env/lib/python2.7/site-packages (from protobuf<4,>=3.5.0.post1->apache-beam==2.7.0.dev0) (40.0.0) Requirement already satisfied: pyparsing>=2.1.4 in /home/jenkins/.local/lib/python2.7/site-packages (from pydot<1.3,>=1.2.0->apache-beam==2.7.0.dev0) (2.2.0) Requirement already satisfied: fasteners>=0.14 in /home/jenkins/.local/lib/python2.7/site-packages (from google-apitools<=0.5.20,>=0.5.18->apache-beam==2.7.0.dev0) (0.14.1) Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.5.2 in /home/jenkins/.local/lib/python2.7/site-packages (from proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.7.0.dev0) (1.5.3) Requirement already satisfied: gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 in /home/jenkins/.local/lib/python2.7/site-packages (from google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (0.15.4) Requirement already satisfied: google-cloud-core<0.26dev,>=0.25.0 in /home/jenkins/.local/lib/python2.7/site-packages (from google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (0.25.0) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/dist-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.7.0.dev0) (2018.4.16) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python2.7/dist-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.7.0.dev0) (3.0.4) Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python2.7/dist-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.7.0.dev0) (2.6) Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python2.7/dist-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.7.0.dev0) (1.22) Requirement already satisfied: monotonic>=0.1 in /home/jenkins/.local/lib/python2.7/site-packages (from fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.7.0.dev0) (1.5) Requirement already satisfied: google-gax<0.16dev,>=0.15.7 in /home/jenkins/.local/lib/python2.7/site-packages (from gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (0.15.16) Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in /home/jenkins/.local/lib/python2.7/site-packages (from gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (0.11.4) Requirement already satisfied: google-auth-httplib2 in /home/jenkins/.local/lib/python2.7/site-packages (from google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (0.0.3) Requirement already satisfied: google-auth<2.0.0dev,>=0.4.0 in /home/jenkins/.local/lib/python2.7/site-packages (from google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (1.5.0) Requirement already satisfied: ply==3.8 in /home/jenkins/.local/lib/python2.7/site-packages (from google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (3.8) Requirement already satisfied: cachetools>=2.0.0 in /home/jenkins/.local/lib/python2.7/site-packages (from google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.7.0.dev0) (2.1.0) Installing collected packages: numpy, apache-beam Found existing installation: numpy 1.13.3 Not uninstalling numpy at /home/jenkins/.local/lib/python2.7/site-packages, outside environment <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/.beam_env> Can't uninstall 'numpy'. No files were found to uninstall. Found existing installation: apache-beam 2.7.0.dev0 Not uninstalling apache-beam at /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Compressed_TextIOIT_HDFS/src/sdks/python, outside environment <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/.beam_env> Can't uninstall 'apache-beam'. No files were found to uninstall. Running setup.py develop for apache-beam Successfully installed apache-beam numpy-1.15.0 [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins7216896748234028238.sh + .beam_env/bin/python src/sdks/python/setup.py sdist --dist-dir=src/sdks/python/target Traceback (most recent call last): File "src/sdks/python/setup.py", line 46, in <module> PACKAGE_VERSION = get_version() File "src/sdks/python/setup.py", line 41, in get_version exec(open(os.path.normpath('./apache_beam/version.py')).read(), global_names) # pylint: disable=exec-used IOError: [Errno 2] No such file or directory: 'apache_beam/version.py' Build step 'Execute shell' marked build as failure
