See <https://builds.apache.org/job/beam_PostCommit_Python_Chicago_Taxi_Dataflow/7/display/redirect?page=changes>
Changes: [hsuryawirawan] Update Beam Katas (Java) course on Stepik [hsuryawirawan] Update Beam Katas (Python) course on Stepik [hsuryawirawan] Update the ParDo OneToMany task description to follow latest convention [ttanay100] unskip ReifyTest.test_window [hsuryawirawan] Add Windowing Adding Timestamp using ParDo Java Kata [hsuryawirawan] Add Windowing Adding Timestamp using WithTimestamps Java Kata [hsuryawirawan] Add Fixed Time Window Java Kata [hsuryawirawan] Add kata description for the Adding Timestamp katas [hsuryawirawan] Rename 'util' package to 'org.apache.beam.learning.katas.util' [hsuryawirawan] Add Triggers "Event Time Triggers" Java kata [hsuryawirawan] Add "DoFn Additional Parameters" Java kata [hsuryawirawan] Move Built-in IOs Task into a package [hsuryawirawan] Add WithKeys Java kata [hsuryawirawan] Change logging library to use log4j2 and add log4j2.xml in the util [hsuryawirawan] Add "Early Triggers" Java kata [hsuryawirawan] Update WithKeys task description javadoc link to use 'current' [hsuryawirawan] Add "Window Accumulation Mode" Java kata [hsuryawirawan] Add package for "Early Triggers" and "Window Accumulation Mode" [hsuryawirawan] Update study_project.xml [hsuryawirawan] Fix course syllabus on Stepik [hsuryawirawan] Fix the wrong file path for TextIO Read kata [hsuryawirawan] Update the allowed lateness and discarding accumulation mode kata [hsuryawirawan] Update course to Stepik [hsuryawirawan] Reupload Python Katas lessons [dcavazos] Add Python snippet for WithTimestamps transform [ehudm] [BEAM-7608] Honor DATASTORE_EMULATOR_HOST env var [dcavazos] Add Python snippet for Regex transform [ttanay100] [BEAM-7437] Add streaming flag to BQ streaming inserts IT test [jozsi] Add Jet Runner to the Get Started page [juta.staes] [BEAM-5315] improve test coverage bigquery special chars [kcweaver] [BEAM-7632] Update Python quickstart guide for Flink and Spark [ttanay100] Change default timeout to 5 mins [alireza4263] [BEAM-7545] Adding RowCount to TextTable. [kcweaver] [BEAM-7656] Add sdk-worker-parallelism arg to flink job server shadow [daniel.o.programmer] Update python containers to beam-master-20190605 [dcrhodes] [BEAM-7666] Memory monitor change [dcrhodes] [BEAM-7666] Adds the counter [lcwik] [BEAM-5605] Update Beam Java SDK backlog to track latest changes in Beam [hannahjiang] BEAM-3645 add ParallelBundleProcessor [hannahjiang] BEAM-3645 reflect comments [hannahjiang] BEAM-3645 add changes from review comments [hannahjiang] BEAM-3645 add thread lock when generating process_bundle_id [valentyn] Use Beam's abstraction of pickler instead of dill in coder tests. [github] Tiny typo fix [htyleo] [BEAM-7665] Support TypeDefinition options in beam.Combine() [boyuanz] Add estimate_size() to source_test.LineSource [kamil.wasilewski] [BEAM-7536] Fixed BQ dataset name in collecting Load Tests metrics [kamil.wasilewski] [BEAM-7504] Added top_count parameter [lukasz.gajowy] [BEAM-4420] Allow connecting to zookeeper using external ip [kamil.wasilewski] [BEAM-7504] Create Combine Python Load Test Jenkins job [iemejia] [BEAM-7640] Rename the package name for amazon-web-services2 from aws to [github] Remove unused NEWLINE in TextSink.TextWriter [ehudm] [BEAM-7679] Add randomness to ITs' BQ dataset name [kamil.wasilewski] [BEAM-7535] Created Jenkins job for BQ performance tests [kamil.wasilewski] [BEAM-7535] Delete existing data if the table already exists [iemejia] [BEAM-7682] Fix Combine.GroupedValues javadoc code snippet [lukasz.gajowy] [BEAM-4420] Add KafkaIO integration test pipeline [iemejia] [BEAM-6740] Add PTransformTranslator for Combine.Globally [iemejia] [BEAM-6740] Add extractAcummulatorCoder for Combine.Globally and fix [iemejia] [BEAM-7640] Change tests to use PayloadTranslator instead of unused [iemejia] [BEAM-6740] Refactor to remove duplicated code in CombineTranslation [boyuanz] Fix RestrictionTracker docstring [htyleo] Revise the comments to only mention the TypeDefinition options. [boyuanz] fix lint [hannahjiang] BEAM-3645 add thread lock [cademarkegard] [BEAM-7690] Port WordCountTest off DoFnTester [cyturel] [BEAM-7683] - fix withQueryFn when split is more than 0 [kamil.wasilewski] [BEAM-7550] Reimplement Python ParDo load test according to the proposal [kamil.wasilewski] [BEAM-7503] Added iteration parameter to CoGBK test in Python [kamil.wasilewski] [BEAM-7503] Created CoGBK Python Load Test Jenkins job [kamil.wasilewski] [BEAM-3959] Added a flake F821 test [yanzhi.wyl] [BEAM-7694]Fix error spelling in annotation of SparkTransformOverrides. [ryan] Consider Elasticsearch as one word in camelCase. [juta.staes] [BEAM-7630] add ITs for writing and reading bytes from pubsub [rezarokni] Edit of Looping Timer Blog to fix issue with Timer State [kcweaver] [BEAM-7708] don't expect SQL shell bundled dependencies to be shadowed [github] [BEAM-7709] Re-use node for explicit flattens [boyuanz] Reformat CamelCase function naming style to underscore style for [kcweaver] [BEAM-7710] [website] remove outdated reference to KeyedCombineFn [heejong] [BEAM-7689] make a temporary directory unique for FileBaseSink [boyuanz] fix lint [yanzhi.wyl] update annotation for PTransformMatcher. [33895511+aromanenko-dev] [BEAM-6480] Adds AvroIO sink for generic records. (#9005) [github] [SQL][Doc] fix broken gradle command. [lcwik] Added new example on how to create a custom unbounded streaming source [chambers] Update Python Dataflow runner to patch side input coders on the unified [iemejia] [BEAM-7653] Add PTransformTranslator for Combine.GroupedValues [pachristopher] Update pyarrow version requirement in setup.py [github] Update code comments to improve readability in docs (#9024) [katarzyna.kucharczyk] [BEAM-5994] Fix condition which allows to publish metrics. [lcwik] [BEAM-7668] Add ability to query a pipeline definition from a gRPC [chamikara] [BEAM-7389] Add Python snippet for Partition transform (#8904) [lcwik] Ensure that publishing vendored artifacts checks the contents of the jar [lcwik] [BEAM-4948, BEAM-6267, BEAM-5559, BEAM-7289] Fix shading of vendored [bmvishwas] [BEAM-7621] Null pointer exception when accessing null row fields in [alireza4263] [BEAM-7729] Fixes the bug by checking the value first before parsing it. [boyuanz] Add _get_file_size and UT [b_m.vishwas] [BEAM-7731] Adding helper function to handle if null else pattern for [sniemitz] Done notifications for BigtableIO.Write [github] Label the ParDos with their name [udim] [BEAM-7437] Raise RuntimeError for PY2 in [ankurgoenka] [BEAM-7546] Increasing environment cache to avoid chances of recreating [ankurgoenka] [BEAM-7736] Free worker when work is skipped [yoshiki.obata] [BEAM-7284] enabled to pickle MappingProxyType in order to pickle [github] Some trivial typos [robertwb] [BEAM-7737] Fix microbenchmark scripts compiled check (#9066) [ehudm] [BEAM-2264] Reuse GCP credentials in GCS calls. [kmj] Fix stream position bug in BigQuery Storage stream source. [lgajowy] [BEAM-6675] Generate JDBC statement and preparedStatementSetter [lukasz.gajowy] [BEAM-4420] Extract method for publishing already collected metrics [iemejia] [BEAM-7715] Mark user facing APIs related to External transforms as [ttanay100] [BEAM-7674] Combine batch and streaming BQ Streaming Insert ITs [udim] [BEAM-7578] add py37 hdfs integration test (#8970) [pabloem] Adding logging and adding one more retry [ryan] [BEAM-4181] Add readFiles transform to TfRecordIO. [lukasz.gajowy] [BEAM-4420] Collect & write metrics to BigQuery and console [iemejia] [BEAM-7557] - Migrate DynamoDBIO to AWS SDK for Java 2 [hannahjiang] BEAM-3645 improve test cases [pabloem] [BEAM-7530] Add it test to read None values from BigQuery (#8875) [github] [BEAM-7499] Fixup for tricky Reify testing issue (#9077) [alireza4263] [BEAM-7545] Reordering Beam Joins and check if the produced join is [github] [BEAM-7641] Collect xunit statistics for Py ITs (#8952) [github] [BEAM-4948, BEAM-6267, BEAM-5559, BEAM-7289] Update the version of guava [udim] [BEAM-7484] Metrics collection in BigQuery perf tests (#8766) [zyichi] Add transform_name_mapping pipeline option for python sdk [valentyn] Split Python 3 Postcommits into several jobs [pabloem] Matching on filename, not directory for fileio [pabloem] Lint fixup [pabloem] Lint fixup [lukasz.gajowy] [BEAM-7775] Remove job_PerformanceTests_Spark [lukasz.gajowy] [BEAM-7775] Remove job_PerformanceTests_Dataflow [robertwb] [BEAM-7060] Support translation of native type variables. [lostluck] Fix documentation on iterable coder spec. [github] Add retractions doc to design doc page. [lostluck] Add the ability to set the service account email for dataflow jobs [dcavazos] Skip DoFn params test in Python 2 on Windows [github] Add last_attempted_record_start to python OffsetRangeTracker (#9058) [ehudm] [BEAM-7784] Fixup for Guava upgrade [robertwb] Refactor portable JobService to allow better sharing of code. [robertwb] Simplify known runner parsing code. [robertwb] [BEAM-7722] Add a Python FlinkRunner that fetches and uses released [je.ik] [BEAM-7785] synchronize watermark update with bundle processing [robertwb] Pull out generic java job server helpers from flink. [robertwb] [BEAM-7060] Translate the typing.Iterable type hint. [robertwb] Minor fixes discovered in migration. [robertwb] [BEAM-7060] Automated replace of typehints with typing. [robertwb] Revert incorrect doc on flink runner version. [jeff] [BEAM-5191] Support for BigQuery clustering [zyichi] Check streaming option when validate transform_name_mapping [github] [BEAM-7747] Close the file handle owned by fastavro.write.Writer in [github] Remove 4 empty spaces from PR template that mess up Python postcommit [github] Relax pydot requirements. [valentyn] Default to PiplelineState.UNKNOWN when job state returned from v1beta3 [valentyn] fixup: Address review feedback. [ttanay100] Replace old badges for Python PostCommit test with split ones [yanzhi.wyl] [BEAM-7696]prepare files to stage also in local master of spark runner. [robertwb] Post-rewrite lint fixes. [robertwb] Manual fixes for over-agressive replace. [robertwb] More conservative typing module translation. [kamil.wasilewski] [BEAM-7502] Create ParDo Python Load Test Jenkins job [kamil.wasilewski] [BEAM-7502] Renamed file with Python GBK Load Test job definition [kamil.wasilewski] [BEAM-7502] Reduced number of iterations to 1 in Java ParDo job [alireza4263] [BEAM-7783] Adding BeamTableStatistics. [gleb] Add setFailOnNoMatchingTests(true) [cmach] [BEAM-7794] DynamoDBIOTest is blocking forever [github] Revert "[BEAM-7060] Migrate to native typing types where possible." [github] Update Python 3 entry in Python SDK roadmap. [robertwb] Revert "Revert "[BEAM-7060] Migrate to native typing types where [robertwb] [BEAM-7798] Unit test for BundleBasedRunner type hints. [robertwb] Temporary workaround for [BEAM-7473] (#9023) [robertwb] [BEAM-7798] Typehints fix for bundle based runner. [robertwb] Make SDFBoundedSource wrapper work with dynamic splitting (#8944) [robertwb] [BEAM-7801] Fix wordcount_xlang postcommit. [kamil.wasilewski] [BEAM-7662] Create Python Combine load test job on Flink [kamil.wasilewski] [BEAM-7661] Create Python coGBK load test job on Flink [kamil.wasilewski] [BEAM-7661] Generate the same number of hot keys from both sources [gleb] [BEAM-6783] byte[] breaks in BeamSQL codegen [gleb] [SQL] Annotate BuiltinStringFunctions with Strict [relax] Merge pull request #8943: Schema conversion cleanup [lostluck] [BEAM-7781] Update from Go Version 1.10 to 1.12 [lostluck] [BEAM-7781] Use correct Go gradle tasks [lostluck] [BEAM-7781] Update to latest GoGradle [aryan.naraghi] Add progress reporting to the BigQuery source [lostluck] [BEAM-7726] Add tests for DataSource protocol [lostluck] [BEAM-7726] Refactor to prepare for state backing. [lostluck] [BEAM-7726] Rename some SideInput code to State [lostluck] [BEAM-7726] Handle State Backed iterables [pabloem] [BEAM-7680] Removed flaky assertion and separated unit tests in [ehudm] [BEAM-7650] Add docker system prune to inventory job [iemejia] [BEAM-7794] Ignore blocking DynamoDBIOtest class [valentyn] Configure console output collection for Python 3.6, 3.7 postcommit [aaltay] [BEAM-7797] Move upload_graph handling logic before template [rohde.samuel] Update the Dataflow API version to 0607 [aaltay] Weaken overly-strict microbenchmark check_complied test. (#9100) [ehudm] [BEAM-7814] Wait for BQ query results in tests [36090911+boyuanzz] Revert "[BEAM-7785] synchronize watermark update with bundle processing" [iemejia] [BEAM-7807] Forbid TestNG imports on Beam classes [iemejia] [BEAM-7807] Update Beam tests to not import TestNG related classes [rohde.samuel] Add hot key detection protos to Windmill [alireza4263] [BEAM-7823] Implementing a Custom NonCumulativeCost Handler [altay] Upgrade google-cloud-core and google-cloud-bigquery dependencies [github] Update nexmark.md [filiperegadas] [BEAM-7830]: make spark default duration consistent with PipelineResult [kenn] [Java] remove unused variable in DirectMetricsTest.java [alireza4263] [BEAM-7783] BeamIOSourceRel using custom BeamStatistics [ehudm] [BEAM-7641] Add xunit results to run_validatescontainer.sh [kenn] [BEAM-7755] adding support for nested rows and arrays in BigQuery to [filiperegadas] fixup! [BEAM-7830]: make spark default duration consistent with [pabloem] [BEAM-6611] BigQuery file loads in Streaming for Python SDK (#8871) [mxm] [BEAM-7366] Restore Spotless check for Flink Runner [github] Nitpicky grammar fix in programming guide [jozsi] Set job name in Jet Runner [b_m.vishwas] [BEAM-7623] Add support to select MAP with Row as values in Beam SQL [je.ik] [BEAM-7574] Use Map-based window combining [je.ik] [BEAM-7574] use Merging and NonMerging accumulating strategies [je.ik] [BEAM-7574] unify Combine.perKey and Combine.globally [je.ik] [BEAM-7574] Explode windows to key when needed [je.ik] [BEAM-7574] Optimize GC performance, finalize implementation [pabloem] [BEAM-7814] Make BigqueryMatcher wait for results. (#9154) [12602502+Ardagan] Provision datasources and dashboards from config. [pabloem] [BEAM-7700] Java transform catalog (#9129) [ryan] Add arbitrary value matchers to MetricsResultMatchers. [ryan] [BEAM-6883] Improve long-running spark streaming test. [github] Merge pull request #9183: [BEAM-7675] Unify Gradle test configuration [iemejia] [BEAM-7837] Add adapter class for API incompatible changes in Spark 3 [chamikara] Sets a default environment for Dataflow runner [github] [BEAM-6202] Change retry policy for dataflow's get_job (#9182) [dcavazos] Fixed type on singleton side inputs sample [altay] Change retry policy for dataflow's list_messages [github] [BEAM-6877] trivial_inference: make remaining tests pass (#8893) [pabloem] [BEAM-7820] Add hot key detection to Batch Dataflow Worker (#9186) [amaliujia] [BEAM-7728] [SQL] Support ParquetTable (#9054) [github] [BEAM-4948] Switches dataflow runner to use a container image with [robertwb] [BEAM-3645] Add multiplexing for python FnApiRunner (#8979) [github] [BEAM-7840] Provide MapTuple and FlatMapTuple for Python 3 users. [gleb] [BEAM-6783] byte[] breaks in BeamSQL codegen ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on apache-beam-jenkins-6 (beam) in workspace <https://builds.apache.org/job/beam_PostCommit_Python_Chicago_Taxi_Dataflow/ws/> No credentials specified Cloning the remote Git repository Cloning repository https://github.com/apache/beam.git > git init > <https://builds.apache.org/job/beam_PostCommit_Python_Chicago_Taxi_Dataflow/ws/src> > # timeout=10 Fetching upstream changes from https://github.com/apache/beam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/apache/beam.git > +refs/heads/*:refs/remotes/origin/* > git config remote.origin.url https://github.com/apache/beam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # > timeout=10 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10 Fetching upstream changes from https://github.com/apache/beam.git > git fetch --tags --progress https://github.com/apache/beam.git > +refs/heads/*:refs/remotes/origin/* > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/* > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision afeb9358ec1fb584d304959c982f99f0212273a1 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f afeb9358ec1fb584d304959c982f99f0212273a1 Commit message: "Merge pull request #9169: [BEAM-6783] byte[] breaks in BeamSQL codegen" > git rev-list --no-walk 0d742936c7a57675f796c385c2186d57507ec0f6 # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 No emails were triggered. [EnvInject] - Executing scripts and injecting environment variables after the SCM step. [EnvInject] - Injecting as environment variables the properties content SPARK_LOCAL_IP=127.0.0.1 [EnvInject] - Variables injected successfully. [Gradle] - Launching build. [src] $ <https://builds.apache.org/job/beam_PostCommit_Python_Chicago_Taxi_Dataflow/ws/src/gradlew> -PgcsRoot=gs://temp-storage-for-perf-tests/chicago-taxi -Prunner=DataflowRunner :sdks:python:test-suites:dataflow:py2:dataflowChicagoTaxiExample Starting a Gradle Daemon (subsequent builds will be faster) > Task :buildSrc:compileJava NO-SOURCE > Task :buildSrc:compileGroovy FROM-CACHE > Task :buildSrc:pluginDescriptors > Task :buildSrc:processResources > Task :buildSrc:classes > Task :buildSrc:jar > Task :buildSrc:assemble > Task :buildSrc:spotlessGroovy > Task :buildSrc:spotlessGroovyCheck > Task :buildSrc:spotlessGroovyGradle > Task :buildSrc:spotlessGroovyGradleCheck > Task :buildSrc:spotlessCheck > Task :buildSrc:pluginUnderTestMetadata > Task :buildSrc:compileTestJava NO-SOURCE > Task :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test NO-SOURCE > Task :buildSrc:validateTaskProperties FROM-CACHE > Task :buildSrc:check > Task :buildSrc:build Configuration on demand is an incubating feature. FAILURE: Build failed with an exception. * What went wrong: Task 'dataflowChicagoTaxiExample' not found in project ':sdks:python:test-suites:dataflow:py2'. * Try: Run gradlew_orig tasks to get a list of available tasks. Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 10s Publishing build scan... https://gradle.com/s/uvzdbo5v5pgr6 Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
