See <https://builds.apache.org/job/beam_PostCommit_XVR_Flink/7/display/redirect?page=changes>
Changes: [hsuryawirawan] Update Beam Katas (Java) course on Stepik [hsuryawirawan] Update Beam Katas (Python) course on Stepik [hsuryawirawan] Update the ParDo OneToMany task description to follow latest convention [lukasz.gajowy] Revert "Merge pull request #8561: [BEAM-6627] Add item and byte counters [lukasz.gajowy] Revert "Merge pull request #8400: [BEAM-6627] Added byte and item [hsuryawirawan] Add README files on how to setup the project for both Java and Python [ttanay100] unskip ReifyTest.test_window [hsuryawirawan] Add Windowing Adding Timestamp using ParDo Java Kata [hsuryawirawan] Add Windowing Adding Timestamp using WithTimestamps Java Kata [hsuryawirawan] Add Fixed Time Window Java Kata [hsuryawirawan] Add kata description for the Adding Timestamp katas [hsuryawirawan] Rename 'util' package to 'org.apache.beam.learning.katas.util' [hsuryawirawan] Add Triggers "Event Time Triggers" Java kata [hsuryawirawan] Add "DoFn Additional Parameters" Java kata [hsuryawirawan] Move Built-in IOs Task into a package [hsuryawirawan] Add WithKeys Java kata [hsuryawirawan] Change logging library to use log4j2 and add log4j2.xml in the util [hsuryawirawan] Add "Early Triggers" Java kata [hsuryawirawan] Update WithKeys task description javadoc link to use 'current' [hsuryawirawan] Add "Window Accumulation Mode" Java kata [hsuryawirawan] Add package for "Early Triggers" and "Window Accumulation Mode" [hsuryawirawan] Update study_project.xml [hsuryawirawan] Fix course syllabus on Stepik [hsuryawirawan] Fix the wrong file path for TextIO Read kata [hsuryawirawan] Update the allowed lateness and discarding accumulation mode kata [hsuryawirawan] Update course to Stepik [hsuryawirawan] Reupload Python Katas lessons [zyichi] Fix erros in py sdk io utils and add unit tests [dcavazos] Add Python snippet for Map transform [dcavazos] Add Python snippet for WithTimestamps transform [ttanay100] Add to docstring of advance_watermark_to_infinity [mxm] Fixed typo and removed whitespace [github] Add a timeout to urlopen calls [dcavazos] Add Python snippet for FlatMap transform [aaltay] [Beam-6696] GroupIntoBatches transform for Python SDK (#8914) [gunnar.schulze] [BEAM-7572] ApproximateUnique.Globally and ApproximateUnique.PerKey [lukasz.gajowy] [BEAM-7307] Fix load tests failures due to mistakes in PR 8881 [aryan.naraghi] Implement splitAtFraction for the BigQuery Storage API [github] Drop experimental from python streaming in the doc [markliu] [BEAM-7598] Do not build Python tar file in run_integration_test.sh [dcavazos] Add Python snippet for Regex transform [dcavazos] Add Python snippet for ParDo transform [ttanay100] [BEAM-7437] Add streaming flag to BQ streaming inserts IT test [aaltay] BEAM-7141: add key value timer callback (#8739) [jozsi] Add Jet Runner to the Get Started page [iemejia] [BEAM-7606] Fix JDBC time conversion tests [robertwb] Update portable schema representation and java SchemaTranslation (#8853) [juta.staes] [BEAM-7326] add documentation bigquery data types [juta.staes] [BEAM-5315] improve test coverage bigquery special chars [lcwik] [BEAM-4948, BEAM-6267, BEAM-5559, BEAM-7289] Update the version of guava [aaltay] [BEAM-7475] update wordcount example (#8803) [github] [BEAM-7013] Add HLL doc link to Beam website [kcweaver] [BEAM-7632] Update Python quickstart guide for Flink and Spark [ttanay100] Change default timeout to 5 mins [heejong] [BEAM-7424] Retry HTTP 429 errors from GCS [markliu] [BEAM-4046, BEAM-7527] Fix benchmark with correct Gradle project [zyichi] [BEAM-7586] Add Integration test for python mongodb io [valentyn] Match Python 3 warning message in __init__.py with the one in setup.py. [kedin] Spotless config update to include java files only under src directory [chamikara] [BEAM-7548] fix flaky tests for ApproximateUnique (#8948) [samuelw] [BEAM-7547] Avoid WindmillStateCache cache hits for stale work. [iemejia] [BEAM-7640] Create amazon-web-services2 module and AwsOptions [alireza4263] [BEAM-7545] Adding RowCount to TextTable. [kcweaver] [BEAM-6692] portable Spark: reshuffle translation [chamikara] [BEAM-7548] Fix flaky tests for ApproximateUnique (#8960) [iemejia] [BEAM-7589] Use only one KinesisProducer instance per JVM [iemejia] [BEAM-7589] Make KinesisIOIT compatible with all other ITs [kcweaver] [BEAM-7656] Add sdk-worker-parallelism arg to flink job server shadow [daniel.o.programmer] Update python containers to beam-master-20190605 [dcrhodes] [BEAM-7666] Memory monitor change [dcrhodes] [BEAM-7666] Adds the counter [lcwik] [BEAM-5605] Update Beam Java SDK backlog to track latest changes in Beam [hannahjiang] BEAM-3645 add ParallelBundleProcessor [hannahjiang] BEAM-3645 reflect comments [hannahjiang] BEAM-3645 add changes from review comments [hannahjiang] BEAM-3645 add thread lock when generating process_bundle_id [valentyn] Use Beam's abstraction of pickler instead of dill in coder tests. [github] Tiny typo fix [htyleo] [BEAM-7665] Support TypeDefinition options in beam.Combine() [boyuanz] Add estimate_size() to source_test.LineSource [kamil.wasilewski] [BEAM-7536] Fixed BQ dataset name in collecting Load Tests metrics [kamil.wasilewski] [BEAM-7504] Added top_count parameter [lukasz.gajowy] [BEAM-4420] Allow connecting to zookeeper using external ip [kamil.wasilewski] [BEAM-7504] Create Combine Python Load Test Jenkins job [iemejia] [BEAM-7640] Rename the package name for amazon-web-services2 from aws to [github] Remove unused NEWLINE in TextSink.TextWriter [ehudm] [BEAM-7679] Add randomness to ITs' BQ dataset name [kamil.wasilewski] [BEAM-7535] Created Jenkins job for BQ performance tests [kamil.wasilewski] [BEAM-7535] Delete existing data if the table already exists [iemejia] [BEAM-7682] Fix Combine.GroupedValues javadoc code snippet [lukasz.gajowy] [BEAM-4420] Add KafkaIO integration test pipeline [iemejia] [BEAM-6740] Add PTransformTranslator for Combine.Globally [iemejia] [BEAM-6740] Add extractAcummulatorCoder for Combine.Globally and fix [iemejia] [BEAM-7640] Change tests to use PayloadTranslator instead of unused [iemejia] [BEAM-6740] Refactor to remove duplicated code in CombineTranslation [boyuanz] Fix RestrictionTracker docstring [htyleo] Revise the comments to only mention the TypeDefinition options. [boyuanz] fix lint [hannahjiang] BEAM-3645 add thread lock [cademarkegard] [BEAM-7690] Port WordCountTest off DoFnTester [cyturel] [BEAM-7683] - fix withQueryFn when split is more than 0 [kamil.wasilewski] [BEAM-7550] Reimplement Python ParDo load test according to the proposal [kamil.wasilewski] [BEAM-7503] Added iteration parameter to CoGBK test in Python [kamil.wasilewski] [BEAM-7503] Created CoGBK Python Load Test Jenkins job [kamil.wasilewski] [BEAM-3959] Added a flake F821 test [yanzhi.wyl] [BEAM-7694]Fix error spelling in annotation of SparkTransformOverrides. [ryan] Consider Elasticsearch as one word in camelCase. [juta.staes] [BEAM-7630] add ITs for writing and reading bytes from pubsub [rezarokni] Edit of Looping Timer Blog to fix issue with Timer State [kcweaver] [BEAM-7708] don't expect SQL shell bundled dependencies to be shadowed [github] [BEAM-7709] Re-use node for explicit flattens [boyuanz] Reformat CamelCase function naming style to underscore style for [kcweaver] [BEAM-7710] [website] remove outdated reference to KeyedCombineFn [heejong] [BEAM-7689] make a temporary directory unique for FileBaseSink [boyuanz] fix lint [yanzhi.wyl] update annotation for PTransformMatcher. [33895511+aromanenko-dev] [BEAM-6480] Adds AvroIO sink for generic records. (#9005) [github] [SQL][Doc] fix broken gradle command. [lcwik] Added new example on how to create a custom unbounded streaming source [chambers] Update Python Dataflow runner to patch side input coders on the unified [iemejia] [BEAM-7653] Add PTransformTranslator for Combine.GroupedValues [pachristopher] Update pyarrow version requirement in setup.py [github] Update code comments to improve readability in docs (#9024) [katarzyna.kucharczyk] [BEAM-5994] Fix condition which allows to publish metrics. [lcwik] [BEAM-7668] Add ability to query a pipeline definition from a gRPC [chamikara] [BEAM-7389] Add Python snippet for Partition transform (#8904) [lcwik] Ensure that publishing vendored artifacts checks the contents of the jar [lcwik] [BEAM-4948, BEAM-6267, BEAM-5559, BEAM-7289] Fix shading of vendored [bmvishwas] [BEAM-7621] Null pointer exception when accessing null row fields in [alireza4263] [BEAM-7729] Fixes the bug by checking the value first before parsing it. [boyuanz] Add _get_file_size and UT [b_m.vishwas] [BEAM-7731] Adding helper function to handle if null else pattern for [sniemitz] Done notifications for BigtableIO.Write [github] Label the ParDos with their name [udim] [BEAM-7437] Raise RuntimeError for PY2 in [ankurgoenka] [BEAM-7546] Increasing environment cache to avoid chances of recreating [ankurgoenka] [BEAM-7736] Free worker when work is skipped [yoshiki.obata] [BEAM-7284] enabled to pickle MappingProxyType in order to pickle [github] Some trivial typos [robertwb] [BEAM-7737] Fix microbenchmark scripts compiled check (#9066) [kmj] Fix stream position bug in BigQuery Storage stream source. [lgajowy] [BEAM-6675] Generate JDBC statement and preparedStatementSetter [lukasz.gajowy] [BEAM-4420] Extract method for publishing already collected metrics [iemejia] [BEAM-7715] Mark user facing APIs related to External transforms as [ttanay100] [BEAM-7674] Combine batch and streaming BQ Streaming Insert ITs [udim] [BEAM-7578] add py37 hdfs integration test (#8970) [pabloem] Adding logging and adding one more retry [lukasz.gajowy] [BEAM-4420] Collect & write metrics to BigQuery and console [iemejia] [BEAM-7557] - Migrate DynamoDBIO to AWS SDK for Java 2 [hannahjiang] BEAM-3645 improve test cases [pabloem] [BEAM-7530] Add it test to read None values from BigQuery (#8875) [github] [BEAM-7499] Fixup for tricky Reify testing issue (#9077) [alireza4263] [BEAM-7545] Reordering Beam Joins and check if the produced join is [github] [BEAM-7641] Collect xunit statistics for Py ITs (#8952) [github] [BEAM-4948, BEAM-6267, BEAM-5559, BEAM-7289] Update the version of guava [udim] [BEAM-7484] Metrics collection in BigQuery perf tests (#8766) [zyichi] Add transform_name_mapping pipeline option for python sdk [valentyn] Split Python 3 Postcommits into several jobs [pabloem] Matching on filename, not directory for fileio [pabloem] Lint fixup [pabloem] Lint fixup [lukasz.gajowy] [BEAM-7775] Remove job_PerformanceTests_Spark [lukasz.gajowy] [BEAM-7775] Remove job_PerformanceTests_Dataflow [robertwb] [BEAM-7060] Support translation of native type variables. [lostluck] Fix documentation on iterable coder spec. [github] Add retractions doc to design doc page. [lostluck] Add the ability to set the service account email for dataflow jobs [dcavazos] Skip DoFn params test in Python 2 on Windows [github] Add last_attempted_record_start to python OffsetRangeTracker (#9058) [ehudm] [BEAM-7784] Fixup for Guava upgrade [robertwb] Refactor portable JobService to allow better sharing of code. [robertwb] Simplify known runner parsing code. [robertwb] [BEAM-7722] Add a Python FlinkRunner that fetches and uses released [je.ik] [BEAM-7785] synchronize watermark update with bundle processing [robertwb] Pull out generic java job server helpers from flink. [robertwb] [BEAM-7060] Translate the typing.Iterable type hint. [robertwb] Minor fixes discovered in migration. [robertwb] [BEAM-7060] Automated replace of typehints with typing. [robertwb] Revert incorrect doc on flink runner version. [jeff] [BEAM-5191] Support for BigQuery clustering [zyichi] Check streaming option when validate transform_name_mapping [github] [BEAM-7747] Close the file handle owned by fastavro.write.Writer in [github] Remove 4 empty spaces from PR template that mess up Python postcommit [github] Relax pydot requirements. [valentyn] Default to PiplelineState.UNKNOWN when job state returned from v1beta3 [valentyn] fixup: Address review feedback. [ttanay100] Replace old badges for Python PostCommit test with split ones [yanzhi.wyl] [BEAM-7696]prepare files to stage also in local master of spark runner. [robertwb] Post-rewrite lint fixes. [robertwb] Manual fixes for over-agressive replace. [robertwb] More conservative typing module translation. [kamil.wasilewski] [BEAM-7502] Create ParDo Python Load Test Jenkins job [kamil.wasilewski] [BEAM-7502] Renamed file with Python GBK Load Test job definition [kamil.wasilewski] [BEAM-7502] Reduced number of iterations to 1 in Java ParDo job [alireza4263] [BEAM-7783] Adding BeamTableStatistics. [gleb] Add setFailOnNoMatchingTests(true) [cmach] [BEAM-7794] DynamoDBIOTest is blocking forever [github] Revert "[BEAM-7060] Migrate to native typing types where possible." [github] Update Python 3 entry in Python SDK roadmap. [robertwb] Revert "Revert "[BEAM-7060] Migrate to native typing types where [robertwb] [BEAM-7798] Unit test for BundleBasedRunner type hints. [robertwb] Temporary workaround for [BEAM-7473] (#9023) [robertwb] [BEAM-7798] Typehints fix for bundle based runner. [robertwb] Make SDFBoundedSource wrapper work with dynamic splitting (#8944) [robertwb] [BEAM-7801] Fix wordcount_xlang postcommit. [kamil.wasilewski] [BEAM-7661] Create Python coGBK load test job on Flink [kamil.wasilewski] [BEAM-7661] Generate the same number of hot keys from both sources [gleb] [BEAM-6783] byte[] breaks in BeamSQL codegen [gleb] [SQL] Annotate BuiltinStringFunctions with Strict [relax] Merge pull request #8943: Schema conversion cleanup [lostluck] [BEAM-7781] Update from Go Version 1.10 to 1.12 [lostluck] [BEAM-7781] Use correct Go gradle tasks [lostluck] [BEAM-7781] Update to latest GoGradle [lostluck] [BEAM-7726] Add tests for DataSource protocol [lostluck] [BEAM-7726] Refactor to prepare for state backing. [lostluck] [BEAM-7726] Rename some SideInput code to State [lostluck] [BEAM-7726] Handle State Backed iterables [pabloem] [BEAM-7680] Removed flaky assertion and separated unit tests in [ehudm] [BEAM-7650] Add docker system prune to inventory job [iemejia] [BEAM-7794] Ignore blocking DynamoDBIOtest class [valentyn] Configure console output collection for Python 3.6, 3.7 postcommit [aaltay] [BEAM-7797] Move upload_graph handling logic before template [rohde.samuel] Update the Dataflow API version to 0607 ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on apache-beam-jenkins-7 (beam) in workspace <https://builds.apache.org/job/beam_PostCommit_XVR_Flink/ws/> No credentials specified Cloning the remote Git repository Cloning repository https://github.com/apache/beam.git > git init <https://builds.apache.org/job/beam_PostCommit_XVR_Flink/ws/src> # > timeout=10 Fetching upstream changes from https://github.com/apache/beam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/apache/beam.git > +refs/heads/*:refs/remotes/origin/* > git config remote.origin.url https://github.com/apache/beam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # > timeout=10 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10 Fetching upstream changes from https://github.com/apache/beam.git > git fetch --tags --progress https://github.com/apache/beam.git > +refs/heads/*:refs/remotes/origin/* > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/* > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 8050915fe5c58a6739378a988df03d5dd3f77ec7 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 8050915fe5c58a6739378a988df03d5dd3f77ec7 Commit message: "Merge pull request #9143: [BEAM-7714] Configure console output collection for Python 3.6, 3.7 postcommit suites." > git rev-list --no-walk 0c04a024a0764682d46dad176ce2d98814872c7d # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 No emails were triggered. [EnvInject] - Executing scripts and injecting environment variables after the SCM step. [EnvInject] - Injecting as environment variables the properties content SPARK_LOCAL_IP=127.0.0.1 [EnvInject] - Variables injected successfully. [Gradle] - Launching build. [src] $ <https://builds.apache.org/job/beam_PostCommit_XVR_Flink/ws/src/gradlew> --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx4g :beam-runners-flink_2.11-job-server:validatesCrossLanguageRunner ############################### WARNING #################################### ## ## ## Deprecation Warning: Gradle command altered. ## ## ## ## You are still using deprecated project namings. Instead of using e.g. ## ## ## ## :beam-sdks-java-core ## ## ## ## you should now use ## ## ## ## :sdks:java:core ## ## ## ## no to reference projects and their tasks. See project-mappings file. ## ## ## ## This compatibility layer will be removed soon. ## ## ## ############################### WARNING #################################### Changed command to gradlew --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx4g :runners:flink:1.5:job-server:validatesCrossLanguageRunner Starting a Gradle Daemon, 1 busy Daemon could not be reused, use --status for details > Task :buildSrc:compileJava NO-SOURCE > Task :buildSrc:compileGroovy FROM-CACHE > Task :buildSrc:pluginDescriptors > Task :buildSrc:processResources > Task :buildSrc:classes > Task :buildSrc:jar > Task :buildSrc:assemble > Task :buildSrc:spotlessGroovy > Task :buildSrc:spotlessGroovyCheck > Task :buildSrc:spotlessGroovyGradle > Task :buildSrc:spotlessGroovyGradleCheck > Task :buildSrc:spotlessCheck > Task :buildSrc:pluginUnderTestMetadata > Task :buildSrc:compileTestJava NO-SOURCE > Task :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test NO-SOURCE > Task :buildSrc:validateTaskProperties FROM-CACHE > Task :buildSrc:check > Task :buildSrc:build Configuration on demand is an incubating feature. FAILURE: Build failed with an exception. * What went wrong: Task 'validatesCrossLanguageRunner' not found in project ':runners:flink:1.5:job-server'. * Try: Run gradlew_orig tasks to get a list of available tasks. Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 10s Publishing build scan... https://gradle.com/s/vyjihz2ef5adg Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
