See <https://builds.apache.org/job/beam_PostCommit_XVR_Flink/2/display/redirect?page=changes>
Changes: [laraschmidt] Test [laraschmidt] Adding synthetic pipeline changes. [ankurgoenka] Update Beam site to use 2.13.0 as the latest release [ankurgoenka] Mark old release archived [lukasz.gajowy] [BEAM-7368] Add method to prepare SDK harness image for jenkins jobs [lukasz.gajowy] [BEAM-7368] Add method to prepare Flink Job server's image for jenkins [lukasz.gajowy] [BEAM-7368] Add cluster Flink creation and teardown steps [lukasz.gajowy] [BEAM-7368] Add GBK load test for Python + Flink [lukasz.gajowy] [BEAM-7368] Specify a cron job for the load test [bhulette] Add support for full unboxing in Convert [laraschmidt] Changes to restriction provider. [lukasz.gajowy] [BEAM-7368] Apply code review suggestions [laraschmidt] Fixing some indents.. [laraschmidt] Fixing wrapping indent [laraschmidt] Fixing more indents [laraschmidt] Some more [laraschmidt] Some more [laraschmidt] And almost done [laraschmidt] Fixing some more things. [laraschmidt] Fixing OffsetRange, some more whitespace issues, and lint issues. [lukasz.gajowy] [BEAM-7368] Use post build script in Jenkins [ttanay100] [BEAM-7388] Reify PTransform for Python SDK [laraschmidt] Review changes. [laraschmidt] FIx for linter [aryan.naraghi] Performs some simple clean-up for the tornadoes demo. [lukecwik] [BEAM-6199] Remove outdated Combine URNs [kamil.wasilewski] [BEAM-7362] Added a performance test for BigQueryIO write [hsuryawirawan] Add side input and side output java katas [hsuryawirawan] Add side input and side output python katas [hsuryawirawan] Modify the TestPipeline declaration to be 'final transient' [hsuryawirawan] Add Create.of transform names in CoGroupByKey kata [hsuryawirawan] Improve the task description of ParDo OneToMany to describe [mxm] [BEAM-2943] Check existence of to-be-staged files in PipelineResources [github] Fixing a typo [ttanay100] [BEAM-7437] BQ integration test for streaming inserts in streaming [juta.staes] [BEAM-7351] increment ack deadline for flaky pubsub test [mxm] Update Flink artifact name in archetypes [mxm] [BEAM-7487] Avoid throwing an exception in FlinkRunnerResult#cancel [aromanenko.dev] [BEAM-7357] Remove stream existance check [lcwik] [BEAM-3606] Remove guava-testlib from sdks/java/core main dependency set [lcwik] [BEAM-3606] Remove unused guava_testlib from dependency sets. [iemejia] [BEAM-7175] Add Spark Portable ValidatesRunner Batch postcommit test [lcwik] [BEAM-6620, BEAM-3606] Stop shading by default. [lcwik] fixup! Fix sdks/java/container Kafka dependency [lcwik] fixup! Fixup JDBC gradle build to use shaded configuration for shaded [github] [BEAM-2857] Implementing WriteToFiles transform for fileio (Python) [mxm] [BEAM-5664] Improve cancellation of portable pipelines [mxm] Resolve Flink artifact name dynamically [ehudm] [BEAM-7432] Set input and output files for sdist [lcwik] Fix Spark ValidatesRunner since the shadowTest classpath before didn't [kenn] TfIdfTest should not be marked ValidatesRunner [kcweaver] [BEAM-7492] [BEAM-7491] add Spark runner to Go SDK [github] Adding a Sequence Generator for SQL, and posting about SQL data sources [kcweaver] add badges for Go-on-Spark test [lcwik] fixup! Fix publishing to use the correct configuration property, also [lcwik] Update python containers to beam-master-20190605 [rezarokni] Increase font and wording for click to expand link. [hsuryawirawan] Update offset for some tasks [laraschmidt] Some small fixes and splitting time between elements. [kenn] Use artifactId as archivesBaseName for vendor modules [aaltay] [BEAM-6356] Add the option to use TFRecord to store cache results using [kenn] Add explicit archivesBaseName to applyJavaNature and [pabloem] Importing hamcrest dependency at matcher runtime [lcwik] [BEAM-7499] Skip test since it doesn't work on DirectRunner. [zyichi] Fix python io developer guide document errors [kenn] Fix archivesBaseName method [aaltay] Fix code block formatting in docstring. (#8756) [robinyqiu] Remove the unnecessary enable_health_checker flag [chamikara] [BEAM-6952] fixes concatenated compressed files bug with python sdk [chamikara] Fixes the version number for concat compressed files fix [peter.back] Using the default partitioner based on parallelism [robertwb] Lookup split requests on transform id rather than constant string. [robertwb] Make the Go SDK more robust with respect to requested split points. [katarzyna.kucharczyk] [BEAM-5995] Fix: cron job naming [lukasz.gajowy] [BEAM-7368] Allow defining number of slots per worker. Set to 1 by [lukasz.gajowy] [BEAM-7368] Added parameters that allow using different flink versions [lukasz.gajowy] [BEAM-7368] increase worker count to be equal to Flink's parallelism [sniemitz] [BEAM-7507] Use TextFormat to escape workIds to a string rather than [ankurgoenka] Beam 2.13.0 Blog Post [melissapa] [BEAM-7449] Add pages documenting common Beam pipeline patterns (with [kenn] Add work-in-progress drain and checkpoint to runner comparison [pabloem] Skip flaky test [relax] Merge pull request #8620: [BEAM-6673] Add schema support to BigQuery [lcwik] [BEAM-7470] Update proto and all SDKs to make the logical data stream [zyichi] Fix CountingSource example stop_position bug [lcwik] [BEAM-7512] Replace deprecated self.assertEquals with self.assertEqual [alireza4263] [BEAM-7511] Fixes the bug in KafkaTable Initialization. [lcwik] [BEAM-7512] Fix up some other deprecated usages of self.assertEquals [chamikara] [BEAM-6821] FileBasedSink improper paths (#8054) [lcwik] [BEAM-6620] Remove references to DEFAULT_SHADOW_CLOSURE since it does [kcweaver] [BEAM-6865] refactor InMemoryBagUserStateFactory to standalone class [hsuryawirawan] Update the task descriptions: formatting, conventions, and styles [hsuryawirawan] Update Beam version to v2.13.0 [pabloem] BEAM-7475: Update programming guide with stateful and timer example [hsuryawirawan] Add BinaryCombineFn Lambda java kata [hsuryawirawan] Add packages for all Java katas [hsuryawirawan] Add composite transform Java kata [hsuryawirawan] Add composite transform Python kata [hsuryawirawan] Add 'public' modifier to early Task classes [hsuryawirawan] Rename Tests classes to TaskTest [hsuryawirawan] Add branching Java kata [hsuryawirawan] Add branching Python kata [hsuryawirawan] Add TextIO Read Java kata [hsuryawirawan] Add the Kata descriptions for Branching and TextIO Read [hsuryawirawan] Add TextIO ReadFromText Python kata [hsuryawirawan] Add Kata description from Branching Python kata [hsuryawirawan] Add Built-in IOs Java kata [hsuryawirawan] Add Built-in IOs Python kata [mxm] [website] Correct date in release blog post [lukasz.gajowy] [BEAM-7368] Bugfix: create proper amount of VMs for Flink Dataproc [lukasz.gajowy] [BEAM-7368] Remove deps to external spark config in Flink init action [alireza4263] [BEAM-7511] Fixes the bug in KafkaTable Initialization, returned the [github] fix minor typo [alireza4263] [BEAM-7511] Fixes the bug in KafkaTable Initialization, returned the [github] Adding Python samples to the Stateful Processing post. (#8363) [github] Update Python dependencies page for 2.13.0 [pabloem] [BEAM-6693] replace mmh3 with default hash function (#8799) [hsuryawirawan] Add missing copyright for Python kata "Built-in IOs" test file [hsuryawirawan] Add rat exclusion for Katas IO txt files [hsuryawirawan] Removed unused entry in Java kata study_project.xml [charith.ellawala] [BEAM-7526] Fix toBeamValue logic in BigQueryUtils [robertwb] Publish SDF progress information on root node. [juta.staes] [BEAM-6769] allow read None bytes [je.ik] [BEAM-7529] Add Sums.ofFloats() and Sums.ofDoubles() [tvanderlippe] Use when/then style for Mockito stubs [pabloem] [BEAM-5995] add Jenkins job with GBK Python load tests (#8151) [kcweaver] [BEAM-7044] portable Spark: support stateful dofns [neville] [BEAM-7533] Fix CoderRegistry for Float [github] Add aliases for machine_type and disk_type flags to match existing [ehudm] [BEAM-7531] Remove and disallow KMS key setting [ehudm] Reduce gradle build verbosity [kenn] Merge #8686: [BEAM-7399] Blog post on looping timers [mxm] [website] Add artifact staging document to design documents [aromanenko.dev] [BEAM-7445] Add search field to Beam website [github] [BEAM-7510] Fixing fileio tests checking JSON serialization (flaky) [github] Remove Python batch support for Pub/Sub from docs [kcweaver] [BEAM-7221] portable Spark: timers [angoenka] [BEAM-7407] Adds portable wordcount test for Python 3 with Flink runner. [rezarokni] Fixed spaces for author in looping blog. Added twitter handle for [github] Use code.jquery.com for jQuery CDN [kanterov] Merge pull request #8552: [BEAM-7268] make sorter extension Hadoop-free [github] Add new proposal to design docs [github] Documenting and refactoring fn api runner code (#8659) [valentyn] Read YAML as binary stream in standard_coders_test. [lcwik] [BEAM-7541] provide runner & filesystem deps for IOIT in runtime [github] [BEAM-5315] Fix Py3 incompatibility in tfrecordio_test. [github] Adding helper functions, and documentation to fn_api_runner classes. [melissapa] [BEAM-7033] Updates for reading with query using BigQuery Storage API [kcweaver] [BEAM-7553] add portability to Spark webpage [iemejia] [BEAM-7043] Add DynamoDBIO [thw] [BEAM-7126] Fix StateRequestHandler type variables to allow for proper [kamil.wasilewski] [BEAM-7402] Added a performance test for BigQuery IO read [mxm] [BEAM-7551] Checkpoint Flink's ImpulseSourceFunction [david.moravek] [BEAM-7561] HdfsFileSystems.match can match a directory. [kanterov] Merge pull request #8858: [BEAM-7542] Fix faulty cast in BigQueryUtils [robertwb] Remove unused **kwargs arguments for various transforms. [github] Adding a Link to The Doc for Cost Estimation [ehudm] [BEAM-6877] Fix trivial_inference for Python 3.x [millsd] Use StateTags.ID_EQUIVALENCE when using comparing StateTags [ehudm] Address review comments [github] Add link to type hints design doc [iemejia] [BEAM-7450] Support unbounded reads with HCatalogIO [valentyn] Add Python 3.6, 3.7 to the list of supported version classifiers. [chamikara] Fixes filesystem_test for Windows. [github] use https [mxm] [BEAM-7144] Fix for rescaling problem on Flink >= 1.6 [je.ik] [BEAM-7543] ReduceByKey.combineBy must accept BinaryFunction<V, V, V> [alireza4263] [BEAM-7513] Implements row estimation for BigQuery. [aaltay] [BEAM-6777] Let HealthzServlet respond actual health information of SDK [amaliujia] support Datetime value conversion. [melissapa] Merge pull request #8836: Add Beam Katas to the website's "Learning [amaliujia] [BEAM-7461] disalbe flaky tests due to flaky [github] Revert "[BEAM-7513] Adding Row Count for Bigquery Table" [ehudm] [BEAM-7579] Use bucket with default key in ITs [github] [BEAM-7467] Add dependency classifier to published pom (#8868) [github] Fixing file naming for windows (#8870) [zyichi] [BEAM-5148] Add MongoDB IO to python sdk [ehudm] Allow more time for IT pipeline to run [ehudm] Address review comments [sniemitz] [BEAM-7507] Avoid more String.format and ByteString.toStringUtf8 in hot [relax] [BEAM-6674] Add schema support to JdbcIO read (#8725) [david.moravek] [BEAM-7561] Validate correct directory matching in LocalFileSystemsTest [owenzhang1990] [BEAM-7467] Do not run gearpump pipeline remotely by default [je.ik] [BEAM-7543] deprecate old #combineBy(Sums.ofLongs(), [dcavazos] Add common code infrastructure for element-wise snippets [alireza4263] [BEAM-7513] Adding RowCount to BigQueryTable. [aaltay] [BEAM-6955] Use base version component of Beam Python SDK version when [kedin] Moving to 2.15.0-SNAPSHOT on master branch. ------------------------------------------ Started by GitHub push by udim [EnvInject] - Loading node environment variables. Building remotely on apache-beam-jenkins-2 (beam) in workspace <https://builds.apache.org/job/beam_PostCommit_XVR_Flink/ws/> No credentials specified Cloning the remote Git repository Cloning repository https://github.com/apache/beam.git > git init <https://builds.apache.org/job/beam_PostCommit_XVR_Flink/ws/src> # > timeout=10 Fetching upstream changes from https://github.com/apache/beam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/apache/beam.git > +refs/heads/*:refs/remotes/origin/* > git config remote.origin.url https://github.com/apache/beam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # > timeout=10 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10 Fetching upstream changes from https://github.com/apache/beam.git > git fetch --tags --progress https://github.com/apache/beam.git > +refs/heads/*:refs/remotes/origin/* > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/* > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision b2fd4e392ede19f03a48997252970b8bba8535f1 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f b2fd4e392ede19f03a48997252970b8bba8535f1 Commit message: "Merge pull request #8892 from riazela/bigquery-rowcount" > git rev-list --no-walk 291f1e9fb5ce5ee4bb7e2519ffe40334fb5c08c5 # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 No emails were triggered. [EnvInject] - Executing scripts and injecting environment variables after the SCM step. [EnvInject] - Injecting as environment variables the properties content SPARK_LOCAL_IP=127.0.0.1 [EnvInject] - Variables injected successfully. [Gradle] - Launching build. [src] $ <https://builds.apache.org/job/beam_PostCommit_XVR_Flink/ws/src/gradlew> --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx4g :beam-runners-flink_2.11-job-server:validatesCrossLanguageRunner ############################### WARNING #################################### ## ## ## Deprecation Warning: Gradle command altered. ## ## ## ## You are still using deprecated project namings. Instead of using e.g. ## ## ## ## :beam-sdks-java-core ## ## ## ## you should now use ## ## ## ## :sdks:java:core ## ## ## ## no to reference projects and their tasks. See project-mappings file. ## ## ## ## This compatibility layer will be removed soon. ## ## ## ############################### WARNING #################################### Changed command to gradlew --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx4g :runners:flink:1.5:job-server:validatesCrossLanguageRunner Starting a Gradle Daemon (subsequent builds will be faster) > Task :buildSrc:compileJava NO-SOURCE > Task :buildSrc:compileGroovy FROM-CACHE > Task :buildSrc:pluginDescriptors > Task :buildSrc:processResources > Task :buildSrc:classes > Task :buildSrc:jar > Task :buildSrc:assemble > Task :buildSrc:spotlessGroovy > Task :buildSrc:spotlessGroovyCheck > Task :buildSrc:spotlessGroovyGradle > Task :buildSrc:spotlessGroovyGradleCheck > Task :buildSrc:spotlessCheck > Task :buildSrc:pluginUnderTestMetadata > Task :buildSrc:compileTestJava NO-SOURCE > Task :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test NO-SOURCE > Task :buildSrc:validateTaskProperties FROM-CACHE > Task :buildSrc:check > Task :buildSrc:build Configuration on demand is an incubating feature. FAILURE: Build failed with an exception. * What went wrong: Task 'validatesCrossLanguageRunner' not found in project ':runners:flink:1.5:job-server'. * Try: Run gradlew_orig tasks to get a list of available tasks. Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 11s Publishing build scan... https://gradle.com/s/cn7hv7dlwxxus Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
