See
<https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/2/display/redirect?page=changes>
Changes:
[robertwb] Supporting infrastructure for dataframes on beam.
[robertwb] Basic deferred data frame implementation.
[robertwb] yapf, py2
[github] typings and docs for expressions.py
[chamikara] Logs BQ insert failures
[crites] Clean up of TestStreamTranscriptTests. Remvoes check for final field in
[ankurgoenka] [BEAM-9485] Raise error when transform urn is not implemented
[crites] Adds clearing of pane info state when windows get merged away.
[zyichi] Update BigQuery source in bigquery_tornadoes example
[mxm] [BEAM-9490] Guard referencing for environment expiration via a lock
[ehudm] [BEAM-9398] runtime_type_check: support setup
[lcwik] [BEAM-9481] fix indentation
[github] [BEAM-8335] Implemented Capture Size limitation (#11050)
[github] [BEAM-9294] Move RowJsonException out of RowJsonSerializer (#11102)
[github] Merge pull request #11046: [BEAM-9442] Properly handle nullable fields
[relax] switch cogbk to use Beam transform
[ankurgoenka] [BEAM-9287] disable validates runner test which uses teststreams
for
[relax] finish join
[sunjincheng121] [BEAM-9299-PR]Upgrade Flink Runner 1.8x to 1.8.3 and 1.9x to
1.9.2
[lcwik] [BEAM-2939] Implement interfaces and concrete watermark estimators
[jfarr1] [BEAM-8374] Add alternate SnsIO PublishResult coders
[ankurgoenka] [BEAM-9499] Sickbay test_multi_triggered_gbk_side_input for
streaming
[kcweaver] [BEAM-9503] Insert missing comma in process worker script.
[robertwb] Minor cleanup, lint.
[robertwb] [BEAM-9433] Create expansion service artifact for common Java IOs.
[thw] [BEAM-9490] Use the lock that belongs to the cache when bundle load
[github] Update Dataflow py container version (#11120)
[github] [BEAM-7923] Streaming support and pipeline pruning when instrumenting a
[github] Update default value in Java snippet
[ankurgoenka] [BEAM-9504] Sickbay streaming test for batch VR
[rohde.samuel] [BEAM-8335] Final PR to merge the InteractiveBeam feature branch
[github] [BEAM-9477] RowCoder should be hashable and picklable (#11088)
[apilloud] [BEAM-8057] Reject Infinite or NaN literals at parse time
[apilloud] [BEAM-7832] Translate ZetaSQL joins without condition
[robertwb] Log in a daemon thread.
[thw] [BEAM-8815] Skip removal of manifest when no artifacts were staged.
[robertwb] [BEAM-9496] Evaluation of deferred dataframes via Beam operations.
[relax] support side-input joins
[relax] support side-input joins
[relax] spotless
[github] [BEAM-9346] Improve the efficiency of TFRecordIO (#11122)
[kawaigin] [BEAM-8335] Refactor IPythonLogHandler
[kcweaver] [BEAM-8866] Use unique temp dir for output of portable word count
tests.
[apilloud] [BEAM-8070] Preserve type for empty array
[kcweaver] [BEAM-9509] Improve error message for bad job server URL.
[relax] make FieldAccessDescriptor always be field-insertion order
[davidyan] [BEAM-9510] Fixing version incompatibities in
[github] Merge pull request #10991 [BEAM-3301] Refactor DoFn validation & allow
[davidyan] Bring the dep versions up to par with
[valentyn] Install typing only on 3.5.2 or earlier versions of Python.
[github] Update dataflow py container ver to 20200317 (#11145)
[kawaigin] [BEAM-7923] Include side effects in p.run
[lcwik] [BEAM-9430] Migrate from ProcessContext#updateWatermark to
[github] [BEAM-9498] Include descriptor and type of unsupported fields in
RowJson
[github] Merge pull request #11149 from [BEAM-9533] Adding tox cloud tests
[github] Flink 1.10 yarn deployment fix (#11146)
[github] [BEAM-9539] Fix copy-pasted comment in load-tests' build.gradle
(#11155)
[lcwik] [BEAM-9540] Rename beam:source:runner:0.1/beam:sink:runner:0.1 to
[github] [BEAM-9526] Add missing unmarshalling in top.LargestPerKey. (#11143)
[robertwb] [BEAM-9535] Remove unused ParDoPayload.Parameters.
[github] Merge pull request #11147 from [BEAM-7923] Support dict and iterable
[coheigea] BEAM-8924 - Update Apache Tika to 1.24
[robertwb] [BEAM-9339] Declare capabilities in the Java SDK.
[kawaigin] [BEAM-7923] Change Transform Label Prefix Syntax
[github] Specify return types of window start/end functions explicitly (#11152)
[apilloud] [BEAM-9511] Uncollect takes arbitrary expressions
[lcwik] [BEAM-9339, BEAM-2939] Drop splittable field from proto, add splittable
[apilloud] [BEAM-9515] Add test
[lcwik] [BEAM-4374] Define the protos for a "short" id mechanism for metrics
[robertwb] [BEAM-9339] Add additional Java capabilities.
[kcweaver] [BEAM-9553] Use latest Flink job server image as default.
[jfarr1] [BEAM-9470] fix flaky unit test in :sdks:java:io:kinesis
[relax] fix side-input joins
[relax] fix bug
[github] Merge pull request #11158 from [BEAM-9533] Fixing tox.ini variants
[relax] remove obsolete test
[alex] [BEAM-9035] BIP-1: Typed options for Row Schema and Field
[iemejia] [BEAM-9279] Refactor HBase to disminish relying on Serializable
wrappers
[iemejia] [BEAM-9279] Make HBase.ReadAll based on Reads instead of HBaseQuery
[github] [BEAM-9551] Environment PB Pointer cleanup (#11164)
[relax] add javadoc
[iemejia] Move CHANGES template related items into template section
[rohde.samuel] Add dependency comment in streaming cache
[robertwb] Fix and test tuple inputs and outputs.
[lcwik] fixup! Address PR comments.
[github] Merge pull request #11166 from [BEAM-7923] Emit info when capture
[github] fix typo at Python Package name (#11098)
[github] [BEAM-9552] Bump TestPubsub subscription creation ACK deadline to 60s
[daniel.o.programmer] [BEAM-3301] Perform SDF validation (missing
RestrictionTrackers).
[kawaigin] Remove the excessive logging from capturable sources property.
[ehudm] [BEAM-8280] Enable type hint annotations
[github] Merge pull request #11128 from [BEAM-9524] Fix for ib.show() executing
[lcwik] [BEAM-9430] Update CHANGES.md to reflect removal of
[piotr.szuberski] [BEAM-9563] Change ToListCombineFn access level to private
[boyuanz] Add Timer to Elements proto representation.
[robertwb] [BEAM-9340] Plumb requirements through Java SDK.
[robertwb] [BEAM-9340] Populate requirements for Java DoFn properties.
[daniel.o.programmer] [BEAM-3301] Bugfix in DoFn validation.
[robertwb] [BEAM-9558] Add an explicit end field to the data channel protos.
[robertwb] [BEAM-9558] Regenerate go protos.
[robertwb] [BEAM-9558] Produce and respect data channel end bit in runners and
[github] Merge pull request #11153 from [BEAM-9537] Adding a new module for
[kcweaver] [BEAM-9446] Retain unknown arguments when using uber jar job server.
[lcwik] [BEAM-9565] Fix threading issue with WatermarkEstimatorsTest
[relax] add unit tests
[relax] update sql transform
[lcwik] [BEAM-9430] Fix coder sent to Dataflow service for non-portable
[kcweaver] Enable '--option=value' and '-option' syntax.
[github] Merge pull request #10990: [BEAM-9569] disable coder inference for rows
[rohde.samuel] Adds a streaming wordcount integration test
[rohde.samuel] Address leftover styling comments from PR/10892
[github] optionally import grpc (#11187)
[github] [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
[rohde.samuel] address comments
[github] Merge pull request #11198 from [BEAM-7923] Obfuscates display ids
[github] Merge pull request #11074: Store logical type values in Row instead of
[piotr.szuberski] [BEAM-9507] Fix python dependency check task
[mxm] [BEAM-9573] Correct computing of watermark hold for timer output
[github] [BEAM-7923] Pop failed transform when error is raised (#11174)
[pabloem] Fixing bqtest
[suztomo] google-api-client 1.30.9
[github] [BEAM-9579] Fix numpy logic operators (#11204)
[mxm] [BEAM-9580] Downgrade Flink version to 1.9 for Nexmark and
[github] Merge pull request #11163 from [BEAM-9548] Add better error handling to
[daniel.o.programmer] [BEAM-3301] Adding restriction trackers and validation.
[ehudm] [BEAM-8280] Type hints via annotations snippets
[rohde.samuel] changed data to be less repetitive
[github] Merge pull request #11215 from [BEAM-9601] unbreaking precommits
[pabloem] Starting refactor of FnApiRunner
[pabloem] fixup
[pabloem] Fix lint issues
[pabloem] Creating file with worker handlers
[pabloem] Fixup
[pabloem] Fixing lint. Formatting
[pabloem] Fixup
[jozo.vilcek] [BEAM-9420] Configurable timeout for blocking kafka API call(s)
[piotr.szuberski] [BEAM-9606] Add missing parameters in combine_test.py gradle
example
[github] Go changes for model updates. (#11211)
[alex] [BEAM-9605] BIP-1: Rename setRowOption to setOption on Option builder
[robertwb] Add base SDK version to environment capabilities for Python and Java.
[apilloud] [BEAM-9609] Upgrade to ZetaSQL 2020.03.2
[robertwb] [BEAM-9340] Validate pipeline requirements in PipelineValidator.
[pabloem] Renaming method to be more appropriate
[daniel.o.programmer] [BEAM-3301] Fix another bug in DoFn validation, in exec.
[github] fix formatter
[github] [BEAM-8078] streaming_wordcount_debugging.py is missing a test (#10914)
[rohde.samuel] [BEAM-9601] Skip the streaming wordcount test because it uses a
[github] Update Go Protos (#11230)
[github] add @RequiresTimeSortedInput to CHANGES.md (#11228)
[github] Add notes to change log.
[ehudm] [BEAM-8078] Disable test_streaming_wordcount_it
[zyichi] [BEAM-9263] Bump up python sdk dataflow environment major versions
[github] [BEAM-9371] Add SideInputLoadTest to Java SDK (#10949)
[github] [BEAM-7505] Add side input load test to Python SDK (#11136)
[mxm] [BEAM-9566] Mitigate performance issue for output timestamp watermark
[robertwb] [BEAM-9614] Add SDK id for go.
[ehudm] [BEAM-8078] Disable test_streaming_wordcount_debugging_it
[github] [BEAM-9495] Make DataCatalogTableProvider AutoCloseable (#11116)
[ehudm] [BEAM-8466] Make strip_iterable more strict
[robertwb] [BEAM-4150] Use explicit map for data channel coders.
[robertwb] [BEAM-4150] Don't window PCollection coders.
[ehudm] [BEAM-5422] Document DynamicDestinations.getTable uniqueness requirement
[ehudm] [BEAM-1894] Remove obsolete EagerRunner test
[github] [BEAM-9574] Ensure that instances of generated namedtuple classes can
be
[github] Update the range for pyarrow
[boyuanz] Remove TimeSpec from proto
[github] add missing bracket
[alex] [BEAM-9605] BIP-1: Rename setRowOption to setOption on Option builder
[lcwik] [BEAM-4374] Update protos related to MonitoringInfo.
[github] Update the Go SDK roadmap for April 2020 (#11239)
[github] Merge pull request #10883: [BEAM-9331] Add better Row builders
[github] [BEAM-8292] Portable Reshuffle for Go SDK (#11197)
[robertwb] Side input signals for legacy worker.
[lostluck] accept generated metrics
[github] Use split instead of rsplit
[lostluck] Remove mType and move type urns to urnToType
[lostluck] add missing pcollection payload
[github] [BEAM-9557] Fix timer window boundary checking (#11252)
[github] Update documentation
[echauchot] [BEAM-5980] Change load-tests build to include spark-sql for spark
[echauchot] [BEAM-9436] avoid one flatmap step and a KV creation per element by
[jozsi] Update Jet version to 4.0
[apilloud] [BEAM-9512] Map anonymous structs to schema
[kawaigin] [BEAM-7923] Fix datatable on notebook reloading
[kyoungha] [BEAM-9325] Added Proper Write Method in UnownedOutputStream
[kyoungha] fix format warning
[valentyn] Fix a Py2/3 incompatibility in profiler.
[kcweaver] [BEAM-9638] Strengthen worker region & zone options tests.
[samuelw] [BEAM-9399] Change the redirection of System.err to be a custom
[kyoungha] [BEAM-9325] reflect comment : inline testing methods
[robinyqiu] Clean up code in ZetaSQLDialectSpecTest
[boyuanz] [BEAM-9454] Add Deduplication PTransform
[robertwb] [BEAM-9577] Rename the Artifact{Staging,Retrieval}Service.
[robertwb] [BEAM-9577] Define the new Artifact{Staging,Retrieval}Service.
[robertwb] [BEAM-9577] Regenerate protos.
[jozsi] Update Jet Runner web page with info about 4.0
[jozsi] Add Beam-Jet compatibility table
[robertwb] [BEAM-9577] Implement the new Artifact{Staging,Retrieval}Services in
[samuelw] Fix missing test import
[kyoungha] [BEAM-9325] reflect comment : Fix JAXBCoder + change test
[chamikara] Refactors X-Lang test pipelines.
[robertwb] [BEAM-9340] Populate requirement for timer families.
[kcweaver] [BEAM-9199] Require Dataflow --region in Python SDK.
[kcweaver] Add --region to tests where needed.
[kcweaver] [BEAM-9199] Require --region option for Dataflow in Java SDK.
[kcweaver] Add --region to Java GCP tests.
[pabloem] [BEAM-9608] Increase reliance on Context Managers for FnApiRunner
[pabloem] Revert "Merge pull request #11104 from y1chi/update_tornado_test"
[daniel.o.programmer] [BEAM-9642] Create runtime invokers for SDF methods.
[lcwik] [BEAM-9668] Disable tests till Dataflow containers are updated.
[github] [BEAM-9652] Ensure that the multipartition write sets the correct coder
[github] [BEAM-8889]add experiment flag use_grpc_for_gcs (#11183)
[robertwb] [BEAM-9322] [BEAM-1833] Better naming for composite transform output
[ameihm] [BEAM-9476] KinesisIO retry LimitExceededException
[github] [BEAM-7923] An indicator of progress in notebooks (#11276)
[robertwb] [BEAM-9577] Add dependency information to provision info.
[robertwb] Update go protos.
[lcwik] [BEAM-9677] Fix path -> url typo in ArtifactUrlPayload
[lcwik] [BEAM-9562] Update missed TimerSpec conversion in Go SDK
[kcweaver] Fix DataflowRunnerTest.
[kcweaver] Fix more Java unit tests missing --region.
[github] [BEAM-9667] Allow metrics in DoFn Setup (#11287)
[amaliujia] add 2.20.0 blog post
[kcweaver] Add --region to DF streaming example tests.
[github] [BEAM-9624] Adds Convert to Accumulators operator for use in combiner
[github] Fix minor typo
[github] Fix minor typo
[github] Merge pull request #11290: [BEAM-9670] Fix nullability widening in
[iemejia] [BEAM-9686] Get default TmpCheckpointDir value from PipelineOptions
[github] [BEAM-4374] Short IDs for the Python SDK (#11286)
[spoorti] [BEAM-9660]: Add an explicit check for integer overflow.
[github] [BEAM-9136]Add licenses for dependencies for Python (#11067)
[boyuanz] Populate source data from SDF
[boyuanz] Update Timer encoding
[mxm] [BEAM-9645] Fix premature removal of Docker container and logs
[alex] [BEAM-9044] Protobuf options to Schema options
[kcweaver] Add unit tests for get_default_gcp_region
[kcweaver] Add --region to Dataflow runner webpage.
[mxm] [BEAM-8201] Cleanup FnServices from DockerEnvironmentFactory and
[kcweaver] lint
[kcweaver] Add --region to more Java tests and examples.
[kcweaver] Add --region to more Python tests and examples.
[kcweaver] format
[robertwb] [BEAM-9577] Update container boot code to stage from dependencies, if
[rohde.samuel] Change delimeter to a dash as it is a reserved symbol in Windows
[valentyn] Fixes platform-dependent assumptions in subprocess_server_test.py.
[valentyn] Switches a test helper to a Py3-version thereof.
[github] Apply suggestions from code review
[robertwb] Use pointer recievers.
[aldaircr] Change: Fixing typos on javadoc
[robertwb] Attempt to stage resources via new API in portable runner.
[robertwb] ResolveArtifact -> ResolveArtifacts
[robertwb] Regenerate protos.
[veblush] Upgrades gcsio to 2.1.2
[github] Merge pull request #11259: Use attachValues in SQL
[alex] Add Beam Schema Options to changelog
[iemejia] [website] Update information about Beam's LTS policy
[alex] [BEAM-9704] Deprecate FieldType metadata
[eekkaaadrian] [BEAM-9705] Go sdk add value length validation checking on write
to
[kcweaver] Remove unrecognized --region option from non-DF tests.
[robertwb] [BEAM-9618] Add protocol for requesting process bundle descriptors.
[robertwb] [BEAM-9618] Update Python to support process bundle descriptor
fetching.
[robertwb] [BEAM-9618] Java FnApiClient support for process bundle descriptor
[github] [BEAM-8019] Python SDK support for cross-langauge pipelines in
Dataflow.
[robertwb] Typo fix.
[github] remove nose (#11307)
[lcwik] [BEAM-4374, BEAM-6189] Delete and remove deprecated Metrics proto
[ecapoccia] [BEAM-9434] Improve Spark runner reshuffle translation to maximize
[github] [BEAM-9685] remove Go SDK container from release process (#11308)
[kcweaver] [BEAM-9716] Alias zone to worker_zone and warn user.
[github] Merge pull request #11226: [BEAM-9557] Fix timer window boundary
[github] Merge pull request #11244 from [BEAM-3097] _ReadFromBigQuery supports
[pabloem] [BEAM-9691] Ensuring BQSource is avoided on FnApi
[pabloem] [BEAM-9715] Ensuring annotations_test passes in all
[github] Name the pipeline_v1 proto import
[github] Update materialize_test.go
[ankurgoenka] [BEAM-9707] Hardcode Unified harness image for fixing dataflow VR
2
[crites] Updates documentation for WINDOWED_VALUE coder.
[rohde.samuel] Fix flaky interactive_runner_test
[github] Merge pull request #11205 [BEAM-9578] Defer expensive artifact
[robertwb] Update go protos.
[github] Fix some Go SDK linter/vet warnings. (#11330)
[robertwb] [BEAM-9577] Plumb resources through Python job service and runner.
[robertwb] [BEAM-9618] Pull bundle descriptors for Go.
[github] [BEAM-9529] Remove datastore.v1, googledatastore (#11175)
[github] Update session.go
[github] Update stage.go
[github] Update server_test.go
[github] Update materialize.go
[github] Update materialize_test.go
[github] Update stage_test.go
[github] Update artifact.go
[github] Update provision.go
[github] Update retrieval.go
[github] Update staging.go
[github] Update translate.go
[github] Update datamgr.go
[github] Update datamgr_test.go
[github] Update logging.go
[github] Update logging_test.go
[github] Update monitoring.go
[github] Update session.go
[github] Update statemgr.go
[github] Update statemgr_test.go
[github] Update replace.go
[github] Update replace_test.go
[github] Update provision.go
[github] Update execute.go
[github] Update job.go
[github] Update translate.go
[github] Update translate.go
[github] Update job.go
[github] Update materialize.go
[mxm] [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline
[kamil.wasilewski] [BEAM-9721] Add --region to Dataflow-based load tests
[kamil.wasilewski] [BEAM-9721] LoadTestConfig: handle --region parameter and
put default
[github] [BEAM-9147] Add a VideoIntelligence transform to Java SDK (#11261)
[mxm] Revert "[BEAM-9580] Downgrade Flink version to 1.9 for Nexmark and
[kcweaver] [BEAM-9714] [Go SDK] Require --region flag in Dataflow runner.
[github] Update translate.go
[mxm] [BEAM-9557] Fix strings used to verify test output
[github] Update session.go
[github] Update materialize_test.go
[mxm] [BEAM-9596] Ensure metrics are available in PipelineResult when the
[samuelw] Ensure that empty messages are not flushed to handler.
[crites] Uses iterable coder for windows and copies all of timestamp encoding
[github] Update session.go (#11352)
[github] [BEAM-9618] Java SDK worker support for pulling bundle descriptors.
[chamikara] Adds nose back under packages needed for testing.
[robertwb] [BEAM-9618] Mark push registration as deprecated.
[github] [Beam-9063]update documentation (#10952)
[kcweaver] [BEAM-9726] [py] Make region optional for non-service Dataflow.
[kcweaver] [BEAM-9726] [java] Make region optional for non-service runner.
[github] [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors. (#11193)
[github] [BEAM-9721]Conditionally add Dataflow region to Dataflow-based
[michael.jacoby] [BEAM-9647] fixes MQTT clientId to long
[lcwik] [BEAM-4374] Fix missing deletion of metrics.
[github] [BEAM-8280] Document Python 3 annotations support (#11232)
[github] [BEAM-9731] Include more detail in passert.Equals errors. (#11359)
[github] [BEAM-9085] Fix performance regression in SyntheticSource on Python 3
[amaliujia] add a known issue
[samuelw] [BEAM-9651] Prevent StreamPool and stream initialization livelock
[boyuanz] [BEAM-9562, BEAM-6274] Fix-up timers to use Elements.Timer proto in
data
[robertwb] Allow unset write threshold for state backed iterable coder.
[github] Revert "[BEAM-9651] Prevent StreamPool and stream initialization
[ankurgoenka] [BEAM-9735] Adding Always trigger and using it in Reshuffle
[samuelw] [BEAM-9651] Prevent StreamPool and stream initialization livelock
[github] [BEAM-9727] Automatically set required experiment flags for dataflow
[github] Update environments.py to add a method to specify container image
[huangry] Include user distritribution into metric-dedicated validate runner
test.
[huangry] update inline comment.
[huangry] To allow only assert on committed metrics, bypassing the attempted
ones.
[huangry] Use distribution matcher for testing.
[kcweaver] Moving to 2.22.0-SNAPSHOT on master branch.
[kamil.wasilewski] [BEAM-8671] Migrate Load Tests to Python 3.7
[michal.walenia] [BEAM-9734] Revert #11122
[github] Add --region to changelog
[pabloem] Fix from_container_image call
[ankurgoenka] TOIL: Update Unified worker image
[boyuanz] [BEAM-9562] Update Element.timer, Element.Timer to Element.timers and
[robertwb] Comments and clarification.
[github] [BEAM-9443] support direct_num_workers=0 (#11372)
[APryor] Correct a typo in documentation
[chamikara] Updates Dataflow stateful DoFn setup to support external transforms
[github] [BEAM-9738] Update dataflow to setup correct docker environment
options.
[github] [BEAM-9136]Add licenses for dependencies for Java (#11243)
[kcweaver] [BEAM-9744] Add missing region option to py perf tests.
[lcwik] [BEAM-9562] Fix output timestamp to be inferred from scheduled time when
[kcweaver] [BEAM-9744] Remove --region option from SQL tests.
[lcwik] [BEAM-2939] Update unbounded source as SDF wrapper to resume
[pabloem] Fixing type names for BQ Avro Tools
[github] Merge pull request #11389 from Refactor the BCJ and capture controls to
[github] [BEAM-i9751] upgrade zetasql to 2020.04.1 (#11410)
[xhan] Documentation bug fix for FlatMapElements#via() SimpleFunction in the
[github] [BEAM-9650] Add PeriodicImpulse Transform and slowly changing side
input
[github] [BEAM-7923] Screendiff Integration Tests (#11338)
[lcwik] fixup! Fix spotbugs warning
[kcweaver] [BEAM-9756] Nexmark: only use --region in Dataflow.
[github] [BEAM-9642] Add SDF execution units. (#11327)
[lcwik] [BEAM-9577] Fix test to create urls from paths which are compatible with
[github] [BEAM-9136] reduce third_party_dependencies size (#11416)
[thw] Fix py37-lint
[thw] Maven compatible publish repository authentication via settings.xml
[github] [BEAM-9746] check for 0 length copies from state (#11413)
[pabloem] Removing underscore from _ReadFromBigQuery to make it external. It
[daniel.o.programmer] [BEAM-9642] Fix infinite recursion.
[kamil.wasilewski] Remove outdated doc for ReadFromBigQuery transform
[ehudm] [BEAM-9119] Disable flaky test
[github] [BEAM-8889] add gRPC suport in GCS connector (behind an
[amaliujia] fixup! update 2.20.0 date
[github] [BEAM-9729, BEAM-8486] Runner-side bundle registration cleanup.
(#11358)
[kcweaver] [BEAM-9765] Exclude module-info.class from vendored Calcite.
[robertwb] [BEAM-9577] Small fixes to portable runner staging.
[robertwb] [BEAM-9768] Gradle command for Python ULR.
[github] Add new release 2.20.0 to beam website (#11285)
[jkw] Fix typo
[pabloem] [BEAM-9769] Ensuring JSON is the default export format for BQ sink
[github] Merge pull request #11151 from [BEAM-9468] Hl7v2 io
[echauchot] Add GBK load tests script for spark structured streaming runner
[echauchot] Add CoGBK load tests script for spark structured streaming runner
[echauchot] Add Combine load tests script for spark structured streaming runner
[echauchot] Add Pardo load tests script for spark structured streaming runner
[github] [BEAM-9678] Create Go SDK introduction kata (#11340)
[github] Merge pull request #11436: Document Beam Schemas (forked PR)
[github] [BEAM-9648]: DirectRunner should return null on timeout (#11275)
[ehudm] [BEAM-9737] Don't use docker create option -u
[github] [BEAM-5379] Ignore go.sum files in RAT checks
[github] [BEAM-5605, BEAM-2939] Add support for FnApiDoFnRunner to handle split
[github] [BEAM-9674] Don't specify selected fields when fetching BigQuery table
[github] [BEAM-9650] Cleanup documentation on side inputs patterns (#11415)
[github] [BEAM-8646] Fix external environment on OS X as well. (#11431)
[github] [BEAM-9743] Fix TFRecordCodec to try harder to read/write (#11397)
[github] Updates and clarifications on type safety. (#11449)
[github] [BEAM-3836] Enable dynamic splitting/checkpointing within the Java SDK
[github] [BEAM-8872] Support split at fraction for OffsetRangeTracker (#11418)
------------------------------------------
[...truncated 8.70 KB...]
> Task :sdks:java:harness:processResources NO-SOURCE
> Task :sdks:java:expansion-service:processResources NO-SOURCE
> Task :vendor:sdks-java-extensions-protobuf:processResources NO-SOURCE
> Task :sdks:java:core:generateAvroJava NO-SOURCE
> Task :sdks:java:io:google-cloud-platform:processResources NO-SOURCE
> Task :runners:core-java:processTestResources NO-SOURCE
> Task :model:job-management:extractProto
> Task :sdks:java:extensions:protobuf:extractProto
> Task :model:fn-execution:extractProto
> Task :sdks:java:extensions:protobuf:processResources NO-SOURCE
> Task :model:job-management:processResources
> Task :runners:flink:1.10:copySourceOverrides
> Task :runners:flink:1.10:copyTestResourcesOverrides NO-SOURCE
> Task :sdks:java:core:generateGrammarSource FROM-CACHE
> Task :model:fn-execution:processResources
> Task :runners:flink:1.10:processResources
> Task :sdks:java:build-tools:compileJava FROM-CACHE
> Task :runners:flink:1.10:copyTestSourceOverrides
> Task :sdks:java:build-tools:processResources
> Task :sdks:java:build-tools:classes
> Task :runners:flink:1.10:processTestResources
> Task :sdks:java:core:processResources
> Task :sdks:java:core:generateTestAvroProtocol NO-SOURCE
> Task :sdks:java:build-tools:jar
> Task :sdks:java:core:generateTestAvroJava
> Task :sdks:java:core:generateTestGrammarSource NO-SOURCE
> Task :sdks:java:core:processTestResources
> Task :model:pipeline:extractIncludeProto
> Task :model:pipeline:extractProto
> Task :model:pipeline:generateProto
> Task :model:pipeline:compileJava FROM-CACHE
> Task :model:pipeline:processResources
> Task :model:pipeline:classes
> Task :model:pipeline:jar
> Task :model:job-management:extractIncludeProto
> Task :model:fn-execution:extractIncludeProto
> Task :model:job-management:generateProto
> Task :model:fn-execution:generateProto
> Task :model:job-management:compileJava FROM-CACHE
> Task :model:job-management:classes
> Task :model:fn-execution:compileJava FROM-CACHE
> Task :model:fn-execution:classes
> Task :model:pipeline:shadowJar
> Task :model:job-management:shadowJar
> Task :model:fn-execution:shadowJar
> Task :sdks:java:core:compileJava FROM-CACHE
> Task :sdks:java:core:classes
> Task :sdks:java:core:shadowJar
> Task :sdks:java:extensions:protobuf:extractIncludeProto
> Task :sdks:java:extensions:protobuf:generateProto NO-SOURCE
> Task :sdks:java:extensions:google-cloud-platform-core:compileJava FROM-CACHE
> Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE
> Task :vendor:sdks-java-extensions-protobuf:compileJava FROM-CACHE
> Task :vendor:sdks-java-extensions-protobuf:classes UP-TO-DATE
> Task :sdks:java:extensions:google-cloud-platform-core:jar
> Task :runners:core-construction-java:compileJava FROM-CACHE
> Task :runners:core-construction-java:classes UP-TO-DATE
> Task :sdks:java:fn-execution:compileJava FROM-CACHE
> Task :sdks:java:fn-execution:classes UP-TO-DATE
> Task :sdks:java:fn-execution:jar
> Task :runners:core-construction-java:jar
> Task :vendor:sdks-java-extensions-protobuf:shadowJar
> Task :sdks:java:expansion-service:compileJava FROM-CACHE
> Task :sdks:java:expansion-service:classes UP-TO-DATE
> Task :sdks:java:expansion-service:jar
> Task :runners:core-java:compileJava FROM-CACHE
> Task :runners:core-java:classes UP-TO-DATE
> Task :sdks:java:extensions:protobuf:compileJava FROM-CACHE
> Task :sdks:java:extensions:protobuf:classes UP-TO-DATE
> Task :runners:core-java:jar
> Task :sdks:java:extensions:protobuf:jar
> Task :sdks:java:harness:compileJava FROM-CACHE
> Task :sdks:java:harness:classes UP-TO-DATE
> Task :sdks:java:harness:jar
> Task :sdks:java:io:google-cloud-platform:compileJava FROM-CACHE
> Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE
> Task :sdks:java:core:compileTestJava FROM-CACHE
> Task :sdks:java:core:testClasses
> Task :sdks:java:io:google-cloud-platform:jar
> Task :sdks:java:core:shadowTestJar
> Task :runners:core-java:compileTestJava FROM-CACHE
> Task :runners:core-java:testClasses UP-TO-DATE
> Task :runners:core-java:testJar
> Task :sdks:java:harness:shadowJar
> Task :runners:java-fn-execution:compileJava FROM-CACHE
> Task :runners:java-fn-execution:classes UP-TO-DATE
> Task :runners:java-fn-execution:jar
> Task :runners:flink:1.10:compileJava FROM-CACHE
> Task :runners:flink:1.10:classes
> Task :runners:flink:1.10:jar
> Task :runners:flink:1.10:compileTestJava FROM-CACHE
> Task :runners:flink:1.10:testClasses
> Task :runners:flink:1.10:testJar
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD SUCCESSFUL in 39s
63 actionable tasks: 43 executed, 20 from cache
Publishing build scan...
https://gradle.com/s/5fyixpeluyfi6
Build step 'Invoke Gradle script' changed build result to SUCCESS
[Gradle] - Launching build.
[src] $
<https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/ws/src/gradlew>
-x shadowJar -x shadowTestJar -x compileJava -x compileTestJava -x jar -x
testJar -x classes -x testClasses
-Dorg.gradle.java.home=/usr/lib/jvm/java-11-openjdk-amd64 --continue
--max-workers=120 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx4g
:runners:flinK:1.10:validatesRunner
Starting a Gradle Daemon, 1 incompatible Daemon could not be reused, use
--status for details
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy FROM-CACHE
> Task :buildSrc:pluginDescriptors UP-TO-DATE
> Task :buildSrc:processResources UP-TO-DATE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar
> Task :buildSrc:assemble
> Task :buildSrc:spotlessGroovy
> Task :buildSrc:spotlessGroovyCheck
> Task :buildSrc:spotlessGroovyGradle
> Task :buildSrc:spotlessGroovyGradleCheck
> Task :buildSrc:spotlessCheck
> Task :buildSrc:pluginUnderTestMetadata
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:validateTaskProperties FROM-CACHE
> Task :buildSrc:check
> Task :buildSrc:build
Configuration on demand is an incubating feature.
> Configure project :sdks:java:container
Found go 1.12 in /usr/bin/go, use it.
> Configure project :sdks:go
Found go 1.12 in /usr/bin/go, use it.
> Configure project :sdks:go:container
Found go 1.12 in /usr/bin/go, use it.
> Configure project :sdks:go:examples
Found go 1.12 in /usr/bin/go, use it.
> Configure project :sdks:go:test
Found go 1.12 in /usr/bin/go, use it.
> Configure project :sdks:python:container
Found go 1.12 in /usr/bin/go, use it.
> Task :runners:flink:1.10:validatesRunnerBatch
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.flink.core.memory.MemoryUtils
(file:/home/jenkins/.gradle/caches/modules-2/files-2.1/org.apache.flink/flink-core/1.10.0/fe708d32c982225b50af6ce3380a0b9b088d0627/flink-core-1.10.0.jar)
to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of
org.apache.flink.core.memory.MemoryUtils
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.flink.core.memory.MemoryUtils
(file:/home/jenkins/.gradle/caches/modules-2/files-2.1/org.apache.flink/flink-core/1.10.0/fe708d32c982225b50af6ce3380a0b9b088d0627/flink-core-1.10.0.jar)
to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of
org.apache.flink.core.memory.MemoryUtils
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
> Task :runners:flink:1.10:validatesRunnerStreaming
> Task :runners:flink:1.10:validatesRunnerBatch
org.apache.beam.sdk.transforms.ParDoSchemaTest >
testSchemaFieldSelectionUnboxing FAILED
org.apache.beam.sdk.Pipeline$PipelineExecutionException at
ParDoSchemaTest.java:542
Caused by: java.lang.IllegalStateException
org.apache.beam.sdk.transforms.ParDoSchemaTest >
testSchemaFieldDescriptorSelectionUnboxing FAILED
org.apache.beam.sdk.Pipeline$PipelineExecutionException at
ParDoSchemaTest.java:583
Caused by: java.lang.IllegalStateException
org.apache.beam.sdk.transforms.SplittableDoFnTest >
testPairWithIndexWindowedTimestampedBounded FAILED
org.apache.beam.sdk.Pipeline$PipelineExecutionException at
SplittableDoFnTest.java:224
Caused by: java.lang.NullPointerException
org.apache.beam.sdk.transforms.SplittableDoFnTest >
testPairWithIndexBasicBounded FAILED
org.apache.beam.sdk.Pipeline$PipelineExecutionException at
SplittableDoFnTest.java:157
Caused by: java.lang.NullPointerException
242 tests completed, 4 failed, 2 skipped
> Task :runners:flink:1.10:validatesRunnerBatch FAILED
> Task :runners:flink:1.10:validatesRunnerStreaming
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner
(file:/home/jenkins/.gradle/caches/modules-2/files-2.1/org.apache.flink/flink-core/1.10.0/fe708d32c982225b50af6ce3380a0b9b088d0627/flink-core-1.10.0.jar)
to field java.lang.String.value
WARNING: Please consider reporting this to the maintainers of
org.apache.flink.api.java.ClosureCleaner
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner
(file:/home/jenkins/.gradle/caches/modules-2/files-2.1/org.apache.flink/flink-core/1.10.0/fe708d32c982225b50af6ce3380a0b9b088d0627/flink-core-1.10.0.jar)
to field java.lang.String.value
WARNING: Please consider reporting this to the maintainers of
org.apache.flink.api.java.ClosureCleaner
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
org.apache.beam.sdk.transforms.ParDoSchemaTest >
testSchemaFieldSelectionUnboxing FAILED
org.apache.beam.sdk.Pipeline$PipelineExecutionException at
ParDoSchemaTest.java:542
Caused by: java.lang.IllegalStateException
org.apache.beam.sdk.transforms.ParDoSchemaTest >
testSchemaFieldDescriptorSelectionUnboxing FAILED
org.apache.beam.sdk.Pipeline$PipelineExecutionException at
ParDoSchemaTest.java:583
Caused by: java.lang.IllegalStateException
org.apache.beam.sdk.transforms.SplittableDoFnTest >
testPairWithIndexBasicUnbounded FAILED
java.lang.AssertionError at SplittableDoFnTest.java:145
Caused by: java.lang.AssertionError
257 tests completed, 3 failed, 3 skipped
> Task :runners:flink:1.10:validatesRunnerStreaming FAILED
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runners:flink:1.10:validatesRunnerBatch'.
> There were failing tests. See the report at:
> file://<https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/ws/src/runners/flink/1.10/build/reports/tests/validatesRunnerBatch/index.html>
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runners:flink:1.10:validatesRunnerStreaming'.
> There were failing tests. See the report at:
> file://<https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/ws/src/runners/flink/1.10/build/reports/tests/validatesRunnerStreaming/index.html>
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4m 0s
2 actionable tasks: 2 executed
Publishing build scan...
https://gradle.com/s/fpollhi5qzci6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
