This is an automated email from the ASF dual-hosted git repository.
echauchot pushed a change to branch spark-runner_structured-streaming
in repository https://gitbox.apache.org/repos/asf/beam.git.
discard 3533779 Fix spotlessJava issues
discard 77cacde Add ReadSourceTranslatorStreaming
discard 91baa65 Cleaning
discard 36a72f7 Use raw Encoder<WindowedValue> also in regular
ReadSourceTranslatorBatch
discard d9869c4 Split batch and streaming sources and translators
discard 8591d63 Run pipeline in batch mode or in streaming mode
discard 87bec8e Move DatasetSourceMock to proper batch mode
discard 017dcb9 clean deps
discard e510711 Use raw WindowedValue so that spark Encoders could work
(temporary)
discard 98d9049 fix mock, wire mock in translators and create a main test.
discard 83f8487 Add source mocks
discard 44fd6c7 Experiment over using spark Catalog to pass in Beam Source
through spark Table
discard e7ed784 Improve type enforcement in ReadSourceTranslator
discard 8bcfa5f Improve exception flow
discard e86247f start source instanciation
discard 8e08c58 Apply spotless
discard f54899b update TODO
discard 3e87c5e Implement read transform
discard b2d37bf Use Iterators.transform() to return Iterable
discard 7b00f7c Add primitive GroupByKeyTranslatorBatch implementation
discard 5ca19f2 Add Flatten transformation translator
discard 59acff8 Create Datasets manipulation methods
discard 26238ce Create PCollections manipulation methods
discard 9061cb0 Add basic pipeline execution. Refactor translatePipeline() to
return the translationContext on which we can run startPipeline()
discard 463178b Added SparkRunnerRegistrar
discard 358170a Add precise TODO for multiple TransformTranslator per
transform URN
discard c567d40 Post-pone batch qualifier in all classes names for readability
discard af6a350 Add TODOs
discard 74054fe Make codestyle and firebug happy
discard 92ed130 apply spotless for e-formatting
discard 41742be Move common translation context components to superclass
discard fc03a4a Move SparkTransformOverrides to correct package
discard 3251ab2 Improve javadocs
discard 8184de0 Make transform translation clearer: renaming, comments
discard 3ce792d Refactoring: -move batch/streaming common translation visitor
and utility methods to PipelineTranslator -rename batch dedicated classes to
Batch* to differentiate with their streaming counterparts -Introduce
TranslationContext for common batch/streaming components
discard 53b2e71 Initialise BatchTranslationContext
discard 5cb1add Organise methods in PipelineTranslator
discard b6b426e Renames: better differenciate pipeline translator for
transform translator
discard b77c7bb Wire node translators with pipeline translator
discard 91f283a Add nodes translators structure
discard 999761a Add global pipeline translation structure
discard ad32304 Start pipeline translation
discard 8b5c33c Add SparkPipelineOptions
discard 0a89b75 Fix missing dep
discard 03d333d Add an empty spark-structured-streaming runner project
targeting spark 2.4.0
add cb4358c [BEAM-6098] Support lookup join symmetric in left/right inputs
add b484402 Merge pull request #7118: [BEAM-6098] Support lookup join
symmetric in left/right inputs
add 7822091 [BEAM-6082] Fix enum for SQL query 5 and 7
add 9d05eed Merge pull request #7133: [BEAM-6082] Fix enum for SQL query
5 and 7
add 7963df4 Updates the script for cutting a release branch
add bedd4d9 Merge pull request #7108: Updates the script for cutting a
release branch
add e662b5f [BEAM-6102] Legacy-worker gradle changes
add adf659e [BEAM-6114] Add isBounded() to BeamRelNode and BeamSqlTable,
use for JOIN
add 048471b Merge pull request #7121: [BEAM-6114] Add isBounded() to
BeamRelNode and BeamSqlTable, use for JOIN
add 148808a [BEAM-6102] Move the worker jar before the pipeline jar when
using the --dataflowWorkerJar option (#7143)
add 02c763b [BEAM-5197] Fix UnboundedSourceWrapper#testWatermarkEmission
(#7138)
add f84feac [BEAM-6058] Adding option for flink configuration directory
and setting config in exectution environment
add c2aaf2d Merge pull request #7031: [BEAM-6058] Adding option for Flink
configuration directory and setting config in execution environment
add d922f70 [BEAM-6102] Fix several packages that were being bundled
without relocation within the Dataflow worker. (#7145)
add 4082ce5 [BEAM-2687] Correctly handle read-before-write semantics for
user state. (#7102)
add 01ef416 Add argument parsing and filtering to coders microbenchmark.
add b76f38b Missing type declaration in default coder.
add 45c114d Optimize sequence coders.
add 18fde2e Restore unicode typing after Py3 changes.
add 0a33d9b Optimize LengthPrefixCoder.
add eb4a65a Optimize WindowedValue encoding.
add 1cc6efc Optimize IntervalWindow encoding.
add 06d4a56 Merge pull request #7130 from Optimize Python coders.
add 400c174 BEAM-6134: MongoDbIO add support for projection
add 62b63fc Merge pull request #7148 from chaimt/BEAM-6134
add 68be16a [BEAM-6100] Collect metrics properly in Load tests (#7087)
add 3fc46b5 [BEAM-5197] Fix possible overflow of timestamp for processing
timer
add 0261783 Add an option to create Dataflow piplines from a snapshot
add 55fae0d Fix merge conflicts
add b9e6fa1 Merge pull request #7092 from dpmills/snapshots
add b1c0993 Update Java Container beam-master-20181128
add 05a7801 Undo accidental commit
b1c09939a54527ca47d02daf742001012a2be149
add 8aff81d Update Dataflow worker container to beam-master-20181128
add df23d47 Merge pull request #7154 from charlesccychen/update-container
add de3bd4af Future-proofing for TensorFlow2 (#7155)
add 004cc54 Revert "Merge pull request #7051 from
markflyhigh/py-move-off-apitools-1"
add d027d96 Merge pull request #7156: [BEAM-6145] Revert "Merge pull
request #7051 from markflyhigh/py-move-off-apitools-1"
add 0b7dc3d [BEAM-6148] ptest support arbitrary runners
add 64a6a3c Merge pull request #7158 from lostluck/ptest
add a440344 [BEAM-4726] Add heap profiling hook
add d9ae90f Merge pull request #7159 from lostluck/heap
add 0a45041 [BEAM-3612] Closurize method invocations
add 1024472 Merge pull request #7161 from lostluck/wrap
add 6cd0d9b [BEAM-3661] Port TriggerExampleTest off DoFnTester
add 2b5a604 Merge pull request #7125: [BEAM-3661] Port TriggerExampleTest
off DoFnTester
add b629727 [SQL] Add support for TableMacro UDF
add 2f0f3f2 Merge pull request #7141 from
kanterov/kanterov_table_functions
add b8d2423 Fix Go lints on Initialized
add 96fe92d Merge pull request #7167 from lostluck/fixtypos
add 4a6527d Revert "Optimize several Python coder implementations."
add fc373df Merge pull request #7166 from apache/revert-7130-fast-coders
add 5b9641d [BEAM-5978] Use dynamic port when starting auto starting
jobserver
add 38e6be9 [BEAM-5978] Adding libltdl7 to flink job server docker
add 65136cf [BEAM-5978] Correctly pick the docker executable
add 63a4c18 [BEAM-5978] Increase portable wordcount threads to avoid dead
lock
add 49c8386 [BEAM-6146] Add precommit for portable python
add 5506335 [BEAM-6146] Portable Python Precommit test
add b06b8e5 Merge pull request #6954: [BEAM-6146] Add portable WordCount
to Python PreCommit
add 4ca0e89 [BEAM-3659] Port UserScoreTest off DoFnTester
add bd81e8b Merge pull request #7126: [BEAM-3659] Port UserScoreTest off
DoFnTester
add d136637 [BEAM-2939] SplittableDoFn Java SDK API Changes (#6969)
add e55c514 [BEAM-6111] Fix flaky PortableTimersExecutionTest (#7171)
add ac10c91 Add packageDeclaration Checkstyle check + fix an issue it
threw up (#7172)
add ad0cb4f Simplifying a few junit assertions (#7139)
add d116dd3 [BEAM-6162] Fix PipelineOptions argparse behavior
add 1768511 Merge pull request #7176 from charlesccychen/fix-argparse
add a9dff65 Merge pull request #7175: [BEAM-5884] Move the nullable
attribute onto FieldType.
add 6119242 [BEAM-6163] Build python boot for mac and support process env
on mac
add 6517025 Merge pull request #7178: [BEAM-6163] Build python boot for
mac and support process env on mac
add 60cc4c1 Merge pull request #7147 : [BEAM-4453] Use constructors to
generate schema POJOs
add 42f39e9 [BEAM-6143] Upgrade to Mongo Java Driver 3.9.1 and update the
API
add b469880 Merge pull request #7151: [BEAM-6143] Upgrade to Mongo Java
Driver 3.9.1 and update the API
add f2d1581 [BEAM-5396] Savepoint restore option in Flink runner.
add 236d0dd Merge pull request #7169: [BEAM-5396] Savepoint restore
option in Flink runner
add 3432c04 [BEAM-6032] Move PortableValidatesRunner configuration out of
BeamModulePlugin (#7173)
add ff93af2 [BEAM-5925] Add .withSocketAndRetryTimeout() and
.withConnectTimeout() to ElasticseachIO.ConnectionConfiguration
add a566874 [BEAM-5925] Set a socket and retry timeout of 1.5min and a
connect timeout of 5s for all connections in all elasticsearch unit tests"
add d98c9f4 Merge pull request #7065 from wscheep/elastic_maxretrytimeout
add aeeb085 [BEAM-5984] Enable publishing load test results to BigQuery
add 9d3daf2 [BEAM-5984] Use more high level method for creating BQ rows
in Nexmark too
add ab3519e [BEAM-5984] Provide generic BigQueryResultPublisher class
add 90ef4eb Merge pull request #7090: [BEAM-5984] Enable publishing load
test results to Big Query
add 4c97d0c [BEAM-6146] Run pre-commit wordcount in batch and streaming
mode.
add b917a86 Merge pull request #7180: [BEAM-6146] Run pre-commit
wordcount in batch and streaming mode
add f040a4e [BEAM-5859] Improve operator names for portable pipelines
add e2e31c9 Merge pull request #7150: [BEAM-5859] Improve operator names
for portable pipelines
add aa4d9bc Add more AVRO utilities to convert between Beam and Avro. Add
schema-conversion utilities as well as a conversion from a Beam Row into
GenericRecord.
add 6b7cf42 Merge pull request #7181 : [BEAM-4454] Add more AVRO
utilities to convert between Beam and Avro.
add 78c1a10 [BEAM-6160] Use service server rather than service (#7168)
add 270ef6a [BEAM-2939] Add support for backlog reporting to byte key and
offset restriction trackers. (#7177)
add 3494a8f [BEAM-5058] Run basic ITs in Python Precommit in parallel
add 3a348e8 Merge pull request #7163 from markflyhigh/py-precommit-it
add 34420cc [BEAM-6174] Kryo dependency removed.
add 54214f8 Merge pull request #7194: [BEAM-6174] Kryo dependency removed.
add 2aa5d07 [BEAM-5778] Add integrations of Metrics API to Big Query for
SyntheticSources load tests in Python SDK
add ee515bf Merge pull request #6943: [BEAM-5778] Add integrations of
Metrics API to Big Query for SyntheticcSources load tests in Python SDK
add 08dafbe [BEAM-1628] Allow empty port for flink master url
add bec7dac Merge pull request #7187: [BEAM-1628] Allow empty port for
flink master url
add 9c018ac [BEAM-6122] Update committer guidelines
add 1c67861 Address review comments.
add 702b9de Move squash paragraph under merging.
add e5d9cf4 Merge pull request #7129: [BEAM-6122] Update committer
guidelines
add 9593adb [BEAM-6077] If available, use max_parallelism for splitting
unbounded source
add 5565b0a [BEAM-6077] Tests for read source translator
add a3a8a32 Merge pull request #7128: [BEAM-6077] If available, use
max_parallelism for splitting unbounded source
add 74ed7ac [BEAM-5462] get rid of <pipeline>.options deprecation
warnings in tests
add 95d0ac5 Merge pull request #6930: [BEAM-5462] get rid of
<pipeline>.options deprecation warnings in tests
add a3d2611 [BEAM-2400] Use getJobId() consistently
add a5b36c5 Merge pull request #7199: [BEAM-2400] Use getJobId()
consistently
add 2f9330c [BEAM-6180] Remove duplicated IdGenerator from runner harness
and use IdGenerator from fnexecution instead. (#7201)
add 681b5cd Merge pull request #7204: [BEAM-5884] Fix FieldType
comparison in BeamSQL
add 33453c2 Revert "Revert "Optimize several Python coder
implementations.""
add 50d8392 [BEAM-6153] Stricter interval window comparison.
add e7ab8c4 Merge pull request #7170 from [BEAM-6153] Re-enable coder
optimization.
add 43fe997 [BEAM-5817] Add SQL bounded side input join to queries that
are actually run
add a3510e0 Merge pull request #7205: [BEAM-5817] Add SQL bounded side
input join to queries that are actually run
add d897c5c BEAM-6151: MongoDbIO add support mongodb server with self
signed ssl
add 9fbe80e Merge branch 'master' into BEAM-6151
add 385f2a1 Merge pull request #7162 from chaimt/BEAM-6151
add 9448dba [BEAM-6182] Disable conscrypt by default (#7203)
add 8f15b88 [BEAM-3912] Add HadoopOutputFormatIO support
add 406f8d7 [BEAM-3912] Remove useless dep
add 86f723e [BEAM-3912] Add HadoopOutputFormatIO support
add 9863c79 [BEAM-3912] Remove useless dep
add fa9cc4a Fix typo in test name
add 757b71e [BEAM-3912] Implement HadoopFormatIO.Write
add 20e3e24 [BEAM-5309] Add streaming support for HadoopFormatIO
add 4adc254 [BEAM-5309] Add streaming support for HadoopFormatIO
add aec6d82 Merge pull request #6691: [BEAM-5309] Add streaming support
for HadoopFormatIO
add af05ee2 Add portable-runner dependency to wordcount example as one of
the defaults.
add 736077c Merge pull request #7213 from [BEAM-6184] Add portable-runner
dependency to example pom.xml
add 60da04a [BEAM-5859] Better handle fused composite stage names.
add 5ae80df Merge pull request #7208: [BEAM-5859] Better handle fused
composite stage names.
add 0edc85e [BEAM-6067] In Python SDK, specify pipeline_proto_coder_id
property in non-Beam-standard CloudObject coders (#7081)
add ecc2d84 Fixup User_COUNTER_URN_PREFIX to contain the trailing: (#7188)
add 75d45e2 [BEAM-6159] Make Dataflow worker use ExecutableStage to
process bundle (#7015)
add a466104 Update PortableTimersExecutionTest to use PAssert, to prevent
a concurrency issue collecting the test results.
add 79df784 Remove extra timeout code from PortableTimersExecutionTest.
add ce15b25 Merge pull request #7214: Update PortableTimersExecutionTest
to use PAssert
add 5e94da3 [BEAM-5920] Add additional owners for Community Metrics
add 5850c00 Merge pull request #7186: [BEAM-5920] Add additional owners
for Community Metrics
add bc859cc [BEAM-6181] Reporting user counters via MonitoringInfos in
Portable Dataflow Runner. (#7202)
add 9159d9b [BEAM-5321] Port transforms package to Python 3 (#7104)
add 4cd1226 Add QueueingBeamFnDataClient and make process, finish and
start run on the same thread to support metrics. (#6786)
add 1f6dd22 [BEAM-6155] Updates the GCS library the Go SDK uses.
add ca6ee63 Merge pull request #7182 from bramp/BEAM-6155
add 4d63cd3 [BEAM-5167] Log unscheduled process bundle requrests
add e6add6a Merge pull request #7192 from [BEAM-5167] Log unscheduled
process bundle requests
add fceeaef Removing some unnecessary parentheses
add 64c62b1 Merge pull request #7185 Removing some unnecessary parentheses
add ee0801f Enabling the ArrayTypeStyle checkstyle module
add ec5602f Merge pull request #7062 Enabling the ArrayTypeStyle
checkstyle module
add 61e8106 Fix translate_pattern test on Python 3.7
add 40977f4 Merge pull request #6739 [BEAM-5787] Fix
test_translate_pattern on Python 3.7
add c314dfe Add a MonitoringInfoSpec proto and
SimpleMonitoringInfoBuilder to provide specs and validate MonitoringInfos are
properly populated.
add a21b196 [BEAM-6194] Follow up with cleanup for
https://github.com/apache/beam/pull/7015 (#7219)
add c3102a9 Fix precommits due to concurrent submissions.
add aa09bbb Merge pull request #7230 Fix precommits due to concurrent
submissions.
add ab59d6d [BEAM-6139] Adding support for BQ GEOGRAPHY data type (#7144)
add ac3a9df Redirect from nexmark page's older location
add a226343 Merge pull request #7225 from udim/patch-2
add 17968a2 [BEAM-6195] Make ProcessRemoteBundleOperation map
PCollectionId into correct OutputReceiver and throws Exception when there is
more than one input PCollection. (#7223)
add 378d907 [BEAM-6167] Add class ReadFromTextWithFilename (Python)
(#7193)
add 36e3f98 [BEAM-4150] Use unwindowed coder in FnApiRunner optimization
phases.
add 6e1a8cd [BEAM-6186] Optimization cleanup: move phase utilities out of
local scope.
add a076444 Merge pull request #7227 from [BEAM-6186] Optimization cleanup
add e147bf9 [BEAM-6120] Support retrieval of large gbk iterables over the
state API.
add c2a9cac Merge pull request #7127 from [BEAM-6120] Large gbk iterables
add c3e636d Add instructions to post-commit policy web page, according to
discussions in dev mailing list.
add f976430 Update website/src/contribute/postcommits-policies-details.md
add 6e3cf84 Merge pull request #7095 from HuangLED/update_postcommit_doc
add 2b95624 Move string literals to the left hand side of the expression
in a few places
add e7b2f30 Merge pull request #6887 from coheigea/string_literals
add 2905227 Clarify usage of PipelineOptions subclass
add e963882 Merge pull request #6872 Clarify snippet for PipelineOptions
subclass
add d4fd5a1 [BEAM-5866] Override structuralValue in ListCoder
add 8b4f60e [BEAM-5866] Override structuralValue in MapCoder
add 7a394cf Merge pull request #6862 from [BEAM-5866] structuralValue in
List/MapCoder
add 8783994 [BEAM-4444] Parquet IO for Python SDK (#6763)
add a07da9e Upgrade to Apache Tika 19.1
add 1c6f145 Merge pull request #6719 Upgrade to Apache Tika 1.19.1
add 8100f32 [BEAM-6079] Add ability for CassandraIO to delete data
add 07d9311 [BEAM-6079] Fix access level and clean up generics issues
add b0aae58 Merge pull request #7064: [BEAM-6079] Add ability for
CassandraIO to delete data
add 010357d [BEAM-3657] Port JoinExamplesTest off DoFnTester
add 29a7917 Merge pull request #7179: [BEAM-3657] Port JoinExamplesTest
off DoFnTester
add f8ef83b [BEAM-5514] BigQueryIO doesn't handle quotaExceeded errors
properly
add b592f94 Merge pull request #7189: [BEAM-5514] BigQueryIO doesn't
handle quotaExceeded errors properly
add dc0946f [BEAM-6201] Move fromJsonString() method to SyntheticOptions
class
add cd8f014 [BEAM-6201] Add SyntheticDataPubSubPublisher pipeline
add b0069eb Merge pull request #7238: [BEAM-6201] Data insertion pipeline
add eb9065b [BEAM-6176] Support IPv6 addresses for Flink master url
add 4d65d72 Merge pull request #7196: [BEAM-6176] Support IPv6 addresses
for Flink master url
add f57def8 [BEAM-6172] Adjust Flink metric names / Add metric reporting
tests
add d80957c Merge pull request #7207: [BEAM-6172] Adjust Flink metric
names / Add metric reporting tests
add 57c248b Remove trailing whitespace.
add 0d2ec39 Quiet c-extension-no-member lint warning.
add edc37f9 Merge pull request #7249: Fix various lint errors in Python
presubmit.
add a9709ff [BEAM-6186] Move combiner lifting out of line.
add 3398a88 [BEAM-4678] Support combiner lifting in portable Flink runner.
add d4237ec Merge pull request #7228 from [BEAM-4678] Combiner lifting in
portable Flink.
add bfd1be9 [BEAM-6178] Adding beam-sdks-java-bom, adding exportJavadoc
flag for applyJavaNature (#7197)
add 17a881b [BEAM-6181] Unexpected metrics should be non-fatal.
add 4160b67 Merge pull request #7250 from [BEAM-6181] Unexpected metrics
non-fatal.
add cb06639 [BEAM-6033] normalize httplib2.Http initialization and usage
add 3a182d6 Merge pull request #7032: [BEAM-6033] normalize httplib2.Http
initialization and usage
add fddb684 [BEAM-5058] remove unused envdir parameter
add e8cbbf6 Merge pull request #7246: [BEAM-5058] remove unused envdir
parameter
add ebe81b9 [BEAM-2943] Fix typo in filename on FlinkRunner page
add 8c8b7c3 Merge pull request #7254: [BEAM-2943] Fix typo in filename on
FlinkRunner page
add 73a4325 Use environment to control worker startup in FnApiRunner.
add efd261a [BEAM-6094] Implement external environment for Portable Beam.
add ec55000 Merge pull request #7078 from [BEAM-6094] External portable
BeamPython.
add 9853bd7 [BEAM-6213] Fix matching of glob patterns in windows local
filesystem
add 289d2b2 Merge pull request #7258: [BEAM-6213] Fix matching of glob
patterns in windows local filesystem
add c00dfa5 Fix combiner lifting for external job service.
add 16b34cf Merge pull request #7260 Fix combiner lifting for external
job service.
add bc11c06 [BEAM-6216] Update flinkMaster URL in the nexmark wep-page to
reflect change in FlinkExecutionEnvironments.
add ffcd9f4 [BEAM-6216] Update flinkMaster URL in nexmark postCommit
script to reflect change in FlinkExecutionEnvironments.
add a53f56a Merge pull request #7261 from echauchot/BEAM-6216-flink-local
add 9e8ac83 [BEAM-6240] Clean up duplicated SocketAddressFactory class.
add 798b3b3 Merge pull request #7241: [BEAM-6204] Clean up duplicated
SocketAddressFactory class.
add 5e506bf [BEAM-6205] Setup gradle task ro run fnapi worker test with
use_executable_stage_bundle_execution
add 5574f47 Merge pull request #7243: [BEAM-6205] Setup gradle task ro
run fnapi worker test with use_execu…
add 8828b16 Add a MonitoringInfoLabelProps proto to attach the proper key
string name to MonitoringInfo label Enums
add a61f2c5 Merge pull request #7242: Add a MonitoringInfoLabelProps
proto to attach the proper key string name to MonitoringInfo label Enums
add 11513c3 Merge remote-tracking branch 'upstream/master' into pr7244
add dc10f75 Merge pull request #7244: [BEAM-6138] Add a
MonitoringInfoSpec proto and SimpleMonitoringInfoBuilder to pro…
add 41719ac Fixing publishing problem introduced in #7197
add 20df151 Merge pull request #7265: Fixing publishing problem
introduced in #7197
add 07bae67 [BEAM-5419] Add Flink multi-version build layout
add 6308a6d [BEAM-5267] Make source Flink v1.6 compatible
add d0254f0 [BEAM-5267] Add Flink v1.6 target for Flink Runner
add 3400ba6 Merge pull request #7229: [BEAM-5419] Add Flink multi-version
builds
add 2a40c57 Add remaining Schema support for AVRO records: * Add
support for SpecificRecord using ByteBuddy codegen. * Add helper methods for
GenericRecord. * Fix uncovered bugs involving nullable support.
add 0a74b17 Add period to sentence.
add 8b20602 Merge pull request #7233: [BEAM-4454] Add remaining
functionality for AVRO schemas
add 3fafec1 [BEAM-5320] [BEAM-6106] Finish Python 3 porting for testing
module (#7262)
add bd0103c [BEAM-5978] Changing parallalim for wordcount to 1
add 1518361 Merge pull request #7174: [BEAM-5978] Changing parallelism
for wordcount to 1
add a89d296 Fix broken beam-sdks-python:test
add 4eb7744 Merge pull request #7273: Fix broken beam-sdks-python:test
add 0b3b9e0 [BEAM-6138] Add User Counter Metric Support to Java SDK
(#6799)
add 1775c19 Stop subclassing user pojos.
add 6280255 spotless.
add f8f9ca5 Merge pull request #7234: [BEAM-4453] Stop subclassing user
POJOs.
add e2db4d5 Updates Beam Website to use 2.9.0 as the latest release
add 0f14b40 Merge pull request #7215: Updates Beam Website to use 2.9.0
as the latest release
add daed1e6 Blog for Apache Beam 2.9.0 release
add 5df8cb2 Merge pull request #7275: Blog for Apache Beam 2.9.0 release
add 2cac7ba Updates blog for 2.9.0 release
add e21b80a Merge pull request #7278: Updates blog for 2.9.0 release
add eae5521 Add reshuffle option to Create.
add a34e459 Merge pull request #7274 Add reshuffle option to Create.
add 0d60a89 [BEAM-6229] Fix LoadTestResult to store propoer timestamp and
runtime
add 88f181c Merge pull request #7283: [BEAM-6229] Fix LoadTestResult to
store propoer timestamp and runtime
add fc38359 [BEAM-6227] Fix GroupByKey with null values in Flink Runner
add 2886473 Merge pull request #7282: [BEAM-6227] Fix GroupByKey with
null values in Flink Runner
add 788ce61 Merge pull request #7267: [BEAM-4454] Support Avro POJO
objects
add 54e2fc1 [BEAM-6206] Add CustomHttpErrors a tool to allow adding
custom errors for specific failing http calls. Plus, add a custom error message
in BigQueryServicesImpl. (#7270)
add 69358c5 [BEAM-6191] Remove redundant error logging for Dataflow
exception handling
add 977080f Merge pull request #7220: [BEAM-6191] Remove redundant error
logging for Dataflow exception handling
add f6c1dd5 [BEAM-6179] Fixing bundle estimation when all xs are same
add 71890da Merge pull request #7280 from angoenka/fix_bundle_estimation
add 5ec695b [BEAM-6150] Superinterface for SerializableFunction allowing
declared exceptions (#7160)
add 4f90294 Remove Gradle from Jenkins job names
add ec3f792 Merge pull request #7286 from swegner/jenkins_gradle
add b7035c1 [BEAM-6225] Setup Jenkins Job to Run VR with ExecutableStage
(#7271)
add 53a5ce7 [BEAM-4594] Support state continuation tokens over the Fn API.
add 39721d2 Merge pull request #7252 from [BEAM-4594] State continuation
tokens.
add f9bc485 Mention portable Flink runner support for state and timers in
2.9.0 release blog
add bd5bbf9 Merge pull request #7294: Mention portable Flink runner
support for state and timers in 2.9.0 release blog
add c26d532 More robust FnApi Runner.
add ca3eb14 Merge pull request #7251 from More robust FnApi Runner.
add e4f6517 [BEAM-6227] Do not compare recovered state against structural
null value
add e108cca Merge pull request #7291: [BEAM-6227] Do not compare
recovered state against structural null value
add 8a333e7 [BEAM-6235] Upgrade AutoValue to version 1.6.3
add 4588c25 Merge pull request #7285: [BEAM-6235] Upgrade AutoValue to
version 1.6.3
add 89ad88d [BEAM-2873] Setting number of shards for writes with runner
determined sharding
add a68f209 Merge pull request #4760: [BEAM-2873] Setting number of
shards for writes with runner determined sharding
add a6d4345 [BEAM-6200] Deprecate old HadoopInputFormatIO, move it into
new HadoopFormatIO
add 67d0f78 Merge pull request #7263: [BEAM-6200] Deprecate old
HadoopInputFormatIO, move it into new HadoopFormatIO
add 2a5cc73 [BEAM-5419] Simplify job-server-container targets
add 1061dc9 Merge pull request #7299: [BEAM-5419] Simplify
job-server-container targets
add da7fec2 [BEAM-6190] Add processing stuck message to Pantheon.
add b070e89 Fixed some style errors and tests that needed an additional
parameter.
add fa522c5 Merge pull request #7240 from dustin12/lullError
add 2555945 Revert "[BEAM-5978] Changing parallelism for wordcount to 1"
add dc62028 [BEAM-6067] Update BeamPython side-input support in the
Dataflow runner for the unified worker. (#7269)
add bc1609d [BEAM-6170] Change Nexmark stuckness warnings to not cause
pipeline failure.
add c29092e Merge pull request #7191 from scwhittle/remove_stuck_error
add 0cdfca4 Avoid creating a variable and immediately returning it.
add af5d7bf Merge pull request #7007 from coheigea/return
add eab1d0e [BEAM-6252] SamzaRunner: Add a registrar to allow customized
DoFnInvoker
add 6a6fbe4 Allow Samza DoFnInvoker to generate configs and pass in
TaskContext
add 319a370 Merge pull request #7301: [BEAM-6252] SamzaRunner: Add a
registrar to allow customized DoFnInvoker
add 473e5a6 [BEAM-6253] SamzaRunner: Add a few customized transforms for
runner use cases
add 6ba3a45 Merge pull request #7302: [BEAM-6253] SamzaRunner: Add a few
customized transforms for runner use cases
add 2563e92 Update Dataflow Python container to 20181218
add 335fdb2 Merge pull request #7306 from charlesccychen/update-container
add a7315c5 [BEAM-6197] Log time for Dataflow GCS upload of staged files
add 095ce13 Merge pull request #7235 from alanmyrvold/uploadGCS
add df4b623 [BEAM-5167] Ensure monitoring thread does not prevent process
exit.
add d3fbf80 Merge pull request #7248 from robertwb/monitor-thread
add 4573c56 Fix time format on Windows.
add e4577ed Merge pull request #7257 Fix time format on Windows.
add 0ef5e66 Update release guide with new Jenkins job name.
add aa0fdc5 Merge pull request #7311: Update release guide with new
Jenkins job name.
add 384de82 WEBSITE: update community nav, add in-person page
add 5845e2a Merge pull request #7314: WEBSITE: update community nav, add
in-person page
add b5223f4 [BEAM-6263] Prevent NullPointer on concurrent JobServer
shutdown
add a50ae05 [BEAM-6263] Prevent port collisions in FlinkJobServerDriveTest
add 5d3b420 [BEAM-6263] Prevent stderr->stdout redirection
add 19f7dfa [BEAM-6263] Restore stderr on exceptions to print the error
add 302b883 Merge pull request #7309: [BEAM-6263] Fix error-prone test
setup for FlinkJobServerDriver
add 6ff3129 [BEAM-6186] Finish moving optimization phases.
add dba30b4 Merge pull request #7281 from [BEAM-6186] Finish moving
optimization phases.
add 6de7cb7 [BEAM-6094] Add loopback environment for java.
add a49c835 More complete exception message.
add 66ff825 Merge pull request #7307 [BEAM-6094] Add loopback environment
for java.
add 6e2ca58 [BEAM-6245] Set translation mode directly on PipelineOptions
add f67ac59 [BEAM-6245] Add integration test for FlinkTransformOverrides
add 85bd95e Merge pull request #7296: [BEAM-6245] Add integration test
for FlinkTransformOverrides
add 8214cb6 [BEAM-5449] Tagging failing ULR ValidatesRunner tests.
add bd68ef6 Merge pull request #7295: [BEAM-5449] Tagging failing ULR
ValidatesRunner tests.
add 0c589db [BEAM-5993] Create SideInput Load test (#7020)
add 61d7e53 Fix documentation for PCollection Type in XMLIO example
add f036699 Merge pull request #7320: [BEAM-6270] Fix documentation for
PCollection Type in XmlIO example
add c0655d1 Rename v1_13_1 to v1p13p1.
add 85ea7ba Depend on local vendored jar instead of the one from Maven
add 2afc42b Disable 'jar' task for vendoring projects.
add a5a139d Merge pull request #7324: [BEAM-6056] Rename vendored guava
relocation to v1p13p1
add 33c85e1 [BEAM-6268] Ignore failed HadoopFormatIOCassandraTest (#7325)
add 8850ad7 Python 3 port io.filesystem module
add 04a647d Merge pull request #7318 from RobbeSneyders/filesystem
add 7217b47 [BEAM-6273] Update dependencies pages with 2.9.0
add 230c7c9 Merge pull request #7327: [BEAM-6273] Update dependencies
pages with 2.9.0
add 9a51eb3 [BEAM-6179] Fixing itterable comparison
add d9e318c Fix whitespace.
add bed5747 Merge pull request #7313 from angoenka/fix_bundle_estimation
add 7248602 [BEAM-6262] KinesisIO - gracefully shutdown executor service
add e9d51ec Merge pull request #7315: [BEAM-6262] KinesisIO - gracefully
shutdown executor service
add 5d305af Fix performance regression.
add c039440 Merge pull request #7331: [BEAM-6276] Fix performance
regression.
add 6c39b61 Adds a link to release notes for Beam 2.9.0.
add 7c8f7ba Merge pull request #7329: Adds a link to release notes for
Beam 2.9.0
add 1e41220 [BEAM-6165] Send metrics to Flink in portable Flink runner
(#7183)
add e07374a [BEAM-5334] Remove unused 'language' argument
add 4290921 Merge pull request #7264 from udim/perf-tests
add 9d4302d [BEAM-5539] Beam Dependency Update Request:
google-cloud-pubsub
add 0644c55 Merge pull request #7268 from ihji/upgrade_pubsub
add b44ecb4 [BEAM-6286] Add SamzaRunner profile to mvn archetype
add fb4b151 Merge pull request #7335: [BEAM-6286] Add SamzaRunner profile
to mvn archetype
add 8d611b4 [BEAM-6212] Add MongoDbIO ordered option
add 094586d Merge pull request #7256: [BEAM-6212] Add MongoDbIO ordered
option
add 7302aef Update Slack invitation on #general vs #beam channel
add 845de99 [BEAM-6283] Convert PortableStateExecutionTest and
PortableExecutionTest to using PAssert
add f1e339b [BEAM-6295] Fix versions in 2.8.0 Java dependencies table
add 941cb27 Merge pull request #7341 from melap/dependencies
add e6e85ed Reimplement GCS copies with rewrites.
add b196397 Merge pull request #7050: [BEAM-5959] Reimplement GCS copies
with rewrites.
add 24aa20f Add toplevel :sqlPostCommit gradle command
add 0f3560a Add Jenkins job to run :sqlPostCommit
add 0b88bca Merge pull request #7338: [BEAM-6288] Add SQL postcommit
add 80b0c6a [BEAM-6295] Fix versions in 2.7.0 Java dependencies table
add 76b180b Merge pull request #7342 from melap/dependencies
add 1c2d631 Update JUnit
add ad43619 Merge pull request #7344: [BEAM-6299] Update JUnit to fix bug
with parameterized tests
add 2f5ba05 Put generated getter/setter/creator classes in the same
package as the class they are modified.
add a02d884 Merge pull request #7345: [BEAM-6300] Put generated
getter/setter/creator classes in the same package as the class they access
add 8bdbb33 Add schema support to AvroIO and PubsubIO. For
backwards-compatibility reasons, Beam schema support must be explicitly enabled
in these sources.
add 4f23004 Remove unneeded @Rule.
add 2681c25 Merge pull request #7290: [BEAM-4454] Support avro schema
inference in sources
add a90dabf [website] Point Slack link to #beam channel instead of
#general
add 9460fee Merge pull request #7346: [website] Point Slack link to #beam
channel instead of #general
add 77791da [BEAM-6239] Add session side input join to Nexmark
add ac8c956 Merge pull request #7287: [BEAM-6239] Add session side input
join to Nexmark
add 0ad4a5d [BEAM-6244] Restore updateProducerProperties
add 9b0d8fb [BEAM-6244] Restore validate
add e636294 Merge pull request #7343: [BEAM-6244] KafkaIO: keep
KafkaIO.Write compatibility with 2.9.0
add 45a61e4 BEAM-6306 Upgrade Jackson to version 2.9.8
add ba01b8e Merge pull request #7352: [BEAM-6306] Upgrade Jackson to
version 2.9.8
add 4ca3cf0 Upgrade to Calcite 1.18
add 718aef7 Merge pull request #7209 from apilloud/upgrade
add 8588d52 Add time usage in seconds for staging files.
add 41eeb39 Merge pull request #7336: Add time usage in seconds for
staging files.
add 6439fb1 Python 3 port io.filesystemio module
add d5638e7 Add apache_beam.io.localfilesystem_test to python 3 test suite
add 7710391 Merge pull request #7326: [BEAM-5315] [BEAM-5627] Python 3
port io.filesystemio module
add be7549c [BEAM-6287] pyarrow is not supported on Windows Python 2
add dc01009 Merge pull request #7337: [BEAM-6287] pyarrow is not
supported on Windows Python 2
add bac909b Treat VarInt encoding as a Beam primitive encoding in
Dataflow runner (#7351)
add f720985 [BEAM-6110] For SQL CoGBK-based join use EARLIEST output
timestamp
add f190152 Merge pull request #7115 from
kennknowles/sql-join-cogbk-timestamps
add ffec485 Flink 1.5.6 upgrade (#7322)
add 671ed3f Update data source for syncing jobs from Jenkins.
add 8f50ac2 Update deployment versions.
add 498b186 Merge pull request #7364 from Ardagan/FixBMetrics
add e413099 [BEAM-4726] Add arity specialization for calling and returns.
add a47b697 Merge pull request #7355 from lostluck/arity
add 643e562 Enforce the checkstyle IllegalThrows rule for throwing Error
+ RuntimeException
add 680d911 Merge pull request #7259 from coheigea/illegal_throws
add 100c561 [BEAM-5918] Fix CastTest
add d7c64e7 Merge pull request #7372: [BEAM-5918] Fix CastTest
add 25865d3 [BEAM-5467] Increase test timeout for portable
ValidatesRunner tests
add d1384b9 Merge pull request #7376: [BEAM-5467] Increase test timeout
for portable ValidatesRunner tests
add 4e8a07b [BEAM-6294] Use Flink rebalance for shuffle.
add 9f2eb34 Merge pull request #7360 [BEAM-6294] Use Flink rebalance for
shuffle.
add 2481ee6 Disable BigQueryIO validation since datasets and tables are
created during runtime.
add 3a0f70e Merge pull request #7368 from boyuanzz/fix_bq
add eab6759 [BEAM-4725] Use unsafe to avoid small allocations to the heap.
add 8f38b46 Merge pull request #7357 from lostluck/smallbuf
add 54d3857 [BEAM-6325] Cast cross compile output from []byte to string
for printing
add 7c9babd Merge pull request #7375 from lostluck/cast
add fb7ae4f [BEAM-5112] Generate code for BeamCalcRel DoFn
add 1ad4aff Remove Beam Interpreter
add e398bee Merge pull request #6417 from apilloud/codegen
add d3a38f5 [BEAM-6316] Fix container image name for PreCommit
PortableWordCount
add 9ad1074 Merge pull request #7377: [BEAM-6316] Fix container image
name for PreCommit PortableWordCount
add 0d50a17 Fix go runtime break
add ca4defe Merge pull request #7379 from lostluck/fixbreak
add ec6384a [BEAM-6329] Address synchronization issue for portable timers
(#7359)
add 1a4db4b [BEAM-5386] Prevent CheckpointMarks from not getting
acknowledged
add ea275e4 [BEAM-5386] Assert that source thread enters sleep instead of
terminating
add f56c86f Merge pull request #7349: [BEAM-5386] Prevent CheckpointMarks
from not getting acknowledged
add 926361b [BEAM-5386] Move assertion out of finally block to not
swallow original exception
add 3b8abca Upgrade vendored gRPC artifact version to 0.2
add 15aa88d Merge pull request #7328: [BEAM-6056] Upgrade vendored gRPC
artifact version to 0.2
add 14781c7 [BEAM-6056] Source vendored grpc dependency from Maven central
add a25b64d Merge pull request #7388: [BEAM-6056] Source vendored grpc
dependency from Maven central
add 095870f Python 3 port io.range_trackers
add 359ddb9 Add io.restriction_trackers_test to Python 3 test suite
add bca5c60 Merge pull request #7358 from RobbeSneyders/trackers
add 5ce0933 Updates release validating to run LeaderBoard example using
Dataflow Streaming Engine
add 5dd597e Merge pull request #7365: [BEAM-6249] Adds an Streaming
Engine based test to release validation
add 5cdf3a7 [BEAM-5315] Python 3 port io.source* and io.concat_source*
modules (#7383)
add 4b039e4 [BEAM-5315] Python 3 port io.filebased_* modules (#7386)
add fc482f1 [BEAM-5959] Add performance testing for writing many files
add 41dd6e1 Merge pull request #7266 from udim/cmek-perf
add a24b1af Move org.apache.beam.runners.samza.util.Base64Serializer to
org.apache.beam.runners.core.serialization.Base64Serializer to be used by other
runners
add 3b8ae00 Fix visibility of deserialize method
add 0783779 Add missing package-info
add 4660895 Merge pull request #7384 from echauchot/Base64Serializer
add c4590a0 split SerializablePipelineOptions into serialization utils
and instance class.
add 5130bcb Merge pull request #7385 from
echauchot/exposeSerializationSerializablePipelineOptions
add a404cee Add paddedcell fix to spotlessJava rules.
add c148c35 Merge pull request #7390: [BEAM-6339] Add paddedcell fix to
spotlessJava rules.
add c028ebc Upgrade html-proofer and dependencies to latest
add 07c279a Remove broken links to datatorrent.com
add b09e721 Fix pydoc link to GoogleCloudOptions
add fd5e321 Remove broken link to atrato.io
add a79ef89 Fix link to internal anchor
add 5466ac0 Remove stale exclusions from HTML link checker.
add a2986cc Merge pull request #7393: [BEAM-5662] Clean up website
html-proofer config
add b02f79f Disable UsesMetricsPusher tests for direct-runner
add f74c979 Fix SplittableDoFnTest#testBoundedness
add 459e730 [BEAM-6352] Ignore tests using Watch PTransform
add 26c73ef [BEAM-6353] Fix TFRecordIOTest
add 92a6c23 [BEAM-6354] Add timeout and ignore hanging tests
add 55ffd97 Add :beam-runners-direct-java:needsRunnerTests to
javaPreCommit
add c591727 Merge pull request #7374: Add
:beam-runners-direct-java:needsRunnerTests to javaPreCommit
add 3948595 [BEAM-5959] Reorder methods according to convention
add 5716dba Merge pull request #7403 from udim/cmek-perf
add 5212b71 [BEAM-6030] Split metrics related options out of
PipelineOptions
add 185cb1a [BEAM-6030] Add Experimental label on MetricsOptions
add bd80118 Merge pull request #7400 from
echauchot/BEAM-6030-metrics-sinks-pipelineOptions
new ce39f93 Add an empty spark-structured-streaming runner project
targeting spark 2.4.0
new 737af2f Fix missing dep
new 018c773 Add SparkPipelineOptions
new 1c97788 Start pipeline translation
new abf4b46 Add global pipeline translation structure
new 28a9422 Add nodes translators structure
new 3a743c2 Wire node translators with pipeline translator
new 051e8dc Renames: better differenciate pipeline translator for
transform translator
new 6695d64 Organise methods in PipelineTranslator
new ec9d634 Initialise BatchTranslationContext
new 476cae8 Refactoring: -move batch/streaming common translation visitor
and utility methods to PipelineTranslator -rename batch dedicated classes to
Batch* to differentiate with their streaming counterparts -Introduce
TranslationContext for common batch/streaming components
new ce484e9 Make transform translation clearer: renaming, comments
new 0033f89 Improve javadocs
new 26f2e4b Move SparkTransformOverrides to correct package
new 4777e22 Move common translation context components to superclass
new 0cfa70d apply spotless for e-formatting
new 91f9ef5 Make codestyle and firebug happy
new 901a1ac Add TODOs
new 2ccccdd Post-pone batch qualifier in all classes names for readability
new b37da3e Add precise TODO for multiple TransformTranslator per
transform URN
new bbf583c Added SparkRunnerRegistrar
new 866ef13 Add basic pipeline execution. Refactor translatePipeline() to
return the translationContext on which we can run startPipeline()
new 7a645e1 Create PCollections manipulation methods
new 31fb182 Create Datasets manipulation methods
new 286d7f3 Add Flatten transformation translator
new 9e6fc2c Add primitive GroupByKeyTranslatorBatch implementation
new 57ce2d1 Use Iterators.transform() to return Iterable
new 4f150da Implement read transform
new 1ec9356 update TODO
new ebbab69 Apply spotless
new d531bb5 start source instanciation
new a3a87b4 Improve exception flow
new b7283d7 Improve type enforcement in ReadSourceTranslator
new e9ac3c3 Experiment over using spark Catalog to pass in Beam Source
through spark Table
new 0452733 Add source mocks
new 8cdc20f fix mock, wire mock in translators and create a main test.
new 1060121 Use raw WindowedValue so that spark Encoders could work
(temporary)
new 1184022 clean deps
new 49ee259 Move DatasetSourceMock to proper batch mode
new 340991e Run pipeline in batch mode or in streaming mode
new 1ca4192 Split batch and streaming sources and translators
new 4e0f7a0 Use raw Encoder<WindowedValue> also in regular
ReadSourceTranslatorBatch
new 758c1ce Cleaning
new 92a104e Add ReadSourceTranslatorStreaming
new 2f5bdd3 Move Source and translator mocks to a mock package.
new 1cea29d Pass Beam Source and PipelineOptions to the spark DataSource
as serialized strings
new 92c94b1 Refactor DatasetSource fields
new d1b549e Wire real SourceTransform and not mock and update the test
new 878ff4e Add missing 0-arg public constructor
new 6392179 Apply spotless
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (3533779)
\
N -- N -- N refs/heads/spark-runner_structured-streaming
(6392179)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 50 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/PULL_REQUEST_TEMPLATE.md | 4 +-
.gitignore | 1 +
.test-infra/jenkins/CommonJobProperties.groovy | 2 +-
.../job_PerformanceTests_FileBasedIO_IT.groovy | 16 +
...job_PerformanceTests_FileBasedIO_IT_HDFS.groovy | 13 +
.../jenkins/job_PerformanceTests_Python.groovy | 2 +-
...GradleBuild.groovy => job_PostCommit_Go.groovy} | 3 +-
...adleBuild.groovy => job_PostCommit_Java.groovy} | 3 +-
.../job_PostCommit_Java_Nexmark_Flink.groovy | 10 +-
...y => job_PostCommit_Java_PortabilityApi.groovy} | 3 +-
...job_PostCommit_Java_ValidatesRunner_Apex.groovy | 5 +-
...PostCommit_Java_ValidatesRunner_Dataflow.groovy | 5 +-
...nner_DataflowPortabilityExecutableStage.groovy} | 11 +-
...ob_PostCommit_Java_ValidatesRunner_Flink.groovy | 3 +-
...PostCommit_Java_ValidatesRunner_Gearpump.groovy | 5 +-
..._ValidatesRunner_PortabilityApi_Dataflow.groovy | 3 +-
...ob_PostCommit_Java_ValidatesRunner_Samza.groovy | 3 +-
...ob_PostCommit_Java_ValidatesRunner_Spark.groovy | 5 +-
...radleBuild.groovy => job_PostCommit_SQL.groovy} | 11 +-
.../jenkins/job_PreCommit_Portable_Python.groovy | 18 +-
...t.groovy => job_Release_NightlySnapshot.groovy} | 3 +-
.../job_beam_PerformanceTests_Analysis.groovy | 2 +
.test-infra/metrics/OWNERS | 3 +
.test-infra/metrics/beamgrafana-deploy.yaml | 2 +-
.../dashboards/stability_critical_jobs_status.json | 4 +-
.test-infra/metrics/sync/jenkins/syncjenkins.py | 3 +-
README.md | 6 +-
build.gradle | 10 +
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 281 ++-
.../org/apache/beam/gradle/GrpcVendoring.groovy | 8 +-
.../org/apache/beam/gradle/VendorJavaPlugin.groovy | 4 +
examples/java/build.gradle | 2 +-
.../beam/examples/complete/TrafficRoutes.java | 3 +-
.../beam/examples/complete/game/UserScore.java | 36 +-
.../beam/examples/cookbook/FilterExamples.java | 3 +-
.../beam/examples/cookbook/TriggerExample.java | 3 +-
.../examples/complete/game/LeaderBoardTest.java | 2 +-
.../beam/examples/complete/game/UserScoreTest.java | 31 +-
.../beam/examples/cookbook/JoinExamplesTest.java | 38 +-
.../beam/examples/cookbook/TriggerExampleTest.java | 23 +-
model/fn-execution/build.gradle | 4 +-
.../fn-execution/src/main/proto/beam_fn_api.proto | 300 ++-
model/job-management/build.gradle | 4 +-
model/pipeline/build.gradle | 2 +-
.../pipeline/src/main/proto/beam_runner_api.proto | 40 +
.../src/main/groovy/MobileGamingCommands.groovy | 20 +-
.../main/groovy/mobilegaming-java-dataflow.groovy | 100 +-
release/src/main/scripts/cut_release_branch.sh | 4 +-
.../operators/ApexProcessFnOperator.java | 35 +-
.../core/construction/ArtifactServiceStager.java | 6 +-
.../beam/runners/core/construction/BeamUrns.java | 2 +-
.../core/construction/CoderTranslation.java | 2 +-
.../core/construction/CombineTranslation.java | 2 +-
.../CreatePCollectionViewTranslation.java | 2 +-
.../core/construction/DisplayDataTranslation.java | 4 +-
.../runners/core/construction/Environments.java | 31 +-
.../construction/ExecutableStageTranslation.java | 90 +
.../construction/PCollectionViewTranslation.java | 2 +-
.../core/construction/PTransformTranslation.java | 6 +
.../core/construction/ParDoTranslation.java | 4 +-
...java => PipelineOptionsSerializationUtils.java} | 34 +-
.../construction/PipelineOptionsTranslation.java | 6 +-
.../runners/core/construction/ReadTranslation.java | 4 +-
.../construction/SerializablePipelineOptions.java | 26 +-
.../runners/core/construction/SplittableParDo.java | 2 +
.../construction/SplittableParDoNaiveBounded.java | 4 +-
.../core/construction/TestStreamTranslation.java | 2 +-
.../core/construction/WindowIntoTranslation.java | 2 +-
.../construction/WindowingStrategyTranslation.java | 8 +-
.../core/construction/WriteFilesTranslation.java | 2 +-
.../graph/GreedyPCollectionFusers.java | 30 +-
.../core/construction/graph/QueryablePipeline.java | 8 +-
.../construction/ArtifactServiceStagerTest.java | 6 +-
.../runners/core/construction/CommonCoderTest.java | 2 +-
.../ExecutableStageTranslationTest.java | 120 ++
.../InMemoryArtifactStagerService.java | 2 +-
.../PipelineOptionsTranslationTest.java | 6 +-
.../construction/WindowIntoTranslationTest.java | 2 +-
.../construction/graph/ProtoOverridesTest.java | 2 +-
.../runners/core/LateDataDroppingDoFnRunner.java | 17 +-
.../apache/beam/runners/core/SideInputHandler.java | 3 +-
.../core/SplittableParDoViaKeyedWorkItems.java | 7 +-
.../runners/core/metrics/DistributionCell.java | 5 +
.../runners/core/metrics/MetricsContainerImpl.java | 19 +
.../beam/runners/core/metrics/MetricsPusher.java | 6 +-
.../beam/runners/core/metrics/NoOpMetricsSink.java | 4 +-
.../core/metrics/SimpleMonitoringInfoBuilder.java | 219 +++
.../core/serialization}/Base64Serializer.java | 4 +-
.../runners/core/serialization}/package-info.java | 4 +-
.../beam/runners/core/ReduceFnRunnerTest.java | 2 +-
.../runners/core/SplittableParDoProcessFnTest.java | 20 +-
.../runners/core/metrics/MetricsPusherTest.java | 7 +-
.../metrics/SimpleMonitoringInfoBuilderTest.java | 87 +
.../beam/runners/core/metrics/TestMetricsSink.java | 4 +-
runners/direct-java/build.gradle | 35 +-
.../runners/direct/ParDoMultiOverrideFactory.java | 32 +-
.../runners/direct/WindowEvaluatorFactory.java | 3 +-
.../runners/direct/portable/ReferenceRunner.java | 15 +-
.../direct/portable/WindowEvaluatorFactory.java | 3 +-
.../LocalFileSystemArtifactRetrievalService.java | 6 +-
.../LocalFileSystemArtifactStagerService.java | 8 +-
.../runners/direct/portable/job/PreparingJob.java | 2 +-
.../portable/job/ReferenceRunnerJobService.java | 6 +-
.../beam/runners/direct/DirectRunnerTest.java | 12 +-
.../direct/UnboundedReadEvaluatorFactoryTest.java | 5 +-
.../runners/direct/WindowEvaluatorFactoryTest.java | 3 +-
.../direct/portable/ReferenceRunnerTest.java | 3 +-
.../portable/RemoteStageEvaluatorFactoryTest.java | 5 +-
...ocalFileSystemArtifactRetrievalServiceTest.java | 4 +-
.../LocalFileSystemArtifactStagerServiceTest.java | 10 +-
.../UnsupportedArtifactRetrievalServiceTest.java | 2 +-
.../job/ReferenceRunnerJobServiceTest.java | 4 +-
runners/extensions-java/metrics/build.gradle | 2 +-
.../extensions/metrics/MetricsGraphiteSink.java | 4 +-
.../extensions/metrics/MetricsHttpSink.java | 4 +-
.../metrics/MetricsGraphiteSinkTest.java | 6 +-
.../extensions/metrics/MetricsHttpSinkTest.java | 6 +-
{model/pipeline => runners/flink/1.6}/build.gradle | 22 +-
.../flink/1.6/job-server-container}/build.gradle | 10 +-
.../flink/1.6/job-server}/build.gradle | 19 +-
runners/flink/build.gradle | 129 +-
.../flink/{build.gradle => flink_runner.gradle} | 34 +-
runners/flink/job-server-container/Dockerfile | 6 +-
runners/flink/job-server-container/build.gradle | 38 +-
...ld.gradle => flink_job_server_container.gradle} | 26 +-
runners/flink/job-server/build.gradle | 80 +-
.../{build.gradle => flink_job_server.gradle} | 79 +-
.../FlinkBatchPortablePipelineTranslator.java | 71 +-
.../runners/flink/FlinkExecutionEnvironments.java | 173 +-
.../beam/runners/flink/FlinkJobInvocation.java | 88 +-
.../apache/beam/runners/flink/FlinkJobInvoker.java | 3 +-
.../beam/runners/flink/FlinkJobServerDriver.java | 25 +-
.../flink/FlinkPipelineExecutionEnvironment.java | 15 +-
.../beam/runners/flink/FlinkPipelineOptions.java | 22 +
.../flink/FlinkPortablePipelineTranslator.java | 19 +-
.../org/apache/beam/runners/flink/FlinkRunner.java | 5 +-
.../flink/FlinkStreamingPipelineTranslator.java | 70 +
.../FlinkStreamingPortablePipelineTranslator.java | 49 +-
.../flink/FlinkStreamingTransformTranslators.java | 26 +-
.../runners/flink/FlinkTransformOverrides.java | 24 +-
....java => PipelineTranslationModeOptimizer.java} | 26 +-
.../beam/runners/flink/metrics/FileReporter.java | 75 +
.../flink/metrics/FlinkMetricContainer.java | 97 +-
.../apache/beam/runners/flink/metrics/Metrics.java | 56 +
.../FlinkDefaultExecutableStageContext.java | 3 +
.../functions/FlinkExecutableStageFunction.java | 23 +-
.../utils/FlinkPipelineTranslatorUtils.java | 2 +-
.../runners/flink/translation/utils/NoopLock.java | 72 +
.../wrappers/streaming/DoFnOperator.java | 3 +-
.../streaming/ExecutableStageDoFnOperator.java | 144 +-
.../streaming/io/UnboundedSourceWrapper.java | 46 +-
.../state/FlinkKeyGroupStateInternals.java | 3 +-
.../streaming/state/FlinkStateInternals.java | 56 +-
.../flink/FlinkExecutionEnvironmentsTest.java | 252 ++-
.../runners/flink/FlinkJobServerDriverTest.java | 27 +-
.../FlinkPipelineExecutionEnvironmentTest.java | 26 +
.../FlinkStreamingTransformTranslatorsTest.java | 238 +++
.../runners/flink/FlinkTransformOverridesTest.java | 116 ++
.../beam/runners/flink/PipelineOptionsTest.java | 3 +
.../PipelineTranslationModeOptimizerTest.java | 63 +
.../beam/runners/flink/PortableExecutionTest.java | 95 +-
.../runners/flink/PortableStateExecutionTest.java | 194 +-
.../runners/flink/PortableTimersExecutionTest.java | 36 +-
.../flink/metrics/FlinkMetricContainerTest.java | 134 ++
.../flink/streaming/BoundedSourceRestoreTest.java | 1 +
.../streaming/ExecutableStageDoFnOperatorTest.java | 2 +-
.../flink/streaming/GroupByWithNullValuesTest.java | 92 +
.../FlinkPipelineTranslatorUtilsTest.java | 44 +
.../FlinkDefaultExecutableStageContextTest.java | 2 +-
.../FlinkExecutableStageFunctionTest.java | 2 +-
.../wrappers/streaming/io}/TestCountingSource.java | 17 +-
.../streaming/io}/UnboundedSourceWrapperTest.java | 150 +-
runners/google-cloud-dataflow-java/build.gradle | 64 +-
.../examples-streaming/build.gradle | 2 +-
.../examples/build.gradle | 6 +-
.../beam/runners/dataflow/DataflowPipelineJob.java | 6 +-
.../dataflow/DataflowPipelineTranslator.java | 2 +-
.../beam/runners/dataflow/DataflowRunner.java | 3 +
.../dataflow/options/DataflowPipelineOptions.java | 8 +
.../runners/dataflow/util/CloudObjectKinds.java | 1 +
.../dataflow/util/CloudObjectTranslators.java | 6 +-
.../beam/runners/dataflow/util/GcsStager.java | 4 +-
.../beam/runners/dataflow/util/PackageUtil.java | 9 +-
.../beam/runners/dataflow/util/GCSUploadMain.java} | 29 +-
.../google-cloud-dataflow-java/worker/build.gradle | 26 +-
.../worker/legacy-worker/build.gradle | 114 +-
.../dataflow/worker/BatchDataflowWorker.java | 59 +-
.../worker/BeamFnMapTaskExecutorFactory.java | 113 +-
.../runners/dataflow/worker/ByteStringCoder.java | 2 +-
.../worker/DataflowMapTaskExecutorFactory.java | 13 +-
.../dataflow/worker/DataflowOperationContext.java | 2 +-
.../dataflow/worker/DataflowRunnerHarness.java | 23 +-
.../worker/DataflowWorkerHarnessHelper.java | 15 +-
.../dataflow/worker/DeltaDistributionCell.java | 5 +
.../runners/dataflow/worker/ExperimentContext.java | 6 +-
...FetchAndFilterStreamingSideInputsOperation.java | 6 +-
.../dataflow/worker/FnApiWindowMappingFn.java | 12 +-
.../worker/GroupAlsoByWindowParDoFnFactory.java | 2 +-
.../worker/IntrinsicMapTaskExecutorFactory.java | 15 +-
.../dataflow/worker/IsmSideInputReader.java | 3 +-
.../worker/MetricTrackingWindmillServerStub.java | 2 +-
.../beam/runners/dataflow/worker/PubsubSink.java | 2 +-
.../beam/runners/dataflow/worker/ReaderCache.java | 2 +-
...HarnessCoderCloudObjectTranslatorRegistrar.java | 2 -
.../dataflow/worker/SdkHarnessRegistries.java | 16 +-
.../dataflow/worker/SdkHarnessRegistry.java | 9 +-
.../beam/runners/dataflow/worker/StateFetcher.java | 2 +-
.../dataflow/worker/StreamingDataflowWorker.java | 53 +-
.../worker/StreamingModeExecutionContext.java | 30 +-
.../dataflow/worker/StreamingSideInputFetcher.java | 4 +-
.../dataflow/worker/WindmillNamespacePrefix.java | 2 +-
.../beam/runners/dataflow/worker/WindmillSink.java | 2 +-
.../dataflow/worker/WindmillStateCache.java | 2 +-
.../dataflow/worker/WindmillStateInternals.java | 2 +-
.../dataflow/worker/WindmillStateReader.java | 2 +-
.../dataflow/worker/WindmillTimerInternals.java | 2 +-
.../dataflow/worker/WorkItemStatusClient.java | 27 +-
.../dataflow/worker/WorkerCustomSources.java | 2 +-
.../FixMultiOutputInfosOnParDoInstructions.java | 8 +-
.../dataflow/worker/fn/BeamFnControlService.java | 2 +-
.../runners/dataflow/worker/fn/ServerFactory.java | 229 ---
.../dataflow/worker/fn/SocketAddressFactory.java | 68 -
.../worker/fn/control/BeamFnMapTaskExecutor.java | 262 ++-
.../fn/control/ProcessRemoteBundleOperation.java | 105 +
.../control/RegisterAndProcessBundleOperation.java | 38 +-
.../worker/fn/data/BeamFnDataGrpcService.java | 14 +-
.../fn/data/RemoteGrpcPortReadOperation.java | 8 +-
.../fn/data/RemoteGrpcPortWriteOperation.java | 9 +-
.../worker/fn/logging/BeamFnLoggingService.java | 4 +-
.../fn/stream/ServerStreamObserverFactory.java | 6 +-
.../graph/CloneAmbiguousFlattensFunction.java | 9 +-
...java => CreateExecutableStageNodeFunction.java} | 345 +---
.../graph/CreateRegisterFnOperationFunction.java | 49 +-
...nsertFetchAndFilterStreamingSideInputNodes.java | 7 +-
.../worker/graph/LengthPrefixUnknownCoders.java | 7 +-
.../worker/graph/MapTaskToNetworkFunction.java | 11 +-
.../beam/runners/dataflow/worker/graph/Nodes.java | 43 +-
.../worker/graph/RegisterNodeFunction.java | 30 +-
.../common/worker/BatchingShuffleEntryReader.java | 3 +-
.../worker/util/common/worker/MapTaskExecutor.java | 2 +-
.../worker/windmill/DirectStreamObserver.java | 4 +-
.../windmill/ForwardingClientResponseObserver.java | 6 +-
.../worker/windmill/GrpcWindmillServer.java | 32 +-
.../worker/windmill/StreamObserverFactory.java | 4 +-
.../runners/dataflow/harness/test/TestStreams.java | 4 +-
.../runners/dataflow/worker/ConcatReaderTest.java | 3 +-
.../runners/dataflow/worker/DataflowMatchers.java | 2 +-
.../worker/DataflowWorkerHarnessHelperTest.java | 2 +-
.../dataflow/worker/FnApiWindowMappingFnTest.java | 4 +-
.../dataflow/worker/GroupingShuffleReaderTest.java | 2 +-
.../IntrinsicMapTaskExecutorFactoryTest.java | 37 +-
.../dataflow/worker/IsmSideInputReaderTest.java | 3 +-
.../runners/dataflow/worker/PubsubReaderTest.java | 2 +-
.../runners/dataflow/worker/PubsubSinkTest.java | 2 +-
.../runners/dataflow/worker/ReaderCacheTest.java | 2 +-
.../dataflow/worker/ShuffleReaderFactoryTest.java | 3 +-
.../runners/dataflow/worker/StateFetcherTest.java | 2 +-
.../worker/StreamingDataflowWorkerTest.java | 6 +-
.../worker/StreamingGroupAlsoByWindowFnsTest.java | 2 +-
...reamingGroupAlsoByWindowsReshuffleDoFnTest.java | 2 +-
.../worker/StreamingModeExecutionContextTest.java | 16 +-
.../worker/StreamingSideInputDoFnRunnerTest.java | 2 +-
.../worker/StreamingSideInputFetcherTest.java | 2 +-
.../dataflow/worker/WindmillKeyedWorkItemTest.java | 2 +-
.../worker/WindmillReaderIteratorBaseTest.java | 2 +-
.../dataflow/worker/WindmillStateCacheTest.java | 2 +-
.../worker/WindmillStateInternalsTest.java | 2 +-
.../dataflow/worker/WindmillStateReaderTest.java | 4 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 4 +-
...FixMultiOutputInfosOnParDoInstructionsTest.java | 20 +-
.../worker/fn/BeamFnControlServiceTest.java | 13 +-
.../dataflow/worker/fn/ServerFactoryTest.java | 244 ---
.../worker/fn/SocketAddressFactoryTest.java | 55 -
.../fn/control/BeamFnMapTaskExecutorTest.java | 219 ++-
.../RegisterAndProcessBundleOperationTest.java | 96 +-
.../SingularProcessBundleProgressTrackerTest.java | 6 +-
.../worker/fn/data/BeamFnDataGrpcServiceTest.java | 32 +-
.../fn/data/RemoteGrpcPortReadOperationTest.java | 14 +-
.../fn/data/RemoteGrpcPortWriteOperationTest.java | 16 +-
.../fn/logging/BeamFnLoggingServiceTest.java | 12 +-
.../fn/stream/ServerStreamObserverFactoryTest.java | 4 +-
.../graph/CloneAmbiguousFlattensFunctionTest.java | 7 +-
.../CreateRegisterFnOperationFunctionTest.java | 6 +-
.../graph/DeduceFlattenLocationsFunctionTest.java | 2 +-
.../graph/DeduceNodeLocationsFunctionTest.java | 4 +-
...tFetchAndFilterStreamingSideInputNodesTest.java | 9 +-
.../graph/LengthPrefixUnknownCodersTest.java | 2 +-
.../worker/graph/MapTaskToNetworkFunctionTest.java | 25 +-
.../runners/dataflow/worker/graph/NodesTest.java | 16 +-
.../RemoveFlattenInstructionsFunctionTest.java | 74 +-
.../ReplacePgbkWithPrecombineFunctionTest.java | 2 +-
.../logging/DataflowWorkerLoggingHandlerTest.java | 2 +-
.../util/common/worker/ReadOperationTest.java | 6 +-
.../worker/windmill/GrpcWindmillServerTest.java | 10 +-
.../worker/windmill/build.gradle | 2 +-
.../apache/beam/runners/fnexecution/FnService.java | 6 +-
.../GrpcContextHeaderAccessorProvider.java | 16 +-
.../beam/runners/fnexecution/GrpcFnServer.java | 19 +-
.../fnexecution/InProcessServerFactory.java | 39 +-
.../beam/runners/fnexecution/ServerFactory.java | 192 +-
.../BeamFileSystemArtifactRetrievalService.java | 10 +-
.../BeamFileSystemArtifactStagingService.java | 10 +-
.../control/DefaultJobBundleFactory.java | 4 +-
.../fnexecution/control/FnApiControlClient.java | 6 +-
.../control/FnApiControlClientPoolService.java | 2 +-
.../control/ProcessBundleDescriptors.java | 2 +-
.../SingleEnvironmentInstanceJobBundleFactory.java | 13 +-
.../runners/fnexecution/data/GrpcDataService.java | 11 +-
...actory.java => ExternalEnvironmentFactory.java} | 116 +-
.../environment/ProcessEnvironmentFactory.java | 2 +-
.../environment/StaticRemoteEnvironment.java | 64 +
.../StaticRemoteEnvironmentFactory.java | 70 +
.../jobsubmission/InMemoryJobService.java | 10 +-
.../fnexecution/jobsubmission/JobInvoker.java | 2 +-
.../fnexecution/jobsubmission/JobPreparation.java | 2 +-
.../fnexecution/logging/GrpcLoggingService.java | 2 +-
.../runners/fnexecution/provisioning/JobInfo.java | 2 +-
.../provisioning/StaticGrpcProvisionService.java | 2 +-
.../splittabledofn/SDFFeederViaStateAndTimers.java | 4 +-
.../fnexecution/state/GrpcStateService.java | 6 +-
.../fnexecution/state/StateRequestHandlers.java | 2 +-
.../GrpcContextHeaderAccessorProviderTest.java | 24 +-
.../runners/fnexecution/ServerFactoryTest.java | 48 +-
.../BeamFileSystemArtifactServicesTest.java | 8 +-
.../control/DefaultJobBundleFactoryTest.java | 4 +-
.../control/FnApiControlClientPoolServiceTest.java | 4 +-
.../control/FnApiControlClientTest.java | 2 +-
.../fnexecution/control/RemoteExecutionTest.java | 220 ++-
...gleEnvironmentInstanceJobBundleFactoryTest.java | 3 +-
.../fnexecution/data/GrpcDataServiceTest.java | 8 +-
.../jobsubmission/InMemoryJobServiceTest.java | 4 +-
.../logging/GrpcLoggingServiceTest.java | 6 +-
.../StaticGrpcProvisionServiceTest.java | 10 +-
.../fnexecution/state/GrpcStateServiceTest.java | 4 +-
.../apache/beam/runners/local/StructuralKey.java | 14 +-
runners/reference/java/build.gradle | 2 +
.../runners/reference/ExternalWorkerService.java | 87 +
.../reference/JobServicePipelineResult.java | 10 +-
.../beam/runners/reference/PortableRunner.java | 34 +-
.../runners/reference/testing/TestJobService.java | 2 +-
.../beam/runners/reference/PortableRunnerTest.java | 4 +-
runners/reference/job-server/build.gradle | 1 +
runners/samza/build.gradle | 2 +-
.../org/apache/beam/runners/samza/SamzaRunner.java | 2 +-
.../runners/samza/adapter/BoundedSourceSystem.java | 2 +-
.../samza/adapter/UnboundedSourceSystem.java | 2 +-
.../apache/beam/runners/samza/runtime/DoFnOp.java | 14 +-
.../beam/runners/samza/runtime/GroupByKeyOp.java | 2 +-
.../samza/runtime/SamzaDoFnInvokerRegistrar.java | 21 +-
.../samza/runtime/SamzaStoreStateInternals.java | 10 +-
.../samza/transforms/GroupWithoutRepartition.java | 60 +
.../samza/transforms/UpdatingCombineFn.java | 19 +-
.../runners/samza/transforms}/package-info.java | 4 +-
.../runners/samza/translation/ConfigBuilder.java | 2 +-
.../runners/samza/translation/ConfigContext.java | 9 +-
.../samza/translation/GroupByKeyTranslator.java | 22 +-
.../translation/ParDoBoundMultiTranslator.java | 17 +
.../runners/samza/translation/ReadTranslator.java | 2 +-
.../samza/translation/SamzaPipelineTranslator.java | 7 +-
.../samza/translation/TranslationContext.java | 47 +-
.../runners/samza/util/PipelineDotRenderer.java | 3 +-
.../translation/TranslationContext.java | 10 +-
.../translation/batch/DatasetSourceBatch.java | 62 +-
.../translation/batch/FlattenTranslatorBatch.java | 2 +-
.../translation/batch/PipelineTranslatorBatch.java | 2 +-
.../batch/ReadSourceTranslatorBatch.java | 34 +-
.../batch/{ => mocks}/DatasetSourceMockBatch.java | 46 +-
.../{ => mocks}/ReadSourceTranslatorMockBatch.java | 22 +-
.../streaming/DatasetSourceStreaming.java | 16 +-
.../streaming/ReadSourceTranslatorStreaming.java | 22 +-
.../spark/structuredstreaming/SourceTest.java | 21 +-
runners/spark/build.gradle | 6 +-
.../org/apache/beam/runners/spark/SparkRunner.java | 4 +-
.../beam/runners/spark/examples/WordCount.java | 4 +-
.../spark/translation/TranslationUtils.java | 5 +-
.../streaming/StreamingTransformTranslator.java | 4 +-
.../runners/spark/ProvidedSparkContextTest.java | 2 +-
.../spark/metrics/SparkMetricsPusherTest.java | 10 +-
sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go | 14 +-
sdks/go/pkg/beam/artifact/gcsproxy/staging.go | 6 +-
sdks/go/pkg/beam/core/graph/coder/int.go | 24 +-
sdks/go/pkg/beam/core/graph/coder/varint.go | 10 +-
sdks/go/pkg/beam/core/graph/fn.go | 11 +
sdks/go/pkg/beam/core/runtime/exec/coder.go | 3 +-
sdks/go/pkg/beam/core/runtime/exec/fn.go | 104 +-
sdks/go/pkg/beam/core/runtime/exec/fn_arity.go | 251 +++
sdks/go/pkg/beam/core/runtime/exec/fn_arity.tmpl | 69 +
sdks/go/pkg/beam/core/runtime/exec/fn_test.go | 242 ++-
sdks/go/pkg/beam/core/runtime/exec/fullvalue.go | 58 +
.../pkg/beam/core/runtime/exec/fullvalue_test.go | 58 +
sdks/go/pkg/beam/core/runtime/init.go | 2 +-
sdks/go/pkg/beam/core/util/ioutilx/read.go | 42 +
.../beam/core/util/ioutilx/{read.go => write.go} | 30 +-
sdks/go/pkg/beam/core/util/reflectx/call.go | 2 +-
sdks/go/pkg/beam/core/util/reflectx/structs.go | 73 +
sdks/go/pkg/beam/forward.go | 4 +-
sdks/go/pkg/beam/io/filesystem/gcs/gcs.go | 70 +-
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 4 +-
.../pkg/beam/runners/dataflow/dataflowlib/stage.go | 4 +-
.../beam/runners/universal/runnerlib/compile.go | 2 +-
sdks/go/pkg/beam/testing/ptest/ptest.go | 43 +-
sdks/go/pkg/beam/util/gcsx/gcs.go | 66 +-
.../util/ioutilx/read.go => util/gcsx/gcs_test.go} | 40 +-
sdks/go/pkg/beam/util/shimx/generate.go | 49 +-
sdks/go/pkg/beam/util/shimx/generate_test.go | 17 +
sdks/go/pkg/beam/util/starcgenx/starcgenx.go | 69 +-
sdks/go/pkg/beam/util/starcgenx/starcgenx_test.go | 4 +-
sdks/go/pkg/beam/x/hooks/perf/perf.go | 68 +-
sdks/java/bom/build.gradle | 122 ++
sdks/java/bom/pom.xml.template | 83 +
sdks/java/build-tools/build.gradle | 2 +-
.../src/main/resources/beam/checkstyle.xml | 10 +
.../java/org/apache/beam/sdk/coders/ListCoder.java | 19 +
.../java/org/apache/beam/sdk/coders/MapCoder.java | 19 +
.../apache/beam/sdk/coders/RowCoderGenerator.java | 4 +-
.../main/java/org/apache/beam/sdk/io/AvroIO.java | 81 +-
.../org/apache/beam/sdk/io/BlockBasedSource.java | 4 +-
.../org/apache/beam/sdk/io/LocalFileSystem.java | 6 +-
.../java/org/apache/beam/sdk/io/TFRecordIO.java | 6 +-
.../main/java/org/apache/beam/sdk/io/TextIO.java | 2 +-
.../org/apache/beam/sdk/metrics/Distribution.java | 2 +
.../java/org/apache/beam/sdk/metrics/Metrics.java | 8 +
.../apache/beam/sdk/metrics/MetricsOptions.java | 83 +
.../apache/beam/sdk/options/PipelineOptions.java | 55 -
.../org/apache/beam/sdk/options/ValueProvider.java | 6 +-
.../apache/beam/sdk/schemas/AvroRecordSchema.java | 53 +
.../apache/beam/sdk/schemas/CachingFactory.java | 55 +
.../java/org/apache/beam/sdk/schemas/Factory.java} | 9 +-
.../beam/sdk/schemas/FieldTypeDescriptors.java | 3 +-
.../apache/beam/sdk/schemas/FieldValueGetter.java | 2 -
.../beam/sdk/schemas/FieldValueGetterFactory.java | 6 +-
.../apache/beam/sdk/schemas/FieldValueSetter.java | 16 -
.../beam/sdk/schemas/FieldValueSetterFactory.java | 6 +-
.../sdk/schemas/FieldValueTypeInformation.java | 224 +++
.../schemas/FieldValueTypeInformationFactory.java} | 12 +-
.../beam/sdk/schemas/FromRowUsingCreator.java | 149 ++
.../sdk/schemas/GetterBasedSchemaProvider.java | 175 +-
.../apache/beam/sdk/schemas/JavaBeanSchema.java | 70 +-
.../apache/beam/sdk/schemas/JavaFieldSchema.java | 42 +-
.../java/org/apache/beam/sdk/schemas/Schema.java | 107 +-
.../schemas/SchemaUserTypeConstructorCreator.java | 44 +
.../beam/sdk/schemas/SchemaUserTypeCreator.java} | 13 +-
.../sdk/schemas/SetterBasedCreatorFactory.java | 57 +
.../beam/sdk/schemas/UserTypeCreatorFactory.java | 14 +-
.../apache/beam/sdk/schemas/transforms/Cast.java | 2 +-
.../apache/beam/sdk/schemas/transforms/Select.java | 7 +-
.../beam/sdk/schemas/utils/AvroByteBuddyUtils.java | 125 ++
.../apache/beam/sdk/schemas/utils/AvroUtils.java | 686 ++++++-
.../beam/sdk/schemas/utils/ByteBuddyUtils.java | 159 +-
...terFactory.java => FieldValueTypeSupplier.java} | 20 +-
.../sdk/schemas/utils/JavaBeanGetterFactory.java | 31 -
.../sdk/schemas/utils/JavaBeanSetterFactory.java | 31 -
.../beam/sdk/schemas/utils/JavaBeanUtils.java | 151 +-
.../apache/beam/sdk/schemas/utils/POJOUtils.java | 211 +-
.../sdk/schemas/utils/PojoValueSetterFactory.java | 31 -
.../beam/sdk/schemas/utils/ReflectUtils.java | 20 +-
.../sdk/schemas/utils/StaticSchemaInference.java | 100 +-
...aflowPortabilityExecutableStageUnsupported.java | 17 +-
.../apache/beam/sdk/testing/UsesSideInputs.java | 16 +-
.../org/apache/beam/sdk/transforms/Contextful.java | 4 +-
.../org/apache/beam/sdk/transforms/Create.java | 9 +-
.../java/org/apache/beam/sdk/transforms/DoFn.java | 49 +-
.../org/apache/beam/sdk/transforms/Filter.java | 23 +-
.../beam/sdk/transforms/FlatMapElements.java | 40 +-
...{SimpleFunction.java => InferableFunction.java} | 49 +-
.../apache/beam/sdk/transforms/MapElements.java | 36 +-
...ializableFunction.java => ProcessFunction.java} | 17 +-
.../beam/sdk/transforms/SerializableFunction.java | 11 +-
.../apache/beam/sdk/transforms/SimpleFunction.java | 38 +-
.../org/apache/beam/sdk/transforms/ToString.java | 8 +-
.../reflect/ByteBuddyDoFnInvokerFactory.java | 6 +-
.../reflect/ByteBuddyOnTimerInvokerFactory.java | 4 +-
.../beam/sdk/transforms/reflect/DoFnInvoker.java | 2 +
.../sdk/transforms/reflect/DoFnSignatures.java | 13 +-
.../sdk/transforms/splittabledofn/Backlog.java | 90 +
.../sdk/transforms/splittabledofn/Backlogs.java | 58 +
.../splittabledofn/ByteKeyRangeTracker.java | 30 +-
.../splittabledofn/OffsetRangeTracker.java | 28 +-
.../splittabledofn/RestrictionTracker.java | 9 +-
.../transforms/splittabledofn/Restrictions.java | 17 +-
.../java/org/apache/beam/sdk/util/CoderUtils.java | 3 +-
.../main/java/org/apache/beam/sdk/values/Row.java | 40 +-
.../org/apache/beam/sdk/values/RowWithGetters.java | 13 +-
.../apache/beam/sdk/values/TypeDescriptors.java | 68 +-
.../avro/org/apache/beam/sdk/schemas/test.avsc | 29 +
.../org/apache/beam/sdk/coders/ListCoderTest.java | 21 +
.../org/apache/beam/sdk/coders/MapCoderTest.java | 21 +
.../java/org/apache/beam/sdk/io/AvroIOTest.java | 2066 ++++++++++----------
.../sdk/io/BoundedReadFromUnboundedSourceTest.java | 4 +-
.../org/apache/beam/sdk/io/CountingSourceTest.java | 13 +-
.../java/org/apache/beam/sdk/io/FileIOTest.java | 2 +
.../sdk/io/SerializableAvroCodecFactoryTest.java | 2 +-
.../org/apache/beam/sdk/io/TFRecordIOTest.java | 11 +-
.../org/apache/beam/sdk/io/TextIOReadTest.java | 2 +
.../apache/beam/sdk/schemas/AvroSchemaTest.java | 368 ++++
.../beam/sdk/schemas/FieldTypeDescriptorsTest.java | 4 +-
.../beam/sdk/schemas/transforms/CastTest.java | 37 +-
.../beam/sdk/schemas/utils/AvroUtilsTest.java | 335 +++-
.../beam/sdk/schemas/utils/JavaBeanUtilsTest.java | 24 +-
.../beam/sdk/schemas/utils/POJOUtilsTest.java | 20 +-
.../org/apache/beam/sdk/testing/PAssertTest.java | 16 +-
.../apache/beam/sdk/transforms/CombineFnsTest.java | 5 +-
.../apache/beam/sdk/transforms/CombineTest.java | 70 +-
.../org/apache/beam/sdk/transforms/FilterTest.java | 17 +
.../beam/sdk/transforms/FlatMapElementsTest.java | 60 +-
.../apache/beam/sdk/transforms/FlattenTest.java | 3 +-
.../apache/beam/sdk/transforms/GroupByKeyTest.java | 11 +-
.../beam/sdk/transforms/MapElementsTest.java | 163 +-
.../org/apache/beam/sdk/transforms/ParDoTest.java | 17 +-
.../org/apache/beam/sdk/transforms/ReifyTest.java | 4 +-
.../beam/sdk/transforms/ReifyTimestampsTest.java | 4 +-
.../apache/beam/sdk/transforms/ReshuffleTest.java | 11 +-
.../beam/sdk/transforms/SplittableDoFnTest.java | 22 +-
.../org/apache/beam/sdk/transforms/ViewTest.java | 2 +
.../org/apache/beam/sdk/transforms/WatchTest.java | 9 +
.../beam/sdk/transforms/join/CoGroupByKeyTest.java | 8 +-
.../sdk/transforms/reflect/DoFnInvokersTest.java | 20 +-
.../reflect/DoFnSignaturesSplittableDoFnTest.java | 52 +-
.../splittabledofn/ByteKeyRangeTrackerTest.java | 40 +
.../splittabledofn/OffsetRangeTrackerTest.java | 35 +-
.../beam/sdk/transforms/windowing/WindowTest.java | 5 +-
.../sdk/transforms/windowing/WindowingTest.java | 9 +-
sdks/java/extensions/euphoria/build.gradle | 7 +-
.../core/translate/BeamMetricsTranslationTest.java | 14 +-
.../sdk/extensions/gcp/options/GcsOptions.java | 10 +
.../sdk/extensions/gcp/storage/GcsFileSystem.java | 20 +
.../org/apache/beam/sdk/util/CustomHttpErrors.java | 141 ++
.../java/org/apache/beam/sdk/util/GcsUtil.java | 31 +-
.../apache/beam/sdk/util/HttpCallCustomError.java | 13 +-
.../org/apache/beam/sdk/util/HttpCallMatcher.java | 16 +-
.../apache/beam/sdk/util/HttpRequestWrapper.java} | 28 +-
.../apache/beam/sdk/util/HttpResponseWrapper.java | 24 +-
.../beam/sdk/util/RetryHttpRequestInitializer.java | 32 +-
.../apache/beam/sdk/util/CustomHttpErrorsTest.java | 128 ++
.../apache/beam/sdk/util/gcsfs/GcsPathTest.java | 3 +-
sdks/java/extensions/kryo/build.gradle | 7 +-
.../beam/sdk/extensions/sorter/SortValues.java | 2 +-
sdks/java/extensions/sql/build.gradle | 4 +-
sdks/java/extensions/sql/jdbc/build.gradle | 7 +-
.../extensions/sql/src/main/codegen/config.fmpp | 291 +++
.../beam/sdk/extensions/sql/BeamSqlTable.java | 3 +
.../sdk/extensions/sql/impl/BeamCalciteTable.java | 9 +-
.../impl/{UdfImpl.java => ScalarFunctionImpl.java} | 34 +-
.../beam/sdk/extensions/sql/impl/UdfImpl.java | 144 +-
.../interpreter/BeamSqlExpressionEnvironment.java | 42 -
.../interpreter/BeamSqlExpressionEnvironments.java | 147 --
.../sql/impl/interpreter/BeamSqlFnExecutor.java | 550 ------
.../operator/BeamSqlBinaryOperator.java | 40 -
.../operator/BeamSqlCaseExpression.java | 64 -
.../operator/BeamSqlCastExpression.java | 138 --
.../operator/BeamSqlCorrelVariableExpression.java | 48 -
.../operator/BeamSqlDefaultExpression.java | 38 -
.../interpreter/operator/BeamSqlDotExpression.java | 57 -
.../interpreter/operator/BeamSqlExpression.java | 79 -
.../operator/BeamSqlInputRefExpression.java | 48 -
.../operator/BeamSqlLocalRefExpression.java | 48 -
.../operator/BeamSqlOperatorExpression.java | 52 -
.../interpreter/operator/BeamSqlPrimitive.java | 180 --
.../interpreter/operator/BeamSqlUdfExpression.java | 89 -
.../interpreter/operator/BeamSqlUnaryOperator.java | 41 -
.../impl/interpreter/operator/DateOperators.java | 164 --
.../impl/interpreter/operator/StringOperators.java | 245 ---
.../arithmetic/BeamSqlArithmeticExpression.java | 125 --
.../arithmetic/BeamSqlDivideExpression.java | 35 -
.../arithmetic/BeamSqlMinusExpression.java | 34 -
.../operator/arithmetic/BeamSqlModExpression.java | 34 -
.../arithmetic/BeamSqlMultiplyExpression.java | 34 -
.../operator/arithmetic/BeamSqlPlusExpression.java | 34 -
.../operator/array/BeamSqlArrayExpression.java | 51 -
.../operator/array/BeamSqlArrayItemExpression.java | 50 -
.../collection/BeamSqlCardinalityExpression.java | 50 -
.../collection/BeamSqlSingleElementExpression.java | 64 -
.../comparison/BeamSqlCompareExpression.java | 97 -
.../comparison/BeamSqlEqualsExpression.java | 53 -
.../comparison/BeamSqlGreaterThanExpression.java | 53 -
.../BeamSqlGreaterThanOrEqualsExpression.java | 53 -
.../comparison/BeamSqlIsNotNullExpression.java | 52 -
.../comparison/BeamSqlIsNullExpression.java | 52 -
.../comparison/BeamSqlLessThanExpression.java | 53 -
.../BeamSqlLessThanOrEqualsExpression.java | 53 -
.../operator/comparison/BeamSqlLikeExpression.java | 51 -
.../comparison/BeamSqlNotEqualsExpression.java | 53 -
.../comparison/BeamSqlNotLikeExpression.java | 52 -
.../date/BeamSqlCurrentDateExpression.java | 49 -
.../date/BeamSqlCurrentTimeExpression.java | 53 -
.../date/BeamSqlCurrentTimestampExpression.java | 53 -
.../date/BeamSqlDatetimeMinusExpression.java | 98 -
.../BeamSqlDatetimeMinusIntervalExpression.java | 77 -
.../date/BeamSqlDatetimePlusExpression.java | 118 --
.../date/BeamSqlIntervalMultiplyExpression.java | 95 -
.../BeamSqlTimestampMinusIntervalExpression.java | 79 -
.../BeamSqlTimestampMinusTimestampExpression.java | 97 -
.../interpreter/operator/date/TimeUnitUtils.java | 63 -
.../operator/logical/BeamSqlAndExpression.java | 47 -
.../operator/logical/BeamSqlNotExpression.java | 48 -
.../operator/logical/BeamSqlOrExpression.java | 48 -
.../operator/map/BeamSqlMapExpression.java | 60 -
.../operator/map/BeamSqlMapItemExpression.java | 49 -
.../operator/math/BeamSqlAbsExpression.java | 63 -
.../operator/math/BeamSqlAcosExpression.java | 38 -
.../operator/math/BeamSqlAsinExpression.java | 38 -
.../operator/math/BeamSqlAtan2Expression.java | 41 -
.../operator/math/BeamSqlAtanExpression.java | 38 -
.../operator/math/BeamSqlCeilExpression.java | 43 -
.../operator/math/BeamSqlCosExpression.java | 38 -
.../operator/math/BeamSqlCotExpression.java | 38 -
.../operator/math/BeamSqlDegreesExpression.java | 38 -
.../operator/math/BeamSqlExpExpression.java | 38 -
.../operator/math/BeamSqlFloorExpression.java | 43 -
.../operator/math/BeamSqlLnExpression.java | 38 -
.../operator/math/BeamSqlLogExpression.java | 38 -
.../operator/math/BeamSqlMathBinaryExpression.java | 63 -
.../operator/math/BeamSqlMathUnaryExpression.java | 57 -
.../operator/math/BeamSqlPiExpression.java | 44 -
.../operator/math/BeamSqlPowerExpression.java | 41 -
.../operator/math/BeamSqlRadiansExpression.java | 38 -
.../operator/math/BeamSqlRandExpression.java | 54 -
.../math/BeamSqlRandIntegerExpression.java | 58 -
.../operator/math/BeamSqlRoundExpression.java | 116 --
.../operator/math/BeamSqlSignExpression.java | 78 -
.../operator/math/BeamSqlSinExpression.java | 38 -
.../operator/math/BeamSqlTanExpression.java | 38 -
.../operator/math/BeamSqlTruncateExpression.java | 88 -
.../interpreter/operator/math/package-info.java | 20 -
.../impl/interpreter/operator/package-info.java | 20 -
.../reinterpret/BeamSqlReinterpretExpression.java | 70 -
.../DatetimeReinterpretConversions.java | 48 -
.../reinterpret/IntegerReinterpretConversions.java | 36 -
.../reinterpret/ReinterpretConversion.java | 112 --
.../operator/reinterpret/Reinterpreter.java | 94 -
.../operator/reinterpret/package-info.java | 20 -
.../operator/row/BeamSqlFieldAccessExpression.java | 69 -
.../interpreter/operator/row/package-info.java | 24 -
.../sql/impl/interpreter/package-info.java | 20 -
.../sql/impl/parser/SqlCreateExternalTable.java | 2 +-
.../BeamJavaTypeFactory.java} | 33 +-
.../sql/impl/planner/BeamRelDataTypeSystem.java | 6 +
.../sdk/extensions/sql/impl/rel/BeamCalcRel.java | 391 +++-
.../extensions/sql/impl/rel/BeamIOSourceRel.java | 5 +
.../sdk/extensions/sql/impl/rel/BeamJoinRel.java | 257 ++-
.../sdk/extensions/sql/impl/rel/BeamRelNode.java | 17 +
.../sql/impl/rel/BeamSetOperatorRelBase.java | 10 +-
.../sql/impl/schema/BeamPCollectionTable.java | 5 +
.../extensions/sql/impl/schema/BeamTableUtils.java | 2 +-
.../sql/impl/transform/BeamJoinTransforms.java | 22 +-
.../impl/transform/agg/CovarianceAccumulator.java | 37 +-
.../sql/impl/udf/BuiltinStringFunctions.java | 8 +-
.../extensions/sql/impl/utils/CalciteUtils.java | 4 +-
.../extensions/sql/impl/utils/SqlTypeUtils.java | 59 -
.../meta/provider/bigquery/BeamBigQueryTable.java | 5 +
.../sql/meta/provider/kafka/BeamKafkaTable.java | 5 +
.../meta/provider/kafka/KafkaTableProvider.java | 3 +-
.../meta/provider/pubsub/PubsubIOJsonTable.java | 5 +
.../provider/pubsub/PubsubJsonTableProvider.java | 7 +-
.../sql/meta/provider/test/TestBoundedTable.java | 5 +
.../sql/meta/provider/test/TestTableProvider.java | 8 +-
.../sql/meta/provider/test/TestUnboundedTable.java | 5 +
.../sql/meta/provider/text/TextTable.java | 5 +
.../sql/meta/provider/text/TextTableProvider.java | 5 +-
.../beam/sdk/extensions/sql/BeamSqlCastTest.java | 33 +-
.../sql/BeamSqlDslSqlStdOperatorsTest.java | 41 +-
.../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java | 43 +-
.../impl/interpreter/BeamSqlFnExecutorTest.java | 204 --
.../interpreter/BeamSqlFnExecutorTestBase.java | 91 -
.../operator/BeamNullExpressionTest.java | 56 -
.../operator/BeamSqlAndOrExpressionTest.java | 70 -
.../operator/BeamSqlCaseExpressionTest.java | 101 -
.../operator/BeamSqlCastExpressionTest.java | 156 --
.../operator/BeamSqlCompareExpressionTest.java | 170 --
.../operator/BeamSqlDotExpressionTest.java | 76 -
.../operator/BeamSqlInputRefExpressionTest.java | 63 -
.../interpreter/operator/BeamSqlPrimitiveTest.java | 95 -
.../operator/BeamSqlReinterpretExpressionTest.java | 130 --
.../operator/BeamSqlUdfExpressionTest.java | 50 -
.../BeamSqlArithmeticExpressionTest.java | 332 ----
.../operator/array/BeamSqlArrayExpressionTest.java | 80 -
.../array/BeamSqlArrayItemExpressionTest.java | 98 -
.../BeamSqlCardinalityExpressionTest.java | 94 -
.../BeamSqlSingleElementExpressionTest.java | 94 -
.../date/BeamSqlCurrentDateExpressionTest.java | 36 -
.../date/BeamSqlCurrentTimeExpressionTest.java | 40 -
.../BeamSqlCurrentTimestampExpressionTest.java | 40 -
.../date/BeamSqlDateExpressionTestBase.java | 36 -
.../date/BeamSqlDatetimeMinusExpressionTest.java | 150 --
...BeamSqlDatetimeMinusIntervalExpressionTest.java | 142 --
.../date/BeamSqlDatetimePlusExpressionTest.java | 186 --
.../BeamSqlIntervalMultiplyExpressionTest.java | 110 --
...eamSqlTimestampMinusIntervalExpressionTest.java | 170 --
...amSqlTimestampMinusTimestampExpressionTest.java | 210 --
.../operator/date/TimeUnitUtilsTest.java | 59 -
.../operator/logical/BeamSqlNotExpressionTest.java | 55 -
.../math/BeamSqlMathBinaryExpressionTest.java | 289 ---
.../math/BeamSqlMathUnaryExpressionTest.java | 446 -----
.../DatetimeReinterpretConversionsTest.java | 68 -
.../IntegerReinterpretConversionsTest.java | 76 -
.../reinterpret/ReinterpretConversionTest.java | 106 -
.../operator/reinterpret/ReinterpreterTest.java | 180 --
.../row/BeamSqlFieldAccessExpressionTest.java | 91 -
.../sql/impl/rel/BeamEnumerableConverterTest.java | 5 +
.../rel/BeamJoinRelUnboundedVsBoundedTest.java | 25 +
.../sql/impl/utils/SqlTypeUtilsTest.java | 76 -
.../BeamSqlComparisonOperatorsIntegrationTest.java | 44 +-
.../BeamSqlDateFunctionsIntegrationTest.java | 32 +-
.../beam/sdk/fn/channel/ManagedChannelFactory.java | 18 +-
.../beam/sdk/fn/channel/SocketAddressFactory.java | 2 +-
.../data/BeamFnDataBufferingOutboundObserver.java | 4 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexer.java | 4 +-
.../beam/sdk/fn/data/RemoteGrpcPortRead.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortWrite.java | 2 +-
.../sdk/fn/splittabledofn/RestrictionTrackers.java | 90 +-
.../sdk/fn/stream/BufferingStreamObserver.java | 4 +-
.../org/apache/beam/sdk/fn/stream/DataStreams.java | 2 +-
.../beam/sdk/fn/stream/DirectStreamObserver.java | 4 +-
.../stream/ForwardingClientResponseObserver.java | 6 +-
.../sdk/fn/stream/OutboundObserverFactory.java | 4 +-
.../sdk/fn/stream/SynchronizedStreamObserver.java | 2 +-
.../fn/test/InProcessManagedChannelFactory.java | 4 +-
.../org/apache/beam/sdk/fn/test/TestStreams.java | 4 +-
.../sdk/fn/windowing/EncodedBoundedWindow.java | 2 +-
.../sdk/fn/channel/ManagedChannelFactoryTest.java | 6 +-
.../sdk/fn/channel/SocketAddressFactoryTest.java | 2 +-
.../BeamFnDataBufferingOutboundObserverTest.java | 2 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortReadTest.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortWriteTest.java | 2 +-
.../fn/splittabledofn/RestrictionTrackersTest.java | 72 +
.../apache/beam/sdk/fn/stream/DataStreamsTest.java | 2 +-
.../ForwardingClientResponseObserverTest.java | 6 +-
.../sdk/fn/stream/OutboundObserverFactoryTest.java | 4 +-
.../sdk/fn/windowing/EncodedBoundedWindowTest.java | 2 +-
sdks/java/harness/build.gradle | 2 +-
.../beam/fn/harness/BoundedSourceRunner.java | 2 +-
.../java/org/apache/beam/fn/harness/FnHarness.java | 2 +-
.../beam/fn/harness/PrecombineGroupingTable.java | 2 +-
.../harness/SplittableProcessElementsRunner.java | 8 +-
.../harness/control/AddHarnessIdInterceptor.java | 8 +-
.../fn/harness/control/BeamFnControlClient.java | 4 +-
.../fn/harness/control/ProcessBundleHandler.java | 58 +-
.../beam/fn/harness/control/RegisterHandler.java | 2 +-
.../beam/fn/harness/data/BeamFnDataGrpcClient.java | 2 +-
.../fn/harness/data/QueueingBeamFnDataClient.java | 182 ++
.../fn/harness/logging/BeamFnLoggingClient.java | 12 +-
.../apache/beam/fn/harness/state/BagUserState.java | 2 +-
.../harness/state/BeamFnStateGrpcClientCache.java | 4 +-
.../beam/fn/harness/state/FnApiStateAccessor.java | 2 +-
.../beam/fn/harness/state/MultimapSideInput.java | 2 +-
.../fn/harness/state/StateFetchingIterators.java | 2 +-
.../stream/HarnessStreamObserverFactories.java | 2 +-
.../beam/fn/harness/BoundedSourceRunnerTest.java | 2 +-
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 138 +-
.../org/apache/beam/fn/harness/FnHarnessTest.java | 6 +-
.../harness/control/BeamFnControlClientTest.java | 8 +-
.../harness/control/ProcessBundleHandlerTest.java | 2 +-
.../fn/harness/data/BeamFnDataGrpcClientTest.java | 14 +-
.../data/BeamFnDataInboundObserverTest.java | 2 +-
...Test.java => QueueingBeamFnDataClientTest.java} | 229 ++-
.../harness/logging/BeamFnLoggingClientTest.java | 16 +-
.../beam/fn/harness/state/BagUserStateTest.java | 2 +-
.../state/BeamFnStateGrpcClientCacheTest.java | 16 +-
.../fn/harness/state/FakeBeamFnStateClient.java | 2 +-
.../fn/harness/state/MultimapSideInputTest.java | 2 +-
.../harness/state/StateFetchingIteratorsTest.java | 2 +-
.../stream/HarnessStreamObserverFactoriesTest.java | 4 +-
.../beam/sdk/io/aws/options/AwsModuleTest.java | 4 +-
.../beam/sdk/io/aws/s3/S3ResourceIdTest.java | 2 +-
.../sdk/io/aws/s3/S3WritableByteChannelTest.java | 2 +-
.../org/apache/beam/sdk/io/aws/sns/SnsIOTest.java | 3 +-
.../apache/beam/sdk/io/cassandra/CassandraIO.java | 161 +-
.../beam/sdk/io/cassandra/CassandraService.java | 16 +-
.../sdk/io/cassandra/CassandraServiceImpl.java | 76 +-
.../beam/sdk/io/cassandra/CassandraIOTest.java | 66 +-
sdks/java/io/common/build.gradle | 2 +-
.../sdk/io/elasticsearch/ElasticsearchIOTest.java | 4 +-
.../sdk/io/elasticsearch/ElasticsearchIOTest.java | 4 +-
.../sdk/io/elasticsearch/ElasticsearchIOTest.java | 4 +-
.../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 74 +-
sdks/java/io/file-based-io-tests/build.gradle | 3 +-
.../io/common/FileBasedIOTestPipelineOptions.java | 19 +
.../java/org/apache/beam/sdk/io/text/TextIOIT.java | 44 +-
.../apache/beam/sdk/io/gcp/bigquery/AvroUtils.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 10 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 4 +-
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 35 +-
.../beam/sdk/io/gcp/bigquery/BigQueryUtils.java | 2 +-
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 2 +-
.../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java | 136 +-
.../beam/sdk/io/gcp/testing/BigqueryClient.java | 8 +-
.../sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java | 2 +-
.../io/gcp/bigquery/BigQueryServicesImplTest.java | 88 +-
.../sdk/io/gcp/bigquery/BigQueryToTableIT.java | 6 +-
.../beam/sdk/io/gcp/datastore/V1TestUtil.java | 2 +-
.../beam/sdk/io/gcp/pubsub/PubsubIOTest.java | 193 ++
.../beam/sdk/io/gcp/spanner/OrderedCodeTest.java | 2 +-
.../build.gradle | 55 +-
.../io/hadoop/format/ExternalSynchronization.java | 62 +
.../sdk/io/hadoop/format/HDFSSynchronization.java | 186 ++
.../beam/sdk/io/hadoop/format/HadoopFormatIO.java | 1987 +++++++++++++++++++
.../beam/sdk/io/hadoop/format/HadoopFormats.java | 243 +++
.../sdk/io/hadoop/format/IterableCombinerFn.java | 140 ++
.../beam/sdk/io/hadoop/format}/package-info.java | 10 +-
.../format/ConfigurableEmployeeInputFormat.java | 126 ++
.../apache/beam/sdk/io/hadoop/format/Employee.java | 87 +
.../sdk/io/hadoop/format}/EmployeeInputFormat.java | 10 +-
.../sdk/io/hadoop/format/EmployeeOutputFormat.java | 73 +
.../io/hadoop/format/HDFSSynchronizationTest.java | 173 ++
.../hadoop/format/HadoopFormatIOCassandraIT.java | 197 ++
.../hadoop/format/HadoopFormatIOCassandraTest.java | 235 +++
.../io/hadoop/format/HadoopFormatIOElasticIT.java | 220 +++
.../hadoop/format/HadoopFormatIOElasticTest.java | 277 +++
.../sdk/io/hadoop/format/HadoopFormatIOIT.java | 189 ++
.../io/hadoop/format/HadoopFormatIOReadTest.java} | 151 +-
.../format/HadoopFormatIOSequenceFileTest.java | 372 ++++
.../hadoop/format/HadoopFormatIOTestOptions.java | 76 +
.../io/hadoop/format/HadoopFormatIOWriteTest.java | 314 +++
.../sdk/io/hadoop/format/IterableCombinerTest.java | 98 +
.../format}/ReuseObjectsEmployeeInputFormat.java | 10 +-
.../sdk/io/hadoop/format}/TestEmployeeDataSet.java | 4 +-
.../sdk/io/hadoop/format}/TestRowDBWritable.java | 13 +-
.../src/test/resources/cassandra.yaml | 0
sdks/java/io/hadoop-input-format/build.gradle | 1 +
.../io/hadoop/inputformat/HadoopInputFormatIO.java | 871 +--------
.../io/hadoop/inputformat/EmployeeInputFormat.java | 6 +-
.../inputformat/HadoopInputFormatIOTest.java | 462 -----
.../ReuseObjectsEmployeeInputFormat.java | 4 +-
.../io/hadoop/inputformat/TestEmployeeDataSet.java | 2 +-
.../io/hadoop/inputformat/TestRowDBWritable.java | 9 +-
.../beam/sdk/io/hbase/HBaseReadSplittableDoFn.java | 3 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 3 +-
.../java/org/apache/beam/sdk/io/jms/JmsIO.java | 29 +-
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 13 +
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 4 +-
.../beam/sdk/io/kinesis/ShardReadersPool.java | 25 +-
.../beam/sdk/io/kinesis/ShardReadersPoolTest.java | 2 +-
sdks/java/io/kudu/build.gradle | 2 +-
sdks/java/io/mongodb/build.gradle | 6 +-
.../beam/sdk/io/mongodb/MongoDbGridFSIO.java | 4 +-
.../org/apache/beam/sdk/io/mongodb/MongoDbIO.java | 186 +-
.../org/apache/beam/sdk/io/mongodb/SSLUtils.java | 75 +
.../beam/sdk/io/mongodb/MongoDBGridFSIOTest.java | 2 +-
.../apache/beam/sdk/io/mongodb/MongoDbIOTest.java | 93 +
sdks/java/io/rabbitmq/build.gradle | 2 +-
sdks/java/io/synthetic/build.gradle | 2 +-
.../beam/sdk/io/synthetic/SyntheticOptions.java | 8 +
sdks/java/io/tika/build.gradle | 2 +-
.../java/org/apache/beam/sdk/io/xml/XmlIO.java | 4 +-
sdks/java/javadoc/build.gradle | 69 +-
sdks/java/maven-archetypes/examples/build.gradle | 2 +-
.../src/main/resources/archetype-resources/pom.xml | 28 +
sdks/java/maven-archetypes/starter/build.gradle | 2 +-
sdks/java/testing/load-tests/build.gradle | 3 +-
.../beam/sdk/loadtests/CoGroupByKeyLoadTest.java | 22 +-
.../apache/beam/sdk/loadtests/CombineLoadTest.java | 22 +-
.../beam/sdk/loadtests/ConsoleResultPublisher.java | 15 +-
.../beam/sdk/loadtests/GroupByKeyLoadTest.java | 13 +-
.../org/apache/beam/sdk/loadtests/LoadTest.java | 64 +-
.../apache/beam/sdk/loadtests/LoadTestOptions.java | 17 +
.../apache/beam/sdk/loadtests/LoadTestResult.java | 67 +
.../apache/beam/sdk/loadtests/ParDoLoadTest.java | 9 +-
.../loadtests/SyntheticDataPubSubPublisher.java | 107 +
.../{MetricsPublisher.java => ByteMonitor.java} | 30 +-
.../{MetricsMonitor.java => TimeMonitor.java} | 20 +-
sdks/java/testing/nexmark/build.gradle | 2 +-
.../java/org/apache/beam/sdk/nexmark/Main.java | 39 +-
.../beam/sdk/nexmark/NexmarkConfiguration.java | 4 +
.../apache/beam/sdk/nexmark/NexmarkLauncher.java | 62 +-
.../org/apache/beam/sdk/nexmark/NexmarkPerf.java | 11 +-
.../apache/beam/sdk/nexmark/NexmarkQueryName.java | 3 +-
.../org/apache/beam/sdk/nexmark/NexmarkUtils.java | 58 +-
.../sdk/nexmark/queries/SessionSideInputJoin.java | 106 +
.../nexmark/queries/SessionSideInputJoinModel.java | 151 ++
.../sdk/nexmark/sources/generator/Generator.java | 3 +-
.../sources/generator/model/PersonGenerator.java | 8 +-
.../beam/sdk/nexmark/PerfsToBigQueryTest.java | 50 +-
.../nexmark/queries/SessionSideInputJoinTest.java | 212 ++
sdks/java/testing/test-utils/build.gradle | 2 +-
.../apache/beam/sdk/testutils/NamedTestResult.java | 76 +
.../org/apache/beam/sdk/testutils/TestResult.java | 16 +-
.../beam/sdk/testutils/metrics/MetricsReader.java | 97 +-
.../apache/beam/sdk/testutils}/package-info.java | 5 +-
.../sdk/testutils/publishing/BigQueryClient.java | 12 +
.../publishing/BigQueryResultsPublisher.java | 60 +
.../sdk/testutils/fakes/FakeBigQueryClient.java | 4 +-
...ient.java => FakeBigQueryResultsPublisher.java} | 44 +-
.../sdk/testutils/metrics/MetricsReaderTest.java | 52 +-
.../publishing/BigQueryResultsPublisherTest.java | 102 +
sdks/python/apache_beam/coders/coder_impl.pxd | 65 +-
sdks/python/apache_beam/coders/coder_impl.py | 169 +-
sdks/python/apache_beam/coders/coders.py | 147 +-
.../apache_beam/coders/coders_test_common.py | 52 +-
.../cookbook/bigquery_tornadoes_it_test.py | 2 +-
.../apache_beam/examples/snippets/snippets.py | 5 +-
sdks/python/apache_beam/internal/http_client.py | 70 +
.../apache_beam/internal/http_client_test.py | 109 ++
sdks/python/apache_beam/io/__init__.py | 1 +
sdks/python/apache_beam/io/filebasedsource_test.py | 20 +-
sdks/python/apache_beam/io/filesystem.py | 4 +-
sdks/python/apache_beam/io/filesystem_test.py | 23 +-
sdks/python/apache_beam/io/filesystemio.py | 4 +-
sdks/python/apache_beam/io/filesystemio_test.py | 33 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 18 +-
sdks/python/apache_beam/io/gcp/bigquery_test.py | 21 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 174 +-
.../apache_beam/io/gcp/gcsio_integration_test.py | 183 ++
sdks/python/apache_beam/io/gcp/gcsio_test.py | 48 +-
sdks/python/apache_beam/io/gcp/pubsub_test.py | 106 +-
sdks/python/apache_beam/io/parquetio.py | 472 +++++
sdks/python/apache_beam/io/parquetio_it_test.py | 176 ++
sdks/python/apache_beam/io/parquetio_test.py | 463 +++++
sdks/python/apache_beam/io/range_trackers.py | 28 +-
sdks/python/apache_beam/io/range_trackers_test.py | 133 +-
.../apache_beam/io/source_test_utils_test.py | 29 +-
sdks/python/apache_beam/io/sources_test.py | 18 +-
sdks/python/apache_beam/io/textio.py | 27 +-
sdks/python/apache_beam/io/textio_test.py | 26 +
sdks/python/apache_beam/io/tfrecordio_test.py | 9 +-
.../python/apache_beam/options/pipeline_options.py | 8 +
.../apache_beam/options/pipeline_options_test.py | 17 +
sdks/python/apache_beam/pipeline.py | 13 +-
sdks/python/apache_beam/portability/python_urns.py | 16 +
.../runners/dataflow/dataflow_runner.py | 106 +-
.../runners/dataflow/internal/apiclient.py | 51 +-
.../apache_beam/runners/dataflow/internal/names.py | 4 +-
.../runners/dataflow/test_dataflow_runner.py | 13 +-
.../apache_beam/runners/direct/direct_runner.py | 14 +-
.../runners/direct/test_direct_runner.py | 10 +-
.../runners/interactive/interactive_runner.py | 12 +-
.../python/apache_beam/runners/pipeline_context.py | 7 +-
.../runners/portability/flink_runner_test.py | 93 +-
.../runners/portability/fn_api_runner.py | 1277 ++++--------
.../runners/portability/fn_api_runner_test.py | 69 +-
.../portability/fn_api_runner_transforms.py | 948 +++++++++
.../apache_beam/runners/portability/job_server.py | 27 +-
.../runners/portability/local_job_service.py | 84 +-
.../runners/portability/local_job_service_main.py | 4 +-
.../runners/portability/portable_runner.py | 153 +-
.../runners/portability/portable_runner_test.py | 45 +-
.../apache_beam/runners/portability/stager.py | 3 +-
sdks/python/apache_beam/runners/runner.py | 14 +-
.../apache_beam/runners/worker/bundle_processor.py | 90 +-
.../apache_beam/runners/worker/data_plane.py | 14 +-
.../apache_beam/runners/worker/operations.py | 2 +
.../apache_beam/runners/worker/sdk_worker.py | 77 +-
.../runners/worker/worker_id_interceptor.py | 5 +-
.../testing/load_tests/co_group_by_key_test.py | 75 +-
.../apache_beam/testing/load_tests/combine_test.py | 69 +-
.../testing/load_tests/group_by_key_test.py | 76 +-
.../testing/load_tests/load_test_metrics_utils.py | 165 +-
.../apache_beam/testing/load_tests/pardo_test.py | 132 +-
.../testing/load_tests/sideinput_test.py | 203 ++
.../apache_beam/testing/synthetic_pipeline.py | 2 +-
.../apache_beam/testing/synthetic_pipeline_test.py | 2 +-
sdks/python/apache_beam/testing/test_utils.py | 7 +-
.../apache_beam/tools/coders_microbenchmark.py | 50 +-
sdks/python/apache_beam/transforms/core.py | 6 +-
.../apache_beam/transforms/ptransform_test.py | 114 +-
.../apache_beam/transforms/userstate_test.py | 32 +-
sdks/python/apache_beam/transforms/util.py | 18 +-
sdks/python/apache_beam/transforms/util_test.py | 8 +
sdks/python/apache_beam/transforms/window.py | 26 +-
sdks/python/apache_beam/utils/proto_utils.py | 8 +-
sdks/python/apache_beam/utils/windowed_value.pxd | 19 +
sdks/python/apache_beam/utils/windowed_value.py | 61 +-
sdks/python/build.gradle | 161 +-
sdks/python/container/base_image_requirements.txt | 2 +-
sdks/python/container/build.gradle | 4 +-
.../python/precommit/dataflow}/build.gradle | 27 +-
sdks/python/scripts/generate_pydoc.sh | 1 +
sdks/python/scripts/run_integration_test.sh | 20 +-
sdks/python/setup.py | 24 +-
sdks/python/tox.ini | 4 +-
settings.gradle | 20 +
vendor/grpc-1_13_1/build.gradle | 4 +-
vendor/sdks-java-extensions-protobuf/build.gradle | 13 +-
website/Gemfile.lock | 16 +-
website/Rakefile | 13 +-
website/_config.yml | 2 +-
website/src/.htaccess | 2 +-
website/src/_data/authors.yml | 3 +
website/src/_includes/section-menu/community.html | 1 +
website/src/_posts/2017-01-09-added-apex-runner.md | 4 +-
website/src/_posts/2018-12-13-beam-2.9.0.md | 62 +
website/src/community/contact-us.md | 2 +-
website/src/community/in-person.md | 47 +
website/src/contribute/committer-guide.md | 37 +-
website/src/contribute/index.md | 25 +-
website/src/contribute/postcommits-guides.md | 2 +-
.../src/contribute/postcommits-policies-details.md | 18 +
website/src/contribute/ptransform-style-guide.md | 4 +-
website/src/contribute/release-guide.md | 2 +-
.../documentation/io/built-in-google-bigquery.md | 2 +-
website/src/documentation/runners/apex.md | 2 +-
website/src/documentation/runners/flink.md | 2 +-
website/src/documentation/sdks/euphoria.md | 4 +-
.../src/documentation/sdks/java-dependencies.md | 497 +++--
website/src/documentation/sdks/nexmark.md | 7 +-
.../src/documentation/sdks/python-dependencies.md | 64 +-
website/src/get-started/downloads.md | 7 +
997 files changed, 28980 insertions(+), 21156 deletions(-)
rename .test-infra/jenkins/{job_PostCommit_Go_GradleBuild.groovy =>
job_PostCommit_Go.groovy} (91%)
copy .test-infra/jenkins/{job_PostCommit_Java_GradleBuild.groovy =>
job_PostCommit_Java.groovy} (92%)
rename
.test-infra/jenkins/{job_PostCommit_Java_PortabilityApi_GradleBuild.groovy =>
job_PostCommit_Java_PortabilityApi.groovy} (94%)
copy
.test-infra/jenkins/{job_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy
=>
job_PostCommit_Java_ValidatesRunner_DataflowPortabilityExecutableStage.groovy}
(83%)
rename .test-infra/jenkins/{job_PostCommit_Java_GradleBuild.groovy =>
job_PostCommit_SQL.groovy} (78%)
rename
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/array/package-info.java
=> .test-infra/jenkins/job_PreCommit_Portable_Python.groovy (71%)
rename .test-infra/jenkins/{job_Release_Gradle_NightlySnapshot.groovy =>
job_Release_NightlySnapshot.groovy} (96%)
copy
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/{SerializablePipelineOptions.java
=> PipelineOptionsSerializationUtils.java} (58%)
create mode 100644
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ExecutableStageTranslationTest.java
create mode 100644
runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleMonitoringInfoBuilder.java
rename runners/{samza/src/main/java/org/apache/beam/runners/samza/util =>
core-java/src/main/java/org/apache/beam/runners/core/serialization}/Base64Serializer.java
(94%)
rename
{sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date
=>
runners/core-java/src/main/java/org/apache/beam/runners/core/serialization}/package-info.java
(89%)
create mode 100644
runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/SimpleMonitoringInfoBuilderTest.java
copy {model/pipeline => runners/flink/1.6}/build.gradle (59%)
copy {sdks/java/build-tools =>
runners/flink/1.6/job-server-container}/build.gradle (79%)
copy {model/pipeline => runners/flink/1.6/job-server}/build.gradle (65%)
copy runners/flink/{build.gradle => flink_runner.gradle} (89%)
copy runners/flink/job-server-container/{build.gradle =>
flink_job_server_container.gradle} (61%)
copy runners/flink/job-server/{build.gradle => flink_job_server.gradle} (50%)
rename
runners/flink/src/main/java/org/apache/beam/runners/flink/{PipelineTranslationOptimizer.java
=> PipelineTranslationModeOptimizer.java} (79%)
create mode 100644
runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FileReporter.java
create mode 100644
runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/Metrics.java
create mode 100644
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/utils/NoopLock.java
create mode 100644
runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslatorsTest.java
create mode 100644
runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkTransformOverridesTest.java
create mode 100644
runners/flink/src/test/java/org/apache/beam/runners/flink/PipelineTranslationModeOptimizerTest.java
create mode 100644
runners/flink/src/test/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainerTest.java
create mode 100644
runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/GroupByWithNullValuesTest.java
create mode 100644
runners/flink/src/test/java/org/apache/beam/runners/flink/translation/FlinkPipelineTranslatorUtilsTest.java
rename runners/flink/src/test/java/org/apache/beam/runners/flink/{streaming =>
translation/wrappers/streaming/io}/TestCountingSource.java (94%)
rename runners/flink/src/test/java/org/apache/beam/runners/flink/{streaming =>
translation/wrappers/streaming/io}/UnboundedSourceWrapperTest.java (82%)
rename
runners/google-cloud-dataflow-java/{worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/IdGeneratorTest.java
=> src/test/java/org/apache/beam/runners/dataflow/util/GCSUploadMain.java}
(55%)
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/ServerFactory.java
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/SocketAddressFactory.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/ProcessRemoteBundleOperation.java
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/{RegisterNodeFunction.java
=> CreateExecutableStageNodeFunction.java} (56%)
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/ServerFactoryTest.java
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/SocketAddressFactoryTest.java
copy
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/{ProcessEnvironmentFactory.java
=> ExternalEnvironmentFactory.java} (59%)
create mode 100644
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/StaticRemoteEnvironment.java
create mode 100644
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/StaticRemoteEnvironmentFactory.java
create mode 100644
runners/reference/java/src/main/java/org/apache/beam/runners/reference/ExternalWorkerService.java
rename
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlOperator.java
=>
runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnInvokerRegistrar.java
(57%)
create mode 100644
runners/samza/src/main/java/org/apache/beam/runners/samza/transforms/GroupWithoutRepartition.java
copy
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldValueGetterFactory.java
=>
runners/samza/src/main/java/org/apache/beam/runners/samza/transforms/UpdatingCombineFn.java
(57%)
rename
{sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic
=>
runners/samza/src/main/java/org/apache/beam/runners/samza/transforms}/package-info.java
(87%)
rename
runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/{
=> mocks}/DatasetSourceMockBatch.java (72%)
rename
runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/{
=> mocks}/ReadSourceTranslatorMockBatch.java (81%)
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/fn_arity.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/fn_arity.tmpl
copy sdks/go/pkg/beam/core/util/ioutilx/{read.go => write.go} (67%)
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/structs.go
copy sdks/go/pkg/beam/{core/util/ioutilx/read.go => util/gcsx/gcs_test.go}
(63%)
create mode 100644 sdks/java/bom/build.gradle
create mode 100644 sdks/java/bom/pom.xml.template
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsOptions.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroRecordSchema.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/CachingFactory.java
rename
sdks/java/{extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/package-info.java
=> core/src/main/java/org/apache/beam/sdk/schemas/Factory.java} (76%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldValueTypeInformation.java
rename
sdks/java/{extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/map/package-info.java
=>
core/src/main/java/org/apache/beam/sdk/schemas/FieldValueTypeInformationFactory.java}
(71%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FromRowUsingCreator.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaUserTypeConstructorCreator.java
rename
sdks/java/{extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/collection/package-info.java
=> core/src/main/java/org/apache/beam/sdk/schemas/SchemaUserTypeCreator.java}
(73%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SetterBasedCreatorFactory.java
copy
runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java
=>
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/UserTypeCreatorFactory.java
(76%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroByteBuddyUtils.java
rename
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/{PojoValueGetterFactory.java
=> FieldValueTypeSupplier.java} (64%)
delete mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanGetterFactory.java
delete mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanSetterFactory.java
delete mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/PojoValueSetterFactory.java
copy
runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java
=>
sdks/java/core/src/main/java/org/apache/beam/sdk/testing/DataflowPortabilityExecutableStageUnsupported.java
(69%)
copy
runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java
=> sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesSideInputs.java
(77%)
copy
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/{SimpleFunction.java
=> InferableFunction.java} (64%)
copy
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/{SerializableFunction.java
=> ProcessFunction.java} (51%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Backlog.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Backlogs.java
copy
runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java
=>
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Restrictions.java
(65%)
create mode 100644
sdks/java/core/src/test/avro/org/apache/beam/sdk/schemas/test.avsc
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AvroSchemaTest.java
create mode 100644
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/CustomHttpErrors.java
copy
runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java
=>
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/HttpCallCustomError.java
(70%)
copy
runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java
=>
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/HttpCallMatcher.java
(67%)
rename
sdks/java/extensions/{sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlExpressionExecutor.java
=>
google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/HttpRequestWrapper.java}
(54%)
rename
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/IdGenerator.java
=>
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/HttpResponseWrapper.java
(55%)
create mode 100644
sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/util/CustomHttpErrorsTest.java
copy
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/{UdfImpl.java
=> ScalarFunctionImpl.java} (84%)
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlExpressionEnvironment.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlExpressionEnvironments.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlBinaryOperator.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCaseExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCastExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCorrelVariableExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlDefaultExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlDotExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlInputRefExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlLocalRefExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlOperatorExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlPrimitive.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlUdfExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlUnaryOperator.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/DateOperators.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/StringOperators.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlArithmeticExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlDivideExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlMinusExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlModExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlMultiplyExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlPlusExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/array/BeamSqlArrayExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/array/BeamSqlArrayItemExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/collection/BeamSqlCardinalityExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/collection/BeamSqlSingleElementExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlCompareExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlEqualsExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlGreaterThanExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlGreaterThanOrEqualsExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlIsNotNullExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlIsNullExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlLessThanExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlLessThanOrEqualsExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlLikeExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlNotEqualsExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/comparison/BeamSqlNotLikeExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlCurrentDateExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlCurrentTimeExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlCurrentTimestampExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimeMinusExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimeMinusIntervalExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimePlusExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlIntervalMultiplyExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlTimestampMinusIntervalExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlTimestampMinusTimestampExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/TimeUnitUtils.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/logical/BeamSqlAndExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/logical/BeamSqlNotExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/logical/BeamSqlOrExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/map/BeamSqlMapExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/map/BeamSqlMapItemExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlAbsExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlAcosExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlAsinExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlAtan2Expression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlAtanExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlCeilExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlCosExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlCotExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlDegreesExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlExpExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlFloorExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlLnExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlLogExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlMathBinaryExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlMathUnaryExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlPiExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlPowerExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlRadiansExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlRandExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlRandIntegerExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlRoundExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlSignExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlSinExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlTanExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlTruncateExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/package-info.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/package-info.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/BeamSqlReinterpretExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/DatetimeReinterpretConversions.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/IntegerReinterpretConversions.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/ReinterpretConversion.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/Reinterpreter.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/package-info.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/row/BeamSqlFieldAccessExpression.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/row/package-info.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/package-info.java
rename
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/{interpreter/operator/logical/BeamSqlLogicalExpression.java
=> planner/BeamJavaTypeFactory.java} (50%)
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SqlTypeUtils.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutorTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutorTestBase.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamNullExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlAndOrExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCaseExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCastExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCompareExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlDotExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlInputRefExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlPrimitiveTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlReinterpretExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlUdfExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlArithmeticExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/array/BeamSqlArrayExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/array/BeamSqlArrayItemExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/collection/BeamSqlCardinalityExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/collection/BeamSqlSingleElementExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlCurrentDateExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlCurrentTimeExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlCurrentTimestampExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDateExpressionTestBase.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimeMinusExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimeMinusIntervalExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlDatetimePlusExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlIntervalMultiplyExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlTimestampMinusIntervalExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlTimestampMinusTimestampExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/TimeUnitUtilsTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/logical/BeamSqlNotExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlMathBinaryExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlMathUnaryExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/DatetimeReinterpretConversionsTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/IntegerReinterpretConversionsTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/ReinterpretConversionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/reinterpret/ReinterpreterTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/row/BeamSqlFieldAccessExpressionTest.java
delete mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/utils/SqlTypeUtilsTest.java
create mode 100644
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
copy
sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/{BeamFnDataGrpcClientTest.java
=> QueueingBeamFnDataClientTest.java} (63%)
copy sdks/java/io/{hadoop-input-format => hadoop-format}/build.gradle (66%)
create mode 100644
sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/ExternalSynchronization.java
create mode 100644
sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HDFSSynchronization.java
create mode 100644
sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java
create mode 100644
sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormats.java
create mode 100644
sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/IterableCombinerFn.java
copy
sdks/java/{extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/logical
=>
io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format}/package-info.java
(76%)
create mode 100644
sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/ConfigurableEmployeeInputFormat.java
create mode 100644
sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/Employee.java
copy
sdks/java/io/{hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat
=>
hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format}/EmployeeInputFormat.java
(93%)
create mode 100644
sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/EmployeeOutputFormat.java
create mode 100644
sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HDFSSynchronizationTest.java
create mode 100644
sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOCassandraIT.java
create mode 100644
sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOCassandraTest.java
create mode 100644
sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOElasticIT.java
create mode 100644
sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOElasticTest.java
create mode 100644
sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOIT.java
copy
sdks/java/io/{hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIOTest.java
=>
hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOReadTest.java}
(85%)
create mode 100644
sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOSequenceFileTest.java
create mode 100644
sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOTestOptions.java
create mode 100644
sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOWriteTest.java
create mode 100644
sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/IterableCombinerTest.java
copy
sdks/java/io/{hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat
=>
hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format}/ReuseObjectsEmployeeInputFormat.java
(93%)
copy
sdks/java/io/{hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat
=>
hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format}/TestEmployeeDataSet.java
(97%)
copy
sdks/java/io/{hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat
=>
hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format}/TestRowDBWritable.java
(88%)
copy sdks/java/io/{hadoop-input-format =>
hadoop-format}/src/test/resources/cassandra.yaml (100%)
create mode 100644
sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/SSLUtils.java
copy
runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java
=>
sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/ConsoleResultPublisher.java
(69%)
create mode 100644
sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/LoadTestResult.java
create mode 100644
sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/SyntheticDataPubSubPublisher.java
rename
sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/metrics/{MetricsPublisher.java
=> ByteMonitor.java} (51%)
rename
sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/metrics/{MetricsMonitor.java
=> TimeMonitor.java} (69%)
create mode 100644
sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoin.java
create mode 100644
sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoinModel.java
create mode 100644
sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoinTest.java
create mode 100644
sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/NamedTestResult.java
rename
runners/flink/src/main/java/org/apache/beam/runners/flink/TranslationMode.java
=>
sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/TestResult.java
(70%)
rename
sdks/java/{extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/logical
=>
testing/test-utils/src/main/java/org/apache/beam/sdk/testutils}/package-info.java
(88%)
create mode 100644
sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/BigQueryResultsPublisher.java
copy
sdks/java/testing/test-utils/src/test/java/org/apache/beam/sdk/testutils/fakes/{FakeBigQueryClient.java
=> FakeBigQueryResultsPublisher.java} (52%)
create mode 100644
sdks/java/testing/test-utils/src/test/java/org/apache/beam/sdk/testutils/publishing/BigQueryResultsPublisherTest.java
create mode 100644 sdks/python/apache_beam/internal/http_client.py
create mode 100644 sdks/python/apache_beam/internal/http_client_test.py
create mode 100644 sdks/python/apache_beam/io/gcp/gcsio_integration_test.py
create mode 100644 sdks/python/apache_beam/io/parquetio.py
create mode 100644 sdks/python/apache_beam/io/parquetio_it_test.py
create mode 100644 sdks/python/apache_beam/io/parquetio_test.py
create mode 100644
sdks/python/apache_beam/runners/portability/fn_api_runner_transforms.py
create mode 100644 sdks/python/apache_beam/testing/load_tests/sideinput_test.py
copy {model/job-management => sdks/python/precommit/dataflow}/build.gradle
(50%)
create mode 100644 website/src/_posts/2018-12-13-beam-2.9.0.md
create mode 100644 website/src/community/in-person.md
