This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to tag nightly-master
in repository https://gitbox.apache.org/repos/asf/beam.git.
*** WARNING: tag nightly-master was modified! ***
from 780cc4f (commit)
to 752bdfd (commit)
from 780cc4f [BEAM-8494] Add python3.8 to the list of container images to
publish. (#12401)
add f95ec86 [BEAM-7390] Add latest code snippets (#12394)
add b976e7b Add a blogpost for Apache Beam 2.23.0. (#12213)
add 88acc52 [BEAM-8543] Dataflow streaming timers are not strictly time
ordered when set earlier mid-bundle (#11924)
add 890bedc [BEAM-10595] Remove rules that not work well with Java UDF
(#12400)
add c92e57b Support creation of empty PCollection in beam CreateList
add 5f9848f Update docstrings
add 1e651d8 Refactor CreateList
add 57133f8 Merge branch 'master' into beam-go-create-empty-pcollection
add 631bbb8 Refactor CreateList
add da49b24 Support creation of empty PCollection in beam CreateList
add 36dc765 Extend elementwise to generic proxy fn.
add ce5d75b Add a more methods to dataframes.
add a37170e Add support for dataframe scalar values.
add 8b08bf2 More dataframes methods and test filters.
add b365608 Reviewer comments.
add de7fa96 Skip dict-order-sensitive tests on 3.5.
add 2496fc4 Merge pull request #11974 [BEAM-9547] Add more methods to
deferred dataframes.
add 1019ef3 Prepare a website update for 2.23.0 (#12212)
add a443fdd Refactor Beam SQL QueryPlanner instantiation to be more type
safe and robust
add 8084d1b Move individual spotbug suppressions out of
spotbugs-filter.xml and remove some global suppressions
add 12631cb [BEAM-10575] Fix rawtypes in FileSystemRegistrar and
implementations
add 45d1a5c [BEAM-10556] Eliminate remaining trivial rawtypes in
sdks/java/io/hadoop-file-system
add eb20f3a [BEAM-10577] Eliminate remaining trivial rawtypes in
sdks/java/io/azure
add ab29122 [BEAM-10575] Eliminate rawtypes from GCP IOs and enable
-Wrawtypes
add c2eb01d Merge pull request #12376: [BEAM-10575] Eliminate legacy
rawtypes from GCP IOs and some others; enable -Wrawtypes -Werror
add 010adc5 [BEAM-10331] Add SnowflakeIO to list of built-in IO
transforms. (#12099)
add 11ea775 Add failing test for count on an empty pcollection
add 2435100 Fix go count on an empty pcollection
add e86062f Merge pull request #12421 from simonepri/beam-go-fix-count
add 596cc06 Convert katas to use with syntax rather than explicit run
call.
add a4af329 Move class definition outside of with statement.
add 6e61a9f Fix placeholder offsets.
add 2ed487bc9 Merge pull request #12362 Convert katas to use with syntax
rather than explicit run call.
add bb6c907 Add Composite transforms to Go SDK katas
add e7ef786 Update stepik
add 62e20f7 [BEAM-9679] Add Composite transforms to Go SDK katas
add b05204e [BEAM-10545] KernelModel and jest tests
add 8f24efa Change the syntax of private _onIOPub to a function
declaration instead of arrow function assignment. This requires binding `this`
when assigning the function to the `future`.
add c64cf34 Merge pull request #12372 from KevinGG/BEAM-10545
add a315672 Bump google cloud bigquery to 1.26.1
add a3e20e7 Merge pull request #12404 from
jiyongjung0/bump_bigquery_1_26_1
add b6ac20d remove redundent precommits.
add 4885e20 Merge pull request #12417 from amaliujia/rw-remove-precommits
add 965eb65 [BEAM-10559] Add apache_beam.examples.sql_taxi (#12399)
add 9af5435 Standardizing BigQuery job names in Beam Python and Java SDKs
add c2fb77d Naming BQ Copy jobs consistently in Python
add a6b88d2 Applying fixes from comments.
add 752bdfd Merge pull request #12082 from Standardizing BigQuery job
names in Beam Python and Java SDKs
No new revisions were added by this update.
Summary of changes:
.../jenkins/job_PreCommit_BeamSQL_ZetaSQL.groovy | 36 -
.../job_PreCommit_BeamSQL_ZetaSQL_Java11.groovy | 41 -
CHANGES.md | 5 +-
.../apache/beam/examples/snippets/Snippets.java | 2 +-
.../beam/examples/complete/AutoCompleteTest.java | 2 +-
.../complete/game/StatefulTeamScoreTest.java | 1 -
.../composite}/cmd/main.go | 5 +-
.../composite/pkg/common/common.go} | 12 +-
.../composite}/pkg/task/task.go | 17 +-
.../composite}/task-info.yaml | 21 +-
.../composite/composite/task-remote-info.yaml | 2 +
.../core_transforms/composite/composite}/task.md | 35 +-
.../composite}/test/task_test.go | 78 +-
.../core_transforms/composite}/lesson-info.yaml | 2 +-
.../composite/lesson-remote-info.yaml | 3 +
.../katas/go/core_transforms/section-info.yaml | 19 +-
.../Aggregation/Count/task-info.yaml | 2 +-
.../Common Transforms/Aggregation/Count/task.py | 9 +-
.../Aggregation/Largest/task-info.yaml | 2 +-
.../Common Transforms/Aggregation/Largest/task.py | 9 +-
.../Aggregation/Mean/task-info.yaml | 2 +-
.../Common Transforms/Aggregation/Mean/task.py | 9 +-
.../Aggregation/Smallest/task-info.yaml | 2 +-
.../Common Transforms/Aggregation/Smallest/task.py | 9 +-
.../Aggregation/Sum/task-info.yaml | 2 +-
.../Common Transforms/Aggregation/Sum/task.py | 9 +-
.../Common Transforms/Filter/Filter/task-info.yaml | 2 +-
.../python/Common Transforms/Filter/Filter/task.py | 9 +-
.../Common Transforms/Filter/ParDo/task-info.yaml | 2 +-
.../python/Common Transforms/Filter/ParDo/task.py | 10 +-
.../WithKeys/WithKeys/task-info.yaml | 2 +-
.../Common Transforms/WithKeys/WithKeys/task.py | 9 +-
.../Branching/Branching/task-info.yaml | 4 +-
.../Core Transforms/Branching/Branching/task.py | 13 +-
.../CoGroupByKey/CoGroupByKey/task.py | 11 +-
.../Combine/Combine PerKey/task-info.yaml | 2 +-
.../Core Transforms/Combine/Combine PerKey/task.py | 11 +-
.../Combine/CombineFn/task-info.yaml | 2 +-
.../Core Transforms/Combine/CombineFn/task.py | 9 +-
.../Combine/Simple Function/task-info.yaml | 2 +-
.../Combine/Simple Function/task.py | 9 +-
.../Composite Transform/task-info.yaml | 2 +-
.../Composite Transform/task.py | 9 +-
.../Core Transforms/Flatten/Flatten/task-info.yaml | 2 +-
.../python/Core Transforms/Flatten/Flatten/task.py | 17 +-
.../GroupByKey/GroupByKey/task-info.yaml | 4 +-
.../Core Transforms/GroupByKey/GroupByKey/task.py | 11 +-
.../Core Transforms/Map/FlatMap/task-info.yaml | 2 +-
.../python/Core Transforms/Map/FlatMap/task.py | 9 +-
.../python/Core Transforms/Map/Map/task-info.yaml | 2 +-
.../katas/python/Core Transforms/Map/Map/task.py | 9 +-
.../Map/ParDo OneToMany/task-info.yaml | 2 +-
.../Core Transforms/Map/ParDo OneToMany/task.py | 9 +-
.../Core Transforms/Map/ParDo/task-info.yaml | 2 +-
.../katas/python/Core Transforms/Map/ParDo/task.py | 9 +-
.../Partition/Partition/task-info.yaml | 2 +-
.../Core Transforms/Partition/Partition/task.py | 13 +-
.../Side Input/Side Input/task-info.yaml | 2 +-
.../Core Transforms/Side Input/Side Input/task.py | 37 +-
.../Side Output/Side Output/task-info.yaml | 4 +-
.../Side Output/Side Output/task.py | 15 +-
.../Examples/Word Count/Word Count/task-info.yaml | 4 +-
.../python/Examples/Word Count/Word Count/task.py | 13 +-
.../python/IO/Built-in IOs/Built-in IOs/task.py | 3 +-
.../python/IO/TextIO/ReadFromText/task-info.yaml | 4 +-
.../katas/python/IO/TextIO/ReadFromText/task.py | 11 +-
.../Hello Beam/Hello Beam/task-info.yaml | 2 +-
.../Introduction/Hello Beam/Hello Beam/task.py | 7 +-
.../Adding Timestamp/ParDo/task-info.yaml | 2 +-
.../Windowing/Adding Timestamp/ParDo/task.py | 25 +-
.../Fixed Time Window/task-info.yaml | 4 +-
.../Fixed Time Window/Fixed Time Window/task.py | 33 +-
.../org/apache/beam/runners/core/DoFnRunner.java | 8 +-
.../apache/beam/runners/core/PaneInfoTracker.java | 3 +
.../org/apache/beam/runners/core/StateMerging.java | 2 +
.../apache/beam/runners/core/SystemReduceFn.java | 2 +
.../apache/beam/runners/core/WatermarkHold.java | 6 +
.../direct/ExecutorServiceParallelExecutor.java | 13 +
.../runners/direct/ParDoMultiOverrideFactory.java | 4 +
.../beam/runners/flink/metrics/FileReporter.java | 3 +
.../beam/runners/dataflow/TestDataflowRunner.java | 2 +
.../runners/dataflow/util/OutputReference.java | 2 +
.../worker/StreamingModeExecutionContext.java | 75 +-
.../dataflow/worker/WindmillTimerInternals.java | 25 +
.../beam/runners/dataflow/worker/graph/Nodes.java | 4 +-
.../JulHandlerPrintStreamAdapterFactory.java | 15 +-
.../jobsubmission/PortablePipelineJarCreator.java | 4 +-
.../beam/runners/spark/metrics/sink/CsvSink.java | 3 +
.../runners/spark/metrics/sink/GraphiteSink.java | 3 +
sdks/go/pkg/beam/create.go | 51 +-
sdks/go/pkg/beam/create_test.go | 26 +
sdks/go/pkg/beam/transforms/stats/count.go | 3 +-
sdks/go/pkg/beam/transforms/stats/count_test.go | 7 +
.../src/main/resources/beam/spotbugs-filter.xml | 426 +--
.../java/org/apache/beam/sdk/coders/AvroCoder.java | 5 +
.../apache/beam/sdk/coders/SerializableCoder.java | 3 +
.../sdk/expansion/ExternalTransformRegistrar.java | 2 +-
.../apache/beam/sdk/io/ClassLoaderFileSystem.java | 2 +-
.../apache/beam/sdk/io/FileSystemRegistrar.java | 2 +-
.../org/apache/beam/sdk/io/GenerateSequence.java | 2 +-
.../beam/sdk/io/LocalFileSystemRegistrar.java | 2 +-
.../beam/sdk/options/ProxyInvocationHandler.java | 10 +
.../java/org/apache/beam/sdk/testing/PAssert.java | 2 +
.../beam/sdk/testing/SerializableMatchers.java | 2 +
.../apache/beam/sdk/testing/WindowSupplier.java | 3 +
.../java/org/apache/beam/sdk/transforms/Mean.java | 4 +
.../org/apache/beam/sdk/util/CombineFnUtil.java | 4 +
.../sdk/util/ExposedByteArrayOutputStream.java | 2 +
.../beam/sdk/io/LocalFileSystemRegistrarTest.java | 3 +-
.../org/apache/beam/sdk/transforms/ParDoTest.java | 69 +-
.../sdk/expansion/service/ExpansionService.java | 14 +-
.../google-cloud-platform-core/build.gradle | 1 -
.../gcp/storage/GcsFileSystemRegistrar.java | 2 +-
.../beam/sdk/extensions/gcp/util/GcsUtil.java | 3 +
.../gcp/storage/GcsFileSystemRegistrarTest.java | 3 +-
.../expansion/ExternalSqlTransformRegistrar.java | 2 +-
.../beam/sdk/extensions/sql/jdbc/BeamSqlLine.java | 5 +-
.../provider/bigquery/BigQueryIOPushDownIT.java | 3 +-
.../beam/sdk/extensions/sql/impl/BeamSqlEnv.java | 31 +-
.../extensions/sql/impl/CalciteQueryPlanner.java | 17 +-
.../beam/sdk/extensions/sql/impl/QueryPlanner.java | 6 +
.../extensions/sql/impl/planner/BeamRuleSets.java | 19 +-
.../sql/impl/rule/IOPushDownRuleTest.java | 3 +-
.../sql/impl/rule/JoinReorderingTest.java | 4 +-
...tTableProviderWithFilterAndProjectPushDown.java | 3 +-
.../test/TestTableProviderWithFilterPushDown.java | 3 +-
.../test/TestTableProviderWithProjectPushDown.java | 3 +-
.../sql/zetasql/ZetaSQLQueryPlanner.java | 41 +-
.../sql/zetasql/ZetaSQLPushDownTest.java | 3 +-
.../extensions/sql/zetasql/ZetaSqlTestBase.java | 3 +-
.../beam/sdk/io/aws/s3/S3FileSystemRegistrar.java | 2 +-
sdks/java/io/azure/build.gradle | 3 +-
.../AzureBlobStoreFileSystemRegistrar.java | 2 +-
.../sdk/io/azure/blobstore/AzfsResourceIdTest.java | 2 +-
sdks/java/io/google-cloud-platform/build.gradle | 1 -
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 26 +-
.../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 36 -
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 28 +-
.../sdk/io/gcp/bigquery/BigQueryQueryHelper.java | 10 +-
.../io/gcp/bigquery/BigQueryQuerySourceDef.java | 7 +-
.../io/gcp/bigquery/BigQueryResourceNaming.java | 113 +
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 6 +-
.../sdk/io/gcp/bigquery/BigQuerySourceBase.java | 6 +-
.../gcp/bigquery/BigQueryStorageStreamSource.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryUtils.java | 11 +-
.../sdk/io/gcp/bigquery/TableRowInfoCoder.java | 4 +-
.../beam/sdk/io/gcp/bigquery/TagWithUniqueIds.java | 5 +-
.../beam/sdk/io/gcp/bigquery/WriteRename.java | 2 +-
.../beam/sdk/io/gcp/bigquery/WriteTables.java | 6 +-
.../io/gcp/healthcare/HealthcareIOErrorCoder.java | 2 +-
.../beam/sdk/io/gcp/pubsub/ExternalRead.java | 2 +-
.../beam/sdk/io/gcp/pubsub/ExternalWrite.java | 2 +-
.../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 4 +-
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 6 +-
.../sdk/io/gcp/testing/FakeBigQueryServices.java | 2 +-
.../sdk/io/gcp/bigquery/BigQueryHelpersTest.java | 4 +-
.../sdk/io/gcp/bigquery/BigQueryIOReadTest.java | 12 +-
.../gcp/bigquery/BigQueryIOStorageQueryTest.java | 12 +-
.../io/gcp/bigquery/BigQueryIOStorageReadTest.java | 2 +-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 7 +-
.../gcp/bigquery/BigQueryResourceNamingTest.java | 73 +
.../sdk/io/gcp/pubsub/PubsubIOExternalTest.java | 2 +-
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 31 +-
sdks/java/io/hadoop-file-system/build.gradle | 1 -
.../sdk/io/hdfs/HadoopFileSystemRegistrar.java | 4 +-
.../sdk/io/hdfs/HadoopFileSystemModuleTest.java | 6 +-
.../sdk/io/hdfs/HadoopFileSystemOptionsTest.java | 4 +-
.../sdk/io/hdfs/HadoopFileSystemRegistrarTest.java | 2 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 5 +
.../beam/sdk/io/jdbc/JdbcReadRowsRegistrar.java | 2 +-
.../beam/sdk/io/jdbc/JdbcWriteRegistrar.java | 2 +-
.../java/org/apache/beam/sdk/io/jms/JmsRecord.java | 10 +
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 20 +-
.../testing/expansion/TestExpansionService.java | 4 +-
sdks/python/apache_beam/dataframe/doctests.py | 62 +-
sdks/python/apache_beam/dataframe/frame_base.py | 109 +-
sdks/python/apache_beam/dataframe/frames.py | 240 +-
.../apache_beam/dataframe/pandas_doctests_test.py | 76 +-
sdks/python/apache_beam/dataframe/transforms.py | 47 +-
.../apache_beam/dataframe/transforms_test.py | 40 +-
sdks/python/apache_beam/examples/sql_taxi.py | 101 +
sdks/python/apache_beam/io/gcp/bigquery.py | 14 +
.../apache_beam/io/gcp/bigquery_file_loads.py | 51 +-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 18 +
.../apache_beam/io/gcp/bigquery_tools_test.py | 39 +
.../apache-beam-jupyterlab-sidepanel/README.md | 26 +
.../{src/index.ts => jest.config.js} | 33 +-
.../apache-beam-jupyterlab-sidepanel/package.json | 9 +-
.../src/__tests__/kernel/KernelModel.test.ts | 124 +
.../src/{index.ts => kernel/KernelCode.ts} | 26 +-
.../src/kernel/KernelModel.ts | 161 +
.../apache-beam-jupyterlab-sidepanel/tsconfig.json | 10 +-
.../apache-beam-jupyterlab-sidepanel/yarn.lock | 3186 +++++++++++++++++++-
sdks/python/apache_beam/transforms/sql.py | 33 +-
sdks/python/container/base_image_requirements.txt | 2 +-
sdks/python/setup.py | 2 +-
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.23.0.md | 68 +
.../transforms/python/aggregation/latest.md | 57 +-
.../en/documentation/transforms/python/overview.md | 2 +-
.../www/site/content/en/get-started/downloads.md | 14 +-
website/www/site/data/io_matrix.yaml | 6 +
.../partials/section-menu/en/documentation.html | 1 +
website/www/site/static/.htaccess | 2 +-
204 files changed, 5466 insertions(+), 1264 deletions(-)
delete mode 100644 .test-infra/jenkins/job_PreCommit_BeamSQL_ZetaSQL.groovy
delete mode 100644
.test-infra/jenkins/job_PreCommit_BeamSQL_ZetaSQL_Java11.groovy
copy learning/katas/go/core_transforms/{combine/simple_function =>
composite/composite}/cmd/main.go (87%)
copy learning/katas/go/core_transforms/{map/pardo/pkg/task/task.go =>
composite/composite/pkg/common/common.go} (80%)
copy learning/katas/go/core_transforms/{combine/simple_function =>
composite/composite}/pkg/task/task.go (69%)
copy learning/katas/go/core_transforms/{combine/combinefn =>
composite/composite}/task-info.yaml (84%)
create mode 100644
learning/katas/go/core_transforms/composite/composite/task-remote-info.yaml
copy learning/katas/{java/Core Transforms/Composite Transform/Composite
Transform => go/core_transforms/composite/composite}/task.md (54%)
copy learning/katas/go/core_transforms/{combine/combine_perkey =>
composite/composite}/test/task_test.go (50%)
copy learning/katas/{java/Core Transforms/Branching =>
go/core_transforms/composite}/lesson-info.yaml (98%)
create mode 100644
learning/katas/go/core_transforms/composite/lesson-remote-info.yaml
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryResourceNaming.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryResourceNamingTest.java
create mode 100644 sdks/python/apache_beam/examples/sql_taxi.py
copy
sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/{src/index.ts
=> jest.config.js} (56%)
create mode 100644
sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/__tests__/kernel/KernelModel.test.ts
copy
sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/{index.ts
=> kernel/KernelCode.ts} (55%)
create mode 100644
sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/src/kernel/KernelModel.ts
create mode 100644 website/www/site/content/en/blog/beam-2.23.0.md
