This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/go_modules/sdks/github.com/aws/aws-sdk-go-v2/feature/s3/manager-1.15.15
in repository https://gitbox.apache.org/repos/asf/beam.git
discard d8f0088e9d7 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks
add 89d1c06e1ea Fix Fhir IO Search IT (#30158)
add a221f98a5f4 fix: fix error handling in segment reader and add an
integration test (#30147)
add ddb93272ab6 Fix git trusted directory errors in website build. (#30171)
add 30917ff7b3e Don't use '-' as a delimiter in BigQuerySinkMetrics
(#30118)
add fb0762352c2 Periodically send perWorkerMetrics on the WorkerMessages
API (#30135)
add 4f09e62f520 Improve varint encoding throughput with unrolled loop
(#29689)
add 5188e37ed44 Bump github.com/opencontainers/runc from 1.1.5 to 1.1.12
in /sdks (#30175)
add 96cf695a101 Fix JdbcIO PreCommit mysql breaking change (#30179)
add 824e06cd967 Allow stale_bq_datasets_cleaner.sh to run to completion,
even when there are failures to grab a dataset (#30153)
add 1d9f6044c76 Remove constraint introduced in #30059 (#30186)
add bdbbef9b54d [YAML] - PubSubLite proto (#30129)
add 1a5466b7df7 Support dict as the input of delete_batch (#30188)
add 574e7bfbc1d Fix script typos for validating docker licenses. (#30176)
add 7fb05cbb5e1 Move the code from sdks-java-fn-execution to sdks-java-core
add e8df60381a3 Merge pull request #30170: Move the code from
sdks-java-fn-execution to sdks-java-core
add 124880bc8ce Fix typo affecting
DataflowPipelineDebugOptions.setStreamingSideInputCacheExpirationMillis (#30195)
add 1f69271044d [Dataflow Streaming] Add a option to use multiple commit
threads (#30194)
add 1183b5d5274 Knowledge lookup prompts: (#29747)
add 801682930e2 Bump google.golang.org/protobuf from 1.31.0 to 1.32.0 in
/sdks (#29913)
add 30a778b9cbe Bump github.com/containerd/containerd from 1.7.7 to 1.7.11
in /sdks (#29823)
add f03b115830b Remove extraneous mocking that was causing NPEs in
DataflowWorkUnitClientTest
add 00f07864784 Remove sdks/java/fn-execution
add 8ec51f181a0 Merge pull request #30193: Remove sdks/java/fn-execution
add 9f8dda274dc Split standalone transform service application from
launcher jar (#30190)
add f2056e72dea Fix stale dataflow prebuilt image cleaner (#30168)
add 34bbbda9f98 Try a different approach for cibuildwheel flake. (#30204)
add 1a1ff756f86 Bump golang.org/x/sync from 0.5.0 to 0.6.0 in /sdks
(#30209)
add c1b3a27ee0c Bump github.com/nats-io/nats-server/v2 from 2.10.7 to
2.10.10 in /sdks (#30210)
add a47b1faa527 [Prism] Implement jobservices.Server Cancel (#30178)
add 7f91baadcf2 Upgrade website node to LTS (#30213)
add d5aa44c9ba9 Handle a user whose account was deleted. (#30192)
add a341eb69abe Added default watermark generation interval (apache#30141)
add cd5f2711c83 Merge pull request #30141: Adding a default watermark emit
interval for FlinkUnboundedSourceReader
add b0f2eebb024 When failing work items during commit, make sure to call
completeWork… (#30215)
add 23dcb7ec1d5 Revert "When failing work items during commit, make sure
to call completeWork…" (#30228)
add 5494f114382 [Dataflow Streaming] Invalidate caches and remove work on
failure before commit (#30229)
add 27f1c0774fd Bump github.com/tetratelabs/wazero from 1.5.0 to 1.6.0 in
/sdks (#30222)
add 78d08235c72 Add grpc-services, grpc-utils to vendored grpc (#30196)
add 3412b4f7a5b fix markdown links (#30227)
add c006c5e2d53 Add Default method for OutputReceiver.outputWindowedValue
(#30220)
add 08a717dc5c2 Creating a Fully Managed Beam Streaming System with Flink
Runner on Kubernetes - Part 3 (#29860)
add 7fa4c99535f Fix to take StreamingDataflowWorkerOptions from external
options (#30232)
add f08058ca75c Bump cloud.google.com/go/spanner from 1.53.1 to 1.56.0 in
/sdks (#30224)
add 7a46686577d Fix a naming conflict of the TransformService test suite
(#30243)
add 4645a76ee2a gcsio: reduce number of get requests in function calls
(#30205)
add f250a843230 Adjust transform-service module names and deps with
sdks/java/core
add 13a70844abb Merge pull request #30237: Adjust transform-service module
names and deps with sdks/java/core
add 4f3963f7878 Bump google.golang.org/grpc from 1.60.1 to 1.61.0 in /sdks
(#30249)
add 2c5d153bf29 Add InferenceAPIEmbeddings to huggingface (#30138)
add c72a9f83fed Fix Jms drop record (#30218)
add a15dd7e7f84 [Go SDK] Use distroless:debian12 (no-ssl) as base image.
(#30011)
add 9b4b509e52c Fail later when duplicate transform translators are on the
classpath
add 382c6dccfcc Merge pull request #30252: Fail later when duplicate
transform translators are on the classpath
add 24efe7619d9 Track windmill current active work budget. (#30048)
add fe262b630b3 [lint] Doc comment exported ErrCancelled. (#30251)
add 9361141c31b Add Drain to Job Management proto (#30157)
add ff272db75e6 [Proposal] Improve DisplayData support in PTransform API
(#30115)
add 619c0d9ca2c [YAML] Add ability to specify JDBC type to read/write
(#30024)
add c281852a58f [YAML] Add documentation to Beam site.
add 562667e4cf5 Add page titles.
add e9202abb7ad Merge pull request #30161 [YAML] Add documentation to Beam
site.
add 2d08b32e674 Add options giving more isolation between grpc streaming
rpcs. (#30233)
add 5a4b4584af0 Downgrade grpc version to 1.60.0. (#30260)
add 4c2b8b1e51e Fix change point analysis unit test (#30244)
add aefcada1103 Update artifacts fetcher to download artifacts locally
using FileSystems (#30202)
add 34cfcc36b5b Update to libraries-bom 26.31.0 , Add support for protos
and enums for spanner dataflow templates (#30181)
add 6d10e0f40fd Fix two flaky tests (#30278)
add 68873c81042 Update Python Dependencies (#30282)
add 371576a3b17 Remove unused StreamingDataflowWorker parameter (#30256)
add 47f777856bf fixed documentation in
sdks/python/apache_beam/io/gcp/pubsub.py WriteToPubSub (#30285)
add fa3249206d0 [JdbcIO] - Adding option for max batch buffering duration
(#30259)
add 702138bf6b8 Remove WriteToFiles is experimental. (#30291)
add b923a67369e fix bug in getProcessingTimesByStepCopy (#30270)
add 514b03bebcc check for cachetoken representing a retry before
activating and completing work (#29082)
add 0ade62cd083 Only populate DataflowHistogramValue::OutlierStats when
overflow/underflow values exist. (#30246)
add 3b09283e45d Bump github.com/docker/go-connections from 0.4.0 to 0.5.0
in /sdks (#30247)
add 8109e85355d Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (d8f0088e9d7)
\
N -- N -- N
refs/heads/dependabot/go_modules/sdks/github.com/aws/aws-sdk-go-v2/feature/s3/manager-1.15.15
(8109e85355d)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
...w.json => beam_PostCommit_Java_DataflowV2.json} | 0
.../beam_PostCommit_TransformService_Direct.yml | 2 +-
.../workflows/beam_PreCommit_Python_Coverage.yml | 2 +
.test-infra/metrics/sync/github/sync_workflows.py | 12 +-
.test-infra/tools/stale_bq_datasets_cleaner.sh | 34 +-
.../tools/stale_dataflow_prebuilt_image_cleaner.sh | 55 +-
CHANGES.md | 2 +
build.gradle.kts | 2 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 21 +-
.../apache/beam/gradle/GrpcVendoring_1_60_1.groovy | 6 +-
contributor-docs/release-guide.md | 6 +-
learning/prompts/README.md | 76 +
learning/prompts/code-explanation/01_io_kafka.md | 65 +
.../02_io_pubsub.md | 0
.../prompts/code-explanation/03_io_bigquery.md | 39 +
.../prompts/code-explanation/04_io_bigtable.md | 44 +
learning/prompts/code-explanation/05_io_spanner.md | 96 +
.../prompts/code-explanation/06_io_tfrecord.md | 63 +
learning/prompts/code-explanation/07_io_json.md | 50 +
learning/prompts/code-explanation/08_io_csv.md | 44 +
learning/prompts/code-explanation/09_io_avro.md | 41 +
learning/prompts/code-explanation/10_io_parquet.md | 39 +
learning/prompts/code-explanation/11_io_jdbc.md | 105 +
learning/prompts/code-generation/01_io_kafka.md | 46 +
learning/prompts/code-generation/03_io_bigquery.md | 54 +
learning/prompts/code-generation/04_io_bigtable.md | 66 +
learning/prompts/code-generation/05_io_spanner.md | 63 +
learning/prompts/code-generation/06_io_tfrecord.md | 60 +
learning/prompts/code-generation/07_io_json.md | 48 +
learning/prompts/code-generation/08_io_csv.md | 47 +
learning/prompts/code-generation/09_io_avro.md | 42 +
learning/prompts/code-generation/10_io_parquet.md | 39 +
learning/prompts/code-generation/11_io_jdbc.md | 75 +
.../prompts/documentation-lookup/18_io_kafka.md | 27 +
.../prompts/documentation-lookup/21_io_bigtable.md | 28 +
.../prompts/documentation-lookup/22_io_spanner.md | 31 +
learning/prompts/documentation-lookup/23_io_gcs.md | 23 +
.../prompts/documentation-lookup/24_io_json.md | 32 +
learning/prompts/documentation-lookup/25_io_csv.md | 32 +
.../prompts/documentation-lookup/26_io_avro.md | 31 +
.../prompts/documentation-lookup/27_io_parquet.md | 31 +
.../prompts/documentation-lookup/28_io_jdbc.md | 52 +
.../model/job_management/v1/beam_job_api.proto | 16 +
runners/core-construction-java/build.gradle | 1 -
.../core/construction/PTransformTranslation.java | 90 +-
.../runners/core/construction/SplittableParDo.java | 10 -
runners/core-java/build.gradle | 1 -
runners/direct-java/build.gradle | 3 -
runners/flink/flink_runner.gradle | 1 -
.../unbounded/FlinkUnboundedSourceReader.java | 36 +-
.../options/DataflowPipelineDebugOptions.java | 8 +-
.../google-cloud-dataflow-java/worker/build.gradle | 8 +-
.../worker/DataflowBatchWorkerHarness.java | 2 +-
.../dataflow/worker/DataflowExecutionContext.java | 42 +-
.../dataflow/worker/DataflowWorkUnitClient.java | 24 +-
.../worker/DataflowWorkerHarnessHelper.java | 8 +-
.../worker/MetricTrackingWindmillServerStub.java | 81 +-
.../MetricsToPerStepNamespaceMetricsConverter.java | 96 +-
.../dataflow/worker/StreamingDataflowWorker.java | 163 +-
.../worker/StreamingModeExecutionContext.java | 23 +-
.../worker/StreamingStepMetricsContainer.java | 8 +-
.../runners/dataflow/worker/WorkUnitClient.java | 17 +-
.../worker/WorkerPipelineOptionsFactory.java | 15 +-
.../dataflow/worker/counters/NameContext.java | 29 +-
.../options/StreamingDataflowWorkerOptions.java | 19 +
.../dataflow/worker/streaming/ActiveWorkState.java | 203 +-
.../worker/streaming/ComputationState.java | 14 +-
.../dataflow/worker/streaming/StageInfo.java | 16 +-
.../runners/dataflow/worker/streaming/Work.java | 12 +-
.../runners/dataflow/worker/streaming/WorkId.java | 31 +-
.../windmill/client/grpc/GrpcWindmillServer.java | 3 +-
.../client/grpc/stubs/IsolationChannel.java | 249 +++
.../client/grpc/stubs/WindmillChannelFactory.java | 8 +-
.../client/grpc/stubs/WindmillStubFactory.java | 17 +-
.../worker/DataflowExecutionContextTest.java | 9 +-
.../worker/DataflowWorkUnitClientTest.java | 132 +-
.../worker/DataflowWorkerHarnessHelperTest.java | 30 +-
...ricsToPerStepNamespaceMetricsConverterTest.java | 25 +-
.../worker/StreamingDataflowWorkerTest.java | 140 +-
.../worker/StreamingModeExecutionContextTest.java | 14 +-
.../worker/StreamingStepMetricsContainerTest.java | 14 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 4 +-
.../worker/WorkerPipelineOptionsFactoryTest.java | 4 +-
.../worker/streaming/ActiveWorkStateTest.java | 277 ++-
.../client/grpc/stubs/IsolationChannelTest.java | 403 ++++
runners/java-fn-execution/build.gradle | 1 -
runners/java-job-service/build.gradle | 1 -
runners/samza/build.gradle | 1 -
runners/spark/spark_runner.gradle | 1 -
scripts/ci/pr-bot/updateReviewers.ts | 1 +
sdks/go.mod | 58 +-
sdks/go.sum | 140 +-
sdks/go/container/Dockerfile | 15 +-
.../beam/model/fnexecution_v1/beam_fn_api.pb.go | 803 ++++---
.../jobmanagement_v1/beam_expansion_api.pb.go | 295 +--
.../beam/model/jobmanagement_v1/beam_job_api.pb.go | 887 ++++----
.../model/jobmanagement_v1/beam_job_api_grpc.pb.go | 38 +
.../beam/model/pipeline_v1/beam_runner_api.pb.go | 2234 ++++++++++----------
sdks/go/pkg/beam/runners/prism/internal/execute.go | 8 +
.../beam/runners/prism/internal/jobservices/job.go | 10 +
.../prism/internal/jobservices/management.go | 31 +
.../prism/internal/jobservices/management_test.go | 34 +
.../prism/internal/jobservices/server_test.go | 61 +
.../container/license_scripts/dep_urls_java.yaml | 2 +-
sdks/java/core/build.gradle | 2 +
.../apache/beam/sdk/jmh/util/VarIntBenchmark.java | 303 +++
.../org/apache/beam/sdk/fn/CancellableQueue.java | 0
.../java/org/apache/beam/sdk/fn/IdGenerator.java | 0
.../java/org/apache/beam/sdk/fn/IdGenerators.java | 0
.../org/apache/beam/sdk/fn/JvmInitializers.java | 0
.../sdk/fn/channel/AddHarnessIdInterceptor.java | 0
.../beam/sdk/fn/channel/ManagedChannelFactory.java | 0
.../beam/sdk/fn/channel/SocketAddressFactory.java | 0
.../apache/beam/sdk/fn/channel/package-info.java | 0
.../sdk/fn/data/BeamFnDataGrpcMultiplexer.java | 0
.../sdk/fn/data/BeamFnDataInboundObserver.java | 0
.../sdk/fn/data/BeamFnDataOutboundAggregator.java | 0
.../beam/sdk/fn/data/CloseableFnDataReceiver.java | 0
.../org/apache/beam/sdk/fn/data/DataEndpoint.java | 0
.../beam/sdk/fn/data/DecodingFnDataReceiver.java | 0
.../apache/beam/sdk/fn/data/FnDataReceiver.java | 0
.../apache/beam/sdk/fn/data/LogicalEndpoint.java | 0
.../beam/sdk/fn/data/RemoteGrpcPortRead.java | 0
.../beam/sdk/fn/data/RemoteGrpcPortWrite.java | 0
.../org/apache/beam/sdk/fn/data/TimerEndpoint.java | 0
.../org/apache/beam/sdk/fn/data/WeightedList.java | 0
.../org/apache/beam/sdk/fn/data/package-info.java | 0
.../java/org/apache/beam/sdk/fn/package-info.java | 0
.../org/apache/beam/sdk/fn/server/FnService.java | 0
.../server/GrpcContextHeaderAccessorProvider.java | 0
.../apache/beam/sdk/fn/server/GrpcFnServer.java | 0
.../apache/beam/sdk/fn/server/HeaderAccessor.java | 0
.../beam/sdk/fn/server/InProcessServerFactory.java | 0
.../apache/beam/sdk/fn/server/ServerFactory.java | 0
.../apache/beam/sdk/fn/server/package-info.java | 0
.../sdk/fn/splittabledofn/RestrictionTrackers.java | 0
.../sdk/fn/splittabledofn/WatermarkEstimators.java | 0
.../beam/sdk/fn/splittabledofn/package-info.java | 0
.../apache/beam/sdk/fn/stream/AdvancingPhaser.java | 0
.../sdk/fn/stream/BufferingStreamObserver.java | 0
.../org/apache/beam/sdk/fn/stream/DataStreams.java | 0
.../beam/sdk/fn/stream/DirectStreamObserver.java | 0
.../stream/ForwardingClientResponseObserver.java | 0
.../sdk/fn/stream/OutboundObserverFactory.java | 0
.../beam/sdk/fn/stream/PrefetchableIterable.java | 0
.../beam/sdk/fn/stream/PrefetchableIterables.java | 0
.../beam/sdk/fn/stream/PrefetchableIterator.java | 0
.../beam/sdk/fn/stream/PrefetchableIterators.java | 0
.../sdk/fn/stream/SynchronizedStreamObserver.java | 0
.../apache/beam/sdk/fn/stream/package-info.java | 0
.../org/apache/beam/sdk/fn/test/TestExecutors.java | 0
.../org/apache/beam/sdk/fn/test/TestStreams.java | 0
.../org/apache/beam/sdk/fn/test/package-info.java | 0
.../sdk/fn/windowing/EncodedBoundedWindow.java | 0
.../apache/beam/sdk/fn/windowing/package-info.java | 0
.../java/org/apache/beam/sdk/transforms/DoFn.java | 7 +-
.../org/apache/beam/sdk/transforms/PTransform.java | 30 +-
.../main/java/org/apache/beam/sdk/util/VarInt.java | 62 +-
.../apache/beam/sdk/fn/CancellableQueueTest.java | 0
.../org/apache/beam/sdk/fn/IdGeneratorsTest.java | 0
.../apache/beam/sdk/fn/JvmInitializersTest.java | 5 +-
.../sdk/fn/channel/ManagedChannelFactoryTest.java | 0
.../sdk/fn/channel/SocketAddressFactoryTest.java | 0
.../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java | 0
.../sdk/fn/data/BeamFnDataInboundObserverTest.java | 25 +-
.../fn/data/BeamFnDataOutboundAggregatorTest.java | 0
.../beam/sdk/fn/data/RemoteGrpcPortReadTest.java | 0
.../beam/sdk/fn/data/RemoteGrpcPortWriteTest.java | 0
.../fn/splittabledofn/RestrictionTrackersTest.java | 0
.../fn/splittabledofn/WatermarkEstimatorsTest.java | 0
.../beam/sdk/fn/stream/AdvancingPhaserTest.java | 0
.../sdk/fn/stream/BufferingStreamObserverTest.java | 0
.../apache/beam/sdk/fn/stream/DataStreamsTest.java | 0
.../sdk/fn/stream/DirectStreamObserverTest.java | 0
.../ForwardingClientResponseObserverTest.java | 0
.../sdk/fn/stream/OutboundObserverFactoryTest.java | 0
.../sdk/fn/stream/PrefetchableIterablesTest.java | 0
.../sdk/fn/stream/PrefetchableIteratorsTest.java | 0
.../apache/beam/sdk/fn/test/TestExecutorsTest.java | 0
.../apache/beam/sdk/fn/test/TestStreamsTest.java | 0
.../sdk/fn/windowing/EncodedBoundedWindowTest.java | 0
.../apache/beam/sdk/transforms/PTransformTest.java | 21 +
.../org/apache/beam/sdk/transforms/ParDoTest.java | 4 +
.../UnboundedScheduledExecutorServiceTest.java | 2 +-
sdks/java/expansion-service/build.gradle | 2 -
.../sdk/expansion/service/ExpansionService.java | 650 +++---
.../ExpansionServiceSchemaTransformProvider.java | 1 -
.../service/JavaClassLookupTransformProvider.java | 1 -
.../sdk/expansion/service/TransformProvider.java | 167 ++
.../expansion/service/ExpansionServiceTest.java | 11 +-
.../beam/sdk/expansion/service/ExternalTest.java | 2 +-
sdks/java/extensions/protobuf/build.gradle | 1 +
.../sdk/extensions/protobuf/ProtoByteUtils.java | 79 +-
.../extensions/protobuf/ProtoByteUtilsTest.java | 72 +
sdks/java/extensions/python/build.gradle | 2 +-
.../extensions/sql/zetasql/BeamZetaSqlCalcRel.java | 12 -
sdks/java/fn-execution/build.gradle | 43 -
sdks/java/harness/build.gradle | 6 +-
sdks/java/harness/jmh/build.gradle | 3 +
.../beam/fn/harness/BeamFnDataWriteRunner.java | 1 -
.../sdk/io/gcp/bigquery/BigQuerySinkMetrics.java | 37 +-
.../bigquery/StorageApiWriteUnshardedRecords.java | 20 -
...ueryStorageWriteApiSchemaTransformProvider.java | 7 -
.../sdk/io/gcp/bigtable/BigtableServiceImpl.java | 20 +-
.../PubsubLiteReadSchemaTransformProvider.java | 84 +-
.../PubsubLiteWriteSchemaTransformProvider.java | 51 +-
.../sdk/io/gcp/spanner/MutationSizeEstimator.java | 4 +
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 12 -
.../beam/sdk/io/gcp/spanner/SpannerSchema.java | 17 +-
.../io/gcp/bigquery/BigQuerySinkMetricsTest.java | 59 +-
.../beam/sdk/io/gcp/bigtable/BigtableReadIT.java | 93 +-
.../beam/sdk/io/gcp/healthcare/FhirIOSearchIT.java | 9 +-
.../gcp/pubsublite/internal/PubsubLiteDlqTest.java | 89 +
.../internal/PubsubLiteWriteDlqTest.java | 31 +
.../io/gcp/spanner/MutationSizeEstimatorTest.java | 54 +
.../beam/sdk/io/gcp/spanner/SpannerSchemaTest.java | 8 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 51 +-
.../io/jdbc/JdbcReadSchemaTransformProvider.java | 36 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java | 19 +
.../io/jdbc/JdbcWriteSchemaTransformProvider.java | 36 +-
.../beam/sdk/io/jdbc/JdbcIOAutoPartitioningIT.java | 5 +-
.../jdbc/JdbcReadSchemaTransformProviderTest.java | 76 +
.../jdbc/JdbcWriteSchemaTransformProviderTest.java | 98 +-
.../apache/beam/sdk/io/jms/JmsCheckpointMark.java | 201 +-
.../java/org/apache/beam/sdk/io/jms/JmsIO.java | 84 +-
.../java/org/apache/beam/sdk/io/jms/JmsIOIT.java | 45 +-
.../java/org/apache/beam/sdk/io/jms/JmsIOTest.java | 109 +-
.../beam/sdk/io/kafka/ReadFromKafkaDoFnTest.java | 11 -
.../beam/sdk/io/pulsar/ReadFromPulsarDoFnTest.java | 13 -
.../ReadFromSparkReceiverWithOffsetDoFnTest.java | 13 -
.../testing/expansion/TestExpansionService.java | 10 +-
.../{launcher => app}/build.gradle | 45 +-
sdks/java/transform-service/build.gradle | 3 +-
sdks/java/transform-service/launcher/build.gradle | 28 +-
.../examples/ml_transform/ml_transform_it_test.py | 4 +-
.../ml_transform/vocab_tfidf_processing.py | 6 +-
sdks/python/apache_beam/io/fileio.py | 2 -
sdks/python/apache_beam/io/gcp/gcsio.py | 55 +-
sdks/python/apache_beam/io/gcp/gcsio_test.py | 85 +-
sdks/python/apache_beam/io/gcp/pubsub.py | 4 +-
.../ml/transforms/embeddings/huggingface.py | 126 +-
.../ml/transforms/embeddings/huggingface_test.py | 51 +
sdks/python/apache_beam/ml/transforms/utils.py | 43 +-
.../testing/analyzers/perf_analysis_test.py | 6 -
.../utils/transform_service_launcher.py | 2 +-
sdks/python/apache_beam/yaml/standard_io.yaml | 26 +-
sdks/python/build.gradle | 37 +-
.../container/py310/base_image_requirements.txt | 49 +-
.../container/py311/base_image_requirements.txt | 49 +-
.../container/py38/base_image_requirements.txt | 47 +-
.../container/py39/base_image_requirements.txt | 49 +-
sdks/python/test-suites/tox/common.gradle | 4 +-
sdks/python/tox.ini | 1 +
settings.gradle.kts | 2 +-
vendor/grpc-1_60_1/build.gradle | 2 +-
website/Dockerfile | 9 +-
.../blog/apache-beam-flink-and-kubernetes-part3.md | 209 ++
.../www/site/content/en/contribute/runner-guide.md | 12 +-
.../content/en/documentation/programming-guide.md | 4 +-
.../content/en/documentation/sdks/yaml-combine.md | 170 ++
.../content/en/documentation/sdks/yaml-errors.md | 204 ++
.../en/documentation/sdks/yaml-inline-python.md | 206 ++
.../site/content/en/documentation/sdks/yaml-udf.md | 252 +++
.../www/site/content/en/documentation/sdks/yaml.md | 591 ++++++
website/www/site/data/pipelines.yaml | 3 +
.../layouts/partials/section-menu/en/sdks.html | 11 +
.../adaptive_scheduler_rescale.png | Bin 0 -> 859268 bytes
.../auto-tuned-worker.png | Bin 0 -> 114222 bytes
.../backlog_growth.png | Bin 0 -> 6568 bytes
.../backlog_time.png | Bin 0 -> 6127 bytes
.../cpurate_desired.png | Bin 0 -> 10111 bytes
.../flink-operator-chaining.png | Bin 0 -> 529398 bytes
.../operator-backlog.png | Bin 0 -> 350363 bytes
.../resource-allocation.png | Bin 0 -> 131782 bytes
.../worker_extra.png | Bin 0 -> 11497 bytes
.../worker_require.png | Bin 0 -> 11975 bytes
.../worker_scaleup.png | Bin 0 -> 7471 bytes
website/www/site/static/images/logos/sdks/yaml.png | Bin 0 -> 6197 bytes
278 files changed, 10284 insertions(+), 3587 deletions(-)
rename .github/trigger_files/{beam_PostCommit_Python_Xlang_Gcp_Dataflow.json
=> beam_PostCommit_Java_DataflowV2.json} (100%)
create mode 100644 learning/prompts/README.md
create mode 100644 learning/prompts/code-explanation/01_io_kafka.md
rename learning/prompts/{code-explaination =>
code-explanation}/02_io_pubsub.md (100%)
create mode 100644 learning/prompts/code-explanation/03_io_bigquery.md
create mode 100644 learning/prompts/code-explanation/04_io_bigtable.md
create mode 100644 learning/prompts/code-explanation/05_io_spanner.md
create mode 100644 learning/prompts/code-explanation/06_io_tfrecord.md
create mode 100644 learning/prompts/code-explanation/07_io_json.md
create mode 100644 learning/prompts/code-explanation/08_io_csv.md
create mode 100644 learning/prompts/code-explanation/09_io_avro.md
create mode 100644 learning/prompts/code-explanation/10_io_parquet.md
create mode 100644 learning/prompts/code-explanation/11_io_jdbc.md
create mode 100644 learning/prompts/code-generation/01_io_kafka.md
create mode 100644 learning/prompts/code-generation/03_io_bigquery.md
create mode 100644 learning/prompts/code-generation/04_io_bigtable.md
create mode 100644 learning/prompts/code-generation/05_io_spanner.md
create mode 100644 learning/prompts/code-generation/06_io_tfrecord.md
create mode 100644 learning/prompts/code-generation/07_io_json.md
create mode 100644 learning/prompts/code-generation/08_io_csv.md
create mode 100644 learning/prompts/code-generation/09_io_avro.md
create mode 100644 learning/prompts/code-generation/10_io_parquet.md
create mode 100644 learning/prompts/code-generation/11_io_jdbc.md
create mode 100644 learning/prompts/documentation-lookup/18_io_kafka.md
create mode 100644 learning/prompts/documentation-lookup/21_io_bigtable.md
create mode 100644 learning/prompts/documentation-lookup/22_io_spanner.md
create mode 100644 learning/prompts/documentation-lookup/23_io_gcs.md
create mode 100644 learning/prompts/documentation-lookup/24_io_json.md
create mode 100644 learning/prompts/documentation-lookup/25_io_csv.md
create mode 100644 learning/prompts/documentation-lookup/26_io_avro.md
create mode 100644 learning/prompts/documentation-lookup/27_io_parquet.md
create mode 100644 learning/prompts/documentation-lookup/28_io_jdbc.md
copy
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/TimerEndpoint.java
=>
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/WorkId.java
(50%)
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/IsolationChannel.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/IsolationChannelTest.java
create mode 100644
sdks/java/core/jmh/src/main/java/org/apache/beam/sdk/jmh/util/VarIntBenchmark.java
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/CancellableQueue.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/IdGenerator.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/IdGenerators.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/JvmInitializers.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/channel/AddHarnessIdInterceptor.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactory.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/channel/SocketAddressFactory.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/channel/package-info.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundAggregator.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/CloseableFnDataReceiver.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/DataEndpoint.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/DecodingFnDataReceiver.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/FnDataReceiver.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/LogicalEndpoint.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortRead.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortWrite.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/TimerEndpoint.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/WeightedList.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/package-info.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/package-info.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/server/FnService.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/server/GrpcContextHeaderAccessorProvider.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/server/GrpcFnServer.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/server/HeaderAccessor.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/server/InProcessServerFactory.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/server/ServerFactory.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/server/package-info.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackers.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/splittabledofn/WatermarkEstimators.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/splittabledofn/package-info.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/AdvancingPhaser.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/BufferingStreamObserver.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/DataStreams.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/DirectStreamObserver.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/ForwardingClientResponseObserver.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/OutboundObserverFactory.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterable.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterables.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterator.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterators.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/SynchronizedStreamObserver.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/package-info.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/test/TestExecutors.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/test/TestStreams.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/test/package-info.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/windowing/EncodedBoundedWindow.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/windowing/package-info.java (100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/CancellableQueueTest.java (100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/IdGeneratorsTest.java (100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/JvmInitializersTest.java (95%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactoryTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/channel/SocketAddressFactoryTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserverTest.java
(91%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundAggregatorTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortReadTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortWriteTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackersTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/splittabledofn/WatermarkEstimatorsTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/stream/AdvancingPhaserTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/stream/BufferingStreamObserverTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/stream/DataStreamsTest.java (100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/stream/DirectStreamObserverTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/stream/ForwardingClientResponseObserverTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/stream/OutboundObserverFactoryTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/stream/PrefetchableIterablesTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/stream/PrefetchableIteratorsTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/test/TestExecutorsTest.java (100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/test/TestStreamsTest.java (100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/windowing/EncodedBoundedWindowTest.java
(100%)
create mode 100644
sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/TransformProvider.java
delete mode 100644 sdks/java/fn-execution/build.gradle
copy sdks/java/transform-service/{launcher => app}/build.gradle (53%)
create mode 100644
website/www/site/content/en/blog/apache-beam-flink-and-kubernetes-part3.md
create mode 100644
website/www/site/content/en/documentation/sdks/yaml-combine.md
create mode 100644
website/www/site/content/en/documentation/sdks/yaml-errors.md
create mode 100644
website/www/site/content/en/documentation/sdks/yaml-inline-python.md
create mode 100644 website/www/site/content/en/documentation/sdks/yaml-udf.md
create mode 100644 website/www/site/content/en/documentation/sdks/yaml.md
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/adaptive_scheduler_rescale.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/auto-tuned-worker.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/backlog_growth.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/backlog_time.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/cpurate_desired.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/flink-operator-chaining.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/operator-backlog.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/resource-allocation.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/worker_extra.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/worker_require.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/worker_scaleup.png
create mode 100644 website/www/site/static/images/logos/sdks/yaml.png
