This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 2f93d8bc199 fix: cover bigquery datetime fraction 1 to 6 or absent
(#32146)
add 780eef98083 Replace StateTag.StateBinder to top level StateBinder in
SparkStateInternals (#31798)
add ab4ada4ff40 Skip most bigtableIO write error handling test on Dataflow
runner (#32048)
add 29de91383f3 Bump cloud.google.com/go/bigtable from 1.28.0 to 1.29.0 in
/sdks (#32151)
add f4e43148118 Bump cloud.google.com/go/pubsub from 1.40.0 to 1.41.0 in
/sdks (#32149)
add 8b6f37b2874 Fix broken Beam Quest URL in README.md (#32145)
add 2ce0ee34470 Added insertion and enrichment pipeline (#31657)
add 9aaf7e41dec fix link (#32156)
add edf4c4f5f19 Bump golang.org/x/sys from 0.23.0 to 0.24.0 in /sdks
(#32150)
add df106091290 Update names.py with container image tag (#32160)
add 74668038c02 Support withFormatRecordOnFailureFunction() for BigQuery
STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE methods (#31659)
No new revisions were added by this update.
Summary of changes:
...beam_PostCommit_Java_ValidatesRunner_Spark.json | 3 +-
...a_ValidatesRunner_SparkStructuredStreaming.json | 3 +-
...stCommit_Java_ValidatesRunner_Spark_Java11.json | 3 +-
CHANGES.md | 1 +
README.md | 2 +-
.../beam-ml/rag_usecase/beam_rag_notebook.ipynb | 1795 ++++++++++++++++++++
.../beam-ml/rag_usecase/chunks_generation.py | 129 ++
.../beam-ml/rag_usecase/redis_connector.py | 349 ++++
.../beam-ml/rag_usecase/redis_enrichment.py | 110 ++
.../org/apache/beam/runners/core/StateTag.java | 5 +-
runners/google-cloud-dataflow-java/build.gradle | 17 +-
.../spark/stateful/SparkStateInternals.java | 108 +-
sdks/go.mod | 26 +-
sdks/go.sum | 60 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 18 +-
.../sdk/io/gcp/bigquery/SplittingIterable.java | 18 +-
.../io/gcp/bigquery/StorageApiConvertMessages.java | 8 +-
.../bigquery/StorageApiDynamicDestinations.java | 2 +-
.../StorageApiDynamicDestinationsBeamRow.java | 16 +-
...StorageApiDynamicDestinationsGenericRecord.java | 20 +-
.../StorageApiDynamicDestinationsProto.java | 31 +-
.../StorageApiDynamicDestinationsTableRow.java | 16 +-
.../io/gcp/bigquery/StorageApiWritePayload.java | 22 +-
.../bigquery/StorageApiWriteUnshardedRecords.java | 65 +-
.../bigquery/StorageApiWritesShardedRecords.java | 26 +-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 336 +++-
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
27 files changed, 3028 insertions(+), 163 deletions(-)
create mode 100644
examples/notebooks/beam-ml/rag_usecase/beam_rag_notebook.ipynb
create mode 100644 examples/notebooks/beam-ml/rag_usecase/chunks_generation.py
create mode 100644 examples/notebooks/beam-ml/rag_usecase/redis_connector.py
create mode 100644 examples/notebooks/beam-ml/rag_usecase/redis_enrichment.py
