sohailshk opened a new pull request, #41282: URL: https://github.com/apache/beam/pull/41282
Fixes #41233 The output functions of MLTransform embeddings created with `columns=` (`_dict_output_fn`), of the RAG embedding adapters (`_add_embedding_fn`) and of `VertexAIMultiModalEmbeddings` (`_multimodal_dict_output_fn`) assigned the embeddings onto the input elements. The programming guide warns that "Changes to input elements may be visible to other DoFns executing within the same bundle, and may cause correctness issues", and that happens here: another consumer of the same PCollection (for example `beam.combiners.ToList()`) received the embedded values. The returned element has to be a different object from the input, so each function now returns a shallow copy (`copy.copy` for the dict, `dataclasses.replace` for the items and the nested Vertex image/video/text objects); the values themselves are not copied. Tests: one per function (the Vertex one uses a stand-in response, no GCP); each fails on master. `pytest apache_beam/ml/transforms/base_test.py apache_beam/ml/rag/embeddings/base_test.py apache_beam/ml/transforms/embeddings/vertex_ai_test.py`: 40 passed, 12 skipped (TFT tests not installed). Two-branch pipeline (input into MLTransform with a test handler that reverses the text, and into `ToList()`): on master the `ToList` branch received `cba`/`def` for the inputs `abc`/`def` on the FnApiRunner and `cba`/`fed` on the default runner (Prism); which elements are hit depends on how the runner fuses the stages. After the change it receives `abc`/`def` on both. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
