Hi Danny, all, Thanks Danny for the detailed review on the doc. Here is a summary of what changed since my last update, for anyone following on the list.
Design changes (doc updated): - Dependency: the writer now goes through pyvespa's feed client again, as an optional apache-beam[vespa] extra, the same pattern as the Qdrant and Milvus writers. This reverses the thin /document/v1 client from my last email; the thin client is now listed under Alternatives. This resolves question 2. - Credentials: auth_token accepts a plain string or an apache_beam.utils.secret.Secret (e.g. GcpSecret), resolved in setup() on the workers. mTLS (cert/key) and auth_token are alternatives; setting both raises ValueError. - Failure handling: the writer fails the bundle on documents that keep failing, as the Qdrant writer does. A dead-letter output would belong in VectorDatabaseWriteTransform, shared by all writers, so it is out of scope here. This resolves question 5. - Testing: the container-based integration test is now part of the first PR. - Future work: Danny pointed out that exposing the writer to Java pipelines through cross-language is an extension of this design rather than an alternative, since it needs the Python implementation first. The doc now has a Future work section for it, and I'll keep the config to types a Java caller can pass across. Progress: - The draft PR is updated to match: https://www.google.com/url?q=https://github.com/apache/beam/pull/40881&source=gmail&ust=1794097627726000&sa=E Still open: 3. Should the container-based IT run with the Milvus/Qdrant ITs (proposed), or postcommit only? 4. Default operation: full put (proposed), or partial update with create=true? Design doc: https://www.google.com/url?q=https://docs.google.com/document/d/1cC5wezcXugmyHDbt-xuP6zrlKia337dJObWNKnZZJvM&source=gmail&ust=1794097627726000&sa=E Thanks again, Wangshu On Fri, Oct 30, 2026 06:01 AM, Danny McCormick via dev <[email protected]> wrote: > Indeed, this went to my spam as well. Chimed in on the doc, but adding > this sounds nice - thanks! > > Thanks, > Dann > > On Wed, Oct 28, 2026 at 1:34 PM Kenneth Knowles <[email protected]> wrote: > >> Hi Wangshu, >> >> Thanks for writing this. I am afraid that gmail flagged this thread as >> spam, so other people may not have seen it. I'm replying to boost it to the >> list if I can. >> https://lists.apache.org/thread.html/7jr1l0ojljzdt3p13396nzp2vlbzzlkx >> >> Kenn >> >> On Tue, Oct 27, 2026 at 1:33 AM Pang Wangshu <[email protected]> >> wrote: >> >>> Hi all, >>> >>> Following up on this proposal: I've opened a tracking issue so the >>> discussion is also visible on GitHub: >>> https://github.com/apache/beam/issues/40837 >>> >>> It has the same summary, the four open questions, and a link to the >>> design doc. Feedback is welcome either here or on the issue. The question >>> I'd most like input on is #2 (pyvespa as an optional extra, which pulls in >>> docker, lxml, fastcore and httpx, vs. a thin client over Vespa's >>> /document/v1 HTTP API), since it decides the dependency footprint of the >>> first PR. >>> >>> Thanks, >>> Wangshu >>> >>> On Sat, Oct 24, 2026 09:49 AM, Pang Wangshu <[email protected]> >>> wrote: >>> >>>> Hi Beam devs, >>>> >>>> I'd like to propose adding Vespa as a write target for the Python RAG >>>> ingestion module (apache_beam.ml.rag.ingestion), alongside the >>>> existing BigQuery, Postgres/AlloyDB/Cloud SQL, MySQL, Spanner, Milvus and >>>> Qdrant configs. I'm currently working on the Qdrant enrichment handler >>>> (#40629) and would like to follow it with Vespa. >>>> >>>> The short version: >>>> - A VespaWriteConfig that plugs into VectorDatabaseWriteTransform and >>>> maps EmbeddableItem to Vespa document fields: id, content, metadata, dense >>>> embedding (indexed tensor) and sparse embedding (mapped tensor). >>>> - Writes go through pyvespa's feed_iterable from a batching DoFn, with >>>> per-document error collection; puts are idempotent by document id, so >>>> bundle retries are safe. >>>> - The module only feeds documents. Users deploy their own Vespa >>>> application package and schema. >>>> - pyvespa (Apache-2.0) as a new optional extra, apache-beam[vespa]. >>>> - A retrieval (enrichment) handler would come as a separate follow-up >>>> PR. >>>> >>>> Design doc (open for comments): >>>> https://docs.google.com/document/d/1cC5wezcXugmyHDbt-xuP6zrlKia337dJObWNKnZZJvM >>>> >>>> >>>> A few questions where I'd value maintainers' input: >>>> 1. Is Vespa a welcome addition to ml.rag ingestion? >>>> 2. pyvespa as an optional extra, or a thin client over Vespa's >>>> /document/v1 HTTP API to keep dependencies small? >>>> 3. testcontainers-python has no Vespa module, so the IT would start >>>> vespaengine/vespa with a generic container and deploy a minimal app package >>>> (tens of seconds). Is it fine to run it with the Milvus/Qdrant ITs, or >>>> should it be postcommit only? >>>> 4. Default operation: full put, or partial update with create=true? >>>> >>>> If there are no objections, I plan to open a GitHub issue and a first >>>> PR in early November. >>>> >>>> Thanks, >>>> Wangshu >>>> >>>
