Hi Beam devs, I'd like to propose adding Vespa as a write target for the Python RAG ingestion module (apache_beam.ml.rag.ingestion), alongside the existing BigQuery, Postgres/AlloyDB/Cloud SQL, MySQL, Spanner, Milvus and Qdrant configs. I'm currently working on the Qdrant enrichment handler (#40629) and would like to follow it with Vespa.
The short version: - A VespaWriteConfig that plugs into VectorDatabaseWriteTransform and maps EmbeddableItem to Vespa document fields: id, content, metadata, dense embedding (indexed tensor) and sparse embedding (mapped tensor). - Writes go through pyvespa's feed_iterable from a batching DoFn, with per-document error collection; puts are idempotent by document id, so bundle retries are safe. - The module only feeds documents. Users deploy their own Vespa application package and schema. - pyvespa (Apache-2.0) as a new optional extra, apache-beam[vespa]. - A retrieval (enrichment) handler would come as a separate follow-up PR. Design doc (open for comments): https://docs.google.com/document/d/1cC5wezcXugmyHDbt-xuP6zrlKia337dJObWNKnZZJvM A few questions where I'd value maintainers' input: 1. Is Vespa a welcome addition to ml.rag ingestion? 2. pyvespa as an optional extra, or a thin client over Vespa's /document/v1 HTTP API to keep dependencies small? 3. testcontainers-python has no Vespa module, so the IT would start vespaengine/vespa with a generic container and deploy a minimal app package (tens of seconds). Is it fine to run it with the Milvus/Qdrant ITs, or should it be postcommit only? 4. Default operation: full put, or partial update with create=true? If there are no objections, I plan to open a GitHub issue and a first PR in early November. Thanks, Wangshu
