Hello Beam Dev Community, I'm happy to share the design document for the Milvus Vector Sink I/O Connector for Apache Beam, developed as part of my GSoC 2025 project.
This sink I/O connector enables both streaming and batch pipelines to persist features into Tecton's online and offline feature stores. It integrates Tecton's capabilities into Beam workflows, allowing pipelines to store newly computed or updated feature data. It is essential for enabling real-time feature serving, continuous feature engineering, and managing the full lifecycle of feature data in modern machine learning systems. Here is the link to the design document: https://docs.google.com/document/d/16MvDv1EQuNosucsLA2KA4PXYxN8GunAHhTOp5HMIcks/edit?usp=sharing This implementation is part of the GSoC 2025 ML Integration project being tracked here: https://github.com/apache/beam/issues/35046 I welcome any feedback, suggestions, or questions about the design approach. Thank you, Mohamed
