Hello Airflow Community, I am starting this discussion to propose the addition of a new community provider for *Voyage AI*.
Voyage AI is a service that specializes in high-performance vector embeddings for AI applications, offering state-of-the-art models for text representation. This provider aims to integrate Voyage AI's capabilities directly into Airflow DAGs, making it easy for users to build powerful AI and RAG (Retrieval-Augmented Generation) pipelines. The initial Pull Request with the implementation, tests, and documentation can be found here: https://github.com/apache/airflow/pull/55671 *Provider Details:* The initial version of the provider includes the following components: - *VoyageAiEmbeddingOperator:* An operator to generate text embeddings using the Voyage AI API. - *VoyageAiHook:* A hook to manage the connection and authentication with the Voyage AI service. *Justification:* As AI-native workflows become more common, integrating with specialized services like Voyage AI is crucial. This provider will enable data engineers and ML practitioners to: - Easily vectorize large datasets as part of their ETL/ELT processes. - Build and automate RAG pipelines that require up-to-date vector embeddings. - Leverage Airflow's orchestration capabilities for complex AI workflows without writing extensive boilerplate code. *Community Provider Checklist:* I have reviewed the criteria for new community providers and believe this submission complies with them: - *Relevant:* The provider integrates a relevant service for the data and AI community. - *License:* All dependencies have licenses compatible with the Apache License 2.0. The main dependency is voyageai which is licensed under Apache 2.0. - *Mature:* Voyage AI is a stable and publicly available service. - *No new core dependencies:* The provider does not add new dependencies to the core Airflow package. - *Naming:* The provider package will be named apache-airflow-providers-voyageai. - *Documentation & Tests:* The PR includes documentation, usage examples, and unit tests for the new components. I am looking forward to hearing your feedback, thoughts, and suggestions on this proposal. Thank you! -- *Arthur Raulino Kretzer* Desenv De Software | Sw CDM | Centro de Convergência Digital e Mecatrônica Fundação CERTI [email protected] (48) 9926-3500 www.certi.org.br <https://www.certi.org.br/> <https://www.facebook.com/FundacaoCerti> <https://www.instagram.com/fundacaocerti/> <https://www.youtube.com/user/FundacaoCERTI> <https://www.linkedin.com/company/fundacao-certi/> Esta mensagem (incluindo arquivos anexos) contém informações confidenciais e é dirigida exclusivamente ao seu destinatário, sendo proibido e sujeito a sanções penais qualquer ato de divulgação, utilização, ou reprodução (total ou parcial) das informações nela contidas, caso não seja seu destinatário.
