Hi Airflow community, I'd like to propose introducing a new community provider for Hugging Face. Integration overview
Hugging Face <https://huggingface.co/> is an open-source AI platform and community that provides tools and infrastructure for building, sharing and deploying machine learning models, datasets and applications. Hugging Face Datasets is widely used in data engineering and machine learning workflows, and a first-class Airflow provider would make dataset orchestration, transformation, and publishing simpler. The provider initially focuses on Hugging Face Datasets, exposing a HuggingFaceHook and native operators for loading datasets with revision pinning, applying user-defined transformations and publishing datasets to the Hugging Face Hub. Future iterations may extend the provider to Hugging Face Models and Inference Endpoints. System tests The provider integrates with the live Hugging Face Hub. System tests are included in the PR and exercise dataset loading, transformation and publishing workflows using the airflow-hf <https://huggingface.co/airflow-hf> Hugging Face organization. Proposed stewards Parag Ekbote, @ParagEkbote <https://github.com/ParagEkbote>, < [email protected]> Quentin Lhoest, @lhoestq <https://github.com/lhoestq>, < [email protected]> Sponsoring committer(s) Kaxil Naik, @kaxil <https://github.com/kaxil>, <[email protected]> Working implementation <https://github.com/apache/airflow/pull/76317> Incubation commitment We commit to: - Maintaining the provider and responding to issues within a reasonable time. - Meeting the incubation health metrics within six months. - Participating in quarterly governance updates. Best regards, Parag Ekbote
