GitHub user ParagEkbote created a discussion: Orchestrating Hugging Face Datasets with Apache Airflow
I've worked on a contribution which integrates Hugging Face Datasets with Apache Airflow through a dedicated provider, bringing dataset operations into Airflow's orchestration model. The provider ships with: - **Dataset operators:** `LoadDatasetOperator`, `MapDatasetOperator`, `StreamDatasetOperator`, and `PushToHubOperator`. - **Hugging Face connection and hook:** Centralized authentication and access to Hub APIs. - **Artifact management:** Durable dataset artifacts with metadata manifests exchanged through XCom. - **Object storage integration:** Support for local and remote storage backends through Airflow's `ObjectStoragePath`. - **OpenLineage integration:** Dataset lineage and provenance tracking for Hugging Face repositories. The integration is designed around familiar Airflow abstractions while retaining the underlying Hugging Face Datasets workflows. A big thank you to [Quentin Lhoest](https://github.com/lhoestq) for his technical stewardship and guidance, [Kaxil Naik](https://github.com/kaxil) for sponsoring and championing the integration and the wider Apache Airflow community for their support throughout the process. **Read the article:** [Load, Transform & Publish at Scale: Introducing the Hugging Face Provider for Apache Airflow](https://huggingface.co/blog/AINovice2005/airflow-hf-provider) GitHub link: https://github.com/apache/airflow/discussions/77028 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
