gopidesupavan commented on PR #62867: URL: https://github.com/apache/airflow/pull/62867#issuecomment-4183853446
Also, I have a question: why are changes related to DataFusion required within the existing providers? As I mentioned earlier, there was an initial idea to introduce a separate provider—something like apache-airflow-providers-apache-datafusion. DataFusion isn’t limited to object storage support alone. There are also table provider capabilities to consider. Currently, DataFusion supports systems like Iceberg, Delta Lake, and Hudi. I’ve also been involved in discussions (https://github.com/datafusion-contrib/datafusion-table-providers ) about integrating these functionalities more directly into DataFusion itself, so that everything works more seamlessly and delivers better performance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
