gyli commented on issue #40962: URL: https://github.com/apache/airflow/issues/40962#issuecomment-2246388296
Hi @raphaelauv, I agree that such operator is a "friendly" XTable trigger and can be built on top of BashOperator. The example code you showed is exactly how I will build it, but my point here is it should be an Airflow community managed provider. Just like Airflow also offers Iceberg hook and DatabricksSQLOperator in the corresponding providers, I believe XTable should also be added, as data engineering industry is embracing unified data format. For example, [Microsoft Fabric and OneLake is adopting XTable](https://clouddamcdnprodep.azureedge.net/gdc/gdc5iZCgN/original). I can understand that this Apache project is still in incubating stage, and Airflow might want to hold until it closer to an industry standard. While from a data engineer perspective, such unified datalake format is the way we are heading to, and I don't see a reason asking users to create custom operator for such work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
