bjoernpollex-sc commented on issue #4743: [AIRFLOW-3871] render Operators template fields recursively URL: https://github.com/apache/airflow/pull/4743#issuecomment-473192025 @ashb Suppose you have a `GCSFileTransformOperator` that applies any given transformation. The problem is that you'll need to duplicate all of the arguments of that transformation on the operator, which needs to propagate them. One way to do this is to have a dictionary for those arguments: ```python transform = GCSFileTransformOperator( source="some/path/{{ds}}/file.xls", target="other/path/{{ds}}/file.csv", transform=ExcelToCSV, transform_args={"quote": True, separator: ","} ) ``` This works, but what if the operator has multiple such components as inputs? Consider the common file-system interfaces proposed in [AIP-14][1]. With those, we can make the operator more generic: ```python transform = FileTransformOperator( source=GCSFileSource, source_args={"path": "some/path/{{ds}}/file.xls", "bucket": "some_gcs_bucket", "conn_id": "gcs_conn"), target=SFTPFileTarget, target_args={"path": "other/path/{{ds}}/file.csv", "conn_id": "sft_conn", "retries": 3}, transform=ExcelToCSV, transform_args={"quote": True, "separator": ","} ) ``` This starts becoming impractical, it would be much simpler to just write: ```python transform = FileTransformOperator( source=GCSFileSource(path="some/path/{{ds}}/file.xls", bucket="some_gcs_bucket", conn_id="gcs_conn"), target=SFTPFileTarget(path="other/path/{{ds}}/file.csv", conn_id="sft_conn", retries=3), transform=ExcelToCSV(quote=True, separator=",") ) ``` This is essentially my use-case in a nutshell. Regarding your concern that this might lead to confusion - that's why I argued for an approach that requires explicit declaration of template fields in all objects/classes that support templates, because that leads to the least surprises (it's explicit, and it follows the current convention for operators). This can easily be added to existing classes, either by monkey-patching or sub-classes, so it's also quite flexible. [1]: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-14+Create+composable+operators+using+common+interfaces
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
