MaksYermak commented on code in PR #49768:
URL: https://github.com/apache/airflow/pull/49768#discussion_r2746300338
##########
providers/google/src/airflow/providers/google/cloud/transfers/s3_to_gcs.py:
##########
@@ -335,7 +336,15 @@ def submit_transfer_jobs(self, files: list[str], gcs_hook:
GCSHook, s3_hook: S3H
return job_names
- def execute_complete(self, context: Context, event: dict[str, Any]) ->
None:
+ @overload
+ def execute_complete(self, context: Context, event: dict[str, Any], files:
None) -> None: ...
+ @overload
+ def execute_complete(
+ self, context: Context, event: dict[str, Any], files: Iterable[str]
+ ) -> list[str]: ...
+ def execute_complete(
+ self, context: Context, event: dict[str, Any], files: Iterable[str] |
None = None
+ ) -> list[str] | None:
Review Comment:
@AlexisBRENON yes, sure
`self.defer()` is a method from the `BaseOperator` and the main purpose of
this method is to say to the Operator that the execution will continue on the
`triggerer` process using `trigger` code. And when execution on `triggerer` is
finished then the code backs to the method specified in the `method_name`
attribute and continues execution on the `worker` process. All attributes in
the `defer` method are needed for start and configure deferable execution. All
business related attributes are better to pass to the `trigger`. Also, as I see
it is not a common practice in Airflow using `kwargs` in `defer` for business
related attributes. I found only one
[operator](https://github.com/apache/airflow/blob/main/providers/http/src/airflow/providers/http/operators/http.py#L305C1-L314C14)
which did it.
Because it is not a common practice and attributes in `defer` are not
intended for business logic I think the solution to pass this attribute to
`trigger` is better.
P.S. We can look for one more solution for this issue. We can make the
`s3_objects` variable as a private attribute for the `S3ToGCSOperator` class
and return it in the `execute_complete` method. But it should be checked that
this attribute will not be `None` after returning to execution on `worker`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
