TobKed commented on pull request #12814:
URL: https://github.com/apache/airflow/pull/12814#issuecomment-763723606
> Is there a commit of the new changes I can review, or do I need to review
the whole PR? I do not recall much from my previous review. What are the new
changes?
@aaltay I squashed commits so first one represents old changes, however I
think review of all changes is required. The most important files are hooks and
operators for Beam and Dataflow:
* `airflow/providers/google/cloud/hooks/dataflow.py`
* `airflow/providers/google/cloud/operators/dataflow.py`
* `airflow/providers/apache/beam/operators/beam.py`
* `airflow/providers/apache/beam/hooks/beam.py`
After change Beam hook handles the logic responsible for **starting**
pipelines for all runners. If runner is Dataflow then Dataflow specific logic
is responsible for handling a waiting for the proper status, cancelling etc.
The background for this change is that user may test pipeline with
`DirectRunner`, then just change runner parameter to `DataflowRunner` to run it
on GCP. Previously he had to change operator from Beam to Dataflow.
All changes are backward compatible so there should be no change in use of
Dataflow operators and hooks for the users.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
