Re: [PR] Added JOB_STATE_CANCELLED and pool_sleep GCP Dataflow Operators [airflow]
github-actions[bot] closed pull request #37364: Added JOB_STATE_CANCELLED and pool_sleep GCP Dataflow Operators URL: https://github.com/apache/airflow/pull/37364 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Added JOB_STATE_CANCELLED and pool_sleep GCP Dataflow Operators [airflow]
github-actions[bot] commented on PR #37364: URL: https://github.com/apache/airflow/pull/37364#issuecomment-2067413940 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Added JOB_STATE_CANCELLED and pool_sleep GCP Dataflow Operators [airflow]
jcamatta commented on PR #37364: URL: https://github.com/apache/airflow/pull/37364#issuecomment-1978669074 Hi, eladkal. If you dont mind, I will try to take this in the weekend since right now i am full of work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Added JOB_STATE_CANCELLED and pool_sleep GCP Dataflow Operators [airflow]
eladkal commented on PR #37364: URL: https://github.com/apache/airflow/pull/37364#issuecomment-1975076963 I fixed some of the static checks and rebased. @jcamatta there are additional static checks that requires fixing. Can you please handle it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Added JOB_STATE_CANCELLED and pool_sleep GCP Dataflow Operators [airflow]
eladkal commented on code in PR #37364:
URL: https://github.com/apache/airflow/pull/37364#discussion_r1510206510
##
tests/providers/google/cloud/operators/test_dataflow.py:
##
@@ -577,6 +578,24 @@ def test_start_with_location(self, dataflow_mock):
assert not kwargs["variables"]
assert kwargs["location"] == TEST_LOCATION
+@pytest.mark.parametrize(
+"status, message",
+[
+("error", f"Dataflow job with id {JOB_ID} has failed its
execution"),
+("stopped", f"Dataflow job with id {JOB_ID} was stopped"),
+("cancelled", f"Dataflow job with id {JOB_ID} was cancelled"),
+]
Review Comment:
```suggestion
],
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Added JOB_STATE_CANCELLED and pool_sleep GCP Dataflow Operators [airflow]
eladkal commented on code in PR #37364: URL: https://github.com/apache/airflow/pull/37364#discussion_r1510206428 ## tests/providers/google/cloud/triggers/test_dataflow.py: ## @@ -125,6 +125,20 @@ async def test_run_loop_return_stopped_event(self, mock_job_status, trigger): actual_event = await trigger.run().asend(None) assert actual_event == expected_event + Review Comment: ```suggestion ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Added JOB_STATE_CANCELLED and pool_sleep GCP Dataflow Operators [airflow]
jcamatta commented on code in PR #37364: URL: https://github.com/apache/airflow/pull/37364#discussion_r1489371929 ## airflow/providers/google/cloud/operators/dataflow.py: ## @@ -835,6 +836,7 @@ def __init__( self.deferrable = deferrable self.expected_terminal_state = expected_terminal_state self.append_job_name = append_job_name +self.poll_sleep = poll_sleep Review Comment: Done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Added JOB_STATE_CANCELLED and pool_sleep GCP Dataflow Operators [airflow]
dirrao commented on code in PR #37364: URL: https://github.com/apache/airflow/pull/37364#discussion_r1489316339 ## airflow/providers/google/cloud/operators/dataflow.py: ## @@ -835,6 +836,7 @@ def __init__( self.deferrable = deferrable self.expected_terminal_state = expected_terminal_state self.append_job_name = append_job_name +self.poll_sleep = poll_sleep Review Comment: Can you update the param description? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Added JOB_STATE_CANCELLED and pool_sleep GCP Dataflow Operators [airflow]
jcamatta commented on PR #37364: URL: https://github.com/apache/airflow/pull/37364#issuecomment-1939869239 @hussein-awala done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Added JOB_STATE_CANCELLED and pool_sleep GCP Dataflow Operators [airflow]
boring-cyborg[bot] commented on PR #37364: URL: https://github.com/apache/airflow/pull/37364#issuecomment-1938861500 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst) Here are some useful points: - Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices). - Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits. Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://s.apache.org/airflow-slack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Added JOB_STATE_CANCELLED and pool_sleep GCP Dataflow Operators [airflow]
jcamatta opened a new pull request, #37364: URL: https://github.com/apache/airflow/pull/37364 Added an 'elif' block in the 'async def run(self)' method of 'TemplateJobStartTrigger', which captures 'JOB_STATE_CANCELLED'. This prevents the triggerer from looping indefinitely when a dataflow is cancelled. This also required updating the 'execute_complete' method of both 'DataflowStartFlexTemplateOperator' and 'DataflowTemplatedJobStartOperator', which now raise an 'AirflowException(event["message"])' when the job has the status "cancelled". So, if you cancel the job in the Google console, this will be reflected in the Airflow task execution. Added 'pool_sleep' as an optional argument to the constructor of 'DataflowStartFlexTemplateOperator', preserving its default value of 10 seconds. This argument is passed to the 'TemplateJobStartTrigger' constructor when the 'self.defer' method is called in 'execute'. Adding a test is very challenging because it requires a FlexTemplate and ClassicTemplate. Then, one must manually cancel it or use the REST API, and observe if this is reflected in the job. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
