amirmor1 opened a new pull request, #69855:
URL: https://github.com/apache/airflow/pull/69855
Add an opt-in `batch_id_prefix` to `DataprocCreateBatchOperator` so task
retries can submit a new Dataproc batch with a unique ID instead of reusing a
static `batch_id`.
### Why
When a task retries with the same `batch_id`, Dataproc can return
`ALREADY_EXISTS` and attach to the old batch. This change provides a built-in
retry-safe path without changing existing explicit `batch_id` behavior.
### What changes
- add optional `batch_id_prefix` parameter
- generate `<batch_id_prefix>-<unique_suffix>` per task attempt
- keep `batch_id` and `batch_id_prefix` mutually exclusive
- keep explicit `batch_id` behavior unchanged
- update Dataproc operator docs
- add unit tests for:
- mutual exclusivity
- prefix-based ID generation
- execute flow with prefix
- backward compatibility for explicit `batch_id`
### Test plan
- [x] `uv run ruff format
providers/google/src/airflow/providers/google/cloud/operators/dataproc.py
providers/google/tests/unit/google/cloud/operators/test_dataproc.py`
- [x] `uv run ruff check --fix
providers/google/src/airflow/providers/google/cloud/operators/dataproc.py
providers/google/tests/unit/google/cloud/operators/test_dataproc.py`
- [x] `uv run --project providers/google pytest
providers/google/tests/unit/google/cloud/operators/test_dataproc.py::TestDataprocCreateBatchOperator::test_execute
providers/google/tests/unit/google/cloud/operators/test_dataproc.py::TestDataprocCreateBatchOperator::test_execute_with_batch_id_prefix
providers/google/tests/unit/google/cloud/operators/test_dataproc.py::TestDataprocCreateBatchOperator::test_build_unique_batch_id_from_prefix_preserves_prefix_verbatim
providers/google/tests/unit/google/cloud/operators/test_dataproc.py::TestDataprocCreateBatchOperator::test_batch_id_and_batch_id_prefix_are_mutually_exclusive
-xvs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
