Colin-Cheng commented on PR #74910:
URL: https://github.com/apache/airflow/pull/74910#issuecomment-6325152859
Thanks for the review @olegkachur-e! I have updated the PR to address your
comments:
1. **Removed `MultiZoneConfig` dataclass & operator changes:** Reverted the
custom `MultiZoneConfig` class and `DataprocCreateClusterOperator.__init__`
changes in
`providers/google/src/airflow/providers/google/cloud/operators/dataproc.py` (as
well as the unit test in `test_dataproc.py`), relying directly on
`google-cloud-dataproc>=5.32.0` passing `cluster_config` through to the API.
2. **Documentation:** Added a **Multi-zone clusters** section with a
`CLUSTER_CONFIG` (`multi_zone_config`) example and links to the official
Dataproc `MultiZoneConfig` and `GceClusterConfig` documentation in
`providers/google/docs/operators/cloud/dataproc.rst`.
3. **System Test (`example_dataproc_cluster_deferrable.py` with
`multi_zone_config`):**
Ran
`providers/google/tests/system/google/cloud/dataproc/example_dataproc_cluster_deferrable.py`
with `CLUSTER_CONFIG` updated to use `multi_zone_config`:
```python
CLUSTER_CONFIG = {
"gce_cluster_config": {
"multi_zone_config": {
"target_shape": "ANY",
},
},
"master_config": {
"num_instances": 1,
"machine_type_uri": "n1-standard-4",
"disk_config": {"boot_disk_type": "pd-standard",
"boot_disk_size_gb": 32},
},
"worker_config": {
"num_instances": 2,
"machine_type_uri": "n1-standard-4",
"disk_config": {"boot_disk_type": "pd-standard",
"boot_disk_size_gb": 32},
},
"secondary_worker_config": {
"num_instances": 1,
"machine_type_uri": "n1-standard-4",
"disk_config": {
"boot_disk_type": "pd-standard",
"boot_disk_size_gb": 32,
},
"is_preemptible": True,
"preemptibility": "PREEMPTIBLE",
},
}
```
**Test execution result (`1 passed`):**
```text
providers/google/tests/system/google/cloud/dataproc/example_dataproc_cluster_deferrable.py
[2026-10-21T19:40:37.211610Z] {dagrun.py:1402} INFO - Marking run <DagRun
dataproc_cluster_def @ 2026-01-01 00:00:00+00:00:
manual__2026-10-21T19:36:15.093321+00:00, state:running, queued_at: None.
run_type: manual> successful
[2026-10-21T19:40:37.212101Z] {dagrun.py:1453} INFO - DagRun Finished:
dag_id=dataproc_cluster_def, logical_date=2026-01-01 00:00:00+00:00,
run_id=manual__2026-10-21T19:36:15.093321+00:00, run_start_date=2026-10-21
19:36:15.206363+00:00, run_end_date=2026-10-21 19:40:37.211635+00:00,
run_duration=262.005272, state=success, external_trigger=True, run_type=manual,
data_interval_start=2026-01-01 00:00:00+00:00, data_interval_end=2026-01-01
00:00:00+00:00,
...
================== 1 passed, 2 warnings in 677.89s (0:11:17)
===================
```
All tasks in `dataproc_cluster_def` (`create_cluster` -> `update_cluster`
-> `delete_cluster`) succeeded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
