jigneshjain25 commented on code in PR #74753:
URL: https://github.com/apache/airflow/pull/74753#discussion_r4289091372
##########
providers/google/docs/operators/cloud/dataproc.rst:
##########
@@ -186,6 +186,43 @@ You can use deferrable mode for this action in order to
run the operator asynchr
:start-after: [START how_to_cloud_dataproc_create_cluster_operator_async]
:end-before: [END how_to_cloud_dataproc_create_cluster_operator_async]
+High Obtainability (HO) mode clusters
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+High Obtainability (HO) mode enables creating Dataproc clusters using abstract
hardware requirements
+(:class:`~airflow.providers.google.cloud.operators.dataproc.FlexibleInstanceGroupConfig`:
number of
+instances, core count, and hardware class) instead of concrete machine types.
The Dataproc backend resolves
+these requirements into a ranked list of widely available machine families to
minimize stockouts and improve
+cluster creation reliability.
+
+High Obtainability clusters have the following characteristics:
+
+* **Mandatory AutoZone**: HO clusters require AutoZone to dynamically select
an optimal zone; do not specify a ``zone``.
+* **Cluster-wide flexible config**: Master and primary workers must both use
flexible configurations (or
+ ``num_workers=0`` for a single-node cluster).
Review Comment:
Applied, thank you! Updated the documentation to accurately reflect
configuring a single-node HO cluster via dataproc:dataproc.allow.zero.workers:
'true' in software_config.properties and omitting worker_flexible_config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
