jigneshjain25 commented on code in PR #74753:
URL: https://github.com/apache/airflow/pull/74753#discussion_r4327413054
##########
providers/google/src/airflow/providers/google/cloud/operators/dataproc.py:
##########
@@ -652,7 +851,24 @@ class
DataprocCreateClusterOperator(GoogleCloudBaseOperator):
values to parameter of DataprocCreateCluster operator.
:param cluster_config: Required. The cluster config to create.
If a dict is provided, it must be of the same form as the protobuf
message
- :class:`~google.cloud.dataproc_v1.types.ClusterConfig`
+ :class:`~google.cloud.dataproc_v1.types.ClusterConfig`.
Review Comment:
Removed the inline HO example block from `DataprocCreateClusterOperator`'s
`cluster_config` docstring. It is now kept clean and consistent with the rest
of the operators, referencing the documentation guide and example DAG.
##########
providers/google/src/airflow/providers/google/cloud/operators/dataproc.py:
##########
@@ -117,6 +118,204 @@ class InstanceFlexibilityPolicy:
instance_selection_list: list[InstanceSelection]
+class HardwareClass(Enum):
+ """
+ Hardware class of a High Obtainability (HO) mode instance group.
+
+ Representation for
+
google.cloud.dataproc.v1#google.cloud.dataproc.v1.FlexibleInstanceGroupConfig.HardwareClass.
+ """
+
+ STANDARD = "STANDARD"
+ PERFORMANCE = "PERFORMANCE"
+
+
+FLEXIBLE_CONFIG_ALLOWED_CORE_COUNTS = (2, 4, 8, 16, 32)
+
+
+@dataclass
+class FlexibleInstanceGroupConfig:
Review Comment:
Completely agree. We have removed the custom `FlexibleInstanceGroupConfig`,
`HardwareClass` enum, `flexible_scale_cluster_update` helper function, and all
client-side validation methods from `dataproc.py`.
Airflow operators now act as pass-through orchestrators passing
`cluster_config`, `cluster`, and `update_mask` directly to `DataprocHook`,
allowing users to provide plain dicts or native `google.cloud.dataproc_v1`
proto types (`FlexibleInstanceGroupConfig`), letting the Dataproc backend API
handle authoritative validation without client-side drift or maintenance
overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
