VladaZakharova commented on code in PR #70922:
URL: https://github.com/apache/airflow/pull/70922#discussion_r4081437512
##########
providers/google/src/airflow/providers/google/cloud/operators/dataproc.py:
##########
@@ -685,6 +685,7 @@ class
DataprocCreateClusterOperator(GoogleCloudBaseOperator):
"labels",
"gcp_conn_id",
"impersonation_chain",
+ "_legacy_cluster_kwargs",
Review Comment:
Looks like previously legacy kwargs built cluster_config during __init__.
Now cluster_config remains None until execute().
Normal task execution should remain compatible, but code that reads
operator.cluster_config before execution—cluster policies, subclasses, custom
validation, or tests—will observe different behavior. Invalid legacy
configuration also moves from Dag parsing time to task execution time.
##########
providers/google/src/airflow/providers/google/cloud/operators/dataproc.py:
##########
@@ -685,6 +685,7 @@ class
DataprocCreateClusterOperator(GoogleCloudBaseOperator):
"labels",
"gcp_conn_id",
"impersonation_chain",
+ "_legacy_cluster_kwargs",
Review Comment:
Also DataprocCreateClusterOperator remains in
validate_operators_init_exemptions.txt. but this PR does not finish this
operator’s part of issue #70296.
The new _legacy_cluster_kwargs field is synthesized in __init__, so it does
not follow the validator’s normal direct-assignment rule. The implementation
and exemption strategy need to be resolved together.
##########
providers/google/src/airflow/providers/google/cloud/operators/dataproc.py:
##########
@@ -722,23 +722,21 @@ def __init__(
AirflowProviderDeprecationWarning,
stacklevel=2,
)
- # Remove result of apply defaults
- if "params" in kwargs:
- del kwargs["params"]
-
- # Create cluster object from kwargs
if project_id is None:
raise AirflowException(
"project_id argument is required when building cluster
from keywords parameters"
)
- kwargs["project_id"] = project_id
- cluster_config = ClusterGenerator(**kwargs).make()
+
+ # Defer building cluster_config until execute(), after templated
fields render.
+ self._legacy_cluster_kwargs: dict | None = dict(kwargs)
Review Comment:
this captures more than Dataproc legacy parameters, including params, dag,
task_group, callbacks, executor configuration, and other BaseOperator values.
Only ClusterGenerator arguments should be captured, no?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
