VictorPlusC commented on a change in pull request #17127:
URL: https://github.com/apache/beam/pull/17127#discussion_r836852439
##########
File path: sdks/python/apache_beam/runners/interactive/interactive_runner.py
##########
@@ -267,9 +267,12 @@ def _get_dataproc_cluster_master_url_if_applicable(
category=DeprecationWarning)
project_id =
(user_pipeline.options.view_as(GoogleCloudOptions).project)
region = (user_pipeline.options.view_as(GoogleCloudOptions).region)
- cluster_name = ie.current_env().clusters.default_cluster_name
- cluster_metadata = MasterURLIdentifier(
- project_id=project_id, region=region, cluster_name=cluster_name)
+ if not project_id:
+ cluster_metadata = ie.current_env().clusters.default_cluster_metadata
+ else:
+ cluster_name = ie.current_env().clusters.default_cluster_name
+ cluster_metadata = MasterURLIdentifier(
+ project_id=project_id, region=region, cluster_name=cluster_name)
Review comment:
This case was added in the scenario the user first uses the JupyterLab
extension to select a default cluster, at this point, we update
ib.clusters.default_cluster_metadata to be the MasterURLIdentifier that is used
by the selected cluster. Afterwards, if they create a new
InteractiveRunner(underlying_runner=FlinkRunner()) pipeline that does not have
a Flink master_url and also does not have a Google Cloud project specified, we
just assign the default Dataproc cluster to that pipeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
