VictorPlusC commented on a change in pull request #16741:
URL: https://github.com/apache/beam/pull/16741#discussion_r807414381
##########
File path:
sdks/python/apache_beam/runners/interactive/dataproc/dataproc_cluster_manager.py
##########
@@ -91,9 +97,10 @@ def create_cluster(self, cluster: dict) -> None:
_LOGGER.info('Cluster created successfully: %s', self._cluster_name)
except Exception as e:
if e.code == 409:
- if self._cluster_name == self.DEFAULT_NAME:
+ if self._cluster_name ==
ie.current_env().clusters.default_cluster_name:
Review comment:
I've made a change to the 409 exception handling such that we now do the
following:
- check if cluster_metadata.cluster_name is default
- if so, attempt to get the master_url
- if master_url exists, log a warning that we're getting it from
storage, but do nothing else (this is because during the instantiation of the
DataprocClusterManager, we assign self.master_url =
self.get_master_url(cluster_metadata), where we first try to match it with the
corresponding master_url. If there is no match, this is defaulted to None)
- else, raise a ValueError
Let me know what you think about this approach!
##########
File path:
sdks/python/apache_beam/runners/interactive/dataproc/dataproc_cluster_manager.py
##########
@@ -91,9 +97,10 @@ def create_cluster(self, cluster: dict) -> None:
_LOGGER.info('Cluster created successfully: %s', self._cluster_name)
except Exception as e:
if e.code == 409:
- if self._cluster_name == self.DEFAULT_NAME:
+ if self._cluster_name ==
ie.current_env().clusters.default_cluster_name:
_LOGGER.info(
- 'Cluster %s already exists. Continuing...', self.DEFAULT_NAME)
+ 'Cluster %s already exists. Continuing...',
+ ie.current_env().clusters.default_cluster_name)
Review comment:
Replaced, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
