[
https://issues.apache.org/jira/browse/BEAM-13973?focusedWorklogId=734279&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-734279
]
ASF GitHub Bot logged work on BEAM-13973:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Mar/22 01:23
Start Date: 01/Mar/22 01:23
Worklog Time Spent: 10m
Work Description: VictorPlusC commented on a change in pull request
#16904:
URL: https://github.com/apache/beam/pull/16904#discussion_r816378042
##########
File path:
sdks/python/apache_beam/runners/interactive/dataproc/dataproc_cluster_manager.py
##########
@@ -186,15 +210,108 @@ def describe(self) -> None:
"""Returns a dictionary describing the cluster."""
return {
'cluster_metadata': self.cluster_metadata,
- 'master_url': self.master_url
+ 'master_url': self.master_url,
+ 'dashboard': self.dashboard
}
- def get_master_url(self, identifier) -> None:
+ def get_cluster_details(self, cluster_metadata: MasterURLIdentifier):
+ """Gets the Dataproc_v1 Cluster object for the current cluster manager."""
+ try:
+ return self._cluster_client.get_cluster(
+ request={
+ 'project_id': cluster_metadata.project_id,
+ 'region': cluster_metadata.region,
+ 'cluster_name': cluster_metadata.cluster_name
+ })
+ except Exception as e:
+ if e.code == 403:
+ _LOGGER.error(
+ 'Due to insufficient project permissions, '
+ 'unable to retrieve information for cluster: %s',
+ cluster_metadata.cluster_name)
+ raise ValueError(
+ 'You cannot view clusters in project: {}'.format(
+ cluster_metadata.project_id))
+ elif e.code == 404:
+ _LOGGER.error(
+ 'Cluster does not exist: %s', cluster_metadata.cluster_name)
+ raise ValueError(
+ 'Cluster was not found: {}'.format(cluster_metadata.cluster_name))
+ else:
+ _LOGGER.error(
+ 'Failed to get information for cluster: %s',
+ cluster_metadata.cluster_name)
+ raise e
+
+ def wait_for_cluster_to_provision(
+ self, cluster_metadata: MasterURLIdentifier) -> None:
+ while self.get_cluster_details(
+ cluster_metadata).status.state.name == 'CREATING':
+ time.sleep(15)
+
+ def get_staging_location(self, cluster_metadata: MasterURLIdentifier) -> str:
+ """Gets the staging bucket of an existing Dataproc cluster."""
+ try:
+ daemon_thread = Thread(
Review comment:
I see, and that sounds good. I've removed it and now I just call the
method directly. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 734279)
Time Spent: 8.5h (was: 8h 20m)
> Link Dataproc Flink master URLs to the InteractiveRunner when FlinkRunner is
> used
> ---------------------------------------------------------------------------------
>
> Key: BEAM-13973
> URL: https://issues.apache.org/jira/browse/BEAM-13973
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
> Reporter: Victor Chen
> Assignee: Victor Chen
> Priority: P2
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
> - Implements logic to get the Flink master from the staging bucket of an
> existing Dataproc cluster by parsing the cluster instantiation output
> - Adds functionality to connect the FlinkRunner to a Dataproc Flink master
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
