[
https://issues.apache.org/jira/browse/BEAM-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victor Chen updated BEAM-13812:
-------------------------------
Description:
* Integrates the DataprocClusterManager previously introduced in:
** [[BEAM-13799] Create an Interactive Beam Dataproc package for users to
manage clusters with]([https://github.com/apache/beam/pull/16691)]
* Adds code to automatically create a Dataproc cluster configured with Docker
and Flink if a user instantiates an InteractiveRunner using
underlying_runner='FlinkRunner'
* By default, the created Dataproc cluster will be called
'interactive-beam-cluster' and will be cleaned up when the Interactive
Environment is interrupted
* Introduces the Clusters class in interactive_beam.py, which acts as a user
interface that provides the user with the following capabilities:
** Changing the default name of the clusters created by Interactive Beam
** Cleaning up a cluster by providing a corresponding pipeline, an option to
force the deletion is provided, in the case where more than one pipeline use
the same cluster
** Getting all pipelines that use the provided master_url
** Receiving a description back for either the cluster corresponding to a
specific pipeline, or the descriptions of all pipelines
*
was:
* Adds code to automatically create a Dataproc cluster configured with Docker
and Flink if a user instantiates an InteractiveRunner using
underlying_runner='FlinkRunner'
* By default, the created Dataproc cluster will be called
'interactive-beam-cluster' and will be cleaned up when the Interactive
Environment is interrupted
* Users will be able to toggle an automatic deletion of the cluster with the
default name, 'interactive-beam-cluster' when the kernel is interrupted
* Introduces the Clusters class in interactive_beam.py, which acts as a user
interface that provides the user with the following capabilities:
** Changing the default name of the clusters created by Interactive Beam
** Cleaning up a cluster by providing a corresponding pipeline, an option to
force the deletion is provided, in the case where more than one pipeline use
the same cluster
** Getting all pipelines that use the provided master_url
** Receiving a description back for either the cluster corresponding to a
specific pipeline, or the descriptions of all pipelines
> Integrate DataprocClusterManager into Interactive Environment
> -------------------------------------------------------------
>
> Key: BEAM-13812
> URL: https://issues.apache.org/jira/browse/BEAM-13812
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
> Reporter: Victor Chen
> Assignee: Victor Chen
> Priority: P2
> Time Spent: 28h 50m
> Remaining Estimate: 0h
>
> * Integrates the DataprocClusterManager previously introduced in:
> ** [[BEAM-13799] Create an Interactive Beam Dataproc package for users to
> manage clusters with]([https://github.com/apache/beam/pull/16691)]
> * Adds code to automatically create a Dataproc cluster configured with
> Docker and Flink if a user instantiates an InteractiveRunner using
> underlying_runner='FlinkRunner'
> * By default, the created Dataproc cluster will be called
> 'interactive-beam-cluster' and will be cleaned up when the Interactive
> Environment is interrupted
> * Introduces the Clusters class in interactive_beam.py, which acts as a user
> interface that provides the user with the following capabilities:
> ** Changing the default name of the clusters created by Interactive Beam
> ** Cleaning up a cluster by providing a corresponding pipeline, an option to
> force the deletion is provided, in the case where more than one pipeline use
> the same cluster
> ** Getting all pipelines that use the provided master_url
> ** Receiving a description back for either the cluster corresponding to a
> specific pipeline, or the descriptions of all pipelines
> *
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
