[ https://issues.apache.org/jira/browse/AIRFLOW-5118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999925#comment-16999925 ]
ASF subversion and git services commented on AIRFLOW-5118: ---------------------------------------------------------- Commit 182556b89e446fa4ef7879030c2bf8371c59c444 in airflow's branch refs/heads/v1-10-stable from Igor Dralyuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=182556b ] [AIRFLOW-5118] Add ability to specify optional components in DataprocClusterCreateOperator (#5821) (cherry picked from commit fd544eb0cbffff77a833ab185049db6c115e822d) > Airflow DataprocClusterCreateOperator does not currently support setting > optional components > -------------------------------------------------------------------------------------------- > > Key: AIRFLOW-5118 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5118 > Project: Apache Airflow > Issue Type: New Feature > Components: operators > Affects Versions: 1.10.3 > Reporter: Omid Vahdaty > Assignee: Igor > Priority: Minor > Fix For: 1.10.7 > > > there need to be an option to install optional components via > DataprocClusterCreateOperator . components such as zeppelin. > From the source code of the DataprocClusterCreateOperator[1], the only > software configs that can be set are the imageVersion and the properties. As > the Zeppelin component needs to be set through softwareConfig > optionalComponents[2], the DataprocClusterCreateOperator does not currently > support setting optional components. > > As a workaround for the time being, you could create your clusters by > directly using the gcloud command rather than the > DataprocClusterCreateOperator . Using the Airflow BashOperator[4], you can > execute gcloud commands that create your Dataproc cluster with the required > optional components. > [1] > [https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py] > > [2] > [https://cloud.google.com/dataproc/docs/reference/rest/v1/ClusterConfig#softwareconfig] > > [3] [https://airflow.apache.org/howto/operator/bash.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)