[ 
https://issues.apache.org/jira/browse/AIRFLOW-5118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907528#comment-16907528
 ] 

ASF GitHub Bot commented on AIRFLOW-5118:
-----------------------------------------

idralyuk commented on pull request #5820: [AIRFLOW-5118] Add ability to specify 
optional components in Dataproc…
URL: https://github.com/apache/airflow/pull/5820
 
 
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Airflow DataprocClusterCreateOperator does not currently support setting 
> optional components
> --------------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-5118
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5118
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: operators
>    Affects Versions: 1.10.3
>            Reporter: Omid Vahdaty
>            Assignee: Igor
>            Priority: Minor
>
> there need to be an option to install optional components via 
> DataprocClusterCreateOperator . components such as zeppelin.
> From the source code of the DataprocClusterCreateOperator[1], the only 
> software configs that can be set are the imageVersion and the properties. As 
> the Zeppelin component needs to be set through softwareConfig 
> optionalComponents[2], the DataprocClusterCreateOperator does not currently 
> support setting optional components. 
>  
> As a workaround for the time being, you could create your clusters by 
> directly using the gcloud command rather than the 
> DataprocClusterCreateOperator . Using the Airflow BashOperator[4], you can 
> execute gcloud commands that create your Dataproc cluster with the required 
> optional components. 
> [1] 
> [https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py]
>  
>  [2] 
> [https://cloud.google.com/dataproc/docs/reference/rest/v1/ClusterConfig#softwareconfig]
>  
> [3] [https://airflow.apache.org/howto/operator/bash.html] 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Reply via email to