pateash commented on issue #16911:
URL: https://github.com/apache/airflow/issues/16911#issuecomment-973827278
> @pateash Sorry, just seen your reply, yes i was trying to update Airflow
from 1.10x to 2.X.
>
> so what is the fix, i saw a merge request
@rajaprakash91 You can generate ClusterConfig with the same arguments from
ClusterGenerator.make() you have been using and then pass that to the operator.
```python
path =
f"gs://goog-dataproc-initialization-actions-{self.cfg.get('region')}/python/pip-install.sh"
cluster_config = ClusterGenerator(
project_id=self.cfg.get('project_id'),
...........
init_actions_uris=[
path
],
metadata={'PIP_PACKAGES': 'pyyaml requests pandas openpyxl'},
properties=properties,
dag=dag
).make()
return DataprocClusterCreateOperator(
task_id='create_dataproc_cluster',
cluster_name='my-cluster',
project_id=self.cfg.get('project_id'),
region=self.cfg.get('region'),
cluster_config=cluster_config,
dag=dag
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
