italovinicius18 commented on issue #28023:
URL: https://github.com/apache/airflow/issues/28023#issuecomment-1333966358
@mdering And how I put this using CLUSTER_CONFIG using ClusterGenerator?
Like this:
```py
CLUSTER_CONFIG = ClusterGenerator(
project_id=PROJECT_ID,
region="us-central1-f",
cluster_name=CLUSTER_NAME,
tags=["dataproc"],
network_uri="",
internal_ip_only = True,
# Add shielded VM config
shielded_instance_config={
"enable_secure_boot": False,
"enable_vtpm": True,
"enable_integrity_monitoring": True,
},
subnetwork_uri= "datalab-network",
config_bucket= "sandbox-processing",
num_workers=2,
storage_bucket="sandbox-processing",
num_masters=1,
master_machine_type="n1-standard-4",
master_disk_type="pd-standard",
master_disk_size=1024,
worker_machine_type="n1-standard-4",
worker_disk_type="pd-standard",
worker_disk_size=1024,
image_version="2.0-debian10",
autoscaling_policy=None,
idle_delete_ttl=7200
#optional_components=['JUPYTER', 'ANACONDA']
#metadata={"gcs-connector-version" : '2.1.1' ,
# "bigquery-connector-version": '1.1.1',
# "spark-bigquery-connector-version": '0.17.2',
# "PIP_PACKAGES" : 'datalab shap oyaml click apache-airflow
apache-airflow-providers-google'
# },
#init_actions_uris
=['gs://goog-dataproc-initialization-actions-us-central1/connectors/connectors.sh','gs://goog-dataproc-initialization-actions-us-central1/python/pip-install.sh']
).make()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
