italovinicius18 commented on issue #28023:
URL: https://github.com/apache/airflow/issues/28023#issuecomment-1335508862
Thank you! But I had the same problem.
I managed to solve the problem using the dataproc api and added a function
to this cluster creation, below is the function created.
```py
def create_cluster_custom(project_id, region, cluster_name):
# Create a client
client = dataproc_v1.ClusterControllerClient(
client_options={"api_endpoint":
"{}-dataproc.googleapis.com:443".format(region)}
)
# Initialize request argument(s)
cluster = dataproc_v1.Cluster()
cluster.project_id = project_id
cluster.cluster_name = cluster_name
cluster_config = dataproc_v1.ClusterConfig()
cluster_config.master_config.num_instances = 1
cluster_config.master_config.machine_type_uri = "n1-standard-4"
cluster_config.worker_config.num_instances = 2
cluster_config.worker_config.machine_type_uri = "n1-standard-4"
# Subnetwork URI
cluster_config.gce_cluster_config.subnetwork_uri = "datalab-network"
# Specify the zone
cluster_config.gce_cluster_config.zone_uri = "us-central1-f"
# Specify the image version
cluster_config.software_config.image_version = "2.0-debian10"
# Set shieled instance
cluster_config.gce_cluster_config.shielded_instance_config.enable_secure_boot =
True
cluster_config.gce_cluster_config.shielded_instance_config.enable_vtpm =
True
cluster_config.gce_cluster_config.shielded_instance_config.enable_integrity_monitoring
= True
# Set internal IP only
cluster_config.gce_cluster_config.internal_ip_only = True
cluster.config = cluster_config
request = dataproc_v1.CreateClusterRequest(
project_id=project_id,
region=region,
cluster=cluster,
)
# Make the request
operation = client.create_cluster(request=request)
print("Waiting for operation to complete...")
response = operation.result()
# Handle the response
print(response)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
