sgringwe opened a new issue #8300: No way to create a Dataflow job with Private IP configuration URL: https://github.com/apache/airflow/issues/8300 <!-- Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions. Don't worry if they're not all applicable; just try to include what you can :-) If you need to include code snippets or logs, please put them in fenced code blocks. If they're super-long, please use the details tag like <details><summary>super-long log</summary> lots of stuff </details> Please delete these comment blocks before submitting the issue. --> **Description** It does not appear that Airflow supports https://cloud.google.com/dataflow/docs/guides/specifying-networks#public_ip_parameter in it's GCP Dataflow operators. Looking at the documentation and at https://github.com/apache/airflow/blob/16d93c9e45e14179c7822fed248743f0c3fd935c/airflow/contrib/hooks/gcp_dataflow_hook.py#L275 it seems that Airflow does not support passing in a private IP configuration for Dataflow jobs. In Terraform, they use IpConfiguration as seen at https://github.com/terraform-providers/terraform-provider-google/blob/master/google/resource_dataflow_job.go#L183. Ideally, Airflow could use the same parameter. Unfortunately, there does not seem to be any documentation for this parameter that is being used https://cloud.google.com/dataflow/docs/reference/rest/v1b3/RuntimeEnvironment. Potentially this is a bug only in their docs? **Use case / motivation** I want to use https://cloud.google.com/dataflow/docs/guides/specifying-networks#public_ip_parameter feature so my VMs do not have public IPs **Related Issues** I don't believe so
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
