[
https://issues.apache.org/jira/browse/AIRFLOW-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17104022#comment-17104022
]
ASF GitHub Bot commented on AIRFLOW-6294:
-----------------------------------------
mik-laj commented on a change in pull request #8809:
URL: https://github.com/apache/airflow/pull/8809#discussion_r422753200
##########
File path: docs/howto/operator/gcp/dataflow.rst
##########
@@ -0,0 +1,120 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Google Cloud Dataflow Operators
+===============================
+
+`Dataflow <https://cloud.google.com/dataflow/>`__ is a managed service for
+executing a wide variety of data processing patterns. These pipelines are
created
+using the Apache Beam programming model which allows for both batch and
streaming.
+
+Airflow provides operators to start/create Java and Python Dataflow jobs.
+
+.. contents::
+ :depth: 1
+ :local:
+
+Prerequisite Tasks
+^^^^^^^^^^^^^^^^^^
+
+.. include:: _partials/prerequisite_tasks.rst
+
+Create a Java Job
+^^^^^^^^^^^^^^^^^
+
+.. _howto/operator:DataflowCreateJavaJobOperator:
+
+Using a JAR on GCS
+""""""""""""""""""
+
+To start a Java Cloud Dataflow job with the JAR on GCS use
+:class:`~airflow.providers.google.cloud.operators.dataflow.DataflowCreateJavaJobOperator`.
+
+.. exampleinclude::
../../../../airflow/providers/google/cloud/example_dags/example_dataflow.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_operator_start_java_job]
+ :end-before: [END howto_operator_start_java_job]
+
+Using a JAR on local
+""""""""""""""""""""
+
+To start a Java Cloud Dataflow job with a JAR on local use
+:class:`~airflow.providers.google.cloud.operators.dataflow.DataflowCreateJavaJobOperator`.
+If the JAR is on GCS, you can download it to your local with
+:class:`~airflow.providers.google.cloud.operators.gcs.GCSToLocalOperator`.
+
+.. exampleinclude::
../../../../airflow/providers/google/cloud/example_dags/example_dataflow.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_operator_start_local_java_job]
+ :end-before: [END howto_operator_start_local_java_job]
+
+Create a Python Job
+^^^^^^^^^^^^^^^^^^^
+
+.. _howto/operator:DataflowCreatePythonJobOperator:
+
+Using a Python file on GCS
+""""""""""""""""""""""""""
+
+To start a Python Cloud Dataflow job with a Python file on GCS use
+:class:`~airflow.providers.google.cloud.operators.dataflow.DataflowCreatePythonJobOperator`.
+
+.. exampleinclude::
../../../../airflow/providers/google/cloud/example_dags/example_dataflow.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_operator_start_python_job]
+ :end-before: [END howto_operator_start_python_job]
+
+Using a Python file on Local
+""""""""""""""""""""""""""""
+
+To start a Python Cloud Dataflow job with a Python file on local use
+:class:`~airflow.providers.google.cloud.operators.dataflow.DataflowCreatePythonJobOperator`.
+If the Python file is on GCS, you can download it to your local with
Review comment:
You do not have to use this operator. If the file is on GCS then it can
be specified using the parameter directly. We have this task at DAG, but only
for testing purposes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Create guide for Dataflow service operator
> ------------------------------------------
>
> Key: AIRFLOW-6294
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6294
> Project: Apache Airflow
> Issue Type: New Feature
> Components: documentation, gcp
> Affects Versions: 1.10.6
> Reporter: Kamil Bregula
> Priority: Minor
> Labels: GoodForNewContributors
>
> Hello,
> A guide that describes how to use Dataflow service perators would be useful.
> Other guides are available:
> https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html
> If anyone is interested in this task, I am willing to provide all the
> necessary tips and information.
> Best regards,
> Kamil
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
