sdevani commented on a change in pull request #4792: [AIRFLOW-3659] Create Google Cloud Transfer Service Operators URL: https://github.com/apache/airflow/pull/4792#discussion_r261402956
########## File path: docs/howto/operator.rst ########## @@ -2327,3 +2327,403 @@ More information See `Google Cloud Vision Product delete documentation <https://googleapis.github.io/google-cloud-python/latest/vision/gapic/v1/api.html#google.cloud.vision_v1.ProductSearchClient.delete_product>`_. + +Google Cloud Transfer Service Operators +--------------------------------------- + +.. _howto/operator:GcpTransferServiceJobCreateOperator: + +GcpTransferServiceJobCreateOperator +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Create a transfer job. + +The function accepts dates in two formats: + +- consistent with `Google API <https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TimeOfDay>`_ :: + + { "year": 2019, "month": 2, "day": 11 } + +- as an :class:`~datetime.datetime` object + +The function accepts time in two formats: + +- consistent with `Google API <https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TimeOfDay>`_ :: + + { "hours": 12, "minutes": 30, "seconds": 0 } + +- as an :class:`~datetime.time` object + +If you want to create a job transfer that copies data from AWS S3 then you must have a connection configured. Information about configuration for AWS is available: :ref:`connection-type-AWS` +The selected connection for AWS can be indicated by the parameter ``aws_conn_id``. + +Arguments +""""""""" + +Some arguments in the example DAG are taken from the OS environment variables: + +.. literalinclude:: ../../airflow/contrib/example_dags/example_gcp_transfer.py + :language: python + :start-after: [START howto_operator_gct_common_variables] + :end-before: [END howto_operator_gct_common_variables] + +Using the operator +"""""""""""""""""" + +.. literalinclude:: ../../airflow/contrib/example_dags/example_gcp_transfer.py + :language: python + :start-after: [START howto_operator_gct_create_job_body_gcp] + :end-before: [END howto_operator_gct_create_job_body_gcp] + +.. literalinclude:: ../../airflow/contrib/example_dags/example_gcp_transfer.py + :language: python + :start-after: [START howto_operator_gct_create_job_body_aws] + :end-before: [END howto_operator_gct_create_job_body_aws] + +.. literalinclude:: ../../airflow/contrib/example_dags/example_gcp_transfer.py + :language: python + :dedent: 4 + :start-after: [START howto_operator_gct_create_job] + :end-before: [END howto_operator_gct_create_job] + +Templating +"""""""""" + +.. literalinclude:: ../../airflow/contrib/operators/gcp_transfer_operator.py + :language: python + :dedent: 4 + :start-after: [START gcp_transfer_job_create_template_fields] + :end-before: [END gcp_transfer_job_create_template_fields] + +More information +"""""""""""""""" + +See `Google Cloud Transfer Service - Method: transferJobs.create +<https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/create>`_. + +.. _howto/operator:GcpTransferServiceJobDeleteOperator: + +GcpTransferServiceJobDeleteOperator +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Deletes a transfer job. + +Arguments +""""""""" + +Some arguments in the example DAG are taken from the OS environment variables: + +.. literalinclude:: ../../airflow/contrib/example_dags/example_gcp_transfer.py + :language: python + :start-after: [START howto_operator_gct_common_variables] Review comment: is this supposed to be gcp? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
