[GitHub] [airflow] vchiapaikeo opened a new pull request, #28959: Add support for write_on_empty in BaseSQLToGCSOperator

[GitBox]

vchiapaikeo opened a new pull request, #28959:
URL: https://github.com/apache/airflow/pull/28959

   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   closes: https://github.com/apache/airflow/issues/28806
   
   Adds support for a write_on_empty param in BaseSQLToGCSOperator which will 
allow the operator to write an empty file if no rows from the query are 
returned.
   
   Test DAG:
   
   ```py
   from airflow import DAG
   
   from airflow.providers.google.cloud.transfers.postgres_to_gcs import 
PostgresToGCSOperator
   
   DEFAULT_TASK_ARGS = {
       "owner": "gcp-data-platform",
       "retries": 1,
       "retry_delay": 10,
       "start_date": "2022-08-01",
   }
   
   with DAG(
       max_active_runs=1,
       concurrency=2,
       catchup=False,
       schedule_interval="@daily",
       dag_id="test_sql_to_gcs_on_empty",
       default_args=DEFAULT_TASK_ARGS,
   ) as dag:
   
       test_basic = PostgresToGCSOperator(
           task_id="test_basic",
           postgres_conn_id="postgres_default",
           sql="""
           SELECT
               task_id,
               dag_id
           FROM sla_miss
           LIMIT 10
           """,
           export_format="csv",
           gcp_conn_id="google_cloud_default",
           bucket="my-bucket",
           filename="vchiapaikeo/sql-to-gcs/empty/csv/file_{}.csv",
           write_on_empty=False,
       )
   ```
   
   Ensured no rows in sla_miss table:
   
   <img width="913" alt="image" 
src="https://user-images.githubusercontent.com/9200263/212562540-d53843f9-5dd3-48e2-a3cb-10d2e945e084.png";>
   
   When param does not exist, ensure no writes to GCS:
   
   <img width="979" alt="image" 
src="https://user-images.githubusercontent.com/9200263/212562568-201dd479-0c6e-4286-a9d9-eb9969dd794f.png";>
   
   When param does exist and is set to False, ensure no writes to GCS:
   
   <img width="892" alt="image" 
src="https://user-images.githubusercontent.com/9200263/212562590-3017a744-a112-4903-b779-99fa69b26906.png";>
   
   
   When param does exist and is set to True, ensure write to GCS occurs:
   
   <img width="1060" alt="image" 
src="https://user-images.githubusercontent.com/9200263/212562610-efb41bca-c173-418a-aa68-a69a60f5a7f6.png";>
   
   Verify data on GCS:
   <img width="818" alt="image" 
src="https://user-images.githubusercontent.com/9200263/212562706-e54e2226-e28b-4693-8586-b0d6bee90d1b.png";>
   
   cc @eladkal 
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org