josh-fell commented on code in PR #23428:
URL: https://github.com/apache/airflow/pull/23428#discussion_r864428831
##########
airflow/providers/amazon/aws/example_dags/example_athena.py:
##########
@@ -82,66 +65,60 @@ def read_results_from_s3(query_execution_id):
dag_id='example_athena',
schedule_interval=None,
start_date=datetime(2021, 1, 1),
- dagrun_timeout=timedelta(minutes=60),
tags=['example'],
catchup=False,
) as dag:
- # [START howto_athena_operator_and_sensor]
- # Using a task-decorated function to create a CSV file in S3
- add_sample_data_to_s3 = add_sample_data_to_s3()
+ upload_sample_data = S3CreateObjectOperator(
+ task_id='upload_sample_data',
+ s3_bucket=S3_BUCKET,
+ s3_key=f'{S3_KEY}/{ATHENA_TABLE}/{SAMPLE_FILENAME}',
+ data=SAMPLE_DATA,
+ replace=True,
+ )
create_table = AthenaOperator(
- task_id='setup__create_table',
+ task_id='create_table',
query=QUERY_CREATE_TABLE,
database=ATHENA_DATABASE,
output_location=f's3://{S3_BUCKET}/{S3_KEY}',
- sleep_time=30,
- max_tries=None,
)
+ # [START howto_athena_operator]
read_table = AthenaOperator(
- task_id='query__read_table',
+ task_id='query_table',
Review Comment:
Since it was brought up on the dev list and generally a good practice in
general with DAG authoring, WDYT about having the variable name and `task_id`
be the same value?
##########
docs/apache-airflow-providers-amazon/operators/athena.rst:
##########
@@ -16,42 +16,56 @@
under the License.
-.. _howto/operator:AthenaOperator:
-
-Amazon Athena Operator
-======================
+Amazon Athena Operators
+=======================
-.. contents::
- :depth: 1
- :local:
+`Amazon Athena <https://aws.amazon.com/athena/>`__ is an interactive query
service
+that makes it easy to analyze data in Amazon Simple Storage Service (S3) using
+standard SQL. Athena is serverless, so there is no infrastructure to setup or
+manage, and you pay only for the queries you run. To get started, simply point
+to your data in S3, define the schema, and start querying using standard SQL.
Prerequisite Tasks
-------------------
+^^^^^^^^^^^^^^^^^^
.. include:: _partials/prerequisite_tasks.rst
-Using Operator
---------------
-Use the
-:class:`~airflow.providers.amazon.aws.operators.athena.AthenaOperator`
-to run a query in Amazon Athena. To get started with Amazon Athena please
visit
-`aws.amazon.com/athena <https://aws.amazon.com/athena>`_
+.. _howto/operator:AthenaOperator:
+
+Athena Operator
+^^^^^^^^^^^^^^^
-In the following example, we create an Athena table and run a query based upon
a CSV file
-created in an S3 bucket and populated with SAMPLE_DATA. The example waits for
the query
-to complete and then drops the created table and deletes the sample CSV file
in the S3
-bucket.
+Use the :class:`~airflow.providers.amazon.aws.operators.athena.AthenaOperator`
+to run a query in Amazon Athena.
+
+
+In the following example, we query an existing Athena table and send the
results to
+an existing Amazon S3 bucket. For more examples of how to use this operator,
please
+see the `Sample DAG
<https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/example_dags/example_athena.py>`__.
.. exampleinclude::
/../../airflow/providers/amazon/aws/example_dags/example_athena.py
:language: python
- :start-after: [START howto_athena_operator_and_sensor]
- :end-before: [END howto_athena_operator_and_sensor]
+ :start-after: [START howto_athena_operator]
+ :end-before: [END howto_athena_operator]
+
+.. _howto/operator:AthenaSensor:
+
+Athena Sensor
+^^^^^^^^^^^^^
+
+Use the :class:`~airflow.providers.amazon.aws.sensors.athena.AthenaSensor`
+to wait for the results of a query in Amazon Athena.
+
+.. exampleinclude::
/../../airflow/providers/amazon/aws/example_dags/example_athena.py
+ :language: python
+ :start-after: [START howto_athena_sensor]
+ :end-before: [END howto_athena_sensor]
Review Comment:
```suggestion
:start-after: [START howto_athena_sensor]
:dedent: 4
:end-before: [END howto_athena_sensor]
```
##########
docs/apache-airflow-providers-amazon/operators/athena.rst:
##########
@@ -16,42 +16,56 @@
under the License.
-.. _howto/operator:AthenaOperator:
-
-Amazon Athena Operator
-======================
+Amazon Athena Operators
+=======================
-.. contents::
- :depth: 1
- :local:
+`Amazon Athena <https://aws.amazon.com/athena/>`__ is an interactive query
service
+that makes it easy to analyze data in Amazon Simple Storage Service (S3) using
+standard SQL. Athena is serverless, so there is no infrastructure to setup or
+manage, and you pay only for the queries you run. To get started, simply point
+to your data in S3, define the schema, and start querying using standard SQL.
Prerequisite Tasks
-------------------
+^^^^^^^^^^^^^^^^^^
.. include:: _partials/prerequisite_tasks.rst
-Using Operator
---------------
-Use the
-:class:`~airflow.providers.amazon.aws.operators.athena.AthenaOperator`
-to run a query in Amazon Athena. To get started with Amazon Athena please
visit
-`aws.amazon.com/athena <https://aws.amazon.com/athena>`_
+.. _howto/operator:AthenaOperator:
+
+Athena Operator
+^^^^^^^^^^^^^^^
-In the following example, we create an Athena table and run a query based upon
a CSV file
-created in an S3 bucket and populated with SAMPLE_DATA. The example waits for
the query
-to complete and then drops the created table and deletes the sample CSV file
in the S3
-bucket.
+Use the :class:`~airflow.providers.amazon.aws.operators.athena.AthenaOperator`
+to run a query in Amazon Athena.
+
+
+In the following example, we query an existing Athena table and send the
results to
+an existing Amazon S3 bucket. For more examples of how to use this operator,
please
+see the `Sample DAG
<https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/example_dags/example_athena.py>`__.
.. exampleinclude::
/../../airflow/providers/amazon/aws/example_dags/example_athena.py
:language: python
- :start-after: [START howto_athena_operator_and_sensor]
- :end-before: [END howto_athena_operator_and_sensor]
+ :start-after: [START howto_athena_operator]
+ :end-before: [END howto_athena_operator]
Review Comment:
```suggestion
:start-after: [START howto_athena_operator]
:dedent: 4
:end-before: [END howto_athena_operator]
```
So the code snippet renders correctly in the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
