phani8996 commented on issue #4111: [AIRFLOW-3266] Add AWS Athena Operator and hook URL: https://github.com/apache/incubator-airflow/pull/4111#issuecomment-438898893 > @phani8996 One (last, I promise!) question: have you run this code yet in a DAG or via `airflow test` etc to check that everything works as expected? Unit tests are great but they don't catch everything. @ashb I moved above code to our production airflow cluster and its working as expected in cases of success queries, failed queries, invalid queries. Here is log snippet of success query. > Log file does not exist: /var/log/airflow/trains.test_dag/repair_task/2018-11-15T02:49:08.016673+00:00/1.log > Fetching from: http://<airflow_ip>:<airflow_port>/log/trains.test_dag/repair_task/2018-11-15T02:49:08.016673+00:00/1.log?num_lines=300 > > Showing only last 300 lines from trains.test_dag/repair_task/2018-11-15T02:49:08.016673+00:00/1.log > > > -------------------------------------------------------------------------------- > Starting attempt 1 of 2 > -------------------------------------------------------------------------------- > > [2018-11-15 02:49:11,350] INFO - Executing <Task(AWSAthenaOperator): repair_task> on 2018-11-15T02:49:08.016673+00:00 > [2018-11-15 02:49:11,350] INFO - Running: ['bash', '-c', 'airflow run trains.test_dag repair_task 2018-11-15T02:49:08.016673+00:00 --job_id 29420 --pool default --raw -sd /mnt/src/airflow-ds/dags/trains/test_dag.py --cfg_path /tmp/tmpeb_ttzdb'] > [2018-11-15 02:49:11,798] INFO - Job 29420: Subtask repair_task [2018-11-15 02:49:11,798] INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800 > [2018-11-15 02:49:12,257] INFO - Job 29420: Subtask repair_task [2018-11-15 02:49:12,256] INFO - Using executor CeleryExecutor > [2018-11-15 02:49:12,384] INFO - Job 29420: Subtask repair_task [2018-11-15 02:49:12,384] INFO - Filling up the DagBag from /mnt/src/airflow-ds/dags/trains/test_dag.py > [2018-11-15 02:49:12,668] INFO - Job 29420: Subtask repair_task [2018-11-15 02:49:12,668] INFO - Running <TaskInstance: trains.test_dag.repair_task 2018-11-15T02:49:08.016673+00:00 [running]> on host <airflow_worker_ip> > [2018-11-15 02:49:12,936] INFO - [2018-11-15 02:49:12,936] INFO - Trial 1: Query is still in an intermediate state - RUNNING > > [2018-11-15 02:49:42,988] INFO - [2018-11-15 02:49:42,987] INFO - Trial 2: Query execution completed. Final state is SUCCEEDED > > [2018-11-15 02:50:11,456] INFO - [2018-11-15 02:50:11,455] INFO - Task exited with return code 0
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
