[
https://issues.apache.org/jira/browse/AIRFLOW-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aizhamal Nurmamat kyzy updated AIRFLOW-1644:
--------------------------------------------
Labels: easyfix hive hive-hooks (was: easyfix)
Component/s: hooks
Moving hive_hooks to hooks, adding hooks label for component refactor.
> HiveOperator dry_run can execute arbitrary HQL
> ----------------------------------------------
>
> Key: AIRFLOW-1644
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1644
> Project: Apache Airflow
> Issue Type: Bug
> Components: hive_hooks, hooks, tests
> Affects Versions: 1.8.0
> Reporter: Cristian Figueroa Rodriguez
> Assignee: Cristian Figueroa Rodriguez
> Priority: Minor
> Labels: easyfix, hive, hive-hooks
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> When running:
> {code:java}
> airflow test <dag_id> <task_id> <ds> --dry_run
> {code}
> If it's a HiveOperator it runs explain plans and executes some set up
> overhead if necessary. However if there are comments before a statement
> airflow recognizes these as set up, independent of the command after the
> comment.
> Example:
> {code:java}
> test_setup = HiveOperator(
> task_id='test_setup',
> hql="""
> CREATE TABLE tmp.test_dry_run_airflow (
> dummy STRING,
> state STRING,
> gender STRING
> );""",
> dag=dag,
> )
> test_comments = HiveOperator(
> task_id='dry_run_hql_with_comments',
> hql="""
> -- A line commenting before the drop.
> DROP TABLE tmp.test_dry_run_airflow;
> CREATE TABLE tmp.test_dry_run_airflow_comments (
> state STRING
> );
> INSERT OVERWRITE TABLE tmp.test_dry_run_airflow_comments
> SELECT state FROM tmp.test_dry_run_airflow;
> """,
> dag=dag,
> )
> cristian_figueroa@ airflow : ~ $ airflow test dag_start_dates_testing
> dry_run_hql_with_comments 2017-07-03 --dry_run
> [2017-09-25 23:59:29,061] {hive_hooks.py:254} INFO - Testing HQL [CREATE
> TABLE tmp.test_dry_run_airflow_comments ( s (...)]
> [2017-09-25 23:59:36,235] {hive_hooks.py:272} INFO - SUCCESS
> [2017-09-25 23:59:36,236] {hive_hooks.py:254} INFO - Testing HQL [INSERT
> OVERWRITE TABLE tmp.test_dry_run_airflow_co (...)]
> [2017-09-25 23:59:43,179] {hive_hooks.py:263} INFO - FAILED:
> SemanticException [Error 10001]: Line 3:27 Table not found
> 'test_dry_run_airflow_comments'
> [2017-09-25 23:59:43,180] {hive_hooks.py:270} INFO - Context :
> INSERT OVERWRITE TABLE tmp.test_dry_run_airflow_comments
> SELECT state FROM tmp.test_dry_run_airflow
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
