[
https://issues.apache.org/jira/browse/AIRFLOW-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393196#comment-16393196
]
Michael Mole commented on AIRFLOW-1153:
---------------------------------------
It looks like you can add it to params if you want to access it in your HQL via
the \{{ param.var }} syntax. However, I try to keep my HQL as executable by
either direct Hive CLI invocation or via Airflow without having to modify the
HQL. For example, let's say I have bar.hql:
bar.hql:
set hivevar:foo=FOO;
select('${hivevar:foo}');
select('${ds}');
select('${params.table}')
I want to be able to run as either:
# beeline/hive --hivevar params.table=foo_table --hivevar ds='2018-03-09' -f
bar.hql
# airflow test my_dag hive_operator_task '2018-03-09'
## the task was created with the HiveOperator with
### hiveconf_jinja_translate=True
### params=\{"table": "foo_table"}
The ${params.table} select will not work because the HiveOperator jinja
templating regex doesn't translate anything with a period (i.e. ${params.table}
does not get translated to \{{ params.table }}). A simple fix would be to
allow the period in the translation.
I think there is a larger issue about cleaning and clarifying the usage around
macros, hiveconf, and hivevars supplied by either macros, user defined macros,
or params. For example, I'd like to be able to supply the needed values to the
following HQL. I can do it via the Hive CLI as such.
hive --hivevar c='C' --hiveconf d='D' --hivevar e="E from hivevar" --hiveconf
e="E from hiveconf" --hivevar ds='2018-03-09' --hivevar
params.table='foo_table' -f test.hql
-- Check local hivevar variable
SET hivevar:a=A;
-- Output: A
select('${a}');
-- Output: A
select('${hivevar:a}');
-- Check local hiveconf variable
SET hiveconf:b=B;
-- Output: B
select('${hiveconf:b}');
-- Check arbitrary hivevar arg
-- Output: C
select('${c}');
-- Output: C
select('${hivevar:c}');
-- Check arbitrary hiveconf arg
-- Output: D
select('${hiveconf:d}');
-- Check arbitrary hivevar and hiveconf with the same name
-- Output: E from hivevar
select('${hivevar:e}');
-- Output: E from hivevar
select('${e}');
-- Output: E from hiveconf
select('${hiveconf:e}');
-- Check airflow macro arg
-- Output: 2018-03-09
select('${ds}');
-- Output: 2018-03-09
select('${hivevar:ds}');
-- Check task parameter arg
-- Output: foo_table
select('${params.table}');
-- Output: foo_table
select('${hivevar:params.table}');
> params in HiveOperator constructor can't be passed into Hive execution context
> ------------------------------------------------------------------------------
>
> Key: AIRFLOW-1153
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1153
> Project: Apache Airflow
> Issue Type: Bug
> Components: hive_hooks, operators
> Affects Versions: Airflow 2.0, Airflow 1.8
> Reporter: Xianping lin
> Priority: Critical
> Labels: easyfix, newbie
>
> params parameter in HiveOperator can't be imported into Hive executation
> context.
> so the following centence won't work, because 'mynumber' doesn't work for
> sql sentence.
> test_hiveoperator = HiveOperator(
> task_id='hive_test',
> hiveconf_jinja_translate=True,
> hql = ''' use myDB;
> INSERT OVERWRITE TABLE t2
> select * from t1 where t1.x > ' ${hiveconf:mynumber}'
> ''',
> params={'mynumber': 2},
> dag=dag
> )
> this modification pass the 'params' in HiveOperator construction to Hive
> sql execution context.
> The the variable definition can pass to hive sql
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
