Re: Hive LLAP Macro and Window Function

[Gopal Vijayaraghavan]

> When LLAP Execution Mode is set to 'only' you can't have a macro and window 
> function in the same select statement. 

The "only" part isn't enforced for the simple select query, but is enforced for 
the complex one (the PTF one).

> select col_1, col_2 from macro_bug where otrim(col_1) is not null;

That becomes a non-LLAP case, because macro_bug is so small - to trigger this 
issue for that query, force it through the Llap decider.

set hive.fetch.task.conversion=none;

I know that's not entirely helpful here, but the point is that the macros are 
not expanded in Hive at compile-time.

  Filter Operator
predicate: otrim(col_1) is not null (type: boolean)

is what I get in the plan, which LLAP recognizes as a "temporary function".

The actual expansion is actually evaluated at runtime by GenericUDFMacro, not 
expanded by the compiler.

If you're 100% sure the temporary function will work, you can take off the 
safety rails and try with 

set hive.llap.skip.compile.udf.check=true;

which will just let LLAP plan deserialization figure out if has the necessary 
classes/fail (& not care about the "name" of the UDF).

Cheers,
Gopal

Re: Hive LLAP Macro and Window Function

[Gopal Vijayaraghavan]

> When LLAP Execution Mode is set to 'only' you can't have a macro and 
window function in the same select statement. 

The "only" part isn't enforced for the simple select query, but is enforced 
for the complex one (the PTF one).

> select col_1, col_2 from macro_bug where otrim(col_1) is not null;

That becomes a non-LLAP case, because macro_bug is so small - to trigger 
this issue for that query, force it through the Llap decider.

set hive.fetch.task.conversion=none;

I know that's not entirely helpful here, but the point is that the macros 
are not expanded in Hive at compile-time.

  Filter Operator
predicate: otrim(col_1) is not null (type: boolean)

is what I get in the plan, which LLAP recognizes as a "temporary function".

The actual expansion is actually evaluated at runtime by GenericUDFMacro, 
not expanded by the compiler.

If you're 100% sure the temporary function will work, you can take off the 
safety rails and try with 

set hive.llap.skip.compile.udf.check=true;

which will just let LLAP plan deserialization figure out if has the 
necessary classes/fail (& not care about the "name" of the UDF).

Cheers,
Gopal

Hive LLAP Macro and Window Function

[Shawn Weeks]

Trying to figure out if the following is a bug or expected behavior. When LLAP 
Execution Mode is set to 'only' you can't have a macro and window function in 
the same select statement. However you can have a macro or a window function 
without an issue. Below is a test case.


use default;

create temporary macro otrim(val string) if(trim(val)='',null,trim(val));

create table macro_bug(col_1 string, col_2 string) stored as orc;

insert into macro_bug values ('ROW1','ROW1'), ('ROW2','ROW2'), ('ROW3','ROW3'), 
('ROW4','ROW4');

select * from macro_bug where otrim(col_1) is not null;

-- Works in Hive 1.x and Hive 2.x
select otrim(col_1), otrim(col_2) from macro_bug where otrim(col_1) is not null;

select col_1, col_2 from macro_bug where otrim(col_1) is not null;

select row_number() over(partition by col_1 order by col_2) from macro_bug 
where col_1 is not null;

-- Works in Hive 1.x and fails in Hive 2.x
select row_number() over(partition by col_1 order by col_2) from macro_bug 
where otrim(col_1) is not null;

drop table macro_bug;

Thanks
Shawn