Re: Unit testing macros

[Per Ullberg]

We ran in to the same issue with oozie hive actions not sharing sessions.
We went for the concatenation approach. We have a java action before the
hive action that reads all hql files from a specification and concatenates
these into one long hql script. We also add some sugar at this point by
setting some custom hive variables (like NOMIAL_TIMESTAMP that allows the
users to not use current_time udf in their scripts etc).

best regards
/Pelle

On Fri, Sep 30, 2016 at 12:12 PM, Staņislavs Rogozins <
stanislavs.rogoz...@gmail.com> wrote:

> Have you taken a look at https://github.com/klarna/HiveRunner?
>
>
> I assume you're referring to this:
> https://github.com/klarna/HiveRunner/blob/ef14a1c181be863cec
> 2278aeb732d9e21c38a2b3/src/test/java/com/klarna/hiverunner/MacroTest.java
>
> It appears to execute the macro creation statement as part of the session,
> using hive library's CLIService class, so, doing the same during normal
> execution would require for the application itself to be implemented in
> Java.
>
> On Fri, Sep 30, 2016 at 12:45 PM, Elliot West  wrote:
>
>> Hi,
>>
>> You can achieve this by storing the macro definition in a separate HQL
>> file and 'import' this as needed. Unfortunately such imports are
>> interpreted by your Hive client and the relevant command varies between
>> client implementations: '!run' in Beeline and 'SOURCE' in Hive CLI. I
>> raised a proposal to create a unified command that is compatible across
>> clients but this has yet to gain any traction: https://issues.apach
>> e.org/jira/browse/HIVE-12703
>>
>> Elliot.
>>
>>
>>
> Actually, I'm developing an Oozie workflow that executes HQL scripts as
>  actions. Doesn't look like it provides support for such features.
> So, it would seem that my current options are either to use shell action
> instead or do some magic with concatenation of scripts before main action's
> execution.
> A first-class command like that sure would be nice.
>
>


-- 

*Per Ullberg*
Data Vault Tech Lead
Odin Uppsala
+46 701612693 <+46+701612693>

Klarna AB (publ)
Sveavägen 46, 111 34 Stockholm
Tel: +46 8 120 120 00 <+46812012000>
Reg no: 556737-0431
klarna.com

Re: Unit testing macros

[Staņislavs Rogozins]

>
> Have you taken a look at https://github.com/klarna/HiveRunner?


I assume you're referring to this:
https://github.com/klarna/HiveRunner/blob/ef14a1c181be863cec2278aeb732d9
e21c38a2b3/src/test/java/com/klarna/hiverunner/MacroTest.java

It appears to execute the macro creation statement as part of the session,
using hive library's CLIService class, so, doing the same during normal
execution would require for the application itself to be implemented in
Java.

On Fri, Sep 30, 2016 at 12:45 PM, Elliot West  wrote:

> Hi,
>
> You can achieve this by storing the macro definition in a separate HQL
> file and 'import' this as needed. Unfortunately such imports are
> interpreted by your Hive client and the relevant command varies between
> client implementations: '!run' in Beeline and 'SOURCE' in Hive CLI. I
> raised a proposal to create a unified command that is compatible across
> clients but this has yet to gain any traction: https://issues.
> apache.org/jira/browse/HIVE-12703
>
> Elliot.
>
>
>
Actually, I'm developing an Oozie workflow that executes HQL scripts as
 actions. Doesn't look like it provides support for such features.
So, it would seem that my current options are either to use shell action
instead or do some magic with concatenation of scripts before main action's
execution.
A first-class command like that sure would be nice.

Re: Unit testing macros

[Elliot West]

Hi,

You can achieve this by storing the macro definition in a separate HQL file
and 'import' this as needed. Unfortunately such imports are interpreted by
your Hive client and the relevant command varies between client
implementations: '!run' in Beeline and 'SOURCE' in Hive CLI. I raised a
proposal to create a unified command that is compatible across clients but
this has yet to gain any traction:
https://issues.apache.org/jira/browse/HIVE-12703

Elliot.


On 30 September 2016 at 10:23, Staņislavs Rogozins <
stanislavs.rogoz...@gmail.com> wrote:

> The Unit testing wiki page
>  
> suggests
> using macros to 'extract and resuse the expressions applied to columns' and
> that they can be 'readily isolated for testing'. However as far as I'm
> aware, right now only temporary macros can be created, that stop existing
> outside of session where they were defined(https://cwiki.apache.
> org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Create/
> DropMacro), thus making it necessary to include the macro in every HQL
> script that uses it. How is it possible to isolate them for testing?
>

Re: Unit testing macros

[Per Ullberg]

Have you taken a look at https://github.com/klarna/HiveRunner?

regards
/Pelle

On Fri, Sep 30, 2016 at 11:23 AM, Staņislavs Rogozins <
stanislavs.rogoz...@gmail.com> wrote:

> The Unit testing wiki page
>  
> suggests
> using macros to 'extract and resuse the expressions applied to columns' and
> that they can be 'readily isolated for testing'. However as far as I'm
> aware, right now only temporary macros can be created, that stop existing
> outside of session where they were defined(https://cwiki.apache.
> org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Create/
> DropMacro), thus making it necessary to include the macro in every HQL
> script that uses it. How is it possible to isolate them for testing?
>



-- 

*Per Ullberg*
Data Vault Tech Lead
Odin Uppsala
+46 701612693 <+46+701612693>

Klarna AB (publ)
Sveavägen 46, 111 34 Stockholm
Tel: +46 8 120 120 00 <+46812012000>
Reg no: 556737-0431
klarna.com