Re: Hive permanent functions are not available in Spark SQL

[Pala M Muthaia]

+user list

On Tue, Sep 29, 2015 at 3:43 PM, Pala M Muthaia  wrote:

> Hi,
>
> I am trying to use internal UDFs that we have added as permanent functions
> to Hive, from within Spark SQL query (using HiveContext), but i encounter
> NoSuchObjectException, i.e. the function could not be found.
>
> However, if i execute 'show functions' command in spark SQL, the permanent
> functions appear in the list.
>
> I am using Spark 1.4.1 with Hive 0.13.1. I tried to debug this by looking
> at the log and code, but it seems both the show functions command as well
> as udf query both go through essentially the same code path, but the former
> can see the UDF but the latter can't.
>
> Any ideas on how to debug/fix this?
>
>
> Thanks,
> pala
>

Hive permanent functions are not available in Spark SQL

[Pala M Muthaia]

Hi,

I am trying to use internal UDFs that we have added as permanent functions
to Hive, from within Spark SQL query (using HiveContext), but i encounter
NoSuchObjectException, i.e. the function could not be found.

However, if i execute 'show functions' command in spark SQL, the permanent
functions appear in the list.

I am using Spark 1.4.1 with Hive 0.13.1. I tried to debug this by looking
at the log and code, but it seems both the show functions command as well
as udf query both go through essentially the same code path, but the former
can see the UDF but the latter can't.

Any ideas on how to debug/fix this?


Thanks,
pala