Hi folks,
I’m running into a strange ClassNotFound error while trying to use one of my
Hive UDFs in Impala.
I’ve defined the UDF:
> create function device_type(string, string, string) returns string location
> 'my/path/to/my-udf-SNAPSHOT.jar'
> symbol='com.criteo.hadoop.hive.udf.UDFUserAgentToDeviceType';
I try to use it in simple queries and it works fine:
> select device_type("a", "b", "c");
Query: select device_type("a", "b", "c")
…
+----------------------------------------+
| device_type('a', 'b', 'c') |
+----------------------------------------+
| Mobile - Other |
+----------------------------------------+
Fetched 1 row(s) in 0.01s
> select ua_device_family, ua_browser_family, ua_os_family,
> device_type(ua_device_family, ua_browser_family, ua_os_family) from
> bi_arbitrage_full limit 1;
Query: select ua_device_family, ua_browser_family, ua_os_family,
device_type(ua_device_family, ua_browser_family, ua_os_family) from
bi_arbitrage_full limit 1
…
+------------------+-------------------+--------------+----------------------------------------------------------------------------+
| ua_device_family | ua_browser_family | ua_os_family |
device_type(ua_device_family, ua_browser_family, ua_os_family) |
+------------------+-------------------+--------------+----------------------------------------------------------------------------+
| iPhone | mobile safari | iOS | iPhone
|
+------------------+-------------------+--------------+----------------------------------------------------------------------------+
Fetched 1 row(s) in 0.73s
Now when I try to run a more complex query (same database)
I get an error:
WARNINGS: ImpalaRuntimeException: Unable to find class.
CAUSED BY: ClassNotFoundException:
com.criteo.hadoop.hive.udf.UDFUserAgentToDeviceType
I turned up the log level on the coordinator to debug. I see the calls being
made to load the UDF and the use of the UDF in the query but no details on the
ClassNotFound.
Has anyone run into a similar issue? We’re running Impala 2.11.0-cdh5.14.0
RELEASE (build d68206561bce6b26762d62c01a78e6cd27aa7690) so I think we should
hopefully be clear of this really old bug -
https://issues.apache.org/jira/browse/IMPALA-695.
Thanks,
-- Piyush
