[
https://issues.apache.org/jira/browse/HIVE-24177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xiaohang.li updated HIVE-24177:
-------------------------------
Attachment: MapredLocalTask.patch
> hive mapjoin throws udf class not found
> ---------------------------------------
>
> Key: HIVE-24177
> URL: https://issues.apache.org/jira/browse/HIVE-24177
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 2.2.0, 2.3.6
> Reporter: xiaohang.li
> Priority: Major
> Attachments: MapredLocalTask.patch
>
>
> when i use mapjoin,hive throws Execution failed with exit status.
> below is my test demo:
> CREATE TABLE `fact`(CREATE TABLE `fact`( `id` int, `name` string) ROW
> FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH
> SERDEPROPERTIES ( 'field.delim'='\t', 'serialization.format'='\t') ;
> vi fact.txt1 lee2 lee2
> load data local inpath 'fact.txt' into table fact;
>
> create table dim_table( id int, name string)ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES (
> 'field.delim'='\t', 'serialization.format'='\t');
> vi dim.txt1 lee2 test3 test34 test45 test56 test67 test8 test9 test10 test
> load data local inpath 'dim.txt' into table dim_table;
> then i create a lowwer udf to Make Upper letters lower
> add jar udfdemo-1.0.jar;
> create temporary function mylower as 'com.test.hive.Lower';
> select *
> from(
> select id,name,mylower(name) lower_name
> from fact
> )t1
> join dim_table t2
> on t2.name='test';
>
> the error is below:
> 2020-09-18T14:22:34,190 INFO [main] mr.MapredLocalTask: Executing: hadoop jar
> /opt/apps/apache-hive-2.2.0-bin/lib/hive-exec-2.2.1-SNAPSHOT.jar
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver -localtask -plan
> file:/tmp/hive/xiaohang.li/7c7f7a15-cac0-4d84-8dda-abc9a952eadb/hive_2020-09-18_14-22-30_557_1389022108873835641-1/-local-10006/plan.xml
> -jobconffile
> file:/tmp/hive/xiaohang.li/7c7f7a15-cac0-4d84-8dda-abc9a952eadb/hive_2020-09-18_14-22-30_557_1389022108873835641-1/-local-10007/jobconf.xml
>
> 2020-09-18 14:39:16 Starting to launch local task to process map join;
> maximum memory = 149133721602020-09-18 14:39:16 Starting to launch local task
> to process map join; maximum memory = 14913372160Execution failed with exit
> status: 2Obtaining error information
> Task failed!Task ID: Stage-4
> the log4j file is thows below exception:
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> com.test.hive.Lower.
> because fact table counts is 2,dim table counts is 10,when execute mapjoin
> fact table is in memory。hive sql is send task off to another
> jvm,ConfVars.HIVEADDEDJARS is not in ExecDriver's args(-libjars **.jar)。need
> in MapredLocalTask.executeInChildVM add
> if(StringUtils.isNotEmpty(conf.getVar(ConfVars.HIVEADDEDJARS)))
> { addlibJar = "-libjars " +conf.getVar(ConfVars.HIVEADDEDJARS); }
> String cmdLine = hadoopExec + " jar " + jarCmd + " -localtask -plan " +
> planPath.toString()
> + " " + isSilent + " " + hiveConfArgs +" "+addlibJar;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
