suribabu-un opened a new issue #4151: URL: https://github.com/apache/hudi/issues/4151
**_Tips before filing an issue_** - Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)? - Join the mailing list to engage in conversations and get faster support at [email protected]. - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly. ** While running aggregate queries in hive (running in EMR cluster) I am getting `ClassNotFoundException: org.apache.hudi.hadoop.HoodieParquetInputFormat` ** I have created hudi tables using a spark application. While trying to run simple aggregation queries using using hive CLI, I am getting below exception (able to run simple non-aggregate queries). `Caused by: java.lang.ClassNotFoundException: org.apache.hudi.hadoop.HoodieParquetInputFormat at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154) ... 44 more` I am using hive CLI which is running as a EMR cluster (emr-6.3.0). Data is stored in S3. Hudi version that came with cluster is 0.7.0 but I tried adding 0.8.0 version which didn't help. Hive execution engine is Tez, **if I change execution engine to mr then even aggregation queries are running fine**. I am setting hive below hive variables and ensured that hive.aux.jars.path has hudi-hadoop-mr-bundle.jar path. set hive.input.format=org.apache.hudi.hadoop.HoodieParquetInputFormat; set hive.tez.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat; **Environment Description** * Hudi version : 0.8.0 * Spark version : 3.1.2 * Hive version : 3.1.2 * Hadoop version : * Storage (HDFS/S3/GCS..) : S3 * Running on Docker? (yes/no) : No **Additional context** **Stacktrace** ```Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Failed to load plan: hdfs://ip-10-0-15-119.ec2.internal:8020/tmp/hive/hadoop/ef4bb1c5-70bb-4367-a215-bf93d21d7798/hive_2021-11-29_09-42-57_913_2458909955818885946-1/hadoop/_tez_scratch_dir/54a0996e-0054-42b3-bfb6-1a259654906e/map.xml at org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:61) at org.apache.hadoop.hive.ql.exec.ObjectCacheWrapper.retrieve(ObjectCacheWrapper.java:45) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:132) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266) ... 15 more Caused by: java.lang.RuntimeException: Failed to load plan: hdfs://ip-10-0-15-119.ec2.internal:8020/tmp/hive/hadoop/ef4bb1c5-70bb-4367-a215-bf93d21d7798/hive_2021-11-29_09-42-57_913_2458909955818885946-1/hadoop/_tez_scratch_dir/54a0996e-0054-42b3-bfb6-1a259654906e/map.xml at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:503) at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:336) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor$1.call(MapRecordProcessor.java:135) at org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:59) ... 18 more Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: org.apache.hudi.hadoop.HoodieParquetInputFormat Serialization trace: inputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc) aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork) at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156) at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:185) at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:326) at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:314) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759) at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObjectOrNull(SerializationUtilities.java:203) at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132) at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790) at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:180) at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161) at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708) at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:218) at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:686) at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:210) at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeObjectByKryo(SerializationUtilities.java:729) at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:613) at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:590) at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:464) ... 21 more Caused by: java.lang.ClassNotFoundException: org.apache.hudi.hadoop.HoodieParquetInputFormat at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154) ... 44 more``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
