Just for curiosity , as you are using Cloudera-Manager hadoop and spark.. How you build shark .....for it??
are you able to read any file from hdfs .......did you tried that out..??? Regards, Arpit Tak On Thu, Apr 17, 2014 at 7:07 PM, ge ko <koenig....@gmail.com> wrote: > Hi, > > the error java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat has been > resolved by adding > parquet-hive-bundle-1.4.1.jar to shark's lib folder. > Now the Hive metastore can be read successfully (also the parquet based > table). > > But if I want to select from that table I receive: > > org.apache.spark.SparkException: Job aborted: Task 0.0:0 failed 4 times > (most recent failure: Exception failure: java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018) > > This is really strange, since the class > org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe is included in > the parquet-hive-bundle-1.4.1.jar ?!?! > ...getting more and more confused ;) > > any help ? > > regards, Gerd > > > On 17 April 2014 11:55, ge ko <koenig....@gmail.com> wrote: > >> Hi, >> >> I want to select from a parquet based table in shark, but receive the >> error: >> >> shark> select * from wl_parquet; >> 14/04/17 11:33:49 INFO shark.SharkCliDriver: Execution Mode: shark >> 14/04/17 11:33:49 INFO ql.Driver: <PERFLOG method=Driver.run> >> 14/04/17 11:33:49 INFO ql.Driver: <PERFLOG method=TimeToSubmit> >> 14/04/17 11:33:49 INFO ql.Driver: <PERFLOG method=compile> >> 14/04/17 11:33:49 INFO parse.ParseDriver: Parsing command: select * from >> wl_parquet >> 14/04/17 11:33:49 INFO parse.ParseDriver: Parse Completed >> 14/04/17 11:33:49 INFO parse.SharkSemanticAnalyzer: Get metadata for >> source tables >> FAILED: Hive Internal Error: >> java.lang.RuntimeException(java.lang.ClassNotFoundException: >> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat) >> 14/04/17 11:33:50 ERROR shark.SharkDriver: FAILED: Hive Internal Error: >> java.lang.RuntimeException(java.lang.ClassNotFoundException: >> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat) >> java.lang.RuntimeException: java.lang.ClassNotFoundException: >> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat >> at >> org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:306) >> at org.apache.hadoop.hive.ql.metadata.Table.<init>(Table.java:99) >> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:988) >> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:891) >> at >> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1083) >> at >> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1059) >> at >> shark.parse.SharkSemanticAnalyzer.analyzeInternal(SharkSemanticAnalyzer.scala:137) >> at >> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:279) >> at shark.SharkDriver.compile(SharkDriver.scala:215) >> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337) >> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) >> at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:338) >> at >> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) >> at shark.SharkCliDriver$.main(SharkCliDriver.scala:235) >> at shark.SharkCliDriver.main(SharkCliDriver.scala) >> Caused by: java.lang.ClassNotFoundException: >> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:270) >> at >> org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:302) >> ... 14 more >> >> I can successfully select from that table with Hive and Impala, but shark >> doesn't work. I am using CDH5 incl. Spark parcel and Shark 0.9.1. >> >> In what jar is this class "hidden", how can I get rid of this exception >> ?!?! >> >> The lib folder of shark contains: >> [root@hadoop-pg-9 shark-0.9.1]# ll lib >> total 180 >> lrwxrwxrwx 1 root root 67 16. Apr 14:17 hive-serdes-1.0-SNAPSHOT.jar >> -> /opt/cloudera/parcels/CDH/lib/hive/lib/hive-serdes-1.0-SNAPSHOT.jar >> -rwxrwxr-x 1 root root 23086 9. Apr 10:57 JavaEWAH-0.4.2.jar >> lrwxrwxrwx 1 root root 53 14. Apr 21:46 parquet-avro.jar -> >> /opt/cloudera/parcels/CDH/lib/hadoop/parquet-avro.jar >> lrwxrwxrwx 1 root root 58 14. Apr 21:46 parquet-cascading.jar -> >> /opt/cloudera/parcels/CDH/lib/hadoop/parquet-cascading.jar >> lrwxrwxrwx 1 root root 55 14. Apr 21:46 parquet-column.jar -> >> /opt/cloudera/parcels/CDH/lib/hadoop/parquet-column.jar >> lrwxrwxrwx 1 root root 55 14. Apr 21:46 parquet-common.jar -> >> /opt/cloudera/parcels/CDH/lib/hadoop/parquet-common.jar >> lrwxrwxrwx 1 root root 57 14. Apr 21:46 parquet-encoding.jar -> >> /opt/cloudera/parcels/CDH/lib/hadoop/parquet-encoding.jar >> lrwxrwxrwx 1 root root 55 14. Apr 21:46 parquet-format.jar -> >> /opt/cloudera/parcels/CDH/lib/hadoop/parquet-format.jar >> lrwxrwxrwx 1 root root 58 14. Apr 21:46 parquet-generator.jar -> >> /opt/cloudera/parcels/CDH/lib/hadoop/parquet-generator.jar >> lrwxrwxrwx 1 root root 62 14. Apr 21:46 parquet-hadoop-bundle.jar -> >> /opt/cloudera/parcels/CDH/lib/hadoop/parquet-hadoop-bundle.jar >> lrwxrwxrwx 1 root root 55 14. Apr 21:46 parquet-hadoop.jar -> >> /opt/cloudera/parcels/CDH/lib/hadoop/parquet-hadoop.jar >> -rw-r--r-- 1 root root 70103 27. Nov 21:24 parquet-hive-1.2.8.jar >> lrwxrwxrwx 1 root root 56 14. Apr 21:46 parquet-scrooge.jar -> >> /opt/cloudera/parcels/CDH/lib/hadoop/parquet-scrooge.jar >> lrwxrwxrwx 1 root root 55 14. Apr 21:46 parquet-thrift.jar -> >> /opt/cloudera/parcels/CDH/lib/hadoop/parquet-thrift.jar >> -rw-rw-r-- 1 root root 76220 9. Apr 10:57 pyrolite.jar >> >> thanks in advance, Gerd >> > >
