Hi, It clearly is a classpath issue! When you do a select * from tab , it works because HIve just fetches the data from HDFS using a FetchTask and doesn't start any MR job (It probably uses json jar in your local hive lib directory to deserialize and limit the rows to 5 and hence not raising any error).
However the error you are facing in the 2nd query involving an MR job is because mapper (on some remote machine which doesn't have json jar in its class path) is unable to locate your json jar ! Did u try doing what Dean suggested? Thanks On Sun, Mar 10, 2013 at 1:49 PM, Sai Sai <saigr...@yahoo.in> wrote: > Just wondering if anyone has any suggestions: > > This executes successfully: > > hive> select * from twitter limit 5; > > This does not work: > > hive> select tweet_id from twitter limit 5; // I have given the exception > info below: > > Here is the output of this: > > hive> select * from twitter limit 5; > OK > > tweet_id created_at text user_id user_screen_name user_lang > 122106088022745088 Fri Oct 07 00:28:54 +0000 2011 wkwkw -_- ayo saja > mba RT @yullyunet: Sepupuuu, kita lanjalan yok.. Kita karokoe-an.. Ajak mas > galih jg kalo dia mau.. "@Dindnf: doremifas 124735434 Dindnf en > 122106088018558976 Fri Oct 07 00:28:54 +0000 2011 @egg486 특별히 > 준비했습니다! 252828803 CocaCola_Korea ko > 122106088026939392 Fri Oct 07 00:28:54 +0000 2011 My offer of free > gobbies for all if @amityaffliction play Blair snitch project still > stands. 168590073 SarahYoungBlood en > 122106088035328001 Fri Oct 07 00:28:54 +0000 2011 the girl nxt to me > in the lib got her headphones in dancing and singing loud af like she the > only one here haha 267296295 MONEYyDREAMS_ en > 122106088005971968 Fri Oct 07 00:28:54 +0000 2011 @KUnYoong_B2UTY > Bị lsao đấy 269182160 b2st_b2utyhp en > Time taken: 0.154 seconds > > This does not work: > > hive> select tweet_id from twitter limit 5; > > > Total MapReduce jobs = 1 > Launching Job 1 out of 1 > Number of reduce tasks is set to 0 since there's no reduce operator > Starting Job = job_201303050432_0094, Tracking URL = > http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094 > Kill Command = /home/satish/work/hadoop-1.0.4/libexec/../bin/hadoop job > -kill job_201303050432_0094 > Hadoop job information for Stage-1: number of mappers: 1; number of > reducers: 0 > 2013-03-10 00:14:44,509 Stage-1 map = 0%, reduce = 0% > 2013-03-10 00:15:14,613 Stage-1 map = 100%, reduce = 100% > Ended Job = job_201303050432_0094 with errors > Error during job, obtaining debugging information... > Job Tracking URL: > http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094 > Examining task ID: task_201303050432_0094_m_000002 (and more) from job > job_201303050432_0094 > > Task with the most failures(4): > ----- > Task ID: > task_201303050432_0094_m_000000 > > URL: > > http://ubuntu:50030/taskdetails.jsp?jobid=job_201303050432_0094&tipid=task_201303050432_0094_m_000000 > ----- > Diagnostic Messages for this Task: > java.lang.RuntimeException: Error in configuring object > at > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: java.lang.reflect.InvocationTargetException > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) > ... 9 more > Caused by: java.lang.RuntimeException: Error in configuring object > at > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) > ... 14 more > Caused by: java.lang.reflect.InvocationTargetException > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) > ... 17 more > Caused by: java.lang.RuntimeException: Map operator initialization failed > at > org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) > ... 22 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.contrib.serde2.JsonSerde > at > org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:420) > at > org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:90) > ... 22 more > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hive.contrib.serde2.JsonSerde > > at java.net.URLClassLoader$1.run(URLClassLoader.java:217) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:205) > at java.lang.ClassLoader.loadClass(ClassLoader.java:321) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) > at java.lang.ClassLoader.loadClass(ClassLoader.java:266) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820) > at > org.apache.hadoop.hive.ql.exec.MapOperator.initObjectInspector(MapOperator.java:243) > at > org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:380) > ... 23 more > > > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.MapRedTask > MapReduce Jobs Launched: > Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL > Total MapReduce CPU Time Spent: 0 msec > > Thanks > Sai > ------------------------------ > *From:* Dean Wampler <dean.wamp...@thinkbiganalytics.com> > *To:* user@hive.apache.org; Sai Sai <saigr...@yahoo.in> > *Sent:* Friday, 8 March 2013 5:22 AM > *Subject:* Re: java.lang.NoClassDefFoundError: > com/jayway/jsonpath/PathUtil > > Unfortunately, you have to also add the json jars to Hive's class path > before it starts, e.g., > > env HADOOP_CLASSPATH=/path/to/lib/*.jar hive > > Use the appropriate path to your lib directory. > > On Fri, Mar 8, 2013 at 4:53 AM, Sai Sai <saigr...@yahoo.in> wrote: > > I have added the jar files successfully like this: > > > hive (testdb)> ADD JAR lib/hive-json-serde-0.3.jar; > Added lib/hive-json-serde-0.3.jar to class path > Added resource: lib/hive-json-serde-0.3.jar > > > hive (testdb)> ADD JAR lib/json-path-0.5.4.jar; > Added lib/json-path-0.5.4.jar to class path > Added resource: lib/json-path-0.5.4.jar > > > hive (testdb)> ADD JAR lib/json-smart-1.0.6.3.jar; > Added lib/json-smart-1.0.6.3.jar to class path > Added resource: lib/json-smart-1.0.6.3.jar > > > After this i am getting this error: > > > CREATE EXTERNAL TABLE IF NOT EXISTS twitter (tweet_id BIGINT,created_at > STRING,text STRING,user_id BIGINT, user_screen_name STRING,user_lang > STRING) ROW FORMAT SERDE "org.apache.hadoop.hive.contrib.serde2.JsonSerde" > WITH SERDEPROPERTIES ( > "tweet_id"="$.id","created_at"="$.created_at","text"="$.text","user_id"="$. > user.id","user_screen_name"="$.user.screen_name", > "user_lang"="$.user.lang") LOCATION '/home/satish/data/twitter/input'; > java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil > at org.apache.hadoop.hive.contrib.serde2.JsonSerde.initialize(Unknown > Source) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:207) > at > org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:266) > at > org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:259) > at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:585) > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:550) > at > org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3698) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:253) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > Caused by: java.lang.ClassNotFoundException: com.jayway.jsonpath.PathUtil > at java.net.URLClassLoader$1.run(URLClassLoader.java:217) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:205) > at java.lang.ClassLoader.loadClass(ClassLoader.java:321) > at java.lang.ClassLoader.loadClass(ClassLoader.java:266) > ... 23 more > FAILED: Execution Error, return code -101 from > org.apache.hadoop.hive.ql.exec.DDLTask > > > Any help would be really appreciated. > Thanks > Sai > > > > > -- > *Dean Wampler, Ph.D.* > thinkbiganalytics.com > +1-312-339-1330 > > > >
