FWIW, the problem in our case was that hive-site.xml enabled map
compression and specified the LZO codec, but we didn't have LZO jar
installed on the new machines. I'm still somewhat puzzled by how the
absence of the LZO library resulted in show tables failing... I guess
the input formats are coupled to the compression codecs?..
The moral here is that if you run into a similar error, (1) look at
what's in your classpath and (2) check if the default hive
configuration works before enabling compression, etc
On Wed, Sep 15, 2010 at 10:27 AM, Edward Capriolo <edlinuxg...@gmail.com> wrote:
> On Wed, Sep 15, 2010 at 1:14 PM, Leo Alekseyev <dnqu...@gmail.com> wrote:
>> This is a "me too" post: we just ran into an identical problem setting
>> up a new cluster using CDH3b2. This is all rather mystifying, because
>> all the correct libraries are there; in fact the hive command line
>> looks something like
>>
>> /usr/java/jdk1.6.0_12/bin/java -Xmx256m -server
>> -Dhadoop.log.dir=/usr/lib/hadoop-0.20/logs
>> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop-0.20
>> -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
>> -Dhadoop.policy.file=hadoop-policy.xml -classpath
>> /usr/lib/hadoop-0.20/conf:/usr/java/jdk1.6.0_12/lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/hadoop-core-0.20.2+320.jar..................
>> org.apache.hadoop.util.RunJar /usr/lib/hive/lib/hive-cli-0.5.0.jar
>> org.apache.hadoop.hive.cli.CliDriver
>>
>> (note the presence of the hadoop-core.jar which contains the class in
>> question)
>>
>> If anyone has pointers on troubleshooting this, I'd appreciate it!
>>
>> --Leo
>>
>> On Tue, Sep 14, 2010 at 1:56 PM, Tianqiang Li <peter...@gmail.com> wrote:
>>> Some more context: I run Hive on a client machine which is NOT one of the
>>> hadoop cluster nodes, I suppose Hive can run well and submit job from a
>>> client machine, so I didn't change hadoop-env.sh on cluster nodes. On this
>>> client machine, hadoop java jobs and pig have been successfully submitted to
>>> cluster and processed.
>>>
>>> Regards,
>>> Peter Li
>>>
>>> On Tue, Sep 14, 2010 at 1:42 PM, Tianqiang Li <peter...@gmail.com> wrote:
>>>>
>>>> Hi, hive-users,
>>>> I am a new Hive users, install hive recently, when I type any
>>>> query-related command in Hive cli, it throws the exception, but create
>>>> table
>>>> are ok:
>>>>
>>>> $ hive
>>>> Hive history file=/tmp/pli/hive_job_log_pli_201009141519_1503313446.txt
>>>> hive> create table test5(a int);
>>>> OK
>>>> Time taken: 2.551 seconds
>>>> hive> show tables;
>>>> OK
>>>> Failed with exception java.io.IOException:java.io.IOException: Cannot
>>>> create an instance of InputFormat class
>>>> org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork!
>>>> Time taken: 0.09 seconds
>>>> hive>
>>>>
>>>> I searched on internet, and there are some info about hive's
>>>> HADOOP_CLASSPATH getting overwritten by some other tools in hadoop-env.sh,
>>>> I
>>>> tried to use appending( export
>>>> HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:other_paths ), didn't fix the problem,
>>>> then I tried to copy the jar files of hadoop libraries (
>>>> hadoop-0.20.1+169.68-core.jar ) which include
>>>> org.apache.hadoop.mapred.TextInputFormat, to $HIVE_HOME/lib/, still the
>>>> problem remain the same, It looks like Hive doesn't know where hadoop's
>>>> libraries are, and failed to new a TextInputFormat object, are there
>>>> someone
>>>> met this before? Any hints on how to work this around are welcome. Thanks.
>>>>
>>>> Here are some more context, I use hadoop 0.20.1+169.68 from cloudera CDH2,
>>>> and Hive 0.4/0.5 from CDH2/3, (both version for hive have the same issues),
>>>> here is the trace stack in log file.
>>>> ------
>>>> 2010-09-14 15:19:58,826 ERROR DataNucleus.Plugin
>>>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
>>>> "org.eclipse.core.resources" but it cannot be resolved.
>>>> 2010-09-14 15:19:58,826 ERROR DataNucleus.Plugin
>>>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
>>>> "org.eclipse.core.resources" but it cannot be resolved.
>>>> 2010-09-14 15:19:58,828 ERROR DataNucleus.Plugin
>>>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
>>>> "org.eclipse.core.runtime" but it cannot be resolved.
>>>> 2010-09-14 15:19:58,828 ERROR DataNucleus.Plugin
>>>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
>>>> "org.eclipse.core.runtime" but it cannot be resolved.
>>>> 2010-09-14 15:19:58,829 ERROR DataNucleus.Plugin
>>>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
>>>> "org.eclipse.text" but it cannot be resolved.
>>>> 2010-09-14 15:19:58,829 ERROR DataNucleus.Plugin
>>>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
>>>> "org.eclipse.text" but it cannot be resolved.
>>>> 2010-09-14 15:20:04,444 ERROR CliDriver
>>>> (SessionState.java:printError(279)) - Failed with exception
>>>> java.io.IOException:java.io.IOException: Cannot create an instance of
>>>> InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in
>>>> mapredWork!
>>>> java.io.IOException: java.io.IOException: Cannot create an instance of
>>>> InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in
>>>> mapredWork!
>>>> at
>>>> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:273)
>>>> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:133)
>>>> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:440)
>>>> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:131)
>>>> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
>>>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:287)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>> Caused by: java.io.IOException: Cannot create an instance of InputFormat
>>>> class org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork!
>>>> at
>>>> org.apache.hadoop.hive.ql.exec.FetchOperator.getInputFormatFromCache(FetchOperator.java:113)
>>>> at
>>>> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:214)
>>>> at
>>>> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:249)
>>>>
>>>> ----
>>>> My hadoop-env.sh :
>>>> export
>>>> HADOOP_CLASSPATH_OTHER=/usr/lib/hadoop-0.20/hadoop-0.20.1+169.68-core.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/*:/usr/lib/hadoop-0.20/*
>>>> export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}$:{HADOOP_CLASSPATH_OTHER}
>>>>
>>>>
>>>>
>>>>
>>>> Regards,
>>>> Peter Li
>>>
>>>
>>
>
> This is likely a cloudera packaging issue, they have support forums
> you might want to contact them. This place is best for help if you can
> reproduce the problem with a hive release or hive trunk.
>
