hi, imcaptor
I try to cp lucene-core-2.3.1.jar to lib, but error also at map task.
-------------------lib-----------------
[EMAIL PROTECTED] hadoop]$ pwd
/home/chenlb/hadoop
[EMAIL PROTECTED] hadoop]$ ls lib
commons-cli-2.0-SNAPSHOT.jar commons-logging-1.0.4.jar jetty-5.1.4.jar
junit-3.8.1.jar kfs-0.1.LICENSE.txt native
commons-codec-1.3.jar commons-logging-api-1.0.4.jar
jetty-5.1.4.LICENSE.txt junit-3.8.1.LICENSE.txt log4j-1.2.13.jar
servlet-api.jar
commons-httpclient-3.0.1.jar jets3t-0.5.0.jar jetty-ext
kfs-0.1.jar lucene-core-2.3.1.jar xmlenc-0.52.jar
-------------------hadoop-site.xml---------------------------------
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000/</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://master:9001/</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/chenlb/hadoop/tmp/</value>
</property>
</configuration>
----------------------------
Thinks!
=====================error=======================
[EMAIL PROTECTED] hadoop]$ bin/hadoop jar hadoop-0.17.1-index.jar -inputPaths
data-in -outputPath index-msg-out -indexPath index -conf index-config.xml
08/10/09 22:33:36 INFO main.UpdateIndex: inputPaths = data-in
08/10/09 22:33:36 INFO main.UpdateIndex: outputPath = index-msg-out
08/10/09 22:33:36 INFO main.UpdateIndex: shards = null
08/10/09 22:33:36 INFO main.UpdateIndex: indexPath = index
08/10/09 22:33:36 INFO main.UpdateIndex: numShards = -1
08/10/09 22:33:36 INFO main.UpdateIndex: numMapTasks= -1
08/10/09 22:33:36 INFO main.UpdateIndex: confPath = index-config.xml
08/10/09 22:33:37 INFO main.UpdateIndex: sea.index.updater =
org.apache.hadoop.contrib.index.mapred.IndexUpdater
08/10/09 22:33:37 INFO mapred.IndexUpdater: mapred.input.dir =
hdfs://master:9000/user/chenlb/data-in
08/10/09 22:33:37 INFO mapred.IndexUpdater: mapred.output.dir =
hdfs://master:9000/user/chenlb/index-msg-out
08/10/09 22:33:37 INFO mapred.IndexUpdater: mapred.map.tasks = 2
08/10/09 22:33:37 INFO mapred.IndexUpdater: mapred.reduce.tasks = 1
08/10/09 22:33:37 INFO mapred.IndexUpdater: 1 shards = [EMAIL PROTECTED]/[EMAIL
PROTECTED]
08/10/09 22:33:37 INFO mapred.IndexUpdater: mapred.input.format.class =
org.apache.hadoop.contrib.index.example.LineDocInputFormat
08/10/09 22:33:38 INFO mapred.FileInputFormat: Total input paths to process : 2
08/10/09 22:33:39 INFO mapred.JobClient: Running job: job_200810092201_0010
08/10/09 22:33:40 INFO mapred.JobClient: map 0% reduce 0%
08/10/09 22:33:48 INFO mapred.JobClient: Task Id :
task_200810092201_0010_m_000000_0, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at
org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:369)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:33:50 INFO mapred.JobClient: Task Id :
task_200810092201_0010_m_000001_0, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at
org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:369)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:33:55 INFO mapred.JobClient: Task Id :
task_200810092201_0010_r_000000_0, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at
org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier.<init>(ReduceTask.java:964)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:327)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:33:56 INFO mapred.JobClient: Task Id :
task_200810092201_0010_m_000000_1, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at
org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:369)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:33:59 INFO mapred.JobClient: Task Id :
task_200810092201_0010_m_000001_1, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at
org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:369)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:34:03 INFO mapred.JobClient: Task Id :
task_200810092201_0010_r_000000_1, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at
org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier.<init>(ReduceTask.java:964)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:327)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:34:05 INFO mapred.JobClient: Task Id :
task_200810092201_0010_m_000000_2, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at
org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:369)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:34:08 INFO mapred.JobClient: Task Id :
task_200810092201_0010_m_000001_2, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at
org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:369)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:34:10 INFO mapred.JobClient: Task Id :
task_200810092201_0010_r_000000_2, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at
org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier.<init>(ReduceTask.java:964)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:327)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:34:13 INFO mapred.JobClient: map 100% reduce 100%
08/10/09 22:34:14 INFO main.UpdateIndex: Elapsed time is 37s
Elapsed time is 37s
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1062)
at
org.apache.hadoop.contrib.index.mapred.IndexUpdater.run(IndexUpdater.java:53)
at
org.apache.hadoop.contrib.index.main.UpdateIndex.main(UpdateIndex.java:263)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)
2008-10-10
chenlbspace
发件人： imcaptor
发送时间： 2008-10-10 17:07:39
收件人： core-user@hadoop.apache.org
抄送：
主题： Re: hadoop index org/apache/lucene/store/Directory NoClassDefFoundError
Try do this.
In the hadoop path,
-bash-3.00$ pwd
/data/hadoop/hadoop-0.18.1
cp src/contrib/index/lib/lucene-core-2.3.1.jar lib
Then you can run the task.
chenlbspace 写道:
> hi,
>
> I try to use contrib/index/hadoop-0.17.1-index.jar build lucene index, but
> Directory NoClassDefFoundError.
>
> How solve? Thinks.
>
> Additional, hadoop run Pseudo-Distributed Mode. and can run "bin/hadoop jar
> hadoop-*-examples.jar grep input output 'dfs[a-z.]+'"
>
> ---------------------------error----------------------------------------------------------------------------------
> [EMAIL PROTECTED] hadoop]$ bin/hadoop jar
> contrib/index/hadoop-0.17.1-index.jar -inputPaths src/contrib/index/sample
> -outputPath index-msg-out -indexPath index -conf
> src/contrib/index/conf/index-config.xml
> 08/10/09 21:41:28 INFO main.UpdateIndex: inputPaths = src/contrib/index/sample
> 08/10/09 21:41:28 INFO main.UpdateIndex: outputPath = index-msg-out
> 08/10/09 21:41:28 INFO main.UpdateIndex: shards = null
> 08/10/09 21:41:28 INFO main.UpdateIndex: indexPath = index
> 08/10/09 21:41:28 INFO main.UpdateIndex: numShards = -1
> 08/10/09 21:41:28 INFO main.UpdateIndex: numMapTasks= -1
> 08/10/09 21:41:28 INFO main.UpdateIndex: confPath =
> src/contrib/index/conf/index-config.xml
> 08/10/09 21:41:28 INFO main.UpdateIndex: Elapsed time is 0s
> Elapsed time is 0s
> java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
> at
> org.apache.hadoop.contrib.index.mapred.IndexUpdateConfiguration.getIndexUpdaterClass(IndexUpdateConfiguration.java:145)
> at
> org.apache.hadoop.contrib.index.main.UpdateIndex.main(UpdateIndex.java:257)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)
> Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> ... 16 more
>
> -----------------------------------------------------------------------------------------------------------------
>
> 2008-10-10
>
>
>
> chenlbspace
>
>
__________ NOD32 3510 (20081010) Information __________
This message was checked by NOD32 antivirus system.
http://www.nod32cn.com
