I modified the cluster-reuters.sh file to include a new case for cvb
I am pretty sure the script works as I had successfully tested on a pseudo
distribution hadoop, but it is failing on a cluster,
elif [ "x$clustertype" == "xcvb" ]; then
echo "cvb!!!"
$MAHOUT seq2sparse \
-i ${WORK_DIR}/reuters-out-seqdir/ \
-o ${WORK_DIR}/reuters-out-seqdir-sparse-cvb -wt tf -seq -nr 3
--namedVector\
\
&& \
$MAHOUT rowid -i ${WORK_DIR}/reuters-out-seqdir-sparse-cvb/tf-vectors -o
${WOR\
K_DIR}/reuters-matrix\
$MAHOUT cvb -i ${WORK_DIR}/reuters-matrix/matrix -o reuters-cvb -k 15 -ow
-x 4\
-dict ${WORK_DIR}/reuters-out-seqdir-sparse-cvb/dictionary.file-* -mt
reuters-c\
vb-tm -dt reuters-cvb-dt
$MAHOUT cvb \
-i ${WORK_DIR}/reuters-out-seqdir-sparse-dirichlet/tfidf-vectors \
-o ${WORK_DIR}/reuters-dirichlet -k 20 -ow -x 20 -a0 2 \
-md
org.apache.mahout.clustering.dirichlet.models.DistanceMeasureClusterDist\
ribution \
-mp org.apache.mahout.math.DenseVector \
-dm org.apache.mahout.common.distance.CosineDistanceMeasure \
On Wed, Feb 13, 2013 at 12:10 PM, Jake Mannix <[email protected]> wrote:
> On Wed, Feb 13, 2013 at 7:37 AM, Yunming Zhang
> <[email protected]>wrote:
>
> > Hi,
> >
> > I ran into this error when I try to run Mahout CVB. I excluded the log4j
> > dependency in hbase in integration/pom.xml,
> >
> > Not sure what is going on here
> >
> > Thanks
> >
> > MAHOUT_LOCAL is set, running locally
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> >
> [jar:file:/home/yz17/mahout/trunk/examples/target/mahout-examples-0.8-SNAPSHOT-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> >
> [jar:file:/home/yz17/mahout/trunk/examples/target/dependency/slf4j-jcl-1.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> >
> [jar:file:/home/yz17/mahout/trunk/examples/target/dependency/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > explanation.
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> > WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please
> > use org.apache.hadoop.log.metrics.EventCounter in all the
> log4j.properties
> > files.
> > 13/02/12 10:07:07 DEBUG conf.Configuration: java.io.IOException: config()
> > at
> > org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)
> > at
> > org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:57)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> >
>
> Looks like you're running SequenceFilesFromDirectory, not CVB:
>
>
> > at
> >
> org.apache.mahout.text.SequenceFilesFromDirectory.main(SequenceFilesFromDirectory.java:53)
> >
>
> What is the command line you're using?
>
>
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> > at
> > org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> > at
> > org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195)
> >
> > 13/02/12 10:07:07 INFO common.AbstractJob: Command line arguments:
> > {--charset=[UTF-8], --chunkSize=[5], --endPhase=[2147483647],
> > --fileFilterClass=[org.apache.mahout.text.PrefixAdditionFilter],
> > --input=[/tmp/mahout-work-yz17/reuters-out], --keyPrefix=[],
> > --output=[/tmp/mahout-work-yz17/reuters-out-seqdir], --startPhase=[0],
> > --tempDir=[temp]}
> > 13/02/12 10:07:07 DEBUG conf.Configuration: java.io.IOException: config()
> > at
> > org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)
> > at
> > org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:184)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:236)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:466)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:452)
> > at
> > org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1494)
> > at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1395)
> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:238)
> > at
> >
> org.apache.mahout.text.SequenceFilesFromDirectory.run(SequenceFilesFromDirectory.java:78)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> > at
> >
> org.apache.mahout.text.SequenceFilesFromDirectory.main(SequenceFilesFromDirectory.java:53)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> > at
> > org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> > at
> > org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195)
> >
> > 13/02/12 10:07:07 DEBUG security.Groups: Creating new Groups object
> > 13/02/12 10:07:07 DEBUG security.Groups: Group mapping
> > impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
> > cacheTimeout=300000
> > 13/02/12 10:07:07 DEBUG security.UserGroupInformation: hadoop login
> > 13/02/12 10:07:07 DEBUG security.UserGroupInformation: hadoop login
> commit
> > 13/02/12 10:07:07 DEBUG security.UserGroupInformation: using local
> > user:UnixPrincipal: yz17
> > 13/02/12 10:07:07 DEBUG security.UserGroupInformation: UGI loginUser:yz17
> > 13/02/12 10:07:07 DEBUG fs.FileSystem: Creating filesystem for file:///
> > 13/02/12 10:07:07 DEBUG util.NativeCodeLoader: Trying to load the
> > custom-built native-hadoop library...
> > 13/02/12 10:07:07 DEBUG util.NativeCodeLoader: Failed to load
> > native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in
> > java.library.path
> > 13/02/12 10:07:07 DEBUG util.NativeCodeLoader:
> >
> java.library.path=/opt/apps/jdk/1.6.0_12/jre/lib/amd64/server:/opt/apps/jdk/1.6.0_12/jre/lib/amd64:/opt/apps/jdk/1.6.0_12/jre/../lib/amd64:/home/yz17/HJ-testrelease/hj.release/lib:/home/yz17/tools/Python-2.7.3//Lib:/opt/apps/mvapich/1.2rc1-intel/lib/shared:/opt/apps/mvapich/1.2rc1-intel/lib:/opt/apps/intel/ics-2011.0.13/mkl/lib/intel64:/opt/apps/intel/ics-2011.0.13/itac/
> >
> 8.0.1.009/lib:/opt/apps/intel/ics-2011.0.13/lib/intel64:/usr/java/packages/lib/amd64:/lib:/usr/lib
> >
> >
>
>
> --
>
> -jake
>
