Try clusterdump -s reuters-clusters/cluster-6 -d... It's expecting a
directory to find the cluster parts in and is quite passive about doing
nothing if it does not find one. This could obviously be improved; at
least an error message would be appropriate. I see it does not extend
AbstractJob either. I'll look into that next week.
Jeff
On 7/16/10 12:24 AM, Robin Anil wrote:
I am trying to run clusterdumper from trunk. seems like its
not outputting anything. Need to investigate
bin/mahout clusterdump -s reuters-clusters/cluster-6/part-r-00000 -d
reuters-vectors/dictionary.file-0 -dt sequencefile -n 10 -b 100
On Fri, Jul 16, 2010 at 7:08 AM, Jeff Eastman<[email protected]>wrote:
Also it looks like you are not passing a clusters-n directory to the
--seqFileDir as you were in your first posting. ClusterDumper won't output
anything if it cannot read clusters from that directory. Also, all the
synthetic control jobs now all call ClusterDumper automatically after
clustering the points.
On 7/15/10 5:58 PM, Jeff Eastman wrote:
Hi Gokul,
Try building and running again. I committed a patch to ClusterDumper which
handles the _log file error when running on Hadoop.
Jeff
On 7/15/10 2:27 PM, Gokul Pillai wrote:
My bad. After setting HADOOP_CONF_DIR and HADOOP_HOME, I now don't get
the
errors.
However, I dont get any output too.
I tried this command too but again no output:
./bin/mahout clusterdump --seqFileDir dirichlet/output/data/ --pointsDir
dirichlet/output/clusteredPoints/ --output dumpOut
Anybody run the clusterdump successfully?
On Thu, Jul 15, 2010 at 2:19 PM, Gokul Pillai<[email protected]>
wrote:
I have Cloudera's CDH3 running on Ubuntu 10.04 version. And I have
Apache
Mahout (0.40 Snapshot version from yesterday).
I was trying to get the clustering examples running based on the wiki
page
https://cwiki.apache.org/confluence/display/MAHOUT/Synthetic+Control+Data.
At the bottom of this page, there is a section that describes how to get
the data out and process it.
Get the data out of HDFS 3
<
https://cwiki.apache.org/confluence/display/MAHOUT/Synthetic+Control+Data#Footnote3>
4
<
https://cwiki.apache.org/confluence/display/MAHOUT/Synthetic+Control+Data#Footnote4>
and
have a look 5
<
https://cwiki.apache.org/confluence/display/MAHOUT/Synthetic+Control+Data#Footnote5>
- All example jobs use *testdata* as input and output to directory *
output*
- Use *bin/hadoop fs -lsr output* to view all outputs. Copy them all
to
your local machine and you can run the ClusterDumper on them.
- Sequence files containing the original points in Vector form are
in *output/data*
- Computed clusters are contained in *output/clusters-i*
- All result clustered points are placed into *
output/clusteredPoints*
So I got the data out of HDFS onto my local and it looks like this:
had...@ubuntu:~/mahoutOutputs$ ls -l dirichlet/output/
total 32
drwxr-xr-x 3 hadoop hadoop 4096 2010-07-13 16:06 clusteredPoints
drwxr-xr-x 2 hadoop hadoop 4096 2010-07-13 16:06 clusters-0
drwxr-xr-x 3 hadoop hadoop 4096 2010-07-13 16:06 clusters-1
drwxr-xr-x 3 hadoop hadoop 4096 2010-07-13 16:06 clusters-2
drwxr-xr-x 3 hadoop hadoop 4096 2010-07-13 16:06 clusters-3
drwxr-xr-x 3 hadoop hadoop 4096 2010-07-13 16:06 clusters-4
drwxr-xr-x 3 hadoop hadoop 4096 2010-07-13 16:06 clusters-5
drwxr-xr-x 3 hadoop hadoop 4096 2010-07-13 16:06 data
However, when I ran clusterdump on this, I get the following error. Any
help on why clusterdump is complaining about a "_logs" folder would be
helpful:
had...@ubuntu:~/mahoutOutputs$ ../mahoutsvn/trunk/bin/mahout
clusterdump
--seqFileDir dirichlet/output/clusters-1 --pointsDir
dirichlet/output/clusteredPoints/ --output dumpOut
no HADOOP_CONF_DIR or HADOOP_HOME set, running locally
Exception in thread "main" java.io.FileNotFoundException:
/home/hadoop/mahoutOutputs/dirichlet/output/clusteredPoints/_logs (Is a
directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at
org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream.<init>(RawLocalFileSystem.java:63)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.<init>(RawLocalFileSystem.java:99)
at
org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:169)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:126)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at
org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1437)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
at
org.apache.mahout.utils.clustering.ClusterDumper.readPoints(ClusterDumper.java:323)
at
org.apache.mahout.utils.clustering.ClusterDumper.init(ClusterDumper.java:93)
at
org.apache.mahout.utils.clustering.ClusterDumper.<init>(ClusterDumper.java:86)
at
org.apache.mahout.utils.clustering.ClusterDumper.main(ClusterDumper.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:175)
Regards
Gokul
