Also it looks like you are not passing a clusters-n directory to the
--seqFileDir as you were in your first posting. ClusterDumper won't
output anything if it cannot read clusters from that directory. Also,
all the synthetic control jobs now all call ClusterDumper automatically
after clustering the points.
On 7/15/10 5:58 PM, Jeff Eastman wrote:
Hi Gokul,
Try building and running again. I committed a patch to ClusterDumper
which handles the _log file error when running on Hadoop.
Jeff
On 7/15/10 2:27 PM, Gokul Pillai wrote:
My bad. After setting HADOOP_CONF_DIR and HADOOP_HOME, I now don't
get the
errors.
However, I dont get any output too.
I tried this command too but again no output:
./bin/mahout clusterdump --seqFileDir dirichlet/output/data/ --pointsDir
dirichlet/output/clusteredPoints/ --output dumpOut
Anybody run the clusterdump successfully?
On Thu, Jul 15, 2010 at 2:19 PM, Gokul Pillai<[email protected]>
wrote:
I have Cloudera's CDH3 running on Ubuntu 10.04 version. And I have
Apache
Mahout (0.40 Snapshot version from yesterday).
I was trying to get the clustering examples running based on the
wiki page
https://cwiki.apache.org/confluence/display/MAHOUT/Synthetic+Control+Data.
At the bottom of this page, there is a section that describes how to
get
the data out and process it.
Get the data out of HDFS 3
<https://cwiki.apache.org/confluence/display/MAHOUT/Synthetic+Control+Data#Footnote3>
4
<https://cwiki.apache.org/confluence/display/MAHOUT/Synthetic+Control+Data#Footnote4>
and
have a look 5
<https://cwiki.apache.org/confluence/display/MAHOUT/Synthetic+Control+Data#Footnote5>
- All example jobs use *testdata* as input and output to
directory *
output*
- Use *bin/hadoop fs -lsr output* to view all outputs. Copy them
all to
your local machine and you can run the ClusterDumper on them.
- Sequence files containing the original points in Vector
form are
in *output/data*
- Computed clusters are contained in *output/clusters-i*
- All result clustered points are placed into *
output/clusteredPoints*
So I got the data out of HDFS onto my local and it looks like this:
had...@ubuntu:~/mahoutOutputs$ ls -l dirichlet/output/
total 32
drwxr-xr-x 3 hadoop hadoop 4096 2010-07-13 16:06 clusteredPoints
drwxr-xr-x 2 hadoop hadoop 4096 2010-07-13 16:06 clusters-0
drwxr-xr-x 3 hadoop hadoop 4096 2010-07-13 16:06 clusters-1
drwxr-xr-x 3 hadoop hadoop 4096 2010-07-13 16:06 clusters-2
drwxr-xr-x 3 hadoop hadoop 4096 2010-07-13 16:06 clusters-3
drwxr-xr-x 3 hadoop hadoop 4096 2010-07-13 16:06 clusters-4
drwxr-xr-x 3 hadoop hadoop 4096 2010-07-13 16:06 clusters-5
drwxr-xr-x 3 hadoop hadoop 4096 2010-07-13 16:06 data
However, when I ran clusterdump on this, I get the following error. Any
help on why clusterdump is complaining about a "_logs" folder would be
helpful:
had...@ubuntu:~/mahoutOutputs$ ../mahoutsvn/trunk/bin/mahout
clusterdump
--seqFileDir dirichlet/output/clusters-1 --pointsDir
dirichlet/output/clusteredPoints/ --output dumpOut
no HADOOP_CONF_DIR or HADOOP_HOME set, running locally
Exception in thread "main" java.io.FileNotFoundException:
/home/hadoop/mahoutOutputs/dirichlet/output/clusteredPoints/_logs (Is a
directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at
org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream.<init>(RawLocalFileSystem.java:63)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.<init>(RawLocalFileSystem.java:99)
at
org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:169)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:126)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at
org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1437)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
at
org.apache.mahout.utils.clustering.ClusterDumper.readPoints(ClusterDumper.java:323)
at
org.apache.mahout.utils.clustering.ClusterDumper.init(ClusterDumper.java:93)
at
org.apache.mahout.utils.clustering.ClusterDumper.<init>(ClusterDumper.java:86)
at
org.apache.mahout.utils.clustering.ClusterDumper.main(ClusterDumper.java:272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at
org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:175)
Regards
Gokul
