Problems running SSVD directly from Java

[Mihai Dascalu]

Hi!


I’ve been experimenting with the SSVDSolver and unfortunately, during runtime, 
I encounter this error:

10648576 [Thread-13] WARN org.apache.hadoop.mapred.LocalJobRunner  - 
job_local1958711697_0001
java.lang.NoClassDefFoundError: org/apache/commons/httpclient/HttpMethod
        at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:546)
Caused by: java.lang.ClassNotFoundException: 
org.apache.commons.httpclient.HttpMethod
        at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 1 more

Exception in thread "Thread-13" java.lang.NoClassDefFoundError: 
org/apache/commons/httpclient/HttpMethod
        at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:562)
Caused by: java.lang.ClassNotFoundException: 
org.apache.commons.httpclient.HttpMethod
        at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 1 more 

The actual invocation is:

public static void runSSVDOnSparseVectors(String inputPath,
                        String outputPath, int rank, int oversampling, int 
blocks,
                        int reduceTasks, int powerIterations, boolean halfSigma)
                        throws IOException {
        Configuration conf = new Configuration();
        SSVDSolver solver = new SSVDSolver(conf, new Path[] { new Path(
                        inputPath) }, new Path(outputPath), blocks, rank, 
oversampling,
                        reduceTasks);
        solver.setQ(powerIterations);
        if (halfSigma) {
                solver.setcUHalfSigma(true);
                solver.setcVHalfSigma(true);
        }
        solver.run();
}

while being invoked with (input.getParent() + “/" + TERM_DOC_MATRIX_NAME, 
input.getParent() + “/" + SVD_FOLDER_NAME, k, 2 * k, Math.min(200000, (int) (3 
* k * 0.01 * Math.max(lsaTraining.getNoDocuments(),lsaTraining.getNoWords()))), 
5, 2, true);

I’m using Mahout 0.10 with httpclient-4.4.1.jar (I tried also 4.2.5 from the 
package archive) on a 48k words X 53k docs matrix.

Any ideas? It works fine with the similar variables if I run the job in command 
line.

Also, how should I tweak the input variables?


Thanks in advance!
Mihai