Hi,
i tried the hash partitioner but it did not work ... see below. Should i
split the file into 96 files or is there another way?
- Set the runtime partitioner
conf.setBoolean("bsp.input.runtime.partitioning", true);
bsp.setPartitioner(ML_DataPartitioner.class);
bsp.setNumBspTask(96);
I am getting the following exception:
Exception in thread "main" org.apache.hama.ipc.RemoteException:
java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 48
at org.apache.hama.bsp.BSPJobClient.readSplitFile(BSPJobClient.java:621)
at org.apache.hama.bsp.JobInProgress.initTasks(JobInProgress.java:261)
at org.apache.hama.bsp.QueueManager.initJob(QueueManager.java:44)
at
org.apache.hama.bsp.SimpleTaskScheduler$JobListener.jobAdded(SimpleTaskScheduler.java:117)
at org.apache.hama.bsp.BSPMaster.addJob(BSPMaster.java:753)
at org.apache.hama.bsp.BSPMaster.submitJob(BSPMaster.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hama.ipc.RPC$Server.call(RPC.java:613)
at org.apache.hama.ipc.Server$Handler$1.run(Server.java:1211)
at org.apache.hama.ipc.Server$Handler$1.run(Server.java:1207)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
at org.apache.hama.ipc.Server$Handler.run(Server.java:1206)
at org.apache.hama.ipc.Client.call(Client.java:908)
at org.apache.hama.ipc.RPC$Invoker.invoke(RPC.java:238)
at com.sun.proxy.$Proxy0.submitJob(Unknown Source)
at org.apache.hama.bsp.BSPJobClient.launchJob(BSPJobClient.java:496)
at
org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:380)
at org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:293)
at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:228)
at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:235)
at
de.distMLP.Train_MultilayerPerceptron.main(Train_MultilayerPerceptron.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hama.util.RunJar.main(RunJar.java:146)
