[
https://issues.apache.org/jira/browse/HAMA-781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719247#comment-13719247
]
Ikhtiyor Ahmedov commented on HAMA-781:
---------------------------------------
Same code affects when multiple inputs given as input.
Code:
{quote}
// set partitionID to rawSplit
if (split.getClass().getName().equals(FileSplit.class.getName())
&& job.getConfiguration().get(Constants.RUNTIME_PARTITIONING_CLASS)
!= null
&& job.get("bsp.partitioning.runner.job") == null) {
LOG.debug(((FileSplit) split).getPath().getName());
String[] extractPartitionID = ((FileSplit) split).getPath().getName()
.split("[-]");
rawSplit.setPartitionID(Integer.parseInt(extractPartitionID[1]));
}
{quote}
Exception:
{quote}java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.hama.bsp.BSPJobClient.writeSplits(BSPJobClient.java:566)
at
org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:342)
at org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:293)
at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:229)
at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:236)
at org.apache.hama.examples.OnlineCF.main(OnlineCF.java:427)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hama.examples.ExampleDriver.main(ExampleDriver.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hama.util.RunJar.main(RunJar.java:146)
{quote}
Example fail cases:
1) When input is non-hdfs format (part-0000, part-0001) and size is big
(usually from local filesystem)
2) When input is given as multiple files in local mode:
{quote}SequenceFileInputFormat.addInputPaths(job,
"/tmp/test.seq,/tmp/test2.seq,/tmp/test3.seq");{quote}
> Setting partition split fails in local mode when file size is big and has a
> runtime partition (HashParitioner)
> --------------------------------------------------------------------------------------------------------------
>
> Key: HAMA-781
> URL: https://issues.apache.org/jira/browse/HAMA-781
> Project: Hama
> Issue Type: Bug
> Components: bsp core
> Reporter: Ikhtiyor Ahmedov
> Priority: Minor
> Attachments: HAMA-781.patch
>
>
> when input partitioner set to HashPartitioner and file size is big in local
> mode; in line 566 of BSPJobClient.java throws index out of bound exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
