Re: [DISCUSS] Hama releases for each hadoop version
I am +1 with this if no additional issue.
On 12 August 2015 at 09:23, Edward J. Yoon wrote:
> Any objections/thoughts?
>
> On Wed, Jul 22, 2015 at 7:48 PM, Edward J. Yoon wrote:
>> Hi,
>>
>> Like http://www.apache.org/dist/spark/spark-1.3.1/, should we create
>> release tarball for each hadoop version?
>>
>> Otherwise, user always need to replace manually hadoop jar and some
>> dependency files in ${HAMA_HOME}/lib folder. Of course, src
>> distributions doesn't matter.
>>
>> --
>> Best Regards, Edward J. Yoon
>
>
>
> --
> Best Regards, Edward J. Yoon
Re: [DISCUSS] Hama releases for each hadoop version
Any objections/thoughts?
On Wed, Jul 22, 2015 at 7:48 PM, Edward J. Yoon wrote:
> Hi,
>
> Like http://www.apache.org/dist/spark/spark-1.3.1/, should we create
> release tarball for each hadoop version?
>
> Otherwise, user always need to replace manually hadoop jar and some
> dependency files in ${HAMA_HOME}/lib folder. Of course, src
> distributions doesn't matter.
>
> --
> Best Regards, Edward J. Yoon
--
Best Regards, Edward J. Yoon
[jira] [Commented] (HAMA-963) ArrayIndexOutOfBoundsException occurs when tasks are greater than splits
[
https://issues.apache.org/jira/browse/HAMA-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14692593#comment-14692593
]
Hudson commented on HAMA-963:
-
SUCCESS: Integrated in Hama-Nightly-for-Hadoop-1.x #1699 (See
[https://builds.apache.org/job/Hama-Nightly-for-Hadoop-1.x/1699/])
HAMA-963. Fix the problem that occurs ArrayIndexOutOfBoundsException (minho:
rev 1695219)
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/ZooKeeperSyncClientImpl.java
* /hama/trunk/yarn/src/main/java/org/apache/hama/bsp/ApplicationMaster.java
* /hama/trunk/yarn/src/main/java/org/apache/hama/bsp/YARNBSPJob.java
> ArrayIndexOutOfBoundsException occurs when tasks are greater than splits
>
>
> Key: HAMA-963
> URL: https://issues.apache.org/jira/browse/HAMA-963
> Project: Hama
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Edward J. Yoon
>Assignee: Minho Kim
>Priority: Blocker
> Fix For: 0.7.1
>
>
> ArrayIndexOutOfBoundsException occurs when the number of tasks are greater
> than the number of splits at ApplicationMaster 950 line.
> {code}
> assignedSplit = splits[taskid.id];
> {code}
> There are two options: Option1. launch additional tasks without input split.
> Option2. adjust the number of tasks as number of input splits.
> I prefer the option 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAMA-963) ArrayIndexOutOfBoundsException occurs when tasks are greater than splits
[
https://issues.apache.org/jira/browse/HAMA-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14692522#comment-14692522
]
Hudson commented on HAMA-963:
-
SUCCESS: Integrated in Hama-Nightly-for-Hadoop-2.x #695 (See
[https://builds.apache.org/job/Hama-Nightly-for-Hadoop-2.x/695/])
HAMA-963. Fix the problem that occurs ArrayIndexOutOfBoundsException (minho:
rev 1695219)
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/ZooKeeperSyncClientImpl.java
* /hama/trunk/yarn/src/main/java/org/apache/hama/bsp/ApplicationMaster.java
* /hama/trunk/yarn/src/main/java/org/apache/hama/bsp/YARNBSPJob.java
> ArrayIndexOutOfBoundsException occurs when tasks are greater than splits
>
>
> Key: HAMA-963
> URL: https://issues.apache.org/jira/browse/HAMA-963
> Project: Hama
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Edward J. Yoon
>Assignee: Minho Kim
>Priority: Blocker
> Fix For: 0.7.1
>
>
> ArrayIndexOutOfBoundsException occurs when the number of tasks are greater
> than the number of splits at ApplicationMaster 950 line.
> {code}
> assignedSplit = splits[taskid.id];
> {code}
> There are two options: Option1. launch additional tasks without input split.
> Option2. adjust the number of tasks as number of input splits.
> I prefer the option 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Problems running Hama v0.7 on Yarn
Thanks for your help!
On Tue, Aug 11, 2015 at 4:51 PM, Minho Kim wrote:
> Hi,
>
> I examine this problem. The issue in jira is as follows.
>
> https://issues.apache.org/jira/browse/HAMA-963
>
> This problem occurs since partition id is allocated to containerId which is
> allocated by YARN.
> ContainerId of the worker tasks begin with 2 because the first containerId
> is allocated to ApplicationMaster.
> It means that splits' array overflow occurs because partitionId is greater
> than the size of splits array.
> I modified that partitionId is allocated to the incremental variable
> started zero using AtomicInteger.
>
> Best regards,
> Minho Kim.
>
>
> 2015-07-27 8:55 GMT+09:00 Edward J. Yoon :
>
>> Thanks for your report. I'll check!
>>
>> On Sun, Jul 26, 2015 at 8:53 PM, Leonidas Fegaras
>> wrote:
>> > I wrote a very small Hama program to test it on a Yarn cluster running
>> on my
>> > laptop to isolate the problem:
>> >
>> > final public class BSPTest extends BSP> > Text, Text> {
>> >
>> > @Override
>> > public final void bsp( BSPPeer> Text,
>> > Text> peer)
>> > throws IOException, InterruptedException,
>> SyncException {
>> > LongWritable key = new LongWritable();
>> > Text value = new Text();
>> > peer.readNext(key,value);
>> > peer.write(key,value);
>> > }
>> >
>> > public static void main ( String[] args ) throws Exception {
>> > HamaConfiguration conf = new HamaConfiguration();
>> > conf.set("yarn.resourcemanager.address","localhost:8032");
>> > YARNBSPJob job = new YARNBSPJob(conf);
>> > job.setMemoryUsedPerTaskInMb(500);
>> > job.setNumBspTask(4);
>> > job.setJobName("test");
>> > job.setBspClass(BSPTest.class);
>> > job.setJarByClass(BSPTest.class);
>> > job.setInputKeyClass(LongWritable.class);
>> > job.setInputValueClass(Text.class);
>> > job.setInputPath(new Path("in"));
>> > job.setInputFormat(TextInputFormat.class);
>> > job.setPartitioner(org.apache.hama.bsp.HashPartitioner.class);
>> > job.set("bsp.min.split.size",Long.toString(1000));
>> > job.setOutputPath(new Path("out"));
>> > job.setOutputKeyClass(LongWritable.class);
>> > job.setOutputValueClass(Text.class);
>> > job.setOutputFormat(TextOutputFormat.class);
>> > job.waitForCompletion(true);
>> > }
>> > }
>> >
>> > where "in" is a small text file stored on HDFS. It does the file
>> > partitioning into 4 files but then it gives me the same error:
>> >
>> > 15/07/26 06:46:25 INFO ipc.Server: IPC Server handler 0 on 1, call
>> > getTask(attempt_appattempt_1437858941768_0042_01__04_4) from
>> > 127.0.0.1:54752: error: java.io.IOException:
>> > java.lang.ArrayIndexOutOfBoundsException: 4
>> > java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 4
>> > at
>> > org.apache.hama.bsp.ApplicationMaster.getTask(ApplicationMaster.java:950)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:497)
>> > at org.apache.hama.ipc.RPC$Server.call(RPC.java:615)
>> > at org.apache.hama.ipc.Server$Handler$1.run(Server.java:1211)
>> > at org.apache.hama.ipc.Server$Handler$1.run(Server.java:1207)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:422)
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> >
>> > I get the same error even when I remove the partitioning and I use 1
>> task.
>> > Leonidas
>> >
>> >
>> > On 07/19/2015 06:55 PM, Edward J. Yoon wrote:
>> >>>
>> >>> It seems that the reason is that I have 1 input block (1 split)
>> but
>> >>> I
>> >>> use 4 tasks.
>> >>
>> >> Thanks for your report, it should be addressed.
>> >>
>> >>> But the Application master shouldn't crash; it should have used 1
>> >>> task instead.
>> >>
>> >> Or, we can launch 1 task and 3 tasks without split. In this case, you
>> >> should distribute the input data yourself within your BSP program.
>> >> Graph package of 0.7.0 partitions vertices into empty tasks directly
>> >> using barrier sync if tasks num is greater than blocks num.
>> >>
>> >>> 2) If I use the PartitioningRunner using:
>> >>> job.setPartitioner(org.apache.hama.bsp.HashPartitioner.class);
>> >>> job.setNumBspTask(4);
>> >>> job.set("bsp.min.split.size","102");
>> >>> it fails because it expects a Long key. Here is the log:
>> >>
>> >> By default, PartitioningRunner reads and re-writes key and value pairs
>> >> based on "bsp.input.key/value.class". I guess your input is Text file
>> >> and so key is automatically Long but you've set M
[jira] [Commented] (HAMA-963) ArrayIndexOutOfBoundsException occurs when tasks are greater than splits
[
https://issues.apache.org/jira/browse/HAMA-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14681361#comment-14681361
]
Hudson commented on HAMA-963:
-
SUCCESS: Integrated in Hama-trunk #450 (See
[https://builds.apache.org/job/Hama-trunk/450/])
HAMA-963. Fix the problem that occurs ArrayIndexOutOfBoundsException (minho:
rev 1695219)
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/sync/ZooKeeperSyncClientImpl.java
* /hama/trunk/yarn/src/main/java/org/apache/hama/bsp/ApplicationMaster.java
* /hama/trunk/yarn/src/main/java/org/apache/hama/bsp/YARNBSPJob.java
> ArrayIndexOutOfBoundsException occurs when tasks are greater than splits
>
>
> Key: HAMA-963
> URL: https://issues.apache.org/jira/browse/HAMA-963
> Project: Hama
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Edward J. Yoon
>Assignee: Minho Kim
>Priority: Blocker
> Fix For: 0.7.1
>
>
> ArrayIndexOutOfBoundsException occurs when the number of tasks are greater
> than the number of splits at ApplicationMaster 950 line.
> {code}
> assignedSplit = splits[taskid.id];
> {code}
> There are two options: Option1. launch additional tasks without input split.
> Option2. adjust the number of tasks as number of input splits.
> I prefer the option 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HAMA-963) ArrayIndexOutOfBoundsException occurs when tasks are greater than splits
[
https://issues.apache.org/jira/browse/HAMA-963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Minho Kim resolved HAMA-963.
Resolution: Fixed
This problem occurs since partition id is allocated to containerId which is
allocated by YARN. ContainerId of the worker tasks begin with 2 because the
first containerId is allocated to ApplicationMaster. It means that splits'
array overflow occurs when you run Hama application on YARN.
I fixed it!
Thanks.
> ArrayIndexOutOfBoundsException occurs when tasks are greater than splits
>
>
> Key: HAMA-963
> URL: https://issues.apache.org/jira/browse/HAMA-963
> Project: Hama
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Edward J. Yoon
>Assignee: Minho Kim
>Priority: Blocker
> Fix For: 0.7.1
>
>
> ArrayIndexOutOfBoundsException occurs when the number of tasks are greater
> than the number of splits at ApplicationMaster 950 line.
> {code}
> assignedSplit = splits[taskid.id];
> {code}
> There are two options: Option1. launch additional tasks without input split.
> Option2. adjust the number of tasks as number of input splits.
> I prefer the option 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
