[
https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655483#action_12655483
]
Edward J. Yoon commented on HAMA-129:
-------------------------------------
5000 * 5000 matrix multiplication, blocks 16, 4 node
----
[d8g053:/root/hama-trunk]# bin/hama examples mult fiveA fiveB 16
08/12/11 11:45:22 INFO hama.AbstractMatrix: Create 4 * 4 blocked matrix
08/12/11 11:45:22 WARN mapred.JobClient: Use GenericOptionsParser for parsing
the arguments. Applications should implement Tool for the same.
08/12/11 11:45:22 WARN mapred.JobClient: Use genericOptions for the option
-libjars
08/12/11 11:45:22 WARN mapred.JobClient: No job jar file set. User classes may
not be found. See JobConf(Class) or JobConf#setJar(String).
08/12/11 11:45:22 INFO mapred.JobClient: Running job: job_200812100956_0045
08/12/11 11:45:23 INFO mapred.JobClient: map 0% reduce 0%
08/12/11 11:46:05 INFO mapred.JobClient: map 25% reduce 0%
08/12/11 11:46:14 INFO mapred.JobClient: map 25% reduce 2%
08/12/11 11:46:15 INFO mapred.JobClient: map 25% reduce 4%
08/12/11 11:46:16 INFO mapred.JobClient: map 25% reduce 6%
08/12/11 11:46:18 INFO mapred.JobClient: map 25% reduce 8%
08/12/11 11:47:58 INFO mapred.JobClient: map 50% reduce 8%
08/12/11 11:48:02 INFO mapred.JobClient: map 75% reduce 8%
08/12/11 11:48:03 INFO mapred.JobClient: map 75% reduce 10%
08/12/11 11:48:07 INFO mapred.JobClient: map 75% reduce 14%
08/12/11 11:48:08 INFO mapred.JobClient: map 75% reduce 18%
08/12/11 11:48:11 INFO mapred.JobClient: map 75% reduce 22%
08/12/11 11:48:15 INFO mapred.JobClient: map 75% reduce 25%
08/12/11 11:49:11 INFO mapred.JobClient: map 100% reduce 25%
08/12/11 11:49:17 INFO mapred.JobClient: map 100% reduce 35%
08/12/11 11:49:18 INFO mapred.JobClient: map 100% reduce 45%
08/12/11 11:49:20 INFO mapred.JobClient: map 100% reduce 47%
08/12/11 11:49:21 INFO mapred.JobClient: map 100% reduce 60%
08/12/11 11:49:22 INFO mapred.JobClient: map 100% reduce 70%
08/12/11 11:49:25 INFO mapred.JobClient: map 100% reduce 72%
08/12/11 11:49:26 INFO mapred.JobClient: map 100% reduce 81%
08/12/11 11:49:27 INFO mapred.JobClient: map 100% reduce 85%
08/12/11 11:49:30 INFO mapred.JobClient: map 100% reduce 87%
08/12/11 11:49:31 INFO mapred.JobClient: map 100% reduce 89%
08/12/11 11:49:33 INFO mapred.JobClient: map 100% reduce 95%
08/12/11 11:49:36 INFO mapred.JobClient: map 100% reduce 97%
08/12/11 11:49:39 INFO mapred.JobClient: Job complete: job_200812100956_0045
08/12/11 11:49:39 INFO mapred.JobClient: Counters: 13
08/12/11 11:49:39 INFO mapred.JobClient: File Systems
08/12/11 11:49:39 INFO mapred.JobClient: Local bytes read=1656774062
08/12/11 11:49:39 INFO mapred.JobClient: Local bytes written=2506547810
08/12/11 11:49:39 INFO mapred.JobClient: Job Counters
08/12/11 11:49:39 INFO mapred.JobClient: Launched reduce tasks=4
08/12/11 11:49:39 INFO mapred.JobClient: Launched map tasks=7
08/12/11 11:49:39 INFO mapred.JobClient: Map-Reduce Framework
08/12/11 11:49:39 INFO mapred.JobClient: Reduce input groups=16
08/12/11 11:49:39 INFO mapred.JobClient: Combine output records=0
08/12/11 11:49:39 INFO mapred.JobClient: Map input records=5000
08/12/11 11:49:39 INFO mapred.JobClient: Reduce output records=0
08/12/11 11:49:39 INFO mapred.JobClient: Map output bytes=844785560
08/12/11 11:49:39 INFO mapred.JobClient: Map input bytes=0
08/12/11 11:49:39 INFO mapred.JobClient: Combine input records=0
08/12/11 11:49:39 INFO mapred.JobClient: Map output records=20000
08/12/11 11:49:39 INFO mapred.JobClient: Reduce input records=20000
08/12/11 11:49:40 INFO hama.AbstractMatrix: Create 4 * 4 blocked matrix
08/12/11 11:49:40 WARN mapred.JobClient: Use GenericOptionsParser for parsing
the arguments. Applications should implement Tool for the same.
08/12/11 11:49:40 WARN mapred.JobClient: Use genericOptions for the option
-libjars
08/12/11 11:49:40 WARN mapred.JobClient: No job jar file set. User classes may
not be found. See JobConf(Class) or JobConf#setJar(String).
08/12/11 11:49:40 INFO mapred.JobClient: Running job: job_200812100956_0046
08/12/11 11:49:41 INFO mapred.JobClient: map 0% reduce 0%
08/12/11 11:50:13 INFO mapred.JobClient: map 25% reduce 0%
08/12/11 11:50:21 INFO mapred.JobClient: map 25% reduce 2%
08/12/11 11:50:23 INFO mapred.JobClient: map 25% reduce 4%
08/12/11 11:50:26 INFO mapred.JobClient: map 25% reduce 6%
08/12/11 11:50:27 INFO mapred.JobClient: map 25% reduce 8%
08/12/11 11:52:20 INFO mapred.JobClient: map 50% reduce 8%
08/12/11 11:52:21 INFO mapred.JobClient: map 75% reduce 8%
08/12/11 11:52:29 INFO mapred.JobClient: map 75% reduce 12%
08/12/11 11:52:32 INFO mapred.JobClient: map 75% reduce 16%
08/12/11 11:52:33 INFO mapred.JobClient: map 75% reduce 25%
08/12/11 11:54:03 INFO mapred.JobClient: map 100% reduce 25%
08/12/11 11:54:11 INFO mapred.JobClient: map 100% reduce 35%
08/12/11 11:54:14 INFO mapred.JobClient: map 100% reduce 37%
08/12/11 11:54:17 INFO mapred.JobClient: map 100% reduce 47%
08/12/11 11:54:18 INFO mapred.JobClient: map 100% reduce 72%
08/12/11 11:54:20 INFO mapred.JobClient: map 100% reduce 75%
08/12/11 11:54:22 INFO mapred.JobClient: map 100% reduce 77%
08/12/11 11:54:23 INFO mapred.JobClient: map 100% reduce 83%
08/12/11 11:54:24 INFO mapred.JobClient: map 100% reduce 85%
08/12/11 11:54:25 INFO mapred.JobClient: map 100% reduce 87%
08/12/11 11:54:26 INFO mapred.JobClient: map 100% reduce 89%
08/12/11 11:54:27 INFO mapred.JobClient: map 100% reduce 95%
08/12/11 11:54:32 INFO mapred.JobClient: Job complete: job_200812100956_0046
08/12/11 11:54:32 INFO mapred.JobClient: Counters: 13
08/12/11 11:54:32 INFO mapred.JobClient: File Systems
08/12/11 11:54:32 INFO mapred.JobClient: Local bytes read=1656774062
08/12/11 11:54:32 INFO mapred.JobClient: Local bytes written=2506547810
08/12/11 11:54:32 INFO mapred.JobClient: Job Counters
08/12/11 11:54:32 INFO mapred.JobClient: Launched reduce tasks=5
08/12/11 11:54:32 INFO mapred.JobClient: Launched map tasks=7
08/12/11 11:54:32 INFO mapred.JobClient: Map-Reduce Framework
08/12/11 11:54:32 INFO mapred.JobClient: Reduce input groups=16
08/12/11 11:54:32 INFO mapred.JobClient: Combine output records=0
08/12/11 11:54:32 INFO mapred.JobClient: Map input records=5000
08/12/11 11:54:32 INFO mapred.JobClient: Reduce output records=0
08/12/11 11:54:32 INFO mapred.JobClient: Map output bytes=844785560
08/12/11 11:54:32 INFO mapred.JobClient: Map input bytes=0
08/12/11 11:54:32 INFO mapred.JobClient: Combine input records=0
08/12/11 11:54:32 INFO mapred.JobClient: Map output records=20000
08/12/11 11:54:32 INFO mapred.JobClient: Reduce input records=20000
08/12/11 11:54:32 INFO hama.AbstractMatrix: Initializing the matrix storage.
08/12/11 11:54:38 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randbcfpq
08/12/11 11:54:38 WARN mapred.JobClient: Use GenericOptionsParser for parsing
the arguments. Applications should implement Tool for the same.
08/12/11 11:54:38 WARN mapred.JobClient: Use genericOptions for the option
-libjars
08/12/11 11:54:38 WARN mapred.JobClient: No job jar file set. User classes may
not be found. See JobConf(Class) or JobConf#setJar(String).
08/12/11 11:54:38 INFO mapred.JobClient: Running job: job_200812100956_0047
08/12/11 11:54:39 INFO mapred.JobClient: map 0% reduce 0%
08/12/11 11:57:30 INFO mapred.JobClient: map 50% reduce 0%
08/12/11 11:57:37 INFO mapred.JobClient: map 50% reduce 4%
08/12/11 11:57:41 INFO mapred.JobClient: map 50% reduce 8%
08/12/11 11:57:42 INFO mapred.JobClient: map 50% reduce 12%
08/12/11 11:57:47 INFO mapred.JobClient: map 50% reduce 16%
08/12/11 12:00:31 INFO mapred.JobClient: map 100% reduce 16%
08/12/11 12:00:38 INFO mapred.JobClient: map 100% reduce 29%
08/12/11 12:00:43 INFO mapred.JobClient: map 100% reduce 42%
08/12/11 12:00:46 INFO mapred.JobClient: map 100% reduce 46%
08/12/11 12:00:47 INFO mapred.JobClient: map 100% reduce 59%
08/12/11 12:00:51 INFO mapred.JobClient: map 100% reduce 68%
08/12/11 12:07:08 INFO mapred.JobClient: map 100% reduce 76%
08/12/11 12:07:39 INFO mapred.JobClient: map 100% reduce 84%
08/12/11 12:07:40 INFO mapred.JobClient: map 100% reduce 92%
08/12/11 12:07:45 INFO mapred.JobClient: Job complete: job_200812100956_0047
08/12/11 12:07:45 INFO mapred.JobClient: Counters: 13
08/12/11 12:07:45 INFO mapred.JobClient: File Systems
08/12/11 12:07:45 INFO mapred.JobClient: Local bytes read=1639585414
08/12/11 12:07:45 INFO mapred.JobClient: Local bytes written=2421198052
08/12/11 12:07:45 INFO mapred.JobClient: Job Counters
08/12/11 12:07:45 INFO mapred.JobClient: Launched reduce tasks=5
08/12/11 12:07:45 INFO mapred.JobClient: Launched map tasks=3
08/12/11 12:07:45 INFO mapred.JobClient: Map-Reduce Framework
08/12/11 12:07:45 INFO mapred.JobClient: Reduce input groups=4
08/12/11 12:07:45 INFO mapred.JobClient: Combine output records=0
08/12/11 12:07:45 INFO mapred.JobClient: Map input records=4
08/12/11 12:07:45 INFO mapred.JobClient: Reduce output records=20000
08/12/11 12:07:45 INFO mapred.JobClient: Map output bytes=800808832
08/12/11 12:07:45 INFO mapred.JobClient: Map input bytes=0
08/12/11 12:07:45 INFO mapred.JobClient: Combine input records=0
08/12/11 12:07:45 INFO mapred.JobClient: Map output records=64
08/12/11 12:07:45 INFO mapred.JobClient: Reduce input records=64
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes
> "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
