Oh, now I get how iterate() works. HAMA-704 is nicely written. On Thu, Mar 14, 2013 at 12:02 AM, Edward J. Yoon <[email protected]> wrote: > I'm reading changes of HAMA-704 again. As a result of adding > DiskVerticesInfo, vertices list is needed to be sorted. I'm not sure > but I think this approach will bring more disadvantages than > advantages. > > On Wed, Mar 13, 2013 at 11:09 PM, Edward J. Yoon <[email protected]> > wrote: >>>>> in loadVertices? Maybe consider feature for coupling storage in user space >>>>> with BSP Messaging[HAMA-734] can avoid double reads and writes. This way >>>>> partitioned or non-partitioned by partitioner, can keep vertices sorted >>>>> with a single read and single write on every peer. >> >> And, as I commented JIRA ticket, I think we can't use messaging system >> for sorting vertices within partition files. >> >> On Wed, Mar 13, 2013 at 11:00 PM, Edward J. Yoon <[email protected]> >> wrote: >>> P.S., (number of splits = number of partitions) is really confuse to >>> me. Even though blocks number is equal to desired tasks number, data >>> should be re-partitioned again. >>> >>> On Wed, Mar 13, 2013 at 10:36 PM, Edward J. Yoon <[email protected]> >>> wrote: >>>> Indeed. If there are already partitioned input files (unsorted) and so >>>> user want to skip pre-partitioning phase, it should be handled in >>>> GraphJobRunner BSP program. Actually, I still don't know why >>>> re-partitioned files need to be Sorted. It's only about >>>> GraphJobRunner. >>>> >>>>> partitioning. (This is outside the scope of graphs. We can have a >>>>> dedicated >>>>> partitioning superstep for graph applications). >>>> >>>> Sorry. I don't understand exactly yet. Do you mean just a partitioning >>>> job based on superstep API? >>>> >>>> By default, 100 tasks will be assigned for partitioning job. >>>> Partitioning job will create 1,000 partitions. Thus, we can execute >>>> the Graph job with 1,000 tasks. >>>> >>>> Let's assume that a input sequence file is 20GB (100 blocks). If I >>>> want to run with 1,000 tasks, what happens? >>>> >>>> On Wed, Mar 13, 2013 at 9:49 PM, Suraj Menon <[email protected]> >>>> wrote: >>>>> I am responding on this thread because of better continuity for >>>>> conversation. We cannot expect the partitions to be sorted every time. >>>>> When >>>>> the number of splits = number of partitions and partitioning is switched >>>>> off by user[HAMA-561], the partitions would not be sorted. Can we do this >>>>> in loadVertices? Maybe consider feature for coupling storage in user space >>>>> with BSP Messaging[HAMA-734] can avoid double reads and writes. This way >>>>> partitioned or non-partitioned by partitioner, can keep vertices sorted >>>>> with a single read and single write on every peer. >>>>> >>>>> Just clearing confusion if any regarding superstep injection for >>>>> partitioning. (This is outside the scope of graphs. We can have a >>>>> dedicated >>>>> partitioning superstep for graph applications). >>>>> Say there are x splits and y number of tasks configured by user. >>>>> >>>>> if x > y >>>>> The y tasks are scheduled with x of them having each of the x splits and >>>>> the remaining with no resource local to them. Then the partitioning >>>>> superstep redistributes the partitions among them to create local >>>>> partitions. Now the question is can we re-initialize a peer's input based >>>>> on this new local part of partition? >>>>> >>>>> if y > x >>>>> works as it works today. >>>>> >>>>> Just putting my points in brainstorming. >>>>> >>>>> -Suraj >>>>> >>>>> >>>>> On Mon, Mar 11, 2013 at 7:39 AM, Edward J. Yoon >>>>> <[email protected]>wrote: >>>>> >>>>>> I just filed here https://issues.apache.org/jira/browse/HAMA-744 >>>>>> >>>>>> On Mon, Mar 11, 2013 at 7:35 PM, Edward J. Yoon <[email protected]> >>>>>> wrote: >>>>>> > Additionally, >>>>>> > >>>>>> >> spilling queue and sorted spilling queue, can we inject the >>>>>> >> partitioning >>>>>> >> superstep as the first superstep and use local memory? >>>>>> > >>>>>> > Can we execute different number of tasks per superstep? >>>>>> > >>>>>> > On Mon, Mar 11, 2013 at 6:56 PM, Edward J. Yoon <[email protected]> >>>>>> wrote: >>>>>> >>> For graph processing, the partitioned files that result from the >>>>>> >>> partitioning job must be sorted. Currently only the partition files >>>>>> >>> in >>>>>> >> >>>>>> >> I see. >>>>>> >> >>>>>> >>> For other partitionings and with regard to our superstep API, Suraj's >>>>>> idea >>>>>> >>> of injecting a preprocessing superstep that partitions the stuff into >>>>>> our >>>>>> >>> messaging system is actually the best. >>>>>> >> >>>>>> >> BTW, if some garbage objects can be accumulated in partitioning step, >>>>>> >> separated partitioning job may not be bad idea. Is there some special >>>>>> >> reason? >>>>>> >> >>>>>> >> On Wed, Mar 6, 2013 at 6:15 PM, Thomas Jungblut >>>>>> >> <[email protected]> wrote: >>>>>> >>> For graph processing, the partitioned files that result from the >>>>>> >>> partitioning job must be sorted. Currently only the partition files >>>>>> >>> in >>>>>> >>> itself are sorted, thus more tasks result in not sorted data in the >>>>>> >>> completed file. This only applies for the graph processing package. >>>>>> >>> So as Suraj told, it would be much more simpler to solve this via >>>>>> >>> messaging, once it is scalable (it will be very very scalable!). So >>>>>> >>> the >>>>>> >>> GraphJobRunner can be partitioning the stuff with a single superstep >>>>>> >>> in >>>>>> >>> setup() as it was before ages ago. The messaging must be sorted >>>>>> >>> anyway >>>>>> for >>>>>> >>> the algorithm so this is a nice side effect and saves us the >>>>>> partitioning >>>>>> >>> job for graph processing. >>>>>> >>> >>>>>> >>> For other partitionings and with regard to our superstep API, Suraj's >>>>>> idea >>>>>> >>> of injecting a preprocessing superstep that partitions the stuff into >>>>>> our >>>>>> >>> messaging system is actually the best. >>>>>> >>> >>>>>> >>> >>>>>> >>> 2013/3/6 Suraj Menon <[email protected]> >>>>>> >>> >>>>>> >>>> No, the partitions we write locally need not be sorted. Sorry for >>>>>> >>>> the >>>>>> >>>> confusion. The Superstep injection is possible with Superstep API. >>>>>> There >>>>>> >>>> are few enhancements needed to make it simpler after I last worked >>>>>> >>>> on >>>>>> it. >>>>>> >>>> We can then look into partitioning superstep being executed before >>>>>> >>>> the >>>>>> >>>> setup of first superstep of submitted job. I think it is feasible. >>>>>> >>>> >>>>>> >>>> On Tue, Mar 5, 2013 at 5:48 AM, Edward J. Yoon >>>>>> >>>> <[email protected] >>>>>> >>>> >wrote: >>>>>> >>>> >>>>>> >>>> > > spilling queue and sorted spilling queue, can we inject the >>>>>> >>>> partitioning >>>>>> >>>> > > superstep as the first superstep and use local memory? >>>>>> >>>> > >>>>>> >>>> > Actually, I wanted to add something before calling BSP.setup() >>>>>> method >>>>>> >>>> > to avoid execute additional BSP job. But, in my opinion, current >>>>>> >>>> > is >>>>>> >>>> > enough. I think, we need to collect more experiences of input >>>>>> >>>> > partitioning on large environments. I'll do. >>>>>> >>>> > >>>>>> >>>> > BTW, I still don't know why it need to be Sorted?! MR-like? >>>>>> >>>> > >>>>>> >>>> > On Thu, Feb 28, 2013 at 11:20 PM, Suraj Menon < >>>>>> [email protected]> >>>>>> >>>> > wrote: >>>>>> >>>> > > Sorry, I am increasing the scope here to outside graph module. >>>>>> When we >>>>>> >>>> > have >>>>>> >>>> > > spilling queue and sorted spilling queue, can we inject the >>>>>> >>>> partitioning >>>>>> >>>> > > superstep as the first superstep and use local memory? >>>>>> >>>> > > Today we have partitioning job within a job and are creating two >>>>>> copies >>>>>> >>>> > of >>>>>> >>>> > > data on HDFS. This could be really costly. Is it possible to >>>>>> create or >>>>>> >>>> > > redistribute the partitions on local memory and initialize the >>>>>> record >>>>>> >>>> > > reader there? >>>>>> >>>> > > The user can run a separate job give in examples area to >>>>>> explicitly >>>>>> >>>> > > repartition the data on HDFS. The deployment question is how >>>>>> >>>> > > much >>>>>> of >>>>>> >>>> disk >>>>>> >>>> > > space gets allocated for local memory usage? Would it be a safe >>>>>> >>>> approach >>>>>> >>>> > > with the limitations? >>>>>> >>>> > > >>>>>> >>>> > > -Suraj >>>>>> >>>> > > >>>>>> >>>> > > On Thu, Feb 28, 2013 at 7:05 AM, Thomas Jungblut >>>>>> >>>> > > <[email protected]>wrote: >>>>>> >>>> > > >>>>>> >>>> > >> yes. Once Suraj added merging of sorted files we can add this >>>>>> >>>> > >> to >>>>>> the >>>>>> >>>> > >> partitioner pretty easily. >>>>>> >>>> > >> >>>>>> >>>> > >> 2013/2/28 Edward J. Yoon <[email protected]> >>>>>> >>>> > >> >>>>>> >>>> > >> > Eh,..... btw, is re-partitioned data really necessary to be >>>>>> Sorted? >>>>>> >>>> > >> > >>>>>> >>>> > >> > On Thu, Feb 28, 2013 at 7:48 PM, Thomas Jungblut >>>>>> >>>> > >> > <[email protected]> wrote: >>>>>> >>>> > >> > > Now I get how the partitioning works, obviously if you >>>>>> >>>> > >> > > merge >>>>>> n >>>>>> >>>> > sorted >>>>>> >>>> > >> > files >>>>>> >>>> > >> > > by just appending to each other, this will result in >>>>>> >>>> > >> > > totally >>>>>> >>>> > unsorted >>>>>> >>>> > >> > data >>>>>> >>>> > >> > > ;-) >>>>>> >>>> > >> > > Why didn't you solve this via messaging? >>>>>> >>>> > >> > > >>>>>> >>>> > >> > > 2013/2/28 Thomas Jungblut <[email protected]> >>>>>> >>>> > >> > > >>>>>> >>>> > >> > >> Seems that they are not correctly sorted: >>>>>> >>>> > >> > >> >>>>>> >>>> > >> > >> vertexID: 50 >>>>>> >>>> > >> > >> vertexID: 52 >>>>>> >>>> > >> > >> vertexID: 54 >>>>>> >>>> > >> > >> vertexID: 56 >>>>>> >>>> > >> > >> vertexID: 58 >>>>>> >>>> > >> > >> vertexID: 61 >>>>>> >>>> > >> > >> ... >>>>>> >>>> > >> > >> vertexID: 78 >>>>>> >>>> > >> > >> vertexID: 81 >>>>>> >>>> > >> > >> vertexID: 83 >>>>>> >>>> > >> > >> vertexID: 85 >>>>>> >>>> > >> > >> ... >>>>>> >>>> > >> > >> vertexID: 94 >>>>>> >>>> > >> > >> vertexID: 96 >>>>>> >>>> > >> > >> vertexID: 98 >>>>>> >>>> > >> > >> vertexID: 1 >>>>>> >>>> > >> > >> vertexID: 10 >>>>>> >>>> > >> > >> vertexID: 12 >>>>>> >>>> > >> > >> vertexID: 14 >>>>>> >>>> > >> > >> vertexID: 16 >>>>>> >>>> > >> > >> vertexID: 18 >>>>>> >>>> > >> > >> vertexID: 21 >>>>>> >>>> > >> > >> vertexID: 23 >>>>>> >>>> > >> > >> vertexID: 25 >>>>>> >>>> > >> > >> vertexID: 27 >>>>>> >>>> > >> > >> vertexID: 29 >>>>>> >>>> > >> > >> vertexID: 3 >>>>>> >>>> > >> > >> >>>>>> >>>> > >> > >> So this won't work then correctly... >>>>>> >>>> > >> > >> >>>>>> >>>> > >> > >> >>>>>> >>>> > >> > >> 2013/2/28 Thomas Jungblut <[email protected]> >>>>>> >>>> > >> > >> >>>>>> >>>> > >> > >>> sure, have fun on your holidays. >>>>>> >>>> > >> > >>> >>>>>> >>>> > >> > >>> >>>>>> >>>> > >> > >>> 2013/2/28 Edward J. Yoon <[email protected]> >>>>>> >>>> > >> > >>> >>>>>> >>>> > >> > >>>> Sure, but if you can fix quickly, please do. March 1 is >>>>>> >>>> > holiday[1] >>>>>> >>>> > >> so >>>>>> >>>> > >> > >>>> I'll appear next week. >>>>>> >>>> > >> > >>>> >>>>>> >>>> > >> > >>>> 1. >>>>>> http://en.wikipedia.org/wiki/Public_holidays_in_South_Korea >>>>>> >>>> > >> > >>>> >>>>>> >>>> > >> > >>>> On Thu, Feb 28, 2013 at 6:36 PM, Thomas Jungblut >>>>>> >>>> > >> > >>>> <[email protected]> wrote: >>>>>> >>>> > >> > >>>> > Maybe 50 is missing from the file, didn't observe if >>>>>> >>>> > >> > >>>> > all >>>>>> >>>> items >>>>>> >>>> > >> were >>>>>> >>>> > >> > >>>> added. >>>>>> >>>> > >> > >>>> > As far as I remember, I copy/pasted the logic of the >>>>>> >>>> > >> > >>>> > ID >>>>>> into >>>>>> >>>> > the >>>>>> >>>> > >> > >>>> fastgen, >>>>>> >>>> > >> > >>>> > want to have a look into it? >>>>>> >>>> > >> > >>>> > >>>>>> >>>> > >> > >>>> > 2013/2/28 Edward J. Yoon <[email protected]> >>>>>> >>>> > >> > >>>> > >>>>>> >>>> > >> > >>>> >> I guess, it's a bug of fastgen, when generate >>>>>> >>>> > >> > >>>> >> adjacency >>>>>> >>>> matrix >>>>>> >>>> > >> into >>>>>> >>>> > >> > >>>> >> multiple files. >>>>>> >>>> > >> > >>>> >> >>>>>> >>>> > >> > >>>> >> On Thu, Feb 28, 2013 at 6:29 PM, Thomas Jungblut >>>>>> >>>> > >> > >>>> >> <[email protected]> wrote: >>>>>> >>>> > >> > >>>> >> > You have two files, are they partitioned correctly? >>>>>> >>>> > >> > >>>> >> > >>>>>> >>>> > >> > >>>> >> > 2013/2/28 Edward J. Yoon <[email protected]> >>>>>> >>>> > >> > >>>> >> > >>>>>> >>>> > >> > >>>> >> >> It looks like a bug. >>>>>> >>>> > >> > >>>> >> >> >>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls -al >>>>>> >>>> > >> /tmp/randomgraph/ >>>>>> >>>> > >> > >>>> >> >> total 44 >>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096 2월 28 18:03 . >>>>>> >>>> > >> > >>>> >> >> drwxrwxrwt 19 root root 20480 2월 28 18:04 .. >>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2243 2월 28 18:01 >>>>>> part-00000 >>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 28 2월 28 18:01 >>>>>> >>>> > .part-00000.crc >>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2251 2월 28 18:01 >>>>>> part-00001 >>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 28 2월 28 18:01 >>>>>> >>>> > .part-00001.crc >>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096 2월 28 18:03 >>>>>> partitions >>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls -al >>>>>> >>>> > >> > >>>> >> /tmp/randomgraph/partitions/ >>>>>> >>>> > >> > >>>> >> >> total 24 >>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096 2월 28 18:03 . >>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096 2월 28 18:03 .. >>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2932 2월 28 18:03 >>>>>> part-00000 >>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 32 2월 28 18:03 >>>>>> >>>> > .part-00000.crc >>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2955 2월 28 18:03 >>>>>> part-00001 >>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 32 2월 28 18:03 >>>>>> >>>> > .part-00001.crc >>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ >>>>>> >>>> > >> > >>>> >> >> >>>>>> >>>> > >> > >>>> >> >> >>>>>> >>>> > >> > >>>> >> >> On Thu, Feb 28, 2013 at 5:27 PM, Edward < >>>>>> >>>> [email protected] >>>>>> >>>> > > >>>>>> >>>> > >> > wrote: >>>>>> >>>> > >> > >>>> >> >> > yes i'll check again >>>>>> >>>> > >> > >>>> >> >> > >>>>>> >>>> > >> > >>>> >> >> > Sent from my iPhone >>>>>> >>>> > >> > >>>> >> >> > >>>>>> >>>> > >> > >>>> >> >> > On Feb 28, 2013, at 5:18 PM, Thomas Jungblut < >>>>>> >>>> > >> > >>>> >> [email protected]> >>>>>> >>>> > >> > >>>> >> >> wrote: >>>>>> >>>> > >> > >>>> >> >> > >>>>>> >>>> > >> > >>>> >> >> >> Can you verify an observation for me please? >>>>>> >>>> > >> > >>>> >> >> >> >>>>>> >>>> > >> > >>>> >> >> >> 2 files are created from fastgen, part-00000 >>>>>> >>>> > >> > >>>> >> >> >> and >>>>>> >>>> > >> part-00001, >>>>>> >>>> > >> > >>>> both >>>>>> >>>> > >> > >>>> >> ~2.2kb >>>>>> >>>> > >> > >>>> >> >> >> sized. >>>>>> >>>> > >> > >>>> >> >> >> In the below partition directory, there is >>>>>> >>>> > >> > >>>> >> >> >> only a >>>>>> >>>> single >>>>>> >>>> > >> > 5.56kb >>>>>> >>>> > >> > >>>> file. >>>>>> >>>> > >> > >>>> >> >> >> >>>>>> >>>> > >> > >>>> >> >> >> Is it intended for the partitioner to write a >>>>>> single >>>>>> >>>> > file >>>>>> >>>> > >> if >>>>>> >>>> > >> > you >>>>>> >>>> > >> > >>>> >> >> configured >>>>>> >>>> > >> > >>>> >> >> >> two? >>>>>> >>>> > >> > >>>> >> >> >> It even reads it as a two files, strange huh? >>>>>> >>>> > >> > >>>> >> >> >> >>>>>> >>>> > >> > >>>> >> >> >> 2013/2/28 Thomas Jungblut < >>>>>> [email protected]> >>>>>> >>>> > >> > >>>> >> >> >> >>>>>> >>>> > >> > >>>> >> >> >>> Will have a look into it. >>>>>> >>>> > >> > >>>> >> >> >>> >>>>>> >>>> > >> > >>>> >> >> >>> gen fastgen 100 10 /tmp/randomgraph 1 >>>>>> >>>> > >> > >>>> >> >> >>> pagerank /tmp/randomgraph /tmp/pageout >>>>>> >>>> > >> > >>>> >> >> >>> >>>>>> >>>> > >> > >>>> >> >> >>> did work for me the last time I profiled, >>>>>> >>>> > >> > >>>> >> >> >>> maybe >>>>>> the >>>>>> >>>> > >> > >>>> partitioning >>>>>> >>>> > >> > >>>> >> >> doesn't >>>>>> >>>> > >> > >>>> >> >> >>> partition correctly with the input or >>>>>> >>>> > >> > >>>> >> >> >>> something >>>>>> else. >>>>>> >>>> > >> > >>>> >> >> >>> >>>>>> >>>> > >> > >>>> >> >> >>> >>>>>> >>>> > >> > >>>> >> >> >>> 2013/2/28 Edward J. Yoon >>>>>> >>>> > >> > >>>> >> >> >>> <[email protected] >>>>>> > >>>>>> >>>> > >> > >>>> >> >> >>> >>>>>> >>>> > >> > >>>> >> >> >>> Fastgen input seems not work for graph >>>>>> >>>> > >> > >>>> >> >> >>> examples. >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >>>>>> :~/workspace/hama-trunk$ >>>>>> >>>> > >> bin/hama >>>>>> >>>> > >> > jar >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar gen >>>>>> >>>> > >> > fastgen >>>>>> >>>> > >> > >>>> 100 10 >>>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph 2 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:02 WARN util.NativeCodeLoader: >>>>>> Unable >>>>>> >>>> > to >>>>>> >>>> > >> > load >>>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your platform... >>>>>> using >>>>>> >>>> > >> > builtin-java >>>>>> >>>> > >> > >>>> >> classes >>>>>> >>>> > >> > >>>> >> >> >>>> where applicable >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO bsp.BSPJobClient: >>>>>> Running >>>>>> >>>> job: >>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO bsp.LocalBSPRunner: >>>>>> Setting >>>>>> >>>> up >>>>>> >>>> > a >>>>>> >>>> > >> new >>>>>> >>>> > >> > >>>> barrier >>>>>> >>>> > >> > >>>> >> >> for 2 >>>>>> >>>> > >> > >>>> >> >> >>>> tasks! >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>>> Current >>>>>> >>>> > >> supersteps >>>>>> >>>> > >> > >>>> >> number: 0 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: The >>>>>> total >>>>>> >>>> > number >>>>>> >>>> > >> > of >>>>>> >>>> > >> > >>>> >> >> supersteps: 0 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>>> Counters: 3 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.JobInProgress$JobCounter >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>>> >>>> > SUPERSTEPS=0 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>>> >>>> > >> > LAUNCHED_TASKS=2 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>>> >>>> > >> > >>>> >> TASK_OUTPUT_RECORDS=100 >>>>>> >>>> > >> > >>>> >> >> >>>> Job Finished in 3.212 seconds >>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >>>>>> :~/workspace/hama-trunk$ >>>>>> >>>> > >> bin/hama >>>>>> >>>> > >> > jar >>>>>> >>>> > >> > >>>> >> >> >>>> examples/target/hama-examples-0.7.0-SNAPSHOT >>>>>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT-javadoc.jar >>>>>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT.jar >>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >>>>>> :~/workspace/hama-trunk$ >>>>>> >>>> > >> bin/hama >>>>>> >>>> > >> > jar >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar >>>>>> >>>> > pagerank >>>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph /tmp/pageour >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 WARN util.NativeCodeLoader: >>>>>> Unable >>>>>> >>>> > to >>>>>> >>>> > >> > load >>>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your platform... >>>>>> using >>>>>> >>>> > >> > builtin-java >>>>>> >>>> > >> > >>>> >> classes >>>>>> >>>> > >> > >>>> >> >> >>>> where applicable >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: >>>>>> Total >>>>>> >>>> > input >>>>>> >>>> > >> > paths >>>>>> >>>> > >> > >>>> to >>>>>> >>>> > >> > >>>> >> >> process >>>>>> >>>> > >> > >>>> >> >> >>>> : 2 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: >>>>>> Total >>>>>> >>>> > input >>>>>> >>>> > >> > paths >>>>>> >>>> > >> > >>>> to >>>>>> >>>> > >> > >>>> >> >> process >>>>>> >>>> > >> > >>>> >> >> >>>> : 2 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO bsp.BSPJobClient: >>>>>> Running >>>>>> >>>> job: >>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO bsp.LocalBSPRunner: >>>>>> Setting >>>>>> >>>> up >>>>>> >>>> > a >>>>>> >>>> > >> new >>>>>> >>>> > >> > >>>> barrier >>>>>> >>>> > >> > >>>> >> >> for 2 >>>>>> >>>> > >> > >>>> >> >> >>>> tasks! >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>> Current >>>>>> >>>> > >> supersteps >>>>>> >>>> > >> > >>>> >> number: 1 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: The >>>>>> total >>>>>> >>>> > number >>>>>> >>>> > >> > of >>>>>> >>>> > >> > >>>> >> >> supersteps: 1 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>> Counters: 6 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.JobInProgress$JobCounter >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>> >>>> > SUPERSTEPS=1 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>> >>>> > >> > LAUNCHED_TASKS=2 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>> >>>> > >> > SUPERSTEP_SUM=4 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>> >>>> > >> > >>>> IO_BYTES_READ=4332 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>> >>>> > >> > >>>> TIME_IN_SYNC_MS=14 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>> >>>> > >> > >>>> TASK_INPUT_RECORDS=100 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.FileInputFormat: >>>>>> Total >>>>>> >>>> > input >>>>>> >>>> > >> > paths >>>>>> >>>> > >> > >>>> to >>>>>> >>>> > >> > >>>> >> >> process >>>>>> >>>> > >> > >>>> >> >> >>>> : 2 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>> Running >>>>>> >>>> job: >>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.LocalBSPRunner: >>>>>> Setting >>>>>> >>>> up >>>>>> >>>> > a >>>>>> >>>> > >> new >>>>>> >>>> > >> > >>>> barrier >>>>>> >>>> > >> > >>>> >> >> for 2 >>>>>> >>>> > >> > >>>> >> >> >>>> tasks! >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: >>>>>> >>>> > >> > >>>> >> >> >>>> 50 >>>>>> >>>> > vertices >>>>>> >>>> > >> > are >>>>>> >>>> > >> > >>>> loaded >>>>>> >>>> > >> > >>>> >> >> into >>>>>> >>>> > >> > >>>> >> >> >>>> local:1 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: >>>>>> >>>> > >> > >>>> >> >> >>>> 50 >>>>>> >>>> > vertices >>>>>> >>>> > >> > are >>>>>> >>>> > >> > >>>> loaded >>>>>> >>>> > >> > >>>> >> >> into >>>>>> >>>> > >> > >>>> >> >> >>>> local:0 >>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 ERROR bsp.LocalBSPRunner: >>>>>> >>>> Exception >>>>>> >>>> > >> > during >>>>>> >>>> > >> > >>>> BSP >>>>>> >>>> > >> > >>>> >> >> >>>> execution! >>>>>> >>>> > >> > >>>> >> >> >>>> java.lang.IllegalArgumentException: Messages >>>>>> must >>>>>> >>>> > never >>>>>> >>>> > >> be >>>>>> >>>> > >> > >>>> behind >>>>>> >>>> > >> > >>>> >> the >>>>>> >>>> > >> > >>>> >> >> >>>> vertex in ID! Current Message ID: 1 vs. 50 >>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> >>>> > >> > >>>> >> >>>>>> >>>> > >> > >>>>>> >>>> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279) >>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> >>>> > >> > >>>> >> >> >>>>>> >>>> > >> > >>>> >> >>>>>> >>>> > >> > >>>> >>>>>> >>>> > >> > >>>>>> >>>> > >>>>>> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225) >>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> >>>> > >> > >>>> >>>>>> >>>> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129) >>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> >>>> > >> > >>>> >> >> >>>>>> >>>> > >> > >>>> >> >>>>>> >>>> > >> > >>>> >>>>>> >>>> > >> > >>>>>> >>>> > >>>>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256) >>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> >>>> > >> > >>>> >> >> >>>>>> >>>> > >> > >>>> >> >>>>>> >>>> > >> > >>>> >>>>>> >>>> > >> > >>>>>> >>>> > >> >>>>>> >>>> > >>>>>> >>>> >>>>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> >>>> > >> > >>>> >> >> >>>>>> >>>> > >> > >>>> >> >>>>>> >>>> > >> > >>>> >>>>>> >>>> > >> > >>>>>> >>>> > >> >>>>>> >>>> > >>>>>> >>>> >>>>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> >>>> > >> > >>>> >>>>>> >>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>> >>>> > >> > >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> >>>> > >> > >>>> >> >> >>>>>> >>>> > >> > >>>> >>>>>> >>>> > >> > >>>>>> >>>> > >>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> >>>> > >> > >>>> >>>>>> >>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>> >>>> > >> > >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> >>>> > >> > >>>> >> >> >>>>>> >>>> > >> > >>>> >> >>>>>> >>>> > >> > >>>> >>>>>> >>>> > >> > >>>>>> >>>> > >> >>>>>> >>>> > >>>>>> >>>> >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> >>>> > >> > >>>> >> >> >>>>>> >>>> > >> > >>>> >> >>>>>> >>>> > >> > >>>> >>>>>> >>>> > >> > >>>>>> >>>> > >> >>>>>> >>>> > >>>>>> >>>> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>> >>>> > >> > >>>> >> >> >>>> java.lang.Thread.run(Thread.java:722) >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>> >>>> > >> > >>>> >> >> >>>> -- >>>>>> >>>> > >> > >>>> >> >> >>>> Best Regards, Edward J. Yoon >>>>>> >>>> > >> > >>>> >> >> >>>> @eddieyoon >>>>>> >>>> > >> > >>>> >> >> >>> >>>>>> >>>> > >> > >>>> >> >> >>> >>>>>> >>>> > >> > >>>> >> >> >>>>>> >>>> > >> > >>>> >> >> >>>>>> >>>> > >> > >>>> >> >> >>>>>> >>>> > >> > >>>> >> >> -- >>>>>> >>>> > >> > >>>> >> >> Best Regards, Edward J. Yoon >>>>>> >>>> > >> > >>>> >> >> @eddieyoon >>>>>> >>>> > >> > >>>> >> >> >>>>>> >>>> > >> > >>>> >> >>>>>> >>>> > >> > >>>> >> >>>>>> >>>> > >> > >>>> >> >>>>>> >>>> > >> > >>>> >> -- >>>>>> >>>> > >> > >>>> >> Best Regards, Edward J. Yoon >>>>>> >>>> > >> > >>>> >> @eddieyoon >>>>>> >>>> > >> > >>>> >> >>>>>> >>>> > >> > >>>> >>>>>> >>>> > >> > >>>> >>>>>> >>>> > >> > >>>> >>>>>> >>>> > >> > >>>> -- >>>>>> >>>> > >> > >>>> Best Regards, Edward J. Yoon >>>>>> >>>> > >> > >>>> @eddieyoon >>>>>> >>>> > >> > >>>> >>>>>> >>>> > >> > >>> >>>>>> >>>> > >> > >>> >>>>>> >>>> > >> > >> >>>>>> >>>> > >> > >>>>>> >>>> > >> > >>>>>> >>>> > >> > >>>>>> >>>> > >> > -- >>>>>> >>>> > >> > Best Regards, Edward J. Yoon >>>>>> >>>> > >> > @eddieyoon >>>>>> >>>> > >> > >>>>>> >>>> > >> >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > -- >>>>>> >>>> > Best Regards, Edward J. Yoon >>>>>> >>>> > @eddieyoon >>>>>> >>>> > >>>>>> >>>> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> -- >>>>>> >> Best Regards, Edward J. Yoon >>>>>> >> @eddieyoon >>>>>> > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > Best Regards, Edward J. Yoon >>>>>> > @eddieyoon >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best Regards, Edward J. Yoon >>>>>> @eddieyoon >>>>>> >>>> >>>> >>>> >>>> -- >>>> Best Regards, Edward J. Yoon >>>> @eddieyoon >>> >>> >>> >>> -- >>> Best Regards, Edward J. Yoon >>> @eddieyoon >> >> >> >> -- >> Best Regards, Edward J. Yoon >> @eddieyoon > > > > -- > Best Regards, Edward J. Yoon > @eddieyoon
-- Best Regards, Edward J. Yoon @eddieyoon
