Personally, I would like to solve this issue by touching DiskVerticesInfo. If we write sorted sub-sets of vertices into multiple files, we can avoid huge memory consumption.
If we want to sort partitioned data using messaging system, idea should be collected. On Thu, Mar 14, 2013 at 10:31 AM, Edward J. Yoon <[email protected]> wrote: > Oh, now I get how iterate() works. HAMA-704 is nicely written. > > On Thu, Mar 14, 2013 at 12:02 AM, Edward J. Yoon <[email protected]> > wrote: >> I'm reading changes of HAMA-704 again. As a result of adding >> DiskVerticesInfo, vertices list is needed to be sorted. I'm not sure >> but I think this approach will bring more disadvantages than >> advantages. >> >> On Wed, Mar 13, 2013 at 11:09 PM, Edward J. Yoon <[email protected]> >> wrote: >>>>>> in loadVertices? Maybe consider feature for coupling storage in user >>>>>> space >>>>>> with BSP Messaging[HAMA-734] can avoid double reads and writes. This way >>>>>> partitioned or non-partitioned by partitioner, can keep vertices sorted >>>>>> with a single read and single write on every peer. >>> >>> And, as I commented JIRA ticket, I think we can't use messaging system >>> for sorting vertices within partition files. >>> >>> On Wed, Mar 13, 2013 at 11:00 PM, Edward J. Yoon <[email protected]> >>> wrote: >>>> P.S., (number of splits = number of partitions) is really confuse to >>>> me. Even though blocks number is equal to desired tasks number, data >>>> should be re-partitioned again. >>>> >>>> On Wed, Mar 13, 2013 at 10:36 PM, Edward J. Yoon <[email protected]> >>>> wrote: >>>>> Indeed. If there are already partitioned input files (unsorted) and so >>>>> user want to skip pre-partitioning phase, it should be handled in >>>>> GraphJobRunner BSP program. Actually, I still don't know why >>>>> re-partitioned files need to be Sorted. It's only about >>>>> GraphJobRunner. >>>>> >>>>>> partitioning. (This is outside the scope of graphs. We can have a >>>>>> dedicated >>>>>> partitioning superstep for graph applications). >>>>> >>>>> Sorry. I don't understand exactly yet. Do you mean just a partitioning >>>>> job based on superstep API? >>>>> >>>>> By default, 100 tasks will be assigned for partitioning job. >>>>> Partitioning job will create 1,000 partitions. Thus, we can execute >>>>> the Graph job with 1,000 tasks. >>>>> >>>>> Let's assume that a input sequence file is 20GB (100 blocks). If I >>>>> want to run with 1,000 tasks, what happens? >>>>> >>>>> On Wed, Mar 13, 2013 at 9:49 PM, Suraj Menon <[email protected]> >>>>> wrote: >>>>>> I am responding on this thread because of better continuity for >>>>>> conversation. We cannot expect the partitions to be sorted every time. >>>>>> When >>>>>> the number of splits = number of partitions and partitioning is switched >>>>>> off by user[HAMA-561], the partitions would not be sorted. Can we do this >>>>>> in loadVertices? Maybe consider feature for coupling storage in user >>>>>> space >>>>>> with BSP Messaging[HAMA-734] can avoid double reads and writes. This way >>>>>> partitioned or non-partitioned by partitioner, can keep vertices sorted >>>>>> with a single read and single write on every peer. >>>>>> >>>>>> Just clearing confusion if any regarding superstep injection for >>>>>> partitioning. (This is outside the scope of graphs. We can have a >>>>>> dedicated >>>>>> partitioning superstep for graph applications). >>>>>> Say there are x splits and y number of tasks configured by user. >>>>>> >>>>>> if x > y >>>>>> The y tasks are scheduled with x of them having each of the x splits and >>>>>> the remaining with no resource local to them. Then the partitioning >>>>>> superstep redistributes the partitions among them to create local >>>>>> partitions. Now the question is can we re-initialize a peer's input based >>>>>> on this new local part of partition? >>>>>> >>>>>> if y > x >>>>>> works as it works today. >>>>>> >>>>>> Just putting my points in brainstorming. >>>>>> >>>>>> -Suraj >>>>>> >>>>>> >>>>>> On Mon, Mar 11, 2013 at 7:39 AM, Edward J. Yoon >>>>>> <[email protected]>wrote: >>>>>> >>>>>>> I just filed here https://issues.apache.org/jira/browse/HAMA-744 >>>>>>> >>>>>>> On Mon, Mar 11, 2013 at 7:35 PM, Edward J. Yoon <[email protected]> >>>>>>> wrote: >>>>>>> > Additionally, >>>>>>> > >>>>>>> >> spilling queue and sorted spilling queue, can we inject the >>>>>>> >> partitioning >>>>>>> >> superstep as the first superstep and use local memory? >>>>>>> > >>>>>>> > Can we execute different number of tasks per superstep? >>>>>>> > >>>>>>> > On Mon, Mar 11, 2013 at 6:56 PM, Edward J. Yoon >>>>>>> > <[email protected]> >>>>>>> wrote: >>>>>>> >>> For graph processing, the partitioned files that result from the >>>>>>> >>> partitioning job must be sorted. Currently only the partition files >>>>>>> >>> in >>>>>>> >> >>>>>>> >> I see. >>>>>>> >> >>>>>>> >>> For other partitionings and with regard to our superstep API, >>>>>>> >>> Suraj's >>>>>>> idea >>>>>>> >>> of injecting a preprocessing superstep that partitions the stuff >>>>>>> >>> into >>>>>>> our >>>>>>> >>> messaging system is actually the best. >>>>>>> >> >>>>>>> >> BTW, if some garbage objects can be accumulated in partitioning step, >>>>>>> >> separated partitioning job may not be bad idea. Is there some special >>>>>>> >> reason? >>>>>>> >> >>>>>>> >> On Wed, Mar 6, 2013 at 6:15 PM, Thomas Jungblut >>>>>>> >> <[email protected]> wrote: >>>>>>> >>> For graph processing, the partitioned files that result from the >>>>>>> >>> partitioning job must be sorted. Currently only the partition files >>>>>>> >>> in >>>>>>> >>> itself are sorted, thus more tasks result in not sorted data in the >>>>>>> >>> completed file. This only applies for the graph processing package. >>>>>>> >>> So as Suraj told, it would be much more simpler to solve this via >>>>>>> >>> messaging, once it is scalable (it will be very very scalable!). So >>>>>>> >>> the >>>>>>> >>> GraphJobRunner can be partitioning the stuff with a single >>>>>>> >>> superstep in >>>>>>> >>> setup() as it was before ages ago. The messaging must be sorted >>>>>>> >>> anyway >>>>>>> for >>>>>>> >>> the algorithm so this is a nice side effect and saves us the >>>>>>> partitioning >>>>>>> >>> job for graph processing. >>>>>>> >>> >>>>>>> >>> For other partitionings and with regard to our superstep API, >>>>>>> >>> Suraj's >>>>>>> idea >>>>>>> >>> of injecting a preprocessing superstep that partitions the stuff >>>>>>> >>> into >>>>>>> our >>>>>>> >>> messaging system is actually the best. >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> 2013/3/6 Suraj Menon <[email protected]> >>>>>>> >>> >>>>>>> >>>> No, the partitions we write locally need not be sorted. Sorry for >>>>>>> >>>> the >>>>>>> >>>> confusion. The Superstep injection is possible with Superstep API. >>>>>>> There >>>>>>> >>>> are few enhancements needed to make it simpler after I last worked >>>>>>> >>>> on >>>>>>> it. >>>>>>> >>>> We can then look into partitioning superstep being executed before >>>>>>> >>>> the >>>>>>> >>>> setup of first superstep of submitted job. I think it is feasible. >>>>>>> >>>> >>>>>>> >>>> On Tue, Mar 5, 2013 at 5:48 AM, Edward J. Yoon >>>>>>> >>>> <[email protected] >>>>>>> >>>> >wrote: >>>>>>> >>>> >>>>>>> >>>> > > spilling queue and sorted spilling queue, can we inject the >>>>>>> >>>> partitioning >>>>>>> >>>> > > superstep as the first superstep and use local memory? >>>>>>> >>>> > >>>>>>> >>>> > Actually, I wanted to add something before calling BSP.setup() >>>>>>> method >>>>>>> >>>> > to avoid execute additional BSP job. But, in my opinion, current >>>>>>> >>>> > is >>>>>>> >>>> > enough. I think, we need to collect more experiences of input >>>>>>> >>>> > partitioning on large environments. I'll do. >>>>>>> >>>> > >>>>>>> >>>> > BTW, I still don't know why it need to be Sorted?! MR-like? >>>>>>> >>>> > >>>>>>> >>>> > On Thu, Feb 28, 2013 at 11:20 PM, Suraj Menon < >>>>>>> [email protected]> >>>>>>> >>>> > wrote: >>>>>>> >>>> > > Sorry, I am increasing the scope here to outside graph module. >>>>>>> When we >>>>>>> >>>> > have >>>>>>> >>>> > > spilling queue and sorted spilling queue, can we inject the >>>>>>> >>>> partitioning >>>>>>> >>>> > > superstep as the first superstep and use local memory? >>>>>>> >>>> > > Today we have partitioning job within a job and are creating >>>>>>> >>>> > > two >>>>>>> copies >>>>>>> >>>> > of >>>>>>> >>>> > > data on HDFS. This could be really costly. Is it possible to >>>>>>> create or >>>>>>> >>>> > > redistribute the partitions on local memory and initialize the >>>>>>> record >>>>>>> >>>> > > reader there? >>>>>>> >>>> > > The user can run a separate job give in examples area to >>>>>>> explicitly >>>>>>> >>>> > > repartition the data on HDFS. The deployment question is how >>>>>>> >>>> > > much >>>>>>> of >>>>>>> >>>> disk >>>>>>> >>>> > > space gets allocated for local memory usage? Would it be a safe >>>>>>> >>>> approach >>>>>>> >>>> > > with the limitations? >>>>>>> >>>> > > >>>>>>> >>>> > > -Suraj >>>>>>> >>>> > > >>>>>>> >>>> > > On Thu, Feb 28, 2013 at 7:05 AM, Thomas Jungblut >>>>>>> >>>> > > <[email protected]>wrote: >>>>>>> >>>> > > >>>>>>> >>>> > >> yes. Once Suraj added merging of sorted files we can add this >>>>>>> >>>> > >> to >>>>>>> the >>>>>>> >>>> > >> partitioner pretty easily. >>>>>>> >>>> > >> >>>>>>> >>>> > >> 2013/2/28 Edward J. Yoon <[email protected]> >>>>>>> >>>> > >> >>>>>>> >>>> > >> > Eh,..... btw, is re-partitioned data really necessary to be >>>>>>> Sorted? >>>>>>> >>>> > >> > >>>>>>> >>>> > >> > On Thu, Feb 28, 2013 at 7:48 PM, Thomas Jungblut >>>>>>> >>>> > >> > <[email protected]> wrote: >>>>>>> >>>> > >> > > Now I get how the partitioning works, obviously if you >>>>>>> >>>> > >> > > merge >>>>>>> n >>>>>>> >>>> > sorted >>>>>>> >>>> > >> > files >>>>>>> >>>> > >> > > by just appending to each other, this will result in >>>>>>> >>>> > >> > > totally >>>>>>> >>>> > unsorted >>>>>>> >>>> > >> > data >>>>>>> >>>> > >> > > ;-) >>>>>>> >>>> > >> > > Why didn't you solve this via messaging? >>>>>>> >>>> > >> > > >>>>>>> >>>> > >> > > 2013/2/28 Thomas Jungblut <[email protected]> >>>>>>> >>>> > >> > > >>>>>>> >>>> > >> > >> Seems that they are not correctly sorted: >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> > >> vertexID: 50 >>>>>>> >>>> > >> > >> vertexID: 52 >>>>>>> >>>> > >> > >> vertexID: 54 >>>>>>> >>>> > >> > >> vertexID: 56 >>>>>>> >>>> > >> > >> vertexID: 58 >>>>>>> >>>> > >> > >> vertexID: 61 >>>>>>> >>>> > >> > >> ... >>>>>>> >>>> > >> > >> vertexID: 78 >>>>>>> >>>> > >> > >> vertexID: 81 >>>>>>> >>>> > >> > >> vertexID: 83 >>>>>>> >>>> > >> > >> vertexID: 85 >>>>>>> >>>> > >> > >> ... >>>>>>> >>>> > >> > >> vertexID: 94 >>>>>>> >>>> > >> > >> vertexID: 96 >>>>>>> >>>> > >> > >> vertexID: 98 >>>>>>> >>>> > >> > >> vertexID: 1 >>>>>>> >>>> > >> > >> vertexID: 10 >>>>>>> >>>> > >> > >> vertexID: 12 >>>>>>> >>>> > >> > >> vertexID: 14 >>>>>>> >>>> > >> > >> vertexID: 16 >>>>>>> >>>> > >> > >> vertexID: 18 >>>>>>> >>>> > >> > >> vertexID: 21 >>>>>>> >>>> > >> > >> vertexID: 23 >>>>>>> >>>> > >> > >> vertexID: 25 >>>>>>> >>>> > >> > >> vertexID: 27 >>>>>>> >>>> > >> > >> vertexID: 29 >>>>>>> >>>> > >> > >> vertexID: 3 >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> > >> So this won't work then correctly... >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> > >> 2013/2/28 Thomas Jungblut <[email protected]> >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> > >>> sure, have fun on your holidays. >>>>>>> >>>> > >> > >>> >>>>>>> >>>> > >> > >>> >>>>>>> >>>> > >> > >>> 2013/2/28 Edward J. Yoon <[email protected]> >>>>>>> >>>> > >> > >>> >>>>>>> >>>> > >> > >>>> Sure, but if you can fix quickly, please do. March 1 is >>>>>>> >>>> > holiday[1] >>>>>>> >>>> > >> so >>>>>>> >>>> > >> > >>>> I'll appear next week. >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> > >> > >>>> 1. >>>>>>> http://en.wikipedia.org/wiki/Public_holidays_in_South_Korea >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> > >> > >>>> On Thu, Feb 28, 2013 at 6:36 PM, Thomas Jungblut >>>>>>> >>>> > >> > >>>> <[email protected]> wrote: >>>>>>> >>>> > >> > >>>> > Maybe 50 is missing from the file, didn't observe if >>>>>>> >>>> > >> > >>>> > all >>>>>>> >>>> items >>>>>>> >>>> > >> were >>>>>>> >>>> > >> > >>>> added. >>>>>>> >>>> > >> > >>>> > As far as I remember, I copy/pasted the logic of the >>>>>>> >>>> > >> > >>>> > ID >>>>>>> into >>>>>>> >>>> > the >>>>>>> >>>> > >> > >>>> fastgen, >>>>>>> >>>> > >> > >>>> > want to have a look into it? >>>>>>> >>>> > >> > >>>> > >>>>>>> >>>> > >> > >>>> > 2013/2/28 Edward J. Yoon <[email protected]> >>>>>>> >>>> > >> > >>>> > >>>>>>> >>>> > >> > >>>> >> I guess, it's a bug of fastgen, when generate >>>>>>> >>>> > >> > >>>> >> adjacency >>>>>>> >>>> matrix >>>>>>> >>>> > >> into >>>>>>> >>>> > >> > >>>> >> multiple files. >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> >> On Thu, Feb 28, 2013 at 6:29 PM, Thomas Jungblut >>>>>>> >>>> > >> > >>>> >> <[email protected]> wrote: >>>>>>> >>>> > >> > >>>> >> > You have two files, are they partitioned >>>>>>> >>>> > >> > >>>> >> > correctly? >>>>>>> >>>> > >> > >>>> >> > >>>>>>> >>>> > >> > >>>> >> > 2013/2/28 Edward J. Yoon <[email protected]> >>>>>>> >>>> > >> > >>>> >> > >>>>>>> >>>> > >> > >>>> >> >> It looks like a bug. >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls -al >>>>>>> >>>> > >> /tmp/randomgraph/ >>>>>>> >>>> > >> > >>>> >> >> total 44 >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096 2월 28 18:03 . >>>>>>> >>>> > >> > >>>> >> >> drwxrwxrwt 19 root root 20480 2월 28 18:04 .. >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2243 2월 28 18:01 >>>>>>> part-00000 >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 28 2월 28 18:01 >>>>>>> >>>> > .part-00000.crc >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2251 2월 28 18:01 >>>>>>> part-00001 >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 28 2월 28 18:01 >>>>>>> >>>> > .part-00001.crc >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096 2월 28 18:03 >>>>>>> partitions >>>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls -al >>>>>>> >>>> > >> > >>>> >> /tmp/randomgraph/partitions/ >>>>>>> >>>> > >> > >>>> >> >> total 24 >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096 2월 28 18:03 . >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096 2월 28 18:03 .. >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2932 2월 28 18:03 >>>>>>> part-00000 >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 32 2월 28 18:03 >>>>>>> >>>> > .part-00000.crc >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2955 2월 28 18:03 >>>>>>> part-00001 >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 32 2월 28 18:03 >>>>>>> >>>> > .part-00001.crc >>>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >> On Thu, Feb 28, 2013 at 5:27 PM, Edward < >>>>>>> >>>> [email protected] >>>>>>> >>>> > > >>>>>>> >>>> > >> > wrote: >>>>>>> >>>> > >> > >>>> >> >> > yes i'll check again >>>>>>> >>>> > >> > >>>> >> >> > >>>>>>> >>>> > >> > >>>> >> >> > Sent from my iPhone >>>>>>> >>>> > >> > >>>> >> >> > >>>>>>> >>>> > >> > >>>> >> >> > On Feb 28, 2013, at 5:18 PM, Thomas Jungblut < >>>>>>> >>>> > >> > >>>> >> [email protected]> >>>>>>> >>>> > >> > >>>> >> >> wrote: >>>>>>> >>>> > >> > >>>> >> >> > >>>>>>> >>>> > >> > >>>> >> >> >> Can you verify an observation for me please? >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >> 2 files are created from fastgen, part-00000 >>>>>>> >>>> > >> > >>>> >> >> >> and >>>>>>> >>>> > >> part-00001, >>>>>>> >>>> > >> > >>>> both >>>>>>> >>>> > >> > >>>> >> ~2.2kb >>>>>>> >>>> > >> > >>>> >> >> >> sized. >>>>>>> >>>> > >> > >>>> >> >> >> In the below partition directory, there is >>>>>>> >>>> > >> > >>>> >> >> >> only a >>>>>>> >>>> single >>>>>>> >>>> > >> > 5.56kb >>>>>>> >>>> > >> > >>>> file. >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >> Is it intended for the partitioner to write a >>>>>>> single >>>>>>> >>>> > file >>>>>>> >>>> > >> if >>>>>>> >>>> > >> > you >>>>>>> >>>> > >> > >>>> >> >> configured >>>>>>> >>>> > >> > >>>> >> >> >> two? >>>>>>> >>>> > >> > >>>> >> >> >> It even reads it as a two files, strange huh? >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >> 2013/2/28 Thomas Jungblut < >>>>>>> [email protected]> >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >>> Will have a look into it. >>>>>>> >>>> > >> > >>>> >> >> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> gen fastgen 100 10 /tmp/randomgraph 1 >>>>>>> >>>> > >> > >>>> >> >> >>> pagerank /tmp/randomgraph /tmp/pageout >>>>>>> >>>> > >> > >>>> >> >> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> did work for me the last time I profiled, >>>>>>> >>>> > >> > >>>> >> >> >>> maybe >>>>>>> the >>>>>>> >>>> > >> > >>>> partitioning >>>>>>> >>>> > >> > >>>> >> >> doesn't >>>>>>> >>>> > >> > >>>> >> >> >>> partition correctly with the input or >>>>>>> >>>> > >> > >>>> >> >> >>> something >>>>>>> else. >>>>>>> >>>> > >> > >>>> >> >> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> 2013/2/28 Edward J. Yoon >>>>>>> >>>> > >> > >>>> >> >> >>> <[email protected] >>>>>>> > >>>>>>> >>>> > >> > >>>> >> >> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> Fastgen input seems not work for graph >>>>>>> >>>> > >> > >>>> >> >> >>> examples. >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >>>>>>> :~/workspace/hama-trunk$ >>>>>>> >>>> > >> bin/hama >>>>>>> >>>> > >> > jar >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar gen >>>>>>> >>>> > >> > fastgen >>>>>>> >>>> > >> > >>>> 100 10 >>>>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph 2 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:02 WARN >>>>>>> >>>> > >> > >>>> >> >> >>>> util.NativeCodeLoader: >>>>>>> Unable >>>>>>> >>>> > to >>>>>>> >>>> > >> > load >>>>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your platform... >>>>>>> using >>>>>>> >>>> > >> > builtin-java >>>>>>> >>>> > >> > >>>> >> classes >>>>>>> >>>> > >> > >>>> >> >> >>>> where applicable >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO bsp.BSPJobClient: >>>>>>> Running >>>>>>> >>>> job: >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO bsp.LocalBSPRunner: >>>>>>> Setting >>>>>>> >>>> up >>>>>>> >>>> > a >>>>>>> >>>> > >> new >>>>>>> >>>> > >> > >>>> barrier >>>>>>> >>>> > >> > >>>> >> >> for 2 >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks! >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>>>> Current >>>>>>> >>>> > >> supersteps >>>>>>> >>>> > >> > >>>> >> number: 0 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: The >>>>>>> total >>>>>>> >>>> > number >>>>>>> >>>> > >> > of >>>>>>> >>>> > >> > >>>> >> >> supersteps: 0 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>>>> Counters: 3 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.JobInProgress$JobCounter >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>>>> >>>> > SUPERSTEPS=0 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>>>> >>>> > >> > LAUNCHED_TASKS=2 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>>>> >>>> > >> > >>>> >> TASK_OUTPUT_RECORDS=100 >>>>>>> >>>> > >> > >>>> >> >> >>>> Job Finished in 3.212 seconds >>>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >>>>>>> :~/workspace/hama-trunk$ >>>>>>> >>>> > >> bin/hama >>>>>>> >>>> > >> > jar >>>>>>> >>>> > >> > >>>> >> >> >>>> examples/target/hama-examples-0.7.0-SNAPSHOT >>>>>>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT-javadoc.jar >>>>>>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT.jar >>>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >>>>>>> :~/workspace/hama-trunk$ >>>>>>> >>>> > >> bin/hama >>>>>>> >>>> > >> > jar >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar >>>>>>> >>>> > pagerank >>>>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph /tmp/pageour >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 WARN >>>>>>> >>>> > >> > >>>> >> >> >>>> util.NativeCodeLoader: >>>>>>> Unable >>>>>>> >>>> > to >>>>>>> >>>> > >> > load >>>>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your platform... >>>>>>> using >>>>>>> >>>> > >> > builtin-java >>>>>>> >>>> > >> > >>>> >> classes >>>>>>> >>>> > >> > >>>> >> >> >>>> where applicable >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: >>>>>>> Total >>>>>>> >>>> > input >>>>>>> >>>> > >> > paths >>>>>>> >>>> > >> > >>>> to >>>>>>> >>>> > >> > >>>> >> >> process >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: >>>>>>> Total >>>>>>> >>>> > input >>>>>>> >>>> > >> > paths >>>>>>> >>>> > >> > >>>> to >>>>>>> >>>> > >> > >>>> >> >> process >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO bsp.BSPJobClient: >>>>>>> Running >>>>>>> >>>> job: >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO bsp.LocalBSPRunner: >>>>>>> Setting >>>>>>> >>>> up >>>>>>> >>>> > a >>>>>>> >>>> > >> new >>>>>>> >>>> > >> > >>>> barrier >>>>>>> >>>> > >> > >>>> >> >> for 2 >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks! >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>>> Current >>>>>>> >>>> > >> supersteps >>>>>>> >>>> > >> > >>>> >> number: 1 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: The >>>>>>> total >>>>>>> >>>> > number >>>>>>> >>>> > >> > of >>>>>>> >>>> > >> > >>>> >> >> supersteps: 1 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>>> Counters: 6 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.JobInProgress$JobCounter >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>>> >>>> > SUPERSTEPS=1 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>>> >>>> > >> > LAUNCHED_TASKS=2 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>>> >>>> > >> > SUPERSTEP_SUM=4 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>>> >>>> > >> > >>>> IO_BYTES_READ=4332 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>>> >>>> > >> > >>>> TIME_IN_SYNC_MS=14 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>>> >>>> > >> > >>>> TASK_INPUT_RECORDS=100 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.FileInputFormat: >>>>>>> Total >>>>>>> >>>> > input >>>>>>> >>>> > >> > paths >>>>>>> >>>> > >> > >>>> to >>>>>>> >>>> > >> > >>>> >> >> process >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>>>> Running >>>>>>> >>>> job: >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.LocalBSPRunner: >>>>>>> Setting >>>>>>> >>>> up >>>>>>> >>>> > a >>>>>>> >>>> > >> new >>>>>>> >>>> > >> > >>>> barrier >>>>>>> >>>> > >> > >>>> >> >> for 2 >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks! >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >>>>>>> >>>> > >> > >>>> >> >> >>>> graph.GraphJobRunner: 50 >>>>>>> >>>> > vertices >>>>>>> >>>> > >> > are >>>>>>> >>>> > >> > >>>> loaded >>>>>>> >>>> > >> > >>>> >> >> into >>>>>>> >>>> > >> > >>>> >> >> >>>> local:1 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >>>>>>> >>>> > >> > >>>> >> >> >>>> graph.GraphJobRunner: 50 >>>>>>> >>>> > vertices >>>>>>> >>>> > >> > are >>>>>>> >>>> > >> > >>>> loaded >>>>>>> >>>> > >> > >>>> >> >> into >>>>>>> >>>> > >> > >>>> >> >> >>>> local:0 >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 ERROR bsp.LocalBSPRunner: >>>>>>> >>>> Exception >>>>>>> >>>> > >> > during >>>>>>> >>>> > >> > >>>> BSP >>>>>>> >>>> > >> > >>>> >> >> >>>> execution! >>>>>>> >>>> > >> > >>>> >> >> >>>> java.lang.IllegalArgumentException: Messages >>>>>>> must >>>>>>> >>>> > never >>>>>>> >>>> > >> be >>>>>>> >>>> > >> > >>>> behind >>>>>>> >>>> > >> > >>>> >> the >>>>>>> >>>> > >> > >>>> >> >> >>>> vertex in ID! Current Message ID: 1 vs. 50 >>>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>>>>> >>>> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279) >>>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> > >> > >>>>>>> >>>> > >>>>>>> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225) >>>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129) >>>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> > >> > >>>>>>> >>>> > >>>>>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256) >>>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> > >> > >>>>>>> >>>> > >> >>>>>>> >>>> > >>>>>>> >>>> >>>>>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >>>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> > >> > >>>>>>> >>>> > >> >>>>>>> >>>> > >>>>>>> >>>> >>>>>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >>>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>>> >>>> > >> > >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> > >> > >>>>>>> >>>> > >>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>>> >>>> > >> > >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> > >> > >>>>>>> >>>> > >> >>>>>>> >>>> > >>>>>>> >>>> >>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> > >> > >>>>>>> >>>> > >> >>>>>>> >>>> > >>>>>>> >>>> >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>>>>>> >>>> > >> > >>>> >> >> >>>> at >>>>>>> >>>> > >> > >>>> >> >> >>>> java.lang.Thread.run(Thread.java:722) >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> -- >>>>>>> >>>> > >> > >>>> >> >> >>>> Best Regards, Edward J. Yoon >>>>>>> >>>> > >> > >>>> >> >> >>>> @eddieyoon >>>>>>> >>>> > >> > >>>> >> >> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >> -- >>>>>>> >>>> > >> > >>>> >> >> Best Regards, Edward J. Yoon >>>>>>> >>>> > >> > >>>> >> >> @eddieyoon >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> >> -- >>>>>>> >>>> > >> > >>>> >> Best Regards, Edward J. Yoon >>>>>>> >>>> > >> > >>>> >> @eddieyoon >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> > >> > >>>> -- >>>>>>> >>>> > >> > >>>> Best Regards, Edward J. Yoon >>>>>>> >>>> > >> > >>>> @eddieyoon >>>>>>> >>>> > >> > >>>> >>>>>>> >>>> > >> > >>> >>>>>>> >>>> > >> > >>> >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> > >>>>>>> >>>> > >> > >>>>>>> >>>> > >> > >>>>>>> >>>> > >> > -- >>>>>>> >>>> > >> > Best Regards, Edward J. Yoon >>>>>>> >>>> > >> > @eddieyoon >>>>>>> >>>> > >> > >>>>>>> >>>> > >> >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > -- >>>>>>> >>>> > Best Regards, Edward J. Yoon >>>>>>> >>>> > @eddieyoon >>>>>>> >>>> > >>>>>>> >>>> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> -- >>>>>>> >> Best Regards, Edward J. Yoon >>>>>>> >> @eddieyoon >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > -- >>>>>>> > Best Regards, Edward J. Yoon >>>>>>> > @eddieyoon >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best Regards, Edward J. Yoon >>>>>>> @eddieyoon >>>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Best Regards, Edward J. Yoon >>>>> @eddieyoon >>>> >>>> >>>> >>>> -- >>>> Best Regards, Edward J. Yoon >>>> @eddieyoon >>> >>> >>> >>> -- >>> Best Regards, Edward J. Yoon >>> @eddieyoon >> >> >> >> -- >> Best Regards, Edward J. Yoon >> @eddieyoon > > > > -- > Best Regards, Edward J. Yoon > @eddieyoon -- Best Regards, Edward J. Yoon @eddieyoon
