>>> in loadVertices? Maybe consider feature for coupling storage in user space >>> with BSP Messaging[HAMA-734] can avoid double reads and writes. This way >>> partitioned or non-partitioned by partitioner, can keep vertices sorted >>> with a single read and single write on every peer.
And, as I commented JIRA ticket, I think we can't use messaging system for sorting vertices within partition files. On Wed, Mar 13, 2013 at 11:00 PM, Edward J. Yoon <[email protected]> wrote: > P.S., (number of splits = number of partitions) is really confuse to > me. Even though blocks number is equal to desired tasks number, data > should be re-partitioned again. > > On Wed, Mar 13, 2013 at 10:36 PM, Edward J. Yoon <[email protected]> > wrote: >> Indeed. If there are already partitioned input files (unsorted) and so >> user want to skip pre-partitioning phase, it should be handled in >> GraphJobRunner BSP program. Actually, I still don't know why >> re-partitioned files need to be Sorted. It's only about >> GraphJobRunner. >> >>> partitioning. (This is outside the scope of graphs. We can have a dedicated >>> partitioning superstep for graph applications). >> >> Sorry. I don't understand exactly yet. Do you mean just a partitioning >> job based on superstep API? >> >> By default, 100 tasks will be assigned for partitioning job. >> Partitioning job will create 1,000 partitions. Thus, we can execute >> the Graph job with 1,000 tasks. >> >> Let's assume that a input sequence file is 20GB (100 blocks). If I >> want to run with 1,000 tasks, what happens? >> >> On Wed, Mar 13, 2013 at 9:49 PM, Suraj Menon <[email protected]> wrote: >>> I am responding on this thread because of better continuity for >>> conversation. We cannot expect the partitions to be sorted every time. When >>> the number of splits = number of partitions and partitioning is switched >>> off by user[HAMA-561], the partitions would not be sorted. Can we do this >>> in loadVertices? Maybe consider feature for coupling storage in user space >>> with BSP Messaging[HAMA-734] can avoid double reads and writes. This way >>> partitioned or non-partitioned by partitioner, can keep vertices sorted >>> with a single read and single write on every peer. >>> >>> Just clearing confusion if any regarding superstep injection for >>> partitioning. (This is outside the scope of graphs. We can have a dedicated >>> partitioning superstep for graph applications). >>> Say there are x splits and y number of tasks configured by user. >>> >>> if x > y >>> The y tasks are scheduled with x of them having each of the x splits and >>> the remaining with no resource local to them. Then the partitioning >>> superstep redistributes the partitions among them to create local >>> partitions. Now the question is can we re-initialize a peer's input based >>> on this new local part of partition? >>> >>> if y > x >>> works as it works today. >>> >>> Just putting my points in brainstorming. >>> >>> -Suraj >>> >>> >>> On Mon, Mar 11, 2013 at 7:39 AM, Edward J. Yoon >>> <[email protected]>wrote: >>> >>>> I just filed here https://issues.apache.org/jira/browse/HAMA-744 >>>> >>>> On Mon, Mar 11, 2013 at 7:35 PM, Edward J. Yoon <[email protected]> >>>> wrote: >>>> > Additionally, >>>> > >>>> >> spilling queue and sorted spilling queue, can we inject the partitioning >>>> >> superstep as the first superstep and use local memory? >>>> > >>>> > Can we execute different number of tasks per superstep? >>>> > >>>> > On Mon, Mar 11, 2013 at 6:56 PM, Edward J. Yoon <[email protected]> >>>> wrote: >>>> >>> For graph processing, the partitioned files that result from the >>>> >>> partitioning job must be sorted. Currently only the partition files in >>>> >> >>>> >> I see. >>>> >> >>>> >>> For other partitionings and with regard to our superstep API, Suraj's >>>> idea >>>> >>> of injecting a preprocessing superstep that partitions the stuff into >>>> our >>>> >>> messaging system is actually the best. >>>> >> >>>> >> BTW, if some garbage objects can be accumulated in partitioning step, >>>> >> separated partitioning job may not be bad idea. Is there some special >>>> >> reason? >>>> >> >>>> >> On Wed, Mar 6, 2013 at 6:15 PM, Thomas Jungblut >>>> >> <[email protected]> wrote: >>>> >>> For graph processing, the partitioned files that result from the >>>> >>> partitioning job must be sorted. Currently only the partition files in >>>> >>> itself are sorted, thus more tasks result in not sorted data in the >>>> >>> completed file. This only applies for the graph processing package. >>>> >>> So as Suraj told, it would be much more simpler to solve this via >>>> >>> messaging, once it is scalable (it will be very very scalable!). So the >>>> >>> GraphJobRunner can be partitioning the stuff with a single superstep in >>>> >>> setup() as it was before ages ago. The messaging must be sorted anyway >>>> for >>>> >>> the algorithm so this is a nice side effect and saves us the >>>> partitioning >>>> >>> job for graph processing. >>>> >>> >>>> >>> For other partitionings and with regard to our superstep API, Suraj's >>>> idea >>>> >>> of injecting a preprocessing superstep that partitions the stuff into >>>> our >>>> >>> messaging system is actually the best. >>>> >>> >>>> >>> >>>> >>> 2013/3/6 Suraj Menon <[email protected]> >>>> >>> >>>> >>>> No, the partitions we write locally need not be sorted. Sorry for the >>>> >>>> confusion. The Superstep injection is possible with Superstep API. >>>> There >>>> >>>> are few enhancements needed to make it simpler after I last worked on >>>> it. >>>> >>>> We can then look into partitioning superstep being executed before the >>>> >>>> setup of first superstep of submitted job. I think it is feasible. >>>> >>>> >>>> >>>> On Tue, Mar 5, 2013 at 5:48 AM, Edward J. Yoon <[email protected] >>>> >>>> >wrote: >>>> >>>> >>>> >>>> > > spilling queue and sorted spilling queue, can we inject the >>>> >>>> partitioning >>>> >>>> > > superstep as the first superstep and use local memory? >>>> >>>> > >>>> >>>> > Actually, I wanted to add something before calling BSP.setup() >>>> method >>>> >>>> > to avoid execute additional BSP job. But, in my opinion, current is >>>> >>>> > enough. I think, we need to collect more experiences of input >>>> >>>> > partitioning on large environments. I'll do. >>>> >>>> > >>>> >>>> > BTW, I still don't know why it need to be Sorted?! MR-like? >>>> >>>> > >>>> >>>> > On Thu, Feb 28, 2013 at 11:20 PM, Suraj Menon < >>>> [email protected]> >>>> >>>> > wrote: >>>> >>>> > > Sorry, I am increasing the scope here to outside graph module. >>>> When we >>>> >>>> > have >>>> >>>> > > spilling queue and sorted spilling queue, can we inject the >>>> >>>> partitioning >>>> >>>> > > superstep as the first superstep and use local memory? >>>> >>>> > > Today we have partitioning job within a job and are creating two >>>> copies >>>> >>>> > of >>>> >>>> > > data on HDFS. This could be really costly. Is it possible to >>>> create or >>>> >>>> > > redistribute the partitions on local memory and initialize the >>>> record >>>> >>>> > > reader there? >>>> >>>> > > The user can run a separate job give in examples area to >>>> explicitly >>>> >>>> > > repartition the data on HDFS. The deployment question is how much >>>> of >>>> >>>> disk >>>> >>>> > > space gets allocated for local memory usage? Would it be a safe >>>> >>>> approach >>>> >>>> > > with the limitations? >>>> >>>> > > >>>> >>>> > > -Suraj >>>> >>>> > > >>>> >>>> > > On Thu, Feb 28, 2013 at 7:05 AM, Thomas Jungblut >>>> >>>> > > <[email protected]>wrote: >>>> >>>> > > >>>> >>>> > >> yes. Once Suraj added merging of sorted files we can add this to >>>> the >>>> >>>> > >> partitioner pretty easily. >>>> >>>> > >> >>>> >>>> > >> 2013/2/28 Edward J. Yoon <[email protected]> >>>> >>>> > >> >>>> >>>> > >> > Eh,..... btw, is re-partitioned data really necessary to be >>>> Sorted? >>>> >>>> > >> > >>>> >>>> > >> > On Thu, Feb 28, 2013 at 7:48 PM, Thomas Jungblut >>>> >>>> > >> > <[email protected]> wrote: >>>> >>>> > >> > > Now I get how the partitioning works, obviously if you merge >>>> n >>>> >>>> > sorted >>>> >>>> > >> > files >>>> >>>> > >> > > by just appending to each other, this will result in totally >>>> >>>> > unsorted >>>> >>>> > >> > data >>>> >>>> > >> > > ;-) >>>> >>>> > >> > > Why didn't you solve this via messaging? >>>> >>>> > >> > > >>>> >>>> > >> > > 2013/2/28 Thomas Jungblut <[email protected]> >>>> >>>> > >> > > >>>> >>>> > >> > >> Seems that they are not correctly sorted: >>>> >>>> > >> > >> >>>> >>>> > >> > >> vertexID: 50 >>>> >>>> > >> > >> vertexID: 52 >>>> >>>> > >> > >> vertexID: 54 >>>> >>>> > >> > >> vertexID: 56 >>>> >>>> > >> > >> vertexID: 58 >>>> >>>> > >> > >> vertexID: 61 >>>> >>>> > >> > >> ... >>>> >>>> > >> > >> vertexID: 78 >>>> >>>> > >> > >> vertexID: 81 >>>> >>>> > >> > >> vertexID: 83 >>>> >>>> > >> > >> vertexID: 85 >>>> >>>> > >> > >> ... >>>> >>>> > >> > >> vertexID: 94 >>>> >>>> > >> > >> vertexID: 96 >>>> >>>> > >> > >> vertexID: 98 >>>> >>>> > >> > >> vertexID: 1 >>>> >>>> > >> > >> vertexID: 10 >>>> >>>> > >> > >> vertexID: 12 >>>> >>>> > >> > >> vertexID: 14 >>>> >>>> > >> > >> vertexID: 16 >>>> >>>> > >> > >> vertexID: 18 >>>> >>>> > >> > >> vertexID: 21 >>>> >>>> > >> > >> vertexID: 23 >>>> >>>> > >> > >> vertexID: 25 >>>> >>>> > >> > >> vertexID: 27 >>>> >>>> > >> > >> vertexID: 29 >>>> >>>> > >> > >> vertexID: 3 >>>> >>>> > >> > >> >>>> >>>> > >> > >> So this won't work then correctly... >>>> >>>> > >> > >> >>>> >>>> > >> > >> >>>> >>>> > >> > >> 2013/2/28 Thomas Jungblut <[email protected]> >>>> >>>> > >> > >> >>>> >>>> > >> > >>> sure, have fun on your holidays. >>>> >>>> > >> > >>> >>>> >>>> > >> > >>> >>>> >>>> > >> > >>> 2013/2/28 Edward J. Yoon <[email protected]> >>>> >>>> > >> > >>> >>>> >>>> > >> > >>>> Sure, but if you can fix quickly, please do. March 1 is >>>> >>>> > holiday[1] >>>> >>>> > >> so >>>> >>>> > >> > >>>> I'll appear next week. >>>> >>>> > >> > >>>> >>>> >>>> > >> > >>>> 1. >>>> http://en.wikipedia.org/wiki/Public_holidays_in_South_Korea >>>> >>>> > >> > >>>> >>>> >>>> > >> > >>>> On Thu, Feb 28, 2013 at 6:36 PM, Thomas Jungblut >>>> >>>> > >> > >>>> <[email protected]> wrote: >>>> >>>> > >> > >>>> > Maybe 50 is missing from the file, didn't observe if all >>>> >>>> items >>>> >>>> > >> were >>>> >>>> > >> > >>>> added. >>>> >>>> > >> > >>>> > As far as I remember, I copy/pasted the logic of the ID >>>> into >>>> >>>> > the >>>> >>>> > >> > >>>> fastgen, >>>> >>>> > >> > >>>> > want to have a look into it? >>>> >>>> > >> > >>>> > >>>> >>>> > >> > >>>> > 2013/2/28 Edward J. Yoon <[email protected]> >>>> >>>> > >> > >>>> > >>>> >>>> > >> > >>>> >> I guess, it's a bug of fastgen, when generate adjacency >>>> >>>> matrix >>>> >>>> > >> into >>>> >>>> > >> > >>>> >> multiple files. >>>> >>>> > >> > >>>> >> >>>> >>>> > >> > >>>> >> On Thu, Feb 28, 2013 at 6:29 PM, Thomas Jungblut >>>> >>>> > >> > >>>> >> <[email protected]> wrote: >>>> >>>> > >> > >>>> >> > You have two files, are they partitioned correctly? >>>> >>>> > >> > >>>> >> > >>>> >>>> > >> > >>>> >> > 2013/2/28 Edward J. Yoon <[email protected]> >>>> >>>> > >> > >>>> >> > >>>> >>>> > >> > >>>> >> >> It looks like a bug. >>>> >>>> > >> > >>>> >> >> >>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls -al >>>> >>>> > >> /tmp/randomgraph/ >>>> >>>> > >> > >>>> >> >> total 44 >>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096 2월 28 18:03 . >>>> >>>> > >> > >>>> >> >> drwxrwxrwt 19 root root 20480 2월 28 18:04 .. >>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2243 2월 28 18:01 >>>> part-00000 >>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 28 2월 28 18:01 >>>> >>>> > .part-00000.crc >>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2251 2월 28 18:01 >>>> part-00001 >>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 28 2월 28 18:01 >>>> >>>> > .part-00001.crc >>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096 2월 28 18:03 >>>> partitions >>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls -al >>>> >>>> > >> > >>>> >> /tmp/randomgraph/partitions/ >>>> >>>> > >> > >>>> >> >> total 24 >>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096 2월 28 18:03 . >>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096 2월 28 18:03 .. >>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2932 2월 28 18:03 >>>> part-00000 >>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 32 2월 28 18:03 >>>> >>>> > .part-00000.crc >>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2955 2월 28 18:03 >>>> part-00001 >>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 32 2월 28 18:03 >>>> >>>> > .part-00001.crc >>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ >>>> >>>> > >> > >>>> >> >> >>>> >>>> > >> > >>>> >> >> >>>> >>>> > >> > >>>> >> >> On Thu, Feb 28, 2013 at 5:27 PM, Edward < >>>> >>>> [email protected] >>>> >>>> > > >>>> >>>> > >> > wrote: >>>> >>>> > >> > >>>> >> >> > yes i'll check again >>>> >>>> > >> > >>>> >> >> > >>>> >>>> > >> > >>>> >> >> > Sent from my iPhone >>>> >>>> > >> > >>>> >> >> > >>>> >>>> > >> > >>>> >> >> > On Feb 28, 2013, at 5:18 PM, Thomas Jungblut < >>>> >>>> > >> > >>>> >> [email protected]> >>>> >>>> > >> > >>>> >> >> wrote: >>>> >>>> > >> > >>>> >> >> > >>>> >>>> > >> > >>>> >> >> >> Can you verify an observation for me please? >>>> >>>> > >> > >>>> >> >> >> >>>> >>>> > >> > >>>> >> >> >> 2 files are created from fastgen, part-00000 and >>>> >>>> > >> part-00001, >>>> >>>> > >> > >>>> both >>>> >>>> > >> > >>>> >> ~2.2kb >>>> >>>> > >> > >>>> >> >> >> sized. >>>> >>>> > >> > >>>> >> >> >> In the below partition directory, there is only a >>>> >>>> single >>>> >>>> > >> > 5.56kb >>>> >>>> > >> > >>>> file. >>>> >>>> > >> > >>>> >> >> >> >>>> >>>> > >> > >>>> >> >> >> Is it intended for the partitioner to write a >>>> single >>>> >>>> > file >>>> >>>> > >> if >>>> >>>> > >> > you >>>> >>>> > >> > >>>> >> >> configured >>>> >>>> > >> > >>>> >> >> >> two? >>>> >>>> > >> > >>>> >> >> >> It even reads it as a two files, strange huh? >>>> >>>> > >> > >>>> >> >> >> >>>> >>>> > >> > >>>> >> >> >> 2013/2/28 Thomas Jungblut < >>>> [email protected]> >>>> >>>> > >> > >>>> >> >> >> >>>> >>>> > >> > >>>> >> >> >>> Will have a look into it. >>>> >>>> > >> > >>>> >> >> >>> >>>> >>>> > >> > >>>> >> >> >>> gen fastgen 100 10 /tmp/randomgraph 1 >>>> >>>> > >> > >>>> >> >> >>> pagerank /tmp/randomgraph /tmp/pageout >>>> >>>> > >> > >>>> >> >> >>> >>>> >>>> > >> > >>>> >> >> >>> did work for me the last time I profiled, maybe >>>> the >>>> >>>> > >> > >>>> partitioning >>>> >>>> > >> > >>>> >> >> doesn't >>>> >>>> > >> > >>>> >> >> >>> partition correctly with the input or something >>>> else. >>>> >>>> > >> > >>>> >> >> >>> >>>> >>>> > >> > >>>> >> >> >>> >>>> >>>> > >> > >>>> >> >> >>> 2013/2/28 Edward J. Yoon <[email protected] >>>> > >>>> >>>> > >> > >>>> >> >> >>> >>>> >>>> > >> > >>>> >> >> >>> Fastgen input seems not work for graph examples. >>>> >>>> > >> > >>>> >> >> >>>> >>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >>>> :~/workspace/hama-trunk$ >>>> >>>> > >> bin/hama >>>> >>>> > >> > jar >>>> >>>> > >> > >>>> >> >> >>>> >>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar gen >>>> >>>> > >> > fastgen >>>> >>>> > >> > >>>> 100 10 >>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph 2 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:02 WARN util.NativeCodeLoader: >>>> Unable >>>> >>>> > to >>>> >>>> > >> > load >>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your platform... >>>> using >>>> >>>> > >> > builtin-java >>>> >>>> > >> > >>>> >> classes >>>> >>>> > >> > >>>> >> >> >>>> where applicable >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO bsp.BSPJobClient: >>>> Running >>>> >>>> job: >>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO bsp.LocalBSPRunner: >>>> Setting >>>> >>>> up >>>> >>>> > a >>>> >>>> > >> new >>>> >>>> > >> > >>>> barrier >>>> >>>> > >> > >>>> >> >> for 2 >>>> >>>> > >> > >>>> >> >> >>>> tasks! >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>> Current >>>> >>>> > >> supersteps >>>> >>>> > >> > >>>> >> number: 0 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: The >>>> total >>>> >>>> > number >>>> >>>> > >> > of >>>> >>>> > >> > >>>> >> >> supersteps: 0 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>> Counters: 3 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.JobInProgress$JobCounter >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>> >>>> > SUPERSTEPS=0 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>> >>>> > >> > LAUNCHED_TASKS=2 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>> >>>> > >> > >>>> >> TASK_OUTPUT_RECORDS=100 >>>> >>>> > >> > >>>> >> >> >>>> Job Finished in 3.212 seconds >>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >>>> :~/workspace/hama-trunk$ >>>> >>>> > >> bin/hama >>>> >>>> > >> > jar >>>> >>>> > >> > >>>> >> >> >>>> examples/target/hama-examples-0.7.0-SNAPSHOT >>>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT-javadoc.jar >>>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT.jar >>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >>>> :~/workspace/hama-trunk$ >>>> >>>> > >> bin/hama >>>> >>>> > >> > jar >>>> >>>> > >> > >>>> >> >> >>>> >>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar >>>> >>>> > pagerank >>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph /tmp/pageour >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 WARN util.NativeCodeLoader: >>>> Unable >>>> >>>> > to >>>> >>>> > >> > load >>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your platform... >>>> using >>>> >>>> > >> > builtin-java >>>> >>>> > >> > >>>> >> classes >>>> >>>> > >> > >>>> >> >> >>>> where applicable >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: >>>> Total >>>> >>>> > input >>>> >>>> > >> > paths >>>> >>>> > >> > >>>> to >>>> >>>> > >> > >>>> >> >> process >>>> >>>> > >> > >>>> >> >> >>>> : 2 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: >>>> Total >>>> >>>> > input >>>> >>>> > >> > paths >>>> >>>> > >> > >>>> to >>>> >>>> > >> > >>>> >> >> process >>>> >>>> > >> > >>>> >> >> >>>> : 2 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO bsp.BSPJobClient: >>>> Running >>>> >>>> job: >>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO bsp.LocalBSPRunner: >>>> Setting >>>> >>>> up >>>> >>>> > a >>>> >>>> > >> new >>>> >>>> > >> > >>>> barrier >>>> >>>> > >> > >>>> >> >> for 2 >>>> >>>> > >> > >>>> >> >> >>>> tasks! >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>> Current >>>> >>>> > >> supersteps >>>> >>>> > >> > >>>> >> number: 1 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: The >>>> total >>>> >>>> > number >>>> >>>> > >> > of >>>> >>>> > >> > >>>> >> >> supersteps: 1 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>> Counters: 6 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.JobInProgress$JobCounter >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>> >>>> > SUPERSTEPS=1 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>> >>>> > >> > LAUNCHED_TASKS=2 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>> >>>> > >> > SUPERSTEP_SUM=4 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>> >>>> > >> > >>>> IO_BYTES_READ=4332 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>> >>>> > >> > >>>> TIME_IN_SYNC_MS=14 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>> >>>> > >> > >>>> TASK_INPUT_RECORDS=100 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.FileInputFormat: >>>> Total >>>> >>>> > input >>>> >>>> > >> > paths >>>> >>>> > >> > >>>> to >>>> >>>> > >> > >>>> >> >> process >>>> >>>> > >> > >>>> >> >> >>>> : 2 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>> Running >>>> >>>> job: >>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.LocalBSPRunner: >>>> Setting >>>> >>>> up >>>> >>>> > a >>>> >>>> > >> new >>>> >>>> > >> > >>>> barrier >>>> >>>> > >> > >>>> >> >> for 2 >>>> >>>> > >> > >>>> >> >> >>>> tasks! >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: 50 >>>> >>>> > vertices >>>> >>>> > >> > are >>>> >>>> > >> > >>>> loaded >>>> >>>> > >> > >>>> >> >> into >>>> >>>> > >> > >>>> >> >> >>>> local:1 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: 50 >>>> >>>> > vertices >>>> >>>> > >> > are >>>> >>>> > >> > >>>> loaded >>>> >>>> > >> > >>>> >> >> into >>>> >>>> > >> > >>>> >> >> >>>> local:0 >>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 ERROR bsp.LocalBSPRunner: >>>> >>>> Exception >>>> >>>> > >> > during >>>> >>>> > >> > >>>> BSP >>>> >>>> > >> > >>>> >> >> >>>> execution! >>>> >>>> > >> > >>>> >> >> >>>> java.lang.IllegalArgumentException: Messages >>>> must >>>> >>>> > never >>>> >>>> > >> be >>>> >>>> > >> > >>>> behind >>>> >>>> > >> > >>>> >> the >>>> >>>> > >> > >>>> >> >> >>>> vertex in ID! Current Message ID: 1 vs. 50 >>>> >>>> > >> > >>>> >> >> >>>> at >>>> >>>> > >> > >>>> >> >> >>>> >>>> >>>> > >> > >>>> >> >>>> >>>> > >> > >>>> >>>> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279) >>>> >>>> > >> > >>>> >> >> >>>> at >>>> >>>> > >> > >>>> >> >> >>>> >>>> >>>> > >> > >>>> >> >> >>>> >>>> > >> > >>>> >> >>>> >>>> > >> > >>>> >>>> >>>> > >> > >>>> >>>> > >>>> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225) >>>> >>>> > >> > >>>> >> >> >>>> at >>>> >>>> > >> > >>>> >> >> >>>> >>>> >>>> > >> > >>>> >>>> >>>> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129) >>>> >>>> > >> > >>>> >> >> >>>> at >>>> >>>> > >> > >>>> >> >> >>>> >>>> >>>> > >> > >>>> >> >> >>>> >>>> > >> > >>>> >> >>>> >>>> > >> > >>>> >>>> >>>> > >> > >>>> >>>> > >>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256) >>>> >>>> > >> > >>>> >> >> >>>> at >>>> >>>> > >> > >>>> >> >> >>>> >>>> >>>> > >> > >>>> >> >> >>>> >>>> > >> > >>>> >> >>>> >>>> > >> > >>>> >>>> >>>> > >> > >>>> >>>> > >> >>>> >>>> > >>>> >>>> >>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >>>> >>>> > >> > >>>> >> >> >>>> at >>>> >>>> > >> > >>>> >> >> >>>> >>>> >>>> > >> > >>>> >> >> >>>> >>>> > >> > >>>> >> >>>> >>>> > >> > >>>> >>>> >>>> > >> > >>>> >>>> > >> >>>> >>>> > >>>> >>>> >>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >>>> >>>> > >> > >>>> >> >> >>>> at >>>> >>>> > >> > >>>> >> >> >>>> >>>> >>>> > >> > >>>> >>>> >>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>> >>>> > >> > >>>> >> >> >>>> at >>>> >>>> > >> > >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>> >>>> > >> > >>>> >> >> >>>> at >>>> >>>> > >> > >>>> >> >> >>>> >>>> >>>> > >> > >>>> >> >> >>>> >>>> > >> > >>>> >>>> >>>> > >> > >>>> >>>> > >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>> >>>> > >> > >>>> >> >> >>>> at >>>> >>>> > >> > >>>> >> >> >>>> >>>> >>>> > >> > >>>> >>>> >>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>> >>>> > >> > >>>> >> >> >>>> at >>>> >>>> > >> > >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>> >>>> > >> > >>>> >> >> >>>> at >>>> >>>> > >> > >>>> >> >> >>>> >>>> >>>> > >> > >>>> >> >> >>>> >>>> > >> > >>>> >> >>>> >>>> > >> > >>>> >>>> >>>> > >> > >>>> >>>> > >> >>>> >>>> > >>>> >>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>>> >>>> > >> > >>>> >> >> >>>> at >>>> >>>> > >> > >>>> >> >> >>>> >>>> >>>> > >> > >>>> >> >> >>>> >>>> > >> > >>>> >> >>>> >>>> > >> > >>>> >>>> >>>> > >> > >>>> >>>> > >> >>>> >>>> > >>>> >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>>> >>>> > >> > >>>> >> >> >>>> at java.lang.Thread.run(Thread.java:722) >>>> >>>> > >> > >>>> >> >> >>>> >>>> >>>> > >> > >>>> >> >> >>>> >>>> >>>> > >> > >>>> >> >> >>>> -- >>>> >>>> > >> > >>>> >> >> >>>> Best Regards, Edward J. Yoon >>>> >>>> > >> > >>>> >> >> >>>> @eddieyoon >>>> >>>> > >> > >>>> >> >> >>> >>>> >>>> > >> > >>>> >> >> >>> >>>> >>>> > >> > >>>> >> >> >>>> >>>> > >> > >>>> >> >> >>>> >>>> > >> > >>>> >> >> >>>> >>>> > >> > >>>> >> >> -- >>>> >>>> > >> > >>>> >> >> Best Regards, Edward J. Yoon >>>> >>>> > >> > >>>> >> >> @eddieyoon >>>> >>>> > >> > >>>> >> >> >>>> >>>> > >> > >>>> >> >>>> >>>> > >> > >>>> >> >>>> >>>> > >> > >>>> >> >>>> >>>> > >> > >>>> >> -- >>>> >>>> > >> > >>>> >> Best Regards, Edward J. Yoon >>>> >>>> > >> > >>>> >> @eddieyoon >>>> >>>> > >> > >>>> >> >>>> >>>> > >> > >>>> >>>> >>>> > >> > >>>> >>>> >>>> > >> > >>>> >>>> >>>> > >> > >>>> -- >>>> >>>> > >> > >>>> Best Regards, Edward J. Yoon >>>> >>>> > >> > >>>> @eddieyoon >>>> >>>> > >> > >>>> >>>> >>>> > >> > >>> >>>> >>>> > >> > >>> >>>> >>>> > >> > >> >>>> >>>> > >> > >>>> >>>> > >> > >>>> >>>> > >> > >>>> >>>> > >> > -- >>>> >>>> > >> > Best Regards, Edward J. Yoon >>>> >>>> > >> > @eddieyoon >>>> >>>> > >> > >>>> >>>> > >> >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > -- >>>> >>>> > Best Regards, Edward J. Yoon >>>> >>>> > @eddieyoon >>>> >>>> > >>>> >>>> >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> Best Regards, Edward J. Yoon >>>> >> @eddieyoon >>>> > >>>> > >>>> > >>>> > -- >>>> > Best Regards, Edward J. Yoon >>>> > @eddieyoon >>>> >>>> >>>> >>>> -- >>>> Best Regards, Edward J. Yoon >>>> @eddieyoon >>>> >> >> >> >> -- >> Best Regards, Edward J. Yoon >> @eddieyoon > > > > -- > Best Regards, Edward J. Yoon > @eddieyoon -- Best Regards, Edward J. Yoon @eddieyoon
