I'm reading changes of HAMA-704 again. As a result of adding DiskVerticesInfo, vertices list is needed to be sorted. I'm not sure but I think this approach will bring more disadvantages than advantages.
On Wed, Mar 13, 2013 at 11:09 PM, Edward J. Yoon <[email protected]> wrote: >>>> in loadVertices? Maybe consider feature for coupling storage in user space >>>> with BSP Messaging[HAMA-734] can avoid double reads and writes. This way >>>> partitioned or non-partitioned by partitioner, can keep vertices sorted >>>> with a single read and single write on every peer. > > And, as I commented JIRA ticket, I think we can't use messaging system > for sorting vertices within partition files. > > On Wed, Mar 13, 2013 at 11:00 PM, Edward J. Yoon <[email protected]> > wrote: >> P.S., (number of splits = number of partitions) is really confuse to >> me. Even though blocks number is equal to desired tasks number, data >> should be re-partitioned again. >> >> On Wed, Mar 13, 2013 at 10:36 PM, Edward J. Yoon <[email protected]> >> wrote: >>> Indeed. If there are already partitioned input files (unsorted) and so >>> user want to skip pre-partitioning phase, it should be handled in >>> GraphJobRunner BSP program. Actually, I still don't know why >>> re-partitioned files need to be Sorted. It's only about >>> GraphJobRunner. >>> >>>> partitioning. (This is outside the scope of graphs. We can have a dedicated >>>> partitioning superstep for graph applications). >>> >>> Sorry. I don't understand exactly yet. Do you mean just a partitioning >>> job based on superstep API? >>> >>> By default, 100 tasks will be assigned for partitioning job. >>> Partitioning job will create 1,000 partitions. Thus, we can execute >>> the Graph job with 1,000 tasks. >>> >>> Let's assume that a input sequence file is 20GB (100 blocks). If I >>> want to run with 1,000 tasks, what happens? >>> >>> On Wed, Mar 13, 2013 at 9:49 PM, Suraj Menon <[email protected]> wrote: >>>> I am responding on this thread because of better continuity for >>>> conversation. We cannot expect the partitions to be sorted every time. When >>>> the number of splits = number of partitions and partitioning is switched >>>> off by user[HAMA-561], the partitions would not be sorted. Can we do this >>>> in loadVertices? Maybe consider feature for coupling storage in user space >>>> with BSP Messaging[HAMA-734] can avoid double reads and writes. This way >>>> partitioned or non-partitioned by partitioner, can keep vertices sorted >>>> with a single read and single write on every peer. >>>> >>>> Just clearing confusion if any regarding superstep injection for >>>> partitioning. (This is outside the scope of graphs. We can have a dedicated >>>> partitioning superstep for graph applications). >>>> Say there are x splits and y number of tasks configured by user. >>>> >>>> if x > y >>>> The y tasks are scheduled with x of them having each of the x splits and >>>> the remaining with no resource local to them. Then the partitioning >>>> superstep redistributes the partitions among them to create local >>>> partitions. Now the question is can we re-initialize a peer's input based >>>> on this new local part of partition? >>>> >>>> if y > x >>>> works as it works today. >>>> >>>> Just putting my points in brainstorming. >>>> >>>> -Suraj >>>> >>>> >>>> On Mon, Mar 11, 2013 at 7:39 AM, Edward J. Yoon >>>> <[email protected]>wrote: >>>> >>>>> I just filed here https://issues.apache.org/jira/browse/HAMA-744 >>>>> >>>>> On Mon, Mar 11, 2013 at 7:35 PM, Edward J. Yoon <[email protected]> >>>>> wrote: >>>>> > Additionally, >>>>> > >>>>> >> spilling queue and sorted spilling queue, can we inject the >>>>> >> partitioning >>>>> >> superstep as the first superstep and use local memory? >>>>> > >>>>> > Can we execute different number of tasks per superstep? >>>>> > >>>>> > On Mon, Mar 11, 2013 at 6:56 PM, Edward J. Yoon <[email protected]> >>>>> wrote: >>>>> >>> For graph processing, the partitioned files that result from the >>>>> >>> partitioning job must be sorted. Currently only the partition files in >>>>> >> >>>>> >> I see. >>>>> >> >>>>> >>> For other partitionings and with regard to our superstep API, Suraj's >>>>> idea >>>>> >>> of injecting a preprocessing superstep that partitions the stuff into >>>>> our >>>>> >>> messaging system is actually the best. >>>>> >> >>>>> >> BTW, if some garbage objects can be accumulated in partitioning step, >>>>> >> separated partitioning job may not be bad idea. Is there some special >>>>> >> reason? >>>>> >> >>>>> >> On Wed, Mar 6, 2013 at 6:15 PM, Thomas Jungblut >>>>> >> <[email protected]> wrote: >>>>> >>> For graph processing, the partitioned files that result from the >>>>> >>> partitioning job must be sorted. Currently only the partition files in >>>>> >>> itself are sorted, thus more tasks result in not sorted data in the >>>>> >>> completed file. This only applies for the graph processing package. >>>>> >>> So as Suraj told, it would be much more simpler to solve this via >>>>> >>> messaging, once it is scalable (it will be very very scalable!). So >>>>> >>> the >>>>> >>> GraphJobRunner can be partitioning the stuff with a single superstep >>>>> >>> in >>>>> >>> setup() as it was before ages ago. The messaging must be sorted anyway >>>>> for >>>>> >>> the algorithm so this is a nice side effect and saves us the >>>>> partitioning >>>>> >>> job for graph processing. >>>>> >>> >>>>> >>> For other partitionings and with regard to our superstep API, Suraj's >>>>> idea >>>>> >>> of injecting a preprocessing superstep that partitions the stuff into >>>>> our >>>>> >>> messaging system is actually the best. >>>>> >>> >>>>> >>> >>>>> >>> 2013/3/6 Suraj Menon <[email protected]> >>>>> >>> >>>>> >>>> No, the partitions we write locally need not be sorted. Sorry for the >>>>> >>>> confusion. The Superstep injection is possible with Superstep API. >>>>> There >>>>> >>>> are few enhancements needed to make it simpler after I last worked on >>>>> it. >>>>> >>>> We can then look into partitioning superstep being executed before >>>>> >>>> the >>>>> >>>> setup of first superstep of submitted job. I think it is feasible. >>>>> >>>> >>>>> >>>> On Tue, Mar 5, 2013 at 5:48 AM, Edward J. Yoon <[email protected] >>>>> >>>> >wrote: >>>>> >>>> >>>>> >>>> > > spilling queue and sorted spilling queue, can we inject the >>>>> >>>> partitioning >>>>> >>>> > > superstep as the first superstep and use local memory? >>>>> >>>> > >>>>> >>>> > Actually, I wanted to add something before calling BSP.setup() >>>>> method >>>>> >>>> > to avoid execute additional BSP job. But, in my opinion, current is >>>>> >>>> > enough. I think, we need to collect more experiences of input >>>>> >>>> > partitioning on large environments. I'll do. >>>>> >>>> > >>>>> >>>> > BTW, I still don't know why it need to be Sorted?! MR-like? >>>>> >>>> > >>>>> >>>> > On Thu, Feb 28, 2013 at 11:20 PM, Suraj Menon < >>>>> [email protected]> >>>>> >>>> > wrote: >>>>> >>>> > > Sorry, I am increasing the scope here to outside graph module. >>>>> When we >>>>> >>>> > have >>>>> >>>> > > spilling queue and sorted spilling queue, can we inject the >>>>> >>>> partitioning >>>>> >>>> > > superstep as the first superstep and use local memory? >>>>> >>>> > > Today we have partitioning job within a job and are creating two >>>>> copies >>>>> >>>> > of >>>>> >>>> > > data on HDFS. This could be really costly. Is it possible to >>>>> create or >>>>> >>>> > > redistribute the partitions on local memory and initialize the >>>>> record >>>>> >>>> > > reader there? >>>>> >>>> > > The user can run a separate job give in examples area to >>>>> explicitly >>>>> >>>> > > repartition the data on HDFS. The deployment question is how much >>>>> of >>>>> >>>> disk >>>>> >>>> > > space gets allocated for local memory usage? Would it be a safe >>>>> >>>> approach >>>>> >>>> > > with the limitations? >>>>> >>>> > > >>>>> >>>> > > -Suraj >>>>> >>>> > > >>>>> >>>> > > On Thu, Feb 28, 2013 at 7:05 AM, Thomas Jungblut >>>>> >>>> > > <[email protected]>wrote: >>>>> >>>> > > >>>>> >>>> > >> yes. Once Suraj added merging of sorted files we can add this to >>>>> the >>>>> >>>> > >> partitioner pretty easily. >>>>> >>>> > >> >>>>> >>>> > >> 2013/2/28 Edward J. Yoon <[email protected]> >>>>> >>>> > >> >>>>> >>>> > >> > Eh,..... btw, is re-partitioned data really necessary to be >>>>> Sorted? >>>>> >>>> > >> > >>>>> >>>> > >> > On Thu, Feb 28, 2013 at 7:48 PM, Thomas Jungblut >>>>> >>>> > >> > <[email protected]> wrote: >>>>> >>>> > >> > > Now I get how the partitioning works, obviously if you merge >>>>> n >>>>> >>>> > sorted >>>>> >>>> > >> > files >>>>> >>>> > >> > > by just appending to each other, this will result in totally >>>>> >>>> > unsorted >>>>> >>>> > >> > data >>>>> >>>> > >> > > ;-) >>>>> >>>> > >> > > Why didn't you solve this via messaging? >>>>> >>>> > >> > > >>>>> >>>> > >> > > 2013/2/28 Thomas Jungblut <[email protected]> >>>>> >>>> > >> > > >>>>> >>>> > >> > >> Seems that they are not correctly sorted: >>>>> >>>> > >> > >> >>>>> >>>> > >> > >> vertexID: 50 >>>>> >>>> > >> > >> vertexID: 52 >>>>> >>>> > >> > >> vertexID: 54 >>>>> >>>> > >> > >> vertexID: 56 >>>>> >>>> > >> > >> vertexID: 58 >>>>> >>>> > >> > >> vertexID: 61 >>>>> >>>> > >> > >> ... >>>>> >>>> > >> > >> vertexID: 78 >>>>> >>>> > >> > >> vertexID: 81 >>>>> >>>> > >> > >> vertexID: 83 >>>>> >>>> > >> > >> vertexID: 85 >>>>> >>>> > >> > >> ... >>>>> >>>> > >> > >> vertexID: 94 >>>>> >>>> > >> > >> vertexID: 96 >>>>> >>>> > >> > >> vertexID: 98 >>>>> >>>> > >> > >> vertexID: 1 >>>>> >>>> > >> > >> vertexID: 10 >>>>> >>>> > >> > >> vertexID: 12 >>>>> >>>> > >> > >> vertexID: 14 >>>>> >>>> > >> > >> vertexID: 16 >>>>> >>>> > >> > >> vertexID: 18 >>>>> >>>> > >> > >> vertexID: 21 >>>>> >>>> > >> > >> vertexID: 23 >>>>> >>>> > >> > >> vertexID: 25 >>>>> >>>> > >> > >> vertexID: 27 >>>>> >>>> > >> > >> vertexID: 29 >>>>> >>>> > >> > >> vertexID: 3 >>>>> >>>> > >> > >> >>>>> >>>> > >> > >> So this won't work then correctly... >>>>> >>>> > >> > >> >>>>> >>>> > >> > >> >>>>> >>>> > >> > >> 2013/2/28 Thomas Jungblut <[email protected]> >>>>> >>>> > >> > >> >>>>> >>>> > >> > >>> sure, have fun on your holidays. >>>>> >>>> > >> > >>> >>>>> >>>> > >> > >>> >>>>> >>>> > >> > >>> 2013/2/28 Edward J. Yoon <[email protected]> >>>>> >>>> > >> > >>> >>>>> >>>> > >> > >>>> Sure, but if you can fix quickly, please do. March 1 is >>>>> >>>> > holiday[1] >>>>> >>>> > >> so >>>>> >>>> > >> > >>>> I'll appear next week. >>>>> >>>> > >> > >>>> >>>>> >>>> > >> > >>>> 1. >>>>> http://en.wikipedia.org/wiki/Public_holidays_in_South_Korea >>>>> >>>> > >> > >>>> >>>>> >>>> > >> > >>>> On Thu, Feb 28, 2013 at 6:36 PM, Thomas Jungblut >>>>> >>>> > >> > >>>> <[email protected]> wrote: >>>>> >>>> > >> > >>>> > Maybe 50 is missing from the file, didn't observe if >>>>> >>>> > >> > >>>> > all >>>>> >>>> items >>>>> >>>> > >> were >>>>> >>>> > >> > >>>> added. >>>>> >>>> > >> > >>>> > As far as I remember, I copy/pasted the logic of the ID >>>>> into >>>>> >>>> > the >>>>> >>>> > >> > >>>> fastgen, >>>>> >>>> > >> > >>>> > want to have a look into it? >>>>> >>>> > >> > >>>> > >>>>> >>>> > >> > >>>> > 2013/2/28 Edward J. Yoon <[email protected]> >>>>> >>>> > >> > >>>> > >>>>> >>>> > >> > >>>> >> I guess, it's a bug of fastgen, when generate >>>>> >>>> > >> > >>>> >> adjacency >>>>> >>>> matrix >>>>> >>>> > >> into >>>>> >>>> > >> > >>>> >> multiple files. >>>>> >>>> > >> > >>>> >> >>>>> >>>> > >> > >>>> >> On Thu, Feb 28, 2013 at 6:29 PM, Thomas Jungblut >>>>> >>>> > >> > >>>> >> <[email protected]> wrote: >>>>> >>>> > >> > >>>> >> > You have two files, are they partitioned correctly? >>>>> >>>> > >> > >>>> >> > >>>>> >>>> > >> > >>>> >> > 2013/2/28 Edward J. Yoon <[email protected]> >>>>> >>>> > >> > >>>> >> > >>>>> >>>> > >> > >>>> >> >> It looks like a bug. >>>>> >>>> > >> > >>>> >> >> >>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls -al >>>>> >>>> > >> /tmp/randomgraph/ >>>>> >>>> > >> > >>>> >> >> total 44 >>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096 2월 28 18:03 . >>>>> >>>> > >> > >>>> >> >> drwxrwxrwt 19 root root 20480 2월 28 18:04 .. >>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2243 2월 28 18:01 >>>>> part-00000 >>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 28 2월 28 18:01 >>>>> >>>> > .part-00000.crc >>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2251 2월 28 18:01 >>>>> part-00001 >>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 28 2월 28 18:01 >>>>> >>>> > .part-00001.crc >>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096 2월 28 18:03 >>>>> partitions >>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls -al >>>>> >>>> > >> > >>>> >> /tmp/randomgraph/partitions/ >>>>> >>>> > >> > >>>> >> >> total 24 >>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096 2월 28 18:03 . >>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096 2월 28 18:03 .. >>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2932 2월 28 18:03 >>>>> part-00000 >>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 32 2월 28 18:03 >>>>> >>>> > .part-00000.crc >>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2955 2월 28 18:03 >>>>> part-00001 >>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 32 2월 28 18:03 >>>>> >>>> > .part-00001.crc >>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ >>>>> >>>> > >> > >>>> >> >> >>>>> >>>> > >> > >>>> >> >> >>>>> >>>> > >> > >>>> >> >> On Thu, Feb 28, 2013 at 5:27 PM, Edward < >>>>> >>>> [email protected] >>>>> >>>> > > >>>>> >>>> > >> > wrote: >>>>> >>>> > >> > >>>> >> >> > yes i'll check again >>>>> >>>> > >> > >>>> >> >> > >>>>> >>>> > >> > >>>> >> >> > Sent from my iPhone >>>>> >>>> > >> > >>>> >> >> > >>>>> >>>> > >> > >>>> >> >> > On Feb 28, 2013, at 5:18 PM, Thomas Jungblut < >>>>> >>>> > >> > >>>> >> [email protected]> >>>>> >>>> > >> > >>>> >> >> wrote: >>>>> >>>> > >> > >>>> >> >> > >>>>> >>>> > >> > >>>> >> >> >> Can you verify an observation for me please? >>>>> >>>> > >> > >>>> >> >> >> >>>>> >>>> > >> > >>>> >> >> >> 2 files are created from fastgen, part-00000 and >>>>> >>>> > >> part-00001, >>>>> >>>> > >> > >>>> both >>>>> >>>> > >> > >>>> >> ~2.2kb >>>>> >>>> > >> > >>>> >> >> >> sized. >>>>> >>>> > >> > >>>> >> >> >> In the below partition directory, there is only >>>>> >>>> > >> > >>>> >> >> >> a >>>>> >>>> single >>>>> >>>> > >> > 5.56kb >>>>> >>>> > >> > >>>> file. >>>>> >>>> > >> > >>>> >> >> >> >>>>> >>>> > >> > >>>> >> >> >> Is it intended for the partitioner to write a >>>>> single >>>>> >>>> > file >>>>> >>>> > >> if >>>>> >>>> > >> > you >>>>> >>>> > >> > >>>> >> >> configured >>>>> >>>> > >> > >>>> >> >> >> two? >>>>> >>>> > >> > >>>> >> >> >> It even reads it as a two files, strange huh? >>>>> >>>> > >> > >>>> >> >> >> >>>>> >>>> > >> > >>>> >> >> >> 2013/2/28 Thomas Jungblut < >>>>> [email protected]> >>>>> >>>> > >> > >>>> >> >> >> >>>>> >>>> > >> > >>>> >> >> >>> Will have a look into it. >>>>> >>>> > >> > >>>> >> >> >>> >>>>> >>>> > >> > >>>> >> >> >>> gen fastgen 100 10 /tmp/randomgraph 1 >>>>> >>>> > >> > >>>> >> >> >>> pagerank /tmp/randomgraph /tmp/pageout >>>>> >>>> > >> > >>>> >> >> >>> >>>>> >>>> > >> > >>>> >> >> >>> did work for me the last time I profiled, maybe >>>>> the >>>>> >>>> > >> > >>>> partitioning >>>>> >>>> > >> > >>>> >> >> doesn't >>>>> >>>> > >> > >>>> >> >> >>> partition correctly with the input or something >>>>> else. >>>>> >>>> > >> > >>>> >> >> >>> >>>>> >>>> > >> > >>>> >> >> >>> >>>>> >>>> > >> > >>>> >> >> >>> 2013/2/28 Edward J. Yoon <[email protected] >>>>> > >>>>> >>>> > >> > >>>> >> >> >>> >>>>> >>>> > >> > >>>> >> >> >>> Fastgen input seems not work for graph >>>>> >>>> > >> > >>>> >> >> >>> examples. >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >>>>> :~/workspace/hama-trunk$ >>>>> >>>> > >> bin/hama >>>>> >>>> > >> > jar >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar gen >>>>> >>>> > >> > fastgen >>>>> >>>> > >> > >>>> 100 10 >>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph 2 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:02 WARN util.NativeCodeLoader: >>>>> Unable >>>>> >>>> > to >>>>> >>>> > >> > load >>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your platform... >>>>> using >>>>> >>>> > >> > builtin-java >>>>> >>>> > >> > >>>> >> classes >>>>> >>>> > >> > >>>> >> >> >>>> where applicable >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO bsp.BSPJobClient: >>>>> Running >>>>> >>>> job: >>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO bsp.LocalBSPRunner: >>>>> Setting >>>>> >>>> up >>>>> >>>> > a >>>>> >>>> > >> new >>>>> >>>> > >> > >>>> barrier >>>>> >>>> > >> > >>>> >> >> for 2 >>>>> >>>> > >> > >>>> >> >> >>>> tasks! >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>> Current >>>>> >>>> > >> supersteps >>>>> >>>> > >> > >>>> >> number: 0 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: The >>>>> total >>>>> >>>> > number >>>>> >>>> > >> > of >>>>> >>>> > >> > >>>> >> >> supersteps: 0 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>> Counters: 3 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.JobInProgress$JobCounter >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>> >>>> > SUPERSTEPS=0 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>> >>>> > >> > LAUNCHED_TASKS=2 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>>>> >>>> > >> > >>>> >> TASK_OUTPUT_RECORDS=100 >>>>> >>>> > >> > >>>> >> >> >>>> Job Finished in 3.212 seconds >>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >>>>> :~/workspace/hama-trunk$ >>>>> >>>> > >> bin/hama >>>>> >>>> > >> > jar >>>>> >>>> > >> > >>>> >> >> >>>> examples/target/hama-examples-0.7.0-SNAPSHOT >>>>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT-javadoc.jar >>>>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT.jar >>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >>>>> :~/workspace/hama-trunk$ >>>>> >>>> > >> bin/hama >>>>> >>>> > >> > jar >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar >>>>> >>>> > pagerank >>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph /tmp/pageour >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 WARN util.NativeCodeLoader: >>>>> Unable >>>>> >>>> > to >>>>> >>>> > >> > load >>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your platform... >>>>> using >>>>> >>>> > >> > builtin-java >>>>> >>>> > >> > >>>> >> classes >>>>> >>>> > >> > >>>> >> >> >>>> where applicable >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: >>>>> Total >>>>> >>>> > input >>>>> >>>> > >> > paths >>>>> >>>> > >> > >>>> to >>>>> >>>> > >> > >>>> >> >> process >>>>> >>>> > >> > >>>> >> >> >>>> : 2 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: >>>>> Total >>>>> >>>> > input >>>>> >>>> > >> > paths >>>>> >>>> > >> > >>>> to >>>>> >>>> > >> > >>>> >> >> process >>>>> >>>> > >> > >>>> >> >> >>>> : 2 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO bsp.BSPJobClient: >>>>> Running >>>>> >>>> job: >>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO bsp.LocalBSPRunner: >>>>> Setting >>>>> >>>> up >>>>> >>>> > a >>>>> >>>> > >> new >>>>> >>>> > >> > >>>> barrier >>>>> >>>> > >> > >>>> >> >> for 2 >>>>> >>>> > >> > >>>> >> >> >>>> tasks! >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>> Current >>>>> >>>> > >> supersteps >>>>> >>>> > >> > >>>> >> number: 1 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: The >>>>> total >>>>> >>>> > number >>>>> >>>> > >> > of >>>>> >>>> > >> > >>>> >> >> supersteps: 1 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>> Counters: 6 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.JobInProgress$JobCounter >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>> >>>> > SUPERSTEPS=1 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>> >>>> > >> > LAUNCHED_TASKS=2 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>> >>>> > >> > SUPERSTEP_SUM=4 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>> >>>> > >> > >>>> IO_BYTES_READ=4332 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>> >>>> > >> > >>>> TIME_IN_SYNC_MS=14 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>> >>>> > >> > >>>> TASK_INPUT_RECORDS=100 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.FileInputFormat: >>>>> Total >>>>> >>>> > input >>>>> >>>> > >> > paths >>>>> >>>> > >> > >>>> to >>>>> >>>> > >> > >>>> >> >> process >>>>> >>>> > >> > >>>> >> >> >>>> : 2 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>>>> Running >>>>> >>>> job: >>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.LocalBSPRunner: >>>>> Setting >>>>> >>>> up >>>>> >>>> > a >>>>> >>>> > >> new >>>>> >>>> > >> > >>>> barrier >>>>> >>>> > >> > >>>> >> >> for 2 >>>>> >>>> > >> > >>>> >> >> >>>> tasks! >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: >>>>> >>>> > >> > >>>> >> >> >>>> 50 >>>>> >>>> > vertices >>>>> >>>> > >> > are >>>>> >>>> > >> > >>>> loaded >>>>> >>>> > >> > >>>> >> >> into >>>>> >>>> > >> > >>>> >> >> >>>> local:1 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: >>>>> >>>> > >> > >>>> >> >> >>>> 50 >>>>> >>>> > vertices >>>>> >>>> > >> > are >>>>> >>>> > >> > >>>> loaded >>>>> >>>> > >> > >>>> >> >> into >>>>> >>>> > >> > >>>> >> >> >>>> local:0 >>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 ERROR bsp.LocalBSPRunner: >>>>> >>>> Exception >>>>> >>>> > >> > during >>>>> >>>> > >> > >>>> BSP >>>>> >>>> > >> > >>>> >> >> >>>> execution! >>>>> >>>> > >> > >>>> >> >> >>>> java.lang.IllegalArgumentException: Messages >>>>> must >>>>> >>>> > never >>>>> >>>> > >> be >>>>> >>>> > >> > >>>> behind >>>>> >>>> > >> > >>>> >> the >>>>> >>>> > >> > >>>> >> >> >>>> vertex in ID! Current Message ID: 1 vs. 50 >>>>> >>>> > >> > >>>> >> >> >>>> at >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> >>>> > >> > >>>> >> >>>>> >>>> > >> > >>>>> >>>> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279) >>>>> >>>> > >> > >>>> >> >> >>>> at >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> >>>> > >> > >>>> >> >> >>>>> >>>> > >> > >>>> >> >>>>> >>>> > >> > >>>> >>>>> >>>> > >> > >>>>> >>>> > >>>>> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225) >>>>> >>>> > >> > >>>> >> >> >>>> at >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> >>>> > >> > >>>> >>>>> >>>> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129) >>>>> >>>> > >> > >>>> >> >> >>>> at >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> >>>> > >> > >>>> >> >> >>>>> >>>> > >> > >>>> >> >>>>> >>>> > >> > >>>> >>>>> >>>> > >> > >>>>> >>>> > >>>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256) >>>>> >>>> > >> > >>>> >> >> >>>> at >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> >>>> > >> > >>>> >> >> >>>>> >>>> > >> > >>>> >> >>>>> >>>> > >> > >>>> >>>>> >>>> > >> > >>>>> >>>> > >> >>>>> >>>> > >>>>> >>>> >>>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >>>>> >>>> > >> > >>>> >> >> >>>> at >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> >>>> > >> > >>>> >> >> >>>>> >>>> > >> > >>>> >> >>>>> >>>> > >> > >>>> >>>>> >>>> > >> > >>>>> >>>> > >> >>>>> >>>> > >>>>> >>>> >>>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >>>>> >>>> > >> > >>>> >> >> >>>> at >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> >>>> > >> > >>>> >>>>> >>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>>> >>>> > >> > >>>> >> >> >>>> at >>>>> >>>> > >> > >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>> >>>> > >> > >>>> >> >> >>>> at >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> >>>> > >> > >>>> >> >> >>>>> >>>> > >> > >>>> >>>>> >>>> > >> > >>>>> >>>> > >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>> >>>> > >> > >>>> >> >> >>>> at >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> >>>> > >> > >>>> >>>>> >>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>>> >>>> > >> > >>>> >> >> >>>> at >>>>> >>>> > >> > >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>> >>>> > >> > >>>> >> >> >>>> at >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> >>>> > >> > >>>> >> >> >>>>> >>>> > >> > >>>> >> >>>>> >>>> > >> > >>>> >>>>> >>>> > >> > >>>>> >>>> > >> >>>>> >>>> > >>>>> >>>> >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>>>> >>>> > >> > >>>> >> >> >>>> at >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> >>>> > >> > >>>> >> >> >>>>> >>>> > >> > >>>> >> >>>>> >>>> > >> > >>>> >>>>> >>>> > >> > >>>>> >>>> > >> >>>>> >>>> > >>>>> >>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>>>> >>>> > >> > >>>> >> >> >>>> at >>>>> >>>> > >> > >>>> >> >> >>>> java.lang.Thread.run(Thread.java:722) >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> >>>> > >> > >>>> >> >> >>>> >>>>> >>>> > >> > >>>> >> >> >>>> -- >>>>> >>>> > >> > >>>> >> >> >>>> Best Regards, Edward J. Yoon >>>>> >>>> > >> > >>>> >> >> >>>> @eddieyoon >>>>> >>>> > >> > >>>> >> >> >>> >>>>> >>>> > >> > >>>> >> >> >>> >>>>> >>>> > >> > >>>> >> >> >>>>> >>>> > >> > >>>> >> >> >>>>> >>>> > >> > >>>> >> >> >>>>> >>>> > >> > >>>> >> >> -- >>>>> >>>> > >> > >>>> >> >> Best Regards, Edward J. Yoon >>>>> >>>> > >> > >>>> >> >> @eddieyoon >>>>> >>>> > >> > >>>> >> >> >>>>> >>>> > >> > >>>> >> >>>>> >>>> > >> > >>>> >> >>>>> >>>> > >> > >>>> >> >>>>> >>>> > >> > >>>> >> -- >>>>> >>>> > >> > >>>> >> Best Regards, Edward J. Yoon >>>>> >>>> > >> > >>>> >> @eddieyoon >>>>> >>>> > >> > >>>> >> >>>>> >>>> > >> > >>>> >>>>> >>>> > >> > >>>> >>>>> >>>> > >> > >>>> >>>>> >>>> > >> > >>>> -- >>>>> >>>> > >> > >>>> Best Regards, Edward J. Yoon >>>>> >>>> > >> > >>>> @eddieyoon >>>>> >>>> > >> > >>>> >>>>> >>>> > >> > >>> >>>>> >>>> > >> > >>> >>>>> >>>> > >> > >> >>>>> >>>> > >> > >>>>> >>>> > >> > >>>>> >>>> > >> > >>>>> >>>> > >> > -- >>>>> >>>> > >> > Best Regards, Edward J. Yoon >>>>> >>>> > >> > @eddieyoon >>>>> >>>> > >> > >>>>> >>>> > >> >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > -- >>>>> >>>> > Best Regards, Edward J. Yoon >>>>> >>>> > @eddieyoon >>>>> >>>> > >>>>> >>>> >>>>> >> >>>>> >> >>>>> >> >>>>> >> -- >>>>> >> Best Regards, Edward J. Yoon >>>>> >> @eddieyoon >>>>> > >>>>> > >>>>> > >>>>> > -- >>>>> > Best Regards, Edward J. Yoon >>>>> > @eddieyoon >>>>> >>>>> >>>>> >>>>> -- >>>>> Best Regards, Edward J. Yoon >>>>> @eddieyoon >>>>> >>> >>> >>> >>> -- >>> Best Regards, Edward J. Yoon >>> @eddieyoon >> >> >> >> -- >> Best Regards, Edward J. Yoon >> @eddieyoon > > > > -- > Best Regards, Edward J. Yoon > @eddieyoon -- Best Regards, Edward J. Yoon @eddieyoon
