P.S., (number of splits = number of partitions) is really confuse to me. Even though blocks number is equal to desired tasks number, data should be re-partitioned again.
On Wed, Mar 13, 2013 at 10:36 PM, Edward J. Yoon <[email protected]> wrote: > Indeed. If there are already partitioned input files (unsorted) and so > user want to skip pre-partitioning phase, it should be handled in > GraphJobRunner BSP program. Actually, I still don't know why > re-partitioned files need to be Sorted. It's only about > GraphJobRunner. > >> partitioning. (This is outside the scope of graphs. We can have a dedicated >> partitioning superstep for graph applications). > > Sorry. I don't understand exactly yet. Do you mean just a partitioning > job based on superstep API? > > By default, 100 tasks will be assigned for partitioning job. > Partitioning job will create 1,000 partitions. Thus, we can execute > the Graph job with 1,000 tasks. > > Let's assume that a input sequence file is 20GB (100 blocks). If I > want to run with 1,000 tasks, what happens? > > On Wed, Mar 13, 2013 at 9:49 PM, Suraj Menon <[email protected]> wrote: >> I am responding on this thread because of better continuity for >> conversation. We cannot expect the partitions to be sorted every time. When >> the number of splits = number of partitions and partitioning is switched >> off by user[HAMA-561], the partitions would not be sorted. Can we do this >> in loadVertices? Maybe consider feature for coupling storage in user space >> with BSP Messaging[HAMA-734] can avoid double reads and writes. This way >> partitioned or non-partitioned by partitioner, can keep vertices sorted >> with a single read and single write on every peer. >> >> Just clearing confusion if any regarding superstep injection for >> partitioning. (This is outside the scope of graphs. We can have a dedicated >> partitioning superstep for graph applications). >> Say there are x splits and y number of tasks configured by user. >> >> if x > y >> The y tasks are scheduled with x of them having each of the x splits and >> the remaining with no resource local to them. Then the partitioning >> superstep redistributes the partitions among them to create local >> partitions. Now the question is can we re-initialize a peer's input based >> on this new local part of partition? >> >> if y > x >> works as it works today. >> >> Just putting my points in brainstorming. >> >> -Suraj >> >> >> On Mon, Mar 11, 2013 at 7:39 AM, Edward J. Yoon <[email protected]>wrote: >> >>> I just filed here https://issues.apache.org/jira/browse/HAMA-744 >>> >>> On Mon, Mar 11, 2013 at 7:35 PM, Edward J. Yoon <[email protected]> >>> wrote: >>> > Additionally, >>> > >>> >> spilling queue and sorted spilling queue, can we inject the partitioning >>> >> superstep as the first superstep and use local memory? >>> > >>> > Can we execute different number of tasks per superstep? >>> > >>> > On Mon, Mar 11, 2013 at 6:56 PM, Edward J. Yoon <[email protected]> >>> wrote: >>> >>> For graph processing, the partitioned files that result from the >>> >>> partitioning job must be sorted. Currently only the partition files in >>> >> >>> >> I see. >>> >> >>> >>> For other partitionings and with regard to our superstep API, Suraj's >>> idea >>> >>> of injecting a preprocessing superstep that partitions the stuff into >>> our >>> >>> messaging system is actually the best. >>> >> >>> >> BTW, if some garbage objects can be accumulated in partitioning step, >>> >> separated partitioning job may not be bad idea. Is there some special >>> >> reason? >>> >> >>> >> On Wed, Mar 6, 2013 at 6:15 PM, Thomas Jungblut >>> >> <[email protected]> wrote: >>> >>> For graph processing, the partitioned files that result from the >>> >>> partitioning job must be sorted. Currently only the partition files in >>> >>> itself are sorted, thus more tasks result in not sorted data in the >>> >>> completed file. This only applies for the graph processing package. >>> >>> So as Suraj told, it would be much more simpler to solve this via >>> >>> messaging, once it is scalable (it will be very very scalable!). So the >>> >>> GraphJobRunner can be partitioning the stuff with a single superstep in >>> >>> setup() as it was before ages ago. The messaging must be sorted anyway >>> for >>> >>> the algorithm so this is a nice side effect and saves us the >>> partitioning >>> >>> job for graph processing. >>> >>> >>> >>> For other partitionings and with regard to our superstep API, Suraj's >>> idea >>> >>> of injecting a preprocessing superstep that partitions the stuff into >>> our >>> >>> messaging system is actually the best. >>> >>> >>> >>> >>> >>> 2013/3/6 Suraj Menon <[email protected]> >>> >>> >>> >>>> No, the partitions we write locally need not be sorted. Sorry for the >>> >>>> confusion. The Superstep injection is possible with Superstep API. >>> There >>> >>>> are few enhancements needed to make it simpler after I last worked on >>> it. >>> >>>> We can then look into partitioning superstep being executed before the >>> >>>> setup of first superstep of submitted job. I think it is feasible. >>> >>>> >>> >>>> On Tue, Mar 5, 2013 at 5:48 AM, Edward J. Yoon <[email protected] >>> >>>> >wrote: >>> >>>> >>> >>>> > > spilling queue and sorted spilling queue, can we inject the >>> >>>> partitioning >>> >>>> > > superstep as the first superstep and use local memory? >>> >>>> > >>> >>>> > Actually, I wanted to add something before calling BSP.setup() >>> method >>> >>>> > to avoid execute additional BSP job. But, in my opinion, current is >>> >>>> > enough. I think, we need to collect more experiences of input >>> >>>> > partitioning on large environments. I'll do. >>> >>>> > >>> >>>> > BTW, I still don't know why it need to be Sorted?! MR-like? >>> >>>> > >>> >>>> > On Thu, Feb 28, 2013 at 11:20 PM, Suraj Menon < >>> [email protected]> >>> >>>> > wrote: >>> >>>> > > Sorry, I am increasing the scope here to outside graph module. >>> When we >>> >>>> > have >>> >>>> > > spilling queue and sorted spilling queue, can we inject the >>> >>>> partitioning >>> >>>> > > superstep as the first superstep and use local memory? >>> >>>> > > Today we have partitioning job within a job and are creating two >>> copies >>> >>>> > of >>> >>>> > > data on HDFS. This could be really costly. Is it possible to >>> create or >>> >>>> > > redistribute the partitions on local memory and initialize the >>> record >>> >>>> > > reader there? >>> >>>> > > The user can run a separate job give in examples area to >>> explicitly >>> >>>> > > repartition the data on HDFS. The deployment question is how much >>> of >>> >>>> disk >>> >>>> > > space gets allocated for local memory usage? Would it be a safe >>> >>>> approach >>> >>>> > > with the limitations? >>> >>>> > > >>> >>>> > > -Suraj >>> >>>> > > >>> >>>> > > On Thu, Feb 28, 2013 at 7:05 AM, Thomas Jungblut >>> >>>> > > <[email protected]>wrote: >>> >>>> > > >>> >>>> > >> yes. Once Suraj added merging of sorted files we can add this to >>> the >>> >>>> > >> partitioner pretty easily. >>> >>>> > >> >>> >>>> > >> 2013/2/28 Edward J. Yoon <[email protected]> >>> >>>> > >> >>> >>>> > >> > Eh,..... btw, is re-partitioned data really necessary to be >>> Sorted? >>> >>>> > >> > >>> >>>> > >> > On Thu, Feb 28, 2013 at 7:48 PM, Thomas Jungblut >>> >>>> > >> > <[email protected]> wrote: >>> >>>> > >> > > Now I get how the partitioning works, obviously if you merge >>> n >>> >>>> > sorted >>> >>>> > >> > files >>> >>>> > >> > > by just appending to each other, this will result in totally >>> >>>> > unsorted >>> >>>> > >> > data >>> >>>> > >> > > ;-) >>> >>>> > >> > > Why didn't you solve this via messaging? >>> >>>> > >> > > >>> >>>> > >> > > 2013/2/28 Thomas Jungblut <[email protected]> >>> >>>> > >> > > >>> >>>> > >> > >> Seems that they are not correctly sorted: >>> >>>> > >> > >> >>> >>>> > >> > >> vertexID: 50 >>> >>>> > >> > >> vertexID: 52 >>> >>>> > >> > >> vertexID: 54 >>> >>>> > >> > >> vertexID: 56 >>> >>>> > >> > >> vertexID: 58 >>> >>>> > >> > >> vertexID: 61 >>> >>>> > >> > >> ... >>> >>>> > >> > >> vertexID: 78 >>> >>>> > >> > >> vertexID: 81 >>> >>>> > >> > >> vertexID: 83 >>> >>>> > >> > >> vertexID: 85 >>> >>>> > >> > >> ... >>> >>>> > >> > >> vertexID: 94 >>> >>>> > >> > >> vertexID: 96 >>> >>>> > >> > >> vertexID: 98 >>> >>>> > >> > >> vertexID: 1 >>> >>>> > >> > >> vertexID: 10 >>> >>>> > >> > >> vertexID: 12 >>> >>>> > >> > >> vertexID: 14 >>> >>>> > >> > >> vertexID: 16 >>> >>>> > >> > >> vertexID: 18 >>> >>>> > >> > >> vertexID: 21 >>> >>>> > >> > >> vertexID: 23 >>> >>>> > >> > >> vertexID: 25 >>> >>>> > >> > >> vertexID: 27 >>> >>>> > >> > >> vertexID: 29 >>> >>>> > >> > >> vertexID: 3 >>> >>>> > >> > >> >>> >>>> > >> > >> So this won't work then correctly... >>> >>>> > >> > >> >>> >>>> > >> > >> >>> >>>> > >> > >> 2013/2/28 Thomas Jungblut <[email protected]> >>> >>>> > >> > >> >>> >>>> > >> > >>> sure, have fun on your holidays. >>> >>>> > >> > >>> >>> >>>> > >> > >>> >>> >>>> > >> > >>> 2013/2/28 Edward J. Yoon <[email protected]> >>> >>>> > >> > >>> >>> >>>> > >> > >>>> Sure, but if you can fix quickly, please do. March 1 is >>> >>>> > holiday[1] >>> >>>> > >> so >>> >>>> > >> > >>>> I'll appear next week. >>> >>>> > >> > >>>> >>> >>>> > >> > >>>> 1. >>> http://en.wikipedia.org/wiki/Public_holidays_in_South_Korea >>> >>>> > >> > >>>> >>> >>>> > >> > >>>> On Thu, Feb 28, 2013 at 6:36 PM, Thomas Jungblut >>> >>>> > >> > >>>> <[email protected]> wrote: >>> >>>> > >> > >>>> > Maybe 50 is missing from the file, didn't observe if all >>> >>>> items >>> >>>> > >> were >>> >>>> > >> > >>>> added. >>> >>>> > >> > >>>> > As far as I remember, I copy/pasted the logic of the ID >>> into >>> >>>> > the >>> >>>> > >> > >>>> fastgen, >>> >>>> > >> > >>>> > want to have a look into it? >>> >>>> > >> > >>>> > >>> >>>> > >> > >>>> > 2013/2/28 Edward J. Yoon <[email protected]> >>> >>>> > >> > >>>> > >>> >>>> > >> > >>>> >> I guess, it's a bug of fastgen, when generate adjacency >>> >>>> matrix >>> >>>> > >> into >>> >>>> > >> > >>>> >> multiple files. >>> >>>> > >> > >>>> >> >>> >>>> > >> > >>>> >> On Thu, Feb 28, 2013 at 6:29 PM, Thomas Jungblut >>> >>>> > >> > >>>> >> <[email protected]> wrote: >>> >>>> > >> > >>>> >> > You have two files, are they partitioned correctly? >>> >>>> > >> > >>>> >> > >>> >>>> > >> > >>>> >> > 2013/2/28 Edward J. Yoon <[email protected]> >>> >>>> > >> > >>>> >> > >>> >>>> > >> > >>>> >> >> It looks like a bug. >>> >>>> > >> > >>>> >> >> >>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls -al >>> >>>> > >> /tmp/randomgraph/ >>> >>>> > >> > >>>> >> >> total 44 >>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096 2월 28 18:03 . >>> >>>> > >> > >>>> >> >> drwxrwxrwt 19 root root 20480 2월 28 18:04 .. >>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2243 2월 28 18:01 >>> part-00000 >>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 28 2월 28 18:01 >>> >>>> > .part-00000.crc >>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2251 2월 28 18:01 >>> part-00001 >>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 28 2월 28 18:01 >>> >>>> > .part-00001.crc >>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096 2월 28 18:03 >>> partitions >>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls -al >>> >>>> > >> > >>>> >> /tmp/randomgraph/partitions/ >>> >>>> > >> > >>>> >> >> total 24 >>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096 2월 28 18:03 . >>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096 2월 28 18:03 .. >>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2932 2월 28 18:03 >>> part-00000 >>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 32 2월 28 18:03 >>> >>>> > .part-00000.crc >>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2955 2월 28 18:03 >>> part-00001 >>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 32 2월 28 18:03 >>> >>>> > .part-00001.crc >>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ >>> >>>> > >> > >>>> >> >> >>> >>>> > >> > >>>> >> >> >>> >>>> > >> > >>>> >> >> On Thu, Feb 28, 2013 at 5:27 PM, Edward < >>> >>>> [email protected] >>> >>>> > > >>> >>>> > >> > wrote: >>> >>>> > >> > >>>> >> >> > yes i'll check again >>> >>>> > >> > >>>> >> >> > >>> >>>> > >> > >>>> >> >> > Sent from my iPhone >>> >>>> > >> > >>>> >> >> > >>> >>>> > >> > >>>> >> >> > On Feb 28, 2013, at 5:18 PM, Thomas Jungblut < >>> >>>> > >> > >>>> >> [email protected]> >>> >>>> > >> > >>>> >> >> wrote: >>> >>>> > >> > >>>> >> >> > >>> >>>> > >> > >>>> >> >> >> Can you verify an observation for me please? >>> >>>> > >> > >>>> >> >> >> >>> >>>> > >> > >>>> >> >> >> 2 files are created from fastgen, part-00000 and >>> >>>> > >> part-00001, >>> >>>> > >> > >>>> both >>> >>>> > >> > >>>> >> ~2.2kb >>> >>>> > >> > >>>> >> >> >> sized. >>> >>>> > >> > >>>> >> >> >> In the below partition directory, there is only a >>> >>>> single >>> >>>> > >> > 5.56kb >>> >>>> > >> > >>>> file. >>> >>>> > >> > >>>> >> >> >> >>> >>>> > >> > >>>> >> >> >> Is it intended for the partitioner to write a >>> single >>> >>>> > file >>> >>>> > >> if >>> >>>> > >> > you >>> >>>> > >> > >>>> >> >> configured >>> >>>> > >> > >>>> >> >> >> two? >>> >>>> > >> > >>>> >> >> >> It even reads it as a two files, strange huh? >>> >>>> > >> > >>>> >> >> >> >>> >>>> > >> > >>>> >> >> >> 2013/2/28 Thomas Jungblut < >>> [email protected]> >>> >>>> > >> > >>>> >> >> >> >>> >>>> > >> > >>>> >> >> >>> Will have a look into it. >>> >>>> > >> > >>>> >> >> >>> >>> >>>> > >> > >>>> >> >> >>> gen fastgen 100 10 /tmp/randomgraph 1 >>> >>>> > >> > >>>> >> >> >>> pagerank /tmp/randomgraph /tmp/pageout >>> >>>> > >> > >>>> >> >> >>> >>> >>>> > >> > >>>> >> >> >>> did work for me the last time I profiled, maybe >>> the >>> >>>> > >> > >>>> partitioning >>> >>>> > >> > >>>> >> >> doesn't >>> >>>> > >> > >>>> >> >> >>> partition correctly with the input or something >>> else. >>> >>>> > >> > >>>> >> >> >>> >>> >>>> > >> > >>>> >> >> >>> >>> >>>> > >> > >>>> >> >> >>> 2013/2/28 Edward J. Yoon <[email protected] >>> > >>> >>>> > >> > >>>> >> >> >>> >>> >>>> > >> > >>>> >> >> >>> Fastgen input seems not work for graph examples. >>> >>>> > >> > >>>> >> >> >>>> >>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >>> :~/workspace/hama-trunk$ >>> >>>> > >> bin/hama >>> >>>> > >> > jar >>> >>>> > >> > >>>> >> >> >>>> >>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar gen >>> >>>> > >> > fastgen >>> >>>> > >> > >>>> 100 10 >>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph 2 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:02 WARN util.NativeCodeLoader: >>> Unable >>> >>>> > to >>> >>>> > >> > load >>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your platform... >>> using >>> >>>> > >> > builtin-java >>> >>>> > >> > >>>> >> classes >>> >>>> > >> > >>>> >> >> >>>> where applicable >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO bsp.BSPJobClient: >>> Running >>> >>>> job: >>> >>>> > >> > >>>> >> >> job_localrunner_0001 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO bsp.LocalBSPRunner: >>> Setting >>> >>>> up >>> >>>> > a >>> >>>> > >> new >>> >>>> > >> > >>>> barrier >>> >>>> > >> > >>>> >> >> for 2 >>> >>>> > >> > >>>> >> >> >>>> tasks! >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>> Current >>> >>>> > >> supersteps >>> >>>> > >> > >>>> >> number: 0 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: The >>> total >>> >>>> > number >>> >>>> > >> > of >>> >>>> > >> > >>>> >> >> supersteps: 0 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>> Counters: 3 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.JobInProgress$JobCounter >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>> >>>> > SUPERSTEPS=0 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>> >>>> > >> > LAUNCHED_TASKS=2 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >>> >>>> > >> > >>>> >> TASK_OUTPUT_RECORDS=100 >>> >>>> > >> > >>>> >> >> >>>> Job Finished in 3.212 seconds >>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >>> :~/workspace/hama-trunk$ >>> >>>> > >> bin/hama >>> >>>> > >> > jar >>> >>>> > >> > >>>> >> >> >>>> examples/target/hama-examples-0.7.0-SNAPSHOT >>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT-javadoc.jar >>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT.jar >>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >>> :~/workspace/hama-trunk$ >>> >>>> > >> bin/hama >>> >>>> > >> > jar >>> >>>> > >> > >>>> >> >> >>>> >>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar >>> >>>> > pagerank >>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph /tmp/pageour >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 WARN util.NativeCodeLoader: >>> Unable >>> >>>> > to >>> >>>> > >> > load >>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your platform... >>> using >>> >>>> > >> > builtin-java >>> >>>> > >> > >>>> >> classes >>> >>>> > >> > >>>> >> >> >>>> where applicable >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: >>> Total >>> >>>> > input >>> >>>> > >> > paths >>> >>>> > >> > >>>> to >>> >>>> > >> > >>>> >> >> process >>> >>>> > >> > >>>> >> >> >>>> : 2 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: >>> Total >>> >>>> > input >>> >>>> > >> > paths >>> >>>> > >> > >>>> to >>> >>>> > >> > >>>> >> >> process >>> >>>> > >> > >>>> >> >> >>>> : 2 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO bsp.BSPJobClient: >>> Running >>> >>>> job: >>> >>>> > >> > >>>> >> >> job_localrunner_0001 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO bsp.LocalBSPRunner: >>> Setting >>> >>>> up >>> >>>> > a >>> >>>> > >> new >>> >>>> > >> > >>>> barrier >>> >>>> > >> > >>>> >> >> for 2 >>> >>>> > >> > >>>> >> >> >>>> tasks! >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>> Current >>> >>>> > >> supersteps >>> >>>> > >> > >>>> >> number: 1 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: The >>> total >>> >>>> > number >>> >>>> > >> > of >>> >>>> > >> > >>>> >> >> supersteps: 1 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>> Counters: 6 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.JobInProgress$JobCounter >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>> >>>> > SUPERSTEPS=1 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>> >>>> > >> > LAUNCHED_TASKS=2 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>> >>>> > >> > SUPERSTEP_SUM=4 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>> >>>> > >> > >>>> IO_BYTES_READ=4332 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>> >>>> > >> > >>>> TIME_IN_SYNC_MS=14 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>> >>>> > >> > >>>> TASK_INPUT_RECORDS=100 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.FileInputFormat: >>> Total >>> >>>> > input >>> >>>> > >> > paths >>> >>>> > >> > >>>> to >>> >>>> > >> > >>>> >> >> process >>> >>>> > >> > >>>> >> >> >>>> : 2 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >>> Running >>> >>>> job: >>> >>>> > >> > >>>> >> >> job_localrunner_0001 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.LocalBSPRunner: >>> Setting >>> >>>> up >>> >>>> > a >>> >>>> > >> new >>> >>>> > >> > >>>> barrier >>> >>>> > >> > >>>> >> >> for 2 >>> >>>> > >> > >>>> >> >> >>>> tasks! >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: 50 >>> >>>> > vertices >>> >>>> > >> > are >>> >>>> > >> > >>>> loaded >>> >>>> > >> > >>>> >> >> into >>> >>>> > >> > >>>> >> >> >>>> local:1 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: 50 >>> >>>> > vertices >>> >>>> > >> > are >>> >>>> > >> > >>>> loaded >>> >>>> > >> > >>>> >> >> into >>> >>>> > >> > >>>> >> >> >>>> local:0 >>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 ERROR bsp.LocalBSPRunner: >>> >>>> Exception >>> >>>> > >> > during >>> >>>> > >> > >>>> BSP >>> >>>> > >> > >>>> >> >> >>>> execution! >>> >>>> > >> > >>>> >> >> >>>> java.lang.IllegalArgumentException: Messages >>> must >>> >>>> > never >>> >>>> > >> be >>> >>>> > >> > >>>> behind >>> >>>> > >> > >>>> >> the >>> >>>> > >> > >>>> >> >> >>>> vertex in ID! Current Message ID: 1 vs. 50 >>> >>>> > >> > >>>> >> >> >>>> at >>> >>>> > >> > >>>> >> >> >>>> >>> >>>> > >> > >>>> >> >>> >>>> > >> > >>> >>>> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279) >>> >>>> > >> > >>>> >> >> >>>> at >>> >>>> > >> > >>>> >> >> >>>> >>> >>>> > >> > >>>> >> >> >>> >>>> > >> > >>>> >> >>> >>>> > >> > >>>> >>> >>>> > >> > >>> >>>> > >>> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225) >>> >>>> > >> > >>>> >> >> >>>> at >>> >>>> > >> > >>>> >> >> >>>> >>> >>>> > >> > >>>> >>> >>>> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129) >>> >>>> > >> > >>>> >> >> >>>> at >>> >>>> > >> > >>>> >> >> >>>> >>> >>>> > >> > >>>> >> >> >>> >>>> > >> > >>>> >> >>> >>>> > >> > >>>> >>> >>>> > >> > >>> >>>> > >>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256) >>> >>>> > >> > >>>> >> >> >>>> at >>> >>>> > >> > >>>> >> >> >>>> >>> >>>> > >> > >>>> >> >> >>> >>>> > >> > >>>> >> >>> >>>> > >> > >>>> >>> >>>> > >> > >>> >>>> > >> >>> >>>> > >>> >>>> >>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >>> >>>> > >> > >>>> >> >> >>>> at >>> >>>> > >> > >>>> >> >> >>>> >>> >>>> > >> > >>>> >> >> >>> >>>> > >> > >>>> >> >>> >>>> > >> > >>>> >>> >>>> > >> > >>> >>>> > >> >>> >>>> > >>> >>>> >>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >>> >>>> > >> > >>>> >> >> >>>> at >>> >>>> > >> > >>>> >> >> >>>> >>> >>>> > >> > >>>> >>> >>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>> >>>> > >> > >>>> >> >> >>>> at >>> >>>> > >> > >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >>> >>>> > >> > >>>> >> >> >>>> at >>> >>>> > >> > >>>> >> >> >>>> >>> >>>> > >> > >>>> >> >> >>> >>>> > >> > >>>> >>> >>>> > >> > >>> >>>> > >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>> >>>> > >> > >>>> >> >> >>>> at >>> >>>> > >> > >>>> >> >> >>>> >>> >>>> > >> > >>>> >>> >>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>> >>>> > >> > >>>> >> >> >>>> at >>> >>>> > >> > >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >>> >>>> > >> > >>>> >> >> >>>> at >>> >>>> > >> > >>>> >> >> >>>> >>> >>>> > >> > >>>> >> >> >>> >>>> > >> > >>>> >> >>> >>>> > >> > >>>> >>> >>>> > >> > >>> >>>> > >> >>> >>>> > >>> >>>> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>> >>>> > >> > >>>> >> >> >>>> at >>> >>>> > >> > >>>> >> >> >>>> >>> >>>> > >> > >>>> >> >> >>> >>>> > >> > >>>> >> >>> >>>> > >> > >>>> >>> >>>> > >> > >>> >>>> > >> >>> >>>> > >>> >>>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>> >>>> > >> > >>>> >> >> >>>> at java.lang.Thread.run(Thread.java:722) >>> >>>> > >> > >>>> >> >> >>>> >>> >>>> > >> > >>>> >> >> >>>> >>> >>>> > >> > >>>> >> >> >>>> -- >>> >>>> > >> > >>>> >> >> >>>> Best Regards, Edward J. Yoon >>> >>>> > >> > >>>> >> >> >>>> @eddieyoon >>> >>>> > >> > >>>> >> >> >>> >>> >>>> > >> > >>>> >> >> >>> >>> >>>> > >> > >>>> >> >> >>> >>>> > >> > >>>> >> >> >>> >>>> > >> > >>>> >> >> >>> >>>> > >> > >>>> >> >> -- >>> >>>> > >> > >>>> >> >> Best Regards, Edward J. Yoon >>> >>>> > >> > >>>> >> >> @eddieyoon >>> >>>> > >> > >>>> >> >> >>> >>>> > >> > >>>> >> >>> >>>> > >> > >>>> >> >>> >>>> > >> > >>>> >> >>> >>>> > >> > >>>> >> -- >>> >>>> > >> > >>>> >> Best Regards, Edward J. Yoon >>> >>>> > >> > >>>> >> @eddieyoon >>> >>>> > >> > >>>> >> >>> >>>> > >> > >>>> >>> >>>> > >> > >>>> >>> >>>> > >> > >>>> >>> >>>> > >> > >>>> -- >>> >>>> > >> > >>>> Best Regards, Edward J. Yoon >>> >>>> > >> > >>>> @eddieyoon >>> >>>> > >> > >>>> >>> >>>> > >> > >>> >>> >>>> > >> > >>> >>> >>>> > >> > >> >>> >>>> > >> > >>> >>>> > >> > >>> >>>> > >> > >>> >>>> > >> > -- >>> >>>> > >> > Best Regards, Edward J. Yoon >>> >>>> > >> > @eddieyoon >>> >>>> > >> > >>> >>>> > >> >>> >>>> > >>> >>>> > >>> >>>> > >>> >>>> > -- >>> >>>> > Best Regards, Edward J. Yoon >>> >>>> > @eddieyoon >>> >>>> > >>> >>>> >>> >> >>> >> >>> >> >>> >> -- >>> >> Best Regards, Edward J. Yoon >>> >> @eddieyoon >>> > >>> > >>> > >>> > -- >>> > Best Regards, Edward J. Yoon >>> > @eddieyoon >>> >>> >>> >>> -- >>> Best Regards, Edward J. Yoon >>> @eddieyoon >>> > > > > -- > Best Regards, Edward J. Yoon > @eddieyoon -- Best Regards, Edward J. Yoon @eddieyoon
