Sorry for my quick and dirty style small patches. However, we should work together in parallel. Please share here if there are some progresses.
On Thu, Mar 14, 2013 at 3:46 PM, Thomas Jungblut <[email protected]> wrote: > Hi Edward, > > before you run riot on all along the codebase, Suraj ist currently working > on that stuff- don't make it more difficult for him rebasing all his > patches the whole time. > He has the plan so that we made to make the stuff working, his part is > currently missing. So don't try to muddle arround there, it will make this > take longer than already needed. > > > > 2013/3/14 Edward J. Yoon <[email protected]> > >> Personally, I would like to solve this issue by touching >> DiskVerticesInfo. If we write sorted sub-sets of vertices into >> multiple files, we can avoid huge memory consumption. >> >> If we want to sort partitioned data using messaging system, idea >> should be collected. >> >> On Thu, Mar 14, 2013 at 10:31 AM, Edward J. Yoon <[email protected]> >> wrote: >> > Oh, now I get how iterate() works. HAMA-704 is nicely written. >> > >> > On Thu, Mar 14, 2013 at 12:02 AM, Edward J. Yoon <[email protected]> >> wrote: >> >> I'm reading changes of HAMA-704 again. As a result of adding >> >> DiskVerticesInfo, vertices list is needed to be sorted. I'm not sure >> >> but I think this approach will bring more disadvantages than >> >> advantages. >> >> >> >> On Wed, Mar 13, 2013 at 11:09 PM, Edward J. Yoon <[email protected]> >> wrote: >> >>>>>> in loadVertices? Maybe consider feature for coupling storage in >> user space >> >>>>>> with BSP Messaging[HAMA-734] can avoid double reads and writes. >> This way >> >>>>>> partitioned or non-partitioned by partitioner, can keep vertices >> sorted >> >>>>>> with a single read and single write on every peer. >> >>> >> >>> And, as I commented JIRA ticket, I think we can't use messaging system >> >>> for sorting vertices within partition files. >> >>> >> >>> On Wed, Mar 13, 2013 at 11:00 PM, Edward J. Yoon < >> [email protected]> wrote: >> >>>> P.S., (number of splits = number of partitions) is really confuse to >> >>>> me. Even though blocks number is equal to desired tasks number, data >> >>>> should be re-partitioned again. >> >>>> >> >>>> On Wed, Mar 13, 2013 at 10:36 PM, Edward J. Yoon < >> [email protected]> wrote: >> >>>>> Indeed. If there are already partitioned input files (unsorted) and >> so >> >>>>> user want to skip pre-partitioning phase, it should be handled in >> >>>>> GraphJobRunner BSP program. Actually, I still don't know why >> >>>>> re-partitioned files need to be Sorted. It's only about >> >>>>> GraphJobRunner. >> >>>>> >> >>>>>> partitioning. (This is outside the scope of graphs. We can have a >> dedicated >> >>>>>> partitioning superstep for graph applications). >> >>>>> >> >>>>> Sorry. I don't understand exactly yet. Do you mean just a >> partitioning >> >>>>> job based on superstep API? >> >>>>> >> >>>>> By default, 100 tasks will be assigned for partitioning job. >> >>>>> Partitioning job will create 1,000 partitions. Thus, we can execute >> >>>>> the Graph job with 1,000 tasks. >> >>>>> >> >>>>> Let's assume that a input sequence file is 20GB (100 blocks). If I >> >>>>> want to run with 1,000 tasks, what happens? >> >>>>> >> >>>>> On Wed, Mar 13, 2013 at 9:49 PM, Suraj Menon <[email protected]> >> wrote: >> >>>>>> I am responding on this thread because of better continuity for >> >>>>>> conversation. We cannot expect the partitions to be sorted every >> time. When >> >>>>>> the number of splits = number of partitions and partitioning is >> switched >> >>>>>> off by user[HAMA-561], the partitions would not be sorted. Can we >> do this >> >>>>>> in loadVertices? Maybe consider feature for coupling storage in >> user space >> >>>>>> with BSP Messaging[HAMA-734] can avoid double reads and writes. >> This way >> >>>>>> partitioned or non-partitioned by partitioner, can keep vertices >> sorted >> >>>>>> with a single read and single write on every peer. >> >>>>>> >> >>>>>> Just clearing confusion if any regarding superstep injection for >> >>>>>> partitioning. (This is outside the scope of graphs. We can have a >> dedicated >> >>>>>> partitioning superstep for graph applications). >> >>>>>> Say there are x splits and y number of tasks configured by user. >> >>>>>> >> >>>>>> if x > y >> >>>>>> The y tasks are scheduled with x of them having each of the x >> splits and >> >>>>>> the remaining with no resource local to them. Then the partitioning >> >>>>>> superstep redistributes the partitions among them to create local >> >>>>>> partitions. Now the question is can we re-initialize a peer's input >> based >> >>>>>> on this new local part of partition? >> >>>>>> >> >>>>>> if y > x >> >>>>>> works as it works today. >> >>>>>> >> >>>>>> Just putting my points in brainstorming. >> >>>>>> >> >>>>>> -Suraj >> >>>>>> >> >>>>>> >> >>>>>> On Mon, Mar 11, 2013 at 7:39 AM, Edward J. Yoon < >> [email protected]>wrote: >> >>>>>> >> >>>>>>> I just filed here https://issues.apache.org/jira/browse/HAMA-744 >> >>>>>>> >> >>>>>>> On Mon, Mar 11, 2013 at 7:35 PM, Edward J. Yoon < >> [email protected]> >> >>>>>>> wrote: >> >>>>>>> > Additionally, >> >>>>>>> > >> >>>>>>> >> spilling queue and sorted spilling queue, can we inject the >> partitioning >> >>>>>>> >> superstep as the first superstep and use local memory? >> >>>>>>> > >> >>>>>>> > Can we execute different number of tasks per superstep? >> >>>>>>> > >> >>>>>>> > On Mon, Mar 11, 2013 at 6:56 PM, Edward J. Yoon < >> [email protected]> >> >>>>>>> wrote: >> >>>>>>> >>> For graph processing, the partitioned files that result from >> the >> >>>>>>> >>> partitioning job must be sorted. Currently only the partition >> files in >> >>>>>>> >> >> >>>>>>> >> I see. >> >>>>>>> >> >> >>>>>>> >>> For other partitionings and with regard to our superstep API, >> Suraj's >> >>>>>>> idea >> >>>>>>> >>> of injecting a preprocessing superstep that partitions the >> stuff into >> >>>>>>> our >> >>>>>>> >>> messaging system is actually the best. >> >>>>>>> >> >> >>>>>>> >> BTW, if some garbage objects can be accumulated in partitioning >> step, >> >>>>>>> >> separated partitioning job may not be bad idea. Is there some >> special >> >>>>>>> >> reason? >> >>>>>>> >> >> >>>>>>> >> On Wed, Mar 6, 2013 at 6:15 PM, Thomas Jungblut >> >>>>>>> >> <[email protected]> wrote: >> >>>>>>> >>> For graph processing, the partitioned files that result from >> the >> >>>>>>> >>> partitioning job must be sorted. Currently only the partition >> files in >> >>>>>>> >>> itself are sorted, thus more tasks result in not sorted data >> in the >> >>>>>>> >>> completed file. This only applies for the graph processing >> package. >> >>>>>>> >>> So as Suraj told, it would be much more simpler to solve this >> via >> >>>>>>> >>> messaging, once it is scalable (it will be very very >> scalable!). So the >> >>>>>>> >>> GraphJobRunner can be partitioning the stuff with a single >> superstep in >> >>>>>>> >>> setup() as it was before ages ago. The messaging must be >> sorted anyway >> >>>>>>> for >> >>>>>>> >>> the algorithm so this is a nice side effect and saves us the >> >>>>>>> partitioning >> >>>>>>> >>> job for graph processing. >> >>>>>>> >>> >> >>>>>>> >>> For other partitionings and with regard to our superstep API, >> Suraj's >> >>>>>>> idea >> >>>>>>> >>> of injecting a preprocessing superstep that partitions the >> stuff into >> >>>>>>> our >> >>>>>>> >>> messaging system is actually the best. >> >>>>>>> >>> >> >>>>>>> >>> >> >>>>>>> >>> 2013/3/6 Suraj Menon <[email protected]> >> >>>>>>> >>> >> >>>>>>> >>>> No, the partitions we write locally need not be sorted. Sorry >> for the >> >>>>>>> >>>> confusion. The Superstep injection is possible with Superstep >> API. >> >>>>>>> There >> >>>>>>> >>>> are few enhancements needed to make it simpler after I last >> worked on >> >>>>>>> it. >> >>>>>>> >>>> We can then look into partitioning superstep being executed >> before the >> >>>>>>> >>>> setup of first superstep of submitted job. I think it is >> feasible. >> >>>>>>> >>>> >> >>>>>>> >>>> On Tue, Mar 5, 2013 at 5:48 AM, Edward J. Yoon < >> [email protected] >> >>>>>>> >>>> >wrote: >> >>>>>>> >>>> >> >>>>>>> >>>> > > spilling queue and sorted spilling queue, can we inject >> the >> >>>>>>> >>>> partitioning >> >>>>>>> >>>> > > superstep as the first superstep and use local memory? >> >>>>>>> >>>> > >> >>>>>>> >>>> > Actually, I wanted to add something before calling >> BSP.setup() >> >>>>>>> method >> >>>>>>> >>>> > to avoid execute additional BSP job. But, in my opinion, >> current is >> >>>>>>> >>>> > enough. I think, we need to collect more experiences of >> input >> >>>>>>> >>>> > partitioning on large environments. I'll do. >> >>>>>>> >>>> > >> >>>>>>> >>>> > BTW, I still don't know why it need to be Sorted?! MR-like? >> >>>>>>> >>>> > >> >>>>>>> >>>> > On Thu, Feb 28, 2013 at 11:20 PM, Suraj Menon < >> >>>>>>> [email protected]> >> >>>>>>> >>>> > wrote: >> >>>>>>> >>>> > > Sorry, I am increasing the scope here to outside graph >> module. >> >>>>>>> When we >> >>>>>>> >>>> > have >> >>>>>>> >>>> > > spilling queue and sorted spilling queue, can we inject >> the >> >>>>>>> >>>> partitioning >> >>>>>>> >>>> > > superstep as the first superstep and use local memory? >> >>>>>>> >>>> > > Today we have partitioning job within a job and are >> creating two >> >>>>>>> copies >> >>>>>>> >>>> > of >> >>>>>>> >>>> > > data on HDFS. This could be really costly. Is it possible >> to >> >>>>>>> create or >> >>>>>>> >>>> > > redistribute the partitions on local memory and >> initialize the >> >>>>>>> record >> >>>>>>> >>>> > > reader there? >> >>>>>>> >>>> > > The user can run a separate job give in examples area to >> >>>>>>> explicitly >> >>>>>>> >>>> > > repartition the data on HDFS. The deployment question is >> how much >> >>>>>>> of >> >>>>>>> >>>> disk >> >>>>>>> >>>> > > space gets allocated for local memory usage? Would it be >> a safe >> >>>>>>> >>>> approach >> >>>>>>> >>>> > > with the limitations? >> >>>>>>> >>>> > > >> >>>>>>> >>>> > > -Suraj >> >>>>>>> >>>> > > >> >>>>>>> >>>> > > On Thu, Feb 28, 2013 at 7:05 AM, Thomas Jungblut >> >>>>>>> >>>> > > <[email protected]>wrote: >> >>>>>>> >>>> > > >> >>>>>>> >>>> > >> yes. Once Suraj added merging of sorted files we can add >> this to >> >>>>>>> the >> >>>>>>> >>>> > >> partitioner pretty easily. >> >>>>>>> >>>> > >> >> >>>>>>> >>>> > >> 2013/2/28 Edward J. Yoon <[email protected]> >> >>>>>>> >>>> > >> >> >>>>>>> >>>> > >> > Eh,..... btw, is re-partitioned data really necessary >> to be >> >>>>>>> Sorted? >> >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> > On Thu, Feb 28, 2013 at 7:48 PM, Thomas Jungblut >> >>>>>>> >>>> > >> > <[email protected]> wrote: >> >>>>>>> >>>> > >> > > Now I get how the partitioning works, obviously if >> you merge >> >>>>>>> n >> >>>>>>> >>>> > sorted >> >>>>>>> >>>> > >> > files >> >>>>>>> >>>> > >> > > by just appending to each other, this will result in >> totally >> >>>>>>> >>>> > unsorted >> >>>>>>> >>>> > >> > data >> >>>>>>> >>>> > >> > > ;-) >> >>>>>>> >>>> > >> > > Why didn't you solve this via messaging? >> >>>>>>> >>>> > >> > > >> >>>>>>> >>>> > >> > > 2013/2/28 Thomas Jungblut <[email protected] >> > >> >>>>>>> >>>> > >> > > >> >>>>>>> >>>> > >> > >> Seems that they are not correctly sorted: >> >>>>>>> >>>> > >> > >> >> >>>>>>> >>>> > >> > >> vertexID: 50 >> >>>>>>> >>>> > >> > >> vertexID: 52 >> >>>>>>> >>>> > >> > >> vertexID: 54 >> >>>>>>> >>>> > >> > >> vertexID: 56 >> >>>>>>> >>>> > >> > >> vertexID: 58 >> >>>>>>> >>>> > >> > >> vertexID: 61 >> >>>>>>> >>>> > >> > >> ... >> >>>>>>> >>>> > >> > >> vertexID: 78 >> >>>>>>> >>>> > >> > >> vertexID: 81 >> >>>>>>> >>>> > >> > >> vertexID: 83 >> >>>>>>> >>>> > >> > >> vertexID: 85 >> >>>>>>> >>>> > >> > >> ... >> >>>>>>> >>>> > >> > >> vertexID: 94 >> >>>>>>> >>>> > >> > >> vertexID: 96 >> >>>>>>> >>>> > >> > >> vertexID: 98 >> >>>>>>> >>>> > >> > >> vertexID: 1 >> >>>>>>> >>>> > >> > >> vertexID: 10 >> >>>>>>> >>>> > >> > >> vertexID: 12 >> >>>>>>> >>>> > >> > >> vertexID: 14 >> >>>>>>> >>>> > >> > >> vertexID: 16 >> >>>>>>> >>>> > >> > >> vertexID: 18 >> >>>>>>> >>>> > >> > >> vertexID: 21 >> >>>>>>> >>>> > >> > >> vertexID: 23 >> >>>>>>> >>>> > >> > >> vertexID: 25 >> >>>>>>> >>>> > >> > >> vertexID: 27 >> >>>>>>> >>>> > >> > >> vertexID: 29 >> >>>>>>> >>>> > >> > >> vertexID: 3 >> >>>>>>> >>>> > >> > >> >> >>>>>>> >>>> > >> > >> So this won't work then correctly... >> >>>>>>> >>>> > >> > >> >> >>>>>>> >>>> > >> > >> >> >>>>>>> >>>> > >> > >> 2013/2/28 Thomas Jungblut < >> [email protected]> >> >>>>>>> >>>> > >> > >> >> >>>>>>> >>>> > >> > >>> sure, have fun on your holidays. >> >>>>>>> >>>> > >> > >>> >> >>>>>>> >>>> > >> > >>> >> >>>>>>> >>>> > >> > >>> 2013/2/28 Edward J. Yoon <[email protected]> >> >>>>>>> >>>> > >> > >>> >> >>>>>>> >>>> > >> > >>>> Sure, but if you can fix quickly, please do. >> March 1 is >> >>>>>>> >>>> > holiday[1] >> >>>>>>> >>>> > >> so >> >>>>>>> >>>> > >> > >>>> I'll appear next week. >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> 1. >> >>>>>>> http://en.wikipedia.org/wiki/Public_holidays_in_South_Korea >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> On Thu, Feb 28, 2013 at 6:36 PM, Thomas Jungblut >> >>>>>>> >>>> > >> > >>>> <[email protected]> wrote: >> >>>>>>> >>>> > >> > >>>> > Maybe 50 is missing from the file, didn't >> observe if all >> >>>>>>> >>>> items >> >>>>>>> >>>> > >> were >> >>>>>>> >>>> > >> > >>>> added. >> >>>>>>> >>>> > >> > >>>> > As far as I remember, I copy/pasted the logic >> of the ID >> >>>>>>> into >> >>>>>>> >>>> > the >> >>>>>>> >>>> > >> > >>>> fastgen, >> >>>>>>> >>>> > >> > >>>> > want to have a look into it? >> >>>>>>> >>>> > >> > >>>> > >> >>>>>>> >>>> > >> > >>>> > 2013/2/28 Edward J. Yoon <[email protected] >> > >> >>>>>>> >>>> > >> > >>>> > >> >>>>>>> >>>> > >> > >>>> >> I guess, it's a bug of fastgen, when generate >> adjacency >> >>>>>>> >>>> matrix >> >>>>>>> >>>> > >> into >> >>>>>>> >>>> > >> > >>>> >> multiple files. >> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> On Thu, Feb 28, 2013 at 6:29 PM, Thomas >> Jungblut >> >>>>>>> >>>> > >> > >>>> >> <[email protected]> wrote: >> >>>>>>> >>>> > >> > >>>> >> > You have two files, are they partitioned >> correctly? >> >>>>>>> >>>> > >> > >>>> >> > >> >>>>>>> >>>> > >> > >>>> >> > 2013/2/28 Edward J. Yoon < >> [email protected]> >> >>>>>>> >>>> > >> > >>>> >> > >> >>>>>>> >>>> > >> > >>>> >> >> It looks like a bug. >> >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls >> -al >> >>>>>>> >>>> > >> /tmp/randomgraph/ >> >>>>>>> >>>> > >> > >>>> >> >> total 44 >> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096 2월 28 >> 18:03 . >> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxrwt 19 root root 20480 2월 28 >> 18:04 .. >> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2243 2월 28 >> 18:01 >> >>>>>>> part-00000 >> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 28 2월 28 >> 18:01 >> >>>>>>> >>>> > .part-00000.crc >> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2251 2월 28 >> 18:01 >> >>>>>>> part-00001 >> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 28 2월 28 >> 18:01 >> >>>>>>> >>>> > .part-00001.crc >> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096 2월 28 >> 18:03 >> >>>>>>> partitions >> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls >> -al >> >>>>>>> >>>> > >> > >>>> >> /tmp/randomgraph/partitions/ >> >>>>>>> >>>> > >> > >>>> >> >> total 24 >> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096 2월 28 >> 18:03 . >> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096 2월 28 >> 18:03 .. >> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2932 2월 28 18:03 >> >>>>>>> part-00000 >> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 32 2월 28 18:03 >> >>>>>>> >>>> > .part-00000.crc >> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2955 2월 28 18:03 >> >>>>>>> part-00001 >> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 32 2월 28 18:03 >> >>>>>>> >>>> > .part-00001.crc >> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ >> >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> On Thu, Feb 28, 2013 at 5:27 PM, Edward < >> >>>>>>> >>>> [email protected] >> >>>>>>> >>>> > > >> >>>>>>> >>>> > >> > wrote: >> >>>>>>> >>>> > >> > >>>> >> >> > yes i'll check again >> >>>>>>> >>>> > >> > >>>> >> >> > >> >>>>>>> >>>> > >> > >>>> >> >> > Sent from my iPhone >> >>>>>>> >>>> > >> > >>>> >> >> > >> >>>>>>> >>>> > >> > >>>> >> >> > On Feb 28, 2013, at 5:18 PM, Thomas >> Jungblut < >> >>>>>>> >>>> > >> > >>>> >> [email protected]> >> >>>>>>> >>>> > >> > >>>> >> >> wrote: >> >>>>>>> >>>> > >> > >>>> >> >> > >> >>>>>>> >>>> > >> > >>>> >> >> >> Can you verify an observation for me >> please? >> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >> 2 files are created from fastgen, >> part-00000 and >> >>>>>>> >>>> > >> part-00001, >> >>>>>>> >>>> > >> > >>>> both >> >>>>>>> >>>> > >> > >>>> >> ~2.2kb >> >>>>>>> >>>> > >> > >>>> >> >> >> sized. >> >>>>>>> >>>> > >> > >>>> >> >> >> In the below partition directory, there >> is only a >> >>>>>>> >>>> single >> >>>>>>> >>>> > >> > 5.56kb >> >>>>>>> >>>> > >> > >>>> file. >> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >> Is it intended for the partitioner to >> write a >> >>>>>>> single >> >>>>>>> >>>> > file >> >>>>>>> >>>> > >> if >> >>>>>>> >>>> > >> > you >> >>>>>>> >>>> > >> > >>>> >> >> configured >> >>>>>>> >>>> > >> > >>>> >> >> >> two? >> >>>>>>> >>>> > >> > >>>> >> >> >> It even reads it as a two files, strange >> huh? >> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >> 2013/2/28 Thomas Jungblut < >> >>>>>>> [email protected]> >> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >>> Will have a look into it. >> >>>>>>> >>>> > >> > >>>> >> >> >>> >> >>>>>>> >>>> > >> > >>>> >> >> >>> gen fastgen 100 10 /tmp/randomgraph 1 >> >>>>>>> >>>> > >> > >>>> >> >> >>> pagerank /tmp/randomgraph /tmp/pageout >> >>>>>>> >>>> > >> > >>>> >> >> >>> >> >>>>>>> >>>> > >> > >>>> >> >> >>> did work for me the last time I >> profiled, maybe >> >>>>>>> the >> >>>>>>> >>>> > >> > >>>> partitioning >> >>>>>>> >>>> > >> > >>>> >> >> doesn't >> >>>>>>> >>>> > >> > >>>> >> >> >>> partition correctly with the input or >> something >> >>>>>>> else. >> >>>>>>> >>>> > >> > >>>> >> >> >>> >> >>>>>>> >>>> > >> > >>>> >> >> >>> >> >>>>>>> >>>> > >> > >>>> >> >> >>> 2013/2/28 Edward J. Yoon < >> [email protected] >> >>>>>>> > >> >>>>>>> >>>> > >> > >>>> >> >> >>> >> >>>>>>> >>>> > >> > >>>> >> >> >>> Fastgen input seems not work for graph >> examples. >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >> >>>>>>> :~/workspace/hama-trunk$ >> >>>>>>> >>>> > >> bin/hama >> >>>>>>> >>>> > >> > jar >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar gen >> >>>>>>> >>>> > >> > fastgen >> >>>>>>> >>>> > >> > >>>> 100 10 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph 2 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:02 WARN >> util.NativeCodeLoader: >> >>>>>>> Unable >> >>>>>>> >>>> > to >> >>>>>>> >>>> > >> > load >> >>>>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your >> platform... >> >>>>>>> using >> >>>>>>> >>>> > >> > builtin-java >> >>>>>>> >>>> > >> > >>>> >> classes >> >>>>>>> >>>> > >> > >>>> >> >> >>>> where applicable >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO >> bsp.BSPJobClient: >> >>>>>>> Running >> >>>>>>> >>>> job: >> >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO >> bsp.LocalBSPRunner: >> >>>>>>> Setting >> >>>>>>> >>>> up >> >>>>>>> >>>> > a >> >>>>>>> >>>> > >> new >> >>>>>>> >>>> > >> > >>>> barrier >> >>>>>>> >>>> > >> > >>>> >> >> for 2 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks! >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO >> bsp.BSPJobClient: >> >>>>>>> Current >> >>>>>>> >>>> > >> supersteps >> >>>>>>> >>>> > >> > >>>> >> number: 0 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO >> bsp.BSPJobClient: The >> >>>>>>> total >> >>>>>>> >>>> > number >> >>>>>>> >>>> > >> > of >> >>>>>>> >>>> > >> > >>>> >> >> supersteps: 0 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO >> bsp.BSPJobClient: >> >>>>>>> Counters: 3 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO >> bsp.BSPJobClient: >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> org.apache.hama.bsp.JobInProgress$JobCounter >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO >> bsp.BSPJobClient: >> >>>>>>> >>>> > SUPERSTEPS=0 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO >> bsp.BSPJobClient: >> >>>>>>> >>>> > >> > LAUNCHED_TASKS=2 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO >> bsp.BSPJobClient: >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO >> bsp.BSPJobClient: >> >>>>>>> >>>> > >> > >>>> >> TASK_OUTPUT_RECORDS=100 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> Job Finished in 3.212 seconds >> >>>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >> >>>>>>> :~/workspace/hama-trunk$ >> >>>>>>> >>>> > >> bin/hama >> >>>>>>> >>>> > >> > jar >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> examples/target/hama-examples-0.7.0-SNAPSHOT >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> hama-examples-0.7.0-SNAPSHOT-javadoc.jar >> >>>>>>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT.jar >> >>>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox >> >>>>>>> :~/workspace/hama-trunk$ >> >>>>>>> >>>> > >> bin/hama >> >>>>>>> >>>> > >> > jar >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar >> >>>>>>> >>>> > pagerank >> >>>>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph /tmp/pageour >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 WARN >> util.NativeCodeLoader: >> >>>>>>> Unable >> >>>>>>> >>>> > to >> >>>>>>> >>>> > >> > load >> >>>>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your >> platform... >> >>>>>>> using >> >>>>>>> >>>> > >> > builtin-java >> >>>>>>> >>>> > >> > >>>> >> classes >> >>>>>>> >>>> > >> > >>>> >> >> >>>> where applicable >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO >> bsp.FileInputFormat: >> >>>>>>> Total >> >>>>>>> >>>> > input >> >>>>>>> >>>> > >> > paths >> >>>>>>> >>>> > >> > >>>> to >> >>>>>>> >>>> > >> > >>>> >> >> process >> >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO >> bsp.FileInputFormat: >> >>>>>>> Total >> >>>>>>> >>>> > input >> >>>>>>> >>>> > >> > paths >> >>>>>>> >>>> > >> > >>>> to >> >>>>>>> >>>> > >> > >>>> >> >> process >> >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO >> bsp.BSPJobClient: >> >>>>>>> Running >> >>>>>>> >>>> job: >> >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO >> bsp.LocalBSPRunner: >> >>>>>>> Setting >> >>>>>>> >>>> up >> >>>>>>> >>>> > a >> >>>>>>> >>>> > >> new >> >>>>>>> >>>> > >> > >>>> barrier >> >>>>>>> >>>> > >> > >>>> >> >> for 2 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks! >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> bsp.BSPJobClient: >> >>>>>>> Current >> >>>>>>> >>>> > >> supersteps >> >>>>>>> >>>> > >> > >>>> >> number: 1 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> bsp.BSPJobClient: The >> >>>>>>> total >> >>>>>>> >>>> > number >> >>>>>>> >>>> > >> > of >> >>>>>>> >>>> > >> > >>>> >> >> supersteps: 1 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> bsp.BSPJobClient: >> >>>>>>> Counters: 6 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> bsp.BSPJobClient: >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> org.apache.hama.bsp.JobInProgress$JobCounter >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> bsp.BSPJobClient: >> >>>>>>> >>>> > SUPERSTEPS=1 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> bsp.BSPJobClient: >> >>>>>>> >>>> > >> > LAUNCHED_TASKS=2 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> bsp.BSPJobClient: >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> bsp.BSPJobClient: >> >>>>>>> >>>> > >> > SUPERSTEP_SUM=4 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> bsp.BSPJobClient: >> >>>>>>> >>>> > >> > >>>> IO_BYTES_READ=4332 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> bsp.BSPJobClient: >> >>>>>>> >>>> > >> > >>>> TIME_IN_SYNC_MS=14 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> bsp.BSPJobClient: >> >>>>>>> >>>> > >> > >>>> TASK_INPUT_RECORDS=100 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> bsp.FileInputFormat: >> >>>>>>> Total >> >>>>>>> >>>> > input >> >>>>>>> >>>> > >> > paths >> >>>>>>> >>>> > >> > >>>> to >> >>>>>>> >>>> > >> > >>>> >> >> process >> >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> bsp.BSPJobClient: >> >>>>>>> Running >> >>>>>>> >>>> job: >> >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> bsp.LocalBSPRunner: >> >>>>>>> Setting >> >>>>>>> >>>> up >> >>>>>>> >>>> > a >> >>>>>>> >>>> > >> new >> >>>>>>> >>>> > >> > >>>> barrier >> >>>>>>> >>>> > >> > >>>> >> >> for 2 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks! >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> graph.GraphJobRunner: 50 >> >>>>>>> >>>> > vertices >> >>>>>>> >>>> > >> > are >> >>>>>>> >>>> > >> > >>>> loaded >> >>>>>>> >>>> > >> > >>>> >> >> into >> >>>>>>> >>>> > >> > >>>> >> >> >>>> local:1 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO >> graph.GraphJobRunner: 50 >> >>>>>>> >>>> > vertices >> >>>>>>> >>>> > >> > are >> >>>>>>> >>>> > >> > >>>> loaded >> >>>>>>> >>>> > >> > >>>> >> >> into >> >>>>>>> >>>> > >> > >>>> >> >> >>>> local:0 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 ERROR >> bsp.LocalBSPRunner: >> >>>>>>> >>>> Exception >> >>>>>>> >>>> > >> > during >> >>>>>>> >>>> > >> > >>>> BSP >> >>>>>>> >>>> > >> > >>>> >> >> >>>> execution! >> >>>>>>> >>>> > >> > >>>> >> >> >>>> java.lang.IllegalArgumentException: >> Messages >> >>>>>>> must >> >>>>>>> >>>> > never >> >>>>>>> >>>> > >> be >> >>>>>>> >>>> > >> > >>>> behind >> >>>>>>> >>>> > >> > >>>> >> the >> >>>>>>> >>>> > >> > >>>> >> >> >>>> vertex in ID! Current Message ID: 1 >> vs. 50 >> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >> >>>>>>> >>>> >> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279) >> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> >>>>>>> >> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225) >> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> >> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129) >> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> >>>>>>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256) >> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> >> >>>>>>> >>>> > >> >>>>>>> >>>> >> >>>>>>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> >> >>>>>>> >>>> > >> >>>>>>> >>>> >> >>>>>>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >>>>>>> >>>> > >> > >>>> >> java.util.concurrent.FutureTask.run(FutureTask.java:166) >> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> >>>>>>> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >>>>>>> >>>> > >> > >>>> >> java.util.concurrent.FutureTask.run(FutureTask.java:166) >> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> >> >>>>>>> >>>> > >> >>>>>>> >>>> >> >>>>>>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> >> >>>>>>> >>>> > >> >>>>>>> >>>> >> >>>>>>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> java.lang.Thread.run(Thread.java:722) >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >>>>>>> >>>> > >> > >>>> >> >> >>>> -- >> >>>>>>> >>>> > >> > >>>> >> >> >>>> Best Regards, Edward J. Yoon >> >>>>>>> >>>> > >> > >>>> >> >> >>>> @eddieyoon >> >>>>>>> >>>> > >> > >>>> >> >> >>> >> >>>>>>> >>>> > >> > >>>> >> >> >>> >> >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> -- >> >>>>>>> >>>> > >> > >>>> >> >> Best Regards, Edward J. Yoon >> >>>>>>> >>>> > >> > >>>> >> >> @eddieyoon >> >>>>>>> >>>> > >> > >>>> >> >> >> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> -- >> >>>>>>> >>>> > >> > >>>> >> Best Regards, Edward J. Yoon >> >>>>>>> >>>> > >> > >>>> >> @eddieyoon >> >>>>>>> >>>> > >> > >>>> >> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>>> -- >> >>>>>>> >>>> > >> > >>>> Best Regards, Edward J. Yoon >> >>>>>>> >>>> > >> > >>>> @eddieyoon >> >>>>>>> >>>> > >> > >>>> >> >>>>>>> >>>> > >> > >>> >> >>>>>>> >>>> > >> > >>> >> >>>>>>> >>>> > >> > >> >> >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> > -- >> >>>>>>> >>>> > >> > Best Regards, Edward J. Yoon >> >>>>>>> >>>> > >> > @eddieyoon >> >>>>>>> >>>> > >> > >> >>>>>>> >>>> > >> >> >>>>>>> >>>> > >> >>>>>>> >>>> > >> >>>>>>> >>>> > >> >>>>>>> >>>> > -- >> >>>>>>> >>>> > Best Regards, Edward J. Yoon >> >>>>>>> >>>> > @eddieyoon >> >>>>>>> >>>> > >> >>>>>>> >>>> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> -- >> >>>>>>> >> Best Regards, Edward J. Yoon >> >>>>>>> >> @eddieyoon >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > -- >> >>>>>>> > Best Regards, Edward J. Yoon >> >>>>>>> > @eddieyoon >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> -- >> >>>>>>> Best Regards, Edward J. Yoon >> >>>>>>> @eddieyoon >> >>>>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> Best Regards, Edward J. Yoon >> >>>>> @eddieyoon >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Best Regards, Edward J. Yoon >> >>>> @eddieyoon >> >>> >> >>> >> >>> >> >>> -- >> >>> Best Regards, Edward J. Yoon >> >>> @eddieyoon >> >> >> >> >> >> >> >> -- >> >> Best Regards, Edward J. Yoon >> >> @eddieyoon >> > >> > >> > >> > -- >> > Best Regards, Edward J. Yoon >> > @eddieyoon >> >> >> >> -- >> Best Regards, Edward J. Yoon >> @eddieyoon >> -- Best Regards, Edward J. Yoon @eddieyoon
