You have 23 issues assigned. Why do you need to work on that? Otherwise Suraj and I branch that issues away and you can play arround.l in trunk how you like. Am 14.03.2013 09:04 schrieb "Edward J. Yoon" <[email protected]>:
> P.S., Please don't say like that. > > No decisions made yet. And if someone have a question or missed > something, you have to try to explain here. Because this is a open > source. Anyone can't say "don't touch trunk bc I'm working on it". > > On Thu, Mar 14, 2013 at 4:37 PM, Edward J. Yoon <[email protected]> > wrote: > > Sorry for my quick and dirty style small patches. > > > > However, we should work together in parallel. Please share here if > > there are some progresses. > > > > On Thu, Mar 14, 2013 at 3:46 PM, Thomas Jungblut > > <[email protected]> wrote: > >> Hi Edward, > >> > >> before you run riot on all along the codebase, Suraj ist currently > working > >> on that stuff- don't make it more difficult for him rebasing all his > >> patches the whole time. > >> He has the plan so that we made to make the stuff working, his part is > >> currently missing. So don't try to muddle arround there, it will make > this > >> take longer than already needed. > >> > >> > >> > >> 2013/3/14 Edward J. Yoon <[email protected]> > >> > >>> Personally, I would like to solve this issue by touching > >>> DiskVerticesInfo. If we write sorted sub-sets of vertices into > >>> multiple files, we can avoid huge memory consumption. > >>> > >>> If we want to sort partitioned data using messaging system, idea > >>> should be collected. > >>> > >>> On Thu, Mar 14, 2013 at 10:31 AM, Edward J. Yoon < > [email protected]> > >>> wrote: > >>> > Oh, now I get how iterate() works. HAMA-704 is nicely written. > >>> > > >>> > On Thu, Mar 14, 2013 at 12:02 AM, Edward J. Yoon < > [email protected]> > >>> wrote: > >>> >> I'm reading changes of HAMA-704 again. As a result of adding > >>> >> DiskVerticesInfo, vertices list is needed to be sorted. I'm not sure > >>> >> but I think this approach will bring more disadvantages than > >>> >> advantages. > >>> >> > >>> >> On Wed, Mar 13, 2013 at 11:09 PM, Edward J. Yoon < > [email protected]> > >>> wrote: > >>> >>>>>> in loadVertices? Maybe consider feature for coupling storage in > >>> user space > >>> >>>>>> with BSP Messaging[HAMA-734] can avoid double reads and writes. > >>> This way > >>> >>>>>> partitioned or non-partitioned by partitioner, can keep vertices > >>> sorted > >>> >>>>>> with a single read and single write on every peer. > >>> >>> > >>> >>> And, as I commented JIRA ticket, I think we can't use messaging > system > >>> >>> for sorting vertices within partition files. > >>> >>> > >>> >>> On Wed, Mar 13, 2013 at 11:00 PM, Edward J. Yoon < > >>> [email protected]> wrote: > >>> >>>> P.S., (number of splits = number of partitions) is really confuse > to > >>> >>>> me. Even though blocks number is equal to desired tasks number, > data > >>> >>>> should be re-partitioned again. > >>> >>>> > >>> >>>> On Wed, Mar 13, 2013 at 10:36 PM, Edward J. Yoon < > >>> [email protected]> wrote: > >>> >>>>> Indeed. If there are already partitioned input files (unsorted) > and > >>> so > >>> >>>>> user want to skip pre-partitioning phase, it should be handled in > >>> >>>>> GraphJobRunner BSP program. Actually, I still don't know why > >>> >>>>> re-partitioned files need to be Sorted. It's only about > >>> >>>>> GraphJobRunner. > >>> >>>>> > >>> >>>>>> partitioning. (This is outside the scope of graphs. We can have > a > >>> dedicated > >>> >>>>>> partitioning superstep for graph applications). > >>> >>>>> > >>> >>>>> Sorry. I don't understand exactly yet. Do you mean just a > >>> partitioning > >>> >>>>> job based on superstep API? > >>> >>>>> > >>> >>>>> By default, 100 tasks will be assigned for partitioning job. > >>> >>>>> Partitioning job will create 1,000 partitions. Thus, we can > execute > >>> >>>>> the Graph job with 1,000 tasks. > >>> >>>>> > >>> >>>>> Let's assume that a input sequence file is 20GB (100 blocks). If > I > >>> >>>>> want to run with 1,000 tasks, what happens? > >>> >>>>> > >>> >>>>> On Wed, Mar 13, 2013 at 9:49 PM, Suraj Menon < > [email protected]> > >>> wrote: > >>> >>>>>> I am responding on this thread because of better continuity for > >>> >>>>>> conversation. We cannot expect the partitions to be sorted every > >>> time. When > >>> >>>>>> the number of splits = number of partitions and partitioning is > >>> switched > >>> >>>>>> off by user[HAMA-561], the partitions would not be sorted. Can > we > >>> do this > >>> >>>>>> in loadVertices? Maybe consider feature for coupling storage in > >>> user space > >>> >>>>>> with BSP Messaging[HAMA-734] can avoid double reads and writes. > >>> This way > >>> >>>>>> partitioned or non-partitioned by partitioner, can keep vertices > >>> sorted > >>> >>>>>> with a single read and single write on every peer. > >>> >>>>>> > >>> >>>>>> Just clearing confusion if any regarding superstep injection for > >>> >>>>>> partitioning. (This is outside the scope of graphs. We can have > a > >>> dedicated > >>> >>>>>> partitioning superstep for graph applications). > >>> >>>>>> Say there are x splits and y number of tasks configured by user. > >>> >>>>>> > >>> >>>>>> if x > y > >>> >>>>>> The y tasks are scheduled with x of them having each of the x > >>> splits and > >>> >>>>>> the remaining with no resource local to them. Then the > partitioning > >>> >>>>>> superstep redistributes the partitions among them to create > local > >>> >>>>>> partitions. Now the question is can we re-initialize a peer's > input > >>> based > >>> >>>>>> on this new local part of partition? > >>> >>>>>> > >>> >>>>>> if y > x > >>> >>>>>> works as it works today. > >>> >>>>>> > >>> >>>>>> Just putting my points in brainstorming. > >>> >>>>>> > >>> >>>>>> -Suraj > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> On Mon, Mar 11, 2013 at 7:39 AM, Edward J. Yoon < > >>> [email protected]>wrote: > >>> >>>>>> > >>> >>>>>>> I just filed here > https://issues.apache.org/jira/browse/HAMA-744 > >>> >>>>>>> > >>> >>>>>>> On Mon, Mar 11, 2013 at 7:35 PM, Edward J. Yoon < > >>> [email protected]> > >>> >>>>>>> wrote: > >>> >>>>>>> > Additionally, > >>> >>>>>>> > > >>> >>>>>>> >> spilling queue and sorted spilling queue, can we inject the > >>> partitioning > >>> >>>>>>> >> superstep as the first superstep and use local memory? > >>> >>>>>>> > > >>> >>>>>>> > Can we execute different number of tasks per superstep? > >>> >>>>>>> > > >>> >>>>>>> > On Mon, Mar 11, 2013 at 6:56 PM, Edward J. Yoon < > >>> [email protected]> > >>> >>>>>>> wrote: > >>> >>>>>>> >>> For graph processing, the partitioned files that result > from > >>> the > >>> >>>>>>> >>> partitioning job must be sorted. Currently only the > partition > >>> files in > >>> >>>>>>> >> > >>> >>>>>>> >> I see. > >>> >>>>>>> >> > >>> >>>>>>> >>> For other partitionings and with regard to our superstep > API, > >>> Suraj's > >>> >>>>>>> idea > >>> >>>>>>> >>> of injecting a preprocessing superstep that partitions the > >>> stuff into > >>> >>>>>>> our > >>> >>>>>>> >>> messaging system is actually the best. > >>> >>>>>>> >> > >>> >>>>>>> >> BTW, if some garbage objects can be accumulated in > partitioning > >>> step, > >>> >>>>>>> >> separated partitioning job may not be bad idea. Is there > some > >>> special > >>> >>>>>>> >> reason? > >>> >>>>>>> >> > >>> >>>>>>> >> On Wed, Mar 6, 2013 at 6:15 PM, Thomas Jungblut > >>> >>>>>>> >> <[email protected]> wrote: > >>> >>>>>>> >>> For graph processing, the partitioned files that result > from > >>> the > >>> >>>>>>> >>> partitioning job must be sorted. Currently only the > partition > >>> files in > >>> >>>>>>> >>> itself are sorted, thus more tasks result in not sorted > data > >>> in the > >>> >>>>>>> >>> completed file. This only applies for the graph processing > >>> package. > >>> >>>>>>> >>> So as Suraj told, it would be much more simpler to solve > this > >>> via > >>> >>>>>>> >>> messaging, once it is scalable (it will be very very > >>> scalable!). So the > >>> >>>>>>> >>> GraphJobRunner can be partitioning the stuff with a single > >>> superstep in > >>> >>>>>>> >>> setup() as it was before ages ago. The messaging must be > >>> sorted anyway > >>> >>>>>>> for > >>> >>>>>>> >>> the algorithm so this is a nice side effect and saves us > the > >>> >>>>>>> partitioning > >>> >>>>>>> >>> job for graph processing. > >>> >>>>>>> >>> > >>> >>>>>>> >>> For other partitionings and with regard to our superstep > API, > >>> Suraj's > >>> >>>>>>> idea > >>> >>>>>>> >>> of injecting a preprocessing superstep that partitions the > >>> stuff into > >>> >>>>>>> our > >>> >>>>>>> >>> messaging system is actually the best. > >>> >>>>>>> >>> > >>> >>>>>>> >>> > >>> >>>>>>> >>> 2013/3/6 Suraj Menon <[email protected]> > >>> >>>>>>> >>> > >>> >>>>>>> >>>> No, the partitions we write locally need not be sorted. > Sorry > >>> for the > >>> >>>>>>> >>>> confusion. The Superstep injection is possible with > Superstep > >>> API. > >>> >>>>>>> There > >>> >>>>>>> >>>> are few enhancements needed to make it simpler after I > last > >>> worked on > >>> >>>>>>> it. > >>> >>>>>>> >>>> We can then look into partitioning superstep being > executed > >>> before the > >>> >>>>>>> >>>> setup of first superstep of submitted job. I think it is > >>> feasible. > >>> >>>>>>> >>>> > >>> >>>>>>> >>>> On Tue, Mar 5, 2013 at 5:48 AM, Edward J. Yoon < > >>> [email protected] > >>> >>>>>>> >>>> >wrote: > >>> >>>>>>> >>>> > >>> >>>>>>> >>>> > > spilling queue and sorted spilling queue, can we > inject > >>> the > >>> >>>>>>> >>>> partitioning > >>> >>>>>>> >>>> > > superstep as the first superstep and use local memory? > >>> >>>>>>> >>>> > > >>> >>>>>>> >>>> > Actually, I wanted to add something before calling > >>> BSP.setup() > >>> >>>>>>> method > >>> >>>>>>> >>>> > to avoid execute additional BSP job. But, in my opinion, > >>> current is > >>> >>>>>>> >>>> > enough. I think, we need to collect more experiences of > >>> input > >>> >>>>>>> >>>> > partitioning on large environments. I'll do. > >>> >>>>>>> >>>> > > >>> >>>>>>> >>>> > BTW, I still don't know why it need to be Sorted?! > MR-like? > >>> >>>>>>> >>>> > > >>> >>>>>>> >>>> > On Thu, Feb 28, 2013 at 11:20 PM, Suraj Menon < > >>> >>>>>>> [email protected]> > >>> >>>>>>> >>>> > wrote: > >>> >>>>>>> >>>> > > Sorry, I am increasing the scope here to outside graph > >>> module. > >>> >>>>>>> When we > >>> >>>>>>> >>>> > have > >>> >>>>>>> >>>> > > spilling queue and sorted spilling queue, can we > inject > >>> the > >>> >>>>>>> >>>> partitioning > >>> >>>>>>> >>>> > > superstep as the first superstep and use local memory? > >>> >>>>>>> >>>> > > Today we have partitioning job within a job and are > >>> creating two > >>> >>>>>>> copies > >>> >>>>>>> >>>> > of > >>> >>>>>>> >>>> > > data on HDFS. This could be really costly. Is it > possible > >>> to > >>> >>>>>>> create or > >>> >>>>>>> >>>> > > redistribute the partitions on local memory and > >>> initialize the > >>> >>>>>>> record > >>> >>>>>>> >>>> > > reader there? > >>> >>>>>>> >>>> > > The user can run a separate job give in examples area > to > >>> >>>>>>> explicitly > >>> >>>>>>> >>>> > > repartition the data on HDFS. The deployment question > is > >>> how much > >>> >>>>>>> of > >>> >>>>>>> >>>> disk > >>> >>>>>>> >>>> > > space gets allocated for local memory usage? Would it > be > >>> a safe > >>> >>>>>>> >>>> approach > >>> >>>>>>> >>>> > > with the limitations? > >>> >>>>>>> >>>> > > > >>> >>>>>>> >>>> > > -Suraj > >>> >>>>>>> >>>> > > > >>> >>>>>>> >>>> > > On Thu, Feb 28, 2013 at 7:05 AM, Thomas Jungblut > >>> >>>>>>> >>>> > > <[email protected]>wrote: > >>> >>>>>>> >>>> > > > >>> >>>>>>> >>>> > >> yes. Once Suraj added merging of sorted files we can > add > >>> this to > >>> >>>>>>> the > >>> >>>>>>> >>>> > >> partitioner pretty easily. > >>> >>>>>>> >>>> > >> > >>> >>>>>>> >>>> > >> 2013/2/28 Edward J. Yoon <[email protected]> > >>> >>>>>>> >>>> > >> > >>> >>>>>>> >>>> > >> > Eh,..... btw, is re-partitioned data really > necessary > >>> to be > >>> >>>>>>> Sorted? > >>> >>>>>>> >>>> > >> > > >>> >>>>>>> >>>> > >> > On Thu, Feb 28, 2013 at 7:48 PM, Thomas Jungblut > >>> >>>>>>> >>>> > >> > <[email protected]> wrote: > >>> >>>>>>> >>>> > >> > > Now I get how the partitioning works, obviously > if > >>> you merge > >>> >>>>>>> n > >>> >>>>>>> >>>> > sorted > >>> >>>>>>> >>>> > >> > files > >>> >>>>>>> >>>> > >> > > by just appending to each other, this will > result in > >>> totally > >>> >>>>>>> >>>> > unsorted > >>> >>>>>>> >>>> > >> > data > >>> >>>>>>> >>>> > >> > > ;-) > >>> >>>>>>> >>>> > >> > > Why didn't you solve this via messaging? > >>> >>>>>>> >>>> > >> > > > >>> >>>>>>> >>>> > >> > > 2013/2/28 Thomas Jungblut < > [email protected] > >>> > > >>> >>>>>>> >>>> > >> > > > >>> >>>>>>> >>>> > >> > >> Seems that they are not correctly sorted: > >>> >>>>>>> >>>> > >> > >> > >>> >>>>>>> >>>> > >> > >> vertexID: 50 > >>> >>>>>>> >>>> > >> > >> vertexID: 52 > >>> >>>>>>> >>>> > >> > >> vertexID: 54 > >>> >>>>>>> >>>> > >> > >> vertexID: 56 > >>> >>>>>>> >>>> > >> > >> vertexID: 58 > >>> >>>>>>> >>>> > >> > >> vertexID: 61 > >>> >>>>>>> >>>> > >> > >> ... > >>> >>>>>>> >>>> > >> > >> vertexID: 78 > >>> >>>>>>> >>>> > >> > >> vertexID: 81 > >>> >>>>>>> >>>> > >> > >> vertexID: 83 > >>> >>>>>>> >>>> > >> > >> vertexID: 85 > >>> >>>>>>> >>>> > >> > >> ... > >>> >>>>>>> >>>> > >> > >> vertexID: 94 > >>> >>>>>>> >>>> > >> > >> vertexID: 96 > >>> >>>>>>> >>>> > >> > >> vertexID: 98 > >>> >>>>>>> >>>> > >> > >> vertexID: 1 > >>> >>>>>>> >>>> > >> > >> vertexID: 10 > >>> >>>>>>> >>>> > >> > >> vertexID: 12 > >>> >>>>>>> >>>> > >> > >> vertexID: 14 > >>> >>>>>>> >>>> > >> > >> vertexID: 16 > >>> >>>>>>> >>>> > >> > >> vertexID: 18 > >>> >>>>>>> >>>> > >> > >> vertexID: 21 > >>> >>>>>>> >>>> > >> > >> vertexID: 23 > >>> >>>>>>> >>>> > >> > >> vertexID: 25 > >>> >>>>>>> >>>> > >> > >> vertexID: 27 > >>> >>>>>>> >>>> > >> > >> vertexID: 29 > >>> >>>>>>> >>>> > >> > >> vertexID: 3 > >>> >>>>>>> >>>> > >> > >> > >>> >>>>>>> >>>> > >> > >> So this won't work then correctly... > >>> >>>>>>> >>>> > >> > >> > >>> >>>>>>> >>>> > >> > >> > >>> >>>>>>> >>>> > >> > >> 2013/2/28 Thomas Jungblut < > >>> [email protected]> > >>> >>>>>>> >>>> > >> > >> > >>> >>>>>>> >>>> > >> > >>> sure, have fun on your holidays. > >>> >>>>>>> >>>> > >> > >>> > >>> >>>>>>> >>>> > >> > >>> > >>> >>>>>>> >>>> > >> > >>> 2013/2/28 Edward J. Yoon < > [email protected]> > >>> >>>>>>> >>>> > >> > >>> > >>> >>>>>>> >>>> > >> > >>>> Sure, but if you can fix quickly, please do. > >>> March 1 is > >>> >>>>>>> >>>> > holiday[1] > >>> >>>>>>> >>>> > >> so > >>> >>>>>>> >>>> > >> > >>>> I'll appear next week. > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > >> > >>>> 1. > >>> >>>>>>> http://en.wikipedia.org/wiki/Public_holidays_in_South_Korea > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > >> > >>>> On Thu, Feb 28, 2013 at 6:36 PM, Thomas > Jungblut > >>> >>>>>>> >>>> > >> > >>>> <[email protected]> wrote: > >>> >>>>>>> >>>> > >> > >>>> > Maybe 50 is missing from the file, didn't > >>> observe if all > >>> >>>>>>> >>>> items > >>> >>>>>>> >>>> > >> were > >>> >>>>>>> >>>> > >> > >>>> added. > >>> >>>>>>> >>>> > >> > >>>> > As far as I remember, I copy/pasted the > logic > >>> of the ID > >>> >>>>>>> into > >>> >>>>>>> >>>> > the > >>> >>>>>>> >>>> > >> > >>>> fastgen, > >>> >>>>>>> >>>> > >> > >>>> > want to have a look into it? > >>> >>>>>>> >>>> > >> > >>>> > > >>> >>>>>>> >>>> > >> > >>>> > 2013/2/28 Edward J. Yoon < > [email protected] > >>> > > >>> >>>>>>> >>>> > >> > >>>> > > >>> >>>>>>> >>>> > >> > >>>> >> I guess, it's a bug of fastgen, when > generate > >>> adjacency > >>> >>>>>>> >>>> matrix > >>> >>>>>>> >>>> > >> into > >>> >>>>>>> >>>> > >> > >>>> >> multiple files. > >>> >>>>>>> >>>> > >> > >>>> >> > >>> >>>>>>> >>>> > >> > >>>> >> On Thu, Feb 28, 2013 at 6:29 PM, Thomas > >>> Jungblut > >>> >>>>>>> >>>> > >> > >>>> >> <[email protected]> wrote: > >>> >>>>>>> >>>> > >> > >>>> >> > You have two files, are they partitioned > >>> correctly? > >>> >>>>>>> >>>> > >> > >>>> >> > > >>> >>>>>>> >>>> > >> > >>>> >> > 2013/2/28 Edward J. Yoon < > >>> [email protected]> > >>> >>>>>>> >>>> > >> > >>>> >> > > >>> >>>>>>> >>>> > >> > >>>> >> >> It looks like a bug. > >>> >>>>>>> >>>> > >> > >>>> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ > ls > >>> -al > >>> >>>>>>> >>>> > >> /tmp/randomgraph/ > >>> >>>>>>> >>>> > >> > >>>> >> >> total 44 > >>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096 2월 28 > >>> 18:03 . > >>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxrwt 19 root root 20480 2월 28 > >>> 18:04 .. > >>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2243 2월 28 > >>> 18:01 > >>> >>>>>>> part-00000 > >>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 28 2월 28 > >>> 18:01 > >>> >>>>>>> >>>> > .part-00000.crc > >>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2251 2월 28 > >>> 18:01 > >>> >>>>>>> part-00001 > >>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 28 2월 28 > >>> 18:01 > >>> >>>>>>> >>>> > .part-00001.crc > >>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096 2월 28 > >>> 18:03 > >>> >>>>>>> partitions > >>> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ > ls > >>> -al > >>> >>>>>>> >>>> > >> > >>>> >> /tmp/randomgraph/partitions/ > >>> >>>>>>> >>>> > >> > >>>> >> >> total 24 > >>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096 2월 28 > >>> 18:03 . > >>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096 2월 28 > >>> 18:03 .. > >>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2932 2월 28 > 18:03 > >>> >>>>>>> part-00000 > >>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 32 2월 28 > 18:03 > >>> >>>>>>> >>>> > .part-00000.crc > >>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2955 2월 28 > 18:03 > >>> >>>>>>> part-00001 > >>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward 32 2월 28 > 18:03 > >>> >>>>>>> >>>> > .part-00001.crc > >>> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ > >>> >>>>>>> >>>> > >> > >>>> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> >> On Thu, Feb 28, 2013 at 5:27 PM, Edward > < > >>> >>>>>>> >>>> [email protected] > >>> >>>>>>> >>>> > > > >>> >>>>>>> >>>> > >> > wrote: > >>> >>>>>>> >>>> > >> > >>>> >> >> > yes i'll check again > >>> >>>>>>> >>>> > >> > >>>> >> >> > > >>> >>>>>>> >>>> > >> > >>>> >> >> > Sent from my iPhone > >>> >>>>>>> >>>> > >> > >>>> >> >> > > >>> >>>>>>> >>>> > >> > >>>> >> >> > On Feb 28, 2013, at 5:18 PM, Thomas > >>> Jungblut < > >>> >>>>>>> >>>> > >> > >>>> >> [email protected]> > >>> >>>>>>> >>>> > >> > >>>> >> >> wrote: > >>> >>>>>>> >>>> > >> > >>>> >> >> > > >>> >>>>>>> >>>> > >> > >>>> >> >> >> Can you verify an observation for me > >>> please? > >>> >>>>>>> >>>> > >> > >>>> >> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> >> >> 2 files are created from fastgen, > >>> part-00000 and > >>> >>>>>>> >>>> > >> part-00001, > >>> >>>>>>> >>>> > >> > >>>> both > >>> >>>>>>> >>>> > >> > >>>> >> ~2.2kb > >>> >>>>>>> >>>> > >> > >>>> >> >> >> sized. > >>> >>>>>>> >>>> > >> > >>>> >> >> >> In the below partition directory, > there > >>> is only a > >>> >>>>>>> >>>> single > >>> >>>>>>> >>>> > >> > 5.56kb > >>> >>>>>>> >>>> > >> > >>>> file. > >>> >>>>>>> >>>> > >> > >>>> >> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> >> >> Is it intended for the partitioner to > >>> write a > >>> >>>>>>> single > >>> >>>>>>> >>>> > file > >>> >>>>>>> >>>> > >> if > >>> >>>>>>> >>>> > >> > you > >>> >>>>>>> >>>> > >> > >>>> >> >> configured > >>> >>>>>>> >>>> > >> > >>>> >> >> >> two? > >>> >>>>>>> >>>> > >> > >>>> >> >> >> It even reads it as a two files, > strange > >>> huh? > >>> >>>>>>> >>>> > >> > >>>> >> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> >> >> 2013/2/28 Thomas Jungblut < > >>> >>>>>>> [email protected]> > >>> >>>>>>> >>>> > >> > >>>> >> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> >> >>> Will have a look into it. > >>> >>>>>>> >>>> > >> > >>>> >> >> >>> > >>> >>>>>>> >>>> > >> > >>>> >> >> >>> gen fastgen 100 10 /tmp/randomgraph > 1 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>> pagerank /tmp/randomgraph > /tmp/pageout > >>> >>>>>>> >>>> > >> > >>>> >> >> >>> > >>> >>>>>>> >>>> > >> > >>>> >> >> >>> did work for me the last time I > >>> profiled, maybe > >>> >>>>>>> the > >>> >>>>>>> >>>> > >> > >>>> partitioning > >>> >>>>>>> >>>> > >> > >>>> >> >> doesn't > >>> >>>>>>> >>>> > >> > >>>> >> >> >>> partition correctly with the input > or > >>> something > >>> >>>>>>> else. > >>> >>>>>>> >>>> > >> > >>>> >> >> >>> > >>> >>>>>>> >>>> > >> > >>>> >> >> >>> > >>> >>>>>>> >>>> > >> > >>>> >> >> >>> 2013/2/28 Edward J. Yoon < > >>> [email protected] > >>> >>>>>>> > > >>> >>>>>>> >>>> > >> > >>>> >> >> >>> > >>> >>>>>>> >>>> > >> > >>>> >> >> >>> Fastgen input seems not work for > graph > >>> examples. > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox > >>> >>>>>>> :~/workspace/hama-trunk$ > >>> >>>>>>> >>>> > >> bin/hama > >>> >>>>>>> >>>> > >> > jar > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar gen > >>> >>>>>>> >>>> > >> > fastgen > >>> >>>>>>> >>>> > >> > >>>> 100 10 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph 2 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:02 WARN > >>> util.NativeCodeLoader: > >>> >>>>>>> Unable > >>> >>>>>>> >>>> > to > >>> >>>>>>> >>>> > >> > load > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your > >>> platform... > >>> >>>>>>> using > >>> >>>>>>> >>>> > >> > builtin-java > >>> >>>>>>> >>>> > >> > >>>> >> classes > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> where applicable > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> Running > >>> >>>>>>> >>>> job: > >>> >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO > >>> bsp.LocalBSPRunner: > >>> >>>>>>> Setting > >>> >>>>>>> >>>> up > >>> >>>>>>> >>>> > a > >>> >>>>>>> >>>> > >> new > >>> >>>>>>> >>>> > >> > >>>> barrier > >>> >>>>>>> >>>> > >> > >>>> >> >> for 2 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks! > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> Current > >>> >>>>>>> >>>> > >> supersteps > >>> >>>>>>> >>>> > >> > >>>> >> number: 0 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO > >>> bsp.BSPJobClient: The > >>> >>>>>>> total > >>> >>>>>>> >>>> > number > >>> >>>>>>> >>>> > >> > of > >>> >>>>>>> >>>> > >> > >>>> >> >> supersteps: 0 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> Counters: 3 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> org.apache.hama.bsp.JobInProgress$JobCounter > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> >>>> > SUPERSTEPS=0 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> >>>> > >> > LAUNCHED_TASKS=2 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> >>>> > >> > >>>> >> TASK_OUTPUT_RECORDS=100 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> Job Finished in 3.212 seconds > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox > >>> >>>>>>> :~/workspace/hama-trunk$ > >>> >>>>>>> >>>> > >> bin/hama > >>> >>>>>>> >>>> > >> > jar > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> examples/target/hama-examples-0.7.0-SNAPSHOT > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> hama-examples-0.7.0-SNAPSHOT-javadoc.jar > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT.jar > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox > >>> >>>>>>> :~/workspace/hama-trunk$ > >>> >>>>>>> >>>> > >> bin/hama > >>> >>>>>>> >>>> > >> > jar > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar > >>> >>>>>>> >>>> > pagerank > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph /tmp/pageour > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 WARN > >>> util.NativeCodeLoader: > >>> >>>>>>> Unable > >>> >>>>>>> >>>> > to > >>> >>>>>>> >>>> > >> > load > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your > >>> platform... > >>> >>>>>>> using > >>> >>>>>>> >>>> > >> > builtin-java > >>> >>>>>>> >>>> > >> > >>>> >> classes > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> where applicable > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO > >>> bsp.FileInputFormat: > >>> >>>>>>> Total > >>> >>>>>>> >>>> > input > >>> >>>>>>> >>>> > >> > paths > >>> >>>>>>> >>>> > >> > >>>> to > >>> >>>>>>> >>>> > >> > >>>> >> >> process > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO > >>> bsp.FileInputFormat: > >>> >>>>>>> Total > >>> >>>>>>> >>>> > input > >>> >>>>>>> >>>> > >> > paths > >>> >>>>>>> >>>> > >> > >>>> to > >>> >>>>>>> >>>> > >> > >>>> >> >> process > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> Running > >>> >>>>>>> >>>> job: > >>> >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO > >>> bsp.LocalBSPRunner: > >>> >>>>>>> Setting > >>> >>>>>>> >>>> up > >>> >>>>>>> >>>> > a > >>> >>>>>>> >>>> > >> new > >>> >>>>>>> >>>> > >> > >>>> barrier > >>> >>>>>>> >>>> > >> > >>>> >> >> for 2 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks! > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> Current > >>> >>>>>>> >>>> > >> supersteps > >>> >>>>>>> >>>> > >> > >>>> >> number: 1 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> bsp.BSPJobClient: The > >>> >>>>>>> total > >>> >>>>>>> >>>> > number > >>> >>>>>>> >>>> > >> > of > >>> >>>>>>> >>>> > >> > >>>> >> >> supersteps: 1 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> Counters: 6 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> org.apache.hama.bsp.JobInProgress$JobCounter > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> >>>> > SUPERSTEPS=1 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> >>>> > >> > LAUNCHED_TASKS=2 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> >>>> > >> > SUPERSTEP_SUM=4 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> >>>> > >> > >>>> IO_BYTES_READ=4332 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> >>>> > >> > >>>> TIME_IN_SYNC_MS=14 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> >>>> > >> > >>>> TASK_INPUT_RECORDS=100 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> bsp.FileInputFormat: > >>> >>>>>>> Total > >>> >>>>>>> >>>> > input > >>> >>>>>>> >>>> > >> > paths > >>> >>>>>>> >>>> > >> > >>>> to > >>> >>>>>>> >>>> > >> > >>>> >> >> process > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> bsp.BSPJobClient: > >>> >>>>>>> Running > >>> >>>>>>> >>>> job: > >>> >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> bsp.LocalBSPRunner: > >>> >>>>>>> Setting > >>> >>>>>>> >>>> up > >>> >>>>>>> >>>> > a > >>> >>>>>>> >>>> > >> new > >>> >>>>>>> >>>> > >> > >>>> barrier > >>> >>>>>>> >>>> > >> > >>>> >> >> for 2 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks! > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> graph.GraphJobRunner: 50 > >>> >>>>>>> >>>> > vertices > >>> >>>>>>> >>>> > >> > are > >>> >>>>>>> >>>> > >> > >>>> loaded > >>> >>>>>>> >>>> > >> > >>>> >> >> into > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> local:1 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO > >>> graph.GraphJobRunner: 50 > >>> >>>>>>> >>>> > vertices > >>> >>>>>>> >>>> > >> > are > >>> >>>>>>> >>>> > >> > >>>> loaded > >>> >>>>>>> >>>> > >> > >>>> >> >> into > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> local:0 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 ERROR > >>> bsp.LocalBSPRunner: > >>> >>>>>>> >>>> Exception > >>> >>>>>>> >>>> > >> > during > >>> >>>>>>> >>>> > >> > >>>> BSP > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> execution! > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> java.lang.IllegalArgumentException: > >>> Messages > >>> >>>>>>> must > >>> >>>>>>> >>>> > never > >>> >>>>>>> >>>> > >> be > >>> >>>>>>> >>>> > >> > >>>> behind > >>> >>>>>>> >>>> > >> > >>>> >> the > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> vertex in ID! Current Message ID: 1 > >>> vs. 50 > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> >>>> > >> > >>>> >> > >>> >>>>>>> >>>> > >> > > >>> >>>>>>> >>>> > >>> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279) > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> >>>> > >> > >>>> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > >> > > >>> >>>>>>> >>>> > > >>> >>>>>>> > >>> > org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225) > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > >>> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129) > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> >>>> > >> > >>>> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > >> > > >>> >>>>>>> >>>> > > >>> >>>>>>> > >>> > org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256) > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> >>>> > >> > >>>> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > >> > > >>> >>>>>>> >>>> > >> > >>> >>>>>>> >>>> > > >>> >>>>>>> >>>> > >>> >>>>>>> > >>> > org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> >>>> > >> > >>>> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > >> > > >>> >>>>>>> >>>> > >> > >>> >>>>>>> >>>> > > >>> >>>>>>> >>>> > >>> >>>>>>> > >>> > org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > > >>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at > >>> >>>>>>> >>>> > >> > >>>> > >>> java.util.concurrent.FutureTask.run(FutureTask.java:166) > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> >>>> > >> > >>>> >> >> > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > >> > > >>> >>>>>>> >>>> > > >>> >>>>>>> > >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > > >>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at > >>> >>>>>>> >>>> > >> > >>>> > >>> java.util.concurrent.FutureTask.run(FutureTask.java:166) > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> >>>> > >> > >>>> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > >> > > >>> >>>>>>> >>>> > >> > >>> >>>>>>> >>>> > > >>> >>>>>>> >>>> > >>> >>>>>>> > >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> >>>> > >> > >>>> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > >> > > >>> >>>>>>> >>>> > >> > >>> >>>>>>> >>>> > > >>> >>>>>>> >>>> > >>> >>>>>>> > >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at > >>> java.lang.Thread.run(Thread.java:722) > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> -- > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> Best Regards, Edward J. Yoon > >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> @eddieyoon > >>> >>>>>>> >>>> > >> > >>>> >> >> >>> > >>> >>>>>>> >>>> > >> > >>>> >> >> >>> > >>> >>>>>>> >>>> > >> > >>>> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> >> -- > >>> >>>>>>> >>>> > >> > >>>> >> >> Best Regards, Edward J. Yoon > >>> >>>>>>> >>>> > >> > >>>> >> >> @eddieyoon > >>> >>>>>>> >>>> > >> > >>>> >> >> > >>> >>>>>>> >>>> > >> > >>>> >> > >>> >>>>>>> >>>> > >> > >>>> >> > >>> >>>>>>> >>>> > >> > >>>> >> > >>> >>>>>>> >>>> > >> > >>>> >> -- > >>> >>>>>>> >>>> > >> > >>>> >> Best Regards, Edward J. Yoon > >>> >>>>>>> >>>> > >> > >>>> >> @eddieyoon > >>> >>>>>>> >>>> > >> > >>>> >> > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > >> > >>>> -- > >>> >>>>>>> >>>> > >> > >>>> Best Regards, Edward J. Yoon > >>> >>>>>>> >>>> > >> > >>>> @eddieyoon > >>> >>>>>>> >>>> > >> > >>>> > >>> >>>>>>> >>>> > >> > >>> > >>> >>>>>>> >>>> > >> > >>> > >>> >>>>>>> >>>> > >> > >> > >>> >>>>>>> >>>> > >> > > >>> >>>>>>> >>>> > >> > > >>> >>>>>>> >>>> > >> > > >>> >>>>>>> >>>> > >> > -- > >>> >>>>>>> >>>> > >> > Best Regards, Edward J. Yoon > >>> >>>>>>> >>>> > >> > @eddieyoon > >>> >>>>>>> >>>> > >> > > >>> >>>>>>> >>>> > >> > >>> >>>>>>> >>>> > > >>> >>>>>>> >>>> > > >>> >>>>>>> >>>> > > >>> >>>>>>> >>>> > -- > >>> >>>>>>> >>>> > Best Regards, Edward J. Yoon > >>> >>>>>>> >>>> > @eddieyoon > >>> >>>>>>> >>>> > > >>> >>>>>>> >>>> > >>> >>>>>>> >> > >>> >>>>>>> >> > >>> >>>>>>> >> > >>> >>>>>>> >> -- > >>> >>>>>>> >> Best Regards, Edward J. Yoon > >>> >>>>>>> >> @eddieyoon > >>> >>>>>>> > > >>> >>>>>>> > > >>> >>>>>>> > > >>> >>>>>>> > -- > >>> >>>>>>> > Best Regards, Edward J. Yoon > >>> >>>>>>> > @eddieyoon > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> -- > >>> >>>>>>> Best Regards, Edward J. Yoon > >>> >>>>>>> @eddieyoon > >>> >>>>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> -- > >>> >>>>> Best Regards, Edward J. Yoon > >>> >>>>> @eddieyoon > >>> >>>> > >>> >>>> > >>> >>>> > >>> >>>> -- > >>> >>>> Best Regards, Edward J. Yoon > >>> >>>> @eddieyoon > >>> >>> > >>> >>> > >>> >>> > >>> >>> -- > >>> >>> Best Regards, Edward J. Yoon > >>> >>> @eddieyoon > >>> >> > >>> >> > >>> >> > >>> >> -- > >>> >> Best Regards, Edward J. Yoon > >>> >> @eddieyoon > >>> > > >>> > > >>> > > >>> > -- > >>> > Best Regards, Edward J. Yoon > >>> > @eddieyoon > >>> > >>> > >>> > >>> -- > >>> Best Regards, Edward J. Yoon > >>> @eddieyoon > >>> > > > > > > > > -- > > Best Regards, Edward J. Yoon > > @eddieyoon > > > > -- > Best Regards, Edward J. Yoon > @eddieyoon >
