Sorry for my quick and dirty style small patches.

However, we should work together in parallel. Please share here if
there are some progresses.

On Thu, Mar 14, 2013 at 3:46 PM, Thomas Jungblut
<[email protected]> wrote:
> Hi Edward,
>
> before you run riot on all along the codebase, Suraj ist currently working
> on that stuff- don't make it more difficult for him rebasing all his
> patches the whole time.
> He has the plan so that we made to make the stuff working, his part is
> currently missing. So don't try to muddle arround there, it will make this
> take longer than already needed.
>
>
>
> 2013/3/14 Edward J. Yoon <[email protected]>
>
>> Personally, I would like to solve this issue by touching
>> DiskVerticesInfo. If we write sorted sub-sets of vertices into
>> multiple files, we can avoid huge memory consumption.
>>
>> If we want to sort partitioned data using messaging system, idea
>> should be collected.
>>
>> On Thu, Mar 14, 2013 at 10:31 AM, Edward J. Yoon <[email protected]>
>> wrote:
>> > Oh, now I get how iterate() works. HAMA-704 is nicely written.
>> >
>> > On Thu, Mar 14, 2013 at 12:02 AM, Edward J. Yoon <[email protected]>
>> wrote:
>> >> I'm reading changes of HAMA-704 again. As a result of adding
>> >> DiskVerticesInfo, vertices list is needed to be sorted. I'm not sure
>> >> but I think this approach will bring more disadvantages than
>> >> advantages.
>> >>
>> >> On Wed, Mar 13, 2013 at 11:09 PM, Edward J. Yoon <[email protected]>
>> wrote:
>> >>>>>> in loadVertices? Maybe consider feature for coupling storage in
>> user space
>> >>>>>> with BSP Messaging[HAMA-734] can avoid double reads and writes.
>> This way
>> >>>>>> partitioned or non-partitioned by partitioner, can keep vertices
>> sorted
>> >>>>>> with a single read and single write on every peer.
>> >>>
>> >>> And, as I commented JIRA ticket, I think we can't use messaging system
>> >>> for sorting vertices within partition files.
>> >>>
>> >>> On Wed, Mar 13, 2013 at 11:00 PM, Edward J. Yoon <
>> [email protected]> wrote:
>> >>>> P.S., (number of splits = number of partitions) is really confuse to
>> >>>> me. Even though blocks number is equal to desired tasks number, data
>> >>>> should be re-partitioned again.
>> >>>>
>> >>>> On Wed, Mar 13, 2013 at 10:36 PM, Edward J. Yoon <
>> [email protected]> wrote:
>> >>>>> Indeed. If there are already partitioned input files (unsorted) and
>> so
>> >>>>> user want to skip pre-partitioning phase, it should be handled in
>> >>>>> GraphJobRunner BSP program. Actually, I still don't know why
>> >>>>> re-partitioned files need to be Sorted. It's only about
>> >>>>> GraphJobRunner.
>> >>>>>
>> >>>>>> partitioning. (This is outside the scope of graphs. We can have a
>> dedicated
>> >>>>>> partitioning superstep for graph applications).
>> >>>>>
>> >>>>> Sorry. I don't understand exactly yet. Do you mean just a
>> partitioning
>> >>>>> job based on superstep API?
>> >>>>>
>> >>>>> By default, 100 tasks will be assigned for partitioning job.
>> >>>>> Partitioning job will create 1,000 partitions. Thus, we can execute
>> >>>>> the Graph job with 1,000 tasks.
>> >>>>>
>> >>>>> Let's assume that a input sequence file is 20GB (100 blocks). If I
>> >>>>> want to run with 1,000 tasks, what happens?
>> >>>>>
>> >>>>> On Wed, Mar 13, 2013 at 9:49 PM, Suraj Menon <[email protected]>
>> wrote:
>> >>>>>> I am responding on this thread because of better continuity for
>> >>>>>> conversation. We cannot expect the partitions to be sorted every
>> time. When
>> >>>>>> the number of splits = number of partitions and partitioning is
>> switched
>> >>>>>> off by user[HAMA-561], the partitions would not be sorted. Can we
>> do this
>> >>>>>> in loadVertices? Maybe consider feature for coupling storage in
>> user space
>> >>>>>> with BSP Messaging[HAMA-734] can avoid double reads and writes.
>> This way
>> >>>>>> partitioned or non-partitioned by partitioner, can keep vertices
>> sorted
>> >>>>>> with a single read and single write on every peer.
>> >>>>>>
>> >>>>>> Just clearing confusion if any regarding superstep injection for
>> >>>>>> partitioning. (This is outside the scope of graphs. We can have a
>> dedicated
>> >>>>>> partitioning superstep for graph applications).
>> >>>>>> Say there are x splits and y number of tasks configured by user.
>> >>>>>>
>> >>>>>> if x > y
>> >>>>>> The y tasks are scheduled with x of them having each of the x
>> splits and
>> >>>>>> the remaining with no resource local to them. Then the partitioning
>> >>>>>> superstep redistributes the partitions among them to create local
>> >>>>>> partitions. Now the question is can we re-initialize a peer's input
>> based
>> >>>>>> on this new local part of partition?
>> >>>>>>
>> >>>>>> if y > x
>> >>>>>> works as it works today.
>> >>>>>>
>> >>>>>> Just putting my points in brainstorming.
>> >>>>>>
>> >>>>>> -Suraj
>> >>>>>>
>> >>>>>>
>> >>>>>> On Mon, Mar 11, 2013 at 7:39 AM, Edward J. Yoon <
>> [email protected]>wrote:
>> >>>>>>
>> >>>>>>> I just filed here https://issues.apache.org/jira/browse/HAMA-744
>> >>>>>>>
>> >>>>>>> On Mon, Mar 11, 2013 at 7:35 PM, Edward J. Yoon <
>> [email protected]>
>> >>>>>>> wrote:
>> >>>>>>> > Additionally,
>> >>>>>>> >
>> >>>>>>> >> spilling queue and sorted spilling queue, can we inject the
>> partitioning
>> >>>>>>> >> superstep as the first superstep and use local memory?
>> >>>>>>> >
>> >>>>>>> > Can we execute different number of tasks per superstep?
>> >>>>>>> >
>> >>>>>>> > On Mon, Mar 11, 2013 at 6:56 PM, Edward J. Yoon <
>> [email protected]>
>> >>>>>>> wrote:
>> >>>>>>> >>> For graph processing, the partitioned files that result from
>> the
>> >>>>>>> >>> partitioning job must be sorted. Currently only the partition
>> files in
>> >>>>>>> >>
>> >>>>>>> >> I see.
>> >>>>>>> >>
>> >>>>>>> >>> For other partitionings and with regard to our superstep API,
>> Suraj's
>> >>>>>>> idea
>> >>>>>>> >>> of injecting a preprocessing superstep that partitions the
>> stuff into
>> >>>>>>> our
>> >>>>>>> >>> messaging system is actually the best.
>> >>>>>>> >>
>> >>>>>>> >> BTW, if some garbage objects can be accumulated in partitioning
>> step,
>> >>>>>>> >> separated partitioning job may not be bad idea. Is there some
>> special
>> >>>>>>> >> reason?
>> >>>>>>> >>
>> >>>>>>> >> On Wed, Mar 6, 2013 at 6:15 PM, Thomas Jungblut
>> >>>>>>> >> <[email protected]> wrote:
>> >>>>>>> >>> For graph processing, the partitioned files that result from
>> the
>> >>>>>>> >>> partitioning job must be sorted. Currently only the partition
>> files in
>> >>>>>>> >>> itself are sorted, thus more tasks result in not sorted data
>> in the
>> >>>>>>> >>> completed file. This only applies for the graph processing
>> package.
>> >>>>>>> >>> So as Suraj told, it would be much more simpler to solve this
>> via
>> >>>>>>> >>> messaging, once it is scalable (it will be very very
>> scalable!). So the
>> >>>>>>> >>> GraphJobRunner can be partitioning the stuff with a single
>> superstep in
>> >>>>>>> >>> setup() as it was before ages ago. The messaging must be
>> sorted anyway
>> >>>>>>> for
>> >>>>>>> >>> the algorithm so this is a nice side effect and saves us the
>> >>>>>>> partitioning
>> >>>>>>> >>> job for graph processing.
>> >>>>>>> >>>
>> >>>>>>> >>> For other partitionings and with regard to our superstep API,
>> Suraj's
>> >>>>>>> idea
>> >>>>>>> >>> of injecting a preprocessing superstep that partitions the
>> stuff into
>> >>>>>>> our
>> >>>>>>> >>> messaging system is actually the best.
>> >>>>>>> >>>
>> >>>>>>> >>>
>> >>>>>>> >>> 2013/3/6 Suraj Menon <[email protected]>
>> >>>>>>> >>>
>> >>>>>>> >>>> No, the partitions we write locally need not be sorted. Sorry
>> for the
>> >>>>>>> >>>> confusion. The Superstep injection is possible with Superstep
>> API.
>> >>>>>>> There
>> >>>>>>> >>>> are few enhancements needed to make it simpler after I last
>> worked on
>> >>>>>>> it.
>> >>>>>>> >>>> We can then look into partitioning superstep being executed
>> before the
>> >>>>>>> >>>> setup of first superstep of submitted job. I think it is
>> feasible.
>> >>>>>>> >>>>
>> >>>>>>> >>>> On Tue, Mar 5, 2013 at 5:48 AM, Edward J. Yoon <
>> [email protected]
>> >>>>>>> >>>> >wrote:
>> >>>>>>> >>>>
>> >>>>>>> >>>> > > spilling queue and sorted spilling queue, can we inject
>> the
>> >>>>>>> >>>> partitioning
>> >>>>>>> >>>> > > superstep as the first superstep and use local memory?
>> >>>>>>> >>>> >
>> >>>>>>> >>>> > Actually, I wanted to add something before calling
>> BSP.setup()
>> >>>>>>> method
>> >>>>>>> >>>> > to avoid execute additional BSP job. But, in my opinion,
>> current is
>> >>>>>>> >>>> > enough. I think, we need to collect more experiences of
>> input
>> >>>>>>> >>>> > partitioning on large environments. I'll do.
>> >>>>>>> >>>> >
>> >>>>>>> >>>> > BTW, I still don't know why it need to be Sorted?! MR-like?
>> >>>>>>> >>>> >
>> >>>>>>> >>>> > On Thu, Feb 28, 2013 at 11:20 PM, Suraj Menon <
>> >>>>>>> [email protected]>
>> >>>>>>> >>>> > wrote:
>> >>>>>>> >>>> > > Sorry, I am increasing the scope here to outside graph
>> module.
>> >>>>>>> When we
>> >>>>>>> >>>> > have
>> >>>>>>> >>>> > > spilling queue and sorted spilling queue, can we inject
>> the
>> >>>>>>> >>>> partitioning
>> >>>>>>> >>>> > > superstep as the first superstep and use local memory?
>> >>>>>>> >>>> > > Today we have partitioning job within a job and are
>> creating two
>> >>>>>>> copies
>> >>>>>>> >>>> > of
>> >>>>>>> >>>> > > data on HDFS. This could be really costly. Is it possible
>> to
>> >>>>>>> create or
>> >>>>>>> >>>> > > redistribute the partitions on local memory and
>> initialize the
>> >>>>>>> record
>> >>>>>>> >>>> > > reader there?
>> >>>>>>> >>>> > > The user can run a separate job give in examples area to
>> >>>>>>> explicitly
>> >>>>>>> >>>> > > repartition the data on HDFS. The deployment question is
>> how much
>> >>>>>>> of
>> >>>>>>> >>>> disk
>> >>>>>>> >>>> > > space gets allocated for local memory usage? Would it be
>> a safe
>> >>>>>>> >>>> approach
>> >>>>>>> >>>> > > with the limitations?
>> >>>>>>> >>>> > >
>> >>>>>>> >>>> > > -Suraj
>> >>>>>>> >>>> > >
>> >>>>>>> >>>> > > On Thu, Feb 28, 2013 at 7:05 AM, Thomas Jungblut
>> >>>>>>> >>>> > > <[email protected]>wrote:
>> >>>>>>> >>>> > >
>> >>>>>>> >>>> > >> yes. Once Suraj added merging of sorted files we can add
>> this to
>> >>>>>>> the
>> >>>>>>> >>>> > >> partitioner pretty easily.
>> >>>>>>> >>>> > >>
>> >>>>>>> >>>> > >> 2013/2/28 Edward J. Yoon <[email protected]>
>> >>>>>>> >>>> > >>
>> >>>>>>> >>>> > >> > Eh,..... btw, is re-partitioned data really necessary
>> to be
>> >>>>>>> Sorted?
>> >>>>>>> >>>> > >> >
>> >>>>>>> >>>> > >> > On Thu, Feb 28, 2013 at 7:48 PM, Thomas Jungblut
>> >>>>>>> >>>> > >> > <[email protected]> wrote:
>> >>>>>>> >>>> > >> > > Now I get how the partitioning works, obviously if
>> you merge
>> >>>>>>> n
>> >>>>>>> >>>> > sorted
>> >>>>>>> >>>> > >> > files
>> >>>>>>> >>>> > >> > > by just appending to each other, this will result in
>> totally
>> >>>>>>> >>>> > unsorted
>> >>>>>>> >>>> > >> > data
>> >>>>>>> >>>> > >> > > ;-)
>> >>>>>>> >>>> > >> > > Why didn't you solve this via messaging?
>> >>>>>>> >>>> > >> > >
>> >>>>>>> >>>> > >> > > 2013/2/28 Thomas Jungblut <[email protected]
>> >
>> >>>>>>> >>>> > >> > >
>> >>>>>>> >>>> > >> > >> Seems that they are not correctly sorted:
>> >>>>>>> >>>> > >> > >>
>> >>>>>>> >>>> > >> > >> vertexID: 50
>> >>>>>>> >>>> > >> > >> vertexID: 52
>> >>>>>>> >>>> > >> > >> vertexID: 54
>> >>>>>>> >>>> > >> > >> vertexID: 56
>> >>>>>>> >>>> > >> > >> vertexID: 58
>> >>>>>>> >>>> > >> > >> vertexID: 61
>> >>>>>>> >>>> > >> > >> ...
>> >>>>>>> >>>> > >> > >> vertexID: 78
>> >>>>>>> >>>> > >> > >> vertexID: 81
>> >>>>>>> >>>> > >> > >> vertexID: 83
>> >>>>>>> >>>> > >> > >> vertexID: 85
>> >>>>>>> >>>> > >> > >> ...
>> >>>>>>> >>>> > >> > >> vertexID: 94
>> >>>>>>> >>>> > >> > >> vertexID: 96
>> >>>>>>> >>>> > >> > >> vertexID: 98
>> >>>>>>> >>>> > >> > >> vertexID: 1
>> >>>>>>> >>>> > >> > >> vertexID: 10
>> >>>>>>> >>>> > >> > >> vertexID: 12
>> >>>>>>> >>>> > >> > >> vertexID: 14
>> >>>>>>> >>>> > >> > >> vertexID: 16
>> >>>>>>> >>>> > >> > >> vertexID: 18
>> >>>>>>> >>>> > >> > >> vertexID: 21
>> >>>>>>> >>>> > >> > >> vertexID: 23
>> >>>>>>> >>>> > >> > >> vertexID: 25
>> >>>>>>> >>>> > >> > >> vertexID: 27
>> >>>>>>> >>>> > >> > >> vertexID: 29
>> >>>>>>> >>>> > >> > >> vertexID: 3
>> >>>>>>> >>>> > >> > >>
>> >>>>>>> >>>> > >> > >> So this won't work then correctly...
>> >>>>>>> >>>> > >> > >>
>> >>>>>>> >>>> > >> > >>
>> >>>>>>> >>>> > >> > >> 2013/2/28 Thomas Jungblut <
>> [email protected]>
>> >>>>>>> >>>> > >> > >>
>> >>>>>>> >>>> > >> > >>> sure, have fun on your holidays.
>> >>>>>>> >>>> > >> > >>>
>> >>>>>>> >>>> > >> > >>>
>> >>>>>>> >>>> > >> > >>> 2013/2/28 Edward J. Yoon <[email protected]>
>> >>>>>>> >>>> > >> > >>>
>> >>>>>>> >>>> > >> > >>>> Sure, but if you can fix quickly, please do.
>> March 1 is
>> >>>>>>> >>>> > holiday[1]
>> >>>>>>> >>>> > >> so
>> >>>>>>> >>>> > >> > >>>> I'll appear next week.
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>> > >> > >>>> 1.
>> >>>>>>> http://en.wikipedia.org/wiki/Public_holidays_in_South_Korea
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>> > >> > >>>> On Thu, Feb 28, 2013 at 6:36 PM, Thomas Jungblut
>> >>>>>>> >>>> > >> > >>>> <[email protected]> wrote:
>> >>>>>>> >>>> > >> > >>>> > Maybe 50 is missing from the file, didn't
>> observe if all
>> >>>>>>> >>>> items
>> >>>>>>> >>>> > >> were
>> >>>>>>> >>>> > >> > >>>> added.
>> >>>>>>> >>>> > >> > >>>> > As far as I remember, I copy/pasted the logic
>> of the ID
>> >>>>>>> into
>> >>>>>>> >>>> > the
>> >>>>>>> >>>> > >> > >>>> fastgen,
>> >>>>>>> >>>> > >> > >>>> > want to have a look into it?
>> >>>>>>> >>>> > >> > >>>> >
>> >>>>>>> >>>> > >> > >>>> > 2013/2/28 Edward J. Yoon <[email protected]
>> >
>> >>>>>>> >>>> > >> > >>>> >
>> >>>>>>> >>>> > >> > >>>> >> I guess, it's a bug of fastgen, when generate
>> adjacency
>> >>>>>>> >>>> matrix
>> >>>>>>> >>>> > >> into
>> >>>>>>> >>>> > >> > >>>> >> multiple files.
>> >>>>>>> >>>> > >> > >>>> >>
>> >>>>>>> >>>> > >> > >>>> >> On Thu, Feb 28, 2013 at 6:29 PM, Thomas
>> Jungblut
>> >>>>>>> >>>> > >> > >>>> >> <[email protected]> wrote:
>> >>>>>>> >>>> > >> > >>>> >> > You have two files, are they partitioned
>> correctly?
>> >>>>>>> >>>> > >> > >>>> >> >
>> >>>>>>> >>>> > >> > >>>> >> > 2013/2/28 Edward J. Yoon <
>> [email protected]>
>> >>>>>>> >>>> > >> > >>>> >> >
>> >>>>>>> >>>> > >> > >>>> >> >> It looks like a bug.
>> >>>>>>> >>>> > >> > >>>> >> >>
>> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls
>> -al
>> >>>>>>> >>>> > >> /tmp/randomgraph/
>> >>>>>>> >>>> > >> > >>>> >> >> total 44
>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x  3 edward edward  4096  2월 28
>> 18:03 .
>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxrwt 19 root   root   20480  2월 28
>> 18:04 ..
>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx  1 edward edward  2243  2월 28
>> 18:01
>> >>>>>>> part-00000
>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r--  1 edward edward    28  2월 28
>> 18:01
>> >>>>>>> >>>> > .part-00000.crc
>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx  1 edward edward  2251  2월 28
>> 18:01
>> >>>>>>> part-00001
>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r--  1 edward edward    28  2월 28
>> 18:01
>> >>>>>>> >>>> > .part-00001.crc
>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x  2 edward edward  4096  2월 28
>> 18:03
>> >>>>>>> partitions
>> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls
>> -al
>> >>>>>>> >>>> > >> > >>>> >> /tmp/randomgraph/partitions/
>> >>>>>>> >>>> > >> > >>>> >> >> total 24
>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096  2월 28
>> 18:03 .
>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096  2월 28
>> 18:03 ..
>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2932  2월 28 18:03
>> >>>>>>> part-00000
>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward   32  2월 28 18:03
>> >>>>>>> >>>> > .part-00000.crc
>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2955  2월 28 18:03
>> >>>>>>> part-00001
>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward   32  2월 28 18:03
>> >>>>>>> >>>> > .part-00001.crc
>> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$
>> >>>>>>> >>>> > >> > >>>> >> >>
>> >>>>>>> >>>> > >> > >>>> >> >>
>> >>>>>>> >>>> > >> > >>>> >> >> On Thu, Feb 28, 2013 at 5:27 PM, Edward <
>> >>>>>>> >>>> [email protected]
>> >>>>>>> >>>> > >
>> >>>>>>> >>>> > >> > wrote:
>> >>>>>>> >>>> > >> > >>>> >> >> > yes i'll check again
>> >>>>>>> >>>> > >> > >>>> >> >> >
>> >>>>>>> >>>> > >> > >>>> >> >> > Sent from my iPhone
>> >>>>>>> >>>> > >> > >>>> >> >> >
>> >>>>>>> >>>> > >> > >>>> >> >> > On Feb 28, 2013, at 5:18 PM, Thomas
>> Jungblut <
>> >>>>>>> >>>> > >> > >>>> >> [email protected]>
>> >>>>>>> >>>> > >> > >>>> >> >> wrote:
>> >>>>>>> >>>> > >> > >>>> >> >> >
>> >>>>>>> >>>> > >> > >>>> >> >> >> Can you verify an observation for me
>> please?
>> >>>>>>> >>>> > >> > >>>> >> >> >>
>> >>>>>>> >>>> > >> > >>>> >> >> >> 2 files are created from fastgen,
>> part-00000 and
>> >>>>>>> >>>> > >> part-00001,
>> >>>>>>> >>>> > >> > >>>> both
>> >>>>>>> >>>> > >> > >>>> >> ~2.2kb
>> >>>>>>> >>>> > >> > >>>> >> >> >> sized.
>> >>>>>>> >>>> > >> > >>>> >> >> >> In the below partition directory, there
>> is only a
>> >>>>>>> >>>> single
>> >>>>>>> >>>> > >> > 5.56kb
>> >>>>>>> >>>> > >> > >>>> file.
>> >>>>>>> >>>> > >> > >>>> >> >> >>
>> >>>>>>> >>>> > >> > >>>> >> >> >> Is it intended for the partitioner to
>> write a
>> >>>>>>> single
>> >>>>>>> >>>> > file
>> >>>>>>> >>>> > >> if
>> >>>>>>> >>>> > >> > you
>> >>>>>>> >>>> > >> > >>>> >> >> configured
>> >>>>>>> >>>> > >> > >>>> >> >> >> two?
>> >>>>>>> >>>> > >> > >>>> >> >> >> It even reads it as a two files, strange
>> huh?
>> >>>>>>> >>>> > >> > >>>> >> >> >>
>> >>>>>>> >>>> > >> > >>>> >> >> >> 2013/2/28 Thomas Jungblut <
>> >>>>>>> [email protected]>
>> >>>>>>> >>>> > >> > >>>> >> >> >>
>> >>>>>>> >>>> > >> > >>>> >> >> >>> Will have a look into it.
>> >>>>>>> >>>> > >> > >>>> >> >> >>>
>> >>>>>>> >>>> > >> > >>>> >> >> >>> gen fastgen 100 10 /tmp/randomgraph 1
>> >>>>>>> >>>> > >> > >>>> >> >> >>> pagerank /tmp/randomgraph /tmp/pageout
>> >>>>>>> >>>> > >> > >>>> >> >> >>>
>> >>>>>>> >>>> > >> > >>>> >> >> >>> did work for me the last time I
>> profiled, maybe
>> >>>>>>> the
>> >>>>>>> >>>> > >> > >>>> partitioning
>> >>>>>>> >>>> > >> > >>>> >> >> doesn't
>> >>>>>>> >>>> > >> > >>>> >> >> >>> partition correctly with the input or
>> something
>> >>>>>>> else.
>> >>>>>>> >>>> > >> > >>>> >> >> >>>
>> >>>>>>> >>>> > >> > >>>> >> >> >>>
>> >>>>>>> >>>> > >> > >>>> >> >> >>> 2013/2/28 Edward J. Yoon <
>> [email protected]
>> >>>>>>> >
>> >>>>>>> >>>> > >> > >>>> >> >> >>>
>> >>>>>>> >>>> > >> > >>>> >> >> >>> Fastgen input seems not work for graph
>> examples.
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox
>> >>>>>>> :~/workspace/hama-trunk$
>> >>>>>>> >>>> > >> bin/hama
>> >>>>>>> >>>> > >> > jar
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar gen
>> >>>>>>> >>>> > >> > fastgen
>> >>>>>>> >>>> > >> > >>>> 100 10
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph 2
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:02 WARN
>> util.NativeCodeLoader:
>> >>>>>>> Unable
>> >>>>>>> >>>> > to
>> >>>>>>> >>>> > >> > load
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your
>> platform...
>> >>>>>>> using
>> >>>>>>> >>>> > >> > builtin-java
>> >>>>>>> >>>> > >> > >>>> >> classes
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> where applicable
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO
>> bsp.BSPJobClient:
>> >>>>>>> Running
>> >>>>>>> >>>> job:
>> >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO
>> bsp.LocalBSPRunner:
>> >>>>>>> Setting
>> >>>>>>> >>>> up
>> >>>>>>> >>>> > a
>> >>>>>>> >>>> > >> new
>> >>>>>>> >>>> > >> > >>>> barrier
>> >>>>>>> >>>> > >> > >>>> >> >> for 2
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks!
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>> bsp.BSPJobClient:
>> >>>>>>> Current
>> >>>>>>> >>>> > >> supersteps
>> >>>>>>> >>>> > >> > >>>> >> number: 0
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>> bsp.BSPJobClient: The
>> >>>>>>> total
>> >>>>>>> >>>> > number
>> >>>>>>> >>>> > >> > of
>> >>>>>>> >>>> > >> > >>>> >> >> supersteps: 0
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>> bsp.BSPJobClient:
>> >>>>>>> Counters: 3
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>> bsp.BSPJobClient:
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> org.apache.hama.bsp.JobInProgress$JobCounter
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>> bsp.BSPJobClient:
>> >>>>>>> >>>> > SUPERSTEPS=0
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>> bsp.BSPJobClient:
>> >>>>>>> >>>> > >> > LAUNCHED_TASKS=2
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>> bsp.BSPJobClient:
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>> bsp.BSPJobClient:
>> >>>>>>> >>>> > >> > >>>> >> TASK_OUTPUT_RECORDS=100
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> Job Finished in 3.212 seconds
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox
>> >>>>>>> :~/workspace/hama-trunk$
>> >>>>>>> >>>> > >> bin/hama
>> >>>>>>> >>>> > >> > jar
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> examples/target/hama-examples-0.7.0-SNAPSHOT
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> hama-examples-0.7.0-SNAPSHOT-javadoc.jar
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT.jar
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox
>> >>>>>>> :~/workspace/hama-trunk$
>> >>>>>>> >>>> > >> bin/hama
>> >>>>>>> >>>> > >> > jar
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar
>> >>>>>>> >>>> > pagerank
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph /tmp/pageour
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 WARN
>> util.NativeCodeLoader:
>> >>>>>>> Unable
>> >>>>>>> >>>> > to
>> >>>>>>> >>>> > >> > load
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your
>> platform...
>> >>>>>>> using
>> >>>>>>> >>>> > >> > builtin-java
>> >>>>>>> >>>> > >> > >>>> >> classes
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> where applicable
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO
>> bsp.FileInputFormat:
>> >>>>>>> Total
>> >>>>>>> >>>> > input
>> >>>>>>> >>>> > >> > paths
>> >>>>>>> >>>> > >> > >>>> to
>> >>>>>>> >>>> > >> > >>>> >> >> process
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO
>> bsp.FileInputFormat:
>> >>>>>>> Total
>> >>>>>>> >>>> > input
>> >>>>>>> >>>> > >> > paths
>> >>>>>>> >>>> > >> > >>>> to
>> >>>>>>> >>>> > >> > >>>> >> >> process
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO
>> bsp.BSPJobClient:
>> >>>>>>> Running
>> >>>>>>> >>>> job:
>> >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO
>> bsp.LocalBSPRunner:
>> >>>>>>> Setting
>> >>>>>>> >>>> up
>> >>>>>>> >>>> > a
>> >>>>>>> >>>> > >> new
>> >>>>>>> >>>> > >> > >>>> barrier
>> >>>>>>> >>>> > >> > >>>> >> >> for 2
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks!
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> bsp.BSPJobClient:
>> >>>>>>> Current
>> >>>>>>> >>>> > >> supersteps
>> >>>>>>> >>>> > >> > >>>> >> number: 1
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> bsp.BSPJobClient: The
>> >>>>>>> total
>> >>>>>>> >>>> > number
>> >>>>>>> >>>> > >> > of
>> >>>>>>> >>>> > >> > >>>> >> >> supersteps: 1
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> bsp.BSPJobClient:
>> >>>>>>> Counters: 6
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> bsp.BSPJobClient:
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> org.apache.hama.bsp.JobInProgress$JobCounter
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> bsp.BSPJobClient:
>> >>>>>>> >>>> > SUPERSTEPS=1
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> bsp.BSPJobClient:
>> >>>>>>> >>>> > >> > LAUNCHED_TASKS=2
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> bsp.BSPJobClient:
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> bsp.BSPJobClient:
>> >>>>>>> >>>> > >> > SUPERSTEP_SUM=4
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> bsp.BSPJobClient:
>> >>>>>>> >>>> > >> > >>>> IO_BYTES_READ=4332
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> bsp.BSPJobClient:
>> >>>>>>> >>>> > >> > >>>> TIME_IN_SYNC_MS=14
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> bsp.BSPJobClient:
>> >>>>>>> >>>> > >> > >>>> TASK_INPUT_RECORDS=100
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> bsp.FileInputFormat:
>> >>>>>>> Total
>> >>>>>>> >>>> > input
>> >>>>>>> >>>> > >> > paths
>> >>>>>>> >>>> > >> > >>>> to
>> >>>>>>> >>>> > >> > >>>> >> >> process
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> bsp.BSPJobClient:
>> >>>>>>> Running
>> >>>>>>> >>>> job:
>> >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> bsp.LocalBSPRunner:
>> >>>>>>> Setting
>> >>>>>>> >>>> up
>> >>>>>>> >>>> > a
>> >>>>>>> >>>> > >> new
>> >>>>>>> >>>> > >> > >>>> barrier
>> >>>>>>> >>>> > >> > >>>> >> >> for 2
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks!
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> graph.GraphJobRunner: 50
>> >>>>>>> >>>> > vertices
>> >>>>>>> >>>> > >> > are
>> >>>>>>> >>>> > >> > >>>> loaded
>> >>>>>>> >>>> > >> > >>>> >> >> into
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> local:1
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>> graph.GraphJobRunner: 50
>> >>>>>>> >>>> > vertices
>> >>>>>>> >>>> > >> > are
>> >>>>>>> >>>> > >> > >>>> loaded
>> >>>>>>> >>>> > >> > >>>> >> >> into
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> local:0
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 ERROR
>> bsp.LocalBSPRunner:
>> >>>>>>> >>>> Exception
>> >>>>>>> >>>> > >> > during
>> >>>>>>> >>>> > >> > >>>> BSP
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> execution!
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> java.lang.IllegalArgumentException:
>> Messages
>> >>>>>>> must
>> >>>>>>> >>>> > never
>> >>>>>>> >>>> > >> be
>> >>>>>>> >>>> > >> > >>>> behind
>> >>>>>>> >>>> > >> > >>>> >> the
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> vertex in ID! Current Message ID: 1
>> vs. 50
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> >>>> > >> > >>>> >>
>> >>>>>>> >>>> > >> >
>> >>>>>>> >>>>
>> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279)
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> >>>> > >> > >>>> >> >>
>> >>>>>>> >>>> > >> > >>>> >>
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>> > >> >
>> >>>>>>> >>>> >
>> >>>>>>>
>> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225)
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>>
>> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129)
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> >>>> > >> > >>>> >> >>
>> >>>>>>> >>>> > >> > >>>> >>
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>> > >> >
>> >>>>>>> >>>> >
>> >>>>>>>
>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256)
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> >>>> > >> > >>>> >> >>
>> >>>>>>> >>>> > >> > >>>> >>
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>> > >> >
>> >>>>>>> >>>> > >>
>> >>>>>>> >>>> >
>> >>>>>>> >>>>
>> >>>>>>>
>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> >>>> > >> > >>>> >> >>
>> >>>>>>> >>>> > >> > >>>> >>
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>> > >> >
>> >>>>>>> >>>> > >>
>> >>>>>>> >>>> >
>> >>>>>>> >>>>
>> >>>>>>>
>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>> >
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>> >>>>>>> >>>> > >> > >>>>
>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> >>>> > >> > >>>> >> >>
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>> > >> >
>> >>>>>>> >>>> >
>> >>>>>>>
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>> >
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>> >>>>>>> >>>> > >> > >>>>
>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> >>>> > >> > >>>> >> >>
>> >>>>>>> >>>> > >> > >>>> >>
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>> > >> >
>> >>>>>>> >>>> > >>
>> >>>>>>> >>>> >
>> >>>>>>> >>>>
>> >>>>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> >>>> > >> > >>>> >> >>
>> >>>>>>> >>>> > >> > >>>> >>
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>> > >> >
>> >>>>>>> >>>> > >>
>> >>>>>>> >>>> >
>> >>>>>>> >>>>
>> >>>>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>> java.lang.Thread.run(Thread.java:722)
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> --
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> Best Regards, Edward J. Yoon
>> >>>>>>> >>>> > >> > >>>> >> >> >>>> @eddieyoon
>> >>>>>>> >>>> > >> > >>>> >> >> >>>
>> >>>>>>> >>>> > >> > >>>> >> >> >>>
>> >>>>>>> >>>> > >> > >>>> >> >>
>> >>>>>>> >>>> > >> > >>>> >> >>
>> >>>>>>> >>>> > >> > >>>> >> >>
>> >>>>>>> >>>> > >> > >>>> >> >> --
>> >>>>>>> >>>> > >> > >>>> >> >> Best Regards, Edward J. Yoon
>> >>>>>>> >>>> > >> > >>>> >> >> @eddieyoon
>> >>>>>>> >>>> > >> > >>>> >> >>
>> >>>>>>> >>>> > >> > >>>> >>
>> >>>>>>> >>>> > >> > >>>> >>
>> >>>>>>> >>>> > >> > >>>> >>
>> >>>>>>> >>>> > >> > >>>> >> --
>> >>>>>>> >>>> > >> > >>>> >> Best Regards, Edward J. Yoon
>> >>>>>>> >>>> > >> > >>>> >> @eddieyoon
>> >>>>>>> >>>> > >> > >>>> >>
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>> > >> > >>>> --
>> >>>>>>> >>>> > >> > >>>> Best Regards, Edward J. Yoon
>> >>>>>>> >>>> > >> > >>>> @eddieyoon
>> >>>>>>> >>>> > >> > >>>>
>> >>>>>>> >>>> > >> > >>>
>> >>>>>>> >>>> > >> > >>>
>> >>>>>>> >>>> > >> > >>
>> >>>>>>> >>>> > >> >
>> >>>>>>> >>>> > >> >
>> >>>>>>> >>>> > >> >
>> >>>>>>> >>>> > >> > --
>> >>>>>>> >>>> > >> > Best Regards, Edward J. Yoon
>> >>>>>>> >>>> > >> > @eddieyoon
>> >>>>>>> >>>> > >> >
>> >>>>>>> >>>> > >>
>> >>>>>>> >>>> >
>> >>>>>>> >>>> >
>> >>>>>>> >>>> >
>> >>>>>>> >>>> > --
>> >>>>>>> >>>> > Best Regards, Edward J. Yoon
>> >>>>>>> >>>> > @eddieyoon
>> >>>>>>> >>>> >
>> >>>>>>> >>>>
>> >>>>>>> >>
>> >>>>>>> >>
>> >>>>>>> >>
>> >>>>>>> >> --
>> >>>>>>> >> Best Regards, Edward J. Yoon
>> >>>>>>> >> @eddieyoon
>> >>>>>>> >
>> >>>>>>> >
>> >>>>>>> >
>> >>>>>>> > --
>> >>>>>>> > Best Regards, Edward J. Yoon
>> >>>>>>> > @eddieyoon
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>> Best Regards, Edward J. Yoon
>> >>>>>>> @eddieyoon
>> >>>>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Best Regards, Edward J. Yoon
>> >>>>> @eddieyoon
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Best Regards, Edward J. Yoon
>> >>>> @eddieyoon
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Best Regards, Edward J. Yoon
>> >>> @eddieyoon
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards, Edward J. Yoon
>> >> @eddieyoon
>> >
>> >
>> >
>> > --
>> > Best Regards, Edward J. Yoon
>> > @eddieyoon
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Reply via email to