Can you verify an observation for me please? 2 files are created from fastgen, part-00000 and part-00001, both ~2.2kb sized. In the below partition directory, there is only a single 5.56kb file.
Is it intended for the partitioner to write a single file if you configured two? It even reads it as a two files, strange huh? 2013/2/28 Thomas Jungblut <[email protected]> > Will have a look into it. > > gen fastgen 100 10 /tmp/randomgraph 1 > pagerank /tmp/randomgraph /tmp/pageout > > did work for me the last time I profiled, maybe the partitioning doesn't > partition correctly with the input or something else. > > > 2013/2/28 Edward J. Yoon <[email protected]> > > Fastgen input seems not work for graph examples. >> >> edward@edward-virtualBox:~/workspace/hama-trunk$ bin/hama jar >> examples/target/hama-examples-0.7.0-SNAPSHOT.jar gen fastgen 100 10 >> /tmp/randomgraph 2 >> 13/02/28 10:32:02 WARN util.NativeCodeLoader: Unable to load >> native-hadoop library for your platform... using builtin-java classes >> where applicable >> 13/02/28 10:32:03 INFO bsp.BSPJobClient: Running job: job_localrunner_0001 >> 13/02/28 10:32:03 INFO bsp.LocalBSPRunner: Setting up a new barrier for 2 >> tasks! >> 13/02/28 10:32:06 INFO bsp.BSPJobClient: Current supersteps number: 0 >> 13/02/28 10:32:06 INFO bsp.BSPJobClient: The total number of supersteps: 0 >> 13/02/28 10:32:06 INFO bsp.BSPJobClient: Counters: 3 >> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >> org.apache.hama.bsp.JobInProgress$JobCounter >> 13/02/28 10:32:06 INFO bsp.BSPJobClient: SUPERSTEPS=0 >> 13/02/28 10:32:06 INFO bsp.BSPJobClient: LAUNCHED_TASKS=2 >> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >> 13/02/28 10:32:06 INFO bsp.BSPJobClient: TASK_OUTPUT_RECORDS=100 >> Job Finished in 3.212 seconds >> edward@edward-virtualBox:~/workspace/hama-trunk$ bin/hama jar >> examples/target/hama-examples-0.7.0-SNAPSHOT >> hama-examples-0.7.0-SNAPSHOT-javadoc.jar >> hama-examples-0.7.0-SNAPSHOT.jar >> edward@edward-virtualBox:~/workspace/hama-trunk$ bin/hama jar >> examples/target/hama-examples-0.7.0-SNAPSHOT.jar pagerank >> /tmp/randomgraph /tmp/pageour >> 13/02/28 10:32:29 WARN util.NativeCodeLoader: Unable to load >> native-hadoop library for your platform... using builtin-java classes >> where applicable >> 13/02/28 10:32:29 INFO bsp.FileInputFormat: Total input paths to process >> : 2 >> 13/02/28 10:32:29 INFO bsp.FileInputFormat: Total input paths to process >> : 2 >> 13/02/28 10:32:30 INFO bsp.BSPJobClient: Running job: job_localrunner_0001 >> 13/02/28 10:32:30 INFO bsp.LocalBSPRunner: Setting up a new barrier for 2 >> tasks! >> 13/02/28 10:32:33 INFO bsp.BSPJobClient: Current supersteps number: 1 >> 13/02/28 10:32:33 INFO bsp.BSPJobClient: The total number of supersteps: 1 >> 13/02/28 10:32:33 INFO bsp.BSPJobClient: Counters: 6 >> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >> org.apache.hama.bsp.JobInProgress$JobCounter >> 13/02/28 10:32:33 INFO bsp.BSPJobClient: SUPERSTEPS=1 >> 13/02/28 10:32:33 INFO bsp.BSPJobClient: LAUNCHED_TASKS=2 >> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >> 13/02/28 10:32:33 INFO bsp.BSPJobClient: SUPERSTEP_SUM=4 >> 13/02/28 10:32:33 INFO bsp.BSPJobClient: IO_BYTES_READ=4332 >> 13/02/28 10:32:33 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=14 >> 13/02/28 10:32:33 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=100 >> 13/02/28 10:32:33 INFO bsp.FileInputFormat: Total input paths to process >> : 2 >> 13/02/28 10:32:33 INFO bsp.BSPJobClient: Running job: job_localrunner_0001 >> 13/02/28 10:32:33 INFO bsp.LocalBSPRunner: Setting up a new barrier for 2 >> tasks! >> 13/02/28 10:32:33 INFO graph.GraphJobRunner: 50 vertices are loaded into >> local:1 >> 13/02/28 10:32:33 INFO graph.GraphJobRunner: 50 vertices are loaded into >> local:0 >> 13/02/28 10:32:33 ERROR bsp.LocalBSPRunner: Exception during BSP >> execution! >> java.lang.IllegalArgumentException: Messages must never be behind the >> vertex in ID! Current Message ID: 1 vs. 50 >> at >> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279) >> at >> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225) >> at >> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129) >> at >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256) >> at >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> at >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> at java.lang.Thread.run(Thread.java:722) >> >> >> -- >> Best Regards, Edward J. Yoon >> @eddieyoon >> > >
