[
https://issues.apache.org/jira/browse/HAMA-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13542905#comment-13542905
]
Suraj Menon commented on HAMA-559:
----------------------------------
I am trying to speed up the reads with PrefetchCache, however it is not final
yet.
In writes where most of the writes are single byte, a small improvements is
made as shown below compared to previous benchmarks:
{noformat}
0% Scenario{vm=java, trial=0, benchmark=Spill, size=10, type=DISK_LIST}
3466.12 ns; σ=11.99 ns @ 3 trials
5% Scenario{vm=java, trial=0, benchmark=Spill, size=100, type=DISK_LIST}
7038.17 ns; σ=163.45 ns @ 10 trials
10% Scenario{vm=java, trial=0, benchmark=Spill, size=1000, type=DISK_LIST}
47895.80 ns; σ=4017.85 ns @ 10 trials
14% Scenario{vm=java, trial=0, benchmark=Spill, size=10000, type=DISK_LIST}
357760.36 ns; σ=353653.20 ns @ 10 trials
19% Scenario{vm=java, trial=0, benchmark=Spill, size=100000, type=DISK_LIST}
3546827.11 ns; σ=164414.76 ns @ 10 trials
24% Scenario{vm=java, trial=0, benchmark=Spill, size=1000000, type=DISK_LIST}
40919933.97 ns; σ=3765935.50 ns @ 10 trials
29% Scenario{vm=java, trial=0, benchmark=Spill, size=10000000, type=DISK_LIST}
350974828.00 ns; σ=15513001.84 ns @ 10 trials
33% Scenario{vm=java, trial=0, benchmark=Spill, size=10, type=SPILLING_BUFFER}
3241.19 ns; σ=9.83 ns @ 3 trials
38% Scenario{vm=java, trial=0, benchmark=Spill, size=100, type=SPILLING_BUFFER}
4929.45 ns; σ=6.30 ns @ 3 trials
43% Scenario{vm=java, trial=0, benchmark=Spill, size=1000,
type=SPILLING_BUFFER} 28712.59 ns; σ=473.22 ns @ 10 trials
48% Scenario{vm=java, trial=0, benchmark=Spill, size=10000,
type=SPILLING_BUFFER} 187271.76 ns; σ=6520.41 ns @ 10 trials
52% Scenario{vm=java, trial=0, benchmark=Spill, size=100000,
type=SPILLING_BUFFER} 1685095.74 ns; σ=10410.03 ns @ 3 trials
57% Scenario{vm=java, trial=0, benchmark=Spill, size=1000000,
type=SPILLING_BUFFER} 16922940.78 ns; σ=237383.64 ns @ 10 trials
62% Scenario{vm=java, trial=0, benchmark=Spill, size=10000000,
type=SPILLING_BUFFER} 169591804.10 ns; σ=6028667.57 ns @ 10 trials
67% Scenario{vm=java, trial=0, benchmark=Spill, size=10, type=DISK_BUFFER}
4195.59 ns; σ=40.50 ns @ 10 trials
71% Scenario{vm=java, trial=0, benchmark=Spill, size=100, type=DISK_BUFFER}
6697.32 ns; σ=43.13 ns @ 3 trials
76% Scenario{vm=java, trial=0, benchmark=Spill, size=1000, type=DISK_BUFFER}
37091.61 ns; σ=155.36 ns @ 3 trials
81% Scenario{vm=java, trial=0, benchmark=Spill, size=10000, type=DISK_BUFFER}
333739.82 ns; σ=3451.49 ns @ 10 trials
86% Scenario{vm=java, trial=0, benchmark=Spill, size=100000, type=DISK_BUFFER}
3209961.86 ns; σ=15172.65 ns @ 3 trials
90% Scenario{vm=java, trial=0, benchmark=Spill, size=1000000, type=DISK_BUFFER}
30691871.71 ns; σ=25893.53 ns @ 3 trials
95% Scenario{vm=java, trial=0, benchmark=Spill, size=10000000,
type=DISK_BUFFER} 317232771.17 ns; σ=2960684.01 ns @ 4 trials
size type us linear runtime
10 DISK_LIST 3.47 =
10 SPILLING_BUFFER 3.24 =
10 DISK_BUFFER 4.20 =
100 DISK_LIST 7.04 =
100 SPILLING_BUFFER 4.93 =
100 DISK_BUFFER 6.70 =
1000 DISK_LIST 47.90 =
1000 SPILLING_BUFFER 28.71 =
1000 DISK_BUFFER 37.09 =
10000 DISK_LIST 357.76 =
10000 SPILLING_BUFFER 187.27 =
10000 DISK_BUFFER 333.74 =
100000 DISK_LIST 3546.83 =
100000 SPILLING_BUFFER 1685.10 =
100000 DISK_BUFFER 3209.96 =
1000000 DISK_LIST 40919.93 ===
1000000 SPILLING_BUFFER 16922.94 =
1000000 DISK_BUFFER 30691.87 ==
10000000 DISK_LIST 350974.83 ==============================
10000000 SPILLING_BUFFER 169591.80 ==============
10000000 DISK_BUFFER 317232.77 ===========================
vm: java
trial: 0
benchmark: Spill
Note: benchmarks printed 48434475 characters to System.out and 0 characters to
System.err. Use --debug to see this output.
{noformat}
> Add a spilling message queue
> ----------------------------
>
> Key: HAMA-559
> URL: https://issues.apache.org/jira/browse/HAMA-559
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.5.0
> Reporter: Thomas Jungblut
> Assignee: Suraj Menon
> Priority: Minor
> Fix For: 0.7.0
>
> Attachments: HAMA-559.patch-v1, HAMA-559.patch-v2,
> spillbench_code.tar.gz, spilling_buffer_cpu_usage_text_write.png,
> SpillingBufferProfile-2012-10-27.snapshot,
> spilling_buffer_profile_cpu_graph_test_write.png,
> spilling_buffer_profile_cpugraph_writeUTF.png,
> spillingbuffer_profile_cpu_writeUTF.png, spilling_buffer_profile_LOCK.JPG,
> spilling_buffer_profile_timesplit_text_write.png,
> spilling_buffer_profile_writeUTF.png
>
>
> After HAMA-521 is done, we can add a spilling queue which just holds the
> messages in RAM that fit into the heap space. The rest can be flushed to disk.
> We may call this a HybridQueue or something like that.
> The benefits should be that we don't have to flush to disk so often and get
> faster. However we may have more GC so it is always overall faster.
> The requirements for this queue also include:
> - The message object once written to the queue (after returning from the
> write call) could be modified, but the changes should not be reflected in the
> messages stored in the queue.
> - For now let's implement a queue that does not support concurrent reading
> and writing. This feature is needed when we implement asynchronous
> communication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
