[
https://issues.apache.org/jira/browse/HAMA-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488231#comment-13488231
]
Suraj Menon commented on HAMA-559:
----------------------------------
Made 3 more changes:
- Removed few ByteBuffer.remaining calls
- Relaxed the restriction that each buffer should fill completely before moving
to next. (I might revert back this change)
- Added a BufferedOutputStream in between SpillingBuffer and SpillingStream.
(This improved the efficiency by around 45%)
The performance numbers are now:
{noformat}
0% Scenario{vm=java, trial=0, benchmark=Spill, size=10, type=DISK_LIST}
3809.61 ns; σ=40.46 ns @ 10 trials
5% Scenario{vm=java, trial=0, benchmark=Spill, size=100, type=DISK_LIST}
7113.85 ns; σ=448.07 ns @ 10 trials
10% Scenario{vm=java, trial=0, benchmark=Spill, size=1000, type=DISK_LIST}
41011.04 ns; σ=1462.04 ns @ 10 trials
14% Scenario{vm=java, trial=0, benchmark=Spill, size=10000, type=DISK_LIST}
357103.20 ns; σ=350967.71 ns @ 10 trials
19% Scenario{vm=java, trial=0, benchmark=Spill, size=100000, type=DISK_LIST}
3606638.74 ns; σ=3339040.16 ns @ 10 trials
24% Scenario{vm=java, trial=0, benchmark=Spill, size=1000000, type=DISK_LIST}
34951212.41 ns; σ=1949660.29 ns @ 10 trials
29% Scenario{vm=java, trial=0, benchmark=Spill, size=10000000, type=DISK_LIST}
354762498.83 ns; σ=13826670.03 ns @ 10 trials
33% Scenario{vm=java, trial=0, benchmark=Spill, size=10, type=SPILLING_BUFFER}
3914.27 ns; σ=17.48 ns @ 3 trials
38% Scenario{vm=java, trial=0, benchmark=Spill, size=100, type=SPILLING_BUFFER}
6114.64 ns; σ=60.98 ns @ 10 trials
43% Scenario{vm=java, trial=0, benchmark=Spill, size=1000,
type=SPILLING_BUFFER} 31973.78 ns; σ=143.74 ns @ 3 trials
48% Scenario{vm=java, trial=0, benchmark=Spill, size=10000,
type=SPILLING_BUFFER} 271619.92 ns; σ=19861.84 ns @ 10 trials
52% Scenario{vm=java, trial=0, benchmark=Spill, size=100000,
type=SPILLING_BUFFER} 2580930.24 ns; σ=3416159.19 ns @ 10 trials
57% Scenario{vm=java, trial=0, benchmark=Spill, size=1000000,
type=SPILLING_BUFFER} 25714962.79 ns; σ=45184.42 ns @ 3 trials
62% Scenario{vm=java, trial=0, benchmark=Spill, size=10000000,
type=SPILLING_BUFFER} 259875751.00 ns; σ=19098186.41 ns @ 10 trials
67% Scenario{vm=java, trial=0, benchmark=Spill, size=10, type=DISK_BUFFER}
3294.61 ns; σ=10.86 ns @ 3 trials
71% Scenario{vm=java, trial=0, benchmark=Spill, size=100, type=DISK_BUFFER}
6442.84 ns; σ=18.10 ns @ 3 trials
76% Scenario{vm=java, trial=0, benchmark=Spill, size=1000, type=DISK_BUFFER}
38160.64 ns; σ=250.75 ns @ 3 trials
81% Scenario{vm=java, trial=0, benchmark=Spill, size=10000, type=DISK_BUFFER}
333411.11 ns; σ=4376.56 ns @ 10 trials
86% Scenario{vm=java, trial=0, benchmark=Spill, size=100000, type=DISK_BUFFER}
3214465.20 ns; σ=8146.50 ns @ 3 trials
90% Scenario{vm=java, trial=0, benchmark=Spill, size=1000000, type=DISK_BUFFER}
30817421.31 ns; σ=21434.85 ns @ 3 trials
95% Scenario{vm=java, trial=0, benchmark=Spill, size=10000000,
type=DISK_BUFFER} 316989389.00 ns; σ=622679.65 ns @ 3 trials
size type us linear runtime
10 DISK_LIST 3.81 =
10 SPILLING_BUFFER 3.91 =
10 DISK_BUFFER 3.29 =
100 DISK_LIST 7.11 =
100 SPILLING_BUFFER 6.11 =
100 DISK_BUFFER 6.44 =
1000 DISK_LIST 41.01 =
1000 SPILLING_BUFFER 31.97 =
1000 DISK_BUFFER 38.16 =
10000 DISK_LIST 357.10 =
10000 SPILLING_BUFFER 271.62 =
10000 DISK_BUFFER 333.41 =
100000 DISK_LIST 3606.64 =
100000 SPILLING_BUFFER 2580.93 =
100000 DISK_BUFFER 3214.47 =
1000000 DISK_LIST 34951.21 ==
1000000 SPILLING_BUFFER 25714.96 ==
1000000 DISK_BUFFER 30817.42 ==
10000000 DISK_LIST 354762.50 ==============================
10000000 SPILLING_BUFFER 259875.75 =====================
10000000 DISK_BUFFER 316989.39 ==========================
vm: java
trial: 0
benchmark: Spill
{noformat}
> Add a spilling message queue
> ----------------------------
>
> Key: HAMA-559
> URL: https://issues.apache.org/jira/browse/HAMA-559
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.5.0
> Reporter: Thomas Jungblut
> Assignee: Suraj Menon
> Priority: Minor
> Fix For: 0.7.0
>
> Attachments: HAMA-559.patch-v1,
> spilling_buffer_cpu_usage_text_write.png,
> SpillingBufferProfile-2012-10-27.snapshot,
> spilling_buffer_profile_cpu_graph_test_write.png,
> spilling_buffer_profile_cpugraph_writeUTF.png,
> spillingbuffer_profile_cpu_writeUTF.png, spilling_buffer_profile_LOCK.JPG,
> spilling_buffer_profile_timesplit_text_write.png,
> spilling_buffer_profile_writeUTF.png
>
>
> After HAMA-521 is done, we can add a spilling queue which just holds the
> messages in RAM that fit into the heap space. The rest can be flushed to disk.
> We may call this a HybridQueue or something like that.
> The benefits should be that we don't have to flush to disk so often and get
> faster. However we may have more GC so it is always overall faster.
> The requirements for this queue also include:
> - The message object once written to the queue (after returning from the
> write call) could be modified, but the changes should not be reflected in the
> messages stored in the queue.
> - For now let's implement a queue that does not support concurrent reading
> and writing. This feature is needed when we implement asynchronous
> communication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
