[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116254#comment-16116254 ] ramkrishna.s.vasudevan commented on HBASE-17049: bq.Old WAL impl is having 5 syncer threads by default and AsyncWAL having 1. Correct? Ur tests changed any of these Ram? No. AsyncFSWAL will have only 1 thread anyway because it depends on the Netty's model now. I will reconfirm with more tests this week. > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116090#comment-16116090 ] Anoop Sam John commented on HBASE-17049: Old WAL impl is having 5 syncer threads by default and AsyncWAL having 1. Correct? Ur tests changed any of these Ram? > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112174#comment-16112174 ] ramkrishna.s.vasudevan commented on HBASE-17049: Some latest tests showed AsyncWAL to be faster. Let me reconfirm later next week with more tests. > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069215#comment-16069215 ] stack commented on HBASE-17049: --- Want this on by default in 2.0. And offheap write path wants it too. Making it critical. > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019647#comment-16019647 ] Anoop Sam John commented on HBASE-17049: Thats good to see Ram.. I will also try with different cases. Hope we can change our default WAL to be asyn wal for 2.0. :-) > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > Attachments: delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[
https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019432#comment-16019432
]
ramkrishna.s.vasudevan commented on HBASE-17049:
I tried running a simple PE test with both asyncfs and default mode. I think on
latest trunk asyncfs wal works faster and has an increased performance.
{code}
asyncfswal
Min: 745028ms Max: 786664ms Avg: 778270ms
default
Min: 848269ms Max: 880460ms Avg: 871433ms
{code}
> Find out why AsyncFSWAL issues much more syncs than FSHLog
> --
>
> Key: HBASE-17049
> URL: https://issues.apache.org/jira/browse/HBASE-17049
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: delay-sync.patch
>
>
> https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15725863#comment-15725863 ] stack commented on HBASE-17049: --- Will do. Doing other tests at mo. Will be back to this soon. > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > Attachments: delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[
https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15725574#comment-15725574
]
Duo Zhang commented on HBASE-17049:
---
I planned to run a single node HBase on a distributed HDFS, but the testing
node which is in the same DC of the HDFS cluster is used by others these days.
Will try this later, or I will just run a PE on a fully distributed
cluster(both HBase and HDFS).
And I have run a single node PE test with multiwal, BoundedGroupingStrategy, 8
wals.
{noformat}
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred
--presplit=50 --size=50 --columns=50 --valueSize=200 --writeToWAL=true
--bloomFilter=NONE randomWrite 50
FSHLog: Min: 392959ms Max: 413928ms Avg: 406991ms
AsyncFSWAL: Min: 359566ms Max: 397469ms Avg: 391964ms
{noformat}
AsyncFSWAL is a bit faster. So I think this is the right way to increase the
performance of AsyncFSWAL.
[~stack] Can you try to see if this also works for you sir? Both [~ram_krish]
and I can get a better result with multiwal(although in ram's test AsyncFSWAL
is still a bit slower).
Thanks.
> Find out why AsyncFSWAL issues much more syncs than FSHLog
> --
>
> Key: HBASE-17049
> URL: https://issues.apache.org/jira/browse/HBASE-17049
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: delay-sync.patch
>
>
> https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723987#comment-15723987 ] Duo Zhang commented on HBASE-17049: --- [~ram_krish] Busy these days, haven't got enough time to setup the test environment yet... > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > Attachments: delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721989#comment-15721989 ] ramkrishna.s.vasudevan commented on HBASE-17049: [~Apache9] Any updates here? > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > Attachments: delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695391#comment-15695391 ] Duo Zhang commented on HBASE-17049: --- Good. At least we are making progress here. And I'd say sorry that I have been stuck by an online service accident so I haven't got enough time to run the multi node test... Will test it this weekend... > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > Attachments: delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[
https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695368#comment-15695368
]
ramkrishna.s.vasudevan commented on HBASE-17049:
With 50G of data still asyncWAL is slower with single node.
{code}
fshlog - multiwal
Min: 364747ms Max: 381766ms Avg: 377440ms
asyncfs - multiwal
Min: 442857ms Max: 459009ms Avg: 454908ms
{code}
But it is not 2x slower as it used to be.
> Find out why AsyncFSWAL issues much more syncs than FSHLog
> --
>
> Key: HBASE-17049
> URL: https://issues.apache.org/jira/browse/HBASE-17049
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: delay-sync.patch
>
>
> https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695104#comment-15695104 ] ramkrishna.s.vasudevan commented on HBASE-17049: When we try with multiWAL and numgroups as 4 we are getting better performance almost in line with FSHLOG with multiWAL. I just tried with 10G data. Will try with bigger data. > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > Attachments: delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[
https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15693114#comment-15693114
]
Duo Zhang commented on HBASE-17049:
---
{quote}
Do you mean multiWAL per region?
{quote}
For a single region it is impossible to use more than one WAL, but I think in
both PE and WALPE we are using at least 50 regions in our current tests?
Yeah I have a cluster with 7 nodes, I could test AsyncFSWAL on it.
Thanks.
> Find out why AsyncFSWAL issues much more syncs than FSHLog
> --
>
> Key: HBASE-17049
> URL: https://issues.apache.org/jira/browse/HBASE-17049
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: delay-sync.patch
>
>
> https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692987#comment-15692987 ] ramkrishna.s.vasudevan commented on HBASE-17049: Do you mean multiWAL per region? Do you have the infra to run with 3 nodes with AsyncWAL (and with PE) to see how much Fanout is helping us. I think may be as there is only one node and there are flushes and compactions this eventloop that waits for response from DN is getting delayed for the response. > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > Attachments: delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692257#comment-15692257 ] ramkrishna.s.vasudevan commented on HBASE-17049: Larger batch size with WALPE helped me. With default batch size I could see lesser performance. I could verify that. Let me check HBASE-17048. > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > Attachments: delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[
https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673648#comment-15673648
]
Duo Zhang commented on HBASE-17049:
---
Delay sync is my last trick to aggregate more syncs. If it does not work, then
I have no idea...
As in the perf result posted by [~stack], it seems that AsyncFSWAL has less
instructions but cost more time. I think it means that the AsyncFSWAL can not
make use of all the cores on the machine. FSHLog does better than us. And I
think it is reasonable. We use one thread to do everything, but FSHLog has one
disruptor thread, five sync threads, one DataStreamer thread, and one Responder
thread.
{noformat}
FSHLog
4987370.861017 task-clock (msec) # 12.922 CPUs utilized
9,934,495,287,070 cycles #1.992 GHz
3,796,677,865,651 instructions #0.38 insns per cycle
AsyncFSWAL
4568572.588814 task-clock (msec) #7.454 CPUs utilized
9,292,754,813,201 cycles #2.034 GHz
3,245,931,999,369 instructions #0.35 insns per cycle
{noformat}
So I think we should try multiwal next? With multiwal, we will have multiple
AsyncFSWAL instance thus we could make use of more cores.
What do you guys think? [~stack] [~ram_krish].
Thanks.
> Find out why AsyncFSWAL issues much more syncs than FSHLog
> --
>
> Key: HBASE-17049
> URL: https://issues.apache.org/jira/browse/HBASE-17049
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: delay-sync.patch
>
>
> https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669890#comment-15669890 ] Duo Zhang commented on HBASE-17049: --- For FSHLog, the sync is done inside DFSOutputStream automatically if the buffer is full so we can not record it with our metrics. > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669845#comment-15669845 ] ramkrishna.s.vasudevan commented on HBASE-17049: One question though. The WALPE histogram should give a correct picture on the sync part, correct? Because they are calculated in the postSync API? > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669565#comment-15669565 ] ramkrishna.s.vasudevan commented on HBASE-17049: This is valid I think. Also almost for every 3 or 4 appends we create one packet and that is being synced. > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669434#comment-15669434 ] stack commented on HBASE-17049: --- Agree. Compare packet sizes and their rates is next I think. Let me see if I can help here. > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667229#comment-15667229 ] Duo Zhang commented on HBASE-17049: --- I think I found a possible reason that why AsyncFSWAL issues 'more' syncs than FSHLog. For FSHLog, if the currentPacket is full, then it will be queued to the dataQueue and processed by DataStreamer. We do not consider this as a sync. But for AsyncFSWAL, every packet that sent to DN will be included when compute the sync count. So the comparison of sync count in our WAL metrics is meaningless. We need to get the number of packets sent to DN for FSHLog and compare it with the sync count for AsyncFSWAL. Thanks. > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[
https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662787#comment-15662787
]
Duo Zhang commented on HBASE-17049:
---
With HBASE-17085 we can perform a little better in the PE.
{noformat}
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred
--presplit=50 --size=50 --columns=50 --valueSize=200 --writeToWAL=true
--bloomFilter=NONE randomWrite 50
FSHLog: Min: 494940ms Max: 527185ms Avg: 520070ms
AsyncFSWAL-old: Min: 899553ms Max: 937701ms Avg: 931112ms
AsyncFSWAL-new:Min: 745193ms Max: 770322ms Avg: 764724ms
{noformat}
Still slower than FSHLog. Need to find out more.
Thanks.
> Find out why AsyncFSWAL issues much more syncs than FSHLog
> --
>
> Key: HBASE-17049
> URL: https://issues.apache.org/jira/browse/HBASE-17049
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
> Fix For: 2.0.0
>
>
> https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661591#comment-15661591 ] Duo Zhang commented on HBASE-17049: --- HBASE-17085 fix a problem which may cause unnecessary DFSOutput.sync. Will test the performance with the patch in place. > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
