[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673492#comment-15673492 ] Hudson commented on HBASE-17085: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1968 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1968/]) HBASE-17085 AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync (zhangduo: rev ab55bad89995c5c34ab0e57e6d5fb9b6b799dfb0) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestSyncFuture.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SyncFuture.java > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085-v2.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15672782#comment-15672782 ] stack commented on HBASE-17085: --- +1 Did a quick pass. Don't forget to undo this: defaultProvider(FSHLogProvider.class), 77 defaultProvider(AsyncFSWALProvider.class), > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085-v2.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15672449#comment-15672449 ] Duo Zhang commented on HBASE-17085: --- Let's fix this issue first? [~stack] [~ram_krish]. I will try other methods to aggregate more syncs after this(limit concurrent syncs, add a delay before issuing sync, etc.). Thanks. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085-v2.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669898#comment-15669898 ] Duo Zhang commented on HBASE-17085: --- For me it does not make any difference... What's the batch size you use? > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085-v2.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669853#comment-15669853 ] ramkrishna.s.vasudevan commented on HBASE-17085: When I changed the wALPE config I could see that AsyncWAL with this patch applied performs atleast 2K ops/sec more than the default size. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085-v2.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669567#comment-15669567 ] ramkrishna.s.vasudevan commented on HBASE-17085: I thought it is only used in tests and not any where else. When I grepped I found that only. Let me check your patch now. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085-v2.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669456#comment-15669456 ] Duo Zhang commented on HBASE-17085: --- Yeah I mean the sync method without txid. It is also called at many places so we need to consider it. Your patch may cause infinite wait when sync() is called. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085-v2.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669360#comment-15669360 ] ramkrishna.s.vasudevan commented on HBASE-17085: My thing was based on the fact that since we call sync(txid), it is sure that when a sync call comes with txid 2 then surely we will have a append with txid 2. So that is when the sync will be called. I tested that with PE tool and things worked fine for me. So do you think of the case where sync() is called without txid? May be I did not test that case as PE actual case is with sync(txid). It can atleast reduce in the number of sync calls. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085-v2.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668746#comment-15668746 ] Duo Zhang commented on HBASE-17085: --- In HBASE-16890 it is 463/272=1.70 and here it is 613/386=1.59. So I think it helps a little? And see my latest comment in HBASE-17049, the sync count metrics of FSHLog and AsyncFSWAL can not be compared directly. Anyway, I will keep trying other methods to aggregate more syncs. Thanks [~stack]. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085-v2.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668744#comment-15668744 ] Duo Zhang commented on HBASE-17085: --- In HBASE-16890 it is 463/272=1.70 and here it is 613/386=1.59. So I think it helps a little? And see my latest comment in HBASE-17049, the sync count metrics of FSHLog and AsyncFSWAL can not be compared directly. Anyway, I will keep trying other methods to aggregate more syncs. Thanks [~stack]. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085-v2.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668360#comment-15668360
]
stack commented on HBASE-17085:
---
Default:
{code}
...
(Hard to copy in histograms because they were logged over..)
...
7555417 2016-11-15 12:14:25,321 INFO [main] wal.WALPerformanceEvaluation:
Summary: threads=100, iterations=10, syncInterval=0 took 382.163s
26166.846ops/s
7555418 2016-11-15 12:14:25,321 DEBUG [main] regionserver.HRegion: Closing
WALPerformanceEvaluation:0,,1479240482214.23f202a47ef65215f3d85a5fe08ce8e3.:
disabling compactions & flushes
7555419 2016-11-15 12:14:25,322 DEBUG [main] regionserver.HRegion: Updates
disabled for region
WALPerformanceEvaluation:0,,1479240482214.23f202a47ef65215f3d85a5fe08ce8e3.
7555420 2016-11-15 12:14:25,326 INFO
[StoreCloserThread-WALPerformanceEvaluation:0,,1479240482214.23f202a47ef65215f3d85a5fe08ce8e3.-1]
regionserver.HStore: Closed cf0
7555421 2016-11-15 12:14:25,326 INFO [main] regionserver.HRegion: Closed
WALPerformanceEvaluation:0,,1479240482214.23f202a47ef65215f3d85a5fe08ce8e3.
7555422 2016-11-15 12:14:25,328 DEBUG [main] wal.FSHLog: Closing WAL writer in
/user/stack/test-data/300d7166-b729-47da-965d-936ec5f7bb43/WALPerformanceEvaluation/WALs/wals
7555423 2016-11-15 12:14:25,328 DEBUG [main] hdfs.DFSClient: DFSClient
writeChunk allocating new packet seqno=1260314,
src=/user/stack/test-data/300d7166-b729-47da-965d-936ec5f7bb43/WALPerformanceEvaluation/WALs/wals/wals.1479240483655,
packetSize=65016, chunksPerPacket=126, bytesCurBlock=16408064
7555424 2016-11-15 12:14:25,328 DEBUG [main] hdfs.DFSClient: Queued packet
1260314
7555425 2016-11-15 12:14:25,328 DEBUG [main] hdfs.DFSClient: Queued packet
1260315
7555426 2016-11-15 12:14:25,328 DEBUG [main] hdfs.DFSClient: Waiting for ack
for: 1260315
7555427 2016-11-15 12:14:25,328 DEBUG [DataStreamer for file
/user/stack/test-data/300d7166-b729-47da-965d-936ec5f7bb43/WALPerformanceEvaluation/WALs/wals/wals.1479240483655
block BP-1837290273-10.17.240.20-1458945429978:blk_1077044408_3304141]
hdfs.DFSClient: DataStreamer block BP-18372902
73-10.17.240.20-1458945429978:blk_1077044408_3304141 sending packet packet
seqno: 1260314 offsetInBlock: 16408064 lastPacketInBlock: false
lastByteOffsetInBlock: 16408265
7555428 2016-11-15 12:14:25,329 DEBUG [ResponseProcessor for block
BP-1837290273-10.17.240.20-1458945429978:blk_1077044408_3304141]
hdfs.DFSClient: DFSClient seqno: 1260314 reply: SUCCESS downstreamAckTimeNanos:
0 flag: 0
7555429 2016-11-15 12:14:25,329 DEBUG [DataStreamer for file
/user/stack/test-data/300d7166-b729-47da-965d-936ec5f7bb43/WALPerformanceEvaluation/WALs/wals/wals.1479240483655
block BP-1837290273-10.17.240.20-1458945429978:blk_1077044408_3304141]
hdfs.DFSClient: DataStreamer block BP-18372902
73-10.17.240.20-1458945429978:blk_1077044408_3304141 sending packet packet
seqno: 1260315 offsetInBlock: 16408265 lastPacketInBlock: true
lastByteOffsetInBlock: 16408265
7555430 2016-11-15 12:14:25,329 DEBUG [ResponseProcessor for block
BP-1837290273-10.17.240.20-1458945429978:blk_1077044408_3304141]
hdfs.DFSClient: DFSClient seqno: 1260315 reply: SUCCESS downstreamAckTimeNanos:
0 flag: 0
7555431 2016-11-15 12:14:25,329 DEBUG [DataStreamer for file
/user/stack/test-data/300d7166-b729-47da-965d-936ec5f7bb43/WALPerformanceEvaluation/WALs/wals/wals.1479240483655
block BP-1837290273-10.17.240.20-1458945429978:blk_1077044408_3304141]
hdfs.DFSClient: Closing old block BP-183729027
3-10.17.240.20-1458945429978:blk_1077044408_3304141
7555432 2016-11-15 12:14:25,344 INFO [main] hbase.MockRegionServerServices:
Shutting down due to request 'test clean up.'
7555433 2016-11-15 12:14:25,344 INFO [main] wal.WALPerformanceEvaluation:
shutting down log roller.
7555434 2016-11-15 12:14:25,345 INFO [WALPerfEval.logRoller]
regionserver.LogRoller: LogRoller exiting.
7555435
7555436 Performance counter stats for './hbase/bin/hbase --config
/home/stack/conf_hbase org.apache.hadoop.hbase.wal.WALPerformanceEvaluation
-threads 100 -iterations 10 -qualifiers 25 -keySize 50 -valueSize 200':
7555437
7555438 4987370.861017 task-clock (msec) # 12.922 CPUs utilized
7555439 18,957,350 context-switches #0.004 M/sec
7555440 2,815,124 cpu-migrations#0.564 K/sec
7555441 10,054,046 page-faults #0.002 M/sec
7555442 9,934,495,287,070 cycles#1.992 GHz
7555443 stalled-cycles-frontend
7555444 stalled-cycles-backend
7555445 3,796,677,865,651 instructions #0.38 insns per cycle
7555446608,865,750,776 branches # 122.082 M/sec
7555447 7,478,401,535 branch-misses #1.23% of all branches
7555448
7555449 385.959774164 seconds time elapsed
{code}
With patch:
{code}
36055
36056 -- Histograms
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667276#comment-15667276
]
Hadoop QA commented on HBASE-17085:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 20s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 27s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 138m 15s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
| | hadoop.hbase.regionserver.TestFSErrorsExposed |
| Timed out junit tests | org.apache.hadoop.hbase.TestIOFencing |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838960/HBASE-17085-v2.patch |
| JIRA Issue | HBASE-17085 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 9aa06e3e2c74 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4d1bff9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4474/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4474/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4474/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4474/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> AsyncFSWAL may issue
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666980#comment-15666980 ] Duo Zhang commented on HBASE-17085: --- [~stack] Could you please try this patch to see if it helps for you? Thanks. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085-v2.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665544#comment-15665544 ] Duo Zhang commented on HBASE-17085: --- If there is no operation after the sync, then the newHighestProcessedAppendTxid will always be 1 and the sync will never be finished? > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664407#comment-15664407 ] ramkrishna.s.vasudevan commented on HBASE-17085: I mean when the txid becomes 2 > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664405#comment-15664405 ] ramkrishna.s.vasudevan commented on HBASE-17085: No prob. When this becomes 2 then sync Wil be called. It is more to minimise sync calls. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664344#comment-15664344
]
Hadoop QA commented on HBASE-17085:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 54s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 53s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 136m 59s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestFSErrorsExposed |
| Timed out junit tests | org.apache.hadoop.hbase.TestHBaseTestingUtility |
| | org.apache.hadoop.hbase.master.TestTableLockManager |
| | org.apache.hadoop.hbase.TestIOFencing |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838758/HBASE-17085-v2.patch |
| JIRA Issue | HBASE-17085 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux cd2526dea5f3 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 9250bf8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4459/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4459/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4459/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4459/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664037#comment-15664037
]
Duo Zhang commented on HBASE-17085:
---
With a larger batch size the total sync count is much less. But it does not
increase the performance.
{noformat}
Summary: threads=100, iterations=25, syncInterval=0 took 875.985s
28539.303ops/s
-- Meters --
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.appendMeter.bytes
count = 173799765420
mean rate = 199006089.64 events/second
1-minute rate = 201041305.95 events/second
5-minute rate = 191330046.98 events/second
15-minute rate = 139227963.92 events/second
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncMeter.syncs
count = 1019110
mean rate = 1166.91 events/second
1-minute rate = 1178.01 events/second
5-minute rate = 1118.39 events/second
15-minute rate = 792.59 events/second
{noformat}
> AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
>
>
> Key: HBASE-17085
> URL: https://issues.apache.org/jira/browse/HBASE-17085
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch,
> HBASE-17085.patch
>
>
> The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync
> if syncFutures is not empty. The SyncFutures in syncFutures can only be
> removed after an AsyncDFSOutput.sync comes back, so before the
> AsyncDFSOutput.sync actually returns, we will always issue an
> AsyncDFSOutput.sync after an append even if there is no new sync request.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664003#comment-15664003 ] Duo Zhang commented on HBASE-17085: --- I tried to set the batchSize to 1MB but got a worse result for PE. For WALPE, it does reduce the total sync count. Let me see if it could make WALPE run faster. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663916#comment-15663916 ] Duo Zhang commented on HBASE-17085: --- What if I have an append with txid 1 and a sync with txid 2? > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663897#comment-15663897
]
ramkrishna.s.vasudevan commented on HBASE-17085:
I tried setting 128MB but still not consistent results. But may be need to
check with bigger data size. In your test can you change the batch size and run
PE? If it performs better it is good.
And in your patch there is a suggestion. Pls check
{code}
if(!syncFutures.isEmpty()) {
long lastTxid = syncFutures.last().getTxid();
if (lastTxid > highestProcessedAppendTxidAtLastSync) {
long temp = newHighestProcessedAppendTxid;
if((temp + 1) > lastTxid)
// we have at least one sync request
sync(writer);
}
}
{code}
See if this is fine. The reason is that we can aggregate more before calling
sync.
> AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
>
>
> Key: HBASE-17085
> URL: https://issues.apache.org/jira/browse/HBASE-17085
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17085-v1.patch, HBASE-17085.patch
>
>
> The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync
> if syncFutures is not empty. The SyncFutures in syncFutures can only be
> removed after an AsyncDFSOutput.sync comes back, so before the
> AsyncDFSOutput.sync actually returns, we will always issue an
> AsyncDFSOutput.sync after an append even if there is no new sync request.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663875#comment-15663875 ] Duo Zhang commented on HBASE-17085: --- As you said above, this is because the batch size. The default batchSize is 64KB, you could try to set a larger value to see if it helps. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663866#comment-15663866 ] ramkrishna.s.vasudevan commented on HBASE-17085: Our comments crossed. Yes what you see is right. But if you see the number of syncFutures it is quite high. My point was that why is that we have so many syncFutures whereas the append txid trailing so much behind. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663856#comment-15663856 ] ramkrishna.s.vasudevan commented on HBASE-17085: 'highestProcessedAppendTxid' increases. But my point was that when the code comes to appendAndSync when we try to process one highestProcessedAppendTxid at that time we have almost 40 to 50 sync future that is greater than the current 'highestProcessedAppendTxid'. so again when the next appendAndSync happens we advance to the next 'highestProcessedAppendTxid' say from 300 to 304 but the syncFuture is now with 375 . Just for eg am saying. So we have lot of syncFuture waiting to get to the done state. Am away mostly for the coming days. Next two days can be online for sometime after that on a vacation. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663855#comment-15663855
]
Duo Zhang commented on HBASE-17085:
---
[~ramkrishna] Could you please confirm your code again? Do you use the latest
master and then apply the v1 patch? I've added a log in sync method and tried a
WALPE run. The txid of the last SyncFuture in syncFutures maybe the same
between calls, but the currentHighestProcessedAppendTxid always increases.
Thanks.
{code}
LOG.info("=" + fileLengthAtLastSync + ":" +
currentHighestProcessedAppendTxid + ":" + (syncFutures.isEmpty() ? -1 :
syncFutures.last().getTxid()));
{code}
fileLength:currentHighestProcessedAppendTxid:syncFutures.last().getTxid()
{noformat}
2016-11-14 20:57:38,644 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=18954972:16712758:16712902
2016-11-14 20:57:38,644 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=19025752:16712778:16712902
2016-11-14 20:57:38,645 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=19096532:16712798:16712902
2016-11-14 20:57:38,645 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=19167312:16712819:16712902
2016-11-14 20:57:38,645 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=19238092:16712839:16712902
2016-11-14 20:57:38,645 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=19308872:16712857:16712902
2016-11-14 20:57:38,646 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=19379652:16712879:16712902
2016-11-14 20:57:38,646 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=19450432:16712899:16712902
2016-11-14 20:57:38,646 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=19457510:16712900:16712902
2016-11-14 20:57:38,646 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=19528290:16712921:16713043
2016-11-14 20:57:38,647 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=19599070:16712941:16713078
2016-11-14 20:57:38,647 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=19669850:16712960:16713078
2016-11-14 20:57:38,647 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=19740630:16712980:16713078
2016-11-14 20:57:38,647 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=19811410:16713000:16713078
2016-11-14 20:57:38,648 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=19882190:16713020:16713078
2016-11-14 20:57:38,648 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=19952970:16713040:16713078
2016-11-14 20:57:38,648 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=20023750:16713060:16713078
2016-11-14 20:57:38,648 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=20080374:16713077:16713078
2016-11-14 20:57:38,649 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=20151154:16713097:16713248
2016-11-14 20:57:38,649 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=20221934:16713117:16713258
2016-11-14 20:57:38,650 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=20292714:16713137:16713258
2016-11-14 20:57:38,650 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=20363494:16713157:16713258
2016-11-14 20:57:38,650 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=20434274:16713177:16713258
2016-11-14 20:57:38,650 INFO [AsyncFSWAL-pool1-t1] wal.AsyncFSWAL:
=20505054:16713196:16713258
{noformat}
> AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
>
>
> Key: HBASE-17085
> URL: https://issues.apache.org/jira/browse/HBASE-17085
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17085-v1.patch, HBASE-17085.patch
>
>
> The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync
> if syncFutures is not empty. The SyncFutures in syncFutures can only be
> removed after an AsyncDFSOutput.sync comes back, so before the
> AsyncDFSOutput.sync actually returns, we will always issue an
> AsyncDFSOutput.sync after an append even if there is no new sync request.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663732#comment-15663732 ] Duo Zhang commented on HBASE-17085: --- This is a bit awkward. If highestProcessedAppendTxid does not increase, we should not call sync again as the writer.getLength will not increase so it will be equal to fileLengthAtLastSync. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663725#comment-15663725 ] ramkrishna.s.vasudevan commented on HBASE-17085: No. Under sync(AsyncWriter) before we call writer.sync. Not in syncCompleted(). The reason may be because we break if the batchSize is reached and so we get a list of appends which is lesser than what is there in sync. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663541#comment-15663541 ] Duo Zhang commented on HBASE-17085: --- Where do you put the logging code? At syncCompleted? > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663473#comment-15663473
]
ramkrishna.s.vasudevan commented on HBASE-17085:
This patch did not help me much. But how ever I can see that the sync is very
slow in catching up with the appends.
At any point of time the appends highestProcessedAppendTxid is very less than
the number of items in the syncfuture.
For eg
{code}
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33525
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33527
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33529
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33531
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33533
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33535
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33537
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33539
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33541
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33543
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33544
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33545
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33548
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33551
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33552
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33553
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33557
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33559
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33560
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33561
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33565
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33567
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33569
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33571
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33573
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33575
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33577
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33579
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33581
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33582
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33583
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33587
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33589
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33590
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33593
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33595
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33597
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33598
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33601
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33603
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33605
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33607
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33609
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33611
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33613
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33614
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33617
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33619
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33620
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33623
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33625
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33627
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33629
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33631
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33632
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33635
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33637
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33639
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33641
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33642
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33645
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33647
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33649
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33651
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33653
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33655
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33657
currenthighestProcessedAppendTxidSync 33527 doneTxid 33524 33659
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663000#comment-15663000
]
Hadoop QA commented on HBASE-17085:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 58s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 117m 0s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService
|
| |
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithCustomVisLabService
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838709/HBASE-17085-v1.patch |
| JIRA Issue | HBASE-17085 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 5e271956dbf9 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 9250bf8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4452/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4452/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4452/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4452/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
>
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662819#comment-15662819
]
ramkrishna.s.vasudevan commented on HBASE-17085:
Am just seeing your -v1 patch.
{code}
if (!syncFutures.isEmpty()
457 && syncFutures.last().getTxid() >
highestProcessedAppendTxidAtLastSync) {
{code}
This seems to be your fix. I can try your patch in my cluster.
{code}
And there is no duplicated record. It is just a pipeline.
{code}
bq.we will always issue an AsyncDFSOutput.sync after an append even if there is
no new sync request.
True. So you mean that since append has happend only that new data will be
again called for sync. And because of that the number of syncs may be more and
here you plan to aggregate more.
> AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
>
>
> Key: HBASE-17085
> URL: https://issues.apache.org/jira/browse/HBASE-17085
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17085-v1.patch, HBASE-17085.patch
>
>
> The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync
> if syncFutures is not empty. The SyncFutures in syncFutures can only be
> removed after an AsyncDFSOutput.sync comes back, so before the
> AsyncDFSOutput.sync actually returns, we will always issue an
> AsyncDFSOutput.sync after an append even if there is no new sync request.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662813#comment-15662813 ] Duo Zhang commented on HBASE-17085: --- I used to tried limiting the concurrent sync requests but it did not help. I think we could try again after this patch is applied. And there is no duplicated record. It is just a pipeline. In FSHLog we also use 5 threads to do syncing(That's why I limited the number of concurrent sync requests to 5) If no concern, let's commit the patch here first? Thanks. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662804#comment-15662804 ] ramkrishna.s.vasudevan commented on HBASE-17085: Can we try limiting the sync that happens? I know you tried with some sync count of 5. So even that did not help. And do think we had duplicate records created as we were syncing even if the current sync did not get completed. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661767#comment-15661767 ] ramkrishna.s.vasudevan commented on HBASE-17085: Thanks for the patch. I don't have system with me. I have seen this issue and was working on a fix. But first tried to do some workaround which did not pay off . My other doubt was if this happens then we will have duplicate entries or is it just empty packets. I was verifying that and as it was week end could not do that > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[
https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661756#comment-15661756
]
Hadoop QA commented on HBASE-17085:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 43s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 34s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 121m 47s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.mapred.TestTableMapReduceUtil
|
| | org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor |
| | org.apache.hadoop.hbase.master.TestWarmupRegion |
| | org.apache.hadoop.hbase.mapred.TestMultiTableSnapshotInputFormat |
| | org.apache.hadoop.hbase.TestPartialResultsFromClientSide |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838688/HBASE-17085.patch |
| JIRA Issue | HBASE-17085 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a6138a0d1f67 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3f919dd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4450/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4450/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4450/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
