[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15166864#comment-15166864 ] Hudson commented on HBASE-15264: FAILURE: Integrated in HBase-Trunk_matrix #737 (See [https://builds.apache.org/job/HBase-Trunk_matrix/737/]) HBASE-15264 Implement a fan out HDFS OutputStream (zhangduo: rev 6e9d355b12a1e666f4d05be02775a01b6754d063) * hbase-server/src/test/java/org/apache/hadoop/hbase/util/FanOutOneBlockAsyncDFSOutputFlushHandler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/FanOutOneBlockAsyncDFSOutputHelper.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFanOutOneBlockAsyncDFSOutput.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/FanOutOneBlockAsyncDFSOutput.java > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264-v5.patch, > HBASE-15264-v6.patch, HBASE-15264-v7.patch, HBASE-15264-v8.patch, > HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163521#comment-15163521
]
Hadoop QA commented on HBASE-15264:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 20s
{color} | {color:red} Patch generated 9 new checkstyle issues in hbase-server
(total was 0, now 9). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 58s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 109m 34s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 109m 28s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 269m 38s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Timed out junit tests |
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
| | org.apache.hadoop.hbase.regionserver.TestHRegion |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-24 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12789551/HBASE-15264-v8.patch |
| JIRA Issue | HBASE-15264 |
| Optional Tests | asflicense javac javadoc unit findbugs
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163164#comment-15163164 ] stack commented on HBASE-15264: --- Says [WARNING] /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FanOutOneBlockAsyncDFSOutput.java:100: warning - End Delimiter } missing for possible See Tag in comment string: "An asynchronous HDFS output stream implementation which fans out data to datanode and only You can fix on commit [~Apache9] > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264-v5.patch, > HBASE-15264-v6.patch, HBASE-15264-v7.patch, HBASE-15264-v8.patch, > HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163156#comment-15163156
]
Hadoop QA commented on HBASE-15264:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 18s
{color} | {color:red} Patch generated 66 new checkstyle issues in hbase-server
(total was 0, now 66). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 44s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 27s
{color} | {color:red} hbase-server-jdk1.8.0_72 with JDK v1.8.0_72 generated 1
new issues (was 1, now 2). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 3m 1s
{color} | {color:red} hbase-server-jdk1.7.0_95 with JDK v1.7.0_95 generated 1
new issues (was 1, now 2). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 4s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 81m 42s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 218m 50s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests | hadoop.hbase.mapreduce.TestImportExport |
| JDK v1.8.0_72 Timed out
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15162923#comment-15162923 ] Duo Zhang commented on HBASE-15264: --- Oh the problem is the @link tag. If I remove it the warning will be gone. Let me rewrite that part. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264-v5.patch, > HBASE-15264-v6.patch, HBASE-15264-v7.patch, HBASE-15264-v8.patch, > HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15162906#comment-15162906 ] Duo Zhang commented on HBASE-15264: --- Let me see what's wrong with the javadoc... > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264-v5.patch, > HBASE-15264-v6.patch, HBASE-15264-v7.patch, HBASE-15264-v8.patch, > HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15162901#comment-15162901
]
Hadoop QA commented on HBASE-15264:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 5m 0s
{color} | {color:red} Patch generated 66 new checkstyle issues in hbase-server
(total was 0, now 66). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 19s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 3m 32s
{color} | {color:red} hbase-server-jdk1.8.0_72 with JDK v1.8.0_72 generated 1
new issues (was 1, now 2). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 4m 21s
{color} | {color:red} hbase-server-jdk1.7.0_95 with JDK v1.7.0_95 generated 1
new issues (was 1, now 2). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 142m 36s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 139m 44s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 345m 31s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests |
hadoop.hbase.client.TestBlockEvictionFromClient |
| JDK v1.7.0_95 Timed
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160164#comment-15160164 ] stack commented on HBASE-15264: --- Ugh. Forget what i said about checkstyle. I forgot what the errors were. Ignore. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264-v5.patch, > HBASE-15264-v6.patch, HBASE-15264-v7.patch, HBASE-15264-v8.patch, > HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160060#comment-15160060 ] Duo Zhang commented on HBASE-15264: --- Sure, I will fix the javadoc issue. For checkstyle, I could fix some. And for the wrong import order error, what is the suggested import order for HBase project? I just follow the eclipse rule... The UT failures are unrelated since I do not change the existing code... > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264-v5.patch, > HBASE-15264-v6.patch, HBASE-15264-v7.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159960#comment-15159960 ] stack commented on HBASE-15264: --- Patch looks good to me. Can fix the checkstyle and javadoc on commit (if you can). The unit test failures are unrelated? If so, go for it [~Apache9] > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264-v5.patch, > HBASE-15264-v6.patch, HBASE-15264-v7.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159955#comment-15159955 ] stack commented on HBASE-15264: --- Ok. Just go for master. Makes sense. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264-v5.patch, > HBASE-15264-v6.patch, HBASE-15264-v7.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159930#comment-15159930
]
Hadoop QA commented on HBASE-15264:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 59s
{color} | {color:red} Patch generated 11 new checkstyle issues in hbase-server
(total was 0, now 11). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 42s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 3m 40s
{color} | {color:red} hbase-server-jdk1.8.0_72 with JDK v1.8.0_72 generated 1
new issues (was 1, now 2). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 4m 28s
{color} | {color:red} hbase-server-jdk1.7.0_95 with JDK v1.7.0_95 generated 1
new issues (was 1, now 2). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 150m 13s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 148m 49s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 384m 9s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Timed out junit tests |
org.apache.hadoop.hbase.client.TestHTableMultiplexer |
| JDK
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158505#comment-15158505
]
Duo Zhang commented on HBASE-15264:
---
Commit this on master is a bit simple for me since I haven't worked well on a
branch yet. Does review board and pre-commit check still work for a commit on
branch other than the several main branches(the pre-commit is broken when I was
doing HDFS-7966...). And the permissions of git repo... If these are all
simple, I would prefer doing this on a feature branch since it is more
flexible. If it is troublesome then I would say just doing this on master since
we only have one big commit, not a series of new patches :)
{quote}
That is a fat bunch of params to pass the constructor:
312 FanOutOneBlockAsyncDFSOutput(Configuration conf, FSUtils fsUtils,
DistributedFileSystem dfs,
313 DFSClient client, ClientProtocol namenode, String clientName, String
src, long fileId,
314 LocatedBlock locatedBlock, EventLoop eventLoop, List
datanodeList,
315 DataChecksum summer, ByteBufAllocator alloc) {
{quote}
The constructor is not suppose to be called by end user, we still need a
private constructor with the builder pattern, so I think you are saying the
Helper.createOutput method. It is same with FileSystem.createNonRecursive
method. But this is a good suggestion, can change to a builder pattern when
moving it to HDFS.
{quote}
What did we say about this:
ByteBuf checksumBuf = alloc.directBuffer(checksumLen);
That we are allocating offheap. when writing always The offheap allocations
are relatively slow... Can use a pool later?
... and here: ByteBuf newBuf =
alloc.directBuffer().ensureWritable(trailingPartialChunkLen);
{quote}
This is done by netty itself. I use PooledByteBufAllocator.
{code:title=FanOutOneBlockAsyncDFSOutputHelper .java}
private static final ByteBufAllocator ALLOC = PooledByteBufAllocator.DEFAULT;
{code}
Will add more comments on how things are going on here. Thanks.
> Implement a fan out HDFS OutputStream
> -
>
> Key: HBASE-15264
> URL: https://issues.apache.org/jira/browse/HBASE-15264
> Project: HBase
> Issue Type: Sub-task
> Components: util, wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch,
> HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264-v5.patch,
> HBASE-15264-v6.patch, HBASE-15264.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158447#comment-15158447
]
stack commented on HBASE-15264:
---
[~Apache9] This patch is four new classes that do not touch any other code.
Could go into master. I think it would be better if the whole working fanout
WAL provider came in as a 'finished' lump so suggest putting these classes on a
'FanOutWAL' branch and then fill in the other necessary pieces around it. When
all running, all current WAL tests with this provider in place are passing, we
can pull into master branch.
But do what is easier on you making progress (master or branch).
I'm game for testing whenever you ready.
On the patch, ok on having the classes in util for reasons above (on commit,
add comment on why core i/o classes are off in a util package...).
That is a fat bunch of params to pass the constructor:
312 FanOutOneBlockAsyncDFSOutput(Configuration conf, FSUtils fsUtils,
DistributedFileSystem dfs,
313 DFSClient client, ClientProtocol namenode, String clientName,
String src, long fileId,
314 LocatedBlock locatedBlock, EventLoop eventLoop, List
datanodeList,
315 DataChecksum summer, ByteBufAllocator alloc) {
Thats how DFSOS takes params? Can't do builder because all params are needed?
FanOutOneBlockAsyncDFSOutput is clean, easy enough to follow what is going on
(I think -- smile -- if I can remember my netty-ese). The helper needs a bit of
commentary. You know how it all works at the moment. Would be good to get some
doc in there while fresh in your mind. Like how DFS_CLIENT_ADAPTOR works and
LeaseManager (Sorry you have to do all this reflection stuff but you have it
nicely contained)... even if it was just a note on why you have to do these
gymnastics and the basic technique you have employed.
What did we say about this:
ByteBuf checksumBuf = alloc.directBuffer(checksumLen);
That we are allocating offheap. when writing always The offheap
allocations are relatively slow... Can use a pool later?
... and here: ByteBuf newBuf =
alloc.directBuffer().ensureWritable(trailingPartialChunkLen);
Tests look good -- yeah, the create w/o create of parent is needed for the
fencing trick.
This looks great [~Apache9]
> Implement a fan out HDFS OutputStream
> -
>
> Key: HBASE-15264
> URL: https://issues.apache.org/jira/browse/HBASE-15264
> Project: HBase
> Issue Type: Sub-task
> Components: util, wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch,
> HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264-v5.patch,
> HBASE-15264-v6.patch, HBASE-15264.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158051#comment-15158051 ] Duo Zhang commented on HBASE-15264: --- [~stack] What do you think we should go next？ On master or a new branch? Thanks. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264-v5.patch, > HBASE-15264-v6.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156412#comment-15156412 ] Duo Zhang commented on HBASE-15264: --- The javadoc and license issue can be fixed when committing. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264-v5.patch, > HBASE-15264-v6.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156269#comment-15156269
]
Hadoop QA commented on HBASE-15264:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 26s
{color} | {color:red} Patch generated 11 new checkstyle issues in hbase-server
(total was 0, now 11). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 44s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 3m 18s
{color} | {color:red} hbase-server-jdk1.8.0_72 with JDK v1.8.0_72 generated 1
new issues (was 1, now 2). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 3m 59s
{color} | {color:red} hbase-server-jdk1.7.0_95 with JDK v1.7.0_95 generated 1
new issues (was 1, now 2). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 165m 46s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 92m 29s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_95.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 15s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 312m 23s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests |
hadoop.hbase.regionserver.TestEncryptionKeyRotation |
| |
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15155257#comment-15155257 ] Duo Zhang commented on HBASE-15264: --- I placed them in regionserver.wal on the POC branch. And here, as you said, I put it in util package to avoid the dependencies on WAL implementation. This could help when moving it to HDFS project. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15155254#comment-15155254 ] Duo Zhang commented on HBASE-15264: --- There is no problem if master renamed our WAL directory because we can not start a new WAL writer any more. This will finally cause the RS abort. And master will only split a log if it is closed(successfully returned from recoverFileLease) so we will not lose any data. asynchronously closing WAL writer is also OK I think. I used to miss the rename stage... Thanks. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15155163#comment-15155163 ] Duo Zhang commented on HBASE-15264: --- Here we just want to split a big patch into several little ones, so we could do the review-update circle in HBASE-15265? That's the place where we modify the existing code and may effect other one's work. It's OK to also modify the FanOutOneBlockAsyncDFSOutput in HBASE-15265 unless we have a very big change which almost rewrites it. So I prefer we commit this to master first and begin the testing work in HBASE-15265. The patch is ready for review(without the retry of creation and shutdown fix). Thanks. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154528#comment-15154528 ] stack commented on HBASE-15264: --- Lets do in a branch first? Then you can commit willy-nilly and we can bring it all into branch in one lump as working WAL provider alternative? Make a branch named for the umbrella JIRA and go for it. Here is some review on the patch: Why are your classes all in util and not in io or under regionserver.wal? These would be better locations? Though, looking through the imports, it is almost as though this set of classes could go into hbase-common rather than into hbase-server. That'd be cool. Might be good to do that so you can be sure you are not dragging in hbase dependency. Will make it easier then upstreaming your work to hdfs. Just a thought (On further review, I see dependency on FSUtil... moving to hbase-common is not important... can do later). Give your class javadoc a reread/edit. You'll find stuff to fix. That is good that you list differences vs default dfsos. ... will be back... > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154511#comment-15154511 ] stack commented on HBASE-15264: --- bq. then we should not start a new one? We might not be able to if the dir has been moved anyways. And IIRC, there is code in recovery that will look for files beyond the one that went 'bad' -- would need to check. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154509#comment-15154509 ] stack commented on HBASE-15264: --- So, WAL goes bad, we abandon it and replay unacked edits into new file. bq. if we failed to close the WAL file current being written , it is likely that we are considered dead on master. So, we should not be writing a new file that is subsequent to the WAL that broke and that we cannot reach anymore? If we 'lose' a WAL -- can't write it or close it -- then we should not start a new one? > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154502#comment-15154502 ] stack commented on HBASE-15264: --- Yes. Can backport working system to branch-1 once all is in. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154014#comment-15154014 ] Duo Zhang commented on HBASE-15264: --- So let's commit this and focus on HBASE-15265? [~stack] Thanks. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153990#comment-15153990 ] Duo Zhang commented on HBASE-15264: --- Oh, master will move the RS's log directory to another place to prevent further creating new WAL files... Seems OK then. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153951#comment-15153951 ] Duo Zhang commented on HBASE-15264: --- Oh, I forgot an important thing, we can not close WAL writer asynchronously and also we can not use recoverFileLease. The reason is that, we use the lease on WAL file to do fencing, which means that, if we failed to close the WAL file current being written , it is likely that we are considered dead on master. So the close must be synchronous, and also, we should close the file normally instead of calling recoverFileLease... Let me retry the recoverAndClose part... Thanks. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task > Components: util, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264-v4.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153845#comment-15153845
]
Hadoop QA commented on HBASE-15264:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 12s
{color} | {color:red} Patch generated 11 new checkstyle issues in hbase-server
(total was 0, now 11). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 83m 0s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 1s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 213m 9s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_95 Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-19 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12788602/HBASE-15264-v4.patch |
| JIRA Issue | HBASE-15264 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 6d6bbec1aec1
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151576#comment-15151576
]
Duo Zhang commented on HBASE-15264:
---
{quote}
You want to commit this Duo Zhang to master branch ahead of provider being done
or you want them all to go in at same time?
{quote}
I think this could go first. But before committing, I want to run some small
tests to confirm that the fan out output works well with the new WALProvider.
So at least I should have a basic patch for HBASE-15265 first(don't worry, it's
almost done)
Thanks.
> Implement a fan out HDFS OutputStream
> -
>
> Key: HBASE-15264
> URL: https://issues.apache.org/jira/browse/HBASE-15264
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch,
> HBASE-15264-v3.patch, HBASE-15264.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15150749#comment-15150749 ] stack commented on HBASE-15264: --- bq. Yes, Payload and RingBufferTruck are the same thing OK. When you have a chance, not a TODO comment so we remember to unify one day. You want to commit this [~Apache9] to master branch ahead of provider being done or you want them all to go in at same time? Thanks. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264-v3.patch, HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15150716#comment-15150716
]
Hadoop QA commented on HBASE-15264:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 7s
{color} | {color:red} Patch generated 12 new checkstyle issues in hbase-server
(total was 0, now 12). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
21m 33s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 79m 27s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 79m 45s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 201m 11s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-17 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12788237/HBASE-15264-v3.patch |
| JIRA Issue | HBASE-15264 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux e0ff521a9687 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool |
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15150003#comment-15150003
]
Duo Zhang commented on HBASE-15264:
---
{quote}
That Payload looks like RingBufferTruck
Is the RingBuffer in your way? Let all the handlers come together on your queue?
{quote}
Yes, Payload and RingBufferTruck are the same thing. Now I only use an
ArrayDeque and create a new Payload instance every time. We could optimize it
later I think.
> Implement a fan out HDFS OutputStream
> -
>
> Key: HBASE-15264
> URL: https://issues.apache.org/jira/browse/HBASE-15264
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch,
> HBASE-15264.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149994#comment-15149994 ] stack commented on HBASE-15264: --- bq. I will stop renew lease for this file first, and then call recoverFileLease. There is that messyness where if you ask to recover the lease while the NN is recovering the lease, it will start the recovery over from scratch. Do you think that with the stopping of the lease renewal, maybe the recovery will run faster. Four seconds seemed safe time to wait on lease recovery IIRC. bq. ...I think finally namenode will close the file because of lease expired even if the recoverFileLease failed? Seems reasonable [~Apache9] We (or I) can test it. If all unacked edits are into the new file, that should work. I think WAL replay will deal properly where a WAL in the middle of a set of WALs needs its lease recovered. That Payload looks like RingBufferTruck Is the RingBuffer in your way? Let all the handlers come together on your queue? Just skimmed AsyncFSWAL. It looks great. I love how it is so little code. Looking forward to trying it out. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149959#comment-15149959
]
Duo Zhang commented on HBASE-15264:
---
[~stack] You can see this, it is almost done.
https://github.com/Apache9/hbase/blob/HBASE-15265/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java
I still use a queue here, the rpc handler will put WAL entry in the queue and
schedule a task to event loop(Of course I have added some batching logic here
to avoid scheduling one task per WAL entry). The task will call write/flush.
See the {{consumer}} field and {{consume}} methods. We will call flush if we
write all queued entries and there are sync requests, or we find that the
buffered bytes is larger than the configured batchSize(64KB default) when
writing.
On error, we just open a new writer and write the unacked WAL entries to the
new writer. We will schedule a task to close the old broken writer. In
recoverAndClose, I will stop renew lease for this file first, and then call
recoverFileLease. With no one renew its lease, I think finally namenode will
close the file because of lease expired even if the recoverFileLease failed?
Thanks.
> Implement a fan out HDFS OutputStream
> -
>
> Key: HBASE-15264
> URL: https://issues.apache.org/jira/browse/HBASE-15264
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch,
> HBASE-15264.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149866#comment-15149866 ] stack commented on HBASE-15264: --- I like 1 thread instead of six. Stick the above list on head of the class too in the class comments! Nice. > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149863#comment-15149863 ] stack commented on HBASE-15264: --- Sweet. Stick on head of the class too. How we know when to flush? We'll do it size and/or time based? We'll plug this in on what was the append/sync side of the ringbuffer? On error, we need to block and wait on the lease recovery? Or is there where we do as you suggest, we put the WAL aside while recovering lease and start a new one. What you thinking for case where we fail to recover lease? Or, you are thinking we accumulate writes in memory without acking the client while lease recovery is going on (Pardon me if missing some of what has been discussed before... I tried to catch up today but these issues are getting long now -- smile). Great stuff [~Apache9] > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149763#comment-15149763
]
Duo Zhang commented on HBASE-15264:
---
There are several advantages
1. The fan out mechanism. This will reduce the latency.
2. The asynchronous WAL could also run in the same EventLoop, we could just
call write and flush inside the EventLoop thread, so generally we only have one
thread to do all the things. {{FSHLog}} have one disruptor consumer thread and
five sync threads default.
3. Fail-fast when connection to datanode error. The WAL implementation could
open new writer ASAP. The recovery mentioned above is only used to make closing
a little faster, does not intend to recover the writer.
4. We may benefit from the {{ByteBufAllocator}} of netty.
And the stream is designed for implementing WAL, so the idea itself has no
obvious downsides. And the current implementation, too many reflections maybe a
problem...
Thanks.
> Implement a fan out HDFS OutputStream
> -
>
> Key: HBASE-15264
> URL: https://issues.apache.org/jira/browse/HBASE-15264
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch,
> HBASE-15264.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149715#comment-15149715
]
Duo Zhang commented on HBASE-15264:
---
Basically, the stream works as follow
1. {{write}} will just copy the data into a buffer without sending it to
datanode, and we do not have a auto flush mechanism, you always need to flush
manually.
2. {{flush}} will fan out data to datanodes and notify the caller using the
CompletionHander with the file length after this flush.
3. {{close}} will complete the file normally.
4. whenever you got an exception when calling {{flush}} or {{close}}, you
should call {{recoverAndClose}} method to close the file. Now we just call
recoverFileLease. As mentioned in HBASE-14790, we need a little change on the
DTP protocol to support client recovery.
There is an {{EventLoop}} assioated with the output. All connections to
datanodes are managed by this EventLoop. For {{write}} and {{flush}}, we will
switch to the EventLoop thread if we are not in the thread currently, and for
{{close}}, {{recoverAndClose}} and the creation method of this class(the
{{createOutput}} method in helper class), you should not call them inside the
EventLoop thread because they call block for a while and you should never block
the EventLoop thread.
> Implement a fan out HDFS OutputStream
> -
>
> Key: HBASE-15264
> URL: https://issues.apache.org/jira/browse/HBASE-15264
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch,
> HBASE-15264.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149526#comment-15149526 ] stack commented on HBASE-15264: --- The checkstyle complaints are mostly silly. The two failures are flakies. I just committed attempts at fixing them Any chance of a few paragraphs on how this Stream works? Will help w/ review. Talk up the why and why this approach is better too. Any downsides [~Apache9]? I put up some superficial review on rb. Thanks [~Apache9] > Implement a fan out HDFS OutputStream > - > > Key: HBASE-15264 > URL: https://issues.apache.org/jira/browse/HBASE-15264 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-15264-v1.patch, HBASE-15264-v2.patch, > HBASE-15264.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148369#comment-15148369
]
Hadoop QA commented on HBASE-15264:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 20s
{color} | {color:red} Patch generated 67 new checkstyle issues in hbase-server
(total was 0, now 67). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
21m 29s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 29s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 5s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 205m 2s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_95 Failed junit tests |
hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
| | hadoop.hbase.mapreduce.TestImportExport |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-16 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12788016/HBASE-15264-v2.patch |
| JIRA Issue | HBASE-15264 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147383#comment-15147383
]
Hadoop QA commented on HBASE-15264:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 8s
{color} | {color:red} Patch generated 67 new checkstyle issues in hbase-server
(total was 0, now 67). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 55s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 23s
{color} | {color:red} hbase-server introduced 1 new FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 105m 53s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 119m 30s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 273m 49s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Possible null pointer dereference of ecnClass in
org.apache.hadoop.hbase.util.FanOutOneBlockAsyncDFSOutputHelper.createPipelineAckStatusGetter()
Dereferenced at FanOutOneBlockAsyncDFSOutputHelper.java:ecnClass in
org.apache.hadoop.hbase.util.FanOutOneBlockAsyncDFSOutputHelper.createPipelineAckStatusGetter()
Dereferenced at FanOutOneBlockAsyncDFSOutputHelper.java:[line 238] |
| JDK v1.8.0_72 Failed junit tests |
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146507#comment-15146507
]
Hadoop QA commented on HBASE-15264:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 15s
{color} | {color:red} Patch generated 22 new checkstyle issues in hbase-server
(total was 0, now 22). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 50s
{color} | {color:red} Patch causes 65 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 39s
{color} | {color:red} Patch causes 65 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 28s
{color} | {color:red} Patch causes 65 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 19s
{color} | {color:red} Patch causes 65 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 8s
{color} | {color:red} Patch causes 65 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 59s
{color} | {color:red} Patch causes 59 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 49s
{color} | {color:red} Patch causes 59 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 38s
{color} | {color:red} Patch causes 59 errors with Hadoop v2.6.3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 26s
{color} | {color:red} hbase-server-jdk1.8.0_72 with JDK v1.8.0_72 generated 1
new issues (was 1, now 2). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} |
[jira] [Commented] (HBASE-15264) Implement a fan out HDFS OutputStream
[
https://issues.apache.org/jira/browse/HBASE-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146518#comment-15146518
]
Duo Zhang commented on HBASE-15264:
---
Let me try addressing the hadoop version issue...
For checkstyle, "total was 0", is it possible? And yeah I could fix some, but
this one
{noformat}
./hbase-server/src/main/java/org/apache/hadoop/hbase/util/FanOutOneBlockAsyncDFSOutput.java:159:33:
error: Inner assignments should be avoided.
{noformat}
{code:title=FanOutOneBlockAsyncDFSOutput.java}
for (Callback cb; (cb = waitingAckQueue.peekFirst()) != null;) {
{code}
I think this is a standard pattern? How do I fix this...
Thanks.
> Implement a fan out HDFS OutputStream
> -
>
> Key: HBASE-15264
> URL: https://issues.apache.org/jira/browse/HBASE-15264
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-15264.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
