[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[ https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921781#comment-16921781 ] Chen Liang commented on HDFS-13541: --- Thanks [~shv]! I've pushed v003 patch of branch-2, with the white space fixed. > NameNode Port based selective encryption > > > Key: HDFS-13541 > URL: https://issues.apache.org/jira/browse/HDFS-13541 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, security >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Labels: release-blocker > Attachments: HDFS-13541-branch-2.001.patch, > HDFS-13541-branch-2.002.patch, HDFS-13541-branch-2.003.patch, > HDFS-13541-branch-3.1.001.patch, HDFS-13541-branch-3.1.002.patch, > HDFS-13541-branch-3.2.001.patch, HDFS-13541-branch-3.2.002.patch, NameNode > Port based selective encryption-v1.pdf > > > Here at LinkedIn, one issue we face is that we need to enforce different > security requirement based on the location of client and the cluster. > Specifically, for clients from outside of the data center, it is required by > regulation that all traffic must be encrypted. But for clients within the > same data center, unencrypted connections are more desired to avoid the high > encryption overhead. > HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335 > introduced WhitelistBasedResolver which solves the same problem. However we > found it difficult to fit into our environment for several reasons. In this > JIRA, on top of pluggable SASL resolver, *we propose a different approach of > running RPC two ports on NameNode, and the two ports will be enforcing > encrypted and unencrypted connections respectively, and the following > DataNode access will simply follow the same behaviour of > encryption/unencryption*. Then by blocking unencrypted port on datacenter > firewall, we can completely block unencrypted external access. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918169#comment-16918169
]
Konstantin Shvachko commented on HDFS-13541:
{{PBHelperClient}} still has a trivial white-space change.
+1 once this is fixed.
> NameNode Port based selective encryption
>
>
> Key: HDFS-13541
> URL: https://issues.apache.org/jira/browse/HDFS-13541
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, security
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Labels: release-blocker
> Attachments: HDFS-13541-branch-2.001.patch,
> HDFS-13541-branch-2.002.patch, HDFS-13541-branch-2.003.patch,
> HDFS-13541-branch-3.1.001.patch, HDFS-13541-branch-3.1.002.patch,
> HDFS-13541-branch-3.2.001.patch, HDFS-13541-branch-3.2.002.patch, NameNode
> Port based selective encryption-v1.pdf
>
>
> Here at LinkedIn, one issue we face is that we need to enforce different
> security requirement based on the location of client and the cluster.
> Specifically, for clients from outside of the data center, it is required by
> regulation that all traffic must be encrypted. But for clients within the
> same data center, unencrypted connections are more desired to avoid the high
> encryption overhead.
> HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335
> introduced WhitelistBasedResolver which solves the same problem. However we
> found it difficult to fit into our environment for several reasons. In this
> JIRA, on top of pluggable SASL resolver, *we propose a different approach of
> running RPC two ports on NameNode, and the two ports will be enforcing
> encrypted and unencrypted connections respectively, and the following
> DataNode access will simply follow the same behaviour of
> encryption/unencryption*. Then by blocking unencrypted port on datacenter
> firewall, we can completely block unencrypted external access.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917492#comment-16917492
]
Hadoop QA commented on HDFS-13541:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 30m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
17s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
11s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
24s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
13s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
48s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
8s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 2m
22s{color} | {color:red} root in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 2m 22s{color} |
{color:red} root in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 22s{color}
| {color:red} root in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
1s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 16m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
1s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 32s{color} | {color:orange} root: The patch generated 4 new + 1654 unchanged
- 9 fixed = 1658 total (was 1663) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
5s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
55s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
30s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
33s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} |
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[ https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917072#comment-16917072 ] Chen Liang commented on HDFS-13541: --- Thanks for the review [~shv] ! Post v003 patch. > NameNode Port based selective encryption > > > Key: HDFS-13541 > URL: https://issues.apache.org/jira/browse/HDFS-13541 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, security >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Labels: release-blocker > Attachments: HDFS-13541-branch-2.001.patch, > HDFS-13541-branch-2.002.patch, HDFS-13541-branch-2.003.patch, > HDFS-13541-branch-3.1.001.patch, HDFS-13541-branch-3.1.002.patch, > HDFS-13541-branch-3.2.001.patch, HDFS-13541-branch-3.2.002.patch, NameNode > Port based selective encryption-v1.pdf > > > Here at LinkedIn, one issue we face is that we need to enforce different > security requirement based on the location of client and the cluster. > Specifically, for clients from outside of the data center, it is required by > regulation that all traffic must be encrypted. But for clients within the > same data center, unencrypted connections are more desired to avoid the high > encryption overhead. > HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335 > introduced WhitelistBasedResolver which solves the same problem. However we > found it difficult to fit into our environment for several reasons. In this > JIRA, on top of pluggable SASL resolver, *we propose a different approach of > running RPC two ports on NameNode, and the two ports will be enforcing > encrypted and unencrypted connections respectively, and the following > DataNode access will simply follow the same behaviour of > encryption/unencryption*. Then by blocking unencrypted port on datacenter > firewall, we can completely block unencrypted external access. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916298#comment-16916298
]
Hadoop QA commented on HDFS-13541:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
7s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
33s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
37s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
11s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
26s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
41s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
50s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 13m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
11s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
11s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 29s{color} | {color:orange} root: The patch generated 12 new + 1656
unchanged - 9 fixed = 1668 total (was 1665) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
36s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
46s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
38s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
|
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916256#comment-16916256
]
Konstantin Shvachko commented on HDFS-13541:
Comments for branch-2 v02 patch:
# {{SaslDataTransferServer}} has a bunch of new unused imports. Not sure where
do they come from.
# Long lines in {{BlockManager}}
# In {{SaslDataTransferClient.checkTrustAndSend()}}
{code}LOG.info("SASL encryption trust check: localHostTrusted = {}, "{code}
is info level, while on trunk it is debug. Should it be debug for branch-2 as
well?
# Do we need any changes in {{PBHelperClient}}?
> NameNode Port based selective encryption
>
>
> Key: HDFS-13541
> URL: https://issues.apache.org/jira/browse/HDFS-13541
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, security
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Labels: release-blocker
> Attachments: HDFS-13541-branch-2.001.patch,
> HDFS-13541-branch-2.002.patch, HDFS-13541-branch-3.1.001.patch,
> HDFS-13541-branch-3.1.002.patch, HDFS-13541-branch-3.2.001.patch,
> HDFS-13541-branch-3.2.002.patch, NameNode Port based selective
> encryption-v1.pdf
>
>
> Here at LinkedIn, one issue we face is that we need to enforce different
> security requirement based on the location of client and the cluster.
> Specifically, for clients from outside of the data center, it is required by
> regulation that all traffic must be encrypted. But for clients within the
> same data center, unencrypted connections are more desired to avoid the high
> encryption overhead.
> HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335
> introduced WhitelistBasedResolver which solves the same problem. However we
> found it difficult to fit into our environment for several reasons. In this
> JIRA, on top of pluggable SASL resolver, *we propose a different approach of
> running RPC two ports on NameNode, and the two ports will be enforcing
> encrypted and unencrypted connections respectively, and the following
> DataNode access will simply follow the same behaviour of
> encryption/unencryption*. Then by blocking unencrypted port on datacenter
> firewall, we can completely block unencrypted external access.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[ https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916193#comment-16916193 ] Chen Liang commented on HDFS-13541: --- The branch-2 v001 patch has a bad backport causing an incompatible change...post v002 patch. > NameNode Port based selective encryption > > > Key: HDFS-13541 > URL: https://issues.apache.org/jira/browse/HDFS-13541 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, security >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Labels: release-blocker > Attachments: HDFS-13541-branch-2.001.patch, > HDFS-13541-branch-2.002.patch, HDFS-13541-branch-3.1.001.patch, > HDFS-13541-branch-3.1.002.patch, HDFS-13541-branch-3.2.001.patch, > HDFS-13541-branch-3.2.002.patch, NameNode Port based selective > encryption-v1.pdf > > > Here at LinkedIn, one issue we face is that we need to enforce different > security requirement based on the location of client and the cluster. > Specifically, for clients from outside of the data center, it is required by > regulation that all traffic must be encrypted. But for clients within the > same data center, unencrypted connections are more desired to avoid the high > encryption overhead. > HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335 > introduced WhitelistBasedResolver which solves the same problem. However we > found it difficult to fit into our environment for several reasons. In this > JIRA, on top of pluggable SASL resolver, *we propose a different approach of > running RPC two ports on NameNode, and the two ports will be enforcing > encrypted and unencrypted connections respectively, and the following > DataNode access will simply follow the same behaviour of > encryption/unencryption*. Then by blocking unencrypted port on datacenter > firewall, we can completely block unencrypted external access. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916119#comment-16916119
]
Hadoop QA commented on HDFS-13541:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
46s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
55s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
35s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
1s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
46s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
51s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
8s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
42s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 10m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 59s{color} | {color:orange} root: The patch generated 12 new + 1655
unchanged - 9 fixed = 1667 total (was 1664) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
49s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
51s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 31s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
34s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 32s{color}
| {color:red}
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[ https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914529#comment-16914529 ] Chen Liang commented on HDFS-13541: --- Thanks [~shv]! I've committed to branch-3.1. > NameNode Port based selective encryption > > > Key: HDFS-13541 > URL: https://issues.apache.org/jira/browse/HDFS-13541 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, security >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Labels: release-blocker > Attachments: HDFS-13541-branch-3.1.001.patch, > HDFS-13541-branch-3.1.002.patch, HDFS-13541-branch-3.2.001.patch, > HDFS-13541-branch-3.2.002.patch, NameNode Port based selective > encryption-v1.pdf > > > Here at LinkedIn, one issue we face is that we need to enforce different > security requirement based on the location of client and the cluster. > Specifically, for clients from outside of the data center, it is required by > regulation that all traffic must be encrypted. But for clients within the > same data center, unencrypted connections are more desired to avoid the high > encryption overhead. > HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335 > introduced WhitelistBasedResolver which solves the same problem. However we > found it difficult to fit into our environment for several reasons. In this > JIRA, on top of pluggable SASL resolver, *we propose a different approach of > running RPC two ports on NameNode, and the two ports will be enforcing > encrypted and unencrypted connections respectively, and the following > DataNode access will simply follow the same behaviour of > encryption/unencryption*. Then by blocking unencrypted port on datacenter > firewall, we can completely block unencrypted external access. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[ https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913816#comment-16913816 ] Konstantin Shvachko commented on HDFS-13541: +1 on v02 patch for branch 3.1. > NameNode Port based selective encryption > > > Key: HDFS-13541 > URL: https://issues.apache.org/jira/browse/HDFS-13541 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, security >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Labels: release-blocker > Attachments: HDFS-13541-branch-3.1.001.patch, > HDFS-13541-branch-3.1.002.patch, HDFS-13541-branch-3.2.001.patch, > HDFS-13541-branch-3.2.002.patch, NameNode Port based selective > encryption-v1.pdf > > > Here at LinkedIn, one issue we face is that we need to enforce different > security requirement based on the location of client and the cluster. > Specifically, for clients from outside of the data center, it is required by > regulation that all traffic must be encrypted. But for clients within the > same data center, unencrypted connections are more desired to avoid the high > encryption overhead. > HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335 > introduced WhitelistBasedResolver which solves the same problem. However we > found it difficult to fit into our environment for several reasons. In this > JIRA, on top of pluggable SASL resolver, *we propose a different approach of > running RPC two ports on NameNode, and the two ports will be enforcing > encrypted and unencrypted connections respectively, and the following > DataNode access will simply follow the same behaviour of > encryption/unencryption*. Then by blocking unencrypted port on datacenter > firewall, we can completely block unencrypted external access. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912922#comment-16912922
]
Hadoop QA commented on HDFS-13541:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 25m
6s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
38s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
48s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
42s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
53s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
26s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
39s{color} | {color:green} branch-3.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 17m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 42s{color} | {color:orange} root: The patch generated 7 new + 1585 unchanged
- 6 fixed = 1592 total (was 1591) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 29s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
57s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}128m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}288m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | hadoop.hdfs.server.diskbalancer.TestDiskBalancer |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912666#comment-16912666
]
Chen Liang commented on HDFS-13541:
---
Thanks [~shv] for the review. Post v002 patch to address white space and the
ordering of conf keys. For the test {{TestJournalNodeSync}} though, I tried
several times in my local run, it succeeded every time.
> NameNode Port based selective encryption
>
>
> Key: HDFS-13541
> URL: https://issues.apache.org/jira/browse/HDFS-13541
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, security
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Labels: release-blocker
> Attachments: HDFS-13541-branch-3.1.001.patch,
> HDFS-13541-branch-3.1.002.patch, HDFS-13541-branch-3.2.001.patch,
> HDFS-13541-branch-3.2.002.patch, NameNode Port based selective
> encryption-v1.pdf
>
>
> Here at LinkedIn, one issue we face is that we need to enforce different
> security requirement based on the location of client and the cluster.
> Specifically, for clients from outside of the data center, it is required by
> regulation that all traffic must be encrypted. But for clients within the
> same data center, unencrypted connections are more desired to avoid the high
> encryption overhead.
> HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335
> introduced WhitelistBasedResolver which solves the same problem. However we
> found it difficult to fit into our environment for several reasons. In this
> JIRA, on top of pluggable SASL resolver, *we propose a different approach of
> running RPC two ports on NameNode, and the two ports will be enforcing
> encrypted and unencrypted connections respectively, and the following
> DataNode access will simply follow the same behaviour of
> encryption/unencryption*. Then by blocking unencrypted port on datacenter
> firewall, we can completely block unencrypted external access.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911832#comment-16911832
]
Konstantin Shvachko commented on HDFS-13541:
Couple things for 3.1 patch:
# In {{hdfs-default.xml}} new auxiliary port and qop variables should go before
{{dfs.namenode.blockreport.queue.size}}, rather than after, as in trunk.
# Noticed some blank line changes.
# {{TestJournalNodeSync}} failed for me with the patch, but passed without.
Worth checking.
> NameNode Port based selective encryption
>
>
> Key: HDFS-13541
> URL: https://issues.apache.org/jira/browse/HDFS-13541
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, security
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Labels: release-blocker
> Attachments: HDFS-13541-branch-3.1.001.patch,
> HDFS-13541-branch-3.2.001.patch, HDFS-13541-branch-3.2.002.patch, NameNode
> Port based selective encryption-v1.pdf
>
>
> Here at LinkedIn, one issue we face is that we need to enforce different
> security requirement based on the location of client and the cluster.
> Specifically, for clients from outside of the data center, it is required by
> regulation that all traffic must be encrypted. But for clients within the
> same data center, unencrypted connections are more desired to avoid the high
> encryption overhead.
> HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335
> introduced WhitelistBasedResolver which solves the same problem. However we
> found it difficult to fit into our environment for several reasons. In this
> JIRA, on top of pluggable SASL resolver, *we propose a different approach of
> running RPC two ports on NameNode, and the two ports will be enforcing
> encrypted and unencrypted connections respectively, and the following
> DataNode access will simply follow the same behaviour of
> encryption/unencryption*. Then by blocking unencrypted port on datacenter
> firewall, we can completely block unencrypted external access.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909560#comment-16909560
]
Hadoop QA commented on HDFS-13541:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
3s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
32s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
37s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
9s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
51s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
10s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | {color:green} branch-3.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 13m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 22s{color} | {color:orange} root: The patch generated 7 new + 1583 unchanged
- 6 fixed = 1590 total (was 1589) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 30s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
46s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 19s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}212m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.util.TestDiskCheckerWithDiskIo |
| | hadoop.util.TestReadWriteDiskValidator |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | hadoop.hdfs.server.diskbalancer.TestDiskBalancer |
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[ https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909310#comment-16909310 ] Chen Liang commented on HDFS-13541: --- Thanks for the review [~shv]! I've committed to branch-3.2, post v001 patch for branch-3.1 > NameNode Port based selective encryption > > > Key: HDFS-13541 > URL: https://issues.apache.org/jira/browse/HDFS-13541 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, security >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-13541-branch-3.1.001.patch, > HDFS-13541-branch-3.2.001.patch, HDFS-13541-branch-3.2.002.patch, NameNode > Port based selective encryption-v1.pdf > > > Here at LinkedIn, one issue we face is that we need to enforce different > security requirement based on the location of client and the cluster. > Specifically, for clients from outside of the data center, it is required by > regulation that all traffic must be encrypted. But for clients within the > same data center, unencrypted connections are more desired to avoid the high > encryption overhead. > HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335 > introduced WhitelistBasedResolver which solves the same problem. However we > found it difficult to fit into our environment for several reasons. In this > JIRA, on top of pluggable SASL resolver, *we propose a different approach of > running RPC two ports on NameNode, and the two ports will be enforcing > encrypted and unencrypted connections respectively, and the following > DataNode access will simply follow the same behaviour of > encryption/unencryption*. Then by blocking unencrypted port on datacenter > firewall, we can completely block unencrypted external access. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907810#comment-16907810
]
Hadoop QA commented on HDFS-13541:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
32s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
24s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
32s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
8s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
27s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
29s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 13m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 34s{color} | {color:orange} root: The patch generated 7 new + 1569 unchanged
- 4 fixed = 1576 total (was 1573) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
0s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
47s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}112m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 1m
5s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}237m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.diskbalancer.TestDiskBalancer |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:63396beab41 |
| JIRA Issue | HDFS-13541 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[ https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907745#comment-16907745 ] Konstantin Shvachko commented on HDFS-13541: +1. Compared most of the files with trunk, looks consistent. > NameNode Port based selective encryption > > > Key: HDFS-13541 > URL: https://issues.apache.org/jira/browse/HDFS-13541 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, security >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-13541-branch-3.2.001.patch, > HDFS-13541-branch-3.2.002.patch, NameNode Port based selective > encryption-v1.pdf > > > Here at LinkedIn, one issue we face is that we need to enforce different > security requirement based on the location of client and the cluster. > Specifically, for clients from outside of the data center, it is required by > regulation that all traffic must be encrypted. But for clients within the > same data center, unencrypted connections are more desired to avoid the high > encryption overhead. > HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335 > introduced WhitelistBasedResolver which solves the same problem. However we > found it difficult to fit into our environment for several reasons. In this > JIRA, on top of pluggable SASL resolver, *we propose a different approach of > running RPC two ports on NameNode, and the two ports will be enforcing > encrypted and unencrypted connections respectively, and the following > DataNode access will simply follow the same behaviour of > encryption/unencryption*. Then by blocking unencrypted port on datacenter > firewall, we can completely block unencrypted external access. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907696#comment-16907696
]
Chen Liang commented on HDFS-13541:
---
Thanks for the review [~shv], post v002 patch. {{TestDiskBalancer}} and
{{TestDirectoryScanner}} were failing in my local run even without the patch,
the other tests all passed.
> NameNode Port based selective encryption
>
>
> Key: HDFS-13541
> URL: https://issues.apache.org/jira/browse/HDFS-13541
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, security
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Attachments: HDFS-13541-branch-3.2.001.patch,
> HDFS-13541-branch-3.2.002.patch, NameNode Port based selective
> encryption-v1.pdf
>
>
> Here at LinkedIn, one issue we face is that we need to enforce different
> security requirement based on the location of client and the cluster.
> Specifically, for clients from outside of the data center, it is required by
> regulation that all traffic must be encrypted. But for clients within the
> same data center, unencrypted connections are more desired to avoid the high
> encryption overhead.
> HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335
> introduced WhitelistBasedResolver which solves the same problem. However we
> found it difficult to fit into our environment for several reasons. In this
> JIRA, on top of pluggable SASL resolver, *we propose a different approach of
> running RPC two ports on NameNode, and the two ports will be enforcing
> encrypted and unencrypted connections respectively, and the following
> DataNode access will simply follow the same behaviour of
> encryption/unencryption*. Then by blocking unencrypted port on datacenter
> firewall, we can completely block unencrypted external access.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907609#comment-16907609
]
Konstantin Shvachko commented on HDFS-13541:
Looks good overall.
# I suggest in {{hdfs-default.xml}} to place new auxiliary part and qop
variable before {{dfs.namenode.blockreport.queue.size}}, rather than after, as
in trunk.
# {{TestDiskBalancer}} and {{TestDirectoryScanner}} failed locally for me.
Probably common problem, but worth checking if it is related to the change.
> NameNode Port based selective encryption
>
>
> Key: HDFS-13541
> URL: https://issues.apache.org/jira/browse/HDFS-13541
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, security
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Attachments: HDFS-13541-branch-3.2.001.patch, NameNode Port based
> selective encryption-v1.pdf
>
>
> Here at LinkedIn, one issue we face is that we need to enforce different
> security requirement based on the location of client and the cluster.
> Specifically, for clients from outside of the data center, it is required by
> regulation that all traffic must be encrypted. But for clients within the
> same data center, unencrypted connections are more desired to avoid the high
> encryption overhead.
> HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335
> introduced WhitelistBasedResolver which solves the same problem. However we
> found it difficult to fit into our environment for several reasons. In this
> JIRA, on top of pluggable SASL resolver, *we propose a different approach of
> running RPC two ports on NameNode, and the two ports will be enforcing
> encrypted and unencrypted connections respectively, and the following
> DataNode access will simply follow the same behaviour of
> encryption/unencryption*. Then by blocking unencrypted port on datacenter
> firewall, we can completely block unencrypted external access.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897696#comment-16897696
]
Hadoop QA commented on HDFS-13541:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
57s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
40s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
49s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
32s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
28s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 13m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 8s{color} | {color:orange} root: The patch generated 7 new + 1566 unchanged
- 4 fixed = 1573 total (was 1570) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
33s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
49s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 36s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}222m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks |
| | hadoop.hdfs.tools.TestDFSAdmin |
| |
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[ https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897581#comment-16897581 ] Chen Liang commented on HDFS-13541: --- Working on backport to branch-3.x and branch-2, post synthesized patch to trigger jenkins tests. > NameNode Port based selective encryption > > > Key: HDFS-13541 > URL: https://issues.apache.org/jira/browse/HDFS-13541 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, security >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-13541-branch-3.2.001.patch, NameNode Port based > selective encryption-v1.pdf > > > Here at LinkedIn, one issue we face is that we need to enforce different > security requirement based on the location of client and the cluster. > Specifically, for clients from outside of the data center, it is required by > regulation that all traffic must be encrypted. But for clients within the > same data center, unencrypted connections are more desired to avoid the high > encryption overhead. > HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335 > introduced WhitelistBasedResolver which solves the same problem. However we > found it difficult to fit into our environment for several reasons. In this > JIRA, on top of pluggable SASL resolver, *we propose a different approach of > running RPC two ports on NameNode, and the two ports will be enforcing > encrypted and unencrypted connections respectively, and the following > DataNode access will simply follow the same behaviour of > encryption/unencryption*. Then by blocking unencrypted port on datacenter > firewall, we can completely block unencrypted external access. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491307#comment-16491307
]
Konstantin Shvachko commented on HDFS-13541:
{quote}I prefer the approach where datanode also listens on two ports, as it
makes the entire approach easy to understand
{quote}
I agree with [~vagarychen] that adding extra port to DNs adds a lot of
complexity to the entire workflow. DNs will need to report to NN both ports
during registration. Then NN will need to selectively return secure or non
secure port to the client depending on the client's connection. It also changes
and complicates configuration of the DNs. I advocate keeping single port on
DNs. Also I think it is logically simpler: NN enforces secure or non secure
communication on the client and it uses it consistently for both NN RPCs and DN
data transfers.
{quote}Encrypting the entire data pipeline is not necessary. I believe, it
should be optimized{quote}
Not sure if it is an optimization. If DN1 receives and encrypted packet it can
send it down to DN2 without any transformation encrypted as it is. As opposed
to decrypting the packet on DN1 and sending it un-encrypted to DN2. So just
keep the pipeline ALL encrypted or ALL not encrypted.
> NameNode Port based selective encryption
>
>
> Key: HDFS-13541
> URL: https://issues.apache.org/jira/browse/HDFS-13541
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, security
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Attachments: NameNode Port based selective encryption-v1.pdf
>
>
> Here at LinkedIn, one issue we face is that we need to enforce different
> security requirement based on the location of client and the cluster.
> Specifically, for clients from outside of the data center, it is required by
> regulation that all traffic must be encrypted. But for clients within the
> same data center, unencrypted connections are more desired to avoid the high
> encryption overhead.
> HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335
> introduced WhitelistBasedResolver which solves the same problem. However we
> found it difficult to fit into our environment for several reasons. In this
> JIRA, on top of pluggable SASL resolver, *we propose a different approach of
> running RPC two ports on NameNode, and the two ports will be enforcing
> encrypted and unencrypted connections respectively, and the following
> DataNode access will simply follow the same behaviour of
> encryption/unencryption*. Then by blocking unencrypted port on datacenter
> firewall, we can completely block unencrypted external access.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491147#comment-16491147
]
Benoy Antony commented on HDFS-13541:
-
{quote}
in our environment, cross data center traffic actually compose a small
fraction of all traffic, having additional DataXceiverServer thread sitting and
listening on every single data node, but being idle most of time does not seem
to be ideal.
{quote}
If its a small fraction, then one option would be to use (1) swebhdfs or (2)
HTTPFS gateway which uses RPC. They work seamlessly for clients via the
FileSystem implementation and swebhdfs scheme.
(1) introduces HTTP protocol overhead and lack of QOS on namenode.
(2) introduces HTTP protocol overhead and an additional hop.
But these are the common solutions to this problem for small fractions of data
which needs to be treated differently. But your use case may be different
which warrants secure and non secure RPC and data transfer.
> NameNode Port based selective encryption
>
>
> Key: HDFS-13541
> URL: https://issues.apache.org/jira/browse/HDFS-13541
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, security
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Attachments: NameNode Port based selective encryption-v1.pdf
>
>
> Here at LinkedIn, one issue we face is that we need to enforce different
> security requirement based on the location of client and the cluster.
> Specifically, for clients from outside of the data center, it is required by
> regulation that all traffic must be encrypted. But for clients within the
> same data center, unencrypted connections are more desired to avoid the high
> encryption overhead.
> HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335
> introduced WhitelistBasedResolver which solves the same problem. However we
> found it difficult to fit into our environment for several reasons. In this
> JIRA, on top of pluggable SASL resolver, *we propose a different approach of
> running RPC two ports on NameNode, and the two ports will be enforcing
> encrypted and unencrypted connections respectively, and the following
> DataNode access will simply follow the same behaviour of
> encryption/unencryption*. Then by blocking unencrypted port on datacenter
> firewall, we can completely block unencrypted external access.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[ https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471159#comment-16471159 ] Chen Liang commented on HDFS-13541: --- Thought a bit more about passing additional fields of the connection to SASL resolver. Passing the Server#connection object will unlikely to work because it is specific to ipc.Server, but SASL resolver is more general, e.g. DN side does not have the ipc.Server instance but still do SASL server side resolution. I will explore alternative ways. > NameNode Port based selective encryption > > > Key: HDFS-13541 > URL: https://issues.apache.org/jira/browse/HDFS-13541 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, security >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: NameNode Port based selective encryption-v1.pdf > > > Here at LinkedIn, one issue we face is that we need to enforce different > security requirement based on the location of client and the cluster. > Specifically, for clients from outside of the data center, it is required by > regulation that all traffic must be encrypted. But for clients within the > same data center, unencrypted connections are more desired to avoid the high > encryption overhead. > HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335 > introduced WhitelistBasedResolver which solves the same problem. However we > found it difficult to fit into our environment for several reasons. In this > JIRA, on top of pluggable SASL resolver, *we propose a different approach of > running RPC two ports on NameNode, and the two ports will be enforcing > encrypted and unencrypted connections respectively, and the following > DataNode access will simply follow the same behaviour of > encryption/unencryption*. Then by blocking unencrypted port on datacenter > firewall, we can completely block unencrypted external access. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[ https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470781#comment-16470781 ] Chen Liang commented on HDFS-13541: --- Thanks a lot for taking a look [~benoyantony]! bq. We could also think of passing additional parameters based on the connection This is a good point. The first thought I had is to pass the entire Server#Connection instance to the resolver, (it also includes the IP address and the ingress port). Other than these, other potentially relevant fields might include user, remote, hostAddress (not sure if hostAddress is set correctly at all though). What do you think? bq. If so, what prevents the external client from replaying the encrypted message from a different connection between an internal client and datanode ? As of now, the main thing that prevents replay attach is the fact that the key expires after 10 min by default. After the key expires, NN and DN will be using different keys and the encrypted message is invalidated. Namely, the attacher has a maximum of 10 min window to reply the encrypted message. We consider this sufficient as for now. If I understood this correctly, I think it is the same rationale behind block access token. i.e. without talking to NN, someone may connect to DN directly replaying block access token, but only possible in that 10 min window. We considered adding more identification info addition to the QOP string, such as client IP, or some timestamp based info. This adds more variable to the message itself. But that also adds more encryption overhead (because the message is larger). Also, adding IP address might be relatively straightforward, other info such as timestamp may be very tricky to manage here. Currently we are inclined not to go this with optimization. Comments on this? bq. Another side effect of derived QOP for data transfer protection is that one cannot enable RPC protection alone with this approach. This is true as in my current POC. Because in our environment NN and DN always do the same protection. But we can add configuration's to allow only enforce RPC protection. We just need to be able to configure DN to ignore the derived QOP. bq. As mentioned in the document, Encrypting the entire data pipeline is not necessary. I believe, it should be optimized Sure, will work on that. bq. I prefer the approach where datanode also listens on two ports, as it makes the entire approach easy to understand On the implementation complexity, it means we will need to change NN-DN communication such that DN informs NN about the new port it has. The DN maintenance code logic seems a bit convoluted now; on the practical side, in our environment, cross data center traffic actually compose a small fraction of all traffic, having additional DataXceiverServer thread sitting and listening on every single datanode, but being idle most of time does not seem to be ideal. [~shv] may have more comments on this, he is on vacation and until next week. In the mean time, I will re-evaluate my other POC patch on this alternative approach. > NameNode Port based selective encryption > > > Key: HDFS-13541 > URL: https://issues.apache.org/jira/browse/HDFS-13541 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, security >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: NameNode Port based selective encryption-v1.pdf > > > Here at LinkedIn, one issue we face is that we need to enforce different > security requirement based on the location of client and the cluster. > Specifically, for clients from outside of the data center, it is required by > regulation that all traffic must be encrypted. But for clients within the > same data center, unencrypted connections are more desired to avoid the high > encryption overhead. > HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335 > introduced WhitelistBasedResolver which solves the same problem. However we > found it difficult to fit into our environment for several reasons. In this > JIRA, on top of pluggable SASL resolver, *we propose a different approach of > running RPC two ports on NameNode, and the two ports will be enforcing > encrypted and unencrypted connections respectively, and the following > DataNode access will simply follow the same behaviour of > encryption/unencryption*. Then by blocking unencrypted port on datacenter > firewall, we can completely block unencrypted external access. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail:
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[ https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470684#comment-16470684 ] Benoy Antony commented on HDFS-13541: - Thanks [~vagarychen]. I have reviewed the document. Overall, I believe, this approach makes the administration of selective encryption simpler. The selection can be done using firewall rules. I have a few comments based on my understanding of the design. # Currently the input to _SaslPropertiesResolver_ methods is ip address. This feature adds ingress port as an input. I believe, it should be made generic so that we do not have to change it in future. We could also think of passing additional parameters based on the connection even though they may not be used in the current implementations of _SaslPropertiesResolver._ # _The_ _selective data transfer protection_ design needs more scrutiny as it's protection is derived from _selective RPC protection_. Based on what I understood, the protection is dictated by the value of _encrypted message_ sent by the client when it handshakes with datanode just before the data transfer. The _encrypted message is either_ _auth_ or _auth_conf_ encrypted by the secret shared between namenode and the datanode. If so, what prevents the external client from replaying the _encrypted message_ from a different connection between an internal client and datanode ? # Another side effect of derived QOP for data transfer protection is that one cannot enable RPC protection alone with this approach. # As mentioned in the document, Encrypting the entire data pipeline is not necessary. I believe, it should be optimized. # All things equal, I prefer the approach where datanode also listens on two ports, as it makes the entire approach easy to understand. It will also solve issues specified in #2 and #3 above. Those issues are the result of QOP of data transfer operation becoming a derivative of RPC operation. > NameNode Port based selective encryption > > > Key: HDFS-13541 > URL: https://issues.apache.org/jira/browse/HDFS-13541 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, security >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: NameNode Port based selective encryption-v1.pdf > > > Here at LinkedIn, one issue we face is that we need to enforce different > security requirement based on the location of client and the cluster. > Specifically, for clients from outside of the data center, it is required by > regulation that all traffic must be encrypted. But for clients within the > same data center, unencrypted connections are more desired to avoid the high > encryption overhead. > HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335 > introduced WhitelistBasedResolver which solves the same problem. However we > found it difficult to fit into our environment for several reasons. In this > JIRA, on top of pluggable SASL resolver, *we propose a different approach of > running RPC two ports on NameNode, and the two ports will be enforcing > encrypted and unencrypted connections respectively, and the following > DataNode access will simply follow the same behaviour of > encryption/unencryption*. Then by blocking unencrypted port on datacenter > firewall, we can completely block unencrypted external access. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[ https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469155#comment-16469155 ] Chen Liang commented on HDFS-13541: --- [~benoyantony], would you be interested in taking a look? cc. [~shv], [~zhz] and [~xkrogen]. > NameNode Port based selective encryption > > > Key: HDFS-13541 > URL: https://issues.apache.org/jira/browse/HDFS-13541 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, security >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: NameNode Port based selective encryption-v1.pdf > > > Here at LinkedIn, one issue we face is that we need to enforce different > security requirement based on the location of client and the cluster. > Specifically, for clients from outside of the data center, it is required by > regulation that all traffic must be encrypted. But for clients within the > same data center, unencrypted connections are more desired to avoid the high > encryption overhead. > HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335 > introduced WhitelistBasedResolver which solves the same problem. However we > found it difficult to fit into our environment for several reasons. In this > JIRA, on top of pluggable SASL resolver, *we propose a different approach of > running RPC two ports on NameNode, and the two ports will be enforcing > encrypted and unencrypted connections respectively, and the following > DataNode access will simply follow the same behaviour of > encryption/unencryption*. Then by blocking unencrypted port on datacenter > firewall, we can completely block unencrypted external access. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
