[jira] [Commented] (HADOOP-17705) S3A to add option fs.s3a.endpoint.region to set AWS region
[
https://issues.apache.org/jira/browse/HADOOP-17705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17622232#comment-17622232
]
Greg Senia commented on HADOOP-17705:
-
[~ste...@apache.org] exactly it fixes it completely!! Not sure why they (AWS)
didn't fix this but makes me think they don't have very good test cases to test
V4 Sig and VPC endpoints as he added one to the test ..
As Cooper Walbrun put it in his PR:
I believe that requests from this SDK to S3 via VPC endpoints are currently
receiving 400 (Bad Request) responses. When I inspect the request headers being
sent in by the SDK, I noticed a difference in the Authorization header:
Authorization: AWS4-HMAC-SHA256
Credential=ACCESS_KEY_ID/20210325/vpce/s3/aws4_request # VPC endpoint
Authorization: AWS4-HMAC-SHA256
Credential=ACCESS_KEY_ID/20210325/us-east-2/s3/aws4_request # normal
The Credential value was being assembled using what the SDK misunderstood as
the region in VPC endpoints, which have the structure
vpce-x-.s3.[region].vpce.amazonaws.com in the case of
S3.
Upon further inspection of the code, it seems the issue originates from the
AwsHostNameUtils class, as you see in the diff of this pull request. I am
however new to this codebase so if I am incorrectly addressing this issue, let
me know. For what it is worth, I did confirm that my test in
AWS4SignerTest.java fails in the absence of my change in AwsHostNameUtils.java
> S3A to add option fs.s3a.endpoint.region to set AWS region
> --
>
> Key: HADOOP-17705
> URL: https://issues.apache.org/jira/browse/HADOOP-17705
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently, AWS region is either constructed via the endpoint URL, by making
> an assumption that the 2nd component after delimiter "." is the region in
> endpoint URL, which doesn't work for private links and sets the default to
> us-east-1 thus causing authorization issue w.r.t the private link.
> The option fs.s3a.endpoint.region allows this to be explicitly set
> h2. how to set the s3 region on older hadoop releases
> For anyone who needs to set the signing region on older versions of the s3a
> client *you do not need this festure*. instead just provide a custom endpoint
> to region mapping json file
> # Download the default region mapping file
> [awssdk_config_default.json|https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/internal/config/awssdk_config_default.json]
> # Add a new regular expression to map the endpoint/hostname to the target
> region
> # Save the file as {{/etc/hadoop/conf/awssdk_config_override.json}}
> # verify basic hadop fs -ls commands work
> # copy to the rest of the cluster.
> # There should be no need to restart any services
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17705) S3A to add option fs.s3a.endpoint.region to set AWS region
[
https://issues.apache.org/jira/browse/HADOOP-17705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17621413#comment-17621413
]
Greg Senia commented on HADOOP-17705:
-
Also fixes it on Hadoop 2.x versions if the SDK is updated to 1.11.45 with the
aws-sdk patched... We have a case open with AWS to force them to address this.
https://github.com/aws/aws-sdk-java/pull/2537
> S3A to add option fs.s3a.endpoint.region to set AWS region
> --
>
> Key: HADOOP-17705
> URL: https://issues.apache.org/jira/browse/HADOOP-17705
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently, AWS region is either constructed via the endpoint URL, by making
> an assumption that the 2nd component after delimiter "." is the region in
> endpoint URL, which doesn't work for private links and sets the default to
> us-east-1 thus causing authorization issue w.r.t the private link.
> The option fs.s3a.endpoint.region allows this to be explicitly set
> h2. how to set the s3 region on older hadoop releases
> For anyone who needs to set the signing region on older versions of the s3a
> client *you do not need this festure*. instead just provide a custom endpoint
> to region mapping json file
> # Download the default region mapping file
> [awssdk_config_default.json|https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/internal/config/awssdk_config_default.json]
> # Add a new regular expression to map the endpoint/hostname to the target
> region
> # Save the file as {{/etc/hadoop/conf/awssdk_config_override.json}}
> # verify basic hadop fs -ls commands work
> # copy to the rest of the cluster.
> # There should be no need to restart any services
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17705) S3A to add option fs.s3a.endpoint.region to set AWS region
[
https://issues.apache.org/jira/browse/HADOOP-17705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17621385#comment-17621385
]
Greg Senia commented on HADOOP-17705:
-
[~ste...@apache.org] apparently this was the real fix to the issue I actually
tested it with Hadoop 3.2.4 and it fixes the problem without any code changes.
I have raised this with our AWS Account Team as a vendor product we are using
hit the same issue that was not utilizing Hadoop-aws code.
https://github.com/aws/aws-sdk-java/pull/2537
> S3A to add option fs.s3a.endpoint.region to set AWS region
> --
>
> Key: HADOOP-17705
> URL: https://issues.apache.org/jira/browse/HADOOP-17705
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently, AWS region is either constructed via the endpoint URL, by making
> an assumption that the 2nd component after delimiter "." is the region in
> endpoint URL, which doesn't work for private links and sets the default to
> us-east-1 thus causing authorization issue w.r.t the private link.
> The option fs.s3a.endpoint.region allows this to be explicitly set
> h2. how to set the s3 region on older hadoop releases
> For anyone who needs to set the signing region on older versions of the s3a
> client *you do not need this festure*. instead just provide a custom endpoint
> to region mapping json file
> # Download the default region mapping file
> [awssdk_config_default.json|https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/internal/config/awssdk_config_default.json]
> # Add a new regular expression to map the endpoint/hostname to the target
> region
> # Save the file as {{/etc/hadoop/conf/awssdk_config_override.json}}
> # verify basic hadop fs -ls commands work
> # copy to the rest of the cluster.
> # There should be no need to restart any services
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17705) S3A to add option fs.s3a.endpoint.region to set AWS region
[
https://issues.apache.org/jira/browse/HADOOP-17705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17618963#comment-17618963
]
Greg Senia commented on HADOOP-17705:
-
[~ste...@apache.org] yes the node is an onprem centos-8 stream node with no aws
tooling installed... So far all the known tests seem to pass. I've been using
this fix a while on the 3.2 branch and even backported it to our 2.x branch
too. PR coming shortly.
> S3A to add option fs.s3a.endpoint.region to set AWS region
> --
>
> Key: HADOOP-17705
> URL: https://issues.apache.org/jira/browse/HADOOP-17705
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently, AWS region is either constructed via the endpoint URL, by making
> an assumption that the 2nd component after delimiter "." is the region in
> endpoint URL, which doesn't work for private links and sets the default to
> us-east-1 thus causing authorization issue w.r.t the private link.
> The option fs.s3a.endpoint.region allows this to be explicitly set
> h2. how to set the s3 region on older hadoop releases
> For anyone who needs to set the signing region on older versions of the s3a
> client *you do not need this festure*. instead just provide a custom endpoint
> to region mapping json file
> # Download the default region mapping file
> [awssdk_config_default.json|https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/internal/config/awssdk_config_default.json]
> # Add a new regular expression to map the endpoint/hostname to the target
> region
> # Save the file as {{/etc/hadoop/conf/awssdk_config_override.json}}
> # verify basic hadop fs -ls commands work
> # copy to the rest of the cluster.
> # There should be no need to restart any services
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17705) S3A to add option fs.s3a.endpoint.region to set AWS region
[
https://issues.apache.org/jira/browse/HADOOP-17705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17617276#comment-17617276
]
Greg Senia commented on HADOOP-17705:
-
[~ste...@apache.org]is it possible to accept a backport I put together for
Hadoop 3.2. I have a few folks who cannot upgrade to Hadoop 3.3.x branch and
need the ability to access S3 using V4 signing via VPC endpoints without having
to mess with overrides and regex's as we have multiple endpoints in different
regions etc.
> S3A to add option fs.s3a.endpoint.region to set AWS region
> --
>
> Key: HADOOP-17705
> URL: https://issues.apache.org/jira/browse/HADOOP-17705
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently, AWS region is either constructed via the endpoint URL, by making
> an assumption that the 2nd component after delimiter "." is the region in
> endpoint URL, which doesn't work for private links and sets the default to
> us-east-1 thus causing authorization issue w.r.t the private link.
> The option fs.s3a.endpoint.region allows this to be explicitly set
> h2. how to set the s3 region on older hadoop releases
> For anyone who needs to set the signing region on older versions of the s3a
> client *you do not need this festure*. instead just provide a custom endpoint
> to region mapping json file
> # Download the default region mapping file
> [awssdk_config_default.json|https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/internal/config/awssdk_config_default.json]
> # Add a new regular expression to map the endpoint/hostname to the target
> region
> # Save the file as {{/etc/hadoop/conf/awssdk_config_override.json}}
> # verify basic hadop fs -ls commands work
> # copy to the rest of the cluster.
> # There should be no need to restart any services
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-16350:
Attachment: (was: HADOOP-16350.patch)
> Ability to tell Hadoop not to request KMS Information from Remote NN
> -
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 2.8.3, 3.0.0, 2.7.6, 3.1.2
>Reporter: Greg Senia
>Priority: Major
> Fix For: 3.3.0
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it did before HADOOP-14104. I can see
> the value in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue
> should of at least had an option specified to allow Hadoop/KMS code to
> operate similar to how it did before by not requesting remote KMSServer URIs
> which would than attempt to get a delegation token even if not operating on
> encrypted zones.
> Error when KMS Server traffic is not allowed between cluster networks per
> enterprise security standard which cannot be changed they denied the request
> for exception so the only solution is to allow a feature to not attempt to
> request tokens.
> {code:java}
> $ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
> -Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
> hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
> hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
> 19/05/29 14:06:09 INFO tools.DistCp: Input Options: DistCpOptions
> {atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, overwrite=false, append=false, useDiff=false,
> fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
> numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
> sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
> preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
> sourceFileListing=null,
> sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
>
> targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
> targetPathExists=true, filtersFile='null', verboseLog=false}
> 19/05/29 14:06:09 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 5093920 for gss2002 on ha-hdfs:unit
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
> token 5093920 for gss2002)
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> kms-dt, Service: ha21d53en.unit.hdp.example.com:9292, Ident: (owner=gss2002,
> renewer=yarn, realUser=, issueDate=1559153170120, maxDate=1559757970120,
> sequenceNumber=237, masterKeyId=2)
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1;
> dirCnt = 0
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Build file listing completed.
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 556079 for gss2002 on ha-hdfs:tech
> 19/05/29 14:06:10 ERROR tools.DistCp: Exception encountered
> java.io.IOException: java.net.NoRouteToHostException: No route to host (Host
> unreachable)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1029)
> at
>
[jira] [Updated] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-16350:
Status: Open (was: Patch Available)
> Ability to tell Hadoop not to request KMS Information from Remote NN
> -
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 3.1.2, 2.7.6, 3.0.0, 2.8.3
>Reporter: Greg Senia
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16350.patch
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it did before HADOOP-14104. I can see
> the value in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue
> should of at least had an option specified to allow Hadoop/KMS code to
> operate similar to how it did before by not requesting remote KMSServer URIs
> which would than attempt to get a delegation token even if not operating on
> encrypted zones.
> Error when KMS Server traffic is not allowed between cluster networks per
> enterprise security standard which cannot be changed they denied the request
> for exception so the only solution is to allow a feature to not attempt to
> request tokens.
> {code:java}
> $ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
> -Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
> hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
> hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
> 19/05/29 14:06:09 INFO tools.DistCp: Input Options: DistCpOptions
> {atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, overwrite=false, append=false, useDiff=false,
> fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
> numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
> sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
> preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
> sourceFileListing=null,
> sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
>
> targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
> targetPathExists=true, filtersFile='null', verboseLog=false}
> 19/05/29 14:06:09 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 5093920 for gss2002 on ha-hdfs:unit
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
> token 5093920 for gss2002)
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> kms-dt, Service: ha21d53en.unit.hdp.example.com:9292, Ident: (owner=gss2002,
> renewer=yarn, realUser=, issueDate=1559153170120, maxDate=1559757970120,
> sequenceNumber=237, masterKeyId=2)
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1;
> dirCnt = 0
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Build file listing completed.
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 556079 for gss2002 on ha-hdfs:tech
> 19/05/29 14:06:10 ERROR tools.DistCp: Exception encountered
> java.io.IOException: java.net.NoRouteToHostException: No route to host (Host
> unreachable)
> at
>
[jira] [Assigned] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia reassigned HADOOP-16350:
---
Assignee: (was: Greg Senia)
> Ability to tell Hadoop not to request KMS Information from Remote NN
> -
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 2.8.3, 3.0.0, 2.7.6, 3.1.2
>Reporter: Greg Senia
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16350.patch
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it did before HADOOP-14104. I can see
> the value in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue
> should of at least had an option specified to allow Hadoop/KMS code to
> operate similar to how it did before by not requesting remote KMSServer URIs
> which would than attempt to get a delegation token even if not operating on
> encrypted zones.
> Error when KMS Server traffic is not allowed between cluster networks per
> enterprise security standard which cannot be changed they denied the request
> for exception so the only solution is to allow a feature to not attempt to
> request tokens.
> {code:java}
> $ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
> -Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
> hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
> hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
> 19/05/29 14:06:09 INFO tools.DistCp: Input Options: DistCpOptions
> {atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, overwrite=false, append=false, useDiff=false,
> fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
> numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
> sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
> preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
> sourceFileListing=null,
> sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
>
> targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
> targetPathExists=true, filtersFile='null', verboseLog=false}
> 19/05/29 14:06:09 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 5093920 for gss2002 on ha-hdfs:unit
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
> token 5093920 for gss2002)
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> kms-dt, Service: ha21d53en.unit.hdp.example.com:9292, Ident: (owner=gss2002,
> renewer=yarn, realUser=, issueDate=1559153170120, maxDate=1559757970120,
> sequenceNumber=237, masterKeyId=2)
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1;
> dirCnt = 0
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Build file listing completed.
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 556079 for gss2002 on ha-hdfs:tech
> 19/05/29 14:06:10 ERROR tools.DistCp: Exception encountered
> java.io.IOException: java.net.NoRouteToHostException: No route to host (Host
> unreachable)
> at
>
[jira] [Commented] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864250#comment-16864250
]
Greg Senia commented on HADOOP-16350:
-
[~ajayydv] unfortunately its not working that way. It goes off and gets the
remote NN's keyprovider info it doesnt matter if its null or not. As the value
being passed into HdfsKmsUtil is which is from this value:
getServerDefaults().getKeyProviderUri(). We built a custom version and add some
debug statements to prove what is happening. The local cluster has zero
information about the Remote Clusters KMS so it's obtaining the information
from the getServerDefaults().getKeyProvider(). There is no known way to make it
not call it as if it doesn't find it its going to go and try to get it. I
proved this last night I attempted to set the value to empty on the distcp
call. It still goes to the remote cluster and we block traffic to that KMS. So
again the only option is a new custom property
*keyProviderUriStr = getServerDefaults().getKeyProviderUri()*
public static URI getKeyProviderUri(UserGroupInformation ugi,
URI namenodeUri, *String keyProviderUriStr*, Configuration conf)
> Ability to tell Hadoop not to request KMS Information from Remote NN
> -
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 2.8.3, 3.0.0, 2.7.6, 3.1.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16350.patch
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it did before HADOOP-14104. I can see
> the value in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue
> should of at least had an option specified to allow Hadoop/KMS code to
> operate similar to how it did before by not requesting remote KMSServer URIs
> which would than attempt to get a delegation token even if not operating on
> encrypted zones.
> Error when KMS Server traffic is not allowed between cluster networks per
> enterprise security standard which cannot be changed they denied the request
> for exception so the only solution is to allow a feature to not attempt to
> request tokens.
> {code:java}
> $ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
> -Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
> hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
> hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
> 19/05/29 14:06:09 INFO tools.DistCp: Input Options: DistCpOptions
> {atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, overwrite=false, append=false, useDiff=false,
> fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
> numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
> sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
> preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
> sourceFileListing=null,
> sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
>
> targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
> targetPathExists=true, filtersFile='null', verboseLog=false}
> 19/05/29 14:06:09 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 5093920 for gss2002 on ha-hdfs:unit
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
> token 5093920 for gss2002)
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt
[jira] [Commented] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863558#comment-16863558
]
Greg Senia commented on HADOOP-16350:
-
And I found why I made the change in the 3.x code line is because in
DFSClient.java the serverdefaults are passed in with no way to override them
hence the need for the custom property. By going and fetching serverDefaults
you automatically have a value from the NN.
{code:java}
/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
@Override
public URI getKeyProviderUri() throws IOException {
return HdfsKMSUtil.getKeyProviderUri(ugi, namenodeUri,
getServerDefaults().getKeyProviderUri(), conf);
}
{code}
And even if you attempt to override it in
/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
isHdfsEncryptionEnabled it will call the above method and return a URI
{code:java}
/**
* Probe for encryption enabled on this filesystem.
* @return true if encryption is enabled
*/
boolean isHDFSEncryptionEnabled() throws IOException {
return getKeyProviderUri() != null;
}{code}
Hence the following code changes below in:
{code:java}
hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsKMSUtil.java
public static URI getKeyProviderUri(UserGroupInformation ugi,
URI namenodeUri, String keyProviderUriStr, Configuration conf)
throws IOException {
URI keyProviderUri = null;
// Lookup the secret in credentials object for namenodeuri.
Credentials credentials = ugi.getCredentials();
Text credsKey = getKeyProviderMapKey(namenodeUri);
byte[] keyProviderUriBytes =
credentials.getSecretKey(credsKey);
if(keyProviderUriBytes != null) {
keyProviderUri =
URI.create(DFSUtilClient.bytes2String(keyProviderUriBytes));
}
if (keyProviderUri == null) {
// NN is old and doesn't report provider, so use conf.
if (keyProviderUriStr == null) {
keyProviderUri = KMSUtil.getKeyProviderUri(conf, keyProviderUriKeyName);
} else if (!keyProviderUriStr.isEmpty()) {
// Check if KMS Traffic to Remote KMS Server is allowed. Default is
allowed
Boolean isRemoteKMSAllowed =
conf.getBoolean(CommonConfigurationKeysPublic.KMS_CLIENT_ALLOW_REMOTE_KMS,
CommonConfigurationKeysPublic.KMS_CLIENT_ALLOW_REMOTE_KMS_DEFAULT);
if (isRemoteKMSAllowed) {
keyProviderUri = URI.create(keyProviderUriStr);
}
}
if (keyProviderUri != null) {
credentials.addSecretKey(
credsKey, DFSUtilClient.string2Bytes(keyProviderUri.toString()));
}
}
return keyProviderUri;
}
{code}
> Ability to tell Hadoop not to request KMS Information from Remote NN
> -
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 2.8.3, 3.0.0, 2.7.6, 3.1.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16350.patch
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it did before HADOOP-14104. I can see
> the value in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue
> should of at least had an option specified to allow Hadoop/KMS code to
> operate similar to how it did before by not requesting remote KMSServer URIs
> which would than attempt to get a delegation token even if not operating on
> encrypted zones.
> Error when KMS Server traffic is not allowed between cluster networks per
> enterprise security standard which cannot be changed they
[jira] [Comment Edited] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863543#comment-16863543
]
Greg Senia edited comment on HADOOP-16350 at 6/13/19 11:51 PM:
---
[~szetszwo] the recommended suggestion won't work with the 2.x line of code.
The custom property is required.. I think that is where the confusion is coming
in here. In Hadoop 2.x code is much different than 3.x I will review 3.x code
again
2.x code:
{code:java}
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
@@ -3670,12 +3670,17 @@
}
// Query the namenode for the key provider uri.
+ Boolean isRemoteKMSAllowed =
+
conf.getBoolean(CommonConfigurationKeysPublic.KMS_CLIENT_ALLOW_REMOTE_KMS,
+
CommonConfigurationKeysPublic.KMS_CLIENT_ALLOW_REMOTE_KMS_DEFAULT);
+ if (isRemoteKMSAllowed) {
FsServerDefaults serverDefaults = getServerDefaults();
- if (serverDefaults.getKeyProviderUri() != null) {
- if (!serverDefaults.getKeyProviderUri().isEmpty()) {
- keyProviderUri = URI.create(serverDefaults.getKeyProviderUri());
+ if (serverDefaults.getKeyProviderUri() != null) {
+ if (!serverDefaults.getKeyProviderUri().isEmpty()) {
+ keyProviderUri = URI.create(serverDefaults.getKeyProviderUri());
+ }
+ return keyProviderUri;
}
- return keyProviderUri;
}
// Last thing is to trust its own conf to be backwards compatible.
{code}
Failure:
{code:java}
[gss2002@ha21t51en ~]$ hadoop distcp -Dhadoop.security.key.provider.path=""
-Ddfs.namenode.kerberos.principal.pattern=*
-Dmapreduce.job.hdfs-servers.token-renewal.exclude=unit
hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
hdfs://unit/processed/public/opendata/samples/distcp_test/distcp_file2.txt
19/06/13 19:22:58 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, overwrite=false, append=false, useDiff=false,
fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
sourceFileListing=null,
sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
targetPath=hdfs://unit/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
targetPathExists=true, filtersFile='null', verboseLog=false} 19/06/13 19:22:59
INFO client.AHSProxy: Connecting to Application History server at
ha21t53mn.tech.hdp.example.com/10.70.33.2:10200 19/06/13 19:22:59 INFO
hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 561611 for gss2002 on
ha-hdfs:tech 19/06/13 19:22:59 INFO security.TokenCache: Got dt for
hdfs://tech; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:tech, Ident:
(HDFS_DELEGATION_TOKEN token 561611 for gss2002) 19/06/13 19:22:59 INFO
security.TokenCache: Got dt for hdfs://tech; Kind: kms-dt, Service:
ha21t53en.tech.hdp.example.com:9292, Ident: (owner=gss2002, renewer=yarn,
realUser=, issueDate=1560468179680, maxDate=1561072979680, sequenceNumber=7787,
masterKeyId=92) 19/06/13 19:23:00 INFO tools.SimpleCopyListing: Paths
(files+dirs) cnt = 1; dirCnt = 0 19/06/13 19:23:00 INFO
tools.SimpleCopyListing: Build file listing completed. 19/06/13 19:23:00 INFO
tools.DistCp: Number of paths in the copy list: 1 19/06/13 19:23:01 INFO
tools.DistCp: Number of paths in the copy list: 1 19/06/13 19:23:01 INFO
client.AHSProxy: Connecting to Application History server at
ha21t53mn.tech.hdp.example.com/10.70.33.2:10200 19/06/13 19:23:01 INFO
hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 5144031 for gss2002 on
ha-hdfs:unit 19/06/13 19:23:01 ERROR tools.DistCp: Exception encountered
java.io.IOException: java.net.NoRouteToHostException: No route to host (Host
unreachable) at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1029)
at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
at
org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2407)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
at
org.apache.hadoop.tools.mapred.CopyOutputFormat.checkOutputSpecs(CopyOutputFormat.java:124)
at
[jira] [Commented] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863543#comment-16863543
]
Greg Senia commented on HADOOP-16350:
-
[~szetszwo] the recommended suggestion won't work. The custom property is
required.. I know you dont want to add it for whatever reason but a feature
that was being used in a commercial product is no longer working so we really
need this custom property added to allow HADOOP-14104 to be reverted.. See
below setting your recommendation fails!!
{code:java}
[gss2002@ha21t51en ~]$ hadoop distcp -Dhadoop.security.key.provider.path=""
-Ddfs.namenode.kerberos.principal.pattern=*
-Dmapreduce.job.hdfs-servers.token-renewal.exclude=unit
hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
hdfs://unit/processed/public/opendata/samples/distcp_test/distcp_file2.txt
19/06/13 19:22:58 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, overwrite=false, append=false, useDiff=false,
fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
sourceFileListing=null,
sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
targetPath=hdfs://unit/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
targetPathExists=true, filtersFile='null', verboseLog=false} 19/06/13 19:22:59
INFO client.AHSProxy: Connecting to Application History server at
ha21t53mn.tech.hdp.example.com/10.70.33.2:10200 19/06/13 19:22:59 INFO
hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 561611 for gss2002 on
ha-hdfs:tech 19/06/13 19:22:59 INFO security.TokenCache: Got dt for
hdfs://tech; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:tech, Ident:
(HDFS_DELEGATION_TOKEN token 561611 for gss2002) 19/06/13 19:22:59 INFO
security.TokenCache: Got dt for hdfs://tech; Kind: kms-dt, Service:
ha21t53en.tech.hdp.example.com:9292, Ident: (owner=gss2002, renewer=yarn,
realUser=, issueDate=1560468179680, maxDate=1561072979680, sequenceNumber=7787,
masterKeyId=92) 19/06/13 19:23:00 INFO tools.SimpleCopyListing: Paths
(files+dirs) cnt = 1; dirCnt = 0 19/06/13 19:23:00 INFO
tools.SimpleCopyListing: Build file listing completed. 19/06/13 19:23:00 INFO
tools.DistCp: Number of paths in the copy list: 1 19/06/13 19:23:01 INFO
tools.DistCp: Number of paths in the copy list: 1 19/06/13 19:23:01 INFO
client.AHSProxy: Connecting to Application History server at
ha21t53mn.tech.hdp.example.com/10.70.33.2:10200 19/06/13 19:23:01 INFO
hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 5144031 for gss2002 on
ha-hdfs:unit 19/06/13 19:23:01 ERROR tools.DistCp: Exception encountered
java.io.IOException: java.net.NoRouteToHostException: No route to host (Host
unreachable) at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1029)
at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
at
org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2407)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
at
org.apache.hadoop.tools.mapred.CopyOutputFormat.checkOutputSpecs(CopyOutputFormat.java:124)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at
org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) at
org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:193) at
org.apache.hadoop.tools.DistCp.execute(DistCp.java:155) at
org.apache.hadoop.tools.DistCp.run(DistCp.java:128) at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at
org.apache.hadoop.tools.DistCp.main(DistCp.java:462) Caused by:
java.net.NoRouteToHostException: No route to host (Host unreachable) at
java.net.PlainSocketImpl.socketConnect(Native Method) at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
[jira] [Commented] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863065#comment-16863065
]
Greg Senia commented on HADOOP-16350:
-
Additional Information showing the working distcp:
*Working with custom property set to false and it can be seen below that only a
delegation token is requested for the local KMS Server which is how it operated
before HADOOP-14104*
[gss2002@ha21t51en ~]$ hadoop distcp
*-Dhadoop.security.kms.client.allow.remote.kms=false*
-Ddfs.namenode.kerberos.principal.pattern=*
-Dmapreduce.job.hdfs-servers.token-renewal.exclude=unit
hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
hdfs://unit/processed/public/opendata/samples/distcp_test/distcp_file2.txt
19/06/13 09:15:30 INFO tools.DistCp: Input Options:
DistCpOptions\{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, overwrite=false, append=false, useDiff=false,
fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
sourceFileListing=null,
sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
targetPath=hdfs://unit/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
targetPathExists=true, filtersFile='null', verboseLog=false}
19/06/13 09:15:30 INFO client.AHSProxy: Connecting to Application History
server at ha21t53mn.tech.hdp.example.com/10.70.33.2:10200
19/06/13 09:15:31 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
561537 for gss2002 on ha-hdfs:tech
19/06/13 09:15:31 INFO security.TokenCache: Got dt for hdfs://tech; Kind:
HDFS_DELEGATION_TOKEN, Service: ha-hdfs:tech, Ident: (HDFS_DELEGATION_TOKEN
token 561537 for gss2002)###
19/06/13 09:15:31 INFO security.TokenCache: Got dt for hdfs://tech; Kind:
kms-dt, Service: ha21t53en.tech.hdp.example.com:9292, Ident: (owner=gss2002,
renewer=yarn, realUser=, issueDate=1560431731476, maxDate=1561036531476,
sequenceNumber=7729, masterKeyId=91)
19/06/13 09:15:32 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1;
dirCnt = 0
19/06/13 09:15:32 INFO tools.SimpleCopyListing: Build file listing completed.
19/06/13 09:15:32 INFO tools.DistCp: Number of paths in the copy list: 1
19/06/13 09:15:32 INFO tools.DistCp: Number of paths in the copy list: 1
19/06/13 09:15:32 INFO client.AHSProxy: Connecting to Application History
server at ha21t53mn.tech.hdp.example.com/10.70.33.2:10200
19/06/13 09:15:32 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
5141150 for gss2002 on ha-hdfs:unit
19/06/13 09:15:32 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
token 5141150 for gss2002)
19/06/13 09:15:32 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
19/06/13 09:15:33 INFO mapreduce.JobSubmitter: number of splits:1
19/06/13 09:15:33 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1560183404146_0051
19/06/13 09:15:33 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN,
Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN token 5141150 for gss2002)
19/06/13 09:15:33 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN,
Service: ha-hdfs:tech, Ident: (HDFS_DELEGATION_TOKEN token 561537 for gss2002)
*19/06/13 09:15:33 INFO mapreduce.JobSubmitter: Kind: kms-dt, Service:
ha21t53en.tech.hdp.example.com:9292, Ident: (owner=gss2002, renewer=yarn,
realUser=, issueDate=1560431731476, maxDate=1561036531476, sequenceNumber=7729,
masterKeyId=91)*
19/06/13 09:15:33 INFO impl.TimelineClientImpl: Timeline service address:
http://ha21t53mn.tech.hdp.example.com:8188/ws/v1/timeline/
19/06/13 09:15:34 INFO impl.YarnClientImpl: Submitted application
application_1560183404146_0051
19/06/13 09:15:34 INFO mapreduce.Job: The url to track the job:
http://ha21t53mn.tech.hdp.example.com:8088/proxy/application_1560183404146_0051/
19/06/13 09:15:34 INFO tools.DistCp: DistCp job-id:
job_1560183404146_0051emote.kms=false
-Ddfs.namenode.kerberos.principal.pattern=* -Dmapreduce.job.hdfs-serv19/06/13
09:15:34 INFO mapreduce.Job: Running job:
job_1560183404146_0051distcp_test/distcp_file.txt
hdfs://unit/processed/public/opendata/samples/dis
19/06/13 09:15:46 INFO mapreduce.Job: Job job_1560183404146_0051 running in
uber mode : false
19/06/13 09:15:46 INFO mapreduce.Job: map 0% reduce 0%
19/06/13 09:15:55 INFO mapreduce.Job: map 100% reduce 0%
19/06/13 09:15:55 INFO mapreduce.Job: Job job_1560183404146_0051 completed
successfully
19/06/13 09:15:55 INFO mapreduce.Job: Counters: 33
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=177893
FILE: Number of read operations=0
FILE: Number of
[jira] [Comment Edited] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863061#comment-16863061
]
Greg Senia edited comment on HADOOP-16350 at 6/13/19 1:15 PM:
--
[~szetszwo] and [~jojochuang] here is what I didn't send along.. If you look at
the code you will see that this allows it to operate as it did before by adding
this param to distcp job it will operate as it did before the patch as that
code path in the if block is what goes to remote server and in our case the
remote server is not even exposed to remote networks.
*hadoop.security.kms.client.allow.remote.kms=false*
hadoop distcp -*Dhadoop.security.kms.client.allow.remote.kms=false*
-Ddfs.namenode.kerberos.principal.pattern=*
-Dmapreduce.job.hdfs-servers.token-renewal.exclude=unit
hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
hdfs://unit/processed/public/opendata/samples/distcp_test/distcp_file2.txt
Here is the code that allows it to work by adding the if block the default is
true so that nothing changes and if you set to false via the prop the remote
call to the remote cluster does not occur
{code:java}
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
index 0fa78c2..77222ee 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
@@ -837,6 +837,16 @@
*
* core-default.xml
*/
+ public static final String KMS_CLIENT_ALLOW_REMOTE_KMS =
+ "hadoop.security.kms.client.allow.remote.kms";
+ /** Default value for KMS_CLIENT_ALLOW_REMOTE_KMS (true)*/
+ public static final boolean KMS_CLIENT_ALLOW_REMOTE_KMS_DEFAULT = true;
+
+ /**
+ * @see
+ *
+ * core-default.xml
+ */
public static final String HADOOP_SECURITY_JAVA_SECURE_RANDOM_ALGORITHM_KEY
=
"hadoop.security.java.secure.random.algorithm";
/** Defalt value for HADOOP_SECURITY_JAVA_SECURE_RANDOM_ALGORITHM_KEY */
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsKMSUtil.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsKMSUtil.java
index 30e8aa7..c6daae0 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsKMSUtil.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsKMSUtil.java
@@ -145,7 +145,13 @@
if (keyProviderUriStr == null) {
keyProviderUri = KMSUtil.getKeyProviderUri(conf,
keyProviderUriKeyName);
} else if (!keyProviderUriStr.isEmpty()) {
-keyProviderUri = URI.create(keyProviderUriStr);
+// Check if KMS Traffic to Remote KMS Server is allowed. Default is
allowed
+ Boolean isRemoteKMSAllowed =
+
conf.getBoolean(CommonConfigurationKeysPublic.KMS_CLIENT_ALLOW_REMOTE_KMS,
+
CommonConfigurationKeysPublic.KMS_CLIENT_ALLOW_REMOTE_KMS_DEFAULT);
+if (isRemoteKMSAllowed) {
+ keyProviderUri = URI.create(keyProviderUriStr);
+}
}
if (keyProviderUri != null) {
credentials.addSecretKey(
{code}
was (Author: gss2002):
[~szetszwo] and [~jojochuang] here is what I didn't send along.. If you look at
the code you will see that this allows it to operate as it did before by adding
this param to distcp job it will operate as it did before the patch as that
code path in the if block is what goes to remote server and in our case the
remote server is not even exposed to remote networks.
*hadoop.security.kms.client.allow.remote.kms=false*
hadoop distcp -*Dhadoop.security.kms.client.allow.remote.kms=false*
-Ddfs.namenode.kerberos.principal.pattern=*
-Dmapreduce.job.hdfs-servers.token-renewal.exclude=unit
hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
hdfs://unit/processed/public/opendata/samples/distcp_test/distcp_file2.txt
Here is the code that allows it to work by adding the if block the default is
true so that nothing changes and if you set to false via the:
{code:java}
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
index 0fa78c2..77222ee 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
@@ -837,6 +837,16 @@
*
* core-default.xml
*/
+ public static final String
[jira] [Commented] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863061#comment-16863061
]
Greg Senia commented on HADOOP-16350:
-
[~szetszwo] and [~jojochuang] here is what I didn't send along.. If you look at
the code you will see that this allows it to operate as it did before by adding
this param to distcp job it will operate as it did before the patch as that
code path in the if block is what goes to remote server and in our case the
remote server is not even exposed to remote networks.
*hadoop.security.kms.client.allow.remote.kms=false*
hadoop distcp -*Dhadoop.security.kms.client.allow.remote.kms=false*
-Ddfs.namenode.kerberos.principal.pattern=*
-Dmapreduce.job.hdfs-servers.token-renewal.exclude=unit
hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
hdfs://unit/processed/public/opendata/samples/distcp_test/distcp_file2.txt
Here is the code that allows it to work by adding the if block the default is
true so that nothing changes and if you set to false via the:
{code:java}
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
index 0fa78c2..77222ee 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
@@ -837,6 +837,16 @@
*
* core-default.xml
*/
+ public static final String KMS_CLIENT_ALLOW_REMOTE_KMS =
+ "hadoop.security.kms.client.allow.remote.kms";
+ /** Default value for KMS_CLIENT_ALLOW_REMOTE_KMS (true)*/
+ public static final boolean KMS_CLIENT_ALLOW_REMOTE_KMS_DEFAULT = true;
+
+ /**
+ * @see
+ *
+ * core-default.xml
+ */
public static final String HADOOP_SECURITY_JAVA_SECURE_RANDOM_ALGORITHM_KEY
=
"hadoop.security.java.secure.random.algorithm";
/** Defalt value for HADOOP_SECURITY_JAVA_SECURE_RANDOM_ALGORITHM_KEY */
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsKMSUtil.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsKMSUtil.java
index 30e8aa7..c6daae0 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsKMSUtil.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsKMSUtil.java
@@ -145,7 +145,13 @@
if (keyProviderUriStr == null) {
keyProviderUri = KMSUtil.getKeyProviderUri(conf,
keyProviderUriKeyName);
} else if (!keyProviderUriStr.isEmpty()) {
-keyProviderUri = URI.create(keyProviderUriStr);
+// Check if KMS Traffic to Remote KMS Server is allowed. Default is
allowed
+ Boolean isRemoteKMSAllowed =
+
conf.getBoolean(CommonConfigurationKeysPublic.KMS_CLIENT_ALLOW_REMOTE_KMS,
+
CommonConfigurationKeysPublic.KMS_CLIENT_ALLOW_REMOTE_KMS_DEFAULT);
+if (isRemoteKMSAllowed) {
+ keyProviderUri = URI.create(keyProviderUriStr);
+}
}
if (keyProviderUri != null) {
credentials.addSecretKey(
{code}
> Ability to tell Hadoop not to request KMS Information from Remote NN
> -
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 2.8.3, 3.0.0, 2.7.6, 3.1.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16350.patch
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it
[jira] [Commented] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861684#comment-16861684
]
Greg Senia commented on HADOOP-16350:
-
[~jojochuang] right now we are protected against that as we are required by our
security team to block KMS traffic from leaving our cluster networks. Aka the
local cluster initiating the distcp has no access to the Remote KMS Server so
no way to get that delegation token even if we tried as the service is blocked.
My original thought was to try to determine if a folder was even using
TDE/Encryption and not even attempt to get a delegation token from either the
local KMS or the remote KMS in that case which seems to be more ideal longer
term. This patch for now does solve our problem to allow us to move forward
with an upgrade but it would be better to solve this long term totally agree.
> Ability to tell Hadoop not to request KMS Information from Remote NN
> -
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 2.8.3, 3.0.0, 2.7.6, 3.1.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16350.patch
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it did before HADOOP-14104. I can see
> the value in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue
> should of at least had an option specified to allow Hadoop/KMS code to
> operate similar to how it did before by not requesting remote KMSServer URIs
> which would than attempt to get a delegation token even if not operating on
> encrypted zones.
> Error when KMS Server traffic is not allowed between cluster networks per
> enterprise security standard which cannot be changed they denied the request
> for exception so the only solution is to allow a feature to not attempt to
> request tokens.
> {code:java}
> $ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
> -Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
> hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
> hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
> 19/05/29 14:06:09 INFO tools.DistCp: Input Options: DistCpOptions
> {atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, overwrite=false, append=false, useDiff=false,
> fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
> numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
> sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
> preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
> sourceFileListing=null,
> sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
>
> targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
> targetPathExists=true, filtersFile='null', verboseLog=false}
> 19/05/29 14:06:09 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 5093920 for gss2002 on ha-hdfs:unit
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
> token 5093920 for gss2002)
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> kms-dt, Service: ha21d53en.unit.hdp.example.com:9292, Ident: (owner=gss2002,
> renewer=yarn, realUser=, issueDate=1559153170120, maxDate=1559757970120,
> sequenceNumber=237, masterKeyId=2)
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1;
> dirCnt = 0
>
[jira] [Comment Edited] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861661#comment-16861661
]
Greg Senia edited comment on HADOOP-16350 at 6/12/19 1:46 AM:
--
[~szetszwo], No problem it solves the interim problem for us. The real problem
is we use KMSClientProvider locally in the cluster so what we are trying to
solve for is to allow for the code to operate is it did before HADOOP-14104. In
our case we want to just disable call that is made we have verified that the
patch I provided when set to false reverts the code to operate is it did before
HADOOP-14104. Please provide more details on how you plan to go about making it
operate as it did before HADOOP-14104 as I am willing to modify the patch but
we need the same functionality that existed before HADOOP-14104. Not trying to
be a pain but our customers and security team basically relied on this feature
to prevent folders that contained TDE/Encrypted data from being moved from
cluster to cluster.
was (Author: gss2002):
[~szetszwo] the problem is we use KMSClientProvider locally in the cluster what
we are trying to solve for is to allow for the code to operate is it did before
HADOOP-14104. In our case we want to just disable call that is made we have
verified that the patch I provided when set to false reverts the code to
operate is it did before HADOOP-14104. Please provide more details on how you
plan to go about making it operate as it did before HADOOP-14104 as I am
willing to modify the patch but we need the same functionality that existed
before HADOOP-14104.
> Ability to tell Hadoop not to request KMS Information from Remote NN
> -
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 2.8.3, 3.0.0, 2.7.6, 3.1.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16350.patch
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it did before HADOOP-14104. I can see
> the value in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue
> should of at least had an option specified to allow Hadoop/KMS code to
> operate similar to how it did before by not requesting remote KMSServer URIs
> which would than attempt to get a delegation token even if not operating on
> encrypted zones.
> Error when KMS Server traffic is not allowed between cluster networks per
> enterprise security standard which cannot be changed they denied the request
> for exception so the only solution is to allow a feature to not attempt to
> request tokens.
> {code:java}
> $ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
> -Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
> hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
> hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
> 19/05/29 14:06:09 INFO tools.DistCp: Input Options: DistCpOptions
> {atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, overwrite=false, append=false, useDiff=false,
> fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
> numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
> sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
> preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
> sourceFileListing=null,
> sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
>
> targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
> targetPathExists=true, filtersFile='null', verboseLog=false}
> 19/05/29 14:06:09 INFO client.AHSProxy: Connecting
[jira] [Commented] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861661#comment-16861661
]
Greg Senia commented on HADOOP-16350:
-
[~szetszwo] the problem is we use KMSClientProvider locally in the cluster what
we are trying to solve for is to allow for the code to operate is it did before
HADOOP-14104. In our case we want to just disable call that is made we have
verified that the patch I provided when set to false reverts the code to
operate is it did before HADOOP-14104. Please provide more details on how you
plan to go about making it operate as it did before HADOOP-14104 as I am
willing to modify the patch but we need the same functionality that existed
before HADOOP-14104.
> Ability to tell Hadoop not to request KMS Information from Remote NN
> -
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 2.8.3, 3.0.0, 2.7.6, 3.1.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16350.patch
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it did before HADOOP-14104. I can see
> the value in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue
> should of at least had an option specified to allow Hadoop/KMS code to
> operate similar to how it did before by not requesting remote KMSServer URIs
> which would than attempt to get a delegation token even if not operating on
> encrypted zones.
> Error when KMS Server traffic is not allowed between cluster networks per
> enterprise security standard which cannot be changed they denied the request
> for exception so the only solution is to allow a feature to not attempt to
> request tokens.
> {code:java}
> $ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
> -Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
> hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
> hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
> 19/05/29 14:06:09 INFO tools.DistCp: Input Options: DistCpOptions
> {atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, overwrite=false, append=false, useDiff=false,
> fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
> numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
> sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
> preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
> sourceFileListing=null,
> sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
>
> targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
> targetPathExists=true, filtersFile='null', verboseLog=false}
> 19/05/29 14:06:09 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 5093920 for gss2002 on ha-hdfs:unit
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
> token 5093920 for gss2002)
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> kms-dt, Service: ha21d53en.unit.hdp.example.com:9292, Ident: (owner=gss2002,
> renewer=yarn, realUser=, issueDate=1559153170120, maxDate=1559757970120,
> sequenceNumber=237, masterKeyId=2)
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1;
> dirCnt = 0
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Build file listing completed.
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the
[jira] [Updated] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-16350:
Fix Version/s: 3.3.0
Status: Patch Available (was: Open)
> Ability to tell Hadoop not to request KMS Information from Remote NN
> -
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 3.1.2, 2.7.6, 3.0.0, 2.8.3
>Reporter: Greg Senia
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16350.patch
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it did before HADOOP-14104. I can see
> the value in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue
> should of at least had an option specified to allow Hadoop/KMS code to
> operate similar to how it did before by not requesting remote KMSServer URIs
> which would than attempt to get a delegation token even if not operating on
> encrypted zones.
> Error when KMS Server traffic is not allowed between cluster networks per
> enterprise security standard which cannot be changed they denied the request
> for exception so the only solution is to allow a feature to not attempt to
> request tokens.
> {code:java}
> $ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
> -Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
> hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
> hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
> 19/05/29 14:06:09 INFO tools.DistCp: Input Options: DistCpOptions
> {atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, overwrite=false, append=false, useDiff=false,
> fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
> numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
> sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
> preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
> sourceFileListing=null,
> sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
>
> targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
> targetPathExists=true, filtersFile='null', verboseLog=false}
> 19/05/29 14:06:09 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 5093920 for gss2002 on ha-hdfs:unit
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
> token 5093920 for gss2002)
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> kms-dt, Service: ha21d53en.unit.hdp.example.com:9292, Ident: (owner=gss2002,
> renewer=yarn, realUser=, issueDate=1559153170120, maxDate=1559757970120,
> sequenceNumber=237, masterKeyId=2)
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1;
> dirCnt = 0
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Build file listing completed.
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 556079 for gss2002 on ha-hdfs:tech
> 19/05/29 14:06:10 ERROR tools.DistCp: Exception encountered
> java.io.IOException: java.net.NoRouteToHostException: No route to host (Host
> unreachable)
> at
>
[jira] [Updated] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-16350:
Attachment: HADOOP-16350.patch
> Ability to tell Hadoop not to request KMS Information from Remote NN
> -
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 2.8.3, 3.0.0, 2.7.6, 3.1.2
>Reporter: Greg Senia
>Priority: Major
> Attachments: HADOOP-16350.patch
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it did before HADOOP-14104. I can see
> the value in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue
> should of at least had an option specified to allow Hadoop/KMS code to
> operate similar to how it did before by not requesting remote KMSServer URIs
> which would than attempt to get a delegation token even if not operating on
> encrypted zones.
> Error when KMS Server traffic is not allowed between cluster networks per
> enterprise security standard which cannot be changed they denied the request
> for exception so the only solution is to allow a feature to not attempt to
> request tokens.
> {code:java}
> $ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
> -Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
> hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
> hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
> 19/05/29 14:06:09 INFO tools.DistCp: Input Options: DistCpOptions
> {atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, overwrite=false, append=false, useDiff=false,
> fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
> numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
> sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
> preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
> sourceFileListing=null,
> sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
>
> targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
> targetPathExists=true, filtersFile='null', verboseLog=false}
> 19/05/29 14:06:09 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 5093920 for gss2002 on ha-hdfs:unit
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
> token 5093920 for gss2002)
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> kms-dt, Service: ha21d53en.unit.hdp.example.com:9292, Ident: (owner=gss2002,
> renewer=yarn, realUser=, issueDate=1559153170120, maxDate=1559757970120,
> sequenceNumber=237, masterKeyId=2)
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1;
> dirCnt = 0
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Build file listing completed.
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 556079 for gss2002 on ha-hdfs:tech
> 19/05/29 14:06:10 ERROR tools.DistCp: Exception encountered
> java.io.IOException: java.net.NoRouteToHostException: No route to host (Host
> unreachable)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1029)
> at
>
[jira] [Created] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
Greg Senia created HADOOP-16350:
---
Summary: Ability to tell Hadoop not to request KMS Information
from Remote NN
Key: HADOOP-16350
URL: https://issues.apache.org/jira/browse/HADOOP-16350
Project: Hadoop Common
Issue Type: Improvement
Components: common, kms
Affects Versions: 3.1.2, 2.7.6, 3.0.0, 2.8.3
Reporter: Greg Senia
Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
NameNode and their associated remote KMSServer delegation token. Many customers
were using this as a security feature to prevent TDE/Encryption Zone data from
being distcped to remote clusters. But there was still a use case to allow
distcp of data residing in folders that are not being encrypted with a
KMSProvider/Encrypted Zone.
So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
now fails as we along with other customers (HDFS-13696) DO NOT allow KMSServer
endpoints to be exposed out of our cluster network as data residing in these
TDE/Zones contain very critical data that cannot be distcped between clusters.
I propose adding a new code block with the following custom property
"hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
keeping current feature of HADOOP-14104 but if specified to "false" will allow
this area of code to operate as it did before HADOOP-14104. I can see the value
in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue should of at
least had an option specified to allow Hadoop/KMS code to operate similar to
how it did before by not requesting remote KMSServer URIs which would than
attempt to get a delegation token even if not operating on encrypted zones.
Error when KMS Server traffic is not allowed between cluster networks per
enterprise security standard which cannot be changed they denied the request
for exception so the only solution is to allow a feature to not attempt to
request tokens.
{code:java}
$ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
-Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
19/05/29 14:06:09 INFO tools.DistCp: Input Options: DistCpOptions
{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, overwrite=false, append=false, useDiff=false,
fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
sourceFileListing=null,
sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
targetPathExists=true, filtersFile='null', verboseLog=false}
19/05/29 14:06:09 INFO client.AHSProxy: Connecting to Application History
server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
5093920 for gss2002 on ha-hdfs:unit
19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
token 5093920 for gss2002)
19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
kms-dt, Service: ha21d53en.unit.hdp.example.com:9292, Ident: (owner=gss2002,
renewer=yarn, realUser=, issueDate=1559153170120, maxDate=1559757970120,
sequenceNumber=237, masterKeyId=2)
19/05/29 14:06:10 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1;
dirCnt = 0
19/05/29 14:06:10 INFO tools.SimpleCopyListing: Build file listing completed.
19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
19/05/29 14:06:10 INFO client.AHSProxy: Connecting to Application History
server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
556079 for gss2002 on ha-hdfs:tech
19/05/29 14:06:10 ERROR tools.DistCp: Exception encountered
java.io.IOException: java.net.NoRouteToHostException: No route to host (Host
unreachable)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1029)
at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
at
org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2407)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
at
[jira] [Commented] (HADOOP-14104) Client should always ask namenode for kms provider path.
[
https://issues.apache.org/jira/browse/HADOOP-14104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851188#comment-16851188
]
Greg Senia commented on HADOOP-14104:
-
Stack trace showing exception:
$ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
-Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
19/05/29 14:06:09 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, overwrite=false, append=false, useDiff=false,
fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
sourceFileListing=null,
sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
targetPathExists=true, filtersFile='null', verboseLog=false}
19/05/29 14:06:09 INFO client.AHSProxy: Connecting to Application History
server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
5093920 for gss2002 on ha-hdfs:unit
19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
token 5093920 for gss2002)
19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
kms-dt, Service: ha21d53en.unit.hdp.example.com:9292, Ident: (owner=gss2002,
renewer=yarn, realUser=, issueDate=1559153170120, maxDate=1559757970120,
sequenceNumber=237, masterKeyId=2)
19/05/29 14:06:10 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1;
dirCnt = 0
19/05/29 14:06:10 INFO tools.SimpleCopyListing: Build file listing completed.
19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
19/05/29 14:06:10 INFO client.AHSProxy: Connecting to Application History
server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
556079 for gss2002 on ha-hdfs:tech
19/05/29 14:06:10 ERROR tools.DistCp: Exception encountered
java.io.IOException: java.net.NoRouteToHostException: No route to host (Host
unreachable)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1029)
at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
at
org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2407)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
at
org.apache.hadoop.tools.mapred.CopyOutputFormat.checkOutputSpecs(CopyOutputFormat.java:124)
at
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:193)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:155)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:128)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:462)
Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
[jira] [Comment Edited] (HADOOP-14104) Client should always ask namenode for kms provider path.
[
https://issues.apache.org/jira/browse/HADOOP-14104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851162#comment-16851162
]
Greg Senia edited comment on HADOOP-14104 at 5/29/19 6:34 PM:
--
[~daryn] or [~shahrs87] or [~xiaochen] I would be curious if there was a reason
this was removed we used distcp and it no longer works as it wants to get to
our Remote Clusters KMS Server which is isolated even if not attempting to
distcp an encrypted zone so by removing "if (dfs.isHDFSEncryptionEnabled())" we
now have situation where 1) folks are able to move TDE/Encrypted data between a
local and remote clusters which is undesirable. And now we have to figure out
how to open our KMSServer in our remote Cluster for moving data that does not
reside in an Encrypted Zone.
Code change
public Token[] addDelegationTokens(
final String renewer, Credentials credentials) throws IOException {
Token[] tokens = super.addDelegationTokens(renewer, credentials);
*if (dfs.isHDFSEncryptionEnabled()) {
* KeyProviderDelegationTokenExtension keyProviderDelegationTokenExtension =
KeyProviderDelegationTokenExtension.
createKeyProviderDelegationTokenExtension(dfs.getKeyProvider());
Token[] kpTokens = keyProviderDelegationTokenExtension.
addDelegationTokens(renewer, credentials);
if (tokens != null && kpTokens != null) {
Token[] all = new Token[tokens.length + kpTokens.length];
System.arraycopy(tokens, 0, all, 0, tokens.length);
System.arraycopy(kpTokens, 0, all, tokens.length, kpTokens.length);
tokens = all;
} else {
tokens = (tokens != null) ? tokens : kpTokens;
}
}
return tokens;
}
vs
@Override
public Token[] addDelegationTokens(
final String renewer, Credentials credentials) throws IOException {
Token[] tokens = super.addDelegationTokens(renewer, credentials);
URI keyProviderUri = dfs.getKeyProviderUri();
if (keyProviderUri != null) {
KeyProviderDelegationTokenExtension keyProviderDelegationTokenExtension =
KeyProviderDelegationTokenExtension.
createKeyProviderDelegationTokenExtension(dfs.getKeyProvider());
Token[] kpTokens = keyProviderDelegationTokenExtension.
addDelegationTokens(renewer, credentials);
credentials.addSecretKey(dfs.getKeyProviderMapKey(),
DFSUtil.string2Bytes(keyProviderUri.toString()));
if (tokens != null && kpTokens != null) {
Token[] all = new Token[tokens.length + kpTokens.length];
System.arraycopy(tokens, 0, all, 0, tokens.length);
System.arraycopy(kpTokens, 0, all, tokens.length, kpTokens.length);
tokens = all;
} else {
tokens = (tokens != null) ? tokens : kpTokens;
}
}
return tokens;
}
was (Author: gss2002):
[~daryn] or [~shahrs87] out of curiosity we are in a situation due to the
removal of "if (dfs.isHDFSEncryptionEnabled())" we now have situation where
folks are able to move TDE'd data between a local and remote cluster which was
undesirable. So previously we were preventing TDE'd data from being moved
between clusters. Now we have to open the remote KMSServers ports which were
previously blocked from the remote cluster. I guess my question is can we add
parameter to prevent distcp or HDFS from looking at remote clusters.
public Token[] addDelegationTokens(
final String renewer, Credentials credentials) throws IOException {
Token[] tokens = super.addDelegationTokens(renewer, credentials);
if (dfs.isHDFSEncryptionEnabled()) {
KeyProviderDelegationTokenExtension keyProviderDelegationTokenExtension =
KeyProviderDelegationTokenExtension.
createKeyProviderDelegationTokenExtension(dfs.getKeyProvider());
Token[] kpTokens = keyProviderDelegationTokenExtension.
addDelegationTokens(renewer, credentials);
if (tokens != null && kpTokens != null) {
Token[] all = new Token[tokens.length + kpTokens.length];
System.arraycopy(tokens, 0, all, 0, tokens.length);
System.arraycopy(kpTokens, 0, all, tokens.length, kpTokens.length);
tokens = all;
} else {
tokens = (tokens != null) ? tokens : kpTokens;
}
}
return tokens;
}
vs
@Override
public Token[] addDelegationTokens(
final String renewer, Credentials credentials) throws IOException {
Token[] tokens = super.addDelegationTokens(renewer, credentials);
URI keyProviderUri = dfs.getKeyProviderUri();
if (keyProviderUri != null) {
KeyProviderDelegationTokenExtension keyProviderDelegationTokenExtension =
KeyProviderDelegationTokenExtension.
createKeyProviderDelegationTokenExtension(dfs.getKeyProvider());
Token[] kpTokens = keyProviderDelegationTokenExtension.
addDelegationTokens(renewer,
[jira] [Commented] (HADOOP-14104) Client should always ask namenode for kms provider path.
[
https://issues.apache.org/jira/browse/HADOOP-14104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851162#comment-16851162
]
Greg Senia commented on HADOOP-14104:
-
[~daryn] or [~shahrs87] out of curiosity we are in a situation due to the
removal of "if (dfs.isHDFSEncryptionEnabled())" we now have situation where
folks are able to move TDE'd data between a local and remote cluster which was
undesirable. So previously we were preventing TDE'd data from being moved
between clusters. Now we have to open the remote KMSServers ports which were
previously blocked from the remote cluster. I guess my question is can we add
parameter to prevent distcp or HDFS from looking at remote clusters.
public Token[] addDelegationTokens(
final String renewer, Credentials credentials) throws IOException {
Token[] tokens = super.addDelegationTokens(renewer, credentials);
if (dfs.isHDFSEncryptionEnabled()) {
KeyProviderDelegationTokenExtension keyProviderDelegationTokenExtension =
KeyProviderDelegationTokenExtension.
createKeyProviderDelegationTokenExtension(dfs.getKeyProvider());
Token[] kpTokens = keyProviderDelegationTokenExtension.
addDelegationTokens(renewer, credentials);
if (tokens != null && kpTokens != null) {
Token[] all = new Token[tokens.length + kpTokens.length];
System.arraycopy(tokens, 0, all, 0, tokens.length);
System.arraycopy(kpTokens, 0, all, tokens.length, kpTokens.length);
tokens = all;
} else {
tokens = (tokens != null) ? tokens : kpTokens;
}
}
return tokens;
}
vs
@Override
public Token[] addDelegationTokens(
final String renewer, Credentials credentials) throws IOException {
Token[] tokens = super.addDelegationTokens(renewer, credentials);
URI keyProviderUri = dfs.getKeyProviderUri();
if (keyProviderUri != null) {
KeyProviderDelegationTokenExtension keyProviderDelegationTokenExtension =
KeyProviderDelegationTokenExtension.
createKeyProviderDelegationTokenExtension(dfs.getKeyProvider());
Token[] kpTokens = keyProviderDelegationTokenExtension.
addDelegationTokens(renewer, credentials);
credentials.addSecretKey(dfs.getKeyProviderMapKey(),
DFSUtil.string2Bytes(keyProviderUri.toString()));
if (tokens != null && kpTokens != null) {
Token[] all = new Token[tokens.length + kpTokens.length];
System.arraycopy(tokens, 0, all, 0, tokens.length);
System.arraycopy(kpTokens, 0, all, tokens.length, kpTokens.length);
tokens = all;
} else {
tokens = (tokens != null) ? tokens : kpTokens;
}
}
return tokens;
}
> Client should always ask namenode for kms provider path.
>
>
> Key: HADOOP-14104
> URL: https://issues.apache.org/jira/browse/HADOOP-14104
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Major
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2
>
> Attachments: HADOOP-14104-branch-2.8.patch,
> HADOOP-14104-branch-2.patch, HADOOP-14104-trunk-v1.patch,
> HADOOP-14104-trunk-v2.patch, HADOOP-14104-trunk-v3.patch,
> HADOOP-14104-trunk-v4.patch, HADOOP-14104-trunk-v5.patch,
> HADOOP-14104-trunk.patch
>
>
> According to current implementation of kms provider in client conf, there can
> only be one kms.
> In multi-cluster environment, if a client is reading encrypted data from
> multiple clusters it will only get kms token for local cluster.
> Not sure whether the target version is correct or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15250) Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[
https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459056#comment-16459056
]
Greg Senia commented on HADOOP-15250:
-
[~ajayydv] Feel free to run with it!!!
> Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
> --
>
> Key: HADOOP-15250
> URL: https://issues.apache.org/jira/browse/HADOOP-15250
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, net
>Affects Versions: 2.7.3, 2.9.0, 3.0.0
> Environment: Multihome cluster with split DNS and rDNS lookup of
> localhost returning non-routable IPAddr
>Reporter: Greg Senia
>Priority: Critical
> Attachments: HADOOP-15250.00.patch, HADOOP-15250.patch
>
>
> We run our Hadoop clusters with two networks attached to each node. These
> network are as follows a server network that is firewalled with firewalld
> allowing inbound traffic: only SSH and things like Knox and Hiveserver2 and
> the HTTP YARN RM/ATS and MR History Server. The second network is the cluster
> network on the second network interface this uses Jumbo frames and is open no
> restrictions and allows all cluster traffic to flow between nodes.
>
> To resolve DNS within the Hadoop Cluster we use DNS Views via BIND so if the
> traffic is originating from nodes with cluster networks we return the
> internal DNS record for the nodes. This all works fine with all the
> multi-homing features added to Hadoop 2.x
> Some logic around views:
> a. The internal view is used by cluster machines when performing lookups. So
> hosts on the cluster network should get answers from the internal view in DNS
> b. The external view is used by non-local-cluster machines when performing
> lookups. So hosts not on the cluster network should get answers from the
> external view in DNS
>
> So this brings me to our problem. We created some firewall rules to allow
> inbound traffic from each clusters server network to allow distcp to occur.
> But we noticed a problem almost immediately that when YARN attempted to talk
> to the Remote Cluster it was binding outgoing traffic to the cluster network
> interface which IS NOT routable. So after researching the code we noticed the
> following in NetUtils.java and Client.java
> Basically in Client.java it looks as if it takes whatever the hostname is and
> attempts to bind to whatever the hostname is resolved to. This is not valid
> in a multi-homed network with one routable interface and one non routable
> interface. After reading through the java.net.Socket documentation it is
> valid to perform socket.bind(null) which will allow the OS routing table and
> DNS to send the traffic to the correct interface. I will also attach the
> nework traces and a test patch for 2.7.x and 3.x code base. I have this test
> fix below in my Hadoop Test Cluster.
> Client.java:
>
> |/*|
> | | * Bind the socket to the host specified in the principal name of the|
> | | * client, to ensure Server matching address of the client connection|
> | | * to host name in principal passed.|
> | | */|
> | |InetSocketAddress bindAddr = null;|
> | |if (ticket != null && ticket.hasKerberosCredentials()) {|
> | |KerberosInfo krbInfo =|
> | |remoteId.getProtocol().getAnnotation(KerberosInfo.class);|
> | |if (krbInfo != null) {|
> | |String principal = ticket.getUserName();|
> | |String host = SecurityUtil.getHostFromPrincipal(principal);|
> | |// If host name is a valid local address then bind socket to it|
> | |{color:#FF}*InetAddress localAddr =
> NetUtils.getLocalInetAddress(host);*{color}|
> |{color:#FF} ** {color}|if (localAddr != null) {|
> | |this.socket.setReuseAddress(true);|
> | |if (LOG.isDebugEnabled()) {|
> | |LOG.debug("Binding " + principal + " to " + localAddr);|
> | |}|
> | |*{color:#FF}bindAddr = new InetSocketAddress(localAddr, 0);{color}*|
> | *{color:#FF}{color}* |*{color:#FF}}{color}*|
> | |}|
> | |}|
>
> So in my Hadoop 2.7.x Cluster I made the following changes and traffic flows
> correctly out the correct interfaces:
>
> diff --git
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
>
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> index e1be271..c5b4a42 100644
> ---
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> +++
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> @@ -305,6 +305,9 @@
> public static final String IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_KEY
> = "ipc.client.fallback-to-simple-auth-allowed";
> public static final boolean
>
[jira] [Commented] (HADOOP-15250) Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[ https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373778#comment-16373778 ] Greg Senia commented on HADOOP-15250: - [~ste...@apache.org] so I have some good news after doing some digging this afternoon. I originally found this IPC Client bind-address issue in an attempt to *distcp* between two clusters that have no explicit cross-realm kerberos trust between them other than they both have a one-way cross realm trust to our Active Directory environment. The example I list below shows how I was able to perform a *distcp* without a cross-realm trust between the two clusters. But I still would like to verify what is occurring here with IPC/NetUtils outbound client connections especially in regards to firewalls and containers. Example/Details: *Distcp with Kerberos between Secure Clusters without Cross-realm Authentication* Two clusters with the realms: *Source Realm (PROD.HDP.EXAMPLE.COM) and Destination Realm (MODL.HDP.EXAMPLE.COM)* Data moves from the *Source Cluster (hdfs://prod) to the Destination Cluster (hdfs://modl)* A *one-way cross-realm* trust exists between *Source Realm (PROD.HDP.EXAMPLE.COM and Active Directory (NT.EXAMPLE.COM), and Destination Realm (MODL.HDP.EXAMPLE.COM) and Active Directory (NT.EXAMPLE.COM).* Both the Source Cluster (prod) and Destination Cluster (modl) are running a Hadoop Distribution with the following patches: https://issues.apache.org/jira/browse/HDFS-7546 https://issues.apache.org/jira/browse/YARN-3021 We set mapreduce.job.hdfs-servers.token-renewal.exclude property which apparently instructs the Resource Managers on either cluster to skip or perform delegation token renewal for NameNode hosts and we also set the dfs.namenode.kerberos.principal.pattern property to * to allow distcp irrespective of the principal patterns of the source and destination clusters *Example of Command that works:* hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=* -Dmapreduce.job.hdfs-servers.token-renewal.exclude=modl hdfs:///public_data hdfs://modl/public_data/gss_test2 > Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong > -- > > Key: HADOOP-15250 > URL: https://issues.apache.org/jira/browse/HADOOP-15250 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, net >Affects Versions: 2.7.3, 2.9.0, 3.0.0 > Environment: Multihome cluster with split DNS and rDNS lookup of > localhost returning non-routable IPAddr >Reporter: Greg Senia >Priority: Critical > Attachments: HADOOP-15250.patch > > > We run our Hadoop clusters with two networks attached to each node. These > network are as follows a server network that is firewalled with firewalld > allowing inbound traffic: only SSH and things like Knox and Hiveserver2 and > the HTTP YARN RM/ATS and MR History Server. The second network is the cluster > network on the second network interface this uses Jumbo frames and is open no > restrictions and allows all cluster traffic to flow between nodes. > > To resolve DNS within the Hadoop Cluster we use DNS Views via BIND so if the > traffic is originating from nodes with cluster networks we return the > internal DNS record for the nodes. This all works fine with all the > multi-homing features added to Hadoop 2.x > Some logic around views: > a. The internal view is used by cluster machines when performing lookups. So > hosts on the cluster network should get answers from the internal view in DNS > b. The external view is used by non-local-cluster machines when performing > lookups. So hosts not on the cluster network should get answers from the > external view in DNS > > So this brings me to our problem. We created some firewall rules to allow > inbound traffic from each clusters server network to allow distcp to occur. > But we noticed a problem almost immediately that when YARN attempted to talk > to the Remote Cluster it was binding outgoing traffic to the cluster network > interface which IS NOT routable. So after researching the code we noticed the > following in NetUtils.java and Client.java > Basically in Client.java it looks as if it takes whatever the hostname is and > attempts to bind to whatever the hostname is resolved to. This is not valid > in a multi-homed network with one routable interface and one non routable > interface. After reading through the java.net.Socket documentation it is > valid to perform socket.bind(null) which will allow the OS routing table and > DNS to send the traffic to the correct interface. I will also attach the > nework traces and a test patch for 2.7.x and 3.x code base. I have this test > fix below in my Hadoop Test Cluster. > Client.java: >
[jira] [Commented] (HADOOP-15250) Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[ https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372935#comment-16372935 ] Greg Senia commented on HADOOP-15250: - These parameters the two DNS ones would never work in a splitview DNS environment with DNS servers correctly configured to determine the hostsname. Remember in our scenerio we want all local traffic to the cluster to use the cluster network and anything destined to the datacenter networks to use the server interfaces. This means you should NOT be binding to a non-routable address. This also raises the question that anyone putting in values to /etc/hosts would cause that to bind incorrectly say like 127.0.0.10. Hadoop should be relying on the OS for DNS and IP routing information like other software stacks in the middleware space do. So I guess the question is. Is Hadoop going to support this multi-homing configuration which is similar to the SDN/Docker setup and our setup here. Nothing in the HWX articles state that it won't: https://community.hortonworks.com/articles/24277/parameters-for-multi-homing.html hadoop.security.dns.nameserver The host name or IP address of the name server (DNS) which a service Node should use to determine its own host name for Kerberos Login. Requires hadoop.security.dns.interface. Most clusters will not require this setting. hadoop.security.dns.interface The name of the Network Interface from which the service should determine its host name for Kerberos login. e.g. eth2. In a multi-homed environment, the setting can be used to affect the _HOST substitution in the service Kerberos principal. If this configuration value is not set, the service will use its default hostname as returned by InetAddress.getLocalHost().getCanonicalHostName(). Most clusters will not require this setting. In regards to hadoop.rpc.protection shouldn't this be what is guarding against the man in the middle not a null check on if a hostname has an ip asscoaited with to bind outbound??? hadoop.rpc.protection privacy A comma-separated list of protection values for secured sasl connections. Possible values are authentication, integrity and privacy. authentication means authentication only and no integrity or privacy; integrity implies authentication and integrity are enabled; and privacy implies all of authentication, integrity and privacy are enabled. hadoop.security.saslproperties.resolver.class can be used to override the hadoop.rpc.protection for a connection at the server side. Here are all of our values for binding to support multi-homed networks per the documentation. Unforunately using the DNS options is not a valid solution with our network design. We did our due diligence spent almost a week forumulating a solution to this problem do not just assume we didn't set the parameters. core-site.xml: ipc.client.nobind.local.addr=true hadoop.rpc.protection=privacy hdfs-site.xml: dfs.client.use.datanode.hostname=true dfs.datanode.use.datanode.hostname=true dfs.namenode.http-bind-host=0.0.0.0 dfs.namenode.https-bind-host=0.0.0.0 dfs.namenode.rpc-bind-host=0.0.0.0 dfs.namenode.lifeline.rpc-bind-host=0.0.0.0 dfs.namenode.servicerpc-bind-host=0.0.0.0 dfs.datanode.address=0.0.0.0:1019 dfs.datanode.http.address=0.0.0.0:1022 dfs.datanode.https.address=0.0.0.0:50475 dfs.datanode.ipc.address=0.0.0.0:8010 dfs.journalnode.http-address=0.0.0.0:8480 dfs.journalnode.https-address=0.0.0.0:8481 dfs.namenode.http-address.tech.nn1=ha21t51nn.tech.hdp.example.com:50070 dfs.namenode.http-address.tech.nn2=ha21t52nn.tech.hdp.example.com:50070 dfs.namenode.http-address.unit.nn1=ha21d51nn.unit.hdp.example.com:50070 dfs.namenode.http-address.unit.nn2=ha21d52nn.unit.hdp.example.com:50070 dfs.namenode.https-address.tech.nn1=ha21t51nn.tech.hdp.example.com:50470 dfs.namenode.https-address.tech.nn2=ha21t52nn.tech.hdp.example.com:50470 dfs.namenode.lifeline.rpc-address.tech.nn1=ha21t51nn.tech.hdp.example.com:8050 dfs.namenode.lifeline.rpc-address.tech.nn2=ha21t52nn.tech.hdp.example.com:8050 dfs.namenode.rpc-address.tech.nn1=ha21t51nn.tech.hdp.example.com:8020 dfs.namenode.rpc-address.tech.nn2=ha21t52nn.tech.hdp.example.com:8020 dfs.namenode.rpc-address.unit.nn1=ha21d51nn.unit.hdp.example.com:8020 dfs.namenode.rpc-address.unit.nn2=ha21d52nn.unit.hdp.example.com:8020 dfs.namenode.servicerpc-address.tech.nn1=ha21t51nn.tech.hdp.example.com:8040 dfs.namenode.servicerpc-address.tech.nn2=ha21t52nn.tech.hdp.example.com:8040 dfs.namenode.servicerpc-address.unit.nn1=ha21d51nn.unit.hdp.example.com:8040 dfs.namenode.servicerpc-address.unit.nn2=ha21d52nn.unit.hdp.example.com:8040 hbase-site.xml: hbase.master.ipc.address=0.0.0.0 hbase.regionserver.ipc.address=0.0.0.0 hbase.master.info.bindAddress=0.0.0.0 mapred-site.xml: mapreduce.jobhistory.bind-host=0.0.0.0 mapreduce.jobhistory.address=ha21t52mn.tech.hdp.example.com:10020
[jira] [Commented] (HADOOP-15250) MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[
https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372818#comment-16372818
]
Greg Senia commented on HADOOP-15250:
-
[~ste...@apache.org] I think by binding to what is resolved by DNS you
inherently break the ability to do Multi-homing with a routable and
non-routable networks where you have split view DNS as in our case. We actually
ran this design two years ago up the support chain of our Hadoop Vendor and was
passed as being just fine. So the thought with splitview DNS is as follows: the
server interface is resolved to ha21d52mn.unit.hdp.example.com when outside the
cluster when inside the cluster ha21d52mn.unit.hdp.newyorklife.com is resolved
to the cluster networks interface this was why we went with the DNS split views
to support multi-homing correctly. If this functionality is NOT supported than
Hadoop Project should remove the multi-homing features as things going to
remote clusters will not work as shown by my trace/logs above. As Unit/Dev
Cluster Network is not routable to our Tech/Test Cluster Network. So traffic
would have to go out the server interfaces to get to another cluster hence why
the splitview DNS is valid and a fix along these lines should at least allow a
flag like the server components do to bind to 0.0.0.0. I do see HADOOP-7215 is
what introduced this but I don't think multi-homing was thought about when this
code was implemented and this code is really is only checking for null's. So
if this is not the right place than I think a solution needs to be determined
to determine how to allow a Client to bind without using a specific IP/hostname
as that is not a valid method with a multi-homed network. Also hadoop traffic
destined for other hosts in the cluster would go via the cluster network and
allow from use of jumbo frames as its not routable.
So when my co-workers and I decided to test this scenario out we modifed the
code to allow traffic to bind to a valid local address depending on where the
traffic needs to be destined this patch is working in our tech/test cluster. As
the java.net.Socket doc shows you can run bind and it will bind to a valid
local address. I mean attempting to bind outside of the OS opens up risks like
we are hitting with distcp attempting to bind to an invalid source address that
is non-routable
[https://docs.oracle.com/javase/8/docs/api/java/net/Socket.html#bind-java.net.SocketAddress-]
public void bind(
[SocketAddress|https://docs.oracle.com/javase/8/docs/api/java/net/SocketAddress.html]
bindpoint) throws
[IOException|https://docs.oracle.com/javase/8/docs/api/java/io/IOException.html]
Binds the socket to a local address.If the address is {{null}}, then the system
will pick up an ephemeral port and a valid local address to bind the socket.
|/**|
| | * Checks if {@code host} is a local host name and return {@link
InetAddress}|
| | * corresponding to that address.|
| | * |
| | * @param host the specified host|
| | * @return a valid local {@link InetAddress} or null|
| | * @throws SocketException if an I/O error occurs|
| | */|
| |public static InetAddress getLocalInetAddress(String host)|
| |throws SocketException {|
| |if (host == null) {|
| |return null;|
| |}|
| |InetAddress addr = null;|
| |try {|
| |addr = SecurityUtil.getByName(host);|
| |if (NetworkInterface.getByInetAddress(addr) == null) {|
| |addr = null; // Not a local address|
| |}|
| |} catch (UnknownHostException ignore) \{ }|
| |return addr;|
| |}|
> MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
>
>
> Key: HADOOP-15250
> URL: https://issues.apache.org/jira/browse/HADOOP-15250
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, net
>Affects Versions: 2.7.3, 2.9.0, 3.0.0
>Reporter: Greg Senia
>Priority: Critical
> Attachments: HADOOP-15250.patch
>
>
> We run our Hadoop clusters with two networks attached to each node. These
> network are as follows a server network that is firewalled with firewalld
> allowing inbound traffic: only SSH and things like Knox and Hiveserver2 and
> the HTTP YARN RM/ATS and MR History Server. The second network is the cluster
> network on the second network interface this uses Jumbo frames and is open no
> restrictions and allows all cluster traffic to flow between nodes.
>
> To resolve DNS within the Hadoop Cluster we use DNS Views via BIND so if the
> traffic is originating from nodes with cluster networks we return the
> internal DNS record for the nodes. This all works fine with all the
> multi-homing features added to Hadoop 2.x
> Some logic around views:
> a. The internal view is used by cluster machines when performing lookups. So
> hosts on the cluster network should get
[jira] [Commented] (HADOOP-15250) MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[
https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372201#comment-16372201
]
Greg Senia commented on HADOOP-15250:
-
To test this patch as a unit test you would need to have a multi-homed network
with one routable interface that only external traffic travels over and one
cluster network interface that is not-routable
> MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
>
>
> Key: HADOOP-15250
> URL: https://issues.apache.org/jira/browse/HADOOP-15250
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, net
>Affects Versions: 2.7.3, 2.9.0, 3.0.0
>Reporter: Greg Senia
>Priority: Critical
> Attachments: HADOOP-15250.patch
>
>
> We run our Hadoop clusters with two networks attached to each node. These
> network are as follows a server network that is firewalled with firewalld
> allowing inbound traffic: only SSH and things like Knox and Hiveserver2 and
> the HTTP YARN RM/ATS and MR History Server. The second network is the cluster
> network on the second network interface this uses Jumbo frames and is open no
> restrictions and allows all cluster traffic to flow between nodes.
>
> To resolve DNS within the Hadoop Cluster we use DNS Views via BIND so if the
> traffic is originating from nodes with cluster networks we return the
> internal DNS record for the nodes. This all works fine with all the
> multi-homing features added to Hadoop 2.x
> Some logic around views:
> a. The internal view is used by cluster machines when performing lookups. So
> hosts on the cluster network should get answers from the internal view in DNS
> b. The external view is used by non-local-cluster machines when performing
> lookups. So hosts not on the cluster network should get answers from the
> external view in DNS
>
> So this brings me to our problem. We created some firewall rules to allow
> inbound traffic from each clusters server network to allow distcp to occur.
> But we noticed a problem almost immediately that when YARN attempted to talk
> to the Remote Cluster it was binding outgoing traffic to the cluster network
> interface which IS NOT routable. So after researching the code we noticed the
> following in NetUtils.java and Client.java
> Basically in Client.java it looks as if it takes whatever the hostname is and
> attempts to bind to whatever the hostname is resolved to. This is not valid
> in a multi-homed network with one routable interface and one non routable
> interface. After reading through the java.net.Socket documentation it is
> valid to perform socket.bind(null) which will allow the OS routing table and
> DNS to send the traffic to the correct interface. I will also attach the
> nework traces and a test patch for 2.7.x and 3.x code base. I have this test
> fix below in my Hadoop Test Cluster.
> Client.java:
>
> |/*|
> | | * Bind the socket to the host specified in the principal name of the|
> | | * client, to ensure Server matching address of the client connection|
> | | * to host name in principal passed.|
> | | */|
> | |InetSocketAddress bindAddr = null;|
> | |if (ticket != null && ticket.hasKerberosCredentials()) {|
> | |KerberosInfo krbInfo =|
> | |remoteId.getProtocol().getAnnotation(KerberosInfo.class);|
> | |if (krbInfo != null) {|
> | |String principal = ticket.getUserName();|
> | |String host = SecurityUtil.getHostFromPrincipal(principal);|
> | |// If host name is a valid local address then bind socket to it|
> | |{color:#FF}*InetAddress localAddr =
> NetUtils.getLocalInetAddress(host);*{color}|
> |{color:#FF} ** {color}|if (localAddr != null) {|
> | |this.socket.setReuseAddress(true);|
> | |if (LOG.isDebugEnabled()) {|
> | |LOG.debug("Binding " + principal + " to " + localAddr);|
> | |}|
> | |*{color:#FF}bindAddr = new InetSocketAddress(localAddr, 0);{color}*|
> | *{color:#FF}{color}* |*{color:#FF}}{color}*|
> | |}|
> | |}|
>
> So in my Hadoop 2.7.x Cluster I made the following changes and traffic flows
> correctly out the correct interfaces:
>
> diff --git
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
>
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> index e1be271..c5b4a42 100644
> ---
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> +++
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> @@ -305,6 +305,9 @@
> public static final String IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_KEY
> = "ipc.client.fallback-to-simple-auth-allowed";
> public static final boolean
>
[jira] [Updated] (HADOOP-15250) MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[
https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-15250:
Status: Patch Available (was: Open)
> MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
>
>
> Key: HADOOP-15250
> URL: https://issues.apache.org/jira/browse/HADOOP-15250
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, net
>Affects Versions: 3.0.0, 2.9.0, 2.7.3
>Reporter: Greg Senia
>Priority: Critical
> Attachments: HADOOP-15250.patch
>
>
> We run our Hadoop clusters with two networks attached to each node. These
> network are as follows a server network that is firewalled with firewalld
> allowing inbound traffic: only SSH and things like Knox and Hiveserver2 and
> the HTTP YARN RM/ATS and MR History Server. The second network is the cluster
> network on the second network interface this uses Jumbo frames and is open no
> restrictions and allows all cluster traffic to flow between nodes.
>
> To resolve DNS within the Hadoop Cluster we use DNS Views via BIND so if the
> traffic is originating from nodes with cluster networks we return the
> internal DNS record for the nodes. This all works fine with all the
> multi-homing features added to Hadoop 2.x
> Some logic around views:
> a. The internal view is used by cluster machines when performing lookups. So
> hosts on the cluster network should get answers from the internal view in DNS
> b. The external view is used by non-local-cluster machines when performing
> lookups. So hosts not on the cluster network should get answers from the
> external view in DNS
>
> So this brings me to our problem. We created some firewall rules to allow
> inbound traffic from each clusters server network to allow distcp to occur.
> But we noticed a problem almost immediately that when YARN attempted to talk
> to the Remote Cluster it was binding outgoing traffic to the cluster network
> interface which IS NOT routable. So after researching the code we noticed the
> following in NetUtils.java and Client.java
> Basically in Client.java it looks as if it takes whatever the hostname is and
> attempts to bind to whatever the hostname is resolved to. This is not valid
> in a multi-homed network with one routable interface and one non routable
> interface. After reading through the java.net.Socket documentation it is
> valid to perform socket.bind(null) which will allow the OS routing table and
> DNS to send the traffic to the correct interface. I will also attach the
> nework traces and a test patch for 2.7.x and 3.x code base. I have this test
> fix below in my Hadoop Test Cluster.
> Client.java:
>
> |/*|
> | | * Bind the socket to the host specified in the principal name of the|
> | | * client, to ensure Server matching address of the client connection|
> | | * to host name in principal passed.|
> | | */|
> | |InetSocketAddress bindAddr = null;|
> | |if (ticket != null && ticket.hasKerberosCredentials()) {|
> | |KerberosInfo krbInfo =|
> | |remoteId.getProtocol().getAnnotation(KerberosInfo.class);|
> | |if (krbInfo != null) {|
> | |String principal = ticket.getUserName();|
> | |String host = SecurityUtil.getHostFromPrincipal(principal);|
> | |// If host name is a valid local address then bind socket to it|
> | |{color:#FF}*InetAddress localAddr =
> NetUtils.getLocalInetAddress(host);*{color}|
> |{color:#FF} ** {color}|if (localAddr != null) {|
> | |this.socket.setReuseAddress(true);|
> | |if (LOG.isDebugEnabled()) {|
> | |LOG.debug("Binding " + principal + " to " + localAddr);|
> | |}|
> | |*{color:#FF}bindAddr = new InetSocketAddress(localAddr, 0);{color}*|
> | *{color:#FF}{color}* |*{color:#FF}}{color}*|
> | |}|
> | |}|
>
> So in my Hadoop 2.7.x Cluster I made the following changes and traffic flows
> correctly out the correct interfaces:
>
> diff --git
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
>
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> index e1be271..c5b4a42 100644
> ---
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> +++
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> @@ -305,6 +305,9 @@
> public static final String IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_KEY
> = "ipc.client.fallback-to-simple-auth-allowed";
> public static final boolean
> IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_DEFAULT = false;
>
> + public static final String IPC_CLIENT_NO_BIND_LOCAL_ADDR_KEY =
> "ipc.client.nobind.local.addr";
> + public static final boolean
[jira] [Updated] (HADOOP-15250) MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[
https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-15250:
Attachment: HADOOP-15250.patch
> MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
>
>
> Key: HADOOP-15250
> URL: https://issues.apache.org/jira/browse/HADOOP-15250
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, net
>Affects Versions: 2.7.3, 2.9.0, 3.0.0
>Reporter: Greg Senia
>Priority: Critical
> Attachments: HADOOP-15250.patch
>
>
> We run our Hadoop clusters with two networks attached to each node. These
> network are as follows a server network that is firewalled with firewalld
> allowing inbound traffic: only SSH and things like Knox and Hiveserver2 and
> the HTTP YARN RM/ATS and MR History Server. The second network is the cluster
> network on the second network interface this uses Jumbo frames and is open no
> restrictions and allows all cluster traffic to flow between nodes.
>
> To resolve DNS within the Hadoop Cluster we use DNS Views via BIND so if the
> traffic is originating from nodes with cluster networks we return the
> internal DNS record for the nodes. This all works fine with all the
> multi-homing features added to Hadoop 2.x
> Some logic around views:
> a. The internal view is used by cluster machines when performing lookups. So
> hosts on the cluster network should get answers from the internal view in DNS
> b. The external view is used by non-local-cluster machines when performing
> lookups. So hosts not on the cluster network should get answers from the
> external view in DNS
>
> So this brings me to our problem. We created some firewall rules to allow
> inbound traffic from each clusters server network to allow distcp to occur.
> But we noticed a problem almost immediately that when YARN attempted to talk
> to the Remote Cluster it was binding outgoing traffic to the cluster network
> interface which IS NOT routable. So after researching the code we noticed the
> following in NetUtils.java and Client.java
> Basically in Client.java it looks as if it takes whatever the hostname is and
> attempts to bind to whatever the hostname is resolved to. This is not valid
> in a multi-homed network with one routable interface and one non routable
> interface. After reading through the java.net.Socket documentation it is
> valid to perform socket.bind(null) which will allow the OS routing table and
> DNS to send the traffic to the correct interface. I will also attach the
> nework traces and a test patch for 2.7.x and 3.x code base. I have this test
> fix below in my Hadoop Test Cluster.
> Client.java:
>
> |/*|
> | | * Bind the socket to the host specified in the principal name of the|
> | | * client, to ensure Server matching address of the client connection|
> | | * to host name in principal passed.|
> | | */|
> | |InetSocketAddress bindAddr = null;|
> | |if (ticket != null && ticket.hasKerberosCredentials()) {|
> | |KerberosInfo krbInfo =|
> | |remoteId.getProtocol().getAnnotation(KerberosInfo.class);|
> | |if (krbInfo != null) {|
> | |String principal = ticket.getUserName();|
> | |String host = SecurityUtil.getHostFromPrincipal(principal);|
> | |// If host name is a valid local address then bind socket to it|
> | |{color:#FF}*InetAddress localAddr =
> NetUtils.getLocalInetAddress(host);*{color}|
> |{color:#FF} ** {color}|if (localAddr != null) {|
> | |this.socket.setReuseAddress(true);|
> | |if (LOG.isDebugEnabled()) {|
> | |LOG.debug("Binding " + principal + " to " + localAddr);|
> | |}|
> | |*{color:#FF}bindAddr = new InetSocketAddress(localAddr, 0);{color}*|
> | *{color:#FF}{color}* |*{color:#FF}}{color}*|
> | |}|
> | |}|
>
> So in my Hadoop 2.7.x Cluster I made the following changes and traffic flows
> correctly out the correct interfaces:
>
> diff --git
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
>
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> index e1be271..c5b4a42 100644
> ---
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> +++
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> @@ -305,6 +305,9 @@
> public static final String IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_KEY
> = "ipc.client.fallback-to-simple-auth-allowed";
> public static final boolean
> IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_DEFAULT = false;
>
> + public static final String IPC_CLIENT_NO_BIND_LOCAL_ADDR_KEY =
> "ipc.client.nobind.local.addr";
> + public static final boolean
[jira] [Commented] (HADOOP-15250) MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[
https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372050#comment-16372050
]
Greg Senia commented on HADOOP-15250:
-
Interface Information. The Interface *{color:#33}eno33559296{color}* is the
cluster interface which is non-routable. Interface *eno16780032* ** is the
server network publically accessible interface. No Hadoop Traffic flows over
this interface unless its traffic originating/terminating outside of the Hadoop
Cluster. I've also included the routing table below and a network trace showing
the wrong IP being used due to the fact that Client.java binds outbound calls
to the hostname it finds in DNS or /etc/hosts.
[root@ha21d52mn yarn]# ifconfig -a
{color:#33}*eno16780032: flags=4163 mtu
1500*{color}
{color:#33} *inet 10.69.81.1 netmask 255.255.240.0 broadcast
10.69.95.255*{color}
inet6 fe80::250:56ff:fe82:4934 prefixlen 64 scopeid 0x20
ether 00:50:56:82:49:34 txqueuelen 1000 (Ethernet)
RX packets 84514982 bytes 25768335637 (23.9 GiB)
RX errors 0 dropped 6172 overruns 0 frame 0
TX packets 83332181 bytes 18600190794 (17.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
{color:#33}*eno33559296: flags=4163 mtu
9000*{color}
{color:#33} *inet 10.70.49.1 netmask 255.255.240.0 broadcast
10.70.63.255*{color}
inet6 fe80::250:56ff:fe82:379c prefixlen 64 scopeid 0x20
ether 00:50:56:82:37:9c txqueuelen 1000 (Ethernet)
RX packets 1649741562 bytes 868670646085 (809.0 GiB)
RX errors 0 dropped 5052 overruns 0 frame 0
TX packets 1248707764 bytes 1782972383010 (1.6 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@ha21d52mn yarn]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
*default gateway 0.0.0.0 UG 0 0 0
eno16780032*
*10.69.80.0 0.0.0.0 255.255.240.0 U 0 0 0
eno16780032*
*10.70.48.0 0.0.0.0 255.255.240.0 U 0 0 0
eno33559296*
link-local 0.0.0.0 255.255.0.0 U 1002 0 0
eno16780032
link-local 0.0.0.0 255.255.0.0 U 1003 0 0
eno33559296
Here is an Example of the attempt to send traffic using the wrong interface
before the patch:
[root@ha21d52mn yarn]# tcpdump -s0 -i eno16780032 -nn host ha21t51nn.tech.hdp
or host ha21t52nn.tech.hdp and port 8020
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno16780032, link-type EN10MB (Ethernet), capture size 65535 bytes
16:27:35.656967 IP 10.70.49.1.34065 > 10.69.49.7.8020: Flags [S], seq 5653335,
win 29200, options [mss 1460,sackOK,TS val 3923129189 ecr 0,nop,wscale 7],
length 0
16:27:36.659542 IP 10.70.49.1.34065 > 10.69.49.7.8020: Flags [S], seq 5653335,
win 29200, options [mss 1460,sackOK,TS val 3923130192 ecr 0,nop,wscale 7],
length 0
16:27:38.663551 IP *10.70.49.1*.34065 > 10.69.49.7.8020: Flags [S], seq
5653335, win 29200, options [mss 1460,sackOK,TS val 3923132196 ecr 0,nop,wscale
7], length 0
16:27:42.675539 IP 10.70.49.1.34065 > 10.69.49.7.8020: Flags [S], seq 5653335,
win 29200, options [mss 1460,sackOK,TS val 3923136208 ecr 0,nop,wscale 7],
length 0
^C
2018-02-21 16:23:55,075 INFO retry.RetryInvocationHandler
(RetryInvocationHandler.java:log(267)) - Exception while invoking
ClientNamenodeProtocolTranslatorPB.renewDelegationToken over
ha21t52nn.tech.hdp.example.com/10.69.49.7:8020 after 1 failover attempts.
Trying to failover after sleeping for 1290ms.
org.apache.hadoop.net.ConnectTimeoutException: Call From
ha21d52mn.unit.hdp.example.com/10.70.49.1 to
ha21t52nn.tech.hdp.example.com:8020 failed on socket timeout exception:
org.apache.hadoop.net.ConnectTimeoutException: 2 millis timeout while
waiting for channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending local=/10.70.49.1:35231
remote=ha21t52nn.tech.hdp.example.com/10.69.49.7:8020]; For more details see:
http://wiki.apache.org/hadoop/SocketTimeout
at sun.reflect.GeneratedConstructorAccessor205.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:751)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1556)
at org.apache.hadoop.ipc.Client.call(Client.java:1496)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at
[jira] [Updated] (HADOOP-15250) MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[
https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-15250:
Summary: MultiHomed Server Network Cluster Network IPC Client Bind Addr
Wrong (was: MultiHomed Server Network Cluster Network )
> MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
>
>
> Key: HADOOP-15250
> URL: https://issues.apache.org/jira/browse/HADOOP-15250
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, net
>Affects Versions: 2.7.3, 2.9.0, 3.0.0
>Reporter: Greg Senia
>Priority: Critical
>
> We run our Hadoop clusters with two networks attached to each node. These
> network are as follows a server network that is firewalled with firewalld
> allowing inbound traffic: only SSH and things like Knox and Hiveserver2 and
> the HTTP YARN RM/ATS and MR History Server. The second network is the cluster
> network on the second network interface this uses Jumbo frames and is open no
> restrictions and allows all cluster traffic to flow between nodes.
>
> To resolve DNS within the Hadoop Cluster we use DNS Views via BIND so if the
> traffic is originating from nodes with cluster networks we return the
> internal DNS record for the nodes. This all works fine with all the
> multi-homing features added to Hadoop 2.x
> Some logic around views:
> a. The internal view is used by cluster machines when performing lookups. So
> hosts on the cluster network should get answers from the internal view in DNS
> b. The external view is used by non-local-cluster machines when performing
> lookups. So hosts not on the cluster network should get answers from the
> external view in DNS
>
> So this brings me to our problem. We created some firewall rules to allow
> inbound traffic from each clusters server network to allow distcp to occur.
> But we noticed a problem almost immediately that when YARN attempted to talk
> to the Remote Cluster it was binding outgoing traffic to the cluster network
> interface which IS NOT routable. So after researching the code we noticed the
> following in NetUtils.java and Client.java
> Basically in Client.java it looks as if it takes whatever the hostname is and
> attempts to bind to whatever the hostname is resolved to. This is not valid
> in a multi-homed network with one routable interface and one non routable
> interface. After reading through the java.net.Socket documentation it is
> valid to perform socket.bind(null) which will allow the OS routing table and
> DNS to send the traffic to the correct interface. I will also attach the
> nework traces and a test patch for 2.7.x and 3.x code base. I have this test
> fix below in my Hadoop Test Cluster.
> Client.java:
>
> |/*|
> | | * Bind the socket to the host specified in the principal name of the|
> | | * client, to ensure Server matching address of the client connection|
> | | * to host name in principal passed.|
> | | */|
> | |InetSocketAddress bindAddr = null;|
> | |if (ticket != null && ticket.hasKerberosCredentials()) {|
> | |KerberosInfo krbInfo =|
> | |remoteId.getProtocol().getAnnotation(KerberosInfo.class);|
> | |if (krbInfo != null) {|
> | |String principal = ticket.getUserName();|
> | |String host = SecurityUtil.getHostFromPrincipal(principal);|
> | |// If host name is a valid local address then bind socket to it|
> | |{color:#FF}*InetAddress localAddr =
> NetUtils.getLocalInetAddress(host);*{color}|
> |{color:#FF} ** {color}|if (localAddr != null) {|
> | |this.socket.setReuseAddress(true);|
> | |if (LOG.isDebugEnabled()) {|
> | |LOG.debug("Binding " + principal + " to " + localAddr);|
> | |}|
> | |*{color:#FF}bindAddr = new InetSocketAddress(localAddr, 0);{color}*|
> | *{color:#FF}{color}* |*{color:#FF}}{color}*|
> | |}|
> | |}|
>
> So in my Hadoop 2.7.x Cluster I made the following changes and traffic flows
> correctly out the correct interfaces:
>
> diff --git
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
>
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> index e1be271..c5b4a42 100644
> ---
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> +++
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> @@ -305,6 +305,9 @@
> public static final String IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_KEY
> = "ipc.client.fallback-to-simple-auth-allowed";
> public static final boolean
> IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_DEFAULT = false;
>
> + public static final String IPC_CLIENT_NO_BIND_LOCAL_ADDR_KEY =
> "ipc.client.nobind.local.addr";
> +
[jira] [Created] (HADOOP-15250) MultiHomed Server Network Cluster Network
Greg Senia created HADOOP-15250:
---
Summary: MultiHomed Server Network Cluster Network
Key: HADOOP-15250
URL: https://issues.apache.org/jira/browse/HADOOP-15250
Project: Hadoop Common
Issue Type: Improvement
Components: ipc, net
Affects Versions: 3.0.0, 2.9.0, 2.7.3
Reporter: Greg Senia
We run our Hadoop clusters with two networks attached to each node. These
network are as follows a server network that is firewalled with firewalld
allowing inbound traffic: only SSH and things like Knox and Hiveserver2 and the
HTTP YARN RM/ATS and MR History Server. The second network is the cluster
network on the second network interface this uses Jumbo frames and is open no
restrictions and allows all cluster traffic to flow between nodes.
To resolve DNS within the Hadoop Cluster we use DNS Views via BIND so if the
traffic is originating from nodes with cluster networks we return the internal
DNS record for the nodes. This all works fine with all the multi-homing
features added to Hadoop 2.x
Some logic around views:
a. The internal view is used by cluster machines when performing lookups. So
hosts on the cluster network should get answers from the internal view in DNS
b. The external view is used by non-local-cluster machines when performing
lookups. So hosts not on the cluster network should get answers from the
external view in DNS
So this brings me to our problem. We created some firewall rules to allow
inbound traffic from each clusters server network to allow distcp to occur. But
we noticed a problem almost immediately that when YARN attempted to talk to the
Remote Cluster it was binding outgoing traffic to the cluster network interface
which IS NOT routable. So after researching the code we noticed the following
in NetUtils.java and Client.java
Basically in Client.java it looks as if it takes whatever the hostname is and
attempts to bind to whatever the hostname is resolved to. This is not valid in
a multi-homed network with one routable interface and one non routable
interface. After reading through the java.net.Socket documentation it is valid
to perform socket.bind(null) which will allow the OS routing table and DNS to
send the traffic to the correct interface. I will also attach the nework traces
and a test patch for 2.7.x and 3.x code base. I have this test fix below in my
Hadoop Test Cluster.
Client.java:
|/*|
| | * Bind the socket to the host specified in the principal name of the|
| | * client, to ensure Server matching address of the client connection|
| | * to host name in principal passed.|
| | */|
| |InetSocketAddress bindAddr = null;|
| |if (ticket != null && ticket.hasKerberosCredentials()) {|
| |KerberosInfo krbInfo =|
| |remoteId.getProtocol().getAnnotation(KerberosInfo.class);|
| |if (krbInfo != null) {|
| |String principal = ticket.getUserName();|
| |String host = SecurityUtil.getHostFromPrincipal(principal);|
| |// If host name is a valid local address then bind socket to it|
| |{color:#FF}*InetAddress localAddr =
NetUtils.getLocalInetAddress(host);*{color}|
|{color:#FF} ** {color}|if (localAddr != null) {|
| |this.socket.setReuseAddress(true);|
| |if (LOG.isDebugEnabled()) {|
| |LOG.debug("Binding " + principal + " to " + localAddr);|
| |}|
| |*{color:#FF}bindAddr = new InetSocketAddress(localAddr, 0);{color}*|
| *{color:#FF}{color}* |*{color:#FF}}{color}*|
| |}|
| |}|
So in my Hadoop 2.7.x Cluster I made the following changes and traffic flows
correctly out the correct interfaces:
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
index e1be271..c5b4a42 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
@@ -305,6 +305,9 @@
public static final String IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_KEY =
"ipc.client.fallback-to-simple-auth-allowed";
public static final boolean
IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_DEFAULT = false;
+ public static final String IPC_CLIENT_NO_BIND_LOCAL_ADDR_KEY =
"ipc.client.nobind.local.addr";
+ public static final boolean IPC_CLIENT_NO_BIND_LOCAL_ADDR_DEFAULT = false;
+
public static final String IPC_CLIENT_CONNECT_MAX_RETRIES_ON_SASL_KEY =
"ipc.client.connect.max.retries.on.sasl";
public static final int IPC_CLIENT_CONNECT_MAX_RETRIES_ON_SASL_DEFAULT =
5;
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
index a6f4eb6..7bfddb7 100644
---
[jira] [Commented] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840625#comment-15840625
]
Greg Senia commented on HADOOP-13988:
-
Is this the error we are
Talking about:
Failed on local exception: java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Cannot get a KDC reply)]; Host
Details : local host is: "bcf70b846b20/172.17.0.2"; destination host is:
"localhost":56245;
Stack Trace
java.io.IOException: Failed on local exception: java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Cannot get a KDC reply)]; Host
Details : local host is: "bcf70b846b20/172.17.0.2"; destination host is:
"localhost":56245;
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
>Assignee: Xiaoyu Yao
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-13988.01.patch, HADOOP-13988.02.patch,
> HADOOP-13988.patch, HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838925#comment-15838925
]
Greg Senia commented on HADOOP-13988:
-
[~xyao] and [~lmccay] thanks for all the help with this issue. I appreciate you
guys digging in and helping get the right fix built.
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
>Assignee: Xiaoyu Yao
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-13988.01.patch, HADOOP-13988.02.patch,
> HADOOP-13988.patch, HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[ https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834756#comment-15834756 ] Greg Senia commented on HADOOP-13988: - [~xyao] and [~lmccay] here is log output from 02.patch 2017-01-23 10:29:17,424 DEBUG security.UserGroupInformation (UserGroupInformation.java:doAs(1744)) - PrivilegedActionException as:knox (auth:TOKEN) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby 2017-01-23 10:29:17,424 DEBUG security.UserGroupInformation (UserGroupInformation.java:doAs(1744)) - PrivilegedActionException as:knox (auth:TOKEN) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby 2017-01-23 10:29:17,426 DEBUG security.UserGroupInformation (UserGroupInformation.java:logPrivilegedAction(1767)) - PrivilegedAction as:knox (auth:TOKEN) from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:758) 2017-01-23 10:29:17,426 DEBUG security.UserGroupInformation (UserGroupInformation.java:logPrivilegedAction(1767)) - PrivilegedAction as:knox (auth:TOKEN) from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:758) 2017-01-23 10:29:17,437 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1774)) - UGI: gss2002 (auth:PROXY) via knox (auth:TOKEN) 2017-01-23 10:29:17,437 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1774)) - UGI: gss2002 (auth:PROXY) via knox (auth:TOKEN) 2017-01-23 10:29:17,437 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1776)) - +RealUGI: knox (auth:TOKEN) 2017-01-23 10:29:17,437 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1776)) - +RealUGI: knox (auth:TOKEN) 2017-01-23 10:29:17,437 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1778)) - +LoginUGI: dn/ha20t5001dn.tech.hdp.example@tech.hdp.example.com (auth:KERBEROS) 2017-01-23 10:29:17,437 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1778)) - +LoginUGI: dn/ha20t5001dn.tech.hdp.example@tech.hdp.example.com (auth:KERBEROS) 2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1780)) - +UGI token:Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:tech, Ident: (HDFS_DELEGATION_TOKEN token 14676 for gss2002) 2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1780)) - +UGI token:Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:tech, Ident: (HDFS_DELEGATION_TOKEN token 14676 for gss2002) 2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1780)) - +UGI token:Kind: HDFS_DELEGATION_TOKEN, Service: 10.70.33.7:8020, Ident: (HDFS_DELEGATION_TOKEN token 14676 for gss2002) 2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1780)) - +UGI token:Kind: HDFS_DELEGATION_TOKEN, Service: 10.70.33.7:8020, Ident: (HDFS_DELEGATION_TOKEN token 14676 for gss2002) 2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1780)) - +UGI token:Kind: HDFS_DELEGATION_TOKEN, Service: 10.70.33.6:8020, Ident: (HDFS_DELEGATION_TOKEN token 14676 for gss2002) 2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1780)) - +UGI token:Kind: HDFS_DELEGATION_TOKEN, Service: 10.70.33.6:8020, Ident: (HDFS_DELEGATION_TOKEN token 14676 for gss2002) 2017-01-23 10:29:17,438 DEBUG kms.KMSClientProvider (KMSClientProvider.java:getActualUgi(1061)) - using loginUser no KMS Delegation Token no Kerberos Credentials 2017-01-23 10:29:17,438 DEBUG kms.KMSClientProvider (KMSClientProvider.java:getActualUgi(1061)) - using loginUser no KMS Delegation Token no Kerberos Credentials 2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation (UserGroupInformation.java:logPrivilegedAction(1767)) - PrivilegedAction as:dn/ha20t5001dn.tech.hdp.example@tech.hdp.example.com (auth:KERBEROS) from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:524) 2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation (UserGroupInformation.java:logPrivilegedAction(1767)) - PrivilegedAction as:dn/ha20t5001dn.tech.hdp.example@tech.hdp.example.com (auth:KERBEROS) from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:524) 2017-01-23 10:29:17,439 DEBUG security.UserGroupInformation (UserGroupInformation.java:getTGT(898)) - Found tgt Ticket (hex) = Client Principal = dn/ha20t5001dn.tech.hdp.example@tech.hdp.example.com Server Principal = krbtgt/tech.hdp.example@tech.hdp.example.com Session Key =
[jira] [Commented] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834630#comment-15834630
]
Greg Senia commented on HADOOP-13988:
-
[~xyao] the second test fix seems to be working. I will leave it in my
environment for a few days to make sure as kerberos tickets expire that the fix
still works.
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.01.patch, HADOOP-13988.02.patch,
> HADOOP-13988.patch, HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831010#comment-15831010
]
Greg Senia commented on HADOOP-13988:
-
[~lmccay] the logs from above are from the patch uploaded an hour ago. Let me
know if it looks like code path is wrong from what I can see the code path is
working correctly and the !equals is definitely working correctly if it wasn't
it would of failed.
Also here is the patch output from my last build about an hour ago with the
updated path from today:
ETG-GSeni-MBP:hadoop-release gss2002$ patch -p1 <
../../kmsfixes/HADOOP-13558.02.patch
patching file
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
Hunk #1 succeeded at 618 with fuzz 1 (offset -14 lines).
Hunk #2 succeeded at 825 (offset -40 lines).
patching file
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java
Hunk #1 succeeded at 31 (offset -1 lines).
Hunk #2 succeeded at 902 with fuzz 2 (offset -111 lines).
ETG-GSeni-MBP:hadoop-release gss2002$ patch -p1 <
../../kmsfixes/HADOOP-13749.00.patch
patching file
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
Hunk #4 succeeded at 901 (offset 2 lines).
Hunk #5 succeeded at 924 (offset 2 lines).
Hunk #6 succeeded at 996 (offset 2 lines).
Hunk #7 succeeded at 1042 (offset 2 lines).
patching file
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
Hunk #1 succeeded at 1768 (offset -55 lines).
patching file
hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
Hunk #1 succeeded at 1825 (offset -8 lines).
Hunk #2 succeeded at 2149 (offset -5 lines).
ETG-GSeni-MBP:hadoop-release gss2002$ patch -p1 < ../../HADOOP-13988.patch
patching file
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
Hunk #1 succeeded at 1052 (offset -10 lines).
patching file
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
Hunk #1 succeeded at 1774 (offset -67 lines).
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch, HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no
[jira] [Commented] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[ https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830995#comment-15830995 ] Greg Senia commented on HADOOP-13988: - yes its running in our cluster. Just put the newest patch out there here is log output from DN getting the request from Knox: 2017-01-19 20:33:12,835 DEBUG security.UserGroupInformation (UserGroupInformation.java:logPrivilegedAction(1767)) - PrivilegedAction as:gss2002 (auth:PROXY) via knox (auth:TOKEN) from:org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114) 2017-01-19 20:33:12,835 DEBUG security.UserGroupInformation (UserGroupInformation.java:logPrivilegedAction(1767)) - PrivilegedAction as:gss2002 (auth:PROXY) via knox (auth:TOKEN) from:org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114) 2017-01-19 20:33:12,873 DEBUG security.SecurityUtil (SecurityUtil.java:setTokenService(421)) - Acquired token Kind: HDFS_DELEGATION_TOKEN, Service: 10.70.33.6:8020, Ident: (HDFS_DELEGATION_TOKEN token 14666 for gss2002) 2017-01-19 20:33:12,873 DEBUG security.SecurityUtil (SecurityUtil.java:setTokenService(421)) - Acquired token Kind: HDFS_DELEGATION_TOKEN, Service: 10.70.33.6:8020, Ident: (HDFS_DELEGATION_TOKEN token 14666 for gss2002) 2017-01-19 20:33:12,874 DEBUG security.SecurityUtil (SecurityUtil.java:setTokenService(421)) - Acquired token Kind: HDFS_DELEGATION_TOKEN, Service: 10.70.33.7:8020, Ident: (HDFS_DELEGATION_TOKEN token 14666 for gss2002) 2017-01-19 20:33:12,874 DEBUG security.SecurityUtil (SecurityUtil.java:setTokenService(421)) - Acquired token Kind: HDFS_DELEGATION_TOKEN, Service: 10.70.33.7:8020, Ident: (HDFS_DELEGATION_TOKEN token 14666 for gss2002) 2017-01-19 20:33:13,061 DEBUG security.UserGroupInformation (UserGroupInformation.java:logPrivilegedAction(1767)) - PrivilegedAction as:knox (auth:TOKEN) from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:758) 2017-01-19 20:33:13,061 DEBUG security.UserGroupInformation (UserGroupInformation.java:logPrivilegedAction(1767)) - PrivilegedAction as:knox (auth:TOKEN) from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:758) 2017-01-19 20:33:13,099 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1774)) - UGI: gss2002 (auth:PROXY) via knox (auth:TOKEN) 2017-01-19 20:33:13,099 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1774)) - UGI: gss2002 (auth:PROXY) via knox (auth:TOKEN) 2017-01-19 20:33:13,100 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1776)) - +RealUGI: knox (auth:TOKEN) 2017-01-19 20:33:13,100 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1776)) - +RealUGI: knox (auth:TOKEN) 2017-01-19 20:33:13,100 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1777)) - +RealUGI: shortName: knox 2017-01-19 20:33:13,100 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1777)) - +RealUGI: shortName: knox 2017-01-19 20:33:13,100 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1780)) - +LoginUGI: dn/ha20t5002dn.tech.hdp.example@tech.hdp.example.com (auth:KERBEROS) 2017-01-19 20:33:13,100 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1780)) - +LoginUGI: dn/ha20t5002dn.tech.hdp.example@tech.hdp.example.com (auth:KERBEROS) 2017-01-19 20:33:13,100 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1781)) - +LoginUGI shortName: hdfs 2017-01-19 20:33:13,100 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1781)) - +LoginUGI shortName: hdfs 2017-01-19 20:33:13,100 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1784)) - +UGI token:Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:tech, Ident: (HDFS_DELEGATION_TOKEN token 14666 for gss2002) 2017-01-19 20:33:13,100 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1784)) - +UGI token:Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:tech, Ident: (HDFS_DELEGATION_TOKEN token 14666 for gss2002) 2017-01-19 20:33:13,100 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1784)) - +UGI token:Kind: HDFS_DELEGATION_TOKEN, Service: 10.70.33.7:8020, Ident: (HDFS_DELEGATION_TOKEN token 14666 for gss2002) 2017-01-19 20:33:13,100 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1784)) - +UGI token:Kind: HDFS_DELEGATION_TOKEN, Service: 10.70.33.7:8020, Ident: (HDFS_DELEGATION_TOKEN token 14666 for gss2002) 2017-01-19 20:33:13,101 DEBUG security.UserGroupInformation (UserGroupInformation.java:logAllUserInfo(1784)) - +UGI token:Kind: HDFS_DELEGATION_TOKEN, Service: 10.70.33.6:8020, Ident: (HDFS_DELEGATION_TOKEN
[jira] [Updated] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-13988:
Status: Patch Available (was: Open)
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.7.3, 2.8.0
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch, HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-13988:
Status: Open (was: Patch Available)
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.7.3, 2.8.0
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch, HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-13988:
Attachment: HADOOP-13988.patch
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch, HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830882#comment-15830882
]
Greg Senia commented on HADOOP-13988:
-
[~xyao] We are currently running the fix patched into our HDP 2.5.3.0 build. We
grabbed the HDP-2.5.3.0-tag from HWX github and recompiled with this fix and
the two fixes this is dependent on. We have been running this fix for over a
week now in our test environment with 2 NNs w/HA and their associated
components 3 JN's and 2 ZKFC's, 2 RM's, 4 DN's/RS's/NM's, 2
HiveServer2/Metastores, 2 HBaseMasters and a node running Knox for WebHDFS,
Oozie and HiveServer2 http access and 1 Node as an Oozie Server. We have a data
ingest framework that runs continuously in this environment and has run with no
issues for the last week since applying the fixes and Knox to WebHDFS at a TDE
file is returned correctly. I will look at adjusting the above code in regards
to logging.
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15826110#comment-15826110
]
Greg Senia commented on HADOOP-13988:
-
Also in regards to test case let me know as this class doesn't have much around
test cases around it.
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823189#comment-15823189
]
Greg Senia commented on HADOOP-13988:
-
[~lmccay] made the changes sorry for delay. I think the test error is not
related to my patch can you verify also:
stGracefulFailoverMultipleZKfcs(org.apache.hadoop.ha.TestZKFailoverController)
Time elapsed: 70.289 sec <<< ERROR!
org.apache.hadoop.ha.ServiceFailedException: Unable to become active. Local
node did not get an opportunity to do so from ZooKeeper, or the local node took
too long to transition to active.
at
org.apache.hadoop.ha.ZKFailoverController.doGracefulFailover(ZKFailoverController.java:693)
at org.apache.hado
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-13988:
Status: Patch Available (was: Open)
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.7.3, 2.8.0
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-13988:
Attachment: HADOOP-13988.patch
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-13988:
Attachment: (was: HADOOP-13988.patch)
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-13988:
Status: Open (was: Patch Available)
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.7.3, 2.8.0
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-13988:
Status: Patch Available (was: Open)
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.7.3, 2.8.0
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-13988:
Attachment: (was: HADOOP-13988.patch)
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-13988:
Attachment: HADOOP-13988.patch
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-13988:
Status: Open (was: Patch Available)
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.7.3, 2.8.0
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822563#comment-15822563
]
Greg Senia commented on HADOOP-13988:
-
[~lmccay] I will fix shortly!
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822301#comment-15822301
]
Greg Senia commented on HADOOP-13988:
-
This patch requires these JIRAs to be included also
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-13988:
Attachment: HADOOP-13988.patch
Initial Patch that is running in our test environment right now across 25 nodes.
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Senia updated HADOOP-13988:
Affects Version/s: 2.8.0
Status: Patch Available (was: Open)
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.7.3, 2.8.0
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822170#comment-15822170
]
Greg Senia commented on HADOOP-13988:
-
[~lmccay] and [~xyao] I have my original patch I will attach it and we can
modify and test from there.
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
Greg Senia created HADOOP-13988:
---
Summary: KMSClientProvider does not work with WebHDFS and Apache
Knox w/ProxyUser
Key: HADOOP-13988
URL: https://issues.apache.org/jira/browse/HADOOP-13988
Project: Hadoop Common
Issue Type: Bug
Components: common, kms
Affects Versions: 2.7.3
Environment: HDP 2.5.3.0
WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
Reporter: Greg Senia
After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider issues
have not been resolved. We put a test build together and applied HADOOP-13558
and HADOOP-13749 these two fixes did still not solve the issue with requests
coming from WebHDFS through to Knox to a TDE zone.
So we added some debug to our build and determined effectively what is
happening here is a double proxy situation which does not seem to work. So we
propose the following fix in getActualUgi Method:
}
// Use current user by default
UserGroupInformation actualUgi = currentUgi;
if (currentUgi.getRealUser() != null) {
// Use real user for proxy user
if (LOG.isDebugEnabled()) {
LOG.debug("using RealUser for proxyUser);
}
actualUgi = currentUgi.getRealUser();
if (getDoAsUser() != null) {
if (LOG.isDebugEnabled()) {
LOG.debug("doAsUser exists");
LOG.debug("currentUGI realUser shortName: {}",
currentUgi.getRealUser().getShortUserName());
LOG.debug("processUGI loginUser shortName: {}",
UserGroupInformation.getLoginUser().getShortUserName());
}
if (currentUgi.getRealUser().getShortUserName() !=
UserGroupInformation.getLoginUser().getShortUserName()) {
if (LOG.isDebugEnabled()) {
LOG.debug("currentUGI.realUser does not match
UGI.processUser);
}
actualUgi = UserGroupInformation.getLoginUser();
if (LOG.isDebugEnabled()) {
LOG.debug("LoginUser for Proxy: {}",
actualUgi.getLoginUser());
}
}
}
} else if (!currentUgiContainsKmsDt() &&
!currentUgi.hasKerberosCredentials()) {
// Use login user for user that does not have either
// Kerberos credential or KMS delegation token for KMS operations
if (LOG.isDebugEnabled()) {
LOG.debug("using loginUser no KMS Delegation Token no Kerberos
Credentials");
}
actualUgi = currentUgi.getLoginUser();
}
return actualUgi;
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143360#comment-15143360 ] Greg Senia commented on HADOOP-9969: [~acmurthy] can we have a quick discussion on this JIRA to find out what is going on with it.. I think Dan or Beth will work to set something up.. > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.5.0, 2.5.2, 2.6.0, 2.6.1, 2.8.0, 2.7.1, > 2.6.2, 2.6.3 > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15139596#comment-15139596 ] Greg Senia commented on HADOOP-9969: [~daryn] I have reached out to IBM JDK Security team to try to get info on if IBM is doing it correctly.. I patched my HDP build from HWX and it seems to solve the issues.. But waiting to hear from IBM JDK folks... Any other info on plans to integrate this into the Core Hadoop build would be great.. thanks > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.5.0, 2.5.2, 2.6.0, 2.6.1, 2.8.0, 2.7.1, > 2.6.2, 2.6.3 > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137124#comment-15137124 ] Greg Senia commented on HADOOP-9969: This also affects IBM JDK8... > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137486#comment-15137486 ] Greg Senia commented on HADOOP-9969: [~crystal_gaoyu] and [~xinwei] I noticed it's stated that there are some other side-effects? Please advise. > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.6.0, 2.6.1, 2.8.0, 2.7.2, 2.6.3 > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Senia updated HADOOP-9969: --- Affects Version/s: 2.7.2 > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.7.2 > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Senia updated HADOOP-9969: --- Affects Version/s: 2.8.0 2.6.0 2.6.1 2.6.3 > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.6.0, 2.6.1, 2.8.0, 2.7.2, 2.6.3 > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Senia updated HADOOP-9969: --- Affects Version/s: 2.5.0 2.5.1 2.5.2 2.7.1 2.6.2 > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.5.0, 2.5.2, 2.6.0, 2.6.1, 2.8.0, 2.7.1, > 2.6.2, 2.6.3 > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Senia updated HADOOP-9969: --- Affects Version/s: (was: 2.7.2) (was: 2.5.1) > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.5.0, 2.5.2, 2.6.0, 2.6.1, 2.8.0, 2.7.1, > 2.6.2, 2.6.3 > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
