[jira] [Created] (HADOOP-15849) Upgrade netty version

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15849:
--

 Summary: Upgrade netty version
 Key: HADOOP-15849
 URL: https://issues.apache.org/jira/browse/HADOOP-15849
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Xiao Chen


We're currently at 3.10.5. It'd be good to upgrade to the latest 3.10.6 release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14238) [Umbrella] Rechecking Guava's object is not exposed to user-facing API

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen resolved HADOOP-14238.

Resolution: Fixed

Closing as I don't see any more work for this at this time

> [Umbrella] Rechecking Guava's object is not exposed to user-facing API
> --
>
> Key: HADOOP-14238
> URL: https://issues.apache.org/jira/browse/HADOOP-14238
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Tsuyoshi Ozawa
>Priority: Critical
>
> This is reported by [~hitesh] on HADOOP-10101.
> At least, AMRMClient#waitFor takes Guava's Supplier instance as an instance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Regarding Hadoop Erasure Coding architecture

[Xiao Chen]

Hi M.V.S.Chaitanya,

Thanks for the interest!

In case you didn't find it, upstream doc

has
the definition. This blog post

may
also help clarify things a bit.

Some answers inline.

Best,
-Xiao


On Mon, Jun 11, 2018 at 9:52 AM Chaitanya M V S 
wrote:

> Hi!
>
> We a group of people trying to understand the architecture of erasure
> coding in Hadoop 3.0. We have been facing difficulties to understand few
> terms and concepts regarding the same.
>
> 1. What do the terms Block, Block Group, Stripe, Cell and Chunk mean in the
> context of erasure coding (these terms have taken different meanings and
> have been used interchangeably over various documentation and blogs)? How
> has this been incorporated in reading and writing of EC data?

Checking the source code is probably the best way to get answers like how
the r/w of EC is done.

>
> 2. How has been the idea/concept of the block from previous versions
> carried over to EC?
>
Block is still largely the actual file on a datanode. In EC, a block group
contains several (9, in case of RS(6,3) ) blocks.

>
> 3. ‎The higher level APIs, that of ErasureCoders and ErasureCodec still
> hasn't been plugged into Hadoop. Also, I haven't found any new Jira
> regarding the same. Can I know if there are any updates or pointers
> regarding the incorporation of these APIs into Hadoop?
>
Not sure I understand what APIs are being referred here. A sample pointer
to hadoop implementation is
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec/ErasureCodec.java,
more can be looked up. :)

>
> 4. How is the datanode for reconstruction work chosen?  Also, how are the
> buffer sizes for the reconstruction work determined?
>
Suggest to look at source code in NN, specifically the BlockManager class.

>
>
> Thanks in advance for your time and considerations.
>
> Regards,
> M.V.S.Chaitanya
>

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Xiao Chen]

Thanks for the effort on this Yongjun.

+1 (binding)

   - Built from src
   - Deployed a pseudo distributed HDFS with KMS
   - Ran basic hdfs commands with encryption
   - Sanity checked webui and logs


-Xiao

On Fri, Jun 8, 2018 at 10:34 AM, Brahma Reddy Battula <
brahmareddy.batt...@hotmail.com> wrote:

> Thanks yongjun zhang for driving this release.
>
> +1 (binding).
>
>
> ---Built from the source
> ---Installed HA cluster
> ---Execute the basic shell commands
> ---Browsed the UI's
> ---Ran sample jobs like pi,wordcount
>
>
>
> 
> From: Yongjun Zhang 
> Sent: Friday, June 8, 2018 1:04 PM
> To: Allen Wittenauer
> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)
>
> BTW, thanks Allen and Steve for discussing and suggestion about the site
> build problem I hit earlier, I did the following step
>
> mvn install -DskipTests
>
> before doing the steps Nanda listed helped to solve the problems.
>
> --Yongjun
>
>
>
>
> On Thu, Jun 7, 2018 at 6:15 PM, Yongjun Zhang  wrote:
>
> > Thank you all very much for the testing, feedback and discussion!
> >
> > I was able to build outside docker, by following the steps Nanda
> > described, I saw the same problem; then I tried 3.0.2 released a while
> > back, it has the same issue.
> >
> > As Allen pointed out, it seems the steps to build site are not correct. I
> > have not figured out the correct steps yet.
> >
> > At this point, I think this issue should not block the 3.0.3 issue. While
> > at the same time we need to figure out the right steps to build the site.
> > Would you please let me know if you think differently?
> >
> > We only have the site build issue reported so far. And we don't have
> > enough PMC votes yet. So need some more PMCs to help.
> >
> > Thanks again, and best regards,
> >
> > --Yongjun
> >
> >
> > On Thu, Jun 7, 2018 at 4:15 PM, Allen Wittenauer <
> a...@effectivemachines.com
> > > wrote:
> >
> >> > On Jun 7, 2018, at 11:47 AM, Steve Loughran 
> >> wrote:
> >> >
> >> > Actually, Yongjun has been really good at helping me get set up for a
> >> 2.7.7 release, including "things you need to do to get GPG working in
> the
> >> docker image”
> >>
> >> *shrugs* I use a different release script after some changes
> >> broke the in-tree version for building on OS X and I couldn’t get the
> fixes
> >> committed upstream.  So not sure what the problems are that you are
> hitting.
> >>
> >> > On Jun 7, 2018, at 1:08 PM, Nandakumar Vadivelu <
> >> nvadiv...@hortonworks.com> wrote:
> >> >
> >> > It will be helpful if we can get the correct steps, and also update
> the
> >> wiki.
> >> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+
> >> Release+Validation
> >>
> >> Yup. Looking forward to seeing it.
> >> -
> >> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >>
> >>
> >
>

[jira] [Created] (HADOOP-15507) Add MapReduce counters about EC bytes read

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15507:
--

 Summary: Add MapReduce counters about EC bytes read
 Key: HADOOP-15507
 URL: https://issues.apache.org/jira/browse/HADOOP-15507
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Xiao Chen
Assignee: Xiao Chen


HDFS has added Erasure Coding support in HDFS-7285. There are HDFS level 
[ReadStatistics|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReadStatistics.java]
 so from DFSClient we can know how much reads are EC/replication.

In order for users to have a better view of how much of their workload is 
impacted by EC, we can expose EC read bytes to File System Counters, and to 
MapReduce's job counters. This way, end users can tell from MR jobs directly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-14445) Delegation tokens are not shared between KMS instances

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen reopened HADOOP-14445:


Reopening Jira as I'm reverting those changes.

Will remove fix versions as I proceed. Some minor conflicts due to HADOOP-14188 
and HADOOP-15313, so building repo and running touched tests before I push.

> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Fix For: 2.10.0, 2.8.4, 3.2.0, 3.1.1, 2.9.2, 3.0.3
>
> Attachments: HADOOP-14445-branch-2.8.002.patch, 
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch, 
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch, 
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch, 
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch, 
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, 
> HADOOP-14445.branch-2.000.precommit.patch, 
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch, 
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch, 
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch, 
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch, 
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch, 
> HADOOP-14445.branch-2.8.006.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do 
> not share delegation tokens. (a client uses KMS address/port as the key for 
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation 
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another 
> KMS instance, by checking the shared secret used to sign the delegation 
> token. To do this, all KMS instances must be able to retrieve the shared 
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share 
> delegation tokens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15431) KMSTokenRenewer should work with KMS_DELEGATION_TOKEN which has ip:port as service

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15431:
--

 Summary: KMSTokenRenewer should work with KMS_DELEGATION_TOKEN 
which has ip:port as service
 Key: HADOOP-15431
 URL: https://issues.apache.org/jira/browse/HADOOP-15431
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Affects Versions: 2.10.0, 2.8.4, 3.2.0, 3.1.1, 2.9.2, 3.0.3
Reporter: Xiao Chen
Assignee: Xiao Chen


Seen a test failure where a MR job failed to submit.
RM log has:
{noformat}
2018-04-30 15:00:17,864 WARN 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: 
Unable to add the application to the delegation token renewer.
java.lang.IllegalArgumentException: Invalid token service IP_ADDR:16000
at 
org.apache.hadoop.util.KMSUtil.createKeyProviderFromTokenService(KMSUtil.java:237)
at 
org.apache.hadoop.crypto.key.kms.KMSTokenRenewer.createKeyProvider(KMSTokenRenewer.java:100)
at 
org.apache.hadoop.crypto.key.kms.KMSTokenRenewer.renew(KMSTokenRenewer.java:57)
at org.apache.hadoop.security.token.Token.renew(Token.java:414)
at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:590)
at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:587)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:585)
at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:463)
at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:79)
at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:894)
at 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:871)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}

while client log has

{noformat}
18/04/30 15:53:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: 
job_1525128478242_0001
18/04/30 15:53:28 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, 
Service: ha-hdfs:ns1, Ident: (token for systest: HDFS_DELEGATION_TOKEN 
owner=syst...@example.com, renewer=yarn, realUser=, issueDate=1525128807236, 
maxDate=1525733607236, sequenceNumber=1038, masterKeyId=20)
18/04/30 15:53:28 INFO mapreduce.JobSubmitter: Kind: HBASE_AUTH_TOKEN, Service: 
621a942b-292f-493d-ba50-f9b783704359, Ident: 
(org.apache.hadoop.hbase.security.token.AuthenticationTokenIdentifier@0)
18/04/30 15:53:28 INFO mapreduce.JobSubmitter: Kind: KMS_DELEGATION_TOKEN, 
Service: IP_ADDR:16000, Ident: 00 07 73 79 73 74 65 73 74 04 79 61 72 6e 00 8a 
01 63 18 c2 c3 d5 8a 01 63 3c cf 47 d5 8e 01 ec 10
18/04/30 15:53:29 INFO mapreduce.JobSubmitter: Cleaning up the staging area 
/user/systest/.staging/job_1525128478242_0001
18/04/30 15:53:29 WARN security.UserGroupInformation: 
PriviledgedActionException as:syst...@example.com (auth:KERBEROS) 
cause:java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: 
Failed to submit application_1525128478242_0001 to YARN : Invalid token service 
IP_ADDR:16000
18/04/30 15:53:29 INFO client.ConnectionManager$HConnectionImplementation: 
Closing master protocol: MasterService
18/04/30 15:53:29 INFO client.ConnectionManager$HConnectionImplementation: 
Closing zookeeper sessionid=0x1630ba2d0001cb5
18/04/30 15:53:29 INFO zookeeper.ZooKeeper: Session: 0x1630ba2d0001cb5 closed
18/04/30 15:53:29 INFO zookeeper.ClientCnxn: EventThread shut down
18/04/30 15:53:29 ERROR util.AbstractHBaseTool: Error running command-line tool
java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to 
submit application_1525128478242_0001 to YARN : Invalid token service 
IP_ADDR:16000
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:336)
at 
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422

Re: [VOTE] Release Apache Hadoop 3.0.2 (RC1)

[Xiao Chen]

Thanks Eddy for the effort!

+1 (binding)

   - Downloaded src tarball and verified checksums
   - Built from src
   - Started a pseudo distributed hdfs cluster
   - Verified basic hdfs operations work
   - Sanity checked webui and logs

Best,
-Xiao


-Xiao

On Fri, Apr 20, 2018 at 1:44 AM, 俊平堵  wrote:

> Thanks Lei for the work!
>
> +1 (binding), base on following verification work:
> - built succeed from source
> - verified signature
> - deployed a pseudo cluster and run some simple MR jobs (PI, sleep,
> terasort, etc.)
> - checked HDFS/YARN daemons' UI
> - Tried some rolling upgrade related features: MR over DistributedCache, NM
> Restart with work preserving, etc.
>
> Thanks,
>
> Junping
>
> 2018-04-17 7:59 GMT+08:00 Lei Xu :
>
> > Hi, All
> >
> > I've created release candidate RC-1 for Apache Hadoop 3.0.2, to
> > address missing source jars in the maven repository in RC-0.
> >
> > Thanks Ajay Kumar for spotting the error.
> >
> > Please note: this is an amendment for Apache Hadoop 3.0.1 release to
> > fix shaded jars in apache maven repository. The codebase of 3.0.2
> > release is the same as 3.0.1.  New bug fixes will be included in
> > Apache Hadoop 3.0.3 instead.
> >
> > The release page is:
> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
> >
> > New RC is available at: http://home.apache.org/~lei/hadoop-3.0.2-RC1/
> >
> > The git tag is release-3.0.2-RC1, and the latest commit is
> > 5c141f7c0f24c12cb8704a6ccc1ff8ec991f41ee, which is the same as RC-0.
> >
> > The maven artifacts are available at:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1102/
> >
> > Please try the release, especially, *verify the maven artifacts*, and
> vote.
> >
> > The vote will run 5 days, ending 4/21/2018.
> >
> > Here is my +1.
> >
> > Best,
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>

[jira] [Resolved] (HADOOP-15401) ConcurrentModificationException on Subject.getPrivateCredentials in UGI constructor

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen resolved HADOOP-15401.

Resolution: Fixed

> ConcurrentModificationException on Subject.getPrivateCredentials in UGI 
> constructor
> ---
>
> Key: HADOOP-15401
> URL: https://issues.apache.org/jira/browse/HADOOP-15401
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 3.1.0, 3.0.3
>Reporter: Xiao Chen
>Priority: Major
>
> Seen a recent exception from KMS client provider as follows:
> {noformat}
> java.io.IOException: java.util.ConcurrentModificationException
> at 
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:488)
> at 
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776)
> at 
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287)
> at 
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283)
> at 
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123)
> at 
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:283)
> at 
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
> at 
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:927)
> at 
> org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:946)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:316)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:311)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:323)
> Caused by: java.util.ConcurrentModificationException
> at 
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070)
> at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399)
> at javax.security.auth.Subject$ClassSet.(Subject.java:1372)
> at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767)
> at 
> org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(KerberosUtil.java:267)
> at 
> org.apache.hadoop.security.UserGroupInformation.(UserGroupInformation.java:715)
> at 
> org.apache.hadoop.security.UserGroupInformation.(UserGroupInformation.java:701)
> at 
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:742)
> at 
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:141)
> at 
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
> at 
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333)
> at 
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:477)
> at 
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:472)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at 
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:471)
> ... 12 more
> {noformat}
> It looks like we have ran into a race modifying jdk Subject class' 
> privCredentials.
> Found [https://bugs.openjdk.java.net/browse/JDK-4892913] but that jira was 
> created before Hadoop
> [~daryn], any thoughts on this?
>  (We have not seen this in versions pre-3.0 yet, but it seems HADOOP-9747 
> would make 

[jira] [Created] (HADOOP-15401) ConcurrentModificationException on Subject.getPrivateCredentials in UGI constructor

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15401:
--

 Summary: ConcurrentModificationException on 
Subject.getPrivateCredentials in UGI constructor
 Key: HADOOP-15401
 URL: https://issues.apache.org/jira/browse/HADOOP-15401
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Xiao Chen


Seen a recent exception from KMS client provider as follows:

{noformat}
java.io.IOException: java.util.ConcurrentModificationException
at 
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:488)
at 
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776)
at 
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287)
at 
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283)
at 
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123)
at 
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:283)
at 
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
at 
org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:927)
at 
org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:946)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:316)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:311)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:323)
Caused by: java.util.ConcurrentModificationException
at 
java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
at java.util.LinkedList$ListItr.next(LinkedList.java:888)
at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070)
at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399)
at javax.security.auth.Subject$ClassSet.(Subject.java:1372)
at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767)
at 
org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(KerberosUtil.java:267)
at 
org.apache.hadoop.security.UserGroupInformation.(UserGroupInformation.java:715)
at 
org.apache.hadoop.security.UserGroupInformation.(UserGroupInformation.java:701)
at 
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:742)
at 
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:141)
at 
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
at 
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333)
at 
org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:477)
at 
org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:472)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at 
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:471)
... 12 more
{noformat}

It looks like we have ran into a race inside jdk's Subject class.

Found https://bugs.openjdk.java.net/browse/JDK-4892913 but that jira was 
created before Hadoop

[~daryn], any thoughts on this?
(With all due respect, we have not seen this in versions without HADOOP-9747 
yet)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15390) Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15390:
--

 Summary: Yarn RM logs flooded by DelegationTokenRenewer trying to 
renew KMS tokens
 Key: HADOOP-15390
 URL: https://issues.apache.org/jira/browse/HADOOP-15390
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Xiao Chen


When looking at a recent issue with [~rkanter] and [~yufeigu], we found that 
the RM log in a cluster was flooded by KMS token renewal errors below:
{noformat}
$ tail -9 hadoop-cmf-yarn-RESOURCEMANAGER.log
2018-04-11 11:34:09,367 WARN 
org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer: keyProvider 
null cannot renew dt.
2018-04-11 11:34:09,367 INFO 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: 
Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt 
owner=user, renewer=yarn, realUser=, issueDate=1522192283334, 
maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0; 
apps=[]], for []
2018-04-11 11:34:09,367 INFO 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: 
Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user, 
renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334, 
sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367 ms, 
appId = []
...
2018-04-11 11:34:09,367 WARN 
org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer: keyProvider 
null cannot renew dt.
2018-04-11 11:34:09,367 INFO 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: 
Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt 
owner=user, renewer=yarn, realUser=, issueDate=1522192283334, 
maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0; 
apps=[]], for []
2018-04-11 11:34:09,367 INFO 
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: 
Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user, 
renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334, 
sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367 ms, 
appId = []
{noformat}

Further inspection shows the KMS IP is from another cluster. The RM is before 
HADOOP-14445, so needs to read from config. The config rightfully doesn't have 
the other cluster's KMS configured.

Although HADOOP-14445 will make this a non-issue by creating the provider from 
token service, we should fix 2 things here:
- KMS token renewer should throw instead of return 0. Returning 0 when not able 
to renew shall be considered a bug in the renewer.
- Yarn RM's {{DelegationTokenRenewer}} service should validate the return and 
not go into this busy loop.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.2 (RC0)

[Xiao Chen]

Thanks Eddy for the effort!

+1 (binding)

   - Downloaded src tarball and verified checksums
   - Built from src
   - Started a pseudo distributed hdfs cluster
   - Verified basic hdfs operations work
   - Sanity checked logs / webui

Best,
-Xiao


On Mon, Apr 9, 2018 at 11:28 AM, Eric Payne 
wrote:

> Thanks a lot for working to produce this release.
>
> +1 (binding)
> Tested the following:
> - built from source and installed on 6-node pseudo-cluster
> - tested Capacity Scheduler FairOrderingPolicy and FifoOrderingPolicy to
> determine that capacity was assigned as expected in each case
> - tested user weights with FifoOrderingPolicy to ensure that weights were
> assigned to users as expected.
>
> Eric Payne
>
>
>
>
>
>
> On Friday, April 6, 2018, 1:17:10 PM CDT, Lei Xu  wrote:
>
>
>
>
>
> Hi, All
>
> I've created release candidate RC-0 for Apache Hadoop 3.0.2.
>
> Please note: this is an amendment for Apache Hadoop 3.0.1 release to
> fix shaded jars in apache maven repository. The codebase of 3.0.2
> release is the same as 3.0.1.  New bug fixes will be included in
> Apache Hadoop 3.0.3 instead.
>
> The release page is:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
>
> New RC is available at: http://home.apache.org/~lei/hadoop-3.0.2-RC0/
>
> The git tag is release-3.0.2-RC0, and the latest commit is
> 5c141f7c0f24c12cb8704a6ccc1ff8ec991f41ee
>
> The maven artifacts are available at
> https://repository.apache.org/content/repositories/orgapachehadoop-1096/
>
> Please try the release, especially, *verify the maven artifacts*, and vote.
>
> The vote will run 5 days, ending 4/11/2018.
>
> Thanks for everyone who helped to spot the error and proposed fixes!
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: Hdfs build on branch-2 are failing.

[Xiao Chen]

Similar to Haibo's link I found https://stackoverflow.com/a/22526046/1338884 to
be working.
Thrown a patch at HADOOP-15375
 to unblock branch-2.

I'm also not sure why this is passing for trunk.

-Xiao

On Thu, Apr 5, 2018 at 3:41 PM, Haibo Chen  wrote:

> Not sure why this did not show up in trunk. A quick google search takes me
> to bower-install-cert-untrusted-error
>  install-cert-untrusted-error>
>
> On Thu, Apr 5, 2018 at 10:40 AM, Vrushali C 
> wrote:
>
> > Seeing the same for branch-2 yarn patches as well.
> >
> > On Fri, Mar 30, 2018 at 12:54 PM, Rushabh Shah  >
> > wrote:
> >
> > > Hi All,
> > > Recently couple of my hdfs builds failed on branch-2.
> > > Slave id# H9 
> > > Builds that failed:
> > > https://builds.apache.org/job/PreCommit-HDFS-Build/23737/console
> > > https://builds.apache.org/job/PreCommit-HDFS-Build/23735/console
> > >
> > > It failed with the following error:
> > >
> > > npm http GET https://registry.npmjs.org/bowernpm http GET
> > > https://registry.npmjs.org/bowernpm http GET
> > > https://registry.npmjs.org/bowernpm ERR! Error: CERT_UNTRUSTED
> > > npm ERR! at SecurePair. (tls.js:1370:32)
> > > npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17)npm ERR!
> > > at SecurePair.maybeInitFinished (tls.js:982:10)npm ERR! at
> > > CleartextStream.read [as _read] (tls.js:469:13)
> > > npm ERR! at CleartextStream.Readable.read
> > > (_stream_readable.js:320:10)npm ERR! at EncryptedStream.write [as
> > > _write] (tls.js:366:25)npm ERR! at doWrite
> > > (_stream_writable.js:223:10)
> > > npm ERR! at writeOrBuffer (_stream_writable.js:213:5)npm ERR!
> > > at EncryptedStream.Writable.write (_stream_writable.js:180:11)
> > > npm ERR! at write (_stream_readable.js:583:24)npm ERR! If you need
> > > help, you may report this log at:
> > > npm ERR! 
> > > npm ERR! or email it to:
> > > npm ERR! npm ERR! System Linux
> > > 3.13.0-143-genericnpm ERR! command "/usr/bin/nodejs" "/usr/bin/npm"
> > > "install" "-g" "bower"
> > > npm ERR! cwd /rootnpm ERR! node -v v0.10.25npm ERR! npm -v 1.3.10npm
> ERR!
> > > npm ERR! Additional logging details can be found in:
> > > npm ERR! /root/npm-debug.log
> > > npm ERR! not ok code 0
> > >
> > >
> > >
> > > The certificate details on https://registry.npmjs.org/bower:
> > >
> > > Not valid before: Thursday, March 15, 2018 at 8:39:52 AM Central
> Daylight
> > > Time
> > > Not valid after: Saturday, June 13, 2020 at 2:06:17 PM Central Daylight
> > > Time
> > >
> > > Far from being an expert on ssl, do we need to change the truststore on
> > > slave also ?
> > >
> > > Appreciate if anyone can help fixing this.
> > >
> > >
> > > Thanks,
> > > Rushabh Shah.
> > >
> >
>

[jira] [Created] (HADOOP-15375) Branch-2 pre-commit failed to build docker image

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15375:
--

 Summary: Branch-2 pre-commit failed to build docker image
 Key: HADOOP-15375
 URL: https://issues.apache.org/jira/browse/HADOOP-15375
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.10.0
Reporter: Xiao Chen
Assignee: Xiao Chen


Branch-2 pre-commit is failing during {{Building base image}}:
{noformat}
...
Step 28/33 : RUN apt-get -y install nodejs && ln -s /usr/bin/nodejs 
/usr/bin/node && apt-get -y install npm && npm set ca null && npm 
install -g bower && npm install -g ember-cli
{noformat}

{noformat}
...
Setting up npm (1.3.10~dfsg-1) ...
npm http GET 
[https://registry.npmjs.org/bower]
npm http GET 
[https://registry.npmjs.org/bower]
npm http GET 
[https://registry.npmjs.org/bower]
npm ERR! Error: CERT_UNTRUSTED
npm ERR! at SecurePair. (tls.js:1370:32)
npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17)
npm ERR! at SecurePair.maybeInitFinished (tls.js:982:10)
npm ERR! at CleartextStream.read [as _read] (tls.js:469:13)
npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10)
npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25)
npm ERR! at doWrite (_stream_writable.js:223:10)
npm ERR! at writeOrBuffer (_stream_writable.js:213:5)
npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11)
npm ERR! at write (_stream_readable.js:583:24)
npm ERR! If you need help, you may report this log at:
npm ERR! <
[http://github.com/isaacs/npm/issues]
>
npm ERR! or email it to:
npm ERR! <n...@googlegroups.com>

npm ERR! System Linux 3.13.0-143-generic
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "bower"
npm ERR! cwd /root
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.3.10
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR! /root/npm-debug.log
npm ERR! not ok code 0
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15359) IPC client could run into JDK deadlock

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15359:
--

 Summary: IPC client could run into JDK deadlock
 Key: HADOOP-15359
 URL: https://issues.apache.org/jira/browse/HADOOP-15359
 Project: Hadoop Common
  Issue Type: Bug
  Components: ipc
Affects Versions: 3.0.0, 2.8.0
Reporter: Xiao Chen


In a recent internal testing, we have found a DFS client hang. Further 
inspecting jstack shows the following:

{noformat}
"IPC Client (552936351) connection toHOSTNAME:8020 from PRINCIPAL" #7468 daemon 
prio=5 os_prio=0 tid=0x7f6bb306c000 nid=0x1c76e waiting for monitor entry 
[0x7f6bc2bd6000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at java.security.Provider.getService(Provider.java:1035)
- waiting to lock <0x80277040> (a sun.security.provider.Sun)
at 
sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:444)
at 
sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:376)
at 
sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:486)
at javax.crypto.Cipher.getInstance(Cipher.java:513)
at 
sun.security.krb5.internal.crypto.dk.Des3DkCrypto.getCipher(Des3DkCrypto.java:202)
at sun.security.krb5.internal.crypto.dk.DkCrypto.dr(DkCrypto.java:484)
at sun.security.krb5.internal.crypto.dk.DkCrypto.dk(DkCrypto.java:447)
at 
sun.security.krb5.internal.crypto.dk.DkCrypto.calculateChecksum(DkCrypto.java:413)
at 
sun.security.krb5.internal.crypto.Des3.calculateChecksum(Des3.java:59)
at 
sun.security.jgss.krb5.CipherHelper.calculateChecksum(CipherHelper.java:231)
at 
sun.security.jgss.krb5.MessageToken.getChecksum(MessageToken.java:466)
at 
sun.security.jgss.krb5.MessageToken.verifySignAndSeqNumber(MessageToken.java:374)
at 
sun.security.jgss.krb5.WrapToken.getDataFromBuffer(WrapToken.java:284)
at sun.security.jgss.krb5.WrapToken.getData(WrapToken.java:209)
at sun.security.jgss.krb5.WrapToken.getData(WrapToken.java:182)
at sun.security.jgss.krb5.Krb5Context.unwrap(Krb5Context.java:1053)
at sun.security.jgss.GSSContextImpl.unwrap(GSSContextImpl.java:403)
at com.sun.security.sasl.gsskerb.GssKrb5Base.unwrap(GssKrb5Base.java:77)
at 
org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.readNextRpcPacket(SaslRpcClient.java:617)
at 
org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.read(SaslRpcClient.java:583)
- locked <0x83444878> (a java.nio.HeapByteBuffer)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at 
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:553)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x834448c0> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at 
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1113)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1006)
{noformat}

and at the end of jstack:
{noformat}
Found one Java-level deadlock:
=
"IPC Parameter Sending Thread #29":
  waiting to lock monitor 0x17ff49f8 (object 0x80277040, a 
sun.security.provider.Sun),
  which is held by UNKNOWN_owner_addr=0x50607000

Java stack information for the threads listed above:
===
"IPC Parameter Sending Thread #29":
at java.security.Provider.getService(Provider.java:1035)
- waiting to lock <0x80277040> (a sun.security.provider.Sun)
at 
sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:437)
at 
sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:376)
at 
sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:486)
at javax.crypto.SecretKeyFactory.nextSpi(SecretKeyFactory.java:293)
- locked <0x834386b8> (a java.lang.Object)
at javax.crypto.SecretKeyFactory.(SecretKeyFactory.java:121)
at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:160)
at 
sun.security.krb5.internal.crypto.dk.Des3DkCrypto.getCipher(Des3DkCrypto.java:187)
at sun.security.krb5.internal.crypto.dk.DkCrypto.dr(DkCrypto.java:484)
at sun.security.krb5.internal.crypto.dk.DkCrypto.dk(DkCrypto.java:447)
at 
sun.security.krb5.internal.crypto.dk.DkCrypto.calculateChecksum(DkCrypto.java:413)
at 
sun.security.krb5.internal.crypto.Des3.calculateChecksum(Des3.java:59)
at 
sun.security.jgss.krb5.CipherHelper.calculateChecksum(CipherHelper.java:231)
at 
sun.secur

[jira] [Created] (HADOOP-15344) LoadBalancingKMSClientProvider#close should not swallow exceptions

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15344:
--

 Summary: LoadBalancingKMSClientProvider#close should not swallow 
exceptions
 Key: HADOOP-15344
 URL: https://issues.apache.org/jira/browse/HADOOP-15344
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Reporter: Xiao Chen


As [~shahrs87]'s comment on HADOOP-14445 says:
{quote}
LoadBalancingKMSCP never throws IOException back. It just swallows all the 
IOException and just logs it.
...
Maybe we might want to return MultipleIOException from 
LoadBalancingKMSCP#close. 
{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC1)

[Xiao Chen]

Thanks Eddy for the effort!

+1 (binding)

   - Downloaded src tarball and verified checksums
   - Built from src
   - Started a pseudo distributed hdfs cluster
   - Verified basic hdfs operations work
   - Verified hdfs encryption basic operations work
   - Sanity checked logs / webui


-Xiao

On Mon, Mar 19, 2018 at 10:45 AM, Lei Xu  wrote:

> Sure, Akira
>
> .mds files are uploaded to http://home.apache.org/~lei/hadoop-3.0.1-RC1/
>
> On Sun, Mar 18, 2018 at 6:04 PM, Akira Ajisaka
>  wrote:
> > Hi Lei,
> >
> > Would you provide SHA checksum files instead of MD5?
> > http://www.apache.org/dev/release-distribution#sigs-and-sums
> >
> > -Akira
> >
> >
> > On 2018/03/18 13:11, Lei Xu wrote:
> >>
> >> Hi, all
> >>
> >> I've created release candidate RC-1 for Apache Hadoop 3.0.1
> >>
> >> Apache Hadoop 3.0.1 will be the first bug fix release for Apache
> >> Hadoop 3.0 release. It includes 49 bug fixes and security fixes, which
> >> include 12
> >> blockers and 17 are critical.
> >>
> >> Please note:
> >> * HDFS-12990. Change default NameNode RPC port back to 8020. It makes
> >> incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
> >> Hadoop 3.0.0 will be deprecated due to this change.
> >>
> >> The release page is:
> >> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
> >>
> >> New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/
> >>
> >> The git tag is release-3.0.1-RC1, and the latest commit is
> >> 496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0
> >>
> >> The maven artifacts are available at:
> >> https://repository.apache.org/content/repositories/
> orgapachehadoop-1081/
> >>
> >> Please try the release and vote; the vote will run for the usual 5
> >> days, ending on 3/22/2017 6pm PST time.
> >>
> >> Thanks!
> >>
> >> -
> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>
> >
>
>
>
> --
> Lei (Eddy) Xu
> Software Engineer, Cloudera
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HADOOP-15321) Reduce the RPC Client max retries on timeouts

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15321:
--

 Summary: Reduce the RPC Client max retries on timeouts
 Key: HADOOP-15321
 URL: https://issues.apache.org/jira/browse/HADOOP-15321
 Project: Hadoop Common
  Issue Type: Improvement
  Components: ipc
Reporter: Xiao Chen
Assignee: Xiao Chen


Currently, the 
[default|https://github.com/apache/hadoop/blob/branch-3.0.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java#L379]
 number of retries when IPC client catch a {{ConnectTimeoutException}} is 45. 
This seems unreasonably high.

Given the IPC client timeout is by default 60 seconds, if a DN host is shutdown 
the client will retry for 45 minutes until aborting. (If host is there but 
process down, it would throw a connection refused immediately, which is cool)

Creating this Jira to discuss whether we can reduce that to a reasonable number.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15317) Improve NetworkTopology chooseRandom's loop

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15317:
--

 Summary: Improve NetworkTopology chooseRandom's loop
 Key: HADOOP-15317
 URL: https://issues.apache.org/jira/browse/HADOOP-15317
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Xiao Chen
Assignee: Xiao Chen


Recently we found a postmortem case where the ANN seems to be in an infinite 
loop. From the logs it seems it just went through a rolling restart, and DNs 
are getting registered.

Later the NN become unresponsive, and from the stacktrace it's inside a 
do-while loop inside {{NetworkTopology#chooseRandom}} - part of what's done in 
HDFS-10320.

Going through the code and logs I'm not able to come up with any theory 
(thought about incorrect locking, or the Node object being modified outside of 
NetworkTopology, both seem impossible) why this is happening, but we should 
eliminate this loop.

stacktrace:
{noformat}
 Stack:
java.util.HashMap.hash(HashMap.java:338)
java.util.HashMap.containsKey(HashMap.java:595)
java.util.HashSet.contains(HashSet.java:203)
org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:786)
org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:732)
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseDataNode(BlockPlacementPolicyDefault.java:757)
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:692)
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:666)
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:573)
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:461)
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:368)
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:243)
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:115)
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4AdditionalDatanode(BlockManager.java:1596)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:3599)
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:717)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15313) TestKMS should close providers

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15313:
--

 Summary: TestKMS should close providers
 Key: HADOOP-15313
 URL: https://issues.apache.org/jira/browse/HADOOP-15313
 Project: Hadoop Common
  Issue Type: Test
  Components: kms, test
Reporter: Xiao Chen
Assignee: Xiao Chen


During the review of HADOOP-14445, [~jojochuang] found that we key providers 
are not closed in tests. Details in [this 
comment|https://issues.apache.org/jira/browse/HADOOP-14445?focusedCommentId=16397824=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16397824].

We should investigate and handle that in all related tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15283) branch-2.8 pre-commit failed to build docker image

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15283:
--

 Summary: branch-2.8 pre-commit failed to build docker image
 Key: HADOOP-15283
 URL: https://issues.apache.org/jira/browse/HADOOP-15283
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Xiao Chen


Not sure about other branches, but branch-2.8 pre-commit is failing at building 
docker images.

See [https://builds.apache.org/job/PreCommit-HDFS-Build/23274]
{noformat}
...
Step 13/31 : RUN mkdir -p /opt/findbugs && curl -L -s -S  
https://sourceforge.net/projects/findbugs/files/findbugs/3.0.1/findbugs-noUpdateChecks-3.0.1.tar.gz/download
  -o /opt/findbugs.tar.gz && tar xzf /opt/findbugs.tar.gz 
--strip-components 1 -C /opt/findbugs
 ---> Running in 59d3581d6f6c

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/sh -c mkdir -p /opt/findbugs && curl -L -s -S  
https://sourceforge.net/projects/findbugs/files/findbugs/3.0.1/findbugs-noUpdateChecks-3.0.1.tar.gz/download
  -o /opt/findbugs.tar.gz && tar xzf /opt/findbugs.tar.gz 
--strip-components 1 -C /opt/findbugs' returned a non-zero code: 2

Total Elapsed time:  11m 54s

ERROR: Docker failed to build image.
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC0)

[Xiao Chen]

Thanks Eddy for driving this!

+1 (binding)

   - Downloaded src tarball and verified md5
   - Built from src
   - Started a pseudo distributed cluster
   - Verified basic hdfs operations work
   - Verified hdfs encryption basic operations work
   - Sanity checked logs

The wiki release page seems to have less items than jira query (p1/p2,
fixed, 3.0.1) though...




-Xiao

On Thu, Feb 15, 2018 at 3:36 PM, Lei Xu  wrote:

> Hi, all
>
> I've created release candidate 0 for Apache Hadoop 3.0.1
>
> Apache Hadoop 3.0.1 will be the first bug fix release for Apache
> Hadoop 3.0 release. It includes 49 bug fixes, which include 10
> blockers and 8 are critical.
>
> Please note:
> * HDFS-12990. Change default NameNode RPC port back to 8020. It makes
> incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
> Hadoop 3.0.0 will be deprecated due to this change.
>
> The release page is:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
>
> New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC0/
>
> The git tag is release-3.0.1-RC0, and the latest commit is
> 494d075055b52b0cc922bc25237e231bb3771c90
>
> The maven artifacts are available:
> https://repository.apache.org/content/repositories/orgapachehadoop-1078/
>
> Please try the release and vote; the vote will run for the usual 5
> days, ending on 2/20/2017 6pm PST time.
>
> Thanks!
>
> --
> Lei (Eddy) Xu
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

[jira] [Reopened] (HADOOP-12897) KerberosAuthenticator.authenticate to include URL on IO failures

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen reopened HADOOP-12897:


> KerberosAuthenticator.authenticate to include URL on IO failures
> 
>
> Key: HADOOP-12897
> URL: https://issues.apache.org/jira/browse/HADOOP-12897
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Ajay Kumar
>Priority: Minor
> Attachments: HADOOP-12897.001.patch, HADOOP-12897.002.patch, 
> HADOOP-12897.003.patch, HADOOP-12897.004.patch, HADOOP-12897.005.patch
>
>
> If {{KerberosAuthenticator.authenticate}} can't connect to the endpoint, you 
> get a stack trace, but without the URL it is trying to talk to.
> That is: it doesn't have any equivalent of the {{NetUtils.wrapException}} 
> handler —which can't be called here as its not in the {{hadoop-auth}} module



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: hadoop common tests failiing

[Xiao Chen]

https://issues.apache.org/jira/browse/HADOOP-12897,  commenting there...

-Xiao

On Wed, Feb 14, 2018 at 8:58 AM, Steve Loughran 
wrote:

> When did a set of tests related to security and exception messages start
> failing?
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/14118/testReport/
>

[jira] [Created] (HADOOP-15234) NPE when initializing KMSWebApp

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15234:
--

 Summary: NPE when initializing KMSWebApp
 Key: HADOOP-15234
 URL: https://issues.apache.org/jira/browse/HADOOP-15234
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Reporter: Xiao Chen


During KMS startup, if the {{keyProvider}} is null, it will NPE inside 
KeyProviderExtension.
{noformat}
java.lang.NullPointerException
at 
org.apache.hadoop.crypto.key.KeyProviderExtension.(KeyProviderExtension.java:43)
at 
org.apache.hadoop.crypto.key.CachingKeyProvider.(CachingKeyProvider.java:93)
at 
org.apache.hadoop.crypto.key.kms.server.KMSWebApp.contextInitialized(KMSWebApp.java:170)
{noformat}

We're investigating the exact scenario that could lead to this, but the NPE and 
log around it can be improved.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: When are incompatible changes acceptable (HDFS-12990)

[Xiao Chen]

Thanks all for the comments, and ATM for initiating the discussion thread.
(I have just returned from a 2-week PTO).

Reading up all the comments here and from HDFS-12990, I think we all agree
having different default NN ports will be inconvenient for all, and
problematic for several cases - ranging from rolling upgrade to various
downstream use cases. In CDH, this was initially reported from downstream
(Impala) testing when the scripts there tries to do RPC with 8020 but NN is
running on 9820. The intuitive was 'change CM to match it'. Later cases pop
up, including the table location in Hive metastore and custom scripts
(including Oozie WFs). The only other real world example we heard so far is
Anu's comment on HDFS-12990, where he did not enjoy keeping separate
scripts for hadoop 2 / 3.

Note that this limits only to NN RPC port (8020 <-> 9820), because other
port changes in HDFS-9427 are indeed switching the default from ephemeral
ports.

The disagreement so far is how to proceed from here.
1. Not fix it at all.

This means everyone on 2.x will run into this issue when they upgrade.

2. Make NN RPC listen to both 8020 and 9820

Nicholas came up with this idea, which by itself smartly solves the
compatibility problems.

The downside of it is, even though things works during/after an upgrade,
people will still have to whack-a-mole their existing 8020's. I agree
adding this will have the side-effect to give NN more flexibility in the
future. We can do this with or without the port change.

3. Change back to 8020

This will make all upgrades from 2.x -> 3.0.1 (if this goes in) free of
this problem, because the original 8020->9820 switch doesn't appear to be a
mature move.

Downside that I summarize up are: a) what about 3.0.0 users b) compat

For a), since we have *just* released 3.0.0, it's safe to say we have
tremendously more users on 2.x than 3.0.0 now. If we make the release notes
clear, this will benefit tremendously more users than harming.
For b), as various others commented, this can be a special case where a
by-definition incompatible change actually fixes a previously problematic
incompatible change. If we can have consensus, and also notify users from
mailing list + release notes, it doesn't weaken our compatibility
guidelines nor surprise the community.


Eric and Nicholas, does this address your concerns?


-Xiao

On Sun, Jan 21, 2018 at 8:27 PM, Akira Ajisaka  wrote:

> Thanks Chris and Daryn for the replies.
>
> First of all, I missed why NN RPC port was moved to 9820.
> HDFS-9427 is to avoid ephemeral port, however, NN RPC port (8020)
> is already out of the range. The change is only to move the
> all ports in the same range, so the change is not really necessary.
>
> I agree the change is disastrous for many users, however,
> reverting the change is also disastrous for 3.0.0 users.
> Therefore, if we are to revert the change, we must notify
> the incompatibility to the users. Adding the notification in the
> release announcement seems to be a good choice. Probably
> users does not carefully read the change logs and they can
> easily miss it, as we missed it in the release process.
>
> Cancelling my -1.
>
> -Akira
>
> On 2018/01/20 7:17, Daryn Sharp wrote:
>
>>  > I'm -1 for reverting HDFS-9427 in 3.x.
>>
>> I'm -1 on not reverting.  If yahoo/oath had the cycles to begin testing
>> 3.0 prior to release, I would have -1'ed this change immediately.  It's
>> already broken our QE testing pipeline.
>>
>>  > The port number is configurable, so if you want to use 8020 for NN RPC
>> port in Hadoop 3.x, you configure this to 8020.
>>
>> No, it's not that easy.  THE DEFAULT IS HARDCODED.  You can only
>> "configure" the port via hardcoding it into all paths.   Which ironically
>> multiple people think shouldn't be done?  Let's starting thinking about the
>> impact to those not running just 1 isolated cluster with fully managed
>> services that can take downtime and be fully upgraded in sync.
>>
>> If the community doesn't revert, I'm not going to tell users to put the
>> port in all their paths.  I'll hack the default back to 8020.  Then I'll
>> have to deal with other users or closed software stacks bundled with a
>> stock 3.0 hadoop client, or using a different 3.0 distro, using the wrong
>> port.   They will break unless they hardcode 8020 port into paths.
>>
>> Let's say I do change to the new port, I still have to tell all my users
>> with 2.x client to hardcode the new port but only after the upgrade.  If
>> the "solution" is listening on the old and new port, it only proves that a
>> port change is frivolous with zero added value.
>>
>> Someone please explain to me how any heterogenous multi-cluster
>> environment benefits from this change?  How does a single cluster
>> environment benefit from this change?  If there are no benefits to anyone,
>> why are we even debating a revert?  Taking a hardline, under the guise of
>> worrying about compatibility for tiny number of 

[jira] [Created] (HADOOP-15154) Abstract new method assertCapability for StreamCapabilities testing

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15154:
--

 Summary: Abstract new method assertCapability for 
StreamCapabilities testing
 Key: HADOOP-15154
 URL: https://issues.apache.org/jira/browse/HADOOP-15154
 Project: Hadoop Common
  Issue Type: Improvement
  Components: test
Reporter: Xiao Chen
Priority: Minor


>From Steve's 
>[comment|https://issues.apache.org/jira/browse/HADOOP-15149?focusedCommentId=16306806=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16306806]:
bq.  it'd have been cleaner for the asserts to have been one in a 
assertCapability(key, StreamCapabilities subject, bool outcome) and had it 
throw meaningful exceptions on a failure

We can consider abstract such a method to a test util class and use it for all 
{{StreamCapabilities}} tests as needed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Xiao Chen]

+1 (binding)

- downloaded src tarball, verified md5
- built from source with jdk1.8.0_112
- started a pseudo cluster with hdfs and kms
- sanity checked encryption related operations working
- sanity checked webui and logs.

-Xiao

On Mon, Dec 11, 2017 at 6:10 PM, Aaron T. Myers  wrote:

> +1 (binding)
>
> - downloaded the src tarball and built the source (-Pdist -Pnative)
> - verified the checksum
> - brought up a secure pseudo distributed cluster
> - did some basic file system operations (mkdir, list, put, cat) and
> confirmed that everything was working
> - confirmed that the web UI worked
>
> Best,
> Aaron
>
> On Fri, Dec 8, 2017 at 12:31 PM, Andrew Wang 
> wrote:
>
> > Hi all,
> >
> > Let me start, as always, by thanking the efforts of all the contributors
> > who contributed to this release, especially those who jumped on the
> issues
> > found in RC0.
> >
> > I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates 302
> > fixed JIRAs since the previous 3.0.0-beta1 release.
> >
> > You can find the artifacts here:
> >
> > http://home.apache.org/~wang/3.0.0-RC1/
> >
> > I've done the traditional testing of building from the source tarball and
> > running a Pi job on a single node cluster. I also verified that the
> shaded
> > jars are not empty.
> >
> > Found one issue that create-release (probably due to the mvn deploy
> change)
> > didn't sign the artifacts, but I fixed that by calling mvn one more time.
> > Available here:
> >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1075/
> >
> > This release will run the standard 5 days, closing on Dec 13th at 12:31pm
> > Pacific. My +1 to start.
> >
> > Best,
> > Andrew
> >
>

[jira] [Resolved] (HADOOP-15100) Configuration#Resource constructor change broke Hive tests

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen resolved HADOOP-15100.

Resolution: Won't Fix

After some discussion with Aihua, he will fix the hive tests to always start 
the minihivekdc before initializing {{HiveConf}}.

> Configuration#Resource constructor change broke Hive tests
> --
>
> Key: HADOOP-15100
> URL: https://issues.apache.org/jira/browse/HADOOP-15100
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.8.3, 2.7.5, 3.0.0, 2.9.1
>Reporter: Xiao Chen
>Priority: Critical
>
> In CDH's C6 rebased testing, the following Hive tests started failing:
> {noformat}
> org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie
> org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie
> org.apache.hive.minikdc.TestHiveAuthFactory.org.apache.hive.minikdc.TestHiveAuthFactory
> org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthHttp.org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthHttp
> org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthHttp.org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthHttp
> org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs
> org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs
> org.apache.hive.minikdc.TestJdbcWithMiniKdc.org.apache.hive.minikdc.TestJdbcWithMiniKdc
> org.apache.hive.minikdc.TestJdbcWithMiniKdc.org.apache.hive.minikdc.TestJdbcWithMiniKdc
> org.apache.hive.minikdc.TestHs2HooksWithMiniKdc.org.apache.hive.minikdc.TestHs2HooksWithMiniKdc
> org.apache.hive.minikdc.TestHs2HooksWithMiniKdc.org.apache.hive.minikdc.TestHs2HooksWithMiniKdc
> org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc
> org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc
> org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthBinary.org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthBinary
> org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthBinary.org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthBinary
> org.apache.hive.minikdc.TestMiniHiveKdc.testLogin
> org.apache.hive.minikdc.TestMiniHiveKdc.testLogin
> org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
> org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
> org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testQueryWithInWithFallbackToORM
> org.apache.hive.jdbc.TestJdbcWithMiniHS2.testSelectThriftSerializeInTasks
> org.apache.hive.jdbc.TestJdbcWithMiniHS2.testEmptyResultsetThriftSerializeInTasks
> org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation2
> org.apache.hive.jdbc.TestJdbcWithMiniHS2.testJoinThriftSerializeInTasks
> org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation
> org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConcurrentStatements
> org.apache.hive.jdbc.TestJdbcWithMiniHS2.testFloatCast2DoubleThriftSerializeInTasks
> org.apache.hive.jdbc.TestJdbcWithMiniHS2.testEnableThriftSerializeInTasks
> org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementParallel
> {noformat}
> The exception is
> {noformat}
> java.lang.ExceptionInInitializerError: null
>   at sun.security.krb5.Config.getRealmFromDNS(Config.java:1102)
>   at sun.security.krb5.Config.getDefaultRealm(Config.java:987)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at 
> org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:110)
>   at 
> org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
>   at 
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:332)
>   at 
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:317)
>   at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:873)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getCurrent

[jira] [Created] (HADOOP-15100) Configuration#Resource constructor change broke Hive tests

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15100:
--

 Summary: Configuration#Resource constructor change broke Hive tests
 Key: HADOOP-15100
 URL: https://issues.apache.org/jira/browse/HADOOP-15100
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.8.3, 2.7.5, 3.0.0, 2.9.1
Reporter: Xiao Chen
Priority: Critical


In CDH's C6 rebased testing, the following Hive tests started failing:
{noformat}
org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie
org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie
org.apache.hive.minikdc.TestHiveAuthFactory.org.apache.hive.minikdc.TestHiveAuthFactory
org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthHttp.org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthHttp
org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthHttp.org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthHttp
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs
org.apache.hive.minikdc.TestJdbcWithMiniKdc.org.apache.hive.minikdc.TestJdbcWithMiniKdc
org.apache.hive.minikdc.TestJdbcWithMiniKdc.org.apache.hive.minikdc.TestJdbcWithMiniKdc
org.apache.hive.minikdc.TestHs2HooksWithMiniKdc.org.apache.hive.minikdc.TestHs2HooksWithMiniKdc
org.apache.hive.minikdc.TestHs2HooksWithMiniKdc.org.apache.hive.minikdc.TestHs2HooksWithMiniKdc
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc
org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthBinary.org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthBinary
org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthBinary.org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthBinary
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testQueryWithInWithFallbackToORM
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testSelectThriftSerializeInTasks
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testEmptyResultsetThriftSerializeInTasks
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation2
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testJoinThriftSerializeInTasks
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConcurrentStatements
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testFloatCast2DoubleThriftSerializeInTasks
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testEnableThriftSerializeInTasks
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementParallel
{noformat}

The exception is
{noformat}
java.lang.ExceptionInInitializerError: null
at sun.security.krb5.Config.getRealmFromDNS(Config.java:1102)
at sun.security.krb5.Config.getDefaultRealm(Config.java:987)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:110)
at 
org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
at 
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:332)
at 
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:317)
at 
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
at 
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:873)
at 
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:740)
at 
org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:261)
at 
org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:229)
at 
org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:221)
at 
org.apache.hadoop.conf.Configuration.addResource(Configuration.java:916)
at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:3864)
at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:3816)
at 
org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie

[jira] [Reopened] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen reopened HADOOP-15012:


> Add readahead, dropbehind, and unbuffer to StreamCapabilities
> -
>
> Key: HADOOP-15012
> URL: https://issues.apache.org/jira/browse/HADOOP-15012
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 3.1.0
>
> Attachments: HADOOP-15012.branch-2.01.patch
>
>
> A split from HADOOP-14872 to track changes that enhance StreamCapabilities 
> class with READAHEAD, DROPBEHIND, and UNBUFFER capability.
> Discussions and code reviews are done in HADOOP-14872.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Xiao Chen]

+1 (binding)

Thanks Andrew!


   - Verified md5 and built from source
   - Started a pseudo distributed cluster with KMS,
   - Performed basic hdfs operations plus encryption related operations
   - Verified logs and webui
   - Confidence from CDH testings (will let Andrew answer officially, but
   we have smokes and nightlies for various downstream projects)


-Xiao

On Mon, Nov 20, 2017 at 3:38 PM, Shane Kumpf 
wrote:

> Thanks, Andrew!
>
> +1 (non-binding)
>
> - Verified checksums and signatures
> - Deployed a single node cluster on CentOS 7.4 using the binary and source
> release
> - Ran hdfs commands
> - Ran pi and distributed shell using the default and docker runtimes
> - Verified the UIs
> - Verified the change log
>
> -Shane
>
>
> On Tue, Nov 14, 2017 at 2:34 PM, Andrew Wang 
> wrote:
>
> > Hi folks,
> >
> > Thanks as always to the many, many contributors who helped with this
> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> > available here:
> >
> > http://people.apache.org/~wang/3.0.0-RC0/
> >
> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> >
> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> > additions include the merge of YARN resource types, API-based
> configuration
> > of the CapacityScheduler, and HDFS router-based federation.
> >
> > I've done my traditional testing with a pseudo cluster and a Pi job. My
> +1
> > to start.
> >
> > Best,
> > Andrew
> >
>

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC2)

[Xiao Chen]

I have no issues now that RC3 is rolling. Thanks for the quick actions.

My point is I'm not sure that's even required. MIT licenses standard format
has that copyright line https://en.wikipedia.org/wiki/MIT_License, and
combined with the licensing-howto I referred earlier, HADOOP-15036 is not
necessary. Though IANAL.

-Xiao

On Mon, Nov 13, 2017 at 3:27 PM, Arun Suresh <arun.sur...@gmail.com> wrote:

> Cancelling the RC2 to fix the LICENSE.txt issue.
> Will be rolling out an RC3 shortly. Given that the delta is just the
> LICENSE fix, we will be carrying over the votes from this thread as well.
>
> Cheers
> -Arun/Subru
>
> On Mon, Nov 13, 2017 at 2:59 PM, Arun Suresh <asur...@apache.org> wrote:
>
>> Hi Xiao,
>> What Anu pointed out was that the copyright was missing in the LICENCE
>> not notice. Don't think we need to file LEGAL, since what we are doing is
>> similar to the other MIT dependencies.
>>
>> Cheers
>> -Arun
>>
>>
>>
>> On Mon, Nov 13, 2017 at 2:53 PM, Xiao Chen <x...@cloudera.com> wrote:
>>
>>> Thanks guys for pulling the RC and prompt reviews.
>>>
>>> From previous L discussions
>>> <https://issues.apache.org/jira/browse/HADOOP-12893?focusedC
>>> ommentId=15284739=com.atlassian.jira.plugin.system.issu
>>> etabpanels:comment-tabpanel#comment-15284739>
>>>  and http://www.apache.org/dev/licensing-howto.html#permissive-deps, my
>>> impression was BSD / MIT doesn't require a Notice. (Usually the copyright
>>> parts go to the notice). For this case, the trickiness comes from the
>>> License itself has this added line of copyright...
>>>
>>> Should we file a LEGAL jira to confirm first? There may (or may not) be
>>> other similar instances.
>>>
>>> -Xiao
>>>
>>> On Mon, Nov 13, 2017 at 2:34 PM, Arun Suresh <asur...@apache.org> wrote:
>>>
>>> > Thanks for notifying Anu,
>>> > We've raised https://issues.apache.org/jira/browse/HADOOP-15036 to
>>> address
>>> > this and have included a patch.
>>> > Can you please take a look and possibly +1 if you are fine ?
>>> >
>>> > Cheers
>>> > -Arun
>>> >
>>> > On Mon, Nov 13, 2017 at 2:19 PM, Anu Engineer <
>>> aengin...@hortonworks.com>
>>> > wrote:
>>> >
>>> > > -1 (binding)
>>> > >
>>> > > Thank you for all the hard work on 2.9 series. Unfortunately, this
>>> is one
>>> > > of the times I have to -1 this release.
>>> > >
>>> > > Looks like HADOOP-14840 added a dependency on “oj! Algorithms -
>>> version
>>> > > 43.0”, but we have just added “oj! Algorithms - version 43.0” to the
>>> > > “LICENSE.txt”. The right addition to the LICENESE.txt should contain
>>> the
>>> > > original MIT License, especially “Copyright (c) 2003-2017
>>> Optimatika”.
>>> > >
>>> > > Please take a look at https://github.com/optimatika/
>>> > > ojAlgo/blob/master/LICENSE
>>> > >
>>> > > I am a +1 after this is fixed.
>>> > >
>>> > > Thanks
>>> > > Anu
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > On 11/13/17, 9:50 AM, "Sunil G" <sun...@apache.org> wrote:
>>> > >
>>> > > +1 (binding)
>>> > >
>>> > > Deployed cluster built from source.
>>> > >
>>> > >
>>> > >
>>> > >- Tested few cases in an HA cluster and tried to do failover
>>> by
>>> > > using
>>> > >rmadmin commands etc. This seems works fine including
>>> submitting
>>> > > apps.
>>> > >- I also tested many MR apps and all are running fine w/o any
>>> > > issues.
>>> > >- Majorly tested below feature sanity too (works fine)
>>> > >   - Application priority
>>> > >   - Application timeout
>>> > >- Tested basic NodeLabel scenarios.
>>> > >   - Added some labels to couple of nodes
>>> > >   - Verified old UI for labels
>>> > >   - Submitted apps to labelled cluster and it works fine.
>>> > >   - Also performed few cli commands related to nodelabel
>>> > >- Verified new YARN UI and 

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC2)

[Xiao Chen]

Thanks guys for pulling the RC and prompt reviews.

>From previous L discussions

 and http://www.apache.org/dev/licensing-howto.html#permissive-deps, my
impression was BSD / MIT doesn't require a Notice. (Usually the copyright
parts go to the notice). For this case, the trickiness comes from the
License itself has this added line of copyright...

Should we file a LEGAL jira to confirm first? There may (or may not) be
other similar instances.

-Xiao

On Mon, Nov 13, 2017 at 2:34 PM, Arun Suresh  wrote:

> Thanks for notifying Anu,
> We've raised https://issues.apache.org/jira/browse/HADOOP-15036 to address
> this and have included a patch.
> Can you please take a look and possibly +1 if you are fine ?
>
> Cheers
> -Arun
>
> On Mon, Nov 13, 2017 at 2:19 PM, Anu Engineer 
> wrote:
>
> > -1 (binding)
> >
> > Thank you for all the hard work on 2.9 series. Unfortunately, this is one
> > of the times I have to -1 this release.
> >
> > Looks like HADOOP-14840 added a dependency on “oj! Algorithms - version
> > 43.0”, but we have just added “oj! Algorithms - version 43.0” to the
> > “LICENSE.txt”. The right addition to the LICENESE.txt should contain the
> > original MIT License, especially “Copyright (c) 2003-2017 Optimatika”.
> >
> > Please take a look at https://github.com/optimatika/
> > ojAlgo/blob/master/LICENSE
> >
> > I am a +1 after this is fixed.
> >
> > Thanks
> > Anu
> >
> >
> >
> >
> > On 11/13/17, 9:50 AM, "Sunil G"  wrote:
> >
> > +1 (binding)
> >
> > Deployed cluster built from source.
> >
> >
> >
> >- Tested few cases in an HA cluster and tried to do failover by
> > using
> >rmadmin commands etc. This seems works fine including submitting
> > apps.
> >- I also tested many MR apps and all are running fine w/o any
> > issues.
> >- Majorly tested below feature sanity too (works fine)
> >   - Application priority
> >   - Application timeout
> >- Tested basic NodeLabel scenarios.
> >   - Added some labels to couple of nodes
> >   - Verified old UI for labels
> >   - Submitted apps to labelled cluster and it works fine.
> >   - Also performed few cli commands related to nodelabel
> >- Verified new YARN UI and accessed various pages when cluster was
> > in
> >use. It seems fine to me.
> >
> >
> > Thanks all folks who participated in this release, appreciate the
> same!
> >
> > - Sunil
> >
> >
> > On Mon, Nov 13, 2017 at 3:01 AM Subru Krishnan 
> > wrote:
> >
> > > Hi Folks,
> > >
> > > Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and
> will
> > be the
> > > starting release for Apache Hadoop 2.9.x line - it includes 30 New
> > Features
> > > with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed
> issues
> > since
> > > 2.8.2.
> > >
> > > More information about the 2.9.0 release plan can be found here:
> > > *
> > > https://cwiki.apache.org/confluence/display/HADOOP/
> > Roadmap#Roadmap-Version2.9
> > > <
> > > https://cwiki.apache.org/confluence/display/HADOOP/
> > Roadmap#Roadmap-Version2.9
> > > >*
> > >
> > > New RC is available at: http://home.apache.org/~
> > asuresh/hadoop-2.9.0-RC2/
> > > <
> > > http://www.google.com/url?q=http%3A%2F%2Fhome.apache.org%
> > 2F~asuresh%2Fhadoop-2.9.0-RC1%2F=D=1=
> > AFQjCNE7BF35IDIMZID3hPqiNglWEVsTpg
> > > >
> > >
> > > The RC tag in git is: release-2.9.0-RC2, and the latest commit id
> is:
> > > 1eb05c1dd48fbc9e4b375a76f2046a59103bbeb1.
> > >
> > > The maven artifacts are available via repository.apache.org at:
> > > https://repository.apache.org/content/repositories/
> > orgapachehadoop-1067/
> > > <
> > > https://www.google.com/url?q=https%3A%2F%2Frepository.
> > apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D&
> > sntz=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg
> > > >
> > >
> > > Please try the release and vote; the vote will run for the usual 5
> > days,
> > > ending on Friday 17th November 2017 2pm PT time.
> > >
> > > We want to give a big shout out to Sunil, Varun, Rohith, Wangda,
> > Vrushali
> > > and Inigo for the extensive testing/validation which helped prepare
> > for
> > > RC2. Do report your results in this vote as it'll be very useful to
> > the
> > > entire community.
> > >
> > > Thanks,
> > > -Subru/Arun
> > >
> >
> >
> >
>

[jira] [Created] (HADOOP-15023) ValueQueue should also validate (lowWatermark * numValues) > 0 on construction

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-15023:
--

 Summary: ValueQueue should also validate (lowWatermark * 
numValues) > 0 on construction
 Key: HADOOP-15023
 URL: https://issues.apache.org/jira/browse/HADOOP-15023
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Xiao Chen
Priority: Minor


ValueQueue has precondition checks for each item independently, but does not 
check {{(int)(lowWatermark * numValues) > 0}}. If the product is low enough, 
casting to int will wrap that to 0, causing problems later when filling / 
getting from the queue.

[code|https://github.com/apache/hadoop/blob/branch-3.0.0-beta1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/ValueQueue.java#L224]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14949) TestKMS#testACLs fails intermitently

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14949:
--

 Summary: TestKMS#testACLs fails intermitently
 Key: HADOOP-14949
 URL: https://issues.apache.org/jira/browse/HADOOP-14949
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms, test
Reporter: Xiao Chen
Assignee: Xiao Chen


We have seen some intermittent failures of this test:

Error Message
{noformat}
java.lang.AssertionError
{noformat}
Stack Trace

{noformat}java.lang.AssertionError: Should not have been able to 
reencryptEncryptedKey
at org.junit.Assert.fail(Assert.java:88)
at 
org.apache.hadoop.crypto.key.kms.server.TestKMS$11$15.run(TestKMS.java:1616)
at 
org.apache.hadoop.crypto.key.kms.server.TestKMS$11$15.run(TestKMS.java:1608)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at 
org.apache.hadoop.crypto.key.kms.server.TestKMS.doAs(TestKMS.java:313)
at 
org.apache.hadoop.crypto.key.kms.server.TestKMS.access$100(TestKMS.java:97)
{noformat}
Standard Output
{noformat}
2017-10-07 09:44:11,112 INFO  log - jetty-6.1.26.cloudera.4
2017-10-07 09:44:11,131 INFO  KMSWebApp - 
-
2017-10-07 09:44:11,131 INFO  KMSWebApp -   Java runtime version : 1.7.0_121-b00
2017-10-07 09:44:11,131 INFO  KMSWebApp -   User: slave
2017-10-07 09:44:11,131 INFO  KMSWebApp -   KMS Hadoop Version: 
2.6.0-cdh5.14.0-SNAPSHOT
2017-10-07 09:44:11,131 INFO  KMSWebApp - 
-
2017-10-07 09:44:11,134 INFO  KMSACLs - 'CREATE' ACL 'CREATE,SET_KEY_MATERIAL'
2017-10-07 09:44:11,134 INFO  KMSACLs - 'DELETE' ACL 'DELETE'
2017-10-07 09:44:11,134 INFO  KMSACLs - 'ROLLOVER' ACL 
'ROLLOVER,SET_KEY_MATERIAL'
2017-10-07 09:44:11,134 INFO  KMSACLs - 'GET' ACL 'GET'
2017-10-07 09:44:11,135 INFO  KMSACLs - 'GET_KEYS' ACL 'GET_KEYS'
2017-10-07 09:44:11,135 INFO  KMSACLs - 'GET_METADATA' ACL 'GET_METADATA'
2017-10-07 09:44:11,135 INFO  KMSACLs - 'SET_KEY_MATERIAL' ACL 
'SET_KEY_MATERIAL'
2017-10-07 09:44:11,135 INFO  KMSACLs - 'GENERATE_EEK' ACL 'GENERATE_EEK'
2017-10-07 09:44:11,135 INFO  KMSACLs - 'DECRYPT_EEK' ACL 'DECRYPT_EEK'
2017-10-07 09:44:11,135 INFO  KMSACLs - KEY_NAME 'k0' KEY_OP 'ALL' ACL '*'
2017-10-07 09:44:11,135 INFO  KMSACLs - KEY_NAME 'k1' KEY_OP 'ALL' ACL '*'
2017-10-07 09:44:11,136 INFO  KMSAudit - No audit logger configured, using 
default.
2017-10-07 09:44:11,137 INFO  KMSAudit - Initializing audit logger class 
org.apache.hadoop.crypto.key.kms.server.SimpleKMSAuditLogger
2017-10-07 09:44:11,137 INFO  KMSWebApp - Initialized KeyProvider 
CachingKeyProvider: 
jceks://file@/tmp/run_tha_testUYG3Cl/hadoop-common-project/hadoop-kms/target/ddbffdf2-e7d8-4e75-982a-debebb227075/kms.keystore
2017-10-07 09:44:11,138 INFO  KMSWebApp - Initialized 
KeyProviderCryptoExtension EagerKeyGeneratorKeyProviderCryptoExtension: 
KeyProviderCryptoExtension: CachingKeyProvider: 
jceks://file@/tmp/run_tha_testUYG3Cl/hadoop-common-project/hadoop-kms/target/ddbffdf2-e7d8-4e75-982a-debebb227075/kms.keystore
2017-10-07 09:44:11,138 INFO  KMSWebApp - Default key bitlength is 128
2017-10-07 09:44:11,138 INFO  KMSWebApp - KMS Started
2017-10-07 09:44:11,141 INFO  PackagesResourceConfig - Scanning for root 
resource and provider classes in the packages:
  org.apache.hadoop.crypto.key.kms.server
2017-10-07 09:44:11,146 INFO  ScanningResourceConfig - Root resource classes 
found:
  class org.apache.hadoop.crypto.key.kms.server.KMS
2017-10-07 09:44:11,146 INFO  ScanningResourceConfig - Provider classes found:
  class org.apache.hadoop.crypto.key.kms.server.KMSJSONWriter
  class org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider
  class org.apache.hadoop.crypto.key.kms.server.KMSJSONReader
2017-10-07 09:44:11,147 INFO  WebApplicationImpl - Initiating Jersey 
application, version 'Jersey: 1.9 09/02/2011 11:17 AM'
2017-10-07 09:44:11,224 INFO  log - Started SocketConnector@localhost:46764
Test KMS running at: http://localhost:46764/kms
2017-10-07 09:44:11,254 INFO  kms-audit - UNAUTHORIZED[op=CREATE_KEY, key=k, 
user=client] 
2017-10-07 09:44:11,255 WARN  KMS - User cli...@example.com (auth:KERBEROS) 
request POST http://localhost:46764/kms/v1/keys caused exception.
2017-10-07 09:44:11,256 WARN  LoadBalancingKMSClientProvider - KMS provider at 
[http://localhost:46764/kms/v1/] threw an IOException [User:client not allowed 
to do 'CREATE_KEY' on 'k']!!
2017-10-07 09:44:11,256 WARN  LoadBalancingKMSClientProvider - Aborting since 
the Request has failed with all KMS providers in the group. !!
2017-10-07 09:44:11,270 INFO  kms-audit - UNAUTHORIZED[op=CREATE_KEY, key=k, 
user=client] 
2017-10-07 09:44:11,270 WARN  KMS - User cli...@example.com (auth:KERBEROS

[jira] [Created] (HADOOP-14944) Add JvmMetrics to KMS

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14944:
--

 Summary: Add JvmMetrics to KMS
 Key: HADOOP-14944
 URL: https://issues.apache.org/jira/browse/HADOOP-14944
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen


Let's make KMS to use {{JvmMetrics}} to report aggregated statistics about heap 
/ GC etc.

This will make statistics monitoring easier, and compatible across the 
tomcat-based (branch-2) KMS and the jetty-based (branch-3) KMS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: H9 build slave is bad

[Xiao Chen]

Running into this again, filed
https://issues.apache.org/jira/browse/INFRA-15194

Noticed some of the HDFS jobs succeeding even the job itself failed and
says 'see HADOOP-13591' in the end. Though not my run wasn't that luck. :(

-Xiao

On Fri, Mar 10, 2017 at 10:19 AM, Sean Busbey  wrote:

> All the precommit builds should be doing the correct thing now for
> making sure we don't render nodes useless. They don't flag the problem
> yet and someone will still need to run the "cleanup" job on nodes
> broken before jenkins runs pick up the new configuration changes.
>
> Probably best if we move to HADOOP-13951 if we need more discussion.
>
> On Thu, Mar 9, 2017 at 5:16 PM, Allen Wittenauer
>  wrote:
> >
> >> On Mar 9, 2017, at 2:15 PM, Andrew Wang 
> wrote:
> >>
> >> H9 is again eating our builds.
> >>
> >
> > H0: https://builds.apache.org/job/PreCommit-HDFS-Build/18652/console
> > H6: https://builds.apache.org/job/PreCommit-HDFS-Build/18646/console
> >
>
>
>
> --
> busbey
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

[Xiao Chen]

Thanks Andrew!

+1 (binding)

   - Verified md5 matches
   - Built from src tarball
   - Started pseudo-distributed hdfs cluster with kms, ran basic operations
   - Sanity checked logs and NN webui

P.S. Obviously Andrew meant to say Oct. 3rd, not November. :)

-Xiao

On Thu, Sep 28, 2017 at 5:04 PM, Andrew Wang 
wrote:

> Hi all,
>
> Let me start, as always, by thanking the many, many contributors who helped
> with this release! I've prepared an RC0 for 3.0.0-beta1:
>
> http://home.apache.org/~wang/3.0.0-beta1-RC0/
>
> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
>
> beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
> improvements, and feature enhancements. Notable additions include the
> addition of YARN Timeline Service v2 alpha2, S3Guard, completion of the
> shaded client, and HDFS erasure coding pluggable policy support.
>
> I've done the traditional testing of running a Pi job on a pseudo cluster.
> My +1 to start.
>
> We're working internally on getting this run through our integration test
> rig. I'm hoping Vijay or Ray can ring in with a +1 once that's complete.
>
> Best,
> Andrew
>

[jira] [Reopened] (HADOOP-14521) KMS client needs retry logic

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen reopened HADOOP-14521:


> KMS client needs retry logic
> 
>
> Key: HADOOP-14521
> URL: https://issues.apache.org/jira/browse/HADOOP-14521
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14521.09.patch, 
> HADOOP-14521-branch-2.8.002.patch, HADOOP-14521-branch-2.8.2.patch, 
> HADOOP-14521-trunk-10.patch, HDFS-11804-branch-2.8.patch, 
> HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, HDFS-11804-trunk-3.patch, 
> HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, HDFS-11804-trunk-6.patch, 
> HDFS-11804-trunk-7.patch, HDFS-11804-trunk-8.patch, HDFS-11804-trunk.patch
>
>
> The kms client appears to have no retry logic – at all.  It's completely 
> decoupled from the ipc retry logic.  This has major impacts if the KMS is 
> unreachable for any reason, including but not limited to network connection 
> issues, timeouts, the +restart during an upgrade+.
> This has some major ramifications:
> # Jobs may fail to submit, although oozie resubmit logic should mask it
> # Non-oozie launchers may experience higher rates if they do not already have 
> retry logic.
> # Tasks reading EZ files will fail, probably be masked by framework reattempts
> # EZ file creation fails after creating a 0-length file – client receives 
> EDEK in the create response, then fails when decrypting the EDEK
> # Bulk hadoop fs copies, and maybe distcp, will prematurely fail



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14841) Add KMS Client retry to handle 'No content to map' EOFExceptions

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14841:
--

 Summary: Add KMS Client retry to handle 'No content to map' 
EOFExceptions
 Key: HADOOP-14841
 URL: https://issues.apache.org/jira/browse/HADOOP-14841
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen


We have seen quite some occurrences when the KMS server is stressed, some of 
the requests would end up getting a 500 return code, with this in the server 
log:
{noformat}
2017-08-31 06:45:33,021 WARN org.apache.hadoop.crypto.key.kms.server.KMS: User 
impala/HOSTNAME@REALM (auth:KERBEROS) request POST 
https://HOSTNAME:16000/kms/v1/keyversion/MNHDKEdWtZWM4vPb0p2bw544vdSRB2gy7APAQURcZns/_eek?eek_op=decrypt
 caused exception.
java.io.EOFException: No content to map to Object due to end of input
at 
org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2444)
at 
org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2396)
at 
org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1648)
at 
org.apache.hadoop.crypto.key.kms.server.KMSJSONReader.readFrom(KMSJSONReader.java:54)
at 
com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
at 
com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
at 
com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
at 
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
at 
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
at 
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at 
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at 
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at 
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at 
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at 
org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:631)
at 
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:301)
at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:579)
at 
org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:130)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191

[jira] [Resolved] (HADOOP-14772) Audit-log delegation token related operations to the KMS

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen resolved HADOOP-14772.

Resolution: Won't Fix

Closing as won't fix for now, due to above comment.

> Audit-log delegation token related operations to the KMS
> 
>
> Key: HADOOP-14772
> URL: https://issues.apache.org/jira/browse/HADOOP-14772
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: kms
>Affects Versions: 2.6.0
>    Reporter: Xiao Chen
>Assignee: Xiao Chen
>
> When inspecting the code, I found that the following methods are not audit 
> logged:
> - getDelegationToken
> - renewDelegationToken
> - cancelDelegationToken
> This jira is to propose add audit logging. A similar jira for HDFS is 
> HDFS-12300



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14779) Refactor decryptEncryptedKey in KeyProviderCryptoExtension

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14779:
--

 Summary: Refactor decryptEncryptedKey in KeyProviderCryptoExtension
 Key: HADOOP-14779
 URL: https://issues.apache.org/jira/browse/HADOOP-14779
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen
Priority: Minor


We could separate out the actual decrypt logic from the 
{{decryptEncryptedKey}}. This enables reencrypt calls to possibly reuse the 
codec.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14772) Audit-log delegation token related operations to the KMS

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14772:
--

 Summary: Audit-log delegation token related operations to the KMS
 Key: HADOOP-14772
 URL: https://issues.apache.org/jira/browse/HADOOP-14772
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen


When inspecting the code, I found that the following methods are not audit 
logged:
- getDelegationToken
- renewDelegationToken
- cancelDelegationToken

This jira is to propose add audit logging. A similar jira for HDFS is HDFS-12300




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14760) Add missing override to LoadBalancingKMSClientProvider

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14760:
--

 Summary: Add missing override to LoadBalancingKMSClientProvider
 Key: HADOOP-14760
 URL: https://issues.apache.org/jira/browse/HADOOP-14760
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Reporter: Xiao Chen
Assignee: Xiao Chen
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14727) Socket not closed properly when reading Configurations with BlockReaderRemote

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14727:
--

 Summary: Socket not closed properly when reading Configurations 
with BlockReaderRemote
 Key: HADOOP-14727
 URL: https://issues.apache.org/jira/browse/HADOOP-14727
 Project: Hadoop Common
  Issue Type: Bug
  Components: conf
Affects Versions: 3.0.0-alpha4, 2.9.0
Reporter: Xiao Chen
Priority: Blocker


This is caught by Cloudera's internal testing over the alpha3 release.

We got report that some hosts ran out of FDs. Triaging that, found out both 
oozie server and Yarn JobHistoryServer have tons of sockets on {{CLOSE_WAIT}} 
state.

[~haibochen] helped then narrow down a consistent reproduction by simply 
visiting the JHS webui, and clicking through a job and its logs.

I then look at the {{BlockReaderRemote}} and related code. After adding a debug 
log whenever a {{Peer}} is created/closed/in/out {{PeerCache}}, it looks like 
all the {{CLOSE_WAIT}} sockets are created from this call stack:
{noformat}
2017-08-02 13:58:59,901 INFO 
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory:  associated peer 
NioInetPeer(Socket[addr=/10.17.196.28,port=20002,localport=42512]) with 
blockreader org.apache.hadoop.hdfs.client.impl.BlockReaderRemote@717ce109
java.lang.Exception: test
at 
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:745)
at 
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:385)
at 
org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:636)
at 
org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:566)
at 
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:749)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:807)
at java.io.DataInputStream.read(DataInputStream.java:149)
at 
com.ctc.wstx.io.StreamBootstrapper.ensureLoaded(StreamBootstrapper.java:482)
at 
com.ctc.wstx.io.StreamBootstrapper.resolveStreamEncoding(StreamBootstrapper.java:306)
at 
com.ctc.wstx.io.StreamBootstrapper.bootstrapInput(StreamBootstrapper.java:167)
at 
com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:573)
at 
com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:633)
at 
com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:647)
at 
com.ctc.wstx.stax.WstxInputFactory.createXMLStreamReader(WstxInputFactory.java:366)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2649)
at 
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2697)
at 
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2662)
at 
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2545)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1076)
at 
org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1126)
at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1344)
at org.apache.hadoop.mapreduce.counters.Limits.init(Limits.java:45)
at org.apache.hadoop.mapreduce.counters.Limits.reset(Limits.java:130)
at 
org.apache.hadoop.mapreduce.v2.hs.CompletedJob.loadFullHistoryData(CompletedJob.java:363)
at 
org.apache.hadoop.mapreduce.v2.hs.CompletedJob.(CompletedJob.java:105)
at 
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager$HistoryFileInfo.loadJob(HistoryFileManager.java:473)
at 
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.loadJob(CachedHistoryStorage.java:180)
at 
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.access$000(CachedHistoryStorage.java:52)
at 
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage$1.load(CachedHistoryStorage.java:103)
at 
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage$1.load(CachedHistoryStorage.java:100)
at 
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at 
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at 
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
at 
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
at 
com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4834)
at 
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:193)
at 
org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:220

[jira] [Created] (HADOOP-14705) Add batched reencryptEncryptedKey interface to KMS

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14705:
--

 Summary: Add batched reencryptEncryptedKey interface to KMS
 Key: HADOOP-14705
 URL: https://issues.apache.org/jira/browse/HADOOP-14705
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Reporter: Xiao Chen
Assignee: Xiao Chen


HADOOP-13827 already enabled the KMS to re-encrypt a {{EncryptedKeyVersion}}.

But as the performance results of HDFS-10899 turns out, communication overhead 
with the KMS occupies the majority of the time. So we need a batched interface 
to re-encrypt multiple EDEKs in 1 call.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14688) Intern strings in KeyVersion and EncryptedKeyVersion

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14688:
--

 Summary: Intern strings in KeyVersion and EncryptedKeyVersion
 Key: HADOOP-14688
 URL: https://issues.apache.org/jira/browse/HADOOP-14688
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Reporter: Xiao Chen
Assignee: Xiao Chen


This is inspired by [~mi...@cloudera.com]'s work on HDFS-11383.

The key names and key version names are usually the same for a bunch of 
{{KeyVersion}} and {{EncryptedKeyVersion}}. We should not create duplicate 
objects for them.

This is more important to HDFS-10899, where we try to re-encrypt all files' 
EDEKs in a given EZ. Those EDEKs all has the same key name, and mostly using no 
more than a couple of key version names.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha4-RC0

[Xiao Chen]

Thanks Andrew!
+1 (non-binding)

   - Verified md5's, checked tarball sizes are reasonable
   - Built source tarball and deployed a pseudo-distributed cluster with
   hdfs/kms
   - Tested basic hdfs/kms operations
   - Sanity checked webuis/logs


-Xiao

On Wed, Jul 5, 2017 at 10:33 PM, John Zhuge  wrote:

> +1 (non-binding)
>
>
>- Verified checksums and signatures of the tarballs
>- Built source with native, Java 1.8.0_131 on Mac OS X 10.12.5
>- Cloud connectors:
>   - A few S3A integration tests
>   - A few ADL live unit tests
>- Deployed both binary and built source to a pseudo cluster, passed the
>following sanity tests in insecure, SSL, and SSL+Kerberos mode:
>   - HDFS basic and ACL
>   - DistCp basic
>   - WordCount (skipped in Kerberos mode)
>   - KMS and HttpFS basic
>
> Thanks Andrew for the great effort!
>
> On Wed, Jul 5, 2017 at 1:33 PM, Eric Payne  invalid>
> wrote:
>
> > Thanks Andrew.
> > I downloaded the source, built it, and installed it onto a pseudo
> > distributed 4-node cluster.
> >
> > I ran mapred and streaming test cases, including sleep and wordcount.
> > +1 (non-binding)
> > -Eric
> >
> >   From: Andrew Wang 
> >  To: "common-dev@hadoop.apache.org" ; "
> > hdfs-...@hadoop.apache.org" ; "
> > mapreduce-...@hadoop.apache.org" ; "
> > yarn-...@hadoop.apache.org" 
> >  Sent: Thursday, June 29, 2017 9:41 PM
> >  Subject: [VOTE] Release Apache Hadoop 3.0.0-alpha4-RC0
> >
> > Hi all,
> >
> > As always, thanks to the many, many contributors who helped with this
> > release! I've prepared an RC0 for 3.0.0-alpha4:
> >
> > http://home.apache.org/~wang/3.0.0-alpha4-RC0/
> >
> > The standard 5-day vote would run until midnight on Tuesday, July 4th.
> > Given that July 4th is a holiday in the US, I expect this vote might have
> > to be extended, but I'd like to close the vote relatively soon after.
> >
> > I've done my traditional testing of a pseudo-distributed cluster with a
> > single task pi job, which was successful.
> >
> > Normally my testing would end there, but I'm slightly more confident this
> > time. At Cloudera, we've successfully packaged and deployed a snapshot
> from
> > a few days ago, and run basic smoke tests. Some bugs found from this
> > include HDFS-11956, which fixes backwards compat with Hadoop 2 clients,
> and
> > the revert of HDFS-11696, which broke NN QJM HA setup.
> >
> > Vijay is working on a test run with a fuller test suite (the results of
> > which we can hopefully post soon).
> >
> > My +1 to start,
> >
> > Best,
> > Andrew
> >
> >
> >
> >
>
>
>
> --
> John
>

[jira] [Resolved] (HADOOP-13872) KMS JMX exception

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen resolved HADOOP-13872.

Resolution: Duplicate

Seems there's also HADOOP-14024 for branch-2, which I don't remember at 
all. re-resolving this as dup then. Thanks.

> KMS JMX exception
> -
>
> Key: HADOOP-13872
> URL: https://issues.apache.org/jira/browse/HADOOP-13872
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Critical
>
> Run KMS in pseudo distributed mode, point browser to 
> http://localhost:16000/kms/jmx?user.name=kms, got "HTTP Status 500 - Servlet 
> execution threw an exception":
> {noformat}
> HTTP Status 500 - Servlet execution threw an exception
> type Exception report
> message Servlet execution threw an exception
> description The server encountered an internal error that prevented it from 
> fulfilling this request.
> exception
> javax.servlet.ServletException: Servlet execution threw an exception
>   
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
>   
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:636)
>   
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:304)
>   
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:588)
>   
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:141)
> root cause
> java.lang.NoClassDefFoundError: 
> org/eclipse/jetty/server/handler/ContextHandler
>   java.lang.ClassLoader.defineClass1(Native Method)
>   java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>   java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   
> org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2946)
>   
> org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1177)
>   
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1665)
>   
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1544)
>   org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:176)
>   javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
>   javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
>   
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
>   
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:636)
>   
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:304)
>   
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:588)
>   
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:141)
> root cause
> java.lang.ClassNotFoundException: 
> org.eclipse.jetty.server.handler.ContextHandler
>   
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1698)
>   
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1544)
>   java.lang.ClassLoader.defineClass1(Native Method)
>   java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>   java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   
> org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2946)
>   
> org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1177)
>   
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1665)
>   
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1544)
>   org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:176)
>   javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
>   javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
>   
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
>   
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:636)
&

[jira] [Reopened] (HADOOP-14515) Specifically configure zookeeper-related log levels in KMS log4j

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen reopened HADOOP-14515:


It seems I missed the main log4j here attaching addendum patch.

> Specifically configure zookeeper-related log levels in KMS log4j
> 
>
> Key: HADOOP-14515
> URL: https://issues.apache.org/jira/browse/HADOOP-14515
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: kms
>Affects Versions: 2.6.0
>    Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2
>
> Attachments: HADOOP-14515.01.patch, HADOOP-14515.addendum.patch
>
>
> When investigating a case, we tried to turn on KMS DEBUG by setting the root 
> logger in the log4j to DEBUG. This ends up making 
> {{org.apache.zookeeper.ClientCnxn}} to generate 199.2M out of a 200M log 
> file, which made the kms.log rotate very quickly.
> We should keep zookeeper's log unaffected by the root logger, and only turn 
> it on when interested.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-13872) KMS JMX exception

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen reopened HADOOP-13872:


> KMS JMX exception
> -
>
> Key: HADOOP-13872
> URL: https://issues.apache.org/jira/browse/HADOOP-13872
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Critical
> Fix For: 3.0.0-alpha2
>
>
> Run KMS in pseudo distributed mode, point browser to 
> http://localhost:16000/kms/jmx?user.name=kms, got "HTTP Status 500 - Servlet 
> execution threw an exception":
> {noformat}
> HTTP Status 500 - Servlet execution threw an exception
> type Exception report
> message Servlet execution threw an exception
> description The server encountered an internal error that prevented it from 
> fulfilling this request.
> exception
> javax.servlet.ServletException: Servlet execution threw an exception
>   
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
>   
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:636)
>   
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:304)
>   
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:588)
>   
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:141)
> root cause
> java.lang.NoClassDefFoundError: 
> org/eclipse/jetty/server/handler/ContextHandler
>   java.lang.ClassLoader.defineClass1(Native Method)
>   java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>   java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   
> org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2946)
>   
> org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1177)
>   
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1665)
>   
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1544)
>   org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:176)
>   javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
>   javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
>   
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
>   
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:636)
>   
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:304)
>   
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:588)
>   
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:141)
> root cause
> java.lang.ClassNotFoundException: 
> org.eclipse.jetty.server.handler.ContextHandler
>   
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1698)
>   
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1544)
>   java.lang.ClassLoader.defineClass1(Native Method)
>   java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>   java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   
> org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2946)
>   
> org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1177)
>   
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1665)
>   
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1544)
>   org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:176)
>   javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
>   javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
>   
> org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
>   
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:636)
>   
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:304)
>   
> org.apache.had

[jira] [Created] (HADOOP-14524) Make CryptoCodec Closeable so it can be cleaned up proactively

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14524:
--

 Summary: Make CryptoCodec Closeable so it can be cleaned up 
proactively
 Key: HADOOP-14524
 URL: https://issues.apache.org/jira/browse/HADOOP-14524
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Xiao Chen
Assignee: Xiao Chen


See HADOOP-14523 for motivation. Credit to [~mi...@cloudera.com] for reporting 
initially on HADOOP-14523.

Basically, the CryptoCodec class is not a closeable, but the 
OpensslAesCtrCryptoCodec implementation of it contains a closeable member (the 
Random object). Currently it is left for {{finalize()}} to clean up, this would 
create problems if OpensslAesCtrCryptoCodec is used with OsSecureRandom, which 
could let OS run out of FDs on {{/dev/urandom}} if too many codecs created.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14497) Logs for KMS delegation token lifecycle

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen resolved HADOOP-14497.

   Resolution: Done
Fix Version/s: 3.0.0-alpha4
   2.9.0

All sub-tasks are done and the items mentioned in the description is complete. 
Specifically:
#1 is improved by subtask 4
#4 is added by subtask 2
#2 and #3 already exists.

So I'm closing this jira as done. Thank you for reporting, [~yzhangal], and 
feel free to reopen of comment if you think there's anything else we should do 
here!

> Logs for KMS delegation token lifecycle
> ---
>
> Key: HADOOP-14497
> URL: https://issues.apache.org/jira/browse/HADOOP-14497
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Yongjun Zhang
>Assignee: Xiao Chen
> Fix For: 2.9.0, 3.0.0-alpha4
>
>
> We run into quite some customer cases about authentication failures related 
> to KMS delegation token. It would be nice to see a log for each stage of the 
> token:
> 1. creation
> 2. renewal
> 3. removal upon cancel
> 4. remove upon expiration
> So that when we correlate the logs for the same DT, we can have a good 
> picture about what's going on, and what could have caused the authentication 
> failure.
> The same is applicable to other delegation tokens.
> NOTE: When log info about delagation token, we don't want leak user's secret 
> info.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14515) Specifically configure org.apache.zookeeper.ClientCnxn in KMS log4j

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14515:
--

 Summary: Specifically configure org.apache.zookeeper.ClientCnxn in 
KMS log4j
 Key: HADOOP-14515
 URL: https://issues.apache.org/jira/browse/HADOOP-14515
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen


When investigating a case, we tried to turn on KMS DEBUG by setting the root 
logger in the log4j to DEBUG. This ends up making 
{{org.apache.zookeeper.ClientCnxn}} to generate 199.2M out of a 200M log file, 
which made the kms.log rotate very quickly.

We should keep zookeeper's log unaffected by the root logger, and only turn it 
on when interested.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13474) Add more details in the log when a token is expired

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen resolved HADOOP-13474.

Resolution: Won't Fix

With more understanding around this area, I think this jira is not necessary.
This is because AuthenticationFilter is usually passing the authentication 
further down to the authentication handler, and that's where we should log more.
Will cover that in HADOOP-13174, so closing this one.

> Add more details in the log when a token is expired
> ---
>
> Key: HADOOP-13474
> URL: https://issues.apache.org/jira/browse/HADOOP-13474
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: security
>Affects Versions: 2.6.0
>    Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13474.01.patch
>
>
> Currently when there's an expired token, we see this from the log:
> {noformat}
> 2016-08-06 07:13:20,807 WARN 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter: 
> AuthenticationToken ignored: AuthenticationToken expired
> 2016-08-06 09:55:48,665 WARN 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter: 
> AuthenticationToken ignored: AuthenticationToken expired
> 2016-08-06 10:01:41,452 WARN 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter: 
> AuthenticationToken ignored: AuthenticationToken expired
> {noformat}
> We should log a better 
> [message|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java#L456],
>  to include more details (e.g. token type, username, tokenid) for 
> trouble-shooting purpose.
> I don't think the additional information exposed will lead to any security 
> concern, since the token is expired anyways.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14480) Remove Oracle JDK usage in Dockerfile

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14480:
--

 Summary: Remove Oracle JDK usage in Dockerfile
 Key: HADOOP-14480
 URL: https://issues.apache.org/jira/browse/HADOOP-14480
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Xiao Chen






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14127) Add log4j configuration to enable logging in hadoop-distcp's tests

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14127:
--

 Summary: Add log4j configuration to enable logging in 
hadoop-distcp's tests
 Key: HADOOP-14127
 URL: https://issues.apache.org/jira/browse/HADOOP-14127
 Project: Hadoop Common
  Issue Type: Improvement
  Components: test
Reporter: Xiao Chen
Assignee: Xiao Chen
Priority: Minor
 Attachments: HADOOP-14127.01.patch

>From review comment of HDFS-9868



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14095) Document caveats about the default JavaKeyStoreProvider in KMS

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14095:
--

 Summary: Document caveats about the default JavaKeyStoreProvider 
in KMS
 Key: HADOOP-14095
 URL: https://issues.apache.org/jira/browse/HADOOP-14095
 Project: Hadoop Common
  Issue Type: Improvement
  Components: documentation, kms
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen


KMS doc provides and example to use JavaKeyStoreProvider. But we should 
document the caveats of using it and setting it up, specifically about keystore 
passwords.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14060) KMS /logs servlet should have access control

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen resolved HADOOP-14060.

Resolution: Duplicate

> KMS /logs servlet should have access control
> 
>
> Key: HADOOP-14060
> URL: https://issues.apache.org/jira/browse/HADOOP-14060
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 3.0.0-alpha3
>Reporter: John Zhuge
>Assignee: John Zhuge
>
> HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine 
> for /conf, /jmx, /logLevel, and /stacks, but not for /logs.
> The code in {{AdminAuthorizedServlet#doGet}} for /logs and 
> {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that 
> /logs should subject to the same access control as intended by the original 
> developer.
> IMHO this could either be my misconfiguration or there is a bug somewhere in 
> {{HttpServer2}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14025) Enhance AccessControlList to reject empty ugi#getGroups

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-14025:
--

 Summary: Enhance AccessControlList to reject empty ugi#getGroups
 Key: HADOOP-14025
 URL: https://issues.apache.org/jira/browse/HADOOP-14025
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen


It would be good to reconsider empty groups when checking user access.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

[Xiao Chen]

Thanks Andrew and the community to work out the alpha2 RC!

+1 (non-binding)

   - Built the source tarball
   - Tested on a pseudo-distributed cluster, basic HDFS operations/sample
   pi job over HDFS encryption zone work.
   - Sanity checked NN and KMS webui
   - Sanity checked NN/DN/KMS logs.


-Xiao

On Wed, Jan 25, 2017 at 9:41 AM, Zhihai Xu  wrote:

> Thanks Andrew for creating release Hadoop 3.0.0-alpha2 RC0
> +1 ( non-binding)
>
> --Downloaded source and built from it.
> --Deployed on a pseudo-distributed cluster.
> --Ran sample MR jobs and tested with basics HDFS operations.
> --Did a sanity check for RM and NM UI.
>
> Best,
> zhihai
>
> On Wed, Jan 25, 2017 at 8:07 AM, Kuhu Shukla  >
> wrote:
>
> > +1 (non-binding)* Built from source* Deployed on a pseudo-distributed
> > cluster (MAC)* Ran wordcount and sleep jobs.
> >
> >
> > On Wednesday, January 25, 2017 3:21 AM, Marton Elek <
> > me...@hortonworks.com> wrote:
> >
> >
> >  Hi,
> >
> > I also did a quick smoketest with the provided 3.0.0-alpha2 binaries:
> >
> > TLDR; It works well
> >
> > Environment:
> >  * 5 hosts, docker based hadoop cluster, every component in separated
> > container (5 datanode/5 nodemanager/...)
> >  * Components are:
> >   * Hdfs/Yarn cluster (upgraded 2.7.3 to 3.0.0-alpha2 using the binary
> > package for vote)
> >   * Zeppelin 0.6.2/0.7.0-RC2
> >   * Spark 2.0.2/2.1.0
> >   * HBase 1.2.4 + zookeeper
> >   * + additional docker containers for configuration management and
> > monitoring
> > * No HA, no kerberos, no wire encryption
> >
> >  * HDFS cluster upgraded successfully from 2.7.3 (with about 200G data)
> >  * Imported 100G data to HBase successfully
> >  * Started Spark jobs to process 1G json from HDFS (using
> > spark-master/slave cluster). It worked even when I used the Zeppelin
> 0.6.2
> > + Spark 2.0.2 (with old hadoop client included). Obviously the old
> version
> > can't use the new Yarn cluster as the token file format has been changed.
> >  * I upgraded my setup to use Zeppelin 0.7.0-RC2/Spark 2.1.0(distribution
> > without hadoop)/hadoop 3.0.0-alpha2. It also worked well: processed the
> > same json files from HDFS with spark jobs (from zeppelin) using yarn
> > cluster (master: yarn deploy-mode: cluster)
> >  * Started spark jobs (with spark submit, master: yarn) to count records
> > from the hbase database: OK
> >  * Started example Mapreduce jobs from distribution over yarn. It was OK
> > but only with specific configuration (see bellow)
> >
> > So my overall impression that it works very well (at least with my
> > 'smalldata')
> >
> > Some notes (none of them are blocking):
> >
> > 1. To run the example mapreduce jobs I defined HADOOP_MAPRED_HOME at
> > command line:
> > ./bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-
> alpha2.jar
> > pi -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME={{HADOOP_
> COMMON_HOME}}"
> > -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME={{HADOOP_COMMON_HOME}}"
> 10
> > 10
> >
> > And in the yarn-site:
> >
> > yarn.nodemanager.env-whitelist: JAVA_HOME,HADOOP_COMMON_HOME,
> > HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_
> > DISTCACHE,HADOOP_YARN_HOME,MAPRED_HOME_DIR
> >
> > I don't know the exact reason for the change, but the 2.7.3 was more
> > userfriendly as the example could be run without specific configuration.
> >
> > For the same reason I didn't start hbase mapreduce job with hbase command
> > line app (There could be some option for hbase to define MAPRED_HOME_DIR
> as
> > well, but by default I got ClassNotFoundException for one of the MR
> class)
> >
> > 2. For the records: The logging and htrace classes are excluded from the
> > shaded hadoop client jar so I added it manually one by one to the spark
> > (spark 2.1.0 distribution without hadoop):
> >
> > RUN wget `cat url` -O spark.tar.gz && tar zxf spark.tar.gz && rm
> > spark.tar.gz && mv spark* spark
> > RUN cp /opt/hadoop/share/hadoop/client/hadoop-client-api-3.0.
> 0-alpha2.jar
> > /opt/spark/jars
> > RUN cp /opt/hadoop/share/hadoop/client/hadoop-client-runtime-
> 3.0.0-alpha2.jar
> > /opt/spark/jars
> > ADD https://repo1.maven.org/maven2/org/slf4j/slf4j-
> > log4j12/1.7.10/slf4j-log4j12-1.7.10.jar /opt/spark/jars
> > ADD https://repo1.maven.org/maven2/org/apache/htrace/
> > htrace-core4/4.1.0-incubating/htrace-core4-4.1.0-incubating.jar
> > /opt/spark/jars
> > ADD https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.
> > 7.10/slf4j-api-1.7.10.jar /opt/spark/jars/
> > ADD https://repo1.maven.org/maven2/log4j/log4j/1.2.17/log4j-1.2.17.jar
> > /opt/spark/jars
> >
> > With this jars files spark 2.1.0 works well with the alpha2 version of
> > HDFS and YARN.
> >
> > 3. The messages "Upgrade in progress. Not yet finalized." wasn't
> > disappeared from the namenode webui but the cluster works well.
> >
> > Most probably I missed to do something, but it's a little bit confusing.
> >
> > (I checked the REST call, it is the jmx 

[jira] [Created] (HADOOP-13954) Some TestFTPContract tests are failing

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13954:
--

 Summary: Some TestFTPContract tests are failing
 Key: HADOOP-13954
 URL: https://issues.apache.org/jira/browse/HADOOP-13954
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs, test
Reporter: Xiao Chen


Tried to follow 
http://hadoop.apache.org/docs/r3.0.0-alpha1/hadoop-project-dist/hadoop-common/filesystem/testing.html
 and run some FTPContract tests locally on MacOS, and failed.

{noformat}
TestFTPContractRename#testRenameFileNonexistentDir:
java.io.IOException: Cannot rename source: 
ftp://localhost/testfpt/testRenameWithNonEmptySubDir/src1 to 
ftp://localhost/testfpt/testRenameWithNonEmptySubDir/dest/src1 -only same 
directory renames are supported

TestFTPContractRename#testRenameNewFileSameDir:
java.lang.AssertionError: rename(ftp://localhost/testfpt/rename_src, 
ftp://localhost/testfpt/rename_dest) returned false

TestFTPContractCreate#testCreatedFileIsEventuallyVisible:
org.apache.hadoop.fs.ftp.FTPException: Client not connected

at 
org.apache.hadoop.fs.ftp.FTPFileSystem$1.close(FTPFileSystem.java:268)
at 
org.apache.hadoop.fs.contract.AbstractContractCreateTest.testCreatedFileIsEventuallyVisible(AbstractContractCreateTest.java:232)
{noformat}

Didn't have time to look into but let's use this jira to track.

Side note, would be great to improve the doc too: ftp config file is not 
{{contract-test-options.xml}} but 
{{hadoop-common-project/hadoop-common/src/test/resources/contract/ftp.xml}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13953) Make FTPFileSystem's data connection mode and transfer mode configurable

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13953:
--

 Summary: Make FTPFileSystem's data connection mode and transfer 
mode configurable
 Key: HADOOP-13953
 URL: https://issues.apache.org/jira/browse/HADOOP-13953
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 0.22.0
Reporter: Xiao Chen
Assignee: Xiao Chen






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13948) Create automated scripts to update LICENSE/NOTICE

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13948:
--

 Summary: Create automated scripts to update LICENSE/NOTICE
 Key: HADOOP-13948
 URL: https://issues.apache.org/jira/browse/HADOOP-13948
 Project: Hadoop Common
  Issue Type: Improvement
  Components: common
Reporter: Xiao Chen
Assignee: Xiao Chen






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13923) Allow changing password on JavaKeyStoreProvider generated keystores

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13923:
--

 Summary: Allow changing password on JavaKeyStoreProvider generated 
keystores 
 Key: HADOOP-13923
 URL: https://issues.apache.org/jira/browse/HADOOP-13923
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen


{{JavaKeyStoreProvider}} generates a jceks keystore file for key storage. 
Although we have different fall backs in {{ProviderUtils#locatePassword}} to 
specify the keystore password, it appears the password itself can never be 
changed after generation.

This jira is to make it possible to change the keystore password.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13911) Remove TRUSTSTORE_PASSWORD related scripts from KMS

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13911:
--

 Summary: Remove TRUSTSTORE_PASSWORD related scripts from KMS
 Key: HADOOP-13911
 URL: https://issues.apache.org/jira/browse/HADOOP-13911
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Affects Versions: 3.0.0-alpha2
Reporter: Xiao Chen
Priority: Minor


Now that HADOOP-13864 is fixed, it's unnecessary to set the truststore 
password. Let's remove it from kms.sh.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13854) KMS should log error details even if a request is malformed

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13854:
--

 Summary: KMS should log error details even if a request is 
malformed
 Key: HADOOP-13854
 URL: https://issues.apache.org/jira/browse/HADOOP-13854
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Affects Versions: 2.6.5
Reporter: Xiao Chen
Assignee: Xiao Chen


It appears if a KMS HTTP request is malformed, it could rejected by tomcat and 
a response is sent by 
[KMSExceptionsProvider|https://github.com/apache/hadoop/blob/branch-3.0.0-alpha1/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java#L99]
 overriding Jersey's ExceptionMapper.

This behavior is okay, but in the logs we'll see an ERROR audit log, but 
nothing in kms.log (or anywhere else). This makes trouble shooting pretty 
painful, let's improve it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13838) KMSTokenRenewer should close providers

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13838:
--

 Summary: KMSTokenRenewer should close providers
 Key: HADOOP-13838
 URL: https://issues.apache.org/jira/browse/HADOOP-13838
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Affects Versions: 2.8.0
Reporter: Xiao Chen
Assignee: Xiao Chen
Priority: Critical






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13819) Directory Scanner should log startup message time correctly

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13819:
--

 Summary: Directory Scanner should log startup message time 
correctly
 Key: HADOOP-13819
 URL: https://issues.apache.org/jira/browse/HADOOP-13819
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.7.1
Reporter: Xiao Chen
Priority: Trivial


When DirectoryScanner is enabled, one can see the following log:
{noformat}
INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Periodic 
Directory Tree Verification scan starting at 1479189895562ms with interval of 
2160ms
{noformat}

The first is epoch time and should not have 'ms'. Or better yet, we can change 
it to a human-readable format.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-7930) Kerberos relogin interval in UserGroupInformation should be configurable

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen reopened HADOOP-7930:
---

Sorry to reopen this. Seeing the 3 comments above, I decided to backport this 
to branch-2 and branch-2.8 (seems we missed branch-2.1.0-beta :) ) This will 
make the backport of HADOOP-13590 easier.

> Kerberos relogin interval in UserGroupInformation should be configurable
> 
>
> Key: HADOOP-7930
> URL: https://issues.apache.org/jira/browse/HADOOP-7930
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.23.1
>Reporter: Alejandro Abdelnur
>Assignee: Robert Kanter
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-7930.patch, HADOOP-7930.patch, HADOOP-7930.patch
>
>
> Currently the check done in the *hasSufficientTimeElapsed()* method is 
> hardcoded to 10 mins wait.
> The wait time should be driven by configuration and its default value, for 
> clients should be 1 min. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13785) Clean up tests that conditionally handle IBM_JAVA

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13785:
--

 Summary: Clean up tests that conditionally handle IBM_JAVA
 Key: HADOOP-13785
 URL: https://issues.apache.org/jira/browse/HADOOP-13785
 Project: Hadoop Common
  Issue Type: Improvement
  Components: test
Affects Versions: 2.8.0
Reporter: Xiao Chen
Priority: Minor


Initially from a [review 
comment|https://issues.apache.org/jira/browse/HADOOP-13590?focusedCommentId=15625141=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15625141]
 by [~ste...@apache.org].

In some tests we see code like below:
{code}
  if (PlatformName.IBM_JAVA) {
options.put("useKeytab", keytab.getPath());
options.put("credsType", "both");
  } else {
options.put("keyTab", keytab.getPath());
options.put("useKeyTab", "true");
options.put("storeKey", "true");
options.put("doNotPrompt", "true");
options.put("useTicketCache", "true");
options.put("renewTGT", "true");
options.put("isInitiator", Boolean.toString(true));
  }
{code}
We should extract a util function to some degree, and clean them up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13775) KMS Client does not encode key names in the URL path correctly

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13775:
--

 Summary: KMS Client does not encode key names in the URL path 
correctly
 Key: HADOOP-13775
 URL: https://issues.apache.org/jira/browse/HADOOP-13775
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen


HADOOP-12962 fixed the KMS server to encode special characters correctly when 
they're part of the 
[URI|https://docs.oracle.com/javase/7/docs/api/java/net/URI.html] query.
It turns out they can also cause trouble when being the URI path. This time 
it's the client-side that's broken.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-13669) KMS Server should log exceptions before throwing

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen reopened HADOOP-13669:


> KMS Server should log exceptions before throwing
> 
>
> Key: HADOOP-13669
> URL: https://issues.apache.org/jira/browse/HADOOP-13669
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: kms
>Reporter: Xiao Chen
>Assignee: Suraj Acharya
>  Labels: supportability
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13369.2.patch, HADOOP-13369.patch, 
> HADOOP-13369.patch.1, HADOOP-13669.addendem2.patch, 
> HADOOP-13669.addendum.patch, trigger.patch
>
>
> In some recent investigation, it turns out when KMS throws an exception (into 
> tomcat), it's not logged anywhere and we can only see the exception message 
> from client-side, but not the stacktrace. Logging the stacktrance would help 
> debugging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13693) Make the SPNEGO initialization OPTIONS message in kms audit log more admin-friendly

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13693:
--

 Summary: Make the SPNEGO initialization OPTIONS message in kms 
audit log more admin-friendly
 Key: HADOOP-13693
 URL: https://issues.apache.org/jira/browse/HADOOP-13693
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Reporter: Xiao Chen
Assignee: Xiao Chen
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13318) Allow configurable connection timeouts for KMSClientProvider

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen resolved HADOOP-13318.

Resolution: Duplicate

> Allow configurable connection timeouts for KMSClientProvider
> 
>
> Key: HADOOP-13318
> URL: https://issues.apache.org/jira/browse/HADOOP-13318
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: kms
>Reporter: Xiao Chen
>Priority: Minor
>  Labels: newbie, supportability
>
> It would be useful to be able to configure timeouts for the http connections 
> in KMSClientProvider. Both readtimeout and connecttimeout would be useful.
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java#L528



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13669) KMS Server should log exceptions before throwing

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13669:
--

 Summary: KMS Server should log exceptions before throwing
 Key: HADOOP-13669
 URL: https://issues.apache.org/jira/browse/HADOOP-13669
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Xiao Chen
Assignee: Suraj Acharya






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13641) Update UGI#spawnAutoRenewalThreadForUserCreds to reduce indentation

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13641:
--

 Summary: Update UGI#spawnAutoRenewalThreadForUserCreds to reduce 
indentation
 Key: HADOOP-13641
 URL: https://issues.apache.org/jira/browse/HADOOP-13641
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Xiao Chen
Priority: Minor


>From [~drankye]'s comment in HADOOP-13590:

Could we return earlier at the beginning so we can avoid at least 2 level of 
indents and make the whole block more readable?
{code}
  /**Spawn a thread to do periodic renewals of kerberos credentials*/
  private void spawnAutoRenewalThreadForUserCreds() {
if (isSecurityEnabled()) {
  //spawn thread only if we have kerb credentials
  if (user.getAuthenticationMethod() == AuthenticationMethod.KERBEROS &&
  !isKeytab) {
...
...
 very deep nested ...
...
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13127) Correctly cache delegation tokens in KMSClientProvider

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen resolved HADOOP-13127.

Resolution: Invalid

With more understanding now, I think this is invalid. The authToken isn't 
supposed to cache a dt.

> Correctly cache delegation tokens in KMSClientProvider
> --
>
> Key: HADOOP-13127
> URL: https://issues.apache.org/jira/browse/HADOOP-13127
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.6.1
>    Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13127.01.patch
>
>
> In the initial implementation of HADOOP-10770, the authToken is updated with 
> delegation tokens during {{KMSClientProvider#addDelegationTokens }} in the 
> following line:
> {code}
> Token token = authUrl.getDelegationToken(url, authToken, renewer);
> {code}
> HADOOP-11482 is a good fix to handle UGI issue, but has a side effect in the 
> following code:
> {code}
> public Token run() throws Exception {
>   // Not using the cached token here.. Creating a new token here
>   // everytime.
>   return authUrl.getDelegationToken(url,
> new DelegationTokenAuthenticatedURL.Token(), renewer, doAsUser);
> }
> {code}
> IIUC, we should do {{setDelegationToken}} on the authToken here to cache it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13590) Retry until TGT expires even if the UGI renewal thread encountered exception

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13590:
--

 Summary: Retry until TGT expires even if the UGI renewal thread 
encountered exception
 Key: HADOOP-13590
 URL: https://issues.apache.org/jira/browse/HADOOP-13590
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 0.22.0
Reporter: Xiao Chen
Assignee: Xiao Chen


The UGI has a background thread to renew the tgt. On exception, it [terminates 
itself|https://github.com/apache/hadoop/blob/bee9f57f5ca9f037ade932c6fd01b0dad47a1296/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L1013-L1014]

If something temporarily goes wrong that results in an IOE, even if it 
recovered no renewal will be done and client will eventually fail to 
authenticate. We should retry with our best effort, until tgt expires, in the 
hope that the error recovers before that.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13582) Implement logExpireToken in ZKDelegationTokenSecretManager

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13582:
--

 Summary: Implement logExpireToken in ZKDelegationTokenSecretManager
 Key: HADOOP-13582
 URL: https://issues.apache.org/jira/browse/HADOOP-13582
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Affects Versions: 2.8.0
Reporter: Xiao Chen
Assignee: Xiao Chen
Priority: Minor


During my investigation on HADOOP-13487, I found it pretty difficult to track 
KMS DTs in ZK, even in debug level.

Adding a simple debug log to make future triages easier.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

[Xiao Chen]

+1 (non-binding)

Thanks Andrew for putting this up! Super excited to see a Hadoop 3 RC.

Verifications done on OS X:
- Verified md5 on all files
- Spot checked release notes and changes
- Built from source
- Verified LICENSE and NOTICE are correctly contained in the jars built
- Started pseudo-distributed HDFS, verified basic operations work.
- Started KMS service, verified basic KMS operations work, a simple
encryption zone works.
- Sanity checked NN webui.

Best,
-Xiao

On Wed, Aug 31, 2016 at 9:25 PM, Rakesh Radhakrishnan 
wrote:

> Thanks for getting this out.
>
> +1 (non-binding)
>
> - downloaded and built tarball from source
> - deployed HDFS-HA cluster and tested few EC file operations
> - executed few hdfs commands including EC commands
> - viewed basic UI
> - ran some of the sample jobs
>
>
> Best Regards,
> Rakesh
> Intel
>
> On Thu, Sep 1, 2016 at 6:19 AM, John Zhuge  wrote:
>
> > +1 (non-binding)
> >
> > - Build source with Java 1.8.0_101 on Centos 6.6 without native
> > - Verify license and notice using the shell script in HADOOP-13374
> > - Deploy a pseudo cluster
> > - Run basic dfs, distcp, ACL, webhdfs commands
> > - Run MapReduce workcount and pi examples
> > - Run balancer
> >
> > Thanks,
> > John
> >
> > John Zhuge
> > Software Engineer, Cloudera
> >
> > On Wed, Aug 31, 2016 at 11:46 AM, Gangumalla, Uma <
> > uma.ganguma...@intel.com>
> > wrote:
> >
> > > +1 (binding).
> > >
> > > Overall it¹s a great effort, Andrew. Thank you for putting all the
> > energy.
> > >
> > > Downloaded and built.
> > > Ran some sample jobs.
> > >
> > > I would love to see all this efforts will lead to get the GA from
> Hadoop
> > > 3.X soon.
> > >
> > > Regards,
> > > Uma
> > >
> > >
> > > On 8/30/16, 8:51 AM, "Andrew Wang"  wrote:
> > >
> > > >Hi all,
> > > >
> > > >Thanks to the combined work of many, many contributors, here's an RC0
> > for
> > > >3.0.0-alpha1:
> > > >
> > > >http://home.apache.org/~wang/3.0.0-alpha1-RC0/
> > > >
> > > >alpha1 is the first in a series of planned alpha releases leading up
> to
> > > >GA.
> > > >The objective is to get an artifact out to downstreams for testing and
> > to
> > > >iterate quickly based on their feedback. So, please keep that in mind
> > when
> > > >voting; hopefully most issues can be addressed by future alphas rather
> > > >than
> > > >future RCs.
> > > >
> > > >Sorry for getting this out on a Tuesday, but I'd still like this vote
> to
> > > >run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll
> > extend
> > > >if we lack the votes.
> > > >
> > > >Please try it out and let me know what you think.
> > > >
> > > >Best,
> > > >Andrew
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

[jira] [Resolved] (HADOOP-13557) UserGroupInformation created from a Subject incorrectly tries to renew the Keberos ticket

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen resolved HADOOP-13557.

Resolution: Duplicate

Seems like a duplicate of HADOOP-13558?
Given there are already some watchers there, I'm closing this one as a dup. 
Let's follow up over there. Thanks for creating this [~tucu00].

> UserGroupInformation created from a Subject incorrectly tries to renew the 
> Keberos ticket
> -
>
> Key: HADOOP-13557
> URL: https://issues.apache.org/jira/browse/HADOOP-13557
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.2, 2.6.4, 3.0.0-alpha2
>Reporter: Alejandro Abdelnur
>
> The UGI {{checkTGTAndReloginFromKeytab()}} method checks certain conditions 
> and if they are met it invokes the {{reloginFromKeytab()}}. The 
> {{reloginFromKeytab()}} method then fails with an {{IOException}} 
> "loginUserFromKeyTab must be done first" because there is no keytab 
> associated with the UGI.
> The {{checkTGTAndReloginFromKeytab()}} method checks if there is a keytab 
> ({{isKeytab}} UGI instance variable) associated with the UGI, if there is one 
> it triggers a call to {{reloginFromKeytab()}}. The problem is that the 
> {{keytabFile}} UGI instance variable is NULL, and that triggers the mentioned 
> {{IOException}}.
> The root of the problem seems to be when creating a UGI via the 
> {{UGI.loginUserFromSubject(Subject)}} method, this method uses the 
> {{UserGroupInformation(Subject)}} constructor, and this constructor does the 
> following to determine if there is a keytab or not.
> {code}
>   this.isKeytab = KerberosUtil.hasKerberosKeyTab(subject);
> {code}
> If the {{Subject}} given had a keytab, then the UGI instance will have the 
> {{isKeytab}} set to TRUE.
> It sets the UGI instance as it would have a keytab because the Subject has a 
> keytab. This has 2 problems:
> First, it does not set the keytab file (and this, having the {{isKeytab}} set 
> to TRUE and the {{keytabFile}) set to NULL is what triggers the 
> {{IOException}} in the method {{reloginFromKeytab()}}.
> Second (and even if the first problem is fixed, this still is a problem), it 
> assumes that because the subject has a keytab it is up to UGI to to the 
> relogin using the keytab. This is incorrect if the UGI was created using the 
> {{UGI.loginUserFromSubject(Subject)}} method. In such case, the owner of the 
> Subject is not the UGI, but the caller, so the caller is responsible for 
> renewing the Kerberos tickets and the UGI should not try to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13539) KMS's zookeeper-based secret manager should be consistent when failed to remove node

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13539:
--

 Summary: KMS's zookeeper-based secret manager should be consistent 
when failed to remove node
 Key: HADOOP-13539
 URL: https://issues.apache.org/jira/browse/HADOOP-13539
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: adding contributor roles timing out again

[Xiao Chen]

Could someone help add Suraj Acharya (suraj dot spa at gmail dot com) to
HADOOP?

Not working for me either (on OSX). The error I got is a pop-up saying: *The
JIRA server could not be contacted. This may be a temporary glitch or the
server may be down.*

+1 on add grouping.
Also, IMHO we should figure out why some of us can add people while some
others cannot. Feels to me it's separate to the size limit issue. Should
this be an INFRA jira?

Thanks!

-Xiao

On Sun, Aug 21, 2016 at 6:38 AM, Chris Nauroth 
wrote:

> I just took care of adding WeiqinYang.
>
> --Chris Nauroth
>
> On 8/21/16, 2:56 AM, "Steve Loughran"  wrote:
>
>
> > On 18 Aug 2016, at 16:39, Chris Nauroth 
> wrote:
> >
> > It’s odd that Firefox didn’t work for you.  My standard workaround
> is to use Firefox, and that’s what I just did successfully for shenyinjie.
> >
> > It’s quite mysterious to me that this problem would be
> browser-specific at all though.
> >
>
> Could you add WeiqingYang
>
> it's not working for me in Chrome, FF or Safari from an OSX box
>
> It's clear that there are too many people in that contributor group.
> We have hit a scale limit in the hadoop coding process.
>
> How about we try to do some grouping, where we just have sub groups,
> hadoop-contributors-1, hadoop-contributors-2, ... and add them as
> contributors; we then edit group membership, adding a new group when the
> current one gets above some chosen size limit?
>
>

[jira] [Created] (HADOOP-13523) Implement a json format audit logger for KMS

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13523:
--

 Summary: Implement a json format audit logger for KMS
 Key: HADOOP-13523
 URL: https://issues.apache.org/jira/browse/HADOOP-13523
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Reporter: Xiao Chen
Assignee: Xiao Chen






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13474) Add more details in the log when a token is expired

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13474:
--

 Summary: Add more details in the log when a token is expired
 Key: HADOOP-13474
 URL: https://issues.apache.org/jira/browse/HADOOP-13474
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen


Currently when there's an expired token, we see this from the log:
{noformat}
2016-08-06 07:13:20,807 WARN 
org.apache.hadoop.security.authentication.server.AuthenticationFilter: 
AuthenticationToken ignored: AuthenticationToken expired
2016-08-06 09:55:48,665 WARN 
org.apache.hadoop.security.authentication.server.AuthenticationFilter: 
AuthenticationToken ignored: AuthenticationToken expired
2016-08-06 10:01:41,452 WARN 
org.apache.hadoop.security.authentication.server.AuthenticationFilter: 
AuthenticationToken ignored: AuthenticationToken expired
{noformat}

We should log a better 
[message|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java#L456],
 to include more details (e.g. token type, username, tokenid) for 
trouble-shooting purpose.
I don't think the additional information exposed will lead to any security 
concern, since the token is expired anyways.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13437) KMS should reload whitelist and default key ACLs when hot-reloading

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13437:
--

 Summary: KMS should reload whitelist and default key ACLs when 
hot-reloading
 Key: HADOOP-13437
 URL: https://issues.apache.org/jira/browse/HADOOP-13437
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen


When hot-reloading, {{KMSACLs#setKeyACLs}} ignores whitelist and default key 
entries if they're present in memory.

We should reload them, hot-reload and cold-start should not have any difference 
in behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Xiao Chen]

+1 (non-binding)

   - Downloaded the binary tarball
   - Verified checksum
   - Verified all jars have LICENSE and NOTICE (using the script
   in HADOOP-13374)
   - Started Pseudo-distributed cluster and verified basic HDFS operations
   work, example Pi job succeed.
   - Started kms with a sample jceks store, verified basic hadoop key
   operations work.


-Xiao

On Mon, Jul 25, 2016 at 11:52 AM, Andrew Wang 
wrote:

> I'll also add that, as a YARN newbie, I did hit two usability issues. These
> are very unlikely to be regressions, and I can file JIRAs if they seem
> fixable.
>
> * I didn't have SSH to localhost set up (new laptop), and when I tried to
> run the Pi job, it'd exit my window manager session. I feel there must be a
> more developer-friendly solution here.
> * If you start the NodeManager and not the RM, the NM has a handler for
> SIGTERM and SIGINT that blocked my Ctrl-C and kill attempts during startup.
> I had to kill -9 it.
>
> On Mon, Jul 25, 2016 at 11:44 AM, Andrew Wang 
> wrote:
>
> > I got asked this off-list, so as a reminder, only PMC votes are binding
> on
> > releases. Everyone is encouraged to vote on releases though!
> >
> > +1 (binding)
> >
> > * Downloaded source, built
> > * Started up HDFS and YARN
> > * Ran Pi job which as usual returned 4, and a little teragen
> >
> > On Mon, Jul 25, 2016 at 11:08 AM, Karthik Kambatla 
> > wrote:
> >
> >> +1 (binding)
> >>
> >> * Downloaded and build from source
> >> * Checked LICENSE and NOTICE
> >> * Pseudo-distributed cluster with FairScheduler
> >> * Ran MR and HDFS tests
> >> * Verified basic UI
> >>
> >> On Sun, Jul 24, 2016 at 1:07 PM, larry mccay  wrote:
> >>
> >> > +1 binding
> >> >
> >> > * downloaded and built from source
> >> > * checked LICENSE and NOTICE files
> >> > * verified signatures
> >> > * ran standalone tests
> >> > * installed pseudo-distributed instance on my mac
> >> > * ran through HDFS and mapreduce tests
> >> > * tested credential command
> >> > * tested webhdfs access through Apache Knox
> >> >
> >> >
> >> > On Fri, Jul 22, 2016 at 10:15 PM, Vinod Kumar Vavilapalli <
> >> > vino...@apache.org> wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > I've created a release candidate RC0 for Apache Hadoop 2.7.3.
> >> > >
> >> > > As discussed before, this is the next maintenance release to follow
> up
> >> > > 2.7.2.
> >> > >
> >> > > The RC is available for validation at:
> >> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <
> >> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
> >> > >
> >> > > The RC tag in git is: release-2.7.3-RC0
> >> > >
> >> > > The maven artifacts are available via repository.apache.org <
> >> > > http://repository.apache.org/> at
> >> > >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/
> >> > <
> >> > >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/
> >> > >
> >> > >
> >> > > The release-notes are inside the tar-balls at location
> >> > >
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> >> > > hosted this at
> >> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html
> <
> >> > >
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> >> > for
> >> > > your quick perusal.
> >> > >
> >> > > As you may have noted, a very long fix-cycle for the License &
> Notice
> >> > > issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop
> >> > release)
> >> > > to slip by quite a bit. This release's related discussion thread is
> >> > linked
> >> > > below: [1].
> >> > >
> >> > > Please try the release and vote; the vote will run for the usual 5
> >> days.
> >> > >
> >> > > Thanks,
> >> > > Vinod
> >> > >
> >> > > [1]: 2.7.3 release plan:
> >> > >
> >> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html
> >> > <
> >> > > http://markmail.org/thread/6yv2fyrs4jlepmmr>
> >> >
> >>
> >
> >
>

[jira] [Created] (HADOOP-13396) Add json format audit logging to KMS

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13396:
--

 Summary: Add json format audit logging to KMS
 Key: HADOOP-13396
 URL: https://issues.apache.org/jira/browse/HADOOP-13396
 Project: Hadoop Common
  Issue Type: New Feature
  Components: kms
Reporter: Xiao Chen
Assignee: Xiao Chen






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13381) KMS clients running in the same JVM should use updated KMS Delegation Token

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13381:
--

 Summary: KMS clients running in the same JVM should use updated 
KMS Delegation Token
 Key: HADOOP-13381
 URL: https://issues.apache.org/jira/browse/HADOOP-13381
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen
Priority: Critical


When {{/tmp}} is setup as an EZ, one may experience YARN log aggregation 
failure after the KMS token is expired. The MR job itself runs find though.

When this happens, YARN NodeManager's log will show {{AuthenticationException}} 
with token is expire / token can't be found in cache, depending on whether the 
expired token is removed by the background or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13374) Add the L verification script

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13374:
--

 Summary: Add the L verification script
 Key: HADOOP-13374
 URL: https://issues.apache.org/jira/browse/HADOOP-13374
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Xiao Chen
Assignee: Xiao Chen


This is the script that's used for L change verification during HADOOP-12893. 
We should commit this as [~ozawa] 
[suggested|https://issues.apache.org/jira/browse/HADOOP-13298?focusedCommentId=15374498=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15374498].

I was 
[initially|https://issues.apache.org/jira/browse/HADOOP-12893?focusedCommentId=15283040=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15283040]
 verifying from an on-fly shell command, and [~andrew.wang] contributed the 
script later in [a comment|
https://issues.apache.org/jira/browse/HADOOP-12893?focusedCommentId=15303281=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15303281],
 so most credit should go to him. :)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13318) Allow configurable connection timeouts for KMSClientProvider

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13318:
--

 Summary: Allow configurable connection timeouts for 
KMSClientProvider
 Key: HADOOP-13318
 URL: https://issues.apache.org/jira/browse/HADOOP-13318
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Reporter: Xiao Chen
Priority: Minor


It would be useful to be able to configure timeouts for the http connections in 
KMSClientProvider. Both readtimeout and connecttimeout would be useful.

https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java#L528



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13317) Add logs to KMS servier-side to improve supportability

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13317:
--

 Summary: Add logs to KMS servier-side to improve supportability
 Key: HADOOP-13317
 URL: https://issues.apache.org/jira/browse/HADOOP-13317
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Reporter: Xiao Chen
Assignee: Xiao Chen
Priority: Minor


[KMS.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java]
 is the main class that serves KMS http requests. There're currently no logs at 
all, making trouble shooting difficult.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13316) Delegation Tokens should not be renewed or retrived using delegation token authentication

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13316:
--

 Summary: Delegation Tokens should not be renewed or retrived using 
delegation token authentication
 Key: HADOOP-13316
 URL: https://issues.apache.org/jira/browse/HADOOP-13316
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms, security
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen
Priority: Blocker






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: hadoop-build-tools/src/main/resources/META-INF/

[Xiao Chen]

FYI - created https://issues.apache.org/jira/browse/HADOOP-13298.

-Xiao

On Mon, Jun 20, 2016 at 12:03 PM, Sean Busbey <bus...@cloudera.com> wrote:

> file a jira please and I'll take a look.
>
> On Fri, Jun 17, 2016 at 4:10 PM, Xiao Chen <x...@cloudera.com> wrote:
> > Thanks Steve for reporting the issue and Sean for the suggestion. This is
> > indeed from HADOOP-12893 (blush).
> >
> > I'm no maven expert so appreciate any recommendations.
> >
> > The reason for the current way is that for the L to be patched into a
> jar,
> > it seems that maven remote resource plugin (which named itself to be the
> > typical Apache licensing way) requires the files to be under
> > src/main/resources. This was mentioned in their example, and I wasn't
> able
> > to trick it to pack things not in there. I wish there were more examples
> to
> > help in our case.
> >
> > So, in HADOOP-12893 I put a step to copy the L into
> > hadoop-build-tools/src/main/resources dir, to allow it get packed into
> the
> > jar. I thought about symlink but don't think it's a good way for Windows
> > builds.
> >
> > It's not committed because we don't want an extra copy of L, we could
> list
> > it in .gitignore.
> >
> >
> > P.S. Tried a bit with Sean's suggestion of making it under
> > target/generated-sources, but couldn't get the plugin to include it. I'm
> > happy to try out more elegant solutions if you have any suggestions.
> >
> > Thanks!
> >
> >
> > -Xiao
> >
> > On Fri, Jun 17, 2016 at 7:34 AM, Sean Busbey <bus...@cloudera.com>
> wrote:
> >>
> >> If it's generated and we're following The Maven Way, it should be in
> >> target. probably in target/generated-sources
> >>
> >> On Fri, Jun 17, 2016 at 9:33 AM, Steve Loughran <ste...@hortonworks.com
> >
> >> wrote:
> >> >
> >> > I see (presumably from the licensing work), that I'm now getting
> >> > hadoop-build-tools/src/main/resources/META-INF/ as an untracked
> directory.
> >> >
> >> > If this is generated, should it be in the source tree? And if so,
> should
> >> > it be committed, or listed in .gitignore?
> >> >
> >> > -
> >> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >> >
> >>
> >>
> >>
> >> --
> >> busbey
> >>
> >> -
> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>
> >
>
>
>
> --
> busbey
>

[jira] [Created] (HADOOP-13298) Fix the leftover L files in hadoop-build-tools/src/main/resources/META-INF/

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13298:
--

 Summary: Fix the leftover L files in 
hadoop-build-tools/src/main/resources/META-INF/
 Key: HADOOP-13298
 URL: https://issues.apache.org/jira/browse/HADOOP-13298
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
Reporter: Xiao Chen
Assignee: Sean Busbey


After HADOOP-12893, an extra copy of LICENSE.txt and NOTICE.txt exists in 
{{hadoop-build-tools/src/main/resources/META-INF/}} after build. We should 
remove it and do it the maven way.

Details in 
https://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201606.mbox/browser

Thanks [~ste...@apache.org] for raising the issue and [~busbey] for offering 
the help!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: hadoop-build-tools/src/main/resources/META-INF/

[Xiao Chen]

Thanks Steve for reporting the issue and Sean for the suggestion. This is
indeed from HADOOP-12893
 (blush).

I'm no maven expert so appreciate any recommendations.

The reason for the current way is that for the L to be patched into a
jar, it seems that maven remote resource plugin
 (which
named itself to be the typical Apache licensing way) requires the files to
be under src/main/resources. This was mentioned in their example
,
and I wasn't able to trick it to pack things not in there. I wish there
were more examples to help in our case.

So, in HADOOP-12893 I put a step to copy the L into
hadoop-build-tools/src/main/resources dir, to allow it get packed into the
jar. I thought about symlink but don't think it's a good way for Windows
builds.

It's not committed because we don't want an extra copy of L, we could
list it in .gitignore.


P.S. Tried a bit with Sean's suggestion of making it under
target/generated-sources, but couldn't get the plugin to include it. I'm
happy to try out more elegant solutions if you have any suggestions.

Thanks!


-Xiao

On Fri, Jun 17, 2016 at 7:34 AM, Sean Busbey  wrote:

> If it's generated and we're following The Maven Way, it should be in
> target. probably in target/generated-sources
>
> On Fri, Jun 17, 2016 at 9:33 AM, Steve Loughran 
> wrote:
> >
> > I see (presumably from the licensing work), that I'm now getting
> hadoop-build-tools/src/main/resources/META-INF/ as an untracked directory.
> >
> > If this is generated, should it be in the source tree? And if so, should
> it be committed, or listed in .gitignore?
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
>
>
>
> --
> busbey
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HADOOP-13258) Fix trailing whitespace in LICENSE.txt for pre-commit.

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13258:
--

 Summary: Fix trailing whitespace in LICENSE.txt for pre-commit.
 Key: HADOOP-13258
 URL: https://issues.apache.org/jira/browse/HADOOP-13258
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0-alpha1
Reporter: Xiao Chen
Assignee: Xiao Chen


HADOOP-12893 auto-generates L by copying the files into resources during 
compile.
Precommit ends up unhappy since those files have lines end in whitespace.
{quote}The patch has 20 line(s) that end in whitespace.{quote}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13255) KMSClientProvider should check and renew tgt when doing delegation token operations.

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13255:
--

 Summary: KMSClientProvider should check and renew tgt when doing 
delegation token operations.
 Key: HADOOP-13255
 URL: https://issues.apache.org/jira/browse/HADOOP-13255
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Reporter: Xiao Chen
Assignee: Xiao Chen






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org