date:20141015

Yi Liu created HADOOP-11204:
---

 Summary: Fix incorrect property in 
hadoop-kms/src/main/conf/kms-site.xml
 Key: HADOOP-11204
 URL: https://issues.apache.org/jira/browse/HADOOP-11204
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Affects Versions: 2.5.0
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Minor


{{hadoop.security.keystore.JavaKeyStoreProvider.password}} doesn't exist, it 
should be {{hadoop.security.keystore.java-keystore-provider.password-file}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11204) Fix incorrect property in hadoop-kms/src/main/conf/kms-site.xml


 [ 
https://issues.apache.org/jira/browse/HADOOP-11204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Liu updated HADOOP-11204:

Description: 
{{hadoop.security.keystore.JavaKeyStoreProvider.password}} doesn't exist, it 
should be {{hadoop.security.keystore.java-keystore-provider.password-file}}

Also check in the KMS user doc, it's correct.

  was:{{hadoop.security.keystore.JavaKeyStoreProvider.password}} doesn't exist, 
it should be {{hadoop.security.keystore.java-keystore-provider.password-file}}


 Fix incorrect property in hadoop-kms/src/main/conf/kms-site.xml
 ---

 Key: HADOOP-11204
 URL: https://issues.apache.org/jira/browse/HADOOP-11204
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Affects Versions: 2.5.0
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Minor

 {{hadoop.security.keystore.JavaKeyStoreProvider.password}} doesn't exist, it 
 should be {{hadoop.security.keystore.java-keystore-provider.password-file}}
 Also check in the KMS user doc, it's correct.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11204) Fix incorrect property in hadoop-kms/src/main/conf/kms-site.xml


 [ 
https://issues.apache.org/jira/browse/HADOOP-11204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Liu updated HADOOP-11204:

Description: 
{{hadoop.security.keystore.JavaKeyStoreProvider.password}} doesn't exist, it 
should be {{hadoop.security.keystore.java-keystore-provider.password-file}}

Also check the KMS user doc, it's correct.

  was:
{{hadoop.security.keystore.JavaKeyStoreProvider.password}} doesn't exist, it 
should be {{hadoop.security.keystore.java-keystore-provider.password-file}}

Also check in the KMS user doc, it's correct.


 Fix incorrect property in hadoop-kms/src/main/conf/kms-site.xml
 ---

 Key: HADOOP-11204
 URL: https://issues.apache.org/jira/browse/HADOOP-11204
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Affects Versions: 2.5.0
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Minor
 Attachments: HADOOP-11204.001.patch


 {{hadoop.security.keystore.JavaKeyStoreProvider.password}} doesn't exist, it 
 should be {{hadoop.security.keystore.java-keystore-provider.password-file}}
 Also check the KMS user doc, it's correct.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11204) Fix incorrect property in hadoop-kms/src/main/conf/kms-site.xml


 [ 
https://issues.apache.org/jira/browse/HADOOP-11204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Liu updated HADOOP-11204:

Attachment: HADOOP-11204.001.patch

 Fix incorrect property in hadoop-kms/src/main/conf/kms-site.xml
 ---

 Key: HADOOP-11204
 URL: https://issues.apache.org/jira/browse/HADOOP-11204
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Affects Versions: 2.5.0
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Minor
 Attachments: HADOOP-11204.001.patch


 {{hadoop.security.keystore.JavaKeyStoreProvider.password}} doesn't exist, it 
 should be {{hadoop.security.keystore.java-keystore-provider.password-file}}
 Also check in the KMS user doc, it's correct.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11182) GraphiteSink emits wrong timestamps


[ 
https://issues.apache.org/jira/browse/HADOOP-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172137#comment-14172137
 ] 

Sascha Coenen commented on HADOOP-11182:


Hi Ravi, Thanks for your help. I will try to make progress.
I can't find an existing TestGraphiteSink file in the trunk 
here 
http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/
or here 
https://github.com/apache/hadoop/tree/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink
Can you advise?
Thanks


 GraphiteSink emits wrong timestamps
 ---

 Key: HADOOP-11182
 URL: https://issues.apache.org/jira/browse/HADOOP-11182
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.5.0, 2.5.1
Reporter: Sascha Coenen
 Attachments: HADOOP-11182-GraphiteSink-v1.patch, HADOOP-11182-v2.patch


 the org.apache.hadoop.metrics2.sink.GraphiteSink class emits metrics at the 
 configured time period, but the timestamps written only change every 128 
 seconds, even it the configured time period in the configuration file is much 
 shorter.
 This is due to a bug in line 93:
 {code:java}
 092// Round the timestamp to second as Graphite accepts it in 
 such format.
 093int timestamp = Math.round(record.timestamp() / 1000.0f);
 {code}
 The timestamp property is a long and is divided by a float which yields a 
 result that is not precise enough and yields same valued results for 
 timestamps that lie up to 128 seconds apart. Also, the result is then written 
 into an int variable.
 One solution would be to divide by 1000.0d, but the best fix would be to not 
 even convert to a decimal format in the first place. Instead one could 
 replace the line with the following:
 {code:java}
long timestamp = record.timestamp() / 1000L;
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11205) ThrottledInputStream should return the actual bandwidth (read rate)

2014-10-15 Thread Raju Bairishetti (JIRA)

Raju Bairishetti created HADOOP-11205:
-

 Summary: ThrottledInputStream should return the actual bandwidth 
(read rate)
 Key: HADOOP-11205
 URL: https://issues.apache.org/jira/browse/HADOOP-11205
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools/distcp
Reporter: Raju Bairishetti
Assignee: Raju Bairishetti


Currently, it is not returning the actual read rate. Due to this, most of the 
time is in idle state.

Behavior: First, it checks whether current bandwidth (number of bytes per 
second) is more than maxBandwidth before reading a chunk of bytes(or byte) from 
buffer. If read rate exceeds max bandwidth then it sleeps for 50ms and resume 
the process after the sleeping period(50ms).

Ex: Assume, both maxBandwidth = 1MBPS and read rate = 1MBPS(i.e. reading 1M 
messages per second())

In the above case,  even if it reads 1.5MB in 1.5 sec which is ideally not 
crossing the max bandwidth but still it goes for sleeping mode as it assumes 
read rate is 1.5M (bytes read/ time i.e. 1.5/1.. time is 1500ms/1000 =1) 
instead of 1(i.e. 1.5/1.5).

Example: 
It does not got to sleep mode till 1 sec as number of bytes read in that 
elapsed time is lesser than maxBandwidth.
when it reads 1M +1 byte/chunk it checks read rate against maxBandwidth. 
when it reads 1M + 2byte /chunk it sleeps for 50ms as read rate is  1
when it reads 1M + 3byte/chunk again it sleeps for 50ms as read rate is  1
...
even if it reads 1.5MB in 1.5 sec but still it goes for sleeping mode as it 
assumes read rate is 1.5M (bytes read/ time i.e. 1.5/1.. time is 1500ms/1000 
=1) instead of 1(i.e. 1.5/1.5).

Cons: it reads for a sec and almost sleeps for a 1sec in an alternate fashion.

getBytesPerSec() method is not returning the actual bandwidth.
Current code: {code}
public long getBytesPerSec() {
long elapsed = (System.currentTimeMillis() - startTime) / 1000;
if (elapsed == 0) {
  return bytesRead;
} else {
  return bytesRead / elapsed;
}
  }
{code}
We should fix the getBytesPerSec() method:
{code}
public long getBytesPerSec() {
long elapsedTimeInMilliSecs = System.currentTimeMillis() - startTime;
if (elapsedTimeInMilliSecs = MILLISECONDS_IN_SEC) {
  return bytesRead;
} else {
  return (bytesRead * MILLISECONDS_IN_SEC)/ elapsedTimeInMilliSecs;
}
  }
{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-9992) Modify the NN loadGenerator to optionally run as a MapReduce job


[ 
https://issues.apache.org/jira/browse/HADOOP-9992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172234#comment-14172234
 ] 

Hadoop QA commented on HADOOP-9992:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12674927/hadoop-9992-v4.patch
  against trunk revision 1220bb7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 5 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 2 new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

  org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing
  
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/4926//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/4926//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/4926//console

This message is automatically generated.

 Modify the NN loadGenerator to optionally run as a MapReduce job
 

 Key: HADOOP-9992
 URL: https://issues.apache.org/jira/browse/HADOOP-9992
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Akshay Radia
Assignee: Akshay Radia
 Attachments: HADOOP-9992.004.patch, hadoop-9992-v2.patch, 
 hadoop-9992-v3.patch, hadoop-9992-v4.patch, hadoop-9992.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11181) o.a.h.security.token.delegation.DelegationTokenManager should be more generalized to handle other DelegationTokenIdentifier


[ 
https://issues.apache.org/jira/browse/HADOOP-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172279#comment-14172279
 ] 

Hudson commented on HADOOP-11181:
-

SUCCESS: Integrated in Hadoop-Yarn-trunk #712 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/712/])
HADOOP-11181. Generalized o.a.h.s.t.d.DelegationTokenManager to handle all 
sub-classes of AbstractDelegationTokenIdentifier. Contributed by Zhijie Shen. 
(zjshen: rev cdce88376a60918dfe2f3bcd82a7666d74992a19)
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestDelegationTokenManager.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestDelegationTokenAuthenticationHandlerWithMocks.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenManager.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java
* hadoop-common-project/hadoop-common/CHANGES.txt


 o.a.h.security.token.delegation.DelegationTokenManager should be more 
 generalized to handle other DelegationTokenIdentifier
 ---

 Key: HADOOP-11181
 URL: https://issues.apache.org/jira/browse/HADOOP-11181
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.6.0

 Attachments: HADOOP-11181.1.patch, HADOOP-11181.2.patch, 
 HADOOP-11181.3.patch, HADOOP-11181.4.patch, HADOOP-11181.5.patch


 While DelegationTokenManager can set external secretManager, it have the 
 assumption that the token is going to be 
 o.a.h.security.token.delegation.DelegationTokenIdentifier, and use 
 DelegationTokenIdentifier method to decode a token. 
 {code}
   @SuppressWarnings(unchecked)
   public UserGroupInformation verifyToken(TokenDelegationTokenIdentifier
   token) throws IOException {
 ByteArrayInputStream buf = new 
 ByteArrayInputStream(token.getIdentifier());
 DataInputStream dis = new DataInputStream(buf);
 DelegationTokenIdentifier id = new DelegationTokenIdentifier(tokenKind);
 id.readFields(dis);
 dis.close();
 secretManager.verifyToken(id, token.getPassword());
 return id.getUser();
   }
 {code}
 It's not going to work it the token kind is other than 
 web.DelegationTokenIdentifier. For example, RM want to reuse it but hook it 
 to RMDelegationTokenSecretManager and RMDelegationTokenIdentifier, which has 
 the customized way to decode a token.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11181) o.a.h.security.token.delegation.DelegationTokenManager should be more generalized to handle other DelegationTokenIdentifier


[ 
https://issues.apache.org/jira/browse/HADOOP-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172379#comment-14172379
 ] 

Hudson commented on HADOOP-11181:
-

FAILURE: Integrated in Hadoop-Hdfs-trunk #1902 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1902/])
HADOOP-11181. Generalized o.a.h.s.t.d.DelegationTokenManager to handle all 
sub-classes of AbstractDelegationTokenIdentifier. Contributed by Zhijie Shen. 
(zjshen: rev cdce88376a60918dfe2f3bcd82a7666d74992a19)
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenManager.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestDelegationTokenManager.java
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestDelegationTokenAuthenticationHandlerWithMocks.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java


 o.a.h.security.token.delegation.DelegationTokenManager should be more 
 generalized to handle other DelegationTokenIdentifier
 ---

 Key: HADOOP-11181
 URL: https://issues.apache.org/jira/browse/HADOOP-11181
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.6.0

 Attachments: HADOOP-11181.1.patch, HADOOP-11181.2.patch, 
 HADOOP-11181.3.patch, HADOOP-11181.4.patch, HADOOP-11181.5.patch


 While DelegationTokenManager can set external secretManager, it have the 
 assumption that the token is going to be 
 o.a.h.security.token.delegation.DelegationTokenIdentifier, and use 
 DelegationTokenIdentifier method to decode a token. 
 {code}
   @SuppressWarnings(unchecked)
   public UserGroupInformation verifyToken(TokenDelegationTokenIdentifier
   token) throws IOException {
 ByteArrayInputStream buf = new 
 ByteArrayInputStream(token.getIdentifier());
 DataInputStream dis = new DataInputStream(buf);
 DelegationTokenIdentifier id = new DelegationTokenIdentifier(tokenKind);
 id.readFields(dis);
 dis.close();
 secretManager.verifyToken(id, token.getPassword());
 return id.getUser();
   }
 {code}
 It's not going to work it the token kind is other than 
 web.DelegationTokenIdentifier. For example, RM want to reuse it but hook it 
 to RMDelegationTokenSecretManager and RMDelegationTokenIdentifier, which has 
 the customized way to decode a token.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11206) TestCryptoCodec.testOpensslAesCtrCryptoCodec fails on master without native code compiled

2014-10-15 Thread Robert Joseph Evans (JIRA)

Robert Joseph Evans created HADOOP-11206:


 Summary: TestCryptoCodec.testOpensslAesCtrCryptoCodec fails on 
master without native code compiled
 Key: HADOOP-11206
 URL: https://issues.apache.org/jira/browse/HADOOP-11206
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Robert Joseph Evans


I tried to run the unit tests recently for another issue, and didn't turn on 
native code.  I got the following error.

{code}
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.71 sec  
FAILURE! - in org.apache.hadoop.crypto.TestCryptoCodec
testOpensslAesCtrCryptoCodec(org.apache.hadoop.crypto.TestCryptoCodec)  Time 
elapsed: 0.064 sec   ERROR!
java.lang.UnsatisfiedLinkError: 
org.apache.hadoop.util.NativeCodeLoader.buildSupportsOpenssl()Z
at org.apache.hadoop.util.NativeCodeLoader.buildSupportsOpenssl(Native 
Method)
at 
org.apache.hadoop.crypto.TestCryptoCodec.testOpensslAesCtrCryptoCodec(TestCryptoCodec.java:66)
{code}

Looks like that test needs an assume that native code is loaded/compiled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-9422) HADOOP_HOME should not be required to be set to be able to launch commands using hadoop.util.Shell

2014-10-15 Thread Robert Joseph Evans (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172433#comment-14172433
 ] 

Robert Joseph Evans commented on HADOOP-9422:
-

We have run into this issue trying to use the latest HBase through storm, and 
also with trying to use slider on a custom app.  It really would be nice to get 
this in to a 2.X release at some point soon, so I don't have to hack it by 
setting HADOOP_HOME=/tmp/ which really just feels wrong to me.

The code compiles and the unit tests (related to this) pass for me (despite the 
age of the patch) I am +1, but as I have not been actively involved in Hadoop 
Development for quite a while I would like others to take a look too. [~jlowe] 
do you mind taking a look too?

I filed HADOOP-11206 for the one test error I found.

 HADOOP_HOME should not be required to be set to be able to launch commands 
 using hadoop.util.Shell
 --

 Key: HADOOP-9422
 URL: https://issues.apache.org/jira/browse/HADOOP-9422
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Hitesh Shah
Assignee: Arpit Gupta
 Attachments: HADOOP-9422.patch


 Not sure why this is an enforced requirement especially in cases where a 
 deployment is done using multiple tar-balls ( one each for 
 common/hdfs/mapreduce/yarn ). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11181) o.a.h.security.token.delegation.DelegationTokenManager should be more generalized to handle other DelegationTokenIdentifier


[ 
https://issues.apache.org/jira/browse/HADOOP-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172457#comment-14172457
 ] 

Hudson commented on HADOOP-11181:
-

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1927 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1927/])
HADOOP-11181. Generalized o.a.h.s.t.d.DelegationTokenManager to handle all 
sub-classes of AbstractDelegationTokenIdentifier. Contributed by Zhijie Shen. 
(zjshen: rev cdce88376a60918dfe2f3bcd82a7666d74992a19)
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestDelegationTokenAuthenticationHandlerWithMocks.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestDelegationTokenManager.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenManager.java


 o.a.h.security.token.delegation.DelegationTokenManager should be more 
 generalized to handle other DelegationTokenIdentifier
 ---

 Key: HADOOP-11181
 URL: https://issues.apache.org/jira/browse/HADOOP-11181
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.6.0

 Attachments: HADOOP-11181.1.patch, HADOOP-11181.2.patch, 
 HADOOP-11181.3.patch, HADOOP-11181.4.patch, HADOOP-11181.5.patch


 While DelegationTokenManager can set external secretManager, it have the 
 assumption that the token is going to be 
 o.a.h.security.token.delegation.DelegationTokenIdentifier, and use 
 DelegationTokenIdentifier method to decode a token. 
 {code}
   @SuppressWarnings(unchecked)
   public UserGroupInformation verifyToken(TokenDelegationTokenIdentifier
   token) throws IOException {
 ByteArrayInputStream buf = new 
 ByteArrayInputStream(token.getIdentifier());
 DataInputStream dis = new DataInputStream(buf);
 DelegationTokenIdentifier id = new DelegationTokenIdentifier(tokenKind);
 id.readFields(dis);
 dis.close();
 secretManager.verifyToken(id, token.getPassword());
 return id.getUser();
   }
 {code}
 It's not going to work it the token kind is other than 
 web.DelegationTokenIdentifier. For example, RM want to reuse it but hook it 
 to RMDelegationTokenSecretManager and RMDelegationTokenIdentifier, which has 
 the customized way to decode a token.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-9422) HADOOP_HOME should not be required to be set to be able to launch commands using hadoop.util.Shell


[ 
https://issues.apache.org/jira/browse/HADOOP-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172498#comment-14172498
 ] 

Hadoop QA commented on HADOOP-9422:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12586372/HADOOP-9422.patch
  against trunk revision 128ace1.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 2 new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-common-project/hadoop-common:

  org.apache.hadoop.ha.TestZKFailoverControllerStress

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/4927//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/4927//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/4927//console

This message is automatically generated.

 HADOOP_HOME should not be required to be set to be able to launch commands 
 using hadoop.util.Shell
 --

 Key: HADOOP-9422
 URL: https://issues.apache.org/jira/browse/HADOOP-9422
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Hitesh Shah
Assignee: Arpit Gupta
 Attachments: HADOOP-9422.patch


 Not sure why this is an enforced requirement especially in cases where a 
 deployment is done using multiple tar-balls ( one each for 
 common/hdfs/mapreduce/yarn ). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor

2014-10-15 Thread Konstantin Shvachko (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172644#comment-14172644
 ] 

Konstantin Shvachko commented on HADOOP-6857:
-

Byron, Could you post some examples (printouts) in support of your use case.

??if we setrep 10 a file when we only have 3 datanodes, -du will still multiply 
fileLength * 10??

This seems to be consistent with how quotas are calculated. DU should report 
numbers consistent with the quotas. We can change diskSpaceConsumed() if people 
want more precise usage info both for du and quotas, but at this point it will 
be a change in semantics. Let's keep it the current way.

 FsShell should report raw disk usage including replication factor
 -

 Key: HADOOP-6857
 URL: https://issues.apache.org/jira/browse/HADOOP-6857
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Reporter: Alex Kozlov
Assignee: Byron Wong
 Attachments: HADOOP-6857.patch, show-space-consumed.txt


 Currently FsShell report HDFS usage with hadoop fs -dus path command.  
 Since replication level is per file level, it would be nice to add raw disk 
 usage including the replication factor (maybe hadoop fs -dus -raw path?). 
  This will allow to assess resource usage more accurately.  -- Alex K



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11182) GraphiteSink emits wrong timestamps


 [ 
https://issues.apache.org/jira/browse/HADOOP-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sascha Coenen updated HADOOP-11182:
---
Status: Open  (was: Patch Available)

 GraphiteSink emits wrong timestamps
 ---

 Key: HADOOP-11182
 URL: https://issues.apache.org/jira/browse/HADOOP-11182
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.5.1, 2.5.0
Reporter: Sascha Coenen
 Attachments: HADOOP-11182-GraphiteSink-v1.patch, HADOOP-11182-v2.patch


 the org.apache.hadoop.metrics2.sink.GraphiteSink class emits metrics at the 
 configured time period, but the timestamps written only change every 128 
 seconds, even it the configured time period in the configuration file is much 
 shorter.
 This is due to a bug in line 93:
 {code:java}
 092// Round the timestamp to second as Graphite accepts it in 
 such format.
 093int timestamp = Math.round(record.timestamp() / 1000.0f);
 {code}
 The timestamp property is a long and is divided by a float which yields a 
 result that is not precise enough and yields same valued results for 
 timestamps that lie up to 128 seconds apart. Also, the result is then written 
 into an int variable.
 One solution would be to divide by 1000.0d, but the best fix would be to not 
 even convert to a decimal format in the first place. Instead one could 
 replace the line with the following:
 {code:java}
long timestamp = record.timestamp() / 1000L;
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11182) GraphiteSink emits wrong timestamps


 [ 
https://issues.apache.org/jira/browse/HADOOP-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sascha Coenen updated HADOOP-11182:
---
Status: Patch Available  (was: Open)

 GraphiteSink emits wrong timestamps
 ---

 Key: HADOOP-11182
 URL: https://issues.apache.org/jira/browse/HADOOP-11182
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.5.1, 2.5.0
Reporter: Sascha Coenen
 Attachments: HADOOP-11182-GraphiteSink-v1.patch, 
 HADOOP-11182-v2.patch, HADOOP-11182-v3.patch


 the org.apache.hadoop.metrics2.sink.GraphiteSink class emits metrics at the 
 configured time period, but the timestamps written only change every 128 
 seconds, even it the configured time period in the configuration file is much 
 shorter.
 This is due to a bug in line 93:
 {code:java}
 092// Round the timestamp to second as Graphite accepts it in 
 such format.
 093int timestamp = Math.round(record.timestamp() / 1000.0f);
 {code}
 The timestamp property is a long and is divided by a float which yields a 
 result that is not precise enough and yields same valued results for 
 timestamps that lie up to 128 seconds apart. Also, the result is then written 
 into an int variable.
 One solution would be to divide by 1000.0d, but the best fix would be to not 
 even convert to a decimal format in the first place. Instead one could 
 replace the line with the following:
 {code:java}
long timestamp = record.timestamp() / 1000L;
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11182) GraphiteSink emits wrong timestamps


 [ 
https://issues.apache.org/jira/browse/HADOOP-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sascha Coenen updated HADOOP-11182:
---
Attachment: HADOOP-11182-v3.patch

 GraphiteSink emits wrong timestamps
 ---

 Key: HADOOP-11182
 URL: https://issues.apache.org/jira/browse/HADOOP-11182
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.5.0, 2.5.1
Reporter: Sascha Coenen
 Attachments: HADOOP-11182-GraphiteSink-v1.patch, 
 HADOOP-11182-v2.patch, HADOOP-11182-v3.patch


 the org.apache.hadoop.metrics2.sink.GraphiteSink class emits metrics at the 
 configured time period, but the timestamps written only change every 128 
 seconds, even it the configured time period in the configuration file is much 
 shorter.
 This is due to a bug in line 93:
 {code:java}
 092// Round the timestamp to second as Graphite accepts it in 
 such format.
 093int timestamp = Math.round(record.timestamp() / 1000.0f);
 {code}
 The timestamp property is a long and is divided by a float which yields a 
 result that is not precise enough and yields same valued results for 
 timestamps that lie up to 128 seconds apart. Also, the result is then written 
 into an int variable.
 One solution would be to divide by 1000.0d, but the best fix would be to not 
 even convert to a decimal format in the first place. Instead one could 
 replace the line with the following:
 {code:java}
long timestamp = record.timestamp() / 1000L;
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11182) GraphiteSink emits wrong timestamps

[
https://issues.apache.org/jira/browse/HADOOP-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172668#comment-14172668
]

Sascha Coenen commented on HADOOP-11182:

Ah. I found the right Unit test class now and moved the new test into that one.
Resubmitting the patch and keeping my fingers crossed.
BTW:
I'm still unable to make use of any local patch testing. The run test-patch
takes like forever and outputs many tests failures but nothing that would
resemble the QA report. The Jenkins console output doesn't contain any findbug
warnings. It just says in there that the findbug report got stored as a local
file on Jenkins which I cannot access.

GraphiteSink emits wrong timestamps
---

Key: HADOOP-11182
URL: https://issues.apache.org/jira/browse/HADOOP-11182
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.5.0, 2.5.1
Reporter: Sascha Coenen
Attachments: HADOOP-11182-GraphiteSink-v1.patch,
HADOOP-11182-v2.patch, HADOOP-11182-v3.patch

the org.apache.hadoop.metrics2.sink.GraphiteSink class emits metrics at the
configured time period, but the timestamps written only change every 128
seconds, even it the configured time period in the configuration file is much
shorter.
This is due to a bug in line 93:
{code:java}
092// Round the timestamp to second as Graphite accepts it in
such format.
093int timestamp = Math.round(record.timestamp() / 1000.0f);
{code}
The timestamp property is a long and is divided by a float which yields a
result that is not precise enough and yields same valued results for
timestamps that lie up to 128 seconds apart. Also, the result is then written
into an int variable.
One solution would be to divide by 1000.0d, but the best fix would be to not
even convert to a decimal format in the first place. Instead one could
replace the line with the following:
{code:java}
long timestamp = record.timestamp() / 1000L;
{code}

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11182) GraphiteSink emits wrong timestamps


[ 
https://issues.apache.org/jira/browse/HADOOP-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172751#comment-14172751
 ] 

Hadoop QA commented on HADOOP-11182:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12675064/HADOOP-11182-v3.patch
  against trunk revision 1862064.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 2 new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/4928//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/4928//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/4928//console

This message is automatically generated.

 GraphiteSink emits wrong timestamps
 ---

 Key: HADOOP-11182
 URL: https://issues.apache.org/jira/browse/HADOOP-11182
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.5.0, 2.5.1
Reporter: Sascha Coenen
 Attachments: HADOOP-11182-GraphiteSink-v1.patch, 
 HADOOP-11182-v2.patch, HADOOP-11182-v3.patch


 the org.apache.hadoop.metrics2.sink.GraphiteSink class emits metrics at the 
 configured time period, but the timestamps written only change every 128 
 seconds, even it the configured time period in the configuration file is much 
 shorter.
 This is due to a bug in line 93:
 {code:java}
 092// Round the timestamp to second as Graphite accepts it in 
 such format.
 093int timestamp = Math.round(record.timestamp() / 1000.0f);
 {code}
 The timestamp property is a long and is divided by a float which yields a 
 result that is not precise enough and yields same valued results for 
 timestamps that lie up to 128 seconds apart. Also, the result is then written 
 into an int variable.
 One solution would be to divide by 1000.0d, but the best fix would be to not 
 even convert to a decimal format in the first place. Instead one could 
 replace the line with the following:
 {code:java}
long timestamp = record.timestamp() / 1000L;
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor

2014-10-15 Thread Byron Wong (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172771#comment-14172771
 ] 

Byron Wong commented on HADOOP-6857:


*Scenario 2*: we still have snapshottable directory /test with same file 
a. We then create a fresh snapshot ss1. We then run {{hadoop fs -rm 
-skipTrash /test/a}}.
{{hadoop fs -du /test}} gives an empty output, as expected.
{{hadoop fs -du -s /test}} outputs:
{code}
41  123  /test
{code}
which makes sense, given that we know about the existence of the snapshot.
However, when we run {{hadoop fs -du -s /test/.snapshot/ss1}}, we get:
{code}
0  0  /test/.snapshot/ss1
{code}
This is inconsistent with the numbers we get when we run {{hadoop fs -du 
/test/.snapshot/ss1}}:
{code}
41  123  /test/.snapshot/ss1/a
{code}
Upon further investigation, we see that running {{hadoop fs -du -s 
/test/.snapshot/anySnapshot}} gives us the information about the current state 
of the real directory. This means that {{hadoop fs -du -s 
/test/.snapshot/anySnapshot}} is equivalent to running {{hadoop fs -du /test/}} 
and adding the numbers up, which is non-intuitive.
For example, let's add a 2 byte, 3 replication file /test/1 (/test/a is still 
deleted). Now {{hadoop fs -du -s /test/.snapshot/ss1}} gives us:
{code}
2  6  /test/.snapshot/ss1
{code}
whereas the results of {{hadoop fs -du /test/.snapshot/ss1}} remains the same:
{code}
41  123  /test/.snapshot/ss1/a
{code}

 FsShell should report raw disk usage including replication factor
 -

 Key: HADOOP-6857
 URL: https://issues.apache.org/jira/browse/HADOOP-6857
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Reporter: Alex Kozlov
Assignee: Byron Wong
 Attachments: HADOOP-6857.patch, show-space-consumed.txt


 Currently FsShell report HDFS usage with hadoop fs -dus path command.  
 Since replication level is per file level, it would be nice to add raw disk 
 usage including the replication factor (maybe hadoop fs -dus -raw path?). 
  This will allow to assess resource usage more accurately.  -- Alex K



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-9422) HADOOP_HOME should not be required to be set to be able to launch commands using hadoop.util.Shell

2014-10-15 Thread Jason Lowe (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172802#comment-14172802
 ] 

Jason Lowe commented on HADOOP-9422:


I think the findbugs and test failures are unrelated.

My only concern with the patch is the original author went out of their way to 
make sure we only computed the value once, and with this patch it will be 
recomputed each time getHadoopHome() is called.  Wondering if it's worth only 
computing it once, e.g.: use System.setProperty(hadoop.home.dir, ...) if that 
property is not already set so subsequent calls can do a quick lookup without 
resorting to filesystem operations in the worst-case every time.

 HADOOP_HOME should not be required to be set to be able to launch commands 
 using hadoop.util.Shell
 --

 Key: HADOOP-9422
 URL: https://issues.apache.org/jira/browse/HADOOP-9422
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Hitesh Shah
Assignee: Arpit Gupta
 Attachments: HADOOP-9422.patch


 Not sure why this is an enforced requirement especially in cases where a 
 deployment is done using multiple tar-balls ( one each for 
 common/hdfs/mapreduce/yarn ). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11194) Ignore .keep files

2014-10-15 Thread Karthik Kambatla (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172851#comment-14172851
 ] 

Karthik Kambatla commented on HADOOP-11194:
---

[~ste...@apache.org] - does that latest patch look good to you? 

 Ignore .keep files
 --

 Key: HADOOP-11194
 URL: https://issues.apache.org/jira/browse/HADOOP-11194
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.5.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: hadoop-11194.patch, hadoop-11194.patch


 Given we don't need to keep empty directories, I suppose we can get rid of 
 the .keep files. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11194) Ignore .keep files

2014-10-15 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172854#comment-14172854
 ] 

Steve Loughran commented on HADOOP-11194:
-

+1

FWIW my concern was that once the .keep in the test resources went away, 
test/resources would go away, which would confuse mvn. If that happens, I can 
stick a different stub file in there, some .properties thing

 Ignore .keep files
 --

 Key: HADOOP-11194
 URL: https://issues.apache.org/jira/browse/HADOOP-11194
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.5.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: hadoop-11194.patch, hadoop-11194.patch


 Given we don't need to keep empty directories, I suppose we can get rid of 
 the .keep files. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11194) Ignore .keep files

2014-10-15 Thread Karthik Kambatla (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172866#comment-14172866
 ] 

Karthik Kambatla commented on HADOOP-11194:
---

Or, you could use dummy.keep file and exclude that from the RAT check? 

You okay with me committing this, I can review any after-effects relatively 
soon? 

 Ignore .keep files
 --

 Key: HADOOP-11194
 URL: https://issues.apache.org/jira/browse/HADOOP-11194
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.5.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: hadoop-11194.patch, hadoop-11194.patch


 Given we don't need to keep empty directories, I suppose we can get rid of 
 the .keep files. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11207) DelegationTokenAuthenticationHandler needs to support DT operations for proxy user

Zhijie Shen created HADOOP-11207:


 Summary: DelegationTokenAuthenticationHandler needs to support DT 
operations for proxy user
 Key: HADOOP-11207
 URL: https://issues.apache.org/jira/browse/HADOOP-11207
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Reporter: Zhijie Shen
Assignee: Zhijie Shen


Currently, DelegationTokenAuthenticationHandler only support DT operations for 
the request user after it passes the authentication. However, it should also 
support the request user to do DT operations on behalf of the proxy user.

Timeline server is using the authentication filter for DT operations instead of 
traditional RPC-based ones. It needs this feature to enable the proxy user to 
use the timeline service (YARN-2676).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11207) DelegationTokenAuthenticationHandler needs to support DT operations for proxy user


 [ 
https://issues.apache.org/jira/browse/HADOOP-11207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijie Shen updated HADOOP-11207:
-
Attachment: HADOOP-11207.1.patch

Create a patch to make the dt auth handler to support proxy user.

 DelegationTokenAuthenticationHandler needs to support DT operations for proxy 
 user
 --

 Key: HADOOP-11207
 URL: https://issues.apache.org/jira/browse/HADOOP-11207
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: HADOOP-11207.1.patch


 Currently, DelegationTokenAuthenticationHandler only support DT operations 
 for the request user after it passes the authentication. However, it should 
 also support the request user to do DT operations on behalf of the proxy user.
 Timeline server is using the authentication filter for DT operations instead 
 of traditional RPC-based ones. It needs this feature to enable the proxy user 
 to use the timeline service (YARN-2676).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11207) DelegationTokenAuthenticationHandler needs to support DT operations for proxy user


 [ 
https://issues.apache.org/jira/browse/HADOOP-11207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijie Shen updated HADOOP-11207:
-
Status: Patch Available  (was: Open)

 DelegationTokenAuthenticationHandler needs to support DT operations for proxy 
 user
 --

 Key: HADOOP-11207
 URL: https://issues.apache.org/jira/browse/HADOOP-11207
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: HADOOP-11207.1.patch


 Currently, DelegationTokenAuthenticationHandler only support DT operations 
 for the request user after it passes the authentication. However, it should 
 also support the request user to do DT operations on behalf of the proxy user.
 Timeline server is using the authentication filter for DT operations instead 
 of traditional RPC-based ones. It needs this feature to enable the proxy user 
 to use the timeline service (YARN-2676).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11182) GraphiteSink emits wrong timestamps

[
https://issues.apache.org/jira/browse/HADOOP-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173091#comment-14173091
]

Ravi Prakash commented on HADOOP-11182:
---

Thanks a lot Sascha! The patch looks good to me. I'll commit it shortly.
Its strange that you are unable to run test-patch successfully. On my laptop it
usually takes 15-20 minutes. I was then able to see the file
newPatchFindbugsWarninghadoop-common.html in /tmp . I am uploading what was
produced for your perusal (which btw I checked had no instance of metrics2) . I
did get 67 warnings but I'm guessing that's probably an error in the
test-patch.

Its probably worthwhile to setup your dev environment properly if you plan on
continuing to contribute. Unfortunately it is a bit of a chore right now :(

GraphiteSink emits wrong timestamps
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11182) GraphiteSink emits wrong timestamps


 [ 
https://issues.apache.org/jira/browse/HADOOP-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated HADOOP-11182:
--
Fix Version/s: 2.6.0

 GraphiteSink emits wrong timestamps
 ---

 Key: HADOOP-11182
 URL: https://issues.apache.org/jira/browse/HADOOP-11182
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.5.0, 2.5.1
Reporter: Sascha Coenen
 Fix For: 2.6.0

 Attachments: HADOOP-11182-GraphiteSink-v1.patch, 
 HADOOP-11182-v2.patch, HADOOP-11182-v3.patch


 the org.apache.hadoop.metrics2.sink.GraphiteSink class emits metrics at the 
 configured time period, but the timestamps written only change every 128 
 seconds, even it the configured time period in the configuration file is much 
 shorter.
 This is due to a bug in line 93:
 {code:java}
 092// Round the timestamp to second as Graphite accepts it in 
 such format.
 093int timestamp = Math.round(record.timestamp() / 1000.0f);
 {code}
 The timestamp property is a long and is divided by a float which yields a 
 result that is not precise enough and yields same valued results for 
 timestamps that lie up to 128 seconds apart. Also, the result is then written 
 into an int variable.
 One solution would be to divide by 1000.0d, but the best fix would be to not 
 even convert to a decimal format in the first place. Instead one could 
 replace the line with the following:
 {code:java}
long timestamp = record.timestamp() / 1000L;
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11182) GraphiteSink emits wrong timestamps


 [ 
https://issues.apache.org/jira/browse/HADOOP-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated HADOOP-11182:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks a lot Sacha! This patch is now committed to branch-2.6, branch-2 and 
trunk

 GraphiteSink emits wrong timestamps
 ---

 Key: HADOOP-11182
 URL: https://issues.apache.org/jira/browse/HADOOP-11182
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.5.0, 2.5.1
Reporter: Sascha Coenen
 Fix For: 2.6.0

 Attachments: HADOOP-11182-GraphiteSink-v1.patch, 
 HADOOP-11182-v2.patch, HADOOP-11182-v3.patch


 the org.apache.hadoop.metrics2.sink.GraphiteSink class emits metrics at the 
 configured time period, but the timestamps written only change every 128 
 seconds, even it the configured time period in the configuration file is much 
 shorter.
 This is due to a bug in line 93:
 {code:java}
 092// Round the timestamp to second as Graphite accepts it in 
 such format.
 093int timestamp = Math.round(record.timestamp() / 1000.0f);
 {code}
 The timestamp property is a long and is divided by a float which yields a 
 result that is not precise enough and yields same valued results for 
 timestamps that lie up to 128 seconds apart. Also, the result is then written 
 into an int variable.
 One solution would be to divide by 1000.0d, but the best fix would be to not 
 even convert to a decimal format in the first place. Instead one could 
 replace the line with the following:
 {code:java}
long timestamp = record.timestamp() / 1000L;
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11182) GraphiteSink emits wrong timestamps


 [ 
https://issues.apache.org/jira/browse/HADOOP-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated HADOOP-11182:
--
Attachment: newPatchFindbugsWarningshadoop-common.html

 GraphiteSink emits wrong timestamps
 ---

 Key: HADOOP-11182
 URL: https://issues.apache.org/jira/browse/HADOOP-11182
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.5.0, 2.5.1
Reporter: Sascha Coenen
 Fix For: 2.6.0

 Attachments: HADOOP-11182-GraphiteSink-v1.patch, 
 HADOOP-11182-v2.patch, HADOOP-11182-v3.patch, 
 newPatchFindbugsWarningshadoop-common.html


 the org.apache.hadoop.metrics2.sink.GraphiteSink class emits metrics at the 
 configured time period, but the timestamps written only change every 128 
 seconds, even it the configured time period in the configuration file is much 
 shorter.
 This is due to a bug in line 93:
 {code:java}
 092// Round the timestamp to second as Graphite accepts it in 
 such format.
 093int timestamp = Math.round(record.timestamp() / 1000.0f);
 {code}
 The timestamp property is a long and is divided by a float which yields a 
 result that is not precise enough and yields same valued results for 
 timestamps that lie up to 128 seconds apart. Also, the result is then written 
 into an int variable.
 One solution would be to divide by 1000.0d, but the best fix would be to not 
 even convert to a decimal format in the first place. Instead one could 
 replace the line with the following:
 {code:java}
long timestamp = record.timestamp() / 1000L;
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11182) GraphiteSink emits wrong timestamps


[ 
https://issues.apache.org/jira/browse/HADOOP-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173108#comment-14173108
 ] 

Zhijie Shen commented on HADOOP-11182:
--

Hi, [~raviprak]. It seems that the jira number is wrongly edited in the commit.
{code}
HADOOP-11181. GraphiteSink emits wrong timestamps (Sascha Coenen via 
raviprak)
{code}

 GraphiteSink emits wrong timestamps
 ---

 Key: HADOOP-11182
 URL: https://issues.apache.org/jira/browse/HADOOP-11182
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.5.0, 2.5.1
Reporter: Sascha Coenen
 Fix For: 2.6.0

 Attachments: HADOOP-11182-GraphiteSink-v1.patch, 
 HADOOP-11182-v2.patch, HADOOP-11182-v3.patch, 
 newPatchFindbugsWarningshadoop-common.html


 the org.apache.hadoop.metrics2.sink.GraphiteSink class emits metrics at the 
 configured time period, but the timestamps written only change every 128 
 seconds, even it the configured time period in the configuration file is much 
 shorter.
 This is due to a bug in line 93:
 {code:java}
 092// Round the timestamp to second as Graphite accepts it in 
 such format.
 093int timestamp = Math.round(record.timestamp() / 1000.0f);
 {code}
 The timestamp property is a long and is divided by a float which yields a 
 result that is not precise enough and yields same valued results for 
 timestamps that lie up to 128 seconds apart. Also, the result is then written 
 into an int variable.
 One solution would be to divide by 1000.0d, but the best fix would be to not 
 even convert to a decimal format in the first place. Instead one could 
 replace the line with the following:
 {code:java}
long timestamp = record.timestamp() / 1000L;
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11181) o.a.h.security.token.delegation.DelegationTokenManager should be more generalized to handle other DelegationTokenIdentifier


[ 
https://issues.apache.org/jira/browse/HADOOP-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173116#comment-14173116
 ] 

Hudson commented on HADOOP-11181:
-

FAILURE: Integrated in Hadoop-trunk-Commit #6268 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/6268/])
HADOOP-11181. GraphiteSink emits wrong timestamps (Sascha Coenen via raviprak) 
(raviprak: rev 466f08792f11c2f95bf293ac9b6facd7018a5bb8)
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGraphiteMetrics.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/GraphiteSink.java
* hadoop-common-project/hadoop-common/CHANGES.txt


 o.a.h.security.token.delegation.DelegationTokenManager should be more 
 generalized to handle other DelegationTokenIdentifier
 ---

 Key: HADOOP-11181
 URL: https://issues.apache.org/jira/browse/HADOOP-11181
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.6.0

 Attachments: HADOOP-11181.1.patch, HADOOP-11181.2.patch, 
 HADOOP-11181.3.patch, HADOOP-11181.4.patch, HADOOP-11181.5.patch


 While DelegationTokenManager can set external secretManager, it have the 
 assumption that the token is going to be 
 o.a.h.security.token.delegation.DelegationTokenIdentifier, and use 
 DelegationTokenIdentifier method to decode a token. 
 {code}
   @SuppressWarnings(unchecked)
   public UserGroupInformation verifyToken(TokenDelegationTokenIdentifier
   token) throws IOException {
 ByteArrayInputStream buf = new 
 ByteArrayInputStream(token.getIdentifier());
 DataInputStream dis = new DataInputStream(buf);
 DelegationTokenIdentifier id = new DelegationTokenIdentifier(tokenKind);
 id.readFields(dis);
 dis.close();
 secretManager.verifyToken(id, token.getPassword());
 return id.getUser();
   }
 {code}
 It's not going to work it the token kind is other than 
 web.DelegationTokenIdentifier. For example, RM want to reuse it but hook it 
 to RMDelegationTokenSecretManager and RMDelegationTokenIdentifier, which has 
 the customized way to decode a token.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11207) DelegationTokenAuthenticationHandler needs to support DT operations for proxy user


[ 
https://issues.apache.org/jira/browse/HADOOP-11207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173120#comment-14173120
 ] 

Hadoop QA commented on HADOOP-11207:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12675130/HADOOP-11207.1.patch
  against trunk revision f19771a.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 2 new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The following test timeouts occurred in 
hadoop-common-project/hadoop-common:

org.apache.hadoop.http.TestHttpServerLifecycle

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/4929//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/4929//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/4929//console

This message is automatically generated.

 DelegationTokenAuthenticationHandler needs to support DT operations for proxy 
 user
 --

 Key: HADOOP-11207
 URL: https://issues.apache.org/jira/browse/HADOOP-11207
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: HADOOP-11207.1.patch


 Currently, DelegationTokenAuthenticationHandler only support DT operations 
 for the request user after it passes the authentication. However, it should 
 also support the request user to do DT operations on behalf of the proxy user.
 Timeline server is using the authentication filter for DT operations instead 
 of traditional RPC-based ones. It needs this feature to enable the proxy user 
 to use the timeline service (YARN-2676).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11207) DelegationTokenAuthenticationHandler needs to support DT operations for proxy user


[ 
https://issues.apache.org/jira/browse/HADOOP-11207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173137#comment-14173137
 ] 

Zhijie Shen commented on HADOOP-11207:
--

The 2 findbugs warnings happened before in HADOOP-11181 too, which are not 
related.

TestHttpServerLifecycle is also an irrelevant test, and it passed locally

 DelegationTokenAuthenticationHandler needs to support DT operations for proxy 
 user
 --

 Key: HADOOP-11207
 URL: https://issues.apache.org/jira/browse/HADOOP-11207
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: HADOOP-11207.1.patch


 Currently, DelegationTokenAuthenticationHandler only support DT operations 
 for the request user after it passes the authentication. However, it should 
 also support the request user to do DT operations on behalf of the proxy user.
 Timeline server is using the authentication filter for DT operations instead 
 of traditional RPC-based ones. It needs this feature to enable the proxy user 
 to use the timeline service (YARN-2676).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-9992) Modify the NN loadGenerator to optionally run as a MapReduce job

2014-10-15 Thread Brandon Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-9992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Li updated HADOOP-9992:
---
Attachment: HADOOP-9992.v5.patch

Uploaded a patch to fix the usage message.

 Modify the NN loadGenerator to optionally run as a MapReduce job
 

 Key: HADOOP-9992
 URL: https://issues.apache.org/jira/browse/HADOOP-9992
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Akshay Radia
Assignee: Akshay Radia
 Attachments: HADOOP-9992.004.patch, HADOOP-9992.v5.patch, 
 hadoop-9992-v2.patch, hadoop-9992-v3.patch, hadoop-9992-v4.patch, 
 hadoop-9992.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-9992) Modify the NN loadGenerator to optionally run as a MapReduce job