[jira] [Commented] (HADOOP-13395) Enhance TestKMSAudit

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398752#comment-15398752
 ] 

Hadoop QA commented on HADOOP-13395:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m  
3s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 15s{color} | {color:orange} hadoop-common-project/hadoop-kms: The patch 
generated 6 new + 14 unchanged - 0 fixed = 20 total (was 14) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
12s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 23s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820870/HADOOP-13395.03.patch 
|
| JIRA Issue | HADOOP-13395 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 7771192a3652 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 8ebf2e9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10115/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-kms.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10115/testReport/ |
| modules | C: hadoop-common-project/hadoop-kms U: 
hadoop-common-project/hadoop-kms |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10115/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Enhance TestKMSAudit
> 
>
> Key: HADOOP-13395
> URL: https://issues.apache.org/jira/browse/HADOOP-13395
> Project: Hadoop Common
>  Issue Type: Test
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Minor
>

[jira] [Commented] (HADOOP-13440) FileContext does not react on changing umask via configuration

[Akira Ajisaka (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398743#comment-15398743
 ] 

Akira Ajisaka commented on HADOOP-13440:


Hi [~ste...@apache.org], would you review this?

> FileContext does not react on changing umask via configuration
> --
>
> Key: HADOOP-13440
> URL: https://issues.apache.org/jira/browse/HADOOP-13440
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Akira Ajisaka
> Attachments: YARN-5425.00.patch
>
>
> After HADOOP-13073, RawLocalFileSystem does react on changing umask but 
> FileContext does not react on changing umask via configuration. 
> TestDirectoryCollection fails by the inconsistent behavior.
> {code}
> java.lang.AssertionError: local dir parent not created with proper 
> permissions expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.TestDirectoryCollection.testCreateDirectories(TestDirectoryCollection.java:113)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13440) FileContext does not react on changing umask via configuration

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated HADOOP-13440:
---
Description: 
After HADOOP-13073, RawLocalFileSystem does react on changing umask but 
FileContext does not react on changing umask via configuration. 
TestDirectoryCollection fails by the inconsistent behavior.
{code}
java.lang.AssertionError: local dir parent not created with proper permissions 
expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at 
org.apache.hadoop.yarn.server.nodemanager.TestDirectoryCollection.testCreateDirectories(TestDirectoryCollection.java:113)
{code}


  was:
{code}
java.lang.AssertionError: local dir parent not created with proper permissions 
expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at 
org.apache.hadoop.yarn.server.nodemanager.TestDirectoryCollection.testCreateDirectories(TestDirectoryCollection.java:113)
{code}



> FileContext does not react on changing umask via configuration
> --
>
> Key: HADOOP-13440
> URL: https://issues.apache.org/jira/browse/HADOOP-13440
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Akira Ajisaka
> Attachments: YARN-5425.00.patch
>
>
> After HADOOP-13073, RawLocalFileSystem does react on changing umask but 
> FileContext does not react on changing umask via configuration. 
> TestDirectoryCollection fails by the inconsistent behavior.
> {code}
> java.lang.AssertionError: local dir parent not created with proper 
> permissions expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.TestDirectoryCollection.testCreateDirectories(TestDirectoryCollection.java:113)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13440) FileContext does not react on changing umask via configuration

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated HADOOP-13440:
---
Summary: FileContext does not react on changing umask via configuration  
(was: TestDirectoryCollection.testCreateDirectories failed)

> FileContext does not react on changing umask via configuration
> --
>
> Key: HADOOP-13440
> URL: https://issues.apache.org/jira/browse/HADOOP-13440
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Akira Ajisaka
> Attachments: YARN-5425.00.patch
>
>
> {code}
> java.lang.AssertionError: local dir parent not created with proper 
> permissions expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.TestDirectoryCollection.testCreateDirectories(TestDirectoryCollection.java:113)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Moved] (HADOOP-13440) TestDirectoryCollection.testCreateDirectories failed

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka moved YARN-5425 to HADOOP-13440:
--

Target Version/s: 3.0.0-alpha2  (was: 3.0.0-alpha2)
 Component/s: (was: nodemanager)
 Key: HADOOP-13440  (was: YARN-5425)
 Project: Hadoop Common  (was: Hadoop YARN)

> TestDirectoryCollection.testCreateDirectories failed
> 
>
> Key: HADOOP-13440
> URL: https://issues.apache.org/jira/browse/HADOOP-13440
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Akira Ajisaka
> Attachments: YARN-5425.00.patch
>
>
> {code}
> java.lang.AssertionError: local dir parent not created with proper 
> permissions expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.TestDirectoryCollection.testCreateDirectories(TestDirectoryCollection.java:113)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13395) Enhance TestKMSAudit

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HADOOP-13395:
---
Attachment: HADOOP-13395.03.patch

Thanks a lot [~andrew.wang] for the review! Getting rid of {{sleep}} is 
definitely better. :)
Patch 3 is attached, addressing your comments. Please take a look and share 
your thoughts.

The major flakiness of the past occurrence is due to the race in different 
threads after an UNAUTHRIZED log - one UNAUTH log itself, and one aggregated OK 
log on the removal listener thread after UNAUTH. I put a comment in the test to 
explain it.



> Enhance TestKMSAudit
> 
>
> Key: HADOOP-13395
> URL: https://issues.apache.org/jira/browse/HADOOP-13395
> Project: Hadoop Common
>  Issue Type: Test
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Minor
> Attachments: HADOOP-13395.01.patch, HADOOP-13395.02.patch, 
> HADOOP-13395.03.patch
>
>
> This jira serves the goals:
> - Enhance existing test cases in TestKMSAudit, to rule out flakiness.
> - Add a new test case about formatting for different events.
> This will help us ensure audit log compatibility when we add a new log format 
> to KMS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13208) S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories

[Aaron Fabbri (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398645#comment-15398645
 ] 

Aaron Fabbri commented on HADOOP-13208:
---

I spent a good amount of time this evening going over patch #18.  Overall looks 
good, considering the previous JIRA included most of the test code.  Are there 
any unit tests still missing (I'm willing to help if so)?

I'm +1 (non-binding).

Some random comments / questions:

{code}
+
 } else {
   LOG.debug("Adding: rd (not a dir): {}", f);
+  result = new ArrayList<>(1);
   result.add(fileStatus);
{code}
How about just returning the singleton array here, to reduce garbage a bit?

{code}
+  private ListObjectsRequest createListObjectsRequest(String key,
+  String delimiter) {
+ListObjectsRequest request = new ListObjectsRequest();
+request.setBucketName(bucket);
+request.setMaxKeys(maxKeys);
+request.setPrefix(key);
+if (delimiter != null) {
+  request.setDelimiter(delimiter);
+}
+return request;
+  }
{code}
Good factorization.

{code}
+  private final class FileStatusListingIterator
+  implements RemoteIterator {
+
+/** Source of objects. */
+private final ObjectListingIterator source;
{code}

I feel like we need to break up S3AFileSystem.java some.  It is getting pretty
large.  We should take any easy opportunities to pull out classes into separate
files.  I'm fine with doing this as a follow-up JIRA.

{code}
+ */
+private boolean requestNextBatch() throws IOException {
+  // look for more object listing batches being available
+  while (source.hasNext()) {
+// if available, retrieve it and build the next status
+if (buildNextStatusBatch(source.next())) {
{code}
Yeah, this logic is non-trivial.  This while loop is key.

{code}
-   */
-  static class AWSAccessKeys {
-private String accessKey = null;
-private String accessSecret = null;
{code}
Bonus deadc0de removal, ok.

{code}
+  @Override
+  protected Configuration createConfiguration() {
+Configuration conf = super.createConfiguration();
+S3ATestUtils.disableFilesystemCaching(conf);
{code}
What is this for?  I see it was introduced in HADOOP-13131, but don't see
any usage of the configuration flag it sets (fs.s3a.impl.disable.cache).


> S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the 
> pseudo-tree of directories
> 
>
> Key: HADOOP-13208
> URL: https://issues.apache.org/jira/browse/HADOOP-13208
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13208-branch-2-001.patch, 
> HADOOP-13208-branch-2-007.patch, HADOOP-13208-branch-2-008.patch, 
> HADOOP-13208-branch-2-009.patch, HADOOP-13208-branch-2-010.patch, 
> HADOOP-13208-branch-2-011.patch, HADOOP-13208-branch-2-012.patch, 
> HADOOP-13208-branch-2-017.patch, HADOOP-13208-branch-2-018.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> A major cost in split calculation against object stores turns out be listing 
> the directory tree itself. That's because against S3, it takes S3A two HEADs 
> and two lists to list the content of any directory path (2 HEADs + 1 list for 
> getFileStatus(); the next list to query the contents).
> Listing a directory could be improved slightly by combining the final two 
> listings. However, a listing of a directory tree will still be 
> O(directories). In contrast, a recursive {{listFiles()}} operation should be 
> implementable by a bulk listing of all descendant paths; one List operation 
> per thousand descendants. 
> As the result of this call is an iterator, the ongoing listing can be 
> implemented within the iterator itself



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-13434) Add quoting to Shell class

[Larry McCay (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398599#comment-15398599
 ] 

Larry McCay edited comment on HADOOP-13434 at 7/29/16 2:22 AM:
---

LGTM - removes bash where there are alternatives and quotes the rest.
+1 - pending jenkins green light.



was (Author: lmccay):
LGTM - removes bash where there are alternatives and quotes the rest.
+1


> Add quoting to Shell class
> --
>
> Key: HADOOP-13434
> URL: https://issues.apache.org/jira/browse/HADOOP-13434
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HADOOP-13434.patch, HADOOP-13434.patch
>
>
> The Shell class makes assumptions that the parameters won't have spaces or 
> other special characters, even when it invokes bash.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13434) Add quoting to Shell class

[Larry McCay (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398599#comment-15398599
 ] 

Larry McCay commented on HADOOP-13434:
--

LGTM - removes bash where there are alternatives and quotes the rest.
+1


> Add quoting to Shell class
> --
>
> Key: HADOOP-13434
> URL: https://issues.apache.org/jira/browse/HADOOP-13434
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HADOOP-13434.patch, HADOOP-13434.patch
>
>
> The Shell class makes assumptions that the parameters won't have spaces or 
> other special characters, even when it invokes bash.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13081) add the ability to create multiple UGIs/subjects from one kerberos login

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398592#comment-15398592
 ] 

Sergey Shelukhin commented on HADOOP-13081:
---

Yes, it's possible to add the test. It fails however, probably due to problems 
with mocks, I will finish it tomorrow, need to run now. Fixed the rest.

> add the ability to create multiple UGIs/subjects from one kerberos login
> 
>
> Key: HADOOP-13081
> URL: https://issues.apache.org/jira/browse/HADOOP-13081
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HADOOP-13081.01.patch, HADOOP-13081.patch
>
>
> We have a scenario where we log in with kerberos as a certain user for some 
> tasks, but also want to add tokens to the resulting UGI that would be 
> specific to each task. We don't want to authenticate with kerberos for every 
> task.
> I am not sure how this can be accomplished with the existing UGI interface. 
> Perhaps some clone method would be helpful, similar to createProxyUser minus 
> the proxy stuff; or it could just relogin anew from ticket cache. 
> getUGIFromTicketCache seems like the best option in existing code, but there 
> doesn't appear to be a consistent way of handling ticket cache location - the 
> above method, that I only see called in test, is using a config setting that 
> is not used anywhere else, and the env variable for the location that is used 
> in the main ticket cache related methods is not set uniformly on all paths - 
> therefore, trying to find the correct ticket cache and passing it via the 
> config setting to getUGIFromTicketCache seems even hackier than doing the 
> clone via reflection ;) Moreover, getUGIFromTicketCache ignores the user 
> parameter on the main path - it logs a warning for multiple principals and 
> then logs in with first available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13381) KMS clients should use KMS Delegation Tokens from current UGI.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398565#comment-15398565
 ] 

Hudson commented on HADOOP-13381:
-

SUCCESS: Integrated in Hadoop-trunk-Commit #10176 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10176/])
HADOOP-13381. KMS clients should use KMS Delegation Tokens from current (xiao: 
rev 8ebf2e95d2053cb94c6ff87ca018811fe8276f2b)
* 
hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java


> KMS clients should use KMS Delegation Tokens from current UGI.
> --
>
> Key: HADOOP-13381
> URL: https://issues.apache.org/jira/browse/HADOOP-13381
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13381.01.patch, HADOOP-13381.02.patch, 
> HADOOP-13381.03.patch, HADOOP-13381.04.patch
>
>
> When {{/tmp}} is setup as an EZ, one may experience YARN log aggregation 
> failure after the very first KMS token is expired. The MR job itself runs 
> fine though.
> When this happens, YARN NodeManager's log will show 
> {{AuthenticationException}} with {{token is expired}} / {{token can't be 
> found in cache}}, depending on whether the expired token is removed by the 
> background or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13381) KMS clients should use KMS Delegation Tokens from current UGI.

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HADOOP-13381:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
   2.9.0
   Status: Resolved  (was: Patch Available)

Committed to trunk and branch-2. There is a minor conflict (imports) when 
backporting to branch-2, compiled and made sure {{TestKMS}} pass before pushing.

Thanks a lot [~asuresh] for the thoughtful reviews, and [~andrew.wang] for 
chiming in!

> KMS clients should use KMS Delegation Tokens from current UGI.
> --
>
> Key: HADOOP-13381
> URL: https://issues.apache.org/jira/browse/HADOOP-13381
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13381.01.patch, HADOOP-13381.02.patch, 
> HADOOP-13381.03.patch, HADOOP-13381.04.patch
>
>
> When {{/tmp}} is setup as an EZ, one may experience YARN log aggregation 
> failure after the very first KMS token is expired. The MR job itself runs 
> fine though.
> When this happens, YARN NodeManager's log will show 
> {{AuthenticationException}} with {{token is expired}} / {{token can't be 
> found in cache}}, depending on whether the expired token is removed by the 
> background or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13381) KMS clients should use KMS Delegation Tokens from current UGI.

[Xiao Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398537#comment-15398537
 ] 

Xiao Chen commented on HADOOP-13381:


Thanks Arun, committing this.

> KMS clients should use KMS Delegation Tokens from current UGI.
> --
>
> Key: HADOOP-13381
> URL: https://issues.apache.org/jira/browse/HADOOP-13381
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HADOOP-13381.01.patch, HADOOP-13381.02.patch, 
> HADOOP-13381.03.patch, HADOOP-13381.04.patch
>
>
> When {{/tmp}} is setup as an EZ, one may experience YARN log aggregation 
> failure after the very first KMS token is expired. The MR job itself runs 
> fine though.
> When this happens, YARN NodeManager's log will show 
> {{AuthenticationException}} with {{token is expired}} / {{token can't be 
> found in cache}}, depending on whether the expired token is removed by the 
> background or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13381) KMS clients should use KMS Delegation Tokens from current UGI.

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HADOOP-13381:
---
Summary: KMS clients should use KMS Delegation Tokens from current UGI.  
(was: KMS clients running in the same JVM should use updated KMS Delegation 
Token)

> KMS clients should use KMS Delegation Tokens from current UGI.
> --
>
> Key: HADOOP-13381
> URL: https://issues.apache.org/jira/browse/HADOOP-13381
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HADOOP-13381.01.patch, HADOOP-13381.02.patch, 
> HADOOP-13381.03.patch, HADOOP-13381.04.patch
>
>
> When {{/tmp}} is setup as an EZ, one may experience YARN log aggregation 
> failure after the very first KMS token is expired. The MR job itself runs 
> fine though.
> When this happens, YARN NodeManager's log will show 
> {{AuthenticationException}} with {{token is expired}} / {{token can't be 
> found in cache}}, depending on whether the expired token is removed by the 
> background or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12588) Fix intermittent test failure of TestGangliaMetrics

[Masatake Iwasaki (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398501#comment-15398501
 ] 

Masatake Iwasaki commented on HADOOP-12588:
---

I filed HADOOP-13439 for following up.

> Fix intermittent test failure of TestGangliaMetrics
> ---
>
> Key: HADOOP-12588
> URL: https://issues.apache.org/jira/browse/HADOOP-12588
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Tsuyoshi Ozawa
>Assignee: Masatake Iwasaki
> Fix For: 2.8.0, 2.7.3
>
> Attachments: HADOOP-12588.001.patch, HADOOP-12588.addendum.02.patch, 
> HADOOP-12588.addendum.03.patch, HADOOP-12588.addendum.patch
>
>
> Jenkins found this test failure on HADOOP-11149.
> {quote}
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.773 sec <<< 
> FAILURE! - in org.apache.hadoop.metrics2.impl.TestGangliaMetrics
> testGangliaMetrics2(org.apache.hadoop.metrics2.impl.TestGangliaMetrics)  Time 
> elapsed: 0.39 sec  <<< FAILURE!
> java.lang.AssertionError: Missing metrics: test.s1rec.Xxx
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at 
> org.apache.hadoop.metrics2.impl.TestGangliaMetrics.checkMetrics(TestGangliaMetrics.java:159)
>   at 
> org.apache.hadoop.metrics2.impl.TestGangliaMetrics.testGangliaMetrics2(TestGangliaMetrics.java:137)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13439) Fix race between TestMetricsSystemImpl and TestGangliaMetrics

[Masatake Iwasaki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HADOOP-13439:
--
Component/s: (was: t)

> Fix race between TestMetricsSystemImpl and TestGangliaMetrics
> -
>
> Key: HADOOP-13439
> URL: https://issues.apache.org/jira/browse/HADOOP-13439
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: test
>Reporter: Masatake Iwasaki
>Priority: Minor
>
> TestGangliaMetrics#testGangliaMetrics2 set *.period to 120 but 8 was used.
> {noformat}
> 2016-06-27 15:21:31,480 INFO  impl.MetricsSystemImpl 
> (MetricsSystemImpl.java:startTimer(375)) - Scheduled snapshot period at 8 
> second(s).
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13439) Fix race between TestMetricsSystemImpl and TestGangliaMetrics

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-13439:
-

 Summary: Fix race between TestMetricsSystemImpl and 
TestGangliaMetrics
 Key: HADOOP-13439
 URL: https://issues.apache.org/jira/browse/HADOOP-13439
 Project: Hadoop Common
  Issue Type: Bug
  Components: t, test
Reporter: Masatake Iwasaki
Priority: Minor


TestGangliaMetrics#testGangliaMetrics2 set *.period to 120 but 8 was used.
{noformat}
2016-06-27 15:21:31,480 INFO  impl.MetricsSystemImpl 
(MetricsSystemImpl.java:startTimer(375)) - Scheduled snapshot period at 8 
second(s).
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13403) AzureNativeFileSystem rename/delete performance improvements

[Chris Nauroth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398461#comment-15398461
 ] 

Chris Nauroth commented on HADOOP-13403:


Hello [~pattipaka].  Thank you for the patch.

I started code reviewing this while doing a full test run with the patch 
against my Azure Storage account.  I saw these failures:

{code}
Tests run: 67, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 591.899 sec 
<<< FAILURE! - in org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
testRenameSigleRenameException(org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads)
  Time elapsed: 7.062 sec  <<< ERROR!
java.lang.Exception: Unexpected exception, expected but 
was
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads.testRenameSigleRenameException(TestFileSystemOperationsWithThreads.java:630)

testDeleteSigleDeleteException(org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads)
  Time elapsed: 4.289 sec  <<< ERROR!
java.lang.Exception: Unexpected exception, expected but 
was
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads.testDeleteSigleDeleteException(TestFileSystemOperationsWithThreads.java:479)

testDeleteSigleDeleteFailure(org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads)
  Time elapsed: 4.403 sec  <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.junit.Assert.assertFalse(Assert.java:74)
at 
org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads.testDeleteSigleDeleteFailure(TestFileSystemOperationsWithThreads.java:455)
{code}

I'll halt my code review at this point so that you can investigate the test 
failures.  Here is my feedback so far, based on what I've read of the patch.

{code}
  threadCount = conf.getInt(config, defaultThreadCount);
{code}

I recommend restructuring this so that the thread count for delete and rename 
are read just once in {{NativeAzureFileSystem#initialize}} and stored to member 
variables.  Accessing {{Configuration}} can be expensive, and contributors 
reading the code generally expect to find all config access in {{initialize}}, 
not throughout other methods.

{code}
  // Enable flat listing option only if depth is unboubded and config 
{code}

Typo: "unbounded".

{code}
Date start = new Date();
// Do something.
Date end = new Date();
long duration = end.getTime() - start.getTime();
{code}

For all occurrences of logic like the above, please switch to using 
{{org.apache.hadoop.util.Time#monotonicNow()}}, which is a wrapper over the JDK 
[{{java.lang.System#nanoTime()}}|http://docs.oracle.com/javase/7/docs/api/java/lang/System.html#nanoTime()].
  By switching to this, the logic won't be prone to issues caused by clock 
drift or resetting time on the host.

{code}
private String threadIdPrefix = "AzureFileSytemThread";
{code}

Typo: "AzureFileSystemThread".

{code}
  enum AzureFileSystemOperation{
Unkown,
Delete,
Rename
  }
{code}

Typo: "Unknown".

{code}
  class AzureFileSystemThreadFactory implements ThreadFactory {
{code}

{code}
  public class AzureFileSystemThreadRunnable implements Runnable {
{code}

{{NativeAzureFileSystem}} is a pretty long class already.  I propose 
refactoring {{AzureFileSystemThreadFactory}} and 
{{AzureFileSystemThreadRunnable}} to top-level classes with package-private 
visibility in separate files.

{code}
if ((ioThreadPool = getThreadPool(threadCount, threadNamePrefix)) != 
null) {
{code}

I don't understand the null check.  {{getThreadPool}} cannot return {{null}}, 
because it simply calls the {{ThreadPoolExecutor}} constructor and returns the 
new object.  I see some tests mock it to return {{null}}, but since this 
condition cannot really happen, why do we need test coverage for it?  The same 
feedback applies to the exception handling.  I do not see any checked 
exceptions thrown from {{getThreadPool}}.  The {{ThreadPoolExecutor}} 
constructor can throw unchecked exceptions if the arguments don't make sense 
(e.g. negative thread pool size), but we can validate the sanity of those 
configuration parameters during {{initialize}} after following the feedback 
above about configuration handling.

{code}
  lastException = new IOException(operation + " failed as operation on 
subfolers and files failed.");
{code}

Typo: "subfolders".

{code}
  @Test(expected=IOException.class)
{code}

Please don't use this technique for writing 

[jira] [Commented] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398459#comment-15398459
 ] 

Mingliang Liu commented on HADOOP-13436:


Nice catch, [~xiaobingo]!

{{RetryPolicy}} states that
{quote}
Implementations of this interface should be immutable.
{quote}
which is not enough. Since the {{RetryPolicy}}'s equality is used by 
{{ConnectionId#equals()}}, every class that implements {{RetryPolicy}} should 
also override {{equals()}} method. For the sake of new retry policies, I 
suggest we also note this in the javadoc of {{RetryPolicy}}. Although I see 
very few immutable classes could perform like a charm without overriding 
{{equals()}}/{{hashCode()}} methods, it's not a strict rule to call a class 
_immutable_.

> RPC connections are leaking due to missing equals override in 
> RetryUtils#getDefaultRetryPolicy
> --
>
> Key: HADOOP-13436
> URL: https://issues.apache.org/jira/browse/HADOOP-13436
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Affects Versions: 2.7.1
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: repro.sh
>
>
> We've noticed RPC connections are increasing dramatically in a Kerberized 
> HDFS cluster with {noformat}dfs.client.retry.policy.enabled{noformat} 
> enabled. Internally,  Client#getConnection is doing lookup relying on 
> ConnectionId#equals, which composes checking Subclass-of-RetryPolicy#equals. 
> If subclasses of RetryPolicy neglect overriding RetryPolicy#equals, every 
> instances of RetryPolicy with equivalent fields' values (e.g. 
> MultipleLinearRandomRetry[6x1ms, 10x6ms]) will lead to a brand new 
> connection because the check will fall back to Object#equals.
> This is stack trace where RetryUtils#getDefaultRetryPolicy is called:
> {noformat}
> at 
> org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(RetryUtils.java:82)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:409)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:315)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:609)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.newDfsClient(WebHdfsHandler.java:272)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.onOpen(WebHdfsHandler.java:215)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.handle(WebHdfsHandler.java:135)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:117)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:114)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:52)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:32)
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> at 
> 

[jira] [Updated] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HADOOP-13436:
---
Description: 
We've noticed RPC connections are increasing dramatically in a Kerberized HDFS 
cluster with {noformat}dfs.client.retry.policy.enabled{noformat} enabled. 
Internally,  Client#getConnection is doing lookup relying on 
ConnectionId#equals, which composes checking Subclass-of-RetryPolicy#equals. If 
subclasses of RetryPolicy neglect overriding RetryPolicy#equals, every 
instances of RetryPolicy with equivalent fields' values (e.g. 
MultipleLinearRandomRetry[6x1ms, 10x6ms]) will lead to a brand new 
connection because the check will fall back to Object#equals.

This is stack trace where RetryUtils#getDefaultRetryPolicy is called:
{noformat}
at 
org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(RetryUtils.java:82)
at 
org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:409)
at 
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:315)
at 
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:678)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:609)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.newDfsClient(WebHdfsHandler.java:272)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.onOpen(WebHdfsHandler.java:215)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.handle(WebHdfsHandler.java:135)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:117)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:114)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114)
at 
org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:52)
at 
org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:32)
at 
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at 
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at 
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at 
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
{noformat}


Three options to fix the problem:
1. All subclasses of RetryPolicy must override equals and hashCode to deliver 
less discriminating equivalence relation, i.e. they are equal if they have 
meaningful equivalent fields' values (e.g. MultipleLinearRandomRetry[6x1ms, 
10x6ms])
2. Change ConnectionId#equals by removing RetryPolicy#equals compoment.
3. Let WebHDFS reuse the DFSClient.

  was:
We've noticed RPC connections are increasing dramatically in a Kerberized HDFS 
cluster with {noformat}dfs.client.retry.policy.enabled{noformat} enabled. 
Internally,  Client#getConnection is 

[jira] [Commented] (HADOOP-13435) Add thread local mechanism for aggregating file system storage stats

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398416#comment-15398416
 ] 

Hadoop QA commented on HADOOP-13435:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m  
4s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 57s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820796/HADOOP-13435.002.patch
 |
| JIRA Issue | HADOOP-13435 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 58f93234c1e0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / a1890c3 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10113/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10113/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add thread local mechanism for aggregating file system storage stats
> 
>
> Key: HADOOP-13435
> URL: https://issues.apache.org/jira/browse/HADOOP-13435
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13435.000.patch, HADOOP-13435.001.patch, 
> HADOOP-13435.002.patch
>
>
> As discussed in [HADOOP-13032], this is to add thread local mechanism for 
> aggregating 

[jira] [Updated] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HADOOP-13436:
---
Description: 
We've noticed RPC connections are increasing dramatically in a Kerberized HDFS 
cluster with {noformat}dfs.client.retry.policy.enabled{noformat} enabled. 
Internally,  Client#getConnection is doing lookup relying on 
ConnectionId#equals, which composes checking Subclass-of-RetryPolicy#equals. If 
subclasses of RetryPolicy neglect overriding RetryPolicy#equals, every 
instances of RetryPolicy with equivalent fields' values (e.g. 
MultipleLinearRandomRetry[6x1ms, 10x6ms]) will lead to a brand new 
connection because the check will fall back to Object#equals.

This is stack trace where RetryUtils#getDefaultRetryPolicy:
{noformat}
at 
org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(RetryUtils.java:82)
at 
org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:409)
at 
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:315)
at 
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:678)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:609)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.newDfsClient(WebHdfsHandler.java:272)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.onOpen(WebHdfsHandler.java:215)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.handle(WebHdfsHandler.java:135)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:117)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:114)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114)
at 
org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:52)
at 
org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:32)
at 
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at 
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at 
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at 
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
{noformat}


Three options to fix the problem:
1. All subclasses of RetryPolicy must override equals and hashCode to deliver 
less discriminating equivalence relation, i.e. they are equal if they have 
meaningful equivalent fields' values (e.g. MultipleLinearRandomRetry[6x1ms, 
10x6ms])
2. Change ConnectionId#equals by removing RetryPolicy#equals compoment.
3. Let WebHDFS reuse the DFSClient.

  was:
We've noticed RPC connections are increasing dramatically in a Kerberized HDFS 
cluster with {noformat}dfs.client.retry.policy.enabled{noformat} enabled. 
Internally,  Client#getConnection is doing lookup 

[jira] [Commented] (HADOOP-13018) Make Kdiag check whether hadoop.token.files points to existent and valid files

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398415#comment-15398415
 ] 

Hadoop QA commented on HADOOP-13018:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  8m 31s{color} 
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 23s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12802265/HADOOP-13018.04.patch 
|
| JIRA Issue | HADOOP-13018 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 5b585171b76a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / a1890c3 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10114/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10114/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10114/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Make Kdiag check whether hadoop.token.files points to existent and valid files
> --
>
> Key: HADOOP-13018
> URL: https://issues.apache.org/jira/browse/HADOOP-13018
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
> 

[jira] [Commented] (HADOOP-13437) KMS should reload whitelist and default key ACLs when hot-reloading

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398405#comment-15398405
 ] 

Andrew Wang commented on HADOOP-13437:
--

Sounds like a good bug! Few review comments though:

* Is the new "continue" case important? Not sure if we should ride over a 
possibly malformed ACL file.
* By removing the if checks, it means if an ACL is specified multiple times, 
we'll use the last one rather than the first one. This seems like a big 
behavior change. I think what would be better is doing a "swap" like we do for 
the key ACLs, and breaking this function up into a few functions for clarity.

> KMS should reload whitelist and default key ACLs when hot-reloading
> ---
>
> Key: HADOOP-13437
> URL: https://issues.apache.org/jira/browse/HADOOP-13437
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13437.01.patch, HADOOP-13437.02.patch
>
>
> When hot-reloading, {{KMSACLs#setKeyACLs}} ignores whitelist and default key 
> entries if they're present in memory.
> We should reload them, hot-reload and cold-start should not have any 
> difference in behavior.
> Credit to [~dilaver] for finding this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13403) AzureNativeFileSystem rename/delete performance improvements

[Chris Nauroth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HADOOP-13403:
---
Assignee: Subramanyam Pattipaka

> AzureNativeFileSystem rename/delete performance improvements
> 
>
> Key: HADOOP-13403
> URL: https://issues.apache.org/jira/browse/HADOOP-13403
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: azure
>Affects Versions: 2.7.2
>Reporter: Subramanyam Pattipaka
>Assignee: Subramanyam Pattipaka
> Fix For: 2.9.0
>
> Attachments: HADOOP-13403-001.patch, HADOOP-13403-002.patch
>
>
> WASB Performance Improvements
> Problem
> ---
> Azure Native File system operations like rename/delete which has large number 
> of directories and/or files in the source directory are experiencing 
> performance issues. Here are possible reasons
> a)We first list all files under source directory hierarchically. This is 
> a serial operation. 
> b)After collecting the entire list of files under a folder, we delete or 
> rename files one by one serially.
> c)There is no logging information available for these costly operations 
> even in DEBUG mode leading to difficulty in understanding wasb performance 
> issues.
> Proposal
> -
> Step 1: Rename and delete operations will generate a list all files under the 
> source folder. We need to use azure flat listing option to get list with 
> single request to azure store. We have introduced config 
> fs.azure.flatlist.enable to enable this option. The default value is 'false' 
> which means flat listing is disabled.
> Step 2: Create thread pool and threads dynamically based on user 
> configuration. These thread pools will be deleted after operation is over.  
> We are introducing introducing two new configs
>   a)  fs.azure.rename.threads : Config to set number of rename 
> threads. Default value is 0 which means no threading.
>   b)  fs.azure.delete.threads: Config to set number of delete 
> threads. Default value is 0 which means no threading.
>   We have provided debug log information on number of threads not used 
> for the operation which can be useful .
>   Failure Scenarios:
>   If we fail to create thread pool due to ANY reason (for example trying 
> create with thread count with large value such as 100), we fall back to 
> serialization operation. 
> Step 3: Bob operations can be done in parallel using multiple threads 
> executing following snippet
>   while ((currentIndex = fileIndex.getAndIncrement()) < files.length) {
>   FileMetadata file = files[currentIndex];
>   Rename/delete(file);
>   }
>   The above strategy depends on the fact that all files are stored in a 
> final array and each thread has to determine synchronized next index to do 
> the job. The advantage of this strategy is that even if user configures large 
> number of unusable threads, we always ensure that work doesn’t get serialized 
> due to lagging threads. 
>   We are logging following information which can be useful for tuning 
> number of threads
>   a) Number of unusable threads
>   b) Time taken by each thread
>   c) Number of files processed by each thread
>   d) Total time taken for the operation
>   Failure Scenarios:
>   Failure to queue a thread execute request shouldn’t be an issue if we 
> can ensure at least one thread has completed execution successfully. If we 
> couldn't schedule one thread then we should take serialization path. 
> Exceptions raised while executing threads are still considered regular 
> exceptions and returned to client as operation failed. Exceptions raised 
> while stopping threads and deleting thread pool shouldn't can be ignored if 
> operation all files are done with out any issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13395) Enhance TestKMSAudit

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398387#comment-15398387
 ] 

Andrew Wang commented on HADOOP-13395:
--

Looks good overall, thanks for working on this Xiao!

Few nitty comments:
* We introduce a new {{interval}} variable that's only used once, could we just 
inline it? Seems a bit confusing too since we also have a {{sleepInterval}} 
variable.
* Can we make the indentation the same for the new or case? This makes it 
easier to diff visually.

In general, if we're trying to reduce flakiness, I'd like to do it by removing 
the need for sleeps entirely. Is there some way to manually trigger aggregation 
instead, perhaps via a VisibleForTesting method? This would also make the test 
deterministic.

> Enhance TestKMSAudit
> 
>
> Key: HADOOP-13395
> URL: https://issues.apache.org/jira/browse/HADOOP-13395
> Project: Hadoop Common
>  Issue Type: Test
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Minor
> Attachments: HADOOP-13395.01.patch, HADOOP-13395.02.patch
>
>
> This jira serves the goals:
> - Enhance existing test cases in TestKMSAudit, to rule out flakiness.
> - Add a new test case about formatting for different events.
> This will help us ensure audit log compatibility when we add a new log format 
> to KMS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398374#comment-15398374
 ] 

Xiaobing Zhou commented on HADOOP-13436:


By pulling out all subclasses of RetryPolicy, only RetryForever and 
TryOnceThenFail don't have any member fields, which means it's not always 
possible to follow the pattern that multiple instances with equivalent fields' 
values are viewed as equal (i.e. being part of ConnectionId#equals) to avoid 
creating new connections. This makes exceptions for option #1, which is a 
dilemma since we can't go to option #2 and #3. Any thoughts? Thanks.

> RPC connections are leaking due to missing equals override in 
> RetryUtils#getDefaultRetryPolicy
> --
>
> Key: HADOOP-13436
> URL: https://issues.apache.org/jira/browse/HADOOP-13436
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Affects Versions: 2.7.1
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: repro.sh
>
>
> We've noticed RPC connections are increasing dramatically in a Kerberized 
> HDFS cluster with {noformat}dfs.client.retry.policy.enabled{noformat} 
> enabled. Internally,  Client#getConnection is doing lookup relying on 
> ConnectionId#equals, which composes checking RetryPolicy#equals. If 
> subclasses of RetryPolicy neglect overriding RetryPolicy#equals, every 
> instances of RetryPolicy with equivalent fields' values (e.g. 
> MultipleLinearRandomRetry[6x1ms, 10x6ms]) will lead to a brand new 
> connection because the check will fall back to Object#equals.
> This is stack trace where RetryUtils#getDefaultRetryPolicy:
> {noformat}
> at 
> org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(RetryUtils.java:82)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:409)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:315)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:609)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.newDfsClient(WebHdfsHandler.java:272)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.onOpen(WebHdfsHandler.java:215)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.handle(WebHdfsHandler.java:135)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:117)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:114)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:52)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:32)
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at 
> 

[jira] [Commented] (HADOOP-13018) Make Kdiag check whether hadoop.token.files points to existent and valid files

[Ravi Prakash (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398340#comment-15398340
 ] 

Ravi Prakash commented on HADOOP-13018:
---

Hi Steve!

I'm not sure what to add to the document. The various options have been 
enumerated and briefly described. However, I haven't added a new option. Its 
just another check, and I don't think the details of the report have been (or 
need to be) documented.

> Make Kdiag check whether hadoop.token.files points to existent and valid files
> --
>
> Key: HADOOP-13018
> URL: https://issues.apache.org/jira/browse/HADOOP-13018
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HADOOP-13018.01.patch, HADOOP-13018.02.patch, 
> HADOOP-13018.03.patch, HADOOP-13018.04.patch
>
>
> Steve proposed that KDiag should fail fast to help debug the case that 
> hadoop.token.files points to a file not found. This JIRA is to affect that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398324#comment-15398324
 ] 

Xiaobing Zhou commented on HADOOP-13436:


Thank you [~daryn] for your inputs. There is a similar patch in my mind.  

bq. re-building strings for hashCode/equals is a horrible thing that should be 
changed.
Does it mean it leads to performance issues?

> RPC connections are leaking due to missing equals override in 
> RetryUtils#getDefaultRetryPolicy
> --
>
> Key: HADOOP-13436
> URL: https://issues.apache.org/jira/browse/HADOOP-13436
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Affects Versions: 2.7.1
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: repro.sh
>
>
> We've noticed RPC connections are increasing dramatically in a Kerberized 
> HDFS cluster with {noformat}dfs.client.retry.policy.enabled{noformat} 
> enabled. Internally,  Client#getConnection is doing lookup relying on 
> ConnectionId#equals, which composes checking RetryPolicy#equals. If 
> subclasses of RetryPolicy neglect overriding RetryPolicy#equals, every 
> instances of RetryPolicy with equivalent fields' values (e.g. 
> MultipleLinearRandomRetry[6x1ms, 10x6ms]) will lead to a brand new 
> connection because the check will fall back to Object#equals.
> This is stack trace where RetryUtils#getDefaultRetryPolicy:
> {noformat}
> at 
> org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(RetryUtils.java:82)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:409)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:315)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:609)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.newDfsClient(WebHdfsHandler.java:272)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.onOpen(WebHdfsHandler.java:215)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.handle(WebHdfsHandler.java:135)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:117)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:114)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:52)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:32)
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at 

[jira] [Updated] (HADOOP-13434) Add quoting to Shell class

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HADOOP-13434:
---
Attachment: HADOOP-13434.patch

This iteration removes the use of bash from the cases where it wasn't necessary.

> Add quoting to Shell class
> --
>
> Key: HADOOP-13434
> URL: https://issues.apache.org/jira/browse/HADOOP-13434
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HADOOP-13434.patch, HADOOP-13434.patch
>
>
> The Shell class makes assumptions that the parameters won't have spaces or 
> other special characters, even when it invokes bash.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13438) Optimize IPC server protobuf decoding

[Daryn Sharp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated HADOOP-13438:
-
Attachment: HADOOP-13438.patch

Server side only changes to ensure protocol not broken (corresponding client 
changes forthcoming).

A new RpcWritable class manages the marshaling of writables and PBs.  The 
Server passes the ByteBuffer for the read request, instead of the byte[] from 
the buffer to eliminate a copy.  The RpcWritables use the ByteBuffer to 
effectively iterate through the byte[] w/o copying it.

> Optimize IPC server protobuf decoding
> -
>
> Key: HADOOP-13438
> URL: https://issues.apache.org/jira/browse/HADOOP-13438
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HADOOP-13438.patch
>
>
> The current use of the protobuf API uses an expensive code path.  The builder 
> uses the parser to instantiate a message, then copies the message into the 
> builder.  The parser is creating multi-layered internally buffering streams 
> that cause excessive byte[] allocations.
> Using the parser directly with a coded input stream backed by the byte[] from 
> the wire will take a fast-path straight to the pb message's ctor.  
> Substantially less garbage is generated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-11540) Raw Reed-Solomon coder using Intel ISA-L library

[Kai Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398286#comment-15398286
 ] 

Kai Zheng commented on HADOOP-11540:


Softly ping [~atm]

> Raw Reed-Solomon coder using Intel ISA-L library
> 
>
> Key: HADOOP-11540
> URL: https://issues.apache.org/jira/browse/HADOOP-11540
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Kai Zheng
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-11540-initial.patch, HADOOP-11540-v1.patch, 
> HADOOP-11540-v10.patch, HADOOP-11540-v11.patch, HADOOP-11540-v12.patch, 
> HADOOP-11540-v13.patch, HADOOP-11540-v2.patch, HADOOP-11540-v4.patch, 
> HADOOP-11540-v5.patch, HADOOP-11540-v6.patch, HADOOP-11540-v7.patch, 
> HADOOP-11540-v8.patch, HADOOP-11540-v9.patch, 
> HADOOP-11540-with-11996-codes.patch, Native Erasure Coder Performance - Intel 
> ISAL-v1.pdf
>
>
> This is to provide RS codec implementation using Intel ISA-L library for 
> encoding and decoding.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13438) Optimize IPC server protobuf decoding

[Daryn Sharp (JIRA)]

Daryn Sharp created HADOOP-13438:


 Summary: Optimize IPC server protobuf decoding
 Key: HADOOP-13438
 URL: https://issues.apache.org/jira/browse/HADOOP-13438
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Daryn Sharp
Assignee: Daryn Sharp


The current use of the protobuf API uses an expensive code path.  The builder 
uses the parser to instantiate a message, then copies the message into the 
builder.  The parser is creating multi-layered internally buffering streams 
that cause excessive byte[] allocations.

Using the parser directly with a coded input stream backed by the byte[] from 
the wire will take a fast-path straight to the pb message's ctor.  
Substantially less garbage is generated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13429) Dispose of unnecessary SASL servers

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398268#comment-15398268
 ] 

Daryn Sharp commented on HADOOP-13429:
--

HttpServer test failure is not related to IPC.

> Dispose of unnecessary SASL servers
> ---
>
> Key: HADOOP-13429
> URL: https://issues.apache.org/jira/browse/HADOOP-13429
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: ipc
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HADOOP-13429.patch
>
>
> The IPC server retains a per-connection SASL server for the duration of the 
> connection.  This causes many unnecessary objects to be promoted to old gen.  
> The SASL server should be disposed of unless required for subsequent 
> encryption.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13428) Fix hadoop-common to generate jdiff

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398256#comment-15398256
 ] 

Hadoop QA commented on HADOOP-13428:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
9s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue}  0m  
2s{color} | {color:blue} The patch file was not named according to hadoop's 
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute 
for instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1234 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  1m 
12s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m  
9s{color} | {color:green} hadoop-project-dist in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 
59s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 37s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820787/HADOOP-13428.2.patch 
, which include jdiff file of hadoop-common from 2.7.3 and update pom to 
compare to 2.7.3. |
| JIRA Issue | HADOOP-13428 |
| Optional Tests |  asflicense  xml  compile  javac  javadoc  mvninstall  
mvnsite  unit  |
| uname | Linux 4eeb97a2dcff 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / a3d0cba |
| Default Java | 1.8.0_101 |
| whitespace | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10112/artifact/patchprocess/whitespace-eol.txt
 |
| whitespace | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10112/artifact/patchprocess/whitespace-tabs.txt
 |
|  Test Results | 

[jira] [Updated] (HADOOP-13435) Add thread local mechanism for aggregating file system storage stats

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HADOOP-13435:
---
Attachment: HADOOP-13435.002.patch

The v2 patch should have no checkstyle warningshopefully

> Add thread local mechanism for aggregating file system storage stats
> 
>
> Key: HADOOP-13435
> URL: https://issues.apache.org/jira/browse/HADOOP-13435
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13435.000.patch, HADOOP-13435.001.patch, 
> HADOOP-13435.002.patch
>
>
> As discussed in [HADOOP-13032], this is to add thread local mechanism for 
> aggregating file system storage stats. This class will also be used in 
> [HADOOP-13031], which is to separate the distance-oriented rack-aware read 
> bytes logic from {{FileSystemStorageStatistics}} to new 
> DFSRackAwareStorageStatistics as it's DFS-specific. After this patch, the 
> {{FileSystemStorageStatistics}} can live without the to-be-removed 
> {{FileSystem$Statistics}} implementation.
> A unit test should also be added.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12588) Fix intermittent test failure of TestGangliaMetrics

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398229#comment-15398229
 ] 

Mingliang Liu commented on HADOOP-12588:


Was this resolved? Seeing this in recent run 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10106/testReport/org.apache.hadoop.metrics2.impl/TestGangliaMetrics/testGangliaMetrics2/
 Thanks.

> Fix intermittent test failure of TestGangliaMetrics
> ---
>
> Key: HADOOP-12588
> URL: https://issues.apache.org/jira/browse/HADOOP-12588
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Tsuyoshi Ozawa
>Assignee: Masatake Iwasaki
> Fix For: 2.8.0, 2.7.3
>
> Attachments: HADOOP-12588.001.patch, HADOOP-12588.addendum.02.patch, 
> HADOOP-12588.addendum.03.patch, HADOOP-12588.addendum.patch
>
>
> Jenkins found this test failure on HADOOP-11149.
> {quote}
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.773 sec <<< 
> FAILURE! - in org.apache.hadoop.metrics2.impl.TestGangliaMetrics
> testGangliaMetrics2(org.apache.hadoop.metrics2.impl.TestGangliaMetrics)  Time 
> elapsed: 0.39 sec  <<< FAILURE!
> java.lang.AssertionError: Missing metrics: test.s1rec.Xxx
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at 
> org.apache.hadoop.metrics2.impl.TestGangliaMetrics.checkMetrics(TestGangliaMetrics.java:159)
>   at 
> org.apache.hadoop.metrics2.impl.TestGangliaMetrics.testGangliaMetrics2(TestGangliaMetrics.java:137)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Issue Comment Deleted] (HADOOP-13434) Add quoting to Shell class

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HADOOP-13434:
---
Comment: was deleted

(was: GitHub user omalley opened a pull request:

https://github.com/apache/hadoop/pull/118

HADOOP-13434. Add bash quoting to Shell class.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omalley/hadoop hadoop-13434

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/118.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #118


commit e0d296a301c1e52a378e674cf7045f4ef3b8c62e
Author: Vinod Kumar Vavilapalli 
Date:   2014-03-08T04:50:31Z

YARN-1787. Fixed help messages for applicationattempt and container 
sub-commands in bin/yarn. Contributed by Zhijie Shen.
svn merge --ignore-ancestry -c 1575482 ../../trunk/


git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1575484 
13f79535-47bb-0310-9956-ffa450edef68

commit 059ba663e2890953fdd5a208d7d7969878ef7887
Author: Arpit Agarwal 
Date:   2014-03-08T16:41:26Z

HDFS-6078. Merging r1575561 from branch-2 to branch-2.4.

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1575562 
13f79535-47bb-0310-9956-ffa450edef68

commit 390ac348271cbb756f3de0ed5ee2951fcc7f34b7
Author: Colin McCabe 
Date:   2014-03-10T02:48:04Z

HDFS-6071. BlockReaderLocal does not return -1 on EOF when doing a 
zero-length read on a short file. (cmccabe)

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1575798 
13f79535-47bb-0310-9956-ffa450edef68

commit f93cc4d3d4edb74a728ba5254274e61c57ae66b9
Author: Jian He 
Date:   2014-03-10T18:05:29Z

Merge r1576026 from branch-2. Fixed ClientRMService#forceKillApplication 
not killing unmanaged application. Contributed by Karthik Kambatla

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576028 
13f79535-47bb-0310-9956-ffa450edef68

commit dc2f782d4fd85c9e416bed1e0098992b3c3a8db5
Author: Andrew Wang 
Date:   2014-03-10T19:05:44Z

HDFS-6070. Cleanup use of ReadStatistics in DFSInputStream.

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576051 
13f79535-47bb-0310-9956-ffa450edef68

commit 5fca55c41ce7ed15c5e542fbbd359f5ac1f2514a
Author: Vinod Kumar Vavilapalli 
Date:   2014-03-10T20:37:37Z

YARN-1788. Fixed a bug in ResourceManager to set the apps-completed and 
apps-killed metrics correctly for killed applications. Contributed by Varun 
Vasudev.
svn merge --ignore-ancestry -c 1576072 ../../trunk/


git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576074 
13f79535-47bb-0310-9956-ffa450edef68

commit d4936ec536920d802ca966869b54f3916fb698e9
Author: Jing Zhao 
Date:   2014-03-10T20:52:22Z

HDFS-6077. Merge change r1576077 from branch-2.

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576080 
13f79535-47bb-0310-9956-ffa450edef68

commit 883c146b4457def2389705409a20bc228fb59357
Author: Chris Nauroth 
Date:   2014-03-10T21:55:07Z

HDFS-6055. Merging change r1576098 from branch-2 to branch-2.4

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576100 
13f79535-47bb-0310-9956-ffa450edef68

commit ffc5328054b9262faefcb36e25eabf991d6e49a8
Author: Chris Nauroth 
Date:   2014-03-10T23:38:50Z

HADOOP-10399. Merging change r1576126 from branch-2 to branch-2.4

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576129 
13f79535-47bb-0310-9956-ffa450edef68

commit 1a5e3d36aa1a2a86a1ef8c76d5720e6349487a65
Author: Tsz-wo Sze 
Date:   2014-03-10T23:40:21Z

svn merge -c 1576128 from branch-2 for HDFS-5535.


git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576130 
13f79535-47bb-0310-9956-ffa450edef68

commit 810188777942f2a3e5e6c3d1ab2ac89912d4b95b
Author: Arpit Agarwal 
Date:   2014-03-11T00:00:22Z

HADOOP-10395. Merging r1576142 from branch-2 to branch-2.4.

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576144 
13f79535-47bb-0310-9956-ffa450edef68

commit 2ad1602b66753bb3cfa5274457a9b21a44374336
Author: Arpit Agarwal 
Date:   2014-03-11T00:04:09Z

HADOOP-10394. Merging r1576146 from branch-2 to branch-2.4.

git-svn-id: 

[jira] [Issue Comment Deleted] (HADOOP-13434) Add quoting to Shell class

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HADOOP-13434:
---
Comment: was deleted

(was: Github user omalley closed the pull request at:

https://github.com/apache/hadoop/pull/118
)

> Add quoting to Shell class
> --
>
> Key: HADOOP-13434
> URL: https://issues.apache.org/jira/browse/HADOOP-13434
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HADOOP-13434.patch
>
>
> The Shell class makes assumptions that the parameters won't have spaces or 
> other special characters, even when it invokes bash.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13435) Add thread local mechanism for aggregating file system storage stats

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398227#comment-15398227
 ] 

Hadoop QA commented on HADOOP-13435:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
9s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 24s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch 
generated 1 new + 127 unchanged - 0 fixed = 128 total (was 127) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 
41s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 50s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820783/HADOOP-13435.001.patch
 |
| JIRA Issue | HADOOP-13435 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 061c56fd126d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 328c855 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10111/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10111/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10111/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add thread local mechanism for aggregating file system storage stats
> 
>
> Key: HADOOP-13435
> URL: https://issues.apache.org/jira/browse/HADOOP-13435
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs
>  

[jira] [Commented] (HADOOP-13381) KMS clients running in the same JVM should use updated KMS Delegation Token

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398197#comment-15398197
 ] 

Arun Suresh commented on HADOOP-13381:
--

+1, Thanks for taking care of this [~xiaochen]

> KMS clients running in the same JVM should use updated KMS Delegation Token
> ---
>
> Key: HADOOP-13381
> URL: https://issues.apache.org/jira/browse/HADOOP-13381
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HADOOP-13381.01.patch, HADOOP-13381.02.patch, 
> HADOOP-13381.03.patch, HADOOP-13381.04.patch
>
>
> When {{/tmp}} is setup as an EZ, one may experience YARN log aggregation 
> failure after the very first KMS token is expired. The MR job itself runs 
> fine though.
> When this happens, YARN NodeManager's log will show 
> {{AuthenticationException}} with {{token is expired}} / {{token can't be 
> found in cache}}, depending on whether the expired token is removed by the 
> background or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13423) Run JDiff on trunk for Hadoop-Common and analyze results

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated HADOOP-13423:

Attachment: 3.0.0-alpha1-hadoop-common-jdiff.zip

Attached generated jdiff report: {{3.0.0-alpha1-hadoop-common-jdiff.zip
}}

> Run JDiff on trunk for Hadoop-Common and analyze results
> 
>
> Key: HADOOP-13423
> URL: https://issues.apache.org/jira/browse/HADOOP-13423
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: 3.0.0-alpha1-hadoop-common-jdiff.zip, 
> 3.0.0-alpha1-jdiff.zip
>
>
> We need to run JDiff and make sure the first 3.0.0 alpha release doesn't 
> include unnecessary API incompatible change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-13423) Run JDiff on trunk for Hadoop-Common and analyze results

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398189#comment-15398189
 ] 

Wangda Tan edited comment on HADOOP-13423 at 7/28/16 8:59 PM:
--

Attached generated jdiff report: {{3.0.0-alpha1-hadoop-common-jdiff.zip}}


was (Author: leftnoteasy):
Attached generated jdiff report: {{3.0.0-alpha1-hadoop-common-jdiff.zip
}}

> Run JDiff on trunk for Hadoop-Common and analyze results
> 
>
> Key: HADOOP-13423
> URL: https://issues.apache.org/jira/browse/HADOOP-13423
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: 3.0.0-alpha1-hadoop-common-jdiff.zip, 
> 3.0.0-alpha1-jdiff.zip
>
>
> We need to run JDiff and make sure the first 3.0.0 alpha release doesn't 
> include unnecessary API incompatible change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13428) Fix hadoop-common to generate jdiff

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated HADOOP-13428:

Status: Patch Available  (was: Open)

> Fix hadoop-common to generate jdiff
> ---
>
> Key: HADOOP-13428
> URL: https://issues.apache.org/jira/browse/HADOOP-13428
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: HADOOP-13428.1.patch, HADOOP-13428.2.patch, 
> metric-system-temp-fix.patch
>
>
> Hadoop-common failed to generate JDiff. We need to fix that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13428) Fix hadoop-common to generate jdiff

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398184#comment-15398184
 ] 

Wangda Tan commented on HADOOP-13428:
-

[~andrew.wang],

The error is caused by a known jdiff bug: 
https://sourceforge.net/p/javadiff/bugs/19/

In MetricSystem, there're two register overloads have same parameter types and 
return types, but one is  and another one is .

However, jdiff cannot recognize it. So I just attached a workaround patch 
https://issues.apache.org/jira/secure/attachment/12820786/metric-system-temp-fix.patch,
 to temporarily remove one of the register method. I think we should not commit 
it to our code base, but we can temporally apply it when generating jdiff.

The steps of generate jdiff is:

1) Apply 
https://issues.apache.org/jira/secure/attachment/12820787/HADOOP-13428.2.patch, 
which include jdiff file of hadoop-common from 2.7.3 and update pom to compare 
to 2.7.3.
2) Apply metric-system temp fix
3) Run {{mvn clean package -Pdocs -DskipTests}}
4) Result: 
- Generated jdiff file: 
{{hadoop-common-project/hadoop-common/target/site/jdiff/xml/Apache_Hadoop_Common_3.0.0-alpha2-SNAPSHOT.xml}}
- Generated jdiff report: 
{{hadoop-common-project/hadoop-common/target/site/jdiff/xml/changes.html}}

> Fix hadoop-common to generate jdiff
> ---
>
> Key: HADOOP-13428
> URL: https://issues.apache.org/jira/browse/HADOOP-13428
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: HADOOP-13428.1.patch, HADOOP-13428.2.patch, 
> metric-system-temp-fix.patch
>
>
> Hadoop-common failed to generate JDiff. We need to fix that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13381) KMS clients running in the same JVM should use updated KMS Delegation Token

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398181#comment-15398181
 ] 

Hadoop QA commented on HADOOP-13381:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m  
1s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
12s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 50s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820780/HADOOP-13381.04.patch 
|
| JIRA Issue | HADOOP-13381 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 0389f1bd4d9b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 26de4f0 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10110/testReport/ |
| modules | C: hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms U: hadoop-common-project |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10110/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> KMS clients running in the same JVM should use updated KMS Delegation Token
> ---
>
>   

[jira] [Updated] (HADOOP-13428) Fix hadoop-common to generate jdiff

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated HADOOP-13428:

Status: Open  (was: Patch Available)

> Fix hadoop-common to generate jdiff
> ---
>
> Key: HADOOP-13428
> URL: https://issues.apache.org/jira/browse/HADOOP-13428
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: HADOOP-13428.1.patch, HADOOP-13428.2.patch, 
> metric-system-temp-fix.patch
>
>
> Hadoop-common failed to generate JDiff. We need to fix that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13428) Fix hadoop-common to generate jdiff

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated HADOOP-13428:

Attachment: metric-system-temp-fix.patch

> Fix hadoop-common to generate jdiff
> ---
>
> Key: HADOOP-13428
> URL: https://issues.apache.org/jira/browse/HADOOP-13428
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: HADOOP-13428.1.patch, HADOOP-13428.2.patch, 
> metric-system-temp-fix.patch
>
>
> Hadoop-common failed to generate JDiff. We need to fix that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13428) Fix hadoop-common to generate jdiff

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated HADOOP-13428:

Attachment: HADOOP-13428.2.patch

> Fix hadoop-common to generate jdiff
> ---
>
> Key: HADOOP-13428
> URL: https://issues.apache.org/jira/browse/HADOOP-13428
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: HADOOP-13428.1.patch, HADOOP-13428.2.patch, 
> metric-system-temp-fix.patch
>
>
> Hadoop-common failed to generate JDiff. We need to fix that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13437) KMS should reload whitelist and default key ACLs when hot-reloading

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398149#comment-15398149
 ] 

Hadoop QA commented on HADOOP-13437:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} hadoop-common-project/hadoop-kms: The patch 
generated 0 new + 8 unchanged - 8 fixed = 8 total (was 16) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m  
7s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 39s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820779/HADOOP-13437.02.patch 
|
| JIRA Issue | HADOOP-13437 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 4ec7420c49bb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 26de4f0 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10109/testReport/ |
| modules | C: hadoop-common-project/hadoop-kms U: 
hadoop-common-project/hadoop-kms |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10109/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> KMS should reload whitelist and default key ACLs when hot-reloading
> ---
>
> Key: HADOOP-13437
> URL: https://issues.apache.org/jira/browse/HADOOP-13437
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13437.01.patch, HADOOP-13437.02.patch
>
>
> When hot-reloading, 

[jira] [Updated] (HADOOP-13437) KMS should reload whitelist and default key ACLs when hot-reloading

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HADOOP-13437:
---
Description: 
When hot-reloading, {{KMSACLs#setKeyACLs}} ignores whitelist and default key 
entries if they're present in memory.

We should reload them, hot-reload and cold-start should not have any difference 
in behavior.
Credit to [~dilaver] for finding this.

  was:
When hot-reloading, {{KMSACLs#setKeyACLs}} ignores whitelist and default key 
entries if they're present in memory.

We should reload them, hot-reload and cold-start should not have any difference 
in behavior.


> KMS should reload whitelist and default key ACLs when hot-reloading
> ---
>
> Key: HADOOP-13437
> URL: https://issues.apache.org/jira/browse/HADOOP-13437
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13437.01.patch, HADOOP-13437.02.patch
>
>
> When hot-reloading, {{KMSACLs#setKeyACLs}} ignores whitelist and default key 
> entries if they're present in memory.
> We should reload them, hot-reload and cold-start should not have any 
> difference in behavior.
> Credit to [~dilaver] for finding this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13435) Add thread local mechanism for aggregating file system storage stats

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HADOOP-13435:
---
Attachment: HADOOP-13435.001.patch

Fix the checkstyle warnings.

> Add thread local mechanism for aggregating file system storage stats
> 
>
> Key: HADOOP-13435
> URL: https://issues.apache.org/jira/browse/HADOOP-13435
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13435.000.patch, HADOOP-13435.001.patch
>
>
> As discussed in [HADOOP-13032], this is to add thread local mechanism for 
> aggregating file system storage stats. This class will also be used in 
> [HADOOP-13031], which is to separate the distance-oriented rack-aware read 
> bytes logic from {{FileSystemStorageStatistics}} to new 
> DFSRackAwareStorageStatistics as it's DFS-specific. After this patch, the 
> {{FileSystemStorageStatistics}} can live without the to-be-removed 
> {{FileSystem$Statistics}} implementation.
> A unit test should also be added.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13381) KMS clients running in the same JVM should use updated KMS Delegation Token

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HADOOP-13381:
---
Attachment: HADOOP-13381.04.patch

Thank you for the nice suggestion, [~asuresh].
Patch 4 attached to address it.

> KMS clients running in the same JVM should use updated KMS Delegation Token
> ---
>
> Key: HADOOP-13381
> URL: https://issues.apache.org/jira/browse/HADOOP-13381
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HADOOP-13381.01.patch, HADOOP-13381.02.patch, 
> HADOOP-13381.03.patch, HADOOP-13381.04.patch
>
>
> When {{/tmp}} is setup as an EZ, one may experience YARN log aggregation 
> failure after the very first KMS token is expired. The MR job itself runs 
> fine though.
> When this happens, YARN NodeManager's log will show 
> {{AuthenticationException}} with {{token is expired}} / {{token can't be 
> found in cache}}, depending on whether the expired token is removed by the 
> background or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13403) AzureNativeFileSystem rename/delete performance improvements

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398117#comment-15398117
 ] 

Hadoop QA commented on HADOOP-13403:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 11s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 
39 new + 43 unchanged - 1 fixed = 82 total (was 44) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 55 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
2s{color} | {color:red} The patch 4 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
39s{color} | {color:red} hadoop-tools/hadoop-azure generated 4 new + 0 
unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
31s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 27s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-azure |
|  |  Redundant nullcheck of ioThreadPool, which is known to be non-null in 
org.apache.hadoop.fs.azure.NativeAzureFileSystem.executeParallel(NativeAzureFileSystem$AzureFileSystemOperation,
 String, FileMetadata[], String, int, Configuration, String, 
NativeAzureFileSystem$AzureFileSystemThreadOperation)  Redundant null check at 
NativeAzureFileSystem.java:is known to be non-null in 
org.apache.hadoop.fs.azure.NativeAzureFileSystem.executeParallel(NativeAzureFileSystem$AzureFileSystemOperation,
 String, FileMetadata[], String, int, Configuration, String, 
NativeAzureFileSystem$AzureFileSystemThreadOperation)  Redundant null check at 
NativeAzureFileSystem.java:[line 910] |
|  |  Should 
org.apache.hadoop.fs.azure.NativeAzureFileSystem$AzureFileSystemThreadFactory 
be a _static_ inner class?  At NativeAzureFileSystem.java:inner class?  At 
NativeAzureFileSystem.java:[lines 717-737] |
|  |  new 
org.apache.hadoop.fs.azure.NativeAzureFileSystem$AzureFileSystemThreadRunnable(NativeAzureFileSystem,
 NativeAzureFileSystem$AzureFileSystemOperation, String, FileMetadata[], 
NativeAzureFileSystem$AzureFileSystemThreadOperation) may expose internal 
representation by storing an externally mutable 

[jira] [Updated] (HADOOP-13437) KMS should reload whitelist and default key ACLs when hot-reloading

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HADOOP-13437:
---
Attachment: HADOOP-13437.02.patch

Fixed checkstyle

> KMS should reload whitelist and default key ACLs when hot-reloading
> ---
>
> Key: HADOOP-13437
> URL: https://issues.apache.org/jira/browse/HADOOP-13437
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13437.01.patch, HADOOP-13437.02.patch
>
>
> When hot-reloading, {{KMSACLs#setKeyACLs}} ignores whitelist and default key 
> entries if they're present in memory.
> We should reload them, hot-reload and cold-start should not have any 
> difference in behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13437) KMS should reload whitelist and default key ACLs when hot-reloading

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398104#comment-15398104
 ] 

Hadoop QA commented on HADOOP-13437:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 12s{color} | {color:orange} hadoop-common-project/hadoop-kms: The patch 
generated 4 new + 11 unchanged - 5 fixed = 15 total (was 16) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m  
4s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 59s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820771/HADOOP-13437.01.patch 
|
| JIRA Issue | HADOOP-13437 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 0606573edf9a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 26de4f0 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10107/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-kms.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10107/testReport/ |
| modules | C: hadoop-common-project/hadoop-kms U: 
hadoop-common-project/hadoop-kms |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10107/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> KMS should reload whitelist and default key ACLs when hot-reloading
> ---
>
> Key: HADOOP-13437
> URL: https://issues.apache.org/jira/browse/HADOOP-13437
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.6.0
>  

[jira] [Updated] (HADOOP-13403) AzureNativeFileSystem rename/delete performance improvements

[Subramanyam Pattipaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Subramanyam Pattipaka updated HADOOP-13403:
---
Status: Patch Available  (was: Open)

> AzureNativeFileSystem rename/delete performance improvements
> 
>
> Key: HADOOP-13403
> URL: https://issues.apache.org/jira/browse/HADOOP-13403
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: azure
>Affects Versions: 2.7.2
>Reporter: Subramanyam Pattipaka
> Fix For: 2.9.0
>
> Attachments: HADOOP-13403-001.patch, HADOOP-13403-002.patch
>
>
> WASB Performance Improvements
> Problem
> ---
> Azure Native File system operations like rename/delete which has large number 
> of directories and/or files in the source directory are experiencing 
> performance issues. Here are possible reasons
> a)We first list all files under source directory hierarchically. This is 
> a serial operation. 
> b)After collecting the entire list of files under a folder, we delete or 
> rename files one by one serially.
> c)There is no logging information available for these costly operations 
> even in DEBUG mode leading to difficulty in understanding wasb performance 
> issues.
> Proposal
> -
> Step 1: Rename and delete operations will generate a list all files under the 
> source folder. We need to use azure flat listing option to get list with 
> single request to azure store. We have introduced config 
> fs.azure.flatlist.enable to enable this option. The default value is 'false' 
> which means flat listing is disabled.
> Step 2: Create thread pool and threads dynamically based on user 
> configuration. These thread pools will be deleted after operation is over.  
> We are introducing introducing two new configs
>   a)  fs.azure.rename.threads : Config to set number of rename 
> threads. Default value is 0 which means no threading.
>   b)  fs.azure.delete.threads: Config to set number of delete 
> threads. Default value is 0 which means no threading.
>   We have provided debug log information on number of threads not used 
> for the operation which can be useful .
>   Failure Scenarios:
>   If we fail to create thread pool due to ANY reason (for example trying 
> create with thread count with large value such as 100), we fall back to 
> serialization operation. 
> Step 3: Bob operations can be done in parallel using multiple threads 
> executing following snippet
>   while ((currentIndex = fileIndex.getAndIncrement()) < files.length) {
>   FileMetadata file = files[currentIndex];
>   Rename/delete(file);
>   }
>   The above strategy depends on the fact that all files are stored in a 
> final array and each thread has to determine synchronized next index to do 
> the job. The advantage of this strategy is that even if user configures large 
> number of unusable threads, we always ensure that work doesn’t get serialized 
> due to lagging threads. 
>   We are logging following information which can be useful for tuning 
> number of threads
>   a) Number of unusable threads
>   b) Time taken by each thread
>   c) Number of files processed by each thread
>   d) Total time taken for the operation
>   Failure Scenarios:
>   Failure to queue a thread execute request shouldn’t be an issue if we 
> can ensure at least one thread has completed execution successfully. If we 
> couldn't schedule one thread then we should take serialization path. 
> Exceptions raised while executing threads are still considered regular 
> exceptions and returned to client as operation failed. Exceptions raised 
> while stopping threads and deleting thread pool shouldn't can be ignored if 
> operation all files are done with out any issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13403) AzureNativeFileSystem rename/delete performance improvements

[Subramanyam Pattipaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Subramanyam Pattipaka updated HADOOP-13403:
---
Affects Version/s: 2.7.2
Fix Version/s: 2.9.0

> AzureNativeFileSystem rename/delete performance improvements
> 
>
> Key: HADOOP-13403
> URL: https://issues.apache.org/jira/browse/HADOOP-13403
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: azure
>Affects Versions: 2.7.2
>Reporter: Subramanyam Pattipaka
> Fix For: 2.9.0
>
> Attachments: HADOOP-13403-001.patch, HADOOP-13403-002.patch
>
>
> WASB Performance Improvements
> Problem
> ---
> Azure Native File system operations like rename/delete which has large number 
> of directories and/or files in the source directory are experiencing 
> performance issues. Here are possible reasons
> a)We first list all files under source directory hierarchically. This is 
> a serial operation. 
> b)After collecting the entire list of files under a folder, we delete or 
> rename files one by one serially.
> c)There is no logging information available for these costly operations 
> even in DEBUG mode leading to difficulty in understanding wasb performance 
> issues.
> Proposal
> -
> Step 1: Rename and delete operations will generate a list all files under the 
> source folder. We need to use azure flat listing option to get list with 
> single request to azure store. We have introduced config 
> fs.azure.flatlist.enable to enable this option. The default value is 'false' 
> which means flat listing is disabled.
> Step 2: Create thread pool and threads dynamically based on user 
> configuration. These thread pools will be deleted after operation is over.  
> We are introducing introducing two new configs
>   a)  fs.azure.rename.threads : Config to set number of rename 
> threads. Default value is 0 which means no threading.
>   b)  fs.azure.delete.threads: Config to set number of delete 
> threads. Default value is 0 which means no threading.
>   We have provided debug log information on number of threads not used 
> for the operation which can be useful .
>   Failure Scenarios:
>   If we fail to create thread pool due to ANY reason (for example trying 
> create with thread count with large value such as 100), we fall back to 
> serialization operation. 
> Step 3: Bob operations can be done in parallel using multiple threads 
> executing following snippet
>   while ((currentIndex = fileIndex.getAndIncrement()) < files.length) {
>   FileMetadata file = files[currentIndex];
>   Rename/delete(file);
>   }
>   The above strategy depends on the fact that all files are stored in a 
> final array and each thread has to determine synchronized next index to do 
> the job. The advantage of this strategy is that even if user configures large 
> number of unusable threads, we always ensure that work doesn’t get serialized 
> due to lagging threads. 
>   We are logging following information which can be useful for tuning 
> number of threads
>   a) Number of unusable threads
>   b) Time taken by each thread
>   c) Number of files processed by each thread
>   d) Total time taken for the operation
>   Failure Scenarios:
>   Failure to queue a thread execute request shouldn’t be an issue if we 
> can ensure at least one thread has completed execution successfully. If we 
> couldn't schedule one thread then we should take serialization path. 
> Exceptions raised while executing threads are still considered regular 
> exceptions and returned to client as operation failed. Exceptions raised 
> while stopping threads and deleting thread pool shouldn't can be ignored if 
> operation all files are done with out any issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13403) AzureNativeFileSystem rename/delete performance improvements

[Subramanyam Pattipaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Subramanyam Pattipaka updated HADOOP-13403:
---
Flags: Patch

> AzureNativeFileSystem rename/delete performance improvements
> 
>
> Key: HADOOP-13403
> URL: https://issues.apache.org/jira/browse/HADOOP-13403
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: azure
>Reporter: Subramanyam Pattipaka
> Attachments: HADOOP-13403-001.patch, HADOOP-13403-002.patch
>
>
> WASB Performance Improvements
> Problem
> ---
> Azure Native File system operations like rename/delete which has large number 
> of directories and/or files in the source directory are experiencing 
> performance issues. Here are possible reasons
> a)We first list all files under source directory hierarchically. This is 
> a serial operation. 
> b)After collecting the entire list of files under a folder, we delete or 
> rename files one by one serially.
> c)There is no logging information available for these costly operations 
> even in DEBUG mode leading to difficulty in understanding wasb performance 
> issues.
> Proposal
> -
> Step 1: Rename and delete operations will generate a list all files under the 
> source folder. We need to use azure flat listing option to get list with 
> single request to azure store. We have introduced config 
> fs.azure.flatlist.enable to enable this option. The default value is 'false' 
> which means flat listing is disabled.
> Step 2: Create thread pool and threads dynamically based on user 
> configuration. These thread pools will be deleted after operation is over.  
> We are introducing introducing two new configs
>   a)  fs.azure.rename.threads : Config to set number of rename 
> threads. Default value is 0 which means no threading.
>   b)  fs.azure.delete.threads: Config to set number of delete 
> threads. Default value is 0 which means no threading.
>   We have provided debug log information on number of threads not used 
> for the operation which can be useful .
>   Failure Scenarios:
>   If we fail to create thread pool due to ANY reason (for example trying 
> create with thread count with large value such as 100), we fall back to 
> serialization operation. 
> Step 3: Bob operations can be done in parallel using multiple threads 
> executing following snippet
>   while ((currentIndex = fileIndex.getAndIncrement()) < files.length) {
>   FileMetadata file = files[currentIndex];
>   Rename/delete(file);
>   }
>   The above strategy depends on the fact that all files are stored in a 
> final array and each thread has to determine synchronized next index to do 
> the job. The advantage of this strategy is that even if user configures large 
> number of unusable threads, we always ensure that work doesn’t get serialized 
> due to lagging threads. 
>   We are logging following information which can be useful for tuning 
> number of threads
>   a) Number of unusable threads
>   b) Time taken by each thread
>   c) Number of files processed by each thread
>   d) Total time taken for the operation
>   Failure Scenarios:
>   Failure to queue a thread execute request shouldn’t be an issue if we 
> can ensure at least one thread has completed execution successfully. If we 
> couldn't schedule one thread then we should take serialization path. 
> Exceptions raised while executing threads are still considered regular 
> exceptions and returned to client as operation failed. Exceptions raised 
> while stopping threads and deleting thread pool shouldn't can be ignored if 
> operation all files are done with out any issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398070#comment-15398070
 ] 

Daryn Sharp commented on HADOOP-13436:
--

We encountered this exact problem, fixed it, but apparently haven't pushed back 
to the community.  I think #1 is the correct approach.  Below is the quick & 
dirty patch we used.  I'd suggest scrubbing all the policies for correctness.  
Plus re-building strings for hashCode/equals is a horrible thing that should be 
changed.

{code}
index 000..5aab5c2
--- /dev/null
+++ b/Y-CHANGES/YHADOOP-977
@@ -0,0 +1 @@
+[YHADOOP-977] Webhdfs causes datanodes to create excessive connections.
index fab406d..032e38e 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryUtils.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryUtils.java
@@ -115,6 +115,16 @@ public RetryAction shouldRetry(Exception e, int retries, 
int failovers,
 }
 
 @Override
+public int hashCode() {
+  return multipleLinearRandomRetry.hashCode();
+}
+
+@Override
+public boolean equals(final Object that) {
+  return this.toString().equals(that.toString());
+}
+
+@Override
 public String toString() {
   return "RetryPolicy[" + multipleLinearRandomRetry + ", "
   + RetryPolicies.TRY_ONCE_THEN_FAIL.getClass().getSimpleName();
{code}


> RPC connections are leaking due to missing equals override in 
> RetryUtils#getDefaultRetryPolicy
> --
>
> Key: HADOOP-13436
> URL: https://issues.apache.org/jira/browse/HADOOP-13436
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Affects Versions: 2.7.1
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: repro.sh
>
>
> We've noticed RPC connections are increasing dramatically in a Kerberized 
> HDFS cluster with {noformat}dfs.client.retry.policy.enabled{noformat} 
> enabled. Internally,  Client#getConnection is doing lookup relying on 
> ConnectionId#equals, which composes checking RetryPolicy#equals. If 
> subclasses of RetryPolicy neglect overriding RetryPolicy#equals, every 
> instances of RetryPolicy with equivalent fields' values (e.g. 
> MultipleLinearRandomRetry[6x1ms, 10x6ms]) will lead to a brand new 
> connection because the check will fall back to Object#equals.
> This is stack trace where RetryUtils#getDefaultRetryPolicy:
> {noformat}
> at 
> org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(RetryUtils.java:82)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:409)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:315)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:609)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.newDfsClient(WebHdfsHandler.java:272)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.onOpen(WebHdfsHandler.java:215)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.handle(WebHdfsHandler.java:135)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:117)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:114)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:52)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:32)
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at 
> 

[jira] [Commented] (HADOOP-13381) KMS clients running in the same JVM should use updated KMS Delegation Token

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398050#comment-15398050
 ] 

Arun Suresh commented on HADOOP-13381:
--

I agree.. I also prefer what you've done in v03.

Minor nit:
You can maybe replace the huge if else with:

{noformat}
UserGroupInformation ugiToUse = (currentUgiContainsKmsDt() && doAsUser == null) 
? currentUgi : actualUgi;
conn = ugiToUse.doAs(new PrivilegedExceptionAction() {
  @Override
  public HttpURLConnection run() throws Exception {
DelegationTokenAuthenticatedURL authUrl =
new DelegationTokenAuthenticatedURL(configurator);
return authUrl.openConnection(url, authToken, doAsUser);
  }
});
{noformat}

> KMS clients running in the same JVM should use updated KMS Delegation Token
> ---
>
> Key: HADOOP-13381
> URL: https://issues.apache.org/jira/browse/HADOOP-13381
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HADOOP-13381.01.patch, HADOOP-13381.02.patch, 
> HADOOP-13381.03.patch
>
>
> When {{/tmp}} is setup as an EZ, one may experience YARN log aggregation 
> failure after the very first KMS token is expired. The MR job itself runs 
> fine though.
> When this happens, YARN NodeManager's log will show 
> {{AuthenticationException}} with {{token is expired}} / {{token can't be 
> found in cache}}, depending on whether the expired token is removed by the 
> background or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13437) KMS should reload whitelist and default key ACLs when hot-reloading

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HADOOP-13437:
---
Attachment: HADOOP-13437.01.patch

Patch 1 to fix this, with a unit test that fails without.

> KMS should reload whitelist and default key ACLs when hot-reloading
> ---
>
> Key: HADOOP-13437
> URL: https://issues.apache.org/jira/browse/HADOOP-13437
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13437.01.patch
>
>
> When hot-reloading, {{KMSACLs#setKeyACLs}} ignores whitelist and default key 
> entries if they're present in memory.
> We should reload them, hot-reload and cold-start should not have any 
> difference in behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13437) KMS should reload whitelist and default key ACLs when hot-reloading

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HADOOP-13437:
---
Status: Patch Available  (was: Open)

> KMS should reload whitelist and default key ACLs when hot-reloading
> ---
>
> Key: HADOOP-13437
> URL: https://issues.apache.org/jira/browse/HADOOP-13437
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13437.01.patch
>
>
> When hot-reloading, {{KMSACLs#setKeyACLs}} ignores whitelist and default key 
> entries if they're present in memory.
> We should reload them, hot-reload and cold-start should not have any 
> difference in behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13437) KMS should reload whitelist and default key ACLs when hot-reloading

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13437:
--

 Summary: KMS should reload whitelist and default key ACLs when 
hot-reloading
 Key: HADOOP-13437
 URL: https://issues.apache.org/jira/browse/HADOOP-13437
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen


When hot-reloading, {{KMSACLs#setKeyACLs}} ignores whitelist and default key 
entries if they're present in memory.

We should reload them, hot-reload and cold-start should not have any difference 
in behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398001#comment-15398001
 ] 

Xiaobing Zhou commented on HADOOP-13436:


[~daryn] what's your thought on this? Thanks. HDFS-7597 cached UGI to avoid new 
connections being created.

> RPC connections are leaking due to missing equals override in 
> RetryUtils#getDefaultRetryPolicy
> --
>
> Key: HADOOP-13436
> URL: https://issues.apache.org/jira/browse/HADOOP-13436
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Affects Versions: 2.7.1
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: repro.sh
>
>
> We've noticed RPC connections are increasing dramatically in a Kerberized 
> HDFS cluster with {noformat}dfs.client.retry.policy.enabled{noformat} 
> enabled. Internally,  Client#getConnection is doing lookup relying on 
> ConnectionId#equals, which composes checking RetryPolicy#equals. If 
> subclasses of RetryPolicy neglect overriding RetryPolicy#equals, every 
> instances of RetryPolicy with equivalent fields' values (e.g. 
> MultipleLinearRandomRetry[6x1ms, 10x6ms]) will lead to a brand new 
> connection because the check will fall back to Object#equals.
> This is stack trace where RetryUtils#getDefaultRetryPolicy:
> {noformat}
> at 
> org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(RetryUtils.java:82)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:409)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:315)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:609)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.newDfsClient(WebHdfsHandler.java:272)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.onOpen(WebHdfsHandler.java:215)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.handle(WebHdfsHandler.java:135)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:117)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:114)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:52)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:32)
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at 
> 

[jira] [Commented] (HADOOP-13435) Add thread local mechanism for aggregating file system storage stats

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397986#comment-15397986
 ] 

Hadoop QA commented on HADOOP-13435:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
9s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 25s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch 
generated 60 new + 127 unchanged - 0 fixed = 187 total (was 127) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  7m 46s{color} 
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 54s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820746/HADOOP-13435.000.patch
 |
| JIRA Issue | HADOOP-13435 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 36b16fe9e290 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 26de4f0 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10106/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10106/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10106/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10106/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add thread local mechanism for aggregating file system storage stats
> 

[jira] [Commented] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397983#comment-15397983
 ] 

Xiaobing Zhou commented on HADOOP-13436:


The option #1 is preferable from my understanding. Option #2 is a high risky 
fundamental change across stack. Option #3 is not universally applicable. They 
are posted here with a hope of sourcing collective opinions. Please feel free 
to comment on it. Thanks.

> RPC connections are leaking due to missing equals override in 
> RetryUtils#getDefaultRetryPolicy
> --
>
> Key: HADOOP-13436
> URL: https://issues.apache.org/jira/browse/HADOOP-13436
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Affects Versions: 2.7.1
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: repro.sh
>
>
> We've noticed RPC connections are increasing dramatically in a Kerberized 
> HDFS cluster with {noformat}dfs.client.retry.policy.enabled{noformat} 
> enabled. Internally,  Client#getConnection is doing lookup relying on 
> ConnectionId#equals, which composes checking RetryPolicy#equals. If 
> subclasses of RetryPolicy neglect overriding RetryPolicy#equals, every 
> instances of RetryPolicy with equivalent fields' values (e.g. 
> MultipleLinearRandomRetry[6x1ms, 10x6ms]) will lead to a brand new 
> connection because the check will fall back to Object#equals.
> This is stack trace where RetryUtils#getDefaultRetryPolicy:
> {noformat}
> at 
> org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(RetryUtils.java:82)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:409)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:315)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:609)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.newDfsClient(WebHdfsHandler.java:272)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.onOpen(WebHdfsHandler.java:215)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.handle(WebHdfsHandler.java:135)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:117)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:114)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:52)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:32)
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> 

[jira] [Commented] (HADOOP-13434) Add quoting to Shell class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397981#comment-15397981
 ] 

ASF GitHub Bot commented on HADOOP-13434:
-

GitHub user omalley opened a pull request:

https://github.com/apache/hadoop/pull/119

HADOOP-13434. Add bash quoting to Shell class.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omalley/hadoop hadoop-13434

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/119.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #119


commit 1ba826fcf3d8a0c94bf8880dd031f376cc0cd417
Author: Owen O'Malley 
Date:   2016-07-28T18:26:21Z

HADOOP-13434. Add bash quoting to Shell class.




> Add quoting to Shell class
> --
>
> Key: HADOOP-13434
> URL: https://issues.apache.org/jira/browse/HADOOP-13434
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HADOOP-13434.patch
>
>
> The Shell class makes assumptions that the parameters won't have spaces or 
> other special characters, even when it invokes bash.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13434) Add quoting to Shell class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397974#comment-15397974
 ] 

ASF GitHub Bot commented on HADOOP-13434:
-

Github user omalley closed the pull request at:

https://github.com/apache/hadoop/pull/118


> Add quoting to Shell class
> --
>
> Key: HADOOP-13434
> URL: https://issues.apache.org/jira/browse/HADOOP-13434
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HADOOP-13434.patch
>
>
> The Shell class makes assumptions that the parameters won't have spaces or 
> other special characters, even when it invokes bash.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13434) Add quoting to Shell class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397972#comment-15397972
 ] 

ASF GitHub Bot commented on HADOOP-13434:
-

GitHub user omalley opened a pull request:

https://github.com/apache/hadoop/pull/118

HADOOP-13434. Add bash quoting to Shell class.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omalley/hadoop hadoop-13434

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/118.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #118


commit e0d296a301c1e52a378e674cf7045f4ef3b8c62e
Author: Vinod Kumar Vavilapalli 
Date:   2014-03-08T04:50:31Z

YARN-1787. Fixed help messages for applicationattempt and container 
sub-commands in bin/yarn. Contributed by Zhijie Shen.
svn merge --ignore-ancestry -c 1575482 ../../trunk/


git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1575484 
13f79535-47bb-0310-9956-ffa450edef68

commit 059ba663e2890953fdd5a208d7d7969878ef7887
Author: Arpit Agarwal 
Date:   2014-03-08T16:41:26Z

HDFS-6078. Merging r1575561 from branch-2 to branch-2.4.

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1575562 
13f79535-47bb-0310-9956-ffa450edef68

commit 390ac348271cbb756f3de0ed5ee2951fcc7f34b7
Author: Colin McCabe 
Date:   2014-03-10T02:48:04Z

HDFS-6071. BlockReaderLocal does not return -1 on EOF when doing a 
zero-length read on a short file. (cmccabe)

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1575798 
13f79535-47bb-0310-9956-ffa450edef68

commit f93cc4d3d4edb74a728ba5254274e61c57ae66b9
Author: Jian He 
Date:   2014-03-10T18:05:29Z

Merge r1576026 from branch-2. Fixed ClientRMService#forceKillApplication 
not killing unmanaged application. Contributed by Karthik Kambatla

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576028 
13f79535-47bb-0310-9956-ffa450edef68

commit dc2f782d4fd85c9e416bed1e0098992b3c3a8db5
Author: Andrew Wang 
Date:   2014-03-10T19:05:44Z

HDFS-6070. Cleanup use of ReadStatistics in DFSInputStream.

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576051 
13f79535-47bb-0310-9956-ffa450edef68

commit 5fca55c41ce7ed15c5e542fbbd359f5ac1f2514a
Author: Vinod Kumar Vavilapalli 
Date:   2014-03-10T20:37:37Z

YARN-1788. Fixed a bug in ResourceManager to set the apps-completed and 
apps-killed metrics correctly for killed applications. Contributed by Varun 
Vasudev.
svn merge --ignore-ancestry -c 1576072 ../../trunk/


git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576074 
13f79535-47bb-0310-9956-ffa450edef68

commit d4936ec536920d802ca966869b54f3916fb698e9
Author: Jing Zhao 
Date:   2014-03-10T20:52:22Z

HDFS-6077. Merge change r1576077 from branch-2.

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576080 
13f79535-47bb-0310-9956-ffa450edef68

commit 883c146b4457def2389705409a20bc228fb59357
Author: Chris Nauroth 
Date:   2014-03-10T21:55:07Z

HDFS-6055. Merging change r1576098 from branch-2 to branch-2.4

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576100 
13f79535-47bb-0310-9956-ffa450edef68

commit ffc5328054b9262faefcb36e25eabf991d6e49a8
Author: Chris Nauroth 
Date:   2014-03-10T23:38:50Z

HADOOP-10399. Merging change r1576126 from branch-2 to branch-2.4

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576129 
13f79535-47bb-0310-9956-ffa450edef68

commit 1a5e3d36aa1a2a86a1ef8c76d5720e6349487a65
Author: Tsz-wo Sze 
Date:   2014-03-10T23:40:21Z

svn merge -c 1576128 from branch-2 for HDFS-5535.


git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576130 
13f79535-47bb-0310-9956-ffa450edef68

commit 810188777942f2a3e5e6c3d1ab2ac89912d4b95b
Author: Arpit Agarwal 
Date:   2014-03-11T00:00:22Z

HADOOP-10395. Merging r1576142 from branch-2 to branch-2.4.

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.4@1576144 
13f79535-47bb-0310-9956-ffa450edef68

commit 2ad1602b66753bb3cfa5274457a9b21a44374336
Author: Arpit Agarwal 
Date:   2014-03-11T00:04:09Z

HADOOP-10394. Merging r1576146 from branch-2 to branch-2.4.

git-svn-id: 

[jira] [Commented] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397970#comment-15397970
 ] 

Xiaobing Zhou commented on HADOOP-13436:


I've posted the script used to reproduce the case as repro.sh

> RPC connections are leaking due to missing equals override in 
> RetryUtils#getDefaultRetryPolicy
> --
>
> Key: HADOOP-13436
> URL: https://issues.apache.org/jira/browse/HADOOP-13436
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Affects Versions: 2.7.1
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: repro.sh
>
>
> We've noticed RPC connections are increasing dramatically in a Kerberized 
> HDFS cluster with {noformat}dfs.client.retry.policy.enabled{noformat} 
> enabled. Internally,  Client#getConnection is doing lookup relying on 
> ConnectionId#equals, which composes checking RetryPolicy#equals. If 
> subclasses of RetryPolicy neglect overriding RetryPolicy#equals, every 
> instances of RetryPolicy with equivalent fields' values (e.g. 
> MultipleLinearRandomRetry[6x1ms, 10x6ms]) will lead to a brand new 
> connection because the check will fall back to Object#equals.
> This is stack trace where RetryUtils#getDefaultRetryPolicy:
> {noformat}
> at 
> org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(RetryUtils.java:82)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:409)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:315)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:609)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.newDfsClient(WebHdfsHandler.java:272)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.onOpen(WebHdfsHandler.java:215)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.handle(WebHdfsHandler.java:135)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:117)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:114)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:52)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:32)
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at 
> 

[jira] [Updated] (HADOOP-13434) Add quoting to Shell class

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HADOOP-13434:
---
Attachment: HADOOP-13434.patch

This patch adds quoting to the Shell class on each command that invokes bash 
with string parameters.

> Add quoting to Shell class
> --
>
> Key: HADOOP-13434
> URL: https://issues.apache.org/jira/browse/HADOOP-13434
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HADOOP-13434.patch
>
>
> The Shell class makes assumptions that the parameters won't have spaces or 
> other special characters, even when it invokes bash.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HADOOP-13436:
---
Attachment: repro.sh

> RPC connections are leaking due to missing equals override in 
> RetryUtils#getDefaultRetryPolicy
> --
>
> Key: HADOOP-13436
> URL: https://issues.apache.org/jira/browse/HADOOP-13436
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Affects Versions: 2.7.1
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: repro.sh
>
>
> We've noticed RPC connections are increasing dramatically in a Kerberized 
> HDFS cluster with {noformat}dfs.client.retry.policy.enabled{noformat} 
> enabled. Internally,  Client#getConnection is doing lookup relying on 
> ConnectionId#equals, which composes checking RetryPolicy#equals. If 
> subclasses of RetryPolicy neglect overriding RetryPolicy#equals, every 
> instances of RetryPolicy with equivalent fields' values (e.g. 
> MultipleLinearRandomRetry[6x1ms, 10x6ms]) will lead to a brand new 
> connection because the check will fall back to Object#equals.
> This is stack trace where RetryUtils#getDefaultRetryPolicy:
> {noformat}
> at 
> org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(RetryUtils.java:82)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:409)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:315)
> at 
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:609)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.newDfsClient(WebHdfsHandler.java:272)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.onOpen(WebHdfsHandler.java:215)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.handle(WebHdfsHandler.java:135)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:117)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:114)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:52)
> at 
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:32)
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at 
> 

[jira] [Updated] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HADOOP-13436:
---
Description: 
We've noticed RPC connections are increasing dramatically in a Kerberized HDFS 
cluster with {noformat}dfs.client.retry.policy.enabled{noformat} enabled. 
Internally,  Client#getConnection is doing lookup relying on 
ConnectionId#equals, which composes checking RetryPolicy#equals. If subclasses 
of RetryPolicy neglect overriding RetryPolicy#equals, every instances of 
RetryPolicy with equivalent fields' values (e.g. 
MultipleLinearRandomRetry[6x1ms, 10x6ms]) will lead to a brand new 
connection because the check will fall back to Object#equals.

This is stack trace where RetryUtils#getDefaultRetryPolicy:
{noformat}
at 
org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(RetryUtils.java:82)
at 
org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:409)
at 
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:315)
at 
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:678)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:609)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.newDfsClient(WebHdfsHandler.java:272)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.onOpen(WebHdfsHandler.java:215)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.handle(WebHdfsHandler.java:135)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:117)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler$1.run(WebHdfsHandler.java:114)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:114)
at 
org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:52)
at 
org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:32)
at 
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at 
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at 
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at 
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
{noformat}


Three options to fix the problem:
1. All subclasses of RetryPolicy must override equals and hashCode to deliver 
less discriminating equivalence relation, i.e. they are equal if they have 
meaningful equivalent fields' values (e.g. MultipleLinearRandomRetry[6x1ms, 
10x6ms])
2. Change ConnectionId#equals by removing RetryPolicy#equals compoment.
3. Let WebHDFS reuse the DFSClient.

> RPC connections are leaking due to missing equals override in 
> RetryUtils#getDefaultRetryPolicy
> --
>
> Key: 

[jira] [Updated] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HADOOP-13436:
---
Component/s: ipc

> RPC connections are leaking due to missing equals override in 
> RetryUtils#getDefaultRetryPolicy
> --
>
> Key: HADOOP-13436
> URL: https://issues.apache.org/jira/browse/HADOOP-13436
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: ipc
>Affects Versions: 2.7.1
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13436) RPC connections are leaking due to missing equals override in RetryUtils#getDefaultRetryPolicy

[Xiaobing Zhou (JIRA)]

Xiaobing Zhou created HADOOP-13436:
--

 Summary: RPC connections are leaking due to missing equals 
override in RetryUtils#getDefaultRetryPolicy
 Key: HADOOP-13436
 URL: https://issues.apache.org/jira/browse/HADOOP-13436
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.7.1
Reporter: Xiaobing Zhou
Assignee: Xiaobing Zhou






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13032) Refactor FileSystem$Statistics to use StorageStatistics

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397901#comment-15397901
 ] 

Mingliang Liu commented on HADOOP-13032:


I filed [HADOOP-13435] for tracking step 1. Thanks.

> Refactor FileSystem$Statistics to use StorageStatistics
> ---
>
> Key: HADOOP-13032
> URL: https://issues.apache.org/jira/browse/HADOOP-13032
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13032.000.patch, HADOOP-13032.001.patch
>
>
> [HADOOP-13065] added a new interface for retrieving FS and FC Statistics. 
> This jira is to track the effort of moving the {{Statistics}} class out of 
> {{FileSystem}}, and make it use that new interface.
> We should keep the thread local implementation. Benefits are:
> # they could be used in both {{FileContext}} and {{FileSystem}}
> # unified stats data structure
> # shorter source code
> Please note this will be an backwards-incompatible change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13435) Add thread local mechanism for aggregating file system storage stats

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HADOOP-13435:
---
Fix Version/s: (was: 2.8.0)

> Add thread local mechanism for aggregating file system storage stats
> 
>
> Key: HADOOP-13435
> URL: https://issues.apache.org/jira/browse/HADOOP-13435
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13435.000.patch
>
>
> As discussed in [HADOOP-13032], this is to add thread local mechanism for 
> aggregating file system storage stats. This class will also be used in 
> [HADOOP-13031], which is to separate the distance-oriented rack-aware read 
> bytes logic from {{FileSystemStorageStatistics}} to new 
> DFSRackAwareStorageStatistics as it's DFS-specific. After this patch, the 
> {{FileSystemStorageStatistics}} can live without the to-be-removed 
> {{FileSystem$Statistics}} implementation.
> A unit test should also be added.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13435) Add thread local mechanism for aggregating file system storage stats

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HADOOP-13435:
---
Status: Patch Available  (was: Open)

> Add thread local mechanism for aggregating file system storage stats
> 
>
> Key: HADOOP-13435
> URL: https://issues.apache.org/jira/browse/HADOOP-13435
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HADOOP-13435.000.patch
>
>
> As discussed in [HADOOP-13032], this is to add thread local mechanism for 
> aggregating file system storage stats. This class will also be used in 
> [HADOOP-13031], which is to separate the distance-oriented rack-aware read 
> bytes logic from {{FileSystemStorageStatistics}} to new 
> DFSRackAwareStorageStatistics as it's DFS-specific. After this patch, the 
> {{FileSystemStorageStatistics}} can live without the to-be-removed 
> {{FileSystem$Statistics}} implementation.
> A unit test should also be added.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13435) Add thread local mechanism for aggregating file system storage stats

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HADOOP-13435:
---
Attachment: HADOOP-13435.000.patch

> Add thread local mechanism for aggregating file system storage stats
> 
>
> Key: HADOOP-13435
> URL: https://issues.apache.org/jira/browse/HADOOP-13435
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HADOOP-13435.000.patch
>
>
> As discussed in [HADOOP-13032], this is to add thread local mechanism for 
> aggregating file system storage stats. This class will also be used in 
> [HADOOP-13031], which is to separate the distance-oriented rack-aware read 
> bytes logic from {{FileSystemStorageStatistics}} to new 
> DFSRackAwareStorageStatistics as it's DFS-specific. After this patch, the 
> {{FileSystemStorageStatistics}} can live without the to-be-removed 
> {{FileSystem$Statistics}} implementation.
> A unit test should also be added.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13435) Add thread local mechanism for aggregating file system storage stats

[Mingliang Liu (JIRA)]

Mingliang Liu created HADOOP-13435:
--

 Summary: Add thread local mechanism for aggregating file system 
storage stats
 Key: HADOOP-13435
 URL: https://issues.apache.org/jira/browse/HADOOP-13435
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs
Reporter: Mingliang Liu
Assignee: Mingliang Liu


As discussed in [HADOOP-13032], this is to add thread local mechanism for 
aggregating file system storage stats. This class will also be used in 
[HADOOP-13031], which is to separate the distance-oriented rack-aware read 
bytes logic from {{FileSystemStorageStatistics}} to new 
DFSRackAwareStorageStatistics as it's DFS-specific. After this patch, the 
{{FileSystemStorageStatistics}} can live without the to-be-removed 
{{FileSystem$Statistics}} implementation.

A unit test should also be added.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13380) TestBasicDiskValidator should not write data to /tmp

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397857#comment-15397857
 ] 

Hadoop QA commented on HADOOP-13380:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
9s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
22s{color} | {color:green} hadoop-common-project/hadoop-common: The patch 
generated 0 new + 25 unchanged - 1 fixed = 25 total (was 26) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m  
3s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 46s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820729/HADOOP-13380.002.patch
 |
| JIRA Issue | HADOOP-13380 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux e1a358837b01 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 7f3c306 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10105/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10105/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> TestBasicDiskValidator should not write data to /tmp
> 
>
> Key: HADOOP-13380
> URL: https://issues.apache.org/jira/browse/HADOOP-13380
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Lei (Eddy) Xu
>Assignee: Yufei Gu
>Priority: Minor
> Attachments: HADOOP-13380.001.patch, HADOOP-13380.002.patch
>
>
> In 

[jira] [Commented] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

[Chris Nauroth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397848#comment-15397848
 ] 

Chris Nauroth commented on HADOOP-9565:
---

FYI, Steve is out for a few weeks, so please do not expect immediate responses 
from him.  (Have fun, Steve.)

> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Pieter Reuse
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch, 
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch, 
> HADOOP-9565-006.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really 
> blobstores, with different atomicity and consistency guarantees, by adding a 
> {{Blobstore}} interface to add to them. 
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that 
> all blobstores implement at server-side copy operation as a substitute for 
> rename.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-11790) leveldb usage should be disabled by default or smarter about platforms

[Alan Burlison (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-11790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397831#comment-15397831
 ] 

Alan Burlison commented on HADOOP-11790:


I do have a recipe to get leveldb to build on Solaris, it needs a patch 
applying to do so, and some hand-copying of .so files after it is built.

The real problem however is leveldbjni. Because it includes .so and DLL files 
inside the JAR file, you need those libraries for *all* the platforms it can 
run on. You also have to force it to statically link against both snappy and 
leveldb. And there's no tarball containing the JNI code, you have to pull it 
down with maven. And then of course it's hard-coded to use a specific autoconf 
version, so you have to regen all that, and then you have to patch the POM so 
it knows about the new platforms you are adding.

Basically, if you don't have access to *all* the platforms it's exceedingly 
difficult to build. And even if you *do* have access to all the platforms, you 
are in a world of hurt anyway, building and then copying libraries and JAR 
files between platforms.

In my opinion, if there's any way leveldb can be removed as a dependency, it 
should be removed. It's pretty much toxic if you need to either build from 
ground up yourself, or are adding support for a new platform.

> leveldb usage should be disabled by default or smarter about platforms
> --
>
> Key: HADOOP-11790
> URL: https://issues.apache.org/jira/browse/HADOOP-11790
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.6.0
> Environment: * any non-x86
> * any OS that isn't Linux, OSX, Windows
>Reporter: Ayappan
>Priority: Critical
>
> The leveldbjni artifact in maven repository has been built for only x86 
> architecture. Due to which some of the testcases are failing in PowerPC. The 
> leveldbjni community has no plans to support other platforms [ 
> https://github.com/fusesource/leveldbjni/issues/54 ]. Right now , the 
> approach is we need to locally built leveldbjni prior to running hadoop 
> testcases. Pushing a PowerPC-specific leveldbjni artifact in central maven 
> repository and making pom.xml to pickup it up while running in PowerPC is 
> another option but i don't know whether this is a suitable one . Any other 
> alternative/solution is there ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

[Chris Nauroth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397825#comment-15397825
 ] 

Chris Nauroth commented on HADOOP-9565:
---

I'm afraid I'm unclear about the goals here, specifically related to the 
definition of Put and Copy operations.

{code}
  public void Put(Path path, OutputStream source, long length, long options)
{code}

This method signature surprised me.  Was the {{OutputStream}} argument meant to 
be an {{InputStream}} for the implementation read?

Like Pieter, it's unclear to me that this is significant as part of the 
{{ObjectStore}} interface.  It looks similar to the {{FileUtil#copy}} utility 
methods, not a primitive operation on {{FileSystem}} or {{ObjectStore}}.  Early 
comments indicate that this could have been a way for S3A to implement a direct 
streaming write instead of spooling to local disk first.  Now that S3A has 
{{S3AFastOutputStream}}, does that meet the need?  Or are you looking to 
provide clients with a guaranteed non-spooling API regardless of configuration?

Is the {{options}} the significant part that I'm missing?  Is the idea that the 
caller passes along options to describe desired semantics, and this gives the 
{{ObjectStore}} implementation the opportunity to throw 
{{UnsatisfiedSemanticsException}}?

Maybe a usage example showing how you expect callers to use Put would clarify 
it.

Regarding Copy, the description indicates that method could substitute for 
rename.  Looking at S3A as a concrete example, the rename already uses S3 
server-side copy.  How do you expect an S3A implementation of Copy would differ 
from the current {{S3AFileSystem#rename}}?  Do you expect it can relax the 
notoriously complex semantics of rename, and therefore trigger fewer S3 calls 
to check metadata?

Perhaps ViewFs could be handled incrementally later based on need.  The 
difficulty is that I think proper ViewFs support will impact the public API 
definition, and then downstream applications need to react by migrating to 
ViewFs-compatible method signatures.  A prior example is 
{{FileSystem#getDelegationToken}} vs. {{FileSystem#addDelegationTokens}}.  Even 
after a long time, there are lingering bugs in downstream applications that 
have not migrated to {{addDelegationTokens}}, making them unusable with ViewFs 
in secured deployments.

> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Pieter Reuse
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch, 
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch, 
> HADOOP-9565-006.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really 
> blobstores, with different atomicity and consistency guarantees, by adding a 
> {{Blobstore}} interface to add to them. 
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that 
> all blobstores implement at server-side copy operation as a substitute for 
> rename.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13380) TestBasicDiskValidator should not write data to /tmp

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated HADOOP-13380:
--
Attachment: HADOOP-13380.002.patch

> TestBasicDiskValidator should not write data to /tmp
> 
>
> Key: HADOOP-13380
> URL: https://issues.apache.org/jira/browse/HADOOP-13380
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Lei (Eddy) Xu
>Assignee: Yufei Gu
>Priority: Minor
> Attachments: HADOOP-13380.001.patch, HADOOP-13380.002.patch
>
>
> In {{TestBasicDiskValidator}}, the following code is confusing
> {code}
>File localDir = File.createTempFile("test", "tmp");
> try {
>if (isDir) {
>// reuse the file path generated by File#createTempFile to create a dir
>   localDir.delete();
>localDir.mkdir();
> }
> {code}
> Btw, as suggested in https://wiki.apache.org/hadoop/CodeReviewChecklist, unit 
> test should not write data into {{/tmp}}:
> bq. * unit tests do not write any temporary files to /tmp (instead, the tests 
> should write to the location specified by the test.build.data system property)
> Finally, should use {{Files}} in these file creation / deletion, so that any 
> error can be thrown as {{IOE}}.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13380) TestBasicDiskValidator should not write data to /tmp

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated HADOOP-13380:
--
Attachment: (was: HADOOP-13380.002.patch)

> TestBasicDiskValidator should not write data to /tmp
> 
>
> Key: HADOOP-13380
> URL: https://issues.apache.org/jira/browse/HADOOP-13380
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Lei (Eddy) Xu
>Assignee: Yufei Gu
>Priority: Minor
> Attachments: HADOOP-13380.001.patch
>
>
> In {{TestBasicDiskValidator}}, the following code is confusing
> {code}
>File localDir = File.createTempFile("test", "tmp");
> try {
>if (isDir) {
>// reuse the file path generated by File#createTempFile to create a dir
>   localDir.delete();
>localDir.mkdir();
> }
> {code}
> Btw, as suggested in https://wiki.apache.org/hadoop/CodeReviewChecklist, unit 
> test should not write data into {{/tmp}}:
> bq. * unit tests do not write any temporary files to /tmp (instead, the tests 
> should write to the location specified by the test.build.data system property)
> Finally, should use {{Files}} in these file creation / deletion, so that any 
> error can be thrown as {{IOE}}.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397660#comment-15397660
 ] 

Hadoop QA commented on HADOOP-12765:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  6s{color} 
| {color:red} HADOOP-12765 does not apply to trunk. Rebase required? Wrong 
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12787415/HADOOP-12765.001.patch
 |
| JIRA Issue | HADOOP-12765 |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10104/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> HttpServer2 should switch to using the non-blocking SslSelectChannelConnector 
> to prevent performance degradation when handling SSL connections
> --
>
> Key: HADOOP-12765
> URL: https://issues.apache.org/jira/browse/HADOOP-12765
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.7.2, 2.6.3
>Reporter: Min Shen
>Assignee: Min Shen
> Attachments: HADOOP-12765.001.patch, HADOOP-12765.001.patch, 
> blocking_1.png, blocking_2.png, unblocking.png
>
>
> The current implementation uses the blocking SslSocketConnector which takes 
> the default maxIdleTime as 200 seconds. We noticed in our cluster that when 
> users use a custom client that accesses the WebHDFS REST APIs through https, 
> it could block all the 250 handler threads in NN jetty server, causing severe 
> performance degradation for accessing WebHDFS and NN web UI. Attached 
> screenshots (blocking_1.png and blocking_2.png) illustrate that when using 
> SslSocketConnector, the jetty handler threads are not released until the 200 
> seconds maxIdleTime has passed. With sufficient number of SSL connections, 
> this issue could render NN HttpServer to become entirely irresponsive.
> We propose to use the non-blocking SslSelectChannelConnector as a fix. We 
> have deployed the attached patch within our cluster, and have seen 
> significant improvement. The attached screenshot (unblocking.png) further 
> illustrates the behavior of NN jetty server after switching to using 
> SslSelectChannelConnector.
> The patch further disables SSLv3 protocol on server side to preserve the 
> spirit of HADOOP-11260.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13073) RawLocalFileSystem does not react on changing umask

[Andras Bokor (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397657#comment-15397657
 ] 

Andras Bokor commented on HADOOP-13073:
---

Sure, I commented on YARN-5425.

> RawLocalFileSystem does not react on changing umask
> ---
>
> Key: HADOOP-13073
> URL: https://issues.apache.org/jira/browse/HADOOP-13073
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Reporter: Andras Bokor
>Assignee: Andras Bokor
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13073.01.patch, HADOOP-13073.02.patch, 
> HADOOP-13073.03.patch, HADOOP-13073.04.patch
>
>
> FileSystemContractBaseTest#testMkdirsWithUmask is changing umask under the 
> filesystem. RawLocalFileSystem reads the config on startup so it will not 
> react if we change the umask.
> It blocks [HADOOP-7363|https://issues.apache.org/jira/browse/HADOOP-7363] 
> since testMkdirsWithUmask test will never work with RawLocalFileSystem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397655#comment-15397655
 ] 

Wei-Chiu Chuang commented on HADOOP-12765:
--

I am interested in this patch. Is there something I can do to help push this 
forward?
Thanks.

> HttpServer2 should switch to using the non-blocking SslSelectChannelConnector 
> to prevent performance degradation when handling SSL connections
> --
>
> Key: HADOOP-12765
> URL: https://issues.apache.org/jira/browse/HADOOP-12765
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.7.2, 2.6.3
>Reporter: Min Shen
>Assignee: Min Shen
> Attachments: HADOOP-12765.001.patch, HADOOP-12765.001.patch, 
> blocking_1.png, blocking_2.png, unblocking.png
>
>
> The current implementation uses the blocking SslSocketConnector which takes 
> the default maxIdleTime as 200 seconds. We noticed in our cluster that when 
> users use a custom client that accesses the WebHDFS REST APIs through https, 
> it could block all the 250 handler threads in NN jetty server, causing severe 
> performance degradation for accessing WebHDFS and NN web UI. Attached 
> screenshots (blocking_1.png and blocking_2.png) illustrate that when using 
> SslSocketConnector, the jetty handler threads are not released until the 200 
> seconds maxIdleTime has passed. With sufficient number of SSL connections, 
> this issue could render NN HttpServer to become entirely irresponsive.
> We propose to use the non-blocking SslSelectChannelConnector as a fix. We 
> have deployed the attached patch within our cluster, and have seen 
> significant improvement. The attached screenshot (unblocking.png) further 
> illustrates the behavior of NN jetty server after switching to using 
> SslSelectChannelConnector.
> The patch further disables SSLv3 protocol on server side to preserve the 
> spirit of HADOOP-11260.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

[Pieter Reuse (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397523#comment-15397523
 ] 

Pieter Reuse commented on HADOOP-9565:
--

Thanks for patch 006, Steve. Having Strings instead of bitmasks is definitely 
an improvement. But maybe an enum is worth considering here.

I don't think the Put command adds functionality. An ObjectStore-object is 
still a FileSystem-object, having the full FileSystem interface available. Or 
am I missing something here?

Thank you for mentioning ViewFs here, Chris. It's important that this patch 
doesn't break the great ease of use ViewFs brings. But considering ViewFs is 
client-side only and this patch only brings performance enhancements, I don't 
think it is worth the extra miles checking the config of ViewFs whether the 
Path belongs to an ObjectStore which is part of a ViewFs instance.

> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Pieter Reuse
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch, 
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch, 
> HADOOP-9565-006.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really 
> blobstores, with different atomicity and consistency guarantees, by adding a 
> {{Blobstore}} interface to add to them. 
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that 
> all blobstores implement at server-side copy operation as a substitute for 
> rename.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13418) Fix javadoc warnings by JDK8 in hadoop-nfs package

[Kai Sasaki (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397502#comment-15397502
 ] 

Kai Sasaki commented on HADOOP-13418:
-

[~jojochuang] Could you review this when you get a chance?

> Fix javadoc warnings by JDK8 in hadoop-nfs package
> --
>
> Key: HADOOP-13418
> URL: https://issues.apache.org/jira/browse/HADOOP-13418
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Attachments: HADOOP-13418.01.patch, HADOOP-13418.02.patch
>
>
> Fix compile warning generated after migrating JDK8.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem

[Oscar Morante (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397490#comment-15397490
 ] 

Oscar Morante commented on HADOOP-13075:


How would you configure it with no encryption?

> Add support for SSE-KMS and SSE-C in s3a filesystem
> ---
>
> Key: HADOOP-13075
> URL: https://issues.apache.org/jira/browse/HADOOP-13075
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Andrew Olson
>Assignee: Federico Czerwinski
>
> S3 provides 3 types of server-side encryption [1],
> * SSE-S3 (Amazon S3-Managed Keys) [2]
> * SSE-KMS (AWS KMS-Managed Keys) [3]
> * SSE-C (Customer-Provided Keys) [4]
> Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 
> (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. 
> With native support in aws-java-sdk already available it should be fairly 
> straightforward [6],[7] to support the other two types of SSE with some 
> additional fs.s3a configuration properties.
> [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
> [2] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
> [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
> [4] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
> [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
> [6] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java
> [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13073) RawLocalFileSystem does not react on changing umask

[Akira Ajisaka (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397394#comment-15397394
 ] 

Akira Ajisaka commented on HADOOP-13073:


Hi [~boky01] and [~ste...@apache.org], this commit broke 
TestDirectoryCollection. Would you check YARN-5425?

> RawLocalFileSystem does not react on changing umask
> ---
>
> Key: HADOOP-13073
> URL: https://issues.apache.org/jira/browse/HADOOP-13073
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Reporter: Andras Bokor
>Assignee: Andras Bokor
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13073.01.patch, HADOOP-13073.02.patch, 
> HADOOP-13073.03.patch, HADOOP-13073.04.patch
>
>
> FileSystemContractBaseTest#testMkdirsWithUmask is changing umask under the 
> filesystem. RawLocalFileSystem reads the config on startup so it will not 
> react if we change the umask.
> It blocks [HADOOP-7363|https://issues.apache.org/jira/browse/HADOOP-7363] 
> since testMkdirsWithUmask test will never work with RawLocalFileSystem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13354) Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft Azure Storage Clients

[Sivaguru Sankaridurg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397386#comment-15397386
 ] 

Sivaguru Sankaridurg commented on HADOOP-13354:
---

Thanks [~cnauroth].

-Siva

> Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft 
> Azure Storage Clients
> ---
>
> Key: HADOOP-13354
> URL: https://issues.apache.org/jira/browse/HADOOP-13354
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Fix For: 2.9.0
>
> Attachments: HADOOP-13354.001.patch, HADOOP-13354.002.patch, 
> HADOOP-13354.003.patch, HADOOP-13354.004.patch, Test-Results-With-4.2.0-fixes
>
>
> Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft 
> Azure Storage Clients.
> We are currently using version 2.2.0 of the SDK.
> Version 4.2.0 brings some breaking changes. 
> Need to fix code to resolve all these breaking changes and certify that 
> everything works properly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13388) Clean up TestLocalFileSystemPermission

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397356#comment-15397356
 ] 

Hadoop QA commented on HADOOP-13388:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
8s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m  
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} hadoop-common-project/hadoop-common: The patch 
generated 0 new + 0 unchanged - 11 fixed = 0 total (was 11) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m  
5s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 35s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820680/HADOOP-13388.01.patch 
|
| JIRA Issue | HADOOP-13388 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 3412b37c66f4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 414fbfa |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10103/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10103/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Clean up TestLocalFileSystemPermission
> --
>
> Key: HADOOP-13388
> URL: https://issues.apache.org/jira/browse/HADOOP-13388
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Trivial
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13388.01.patch
>
>
> I see more problems with 

[jira] [Commented] (HADOOP-13263) Reload cached groups in background after expiry

[Stephen O'Donnell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397341#comment-15397341
 ] 

Stephen O'Donnell commented on HADOOP-13263:


[~arpitagarwal] Looks good to me.

> Reload cached groups in background after expiry
> ---
>
> Key: HADOOP-13263
> URL: https://issues.apache.org/jira/browse/HADOOP-13263
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
> Fix For: 2.8.0
>
> Attachments: HADOOP-13263.001.patch, HADOOP-13263.002.patch, 
> HADOOP-13263.003.patch, HADOOP-13263.004.patch, HADOOP-13263.005.patch, 
> HADOOP-13263.006.patch, HADOOP-13263.007.patch
>
>
> In HADOOP-11238 the Guava cache was introduced to allow refreshes on the 
> Namenode group cache to run in the background, avoiding many slow group 
> lookups. Even with this change, I have seen quite a few clusters with issues 
> due to slow group lookups. The problem is most prevalent in HA clusters, 
> where a slow group lookup on the hdfs user can fail to return for over 45 
> seconds causing the Failover Controller to kill it.
> The way the current Guava cache implementation works is approximately:
> 1) On initial load, the first thread to request groups for a given user 
> blocks until it returns. Any subsequent threads requesting that user block 
> until that first thread populates the cache.
> 2) When the key expires, the first thread to hit the cache after expiry 
> blocks. While it is blocked, other threads will return the old value.
> I feel it is this blocking thread that still gives the Namenode issues on 
> slow group lookups. If the call from the FC is the one that blocks and 
> lookups are slow, if can cause the NN to be killed.
> Guava has the ability to refresh expired keys completely in the background, 
> where the first thread that hits an expired key schedules a background cache 
> reload, but still returns the old value. Then the cache is eventually 
> updated. This patch introduces this background reload feature. There are two 
> new parameters:
> 1) hadoop.security.groups.cache.background.reload - default false to keep the 
> current behaviour. Set to true to enable a small thread pool and background 
> refresh for expired keys
> 2) hadoop.security.groups.cache.background.reload.threads - only relevant if 
> the above is set to true. Controls how many threads are in the background 
> refresh pool. Default is 1, which is likely to be enough.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13388) Clean up TestLocalFileSystemPermission

[Andras Bokor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Bokor updated HADOOP-13388:
--
Status: Patch Available  (was: Open)

> Clean up TestLocalFileSystemPermission
> --
>
> Key: HADOOP-13388
> URL: https://issues.apache.org/jira/browse/HADOOP-13388
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Trivial
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13388.01.patch
>
>
> I see more problems with {{TestLocalFileSystemPermission}}:
> * Many checkstyle warnings
> * Relays on JUnit3 so Assume framework cannot be used for Windows checks.
> * In the tests in case of exception we get an error message but the test 
> itself will pass (because of the return).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13388) Clean up TestLocalFileSystemPermission

[Andras Bokor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Bokor updated HADOOP-13388:
--
Attachment: HADOOP-13388.01.patch

Uploading 1st patch.
I removed the catch blocks because it does make sense for me to let a test pass 
if we get an exception. Do I misunderstand something?

> Clean up TestLocalFileSystemPermission
> --
>
> Key: HADOOP-13388
> URL: https://issues.apache.org/jira/browse/HADOOP-13388
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Trivial
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13388.01.patch
>
>
> I see more problems with {{TestLocalFileSystemPermission}}:
> * Many checkstyle warnings
> * Relays on JUnit3 so Assume framework cannot be used for Windows checks.
> * In the tests in case of exception we get an error message but the test 
> itself will pass (because of the return).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13191) FileSystem#listStatus should not return null

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397234#comment-15397234
 ] 

Hadoop QA commented on HADOOP-13191:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
41s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  5m 
16s{color} | {color:red} hadoop-common in trunk failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
33s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 22m  8s{color} 
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
3s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m 
50s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
28s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}113m 37s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.http.TestHttpServerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820628/HADOOP-13191.004.patch
 |
| JIRA Issue | HADOOP-13191 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 5b730640303b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 8d06bda |
| Default Java | 1.8.0_101 |
| mvnsite | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10102/artifact/patchprocess/branch-mvnsite-hadoop-common-project_hadoop-common.txt
 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10102/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
 |
|  Test Results