[jira] [Created] (HADOOP-13032) Refactor FileSystem$Statistics as a separate source file

[Mingliang Liu (JIRA)]

Mingliang Liu created HADOOP-13032:
--

 Summary: Refactor FileSystem$Statistics as a separate source file
 Key: HADOOP-13032
 URL: https://issues.apache.org/jira/browse/HADOOP-13032
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Priority: Minor


This jira is to track the effort of moving the {{Statistics}} class out of 
{{FileSystem}} for:
# shorter source code ({{FileSystem.java}} is gonna have 3780 LoC after 
[HDFS-10175])
# simpler class structure (the {{Statistics}} is not very straightforward as 
it's optimized for performance in [HDFS-5276] and evloving rapidly recently, 
see [HDFS-9579], [HDFS-10175] and [MAPREDUCE-6660]).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-13031) Refactor the code that maintains rack-aware counters in FileSystem$Statistics

[Mingliang Liu (JIRA)]

Mingliang Liu created HADOOP-13031:
--

 Summary: Refactor the code that maintains rack-aware counters in 
FileSystem$Statistics
 Key: HADOOP-13031
 URL: https://issues.apache.org/jira/browse/HADOOP-13031
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Mingliang Liu
Priority: Minor


According to discussion in [HDFS-10175], using a composite (e.g. enum map, 
array) data structure to manage the distance->bytesRead mapping will probably 
make the code simpler.

# {{StatisticsData}} will be a bit shorter by delegating the operations to the 
composite data structure.
# The {{incrementBytesReadByDistance(int distance, long newBytes)}} and 
{{getBytesReadByDistance(int distance)}} which switch-case all hard-code 
variables, may be simplified as we can set/get the {{bytesRead}} by distance 
directly from map/array.

This jira is to track the discussion and effort of refactoring the code that 
maintains rack-aware counters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-13030) KMS startup script failed to handle special characters

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13030:
--

 Summary: KMS startup script failed to handle special characters
 Key: HADOOP-13030
 URL: https://issues.apache.org/jira/browse/HADOOP-13030
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen


{{kms.sh}} currently cannot handle special characters.

{code}
 sed -e 's/_kms_ssl_keystore_pass_/'${KMS_SSL_KEYSTORE_PASS}'/g' \
-e 's/_kms_ssl_truststore_pass_/'${KMS_SSL_TRUSTSTORE_PASS}'/g' \
"${HADOOP_CATALINA_HOME}/conf/ssl-server.xml.conf" \
> "${HADOOP_CATALINA_HOME}/conf/ssl-server.xml"
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : Hadoop-Common-trunk #2637

[Apache Jenkins Server]

See 

[jira] [Created] (HADOOP-13029) Have FairCallQueue try all lower priority sub queues before backoff

[Ming Ma (JIRA)]

Ming Ma created HADOOP-13029:


 Summary: Have FairCallQueue try all lower priority sub queues 
before backoff
 Key: HADOOP-13029
 URL: https://issues.apache.org/jira/browse/HADOOP-13029
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ming Ma


Currently if FairCallQueue and backoff are enabled, backoff will kick in as 
soon as the assigned sub queue is filled up.

{noformat}
  /**
   * Put and offer follow the same pattern:
   * 1. Get the assigned priorityLevel from the call by scheduler
   * 2. Get the nth sub-queue matching this priorityLevel
   * 3. delegate the call to this sub-queue.
   *
   * But differ in how they handle overflow:
   * - Put will move on to the next queue until it lands on the last queue
   * - Offer does not attempt other queues on overflow
   */
{noformat}

Seems it is better to try lower priority sub queues when the assigned sub queue 
is full, just like the case when backoff is disabled. This will give regular 
users more opportunities and allow the cluster to be configured with smaller 
call queue length. [~chrili], [~arpitagarwal], what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: Hadoop-Common-trunk #2636

[Apache Jenkins Server]

See 

Changes:

[jlowe] YARN-4940. yarn node -list -all failed if RM start with decommissioned

--
[...truncated 5139 lines...]
Running org.apache.hadoop.security.TestUGIWithExternalKdc
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.177 sec - in 
org.apache.hadoop.security.TestUGIWithExternalKdc
Running org.apache.hadoop.security.http.TestRestCsrfPreventionFilter
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.116 sec - in 
org.apache.hadoop.security.http.TestRestCsrfPreventionFilter
Running org.apache.hadoop.security.http.TestCrossOriginFilter
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.177 sec - in 
org.apache.hadoop.security.http.TestCrossOriginFilter
Running org.apache.hadoop.security.TestProxyUserFromEnv
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.414 sec - in 
org.apache.hadoop.security.TestProxyUserFromEnv
Running org.apache.hadoop.security.authorize.TestProxyUsers
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.894 sec - in 
org.apache.hadoop.security.authorize.TestProxyUsers
Running org.apache.hadoop.security.authorize.TestProxyServers
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.265 sec - in 
org.apache.hadoop.security.authorize.TestProxyServers
Running org.apache.hadoop.security.authorize.TestServiceAuthorization
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.991 sec - in 
org.apache.hadoop.security.authorize.TestServiceAuthorization
Running org.apache.hadoop.security.authorize.TestAccessControlList
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.425 sec - in 
org.apache.hadoop.security.authorize.TestAccessControlList
Running org.apache.hadoop.security.alias.TestCredShell
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.266 sec - in 
org.apache.hadoop.security.alias.TestCredShell
Running org.apache.hadoop.security.alias.TestCredentialProviderFactory
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.389 sec - in 
org.apache.hadoop.security.alias.TestCredentialProviderFactory
Running org.apache.hadoop.security.alias.TestCredentialProvider
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.822 sec - in 
org.apache.hadoop.security.alias.TestCredentialProvider
Running org.apache.hadoop.security.TestAuthenticationFilter
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.675 sec - in 
org.apache.hadoop.security.TestAuthenticationFilter
Running org.apache.hadoop.security.TestLdapGroupsMapping
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.574 sec - in 
org.apache.hadoop.security.TestLdapGroupsMapping
Running org.apache.hadoop.security.token.TestToken
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.435 sec - in 
org.apache.hadoop.security.token.TestToken
Running org.apache.hadoop.security.token.delegation.TestDelegationToken
Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.871 sec - 
in org.apache.hadoop.security.token.delegation.TestDelegationToken
Running 
org.apache.hadoop.security.token.delegation.web.TestDelegationTokenAuthenticationHandlerWithMocks
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.405 sec - in 
org.apache.hadoop.security.token.delegation.web.TestDelegationTokenAuthenticationHandlerWithMocks
Running 
org.apache.hadoop.security.token.delegation.web.TestDelegationTokenManager
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.681 sec - in 
org.apache.hadoop.security.token.delegation.web.TestDelegationTokenManager
Running org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.144 sec - 
in org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken
Running 
org.apache.hadoop.security.token.delegation.TestZKDelegationTokenSecretManager
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 86.499 sec <<< 
FAILURE! - in 
org.apache.hadoop.security.token.delegation.TestZKDelegationTokenSecretManager
testMultiNodeOperations(org.apache.hadoop.security.token.delegation.TestZKDelegationTokenSecretManager)
  Time elapsed: 13.082 sec  <<< FAILURE!
java.lang.AssertionError: Expected InvalidToken
at org.junit.Assert.fail(Assert.java:88)
at 
org.apache.hadoop.security.token.delegation.TestZKDelegationTokenSecretManager.testMultiNodeOperations(TestZKDelegationTokenSecretManager.java:113)

Running org.apache.hadoop.security.TestShellBasedUnixGroupsMapping
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.91 sec - in 
org.apache.hadoop.security.TestShellBasedUnixGroupsMapping
Running org.apache.hadoop.security.TestShellBasedIdMapping
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.304 sec - in 
org.apache.hadoop.security.TestShellBasedIdMapping

Jenkins build is back to normal : Hadoop-common-trunk-Java8 #1348

[Apache Jenkins Server]

See 

Build failed in Jenkins: Hadoop-common-trunk-Java8 #1347

[Apache Jenkins Server]

See 

Changes:

[naganarasimha_gr] YARN-4909. Fix intermittent failures of TestRMWebServices And

--
[...truncated 3871 lines...]
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
hadoop-minikdc ---
[INFO] Compiling 2 source files to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-minikdc ---
[INFO] Surefire report directory: 


---
 T E S T S
---

---
 T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.839 sec - in 
org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.minikdc.TestMiniKdc
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.062 sec - in 
org.apache.hadoop.minikdc.TestMiniKdc

Results :

Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ 
hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ hadoop-minikdc 
---
[INFO] 
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ 
hadoop-minikdc ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-minikdc ---
[INFO] 
Loading source files for package org.apache.hadoop.minikdc...
Constructing Javadoc information...
Standard Doclet version 1.8.0
Building tree for all the packages and classes...
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Building index for all the packages and classes...
Generating 

Generating 

Generating 

Building index for all classes...
Generating 

Build failed in Jenkins: Hadoop-common-trunk-Java8 #1346

[Apache Jenkins Server]

See 

Changes:

[jing9] HDFS-10283.

[jing9] HDFS-10293. StripedFileTestUtil#readAll flaky. Contributed by Mingliang

--
[...truncated 3871 lines...]
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
hadoop-minikdc ---
[INFO] Compiling 2 source files to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-minikdc ---
[INFO] Surefire report directory: 


---
 T E S T S
---

---
 T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.431 sec - in 
org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.minikdc.TestMiniKdc
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.227 sec - in 
org.apache.hadoop.minikdc.TestMiniKdc

Results :

Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ 
hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ hadoop-minikdc 
---
[INFO] 
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ 
hadoop-minikdc ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-minikdc ---
[INFO] 
Loading source files for package org.apache.hadoop.minikdc...
Constructing Javadoc information...
Standard Doclet version 1.8.0
Building tree for all the packages and classes...
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Building index for all the packages and classes...
Generating 

Generating 

Generating 

Building index for all classes...
Generating 

Build failed in Jenkins: Hadoop-common-trunk-Java8 #1345

[Apache Jenkins Server]

See 

Changes:

[stevel] HADOOP-13026 Should not wrap IOExceptions into a 
AuthenticationException

--
[...truncated 3882 lines...]
---
 T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.371 sec - in 
org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.minikdc.TestMiniKdc
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.111 sec - in 
org.apache.hadoop.minikdc.TestMiniKdc

Results :

Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ 
hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ hadoop-minikdc 
---
[INFO] 
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ 
hadoop-minikdc ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-minikdc ---
[INFO] 
Loading source files for package org.apache.hadoop.minikdc...
Constructing Javadoc information...
Standard Doclet version 1.8.0
Building tree for all the packages and classes...
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Building index for all the packages and classes...
Generating 

Generating 

Generating 

Building index for all classes...
Generating 

Generating 

Generating 

Generating 

2 warnings
[WARNING] Javadoc Warnings
[WARNING] 
/home/jenkins/.m2/repository/org/apache/directory/api/api-ldap-model/1.0.0-M20/api-ldap-model-1.0.0-M20.jar(org/apache/directory/api/ldap/model/name/Dn.class)

[jira] [Created] (HADOOP-13028) add counter and timer metrics for S3A operations

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-13028:
---

 Summary: add counter and timer metrics for S3A operations
 Key: HADOOP-13028
 URL: https://issues.apache.org/jira/browse/HADOOP-13028
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3, metrics
Affects Versions: 2.8.0
Reporter: Steve Loughran
Priority: Minor


against S3 (and other object stores), opening connections can be expensive, 
closing connections may be expensive (a sign of a regression). 

S3A FS and individual input streams should have counters of the # of 
open/close/failure+reconnect operations, timers of how long things take. This 
can be used downstream to measure efficiency of the code (how often connections 
are being made), connection reliability, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)