[jira] [Comment Edited] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677635#comment-16677635 ] Wanqiang Ji edited comment on HADOOP-15907 at 11/7/18 6:31 AM: --- Hi [~cheersyang] Thanks for your suggestion, I had update the patch. Please help to review again, thx~ Pending on Jenkins. was (Author: jiwq): Hi [~cheersyang] Thanks for you suggestion, I had update the patch. Please help to review again, thx~ Pending on Jenkins. > Add YARN module to the main modules section in BUILDING.txt > --- > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Trivial > Attachments: HADOOP-15907.001.patch, HADOOP-15907.002.patch, > HADOOP-15907.003.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji reassigned HADOOP-15907: Assignee: Wanqiang Ji > Add YARN module to the main modules section in BUILDING.txt > --- > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Trivial > Attachments: HADOOP-15907.001.patch, HADOOP-15907.002.patch, > HADOOP-15907.003.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local
[
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677680#comment-16677680
]
Akira Ajisaka commented on HADOOP-15908:
Thank you for the patch, [~oshevchenko].
-1, we have to include LICENSE.txt and NOTICE.txt in the jar files.
before
{noformat}
$ jar tf
hadoop-common-project/hadoop-common/target/hadoop-common-3.3.0-SNAPSHOT.jar |
grep LICENSE.txt
META-INF/LICENSE.txt
{noformat}
after
{noformat}
$ jar tf
hadoop-common-project/hadoop-common/target/hadoop-common-3.3.0-SNAPSHOT.jar |
grep LICENSE.txt
{noformat}
> hadoop-build-tools jar is downloaded from remote repository instead of using
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15908.001.patch
>
>
> HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml
> to verify LICENSE.txt and NOTICE.txt files which includes
> "hadoop-build-tools" remote resource bundles.
> {code}
>
> org.apache.maven.plugins
> maven-remote-resources-plugin
> ${maven-remote-resources-plugin.version}
>
>
>
> org.apache.hadoop:hadoop-build-tools:${hadoop.version}
>
>
>
>
> org.apache.hadoop
> hadoop-build-tools
> ${hadoop.version}
>
>
>
>
>
> process
>
>
>
>
> {code}
> If we build only some module we always download " hadoop-build-tools" from
> maven repository.
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @
> hadoop-annotations ---
> Downloading from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> (791 B at 684 B/s)
> Downloading from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> (609 B at 547 B/s)
> Downloading from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> (791 B at 343 B/s)
> Downloading from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> Downloaded from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> (0 B at 0 B/s)
> {noformat}
> If "hadoop-build-tools" jar doesn't exist in maven repository (for example we
> try to build new version locally before repository will be created ) we can't
> build some module:
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default)
> on project hadoop-annotations: Execution default of goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed:
> Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of
> its dependencies could not be resolved: Failure to find
> org.apache.hadoop:hadoop-build-tools:jar:3.2.0 in
> https://repo.maven.apache.org/maven2 was cached in the local repository,
> resolution will not be reattempted until the update interval of central has
> elapsed or updates are forced -> [Help 1]
> {noformat}
> Therefore, we need to limit execution of the Remote Resources Plugin only in
> the root directory in which the build was run.
> To accomplish this, we can use the "runOnlyAtExecutionRoot parameter"
> From maven documentation
> http://maven.apache.org/plugins/maven-remote-resources-plugin/usage.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[
https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677652#comment-16677652
]
Hadoop QA commented on HADOOP-15907:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15907 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947173/HADOOP-15907.003.patch
|
| Optional Tests | dupname asflicense |
| uname | Linux 6160981a91e7 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 482716e |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 364 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15481/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add YARN module to the main modules section in BUILDING.txt
> ---
>
> Key: HADOOP-15907
> URL: https://issues.apache.org/jira/browse/HADOOP-15907
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wanqiang Ji
>Priority: Trivial
> Attachments: HADOOP-15907.001.patch, HADOOP-15907.002.patch,
> HADOOP-15907.003.patch
>
>
> In the Maven main modules section, I found the missing main module YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15876) Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore
[
https://issues.apache.org/jira/browse/HADOOP-15876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677619#comment-16677619
]
Dedunu Dhananjaya edited comment on HADOOP-15876 at 11/7/18 3:46 AM:
-
[~DanielZhou] I created a patch file but I can't attach it. It doesn't work it
seems.
was (Author: dedunumax):
Sure, Will create a patch an attach.
> Use keySet().removeAll() to remove multiple keys from Map in
> AzureBlobFileSystemStore
> -
>
> Key: HADOOP-15876
> URL: https://issues.apache.org/jira/browse/HADOOP-15876
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Ted Yu
>Priority: Minor
>
> Looking at
> hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
> , {{removeDefaultAcl}} in particular:
> {code}
> for (Map.Entry defaultAclEntry :
> defaultAclEntries.entrySet()) {
> aclEntries.remove(defaultAclEntry.getKey());
> }
> {code}
> The above operation can be written this way:
> {code}
> aclEntries.keySet().removeAll(defaultAclEntries.keySet());
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677635#comment-16677635 ] Wanqiang Ji commented on HADOOP-15907: -- Hi [~cheersyang] Thanks for you suggestion, I had update the patch. Please help to review again, thx~ > Add YARN module to the main modules section in BUILDING.txt > --- > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Priority: Trivial > Attachments: HADOOP-15907.001.patch, HADOOP-15907.002.patch, > HADOOP-15907.003.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677635#comment-16677635 ] Wanqiang Ji edited comment on HADOOP-15907 at 11/7/18 3:42 AM: --- Hi [~cheersyang] Thanks for you suggestion, I had update the patch. Please help to review again, thx~ Pending on Jenkins. was (Author: jiwq): Hi [~cheersyang] Thanks for you suggestion, I had update the patch. Please help to review again, thx~ > Add YARN module to the main modules section in BUILDING.txt > --- > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Priority: Trivial > Attachments: HADOOP-15907.001.patch, HADOOP-15907.002.patch, > HADOOP-15907.003.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji updated HADOOP-15907: - Attachment: HADOOP-15907.003.patch > Add YARN module to the main modules section in BUILDING.txt > --- > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Priority: Trivial > Attachments: HADOOP-15907.001.patch, HADOOP-15907.002.patch, > HADOOP-15907.003.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-15876) Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore
[
https://issues.apache.org/jira/browse/HADOOP-15876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dedunu Dhananjaya updated HADOOP-15876:
---
Comment: was deleted
(was: Attaching the patch file)
> Use keySet().removeAll() to remove multiple keys from Map in
> AzureBlobFileSystemStore
> -
>
> Key: HADOOP-15876
> URL: https://issues.apache.org/jira/browse/HADOOP-15876
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Ted Yu
>Priority: Minor
>
> Looking at
> hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
> , {{removeDefaultAcl}} in particular:
> {code}
> for (Map.Entry defaultAclEntry :
> defaultAclEntries.entrySet()) {
> aclEntries.remove(defaultAclEntry.getKey());
> }
> {code}
> The above operation can be written this way:
> {code}
> aclEntries.keySet().removeAll(defaultAclEntries.keySet());
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15876) Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore
[
https://issues.apache.org/jira/browse/HADOOP-15876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677632#comment-16677632
]
Dedunu Dhananjaya commented on HADOOP-15876:
Attaching the patch file
> Use keySet().removeAll() to remove multiple keys from Map in
> AzureBlobFileSystemStore
> -
>
> Key: HADOOP-15876
> URL: https://issues.apache.org/jira/browse/HADOOP-15876
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Ted Yu
>Priority: Minor
>
> Looking at
> hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
> , {{removeDefaultAcl}} in particular:
> {code}
> for (Map.Entry defaultAclEntry :
> defaultAclEntries.entrySet()) {
> aclEntries.remove(defaultAclEntry.getKey());
> }
> {code}
> The above operation can be written this way:
> {code}
> aclEntries.keySet().removeAll(defaultAclEntries.keySet());
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15895) [JDK9+] Add missing javax.annotation-api dependency to hadoop-yarn-csi
[
https://issues.apache.org/jira/browse/HADOOP-15895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677622#comment-16677622
]
Weiwei Yang commented on HADOOP-15895:
--
I also noticed this when I tried to compile with java 11, thanks for getting
this done [~tasanuma0829].
> [JDK9+] Add missing javax.annotation-api dependency to hadoop-yarn-csi
> --
>
> Key: HADOOP-15895
> URL: https://issues.apache.org/jira/browse/HADOOP-15895
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15895.1.patch
>
>
> Javadoc build fails in hadoop-yarn-csi due to missing {{javax.annotation}}.
> {noformat}
> $ mvn javadoc:javadoc --projects
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi
> ...
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:jar (module-javadocs) on
> project hadoop-yarn-csi: MavenReportException: Error while generating Javadoc:
> [ERROR] Exit code: 1 -
> /hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/target/generated-sources/protobuf/grpc-java/csi/v0/IdentityGrpc.java:20:
> error: cannot find symbol
> [ERROR] @javax.annotation.Generated(
> [ERROR] ^
> [ERROR] symbol: class Generated
> [ERROR] location: package javax.annotation
> [ERROR]
> /hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/target/generated-sources/protobuf/grpc-java/csi/v0/ControllerGrpc.java:20:
> error: cannot find symbol
> [ERROR] @javax.annotation.Generated(
> [ERROR] ^
> [ERROR] symbol: class Generated
> [ERROR] location: package javax.annotation
> [ERROR]
> /hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/target/generated-sources/protobuf/grpc-java/csi/v0/NodeGrpc.java:20:
> error: cannot find symbol
> [ERROR] @javax.annotation.Generated(
> [ERROR] ^
> [ERROR] symbol: class Generated
> [ERROR] location: package javax.annotation
> [ERROR]
> [ERROR] Command line was: /usr/java/jdk-9.0.4/bin/javadoc @options @packages
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15876) Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore
[
https://issues.apache.org/jira/browse/HADOOP-15876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677619#comment-16677619
]
Dedunu Dhananjaya commented on HADOOP-15876:
Sure, Will create a patch an attach.
> Use keySet().removeAll() to remove multiple keys from Map in
> AzureBlobFileSystemStore
> -
>
> Key: HADOOP-15876
> URL: https://issues.apache.org/jira/browse/HADOOP-15876
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Ted Yu
>Priority: Minor
>
> Looking at
> hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
> , {{removeDefaultAcl}} in particular:
> {code}
> for (Map.Entry defaultAclEntry :
> defaultAclEntries.entrySet()) {
> aclEntries.remove(defaultAclEntry.getKey());
> }
> {code}
> The above operation can be written this way:
> {code}
> aclEntries.keySet().removeAll(defaultAclEntries.keySet());
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677603#comment-16677603 ] Weiwei Yang commented on HADOOP-15907: -- Hi [~jiwq] Mostly seems good, some nitpicks * hadoop-hdds: Hadoop Distributed Data Store * hadoop-minicluster: Hadoop minicluster artifacts * hadoop-client-modules: Hadoop client modules * hadoop-cloud-storage-project: Generates artifacts to access cloud storage like aws, azure, etc. > Add YARN module to the main modules section in BUILDING.txt > --- > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Priority: Trivial > Attachments: HADOOP-15907.001.patch, HADOOP-15907.002.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15876) Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore
[
https://issues.apache.org/jira/browse/HADOOP-15876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677429#comment-16677429
]
Da Zhou commented on HADOOP-15876:
--
Hi [~dedunumax], thanks for providing this fix.
Instead of creating a PR, could you attach the patch here? Also it will be
great if you can paste the tests results here.
For the cmds you need to run mvn tests, please check
[here|https://issues.apache.org/jira/browse/HADOOP-15740?focusedCommentId=16611070=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16611070].
> Use keySet().removeAll() to remove multiple keys from Map in
> AzureBlobFileSystemStore
> -
>
> Key: HADOOP-15876
> URL: https://issues.apache.org/jira/browse/HADOOP-15876
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Ted Yu
>Priority: Minor
>
> Looking at
> hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
> , {{removeDefaultAcl}} in particular:
> {code}
> for (Map.Entry defaultAclEntry :
> defaultAclEntries.entrySet()) {
> aclEntries.remove(defaultAclEntry.getKey());
> }
> {code}
> The above operation can be written this way:
> {code}
> aclEntries.keySet().removeAll(defaultAclEntries.keySet());
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15860) ABFS: Trailing period in file names gets ignored for some operations.
[
https://issues.apache.org/jira/browse/HADOOP-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677376#comment-16677376
]
Da Zhou edited comment on HADOOP-15860 at 11/6/18 11:35 PM:
[~mackrorysd], sorry for the late reply, this Jira was postponed due to a
conflict.
I find that Blob name end with period is not recommended, according to [Azure
doc:|https://docs.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata]：
{noformat}
Avoid blob names that end with a dot (.), a forward slash (/), or a sequence or
combination of the two.
{noformat}
I am checking if this is some restriction or a bug.
was (Author: danielzhou):
[~mackrorysd], actually I found that Blob name end with period is not
recommended, according to [Azure
doc:|https://docs.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata]：
{noformat}
Avoid blob names that end with a dot (.), a forward slash (/), or a sequence or
combination of the two.
{noformat}
> ABFS: Trailing period in file names gets ignored for some operations.
> -
>
> Key: HADOOP-15860
> URL: https://issues.apache.org/jira/browse/HADOOP-15860
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/adl
>Affects Versions: 3.2.0
>Reporter: Sean Mackrory
>Priority: Major
> Attachments: trailing-periods.patch
>
>
> If you create a directory with a trailing period (e.g. '/test.') the period
> is silently dropped, and will be listed as simply '/test'. '/test.test'
> appears to work just fine.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15860) ABFS: Trailing period in file names gets ignored for some operations.
[
https://issues.apache.org/jira/browse/HADOOP-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677376#comment-16677376
]
Da Zhou commented on HADOOP-15860:
--
[~mackrorysd], actually I found that Blob name end with period is not
recommended, according to [Azure
doc:|https://docs.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata]：
{noformat}
Avoid blob names that end with a dot (.), a forward slash (/), or a sequence or
combination of the two.
{noformat}
> ABFS: Trailing period in file names gets ignored for some operations.
> -
>
> Key: HADOOP-15860
> URL: https://issues.apache.org/jira/browse/HADOOP-15860
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/adl
>Affects Versions: 3.2.0
>Reporter: Sean Mackrory
>Priority: Major
> Attachments: trailing-periods.patch
>
>
> If you create a directory with a trailing period (e.g. '/test.') the period
> is silently dropped, and will be listed as simply '/test'. '/test.test'
> appears to work just fine.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14128) ChecksumFs should override rename with overwrite flag
[ https://issues.apache.org/jira/browse/HADOOP-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677332#comment-16677332 ] Hrishikesh Gadre commented on HADOOP-14128: --- [~mathieu_c] given that the last pre-commit run is successful with my patch containing your test only changes (but not the actual fix), I am inclined to close this jira as "Not a Bug". Please let me know if you have any thoughts otherwise. CC [~xiaochen] > ChecksumFs should override rename with overwrite flag > - > > Key: HADOOP-14128 > URL: https://issues.apache.org/jira/browse/HADOOP-14128 > Project: Hadoop Common > Issue Type: Bug > Components: common, fs >Affects Versions: 2.8.1 >Reporter: Mathieu Chataigner >Assignee: Hrishikesh Gadre >Priority: Major > Attachments: HADOOP-14128-003.patch, > HADOOP-14128-branch-2.8.1-001.patch, HADOOP-14128-branch2.8.1-001.patch, > HADOOP-14128.001.patch, HADOOP-14128.002.patch > > > When I call FileContext.rename(src, dst, Options.Rename.OVERWRITE) on a > LocalFs (which extends ChecksumFs), it does not update crc files. > Every subsequent read on moved files will result in failures due to crc > missmatch. > One solution is to override rename(src, dst, overwrite) the same way it's > done with rename(src, dst) and moving crc files accordingly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14128) ChecksumFs should override rename with overwrite flag
[
https://issues.apache.org/jira/browse/HADOOP-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677310#comment-16677310
]
Hadoop QA commented on HADOOP-14128:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 72 unchanged - 0 fixed = 73 total (was 72) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
47s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-14128 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947119/HADOOP-14128-003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux fe7cc404c91b 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 08d69d9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15480/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15480/testReport/ |
| Max. process+thread count | 1765 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15480/console |
| Powered by |
[jira] [Updated] (HADOOP-14128) ChecksumFs should override rename with overwrite flag
[ https://issues.apache.org/jira/browse/HADOOP-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hrishikesh Gadre updated HADOOP-14128: -- Attachment: HADOOP-14128-003.patch > ChecksumFs should override rename with overwrite flag > - > > Key: HADOOP-14128 > URL: https://issues.apache.org/jira/browse/HADOOP-14128 > Project: Hadoop Common > Issue Type: Bug > Components: common, fs >Affects Versions: 2.8.1 >Reporter: Mathieu Chataigner >Assignee: Hrishikesh Gadre >Priority: Major > Attachments: HADOOP-14128-003.patch, > HADOOP-14128-branch-2.8.1-001.patch, HADOOP-14128-branch2.8.1-001.patch, > HADOOP-14128.001.patch, HADOOP-14128.002.patch > > > When I call FileContext.rename(src, dst, Options.Rename.OVERWRITE) on a > LocalFs (which extends ChecksumFs), it does not update crc files. > Every subsequent read on moved files will result in failures due to crc > missmatch. > One solution is to override rename(src, dst, overwrite) the same way it's > done with rename(src, dst) and moving crc files accordingly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14128) ChecksumFs should override rename with overwrite flag
[ https://issues.apache.org/jira/browse/HADOOP-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677174#comment-16677174 ] Hrishikesh Gadre commented on HADOOP-14128: --- [~xiaochen] mentioned that the pre-commit build for branch-2.8 is broken. Let me resubmit the patch against the trunk. > ChecksumFs should override rename with overwrite flag > - > > Key: HADOOP-14128 > URL: https://issues.apache.org/jira/browse/HADOOP-14128 > Project: Hadoop Common > Issue Type: Bug > Components: common, fs >Affects Versions: 2.8.1 >Reporter: Mathieu Chataigner >Assignee: Hrishikesh Gadre >Priority: Major > Attachments: HADOOP-14128-branch-2.8.1-001.patch, > HADOOP-14128-branch2.8.1-001.patch, HADOOP-14128.001.patch, > HADOOP-14128.002.patch > > > When I call FileContext.rename(src, dst, Options.Rename.OVERWRITE) on a > LocalFs (which extends ChecksumFs), it does not update crc files. > Every subsequent read on moved files will result in failures due to crc > missmatch. > One solution is to override rename(src, dst, overwrite) the same way it's > done with rename(src, dst) and moving crc files accordingly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[
https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676863#comment-16676863
]
Hadoop QA commented on HADOOP-15907:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15907 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947082/HADOOP-15907.002.patch
|
| Optional Tests | dupname asflicense |
| uname | Linux d75a121140e6 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 49412a1 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 307 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15479/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add YARN module to the main modules section in BUILDING.txt
> ---
>
> Key: HADOOP-15907
> URL: https://issues.apache.org/jira/browse/HADOOP-15907
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wanqiang Ji
>Priority: Trivial
> Attachments: HADOOP-15907.001.patch, HADOOP-15907.002.patch
>
>
> In the Maven main modules section, I found the missing main module YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676826#comment-16676826 ] Wanqiang Ji edited comment on HADOOP-15907 at 11/6/18 2:34 PM: --- Hi, [~cheersyang]. The new patch updated, please help to review this. Thx~ Pending on Jenkins. was (Author: jiwq): Hi, [~cheersyang]. The new patch updated, please help to review this. Thx~ > Add YARN module to the main modules section in BUILDING.txt > --- > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Priority: Trivial > Attachments: HADOOP-15907.001.patch, HADOOP-15907.002.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji updated HADOOP-15907: - Comment: was deleted (was: [~cheersyang] OK, I will submit new patch.) > Add YARN module to the main modules section in BUILDING.txt > --- > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Priority: Trivial > Attachments: HADOOP-15907.001.patch, HADOOP-15907.002.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676826#comment-16676826 ] Wanqiang Ji commented on HADOOP-15907: -- Hi, [~cheersyang]. The new patch updated, please help to review this. Thx~ > Add YARN module to the main modules section in BUILDING.txt > --- > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Priority: Trivial > Attachments: HADOOP-15907.001.patch, HADOOP-15907.002.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji updated HADOOP-15907: - Attachment: HADOOP-15907.002.patch > Add YARN module to the main modules section in BUILDING.txt > --- > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Priority: Trivial > Attachments: HADOOP-15907.001.patch, HADOOP-15907.002.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local
[
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676810#comment-16676810
]
Hadoop QA commented on HADOOP-15908:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
11s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15908 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947060/HADOOP-15865.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 978368c40974 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 49412a1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15477/testReport/ |
| Max. process+thread count | 1516 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15477/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> hadoop-build-tools
[jira] [Commented] (HADOOP-15865) ConcurrentModificationException in Configuration.overlay() method
[
https://issues.apache.org/jira/browse/HADOOP-15865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676790#comment-16676790
]
Oleksandr Shevchenko commented on HADOOP-15865:
---
[~ajisakaa] Thanks a lot for the review! [~andrew.wang] Could you please
double-check this?
Thanks.
> ConcurrentModificationException in Configuration.overlay() method
> -
>
> Key: HADOOP-15865
> URL: https://issues.apache.org/jira/browse/HADOOP-15865
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Major
> Attachments: HADOOP-15865.001.patch
>
>
> Configuration.overlay() is not thread-safe and can be the cause of
> ConcurrentModificationException since we use iteration over Properties
> object.
> {code}
> private void overlay(Properties to, Properties from) {
> for (Entry entry: from.entrySet()) {
> to.put(entry.getKey(), entry.getValue());
> }
> }
> {code}
> Properties class is thread-safe but iterator is not. We should manually
> synchronize on the returned set of entries which we use for iteration.
> We faced with ResourceManger fails during recovery caused by
> ConcurrentModificationException:
> {noformat}
> 2018-10-12 08:00:56,968 INFO org.apache.hadoop.service.AbstractService:
> Service ResourceManager failed in state STARTED; cause:
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at java.util.Hashtable$Enumerator.next(Hashtable.java:1383)
> at org.apache.hadoop.conf.Configuration.overlay(Configuration.java:2801)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2696)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2632)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2528)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1062)
> at
> org.apache.hadoop.conf.Configuration.getStringCollection(Configuration.java:1914)
> at
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:53)
> at
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2043)
> at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2023)
> at
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.getPassword(WebAppUtils.java:452)
> at
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.loadSslConfiguration(WebAppUtils.java:428)
> at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:293)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1017)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1117)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1251)
> 2018-10-12 08:00:56,968 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager:
> removing RMDelegation token with sequence number: 3489914
> 2018-10-12 08:00:56,968 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Removing
> RMDelegationToken and SequenceNumber
> 2018-10-12 08:00:56,968 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore:
> Removing RMDelegationToken_3489914
> 2018-10-12 08:00:56,969 INFO org.apache.hadoop.ipc.Server: Stopping server on
> 8032
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676782#comment-16676782 ] Wanqiang Ji commented on HADOOP-15907: -- [~cheersyang] OK, I will submit new patch. > Add YARN module to the main modules section in BUILDING.txt > --- > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Priority: Trivial > Attachments: HADOOP-15907.001.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local
[
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676778#comment-16676778
]
Hadoop QA commented on HADOOP-15908:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
33m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15908 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947061/HADOOP-15908.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux e52eceb7ad37 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 49412a1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15478/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 1) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15478/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> hadoop-build-tools jar is downloaded from remote repository instead of using
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15908.001.patch
>
>
>
[jira] [Commented] (HADOOP-15573) s3guard set-capacity to not retry on an access denied exception
[
https://issues.apache.org/jira/browse/HADOOP-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676774#comment-16676774
]
Adam Antal commented on HADOOP-15573:
-
It looks to me that this issue has been fixed in HADOOP-15583 as this piece of
code got added to {{S3AUtils.translateDynamoDBException}}:
{code:java}
final int statusCode = ddbException.getStatusCode();
final String errorCode = ddbException.getErrorCode();
IOException result = null;
// 400 gets used a lot by DDB
if (statusCode == 400) {
switch (errorCode) {
case "AccessDeniedException":
result = (IOException) new AccessDeniedException(
path,
null,
ddbException.toString())
.initCause(ddbException);
break;
default:
result = new AWSBadRequestException(message, ddbException);
}
}
{code}
> s3guard set-capacity to not retry on an access denied exception
> ---
>
> Key: HADOOP-15573
> URL: https://issues.apache.org/jira/browse/HADOOP-15573
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Priority: Minor
>
> when you call {{hadoop s3guard set-capacity}} with restricted access, you are
> (correctly) blocked by AWS, but the client keeps retrying. It should fail
> fast on a 400/AccessDenied
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15586) Fix wrong log statement in AbstractService
[
https://issues.apache.org/jira/browse/HADOOP-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676753#comment-16676753
]
Sunil Govindan commented on HADOOP-15586:
-
Updated Fixed versions.
> Fix wrong log statement in AbstractService
> --
>
> Key: HADOOP-15586
> URL: https://issues.apache.org/jira/browse/HADOOP-15586
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 2.9.0, 3.1.0
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Fix For: 3.2.0, 3.3.0
>
> Attachments: HADOOP-15586-001.patch, HADOOP-15586-002.patch,
> HADOOP-15586-003.patch
>
>
> There are some wrong logging statements in AbstractService, here is one
> example:
> {code:java}
> LOG.debug("noteFailure {}" + exception);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15586) Fix wrong log statement in AbstractService
[
https://issues.apache.org/jira/browse/HADOOP-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan updated HADOOP-15586:
Fix Version/s: 3.3.0
3.2.0
> Fix wrong log statement in AbstractService
> --
>
> Key: HADOOP-15586
> URL: https://issues.apache.org/jira/browse/HADOOP-15586
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 2.9.0, 3.1.0
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Fix For: 3.2.0, 3.3.0
>
> Attachments: HADOOP-15586-001.patch, HADOOP-15586-002.patch,
> HADOOP-15586-003.patch
>
>
> There are some wrong logging statements in AbstractService, here is one
> example:
> {code:java}
> LOG.debug("noteFailure {}" + exception);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15740) ABFS: Check variable names during initialization of AbfsClientThrottlingIntercept
[ https://issues.apache.org/jira/browse/HADOOP-15740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15740: Fix Version/s: 3.2.0 > ABFS: Check variable names during initialization of > AbfsClientThrottlingIntercept > -- > > Key: HADOOP-15740 > URL: https://issues.apache.org/jira/browse/HADOOP-15740 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sneha Varma >Assignee: Sneha Varma >Priority: Minor > Fix For: 3.2.0 > > Attachments: HADOOP-15740-HADOOP-15407-001.patch > > > In the initializeSingleton function of the AbfsClientThrottlingIntercept the > local variable name is same as global variable isAutoThrottlingEnabled > because of which the global isAutoThrottlingEnabled is not being set to true. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15356) Make HTTP timeout configurable in ADLS Connector
[ https://issues.apache.org/jira/browse/HADOOP-15356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15356: Fix Version/s: 3.3.0 3.2.0 > Make HTTP timeout configurable in ADLS Connector > > > Key: HADOOP-15356 > URL: https://issues.apache.org/jira/browse/HADOOP-15356 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Atul Sikaria >Assignee: Atul Sikaria >Priority: Major > Fix For: 3.2.0, 3.3.0 > > Attachments: HADOOP-15356.001.patch, HADOOP-15356.002.patch, > HADOOP-15356.003.patch, HADOOP-15356.004.patch, HADOOP-15356.005.patch > > > Currently the HTTP timeout for the connections to ADLS are not configurable > in Hadoop. This patch enables the timeouts to be configurable based on a > core-site config setting. Also, up the ADLS SDK version to 2.2.8, that has > default value of 60 seconds - any optimizations to that setting can now be > done in Hadoop through core-site. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15356) Make HTTP timeout configurable in ADLS Connector
[ https://issues.apache.org/jira/browse/HADOOP-15356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676736#comment-16676736 ] Sunil Govindan commented on HADOOP-15356: - Updated Fixed version to 3.3.0 and 3.2.0 > Make HTTP timeout configurable in ADLS Connector > > > Key: HADOOP-15356 > URL: https://issues.apache.org/jira/browse/HADOOP-15356 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Atul Sikaria >Assignee: Atul Sikaria >Priority: Major > Fix For: 3.2.0, 3.3.0 > > Attachments: HADOOP-15356.001.patch, HADOOP-15356.002.patch, > HADOOP-15356.003.patch, HADOOP-15356.004.patch, HADOOP-15356.005.patch > > > Currently the HTTP timeout for the connections to ADLS are not configurable > in Hadoop. This patch enables the timeouts to be configurable based on a > core-site config setting. Also, up the ADLS SDK version to 2.2.8, that has > default value of 60 seconds - any optimizations to that setting can now be > done in Hadoop through core-site. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local
[
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksandr Shevchenko updated HADOOP-15908:
--
Attachment: (was: HADOOP-15865.001.patch)
> hadoop-build-tools jar is downloaded from remote repository instead of using
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15908.001.patch
>
>
> HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml
> to verify LICENSE.txt and NOTICE.txt files which includes
> "hadoop-build-tools" remote resource bundles.
> {code}
>
> org.apache.maven.plugins
> maven-remote-resources-plugin
> ${maven-remote-resources-plugin.version}
>
>
>
> org.apache.hadoop:hadoop-build-tools:${hadoop.version}
>
>
>
>
> org.apache.hadoop
> hadoop-build-tools
> ${hadoop.version}
>
>
>
>
>
> process
>
>
>
>
> {code}
> If we build only some module we always download " hadoop-build-tools" from
> maven repository.
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @
> hadoop-annotations ---
> Downloading from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> (791 B at 684 B/s)
> Downloading from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> (609 B at 547 B/s)
> Downloading from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> (791 B at 343 B/s)
> Downloading from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> Downloaded from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> (0 B at 0 B/s)
> {noformat}
> If "hadoop-build-tools" jar doesn't exist in maven repository (for example we
> try to build new version locally before repository will be created ) we can't
> build some module:
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default)
> on project hadoop-annotations: Execution default of goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed:
> Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of
> its dependencies could not be resolved: Failure to find
> org.apache.hadoop:hadoop-build-tools:jar:3.2.0 in
> https://repo.maven.apache.org/maven2 was cached in the local repository,
> resolution will not be reattempted until the update interval of central has
> elapsed or updates are forced -> [Help 1]
> {noformat}
> Therefore, we need to limit execution of the Remote Resources Plugin only in
> the root directory in which the build was run.
> To accomplish this, we can use the "runOnlyAtExecutionRoot parameter"
> From maven documentation
> http://maven.apache.org/plugins/maven-remote-resources-plugin/usage.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local
[
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksandr Shevchenko updated HADOOP-15908:
--
Attachment: HADOOP-15908.001.patch
Status: Patch Available (was: Open)
> hadoop-build-tools jar is downloaded from remote repository instead of using
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15908.001.patch
>
>
> HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml
> to verify LICENSE.txt and NOTICE.txt files which includes
> "hadoop-build-tools" remote resource bundles.
> {code}
>
> org.apache.maven.plugins
> maven-remote-resources-plugin
> ${maven-remote-resources-plugin.version}
>
>
>
> org.apache.hadoop:hadoop-build-tools:${hadoop.version}
>
>
>
>
> org.apache.hadoop
> hadoop-build-tools
> ${hadoop.version}
>
>
>
>
>
> process
>
>
>
>
> {code}
> If we build only some module we always download " hadoop-build-tools" from
> maven repository.
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @
> hadoop-annotations ---
> Downloading from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> (791 B at 684 B/s)
> Downloading from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> (609 B at 547 B/s)
> Downloading from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> (791 B at 343 B/s)
> Downloading from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> Downloaded from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> (0 B at 0 B/s)
> {noformat}
> If "hadoop-build-tools" jar doesn't exist in maven repository (for example we
> try to build new version locally before repository will be created ) we can't
> build some module:
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default)
> on project hadoop-annotations: Execution default of goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed:
> Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of
> its dependencies could not be resolved: Failure to find
> org.apache.hadoop:hadoop-build-tools:jar:3.2.0 in
> https://repo.maven.apache.org/maven2 was cached in the local repository,
> resolution will not be reattempted until the update interval of central has
> elapsed or updates are forced -> [Help 1]
> {noformat}
> Therefore, we need to limit execution of the Remote Resources Plugin only in
> the root directory in which the build was run.
> To accomplish this, we can use the "runOnlyAtExecutionRoot parameter"
> From maven documentation
> http://maven.apache.org/plugins/maven-remote-resources-plugin/usage.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local
[
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksandr Shevchenko updated HADOOP-15908:
--
Status: Open (was: Patch Available)
> hadoop-build-tools jar is downloaded from remote repository instead of using
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15865.001.patch
>
>
> HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml
> to verify LICENSE.txt and NOTICE.txt files which includes
> "hadoop-build-tools" remote resource bundles.
> {code}
>
> org.apache.maven.plugins
> maven-remote-resources-plugin
> ${maven-remote-resources-plugin.version}
>
>
>
> org.apache.hadoop:hadoop-build-tools:${hadoop.version}
>
>
>
>
> org.apache.hadoop
> hadoop-build-tools
> ${hadoop.version}
>
>
>
>
>
> process
>
>
>
>
> {code}
> If we build only some module we always download " hadoop-build-tools" from
> maven repository.
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @
> hadoop-annotations ---
> Downloading from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> (791 B at 684 B/s)
> Downloading from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> (609 B at 547 B/s)
> Downloading from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> (791 B at 343 B/s)
> Downloading from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> Downloaded from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> (0 B at 0 B/s)
> {noformat}
> If "hadoop-build-tools" jar doesn't exist in maven repository (for example we
> try to build new version locally before repository will be created ) we can't
> build some module:
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default)
> on project hadoop-annotations: Execution default of goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed:
> Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of
> its dependencies could not be resolved: Failure to find
> org.apache.hadoop:hadoop-build-tools:jar:3.2.0 in
> https://repo.maven.apache.org/maven2 was cached in the local repository,
> resolution will not be reattempted until the update interval of central has
> elapsed or updates are forced -> [Help 1]
> {noformat}
> Therefore, we need to limit execution of the Remote Resources Plugin only in
> the root directory in which the build was run.
> To accomplish this, we can use the "runOnlyAtExecutionRoot parameter"
> From maven documentation
> http://maven.apache.org/plugins/maven-remote-resources-plugin/usage.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local
[
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksandr Shevchenko updated HADOOP-15908:
--
Attachment: HADOOP-15865.001.patch
Status: Patch Available (was: Open)
Patch is attached.
Could someone review the proposed changes?
Thanks.
> hadoop-build-tools jar is downloaded from remote repository instead of using
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15865.001.patch
>
>
> HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml
> to verify LICENSE.txt and NOTICE.txt files which includes
> "hadoop-build-tools" remote resource bundles.
> {code}
>
> org.apache.maven.plugins
> maven-remote-resources-plugin
> ${maven-remote-resources-plugin.version}
>
>
>
> org.apache.hadoop:hadoop-build-tools:${hadoop.version}
>
>
>
>
> org.apache.hadoop
> hadoop-build-tools
> ${hadoop.version}
>
>
>
>
>
> process
>
>
>
>
> {code}
> If we build only some module we always download " hadoop-build-tools" from
> maven repository.
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @
> hadoop-annotations ---
> Downloading from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> (791 B at 684 B/s)
> Downloading from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots:
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> (609 B at 547 B/s)
> Downloading from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> (791 B at 343 B/s)
> Downloading from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> Downloaded from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> (0 B at 0 B/s)
> {noformat}
> If "hadoop-build-tools" jar doesn't exist in maven repository (for example we
> try to build new version locally before repository will be created ) we can't
> build some module:
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default)
> on project hadoop-annotations: Execution default of goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed:
> Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of
> its dependencies could not be resolved: Failure to find
> org.apache.hadoop:hadoop-build-tools:jar:3.2.0 in
> https://repo.maven.apache.org/maven2 was cached in the local repository,
> resolution will not be reattempted until the update interval of central has
> elapsed or updates are forced -> [Help 1]
> {noformat}
> Therefore, we need to limit execution of the Remote Resources Plugin only in
> the root directory in which the build was run.
> To accomplish this, we can use the "runOnlyAtExecutionRoot parameter"
> From maven documentation
> http://maven.apache.org/plugins/maven-remote-resources-plugin/usage.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local
Oleksandr Shevchenko created HADOOP-15908:
-
Summary: hadoop-build-tools jar is downloaded from remote
repository instead of using from local
Key: HADOOP-15908
URL: https://issues.apache.org/jira/browse/HADOOP-15908
Project: Hadoop Common
Issue Type: Bug
Reporter: Oleksandr Shevchenko
HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml to
verify LICENSE.txt and NOTICE.txt files which includes "hadoop-build-tools"
remote resource bundles.
{code}
org.apache.maven.plugins
maven-remote-resources-plugin
${maven-remote-resources-plugin.version}
org.apache.hadoop:hadoop-build-tools:${hadoop.version}
org.apache.hadoop
hadoop-build-tools
${hadoop.version}
process
{code}
If we build only some module we always download " hadoop-build-tools" from
maven repository.
For example run:
cd hadoop-common-project/
mvn test
Then we will get the following output:
{noformat}
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
hadoop-annotations ---
Downloading from apache.snapshots:
http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots:
http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
(791 B at 684 B/s)
Downloading from apache.snapshots:
http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots:
http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
(609 B at 547 B/s)
Downloading from apache.snapshots.https:
https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots.https:
https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
(791 B at 343 B/s)
Downloading from apache.snapshots.https:
https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
Downloaded from apache.snapshots.https:
https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
(0 B at 0 B/s)
{noformat}
If "hadoop-build-tools" jar doesn't exist in maven repository (for example we
try to build new version locally before repository will be created ) we can't
build some module:
For example run:
cd hadoop-common-project/
mvn test
Then we will get the following output:
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on
project hadoop-annotations: Execution default of goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed:
Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of its
dependencies could not be resolved: Failure to find
org.apache.hadoop:hadoop-build-tools:jar:3.2.0 in
https://repo.maven.apache.org/maven2 was cached in the local repository,
resolution will not be reattempted until the update interval of central has
elapsed or updates are forced -> [Help 1]
{noformat}
Therefore, we need to limit execution of the Remote Resources Plugin only in
the root directory in which the build was run.
To accomplish this, we can use the "runOnlyAtExecutionRoot parameter"
>From maven documentation
>http://maven.apache.org/plugins/maven-remote-resources-plugin/usage.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676686#comment-16676686 ] Weiwei Yang commented on HADOOP-15907: -- Thanks [~jiwq]. Apparently we are missing more than YARN, like following * hadoop-builld-tools * hadoop-client * hadoop-client-modules * hadoop-cloud-storage-project * hadoop-hdds * hadoop-maven-plugins * hadoop-minicluster * hadoop-ozone Can you add them all? > Add YARN module to the main modules section in BUILDING.txt > --- > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Priority: Trivial > Attachments: HADOOP-15907.001.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15902) [JDK 11] Specify the HTML version of Javadoc to 4.01
[
https://issues.apache.org/jira/browse/HADOOP-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676680#comment-16676680
]
Takanobu Asanuma commented on HADOOP-15902:
---
Thanks [~ajisakaa]!
> [JDK 11] Specify the HTML version of Javadoc to 4.01
>
>
> Key: HADOOP-15902
> URL: https://issues.apache.org/jira/browse/HADOOP-15902
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15902.1.patch, HADOOP-15902.2.patch
>
>
> From JDK9, Javadoc has options to specify the HTML version.
> {noformat}
> $ javadoc --help
> ...
> -html4Generate HTML 4.01 output
> -html5Generate HTML 5 output
> {noformat}
> JDK8 supports only HTML 4.01 output. The default html version of JDK9 and
> JDK10 is 4.01. That of JDK11 is 5. Let's unify the version to 4.01 for now.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15367) Update the initialization code in the docker hadoop-runner baseimage
[ https://issues.apache.org/jira/browse/HADOOP-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676681#comment-16676681 ] Elek, Marton commented on HADOOP-15367: --- It's committed to the docker-hadoop-runner branch. I removed the fixed version. Sorry for the confusion. > Update the initialization code in the docker hadoop-runner baseimage > - > > Key: HADOOP-15367 > URL: https://issues.apache.org/jira/browse/HADOOP-15367 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15367-hadoop-docker-runner.001.patch, > HADOOP-15367-hadoop-docker-runner.002.patch > > > The hadoop-runner baseimage contains initialization code for both the HDFS > namenode/datanode and Ozone/Hdds scm/ksm. > The script name for Ozone/Hdds is changed (from oz to ozone) therefore we > need to updated the base image. > This commit also would be a test for the dockerhub automated build. > Please apply the patch on the top of the _docker-hadoop-runner_ branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15902) [JDK 11] Specify the HTML version of Javadoc to 4.01
[
https://issues.apache.org/jira/browse/HADOOP-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676675#comment-16676675
]
Hudson commented on HADOOP-15902:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15368 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15368/])
HADOOP-15902. [JDK 11] Specify the HTML version of Javadoc to 4.01. (aajisaka:
rev 78e43bccf655153d5fc8fc891d59c536354d8031)
* (edit) hadoop-project/pom.xml
> [JDK 11] Specify the HTML version of Javadoc to 4.01
>
>
> Key: HADOOP-15902
> URL: https://issues.apache.org/jira/browse/HADOOP-15902
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15902.1.patch, HADOOP-15902.2.patch
>
>
> From JDK9, Javadoc has options to specify the HTML version.
> {noformat}
> $ javadoc --help
> ...
> -html4Generate HTML 4.01 output
> -html5Generate HTML 5 output
> {noformat}
> JDK8 supports only HTML 4.01 output. The default html version of JDK9 and
> JDK10 is 4.01. That of JDK11 is 5. Let's unify the version to 4.01 for now.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15901) IPC Client and Server should use Time.monotonicNow() for elapsed times.
[
https://issues.apache.org/jira/browse/HADOOP-15901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676673#comment-16676673
]
Steve Loughran commented on HADOOP-15901:
-
bq. This blog was really informative.
doesn't look at the issue of multisocket systems whose return value of RDTSCP
is inconsistent across sockets, even though it is (now( consistent across cores
in a single socket.
bq. So what we should do? Remove all usages of System.nanoTime() in Hadoop?
don't know.
big issue with time is in VMs where the clock can suddenly jump forwards
(sometimes even backwards), far more frequently than you see with NTP —the
latter can be set to slowly sync up at a rate less likely to break things
(though again, its clock can go backwards.
Thoughts:
* if any clock goes backwards, we are in trouble, especially in any code which
uses the diff and assumes its always positive.
* if a clock leaps forward through a NTP change, that's a spurious failure.
Proposed: people set clock correction rate to be low
* if a clock leaps forward through a VM suspent/resume, then it may be that
wall time has suddenly moved. things should (correctly) time out here
* I don't think we can rely on nanotime being consistent, but like you say,
currentTimeMillis isn't monotonic either
> IPC Client and Server should use Time.monotonicNow() for elapsed times.
> ---
>
> Key: HADOOP-15901
> URL: https://issues.apache.org/jira/browse/HADOOP-15901
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc, metrics
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Major
> Attachments: HADOOP-15901-01.patch
>
>
> Client.java and Server.java uses {{Time.now()}} to calculate the elapsed
> times/timeouts. This could result in undesired results when system clock's
> time changes.
> {{Time.monotonicNow()}} should be used for elapsed time calculations within
> same JVM.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15902) [JDK 11] Specify the HTML version of Javadoc to 4.01
[
https://issues.apache.org/jira/browse/HADOOP-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15902:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
Committed this to trunk. Thanks [~tasanuma0829] for the contribution!
> [JDK 11] Specify the HTML version of Javadoc to 4.01
>
>
> Key: HADOOP-15902
> URL: https://issues.apache.org/jira/browse/HADOOP-15902
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15902.1.patch, HADOOP-15902.2.patch
>
>
> From JDK9, Javadoc has options to specify the HTML version.
> {noformat}
> $ javadoc --help
> ...
> -html4Generate HTML 4.01 output
> -html5Generate HTML 5 output
> {noformat}
> JDK8 supports only HTML 4.01 output. The default html version of JDK9 and
> JDK10 is 4.01. That of JDK11 is 5. Let's unify the version to 4.01 for now.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15904) [JDK 11] Javadoc build failed due to "bad use of '>'" in hadoop-hdfs
[
https://issues.apache.org/jira/browse/HADOOP-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1667#comment-1667
]
Akira Ajisaka commented on HADOOP-15904:
I forgot to add "Contributed by Takanobu Asanuma." to the commit message. Sorry
for that.
> [JDK 11] Javadoc build failed due to "bad use of '>'" in hadoop-hdfs
>
>
> Key: HADOOP-15904
> URL: https://issues.apache.org/jira/browse/HADOOP-15904
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15904.1.patch
>
>
> {noformat}
> $ mvn javadoc:javadoc -pl hadoop-hdfs-project/hadoop-hdfs
> ...
> [ERROR]
> /hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/planner/package-info.java:32:
> error: bad use of '>'
> [ERROR] * // Creates a plan , like move 20 GB data from v1 -> v2
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15902) [JDK 11] Specify the HTML version of Javadoc to 4.01
[
https://issues.apache.org/jira/browse/HADOOP-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676612#comment-16676612
]
Hadoop QA commented on HADOOP-15902:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
32m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
13s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15902 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947045/HADOOP-15902.2.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux 8da8bf8010a0 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6430c98 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15476/testReport/ |
| Max. process+thread count | 338 (vs. ulimit of 1) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15476/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> [JDK 11] Specify the HTML version of Javadoc to 4.01
>
>
> Key: HADOOP-15902
> URL: https://issues.apache.org/jira/browse/HADOOP-15902
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HADOOP-15902.1.patch, HADOOP-15902.2.patch
>
>
> From JDK9, Javadoc has
[jira] [Commented] (HADOOP-15902) [JDK 11] Specify the HTML version of Javadoc to 4.01
[
https://issues.apache.org/jira/browse/HADOOP-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676507#comment-16676507
]
Akira Ajisaka commented on HADOOP-15902:
+1
> [JDK 11] Specify the HTML version of Javadoc to 4.01
>
>
> Key: HADOOP-15902
> URL: https://issues.apache.org/jira/browse/HADOOP-15902
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HADOOP-15902.1.patch, HADOOP-15902.2.patch
>
>
> From JDK9, Javadoc has options to specify the HTML version.
> {noformat}
> $ javadoc --help
> ...
> -html4Generate HTML 4.01 output
> -html5Generate HTML 5 output
> {noformat}
> JDK8 supports only HTML 4.01 output. The default html version of JDK9 and
> JDK10 is 4.01. That of JDK11 is 5. Let's unify the version to 4.01 for now.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15623) Compiling hadoop-azure fails with jdk10 (javax javascript)
[
https://issues.apache.org/jira/browse/HADOOP-15623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved HADOOP-15623.
Resolution: Cannot Reproduce
Closing. Please reopen this if the problem still exists.
> Compiling hadoop-azure fails with jdk10 (javax javascript)
> --
>
> Key: HADOOP-15623
> URL: https://issues.apache.org/jira/browse/HADOOP-15623
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Reporter: Ewan Higgs
>Priority: Major
>
> {code}
> $ java -version
> java version "10.0.1" 2018-04-17
> Java(TM) SE Runtime Environment 18.3 (build 10.0.1+10)
> Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.1+10, mixed mode)
> {code}
> {code}
> $ mvn install -DskipShade -Dmaven.javadoc.skip=true -Djava.awt.headless=true
> -DskipTests -rf :hadoop-azure
> ...
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.7:run
> (create-parallel-tests-dirs) on project hadoop-azure: An Ant BuildException
> has occured: Unable to create javax script engine for javascript
> [ERROR] around Ant part
[jira] [Commented] (HADOOP-15902) [JDK 11] Specify the HTML version of Javadoc to 4.01
[
https://issues.apache.org/jira/browse/HADOOP-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676488#comment-16676488
]
Takanobu Asanuma commented on HADOOP-15902:
---
[~ajisakaa] Thanks for the review. Updated the patch addressing your comment.
> [JDK 11] Specify the HTML version of Javadoc to 4.01
>
>
> Key: HADOOP-15902
> URL: https://issues.apache.org/jira/browse/HADOOP-15902
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HADOOP-15902.1.patch, HADOOP-15902.2.patch
>
>
> From JDK9, Javadoc has options to specify the HTML version.
> {noformat}
> $ javadoc --help
> ...
> -html4Generate HTML 4.01 output
> -html5Generate HTML 5 output
> {noformat}
> JDK8 supports only HTML 4.01 output. The default html version of JDK9 and
> JDK10 is 4.01. That of JDK11 is 5. Let's unify the version to 4.01 for now.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15904) [JDK 11] Javadoc build failed due to "bad use of '>'" in hadoop-hdfs
[
https://issues.apache.org/jira/browse/HADOOP-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676485#comment-16676485
]
Takanobu Asanuma commented on HADOOP-15904:
---
Thanks [~anu] for the review, and thanks [~ajisakaa] for the commit!
> [JDK 11] Javadoc build failed due to "bad use of '>'" in hadoop-hdfs
>
>
> Key: HADOOP-15904
> URL: https://issues.apache.org/jira/browse/HADOOP-15904
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15904.1.patch
>
>
> {noformat}
> $ mvn javadoc:javadoc -pl hadoop-hdfs-project/hadoop-hdfs
> ...
> [ERROR]
> /hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/planner/package-info.java:32:
> error: bad use of '>'
> [ERROR] * // Creates a plan , like move 20 GB data from v1 -> v2
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15902) [JDK 11] Specify the HTML version of Javadoc to 4.01
[
https://issues.apache.org/jira/browse/HADOOP-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HADOOP-15902:
--
Attachment: HADOOP-15902.2.patch
> [JDK 11] Specify the HTML version of Javadoc to 4.01
>
>
> Key: HADOOP-15902
> URL: https://issues.apache.org/jira/browse/HADOOP-15902
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HADOOP-15902.1.patch, HADOOP-15902.2.patch
>
>
> From JDK9, Javadoc has options to specify the HTML version.
> {noformat}
> $ javadoc --help
> ...
> -html4Generate HTML 4.01 output
> -html5Generate HTML 5 output
> {noformat}
> JDK8 supports only HTML 4.01 output. The default html version of JDK9 and
> JDK10 is 4.01. That of JDK11 is 5. Let's unify the version to 4.01 for now.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15349) S3Guard DDB retryBackoff to be more informative on limits exceeded
[ https://issues.apache.org/jira/browse/HADOOP-15349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15349: Fix Version/s: 3.3.0 3.2.0 > S3Guard DDB retryBackoff to be more informative on limits exceeded > -- > > Key: HADOOP-15349 > URL: https://issues.apache.org/jira/browse/HADOOP-15349 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Fix For: 3.2.0, 3.3.0 > > Attachments: HADOOP-15349.001.patch, failure.log > > > When S3Guard can't update the DB and so throws an IOE after the retry limit > is exceeded, it's not at all informative. Improve logging & exception -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15349) S3Guard DDB retryBackoff to be more informative on limits exceeded
[ https://issues.apache.org/jira/browse/HADOOP-15349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676483#comment-16676483 ] Sunil Govindan commented on HADOOP-15349: - I added 3.2.0 and 3.3.0 to Fixed version. Pls add if i missed any versions. > S3Guard DDB retryBackoff to be more informative on limits exceeded > -- > > Key: HADOOP-15349 > URL: https://issues.apache.org/jira/browse/HADOOP-15349 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Fix For: 3.2.0, 3.3.0 > > Attachments: HADOOP-15349.001.patch, failure.log > > > When S3Guard can't update the DB and so throws an IOE after the retry limit > is exceeded, it's not at all informative. Improve logging & exception -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[
https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676477#comment-16676477
]
Hadoop QA commented on HADOOP-15907:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 41s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15907 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947038/HADOOP-15907.001.patch
|
| Optional Tests | dupname asflicense |
| uname | Linux 09e3cc1e1f34 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5d6554c |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 305 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15475/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add YARN module to the main modules section in BUILDING.txt
> ---
>
> Key: HADOOP-15907
> URL: https://issues.apache.org/jira/browse/HADOOP-15907
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wanqiang Ji
>Priority: Trivial
> Attachments: HADOOP-15907.001.patch
>
>
> In the Maven main modules section, I found the missing main module YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15904) [JDK 11] Javadoc build failed due to "bad use of '>'" in hadoop-hdfs
[
https://issues.apache.org/jira/browse/HADOOP-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676478#comment-16676478
]
Hudson commented on HADOOP-15904:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15367 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15367/])
HADOOP-15904. [JDK 11] Javadoc build failed due to bad use of '>' in (aajisaka:
rev 6430c9890aae46aeb2f41512ee442adfda5189dc)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/planner/package-info.java
> [JDK 11] Javadoc build failed due to "bad use of '>'" in hadoop-hdfs
>
>
> Key: HADOOP-15904
> URL: https://issues.apache.org/jira/browse/HADOOP-15904
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15904.1.patch
>
>
> {noformat}
> $ mvn javadoc:javadoc -pl hadoop-hdfs-project/hadoop-hdfs
> ...
> [ERROR]
> /hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/planner/package-info.java:32:
> error: bad use of '>'
> [ERROR] * // Creates a plan , like move 20 GB data from v1 -> v2
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15668) mvn test goal fails on HADOOP-15407 branch
[
https://issues.apache.org/jira/browse/HADOOP-15668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan resolved HADOOP-15668.
-
Resolution: Not A Problem
Closed with correct status
> mvn test goal fails on HADOOP-15407 branch
> --
>
> Key: HADOOP-15668
> URL: https://issues.apache.org/jira/browse/HADOOP-15668
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
>
> It's very easy to reproduce:
> {code}cd hadoop-common-project/hadoop-common/
> mvn test -Dtest=Whatever{code}
> The error is due to
> {code} [exec] Running bats -t hadoop_stop_daemon.bats
> [exec] 1..2
> [exec] ok 1 hadoop_stop_daemon_changing_pid
> [exec] not ok 2 hadoop_stop_daemon_force_kill
> [exec] # (in test file hadoop_stop_daemon.bats, line 43)
> [exec] # `[ -f ${TMP}/pidfile ]' failed
> [exec] # bindir:
> /Users/abokor/work/hadoop/hadoop-common-project/hadoop-common/src/test/scripts
> [exec] # sh:
> /Users/abokor/work/hadoop/hadoop-common-project/hadoop-common/src/test/scripts/process_with_sigterm_trap.sh:
> No such file or directory{code}
> This happens because actually 3 commits belong to HADOOP-15527 but
> HADOOP-15407 branch contains only one of them so the test won't find
> process_with_sigterm_trap.sh.
> I am not sure what is the best practice to solve this kind of issues. Is
> patch required or can somebody just cherry-pick the missing commits?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-15668) mvn test goal fails on HADOOP-15407 branch
[
https://issues.apache.org/jira/browse/HADOOP-15668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan reopened HADOOP-15668:
-
> mvn test goal fails on HADOOP-15407 branch
> --
>
> Key: HADOOP-15668
> URL: https://issues.apache.org/jira/browse/HADOOP-15668
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
>
> It's very easy to reproduce:
> {code}cd hadoop-common-project/hadoop-common/
> mvn test -Dtest=Whatever{code}
> The error is due to
> {code} [exec] Running bats -t hadoop_stop_daemon.bats
> [exec] 1..2
> [exec] ok 1 hadoop_stop_daemon_changing_pid
> [exec] not ok 2 hadoop_stop_daemon_force_kill
> [exec] # (in test file hadoop_stop_daemon.bats, line 43)
> [exec] # `[ -f ${TMP}/pidfile ]' failed
> [exec] # bindir:
> /Users/abokor/work/hadoop/hadoop-common-project/hadoop-common/src/test/scripts
> [exec] # sh:
> /Users/abokor/work/hadoop/hadoop-common-project/hadoop-common/src/test/scripts/process_with_sigterm_trap.sh:
> No such file or directory{code}
> This happens because actually 3 commits belong to HADOOP-15527 but
> HADOOP-15407 branch contains only one of them so the test won't find
> process_with_sigterm_trap.sh.
> I am not sure what is the best practice to solve this kind of issues. Is
> patch required or can somebody just cherry-pick the missing commits?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15904) [JDK 11] Javadoc build failed due to "bad use of '>'" in hadoop-hdfs
[
https://issues.apache.org/jira/browse/HADOOP-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15904:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks [~tasanuma0829] and [~anu]!
> [JDK 11] Javadoc build failed due to "bad use of '>'" in hadoop-hdfs
>
>
> Key: HADOOP-15904
> URL: https://issues.apache.org/jira/browse/HADOOP-15904
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15904.1.patch
>
>
> {noformat}
> $ mvn javadoc:javadoc -pl hadoop-hdfs-project/hadoop-hdfs
> ...
> [ERROR]
> /hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/planner/package-info.java:32:
> error: bad use of '>'
> [ERROR] * // Creates a plan , like move 20 GB data from v1 -> v2
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15904) [JDK 11] Javadoc build failed due to "bad use of '>'" in hadoop-hdfs
[
https://issues.apache.org/jira/browse/HADOOP-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676423#comment-16676423
]
Akira Ajisaka commented on HADOOP-15904:
+1
> [JDK 11] Javadoc build failed due to "bad use of '>'" in hadoop-hdfs
>
>
> Key: HADOOP-15904
> URL: https://issues.apache.org/jira/browse/HADOOP-15904
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HADOOP-15904.1.patch
>
>
> {noformat}
> $ mvn javadoc:javadoc -pl hadoop-hdfs-project/hadoop-hdfs
> ...
> [ERROR]
> /hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/planner/package-info.java:32:
> error: bad use of '>'
> [ERROR] * // Creates a plan , like move 20 GB data from v1 -> v2
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15902) [JDK 11] Specify the HTML version of Javadoc to 4.01
[
https://issues.apache.org/jira/browse/HADOOP-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676421#comment-16676421
]
Akira Ajisaka commented on HADOOP-15902:
Would you add a TODO comment to remove this setting when Apache Hadoop drops
JDK8 support?
> [JDK 11] Specify the HTML version of Javadoc to 4.01
>
>
> Key: HADOOP-15902
> URL: https://issues.apache.org/jira/browse/HADOOP-15902
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HADOOP-15902.1.patch
>
>
> From JDK9, Javadoc has options to specify the HTML version.
> {noformat}
> $ javadoc --help
> ...
> -html4Generate HTML 4.01 output
> -html5Generate HTML 5 output
> {noformat}
> JDK8 supports only HTML 4.01 output. The default html version of JDK9 and
> JDK10 is 4.01. That of JDK11 is 5. Let's unify the version to 4.01 for now.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji updated HADOOP-15907: - Priority: Trivial (was: Major) > Add YARN module to the main modules section in BUILDING.txt > --- > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Priority: Trivial > Attachments: HADOOP-15907.001.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676416#comment-16676416 ] Wanqiang Ji commented on HADOOP-15907: -- [~cheersyang] Can you help to review this? > Add YARN module to the main modules section in BUILDING.txt > --- > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Priority: Major > Attachments: HADOOP-15907.001.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15907) Add YARN module to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji updated HADOOP-15907: - Summary: Add YARN module to the main modules section in BUILDING.txt (was: Add YARN modules to the main modules section in BUILDING.txt) > Add YARN module to the main modules section in BUILDING.txt > --- > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Priority: Major > Attachments: HADOOP-15907.001.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15907) Add YARN modules to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji updated HADOOP-15907: - Status: Patch Available (was: Open) > Add YARN modules to the main modules section in BUILDING.txt > > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Priority: Major > Attachments: HADOOP-15907.001.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15907) Add YARN modules to the main modules section in BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji updated HADOOP-15907: - Attachment: HADOOP-15907.001.patch > Add YARN modules to the main modules section in BUILDING.txt > > > Key: HADOOP-15907 > URL: https://issues.apache.org/jira/browse/HADOOP-15907 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wanqiang Ji >Priority: Major > Attachments: HADOOP-15907.001.patch > > > In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15907) Add YARN modules to the main modules section in BUILDING.txt
Wanqiang Ji created HADOOP-15907: Summary: Add YARN modules to the main modules section in BUILDING.txt Key: HADOOP-15907 URL: https://issues.apache.org/jira/browse/HADOOP-15907 Project: Hadoop Common Issue Type: Improvement Reporter: Wanqiang Ji In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15906) [JDK 11] Fix compilation failures with jdk 11
[
https://issues.apache.org/jira/browse/HADOOP-15906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15906:
---
Issue Type: Bug (was: Sub-task)
Parent: (was: HADOOP-15338)
> [JDK 11] Fix compilation failures with jdk 11
> -
>
> Key: HADOOP-15906
> URL: https://issues.apache.org/jira/browse/HADOOP-15906
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Lokesh Jain
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-15906.001.patch
>
>
> This Jira aims to fix the compilation failures with jdk 11. The compilation
> failure occurs in hadoop-hdds and hadoop-ozone module.
> {code:java}
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-hdds/common/target/generated-sources/java/org/apache/hadoop/hdds/protocol/datanode/proto/XceiverClientProtocolServiceGrpc.java:[20,18]
> cannot find symbol
> symbol: class Generated
> location: package javax.annotation
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-hdds/common/target/generated-sources/java/org/apache/hadoop/hdds/protocol/datanode/proto/IntraDatanodeProtocolServiceGrpc.java:[20,18]
> cannot find symbol
> symbol: class Generated
> location: package javax.annotation
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[20,33]
> package javax.xml.bind.annotation does not exist
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[21,33]
> package javax.xml.bind.annotation does not exist
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[22,33]
> package javax.xml.bind.annotation does not exist
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[23,42]
> package javax.xml.bind.annotation.adapters does not exist
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15906) [JDK 11] Fix compilation failures with jdk 11
[
https://issues.apache.org/jira/browse/HADOOP-15906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676392#comment-16676392
]
Akira Ajisaka commented on HADOOP-15906:
Moving this issue to HDDS project. This is very similar to HADOOP-15895.
> [JDK 11] Fix compilation failures with jdk 11
> -
>
> Key: HADOOP-15906
> URL: https://issues.apache.org/jira/browse/HADOOP-15906
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Lokesh Jain
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-15906.001.patch
>
>
> This Jira aims to fix the compilation failures with jdk 11. The compilation
> failure occurs in hadoop-hdds and hadoop-ozone module.
> {code:java}
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-hdds/common/target/generated-sources/java/org/apache/hadoop/hdds/protocol/datanode/proto/XceiverClientProtocolServiceGrpc.java:[20,18]
> cannot find symbol
> symbol: class Generated
> location: package javax.annotation
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-hdds/common/target/generated-sources/java/org/apache/hadoop/hdds/protocol/datanode/proto/IntraDatanodeProtocolServiceGrpc.java:[20,18]
> cannot find symbol
> symbol: class Generated
> location: package javax.annotation
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[20,33]
> package javax.xml.bind.annotation does not exist
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[21,33]
> package javax.xml.bind.annotation does not exist
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[22,33]
> package javax.xml.bind.annotation does not exist
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[23,42]
> package javax.xml.bind.annotation.adapters does not exist
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15906) [JDK 11] Fix compilation failures with jdk 11
[
https://issues.apache.org/jira/browse/HADOOP-15906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reassigned HADOOP-15906:
--
Assignee: Akira Ajisaka (was: Lokesh Jain)
> [JDK 11] Fix compilation failures with jdk 11
> -
>
> Key: HADOOP-15906
> URL: https://issues.apache.org/jira/browse/HADOOP-15906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Reporter: Lokesh Jain
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-15906.001.patch
>
>
> This Jira aims to fix the compilation failures with jdk 11. The compilation
> failure occurs in hadoop-hdds and hadoop-ozone module.
> {code:java}
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-hdds/common/target/generated-sources/java/org/apache/hadoop/hdds/protocol/datanode/proto/XceiverClientProtocolServiceGrpc.java:[20,18]
> cannot find symbol
> symbol: class Generated
> location: package javax.annotation
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-hdds/common/target/generated-sources/java/org/apache/hadoop/hdds/protocol/datanode/proto/IntraDatanodeProtocolServiceGrpc.java:[20,18]
> cannot find symbol
> symbol: class Generated
> location: package javax.annotation
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[20,33]
> package javax.xml.bind.annotation does not exist
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[21,33]
> package javax.xml.bind.annotation does not exist
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[22,33]
> package javax.xml.bind.annotation does not exist
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[23,42]
> package javax.xml.bind.annotation.adapters does not exist
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15688) ABFS: InputStream wrapped in FSDataInputStream twice
[ https://issues.apache.org/jira/browse/HADOOP-15688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15688: Fix Version/s: 3.2.0 > ABFS: InputStream wrapped in FSDataInputStream twice > > > Key: HADOOP-15688 > URL: https://issues.apache.org/jira/browse/HADOOP-15688 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15688-HADOOP-15407-002.patch, > HADOOP-15688.001.patch > > > I can't read Parquet files from ABFS. It has 2 different implementations to > read seekable streams, and it'll use the one that uses ByteBuffer reads if it > can. It currently decides to use the ByteBuffer read implementation because > the FSDataInputStream it gets back wraps another FSDataInputStream, which > implements ByteBufferReadable. > That's not the most robust way to check that ByteBufferReads are supported by > the ultimately underlying InputStream, but it's unnecessary and probably a > mistake to double-wrap the InputStream, so let's not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[ https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15659: Fix Version/s: 3.2.0 > ABFS: Code changes for bug fix and new tests > > > Key: HADOOP-15659 > URL: https://issues.apache.org/jira/browse/HADOOP-15659 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15659-HADOOP-15407-001.patch, > HADOOP-15659-HADOOP-15407-002.patch, HADOOP-15659-HADOOP-15407-003.patch > > > - add bug fixes. > - remove unnecessary dependencies. > - add new tests for code changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15669) ABFS: Improve HTTPS Performance
[ https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676368#comment-16676368 ] Sunil Govindan commented on HADOOP-15669: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Improve HTTPS Performance > --- > > Key: HADOOP-15669 > URL: https://issues.apache.org/jira/browse/HADOOP-15669 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Vishwajeet Dusane >Priority: Major > Fix For: 3.2.0 > > Attachments: ABFS - Improve HTTPS Performance Over Java Based > Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, > HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch, > HADOOP-15669-HADOOP-15407-04.patch > > > We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP. Lets > perform a detailed measurement and see what can be done to improve throughput. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[ https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676370#comment-16676370 ] Sunil Govindan commented on HADOOP-15659: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Code changes for bug fix and new tests > > > Key: HADOOP-15659 > URL: https://issues.apache.org/jira/browse/HADOOP-15659 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15659-HADOOP-15407-001.patch, > HADOOP-15659-HADOOP-15407-002.patch, HADOOP-15659-HADOOP-15407-003.patch > > > - add bug fixes. > - remove unnecessary dependencies. > - add new tests for code changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15660) ABFS: Add support for OAuth
[ https://issues.apache.org/jira/browse/HADOOP-15660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15660: Fix Version/s: 3.2.0 > ABFS: Add support for OAuth > --- > > Key: HADOOP-15660 > URL: https://issues.apache.org/jira/browse/HADOOP-15660 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15660-HADOOP-15407-001.patch, > HADOOP-15660-HADOOP-15407-002.patch, HADOOP-15660-HADOOP-15407-003.patch > > > - Add support for OAuth -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15669) ABFS: Improve HTTPS Performance
[ https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15669: Fix Version/s: 3.2.0 > ABFS: Improve HTTPS Performance > --- > > Key: HADOOP-15669 > URL: https://issues.apache.org/jira/browse/HADOOP-15669 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Vishwajeet Dusane >Priority: Major > Fix For: 3.2.0 > > Attachments: ABFS - Improve HTTPS Performance Over Java Based > Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, > HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch, > HADOOP-15669-HADOOP-15407-04.patch > > > We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP. Lets > perform a detailed measurement and see what can be done to improve throughput. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15661) ABFS: Add support for ACL
[ https://issues.apache.org/jira/browse/HADOOP-15661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676366#comment-16676366 ] Sunil Govindan commented on HADOOP-15661: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Add support for ACL > - > > Key: HADOOP-15661 > URL: https://issues.apache.org/jira/browse/HADOOP-15661 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15661-HADOOP-15407-001.patch, > HADOOP-15661-HADOOP-15407-002.patch, HADOOP-15661-HADOOP-15407-003.patch, > HADOOP-15661-HADOOP-15407-004.patch, HADOOP-15661-HADOOP-15407-005.patch > > > - Add support for ACL -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15661) ABFS: Add support for ACL
[ https://issues.apache.org/jira/browse/HADOOP-15661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15661: Fix Version/s: 3.2.0 > ABFS: Add support for ACL > - > > Key: HADOOP-15661 > URL: https://issues.apache.org/jira/browse/HADOOP-15661 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15661-HADOOP-15407-001.patch, > HADOOP-15661-HADOOP-15407-002.patch, HADOOP-15661-HADOOP-15407-003.patch, > HADOOP-15661-HADOOP-15407-004.patch, HADOOP-15661-HADOOP-15407-005.patch > > > - Add support for ACL -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15660) ABFS: Add support for OAuth
[ https://issues.apache.org/jira/browse/HADOOP-15660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676367#comment-16676367 ] Sunil Govindan commented on HADOOP-15660: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Add support for OAuth > --- > > Key: HADOOP-15660 > URL: https://issues.apache.org/jira/browse/HADOOP-15660 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15660-HADOOP-15407-001.patch, > HADOOP-15660-HADOOP-15407-002.patch, HADOOP-15660-HADOOP-15407-003.patch > > > - Add support for OAuth -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15682) ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle
[ https://issues.apache.org/jira/browse/HADOOP-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676365#comment-16676365 ] Sunil Govindan commented on HADOOP-15682: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle > > > Key: HADOOP-15682 > URL: https://issues.apache.org/jira/browse/HADOOP-15682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15682-HADOOP-15407-001.patch > > > Add support for the new StreamCapabilities interface. This work is similar > to what was done for WASB > [HADOOP-15677|https://jira.apache.org/jira/browse/HADOOP-15677]. > Also fix javadoc and checkstyle errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15682) ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle
[ https://issues.apache.org/jira/browse/HADOOP-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15682: Fix Version/s: 3.2.0 > ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle > > > Key: HADOOP-15682 > URL: https://issues.apache.org/jira/browse/HADOOP-15682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15682-HADOOP-15407-001.patch > > > Add support for the new StreamCapabilities interface. This work is similar > to what was done for WASB > [HADOOP-15677|https://jira.apache.org/jira/browse/HADOOP-15677]. > Also fix javadoc and checkstyle errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15906) [JDK 11] Fix compilation failures with jdk 11
[
https://issues.apache.org/jira/browse/HADOOP-15906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676363#comment-16676363
]
Lokesh Jain commented on HADOOP-15906:
--
I have uploaded v1 patch which adds dependency on javax.annotation-api and
jaxb-api in hadoop-hdds/common and hadoop-ozone/s3gateway pom.xml. There were a
lot of javadoc failures therefore I had to skip javadoc in the build.
{code:java}
mvn clean install -DskipTests -Phdds -Pdist -Dmaven.javadoc.skip=true
{code}
> [JDK 11] Fix compilation failures with jdk 11
> -
>
> Key: HADOOP-15906
> URL: https://issues.apache.org/jira/browse/HADOOP-15906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>Priority: Major
> Attachments: HADOOP-15906.001.patch
>
>
> This Jira aims to fix the compilation failures with jdk 11. The compilation
> failure occurs in hadoop-hdds and hadoop-ozone module.
> {code:java}
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-hdds/common/target/generated-sources/java/org/apache/hadoop/hdds/protocol/datanode/proto/XceiverClientProtocolServiceGrpc.java:[20,18]
> cannot find symbol
> symbol: class Generated
> location: package javax.annotation
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-hdds/common/target/generated-sources/java/org/apache/hadoop/hdds/protocol/datanode/proto/IntraDatanodeProtocolServiceGrpc.java:[20,18]
> cannot find symbol
> symbol: class Generated
> location: package javax.annotation
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[20,33]
> package javax.xml.bind.annotation does not exist
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[21,33]
> package javax.xml.bind.annotation does not exist
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[22,33]
> package javax.xml.bind.annotation does not exist
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[23,42]
> package javax.xml.bind.annotation.adapters does not exist
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15663) ABFS: Simplify configuration
[ https://issues.apache.org/jira/browse/HADOOP-15663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15663: Fix Version/s: 3.2.0 > ABFS: Simplify configuration > > > Key: HADOOP-15663 > URL: https://issues.apache.org/jira/browse/HADOOP-15663 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15663-HADOOP-15407-001.patch, > HADOOP-15663-HADOOP-15407-002.patch, HADOOP-15663-HADOOP-15407-003.patch, > HADOOP-15663-HADOOP-15407-004.patch, HADOOP-15663-HADOOP-15407-005.patch > > > Configuration for WASB and ABFS is too complex. The current approach is to > use four files for test configuration. > Both WASB and ABFS have basic test configuration which is committed to the > repo (azure-test.xml and azure-bfs-test.xml). Currently these contain the > fs.AbstractFileSystem.[scheme].impl configuration, but otherwise are empty > except for an include reference to a file containing the endpoint > credentials. > Both WASB and ABFS have endpoint credential configuration files > (azure-auth-keys.xml and azure-bfs-auth-keys.xml). These have been added to > .gitignore to prevent them from accidentally being submitted in a patch, > which would leak the developers storage account credentials. These files > contain account names, storage account keys, and service endpoints. > There is some overlap of the configuration for WASB and ABFS, where they use > the same property name but use different values. > 1) Let's reduce the number of test configuration files to one, if possible. > 2) Let's simplify the account name, key, and endpoint configuration for WASB > and ABFS if possible, but still support the legacy way of doing it, which is > very error prone. > 3) Let's improve error handling, so that typos or misconfiguration are not so > difficult to troubleshoot. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15664) ABFS: Reduce test run time via parallelization and grouping
[ https://issues.apache.org/jira/browse/HADOOP-15664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15664: Fix Version/s: 3.2.0 > ABFS: Reduce test run time via parallelization and grouping > --- > > Key: HADOOP-15664 > URL: https://issues.apache.org/jira/browse/HADOOP-15664 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15664-HADOOP-15407-001.patch, > HADOOP-15664-HADOOP-15407-002.patch, HADOOP-15664-HADOOP-15407-003.patch, > HADOOP-15664-HADOOP-15407-004.patch, HADOOP-15664-HADOOP-15407-005.patch > > > 1) Let's reduce the total test runtime by improving parallelization of the > tests. > 2) Let's make it possible to select WASB tests, ABFS tests, or both so > developers can run only the tests appropriate for the change they've made. > 3) Update the testing-azure.md accordingly -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15692) ABFS: extensible support for custom oauth
[ https://issues.apache.org/jira/browse/HADOOP-15692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676361#comment-16676361 ] Sunil Govindan commented on HADOOP-15692: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: extensible support for custom oauth > - > > Key: HADOOP-15692 > URL: https://issues.apache.org/jira/browse/HADOOP-15692 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15692-HADOOP-15407-001.patch, > HADOOP-15692-HADOOP-15407-002.patch > > > ABFS supports oauth in various forms and needs to export interfaces for > customization of FileSystem.getDelegationToken and getAccessToken. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15664) ABFS: Reduce test run time via parallelization and grouping
[ https://issues.apache.org/jira/browse/HADOOP-15664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676359#comment-16676359 ] Sunil Govindan commented on HADOOP-15664: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Reduce test run time via parallelization and grouping > --- > > Key: HADOOP-15664 > URL: https://issues.apache.org/jira/browse/HADOOP-15664 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15664-HADOOP-15407-001.patch, > HADOOP-15664-HADOOP-15407-002.patch, HADOOP-15664-HADOOP-15407-003.patch, > HADOOP-15664-HADOOP-15407-004.patch, HADOOP-15664-HADOOP-15407-005.patch > > > 1) Let's reduce the total test runtime by improving parallelization of the > tests. > 2) Let's make it possible to select WASB tests, ABFS tests, or both so > developers can run only the tests appropriate for the change they've made. > 3) Update the testing-azure.md accordingly -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15692) ABFS: extensible support for custom oauth
[ https://issues.apache.org/jira/browse/HADOOP-15692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15692: Fix Version/s: 3.2.0 > ABFS: extensible support for custom oauth > - > > Key: HADOOP-15692 > URL: https://issues.apache.org/jira/browse/HADOOP-15692 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15692-HADOOP-15407-001.patch, > HADOOP-15692-HADOOP-15407-002.patch > > > ABFS supports oauth in various forms and needs to export interfaces for > customization of FileSystem.getDelegationToken and getAccessToken. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15663) ABFS: Simplify configuration
[ https://issues.apache.org/jira/browse/HADOOP-15663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676360#comment-16676360 ] Sunil Govindan commented on HADOOP-15663: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Simplify configuration > > > Key: HADOOP-15663 > URL: https://issues.apache.org/jira/browse/HADOOP-15663 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15663-HADOOP-15407-001.patch, > HADOOP-15663-HADOOP-15407-002.patch, HADOOP-15663-HADOOP-15407-003.patch, > HADOOP-15663-HADOOP-15407-004.patch, HADOOP-15663-HADOOP-15407-005.patch > > > Configuration for WASB and ABFS is too complex. The current approach is to > use four files for test configuration. > Both WASB and ABFS have basic test configuration which is committed to the > repo (azure-test.xml and azure-bfs-test.xml). Currently these contain the > fs.AbstractFileSystem.[scheme].impl configuration, but otherwise are empty > except for an include reference to a file containing the endpoint > credentials. > Both WASB and ABFS have endpoint credential configuration files > (azure-auth-keys.xml and azure-bfs-auth-keys.xml). These have been added to > .gitignore to prevent them from accidentally being submitted in a patch, > which would leak the developers storage account credentials. These files > contain account names, storage account keys, and service endpoints. > There is some overlap of the configuration for WASB and ABFS, where they use > the same property name but use different values. > 1) Let's reduce the number of test configuration files to one, if possible. > 2) Let's simplify the account name, key, and endpoint configuration for WASB > and ABFS if possible, but still support the legacy way of doing it, which is > very error prone. > 3) Let's improve error handling, so that typos or misconfiguration are not so > difficult to troubleshoot. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15745) Add ABFS configuration to ConfigRedactor
[ https://issues.apache.org/jira/browse/HADOOP-15745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15745: Fix Version/s: 3.2.0 > Add ABFS configuration to ConfigRedactor > > > Key: HADOOP-15745 > URL: https://issues.apache.org/jira/browse/HADOOP-15745 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15735-HADOOP-15407.001.patch, > HADOOP-15745-HADOOP-15407-002.patch > > > Sensitive information like credentials should be detected by ConfigRedactor > so they never appear in logs or other channels. ABFS credentials are not all > currently detected correctly so we should amend the default list of config > patterns. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15703) ABFS - Implement client-side throttling
[ https://issues.apache.org/jira/browse/HADOOP-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676358#comment-16676358 ] Sunil Govindan commented on HADOOP-15703: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS - Implement client-side throttling > > > Key: HADOOP-15703 > URL: https://issues.apache.org/jira/browse/HADOOP-15703 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sneha Varma >Assignee: Thomas Marquardt >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15703-HADOOP-15407-001.patch, > HADOOP-15703-HADOOP-15407-002.patch, HADOOP-15703-HADOOP-15407-003.patch, > HADOOP-15703-HADOOP-15407-004.patch > > > Big data workloads frequently exceed the AzureBlobFS max ingress and egress > limits > (https://docs.microsoft.com/en-us/azure/storage/common/storage-scalability-targets). > For example, the max ingress limit for a GRS account in the United States is > currently 10 Gbps. When the limit is exceeded, the AzureBlobFS service fails > a percentage of incoming requests, and this causes the client to initiate the > retry policy. The retry policy delays requests by sleeping, but the sleep > duration is independent of the client throughput and account limit. This > results in low throughput, due to the high number of failed requests and > thrashing causes by the retry policy. > To fix this, we introduce a client-side throttle which minimizes failed > requests and maximizes throughput. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15728) ABFS: Add backward compatibility to handle Unsupported Operation for storage account with no namespace feature
[ https://issues.apache.org/jira/browse/HADOOP-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15728: Fix Version/s: 3.2.0 > ABFS: Add backward compatibility to handle Unsupported Operation for storage > account with no namespace feature > -- > > Key: HADOOP-15728 > URL: https://issues.apache.org/jira/browse/HADOOP-15728 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15728-HADOOP-15407-001.patch, > HADOOP-15728-HADOOP-15407-002.patch, HADOOP-15728-HADOOP-15407-003.patch > > > For non-namespaceenbaled account, UnsupportedOperationException is thrown > when setOwner() or setPermission() is called, which cause problems for others. > They should behave the same as before, so should make them no-op if the > storage account is not namespace enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15694) ABFS: Allow OAuth credentials to not be tied to accounts
[ https://issues.apache.org/jira/browse/HADOOP-15694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676355#comment-16676355 ] Sunil Govindan commented on HADOOP-15694: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Allow OAuth credentials to not be tied to accounts > > > Key: HADOOP-15694 > URL: https://issues.apache.org/jira/browse/HADOOP-15694 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15694-HADOOP-15407-005.patch, > HADOOP-15694-HADOOP-15407.003.patch, HADOOP-15694-HADOOP-15407.004.patch, > HADOOP-15694-HADOOP-15407.006.patch, HADOOP-15694-HADOOP-15407.007.patch, > HADOOP-15694-HADOOP-15407.008.patch, HADOOP-15694.001.patch, > HADOOP-15694.002.patch, HADOOP-15694.003.patch > > > Now that there's OAuth support, it's possible to have a notion of identity > that's distinct from the account itself. If a cluster is configured via OAuth > with it's own identity, it's likely operators will want to use that identity > regardless of which storage account a job uses. > So OAuth configs right now (and probably others) are looked up with > .. I propose that we add a function for looking up these > configs that returns an account-specific value if it exists, but in the event > it does not will also try to return , if that exists. > I can work on a patch for this if nobody has any objections. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15728) ABFS: Add backward compatibility to handle Unsupported Operation for storage account with no namespace feature
[ https://issues.apache.org/jira/browse/HADOOP-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676356#comment-16676356 ] Sunil Govindan commented on HADOOP-15728: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Add backward compatibility to handle Unsupported Operation for storage > account with no namespace feature > -- > > Key: HADOOP-15728 > URL: https://issues.apache.org/jira/browse/HADOOP-15728 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15728-HADOOP-15407-001.patch, > HADOOP-15728-HADOOP-15407-002.patch, HADOOP-15728-HADOOP-15407-003.patch > > > For non-namespaceenbaled account, UnsupportedOperationException is thrown > when setOwner() or setPermission() is called, which cause problems for others. > They should behave the same as before, so should make them no-op if the > storage account is not namespace enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15703) ABFS - Implement client-side throttling
[ https://issues.apache.org/jira/browse/HADOOP-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15703: Fix Version/s: 3.2.0 > ABFS - Implement client-side throttling > > > Key: HADOOP-15703 > URL: https://issues.apache.org/jira/browse/HADOOP-15703 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sneha Varma >Assignee: Thomas Marquardt >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15703-HADOOP-15407-001.patch, > HADOOP-15703-HADOOP-15407-002.patch, HADOOP-15703-HADOOP-15407-003.patch, > HADOOP-15703-HADOOP-15407-004.patch > > > Big data workloads frequently exceed the AzureBlobFS max ingress and egress > limits > (https://docs.microsoft.com/en-us/azure/storage/common/storage-scalability-targets). > For example, the max ingress limit for a GRS account in the United States is > currently 10 Gbps. When the limit is exceeded, the AzureBlobFS service fails > a percentage of incoming requests, and this causes the client to initiate the > retry policy. The retry policy delays requests by sleeping, but the sleep > duration is independent of the client throughput and account limit. This > results in low throughput, due to the high number of failed requests and > thrashing causes by the retry policy. > To fix this, we introduce a client-side throttle which minimizes failed > requests and maximizes throughput. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15745) Add ABFS configuration to ConfigRedactor
[ https://issues.apache.org/jira/browse/HADOOP-15745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676357#comment-16676357 ] Sunil Govindan commented on HADOOP-15745: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > Add ABFS configuration to ConfigRedactor > > > Key: HADOOP-15745 > URL: https://issues.apache.org/jira/browse/HADOOP-15745 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15735-HADOOP-15407.001.patch, > HADOOP-15745-HADOOP-15407-002.patch > > > Sensitive information like credentials should be detected by ConfigRedactor > so they never appear in logs or other channels. ABFS credentials are not all > currently detected correctly so we should amend the default list of config > patterns. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15906) [JDK 11] Fix compilation failures with jdk 11
[
https://issues.apache.org/jira/browse/HADOOP-15906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lokesh Jain updated HADOOP-15906:
-
Status: Patch Available (was: Open)
> [JDK 11] Fix compilation failures with jdk 11
> -
>
> Key: HADOOP-15906
> URL: https://issues.apache.org/jira/browse/HADOOP-15906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>Priority: Major
> Attachments: HADOOP-15906.001.patch
>
>
> This Jira aims to fix the compilation failures with jdk 11. The compilation
> failure occurs in hadoop-hdds and hadoop-ozone module.
> {code:java}
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-hdds/common/target/generated-sources/java/org/apache/hadoop/hdds/protocol/datanode/proto/XceiverClientProtocolServiceGrpc.java:[20,18]
> cannot find symbol
> symbol: class Generated
> location: package javax.annotation
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-hdds/common/target/generated-sources/java/org/apache/hadoop/hdds/protocol/datanode/proto/IntraDatanodeProtocolServiceGrpc.java:[20,18]
> cannot find symbol
> symbol: class Generated
> location: package javax.annotation
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[20,33]
> package javax.xml.bind.annotation does not exist
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[21,33]
> package javax.xml.bind.annotation does not exist
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[22,33]
> package javax.xml.bind.annotation does not exist
> [ERROR]
> /Users/ljain/codebase/apache/hadoop/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/commontypes/BucketMetadata.java:[23,42]
> package javax.xml.bind.annotation.adapters does not exist
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15757) ABFS: remove dependency on common-codec Base64
[ https://issues.apache.org/jira/browse/HADOOP-15757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676352#comment-16676352 ] Sunil Govindan commented on HADOOP-15757: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: remove dependency on common-codec Base64 > -- > > Key: HADOOP-15757 > URL: https://issues.apache.org/jira/browse/HADOOP-15757 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15757-HADOOP-15407-001.patch, > HADOOP-15757-HADOOP-15407-002.patch > > > Currently ABFS relies on common-codec Base64, because different versions of > common-codec are widely used and some are missing the methods needed by ABFS, > it cause lots of "no such method" exception in customer's env, hence we > decide to add util for Base64 to avoid such issues in future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15753) ABFS: support path "abfs://mycluster/file/path"
[ https://issues.apache.org/jira/browse/HADOOP-15753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676353#comment-16676353 ] Sunil Govindan commented on HADOOP-15753: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: support path "abfs://mycluster/file/path" > --- > > Key: HADOOP-15753 > URL: https://issues.apache.org/jira/browse/HADOOP-15753 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15753-HADOOP-15407-001.patch > > > WASB support path format: "wasb://mycluster/file/path", but ABFS doesn't, > which caused some issues for customer. I will add support for this path > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15753) ABFS: support path "abfs://mycluster/file/path"
[ https://issues.apache.org/jira/browse/HADOOP-15753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15753: Fix Version/s: 3.2.0 > ABFS: support path "abfs://mycluster/file/path" > --- > > Key: HADOOP-15753 > URL: https://issues.apache.org/jira/browse/HADOOP-15753 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15753-HADOOP-15407-001.patch > > > WASB support path format: "wasb://mycluster/file/path", but ABFS doesn't, > which caused some issues for customer. I will add support for this path > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
