[jira] [Commented] (HADOOP-14667) Flexible Visual Studio support
[
https://issues.apache.org/jira/browse/HADOOP-14667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151403#comment-16151403
]
Hadoop QA commented on HADOOP-14667:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m
1s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m
22s{color} | {color:red} root in trunk failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m
22s{color} | {color:red} root in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 44s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestKDiag |
| | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14667 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885053/HADOOP-14667.05.patch
|
| Optional Tests | asflicense mvnsite unit compile javac javadoc
mvninstall xml |
| uname | Linux 23f645bae94e 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 275980b |
| Default Java | 1.8.0_144 |
| javadoc |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13156/artifact/patchprocess/branch-javadoc-root.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13156/artifact/patchprocess/patch-javadoc-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13156/artifact/patchprocess/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13156/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-native-client . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13156/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Flexible Visual Studio support
> --
>
> Key: HADOOP-14667
> URL: https://issues.apache.org/jira/browse/HADOOP-14667
> Project: Hadoop Common
>
[jira] [Commented] (HADOOP-14667) Flexible Visual Studio support
[
https://issues.apache.org/jira/browse/HADOOP-14667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151398#comment-16151398
]
Allen Wittenauer commented on HADOOP-14667:
---
The next pass (unit tests) also does a clean and works. Umm, ok... also, the
"don't convert again code" kicked in:
{code}
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-os) @ hadoop-common
---
[INFO]
[INFO] --- exec-maven-plugin:1.3.1:exec (convert-ms-winutils) @ hadoop-common
---
F:\Microsoft\Visual Studio CE 2015\Common7\IDE\devenv.com
F:\Microsoft\Visual Studio CE 2015\Common7\IDE\devenv.exe
"Solution files already upgraded."
[INFO]
[INFO] --- exec-maven-plugin:1.3.1:exec (convert-ms-native-dll) @ hadoop-common
---
F:\Microsoft\Visual Studio CE 2015\Common7\IDE\devenv.com
F:\Microsoft\Visual Studio CE 2015\Common7\IDE\devenv.exe
"Solution files already upgraded."
[INFO]
[INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:protoc (compile-protoc) @
hadoop-common ---
{code}
> Flexible Visual Studio support
> --
>
> Key: HADOOP-14667
> URL: https://issues.apache.org/jira/browse/HADOOP-14667
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0-beta1
> Environment: Windows
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Attachments: HADOOP-14667.00.patch, HADOOP-14667.01.patch,
> HADOOP-14667.02.patch, HADOOP-14667.03.patch, HADOOP-14667.04.patch,
> HADOOP-14667.05.patch
>
>
> Is it time to upgrade the Windows native project files to use something more
> modern than Visual Studio 2010?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14667) Flexible Visual Studio support
[
https://issues.apache.org/jira/browse/HADOOP-14667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151387#comment-16151387
]
Allen Wittenauer commented on HADOOP-14667:
---
>From the current test on the ASF Windows boxes, some relevant logs:
https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-trunk-win/ws/out/patch-mvninstall-root.txt
Conversion:
{code}
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-os) @ hadoop-common
---
[INFO]
[INFO] --- exec-maven-plugin:1.3.1:exec (convert-ms-winutils) @ hadoop-common
---
F:\Microsoft\Visual Studio CE 2015\Common7\IDE\devenv.com
F:\Microsoft\Visual Studio CE 2015\Common7\IDE\devenv.exe
Microsoft Visual Studio 2015 Version 14.0.25420.1.
Copyright (C) Microsoft Corp. All rights reserved.
Upgrading project 'winutils'...
Configuration 'Debug|Win32': changing Platform Toolset to 'v140' (was
'v100').
Configuration 'Debug|x64': changing Platform Toolset to 'v140' (was
'v100').
Configuration 'Release|Win32': changing Platform Toolset to 'v140' (was
'v100').
Configuration 'Release|x64': changing Platform Toolset to 'v140' (was
'v100').
Upgrading project 'libwinutils'...
Configuration 'Debug|Win32': changing Platform Toolset to 'v140' (was
'v100').
Configuration 'Debug|x64': changing Platform Toolset to 'v140' (was
'v100').
Configuration 'Release|Win32': changing Platform Toolset to 'v140' (was
'v100').
Configuration 'Release|x64': changing Platform Toolset to 'v140' (was
'v100').
F:\jenkins\jenkins-slave\workspace\hadoop-trunk-win\s\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj
: warning : The build tools for Visual Studio 2015 (v140) cannot be found.
Install Visual Studio 2015 (v140) to build using the Visual Studio 2015 (v140)
build tools.
Migration completed successfully, but some warnings were detected during
migration.
For more information, see the migration report:
F:\jenkins\jenkins-slave\workspace\hadoop-trunk-win\s\hadoop-common-project\hadoop-common\src\main\winutils\UpgradeLog.htm
[INFO]
[INFO] --- exec-maven-plugin:1.3.1:exec (convert-ms-native-dll) @ hadoop-common
---
F:\Microsoft\Visual Studio CE 2015\Common7\IDE\devenv.com
F:\Microsoft\Visual Studio CE 2015\Common7\IDE\devenv.exe
Microsoft Visual Studio 2015 Version 14.0.25420.1.
Copyright (C) Microsoft Corp. All rights reserved.
Upgrading project 'native'...
Configuration 'Release|Win32': changing Platform Toolset to 'v140' (was
'v100').
Configuration 'Release|x64': changing Platform Toolset to 'v140' (was
'v100').
Migration completed successfully, but some warnings were detected during
migration.
For more information, see the migration report:
F:\jenkins\jenkins-slave\workspace\hadoop-trunk-win\s\hadoop-common-project\hadoop-common\src\main\native\UpgradeLog.htm
[INFO]
[INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:protoc (compile-protoc) @
hadoop-common ---
[INFO] Wrote protoc checksums to file
F:\jenkins\jenkins-slave\workspace\hadoop-trunk-win\s\hadoop-common-project\hadoop-common\target\hadoop-maven-plugins-protoc-checksums.json
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-common
---
{code}
Compilation:
{code}
[INFO] --- exec-maven-plugin:1.3.1:exec (compile-ms-winutils) @ hadoop-common
---
Building the projects in this solution one at a time. To enable parallel build,
please add the "/m" switch.
Build started 9/2/2017 2:52:27 AM.
Project
"F:\jenkins\jenkins-slave\workspace\hadoop-trunk-win\s\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln"
on node 1 (default targets).
ValidateSolutionConfiguration:
Building solution configuration "Release|X64".
...
Build succeeded.
...
5 Warning(s)
0 Error(s)
Time Elapsed 00:00:13.39
{code}
Yetus then does a second (compile) pass. Which fails. :(
{code}
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-common ---
[INFO] Deleting
F:\jenkins\jenkins-slave\workspace\hadoop-trunk-win\s\hadoop-common-project\hadoop-common\target
{code}
I guess something still has a lock on that directory?
> Flexible Visual Studio support
> --
>
> Key: HADOOP-14667
> URL: https://issues.apache.org/jira/browse/HADOOP-14667
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0-beta1
> Environment: Windows
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Attachments: HADOOP-14667.00.patch, HADOOP-14667.01.patch,
> HADOOP-14667.02.patch, HADOOP-14667.03.patch, HADOOP-14667.04.patch,
> HADOOP-14667.05.patch
>
>
> Is it time to upgrade the Windows native project files to use something more
> modern than Visual Studio 2010?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HADOOP-14667) Flexible Visual Studio support
[ https://issues.apache.org/jira/browse/HADOOP-14667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-14667: -- Flags: Important > Flexible Visual Studio support > -- > > Key: HADOOP-14667 > URL: https://issues.apache.org/jira/browse/HADOOP-14667 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-beta1 > Environment: Windows >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14667.00.patch, HADOOP-14667.01.patch, > HADOOP-14667.02.patch, HADOOP-14667.03.patch, HADOOP-14667.04.patch, > HADOOP-14667.05.patch > > > Is it time to upgrade the Windows native project files to use something more > modern than Visual Studio 2010? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14667) Flexible Visual Studio support
[ https://issues.apache.org/jira/browse/HADOOP-14667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-14667: -- Release Note: This change updates the Microsoft Windows build directions to be more flexible with regards to Visual Studio compiler versions: * Any version of Visual Studio 2010 Pro or higher may be used. * MSBuild Solution files are converted to the version of VS at build time * Example command file to set command paths prior to using maven so that conversion works Additionally, Snappy and ISA-L that use bin as the location of the DLL will now be recognized without having to set their respective lib paths if the prefix is set. Note to contributors: It is very important that solutions for any patches remain at the VS 2010-level. was: This change updates the Microsoft Windows build directions to be more flexible with regards to Visual Studio compiler versions: * Any version of Visual Studio 2010 Pro or higher may be used. * Example command files to set command paths and upgrade Visual Studio solution files are now located in dev-support/ Additionally, Snappy and ISA-L that use bin as the location of the DLL will now be recognized without having to set their respective lib paths if the prefix is set. > Flexible Visual Studio support > -- > > Key: HADOOP-14667 > URL: https://issues.apache.org/jira/browse/HADOOP-14667 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-beta1 > Environment: Windows >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14667.00.patch, HADOOP-14667.01.patch, > HADOOP-14667.02.patch, HADOOP-14667.03.patch, HADOOP-14667.04.patch, > HADOOP-14667.05.patch > > > Is it time to upgrade the Windows native project files to use something more > modern than Visual Studio 2010? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14696) parallel tests don't work for Windows
[
https://issues.apache.org/jira/browse/HADOOP-14696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-14696:
--
Attachment: HADOOP-14696.04.patch
> parallel tests don't work for Windows
> -
>
> Key: HADOOP-14696
> URL: https://issues.apache.org/jira/browse/HADOOP-14696
> Project: Hadoop Common
> Issue Type: Test
> Components: test
>Affects Versions: 3.0.0-beta1
> Environment: Windows
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Minor
> Attachments: HADOOP-14696-002.patch, HADOOP-14696-003.patch,
> HADOOP-14696.00.patch, HADOOP-14696.01.patch, HADOOP-14696.04.patch
>
>
> If hadoop-common-project/hadoop-common is run with the -Pparallel-tests flag,
> it fails in create-parallel-tests-dirs from the pom.xml
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.7:run
> (create-parallel-tests-dirs) on project hadoop-common: An Ant BuildException
> has occured: Directory
> F:\jenkins\jenkins-slave\workspace\hadoop-trunk-win\s\hadoop-common-project\hadoop-common\jenkinsjenkins-slaveworkspacehadoop-trunk-winshadoop-common-projecthadoop-common
> arget\test\data\1 creation was not successful for an unknown reason
> [ERROR] around Ant part
[jira] [Updated] (HADOOP-14667) Flexible Visual Studio support
[ https://issues.apache.org/jira/browse/HADOOP-14667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-14667: -- Attachment: HADOOP-14667.05.patch -05: * convert solution files on the fly > Flexible Visual Studio support > -- > > Key: HADOOP-14667 > URL: https://issues.apache.org/jira/browse/HADOOP-14667 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-beta1 > Environment: Windows >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14667.00.patch, HADOOP-14667.01.patch, > HADOOP-14667.02.patch, HADOOP-14667.03.patch, HADOOP-14667.04.patch, > HADOOP-14667.05.patch > > > Is it time to upgrade the Windows native project files to use something more > modern than Visual Studio 2010? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14830) Write some non-Docker-based instructions on how to build on Mac OS X
Allen Wittenauer created HADOOP-14830: - Summary: Write some non-Docker-based instructions on how to build on Mac OS X Key: HADOOP-14830 URL: https://issues.apache.org/jira/browse/HADOOP-14830 Project: Hadoop Common Issue Type: Improvement Components: build, documentation Affects Versions: 3.0.0-beta1 Reporter: Allen Wittenauer Assignee: Allen Wittenauer We should have some decent documentation on how to build on OS X, now that almost everything works again. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-14136) Default FS For ViewFS
[
https://issues.apache.org/jira/browse/HADOOP-14136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-14136 started by Manoj Govindassamy.
---
> Default FS For ViewFS
> -
>
> Key: HADOOP-14136
> URL: https://issues.apache.org/jira/browse/HADOOP-14136
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, viewfs
>Reporter: Erik Krogen
>Assignee: Manoj Govindassamy
>
> It would be useful if ViewFS had the ability to designate one FileSystem as a
> "primary"/"default" to fall back to in the case that an entry in the mount
> table is not found. Consider the situation when you have a mount table that
> looks like:
> {code}
> /data -> hdfs://nn1/data
> /logs -> hdfs://nn1/logs
> /user -> hdfs://nn1/user
> /remote -> hdfs://nn2/remote
> {code}
> {{nn1}} here is being used as the primary, with a specific directory 'remote'
> being offloaded to another namenode. This works, but if we want to add
> another top-level directory to {{nn1}}, we have to update all of our
> client-side mount tables. Merge links (HADOOP-8298) could be used to achieve
> this but they do not seem to be coming soon; this special case of a default
> FS is much simpler - try to resolve through the VFS mount table, if not
> found, then resolve to the defaultFS.
> There is a good discussion of this at
> https://issues.apache.org/jira/browse/HADOOP-13055?focusedCommentId=15733822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15733822
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14829) Path should support colon
[ https://issues.apache.org/jira/browse/HADOOP-14829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuliya Feldman updated HADOOP-14829: Issue Type: Sub-task (was: Improvement) Parent: HADOOP-3257 > Path should support colon > - > > Key: HADOOP-14829 > URL: https://issues.apache.org/jira/browse/HADOOP-14829 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Yuliya Feldman > > Object storages today support colon ( : ) in names, while Hadoop Path does > not support it. > This JIRA is to allow Path to support colon in URI -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14829) Path should support colon
[ https://issues.apache.org/jira/browse/HADOOP-14829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuliya Feldman updated HADOOP-14829: Description: Object storages today support colon ( : ) in names, while Hadoop Path does not support it. This JIRA is to allow Path to support colon in URI was: Object storages today support colon (:) in names, while Hadoop Path does not support it. This JIRA is to allow Path to support colon in URI > Path should support colon > - > > Key: HADOOP-14829 > URL: https://issues.apache.org/jira/browse/HADOOP-14829 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Yuliya Feldman > > Object storages today support colon ( : ) in names, while Hadoop Path does > not support it. > This JIRA is to allow Path to support colon in URI -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14829) Path should support colon
Yuliya Feldman created HADOOP-14829: --- Summary: Path should support colon Key: HADOOP-14829 URL: https://issues.apache.org/jira/browse/HADOOP-14829 Project: Hadoop Common Issue Type: Improvement Components: common Reporter: Yuliya Feldman Object storages today support colon (:) in names, while Hadoop Path does not support it. This JIRA is to allow Path to support colon in URI -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13421) Switch to v2 of the S3 List Objects API in S3A
[
https://issues.apache.org/jira/browse/HADOOP-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151302#comment-16151302
]
Aaron Fabbri commented on HADOOP-13421:
---
Whoever added the forced list response paging to ITestS3AContractGetFileStatus,
thank you. Was going to add that and see it is already there.
Also explains why that test was timing out with v2 list.. not just slow home
internet.. I needed to change this bit:
{noformat}
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
index e8b739432d1..eb80d37a12f 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
@@ -1113,7 +1113,7 @@ protected ListObjectsV2Result
continueListObjects(ListObjectsV2Request req,
ListObjectsV2Result objects) {
incrementStatistic(OBJECT_CONTINUE_LIST_REQUESTS);
incrementReadOperations();
-req.setContinuationToken(objects.getContinuationToken());
+req.setContinuationToken(objects.getNextContinuationToken());
return s3.listObjectsV2(req);
}
{noformat}
So, the v2 response has two continuation token fields, {{ContinuationToken}}
and {{NextContinuationToken}}. Turns out i was using the former and retrieving
the same 2 results over and over. Gave me a giggle, had to share.. V2 patch
coming soon.
> Switch to v2 of the S3 List Objects API in S3A
> --
>
> Key: HADOOP-13421
> URL: https://issues.apache.org/jira/browse/HADOOP-13421
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steven K. Wong
>Assignee: Aaron Fabbri
>Priority: Minor
> Attachments: HADOOP-13421-HADOOP-13345.001.patch
>
>
> Unlike [version
> 1|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html] of the
> S3 List Objects API, [version
> 2|http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html] by
> default does not fetch object owner information, which S3A doesn't need
> anyway. By switching to v2, there will be less data to transfer/process.
> Also, it should be more robust when listing a versioned bucket with "a large
> number of delete markers" ([according to
> AWS|https://aws.amazon.com/releasenotes/Java/0735652458007581]).
> Methods in S3AFileSystem that use this API include:
> * getFileStatus(Path)
> * innerDelete(Path, boolean)
> * innerListStatus(Path)
> * innerRename(Path, Path)
> Requires AWS SDK 1.10.75 or later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151289#comment-16151289
]
Ping Liu commented on HADOOP-14600:
---
Yeah, it must be automatically included with MingW, Visual Studio, or some
other installation. Thanks for telling me that! It's good to know all of
these especially when using command line.
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
>
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
>Reporter: Steve Loughran
>Assignee: Ping Liu
> Attachments: HADOOP-14600.001.patch,
> TestRawLocalFileSystemContract.java
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13916) Document how downstream clients should make use of the new shaded client artifacts
[ https://issues.apache.org/jira/browse/HADOOP-13916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151283#comment-16151283 ] Sean Busbey commented on HADOOP-13916: -- Thanks Chris! > Document how downstream clients should make use of the new shaded client > artifacts > -- > > Key: HADOOP-13916 > URL: https://issues.apache.org/jira/browse/HADOOP-13916 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Sean Busbey >Assignee: Sean Busbey > > provide a quickstart that walks through using the new shaded dependencies > with Maven to create a simple downstream project. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13916) Document how downstream clients should make use of the new shaded client artifacts
[ https://issues.apache.org/jira/browse/HADOOP-13916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151276#comment-16151276 ] Chris Douglas commented on HADOOP-13916: OK. If it'd make sense for BigTop/Yetus to be a consumer or coordinating entity for such things, I'm sure we can work it out later. Thanks [~busbey]. The repo is [created|https://git1-us-west.apache.org/repos/asf?p=hadoop-downstream-tests.git]. You should have all the necessary privs. > Document how downstream clients should make use of the new shaded client > artifacts > -- > > Key: HADOOP-13916 > URL: https://issues.apache.org/jira/browse/HADOOP-13916 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Sean Busbey >Assignee: Sean Busbey > > provide a quickstart that walks through using the new shaded dependencies > with Maven to create a simple downstream project. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151275#comment-16151275
]
Allen Wittenauer commented on HADOOP-14600:
---
bq. As I checked Program Files, I found there is another Git installation just
as you mentioned!
It's good to know my spyware is still working. ;)
In reality, MingW (which is what that is) comes with the KitWare cmake binary
or maybe the git binary package. Most people who go down this path have it
installed already so it was a safe hunch.
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
>
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
>Reporter: Steve Loughran
>Assignee: Ping Liu
> Attachments: HADOOP-14600.001.patch,
> TestRawLocalFileSystemContract.java
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151255#comment-16151255
]
Ping Liu commented on HADOOP-14600:
---
Thanks [~aw]! I have both Cygwin and Git. But Neither has /usr/bin. As I
checked Program Files, I found there is another Git installation just as you
mentioned! I can try this one.
But you are right. I got lots of cuts and blood with getting mvn test
-Dtest=foo running on Windows. I'll try to run test-patch on Linux and come
back test the same scenario on Windows probably with mvn test -Dtest=foo one by
one as a workaround.
Thanks!
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
>
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
>Reporter: Steve Loughran
>Assignee: Ping Liu
> Attachments: HADOOP-14600.001.patch,
> TestRawLocalFileSystemContract.java
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14667) Flexible Visual Studio support
[ https://issues.apache.org/jira/browse/HADOOP-14667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151245#comment-16151245 ] Allen Wittenauer commented on HADOOP-14667: --- Now that HADOOP-14670 has gone in, this is no longer an incompatible change. :D > Flexible Visual Studio support > -- > > Key: HADOOP-14667 > URL: https://issues.apache.org/jira/browse/HADOOP-14667 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-beta1 > Environment: Windows >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14667.00.patch, HADOOP-14667.01.patch, > HADOOP-14667.02.patch, HADOOP-14667.03.patch, HADOOP-14667.04.patch > > > Is it time to upgrade the Windows native project files to use something more > modern than Visual Studio 2010? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14667) Flexible Visual Studio support
[ https://issues.apache.org/jira/browse/HADOOP-14667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-14667: -- Hadoop Flags: (was: Incompatible change) > Flexible Visual Studio support > -- > > Key: HADOOP-14667 > URL: https://issues.apache.org/jira/browse/HADOOP-14667 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-beta1 > Environment: Windows >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14667.00.patch, HADOOP-14667.01.patch, > HADOOP-14667.02.patch, HADOOP-14667.03.patch, HADOOP-14667.04.patch > > > Is it time to upgrade the Windows native project files to use something more > modern than Visual Studio 2010? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14667) Flexible Visual Studio support
[ https://issues.apache.org/jira/browse/HADOOP-14667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-14667: -- Release Note: This change updates the Microsoft Windows build directions to be more flexible with regards to Visual Studio compiler versions: * Any version of Visual Studio 2010 Pro or higher may be used. * Example command files to set command paths and upgrade Visual Studio solution files are now located in dev-support/ Additionally, Snappy and ISA-L that use bin as the location of the DLL will now be recognized without having to set their respective lib paths if the prefix is set. was: This change updates the Microsoft Windows build directions to be more flexible with regards to Visual Studio compiler versions: * CMake v3.1 or higher is now required. * Any version of Visual Studio 2010 Pro or higher may be used. * Example command files to set command paths and upgrade Visual Studio solution files are now located in dev-support/ Additionally, Snappy and ISA-L that use bin as the location of the DLL will now be recognized without having to set their respective lib paths if the prefix is set. > Flexible Visual Studio support > -- > > Key: HADOOP-14667 > URL: https://issues.apache.org/jira/browse/HADOOP-14667 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-beta1 > Environment: Windows >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14667.00.patch, HADOOP-14667.01.patch, > HADOOP-14667.02.patch, HADOOP-14667.03.patch, HADOOP-14667.04.patch > > > Is it time to upgrade the Windows native project files to use something more > modern than Visual Studio 2010? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[
https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151242#comment-16151242
]
Sean Busbey edited comment on HADOOP-14771 at 9/1/17 10:27 PM:
---
{quote}
HADOOP-14238 mentions that Guava is exposed in an AMRMClient API. I assume this
matters if we shade yarn-client?
We probably do need to do a sweep of the public APIs for additional third-party
library exposure.
{quote}
It matters. I'll see if there's a tool we can use to automate checking for
these things. A first-pass approach can be to just keep the shaded version of
the library as the thing that's in the API (though this is obviously less than
optimal).
Since this was a thing called out early in the shading discussion (I think
maybe by Steve L?) I assume this is a larger concern than just yarn-client and
thus shouldn't hold up this change. If no one disagrees with that, I'll make a
new subtask to figure out our exposure.
was (Author: busbey):
> HADOOP-14238 mentions that Guava is exposed in an AMRMClient API. I assume
> this matters if we shade yarn-client?
> We probably do need to do a sweep of the public APIs for additional
> third-party library exposure.
It matters. I'll see if there's a tool we can use to automate checking for
these things. A first-pass approach can be to just keep the shaded version of
the library as the thing that's in the API (though this is obviously less than
optimal).
Since this was a thing called out early in the shading discussion (I think
maybe by Steve L?) I assume this is a larger concern than just yarn-client and
thus shouldn't hold up this change. If no one disagrees with that, I'll make a
new subtask to figure out our exposure.
> hadoop-client does not include hadoop-yarn-client
> -
>
> Key: HADOOP-14771
> URL: https://issues.apache.org/jira/browse/HADOOP-14771
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common
>Reporter: Haibo Chen
>Assignee: Ajay Kumar
>Priority: Blocker
> Attachments: HADOOP-14771.01.patch
>
>
> The hadoop-client does not include hadoop-yarn-client, thus, the shared
> hadoop-client is incomplete.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151242#comment-16151242 ] Sean Busbey commented on HADOOP-14771: -- > HADOOP-14238 mentions that Guava is exposed in an AMRMClient API. I assume > this matters if we shade yarn-client? > We probably do need to do a sweep of the public APIs for additional > third-party library exposure. It matters. I'll see if there's a tool we can use to automate checking for these things. A first-pass approach can be to just keep the shaded version of the library as the thing that's in the API (though this is obviously less than optimal). Since this was a thing called out early in the shading discussion (I think maybe by Steve L?) I assume this is a larger concern than just yarn-client and thus shouldn't hold up this change. If no one disagrees with that, I'll make a new subtask to figure out our exposure. > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-14771.01.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151203#comment-16151203
]
Allen Wittenauer commented on HADOOP-14600:
---
You actually can run test-patch on Windows. I've got a daily job that runs on
the ASF infra now. \Program Files\Git\usr\bin has almost everything you need,
outside of cmake, protoc, maven, and whatever features you want. But there are
a lot of gotchas. Beyond that, there are a lot of problems with just using
mvn test on Windows. It's probably better just to specifically use mvn test
-Dtest=foo and test your specific foo at the moment.
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
>
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
>Reporter: Steve Loughran
>Assignee: Ping Liu
> Attachments: HADOOP-14600.001.patch,
> TestRawLocalFileSystemContract.java
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151199#comment-16151199 ] Andrew Wang commented on HADOOP-14771: -- HADOOP-14238 mentions that Guava is exposed in an AMRMClient API. I assume this matters if we shade yarn-client? We probably do need to do a sweep of the public APIs for additional third-party library exposure. > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-14771.01.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests
[
https://issues.apache.org/jira/browse/HADOOP-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151198#comment-16151198
]
Hadoop QA commented on HADOOP-14827:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 20s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
| | hadoop.test.TestLambdaTestUtils |
| | hadoop.security.TestShellBasedUnixGroupsMapping |
| | hadoop.ha.TestZKFailoverController |
| | hadoop.security.TestKDiag |
| | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14827 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885011/HADOOP-14827.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d60fb4dcf114 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c5281a8 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13155/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13155/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13155/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Allow StopWatch to accept a Timer parameter for tests
> -
>
>
[jira] [Updated] (HADOOP-13250) jdiff and dependency reports aren't linked in site web pages
[ https://issues.apache.org/jira/browse/HADOOP-13250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-13250: - Priority: Major (was: Critical) > jdiff and dependency reports aren't linked in site web pages > > > Key: HADOOP-13250 > URL: https://issues.apache.org/jira/browse/HADOOP-13250 > Project: Hadoop Common > Issue Type: Bug > Components: build, documentation >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Vinod Kumar Vavilapalli > > Even though they are in the site tar ball (after HADOOP-13245), they aren't > actually reachable. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13948) Create automated scripts to update LICENSE/NOTICE
[ https://issues.apache.org/jira/browse/HADOOP-13948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151152#comment-16151152 ] Andrew Wang commented on HADOOP-13948: -- Hi [~xiaochen], another ping, do you have time to revive this JIRA? > Create automated scripts to update LICENSE/NOTICE > - > > Key: HADOOP-13948 > URL: https://issues.apache.org/jira/browse/HADOOP-13948 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Critical > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14828) RetryUpToMaximumTimeWithFixedSleep is not bounded by maximum time
Jonathan Hung created HADOOP-14828:
--
Summary: RetryUpToMaximumTimeWithFixedSleep is not bounded by
maximum time
Key: HADOOP-14828
URL: https://issues.apache.org/jira/browse/HADOOP-14828
Project: Hadoop Common
Issue Type: Bug
Reporter: Jonathan Hung
In RetryPolicies.java, RetryUpToMaximumTimeWithFixedSleep is converted to a
RetryUpToMaximumCountWithFixedSleep, whose count is the maxTime / sleepTime:
{noformat}public RetryUpToMaximumTimeWithFixedSleep(long maxTime, long
sleepTime,
TimeUnit timeUnit) {
super((int) (maxTime / sleepTime), sleepTime, timeUnit);
this.maxTime = maxTime;
this.timeUnit = timeUnit;
}
{noformat}
But if retries take a long time, then the maxTime passed to the
RetryUpToMaximumTimeWithFixedSleep is exceeded.
As an example, while doing NM restarts, we saw an issue where the NMProxy
creates a retry policy which specifies a maximum wait time of 15 minutes and a
10 sec interval (which is converted to a MaximumCount policy with 15 min / 10
sec = 90 tries). But each NMProxy retry policy invokes o.a.h.ipc.Client's retry
policy: {noformat} if (connectionRetryPolicy == null) {
final int max = conf.getInt(
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_MAX_RETRIES_KEY,
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_MAX_RETRIES_DEFAULT);
final int retryInterval = conf.getInt(
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_RETRY_INTERVAL_KEY,
CommonConfigurationKeysPublic
.IPC_CLIENT_CONNECT_RETRY_INTERVAL_DEFAULT);
connectionRetryPolicy =
RetryPolicies.retryUpToMaximumCountWithFixedSleep(
max, retryInterval, TimeUnit.MILLISECONDS);
}{noformat}
So the time it takes the NMProxy to fail is actually (90 retries) * (10 sec
NMProxy interval + o.a.h.ipc.Client retry time). In the default case, ipc
client retries 10 times with a 1 sec interval, meaning the time it takes for
NMProxy to fail is (90)(10 sec + 10 sec) = 30 min instead of the 15 min
specified by NMProxy configuration.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests
[
https://issues.apache.org/jira/browse/HADOOP-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151099#comment-16151099
]
Wei-Chiu Chuang commented on HADOOP-14827:
--
Haven't reviewed the patch, but I like the proposal. Thanks Erik.
> Allow StopWatch to accept a Timer parameter for tests
> -
>
> Key: HADOOP-14827
> URL: https://issues.apache.org/jira/browse/HADOOP-14827
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, test
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-14827.000.patch
>
>
> {{StopWatch}} should optionally accept a {{Timer}} parameter rather than
> directly using {{Time}} so that its behavior can be controlled during tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests
[
https://issues.apache.org/jira/browse/HADOOP-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HADOOP-14827:
-
Status: Patch Available (was: In Progress)
> Allow StopWatch to accept a Timer parameter for tests
> -
>
> Key: HADOOP-14827
> URL: https://issues.apache.org/jira/browse/HADOOP-14827
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, test
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-14827.000.patch
>
>
> {{StopWatch}} should accept a {{Timer}} parameter rather than directly using
> {{Time}} so that its behavior can be controlled during tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests
[
https://issues.apache.org/jira/browse/HADOOP-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HADOOP-14827:
-
Description: {{StopWatch}} should optionally accept a {{Timer}} parameter
rather than directly using {{Time}} so that its behavior can be controlled
during tests. (was: {{StopWatch}} should accept a {{Timer}} parameter rather
than directly using {{Time}} so that its behavior can be controlled during
tests.)
> Allow StopWatch to accept a Timer parameter for tests
> -
>
> Key: HADOOP-14827
> URL: https://issues.apache.org/jira/browse/HADOOP-14827
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, test
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-14827.000.patch
>
>
> {{StopWatch}} should optionally accept a {{Timer}} parameter rather than
> directly using {{Time}} so that its behavior can be controlled during tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests
[
https://issues.apache.org/jira/browse/HADOOP-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HADOOP-14827:
-
Attachment: HADOOP-14827.000.patch
> Allow StopWatch to accept a Timer parameter for tests
> -
>
> Key: HADOOP-14827
> URL: https://issues.apache.org/jira/browse/HADOOP-14827
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, test
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-14827.000.patch
>
>
> {{StopWatch}} should accept a {{Timer}} parameter rather than directly using
> {{Time}} so that its behavior can be controlled during tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests
[
https://issues.apache.org/jira/browse/HADOOP-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-14827 started by Erik Krogen.
> Allow StopWatch to accept a Timer parameter for tests
> -
>
> Key: HADOOP-14827
> URL: https://issues.apache.org/jira/browse/HADOOP-14827
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, test
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
>
> {{StopWatch}} should accept a {{Timer}} parameter rather than directly using
> {{Time}} so that its behavior can be controlled during tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests
[
https://issues.apache.org/jira/browse/HADOOP-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151091#comment-16151091
]
Erik Krogen commented on HADOOP-14827:
--
I particularly want this to be able to create a reliable unit test for
HDFS-12323.
> Allow StopWatch to accept a Timer parameter for tests
> -
>
> Key: HADOOP-14827
> URL: https://issues.apache.org/jira/browse/HADOOP-14827
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, test
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
>
> {{StopWatch}} should accept a {{Timer}} parameter rather than directly using
> {{Time}} so that its behavior can be controlled during tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14827) Allow StopWatch to accept a Timer parameter for tests
Erik Krogen created HADOOP-14827:
Summary: Allow StopWatch to accept a Timer parameter for tests
Key: HADOOP-14827
URL: https://issues.apache.org/jira/browse/HADOOP-14827
Project: Hadoop Common
Issue Type: Improvement
Components: common, test
Reporter: Erik Krogen
Assignee: Erik Krogen
Priority: Minor
{{StopWatch}} should accept a {{Timer}} parameter rather than directly using
{{Time}} so that its behavior can be controlled during tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151090#comment-16151090
]
Ping Liu commented on HADOOP-14600:
---
Now I am trying to do the patch test on my Windows. It looks like
dev-support/bin/test-patch is a BASH script and cannot be run on Windows. Is
there any guide on how to run it on Windows or the patch test is not expected
to be run on Windows?
CC: [~aw]
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
>
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
>Reporter: Steve Loughran
>Assignee: Ping Liu
> Attachments: HADOOP-14600.001.patch,
> TestRawLocalFileSystemContract.java
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150975#comment-16150975 ] Steve Loughran commented on HADOOP-13345: - Filed HADOOP-14826. > S3Guard: Improved Consistency for S3A > - > > Key: HADOOP-13345 > URL: https://issues.apache.org/jira/browse/HADOOP-13345 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-13345.prototype1.patch, s3c.001.patch, > S3C-ConsistentListingonS3-Design.pdf, S3GuardImprovedConsistencyforS3A.pdf, > S3GuardImprovedConsistencyforS3AV2.pdf > > > This issue proposes S3Guard, a new feature of S3A, to provide an option for a > stronger consistency model than what is currently offered. The solution > coordinates with a strongly consistent external store to resolve > inconsistencies caused by the S3 eventual consistency model. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14826) review S3 docs prior to 3.0-beta-1
[ https://issues.apache.org/jira/browse/HADOOP-14826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-14826: --- Assignee: Steve Loughran Priority: Blocker (was: Major) Summary: review S3 docs prior to 3.0-beta-1 (was: reviewS3 docs prior to 3.0-beta-1) > review S3 docs prior to 3.0-beta-1 > -- > > Key: HADOOP-14826 > URL: https://issues.apache.org/jira/browse/HADOOP-14826 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > > The hadoop-aws docs need a review and update > * things marked as stabilizing (fast upload, .fadvise ..) can be considered > stable > * move s3n docs off to the side > * add a "how to move from s3n to s3a" para -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14820) Wasb mkdirs security checks inconsistent with HDFS
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150959#comment-16150959
]
Hadoop QA commented on HADOOP-14820:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 1
new + 80 unchanged - 1 fixed = 81 total (was 81) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
16s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14820 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884944/HADOOP-14820-007.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1e2c6997393d 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 063b6d0 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13154/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13154/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13154/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Wasb mkdirs security checks inconsistent with HDFS
> --
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs,
[jira] [Comment Edited] (HADOOP-14821) Executing the command 'hdfs -Dhadoop.security.credential.provider.path=file1.jceks,file2.jceks' fails if permission is denied to some files
[
https://issues.apache.org/jira/browse/HADOOP-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150952#comment-16150952
]
Aaron Fabbri edited comment on HADOOP-14821 at 9/1/17 6:12 PM:
---
Edit: I see what you are saying: Failure to access one file causes it to fail
even if subsequent files to allow access.
Also, I thought S3A support for credential providers landed around hadoop 2.8?
(HADOOP-12548) Not sure this will work in 2.7.x even when you get past the
permission issue. Maybe it was backported?
was (Author: fabbri):
It looks like a permissions issue on your .jceks file: it is not readable by
the user your are running the hadoop command as, right?
Also, I thought S3A support for credential providers landed around hadoop 2.8?
(HADOOP-12548) Not sure this will work in 2.7.x even when you get past the
permission issue.
> Executing the command 'hdfs
> -Dhadoop.security.credential.provider.path=file1.jceks,file2.jceks' fails if
> permission is denied to some files
> ---
>
> Key: HADOOP-14821
> URL: https://issues.apache.org/jira/browse/HADOOP-14821
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3, hdfs-client, security
>Affects Versions: 2.7.3
> Environment: hadoop-common-2.7.3.2.6.0.11-1
>Reporter: Ernani Pereira de Mattos Junior
>Priority: Critical
> Labels: features
>
> ===
> Request Use Case:
> UC1:
> The customer has the path to a directory and subdirectories full of keys. The
> customer knows that he does not have the access to all the keys, but ignoring
> this problem, the customer makes a list of the keys.
> UC1.2:
> The customer in a FIFO manner, try his access to the key provided on the
> list. If the access is granted locally then he can try the login on the s3a.
> UC1.2:
> The customer in a FIFO manner, try his access to the key provided on the
> list. If the access is not granted locally then he will skip the login on the
> s3a and try the next key on the list.
> ===
> For now, the UC1.2 fails with below exception and does not try the next key:
> {code}
> $ hdfs --loglevel DEBUG dfs
> -Dhadoop.security.credential.provider.path=jceks://hdfs/tmp/aws.jceks,jceks://hdfs/tmp/awst.jceks
> -ls s3a://av-dl-hwx-nprod-anhffpoc-enriched/hive/e_ceod/
> Not retrying because try once and fail.
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> Permission denied: user=502549376, access=READ,
> inode="/tmp/aws.jceks":admin:hdfs:-rwx--
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14821) Executing the command 'hdfs -Dhadoop.security.credential.provider.path=file1.jceks,file2.jceks' fails if permission is denied to some files
[
https://issues.apache.org/jira/browse/HADOOP-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150952#comment-16150952
]
Aaron Fabbri commented on HADOOP-14821:
---
It looks like a permissions issue on your .jceks file: it is not readable by
the user your are running the hadoop command as, right?
Also, I thought S3A support for credential providers landed around hadoop 2.8?
(HADOOP-12548) Not sure this will work in 2.7.x even when you get past the
permission issue.
> Executing the command 'hdfs
> -Dhadoop.security.credential.provider.path=file1.jceks,file2.jceks' fails if
> permission is denied to some files
> ---
>
> Key: HADOOP-14821
> URL: https://issues.apache.org/jira/browse/HADOOP-14821
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3, hdfs-client, security
>Affects Versions: 2.7.3
> Environment: hadoop-common-2.7.3.2.6.0.11-1
>Reporter: Ernani Pereira de Mattos Junior
>Priority: Critical
> Labels: features
>
> ===
> Request Use Case:
> UC1:
> The customer has the path to a directory and subdirectories full of keys. The
> customer knows that he does not have the access to all the keys, but ignoring
> this problem, the customer makes a list of the keys.
> UC1.2:
> The customer in a FIFO manner, try his access to the key provided on the
> list. If the access is granted locally then he can try the login on the s3a.
> UC1.2:
> The customer in a FIFO manner, try his access to the key provided on the
> list. If the access is not granted locally then he will skip the login on the
> s3a and try the next key on the list.
> ===
> For now, the UC1.2 fails with below exception and does not try the next key:
> {code}
> $ hdfs --loglevel DEBUG dfs
> -Dhadoop.security.credential.provider.path=jceks://hdfs/tmp/aws.jceks,jceks://hdfs/tmp/awst.jceks
> -ls s3a://av-dl-hwx-nprod-anhffpoc-enriched/hive/e_ceod/
> Not retrying because try once and fail.
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> Permission denied: user=502549376, access=READ,
> inode="/tmp/aws.jceks":admin:hdfs:-rwx--
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14674) Correct javadoc for getRandomizedTempPath
[
https://issues.apache.org/jira/browse/HADOOP-14674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150950#comment-16150950
]
Hudson commented on HADOOP-14674:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12301 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12301/])
HADOOP-14674. Correct javadoc for getRandomizedTempPath. Contributed by
(jitendra: rev 063b6d0c93d700a57a7c6c29fdd1bcdecd0b9dc0)
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java
> Correct javadoc for getRandomizedTempPath
> -
>
> Key: HADOOP-14674
> URL: https://issues.apache.org/jira/browse/HADOOP-14674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14674.001.patch, HADOOP-14674.002.patch
>
>
> getRandomizedTempPath has incorrect javadoc where the javadoc specifies a
> parameter to the function however the function doesnt expects one.
> {code}
> /**
>* Get a temp path. This may or may not be relative; it depends on what the
>* {@link #SYSPROP_TEST_DATA_DIR} is set to. If unset, it returns a path
>* under the relative path {@link #DEFAULT_TEST_DATA_PATH}
>* @param subpath sub path, with no leading "/" character
>* @return a string to use in paths
>*/
> public static String getRandomizedTempPath() {
> return getTempPath(RandomStringUtils.randomAlphanumeric(10));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150948#comment-16150948
]
Hadoop QA commented on HADOOP-14600:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 5 new + 156 unchanged - 0 fixed = 161 total (was 156) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 29s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
| | hadoop.fs.sftp.TestSFTPFileSystem |
| | hadoop.security.TestKDiag |
| | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14600 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884977/HADOOP-14600.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 568b8f8ee110 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 99a7f5d |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13153/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13153/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13153/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
[jira] [Updated] (HADOOP-14674) Correct javadoc for getRandomizedTempPath
[
https://issues.apache.org/jira/browse/HADOOP-14674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-14674:
--
Fix Version/s: 3.0.0-beta1
> Correct javadoc for getRandomizedTempPath
> -
>
> Key: HADOOP-14674
> URL: https://issues.apache.org/jira/browse/HADOOP-14674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14674.001.patch, HADOOP-14674.002.patch
>
>
> getRandomizedTempPath has incorrect javadoc where the javadoc specifies a
> parameter to the function however the function doesnt expects one.
> {code}
> /**
>* Get a temp path. This may or may not be relative; it depends on what the
>* {@link #SYSPROP_TEST_DATA_DIR} is set to. If unset, it returns a path
>* under the relative path {@link #DEFAULT_TEST_DATA_PATH}
>* @param subpath sub path, with no leading "/" character
>* @return a string to use in paths
>*/
> public static String getRandomizedTempPath() {
> return getTempPath(RandomStringUtils.randomAlphanumeric(10));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14674) Correct javadoc for getRandomizedTempPath
[
https://issues.apache.org/jira/browse/HADOOP-14674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HADOOP-14674:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Correct javadoc for getRandomizedTempPath
> -
>
> Key: HADOOP-14674
> URL: https://issues.apache.org/jira/browse/HADOOP-14674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-14674.001.patch, HADOOP-14674.002.patch
>
>
> getRandomizedTempPath has incorrect javadoc where the javadoc specifies a
> parameter to the function however the function doesnt expects one.
> {code}
> /**
>* Get a temp path. This may or may not be relative; it depends on what the
>* {@link #SYSPROP_TEST_DATA_DIR} is set to. If unset, it returns a path
>* under the relative path {@link #DEFAULT_TEST_DATA_PATH}
>* @param subpath sub path, with no leading "/" character
>* @return a string to use in paths
>*/
> public static String getRandomizedTempPath() {
> return getTempPath(RandomStringUtils.randomAlphanumeric(10));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14674) Correct javadoc for getRandomizedTempPath
[
https://issues.apache.org/jira/browse/HADOOP-14674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150929#comment-16150929
]
Mukul Kumar Singh commented on HADOOP-14674:
The v2 patch has been committed to trunk. Thanks for the commit [~jnp].
> Correct javadoc for getRandomizedTempPath
> -
>
> Key: HADOOP-14674
> URL: https://issues.apache.org/jira/browse/HADOOP-14674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-14674.001.patch, HADOOP-14674.002.patch
>
>
> getRandomizedTempPath has incorrect javadoc where the javadoc specifies a
> parameter to the function however the function doesnt expects one.
> {code}
> /**
>* Get a temp path. This may or may not be relative; it depends on what the
>* {@link #SYSPROP_TEST_DATA_DIR} is set to. If unset, it returns a path
>* under the relative path {@link #DEFAULT_TEST_DATA_PATH}
>* @param subpath sub path, with no leading "/" character
>* @return a string to use in paths
>*/
> public static String getRandomizedTempPath() {
> return getTempPath(RandomStringUtils.randomAlphanumeric(10));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14089) Shaded Hadoop client runtime includes non-shaded classes
[
https://issues.apache.org/jira/browse/HADOOP-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HADOOP-14089:
-
Status: In Progress (was: Patch Available)
> Shaded Hadoop client runtime includes non-shaded classes
>
>
> Key: HADOOP-14089
> URL: https://issues.apache.org/jira/browse/HADOOP-14089
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha2
>Reporter: David Phillips
>Assignee: Sean Busbey
>Priority: Critical
> Attachments: HADOOP-14089.WIP.0.patch
>
>
> The jar includes things like {{assets}}, {{okio}}, {{javax/annotation}},
> {{javax/ws}}, {{mozilla}}, etc.
> An easy way to verify this is to look at the contents of the jar:
> {code}
> jar tf hadoop-client-runtime-xxx.jar | sort | grep -v '^org/apache/hadoop'
> {code}
> For standard dependencies, such as the JSR 305 {{javax.annotation}} or JAX-RS
> {{javax.ws}}, it makes sense for those to be normal dependencies in the POM
> -- they are standard, so version conflicts shouldn't be a problem. The JSR
> 305 annotations can be {{true}} since they aren't needed
> at runtime (this is what Guava does).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13919) add integration tests for shaded client based on use by Spark
[ https://issues.apache.org/jira/browse/HADOOP-13919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reassigned HADOOP-13919: Assignee: (was: Sean Busbey) > add integration tests for shaded client based on use by Spark > - > > Key: HADOOP-13919 > URL: https://issues.apache.org/jira/browse/HADOOP-13919 > Project: Hadoop Common > Issue Type: Improvement > Components: test >Affects Versions: 3.0.0-alpha1 >Reporter: Sean Busbey >Priority: Minor > > Look at the tests that Spark runs against the Hadoop Minicluster and make > sure that functionality is tested in our integration tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13919) add integration tests for shaded client based on use by Spark
[ https://issues.apache.org/jira/browse/HADOOP-13919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HADOOP-13919: - Issue Type: Improvement (was: Sub-task) Parent: (was: HADOOP-11656) > add integration tests for shaded client based on use by Spark > - > > Key: HADOOP-13919 > URL: https://issues.apache.org/jira/browse/HADOOP-13919 > Project: Hadoop Common > Issue Type: Improvement > Components: test >Affects Versions: 3.0.0-alpha1 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Minor > > Look at the tests that Spark runs against the Hadoop Minicluster and make > sure that functionality is tested in our integration tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14820) Wasb mkdirs security checks inconsistent with HDFS
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14820:
Status: Patch Available (was: Open)
> Wasb mkdirs security checks inconsistent with HDFS
> --
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch,
> HADOOP-14820.003.patch, HADOOP-14820.004.patch, HADOOP-14820.005.patch,
> HADOOP-14820-006.patch, HADOOP-14820-007.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14731) Update gitignore to exclude output of site build
[ https://issues.apache.org/jira/browse/HADOOP-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150905#comment-16150905 ] Andrew Wang commented on HADOOP-14731: -- The site build spews files across the src/site directories, which I agree is not optimal. We really should figure out how to get them from target, but like I said, this is a quick fix. > Update gitignore to exclude output of site build > > > Key: HADOOP-14731 > URL: https://issues.apache.org/jira/browse/HADOOP-14731 > Project: Hadoop Common > Issue Type: Improvement > Components: build, site >Affects Versions: 3.0.0-alpha3 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HADOOP-14731.001.patch > > > Site build generates a bunch of files that aren't caught by gitignore, let's > update. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14826) reviewS3 docs prior to 3.0-beta-1
Steve Loughran created HADOOP-14826: --- Summary: reviewS3 docs prior to 3.0-beta-1 Key: HADOOP-14826 URL: https://issues.apache.org/jira/browse/HADOOP-14826 Project: Hadoop Common Issue Type: Sub-task Components: documentation, fs/s3 Affects Versions: 3.0.0-beta1 Reporter: Steve Loughran The hadoop-aws docs need a review and update * things marked as stabilizing (fast upload, .fadvise ..) can be considered stable * move s3n docs off to the side * add a "how to move from s3n to s3a" para -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13916) Document how downstream clients should make use of the new shaded client artifacts
[ https://issues.apache.org/jira/browse/HADOOP-13916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150892#comment-16150892 ] Sean Busbey commented on HADOOP-13916: -- I'm not sure. Maybe worth a review of how different projects do this. My intuition says there won't be much overlap in project bits outside of poms since it's so dependent on the individual thing you're testing. But maybe BigTop would make sense, given that even the hbase one already is looking across HBase, Spark, and Hadoop. > Document how downstream clients should make use of the new shaded client > artifacts > -- > > Key: HADOOP-13916 > URL: https://issues.apache.org/jira/browse/HADOOP-13916 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Sean Busbey >Assignee: Sean Busbey > > provide a quickstart that walks through using the new shaded dependencies > with Maven to create a simple downstream project. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14781) Clarify that HADOOP_CONF_DIR shouldn't actually be set in hadoop-env.sh
[ https://issues.apache.org/jira/browse/HADOOP-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150893#comment-16150893 ] Andrew Wang commented on HADOOP-14781: -- Thanks Allen! +1 again after the fact for ultimate clarity. > Clarify that HADOOP_CONF_DIR shouldn't actually be set in hadoop-env.sh > --- > > Key: HADOOP-14781 > URL: https://issues.apache.org/jira/browse/HADOOP-14781 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, scripts >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14781.00.patch > > > We should be more explicit in the documentation in hadoop-env.sh that > HADOOP_CONF_DIR: > * shouldn't actually be set in this file > * is really intended for something "outside" of this file to set > * will break --config if the pointed to configs don't also set > HADOOP_CONF_DIR appropriately -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13204) Über-jira: S3a phase III: scale and tuning
[ https://issues.apache.org/jira/browse/HADOOP-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13204: Target Version/s: 2.9.0, 3.0.0-beta1 (was: 2.9.0) > Über-jira: S3a phase III: scale and tuning > -- > > Key: HADOOP-13204 > URL: https://issues.apache.org/jira/browse/HADOOP-13204 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > S3A Phase III work; post 2.8. > Areas could include > * customisation > * performance enhancement > * management and metrics > * error/failure handling > And of course any bugs which surface -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13916) Document how downstream clients should make use of the new shaded client artifacts
[
https://issues.apache.org/jira/browse/HADOOP-13916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150883#comment-16150883
]
Chris Douglas commented on HADOOP-13916:
Thanks for the pointer to Stack's repo; I get it, now. Requested
{{hadoop-downstream-tests}}.
Does this overlap with Bigtop or Yetus? I mean, if more projects create similar
repositories, we'll likely end up sharing (or at least copy/pasting) similar
machinery.
> Document how downstream clients should make use of the new shaded client
> artifacts
> --
>
> Key: HADOOP-13916
> URL: https://issues.apache.org/jira/browse/HADOOP-13916
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>
> provide a quickstart that walks through using the new shaded dependencies
> with Maven to create a simple downstream project.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150884#comment-16150884 ] Andrew Wang commented on HADOOP-13345: -- Great to see this merged :) Steve, could you file a JIRA for the doc enhancements and raise it as a blocker for beta1? I can do a prompt review of whatever gets posted there. > S3Guard: Improved Consistency for S3A > - > > Key: HADOOP-13345 > URL: https://issues.apache.org/jira/browse/HADOOP-13345 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-13345.prototype1.patch, s3c.001.patch, > S3C-ConsistentListingonS3-Design.pdf, S3GuardImprovedConsistencyforS3A.pdf, > S3GuardImprovedConsistencyforS3AV2.pdf > > > This issue proposes S3Guard, a new feature of S3A, to provide an option for a > stronger consistency model than what is currently offered. The solution > coordinates with a strongly consistent external store to resolve > inconsistencies caused by the S3 eventual consistency model. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14600:
Status: Patch Available (was: Open)
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
>
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
>Reporter: Steve Loughran
>Assignee: Ping Liu
> Attachments: HADOOP-14600.001.patch,
> TestRawLocalFileSystemContract.java
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150822#comment-16150822
]
Ping Liu commented on HADOOP-14600:
---
Hi [~jzhuge], thanks for your help!
The patch and the test file are now attached.
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
>
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
>Reporter: Steve Loughran
>Assignee: Ping Liu
> Attachments: HADOOP-14600.001.patch,
> TestRawLocalFileSystemContract.java
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ping Liu updated HADOOP-14600:
--
Attachment: HADOOP-14600.001.patch
TestRawLocalFileSystemContract.java
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
>
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
>Reporter: Steve Loughran
>Assignee: Ping Liu
> Attachments: HADOOP-14600.001.patch,
> TestRawLocalFileSystemContract.java
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150811#comment-16150811
]
John Zhuge edited comment on HADOOP-14600 at 9/1/17 4:45 PM:
-
Please follow the guidelines in
https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch. The patch
file name should be: *HADOOP-14600.NNN.patch*, e.g., *HADOOP-14600.001.patch*.
was (Author: jzhuge):
Please follow the guidelines in
https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch. The patch
file name should be: *HADOOP-14600.001.patch*.
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
>
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
>Reporter: Steve Loughran
>Assignee: Ping Liu
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150811#comment-16150811
]
John Zhuge commented on HADOOP-14600:
-
Please follow the guidelines in
https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch. The patch
file name should be: *HADOOP-14600.001.patch*.
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
>
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
>Reporter: Steve Loughran
>Assignee: Ping Liu
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150810#comment-16150810
]
John Zhuge commented on HADOOP-14600:
-
Hi [~myapachejira], added you to the contributor list and assigned the jira to
you. Thank you for working on the issue.
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
>
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
>Reporter: Steve Loughran
>Assignee: Ping Liu
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge reassigned HADOOP-14600:
---
Assignee: Ping Liu
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
>
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
>Reporter: Steve Loughran
>Assignee: Ping Liu
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14824) Update ADLS SDK to 2.2.2 for MSI fix
[ https://issues.apache.org/jira/browse/HADOOP-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150794#comment-16150794 ] Atul Sikaria commented on HADOOP-14824: --- The change is in MSI (one of the auth mechanisms), not in ADLS - this patch just ensures that users of MSI don't break. Also, this impacts only people who use MSI as the auth mechanism. Given MSI is very recent (GA is later this month), we don't expect there to be many users. > Update ADLS SDK to 2.2.2 for MSI fix > > > Key: HADOOP-14824 > URL: https://issues.apache.org/jira/browse/HADOOP-14824 > Project: Hadoop Common > Issue Type: Bug > Components: fs/adl >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Fix For: 2.9.0, 3.0.0-beta1, 2.8.3 > > Attachments: HADOOP-14824.001.patch > > > Recently there was a change to MSI-based authentication in Azure VMs: the > auth request to localhost now requires a new header (Metadata) with a fixed > value (true). Existing clients that do not send this header will fail to get > token. > To address this, the datalake SDK was updated in version 2.2.2 to send this > header in the token request. > This patch is to use the updated sdk in Hadoop. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14824) Update ADLS SDK to 2.2.2 for MSI fix
[ https://issues.apache.org/jira/browse/HADOOP-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150767#comment-16150767 ] Steve Loughran edited comment on HADOOP-14824 at 9/1/17 3:59 PM: - Presumably this auth change in ADL means everyone using any release of hadoop with ADL is going to find their code stops working without the library update? was (Author: ste...@apache.org): Presumably this auth change means everyone using any release of hadoop with ADL is going to find their code stops working > Update ADLS SDK to 2.2.2 for MSI fix > > > Key: HADOOP-14824 > URL: https://issues.apache.org/jira/browse/HADOOP-14824 > Project: Hadoop Common > Issue Type: Bug > Components: fs/adl >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Fix For: 2.9.0, 3.0.0-beta1, 2.8.3 > > Attachments: HADOOP-14824.001.patch > > > Recently there was a change to MSI-based authentication in Azure VMs: the > auth request to localhost now requires a new header (Metadata) with a fixed > value (true). Existing clients that do not send this header will fail to get > token. > To address this, the datalake SDK was updated in version 2.2.2 to send this > header in the token request. > This patch is to use the updated sdk in Hadoop. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14824) Update ADLS SDK to 2.2.2 for MSI fix
[ https://issues.apache.org/jira/browse/HADOOP-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150767#comment-16150767 ] Steve Loughran commented on HADOOP-14824: - Presumably this auth change means everyone using any release of hadoop with ADL is going to find their code stops working > Update ADLS SDK to 2.2.2 for MSI fix > > > Key: HADOOP-14824 > URL: https://issues.apache.org/jira/browse/HADOOP-14824 > Project: Hadoop Common > Issue Type: Bug > Components: fs/adl >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Fix For: 2.9.0, 3.0.0-beta1, 2.8.3 > > Attachments: HADOOP-14824.001.patch > > > Recently there was a change to MSI-based authentication in Azure VMs: the > auth request to localhost now requires a new header (Metadata) with a fixed > value (true). Existing clients that do not send this header will fail to get > token. > To address this, the datalake SDK was updated in version 2.2.2 to send this > header in the token request. > This patch is to use the updated sdk in Hadoop. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14414) Calling maven-site-plugin directly for docs profile is unnecessary
[
https://issues.apache.org/jira/browse/HADOOP-14414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150759#comment-16150759
]
Hudson commented on HADOOP-14414:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12299 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12299/])
HADOOP-14414. Calling maven-site-plugin directly for docs profile is (aw: rev
a3fee475f72c8548aafd8574da1d3f1745f0471d)
* (edit) hadoop-tools/hadoop-sls/pom.xml
* (edit) hadoop-common-project/hadoop-auth/pom.xml
* (edit) hadoop-common-project/hadoop-kms/pom.xml
* (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
> Calling maven-site-plugin directly for docs profile is unnecessary
> --
>
> Key: HADOOP-14414
> URL: https://issues.apache.org/jira/browse/HADOOP-14414
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14414.01.patch, HADOOP-14414.02.patch,
> missingFiles
>
>
> For a few modules:
> * hadoop-auth
> * hadoop-kms
> * hadoop-hdfs-httpfs
> * hadoop-sls
> we call {{mave-site-plugin}} directly when docs profile is active.
> In main pom we use {{excludeDefaults}} in reporting section and allow only
> javadoc and dependency-plugin for the report. Since javadoc plugin is set to
> {{inherited}} false it won't be called on individual child modules. So
> actually {{maven-dependency-plugin:analyze-report}} is the only additional
> goal which will run.
> I debugged the process with {{mvn clean package -DskipTests
> -Dmaven.javadoc.skip=true -DskipShade -Pdocs -X}} command and in all the 4
> affected modules I found the following configuration for site plugin:
> {code}
>
> org.apache.maven.plugins
> maven-dependency-plugin
> 2.10
>
>
> default
>
> analyze-report
>
>
>
>
> {code}
> At this point I do not see the purpose of calling {{mave-site-plugin}} for
> docs profile. It does not contain useful information. Or if it does why don't
> we call for other modules? It's inconsistent.
> Considering to remove.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12809) Move to tomcat 8.0.21 to support different passwords for key and keystore on HTTPFS/KMS using SSL
[ https://issues.apache.org/jira/browse/HADOOP-12809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-12809: -- Resolution: Not A Problem Status: Resolved (was: Patch Available) Tomcat is no longer present in hadoop trunk. Closing this as "Not a Problem" given 3.x will remove tomcat completely. > Move to tomcat 8.0.21 to support different passwords for key and keystore on > HTTPFS/KMS using SSL > - > > Key: HADOOP-12809 > URL: https://issues.apache.org/jira/browse/HADOOP-12809 > Project: Hadoop Common > Issue Type: Improvement > Components: conf, kms, scripts >Affects Versions: 2.7.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-8509.patch > > > Currently, SSL for HTTPFS requires that keystore/truststore and key passwords > be the same. This is a limitation from Tomcat version 6, which didn't have > support for different passwords. From Tomcat 7, this is now possible by > defining "keyPass" property for "Connector" configuration on Tomcat's > server.xml file. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14414) Calling maven-site-plugin directly for docs profile is unnecessary
[
https://issues.apache.org/jira/browse/HADOOP-14414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-14414:
--
Resolution: Fixed
Fix Version/s: 3.0.0-beta1
Status: Resolved (was: Patch Available)
+1 committing to trunk.
Thanks for the contribution!
> Calling maven-site-plugin directly for docs profile is unnecessary
> --
>
> Key: HADOOP-14414
> URL: https://issues.apache.org/jira/browse/HADOOP-14414
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14414.01.patch, HADOOP-14414.02.patch,
> missingFiles
>
>
> For a few modules:
> * hadoop-auth
> * hadoop-kms
> * hadoop-hdfs-httpfs
> * hadoop-sls
> we call {{mave-site-plugin}} directly when docs profile is active.
> In main pom we use {{excludeDefaults}} in reporting section and allow only
> javadoc and dependency-plugin for the report. Since javadoc plugin is set to
> {{inherited}} false it won't be called on individual child modules. So
> actually {{maven-dependency-plugin:analyze-report}} is the only additional
> goal which will run.
> I debugged the process with {{mvn clean package -DskipTests
> -Dmaven.javadoc.skip=true -DskipShade -Pdocs -X}} command and in all the 4
> affected modules I found the following configuration for site plugin:
> {code}
>
> org.apache.maven.plugins
> maven-dependency-plugin
> 2.10
>
>
> default
>
> analyze-report
>
>
>
>
> {code}
> At this point I do not see the purpose of calling {{mave-site-plugin}} for
> docs profile. It does not contain useful information. Or if it does why don't
> we call for other modules? It's inconsistent.
> Considering to remove.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11880) Update test-patch to leverage rewrite in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11880: -- Resolution: Not A Problem Status: Resolved (was: Patch Available) > Update test-patch to leverage rewrite in Hadoop > --- > > Key: HADOOP-11880 > URL: https://issues.apache.org/jira/browse/HADOOP-11880 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Allen Wittenauer > Attachments: HDFS-9263-001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14824) Update ADLS SDK to 2.2.2 for MSI fix
[ https://issues.apache.org/jira/browse/HADOOP-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150693#comment-16150693 ] John Zhuge commented on HADOOP-14824: - [~steve_l] All live unit tests passed in 3 branches committed. Sorry for the rush. The patch is to handle an ADLS upgrade on Thursday which would break MSI feature. See detailed SDK change in https://github.com/Azure/azure-data-lake-store-java/commit/81f0016e91d6e9e6bdf35e08917b8aa1895f3389. It should only have limited impact. > Update ADLS SDK to 2.2.2 for MSI fix > > > Key: HADOOP-14824 > URL: https://issues.apache.org/jira/browse/HADOOP-14824 > Project: Hadoop Common > Issue Type: Bug > Components: fs/adl >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Fix For: 2.9.0, 3.0.0-beta1, 2.8.3 > > Attachments: HADOOP-14824.001.patch > > > Recently there was a change to MSI-based authentication in Azure VMs: the > auth request to localhost now requires a new header (Metadata) with a fixed > value (true). Existing clients that do not send this header will fail to get > token. > To address this, the datalake SDK was updated in version 2.2.2 to send this > header in the token request. > This patch is to use the updated sdk in Hadoop. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13345) S3Guard: Improved Consistency for S3A
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-13345. - Resolution: Fixed Fix Version/s: 3.0.0-beta1 Closing as fixed. I've added some release notes. Andrew: no bit in the site docs though, other than the bits off the s3 docs. Which, having just read, need a rework anyway, as things which are marked as "stabilising" can now be considered stable > S3Guard: Improved Consistency for S3A > - > > Key: HADOOP-13345 > URL: https://issues.apache.org/jira/browse/HADOOP-13345 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-13345.prototype1.patch, s3c.001.patch, > S3C-ConsistentListingonS3-Design.pdf, S3GuardImprovedConsistencyforS3A.pdf, > S3GuardImprovedConsistencyforS3AV2.pdf > > > This issue proposes S3Guard, a new feature of S3A, to provide an option for a > stronger consistency model than what is currently offered. The solution > coordinates with a strongly consistent external store to resolve > inconsistencies caused by the S3 eventual consistency model. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13345) S3Guard: Improved Consistency for S3A
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13345: Affects Version/s: 2.8.1 Release Note: S3Guard (pronounced see-guard) is a new feature for the S3A connector to Amazon S3, which uses DynamoDB for a high performance and consistent metadata repository. Essentially: S3Guard caches directory information, so your S3A clients get faster lookups and resilience to inconsistency between S3 list operations and the status of objects. When files are created, with S3Guard, they'll always be found. S3Guard does not address update consistency: if a file is updated, while the directory information will be updated, calling open() on the path may still return the old data. Similarly, deleted objects may also potentially be opened. Please consult the S3Guard documentation in the Amazon S3 section of our documentation. Note: part of this update includes moving to a new version of the AWS SDK 1.11, one which includes the Dynamo DB client and its a shaded version of Jackson 2. The large aws-sdk-bundle JAR is needed to use the S3A client with or without S3Guard enabled. The good news: because Jackson is shaded, there will be no conflict between any Jackson version used in your application and that which the AWS SDK needs. > S3Guard: Improved Consistency for S3A > - > > Key: HADOOP-13345 > URL: https://issues.apache.org/jira/browse/HADOOP-13345 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-13345.prototype1.patch, s3c.001.patch, > S3C-ConsistentListingonS3-Design.pdf, S3GuardImprovedConsistencyforS3A.pdf, > S3GuardImprovedConsistencyforS3AV2.pdf > > > This issue proposes S3Guard, a new feature of S3A, to provide an option for a > stronger consistency model than what is currently offered. The solution > coordinates with a strongly consistent external store to resolve > inconsistencies caused by the S3 eventual consistency model. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14759) S3GuardTool prune to prune specific bucket entries
[
https://issues.apache.org/jira/browse/HADOOP-14759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14759:
Parent Issue: HADOOP-14825 (was: HADOOP-13345)
> S3GuardTool prune to prune specific bucket entries
> --
>
> Key: HADOOP-14759
> URL: https://issues.apache.org/jira/browse/HADOOP-14759
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Priority: Minor
>
> Users may think that when you provide a URI to a bucket, you are pruning all
> entries in the table *for that bucket*. In fact you are purging all entries
> across all buckets in the table:
> {code}
> hadoop s3guard prune -days 7 s3a://ireland-1
> {code}
> It should be restricted to that bucket, unless you specify otherwise
> +maybe also add a hard date rather than a relative one
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14800) eliminate double stack trace on some s3guard CLI failures
[
https://issues.apache.org/jira/browse/HADOOP-14800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-14800.
-
Resolution: Duplicate
> eliminate double stack trace on some s3guard CLI failures
> -
>
> Key: HADOOP-14800
> URL: https://issues.apache.org/jira/browse/HADOOP-14800
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Priority: Minor
> Fix For: 3.1.0
>
>
> {{s3guard destroy}] when there's no bucket ends up double-listing the stack
> trace, which is somewhat confusing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14800) eliminate double stack trace on some s3guard CLI failures
[
https://issues.apache.org/jira/browse/HADOOP-14800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14800:
Parent Issue: HADOOP-14825 (was: HADOOP-13345)
> eliminate double stack trace on some s3guard CLI failures
> -
>
> Key: HADOOP-14800
> URL: https://issues.apache.org/jira/browse/HADOOP-14800
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Priority: Minor
> Fix For: 3.1.0
>
>
> {{s3guard destroy}] when there's no bucket ends up double-listing the stack
> trace, which is somewhat confusing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14585) Ensure controls in-place to prevent clients with significant clock skews pruning aggressively
[
https://issues.apache.org/jira/browse/HADOOP-14585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14585:
Priority: Minor (was: Major)
> Ensure controls in-place to prevent clients with significant clock skews
> pruning aggressively
> -
>
> Key: HADOOP-14585
> URL: https://issues.apache.org/jira/browse/HADOOP-14585
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Sean Mackrory
>Priority: Minor
> Fix For: 3.1.0
>
>
> From discussion on HADOOP-14499:
> {quote}
> bear in mind that we can't guarantee that the clocks of all clients are in
> sync; you don't want a client whose TZ setting is wrong to aggressively prune
> things. Had that happen in production with files in shared filestore. This is
> why ant -diagnostics checks time consistency with temp files...
> {quote}
> {quote}
> temp files work on a shared FS. AWS is actually somewhat sensitive to clocks:
> if your VM is too far out of time then auth actually fails, its ~+-15
> minutes. There's some stuff in the Java SDK to actually calculate and adjust
> clock skew, presumably parsing the timestamp of a failure, calculating the
> difference and retrying. Which means that the field in SDKGlobalConfiguration
> could help identify the difference between local time and AWS time.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14585) Ensure controls in-place to prevent clients with significant clock skews pruning aggressively
[
https://issues.apache.org/jira/browse/HADOOP-14585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14585:
Parent Issue: HADOOP-14825 (was: HADOOP-13345)
> Ensure controls in-place to prevent clients with significant clock skews
> pruning aggressively
> -
>
> Key: HADOOP-14585
> URL: https://issues.apache.org/jira/browse/HADOOP-14585
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Sean Mackrory
> Fix For: 3.1.0
>
>
> From discussion on HADOOP-14499:
> {quote}
> bear in mind that we can't guarantee that the clocks of all clients are in
> sync; you don't want a client whose TZ setting is wrong to aggressively prune
> things. Had that happen in production with files in shared filestore. This is
> why ant -diagnostics checks time consistency with temp files...
> {quote}
> {quote}
> temp files work on a shared FS. AWS is actually somewhat sensitive to clocks:
> if your VM is too far out of time then auth actually fails, its ~+-15
> minutes. There's some stuff in the Java SDK to actually calculate and adjust
> clock skew, presumably parsing the timestamp of a failure, calculating the
> difference and retrying. Which means that the field in SDKGlobalConfiguration
> could help identify the difference between local time and AWS time.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14757) S3AFileSystem.innerRename() to size metadatastore lists better
[
https://issues.apache.org/jira/browse/HADOOP-14757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14757:
Parent Issue: HADOOP-14825 (was: HADOOP-13345)
> S3AFileSystem.innerRename() to size metadatastore lists better
> --
>
> Key: HADOOP-14757
> URL: https://issues.apache.org/jira/browse/HADOOP-14757
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Priority: Minor
> Fix For: 3.1.0
>
>
> In {{S3AFileSystem.innerRename()}}, various ArrayLists are created to track
> paths to update; these are created with the default size. It could/should be
> possible to allocate better, so avoid expensive array growth & copy
> operations while iterating through the list of entries.
> # for a single file copy, sizes == 1
> # for a recursive copy, the outcome of the first real LIST will either
> provide the actual size, or, if the list == the max response, a very large
> minimum size.
> For #2, we'd need to get the hint of iterable length rather than just iterate
> through...some interface {{{IterableLength.expectedMinimumSize()}} could do
> that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14750) s3guard to provide better diags on ddb init failures
[ https://issues.apache.org/jira/browse/HADOOP-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14750: Parent Issue: HADOOP-14825 (was: HADOOP-13345) > s3guard to provide better diags on ddb init failures > > > Key: HADOOP-14750 > URL: https://issues.apache.org/jira/browse/HADOOP-14750 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Priority: Minor > > When you can't connect to DDB you get an Http exception; it'd be good to > include more info here (table name & region in particular) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14758) S3GuardTool.prune to handle UnsupportedOperationException
[
https://issues.apache.org/jira/browse/HADOOP-14758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14758:
Parent Issue: HADOOP-14825 (was: HADOOP-13345)
> S3GuardTool.prune to handle UnsupportedOperationException
> -
>
> Key: HADOOP-14758
> URL: https://issues.apache.org/jira/browse/HADOOP-14758
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Priority: Trivial
>
> {{MetadataStore.prune()}} may throw {{UnsupportedOperationException}} if not
> supported.
> {{S3GuardTool.prune}} should recognise this, catch it and treat it
> differently from any other failure, e.g. inform and return 0 as its a no-op
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14734) add option to tag DDB table(s) created
[ https://issues.apache.org/jira/browse/HADOOP-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14734: Parent Issue: HADOOP-14825 (was: HADOOP-13345) > add option to tag DDB table(s) created > -- > > Key: HADOOP-14734 > URL: https://issues.apache.org/jira/browse/HADOOP-14734 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Priority: Minor > > Many organisations have a "no untagged" resource policy; s3guard runs into > this when a table is created untagged. If there's a strict "delete untagged > resources" policy, the tables will go without warning. > Proposed: we add an option which can be used to declare the tags for a table > when created, use it in creation. No need to worry about updating/viewing > tags, as the AWS console can do that -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14592) ITestS3ATemporaryCredentials to cover all ddb metastore ops with session credentials
[
https://issues.apache.org/jira/browse/HADOOP-14592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14592:
Parent Issue: HADOOP-14825 (was: HADOOP-13345)
> ITestS3ATemporaryCredentials to cover all ddb metastore ops with session
> credentials
>
>
> Key: HADOOP-14592
> URL: https://issues.apache.org/jira/browse/HADOOP-14592
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Priority: Minor
>
> {{ITestS3ATemporaryCredentials}} tests a couple of operations with temp
> credentials, but for completeness it should perform all operations which will
> access dynamo DB, so verifying that temporary credentials work everywhere.
> I know this is implicit from anyone running the tests in an EC2 container and
> picking up the IAM details, so I'm not expecting the tests to fail —but they
> should be there
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14468) S3Guard: make short-circuit getFileStatus() configurable
[
https://issues.apache.org/jira/browse/HADOOP-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14468:
Parent Issue: HADOOP-14825 (was: HADOOP-13345)
> S3Guard: make short-circuit getFileStatus() configurable
>
>
> Key: HADOOP-14468
> URL: https://issues.apache.org/jira/browse/HADOOP-14468
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
>Priority: Minor
>
> Currently, when S3Guard is enabled, getFileStatus() will skip S3 if it gets a
> result from the MetadataStore (e.g. dynamodb) first.
> I would like to add a new parameter
> {{fs.s3a.metadatastore.getfilestatus.authoritative}} which, when true, keeps
> the current behavior. When false, S3AFileSystem will check both S3 and the
> MetadataStore.
> I'm not sure yet if we want to have this behavior the same for all callers of
> getFileStatus(), or if we only want to check both S3 and MetadataStore for
> some internal callers such as open().
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14577) ITestS3AInconsistency.testGetFileStatus failing in -DS3guard test runs
[
https://issues.apache.org/jira/browse/HADOOP-14577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14577:
Parent Issue: HADOOP-14825 (was: HADOOP-13345)
> ITestS3AInconsistency.testGetFileStatus failing in -DS3guard test runs
> --
>
> Key: HADOOP-14577
> URL: https://issues.apache.org/jira/browse/HADOOP-14577
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.0.0-beta1
>Reporter: Sean Mackrory
>Priority: Minor
>
> This test is failing for me when run individually or in parallel (with
> -Ds3guard). Even if I revert back to the commit that introduced it. I thought
> I had successful test runs on that before and haven't changed anything in my
> test configuration.
> {code}Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.671
> sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AInconsistency
> testGetFileStatus(org.apache.hadoop.fs.s3a.ITestS3AInconsistency) Time
> elapsed: 4.475 sec <<< FAILURE!
> java.lang.AssertionError: S3Guard failed to list parent of inconsistent child.
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.fs.s3a.ITestS3AInconsistency.testGetFileStatus(ITestS3AInconsistency.java:83){code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14577) ITestS3AInconsistency.testGetFileStatus failing in -DS3guard test runs
[
https://issues.apache.org/jira/browse/HADOOP-14577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14577:
Component/s: test
> ITestS3AInconsistency.testGetFileStatus failing in -DS3guard test runs
> --
>
> Key: HADOOP-14577
> URL: https://issues.apache.org/jira/browse/HADOOP-14577
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.0.0-beta1
>Reporter: Sean Mackrory
>Priority: Minor
>
> This test is failing for me when run individually or in parallel (with
> -Ds3guard). Even if I revert back to the commit that introduced it. I thought
> I had successful test runs on that before and haven't changed anything in my
> test configuration.
> {code}Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.671
> sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AInconsistency
> testGetFileStatus(org.apache.hadoop.fs.s3a.ITestS3AInconsistency) Time
> elapsed: 4.475 sec <<< FAILURE!
> java.lang.AssertionError: S3Guard failed to list parent of inconsistent child.
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.fs.s3a.ITestS3AInconsistency.testGetFileStatus(ITestS3AInconsistency.java:83){code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14576) DynamoDB tables may leave ACTIVE state after initial connection
[ https://issues.apache.org/jira/browse/HADOOP-14576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150616#comment-16150616 ] Steve Loughran commented on HADOOP-14576: - DDB can -> updating after you change the capacity (as HADOOP-14220 will let you do). The retry handler needs to recognise the special case of update-in-progress and have a long retry for it > DynamoDB tables may leave ACTIVE state after initial connection > --- > > Key: HADOOP-14576 > URL: https://issues.apache.org/jira/browse/HADOOP-14576 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Sean Mackrory > > We currently only anticipate tables not being in the ACTIVE state when first > connecting. It is possible for a table to be in the ACTIVE state and move to > an UPDATING state during partitioning events. Attempts to read or write > during that time will result in an AmazonServerException getting thrown. We > should try to handle that better... -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14576) DynamoDB tables may leave ACTIVE state after initial connection
[ https://issues.apache.org/jira/browse/HADOOP-14576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14576: Parent Issue: HADOOP-14825 (was: HADOOP-13345) > DynamoDB tables may leave ACTIVE state after initial connection > --- > > Key: HADOOP-14576 > URL: https://issues.apache.org/jira/browse/HADOOP-14576 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Sean Mackrory > > We currently only anticipate tables not being in the ACTIVE state when first > connecting. It is possible for a table to be in the ACTIVE state and move to > an UPDATING state during partitioning events. Attempts to read or write > during that time will result in an AmazonServerException getting thrown. We > should try to handle that better... -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14335) Improve DynamoDB schema update story
[ https://issues.apache.org/jira/browse/HADOOP-14335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14335: Parent Issue: HADOOP-14825 (was: HADOOP-13345) > Improve DynamoDB schema update story > > > Key: HADOOP-14335 > URL: https://issues.apache.org/jira/browse/HADOOP-14335 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > > On HADOOP-13760 I'm realizing that changes to the DynamoDB schema aren't > great to deal with. Currently a build of Hadoop is hard-coded to a specific > schema version. So if you upgrade from one to the next you have to upgrade > everything (and then update the version in the table - which we don't have a > tool or document for) before you can keep using S3Guard. We could possibly > also make the definition of compatibility a bit more flexible, but it's going > to be very tough to do that without knowing what kind of future schema > changes we might want ahead of time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14425) Add more s3guard metrics
[ https://issues.apache.org/jira/browse/HADOOP-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14425: Parent Issue: HADOOP-14825 (was: HADOOP-14420) > Add more s3guard metrics > > > Key: HADOOP-14425 > URL: https://issues.apache.org/jira/browse/HADOOP-14425 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Ai Deng > > The metrics suggested to add: > Status: > S3GUARD_METADATASTORE_ENABLED > S3GUARD_METADATASTORE_IS_AUTHORITATIVE > Operations: > S3GUARD_METADATASTORE_INITIALIZATION > S3GUARD_METADATASTORE_DELETE_PATH > S3GUARD_METADATASTORE_DELETE_PATH_LATENCY > S3GUARD_METADATASTORE_DELETE_SUBTREE_PATCH > S3GUARD_METADATASTORE_GET_PATH > S3GUARD_METADATASTORE_GET_PATH_LATENCY > S3GUARD_METADATASTORE_GET_CHILDREN_PATH > S3GUARD_METADATASTORE_GET_CHILDREN_PATH_LATENCY > S3GUARD_METADATASTORE_MOVE_PATH > S3GUARD_METADATASTORE_PUT_PATH > S3GUARD_METADATASTORE_PUT_PATH_LATENCY > S3GUARD_METADATASTORE_CLOSE > S3GUARD_METADATASTORE_DESTORY > From S3Guard: > S3GUARD_METADATASTORE_MERGE_DIRECTORY > For the failures: > S3GUARD_METADATASTORE_DELETE_FAILURE > S3GUARD_METADATASTORE_GET_FAILURE > S3GUARD_METADATASTORE_PUT_FAILURE > Etc: > S3GUARD_METADATASTORE_PUT_RETRY_TIMES -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14425) Add more s3guard metrics
[ https://issues.apache.org/jira/browse/HADOOP-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14425: Affects Version/s: 3.1.0 > Add more s3guard metrics > > > Key: HADOOP-14425 > URL: https://issues.apache.org/jira/browse/HADOOP-14425 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Ai Deng > > The metrics suggested to add: > Status: > S3GUARD_METADATASTORE_ENABLED > S3GUARD_METADATASTORE_IS_AUTHORITATIVE > Operations: > S3GUARD_METADATASTORE_INITIALIZATION > S3GUARD_METADATASTORE_DELETE_PATH > S3GUARD_METADATASTORE_DELETE_PATH_LATENCY > S3GUARD_METADATASTORE_DELETE_SUBTREE_PATCH > S3GUARD_METADATASTORE_GET_PATH > S3GUARD_METADATASTORE_GET_PATH_LATENCY > S3GUARD_METADATASTORE_GET_CHILDREN_PATH > S3GUARD_METADATASTORE_GET_CHILDREN_PATH_LATENCY > S3GUARD_METADATASTORE_MOVE_PATH > S3GUARD_METADATASTORE_PUT_PATH > S3GUARD_METADATASTORE_PUT_PATH_LATENCY > S3GUARD_METADATASTORE_CLOSE > S3GUARD_METADATASTORE_DESTORY > From S3Guard: > S3GUARD_METADATASTORE_MERGE_DIRECTORY > For the failures: > S3GUARD_METADATASTORE_DELETE_FAILURE > S3GUARD_METADATASTORE_GET_FAILURE > S3GUARD_METADATASTORE_PUT_FAILURE > Etc: > S3GUARD_METADATASTORE_PUT_RETRY_TIMES -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14425) Add more s3guard metrics
[ https://issues.apache.org/jira/browse/HADOOP-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14425: Parent Issue: HADOOP-14420 (was: HADOOP-13345) > Add more s3guard metrics > > > Key: HADOOP-14425 > URL: https://issues.apache.org/jira/browse/HADOOP-14425 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Ai Deng > > The metrics suggested to add: > Status: > S3GUARD_METADATASTORE_ENABLED > S3GUARD_METADATASTORE_IS_AUTHORITATIVE > Operations: > S3GUARD_METADATASTORE_INITIALIZATION > S3GUARD_METADATASTORE_DELETE_PATH > S3GUARD_METADATASTORE_DELETE_PATH_LATENCY > S3GUARD_METADATASTORE_DELETE_SUBTREE_PATCH > S3GUARD_METADATASTORE_GET_PATH > S3GUARD_METADATASTORE_GET_PATH_LATENCY > S3GUARD_METADATASTORE_GET_CHILDREN_PATH > S3GUARD_METADATASTORE_GET_CHILDREN_PATH_LATENCY > S3GUARD_METADATASTORE_MOVE_PATH > S3GUARD_METADATASTORE_PUT_PATH > S3GUARD_METADATASTORE_PUT_PATH_LATENCY > S3GUARD_METADATASTORE_CLOSE > S3GUARD_METADATASTORE_DESTORY > From S3Guard: > S3GUARD_METADATASTORE_MERGE_DIRECTORY > For the failures: > S3GUARD_METADATASTORE_DELETE_FAILURE > S3GUARD_METADATASTORE_GET_FAILURE > S3GUARD_METADATASTORE_PUT_FAILURE > Etc: > S3GUARD_METADATASTORE_PUT_RETRY_TIMES -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14758) S3GuardTool.prune to handle UnsupportedOperationException
[
https://issues.apache.org/jira/browse/HADOOP-14758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14758:
Affects Version/s: 3.0.0-beta1
Priority: Trivial (was: Major)
> S3GuardTool.prune to handle UnsupportedOperationException
> -
>
> Key: HADOOP-14758
> URL: https://issues.apache.org/jira/browse/HADOOP-14758
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Priority: Trivial
>
> {{MetadataStore.prune()}} may throw {{UnsupportedOperationException}} if not
> supported.
> {{S3GuardTool.prune}} should recognise this, catch it and treat it
> differently from any other failure, e.g. inform and return 0 as its a no-op
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14758) S3GuardTool.prune to handle UnsupportedOperationException
[
https://issues.apache.org/jira/browse/HADOOP-14758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150605#comment-16150605
]
Steve Loughran commented on HADOOP-14758:
-
trivial; can do this in HADOOP-14420
> S3GuardTool.prune to handle UnsupportedOperationException
> -
>
> Key: HADOOP-14758
> URL: https://issues.apache.org/jira/browse/HADOOP-14758
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>
> {{MetadataStore.prune()}} may throw {{UnsupportedOperationException}} if not
> supported.
> {{S3GuardTool.prune}} should recognise this, catch it and treat it
> differently from any other failure, e.g. inform and return 0 as its a no-op
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14220: Parent Issue: HADOOP-14825 (was: HADOOP-13345) > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14220-HADOOP-13345-001.patch, > HADOOP-14220-HADOOP-13345-002.patch, HADOOP-14220-HADOOP-13345-003.patch, > HADOOP-14220-HADOOP-13345-004.patch, HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14158) Possible for modified configuration to leak into metadatastore in S3GuardTool
[ https://issues.apache.org/jira/browse/HADOOP-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14158: Affects Version/s: 3.0.0-beta1 Priority: Minor (was: Major) > Possible for modified configuration to leak into metadatastore in S3GuardTool > - > > Key: HADOOP-14158 > URL: https://issues.apache.org/jira/browse/HADOOP-14158 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Sean Mackrory >Priority: Minor > > It doesn't appear to do it when run from the command-line, but when running > the S3GuardTool.run (i.e. the parent function of most of the functions used > in the unit tests) from a unit test, you end up with a NullMetadataStore, > regardless of what else was configured. > We create an instance of S3AFileSystem with the metadata store implementation > overridden to NullMetadataStore so that we have distinct interfaces to S3 and > the metadata store. S3Guard can later be called using this filesystem, > causing it to pick up the filesystem's configuration, which instructs it to > use the NullMetadataStore implementation. This shouldn't be possible. > It is unknown if this happens in any real-world scenario - I've been unable > to reproduce the problem from the command-line. But it definitely happens in > a test, it shouldn't, and fixing this will at least allow HADOOP-14145 to > have an automated test. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14158) Possible for modified configuration to leak into metadatastore in S3GuardTool
[ https://issues.apache.org/jira/browse/HADOOP-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14158: Parent Issue: HADOOP-14825 (was: HADOOP-13345) > Possible for modified configuration to leak into metadatastore in S3GuardTool > - > > Key: HADOOP-14158 > URL: https://issues.apache.org/jira/browse/HADOOP-14158 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Sean Mackrory >Priority: Minor > > It doesn't appear to do it when run from the command-line, but when running > the S3GuardTool.run (i.e. the parent function of most of the functions used > in the unit tests) from a unit test, you end up with a NullMetadataStore, > regardless of what else was configured. > We create an instance of S3AFileSystem with the metadata store implementation > overridden to NullMetadataStore so that we have distinct interfaces to S3 and > the metadata store. S3Guard can later be called using this filesystem, > causing it to pick up the filesystem's configuration, which instructs it to > use the NullMetadataStore implementation. This shouldn't be possible. > It is unknown if this happens in any real-world scenario - I've been unable > to reproduce the problem from the command-line. But it definitely happens in > a test, it shouldn't, and fixing this will at least allow HADOOP-14145 to > have an automated test. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14154) Set isAuthoritative flag when creating DirListingMetadata in DynamoDBMetaStore
[
https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14154:
Parent Issue: HADOOP-14825 (was: HADOOP-13345)
> Set isAuthoritative flag when creating DirListingMetadata in DynamoDBMetaStore
> --
>
> Key: HADOOP-14154
> URL: https://issues.apache.org/jira/browse/HADOOP-14154
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HADOOP-14154-HADOOP-13345.001.patch,
> HADOOP-14154-HADOOP-13345.002.patch
>
>
> Currently {{DynamoDBMetaStore::listChildren}} does not populate
> {{isAuthoritative}} flag when creating {{DirListingMetadata}}.
> This causes additional S3 lookups even when users have enabled
> {{fs.s3a.metadatastore.authoritative}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
