[jira] [Commented] (HADOOP-16055) Upgrade AWS SDK to fix license issue in branch-2.8 and branch-2.7
[ https://issues.apache.org/jira/browse/HADOOP-16055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749633#comment-16749633 ] Akira Ajisaka commented on HADOOP-16055: JUnit tests passed except ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRecursiveRootListing and ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive. > Upgrade AWS SDK to fix license issue in branch-2.8 and branch-2.7 > - > > Key: HADOOP-16055 > URL: https://issues.apache.org/jira/browse/HADOOP-16055 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Blocker > Attachments: HADOOP-16055-branch-2-01.patch, > HADOOP-16055-branch-2.8-01.patch, HADOOP-16055-branch-2.8-02.patch > > > Per HADOOP-13794, we must exclude the JSON license. > The upgrade will contain incompatible changes, however, the license issue is > much more important. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16055) Upgrade AWS SDK to fix license issue in branch-2.8 and branch-2.7
[
https://issues.apache.org/jira/browse/HADOOP-16055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749634#comment-16749634
]
Akira Ajisaka commented on HADOOP-16055:
Ran the following command
{noformat}
$ mvn -Dparallel-tests -DtestsThreadCount=8 clean verify -Pscale -Ddynamo
-Ds3guard
$ mvn -Dparallel-tests -DtestsThreadCount=8 clean verify -Pscale
{noformat}
with setting fs.s3a.server-side-encryption.key to my KMS key.
> Upgrade AWS SDK to fix license issue in branch-2.8 and branch-2.7
> -
>
> Key: HADOOP-16055
> URL: https://issues.apache.org/jira/browse/HADOOP-16055
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Blocker
> Attachments: HADOOP-16055-branch-2-01.patch,
> HADOOP-16055-branch-2.8-01.patch, HADOOP-16055-branch-2.8-02.patch
>
>
> Per HADOOP-13794, we must exclude the JSON license.
> The upgrade will contain incompatible changes, however, the license issue is
> much more important.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16065) -Ddynamodb should be -Ddynamo in AWS SDK testing document
[
https://issues.apache.org/jira/browse/HADOOP-16065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749613#comment-16749613
]
Hadoop QA commented on HADOOP-16065:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 20m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
34m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16065 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955906/HADOOP-16065.01.patch
|
| Optional Tests | dupname asflicense mvnsite |
| uname | Linux 9831bae7465c 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e3e076d |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 339 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15827/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> -Ddynamodb should be -Ddynamo in AWS SDK testing document
> -
>
> Key: HADOOP-16065
> URL: https://issues.apache.org/jira/browse/HADOOP-16065
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-16065.01.patch
>
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
> {{-Ddynamodb}} should be {{-Ddynamo}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16065) -Ddynamodb should be -Ddynamo in AWS SDK testing document
[
https://issues.apache.org/jira/browse/HADOOP-16065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749623#comment-16749623
]
Akira Ajisaka commented on HADOOP-16065:
Hi [~ste...@apache.org], would you review this?
> -Ddynamodb should be -Ddynamo in AWS SDK testing document
> -
>
> Key: HADOOP-16065
> URL: https://issues.apache.org/jira/browse/HADOOP-16065
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-16065.01.patch
>
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
> {{-Ddynamodb}} should be {{-Ddynamo}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] aajisaka closed pull request #139: YARN-5721. NPE at AMRMClientImpl.getMatchingRequests
aajisaka closed pull request #139: YARN-5721. NPE at AMRMClientImpl.getMatchingRequests URL: https://github.com/apache/hadoop/pull/139 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] aajisaka commented on issue #139: YARN-5721. NPE at AMRMClientImpl.getMatchingRequests
aajisaka commented on issue #139: YARN-5721. NPE at AMRMClientImpl.getMatchingRequests URL: https://github.com/apache/hadoop/pull/139#issuecomment-456699050 Duplicated and fixed by https://issues.apache.org/jira/browse/YARN-5753. Closing This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] aajisaka commented on issue #39: HADOOP-12527. Upgrade Avro dependency to 1.7.7.
aajisaka commented on issue #39: HADOOP-12527. Upgrade Avro dependency to 1.7.7. URL: https://github.com/apache/hadoop/pull/39#issuecomment-456698718 Duplicated and fixed by https://issues.apache.org/jira/browse/HADOOP-14992. Closing. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] aajisaka closed pull request #39: HADOOP-12527. Upgrade Avro dependency to 1.7.7.
aajisaka closed pull request #39: HADOOP-12527. Upgrade Avro dependency to 1.7.7. URL: https://github.com/apache/hadoop/pull/39 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] aajisaka closed pull request #84: HADOOP-12927. Update netty-all to 4.0.34.Final
aajisaka closed pull request #84: HADOOP-12927. Update netty-all to 4.0.34.Final URL: https://github.com/apache/hadoop/pull/84 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] aajisaka commented on issue #84: HADOOP-12927. Update netty-all to 4.0.34.Final
aajisaka commented on issue #84: HADOOP-12927. Update netty-all to 4.0.34.Final URL: https://github.com/apache/hadoop/pull/84#issuecomment-456698265 Duplicated and fixed by HADOOP-13866. Closing. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-11219) Upgrade to netty 4
[ https://issues.apache.org/jira/browse/HADOOP-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reopened HADOOP-11219: Assignee: Haohui Mai Sorry, some modules are still using Netty 3. Reopening. > Upgrade to netty 4 > -- > > Key: HADOOP-11219 > URL: https://issues.apache.org/jira/browse/HADOOP-11219 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Major > > This is an umbrella jira to track the effort of upgrading to Netty 4. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-11219) Upgrade to netty 4
[ https://issues.apache.org/jira/browse/HADOOP-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HADOOP-11219. Resolution: Duplicate Assignee: (was: Haohui Mai) Closing this as duplicate. > Upgrade to netty 4 > -- > > Key: HADOOP-11219 > URL: https://issues.apache.org/jira/browse/HADOOP-11219 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Haohui Mai >Priority: Major > > This is an umbrella jira to track the effort of upgrading to Netty 4. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] aajisaka commented on issue #127: Branch 2.7.1
aajisaka commented on issue #127: Branch 2.7.1 URL: https://github.com/apache/hadoop/pull/127#issuecomment-456694290 There is no need to merge commits into trunk from branch-2.7.1. Closing. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] aajisaka closed pull request #127: Branch 2.7.1
aajisaka closed pull request #127: Branch 2.7.1 URL: https://github.com/apache/hadoop/pull/127 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] aajisaka commented on issue #113: [HADOOP-13075] Adding support for SSE-KMS and SSE-C
aajisaka commented on issue #113: [HADOOP-13075] Adding support for SSE-KMS and SSE-C URL: https://github.com/apache/hadoop/pull/113#issuecomment-456693792 HADOOP-13075 has been fixed. Closing. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] aajisaka closed pull request #113: [HADOOP-13075] Adding support for SSE-KMS and SSE-C
aajisaka closed pull request #113: [HADOOP-13075] Adding support for SSE-KMS and SSE-C URL: https://github.com/apache/hadoop/pull/113 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] aajisaka closed pull request #110: [HADOOP-13075] Adding support for SSE-KMS, SSE-C and SSE-S3
aajisaka closed pull request #110: [HADOOP-13075] Adding support for SSE-KMS, SSE-C and SSE-S3 URL: https://github.com/apache/hadoop/pull/110 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] aajisaka commented on issue #110: [HADOOP-13075] Adding support for SSE-KMS, SSE-C and SSE-S3
aajisaka commented on issue #110: [HADOOP-13075] Adding support for SSE-KMS, SSE-C and SSE-S3 URL: https://github.com/apache/hadoop/pull/110#issuecomment-456693576 HADOOP-13075 has been fixed. Closing. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16065) -Ddynamodb should be -Ddynamo in AWS SDK testing document
[
https://issues.apache.org/jira/browse/HADOOP-16065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-16065:
---
Status: Patch Available (was: Open)
> -Ddynamodb should be -Ddynamo in AWS SDK testing document
> -
>
> Key: HADOOP-16065
> URL: https://issues.apache.org/jira/browse/HADOOP-16065
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-16065.01.patch
>
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
> {{-Ddynamodb}} should be {{-Ddynamo}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16065) -Ddynamodb should be -Ddynamo in AWS SDK testing document
[
https://issues.apache.org/jira/browse/HADOOP-16065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-16065:
---
Attachment: HADOOP-16065.01.patch
> -Ddynamodb should be -Ddynamo in AWS SDK testing document
> -
>
> Key: HADOOP-16065
> URL: https://issues.apache.org/jira/browse/HADOOP-16065
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-16065.01.patch
>
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
> {{-Ddynamodb}} should be {{-Ddynamo}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16065) -Ddynamodb should be -Ddynamo in AWS SDK testing document
[
https://issues.apache.org/jira/browse/HADOOP-16065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reassigned HADOOP-16065:
--
Assignee: Akira Ajisaka
> -Ddynamodb should be -Ddynamo in AWS SDK testing document
> -
>
> Key: HADOOP-16065
> URL: https://issues.apache.org/jira/browse/HADOOP-16065
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Labels: newbie
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
> {{-Ddynamodb}} should be {{-Ddynamo}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16055) Upgrade AWS SDK to fix license issue in branch-2.8 and branch-2.7
[ https://issues.apache.org/jira/browse/HADOOP-16055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749545#comment-16749545 ] Akira Ajisaka commented on HADOOP-16055: Attached a patch. Now testing in Tokyo region. > Upgrade AWS SDK to fix license issue in branch-2.8 and branch-2.7 > - > > Key: HADOOP-16055 > URL: https://issues.apache.org/jira/browse/HADOOP-16055 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Blocker > Attachments: HADOOP-16055-branch-2-01.patch, > HADOOP-16055-branch-2.8-01.patch, HADOOP-16055-branch-2.8-02.patch > > > Per HADOOP-13794, we must exclude the JSON license. > The upgrade will contain incompatible changes, however, the license issue is > much more important. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16065) -Ddynamodb should be -Ddynamo in AWS SDK testing document
Akira Ajisaka created HADOOP-16065:
--
Summary: -Ddynamodb should be -Ddynamo in AWS SDK testing document
Key: HADOOP-16065
URL: https://issues.apache.org/jira/browse/HADOOP-16065
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Reporter: Akira Ajisaka
{{-Ddynamodb}} should be {{-Ddynamo}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16055) Upgrade AWS SDK to fix license issue in branch-2.8 and branch-2.7
[ https://issues.apache.org/jira/browse/HADOOP-16055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-16055: --- Attachment: HADOOP-16055-branch-2-01.patch > Upgrade AWS SDK to fix license issue in branch-2.8 and branch-2.7 > - > > Key: HADOOP-16055 > URL: https://issues.apache.org/jira/browse/HADOOP-16055 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Blocker > Attachments: HADOOP-16055-branch-2-01.patch, > HADOOP-16055-branch-2.8-01.patch, HADOOP-16055-branch-2.8-02.patch > > > Per HADOOP-13794, we must exclude the JSON license. > The upgrade will contain incompatible changes, however, the license issue is > much more important. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16055) Upgrade AWS SDK to fix license issue in branch-2.8 and branch-2.7
[ https://issues.apache.org/jira/browse/HADOOP-16055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749543#comment-16749543 ] Akira Ajisaka commented on HADOOP-16055: bq. something to discuss. If it's kept alive, maybe exclude the aws SDK From the .tarball, though that leaves the problem about POMs open. Yes, let's discuss in common-dev ML. > Upgrade AWS SDK to fix license issue in branch-2.8 and branch-2.7 > - > > Key: HADOOP-16055 > URL: https://issues.apache.org/jira/browse/HADOOP-16055 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Blocker > Attachments: HADOOP-16055-branch-2.8-01.patch, > HADOOP-16055-branch-2.8-02.patch > > > Per HADOOP-13794, we must exclude the JSON license. > The upgrade will contain incompatible changes, however, the license issue is > much more important. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16055) Upgrade AWS SDK to fix license issue in branch-2.8 and branch-2.7
[ https://issues.apache.org/jira/browse/HADOOP-16055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749542#comment-16749542 ] Akira Ajisaka commented on HADOOP-16055: Thanks. I'd like to upgrade to 271 in branch-2 first, and then backport them to branch-2.9 and 2.8. > Upgrade AWS SDK to fix license issue in branch-2.8 and branch-2.7 > - > > Key: HADOOP-16055 > URL: https://issues.apache.org/jira/browse/HADOOP-16055 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Blocker > Attachments: HADOOP-16055-branch-2.8-01.patch, > HADOOP-16055-branch-2.8-02.patch > > > Per HADOOP-13794, we must exclude the JSON license. > The upgrade will contain incompatible changes, however, the license issue is > much more important. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16065) -Ddynamodb should be -Ddynamo in AWS SDK testing document
[
https://issues.apache.org/jira/browse/HADOOP-16065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-16065:
---
Description:
https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
{{-Ddynamodb}} should be {{-Ddynamo}}.
was:{{-Ddynamodb}} should be {{-Ddynamo}}.
> -Ddynamodb should be -Ddynamo in AWS SDK testing document
> -
>
> Key: HADOOP-16065
> URL: https://issues.apache.org/jira/browse/HADOOP-16065
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: Akira Ajisaka
>Priority: Minor
> Labels: newbie
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
> {{-Ddynamodb}} should be {{-Ddynamo}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15922) DelegationTokenAuthenticationFilter get wrong doAsUser since it does not decode URL
[
https://issues.apache.org/jira/browse/HADOOP-15922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749341#comment-16749341
]
Hudson commented on HADOOP-15922:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15805 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15805/])
HADOOP-15922. Fixed DelegationTokenAuthenticator URL decoding for doAs (eyang:
rev 0dd35e218fd4d6c660fd064e893be3112c546c9f)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java
* (edit)
hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
> DelegationTokenAuthenticationFilter get wrong doAsUser since it does not
> decode URL
> ---
>
> Key: HADOOP-15922
> URL: https://issues.apache.org/jira/browse/HADOOP-15922
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Fix For: 3.3.0, 3.1.2, 3.2.1
>
> Attachments: HADOOP-15922.001.patch, HADOOP-15922.002.patch,
> HADOOP-15922.003.patch, HADOOP-15922.004.patch, HADOOP-15922.005.patch,
> HADOOP-15922.006.patch, HADOOP-15922.007.patch
>
>
> DelegationTokenAuthenticationFilter get wrong doAsUser when proxy user from
> client is complete kerberos name (e.g., user/hostn...@realm.com, actually it
> is acceptable), because DelegationTokenAuthenticationFilter does not decode
> DOAS parameter in URL which is encoded by {{URLEncoder}} at client.
> e.g. KMS as example:
> a. KMSClientProvider creates connection to KMS Server using
> DelegationTokenAuthenticatedURL#openConnection.
> b. If KMSClientProvider is a doAsUser, KMSClientProvider will put {{doas}}
> with url encoded user as one parameter of http request.
> {code:java}
> // proxyuser
> if (doAs != null) {
> extraParams.put(DO_AS, URLEncoder.encode(doAs, "UTF-8"));
> }
> {code}
> c. when KMS server receives the request, it does not decode the proxy user.
> As result, KMS Server will get the wrong proxy user if this proxy user is
> complete Kerberos Name or it includes some special character. Some other
> authentication and authorization exception will throws next to it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15922) DelegationTokenAuthenticationFilter get wrong doAsUser since it does not decode URL
[
https://issues.apache.org/jira/browse/HADOOP-15922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HADOOP-15922:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thank you [~hexiaoqiao] for the patch.
I committed patch 007 to trunk.
> DelegationTokenAuthenticationFilter get wrong doAsUser since it does not
> decode URL
> ---
>
> Key: HADOOP-15922
> URL: https://issues.apache.org/jira/browse/HADOOP-15922
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Fix For: 3.3.0, 3.1.2, 3.2.1
>
> Attachments: HADOOP-15922.001.patch, HADOOP-15922.002.patch,
> HADOOP-15922.003.patch, HADOOP-15922.004.patch, HADOOP-15922.005.patch,
> HADOOP-15922.006.patch, HADOOP-15922.007.patch
>
>
> DelegationTokenAuthenticationFilter get wrong doAsUser when proxy user from
> client is complete kerberos name (e.g., user/hostn...@realm.com, actually it
> is acceptable), because DelegationTokenAuthenticationFilter does not decode
> DOAS parameter in URL which is encoded by {{URLEncoder}} at client.
> e.g. KMS as example:
> a. KMSClientProvider creates connection to KMS Server using
> DelegationTokenAuthenticatedURL#openConnection.
> b. If KMSClientProvider is a doAsUser, KMSClientProvider will put {{doas}}
> with url encoded user as one parameter of http request.
> {code:java}
> // proxyuser
> if (doAs != null) {
> extraParams.put(DO_AS, URLEncoder.encode(doAs, "UTF-8"));
> }
> {code}
> c. when KMS server receives the request, it does not decode the proxy user.
> As result, KMS Server will get the wrong proxy user if this proxy user is
> complete Kerberos Name or it includes some special character. Some other
> authentication and authorization exception will throws next to it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15922) DelegationTokenAuthenticationFilter get wrong doAsUser since it does not decode URL
[
https://issues.apache.org/jira/browse/HADOOP-15922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749332#comment-16749332
]
Eric Yang edited comment on HADOOP-15922 at 1/23/19 12:40 AM:
--
Thank you [~hexiaoqiao] for the patch.
I committed patch 007 to trunk, branch-3.2, branch-3.1.
was (Author: eyang):
Thank you [~hexiaoqiao] for the patch.
I committed patch 007 to trunk.
> DelegationTokenAuthenticationFilter get wrong doAsUser since it does not
> decode URL
> ---
>
> Key: HADOOP-15922
> URL: https://issues.apache.org/jira/browse/HADOOP-15922
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Fix For: 3.3.0, 3.1.2, 3.2.1
>
> Attachments: HADOOP-15922.001.patch, HADOOP-15922.002.patch,
> HADOOP-15922.003.patch, HADOOP-15922.004.patch, HADOOP-15922.005.patch,
> HADOOP-15922.006.patch, HADOOP-15922.007.patch
>
>
> DelegationTokenAuthenticationFilter get wrong doAsUser when proxy user from
> client is complete kerberos name (e.g., user/hostn...@realm.com, actually it
> is acceptable), because DelegationTokenAuthenticationFilter does not decode
> DOAS parameter in URL which is encoded by {{URLEncoder}} at client.
> e.g. KMS as example:
> a. KMSClientProvider creates connection to KMS Server using
> DelegationTokenAuthenticatedURL#openConnection.
> b. If KMSClientProvider is a doAsUser, KMSClientProvider will put {{doas}}
> with url encoded user as one parameter of http request.
> {code:java}
> // proxyuser
> if (doAs != null) {
> extraParams.put(DO_AS, URLEncoder.encode(doAs, "UTF-8"));
> }
> {code}
> c. when KMS server receives the request, it does not decode the proxy user.
> As result, KMS Server will get the wrong proxy user if this proxy user is
> complete Kerberos Name or it includes some special character. Some other
> authentication and authorization exception will throws next to it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15922) DelegationTokenAuthenticationFilter get wrong doAsUser since it does not decode URL
[
https://issues.apache.org/jira/browse/HADOOP-15922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749327#comment-16749327
]
Hadoop QA commented on HADOOP-15922:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 1s{color} | {color:orange} hadoop-common-project: The patch generated 2 new
+ 97 unchanged - 0 fixed = 99 total (was 97) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
6s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
28s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}107m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15922 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12953865/HADOOP-15922.007.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 1df15b01f351 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0ef54f7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[GitHub] elek commented on issue #469: HADOOP-15281: Add -directwrite DistCp option
elek commented on issue #469: HADOOP-15281: Add -directwrite DistCp option URL: https://github.com/apache/hadoop/pull/469#issuecomment-456594691 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15281) Distcp to add no-rename copy option
[
https://issues.apache.org/jira/browse/HADOOP-15281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749237#comment-16749237
]
Andrew Olson commented on HADOOP-15281:
---
[~ste...@apache.org] I don't seem to have permission to attach a patch to this
issue. Here's a pull request: https://github.com/apache/hadoop/pull/469
> Distcp to add no-rename copy option
> ---
>
> Key: HADOOP-15281
> URL: https://issues.apache.org/jira/browse/HADOOP-15281
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Priority: Major
>
> Currently Distcp uploads a file by two strategies
> # append parts
> # copy to temp then rename
> option 2 executes the following sequence in {{promoteTmpToTarget}}
> {code}
> if ((fs.exists(target) && !fs.delete(target, false))
> || (!fs.exists(target.getParent()) && !fs.mkdirs(target.getParent()))
> || !fs.rename(tmpTarget, target)) {
> throw new IOException("Failed to promote tmp-file:" + tmpTarget
> + " to: " + target);
> }
> {code}
> For any object store, that's a lot of HTTP requests; for S3A you are looking
> at 12+ requests and an O(data) copy call.
> This is not a good upload strategy for any store which manifests its output
> atomically at the end of the write().
> Proposed: add a switch to write direct to the dest path. either a conf option
> or a CLI option
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] noslowerdna opened a new pull request #469: HADOOP-15281: Add -directwrite DistCp option
noslowerdna opened a new pull request #469: HADOOP-15281: Add -directwrite DistCp option URL: https://github.com/apache/hadoop/pull/469 Adds a `-directwrite` option to DistCp as suggested in [HADOOP-15281](https://issues.apache.org/jira/browse/HADOOP-15281). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15281) Distcp to add no-rename copy option
[
https://issues.apache.org/jira/browse/HADOOP-15281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749222#comment-16749222
]
Andrew Olson commented on HADOOP-15281:
---
Attaching patch file
> Distcp to add no-rename copy option
> ---
>
> Key: HADOOP-15281
> URL: https://issues.apache.org/jira/browse/HADOOP-15281
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Priority: Major
>
> Currently Distcp uploads a file by two strategies
> # append parts
> # copy to temp then rename
> option 2 executes the following sequence in {{promoteTmpToTarget}}
> {code}
> if ((fs.exists(target) && !fs.delete(target, false))
> || (!fs.exists(target.getParent()) && !fs.mkdirs(target.getParent()))
> || !fs.rename(tmpTarget, target)) {
> throw new IOException("Failed to promote tmp-file:" + tmpTarget
> + " to: " + target);
> }
> {code}
> For any object store, that's a lot of HTTP requests; for S3A you are looking
> at 12+ requests and an O(data) copy call.
> This is not a good upload strategy for any store which manifests its output
> atomically at the end of the write().
> Proposed: add a switch to write direct to the dest path. either a conf option
> or a CLI option
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-15281) Distcp to add no-rename copy option
[
https://issues.apache.org/jira/browse/HADOOP-15281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Olson updated HADOOP-15281:
--
Comment: was deleted
(was: Attaching patch file)
> Distcp to add no-rename copy option
> ---
>
> Key: HADOOP-15281
> URL: https://issues.apache.org/jira/browse/HADOOP-15281
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Priority: Major
>
> Currently Distcp uploads a file by two strategies
> # append parts
> # copy to temp then rename
> option 2 executes the following sequence in {{promoteTmpToTarget}}
> {code}
> if ((fs.exists(target) && !fs.delete(target, false))
> || (!fs.exists(target.getParent()) && !fs.mkdirs(target.getParent()))
> || !fs.rename(tmpTarget, target)) {
> throw new IOException("Failed to promote tmp-file:" + tmpTarget
> + " to: " + target);
> }
> {code}
> For any object store, that's a lot of HTTP requests; for S3A you are looking
> at 12+ requests and an O(data) copy call.
> This is not a good upload strategy for any store which manifests its output
> atomically at the end of the write().
> Proposed: add a switch to write direct to the dest path. either a conf option
> or a CLI option
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-16048) ABFS: Fix Date format parser
[ https://issues.apache.org/jira/browse/HADOOP-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-16048: - Comment: was deleted (was: Thanks, I will leave it empty in future before commit.) > ABFS: Fix Date format parser > > > Key: HADOOP-16048 > URL: https://issues.apache.org/jira/browse/HADOOP-16048 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure > Environment: Parsing of the US date does not work on a machine with > European time format, need to set "Locale.*_US_*" when creating > SimpleDateFormat. >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-16048-001.patch > > > AbfsFileSystemStore.parseLastModifiedTime needs to be locale insensitive -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16048) ABFS: Fix Date format parser
[ https://issues.apache.org/jira/browse/HADOOP-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749028#comment-16749028 ] Da Zhou commented on HADOOP-16048: -- Thanks, I will leave it empty in future before the commit. > ABFS: Fix Date format parser > > > Key: HADOOP-16048 > URL: https://issues.apache.org/jira/browse/HADOOP-16048 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure > Environment: Parsing of the US date does not work on a machine with > European time format, need to set "Locale.*_US_*" when creating > SimpleDateFormat. >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-16048-001.patch > > > AbfsFileSystemStore.parseLastModifiedTime needs to be locale insensitive -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16048) ABFS: Fix Date format parser
[ https://issues.apache.org/jira/browse/HADOOP-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-16048: - Fix Version/s: 3.2.1 > ABFS: Fix Date format parser > > > Key: HADOOP-16048 > URL: https://issues.apache.org/jira/browse/HADOOP-16048 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure > Environment: Parsing of the US date does not work on a machine with > European time format, need to set "Locale.*_US_*" when creating > SimpleDateFormat. >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-16048-001.patch > > > AbfsFileSystemStore.parseLastModifiedTime needs to be locale insensitive -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16048) ABFS: Fix Date format parser
[ https://issues.apache.org/jira/browse/HADOOP-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749027#comment-16749027 ] Da Zhou commented on HADOOP-16048: -- Thanks, I will leave it empty in future before commit. > ABFS: Fix Date format parser > > > Key: HADOOP-16048 > URL: https://issues.apache.org/jira/browse/HADOOP-16048 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure > Environment: Parsing of the US date does not work on a machine with > European time format, need to set "Locale.*_US_*" when creating > SimpleDateFormat. >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-16048-001.patch > > > AbfsFileSystemStore.parseLastModifiedTime needs to be locale insensitive -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16048) ABFS: Fix Date format parser
[ https://issues.apache.org/jira/browse/HADOOP-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-16048: - Fix Version/s: (was: 3.2.1) > ABFS: Fix Date format parser > > > Key: HADOOP-16048 > URL: https://issues.apache.org/jira/browse/HADOOP-16048 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure > Environment: Parsing of the US date does not work on a machine with > European time format, need to set "Locale.*_US_*" when creating > SimpleDateFormat. >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-16048-001.patch > > > AbfsFileSystemStore.parseLastModifiedTime needs to be locale insensitive -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16039) backport HADOOP-15965, "Upgrade ADLS SDK to 2.3.3" to branch-2
[
https://issues.apache.org/jira/browse/HADOOP-16039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749004#comment-16749004
]
Hadoop QA commented on HADOOP-16039:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
59s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s{color}
| {color:red} hadoop-tools_hadoop-azure-datalake generated 1 new + 2 unchanged
- 0 fixed = 3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
51s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:a5f678f |
| JIRA Issue | HADOOP-16039 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955833/HADOOP-16039-branch-2-001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux dd3792bd7c8b 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / d3b06d1 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_181 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15824/artifact/out/diff-compile-javac-hadoop-tools_hadoop-azure-datalake.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15824/testReport/ |
| Max. process+thread count | 113 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure-datalake U:
hadoop-tools/hadoop-azure-datalake |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15824/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> backport HADOOP-15965, "Upgrade ADLS SDK to 2.3.3" to branch-2
> --
>
> Key: HADOOP-16039
> URL: https://issues.apache.org/jira/browse/HADOOP-16039
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments:
[jira] [Updated] (HADOOP-16039) backport HADOOP-15965, "Upgrade ADLS SDK to 2.3.3" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16039: Attachment: HADOOP-16039-branch-2-001.patch > backport HADOOP-15965, "Upgrade ADLS SDK to 2.3.3" to branch-2 > -- > > Key: HADOOP-16039 > URL: https://issues.apache.org/jira/browse/HADOOP-16039 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.9.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-16039-001.patch, HADOOP-16039-branch-2-001.patch > > > Backport the ADLS SDK 2.3.3 update to branch-2, retest. etc -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16039) backport HADOOP-15965, "Upgrade ADLS SDK to 2.3.3" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16039: Status: Patch Available (was: Open) patch 001; file renamed as noted, ADLS doesn't like me right now. If others can test this, it will stop regressions > backport HADOOP-15965, "Upgrade ADLS SDK to 2.3.3" to branch-2 > -- > > Key: HADOOP-16039 > URL: https://issues.apache.org/jira/browse/HADOOP-16039 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.9.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-16039-001.patch, HADOOP-16039-branch-2-001.patch > > > Backport the ADLS SDK 2.3.3 update to branch-2, retest. etc -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16039) backport HADOOP-15965, "Upgrade ADLS SDK to 2.3.3" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16039: Status: Open (was: Patch Available) > backport HADOOP-15965, "Upgrade ADLS SDK to 2.3.3" to branch-2 > -- > > Key: HADOOP-16039 > URL: https://issues.apache.org/jira/browse/HADOOP-16039 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.9.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-16039-001.patch > > > Backport the ADLS SDK 2.3.3 update to branch-2, retest. etc -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16059) Use SASL Factories Cache to Improove Performance
[
https://issues.apache.org/jira/browse/HADOOP-16059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748978#comment-16748978
]
Hadoop QA commented on HADOOP-16059:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
29s{color} | {color:green} root: The patch generated 0 new + 156 unchanged - 6
fixed = 156 total (was 162) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
7s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
48s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16059 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955811/HADOOP-16059-04.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e84d85d00be4 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6f0756f |
| maven | version: Apache Maven 3.3.9 |
| Default
[jira] [Commented] (HADOOP-16048) ABFS: Fix Date format parser
[ https://issues.apache.org/jira/browse/HADOOP-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748898#comment-16748898 ] Hudson commented on HADOOP-16048: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #15801 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15801/]) HADOOP-16048. ABFS: Fix Date format parser. (stevel: rev 00ad9e23e88d1e1b1f887b6c9473a7a924a95a97) * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java > ABFS: Fix Date format parser > > > Key: HADOOP-16048 > URL: https://issues.apache.org/jira/browse/HADOOP-16048 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure > Environment: Parsing of the US date does not work on a machine with > European time format, need to set "Locale.*_US_*" when creating > SimpleDateFormat. >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-16048-001.patch > > > AbfsFileSystemStore.parseLastModifiedTime needs to be locale insensitive -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16048) ABFS: Fix Date format parser
[ https://issues.apache.org/jira/browse/HADOOP-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16048: Resolution: Fixed Target Version/s: 3.2.1 Status: Resolved (was: Patch Available) +1, committed BTW, can you tag the "target" version for where you want a patch to go, leaving fix version until the commit. We use the fix version for generating of change reports, so marking uncommitted code as fixed for a version only causes confusion. Thanks > ABFS: Fix Date format parser > > > Key: HADOOP-16048 > URL: https://issues.apache.org/jira/browse/HADOOP-16048 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure > Environment: Parsing of the US date does not work on a machine with > European time format, need to set "Locale.*_US_*" when creating > SimpleDateFormat. >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-16048-001.patch > > > AbfsFileSystemStore.parseLastModifiedTime needs to be locale insensitive -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16059) Use SASL Factories Cache to Improove Performance
[ https://issues.apache.org/jira/browse/HADOOP-16059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HADOOP-16059: -- Attachment: HADOOP-16059-04.patch > Use SASL Factories Cache to Improove Performance > > > Key: HADOOP-16059 > URL: https://issues.apache.org/jira/browse/HADOOP-16059 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Critical > Attachments: HADOOP-16059-01.patch, HADOOP-16059-02.patch, > HADOOP-16059-02.patch, HADOOP-16059-03.patch, HADOOP-16059-04.patch > > > SASL Client factories can be cached and SASL Server Factories and SASL Client > Factories can be together extended at SaslParticipant to improve performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth
[ https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748868#comment-16748868 ] junhua gu edited comment on HADOOP-15954 at 1/22/19 4:06 PM: - *AbfsConfiguration.java* minor: public String getString(String key, String defaultValue) can be removed since it is no longer used. This patch looks good to me, +1 was (Author: thanatosjug): *AbfsConfiguration.java* minor: public String getString(String key, String defaultValue) can be removed since it is no longer used. The rest looks good to me, +1 > ABFS: Enable owner and group conversion for MSI and login user using OAuth > -- > > Key: HADOOP-15954 > URL: https://issues.apache.org/jira/browse/HADOOP-15954 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: junhua gu >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, > HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, > HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, > HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, > HADOOP-15954-012.patch > > > Add support for overwriting owner and group in set/get operations to be the > service principal id when OAuth is used. Add support for upn short name > format. > > Add Standard Transformer for SharedKey / Service > Add interface provides an extensible model for customizing the acquisition of > Identity Transformer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth
[ https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748868#comment-16748868 ] junhua gu commented on HADOOP-15954: *AbfsConfiguration.java* minor: public String getString(String key, String defaultValue) can be removed since it is no longer used. The rest looks good to me, +1 > ABFS: Enable owner and group conversion for MSI and login user using OAuth > -- > > Key: HADOOP-15954 > URL: https://issues.apache.org/jira/browse/HADOOP-15954 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: junhua gu >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, > HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, > HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, > HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch, > HADOOP-15954-012.patch > > > Add support for overwriting owner and group in set/get operations to be the > service principal id when OAuth is used. Add support for upn short name > format. > > Add Standard Transformer for SharedKey / Service > Add interface provides an extensible model for customizing the acquisition of > Identity Transformer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16059) Use SASL Factories Cache to Improove Performance
[
https://issues.apache.org/jira/browse/HADOOP-16059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748862#comment-16748862
]
Hadoop QA commented on HADOOP-16059:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 6m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
4s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 0s{color} | {color:orange} root: The patch generated 5 new + 156 unchanged
- 6 fixed = 161 total (was 162) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
48s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 40s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
56s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
| | Write to static field
org.apache.hadoop.security.SaslRpcClient.saslFactory from instance method new
org.apache.hadoop.security.SaslRpcClient(UserGroupInformation, Class,
InetSocketAddress, Configuration) At SaslRpcClient.java:from instance method
new org.apache.hadoop.security.SaslRpcClient(UserGroupInformation, Class,
InetSocketAddress, Configuration) At SaslRpcClient.java:[line 125] |
| Failed junit tests |
hadoop.security.token.delegation.TestZKDelegationTokenSecretManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |
[jira] [Commented] (HADOOP-16064) Load configuration values from external sources
[
https://issues.apache.org/jira/browse/HADOOP-16064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748760#comment-16748760
]
Steve Loughran commented on HADOOP-16064:
-
Interesting idea.
This is still a fairly raw patch and it's going to have to go through a fair
few iterations before it is ready to go in. Configuration is such as stable and
foundational class that we need to tread very carefully when doing radical
things. I'm not saying there aren't benefits, only that we mustn't rush into
supporting something which turns out not to be ready.
There are some serious security implications here. I hope the (forthcoming)
documentation has a section here. [~lmccay] will no doubt have opinions.
And failure handling, obviously. I don't see any here other than
e.printStackTrace.
patch-wise
* ozone/hdds pom changes are urgent and should go on their own patch.
* mark everything as private/unstable
* import ordering should be [java, third-party, org-apache, static].
* use final fields where possible
* a lot of the class names are too generic "env", "stub". Please make them
clearer
* and add the javadocs to the classes, methods and production packages
h3. {{ConfigurationLocator}}
* must use logs for printing
* go on, add some javadocs
* L71 toLowerCase() needs a locale
h3. {{ConfigurationLocator}}
lacks resilience to/strategy for failure for service implementations to load.
h3. {{Env}}
* I don't like this class name; too close to system stuff, please use something
a bit longer/clearer
* what does the prefix field do?
h3. {{HadoopWeb}}
* again, name bears ~no relation to what it does, which is "load hadoop
configuration from remote site"
* and again, needs to log errors through SLF4J
* With the URL which is causing problems. Without this your support team will
never forgive you.
I don't see any resilience to the classic problem of "page returns HTML". check
your content-type before trying to parse it
h3. TestHadoopWeb
* doesn't shut down server.
* doesn't stress the code with invalid URLs, error responses, non-XML data
Proposed:
* do the server setup/teardown in before/after class.
*
> Load configuration values from external sources
> ---
>
> Key: HADOOP-16064
> URL: https://issues.apache.org/jira/browse/HADOOP-16064
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-16064.001.patch
>
>
> This is a proposal to improve the Configuration.java to load configuration
> from external sources (kubernetes config map, external http reqeust, any
> cluster manager like ambari, etc.)
> I will attach a patch to illustrate the proposed solution, but please comment
> the concept first, the patch is just poc and not fully implemented.
> *Goals:*
> * Load the configuration files (core-site.xml/hdfs-site.xml/...) from
> external locations instead of the classpath (classpath remains the default)
> * Make the configuration loading extensible
> * Make it in an backward-compatible way with minimal change in the existing
> Configuration.java
> *Use-cases:*
> 1.) load configuration from the namenode ([http://namenode:9878/conf]). With
> this approach only the namenode should be configured, other components
> require only the url of the namenode
> 2.) Read configuration directly from kubernetes config-map (or mesos)
> 3.) Read configuration from any external cluster management (such as Apache
> Ambari or any equivalent)
> 4.) as of now in the hadoop docker images we transform environment variables
> (such as HDFS-SITE.XML_fs.defaultFs) to configuration xml files with the help
> of a python script. With the proposed implementation it would be possible to
> read the configuration directly from the system environment variables.
> *Problem:*
> The existing Configuration.java can read configuration from multiple sources.
> But most of the time it's used to load predefined config names
> ("core-site.xml" and "hdfs-site.xml") without configuration location. In this
> case the files will be loaded from the classpath.
> I propose to add additional option to define the default location of
> core-site.xml and hdfs-site.xml (any configuration which is defined by string
> name) to use external sources in the classpath.
> The configuration loading requires implementation + configuration (where are
> the external configs). We can't use regular configuration to configure the
> config loader (chicken/egg).
> I propose to use a new environment variable HADOOP_CONF_SOURCE
> The environment variable could contain a URL, where the schema of the url can
> define the config source and all the other parts can configure the access to
> the resource.
> Examples:
>
[jira] [Updated] (HADOOP-16059) Use SASL Factories Cache to Improove Performance
[ https://issues.apache.org/jira/browse/HADOOP-16059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HADOOP-16059: -- Attachment: HADOOP-16059-03.patch > Use SASL Factories Cache to Improove Performance > > > Key: HADOOP-16059 > URL: https://issues.apache.org/jira/browse/HADOOP-16059 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Critical > Attachments: HADOOP-16059-01.patch, HADOOP-16059-02.patch, > HADOOP-16059-02.patch, HADOOP-16059-03.patch > > > SASL Client factories can be cached and SASL Server Factories and SASL Client > Factories can be together extended at SaslParticipant to improve performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16064) Load configuration values from external sources
[
https://issues.apache.org/jira/browse/HADOOP-16064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HADOOP-16064:
--
Attachment: HADOOP-16064.001.patch
> Load configuration values from external sources
> ---
>
> Key: HADOOP-16064
> URL: https://issues.apache.org/jira/browse/HADOOP-16064
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-16064.001.patch
>
>
> This is a proposal to improve the Configuration.java to load configuration
> from external sources (kubernetes config map, external http reqeust, any
> cluster manager like ambari, etc.)
> I will attach a patch to illustrate the proposed solution, but please comment
> the concept first, the patch is just poc and not fully implemented.
> *Goals:*
> * Load the configuration files (core-site.xml/hdfs-site.xml/...) from
> external locations instead of the classpath (classpath remains the default)
> * Make the configuration loading extensible
> * Make it in an backward-compatible way with minimal change in the existing
> Configuration.java
> *Use-cases:*
> 1.) load configuration from the namenode ([http://namenode:9878/conf]). With
> this approach only the namenode should be configured, other components
> require only the url of the namenode
> 2.) Read configuration directly from kubernetes config-map (or mesos)
> 3.) Read configuration from any external cluster management (such as Apache
> Ambari or any equivalent)
> 4.) as of now in the hadoop docker images we transform environment variables
> (such as HDFS-SITE.XML_fs.defaultFs) to configuration xml files with the help
> of a python script. With the proposed implementation it would be possible to
> read the configuration directly from the system environment variables.
> *Problem:*
> The existing Configuration.java can read configuration from multiple sources.
> But most of the time it's used to load predefined config names
> ("core-site.xml" and "hdfs-site.xml") without configuration location. In this
> case the files will be loaded from the classpath.
> I propose to add additional option to define the default location of
> core-site.xml and hdfs-site.xml (any configuration which is defined by string
> name) to use external sources in the classpath.
> The configuration loading requires implementation + configuration (where are
> the external configs). We can't use regular configuration to configure the
> config loader (chicken/egg).
> I propose to use a new environment variable HADOOP_CONF_SOURCE
> The environment variable could contain a URL, where the schema of the url can
> define the config source and all the other parts can configure the access to
> the resource.
> Examples:
> HADOOP_CONF_SOURCE=hadoop-[http://namenode:9878/conf]
> HADOOP_CONF_SOURCE=env://prefix
> HADOOP_CONF_SOURCE=k8s://config-map-name
> The ConfigurationSource interface can be as easy as:
> {code:java}
> /**
> * Interface to load hadoop configuration from custom location.
> */
> public interface ConfigurationSource {
> /**
> * Method will be called one with the defined configuration url.
> *
> * @param uri
> */
> void initialize(URI uri) throws IOException;
> /**
> * Method will be called to load a specific configuration resource.
> *
> * @param name of the configuration resource (eg. hdfs-site.xml)
> * @return List of loaded configuraiton key and values.
> */
> List readConfiguration(String name);
> }{code}
> We can choose the right implementation based the schema of the uri and with
> Java Service Provider Interface mechanism
> (META-INF/services/org.apache.hadoop.conf.ConfigurationSource)
> It could be with minimal modification in the Configuration.java (see the
> attached patch as an example)
> The patch contains two example implementation:
> *hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/location/Env.java*
> This can load configuration from environment variables based on a naming
> convention (eg. HDFS-SITE.XML_hdfs.dfs.key=value)
> *hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/location/HadoopWeb.java*
> This implementation can load the configuration from a /conf servlet of any
> Hadoop components.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16064) Load configuration values from external sources
[
https://issues.apache.org/jira/browse/HADOOP-16064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HADOOP-16064:
--
Description:
This is a proposal to improve the Configuration.java to load configuration from
external sources (kubernetes config map, external http reqeust, any cluster
manager like ambari, etc.)
I will attach a patch to illustrate the proposed solution, but please comment
the concept first, the patch is just poc and not fully implemented.
*Goals:*
* Load the configuration files (core-site.xml/hdfs-site.xml/...) from external
locations instead of the classpath (classpath remains the default)
* Make the configuration loading extensible
* Make it in an backward-compatible way with minimal change in the existing
Configuration.java
*Use-cases:*
1.) load configuration from the namenode ([http://namenode:9878/conf]). With
this approach only the namenode should be configured, other components require
only the url of the namenode
2.) Read configuration directly from kubernetes config-map (or mesos)
3.) Read configuration from any external cluster management (such as Apache
Ambari or any equivalent)
4.) as of now in the hadoop docker images we transform environment variables
(such as HDFS-SITE.XML_fs.defaultFs) to configuration xml files with the help
of a python script. With the proposed implementation it would be possible to
read the configuration directly from the system environment variables.
*Problem:*
The existing Configuration.java can read configuration from multiple sources.
But most of the time it's used to load predefined config names ("core-site.xml"
and "hdfs-site.xml") without configuration location. In this case the files
will be loaded from the classpath.
I propose to add additional option to define the default location of
core-site.xml and hdfs-site.xml (any configuration which is defined by string
name) to use external sources in the classpath.
The configuration loading requires implementation + configuration (where are
the external configs). We can't use regular configuration to configure the
config loader (chicken/egg).
I propose to use a new environment variable HADOOP_CONF_SOURCE
The environment variable could contain a URL, where the schema of the url can
define the config source and all the other parts can configure the access to
the resource.
Examples:
HADOOP_CONF_SOURCE=hadoop-[http://namenode:9878/conf]
HADOOP_CONF_SOURCE=env://prefix
HADOOP_CONF_SOURCE=k8s://config-map-name
The ConfigurationSource interface can be as easy as:
{code:java}
/**
* Interface to load hadoop configuration from custom location.
*/
public interface ConfigurationSource {
/**
* Method will be called one with the defined configuration url.
*
* @param uri
*/
void initialize(URI uri) throws IOException;
/**
* Method will be called to load a specific configuration resource.
*
* @param name of the configuration resource (eg. hdfs-site.xml)
* @return List of loaded configuraiton key and values.
*/
List readConfiguration(String name);
}{code}
We can choose the right implementation based the schema of the uri and with
Java Service Provider Interface mechanism
(META-INF/services/org.apache.hadoop.conf.ConfigurationSource)
It could be with minimal modification in the Configuration.java (see the
attached patch as an example)
The patch contains two example implementation:
*hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/location/Env.java*
This can load configuration from environment variables based on a naming
convention (eg. HDFS-SITE.XML_hdfs.dfs.key=value)
*hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/location/HadoopWeb.java*
This implementation can load the configuration from a /conf servlet of any
Hadoop components.
was:
This is a proposal to improve the Configuration.java to load configuration from
external sources (kubernetes config map, external http reqeust, any cluster
manager like ambari, etc.)
I will attach a patch to illustrate the proposed solution, but please comment
the concept first, the patch is just poc and not fully implemented.
*Goals:*
* **Load the configuration files (core-site.xml/hdfs-site.xml/...) from
external locations instead of the classpath (classpath remains the default)
* Make the configuration loading extensible
* Make it in an backward-compatible way with minimal change in the existing
Configuration.java
*Use-cases:*
1.) load configuration from the namenode ([http://namenode:9878/conf]). With
this approach only the namenode should be configured, other components require
only the url of the namenode
2.) Read configuration directly from kubernetes config-map (or mesos)
3.) Read configuration from any external cluster management (such as Apache
Ambari or any equivalent)
4.) as of now in the hadoop docker images we
[jira] [Created] (HADOOP-16064) Load configuration values from external sources
Elek, Marton created HADOOP-16064:
-
Summary: Load configuration values from external sources
Key: HADOOP-16064
URL: https://issues.apache.org/jira/browse/HADOOP-16064
Project: Hadoop Common
Issue Type: Improvement
Reporter: Elek, Marton
This is a proposal to improve the Configuration.java to load configuration from
external sources (kubernetes config map, external http reqeust, any cluster
manager like ambari, etc.)
I will attach a patch to illustrate the proposed solution, but please comment
the concept first, the patch is just poc and not fully implemented.
*Goals:*
* **Load the configuration files (core-site.xml/hdfs-site.xml/...) from
external locations instead of the classpath (classpath remains the default)
* Make the configuration loading extensible
* Make it in an backward-compatible way with minimal change in the existing
Configuration.java
*Use-cases:*
1.) load configuration from the namenode ([http://namenode:9878/conf]). With
this approach only the namenode should be configured, other components require
only the url of the namenode
2.) Read configuration directly from kubernetes config-map (or mesos)
3.) Read configuration from any external cluster management (such as Apache
Ambari or any equivalent)
4.) as of now in the hadoop docker images we transform environment variables
(such as HDFS-SITE.XML_fs.defaultFs) to configuration xml files with the help
of a python script. With the proposed implementation it would be possible to
read the configuration directly from the system environment variables.
*Problem:*
The existing Configuration.java can read configuration from multiple sources.
But most of the time it's used to load predefined config names ("core-site.xml"
and "hdfs-site.xml") without configuration location. In this case the files
will be loaded from the classpath.
I propose to add additional option to define the default location of
core-site.xml and hdfs-site.xml (any configuration which is defined by string
name) to use external sources in the classpath.
The configuration loading requires implementation + configuration (where are
the external configs). We can't use regular configuration to configure the
config loader (chicken/egg).
I propose to use a new environment variable HADOOP_CONF_SOURCE
The environment variable could contain a URL, where the schema of the url can
define the config source and all the other parts can configure the access to
the resource.
Examples:
HADOOP_CONF_SOURCE=hadoop-[http://namenode:9878/conf]
HADOOP_CONF_SOURCE=env://prefix
HADOOP_CONF_SOURCE=k8s://config-map-name
The ConfigurationSource interface can be as easy as:
{code:java}
/**
* Interface to load hadoop configuration from custom location.
*/
public interface ConfigurationSource {
/**
* Method will be called one with the defined configuration url.
*
* @param uri
*/
void initialize(URI uri) throws IOException;
/**
* Method will be called to load a specific configuration resource.
*
* @param name of the configuration resource (eg. hdfs-site.xml)
* @return List of loaded configuraiton key and values.
*/
List readConfiguration(String name);
}{code}
We can choose the right implementation based the schema of the uri and with
Java Service Provider Interface mechanism
(META-INF/services/org.apache.hadoop.conf.ConfigurationSource)
It could be with minimal modification in the Configuration.java (see the
attached patch as an example)
The patch contains two example implementation:
*hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/location/Env.java*
This can load configuration from environment variables based on a naming
convention (eg. HDFS-SITE.XML_hdfs.dfs.key=value)
*hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/location/HadoopWeb.java*
This implementation can load the configuration from a /conf servlet of any
Hadoop components.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16064) Load configuration values from external sources
[
https://issues.apache.org/jira/browse/HADOOP-16064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton reassigned HADOOP-16064:
-
Assignee: Elek, Marton
> Load configuration values from external sources
> ---
>
> Key: HADOOP-16064
> URL: https://issues.apache.org/jira/browse/HADOOP-16064
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
>
> This is a proposal to improve the Configuration.java to load configuration
> from external sources (kubernetes config map, external http reqeust, any
> cluster manager like ambari, etc.)
> I will attach a patch to illustrate the proposed solution, but please comment
> the concept first, the patch is just poc and not fully implemented.
> *Goals:*
> * **Load the configuration files (core-site.xml/hdfs-site.xml/...) from
> external locations instead of the classpath (classpath remains the default)
> * Make the configuration loading extensible
> * Make it in an backward-compatible way with minimal change in the existing
> Configuration.java
> *Use-cases:*
> 1.) load configuration from the namenode ([http://namenode:9878/conf]). With
> this approach only the namenode should be configured, other components
> require only the url of the namenode
> 2.) Read configuration directly from kubernetes config-map (or mesos)
> 3.) Read configuration from any external cluster management (such as Apache
> Ambari or any equivalent)
> 4.) as of now in the hadoop docker images we transform environment variables
> (such as HDFS-SITE.XML_fs.defaultFs) to configuration xml files with the help
> of a python script. With the proposed implementation it would be possible to
> read the configuration directly from the system environment variables.
> *Problem:*
> The existing Configuration.java can read configuration from multiple sources.
> But most of the time it's used to load predefined config names
> ("core-site.xml" and "hdfs-site.xml") without configuration location. In this
> case the files will be loaded from the classpath.
> I propose to add additional option to define the default location of
> core-site.xml and hdfs-site.xml (any configuration which is defined by string
> name) to use external sources in the classpath.
> The configuration loading requires implementation + configuration (where are
> the external configs). We can't use regular configuration to configure the
> config loader (chicken/egg).
> I propose to use a new environment variable HADOOP_CONF_SOURCE
> The environment variable could contain a URL, where the schema of the url can
> define the config source and all the other parts can configure the access to
> the resource.
> Examples:
> HADOOP_CONF_SOURCE=hadoop-[http://namenode:9878/conf]
> HADOOP_CONF_SOURCE=env://prefix
> HADOOP_CONF_SOURCE=k8s://config-map-name
> The ConfigurationSource interface can be as easy as:
> {code:java}
> /**
> * Interface to load hadoop configuration from custom location.
> */
> public interface ConfigurationSource {
> /**
> * Method will be called one with the defined configuration url.
> *
> * @param uri
> */
> void initialize(URI uri) throws IOException;
> /**
> * Method will be called to load a specific configuration resource.
> *
> * @param name of the configuration resource (eg. hdfs-site.xml)
> * @return List of loaded configuraiton key and values.
> */
> List readConfiguration(String name);
> }{code}
> We can choose the right implementation based the schema of the uri and with
> Java Service Provider Interface mechanism
> (META-INF/services/org.apache.hadoop.conf.ConfigurationSource)
> It could be with minimal modification in the Configuration.java (see the
> attached patch as an example)
> The patch contains two example implementation:
> *hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/location/Env.java*
> This can load configuration from environment variables based on a naming
> convention (eg. HDFS-SITE.XML_hdfs.dfs.key=value)
> *hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/location/HadoopWeb.java*
> This implementation can load the configuration from a /conf servlet of any
> Hadoop components.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14898) Create official Docker images for development and testing features
[
https://issues.apache.org/jira/browse/HADOOP-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748677#comment-16748677
]
Elek, Marton commented on HADOOP-14898:
---
Created a follow-up HADOOP-16063. Please check if you are interested.
> Create official Docker images for development and testing features
> ---
>
> Key: HADOOP-14898
> URL: https://issues.apache.org/jira/browse/HADOOP-14898
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HADOOP-14898.001.tar.gz, HADOOP-14898.002.tar.gz,
> HADOOP-14898.003.tgz, docker_design.pdf
>
>
> This is the original mail from the mailing list:
> {code}
> TL;DR: I propose to create official hadoop images and upload them to the
> dockerhub.
> GOAL/SCOPE: I would like improve the existing documentation with easy-to-use
> docker based recipes to start hadoop clusters with various configuration.
> The images also could be used to test experimental features. For example
> ozone could be tested easily with these compose file and configuration:
> https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> Or even the configuration could be included in the compose file:
> https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml
> I would like to create separated example compose files for federation, ha,
> metrics usage, etc. to make it easier to try out and understand the features.
> CONTEXT: There is an existing Jira
> https://issues.apache.org/jira/browse/HADOOP-13397
> But it’s about a tool to generate production quality docker images (multiple
> types, in a flexible way). If no objections, I will create a separated issue
> to create simplified docker images for rapid prototyping and investigating
> new features. And register the branch to the dockerhub to create the images
> automatically.
> MY BACKGROUND: I am working with docker based hadoop/spark clusters quite a
> while and run them succesfully in different environments (kubernetes,
> docker-swarm, nomad-based scheduling, etc.) My work is available from here:
> https://github.com/flokkr but they could handle more complex use cases (eg.
> instrumenting java processes with btrace, or read/reload configuration from
> consul).
> And IMHO in the official hadoop documentation it’s better to suggest to use
> official apache docker images and not external ones (which could be changed).
> {code}
> The next list will enumerate the key decision points regarding to docker
> image creating
> A. automated dockerhub build / jenkins build
> Docker images could be built on the dockerhub (a branch pattern should be
> defined for a github repository and the location of the Docker files) or
> could be built on a CI server and pushed.
> The second one is more flexible (it's more easy to create matrix build, for
> example)
> The first one had the advantage that we can get an additional flag on the
> dockerhub that the build is automated (and built from the source by the
> dockerhub).
> The decision is easy as ASF supports the first approach: (see
> https://issues.apache.org/jira/browse/INFRA-12781?focusedCommentId=15824096=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15824096)
> B. source: binary distribution or source build
> The second question is about creating the docker image. One option is to
> build the software on the fly during the creation of the docker image the
> other one is to use the binary releases.
> I suggest to use the second approach as:
> 1. In that case the hadoop:2.7.3 could contain exactly the same hadoop
> distrubution as the downloadable one
> 2. We don't need to add development tools to the image, the image could be
> more smaller (which is important as the goal for this image to getting
> started as fast as possible)
> 3. The docker definition will be more simple (and more easy to maintain)
> Usually this approach is used in other projects (I checked Apache Zeppelin
> and Apache Nutch)
> C. branch usage
> Other question is the location of the Docker file. It could be on the
> official source-code branches (branch-2, trunk, etc.) or we can create
> separated branches for the dockerhub (eg. docker/2.7 docker/2.8 docker/3.0)
> For the first approach it's easier to find the docker images, but it's less
> flexible. For example if we had a Dockerfile for on the source code it should
> be used for every release (for example the Docker file from the tag
> release-3.0.0 should be used for the 3.0 hadoop docker image). In that case
> the release process is much more harder: in case of a Dockerfile error (which
> could be test on dockerhub only after the taging), a new release should be
>
[jira] [Updated] (HADOOP-15259) Provide docker file for the development builds
[ https://issues.apache.org/jira/browse/HADOOP-15259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15259: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) Closing it as of now as there was no reviewer. Created HADOOP-16063 to do similar works. > Provide docker file for the development builds > -- > > Key: HADOOP-15259 > URL: https://issues.apache.org/jira/browse/HADOOP-15259 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15259.001.patch > > > An other use case for using docker image is creating custom docker image > (base image + custom hadoop build). The custom image could be used to test > easily the hadoop build on external dockerized cluster (eg. Kubernetes) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15257) Provide example docker compose file for developer builds
[ https://issues.apache.org/jira/browse/HADOOP-15257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15257: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) Closing it as of now as there was no reviewer. Created HADOOP-16063 to do similar works. > Provide example docker compose file for developer builds > > > Key: HADOOP-15257 > URL: https://issues.apache.org/jira/browse/HADOOP-15257 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15257.001.patch > > > This issue is about creating example docker-compose files which use the > latest build from the hadoop-dist directory. > These docker-compose files would help to run a specific hadoop cluster based > on the latest custom build without the need to build customized docker image > (with mounting hadoop fro hadoop-dist to the container -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15258) Create example docker-compose file for documentations
[ https://issues.apache.org/jira/browse/HADOOP-15258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15258: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) Closing it as of now as there was no reviewer. Created HADOOP-16063 to do similar works. > Create example docker-compose file for documentations > - > > Key: HADOOP-15258 > URL: https://issues.apache.org/jira/browse/HADOOP-15258 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15258.001.patch > > > An other user case for docker is to use it in the documentation. For example > in the HA documentation we can provide an example docker-compose file and > configuration with all the required settings to getting started easily with > an HA cluster. > 1. I would add an example to a documetation page > 2. It will use the hadoop3 image (which contains latest hadoop3) as the user > of the documentation may not build a hadoop -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16063) Docker based pseudo-cluster definitions and test scripts for Hdfs/Yarn
Elek, Marton created HADOOP-16063: - Summary: Docker based pseudo-cluster definitions and test scripts for Hdfs/Yarn Key: HADOOP-16063 URL: https://issues.apache.org/jira/browse/HADOOP-16063 Project: Hadoop Common Issue Type: New Feature Reporter: Elek, Marton During the recent releases of Apache Hadoop Ozone we had multiple experiments using docker/docker-compose to support the development of ozone. As of now the hadoop-ozone distribution contains two directories in additional the regular hadoop directories (bin, share/lib, etc h3. compose The ./compose directory of the distribution contains different type of pseudo-cluster definitions. To start an ozone cluster is as easy as "cd compose/ozone && docker-compose up-d" The clusters also could be scaled up and down (docker-compose scale datanode=3) There are multiple cluster definitions for different use cases (for example ozone+s3 or hdfs+ozone). The docker-compose files are based on apache/hadoop-runner image which is an "empty" image. It doesnt' contain any hadoop distribution. Instead the current hadoop is used (the ../.. is mapped as a volume at /opt/hadoop) With this approach it's very easy to 1) start a cluster from the distribution 2) test any patch from the dev tree, as after any build a new cluster can be started easily (with multiple nodes and datanodes) h3. smoketest We also started to use a simple robotframework based test suite. (see ./smoketest directory). It's a high level test definition very similar to the smoketests which are executed manually by the contributors during a release vote. But it's a formal definition to start cluster from different docker-compose definitions and execute simple shell scripts (and compare the output). I believe that both approaches helped a lot during the development of ozone and I propose to do the same improvements on the main hadoop distribution. I propose to provide docker-compose based example cluster definitions for yarn/hdfs and for different use cases (simple hdfs, router based federation, etc.) It can help to understand the different configuration and try out new features with predefined config set. Long term we can also add robottests to help the release votes (basic wordcount/mr tests could be scripted) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16061) Update Apache Yetus to 0.9.0
[ https://issues.apache.org/jira/browse/HADOOP-16061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748602#comment-16748602 ] Steve Loughran commented on HADOOP-16061: - I've created the github A/C: [https://github.com/hadoop-yetus] ; INFRA-17722 covers getting a new apache.org email address for this. > Update Apache Yetus to 0.9.0 > > > Key: HADOOP-16061 > URL: https://issues.apache.org/jira/browse/HADOOP-16061 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Priority: Major > > Yetus 0.9.0 is out. Let's upgrade. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] steveloughran commented on issue #163: HADOOP-13227 outputstream specification
steveloughran commented on issue #163: HADOOP-13227 outputstream specification URL: https://github.com/apache/hadoop/pull/163#issuecomment-456358171 @elek that feature doesn't come yet This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
