[jira] [Updated] (HADOOP-15945) ABFS: replace $superuser with local user
[ https://issues.apache.org/jira/browse/HADOOP-15945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-15945: - Summary: ABFS: replace $superuser with local user (was: ABFS: replace superuser with local user) > ABFS: replace $superuser with local user > > > Key: HADOOP-15945 > URL: https://issues.apache.org/jira/browse/HADOOP-15945 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > > if the owner of file/path is $superUser, we should replace it with local > user. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15945) ABFS: replace superuser with local user
[ https://issues.apache.org/jira/browse/HADOOP-15945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-15945: - Summary: ABFS: replace superuser with local user (was: ABFS: replace superuser with local user for GET request result.) > ABFS: replace superuser with local user > --- > > Key: HADOOP-15945 > URL: https://issues.apache.org/jira/browse/HADOOP-15945 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > > if the owner of file/path is $superUser, we should replace it with local > user. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15945) ABFS: replace superuser with local user for GET request result.
Da Zhou created HADOOP-15945: Summary: ABFS: replace superuser with local user for GET request result. Key: HADOOP-15945 URL: https://issues.apache.org/jira/browse/HADOOP-15945 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Reporter: Da Zhou Assignee: Da Zhou Fix For: 3.2.0 if the owner of file/path is $superUser, we should replace it with local user. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693742#comment-16693742 ] Steve Loughran commented on HADOOP-15870: - Afraid you'll need to attach it to HADOOP-15920, because once yetus sees a github PR it kicks into github mode, but it doesn't quite do what we need after doing that, cancel the patch there and hit "submit patch" Again; the act of resubmitting the patch triggers the rerun > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15920) get patch for S3a nextReadPos(), through Yetus
[ https://issues.apache.org/jira/browse/HADOOP-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee reassigned HADOOP-15920: -- Assignee: lqjacklee > get patch for S3a nextReadPos(), through Yetus > -- > > Key: HADOOP-15920 > URL: https://issues.apache.org/jira/browse/HADOOP-15920 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.1 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-001.diff, HADOOP-15870-002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15945) ABFS: replace $superuser with local user
[ https://issues.apache.org/jira/browse/HADOOP-15945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou resolved HADOOP-15945. -- Resolution: Duplicate This is already covered in HADOOP-15872 > ABFS: replace $superuser with local user > > > Key: HADOOP-15945 > URL: https://issues.apache.org/jira/browse/HADOOP-15945 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > > if the owner of file/path is $superUser, we should replace it with local > user. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15920) get patch for S3a nextReadPos(), through Yetus
[ https://issues.apache.org/jira/browse/HADOOP-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15920: --- Attachment: HADOOP-15870-002.patch > get patch for S3a nextReadPos(), through Yetus > -- > > Key: HADOOP-15920 > URL: https://issues.apache.org/jira/browse/HADOOP-15920 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.1 >Reporter: Steve Loughran >Priority: Major > Attachments: HADOOP-15870-001.diff, HADOOP-15870-002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693721#comment-16693721
]
Íñigo Goiri commented on HADOOP-14556:
--
Thanks [~ste...@apache.org] for [^HADOOP-14556-021.patch].
* I think we can solve most of the checkstyle issues left here )specially left
spaces).
* {{to directly authenticate with S3 and DynamoDB services..}}, you have a
double dot here.
* The indices in core-default.xml are dupe, I'm not sure if there's a point on
having a number; anyway, if we keep it we should change the numbers.
* The {{assertEquals}} message in {{AbstractContractGetFileStatusTest}} should
have a space or something in between.
* {{AWSCredentialProviderList}} now does LOG and throw, I think we should keep
it as just throw.
* Constants#DEFAULT_ASSUMED_ROLE_STS_ENDPOINT should fit in one line.
* S3AUtils#getCanonicalServiceURI has comments left behind.
* IAMInstanceCredentialsProvider#getCredentials leaves comments behind.
* In MarshalledCredentialProvider#NAME can use the class.getName() or one of
the variations?
* MarshalledCredentials#equals can use EqualsBuilder (same for
EncryptionSecrets).
* We use the {{Collections.unmodifiableList(Arrays.asList(}} a lot, it might be
worth adding a method to define constant lists.
In general I can provide a very high level review but somebody else should
review deeper.
In any case, general comments:
* The documentation seems to cover the added cases well; it might be worth
doing a full pass to these documents once everything is finished up.
* The unit tests cover the basic cases well.
* Very long patch and even though there are a bunch of interfaces which are
pretty verbose, there is a lot here. I'm not sure if there are ways to split
it. For example the utilities to fetch the DT.
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch,
> HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch,
> HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch,
> HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
> HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch,
> HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch,
> HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch,
> HADOOP-14556-021.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693267#comment-16693267
]
Hadoop QA commented on HADOOP-14556:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 38 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
43s{color} | {color:green} root generated 0 new + 1448 unchanged - 1 fixed =
1448 total (was 1449) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 21s{color} | {color:orange} root: The patch generated 14 new + 167 unchanged
- 11 fixed = 181 total (was 178) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 97 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
17s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
16s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
41s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
[jira] [Updated] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14556:
Status: Open (was: Patch Available)
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch,
> HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch,
> HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch,
> HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
> HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch,
> HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch,
> HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch,
> HADOOP-14556-021.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693229#comment-16693229 ] lqjacklee commented on HADOOP-15870: [^HADOOP-15870-002.patch] > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15870: --- Attachment: HADOOP-15870-002.patch > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15940) ABFS: For HNS account, avoid unnecessary get call when doing Rename
[ https://issues.apache.org/jira/browse/HADOOP-15940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693074#comment-16693074 ] Steve Loughran commented on HADOOP-15940: - If there's two possible "valid" actions for the rename to self, then really there should be another contract option which declares which codepath an FS takes, and the standard test changed to reading it and requiring the specific outcome * makes the behaviour chosen explicit in the XML * makes it easy for other stores to adopt the same behaviour * by adding an assert that the new outcome must occur, catches regressions. Interesting that the handling of a missing source didn't raise an exception -I'd have expected that. FWIW, part of the history of those contract tests is that the s3 and swift connectors didn't check for a rename under self (who would do that?) and after copying the files to the destination, would then recursively delete the entire source tree, so losing the renamed data. Luckily, nobody ever seems to hae tried to do this in the wild, which shows how our filesystem mental model prevents us wandering into dangerous places > ABFS: For HNS account, avoid unnecessary get call when doing Rename > --- > > Key: HADOOP-15940 > URL: https://issues.apache.org/jira/browse/HADOOP-15940 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-15940-001.patch > > > When rename, there is always a GET dst file status call, this is not > necessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14556:
Status: Patch Available (was: Open)
Patch 021: address checkstyle and javadoc warnings from patch 020
No other changes, not done any retest (yet).
This patch is really, really, ready for review. I am successfully using it in
real Distcp jobs
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch,
> HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch,
> HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch,
> HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
> HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch,
> HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch,
> HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch,
> HADOOP-14556-021.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14556:
Attachment: HADOOP-14556-021.patch
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch,
> HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch,
> HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch,
> HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
> HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch,
> HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch,
> HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch,
> HADOOP-14556-021.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14739) Update start-build-env.sh and build instruction for docker for Mac instead of docker toolbox.
[ https://issues.apache.org/jira/browse/HADOOP-14739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693378#comment-16693378 ] Hudson commented on HADOOP-14739: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15474 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15474/]) HADOOP-14739. Update start-build-env.sh and build instruction for docker (aajisaka: rev 10b5da85fab41c68c2c223a3a8749be9719908f6) * (edit) BUILDING.txt * (edit) start-build-env.sh > Update start-build-env.sh and build instruction for docker for Mac instead of > docker toolbox. > - > > Key: HADOOP-14739 > URL: https://issues.apache.org/jira/browse/HADOOP-14739 > Project: Hadoop Common > Issue Type: Improvement > Components: build, documentation >Reporter: Akira Ajisaka >Assignee: Dinesh Chitlangia >Priority: Minor > Labels: newbie > Fix For: 3.3.0 > > Attachments: HADOOP-14739.001.patch, HADOOP-14739.002.patch, > HADOOP-14739.003.patch > > > HADOOP-12575 added build instruction for docker toolbox. > Now Docker for Mac (https://www.docker.com/docker-mac) is available and it > can skip some procedures written in BUILDING.txt. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15650) Add custom InstanceProfileCredentialsProvider with more resilience to throttling
[
https://issues.apache.org/jira/browse/HADOOP-15650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693332#comment-16693332
]
lqjacklee commented on HADOOP-15650:
[~ste...@apache.org] There are two ways to implement the function :
1, Invoke method by reflection, and use the dynamic proxy add monitor and log
feature.
2, Rewrite the logic, and not use the AWS code.
which one do you prefer?
> Add custom InstanceProfileCredentialsProvider with more resilience to
> throttling
>
>
> Key: HADOOP-15650
> URL: https://issues.apache.org/jira/browse/HADOOP-15650
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
>
> Add our own InstanceProfileCredentialsProvider class which uses the AWS
> implementation to retrieve credentials from EC2's instance info, but more
> resilient to overloading.
> # pass in client config with retry logic (HADOOP-15603)
> # use Invoke.retry() to retry
> # log/measure failures
> # maybe use the Async feature of the AWS SDK class, so that credential
> renewer doesn't block IO.
> # be shared amongst all AWS auth chains which need these credentials.
> The singleton we current use for IAM auth doesn't do async, which is good as
> it ensures that we don't prematurely close it when
> {{AWSCredentialProviderList.close()}} closes its children.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14739) Add build instruction for docker for Mac instead of docker toolbox.
[ https://issues.apache.org/jira/browse/HADOOP-14739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693335#comment-16693335 ] Akira Ajisaka commented on HADOOP-14739: +1 > Add build instruction for docker for Mac instead of docker toolbox. > --- > > Key: HADOOP-14739 > URL: https://issues.apache.org/jira/browse/HADOOP-14739 > Project: Hadoop Common > Issue Type: Improvement > Components: build, documentation >Reporter: Akira Ajisaka >Assignee: Dinesh Chitlangia >Priority: Minor > Labels: newbie > Attachments: HADOOP-14739.001.patch, HADOOP-14739.002.patch, > HADOOP-14739.003.patch > > > HADOOP-12575 added build instruction for docker toolbox. > Now Docker for Mac (https://www.docker.com/docker-mac) is available and it > can skip some procedures written in BUILDING.txt. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14739) Update start-build-env.sh and build instruction for docker for Mac instead of docker toolbox.
[ https://issues.apache.org/jira/browse/HADOOP-14739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-14739: --- Hadoop Flags: Reviewed Summary: Update start-build-env.sh and build instruction for docker for Mac instead of docker toolbox. (was: Add build instruction for docker for Mac instead of docker toolbox.) > Update start-build-env.sh and build instruction for docker for Mac instead of > docker toolbox. > - > > Key: HADOOP-14739 > URL: https://issues.apache.org/jira/browse/HADOOP-14739 > Project: Hadoop Common > Issue Type: Improvement > Components: build, documentation >Reporter: Akira Ajisaka >Assignee: Dinesh Chitlangia >Priority: Minor > Labels: newbie > Attachments: HADOOP-14739.001.patch, HADOOP-14739.002.patch, > HADOOP-14739.003.patch > > > HADOOP-12575 added build instruction for docker toolbox. > Now Docker for Mac (https://www.docker.com/docker-mac) is available and it > can skip some procedures written in BUILDING.txt. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14739) Update start-build-env.sh and build instruction for docker for Mac instead of docker toolbox.
[ https://issues.apache.org/jira/browse/HADOOP-14739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-14739: --- Resolution: Fixed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) Committed this to trunk. Thanks [~dineshchitlangia] for the contribution and thanks [~elek] for the help. > Update start-build-env.sh and build instruction for docker for Mac instead of > docker toolbox. > - > > Key: HADOOP-14739 > URL: https://issues.apache.org/jira/browse/HADOOP-14739 > Project: Hadoop Common > Issue Type: Improvement > Components: build, documentation >Reporter: Akira Ajisaka >Assignee: Dinesh Chitlangia >Priority: Minor > Labels: newbie > Fix For: 3.3.0 > > Attachments: HADOOP-14739.001.patch, HADOOP-14739.002.patch, > HADOOP-14739.003.patch > > > HADOOP-12575 added build instruction for docker toolbox. > Now Docker for Mac (https://www.docker.com/docker-mac) is available and it > can skip some procedures written in BUILDING.txt. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14739) Update start-build-env.sh and build instruction for docker for Mac instead of docker toolbox.
[ https://issues.apache.org/jira/browse/HADOOP-14739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693347#comment-16693347 ] Dinesh Chitlangia commented on HADOOP-14739: [~ajisakaa] thank you for review and commit > Update start-build-env.sh and build instruction for docker for Mac instead of > docker toolbox. > - > > Key: HADOOP-14739 > URL: https://issues.apache.org/jira/browse/HADOOP-14739 > Project: Hadoop Common > Issue Type: Improvement > Components: build, documentation >Reporter: Akira Ajisaka >Assignee: Dinesh Chitlangia >Priority: Minor > Labels: newbie > Fix For: 3.3.0 > > Attachments: HADOOP-14739.001.patch, HADOOP-14739.002.patch, > HADOOP-14739.003.patch > > > HADOOP-12575 added build instruction for docker toolbox. > Now Docker for Mac (https://www.docker.com/docker-mac) is available and it > can skip some procedures written in BUILDING.txt. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15650) Add custom InstanceProfileCredentialsProvider with more resilience to throttling
[
https://issues.apache.org/jira/browse/HADOOP-15650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693399#comment-16693399
]
Steve Loughran commented on HADOOP-15650:
-
aah. In HADOOP-14556 I've actually started my own provider for this; had
forgotten about the retry code there. I think I'll leave that as a separate
patch -switching to async refresh would actually be better as it'd eliminate
latencies.
As to what to do: just wrap the existing getCredentials() Call in an Invoker
with a retry policy which will only retry on throttle exceptions, fail fast on
everything else, especially network failures
> Add custom InstanceProfileCredentialsProvider with more resilience to
> throttling
>
>
> Key: HADOOP-15650
> URL: https://issues.apache.org/jira/browse/HADOOP-15650
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
>
> Add our own InstanceProfileCredentialsProvider class which uses the AWS
> implementation to retrieve credentials from EC2's instance info, but more
> resilient to overloading.
> # pass in client config with retry logic (HADOOP-15603)
> # use Invoke.retry() to retry
> # log/measure failures
> # maybe use the Async feature of the AWS SDK class, so that credential
> renewer doesn't block IO.
> # be shared amongst all AWS auth chains which need these credentials.
> The singleton we current use for IAM auth doesn't do async, which is good as
> it ensures that we don't prematurely close it when
> {{AWSCredentialProviderList.close()}} closes its children.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
