[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17488019#comment-17488019 ] Daniel Carl Jones commented on HADOOP-14661: I'd like to pick this ticket up if that's OK. It looks fairly straight forward, though I have some questions about pre-existing requester pays flags in the code base that aren't yet hooked up. There's a flag in [S3ClientCreationParameters|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ClientFactory.java#L103-L106] (added March 2021) and another in [RequestFactoryImpl|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java#L109-L114] (added May 2021, as part of auditor changes). Is there a preference which one should be hooked up? > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17299787#comment-17299787 ] Steve Loughran commented on HADOOP-14661: - HADOOP-17511 is lining up for this patch -all S3 requests will be from a request factory...this is where the option will be set > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236904#comment-17236904 ] Asier Arambarri Beldarrain commented on HADOOP-14661: - Hi Steve, I just send you a mail-message; Please feel free to ask for further work! > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236657#comment-17236657 ] Steve Loughran commented on HADOOP-14661: - Tell you what -if I get a chance I'll put together a PR with your header change and test I'll write myself. You'll get the credit > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235614#comment-17235614 ] Asier Arambarri Beldarrain commented on HADOOP-14661: - [~ste...@apache.org] I would love to work on this if needed, even if I am not really a hadoop/spark expert (I mostly work with Kafka). Anyway, thanks for your words and please do not hesitate asking me for some further work. Also thanks for your comment on the awscredentialsprovider (I believe my coworkers are not having this problem, but I'd tell em to have a look) > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235493#comment-17235493 ] Steve Loughran commented on HADOOP-14661: - (I see looking @ your AWS issue comments that our request factory plan wouldn't have worked. And we would have had to find a requester pays bucket with > 1 page of entries to find the list problem. nice. FWIW the ability to overwrite the factory was mainly to let us plug in an inconsistent s3 client. you've found a new use. Be aware of a problem you may hit with spark and hive's isolated classloader https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md#class-does-not-implement-awscredentialsprovider > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235490#comment-17235490 ] Steve Loughran commented on HADOOP-14661: - Oh, that an *Awesome* way to fix this! I'd assumed that every s3 request we built would need the header and I was going to have to get the factored out request factory of HADOOP-16848 in before worrying about it -I'd be patching every request, the way we do with encryption settings on read/write/copy. Setting it once and for all, well, that's easier isn't it? I'd propose we formalise this with the default factory reading in an fs.s3a option to enable requester pays from the config, setting it and then creating the AWS console. Pretty much what you've done, just: wrapping in a config and then enabling if set. Testing would be fun. CommonCrawl provide some requester pays buckets, we'd need to do the LIST/HEAD/GET sequence to validate the read path., and assume if they work, so will writing (which is much more niche, isn't it?) Do you fancy working on this? > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[
https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234615#comment-17234615
]
Hadoop QA commented on HADOOP-14661:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s{color}
| | {color:red} HADOOP-14661 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14661 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877218/HADOOP-14661.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/120/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> S3A to support Requester Pays Buckets
> -
>
> Key: HADOOP-14661
> URL: https://issues.apache.org/jira/browse/HADOOP-14661
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common, util
>Affects Versions: 3.0.0-alpha3
>Reporter: Mandus Momberg
>Assignee: Mandus Momberg
>Priority: Minor
> Attachments: HADOOP-14661.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Amazon S3 has the ability to charge the requester for the cost of accessing
> S3. This is called Requester Pays Buckets.
> In order to access these buckets, each request needs to be signed with a
> specific header.
> http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[
https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234612#comment-17234612
]
Asier Arambarri Beldarrain commented on HADOOP-14661:
-
This is the solution for my case, regarding this error with S3A and Spark (but
I believe it can be replicated in other envs).
The key is the *fs.s3a.s3.client.factory.impl* properties' value. By default,
this value is set to *DefaultS3ClientFactory.*
So what's wrong with this factory? Well, it doesn't include any requester-pays
related header, as seen in its source code:
{color:#FF}[https://github.com/apache/hadoop/blob/e02b102/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java|https://github.com/apache/hadoop/blob/e02b102/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java].]{color}
{color:#172b4d}The solution is to implement a custom client factory extending
the default one but {color:#de350b}*adding the request-payer header*
{color:#172b4d}in the awsConfig{color}{color}.{color}
{color:#172b4d} ~public class RequestPayerS3ClientFactory extends
DefaultS3ClientFactory {~
~@Override~
~protected AmazonS3 newAmazonS3Client(AWSCredentialsProvider
credentials, ClientConfiguration awsConf)~
~{~
~awsConf.addHeader("x-amz-request-payer","requester");~
~return new AmazonS3Client(credentials, awsConf);~{color}
~{color:#172b4d}}
}{color}~
_{color:#172b4d}This is just a simplification, as you could also check if the
value is set to true before adding the header. This factory asssumes all
request will be payed by the requester if needed.{color}_
{color:#172b4d}Once you *compile the class and add it to your classpath*, set
the new hadoopConfiguration:{color}
*{color:#172b4d}~{color:#57d9a3}spark.sparkContext{color}.hadoopConfiguration.set("fs.s3a.s3.client.factory.impl",
"your.package.~~RequestPayerS3ClientFactory~ ~")~{color}*
{color:#172b4d}This way the S3 requests will call the overriden
newAmazonS3Client method, now including the x-amz-request-payer header.{color}
> S3A to support Requester Pays Buckets
> -
>
> Key: HADOOP-14661
> URL: https://issues.apache.org/jira/browse/HADOOP-14661
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common, util
>Affects Versions: 3.0.0-alpha3
>Reporter: Mandus Momberg
>Assignee: Mandus Momberg
>Priority: Minor
> Attachments: HADOOP-14661.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Amazon S3 has the ability to charge the requester for the cost of accessing
> S3. This is called Requester Pays Buckets.
> In order to access these buckets, each request needs to be signed with a
> specific header.
> http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[
https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044799#comment-17044799
]
Hadoop QA commented on HADOOP-14661:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HADOOP-14661 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14661 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877218/HADOOP-14661.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16770/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> S3A to support Requester Pays Buckets
> -
>
> Key: HADOOP-14661
> URL: https://issues.apache.org/jira/browse/HADOOP-14661
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common, util
>Affects Versions: 3.0.0-alpha3
>Reporter: Mandus Momberg
>Assignee: Mandus Momberg
>Priority: Minor
> Attachments: HADOOP-14661.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Amazon S3 has the ability to charge the requester for the cost of accessing
> S3. This is called Requester Pays Buckets.
> In order to access these buckets, each request needs to be signed with a
> specific header.
> http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916841#comment-16916841 ] Steve Loughran commented on HADOOP-14661: - delete needs this too; I'd always thought you weren't billed for deletions > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[
https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16858495#comment-16858495
]
Hadoop QA commented on HADOOP-14661:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HADOOP-14661 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14661 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877218/HADOOP-14661.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16302/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> S3A to support Requester Pays Buckets
> -
>
> Key: HADOOP-14661
> URL: https://issues.apache.org/jira/browse/HADOOP-14661
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common, util
>Affects Versions: 3.0.0-alpha3
>Reporter: Mandus Momberg
>Assignee: Mandus Momberg
>Priority: Minor
> Attachments: HADOOP-14661.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Amazon S3 has the ability to charge the requester for the cost of accessing
> S3. This is called Requester Pays Buckets.
> In order to access these buckets, each request needs to be signed with a
> specific header.
> http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16858489#comment-16858489 ] Steve Loughran commented on HADOOP-14661: - Looks like we need to set the requester pays flag on all operations, and the SDK doesn't explicitly let you do it yet...we'd need to add it for all the paged list calls by explicitly setting the header. > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[
https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714013#comment-16714013
]
Hadoop QA commented on HADOOP-14661:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HADOOP-14661 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14661 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877218/HADOOP-14661.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15621/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> S3A to support Requester Pays Buckets
> -
>
> Key: HADOOP-14661
> URL: https://issues.apache.org/jira/browse/HADOOP-14661
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common, util
>Affects Versions: 3.0.0-alpha3
>Reporter: Mandus Momberg
>Assignee: Mandus Momberg
>Priority: Minor
> Attachments: HADOOP-14661.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Amazon S3 has the ability to charge the requester for the cost of accessing
> S3. This is called Requester Pays Buckets.
> In order to access these buckets, each request needs to be signed with a
> specific header.
> http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714011#comment-16714011 ] Steve Loughran commented on HADOOP-14661: - Still want to get this in, still hoping someone adds tests. I think some of the common crawl data is requester pays Here's the help I need # Someone needs to write a test to try and read requester-pays data, *before this patch goes in* # Error message you get is noted, added to troubleshooting.md, and we make sure that the S3ARetryPolicy does not attempt to retry here # then apply this patch, verify test # then add an extended test: seek around a requester pays file, first to EOF-1, read a byte, then seek to 0 and read another byte. This is needed to verify that GET requests triggered within in an input stream always sets the header > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582870#comment-16582870 ] Steve Loughran commented on HADOOP-14661: - I'm not giving up on this BTW, just thinking of best way to test. Some datasets are requester pays, all we should need to do is # add a test config option identifying a non-zero requester pays path (if empty: skip test) # open it, read a byte, close it correct? > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[
https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570975#comment-16570975
]
genericqa commented on HADOOP-14661:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-14661 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14661 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877218/HADOOP-14661.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14995/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3A to support Requester Pays Buckets
> -
>
> Key: HADOOP-14661
> URL: https://issues.apache.org/jira/browse/HADOOP-14661
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common, util
>Affects Versions: 3.0.0-alpha3
>Reporter: Mandus Momberg
>Assignee: Mandus Momberg
>Priority: Minor
> Attachments: HADOOP-14661.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Amazon S3 has the ability to charge the requester for the cost of accessing
> S3. This is called Requester Pays Buckets.
> In order to access these buckets, each request needs to be signed with a
> specific header.
> http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[
https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360887#comment-16360887
]
genericqa commented on HADOOP-14661:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-14661 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14661 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877218/HADOOP-14661.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14101/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3A to support Requester Pays Buckets
> -
>
> Key: HADOOP-14661
> URL: https://issues.apache.org/jira/browse/HADOOP-14661
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common, util
>Affects Versions: 3.0.0-alpha3
>Reporter: Mandus Momberg
>Assignee: Mandus Momberg
>Priority: Minor
> Attachments: HADOOP-14661.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Amazon S3 has the ability to charge the requester for the cost of accessing
> S3. This is called Requester Pays Buckets.
> In order to access these buckets, each request needs to be signed with a
> specific header.
> http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[
https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184099#comment-16184099
]
Hadoop QA commented on HADOOP-14661:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-14661 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14661 |
| GITHUB PR | https://github.com/apache/hadoop/pull/250 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13396/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3A to support Requester Pays Buckets
> -
>
> Key: HADOOP-14661
> URL: https://issues.apache.org/jira/browse/HADOOP-14661
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common, util
>Affects Versions: 3.0.0-alpha3
>Reporter: Mandus Momberg
>Assignee: Mandus Momberg
>Priority: Minor
> Attachments: HADOOP-14661.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Amazon S3 has the ability to charge the requester for the cost of accessing
> S3. This is called Requester Pays Buckets.
> In order to access these buckets, each request needs to be signed with a
> specific header.
> http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets using
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160041#comment-16160041 ] Steve Loughran commented on HADOOP-14661: - no worries > S3A to support Requester Pays Buckets using > --- > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets using
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159754#comment-16159754 ] Mandus Momberg commented on HADOOP-14661: - [~steve_l], I doubt I'll be able to hit the deadline for 3.0, my apologies. I have been extremely busy at work and have not had time to add the additional tests and clean up the code. > S3A to support Requester Pays Buckets using > --- > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets using
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156748#comment-16156748 ] Steve Loughran commented on HADOOP-14661: - [~mandusm] have you had a chance to look at this...we're in the closedown phase for Hadoop 3.0 but there's still a day or two to get this in > S3A to support Requester Pays Buckets using > --- > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets using
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111473#comment-16111473 ] Mandus Momberg commented on HADOOP-14661: - Cool. I'll do all of this. I've been tied up with a bunch of stuff, so haven't been able to make the changes. Should be able to do this in a week or two. > S3A to support Requester Pays Buckets using > --- > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets using
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110804#comment-16110804 ] Steve Loughran commented on HADOOP-14661: - + a mention in the s3a docs > S3A to support Requester Pays Buckets using > --- > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
