[jira] [Comment Edited] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189527#comment-16189527 ] Steve Loughran edited comment on HADOOP-14220 at 10/3/17 10:37 AM: --- patch 016; patch 015 backported to branch-2. Usual test tweaks: closures -> callable, final references to variables tested: s3a ireland, dynamodb, w/ SSE-KMS to round things out was (Author: ste...@apache.org): patch 016; patch 015 backported to branch-2. Usual test tweaks: closures -> callable, final references to variables > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14220-006.patch, HADOOP-14220-008.patch, > HADOOP-14220-009.patch, HADOOP-14220-010.patch, HADOOP-14220-012.patch, > HADOOP-14220-013.patch, HADOOP-14220-013.patch, HADOOP-14220-014.patch, > HADOOP-14220-015.patch, HADOOP-14220-016.patch, > HADOOP-14220-branch-2-016.patch, HADOOP-14220-HADOOP-13345-001.patch, > HADOOP-14220-HADOOP-13345-002.patch, HADOOP-14220-HADOOP-13345-003.patch, > HADOOP-14220-HADOOP-13345-004.patch, HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168258#comment-16168258 ] Aaron Fabbri edited comment on HADOOP-14220 at 9/15/17 5:53 PM: Thanks for the update. I will try to test and review this today. was (Author: fabbri): Thanks for an update. I will try to test and review this today. > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14220-006.patch, HADOOP-14220-008.patch, > HADOOP-14220-009.patch, HADOOP-14220-010.patch, HADOOP-14220-012.patch, > HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch, > HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch, > HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156031#comment-16156031 ] Steve Loughran edited comment on HADOOP-14220 at 9/7/17 3:25 PM: - bq. On that topic, is there a way to mark CLI interface as unstable? mention it in the docs :) you are right about the troubleshooting docs, I'll do that. But only after I've got the doc reorg of -HADOOP-14220- in.(correction HADOOP-14738) I'll be adding some more CLI options related to the committer next, I think I'll want to list & drop multipart uploads under a path, maybe do some more low-levelness. And at some point soon we'll need to think of a "hadoop cloudstore" CLI entry point for the stores & options related to them, such as my HADOOP-14766 util was (Author: ste...@apache.org): bq. On that topic, is there a way to mark CLI interface as unstable? mention it in the docs :) you are right about the troubleshooting docs, I'll do that. But only after I've got the doc reorg of HADOOP-14220 in. I'll be adding some more CLI options related to the committer next, I think I'll want to list & drop multipart uploads under a path, maybe do some more low-levelness. And at some point soon we'll need to think of a "hadoop cloudstore" CLI entry point for the stores & options related to them, such as my HADOOP-14766 util > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14220-006.patch, > HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch, > HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch, > HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[
https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156001#comment-16156001
]
Aaron Fabbri edited comment on HADOOP-14220 at 9/6/17 8:38 PM:
---
Great stuff [~ste...@apache.org]. Thanks for doing this. I'm +1 on this patch
(non-binding) assuming yetus is happy. Minor suggestions below that you could
roll in before committing (comment typo, maybe add sentence to main site docs).
Reading the patch.
{noformat}
+ /**
+ * Build the exception to raise on a bad store/bucket staet
{noformat}
typo at end.
{noformat}
+hadoop s3guard bucket-info [ -guarded ] [-unguarded] [-auth] [-nonauth]
[-encryption ENCRYPTION] s3a://BUCKET
{noformat}
Yeah, I'd wanted us to use "s3a" instead of "s3guard" for the entry point
originally. There seems to be a good deal of overhead to doing a new hadoop
subcommand though. On that topic, is there a way to mark CLI interface as
unstable?
Overall I'm ok with "s3guard tool has some options that are useful even for
non-guarded buckets". May want to add a hint in the main s3a index.md /
troubleshooting to "see the bucket-info s3guard command"?
I can test this stuff out a little after I finish the other patch I'm testing,
if you like.
was (Author: fabbri):
Great stuff [~ste...@apache.org]. Thanks for doing this. I'm +1 on this patch
(non-binding). Minor suggestions below that you could roll in before
committing (comment typo, maybe add sentence to main site docs).
Reading the patch.
{noformat}
+ /**
+ * Build the exception to raise on a bad store/bucket staet
{noformat}
typo at end.
{noformat}
+hadoop s3guard bucket-info [ -guarded ] [-unguarded] [-auth] [-nonauth]
[-encryption ENCRYPTION] s3a://BUCKET
{noformat}
Yeah, I'd wanted us to use "s3a" instead of "s3guard" for the entry point
originally. There seems to be a good deal of overhead to doing a new hadoop
subcommand though. On that topic, is there a way to mark CLI interface as
unstable?
Overall I'm ok with "s3guard tool has some options that are useful even for
non-guarded buckets". May want to add a hint in the main s3a index.md /
troubleshooting to "see the bucket-info s3guard command"?
I can test this stuff out a little after I finish the other patch I'm testing,
if you like.
> Enhance S3GuardTool with bucket-info and set-capacity commands, tests
> -
>
> Key: HADOOP-14220
> URL: https://issues.apache.org/jira/browse/HADOOP-14220
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14220-006.patch,
> HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch,
> HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch,
> HADOOP-14220-HADOOP-13345-005.patch
>
>
> Add a diagnostics command to s3guard which does whatever we need to diagnose
> problems for a specific (named) s3a url. This is something which can be
> attached to bug reports as well as used by developers.
> * Properties to log (with provenance attribute, which can track bucket
> overrides: s3guard metastore setup, autocreate, capacity,
> * table present/absent
> * # of keys in DDB table for that bucket?
> * any other stats?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[
https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156001#comment-16156001
]
Aaron Fabbri edited comment on HADOOP-14220 at 9/6/17 8:37 PM:
---
Great stuff [~ste...@apache.org]. Thanks for doing this. I'm +1 on this patch
(non-binding). Minor suggestions below that you could roll in before
committing (comment typo, maybe add sentence to main site docs).
Reading the patch.
{noformat}
+ /**
+ * Build the exception to raise on a bad store/bucket staet
{noformat}
typo at end.
{noformat}
+hadoop s3guard bucket-info [ -guarded ] [-unguarded] [-auth] [-nonauth]
[-encryption ENCRYPTION] s3a://BUCKET
{noformat}
Yeah, I'd wanted us to use "s3a" instead of "s3guard" for the entry point
originally. There seems to be a good deal of overhead to doing a new hadoop
subcommand though. On that topic, is there a way to mark CLI interface as
unstable?
Overall I'm ok with "s3guard tool has some options that are useful even for
non-guarded buckets". May want to add a hint in the main s3a index.md /
troubleshooting to "see the bucket-info s3guard command"?
I can test this stuff out a little after I finish the other patch I'm testing,
if you like.
was (Author: fabbri):
Great stuff [~ste...@apache.org]. Thanks for doing this. I'm +1 on this patch
(non-binding). Minor suggestions below that you could roll in before
committing (comment typo, maybe add sentence to main site docs).
Reading the patch.
{noformat}
+ /**
+ * Build the exception to raise on a bad store/bucket staet
{noformat}
{noformat}
+hadoop s3guard bucket-info [ -guarded ] [-unguarded] [-auth] [-nonauth]
[-encryption ENCRYPTION] s3a://BUCKET
{noformat}
Yeah, I'd wanted us to use "s3a" instead of "s3guard" for the entry point
originally. There seems to be a good deal of overhead to doing a new hadoop
subcommand though. On that topic, is there a way to mark CLI interface as
unstable?
Overall I'm ok with "s3guard tool has some options that are useful even for
non-guarded buckets". May want to add a hint in the main s3a index.md /
troubleshooting to "see the bucket-info s3guard command"?
I can test this stuff out a little after I finish the other patch I'm testing,
if you like.
> Enhance S3GuardTool with bucket-info and set-capacity commands, tests
> -
>
> Key: HADOOP-14220
> URL: https://issues.apache.org/jira/browse/HADOOP-14220
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14220-006.patch,
> HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch,
> HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch,
> HADOOP-14220-HADOOP-13345-005.patch
>
>
> Add a diagnostics command to s3guard which does whatever we need to diagnose
> problems for a specific (named) s3a url. This is something which can be
> attached to bug reports as well as used by developers.
> * Properties to log (with provenance attribute, which can track bucket
> overrides: s3guard metastore setup, autocreate, capacity,
> * table present/absent
> * # of keys in DDB table for that bucket?
> * any other stats?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
