[
https://issues.apache.org/jira/browse/HADOOP-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705643#comment-15705643
]
Steve Loughran commented on HADOOP-13650:
-----------------------------------------
* {{S3GuardTool.main}} needs to catch, log all exceptions. Ideally return a
different exit code for an exception than from a usage command; this makes
testing/diags easier.
* be good to have ~aw to review the bash scripts. Are these testable? At the
very least, something that goes as far as a usage command in the java code can
do basic installation testing.
* we have the option of JCommander here for arg parsing. just a thought.
* might be good to have the option of printing the diff out in a way that's
easy to parse downstream. Is the current output machine parseable?
* For the comparison, a {{listFiles(recursive=true)}} is much faster to list s3
buckets...but I'm not sure how well it will work here, because the iterator
returned from the metastore may return differently ordered results from the FS;
you don't want to build up large structures in memory. Even so, I'm not sure
how well the existing code can handle a path with a few million blobs directly
under a path. some s3 buckets do have this, apparently.
* Maybe an operation to verify that the metastore is in sync with s3, return a
specific error code to indicate this is not the case. That's something you can
use at installation, preflight, etc.
> S3Guard: Provide command line tools to manipulate metadata store.
> -----------------------------------------------------------------
>
> Key: HADOOP-13650
> URL: https://issues.apache.org/jira/browse/HADOOP-13650
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Lei (Eddy) Xu
> Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-13650-HADOOP-13345.000.patch,
> HADOOP-13650-HADOOP-13345.001.patch, HADOOP-13650-HADOOP-13345.002.patch
>
>
> Similar systems like EMRFS has the CLI tools to manipulate the metadata
> store, i.e., create or delete metadata store, or {{import}}, {{sync}} the
> file metadata between metadata store and S3.
> http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emrfs-cli-reference.html
> S3Guard should offer similar functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
