[
https://issues.apache.org/jira/browse/HADOOP-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16256714#comment-16256714
]
Genmao Yu edited comment on HADOOP-15038 at 11/17/17 9:31 AM:
--------------------------------------------------------------
[~fabbri] Sorry for the late response, and thanks for your comments.
bq. I originally intended for this to be possible, just didn't want to pollute
public hadoop common until it was needed by another client.
What about move it to a separate module? like {{hadoop-metastore}} inside
{{hadoop-tools}}.
bq. It might be a good idea to use HADOOP-14098 as a top-level "umbrella" JIRA
for your effort, and create subtasks ...
Absolutely yes. Now I did these works offline and made a POC. As soon as I can
confirm the validity of metastore (call it this way first) for performance
improving, I will begin to provide this feature into Aliyun OSS fs. After all,
I want use metastore to improve performance of metadata operation. There are
still many blocks, like {{rename}}. It is not a metadata operation issue, but
can be done with metastore.
bq. Can you comment on when you will need this?
Maybe, separating and my work can be done in parallel. Is there any concern?
Besides, I can help you to do separating work.
was (Author: unclegen):
[~fabbri] Sorry for the late response, and thanks for your comments.
bq. I originally intended for this to be possible, just didn't want to pollute
public hadoop common until it was needed by another client.
What about move it to a separate module? like {{hadoop-metastore}} inside
{{hadoop-tools}}.
bq. It might be a good idea to use HADOOP-14098 as a top-level "umbrella" JIRA
for your effort, and create subtasks ...
Absolutely yes. Now I did these works offline and made a POC. As soon as I can
confirm the validity of metastore (call it this way first) for performance
improving. After all, I want use metastore to improve performance of metadata
operation. There are still many blocks, like {{rename}}. It is not a metadata
operation issue, but can be done with metastore.
bq. Can you comment on when you will need this?
Maybe, separating and my work can be done in parallel. Is there any concern?
Besides, I can help you to do separating work.
> Abstract MetadataStore in S3Guard into a common module.
> -------------------------------------------------------
>
> Key: HADOOP-15038
> URL: https://issues.apache.org/jira/browse/HADOOP-15038
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Affects Versions: 3.0.0-beta1
> Reporter: Genmao Yu
>
> Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}}
> into a common module.
> Based on this work, other filesystem or object store can implement their own
> metastore for optimization (known issues like consistency problem and
> metadata operation performance). [~ste...@apache.org] and other guys have
> done many base and great works in {{S3Guard}}. It is very helpful to start
> work. I did some perf test in HADOOP-14098, and started related work for
> Aliyun OSS. Indeed there are still works to do for {{S3Guard}}, like
> metadata cache inconsistent with S3 and so on. It also will be a problem for
> other object store. However, we can do these works in parallel.
> [~ste...@apache.org] [~fabbri] [~drankye] Any suggestion is appreciated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
