[
https://issues.apache.org/jira/browse/HADOOP-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16247399#comment-16247399
]
Genmao Yu commented on HADOOP-14098:
------------------------------------
h3. AliyunOSS FS + metastore POC
1. metastore: a common module of {{MetadataStore}} interface to manager
metadata of OSS object. Other filesystem or object store can manager their
metadata based on specific implementation of metastore
2. In this POC, I implemented one common metastore based on HDFS, i.e. we use
HDFS to manager metadata of object store. One advantage is the semantic
consistency, we can use HDFS backened metastore to manage metadata just like
use HDFS. Another advantage is this HDFS based metastore is platform
independent, i.e. we can use it to manager any object store metadata.
h4. Goals:
- Test if metastore can accelerate the performance of metadata operation.
h4. Non-Goals:
- This test is not a pressure test of cluster. The test result does not
indicate any platform or product performance.
| operation|unit|oss fs + metastore|oss fs|hdfs|
|CreateOp| | | | |
| |operations/sec|0.243|0.252|1.84|
| |successes/sec|0.152|0.161|0.324|
|DeleteOp| | | | |
| |operations/sec|52.479|43.499|328.515|
| |successes/sec|19.438|16.947|162.943|
|ListOp| | | | |
| |directory entries/sec|303.461|51.949|386.441|
| |operations/sec|250.05|37.191|235.405|
| |successes/sec|247.85|37.043|233.898|
|MkdirOp| | | | |
| |operations/sec|227.853|73.738|225.388|
| |successes/sec|227.853|73.738|225.388|
|ReadOp| | | | |
| |operations/sec|2.039|1.606|47.426|
| |successes/sec|0.127|0.132|6.222|
|RenameOp| | | | |
| |operations/sec|2.542|2.986|578.972|
| |successes/sec|0.578|0.507|163.965|
|AppendOp| |No Support|No Support| |
With metastore, we can find：
- {{ListOp}} and {{MkdirOp}} have a similar performance with HDFS.
- Others like {{RenameOp}} {{ReadOp}} {{DeleteOp}} still are much worse than
HDFS.
> AliyunOSS: improve the performance of object metadata operation
> ---------------------------------------------------------------
>
> Key: HADOOP-14098
> URL: https://issues.apache.org/jira/browse/HADOOP-14098
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
> Affects Versions: 3.0.0-alpha2
> Reporter: Genmao Yu
> Assignee: Genmao Yu
>
> Open this JIRA to research and address the potential request performance
> issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
