[
https://issues.apache.org/jira/browse/HADOOP-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210796#comment-16210796
]
Steve Loughran commented on HADOOP-14098:
-----------------------------------------
My view?
# It's a major piece of work & you are going to need to recruit people to
collaborate on & review
# some of the metastore design was always intended to be neutral of back end,
but we didn't follow it all the way through
See what [~fabbri], [~mackrorysd], [~cnauroth] & [~liuml07] have to say on the
matter.
Based on the TCP-DS benchmarking we've done in the past, speeding up the read
pipeline through seek optimisation has the biggest benefits, then incremental
writes of data. metadata caching &c boosts query setup irrespective of
destination, while faster commiters stop the job commit process being an
O(data) thing. I'd make sure the read/write code was fast first. Take a look at
wasb's latest work there, where it works out from the seek patten whether you
want to switch to random IO or not: on the first backwards seek it changes mode
> AliyunOSS: improve the performance of object metadata operation
> ---------------------------------------------------------------
>
> Key: HADOOP-14098
> URL: https://issues.apache.org/jira/browse/HADOOP-14098
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
> Affects Versions: 3.0.0-alpha2
> Reporter: Genmao Yu
> Assignee: Genmao Yu
>
> Open this JIRA to research and address the potential request performance
> issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
