[
https://issues.apache.org/jira/browse/HADOOP-19229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-19229.
-------------------------------------
Release Note: This is exactly what `"fs.s3a.vectored.read.min.seek.size"
does. We set it to 4K; maybe we should review it. The facebook Velox paper says
that 20kB is better for cloud storage
Resolution: Fixed
> Vector IO: have a max distance between ranges to read
> -----------------------------------------------------
>
> Key: HADOOP-19229
> URL: https://issues.apache.org/jira/browse/HADOOP-19229
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.1
> Reporter: Steve Loughran
> Priority: Major
>
> vector iO has a max size to coalesce ranges, but it also needs a maximum gap
> between ranges to justify the merge. Right now we could have a read where two
> vectors of size 8 bytes can be merged with a 1 MB gap between them -and
> that's wasteful.
> We could also consider an "efficiency" metric which looks at the ratio of
> bytes-read to bytes-discarded. Not sure what we'd do with it, but we could
> track it as an IOStat
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
