[
https://issues.apache.org/jira/browse/HADOOP-14965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218478#comment-16218478
]
Steve Loughran commented on HADOOP-14965:
-----------------------------------------
BTW, been testing on some downstream spark commit work. This policy is slightly
worse for columnar data than random IO in that it *may* add one extra abort()
call to the read, if the access pattern is open/seek(EOF-offset),
read(footer)...it depends on whether EOF-offset+len(footer) >
fs.s3a.readahead.range. However, once that backwards seek has happened and you
are in random IO, then its perf is inherently equivalent. And it delivers
comparable perf on sequential IO of .csv.gz, which is what highlit that an
adaptive policy is needed. If you have to declare upfront what the read policy
is for a s3 bucket for the life of the process, then you have to decide upfront
whether to go random or serial, when really it is driven by the data source
> s3a input stream "normal" fadvise mode to be adaptive
> -----------------------------------------------------
>
> Key: HADOOP-14965
> URL: https://issues.apache.org/jira/browse/HADOOP-14965
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-14965-001.patch
>
>
> HADOOP-14535 added seek optimisation to wasb, but rather than require the
> caller to declare sequential vs random, it works out for itself.
> # defaults to sequential, lazy seek
> # if the caller ever seeks backwards, switches to random IO.
> This means that on the use pattern of columnar stores: of go to end of file,
> read summary, then go to columns and work forwards, will switch to random IO
> after that first seek back (cost: one aborted HTTP connection)/.
> Where this should benefit the most is in downstream apps where you are
> working with different data sources in the same object store/running of the
> same app config, but have different read patterns. I'm seeing exactly this in
> some of my spark tests, where it's near impossible to set things up so that
> .gz files are read sequentially, but ORC data is read in random IO
> I propose the "normal" fadvise => adaptive, sequential==sequential always,
> random => random from the outset.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
