[
https://issues.apache.org/jira/browse/HADOOP-17038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212916#comment-17212916
]
Anoop Sam John commented on HADOOP-17038:
-----------------------------------------
New PR based on the suggestion from [[email protected]]. Using new openFile
API to disable buffered reads while preads. The API is marked
InterfaceStability.Unstable as of now. Will this be changed ? Thanks for the
suggestions.
Tests passed in Azure ADL Gen2 premium storage account in East US.
I have an HBase PE test results on a 3 node cluster. Will give that charts in
a while. We see 2x gains. Will give cluster details and hbase file details.
> Support disabling buffered reads in ABFS positional reads
> ---------------------------------------------------------
>
> Key: HADOOP-17038
> URL: https://issues.apache.org/jira/browse/HADOOP-17038
> Project: Hadoop Common
> Issue Type: Sub-task
> Reporter: Anoop Sam John
> Assignee: Anoop Sam John
> Priority: Major
> Labels: HBase, abfsactive, pull-request-available
> Attachments: HBase Perf Test Report.xlsx, screenshot-1.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Right now it will do a seek to the position , read and then seek back to the
> old position. (As per the impl in the super class)
> In HBase kind of workloads we rely mostly on short preads. (like 64 KB size
> by default). So would be ideal to support a pure pos read API which will not
> even keep the data in a buffer but will only read the required data as what
> is asked for by the caller. (Not reading ahead more data as per the read size
> config)
> Allow an optional boolean config to be specified while opening file for read
> using which buffered pread can be disabled.
> FutureDataInputStreamBuilder openFile(Path path)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
