[
https://issues.apache.org/jira/browse/HADOOP-17347?focusedWorklogId=519577&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-519577
]
ASF GitHub Bot logged work on HADOOP-17347:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 03/Dec/20 11:49
Start Date: 03/Dec/20 11:49
Worklog Time Spent: 10m
Work Description: snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r535129578
##########
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -141,7 +154,7 @@ public synchronized int read(final byte[] b, final int off,
final int len) throw
return totalReadBytes > 0 ? totalReadBytes : lastReadBytes;
}
- private int readOneBlock(final byte[] b, final int off, final int len)
throws IOException {
+ private int validate(byte[] b, int off, int len) throws IOException {
Review comment:
Returning int from validate which is used by caller as bytes read is
wrong. Any failure in validate should return Boolean false and the caller in
turn can convert it to -1 for bytes read.
##########
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -517,6 +527,14 @@ public int getWriteBufferSize() {
return this.writeBufferSize;
}
+ public boolean readSmallFilesCompletely() {
Review comment:
In that case, skipping readAhead wont be ideal when reads are done by
means of this optimizations. if 100 MB is set as buffer size, current logic is
forcing a single server read request for 100 MB.
##########
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -161,6 +174,14 @@ private int readOneBlock(final byte[] b, final int off,
final int len) throws IO
if (off < 0 || len < 0 || len > b.length - off) {
throw new IndexOutOfBoundsException();
}
+ return 1; // 1 indicate success
+ }
+
+ private int readOneBlock(final byte[] b, final int off, final int len)
throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
Review comment:
see comment above
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 519577)
Time Spent: 1h 40m (was: 1.5h)
> ABFS: Read optimizations
> ------------------------
>
> Key: HADOOP-17347
> URL: https://issues.apache.org/jira/browse/HADOOP-17347
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.4.0
> Reporter: Bilahari T H
> Assignee: Bilahari T H
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Optimize read performance for the following scenarios
> # Read small files completely
> Files that are of size smaller than the read buffer size can be considered
> as small files. In case of such files it would be better to read the full
> file into the AbfsInputStream buffer.
> # Read last block if the read is for footer
> If the read is for the last 8 bytes, read the full file.
> This will optimize reads for parquet files. [Parquet file
> format|https://www.ellicium.com/parquet-file-format-structure/]
> Both these optimizations will be present under configs as follows
> # fs.azure.read.smallfilescompletely
> # fs.azure.read.optimizefooterread
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
