[
https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=499775&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499775
]
ASF GitHub Bot logged work on HADOOP-17301:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 13/Oct/20 02:31
Start Date: 13/Oct/20 02:31
Worklog Time Spent: 10m
Work Description: snvijaya commented on a change in pull request #2369:
URL: https://github.com/apache/hadoop/pull/2369#discussion_r503631775
##########
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##########
@@ -182,7 +183,39 @@ public void testFailedReadAhead() throws Exception {
checkEvictedStatus(inputStream, 0, false);
}
+ @Test
+ public void testFailedReadAheadEviction() throws Exception {
+ AbfsClient client = getMockAbfsClient();
+ AbfsRestOperation successOp = getMockRestOp();
+
ReadBufferManager.setThresholdAgeMilliseconds(INCREASED_READ_BUFFER_AGE_THRESHOLD);
+ // Stub :
+ // Read request leads to 3 readahead calls: Fail all 3
readahead-client.read()
+ // Actual read request fails with the failure in readahead thread
+ doThrow(new TimeoutException("Internal Server error"))
+ .when(client)
+ .read(any(String.class), any(Long.class), any(byte[].class),
+ any(Integer.class), any(Integer.class), any(String.class),
+ any(String.class));
+
+ AbfsInputStream inputStream = getAbfsInputStream(client,
"testFailedReadAheadEviction.txt");
+
+ // Add a failed buffer to completed queue and set to no free buffers to
read ahead.
+ ReadBuffer buff = new ReadBuffer();
+ buff.setStatus(
+
org.apache.hadoop.fs.azurebfs.contracts.services.ReadBufferStatus.READ_FAILED);
Review comment:
Done
##########
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##########
@@ -264,12 +297,24 @@ public void testSuccessfulReadAhead() throws Exception {
any(String.class));
AbfsInputStream inputStream = getAbfsInputStream(client,
"testSuccessfulReadAhead.txt");
+ int beforeReadCompletedListSize =
ReadBufferManager.getBufferManager().getCompletedReadListSize();
// First read request that triggers readAheads.
inputStream.read(new byte[ONE_KB]);
// Only the 3 readAhead threads should have triggered client.read
verifyReadCallCount(client, 3);
+ int newAdditionsToCompletedRead =
+ ReadBufferManager.getBufferManager().getCompletedReadListSize()
+ - beforeReadCompletedListSize;
+ // read buffer might be dumped if the ReadBufferManager getblock preceded
+ // the action of buffer being picked for reading from readaheadqueue, so
that
+ // inputstream can proceed with read and not be blocked on readahead thread
+ // availability. So the count of buffers in completedReadQueue for the
stream
+ // can be same or lesser than the requests triggered to queue readahead.
+ assertTrue(
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 499775)
Time Spent: 2h 20m (was: 2h 10m)
> ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
> ----------------------------------------------------------------------------
>
> Key: HADOOP-17301
> URL: https://issues.apache.org/jira/browse/HADOOP-17301
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.3.0
> Reporter: Sneha Vijayarajan
> Assignee: Sneha Vijayarajan
> Priority: Critical
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> When reads done by readahead buffers failed, the exceptions where dropped and
> the failure was not getting reported to the calling app.
> Jira HADOOP-16852: Report read-ahead error back
> tried to handle the scenario by reporting the error back to calling app. But
> the commit has introduced a bug which can lead to ReadBuffer being injected
> into read completed queue twice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
