[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[ https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=500116=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500116 ] ASF GitHub Bot logged work on HADOOP-17301: --- Author: ASF GitHub Bot Created on: 13/Oct/20 15:31 Start Date: 13/Oct/20 15:31 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2369: URL: https://github.com/apache/hadoop/pull/2369#issuecomment-707822816 +1, merged to trunk. @snvijaya -before I backport it, can you do a cherrypick of this into branch-3.3 and verify all the tests are good? thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 500116) Time Spent: 3h 20m (was: 3h 10m) > ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back > > > Key: HADOOP-17301 > URL: https://issues.apache.org/jira/browse/HADOOP-17301 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Critical > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > When reads done by readahead buffers failed, the exceptions where dropped and > the failure was not getting reported to the calling app. > Jira HADOOP-16852: Report read-ahead error back > tried to handle the scenario by reporting the error back to calling app. But > the commit has introduced a bug which can lead to ReadBuffer being injected > into read completed queue twice. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[ https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=500115=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500115 ] ASF GitHub Bot logged work on HADOOP-17301: --- Author: ASF GitHub Bot Created on: 13/Oct/20 15:30 Start Date: 13/Oct/20 15:30 Worklog Time Spent: 10m Work Description: steveloughran merged pull request #2369: URL: https://github.com/apache/hadoop/pull/2369 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 500115) Time Spent: 3h 10m (was: 3h) > ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back > > > Key: HADOOP-17301 > URL: https://issues.apache.org/jira/browse/HADOOP-17301 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Critical > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > When reads done by readahead buffers failed, the exceptions where dropped and > the failure was not getting reported to the calling app. > Jira HADOOP-16852: Report read-ahead error back > tried to handle the scenario by reporting the error back to calling app. But > the commit has introduced a bug which can lead to ReadBuffer being injected > into read completed queue twice. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[
https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=500114=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500114
]
ASF GitHub Bot logged work on HADOOP-17301:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 15:29
Start Date: 13/Oct/20 15:29
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #2369:
URL: https://github.com/apache/hadoop/pull/2369#issuecomment-705474826
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 21s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 20s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 1s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 56s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 71m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2369 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8b0a2d453311 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 735e85a1d20 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/2/testReport/ |
| Max. process+thread count | 419 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[ https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=499795=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499795 ] ASF GitHub Bot logged work on HADOOP-17301: --- Author: ASF GitHub Bot Created on: 13/Oct/20 03:52 Start Date: 13/Oct/20 03:52 Worklog Time Spent: 10m Work Description: snvijaya commented on pull request #2369: URL: https://github.com/apache/hadoop/pull/2369#issuecomment-707468028 @steveloughran - Thanks for your review. I have updated this PR with the suggestions. Kindly request your review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 499795) Time Spent: 2h 50m (was: 2h 40m) > ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back > > > Key: HADOOP-17301 > URL: https://issues.apache.org/jira/browse/HADOOP-17301 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Critical > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > When reads done by readahead buffers failed, the exceptions where dropped and > the failure was not getting reported to the calling app. > Jira HADOOP-16852: Report read-ahead error back > tried to handle the scenario by reporting the error back to calling app. But > the commit has introduced a bug which can lead to ReadBuffer being injected > into read completed queue twice. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[ https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=499794=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499794 ] ASF GitHub Bot logged work on HADOOP-17301: --- Author: ASF GitHub Bot Created on: 13/Oct/20 03:51 Start Date: 13/Oct/20 03:51 Worklog Time Spent: 10m Work Description: snvijaya commented on pull request #2369: URL: https://github.com/apache/hadoop/pull/2369#issuecomment-707467838 Test results from accounts on East US 2 region: ### NON-HNS: SharedKey: [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 245 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 ### HNS: SharedKey: [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 24 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 OAuth: [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 499794) Time Spent: 2h 40m (was: 2.5h) > ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back > > > Key: HADOOP-17301 > URL: https://issues.apache.org/jira/browse/HADOOP-17301 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Critical > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > When reads done by readahead buffers failed, the exceptions where dropped and > the failure was not getting reported to the calling app. > Jira HADOOP-16852: Report read-ahead error back > tried to handle the scenario by reporting the error back to calling app. But > the commit has introduced a bug which can lead to ReadBuffer being injected > into read completed queue twice. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[
https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=499790=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499790
]
ASF GitHub Bot logged work on HADOOP-17301:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 03:42
Start Date: 13/Oct/20 03:42
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2369:
URL: https://github.com/apache/hadoop/pull/2369#issuecomment-707465367
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 46s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 7s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 58s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 4s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 57s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 27s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 71m 41s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2369 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 61d0a03f4d37 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b3786d6c3cc |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/3/testReport/ |
| Max. process+thread count | 430 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/3/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.1.3 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[
https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=499773=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499773
]
ASF GitHub Bot logged work on HADOOP-17301:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 02:31
Start Date: 13/Oct/20 02:31
Worklog Time Spent: 10m
Work Description: snvijaya commented on a change in pull request #2369:
URL: https://github.com/apache/hadoop/pull/2369#discussion_r503631680
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java
##
@@ -242,13 +243,29 @@ private synchronized boolean tryEvict() {
}
// next, try any old nodes that have not been consumed
+// Failed read buffers (with buffer index=-1) that are older than
+// thresholdAge should be cleaned up, but at the same time should not
+// report successful eviction.
+// Queue logic expects that a buffer is freed up for read ahead when
+// eviction is successful, whereas a failed ReadBuffer would have released
+// its buffer when its status was set to READ_FAILED.
long earliestBirthday = Long.MAX_VALUE;
+ArrayList oldFailedBuffers = new ArrayList<>();
for (ReadBuffer buf : completedReadList) {
- if (buf.getTimeStamp() < earliestBirthday) {
+ if ((buf.getBufferindex() != -1)
+ && (buf.getTimeStamp() < earliestBirthday)) {
nodeToEvict = buf;
earliestBirthday = buf.getTimeStamp();
+ } else if ((buf.getBufferindex() == -1)
+ && (currentTimeMillis() - buf.getTimeStamp()) >
thresholdAgeMilliseconds) {
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 499773)
Time Spent: 2h (was: 1h 50m)
> ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
>
>
> Key: HADOOP-17301
> URL: https://issues.apache.org/jira/browse/HADOOP-17301
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> When reads done by readahead buffers failed, the exceptions where dropped and
> the failure was not getting reported to the calling app.
> Jira HADOOP-16852: Report read-ahead error back
> tried to handle the scenario by reporting the error back to calling app. But
> the commit has introduced a bug which can lead to ReadBuffer being injected
> into read completed queue twice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[
https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=499774=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499774
]
ASF GitHub Bot logged work on HADOOP-17301:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 02:31
Start Date: 13/Oct/20 02:31
Worklog Time Spent: 10m
Work Description: snvijaya commented on a change in pull request #2369:
URL: https://github.com/apache/hadoop/pull/2369#discussion_r503631736
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java
##
@@ -464,4 +480,10 @@ int getCompletedReadListSize() {
void callTryEvict() {
tryEvict();
}
+
+ @VisibleForTesting
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 499774)
Time Spent: 2h 10m (was: 2h)
> ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
>
>
> Key: HADOOP-17301
> URL: https://issues.apache.org/jira/browse/HADOOP-17301
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> When reads done by readahead buffers failed, the exceptions where dropped and
> the failure was not getting reported to the calling app.
> Jira HADOOP-16852: Report read-ahead error back
> tried to handle the scenario by reporting the error back to calling app. But
> the commit has introduced a bug which can lead to ReadBuffer being injected
> into read completed queue twice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[
https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=499775=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499775
]
ASF GitHub Bot logged work on HADOOP-17301:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 02:31
Start Date: 13/Oct/20 02:31
Worklog Time Spent: 10m
Work Description: snvijaya commented on a change in pull request #2369:
URL: https://github.com/apache/hadoop/pull/2369#discussion_r503631775
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -182,7 +183,39 @@ public void testFailedReadAhead() throws Exception {
checkEvictedStatus(inputStream, 0, false);
}
+ @Test
+ public void testFailedReadAheadEviction() throws Exception {
+AbfsClient client = getMockAbfsClient();
+AbfsRestOperation successOp = getMockRestOp();
+
ReadBufferManager.setThresholdAgeMilliseconds(INCREASED_READ_BUFFER_AGE_THRESHOLD);
+// Stub :
+// Read request leads to 3 readahead calls: Fail all 3
readahead-client.read()
+// Actual read request fails with the failure in readahead thread
+doThrow(new TimeoutException("Internal Server error"))
+.when(client)
+.read(any(String.class), any(Long.class), any(byte[].class),
+any(Integer.class), any(Integer.class), any(String.class),
+any(String.class));
+
+AbfsInputStream inputStream = getAbfsInputStream(client,
"testFailedReadAheadEviction.txt");
+
+// Add a failed buffer to completed queue and set to no free buffers to
read ahead.
+ReadBuffer buff = new ReadBuffer();
+buff.setStatus(
+
org.apache.hadoop.fs.azurebfs.contracts.services.ReadBufferStatus.READ_FAILED);
Review comment:
Done
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -264,12 +297,24 @@ public void testSuccessfulReadAhead() throws Exception {
any(String.class));
AbfsInputStream inputStream = getAbfsInputStream(client,
"testSuccessfulReadAhead.txt");
+int beforeReadCompletedListSize =
ReadBufferManager.getBufferManager().getCompletedReadListSize();
// First read request that triggers readAheads.
inputStream.read(new byte[ONE_KB]);
// Only the 3 readAhead threads should have triggered client.read
verifyReadCallCount(client, 3);
+int newAdditionsToCompletedRead =
+ReadBufferManager.getBufferManager().getCompletedReadListSize()
+- beforeReadCompletedListSize;
+// read buffer might be dumped if the ReadBufferManager getblock preceded
+// the action of buffer being picked for reading from readaheadqueue, so
that
+// inputstream can proceed with read and not be blocked on readahead thread
+// availability. So the count of buffers in completedReadQueue for the
stream
+// can be same or lesser than the requests triggered to queue readahead.
+assertTrue(
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 499775)
Time Spent: 2h 20m (was: 2h 10m)
> ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
>
>
> Key: HADOOP-17301
> URL: https://issues.apache.org/jira/browse/HADOOP-17301
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> When reads done by readahead buffers failed, the exceptions where dropped and
> the failure was not getting reported to the calling app.
> Jira HADOOP-16852: Report read-ahead error back
> tried to handle the scenario by reporting the error back to calling app. But
> the commit has introduced a bug which can lead to ReadBuffer being injected
> into read completed queue twice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[
https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=499772=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499772
]
ASF GitHub Bot logged work on HADOOP-17301:
---
Author: ASF GitHub Bot
Created on: 13/Oct/20 02:30
Start Date: 13/Oct/20 02:30
Worklog Time Spent: 10m
Work Description: snvijaya commented on a change in pull request #2369:
URL: https://github.com/apache/hadoop/pull/2369#discussion_r503631485
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java
##
@@ -464,4 +480,10 @@ int getCompletedReadListSize() {
void callTryEvict() {
tryEvict();
}
+
+ @VisibleForTesting
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 499772)
Time Spent: 1h 50m (was: 1h 40m)
> ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
>
>
> Key: HADOOP-17301
> URL: https://issues.apache.org/jira/browse/HADOOP-17301
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> When reads done by readahead buffers failed, the exceptions where dropped and
> the failure was not getting reported to the calling app.
> Jira HADOOP-16852: Report read-ahead error back
> tried to handle the scenario by reporting the error back to calling app. But
> the commit has introduced a bug which can lead to ReadBuffer being injected
> into read completed queue twice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[ https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=498266=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-498266 ] ASF GitHub Bot logged work on HADOOP-17301: --- Author: ASF GitHub Bot Created on: 09/Oct/20 13:53 Start Date: 09/Oct/20 13:53 Worklog Time Spent: 10m Work Description: snvijaya commented on pull request #2369: URL: https://github.com/apache/hadoop/pull/2369#issuecomment-705436858 Tests were run on accounts on East US 2 region. ### HNS-OAuth [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 ### HNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 24 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 ### NonHNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 245 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 498266) Time Spent: 1h 40m (was: 1.5h) > ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back > > > Key: HADOOP-17301 > URL: https://issues.apache.org/jira/browse/HADOOP-17301 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Critical > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > When reads done by readahead buffers failed, the exceptions where dropped and > the failure was not getting reported to the calling app. > Jira HADOOP-16852: Report read-ahead error back > tried to handle the scenario by reporting the error back to calling app. But > the commit has introduced a bug which can lead to ReadBuffer being injected > into read completed queue twice. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[
https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=498237=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-498237
]
ASF GitHub Bot logged work on HADOOP-17301:
---
Author: ASF GitHub Bot
Created on: 09/Oct/20 13:51
Start Date: 09/Oct/20 13:51
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #2369:
URL: https://github.com/apache/hadoop/pull/2369#issuecomment-705303355
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 29m 10s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 31s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 26s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 10s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 6s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| -0 :warning: | checkstyle | 0m 20s |
[/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 4 new + 0 unchanged - 0
fixed = 4 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/1/artifact/out/whitespace-tabs.txt)
| The patch 3 line(s) with tabs. |
| +1 :green_heart: | shadedclient | 17m 31s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 11s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 34s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 117m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2369 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a8cdbf86d0f6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / df4006eb813 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[
https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=498094=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-498094
]
ASF GitHub Bot logged work on HADOOP-17301:
---
Author: ASF GitHub Bot
Created on: 09/Oct/20 13:39
Start Date: 09/Oct/20 13:39
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2369:
URL: https://github.com/apache/hadoop/pull/2369#issuecomment-705474826
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 21s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 20s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 1s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 56s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 71m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2369 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8b0a2d453311 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 735e85a1d20 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/2/testReport/ |
| Max. process+thread count | 419 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[
https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=498000=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-498000
]
ASF GitHub Bot logged work on HADOOP-17301:
---
Author: ASF GitHub Bot
Created on: 09/Oct/20 13:26
Start Date: 09/Oct/20 13:26
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#2369:
URL: https://github.com/apache/hadoop/pull/2369#discussion_r501695425
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java
##
@@ -242,13 +243,29 @@ private synchronized boolean tryEvict() {
}
// next, try any old nodes that have not been consumed
+// Failed read buffers (with buffer index=-1) that are older than
+// thresholdAge should be cleaned up, but at the same time should not
+// report successful eviction.
+// Queue logic expects that a buffer is freed up for read ahead when
+// eviction is successful, whereas a failed ReadBuffer would have released
+// its buffer when its status was set to READ_FAILED.
long earliestBirthday = Long.MAX_VALUE;
+ArrayList oldFailedBuffers = new ArrayList<>();
for (ReadBuffer buf : completedReadList) {
- if (buf.getTimeStamp() < earliestBirthday) {
+ if ((buf.getBufferindex() != -1)
+ && (buf.getTimeStamp() < earliestBirthday)) {
nodeToEvict = buf;
earliestBirthday = buf.getTimeStamp();
+ } else if ((buf.getBufferindex() == -1)
+ && (currentTimeMillis() - buf.getTimeStamp()) >
thresholdAgeMilliseconds) {
Review comment:
pull `currentTimeMillis()` outside the for loop as its an OS call with
potential cost, and things probably work best if the same value is used through
the loop and the code at L269
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -264,12 +297,24 @@ public void testSuccessfulReadAhead() throws Exception {
any(String.class));
AbfsInputStream inputStream = getAbfsInputStream(client,
"testSuccessfulReadAhead.txt");
+int beforeReadCompletedListSize =
ReadBufferManager.getBufferManager().getCompletedReadListSize();
// First read request that triggers readAheads.
inputStream.read(new byte[ONE_KB]);
// Only the 3 readAhead threads should have triggered client.read
verifyReadCallCount(client, 3);
+int newAdditionsToCompletedRead =
+ReadBufferManager.getBufferManager().getCompletedReadListSize()
+- beforeReadCompletedListSize;
+// read buffer might be dumped if the ReadBufferManager getblock preceded
+// the action of buffer being picked for reading from readaheadqueue, so
that
+// inputstream can proceed with read and not be blocked on readahead thread
+// availability. So the count of buffers in completedReadQueue for the
stream
+// can be same or lesser than the requests triggered to queue readahead.
+assertTrue(
Review comment:
use Assertions.assertThat with an explicit `isLessThanOrEqualTo(3)`
assertion.
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -182,7 +183,39 @@ public void testFailedReadAhead() throws Exception {
checkEvictedStatus(inputStream, 0, false);
}
+ @Test
+ public void testFailedReadAheadEviction() throws Exception {
+AbfsClient client = getMockAbfsClient();
+AbfsRestOperation successOp = getMockRestOp();
+
ReadBufferManager.setThresholdAgeMilliseconds(INCREASED_READ_BUFFER_AGE_THRESHOLD);
+// Stub :
+// Read request leads to 3 readahead calls: Fail all 3
readahead-client.read()
+// Actual read request fails with the failure in readahead thread
+doThrow(new TimeoutException("Internal Server error"))
+.when(client)
+.read(any(String.class), any(Long.class), any(byte[].class),
+any(Integer.class), any(Integer.class), any(String.class),
+any(String.class));
+
+AbfsInputStream inputStream = getAbfsInputStream(client,
"testFailedReadAheadEviction.txt");
+
+// Add a failed buffer to completed queue and set to no free buffers to
read ahead.
+ReadBuffer buff = new ReadBuffer();
+buff.setStatus(
+
org.apache.hadoop.fs.azurebfs.contracts.services.ReadBufferStatus.READ_FAILED);
Review comment:
import the field rather than a full reference
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java
##
@@ -464,4 +480,10 @@ int getCompletedReadListSize() {
void callTryEvict() {
tryEvict();
}
+
+ @VisibleForTesting
Review comment:
add (minimal) javadoc
##
File path:
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[
https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=497403=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-497403
]
ASF GitHub Bot logged work on HADOOP-17301:
---
Author: ASF GitHub Bot
Created on: 08/Oct/20 15:15
Start Date: 08/Oct/20 15:15
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#2369:
URL: https://github.com/apache/hadoop/pull/2369#discussion_r501695425
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java
##
@@ -242,13 +243,29 @@ private synchronized boolean tryEvict() {
}
// next, try any old nodes that have not been consumed
+// Failed read buffers (with buffer index=-1) that are older than
+// thresholdAge should be cleaned up, but at the same time should not
+// report successful eviction.
+// Queue logic expects that a buffer is freed up for read ahead when
+// eviction is successful, whereas a failed ReadBuffer would have released
+// its buffer when its status was set to READ_FAILED.
long earliestBirthday = Long.MAX_VALUE;
+ArrayList oldFailedBuffers = new ArrayList<>();
for (ReadBuffer buf : completedReadList) {
- if (buf.getTimeStamp() < earliestBirthday) {
+ if ((buf.getBufferindex() != -1)
+ && (buf.getTimeStamp() < earliestBirthday)) {
nodeToEvict = buf;
earliestBirthday = buf.getTimeStamp();
+ } else if ((buf.getBufferindex() == -1)
+ && (currentTimeMillis() - buf.getTimeStamp()) >
thresholdAgeMilliseconds) {
Review comment:
pull `currentTimeMillis()` outside the for loop as its an OS call with
potential cost, and things probably work best if the same value is used through
the loop and the code at L269
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -264,12 +297,24 @@ public void testSuccessfulReadAhead() throws Exception {
any(String.class));
AbfsInputStream inputStream = getAbfsInputStream(client,
"testSuccessfulReadAhead.txt");
+int beforeReadCompletedListSize =
ReadBufferManager.getBufferManager().getCompletedReadListSize();
// First read request that triggers readAheads.
inputStream.read(new byte[ONE_KB]);
// Only the 3 readAhead threads should have triggered client.read
verifyReadCallCount(client, 3);
+int newAdditionsToCompletedRead =
+ReadBufferManager.getBufferManager().getCompletedReadListSize()
+- beforeReadCompletedListSize;
+// read buffer might be dumped if the ReadBufferManager getblock preceded
+// the action of buffer being picked for reading from readaheadqueue, so
that
+// inputstream can proceed with read and not be blocked on readahead thread
+// availability. So the count of buffers in completedReadQueue for the
stream
+// can be same or lesser than the requests triggered to queue readahead.
+assertTrue(
Review comment:
use Assertions.assertThat with an explicit `isLessThanOrEqualTo(3)`
assertion.
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -182,7 +183,39 @@ public void testFailedReadAhead() throws Exception {
checkEvictedStatus(inputStream, 0, false);
}
+ @Test
+ public void testFailedReadAheadEviction() throws Exception {
+AbfsClient client = getMockAbfsClient();
+AbfsRestOperation successOp = getMockRestOp();
+
ReadBufferManager.setThresholdAgeMilliseconds(INCREASED_READ_BUFFER_AGE_THRESHOLD);
+// Stub :
+// Read request leads to 3 readahead calls: Fail all 3
readahead-client.read()
+// Actual read request fails with the failure in readahead thread
+doThrow(new TimeoutException("Internal Server error"))
+.when(client)
+.read(any(String.class), any(Long.class), any(byte[].class),
+any(Integer.class), any(Integer.class), any(String.class),
+any(String.class));
+
+AbfsInputStream inputStream = getAbfsInputStream(client,
"testFailedReadAheadEviction.txt");
+
+// Add a failed buffer to completed queue and set to no free buffers to
read ahead.
+ReadBuffer buff = new ReadBuffer();
+buff.setStatus(
+
org.apache.hadoop.fs.azurebfs.contracts.services.ReadBufferStatus.READ_FAILED);
Review comment:
import the field rather than a full reference
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java
##
@@ -464,4 +480,10 @@ int getCompletedReadListSize() {
void callTryEvict() {
tryEvict();
}
+
+ @VisibleForTesting
Review comment:
add (minimal) javadoc
##
File path:
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[
https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=497318=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-497318
]
ASF GitHub Bot logged work on HADOOP-17301:
---
Author: ASF GitHub Bot
Created on: 08/Oct/20 13:01
Start Date: 08/Oct/20 13:01
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #2369:
URL: https://github.com/apache/hadoop/pull/2369#issuecomment-705303355
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 29m 10s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 31s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 26s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 10s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 6s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| -0 :warning: | checkstyle | 0m 20s |
[/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 4 new + 0 unchanged - 0
fixed = 4 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/1/artifact/out/whitespace-tabs.txt)
| The patch 3 line(s) with tabs. |
| +1 :green_heart: | shadedclient | 17m 31s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 11s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 34s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 117m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2369 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a8cdbf86d0f6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / df4006eb813 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[
https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=497216=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-497216
]
ASF GitHub Bot logged work on HADOOP-17301:
---
Author: ASF GitHub Bot
Created on: 08/Oct/20 10:17
Start Date: 08/Oct/20 10:17
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2369:
URL: https://github.com/apache/hadoop/pull/2369#issuecomment-705474826
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 21s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 20s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 1s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 56s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 71m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2369 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8b0a2d453311 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 735e85a1d20 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/2/testReport/ |
| Max. process+thread count | 419 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[ https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=497191=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-497191 ] ASF GitHub Bot logged work on HADOOP-17301: --- Author: ASF GitHub Bot Created on: 08/Oct/20 09:07 Start Date: 08/Oct/20 09:07 Worklog Time Spent: 10m Work Description: snvijaya commented on pull request #2369: URL: https://github.com/apache/hadoop/pull/2369#issuecomment-705436858 Tests were run on accounts on East US 2 region. ### HNS-OAuth [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 ### HNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 24 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 ### NonHNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 245 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 497191) Time Spent: 0.5h (was: 20m) > ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back > > > Key: HADOOP-17301 > URL: https://issues.apache.org/jira/browse/HADOOP-17301 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Critical > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > When reads done by readahead buffers failed, the exceptions where dropped and > the failure was not getting reported to the calling app. > Jira HADOOP-16852: Report read-ahead error back > tried to handle the scenario by reporting the error back to calling app. But > the commit has introduced a bug which can lead to ReadBuffer being injected > into read completed queue twice. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[
https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=497056=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-497056
]
ASF GitHub Bot logged work on HADOOP-17301:
---
Author: ASF GitHub Bot
Created on: 08/Oct/20 03:11
Start Date: 08/Oct/20 03:11
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2369:
URL: https://github.com/apache/hadoop/pull/2369#issuecomment-705303355
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 29m 10s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 31s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 26s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 10s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 6s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| -0 :warning: | checkstyle | 0m 20s |
[/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 4 new + 0 unchanged - 0
fixed = 4 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/1/artifact/out/whitespace-tabs.txt)
| The patch 3 line(s) with tabs. |
| +1 :green_heart: | shadedclient | 17m 31s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 11s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 34s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 117m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2369/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2369 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a8cdbf86d0f6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / df4006eb813 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
[jira] [Work logged] (HADOOP-17301) ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back
[ https://issues.apache.org/jira/browse/HADOOP-17301?focusedWorklogId=497030=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-497030 ] ASF GitHub Bot logged work on HADOOP-17301: --- Author: ASF GitHub Bot Created on: 08/Oct/20 01:05 Start Date: 08/Oct/20 01:05 Worklog Time Spent: 10m Work Description: snvijaya opened a new pull request #2369: URL: https://github.com/apache/hadoop/pull/2369 When reads done by readahead buffers failed, the exceptions where dropped and the failure was not getting reported to the calling app. Jira HADOOP-16852: Report read-ahead error back tried to handle the scenario by reporting the error back to calling app. But the commit has introduced a bug which can lead to ReadBuffer being injected into read completed queue twice when it has finished the store operation. Additionally, in a scenario where all readahead buffers are exhausted and the buffer chosen to evict is one which is failed read, there is no buffer returned for other reads to use. But successful eviction leads the queuing logic to determine there is a free buffer and while fetching the buffer index from free list, can lead to EmptyStack exceptions. This PR fixes both these issues and also has added test checks for both scenarios. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 497030) Remaining Estimate: 0h Time Spent: 10m > ABFS: Fix bug introduced in HADOOP-16852 which reports read-ahead error back > > > Key: HADOOP-17301 > URL: https://issues.apache.org/jira/browse/HADOOP-17301 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Critical > Time Spent: 10m > Remaining Estimate: 0h > > When reads done by readahead buffers failed, the exceptions where dropped and > the failure was not getting reported to the calling app. > Jira HADOOP-16852: Report read-ahead error back > tried to handle the scenario by reporting the error back to calling app. But > the commit has introduced a bug which can lead to ReadBuffer being injected > into read completed queue twice. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
