steveloughran commented on code in PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#discussion_r1272698192
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/CachingBlockManager.java:
##########
@@ -298,16 +306,26 @@ public void requestPrefetch(int blockNumber) {
/**
* Requests cancellation of any previously issued prefetch requests.
+ * If the reason was switching to random IO, any active prefetched blocks
+ * are still cached.
+ * @param reason why?
*/
@Override
- public void cancelPrefetches() {
+ public void cancelPrefetches(final CancelReason reason) {
+ LOG.debug("Cancelling prefetches {}", reason);
BlockOperations.Operation op = ops.cancelPrefetches();
- for (BufferData data : bufferPool.getAll()) {
- // We add blocks being prefetched to the local cache so that the
prefetch is not wasted.
- if (data.stateEqualsOneOf(BufferData.State.PREFETCHING,
BufferData.State.READY)) {
- requestCaching(data);
+ if (reason == CancelReason.RandomIO) {
+ for (BufferData data : bufferPool.getAll()) {
+ // We add blocks being prefetched to the local cache so that the
prefetch is not wasted.
+ // this only done if the reason is random IO-related, not due to
close/unbuffer
+ if (data.stateEqualsOneOf(BufferData.State.PREFETCHING,
BufferData.State.READY)) {
+ requestCaching(data);
+ }
}
+ } else {
+ // free the buffers
+ bufferPool.getAll().forEach(BufferData::setDone);
Review Comment:
we had so much grief with the abfs prefetch premature release bug that I am
scared now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
