[
https://issues.apache.org/jira/browse/HADOOP-19946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141844#comment-18141844
]
ASF GitHub Bot commented on HADOOP-19946:
-----------------------------------------
anmolanmol1234 commented on code in PR #8830:
URL: https://github.com/apache/hadoop/pull/8830#discussion_r4522583049
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManagerV2.java:
##########
@@ -890,43 +1119,128 @@ private ReadBuffer clearFromReadAheadQueue(final String
eTag,
*/
private int getBlockFromCompletedQueue(final String eTag, final long
position,
final int length, final byte[] buffer) throws IOException {
- ReadBuffer buf = getBufferFromCompletedQueue(eTag, position);
+ // Find any child buffer from the window that covers the requested position
+ ReadBuffer anyChild = getBufferFromCompletedQueue(eTag, position);
- if (buf == null) {
+ if (anyChild == null) {
+ // No buffer covers this position - cache miss
+ printTraceLog(
+ "No buffer in completed queue covers position: {} for eTag: {}",
+ position, eTag);
return 0;
}
- buf.startReading(); // atomic increment of refCount.
+ // Get the parent and its children
+ ReadBuffer parent = anyChild.getParentBuffer();
+
+ if (parent == null) {
+ // This is a standalone buffer (not part of a parent/child structure)
+ // Fall back to simple single-buffer copy
+ return copyFromSingleBuffer(anyChild, position, length, buffer);
+ }
+
+ List<ReadBuffer> children = parent.getChildBufferList();
+ // Increment refCount on ALL children first to prevent eviction
+ for (ReadBuffer child : children) {
+ child.startReading();
+ }
+
+ try {
+ for (ReadBuffer child : children) {
+ if (child.getStatus() == ReadBufferStatus.READ_FAILED) {
Review Comment:
getBufferFromCompletedQueue now returns any completed child whose parent
window covers the position. The loop here only checks for READ_FAILED, then
copies from the shared array on the assumption that all children succeeded. A
sibling can still be NOT_AVAILABLE in the read-ahead queue or in progress.
waitForProcess only waits when it finds an in-progress buffer, so it may not
wait when siblings are only queued. That lets the copy return unfilled bytes.
Please require every child to be AVAILABLE before copying, and otherwise return
0 so the caller does a direct read.
> ABFS: Enable data locality aware reads for Azure Blob Storage
> -------------------------------------------------------------
>
> Key: HADOOP-19946
> URL: https://issues.apache.org/jira/browse/HADOOP-19946
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/azure
> Reporter: Manish Bhatt
> Assignee: Manish Bhatt
> Priority: Major
> Labels: pull-request-available
>
> Introduce data locality support in ABFS to improve read performance for Azure
> Blob Storage by routing read requests to the endpoint that hosts the required
> data range. The goal is to reduce latency, improve throughput, and make read
> operations more efficient for large files and distributed workloads. This
> work adds blob layout discovery, range-to-endpoint mapping, locality-aware
> range selection, and caching of layout metadata to avoid repeated lookups. It
> also enables segmented read-ahead aligned with locality boundaries and adds
> safeguards for concurrency, eviction, timeout handling, and failure
> propagation needed for reliable production-scale reads.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
