[
https://issues.apache.org/jira/browse/HADOOP-18106?focusedWorklogId=783525&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783525
]
ASF GitHub Bot logged work on HADOOP-18106:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 21/Jun/22 18:53
Start Date: 21/Jun/22 18:53
Worklog Time Spent: 10m
Work Description: mukund-thakur commented on PR #4445:
URL: https://github.com/apache/hadoop/pull/4445#issuecomment-1162193283
> on the EOF exception what happens with
>
> * vectored reads which start before and end after EOF?
> * vectored reads which start and end after EOF
> * vectored reads with two ranges in draining distance, the second of which
is beyond the EOF?
>
> should all be tested in the contract tests, if not done already
All these will cause EOF because we validate ranges in the start only. Tests
are there but not exactly like these. I can add some more next time.
Issue Time Tracking
-------------------
Worklog Id: (was: 783525)
Time Spent: 4h (was: 3h 50m)
> Handle memory fragmentation in S3 Vectored IO implementation.
> -------------------------------------------------------------
>
> Key: HADOOP-18106
> URL: https://issues.apache.org/jira/browse/HADOOP-18106
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Mukund Thakur
> Assignee: Mukund Thakur
> Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> As we have implemented merging of ranges in the S3AInputStream implementation
> of vectored IO api, it can lead to memory fragmentation. Let me explain by
> example.
>
> Suppose client requests for 3 ranges.
> 0-500, 700-1000 and 1200-1500.
> Now because of merging, all the above ranges will get merged into one and we
> will allocate a big byte buffer of 0-1500 size but return sliced byte buffers
> for the desired ranges.
> Now once the client is done reading all the ranges, it will only be able to
> free the memory for requested ranges and memory of the gaps will never be
> released for eg here (500-700 and 1000-1200).
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
