[
https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743723#comment-17743723
]
Steve Loughran commented on HADOOP-18291:
-----------------------------------------
[~quanli] how is it failing? what is your test setup (do you have that landsat
csv file reachable or something else configured to use it?}
it is working for me remotely although i haven't yet backported internally (the
initial prefetching code is in there:
bq. not sure if reviewers wrote code via review this much comments.
we reviewed it. and ran the tests first; aws s3 london with direct
region/endpoint settings and -Dscale, inside vpn and outside it
# If the problems are only test related, then HADOOP-18805 is the place to fix.
let's get on to that. I want to move off landsat entirely
# I have a PR up for HADOOP-18184 which moves the tests and extends them,
splitting out landsat io into -Dscale.
# [~quanli] we will need your involvement testing that to make sure it is good
in your setup. Could you start with the HADOOP-18184 PR with/without -Dscale to
see if the problems are isolated to landsat tests?
> S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
> --------------------------------------------------------------
>
> Key: HADOOP-18291
> URL: https://issues.apache.org/jira/browse/HADOOP-18291
> Project: Hadoop Common
> Issue Type: Sub-task
> Affects Versions: 3.4.0
> Reporter: Ahmar Suhail
> Assignee: Viraj Jasani
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.9
>
>
> Currently there is no limit on the size of disk cache. This means we could
> have a large number of files on files, especially for access patterns that
> are very random and do not always read the block fully.
>
> eg:
> in.seek(5);
> in.read();
> in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read
> in.read();
> in.seek(2 * blockSize + 10) // block 1 gets saved to disk
> .. and so on
>
> The in memory cache is bounded, and by default has a limit of 72MB (9
> blocks). When a block is fully read, and a seek is issued it's released
> [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109].
> We can also delete the on disk file for the block here if it exists.
>
> Also maybe add an upper limit on disk space, and delete the file which stores
> data of the block furthest from the current block (similar to the in memory
> cache) when this limit is reached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
