[
https://issues.apache.org/jira/browse/HADOOP-18231?focusedWorklogId=771381&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-771381
]
ASF GitHub Bot logged work on HADOOP-18231:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 17/May/22 14:21
Start Date: 17/May/22 14:21
Worklog Time Spent: 10m
Work Description: ahmarsuhail commented on PR #4305:
URL: https://github.com/apache/hadoop/pull/4305#issuecomment-1128933678
Thanks @monthonk. As discussed, instead of using
`landsat-pds/scene_list.gz`, I tried creating a smaller file (size 16k) and
setting the block size to 4K, which would make the tests faster. But currently
if you try to set the block size < default size (8M), validation fails
[here](https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L487)
. The min allowed block size currently is `PREFETCH_BLOCK_DEFAULT_SIZE` , I'm
not sure if this is something we want to update.
Issue Time Tracking
-------------------
Worklog Id: (was: 771381)
Time Spent: 1h 20m (was: 1h 10m)
> tests in ITestS3AInputStreamPerformance are failing
> ----------------------------------------------------
>
> Key: HADOOP-18231
> URL: https://issues.apache.org/jira/browse/HADOOP-18231
> Project: Hadoop Common
> Issue Type: Sub-task
> Reporter: Ahmar Suhail
> Assignee: Ahmar Suhail
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The following tests are failing when prefetching is enabled:
> testRandomIORandomPolicy - expects stream to be opened 4 times (once for
> every random read), but prefetching will only open twice.
> testDecompressionSequential128K - expects stream to be opened once, but
> prefetching will open once for each block the file has. landsat file used in
> the test has size 42MB, prefetching block size = 8MB, expected open count is
> 6.
> testReadWithNormalPolicy - same as above.
> testRandomIONormalPolicy - executes random IO, but with a normal policy.
> S3AInputStream will abort the stream and change the policy, prefetching
> handles random IO by caching blocks so doesn't do any of that.
> testRandomReadOverBuffer - multiple assertions failing here, also depends a
> lot on readAhead values, not very relevant for prefetching
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
