[
https://issues.apache.org/jira/browse/HADOOP-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743344#comment-17743344
]
Viraj Jasani edited comment on HADOOP-18805 at 7/15/23 6:48 AM:
----------------------------------------------------------------
sorry Steve, i was not aware you already created this Jira, i created addendum
for letting LRU test depend on small file rather than large one:
[https://github.com/apache/hadoop/pull/5843]
{quote}also, and this is very, very important, they can't validate the data
{quote}
i was about to create a sub-task for this as i am planning to refactor Entry to
it's own class and have the contents of the linked list data tested in UT
(discussed with Mehakmeet in the earlier part of the review). maybe i can do
the work as part of this Jira.
are you fine with?
* the above addendum PR for using small file in the test (so that we don't
need to put the test under -scale)
* this Jira to refactor Entry and allowing a UT to test the contents of the
linked list
if you think above PR is not good for an addendum and should rather be linked
to this Jira, i can change PR title to reflect this Jira number and i can
create another sub-task to write simple UT that can test contents of the linked
list from head to tail.
was (Author: vjasani):
sorry Steve, i was not aware you already created this Jira, i created addendum
for letting LRU test depend on small file rather than large one:
[https://github.com/apache/hadoop/pull/5843]
{quote}also, and this is very, very important, they can't validate the data
{quote}
i was about to create a sub-task for this as i am planning to refactor Entry to
it's own class and have the contents of the linked list data tested in UT
(discussed with Mehakmeet in the earlier part of the review). maybe i can do
the work as part of this Jira.
are you fine with the above addendum PR taking care of using small file in the
test (so that we don't need to put the test under -scale) and this Jira being
used for refactoring Entry and allowing a UT to test the contents of the linked
list?
> s3a large file prefetch tests are too slow, don't validate data
> ---------------------------------------------------------------
>
> Key: HADOOP-18805
> URL: https://issues.apache.org/jira/browse/HADOOP-18805
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Affects Versions: 3.3.9
> Reporter: Steve Loughran
> Priority: Major
>
> the large file prefetch tests (including LRU cache eviction) are really slow.
> moving under -scale may hide the problem for most runs, but they are still
> too slow, can time out, etc etc.
> also, and this is very, very important, they can't validate the data.
> Better:
> * test on smaller files by setting a very small block size (1k bytes or less)
> just to force paged reads of a small 16k file.
> * with known contents to the values of all forms of read can be validated
> * maybe the LRU tests can work with a fake remote object which can then be
> used in a unit test
> * extend one of the huge file tests to read from there -including s3-CSE
> encryption coverage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
