[
https://issues.apache.org/jira/browse/HADOOP-19043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808783#comment-17808783
]
ASF GitHub Bot commented on HADOOP-19043:
-----------------------------------------
steveloughran commented on code in PR #6465:
URL: https://github.com/apache/hadoop/pull/6465#discussion_r1459580119
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:
##########
@@ -161,7 +171,11 @@ public void testOpenFileWithStatusOfOtherFS() throws
Throwable {
@Test
public void testStreamIsNotChecksummed() throws Throwable {
describe("Verify that an opened stream is not checksummed");
+
+ // if prefetching is enabled, skip this test
+ assumeNoPrefetching();
S3AFileSystem fs = getFileSystem();
+
Review Comment:
cut just to reduce diff
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:
##########
@@ -420,31 +443,56 @@ public void testVectorReadPastEOF() throws Throwable {
TimeUnit.SECONDS,
range.getData());
assertS3StreamClosed(in);
- return "vector read past EOF";
+ return "vector read past EOF with " + in;
}
},
- with(Statistic.ACTION_HTTP_GET_REQUEST, 1));
+ always(),
+ probe(!prefetching, Statistic.ACTION_HTTP_GET_REQUEST, 1));
+ }
+
+ /**
+ * Probe the FS for supporting prefetching.
+ * @return true if the fs has prefetching enabled.
+ */
+ private boolean prefetching() {
+ return getFileSystem().getConf().getBoolean(
+ PREFETCH_ENABLED_KEY, PREFETCH_ENABLED_DEFAULT);
+ }
+
+ /**
+ * Skip the test if prefetching is enabled.
+ */
+ private void assumeNoPrefetching(){
+ if (prefetching) {
+ skip("Prefetching is enabled");
+ }
}
/**
* Assert that the inner S3 Stream is closed.
* @param in input stream
*/
private static void assertS3StreamClosed(final FSDataInputStream in) {
- S3AInputStream s3ain = (S3AInputStream) in.getWrappedStream();
- Assertions.assertThat(s3ain.isObjectStreamOpen())
- .describedAs("stream is open")
- .isFalse();
+ final InputStream wrapped = in.getWrappedStream();
+ if (wrapped instanceof S3AInputStream) {
+ S3AInputStream s3ain = (S3AInputStream) wrapped;
+ Assertions.assertThat(s3ain.isObjectStreamOpen())
+ .describedAs("stream is open")
Review Comment:
should put the string value of the stream in the exception text here; same
below
> S3A: Regression: ITestS3AOpenCost fails on prefetch test runs
> -------------------------------------------------------------
>
> Key: HADOOP-19043
> URL: https://issues.apache.org/jira/browse/HADOOP-19043
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Labels: pull-request-available
>
> Getting test failures in the new ITestS3AOpenCost tests when run with
> {{-Dprefetch}}
> Thought I'd tested this, but clearly not
> * class cast failures on asserts (fix: skip)
> * bytes read different in one test: (fix: identify and address)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
