[
https://issues.apache.org/jira/browse/HADOOP-18366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17575356#comment-17575356
]
Ahmar Suhail commented on HADOOP-18366:
---------------------------------------
Timeout issue appears to be because of logging. Seek in SelectInputStream works
by reading through the stream byte by byte. Each read operation calls
{code:java}
once("read()", uri, () -> wrappedStream.read());{code}
and then logs
{code:java}
`s3a.Invoker (DurationInfo.java:close(101)) - read(): duration 0:00.000s`{code}
in `DurationInfo`. For a ~42MB that's a lot of logging, either to the console
when running the test via IDE or to the output file when running via maven.
With debug logging disabled, test is running in 17s for me right now.
No idea why this was working before and has suddenly stopped working. The
dataset has not grown, the scene_list.gz file hasn't been modified since 2018.
For now, I've fixed by reducing the amount of data we read to 1MB. But also
wondering why we have this test and if we can remove it? It is mostly checking
seek behaviour which is already tested in ITestS3Select.testSelectSeek().
Another possible solution could be to recommend to run scale tests with debug
logging off. Not sure if we can disable logging in the test?
> ITestS3Select.testSelectSeekFullLandsat is timing out
> -----------------------------------------------------
>
> Key: HADOOP-18366
> URL: https://issues.apache.org/jira/browse/HADOOP-18366
> Project: Hadoop Common
> Issue Type: Sub-task
> Reporter: Ahmar Suhail
> Assignee: Ahmar Suhail
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> ITestS3Select.testSelectSeekFullLandsat is timing out. When I run it
> separately on my IDE, I get the following output repeatedly till I stop the
> test manually:
>
> {code:java}
> 2022-07-25 15:15:37,463 [JUnit-testSelectSeekFullLandsat] DEBUG s3a.Invoker
> (DurationInfo.java:close(101)) - read(): duration 0:00.000s
> 2022-07-25 15:15:37,463 [JUnit-testSelectSeekFullLandsat] DEBUG s3a.Invoker
> (DurationInfo.java:<init>(80)) - Starting: read()
> 2022-07-25 15:15:37,463 [JUnit-testSelectSeekFullLandsat] DEBUG s3a.Invoker
> (DurationInfo.java:close(101)) - read(): duration 0:00.000s
> 2022-07-25 15:15:37,463 [JUnit-testSelectSeekFullLandsat] DEBUG s3a.Invoker
> (DurationInfo.java:<init>(80)) - Starting: read()
> 2022-07-25 15:15:37,463 [JUnit-testSelectSeekFullLandsat] DEBUG s3a.Invoker
> (DurationInfo.java:close(101)) - read(): duration 0:00.000s {code}
> When running the entire test suite on my EC2, I get:
> {code:java}
> Time elapsed: 600.01 s <<< ERROR!
> org.junit.runners.model.TestTimedOutException: test timed out after 600000
> milliseconds {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
