steveloughran commented on PR #5548:
URL: https://github.com/apache/hadoop/pull/5548#issuecomment-1517730579
Tested s3 london; I'm now playing with both prefetch and non-prefetching
test runs.
At least with my local test setup, its a bit slower with -Dprefetch, but
remember these aren't benchmarks, these are unit tests where the connections
are all very short lived, the FS instances don't even have time to build up
that http pool.
### prefetch
```
time mvit -Dparallel-tests -DtestsThreadCount=8 -Dscale
-Dfs.s3a.scale.test.huge.filesize=100M -Dprefetch
________________________________________________________
Executed in 16.32 mins fish external
usr time 20.57 mins 0.14 millis 20.57 mins
sys time 3.24 mins 1.70 millis 3.24 mins
```
one failure there, which is HADOOP-18697
```
[ERROR]
testRandomReadLargeFile(org.apache.hadoop.fs.s3a.ITestS3APrefetchingInputStream)
Time elapsed: 25.738 s <<< FAILURE!
org.junit.ComparisonFailure: [Gauge named stream_read_blocks_in_cache with
expected value 0] expected:<[0]L> but was:<[1]L>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
org.apache.hadoop.fs.statistics.IOStatisticAssertions.verifyStatisticValue(IOStatisticAssertions.java:257)
at
org.apache.hadoop.fs.statistics.IOStatisticAssertions.verifyStatisticGaugeValue(IOStatisticAssertions.java:190)
at
org.apache.hadoop.fs.s3a.ITestS3APrefetchingInputStream.testRandomReadLargeFile(ITestS3APrefetchingInputStream.java:210)
```
## no prefetch
```
time mvit -Dparallel-tests -DtestsThreadCount=8 -Dscale
-Dfs.s3a.scale.test.huge.filesize=100M
________________________________________________________
Executed in 15.91 mins fish external
usr time 20.34 mins 0.15 millis 20.34 mins
sys time 3.24 mins 1.90 millis 3.24 mins
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
