steveloughran commented on issue #3076:
URL: https://github.com/apache/parquet-java/issues/3076#issuecomment-2498812581
I think we could do some micro benchmarking here as well.
What we wantis have a skip size such that it is faster to discard the data
it is to wait for/acquire an http connection and download the data across
multiple threads.
Acquisition time is trouble given the http connection pool size and wait
times which may be imposed by the actions of other threads. Same for that
thread pool scheduling overhead.
Download time is a function of bandwidth alone.
We could ignore the https and thread delays and focus on the time from GET
to "first byte" -which would be entirely that imposed by the cloud store
itself. Then all we care about is that the time to download skip the data is
less than the GET-to-first-byte latency.
which is will be when
```
bytes-to-skip/bandwidth < t(GET-to-first-byte)
```
We could benchmark this: something to preheat the pool with a few head
calls, then a set of single byte read() calls to different parts of a large
file, with time to return being considered time to first byte.
then work out download bandwidth and so how many bytes match that
time-to-first-byte
I will take a small PR to cloudstore for this; it'd be interesting to see
what the local fs values are as well as the different cloud stores, local and
remote. Even: do different VM types matter?
@Arnaud-Nauwynck
try setting `http.maxConnections` a number like 100, this will improve the
cacheing of the http connections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
