[
https://issues.apache.org/jira/browse/ARROW-8562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090742#comment-17090742
]
Mayur Srivastava commented on ARROW-8562:
-----------------------------------------
[~apitrou], [~lidavidm], I've created a PR for this work:
[https://github.com/apache/arrow/pull/7022]
Please take a look when you get a chance.
Thanks,
Mayur
> [C++] IO: Parameterize I/O coalescing using S3 storage metrics
> --------------------------------------------------------------
>
> Key: ARROW-8562
> URL: https://issues.apache.org/jira/browse/ARROW-8562
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Mayur Srivastava
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Related to https://issues.apache.org/jira/browse/ARROW-7995
> The adaptive I/O coalescing algorithm uses two parameters:
> 1. max_io_gap: Max I/O gap/hole size in bytes
> 2. ideal_request_size = Ideal I/O Request size in bytes
> These parameters can be derived from S3 metrics as described below:
> In an S3 compatible storage, there are two main metrics:
> 1. Seek-time or Time-To-First-Byte (TTFB) in seconds: call setup latency of
> a new S3 request
> 2. Transfer Bandwidth (BW) for data in bytes/sec
> 1. Computing max_io_gap:
> max_io_gap = TTFB * BW
> This is also called Bandwidth-Delay-Product (BDP).
> Two byte ranges that have a gap can still be mapped to the same read if the
> gap is less than the bandwidt-delay product [TTFB * TransferBandwidth], i.e.
> if the Time-To-First-Byte (or call setup latency of a new S3 request) is
> expected to be greater than just reading and discarding the extra bytes on an
> existing HTTP request.
> 2. Computing ideal_request_size:
> We want to have high bandwidth utilization per S3 connections, i.e. transfer
> large amounts of data to amortize the seek overhead.
> But, we also want to leverage parallelism by slicing very large IO chunks.
> We define two more config parameters with suggested default values to control
> the slice size and seek to balance the two effects with the goal of
> maximizing net data load performance.
> BW_util (ideal bandwidth utilization):
> This means what fraction of per connection bandwidth should be utilized to
> maximize net data load.
> A good default value is 90% or 0.9.
> MAX_IDEAL_REQUEST_SIZE:
> This means what is the maximum single request size (in bytes) to maximize
> net data load.
> A good default value is 64 MiB.
> The amount of data that needs to be transferred in a single S3 get_object
> request to achieve effective bandwidth eff_BW = BW_util * BW is as follows:
> eff_BW = ideal_request_size / (TTFB + ideal_request_size / BW)
> Substituting TTFB = max_io_gap / BW and eff_BW = BW_util * BW, we get the
> following result:
> ideal_request_size = max_io_gap * BW_util / (1 - BW_util)
> Applying the MAX_IDEAL_REQUEST_SIZE, we get the following:
> ideal_request_size = min(MAX_IDEAL_REQUEST_SIZE, max_io_gap * BW_util / (1 -
> BW_util))
> The proposal is to create a named constructor in the io::CacheOptions (PR:
> [https://github.com/apache/arrow/pull/6744] created by [~lidavidm]) to
> compute max_io_gap and ideal_request_size from TTFB and BW which will then be
> passed to reader to configure the I/O coalescing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
