Todd Lipcon updated KUDU-1693:
    Component/s: perf

> Flush write operations on per-TS basis and add corresponding limit on the 
> buffer space
> --------------------------------------------------------------------------------------
>                 Key: KUDU-1693
>                 URL: https://issues.apache.org/jira/browse/KUDU-1693
>             Project: Kudu
>          Issue Type: Improvement
>          Components: client, perf
>    Affects Versions: 1.0.0
>            Reporter: Alexey Serbin
>            Priority: Major
> Currently, the Kudu C++ client buffers incoming operations regardless of 
> their destination tablet server.  Accordingly, it's possible to set limit on 
> the _total_ buffer space, not per tablet server.  This approach works but 
> there is room for improvement: there are real-world scenarios where per-TS 
> buffering would be more robust.  Besides, tablet servers impose limit on the 
> RPC operations size.
> Grouping write operations on per-tablet-server basis would be beneficial for 
> 'one-out-of-many lagging tablet server' scenario.  There, all tablet servers 
> for a table perform well except for one which runs slow due to excessive IO, 
> network issues, failing disk, etc.  The problem is that the lagging server 
> hinders the overall performance.  This is due to the current approach to the 
> buffer turnaround: a buffer is considered 'flushed' and its space is 
> reclaimed at once when _all_ operations in the buffer are completed.  So, if 
> 1000 operations have already been sent but there is 1 operation still in 
> progress, the whole buffer space is 'locked' and cannot be used.
> Accordingly, introducing per-tablet-server buffer limit would help to address 
> scenarios with concurrent writes into tables with extremely diverse partition 
> factors (like 2 and 100).   E.g., consider a case when incoming write 
> operations for tables with diverse partition factors are intermixed in the 
> context of one session.  The problem is that setting the total buffer space 
> limit high is beneficial for the writes into the table with many partitions 
> (assuming those writes are evenly distributed across participating tablets), 
> but it may be over the server-side's limit for max transaction size if those 
> writes are targeted for the table with a few partitions.

This message was sent by Atlassian JIRA

Reply via email to