David Alves created KUDU-1702:
Summary: Document/Implement read-your-writes for Impala/Spark etc.
Issue Type: Sub-task
Affects Versions: 1.1.0
Reporter: David Alves
Assignee: David Alves
Engines like Impala/Spark use many independent client instances, so we should
provide a way to have read-your-writes across many independent client
instances, which translates to provide a way to get linearizable behavior.
At first this can be done using the APIs that are already available. For
instance if the objective is to be sure to have the results of a write in a a
following scan, the following steps can be taken:
- After a write the engine should collect the last observed timestamps from
- The engine's coordinator then takes the max of those timestamps, adds 1 and
uses that as a snapshot scan timestamp.
One important pre-requisite of the behavior above is that scans be done in
READ_AT_SNAPSHOT mode. Also the steps above currently don't actually guarantee
the expected behavior, but should as the currently anomalies are taken care of
(as part of KUDU-430).
In the immediate future we'll add APIs to the Kudu client so as to make the
inner workings of getting this behavior oblivious to the engine. The steps will
still be the same, i.e. timestamps or timestamp tokens will still be passed
around, but the kudu client will encapsulate the choice of the timestamp for
Later we add a way to obtain this behavior without timestamp propagation,
either by doing a write-side commit-wait, where clients finish their writes by
waiting out the clock error thus making sure any future operation will have a
higher timestamp; or by making read-side commit wait, where we provide an api
on the kudu client for the engine to perform a similar call before the scan
call to obtain a scan timestamp.
This message was sent by Atlassian JIRA