Yes. Thanks for explaining. On Friday, May 20, 2016, Ufuk Celebi <u...@apache.org> wrote:
> On Thu, May 19, 2016 at 8:54 PM, Abhishek R. Singh > <abhis...@tetrationanalytics.com <javascript:;>> wrote: > > If you can take atomic in-memory copies, then it works (at the cost of > > doubling your instantaneous memory). For larger state (say rocks DB), > won’t > > you have to stop the world (atomic snapshot) and make a copy? Doesn’t > that > > make it synchronous, instead of background/async? > > Hey Abhishek, > > that's correct. There are two variants for RocksDB: > > - semi-async (default): snapshot is taking via RocksDB backup feature > to backup to a directory (sync). This is then copied to the final > checkpoint location (async, e.g copy to HDFS). > > - fully-async: snapshot is taking via RocksDB snapshot feature (sync, > but no full copy and essentially "free"). With this snapshot we > iterate over all k/v-pairs and copy them to the final checkpoint > location (async, e.g. copy to HDFS). > > You enable the second variant via: > rocksDbBackend.enableFullyAsyncSnapshots(); > > This is only part of the 1.1-SNAPSHOT version though. > > I'm not too familiar with the performance characteristics of both > variants, but maybe Aljoscha can chime in. > > Does this clarify things for you? > > – Ufuk >