GitHub user rmatharu opened a pull request:
https://github.com/apache/samza/pull/892
Samza-2018: Using Rocksdb bulk load only for clean-slate restores
Rocksdb has a problem that its bulk-load option cannot be used on existing
stores seamlessly.
See https://github.com/facebook/rocksdb/issues/2734
Because of this when host-affinity puts a container onto an old host, the
restore path's bulk-load mode fails.
To fix this,
We use rocksdb's bulk load option only when the underlying storeDirectory
does not exist.
If the storeDirectory exists, then we open the store without bulk-load
options.
Note that, an existing storeDirectory may also be "stale", in which case we
delete it, and should ideally be using the bulk load option. We will add this
additional optimization separately.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rmatharu/samza rocksdbbugfix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/892.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #892
----
commit 2c09b081d17848a357c0a3a8ecafbe36da6abd39
Author: Ray Matharu <rmatharu@...>
Date: 2019-01-24T02:16:05Z
Rocksdb bug fix
----
---
