Thanks a lot to Nick for further suggestion. Please see below my embedded response about API nits.
> On Mar 19, 2020, at 6:01 AM, Nick Vatamaniuc <vatam...@gmail.com> wrote: > I like the API shape, and the implementation looks pretty small and > easy so far. Bonus points for using the HCA to hopefully get some > performance improvement from smaller keys overall. That was Paul's > original idea all along I believe. > > Was wondering about a few minor API nits: > > * Maybe use `timestamp` instead of `deleted_when` since we used > `timestamp` in the rest of the API description? Yes, I am struggling about this field name, and changed it from `deleted_ts` to `deleted_when`. To keep consistent, `timestamp` is better than `deleted_when`. I like this. > > * Our db instances have unique a `uuid` (instance id) attribute > internally, we just don't surface it in the API. So when we re-create > a db with the same name it gets a new `uuid`. I could see using that > to identify individual deleted db instances when we restore them, as > opposed to using timestamps: `/{db}/_restore/{DbUuid}`. However, > because we don't already surface that attribute in the API it would be > a bit more noise too... So I think that argues for keeping timestamp > as the id, but thought I'd mentioned and see if others have thoughts > on it anyway. Uuid is another alternative for restore. However, I think that timestamp might give more hints to help users to restore database. It is with more meaning. > > Concerning the backup implementation. I think that's still an option! > In other words the soft deletion API can still be the same, and > eventually, once we get backup implemented, soft-deleted instances > could immediately (or transparently in the background) become backups. > Users might just see an extra metadata rows in `_deleted_dbs_info` > something like "backed to blobstore foo as 1 day ago .... So they know > restoring it won't be a single transaction but might take a while. > > FDB backup does have a local `file://</absolute/path/to/base_dir>` > option for URLs [1] so that might be useful in embedded scenarios. And > someone has probably created some sort of local filesystem S3 shim (S0 > ;-) ) we could adapt perhaps.... > > Cheers, > -Nick > > [1] https://apple.github.io/foundationdb/backups.html#backup-urls > > On Wed, Mar 18, 2020 at 5:06 PM Paul Davis <paul.joseph.da...@gmail.com> wr= > ote: >> >> Alex, >> >> The first con I see for that approach is that its not soft-deletion. >> Its actual deletion with an API for restoration. Which, fair enough, >> is probably a feature we should consider supporting for CouchDB >> installations that are based on FoundationDB. >> >> The second major con is that it relies on CouchDB being based on >> FoundationDB. Part of CouchDB's design philosophy is that the internet >> may or may not exist, and if it does exist that it may or may not be >> reliable. There are lots of deployments of CouchDB that are part of a >> desktop application or POS installation that may see internet only >> periodically if at all so an S3 backup solution is out. There also may >> come a time that there's a flavor of CouchDB that uses LevelDB or >> SQLite or FDBLite (I just made that up, any idea how hard it'd be?) >> for these sorts of embedded deployments where fdbrestore/fdbbackup >> wouldn't be feasible. >> >> Then the last major con I see is the time-to-restore disparity. With >> soft-deletion restoration is a few milliseconds. Streaming from S3 >> will obviously depend on the size of the database and obviously be >> orders of magnitude longer. >> >> On the pro side for the soft-delete on FoundationDB is that the first >> draft of the RFC is 108 lines [1]. We obviously can't say for sure how >> big or involved the fdbrestore approach would be but I think we'd all >> agree it'd be bigger. >> >> Paul >> >> [1] https://github.com/apache/couchdb/pull/2666 >> >> >> On Wed, Mar 18, 2020 at 2:31 PM Alex Miller >> <alexmil...@apple.com.invalid> wrote: >>> >>> Let me perhaps paint an alternative RFC: >>> >>> 1) `DELETE /{db}` >>> >>> If soft-deletion is enabled, delete the database subspace, and also rec= > ord into ?DELETED_DBS the timestamp of the commit and the database subspace= > prefix >>> >>> 2) `GET /{db}/_deleted_dbs_info` >>> >>> Return the timestamp (and whatever other info one should record) of del= > eted databases. >>> >>> 3) `PUT /{db}/_restore/{deletedTS}` >>> >>> Invoke `fdbrestore -k` to do a key range restricted restore into the cu= > rrent cluster of the deleted subspace prefix at versionstamp-1. Wait for i= > t to complete, and return 200 when completed. >>> >>> And this would all rely on having a continuous backup configured and ru= > nning that would hold a minimum of 48 hours of changes. >>> >>> >>> Now, I don=E2=80=99t actually deal with backups often so my memory on c= > urrent caveats is a bit fuzzy. I think there might be a couple complicatio= > ns here that I=E2=80=99ve missed, like=E2=80=A6 >>> * There not being key range restricted locking of the database >>> * A key range restore is currently suboptimal in that it doesn=E2=80=99= > t do obvious filtering that it could to cut down on the amount of data it r= > eads >>> >>> But, neither of these seem heavily blocking, as they could be tackled q= > uickly, particularly if you leverage some upstream relationships ;). Backu= > p and restore has been the general answer to accidental data deletion (or c= > orruption) on FDB, and I could paint some attractive looking pros of this a= > pproach: backup files are more disk space efficient, soft deleted data coul= > d be offloaded to an S3-compatible store, it would be free if FDB is alread= > y configured to take backups. I was just curious to hear a bit more detail= > on your/Peng=E2=80=99s side of the reasons for preferring to build soft de= > letion on top of FDB (and thus have also intentionally withheld more of the= > cons of this approach, or the pros of yours). >>> >>>> On Mar 18, 2020, at 11:59, Paul Davis <paul.joseph.da...@gmail.com> w= > rote: >>>> >>>> Alex, >>>> >>>> All joking aside, soft-deletion's target use case is accidental >>>> deletions. This isn't a replacement for backup/restore which will >>>> still happen for all the usual reasons. >>>> >>>> Paul >>>> >>>> On Wed, Mar 18, 2020 at 1:42 PM Paul Davis <paul.joseph.davis@gmail.c= > om> wrote: >>>>> >>>>> On Wed, Mar 18, 2020 at 1:29 PM Alex Miller >>>>> <alexmil...@apple.com.invalid> wrote: >>>>>> >>>>>> >>>>>>> On Mar 18, 2020, at 05:04, jiangph <jiangpeng...@hotmail.com> wrot= > e: >>>>>>> >>>>>>> Instead of automatically and immediately removing data and index i= > n database after a delete operation, soft-deletion allows to restore the de= > leted data back to original state due to a =E2=80=9Cfat finger=E2=80=9Dor u= > ndesired delete operation, up to defined periods, such as 48 hours. >>>>>>> >>>>>>> In CouchDB 3.0, soft-deletion of database is implemented in [1]. T= > he .couch file is renamed with the .<timestamp>.deleted.couch file after so= > ft-deletion is enabled, and such file can be changed back to .couch for the= > purpose of restore. If restore is not needed and some specified period pas= > sed, the .<timestamp>.deleted.couch file can be deleted to achieve deletion= > of database permanently. >>>>>>> >>>>>>> In CouchDB 4.0, with the introduction of FoundationDB, the data mo= > del and storage is changed. In order to support soft-deletion, we propose b= > elow solution and then implement them. >>>>>> >>>>>> >>>>>> >>>>>> I=E2=80=99ve sort of hand waved some answers to this in my head, bu= > t would you mind expanding a bit on the advantages of keeping soft-deleted = > data in FoundationDB as opposed to actually deleting it and relying on Foun= > dationDB=E2=80=99s backup and restore to recover it if needed? >>>>> >>>>> From: Panicked User >>>>> To: Customer Support >>>>> Subject: URGENT! EMERGENCY DATABASE RESTORE! >>>>> >>>>> Dear, >>>>> >>>>> I have accidentally deleted my Very Important Database and need to >>>>> have it restored ASAP! Without this mission critical database my >>>>> company is completely offline which is costing $1B an hour!!!!! >>>>> >>>>> Please respond ASAP! >>>>> >>>>> Sincerely, >>>>> Panicky McPanics >>>