Hi,
There is a fundamental incompatibility between CouchDB using couch_file/btree and CouchDB using FDB. The choice at hand here is between two different forms of compatibility break; 1) All responses that were over a single snapshot in CouchDB 1/2/3 will still be over a single snapshot in CouchDB 4, but necessarily limited in the amount of data they can return and time they can take. This means that some requests will fail that previously succeeded. As an example, hitting _all_docs without a suitably small “limit” parameter will either a) return a 400 Bad Request right out of the gate or b) be abruptly terminated mid-response when one of the FDB limits is reached (5 second transaction duration or 10MB of data: https://apple.github.io/foundationdb/known-limitations.html <https://apple.github.io/foundationdb/known-limitations.html>). 2) Some responses that were over a single snapshot in CouchDB 1/2/3 will potentially be over multiple snapshots in CouchDB 4, so clients will sometimes see incoherent responses. As an example, an _all_docs response running concurrently with doc inserts will see none, some or all of those inserts, depending on the doc _ids of those inserts and how far along the _all_docs response has progressed. Two _all_docs responses running concurrently with those inserts could see a different subset of those concurrently running inserts (based on when the restart_tx code is called and the GRV grabbed at that point. I vastly prefer the 1) scenario. To bridge the gap that Nick describes I agree it would be acceptable to drop the snapshot requirement for all _changes responses, not just the continuous mode. All correctly-implemented consumers of the changes feed should be able to handle that. For 3.x -> 4.x replication, we could make a 3.x minor release that solely enhances the replicator in whatever way would be necessary to restore replication compatibility. We’ve done this before at least once. To Will’s points, we would need to decide if CouchDB 4 will have a “compatibility mode” at all (taken to mean that no adjustment is needed by the client whatsoever). Beyond replication, I don’t see how it could be done, and I don’t think it should be a goal. We shouldn’t be _incompatible_ capriciously, however. But, at base, this is a major version bump (the classic signal of potential incompatibility), a very significant amount of new code, and a completely new storage backend with constraints that preclude CouchDB 1.x semantics. Anyway, this is a VOTE thread and not a DISCUSS thread. I think it’s fair to say the proposal has failed and so this thread is over. We do need to make a project level decision on this topic before CouchDB 4 can be released. B. > On 10 Jan 2021, at 07:26, Joan Touzet <woh...@apache.org> wrote: > > If this proposal means v3.x replicators can't replicate one-shot / normal / > non-continuous changes from 4.x+ endpoints, that sounds like a big break in > compatibility. > > I'm -0.5, tending towards -1, but mostly because I'm having trouble > understanding if it's even possible - unless a proposal is being made to > release a 3.2 that introduces replication compatibility with 4.x in tandem. > > -Joan > > On 2021-01-09 6:45 p.m., Nick Vatamaniuc wrote: >>> I withdraw my vote until I can get a clearer view. Nick would you mind >> re-stating? >> Not at all! The longer version and other considerations was stated in >> my last reply to the discussion thread so I assumed that was accepted >> as a consensus since nobody replied arguing otherwise. >> https://lists.apache.org/thread.html/r45bff6ca4339f775df631f47e77657afbca83ee0ef03c6aa1a1d45cb%40%3Cdev.couchdb.apache.org%3E >> But the gist of it is that existing (< 3.x) replicators won't be able >> to replicate non-continuous (normal) changes from >= 4.x endpoints. >> Regards, >> -Nick >> On Sat, Jan 9, 2021 at 1:26 AM Joan Touzet <woh...@apache.org> wrote: >>> >>> Wait, what? I thought you agreed with this approach in that thread. >>> >>> I withdraw my vote until I can get a clearer view. Nick would you mind >>> re-stating? >>> >>> -Joan >>> >>> On 2021-01-08 11:37 p.m., Nick V wrote: >>>> +1 for 1 through 3 >>>> >>>> -1 for 4 as I think the exception should apply to normal change feeds as >>>> well, as described in the thread >>>> >>>> Cheers, >>>> -Nick >>>> >>>>> On Jan 8, 2021, at 17:12, Joan Touzet <woh...@apache.org> wrote: >>>>> >>>>> Thanks, then it's a solid +1 from me. >>>>> >>>>> -Joan >>>>> >>>>>> On 2021-01-08 4:13 p.m., Robert Newson wrote: >>>>>> You are probably thinking of a possible “group commit”. That is >>>>>> anticipated and not contradicted by this proposal. This proposal is >>>>>> explicitly about not using multiple states of the database for a single >>>>>> doc lookup, view query, etc. >>>>>>>> On 8 Jan 2021, at 19:53, Joan Touzet <woh...@apache.org> wrote: >>>>>>> >>>>>>> +1. >>>>>>> >>>>>>> This is for now I presume, as I thought that there was feeling about >>>>>>> relaxing this restriction somewhat for the 5.0 timeframe? Memory's dim. >>>>>>> >>>>>>> -Joan >>>>>>> >>>>>>> On 07/01/2021 06:00, Robert Newson wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> Following on from the discussion at >>>>>>>> https://lists.apache.org/thread.html/rac6c90c4ae03dc055c7e8be6eca1c1e173cf2f98d2afe6d018e62d29%40%3Cdev.couchdb.apache.org%3E >>>>>>>> >>>>>>>> <https://lists.apache.org/thread.html/rac6c90c4ae03dc055c7e8be6eca1c1e173cf2f98d2afe6d018e62d29@%3Cdev.couchdb.apache.org%3E> >>>>>>>> >>>>>>>> The proposal is; >>>>>>>> >>>>>>>> "With the exception of the changes endpoint when in feed=continuous >>>>>>>> mode, that all data-bearing responses from CouchDB are constructed >>>>>>>> from a single, immutable snapshot of the database at the time of the >>>>>>>> request.” >>>>>>>> >>>>>>>> Paul Davis summarised the discussion in four bullet points, reiterated >>>>>>>> here for context; >>>>>>>> >>>>>>>> 1. A single CouchDB API call should map to a single FDB transaction >>>>>>>> 2. We absolutely do not want to return a valid JSON response to any >>>>>>>> streaming API that hit a transaction boundary (because data >>>>>>>> loss/corruption) >>>>>>>> 3. We're willing to change the API requirements so that 2 is not an >>>>>>>> issue. >>>>>>>> 4. None of this applies to continuous changes since that API call was >>>>>>>> never a single snapshot. >>>>>>>> >>>>>>>> >>>>>>>> Please vote accordingly, we’ll run this as lazy consensus per the >>>>>>>> bylaws (https://couchdb.apache.org/bylaws.html#lazy >>>>>>>> <https://couchdb.apache.org/bylaws.html#lazy>) >>>>>>>> >>>>>>>> B. >>>>>>>> >>>>>>>>
