Re: Important update on couchdb's foundationdb work
Hi All, I think sticking with 3.x is the better way to go for the CouchDB community. If anyone wants to continue with the FDB-CouchDB work. I think the first place to start there is to change the user-facing API to better fit with FDB. We added some really nice unit tests for FDB-CouchDB, it might be worth investigating if we could move some of the unit tests to 3.x. The one feature I loved with FDB-CouchDB was that the mango indexes were updated in the transaction that the document was added. Meaning the mango indexes were always up to date. I will really miss that feature. Finally, I've seen some mention around the pluggable storage engine for 3.x. I wrote a PoC Rockdbs version of it that passed most the pluggable storage tests https://github.com/garrensmith/couch_rocks With the pluggable storage engine you will most likely run into the same issue we had with FDB. CouchDB relies on a B-Tree to do anything reduce related. I think what could be possible with the pluggable storage engine is to write the couch_store in something like Rust to see if you get a performance improvement. Or to write an in-memory one. However, I don't think the storage engine is the real bottleneck in CouchDB. Cheers Garren On Fri, Mar 18, 2022 at 9:44 AM Johs Ensby wrote: > Hi, > I want to second the statements (Fynn, Ronny and others) that I see as > support for option 1 in Jan’s three ways forward: > > 1. Follow IBM in abandoning FDB-Couch, refocus all effort on Erlang-Couch > (3.x). > > “Follow IBM” in the meaning “staying together and have the benefit of a > big brand supporter” is a good thing, also. > > I still think couchDB is unique, but simplicity for devops is the key > benefit. > Replication, offline first, distributed systems, built-in blockchain, > attachments and designdocs that allow full-fledged apps to travel with the > data, > … the feature list goes on and on, and the heritage from the project’s > start is beautiful and still very future-oriented. > > Best regards > Johs > > > On 17 Mar 2022, at 21:25, Jan Lehnardt wrote: > > > > Heya Alex, > > > > I think all these are fair assessments and I can’t think of anything > atop. > > > > Best > > Jan > > — > > > > > >> On 16. Mar 2022, at 23:54, Alex Miller wrote: > >> > >> (With my hat on as a liaison from the FoundationDB community...) > >> > >> I imagine that, like all corporate decisions, stopping work on > >> CouchDB-FDB was some part business reasons (which are entirely out of > >> scope for this discussion) and some part technical reasons. I wanted > >> to try and make sure to collect the feedback about any technical > >> motivations that might have dissuaded the continued work on > >> CouchDB-FDB as CouchDB 4.x. Thus far from watching the thread, I'm > >> at: > >> > >> * Long lived read-only snapshots are required to maintain CouchDB 3.x > >> semantics, and that feature has yet to be implemented in FDB.[1] > >> And this sounds like probably the largest concern. > >> * FDB cluster administration at scale isn't well documented.[2] > >> * The governance model isn't inspiring confidence, though the project > >> decisions haven't inspired distrust either.[3] > >> * The skill sets required to modify a layer (CouchDB) is different > >> than modifying the underlying database (FDB), and it's unclear what > >> degree of features can be requested and completed from the latter.[4] > >> Additionally, there's no consulting/FDB-as-a-service company from > >> which one could potentially pay for core changes. > >> * FoundationDB has been eager to utilize new hardware features for > >> performance, and doesn't degrade gracefully on older hardware.[5] > >> > >> Is there anything else that one might highlight as "If XYZ was > >> improved, a finished CouchDB-FDB would look like a more tenable > >> CouchDB 4.x"? > >> > >> Thanks, > >> Alex > >> > >> > >> [1]: > >> On Sat, Mar 12, 2022 at 1:26 AM Jan Lehnardt wrote: > >>> I know for some the 4.x release is highly anticipated and we as a > project hoped to make a generational jump for our underlying storage and > distribution technologies. During initial discussions about FDB-Couch and > during its development, we anticipated certain developments on the FDB side > (especially allowing longer transactions for consistent _changes responses > with their new Redwood storage engine). It is my understanding that these > developments have not materialised in the way we would like them. > >> > >> [2]: > >> On Sat, Mar 12, 2022 at 1:26 AM Jan Lehnardt wrote: > >>> We also learned that operating a FDB cluster is a significant effort > that somewhat goes against CouchDB’s mostly “just works” nature. We had > asked the IBM team to share their operational FDB learnings with the > CouchDB project, so we can build up community knowledge around this, but > this has not materialised either. > >> > >> [3]: > On 13 Mar 2022, at 11:17, Reddy B. wrote: > And this fact is also the cause of the unease I personally have this >
Re: Important update on couchdb's foundationdb work
Hi, I want to second the statements (Fynn, Ronny and others) that I see as support for option 1 in Jan’s three ways forward: 1. Follow IBM in abandoning FDB-Couch, refocus all effort on Erlang-Couch (3.x). “Follow IBM” in the meaning “staying together and have the benefit of a big brand supporter” is a good thing, also. I still think couchDB is unique, but simplicity for devops is the key benefit. Replication, offline first, distributed systems, built-in blockchain, attachments and designdocs that allow full-fledged apps to travel with the data, … the feature list goes on and on, and the heritage from the project’s start is beautiful and still very future-oriented. Best regards Johs > On 17 Mar 2022, at 21:25, Jan Lehnardt wrote: > > Heya Alex, > > I think all these are fair assessments and I can’t think of anything atop. > > Best > Jan > — > > >> On 16. Mar 2022, at 23:54, Alex Miller wrote: >> >> (With my hat on as a liaison from the FoundationDB community...) >> >> I imagine that, like all corporate decisions, stopping work on >> CouchDB-FDB was some part business reasons (which are entirely out of >> scope for this discussion) and some part technical reasons. I wanted >> to try and make sure to collect the feedback about any technical >> motivations that might have dissuaded the continued work on >> CouchDB-FDB as CouchDB 4.x. Thus far from watching the thread, I'm >> at: >> >> * Long lived read-only snapshots are required to maintain CouchDB 3.x >> semantics, and that feature has yet to be implemented in FDB.[1] >> And this sounds like probably the largest concern. >> * FDB cluster administration at scale isn't well documented.[2] >> * The governance model isn't inspiring confidence, though the project >> decisions haven't inspired distrust either.[3] >> * The skill sets required to modify a layer (CouchDB) is different >> than modifying the underlying database (FDB), and it's unclear what >> degree of features can be requested and completed from the latter.[4] >> Additionally, there's no consulting/FDB-as-a-service company from >> which one could potentially pay for core changes. >> * FoundationDB has been eager to utilize new hardware features for >> performance, and doesn't degrade gracefully on older hardware.[5] >> >> Is there anything else that one might highlight as "If XYZ was >> improved, a finished CouchDB-FDB would look like a more tenable >> CouchDB 4.x"? >> >> Thanks, >> Alex >> >> >> [1]: >> On Sat, Mar 12, 2022 at 1:26 AM Jan Lehnardt wrote: >>> I know for some the 4.x release is highly anticipated and we as a project >>> hoped to make a generational jump for our underlying storage and >>> distribution technologies. During initial discussions about FDB-Couch and >>> during its development, we anticipated certain developments on the FDB side >>> (especially allowing longer transactions for consistent _changes responses >>> with their new Redwood storage engine). It is my understanding that these >>> developments have not materialised in the way we would like them. >> >> [2]: >> On Sat, Mar 12, 2022 at 1:26 AM Jan Lehnardt wrote: >>> We also learned that operating a FDB cluster is a significant effort that >>> somewhat goes against CouchDB’s mostly “just works” nature. We had asked >>> the IBM team to share their operational FDB learnings with the CouchDB >>> project, so we can build up community knowledge around this, but this has >>> not materialised either. >> >> [3]: On 13 Mar 2022, at 11:17, Reddy B. wrote: And this fact is also the cause of the unease I personally have this FoundationDB migration: it looks like CouchDB will have much less control over its destiny and even philosophy. This is different from say an encrypted messaging app deciding to replace its home-made encryption with an established and more robust open-source solution. From day 1, I feel like this project will end up in FoundationDB integrating CouchDB rather than the other way around. I even suspected that maybe the dev team was no longer interested in CouchDB and wanted to find it a new home. My friendly feedback as a user is that I trust the Apache governance model much more than I trust Apple, especially when the welcome meal they have offer me is that features will be removed and limitations introduced. The political background and what I would call "corporate risk" (key capabilities not implemented by upstream, change in priority or vision, difficulty to affect the roadmap of upstream etc...), is also a key factor when choosing a DB solution as a user. >>> >>> On Sun, Mar 13, 2022 at 4:39 AM Robert Newson wrote: >>> I think it’s reasonable to worry about tying CouchDB to FoundationDB for >>> some of the reasons you mentioned, but not all of them. We did worry, at >>> the start, at the lack of a governance policy around FoundationDB; >>> something that would help
Re: Important update on couchdb's foundationdb work
Heya Alex, I think all these are fair assessments and I can’t think of anything atop. Best Jan — > On 16. Mar 2022, at 23:54, Alex Miller wrote: > > (With my hat on as a liaison from the FoundationDB community...) > > I imagine that, like all corporate decisions, stopping work on > CouchDB-FDB was some part business reasons (which are entirely out of > scope for this discussion) and some part technical reasons. I wanted > to try and make sure to collect the feedback about any technical > motivations that might have dissuaded the continued work on > CouchDB-FDB as CouchDB 4.x. Thus far from watching the thread, I'm > at: > > * Long lived read-only snapshots are required to maintain CouchDB 3.x > semantics, and that feature has yet to be implemented in FDB.[1] > And this sounds like probably the largest concern. > * FDB cluster administration at scale isn't well documented.[2] > * The governance model isn't inspiring confidence, though the project > decisions haven't inspired distrust either.[3] > * The skill sets required to modify a layer (CouchDB) is different > than modifying the underlying database (FDB), and it's unclear what > degree of features can be requested and completed from the latter.[4] > Additionally, there's no consulting/FDB-as-a-service company from > which one could potentially pay for core changes. > * FoundationDB has been eager to utilize new hardware features for > performance, and doesn't degrade gracefully on older hardware.[5] > > Is there anything else that one might highlight as "If XYZ was > improved, a finished CouchDB-FDB would look like a more tenable > CouchDB 4.x"? > > Thanks, > Alex > > > [1]: > On Sat, Mar 12, 2022 at 1:26 AM Jan Lehnardt wrote: >> I know for some the 4.x release is highly anticipated and we as a project >> hoped to make a generational jump for our underlying storage and >> distribution technologies. During initial discussions about FDB-Couch and >> during its development, we anticipated certain developments on the FDB side >> (especially allowing longer transactions for consistent _changes responses >> with their new Redwood storage engine). It is my understanding that these >> developments have not materialised in the way we would like them. > > [2]: > On Sat, Mar 12, 2022 at 1:26 AM Jan Lehnardt wrote: >> We also learned that operating a FDB cluster is a significant effort that >> somewhat goes against CouchDB’s mostly “just works” nature. We had asked the >> IBM team to share their operational FDB learnings with the CouchDB project, >> so we can build up community knowledge around this, but this has not >> materialised either. > > [3]: >>> On 13 Mar 2022, at 11:17, Reddy B. wrote: >>> And this fact is also the cause of the unease I personally have this >>> FoundationDB migration: it looks like CouchDB will have much less control >>> over its destiny and even philosophy. This is different from say an >>> encrypted messaging app deciding to replace its home-made encryption with >>> an established and more robust open-source solution. From day 1, I feel >>> like this project will end up in FoundationDB integrating CouchDB rather >>> than the other way around. I even suspected that maybe the dev team was no >>> longer interested in CouchDB and wanted to find it a new home. >>> >>> My friendly feedback as a user is that I trust the Apache governance model >>> much more than I trust Apple, especially when the welcome meal they have >>> offer me is that features will be removed and limitations introduced. The >>> political background and what I would call "corporate risk" (key >>> capabilities not implemented by upstream, change in priority or vision, >>> difficulty to affect the roadmap of upstream etc...), is also a key factor >>> when choosing a DB solution as a user. >> >> On Sun, Mar 13, 2022 at 4:39 AM Robert Newson wrote: >> I think it’s reasonable to worry about tying CouchDB to FoundationDB for >> some of the reasons you mentioned, but not all of them. We did worry, at the >> start, at the lack of a governance policy around FoundationDB; something >> that would help ensure the project is not beholden to a single corporate >> entity that might abandon the project or take it in places that make it >> unsuitable for CouchDB in the future. There hasn’t been much progress on >> that, but likewise the project has stayed true to form. > > [4]: >>> On 13 Mar 2022, at 11:17, Reddy B. wrote: >>> If even you guys weren't treated as a priority, I doubt that my feature >>> requests and other input will matter even one bit as a user. And I would >>> have zero chance of having the expertize required to modify the FDB core >>> myself and get my changes approved to make my CouchDB Layer- related >>> request possible. While right now I get can get my hands dirty and >>> eventually get something done if I really want to. The governance here is >>> very friendly, welcoming and inspiring trust. > > I do
Re: Important update on couchdb's foundationdb work
Hi all, I’m also an end user and maintainer of a fork of Colin Skow’s Superlogin (https://github.com/perfood/couch-auth) who is running a few single node CouchDBs with a few thousand per-User DBs. I agree with the general tone of Ronny’s mail - keeping things simple and having a low entry barrier for folks who want to write apps with CouchDB without worrying about the backend is how I’ve gotten into using it. Same for a few users of my library that I’ve been in touch with. I’m very excited about Jan resuming the work on document based access control. To me, this would make the difference between „DB per user is clunky + annoying, but works OK if you want to build an offline-first app“ to „data is all in one place, we can properly query data across users and I’d recommend CouchDB to anyone“. Kind regards and thanks to you devs, Fynn > Am 16.03.2022 um 14:03 schrieb Ronny Berndt : > > Hi, > > I think we have to look at the discussion from different angles. > It would be interesting to know how CouchDB is used in general. > > - How many users use CouchDB as a single-node database? > - How many users use CouchDB as a clustered database? > - How many CouchDB users use a cluster and reach the limits of CouchDB? > > (My) view as an end user or as a developer using CouchDB as a single-node > data backend: > > Try to keep things simple. Normally, as a user, I don't want or need to know > how a kernel of an operating system, or an engine of a car works. When i > press the > "start" button, it should run and work. The same applies to the storage of > data through CouchDB. These should somehow be saved as quickly and as best > as possible, but how > exactly what works doesn't matter to me as an end user. > > Furthermore, it should be easy to get started with installing and > administering CouchDB. > There are already countless setting options to adapt CouchDB to your needs. > My observation on Slack is that most (myself included) have much simpler > problems. > CORS-, Authentication-, Mango view-, Javascript view-, Q/N-, "bad redable > and formatted" (erlang) > error message-, DB-Backup-, Fauxton-, Replicator-, and Query-parameter > questions, to name a few. > > Development using FoundationDB as the storage backend was fairly silent. I > have no > feeling, or it is not entirely clear to me what advantages (or > disadvantages) FDB could bring as an end user, > but I don't want to have to worry about configuring a FoundationDB cluster > yet - but > maybe I'm just misjudging it and it wouldn't be like that at all. > > As some have pointed out, there are many small and useful improvements to > CouchDB 3.x > exist, which probably makes more sense for the majority of users. Robert > wrote it > that the possibility of an alternative storage backend has existed since > around 2016 and none was implemented > until now. My gut feeling is that it probably doesn't make sense to have > CouchDB-FDB in active > development state. Perhaps the best of what's going on now should be > carried over into CouchDB 3.x. > Others have raised many interesting points that I think make more sense to > implement than CouchDB-FDB. > > CouchDB is currently being pushed forward by a few developers, which is > probably due to the fact > that there are few Erlang developers who are also actively using CouchDB > and can push the development forward. > > - Ronny > > > Am Di., 15. März 2022 um 22:52 Uhr schrieb Juan José Rodríguez < > jjrod...@gmail.com>: > >> Hi, >> >> My team has been using CouchDB in different projects to support >> synchronization in mobile apps. We don't use a db-per-user pattern strictly >> but something that comes pretty close. >> >> So far we have not encountered the limits and we are comfortable with >> CouchDB 3.x., we were not particularly interested in incorporating >> FoundationDB as we intuited a significant increase in the complexity of the >> architecture which, at least in our case, was not necessary (admittedly in >> exchange for very interesting functionalities). >> >> We prefer to refocus development on CouchDB 3.x, we do not have a clear >> position on the CouchDB-FDB option but would be concerned about any option >> that would result in dividing the project efforts. >> >> We believe that CouchDB 3.x has a lot of room for development with features >> that can help increase its adoption and improve its usage at least in our >> context: >> - Incorporate support for mobile sync clients, e.g. allow initial syncs >> without deletes. >> - Rethink the db-per-user pattern, perhaps as an idea partitioned _changes >> streams could help improve this aspect. >> - Allow permanent document deletions or something like expiration policies >> for deleted content. Any functionality that helps to keep databases >> compact. >> - Perhaps, better support for conflict resolution >> >> Thanks, >> Juanjo >> >> El jue, 10 mar 2022 a las 17:24, Robert Newson () >> escribió: >> >>> Hi, >>> >>> For those
Re: Important update on couchdb's foundationdb work
(With my hat on as a liaison from the FoundationDB community...) I imagine that, like all corporate decisions, stopping work on CouchDB-FDB was some part business reasons (which are entirely out of scope for this discussion) and some part technical reasons. I wanted to try and make sure to collect the feedback about any technical motivations that might have dissuaded the continued work on CouchDB-FDB as CouchDB 4.x. Thus far from watching the thread, I'm at: * Long lived read-only snapshots are required to maintain CouchDB 3.x semantics, and that feature has yet to be implemented in FDB.[1] And this sounds like probably the largest concern. * FDB cluster administration at scale isn't well documented.[2] * The governance model isn't inspiring confidence, though the project decisions haven't inspired distrust either.[3] * The skill sets required to modify a layer (CouchDB) is different than modifying the underlying database (FDB), and it's unclear what degree of features can be requested and completed from the latter.[4] Additionally, there's no consulting/FDB-as-a-service company from which one could potentially pay for core changes. * FoundationDB has been eager to utilize new hardware features for performance, and doesn't degrade gracefully on older hardware.[5] Is there anything else that one might highlight as "If XYZ was improved, a finished CouchDB-FDB would look like a more tenable CouchDB 4.x"? Thanks, Alex [1]: On Sat, Mar 12, 2022 at 1:26 AM Jan Lehnardt wrote: > I know for some the 4.x release is highly anticipated and we as a project > hoped to make a generational jump for our underlying storage and distribution > technologies. During initial discussions about FDB-Couch and during its > development, we anticipated certain developments on the FDB side (especially > allowing longer transactions for consistent _changes responses with their new > Redwood storage engine). It is my understanding that these developments have > not materialised in the way we would like them. [2]: On Sat, Mar 12, 2022 at 1:26 AM Jan Lehnardt wrote: > We also learned that operating a FDB cluster is a significant effort that > somewhat goes against CouchDB’s mostly “just works” nature. We had asked the > IBM team to share their operational FDB learnings with the CouchDB project, > so we can build up community knowledge around this, but this has not > materialised either. [3]: > > On 13 Mar 2022, at 11:17, Reddy B. wrote: > > And this fact is also the cause of the unease I personally have this > > FoundationDB migration: it looks like CouchDB will have much less control > > over its destiny and even philosophy. This is different from say an > > encrypted messaging app deciding to replace its home-made encryption with > > an established and more robust open-source solution. From day 1, I feel > > like this project will end up in FoundationDB integrating CouchDB rather > > than the other way around. I even suspected that maybe the dev team was no > > longer interested in CouchDB and wanted to find it a new home. > > > > My friendly feedback as a user is that I trust the Apache governance model > > much more than I trust Apple, especially when the welcome meal they have > > offer me is that features will be removed and limitations introduced. The > > political background and what I would call "corporate risk" (key > > capabilities not implemented by upstream, change in priority or vision, > > difficulty to affect the roadmap of upstream etc...), is also a key factor > > when choosing a DB solution as a user. > > On Sun, Mar 13, 2022 at 4:39 AM Robert Newson wrote: > I think it’s reasonable to worry about tying CouchDB to FoundationDB for some > of the reasons you mentioned, but not all of them. We did worry, at the > start, at the lack of a governance policy around FoundationDB; something that > would help ensure the project is not beholden to a single corporate entity > that might abandon the project or take it in places that make it unsuitable > for CouchDB in the future. There hasn’t been much progress on that, but > likewise the project has stayed true to form. [4]: > > On 13 Mar 2022, at 11:17, Reddy B. wrote: > > If even you guys weren't treated as a priority, I doubt that my feature > > requests and other input will matter even one bit as a user. And I would > > have zero chance of having the expertize required to modify the FDB core > > myself and get my changes approved to make my CouchDB Layer- related > > request possible. While right now I get can get my hands dirty and > > eventually get something done if I really want to. The governance here is > > very friendly, welcoming and inspiring trust. I do feel the need to call out that there were a few contributions to FDB from CouchDB folk, and IBM Research Zurich (Diego Didona, et. al) in particular contributed some great storage engine improvements. [5]: On Tue, Mar 15, 2022 at 5:33 AM Will Young wrote: >
Re: Important update on couchdb's foundationdb work
Just clarifying one point further down > On 16. Mar 2022, at 14:03, Ronny Berndt wrote: > > Hi, > > I think we have to look at the discussion from different angles. > It would be interesting to know how CouchDB is used in general. > > - How many users use CouchDB as a single-node database? > - How many users use CouchDB as a clustered database? > - How many CouchDB users use a cluster and reach the limits of CouchDB? > > (My) view as an end user or as a developer using CouchDB as a single-node > data backend: > > Try to keep things simple. Normally, as a user, I don't want or need to know > how a kernel of an operating system, or an engine of a car works. When i > press the > "start" button, it should run and work. The same applies to the storage of > data through CouchDB. These should somehow be saved as quickly and as best > as possible, but how > exactly what works doesn't matter to me as an end user. > > Furthermore, it should be easy to get started with installing and > administering CouchDB. > There are already countless setting options to adapt CouchDB to your needs. > My observation on Slack is that most (myself included) have much simpler > problems. > CORS-, Authentication-, Mango view-, Javascript view-, Q/N-, "bad redable > and formatted" (erlang) > error message-, DB-Backup-, Fauxton-, Replicator-, and Query-parameter > questions, to name a few. > > Development using FoundationDB as the storage backend was fairly silent. I > have no > feeling, or it is not entirely clear to me what advantages (or > disadvantages) FDB could bring as an end user, > but I don't want to have to worry about configuring a FoundationDB cluster > yet - but > maybe I'm just misjudging it and it wouldn't be like that at all. > > As some have pointed out, there are many small and useful improvements to > CouchDB 3.x > exist, which probably makes more sense for the majority of users. > Robert > wrote it > that the possibility of an alternative storage backend has existed since > around 2016 and none was implemented > until now. The original suggestion in this thread was thinking of CouchDB as a layer on top of a distributed data store, like FDB-Couch, but with other “backends” that cover distribution in a network and data storage. The current 3.x codebase has a feature “pluggable storage engines” that ships with the default implementation `couch_bt_engine`. There is one early and affect abandoned prototype of an in-memory implementation (hello Chewbranca ;) and I have wrote ~85% of a SQLite-based storage engine that I have not open- sourced, and I don’t know if I will be able to, but it exists and largely works. Note that the storage here is on each CouchDB node, that’s a part of the CouchDB layer cake that comes *after* distribution. In contrast, FDB takes over distribution *and* storage for us. Let’s not conflate the two concepts, and take the (lack of) interest in one as an outlook for usefulness of the other. (I think PSE for Erlang-CouchDB is useful and I hope we get at least the two variants I mentioned above properly released at some point, but that’s for a future roadmap thread, a RocksDB variant would be feasible too). What I don’t really see is an abstraction layer that allows end-users to choose between multiple distributed data stores like FDB (say DynamoDB or CosmosDB), even though that would be *really* interesting for CouchDB-interop. Those implementations would be either a lot of work, or would be so generic that they have to eschew many underlying-system-specific optimisations[1], that’s why don’t think these would appear any time soon. But they *could* feasibly be done, given enough developer time/interest. [1]: I have some experience writing a PouchDB storage backend for CosmosDB using the leveldown API and while it works, it is not taking advantage of running on a database as sophisticated as CosmosDB (née DocumentDB). Best Jan > My gut feeling is that it probably doesn't make sense to have > CouchDB-FDB in active > development state. Perhaps the best of what's going on now should be > carried over into CouchDB 3.x. > Others have raised many interesting points that I think make more sense to > implement than CouchDB-FDB. > > CouchDB is currently being pushed forward by a few developers, which is > probably due to the fact > that there are few Erlang developers who are also actively using CouchDB > and can push the development forward. > > - Ronny > > > Am Di., 15. März 2022 um 22:52 Uhr schrieb Juan José Rodríguez < > jjrod...@gmail.com>: > >> Hi, >> >> My team has been using CouchDB in different projects to support >> synchronization in mobile apps. We don't use a db-per-user pattern strictly >> but something that comes pretty close. >> >> So far we have not encountered the limits and we are comfortable with >> CouchDB 3.x., we were not particularly interested in incorporating >> FoundationDB as we intuited a significant increase in the complexity of the >> architecture
Re: Important update on couchdb's foundationdb work
Hi, I think we have to look at the discussion from different angles. It would be interesting to know how CouchDB is used in general. - How many users use CouchDB as a single-node database? - How many users use CouchDB as a clustered database? - How many CouchDB users use a cluster and reach the limits of CouchDB? (My) view as an end user or as a developer using CouchDB as a single-node data backend: Try to keep things simple. Normally, as a user, I don't want or need to know how a kernel of an operating system, or an engine of a car works. When i press the "start" button, it should run and work. The same applies to the storage of data through CouchDB. These should somehow be saved as quickly and as best as possible, but how exactly what works doesn't matter to me as an end user. Furthermore, it should be easy to get started with installing and administering CouchDB. There are already countless setting options to adapt CouchDB to your needs. My observation on Slack is that most (myself included) have much simpler problems. CORS-, Authentication-, Mango view-, Javascript view-, Q/N-, "bad redable and formatted" (erlang) error message-, DB-Backup-, Fauxton-, Replicator-, and Query-parameter questions, to name a few. Development using FoundationDB as the storage backend was fairly silent. I have no feeling, or it is not entirely clear to me what advantages (or disadvantages) FDB could bring as an end user, but I don't want to have to worry about configuring a FoundationDB cluster yet - but maybe I'm just misjudging it and it wouldn't be like that at all. As some have pointed out, there are many small and useful improvements to CouchDB 3.x exist, which probably makes more sense for the majority of users. Robert wrote it that the possibility of an alternative storage backend has existed since around 2016 and none was implemented until now. My gut feeling is that it probably doesn't make sense to have CouchDB-FDB in active development state. Perhaps the best of what's going on now should be carried over into CouchDB 3.x. Others have raised many interesting points that I think make more sense to implement than CouchDB-FDB. CouchDB is currently being pushed forward by a few developers, which is probably due to the fact that there are few Erlang developers who are also actively using CouchDB and can push the development forward. - Ronny Am Di., 15. März 2022 um 22:52 Uhr schrieb Juan José Rodríguez < jjrod...@gmail.com>: > Hi, > > My team has been using CouchDB in different projects to support > synchronization in mobile apps. We don't use a db-per-user pattern strictly > but something that comes pretty close. > > So far we have not encountered the limits and we are comfortable with > CouchDB 3.x., we were not particularly interested in incorporating > FoundationDB as we intuited a significant increase in the complexity of the > architecture which, at least in our case, was not necessary (admittedly in > exchange for very interesting functionalities). > > We prefer to refocus development on CouchDB 3.x, we do not have a clear > position on the CouchDB-FDB option but would be concerned about any option > that would result in dividing the project efforts. > > We believe that CouchDB 3.x has a lot of room for development with features > that can help increase its adoption and improve its usage at least in our > context: > - Incorporate support for mobile sync clients, e.g. allow initial syncs > without deletes. > - Rethink the db-per-user pattern, perhaps as an idea partitioned _changes > streams could help improve this aspect. > - Allow permanent document deletions or something like expiration policies > for deleted content. Any functionality that helps to keep databases > compact. > - Perhaps, better support for conflict resolution > > Thanks, > Juanjo > > El jue, 10 mar 2022 a las 17:24, Robert Newson () > escribió: > > > Hi, > > > > For those that are following closely, and particularly those that build > or > > use CouchDB from our git repo, you'll be aware that CouchDB embarked on > an > > attempt to build a next-generation version of CouchDB using the > > FoundationDB database engine as its new base. > > > > The principal sponsors of this work, the Cloudant team at IBM, have > > informed us that, unfortunately, they will not be continuing to fund the > > development of this version and are refocusing their efforts on CouchDB > 3.x. > > > > Cloudant developers will continue to contribute as they always have done > > and the CouchDB PMC thanks them for their efforts. > > > > As the Project Management Committee for the CouchDB project, we are now > > asking the developer community how we’d like to proceed in light of this > > new information. > > > > Regards, > > Robert Newson > > Apache CouchDB PMC > > > > >
Re: Important update on couchdb's foundationdb work
Hi, My team has been using CouchDB in different projects to support synchronization in mobile apps. We don't use a db-per-user pattern strictly but something that comes pretty close. So far we have not encountered the limits and we are comfortable with CouchDB 3.x., we were not particularly interested in incorporating FoundationDB as we intuited a significant increase in the complexity of the architecture which, at least in our case, was not necessary (admittedly in exchange for very interesting functionalities). We prefer to refocus development on CouchDB 3.x, we do not have a clear position on the CouchDB-FDB option but would be concerned about any option that would result in dividing the project efforts. We believe that CouchDB 3.x has a lot of room for development with features that can help increase its adoption and improve its usage at least in our context: - Incorporate support for mobile sync clients, e.g. allow initial syncs without deletes. - Rethink the db-per-user pattern, perhaps as an idea partitioned _changes streams could help improve this aspect. - Allow permanent document deletions or something like expiration policies for deleted content. Any functionality that helps to keep databases compact. - Perhaps, better support for conflict resolution Thanks, Juanjo El jue, 10 mar 2022 a las 17:24, Robert Newson () escribió: > Hi, > > For those that are following closely, and particularly those that build or > use CouchDB from our git repo, you'll be aware that CouchDB embarked on an > attempt to build a next-generation version of CouchDB using the > FoundationDB database engine as its new base. > > The principal sponsors of this work, the Cloudant team at IBM, have > informed us that, unfortunately, they will not be continuing to fund the > development of this version and are refocusing their efforts on CouchDB 3.x. > > Cloudant developers will continue to contribute as they always have done > and the CouchDB PMC thanks them for their efforts. > > As the Project Management Committee for the CouchDB project, we are now > asking the developer community how we’d like to proceed in light of this > new information. > > Regards, > Robert Newson > Apache CouchDB PMC > >
Re: Important update on couchdb's foundationdb work
Hi, Personally, I had no specific need in mind for foundationdb to address, I'm well served by the current storage atop ZFS, so I mostly see downsides in adding any storage that adds new restrictions, non-portable dependencies, or puts groups of features into silos. I.e. foundationdb's requirement for avx was a headache for me as I have some older x86 as well as Arm, where avx2neon seemed experimental. Several host local DBs like RocksDB and Sqlite seem like they might provide a modest benefit while less likely to create a split somewhere between anyone using couch for embedded, anyone building a modest cluster and anyone building a full scale cloud DB service. So if there is an interest in continuing to develop an alternative to the current storage, I would prefer investigations into the benefits of a direct/local RocksDB storage engine rather than having a portion of the community invest more time into continuing a CouchDB-FDB project. -Will Am Mo., 14. März 2022 um 12:19 Uhr schrieb Robert Newson : > > Hi, > > That already happened. “Pluggable storage engine” was introduced in 2016 > (https://github.com/apache/couchdb/commit/f6a771147ba488f80a7d29491263d19088d0eefb). > > No alternative backends have yet been contributed. > > B. > > > On 13 Mar 2022, at 16:27, Chintan Mishra from Rebhu > > wrote: > > > > As a user, my team and I were keenly looking forward to CouchDB v4 with > > FoundationDB. > > > > Given the current situation, it is only reasonable to come up with a best > > alternative. > > > > How about refactoring CouchDB to work with multiple storage engines? > > > > The default CouchDB will support whatever the PMC agrees upon. Whereas the > > community can tinker with different backend storage engines. So, the > > FoundationDB can be one of the backing engines that get used with CouchDB. > > Other storage engines can be RocksDB, Apache Derby, etc. > > > > Thank you. > > > > -- > > Chintan Mishra > > > > On 13/03/22 17:09, Robert Newson wrote: > >> Thank you for this feedback. > >> I think it’s reasonable to worry about tying CouchDB to FoundationDB for > >> some of the reasons you mentioned, but not all of them. We did worry, at > >> the start, at the lack of a governance policy around FoundationDB; > >> something that would help ensure the project is not beholden to a single > >> corporate entity that might abandon the project or take it in places that > >> make it unsuitable for CouchDB in the future. There hasn’t been much > >> progress on that, but likewise the project has stayed true to form. > >> CouchDB is critically dependent on Erlang/OTP, among other components, > >> which similarly lack the kind of governance or oversight that Apache > >> projects themselves work within. At no point have I feared the "project > >> will end up in FoundationDB integrating CouchDB rather than the other way > >> around”. FoundationDB is not a database, it is explicitly only > >> foundational support to build databases on top of. > >> "If even you guys weren't treated as a priority, I doubt that my feature > >> requests and other input will matter even one bit as a user.” - I’m not > >> sure who you refer to with “you guys”, but I remind everyone that the > >> CouchDB contributors from IBM Cloudant are the main contributors to > >> CouchDB 2.0 and 3.0, have been so for years and are in, many cases, either > >> CouchDB committers or PMC members. They are “us” as much as any other > >> contributor. That the Cloudant team has moved focus from CouchDB 4.0 (as > >> it would have been) to 3.0 is a re-establishment of the status quo ante. > >> "I doubt that my feature requests and other input will matter even one bit > >> as a user.” — I strongly disagree here. Community contributions are hugely > >> valuable and valued, the rewrite of the lower layers of CouchDB would not > >> have changed that significantly. CouchDB-FDB is still written in Erlang, > >> the http layer is largely the same code as before. The parts that interact > >> with FoundationDB are confined to a single library application (erlfdb) > >> which exposes the C language bindings as Erlang functions and data > >> structures. Unless you are working at that level you can mostly ignore it. > >> Finally, while I don’t think we’ve explicitly described it this way, > >> CouchDB-FDB effectively _is_ a “layer” on top of FDB in the same sense > >> that their “document layer” (which is mongo-like) is. > >> B. > >>> On 13 Mar 2022, at 11:17, Reddy B. wrote: > >>> > >>> Hello! > >>> > >>> Thanks a lot for this update and overview of the situation. As users (our > >>> company has been using couchdb since 2015 circa as the main database of > >>> our 3 tier web apps), I feel it may be preferable to move the couchdb-fdb > >>> work to a separate project having a different name. As Janh has > >>> mentioned, the internals and daily management of FDB may with certain > >>> regards be at odds with the philosophy and user
Re: Important update on couchdb's foundationdb work
Hi, That already happened. “Pluggable storage engine” was introduced in 2016 (https://github.com/apache/couchdb/commit/f6a771147ba488f80a7d29491263d19088d0eefb). No alternative backends have yet been contributed. B. > On 13 Mar 2022, at 16:27, Chintan Mishra from Rebhu wrote: > > As a user, my team and I were keenly looking forward to CouchDB v4 with > FoundationDB. > > Given the current situation, it is only reasonable to come up with a best > alternative. > > How about refactoring CouchDB to work with multiple storage engines? > > The default CouchDB will support whatever the PMC agrees upon. Whereas the > community can tinker with different backend storage engines. So, the > FoundationDB can be one of the backing engines that get used with CouchDB. > Other storage engines can be RocksDB, Apache Derby, etc. > > Thank you. > > -- > Chintan Mishra > > On 13/03/22 17:09, Robert Newson wrote: >> Thank you for this feedback. >> I think it’s reasonable to worry about tying CouchDB to FoundationDB for >> some of the reasons you mentioned, but not all of them. We did worry, at the >> start, at the lack of a governance policy around FoundationDB; something >> that would help ensure the project is not beholden to a single corporate >> entity that might abandon the project or take it in places that make it >> unsuitable for CouchDB in the future. There hasn’t been much progress on >> that, but likewise the project has stayed true to form. >> CouchDB is critically dependent on Erlang/OTP, among other components, which >> similarly lack the kind of governance or oversight that Apache projects >> themselves work within. At no point have I feared the "project will end up >> in FoundationDB integrating CouchDB rather than the other way around”. >> FoundationDB is not a database, it is explicitly only foundational support >> to build databases on top of. >> "If even you guys weren't treated as a priority, I doubt that my feature >> requests and other input will matter even one bit as a user.” - I’m not sure >> who you refer to with “you guys”, but I remind everyone that the CouchDB >> contributors from IBM Cloudant are the main contributors to CouchDB 2.0 and >> 3.0, have been so for years and are in, many cases, either CouchDB >> committers or PMC members. They are “us” as much as any other contributor. >> That the Cloudant team has moved focus from CouchDB 4.0 (as it would have >> been) to 3.0 is a re-establishment of the status quo ante. >> "I doubt that my feature requests and other input will matter even one bit >> as a user.” — I strongly disagree here. Community contributions are hugely >> valuable and valued, the rewrite of the lower layers of CouchDB would not >> have changed that significantly. CouchDB-FDB is still written in Erlang, the >> http layer is largely the same code as before. The parts that interact with >> FoundationDB are confined to a single library application (erlfdb) which >> exposes the C language bindings as Erlang functions and data structures. >> Unless you are working at that level you can mostly ignore it. >> Finally, while I don’t think we’ve explicitly described it this way, >> CouchDB-FDB effectively _is_ a “layer” on top of FDB in the same sense that >> their “document layer” (which is mongo-like) is. >> B. >>> On 13 Mar 2022, at 11:17, Reddy B. wrote: >>> >>> Hello! >>> >>> Thanks a lot for this update and overview of the situation. As users (our >>> company has been using couchdb since 2015 circa as the main database of our >>> 3 tier web apps), I feel it may be preferable to move the couchdb-fdb work >>> to a separate project having a different name. As Janh has mentioned, the >>> internals and daily management of FDB may with certain regards be at odds >>> with the philosophy and user experience that couchdb wants to provide. >>> >>> Moving this effort to a different project would give people interested in >>> this effort more flexibility to introduce breaking changes and limitations >>> taking full advantage of the philosophy of FDB. I feel the idea that: if >>> you have outscaled CouchDB, move to couchdb-fdb (or another more >>> specialized DB) is the right idea. Couchdb-fdb advantage compared to >>> alternative would simply be that it implements both the replication >>> protocol and the HTTP API. >>> >>> This project may/should even "simply" become something under the umbrella >>> of the FoundationDB layer similar to the MongoDB-compatible document layer >>> of FoundationDB [1]. >>> >>> And this fact is also the cause of the unease I personally have this >>> FoundationDB migration: it looks like CouchDB will have much less control >>> over its destiny and even philosophy. This is different from say an >>> encrypted messaging app deciding to replace its home-made encryption with >>> an established and more robust open-source solution. From day 1, I feel >>> like this project will end up in FoundationDB
Re: Important update on couchdb's foundationdb work
As a user, my team and I were keenly looking forward to CouchDB v4 with FoundationDB. Given the current situation, it is only reasonable to come up with a best alternative. How about refactoring CouchDB to work with multiple storage engines? The default CouchDB will support whatever the PMC agrees upon. Whereas the community can tinker with different backend storage engines. So, the FoundationDB can be one of the backing engines that get used with CouchDB. Other storage engines can be RocksDB, Apache Derby, etc. Thank you. -- Chintan Mishra On 13/03/22 17:09, Robert Newson wrote: Thank you for this feedback. I think it’s reasonable to worry about tying CouchDB to FoundationDB for some of the reasons you mentioned, but not all of them. We did worry, at the start, at the lack of a governance policy around FoundationDB; something that would help ensure the project is not beholden to a single corporate entity that might abandon the project or take it in places that make it unsuitable for CouchDB in the future. There hasn’t been much progress on that, but likewise the project has stayed true to form. CouchDB is critically dependent on Erlang/OTP, among other components, which similarly lack the kind of governance or oversight that Apache projects themselves work within. At no point have I feared the "project will end up in FoundationDB integrating CouchDB rather than the other way around”. FoundationDB is not a database, it is explicitly only foundational support to build databases on top of. "If even you guys weren't treated as a priority, I doubt that my feature requests and other input will matter even one bit as a user.” - I’m not sure who you refer to with “you guys”, but I remind everyone that the CouchDB contributors from IBM Cloudant are the main contributors to CouchDB 2.0 and 3.0, have been so for years and are in, many cases, either CouchDB committers or PMC members. They are “us” as much as any other contributor. That the Cloudant team has moved focus from CouchDB 4.0 (as it would have been) to 3.0 is a re-establishment of the status quo ante. "I doubt that my feature requests and other input will matter even one bit as a user.” — I strongly disagree here. Community contributions are hugely valuable and valued, the rewrite of the lower layers of CouchDB would not have changed that significantly. CouchDB-FDB is still written in Erlang, the http layer is largely the same code as before. The parts that interact with FoundationDB are confined to a single library application (erlfdb) which exposes the C language bindings as Erlang functions and data structures. Unless you are working at that level you can mostly ignore it. Finally, while I don’t think we’ve explicitly described it this way, CouchDB-FDB effectively _is_ a “layer” on top of FDB in the same sense that their “document layer” (which is mongo-like) is. B. On 13 Mar 2022, at 11:17, Reddy B. wrote: Hello! Thanks a lot for this update and overview of the situation. As users (our company has been using couchdb since 2015 circa as the main database of our 3 tier web apps), I feel it may be preferable to move the couchdb-fdb work to a separate project having a different name. As Janh has mentioned, the internals and daily management of FDB may with certain regards be at odds with the philosophy and user experience that couchdb wants to provide. Moving this effort to a different project would give people interested in this effort more flexibility to introduce breaking changes and limitations taking full advantage of the philosophy of FDB. I feel the idea that: if you have outscaled CouchDB, move to couchdb-fdb (or another more specialized DB) is the right idea. Couchdb-fdb advantage compared to alternative would simply be that it implements both the replication protocol and the HTTP API. This project may/should even "simply" become something under the umbrella of the FoundationDB layer similar to the MongoDB-compatible document layer of FoundationDB [1]. And this fact is also the cause of the unease I personally have this FoundationDB migration: it looks like CouchDB will have much less control over its destiny and even philosophy. This is different from say an encrypted messaging app deciding to replace its home-made encryption with an established and more robust open-source solution. From day 1, I feel like this project will end up in FoundationDB integrating CouchDB rather than the other way around. I even suspected that maybe the dev team was no longer interested in CouchDB and wanted to find it a new home. My friendly feedback as a user is that I trust the Apache governance model much more than I trust Apple, especially when the welcome meal they have offer me is that features will be removed and limitations introduced. The political background and what I would call "corporate risk" (key capabilities not implemented by upstream, change in priority or vision,
Re: Important update on couchdb's foundationdb work
Thank you for this feedback. I think it’s reasonable to worry about tying CouchDB to FoundationDB for some of the reasons you mentioned, but not all of them. We did worry, at the start, at the lack of a governance policy around FoundationDB; something that would help ensure the project is not beholden to a single corporate entity that might abandon the project or take it in places that make it unsuitable for CouchDB in the future. There hasn’t been much progress on that, but likewise the project has stayed true to form. CouchDB is critically dependent on Erlang/OTP, among other components, which similarly lack the kind of governance or oversight that Apache projects themselves work within. At no point have I feared the "project will end up in FoundationDB integrating CouchDB rather than the other way around”. FoundationDB is not a database, it is explicitly only foundational support to build databases on top of. "If even you guys weren't treated as a priority, I doubt that my feature requests and other input will matter even one bit as a user.” - I’m not sure who you refer to with “you guys”, but I remind everyone that the CouchDB contributors from IBM Cloudant are the main contributors to CouchDB 2.0 and 3.0, have been so for years and are in, many cases, either CouchDB committers or PMC members. They are “us” as much as any other contributor. That the Cloudant team has moved focus from CouchDB 4.0 (as it would have been) to 3.0 is a re-establishment of the status quo ante. "I doubt that my feature requests and other input will matter even one bit as a user.” — I strongly disagree here. Community contributions are hugely valuable and valued, the rewrite of the lower layers of CouchDB would not have changed that significantly. CouchDB-FDB is still written in Erlang, the http layer is largely the same code as before. The parts that interact with FoundationDB are confined to a single library application (erlfdb) which exposes the C language bindings as Erlang functions and data structures. Unless you are working at that level you can mostly ignore it. Finally, while I don’t think we’ve explicitly described it this way, CouchDB-FDB effectively _is_ a “layer” on top of FDB in the same sense that their “document layer” (which is mongo-like) is. B. > On 13 Mar 2022, at 11:17, Reddy B. wrote: > > Hello! > > Thanks a lot for this update and overview of the situation. As users (our > company has been using couchdb since 2015 circa as the main database of our 3 > tier web apps), I feel it may be preferable to move the couchdb-fdb work to a > separate project having a different name. As Janh has mentioned, the > internals and daily management of FDB may with certain regards be at odds > with the philosophy and user experience that couchdb wants to provide. > > Moving this effort to a different project would give people interested in > this effort more flexibility to introduce breaking changes and limitations > taking full advantage of the philosophy of FDB. I feel the idea that: if you > have outscaled CouchDB, move to couchdb-fdb (or another more specialized DB) > is the right idea. Couchdb-fdb advantage compared to alternative would simply > be that it implements both the replication protocol and the HTTP API. > > This project may/should even "simply" become something under the umbrella of > the FoundationDB layer similar to the MongoDB-compatible document layer of > FoundationDB [1]. > > And this fact is also the cause of the unease I personally have this > FoundationDB migration: it looks like CouchDB will have much less control > over its destiny and even philosophy. This is different from say an encrypted > messaging app deciding to replace its home-made encryption with an > established and more robust open-source solution. From day 1, I feel like > this project will end up in FoundationDB integrating CouchDB rather than the > other way around. I even suspected that maybe the dev team was no longer > interested in CouchDB and wanted to find it a new home. > > My friendly feedback as a user is that I trust the Apache governance model > much more than I trust Apple, especially when the welcome meal they have > offer me is that features will be removed and limitations introduced. The > political background and what I would call "corporate risk" (key capabilities > not implemented by upstream, change in priority or vision, difficulty to > affect the roadmap of upstream etc...), is also a key factor when choosing a > DB solution as a user. > > If even you guys weren't treated as a priority, I doubt that my feature > requests and other input will matter even one bit as a user. And I would have > zero chance of having the expertize required to modify the FDB core myself > and get my changes approved to make my CouchDB Layer- related request > possible. While right now I get can get my hands dirty and eventually get > something done if I really
Thanks for getting in touch! We've received your email: Re: Important update on couchdb's foundationdb work
Hi there, We have received your request #1692199 and we'll start working on it as soon as possible. We usually respond within 24 hours, but please keep in mind that there may be a delay in response over the weekend. In the meantime, check our Help Center at https://support.ecwid.com/hc/en-us Thank you, Ecwid Customer Care Team
Re: Important update on couchdb's foundationdb work
Hello! Thanks a lot for this update and overview of the situation. As users (our company has been using couchdb since 2015 circa as the main database of our 3 tier web apps), I feel it may be preferable to move the couchdb-fdb work to a separate project having a different name. As Janh has mentioned, the internals and daily management of FDB may with certain regards be at odds with the philosophy and user experience that couchdb wants to provide. Moving this effort to a different project would give people interested in this effort more flexibility to introduce breaking changes and limitations taking full advantage of the philosophy of FDB. I feel the idea that: if you have outscaled CouchDB, move to couchdb-fdb (or another more specialized DB) is the right idea. Couchdb-fdb advantage compared to alternative would simply be that it implements both the replication protocol and the HTTP API. This project may/should even "simply" become something under the umbrella of the FoundationDB layer similar to the MongoDB-compatible document layer of FoundationDB [1]. And this fact is also the cause of the unease I personally have this FoundationDB migration: it looks like CouchDB will have much less control over its destiny and even philosophy. This is different from say an encrypted messaging app deciding to replace its home-made encryption with an established and more robust open-source solution. From day 1, I feel like this project will end up in FoundationDB integrating CouchDB rather than the other way around. I even suspected that maybe the dev team was no longer interested in CouchDB and wanted to find it a new home. My friendly feedback as a user is that I trust the Apache governance model much more than I trust Apple, especially when the welcome meal they have offer me is that features will be removed and limitations introduced. The political background and what I would call "corporate risk" (key capabilities not implemented by upstream, change in priority or vision, difficulty to affect the roadmap of upstream etc...), is also a key factor when choosing a DB solution as a user. If even you guys weren't treated as a priority, I doubt that my feature requests and other input will matter even one bit as a user. And I would have zero chance of having the expertize required to modify the FDB core myself and get my changes approved to make my CouchDB Layer- related request possible. While right now I get can get my hands dirty and eventually get something done if I really want to. The governance here is very friendly, welcoming and inspiring trust. So to summarize, I feel that to realize the full potential of this vision rather than settling on compromises not satisfying anyone, it may be better to treat it as a separate project and let CouchDB remain CouchDB. I also feel that the project would lose too much control and sovereignty with such a migration, especially in light of the facts reported. The scaling challenges and limitations that motivated this effort may probably be addressed differently with a fresh outlook. For example, nowadays, there are even application-level middleware libraries like Microsoft Orleans being able to coordinate ACID distributed transactions from the application layer. My point is, challenges may be able to be overcome overtime by approaching things in a creative manner. Users may be able to workaround some of them by adjusting the topology of their clusters (using single writer, huge single node with distributed file systems etc...), for other challenges application-layer solutions may exist, or the solution may simply be shipping extremely user friendly graphical management tools making for example things such as conflict resolution a breeze for the admin. My 2 cents [1]: https://github.com/FoundationDB/fdb-document-layer 12 mars 2022 10:26:35 Jan Lehnardt : > Thanks Bob for passing this along. > > I’m looking forward to renewed interest in the 3.x codebase :) > > For our 4.x plans, we’ll have to discuss here what we want to do with it and > I’m looking at everyone for input here. Even if you’ve never spoken up on > this list before, I’d lie to hear from you. > > * * * > > First off, as a project, CouchDB is not obliged to follow IBMs lead and > abandon the FDB-CouchDB effort. At the same time, it is not obliged to take > what they leave behind and finish it. > > I know for some the 4.x release is highly anticipated and we as a project > hoped to make a generational jump for our underlying storage and distribution > technologies. During initial discussions about FDB-Couch and during its > development, we anticipated certain developments on the FDB side (especially > allowing longer transactions for consistent _changes responses with their new > Redwood storage engine). It is my understanding that these developments have > not materialised in the way we would like them. The consequence is that there > are
Re: Important update on couchdb's foundationdb work
> the most is the scaling limits of the database-per-user pattern I had a discussion about replacing the db-per-user approach with a single partitioned db having a username as a partition key for each document. Looks easy to enforce Couch-side, both for query URLs and for new doc _id-s. Sort of special security mode. Anyway, we didn’t proceed with the concept, so I'm just leaving it here. ermouth
Re: Important update on couchdb's foundationdb work
Thank you for that, those are the issues that are turning over in my mind also. FoundationDB base would have improved CouchDB in a number of very helpful ways, beyond the scalability benefits themselves (though you are right that running FoundationDB in production is not a trivial undertaking). Namely, we would return to CouchDB 1.x era consistency within a cluster. That is, there’d be no such thing as a “rewind” and no chance of introducing a conflict within a cluster (though you still can _between_ clusters). Additionally, the representation of a document’s revision tree in FDB is superior to CouchDB 3.x’s, as each edit branch is stored separately. In a lot of cases this means documents can be read and updated without loading up the entire revision tree. If we’re not pursuing the couchdb-fdb work further, we should consider implementing some or all of the above again. All of those changes are possible but some are considerably harder than others, and we might have to pick our battles. Some time ago I made a wishlist of breaking changes for CouchDB 3.x that address a number of the gotchas our users frequently encounter. Other wishlists exist, I know you have one yourself, but I present it below; 1) True delete by default: Remove branches and documents entirely after deletion and once all replications, indexes and other observers have checkpointed past the sequence. This also removes the ability to store information in a deleted document. 2) Flatten the revtree: Store each edit branch as a separate k/v entry (we do this on fdb main, so backport the concept) 3) Optional automatic deletion of losing revs after a period of significant wall-clock time 4) Can we make the N replicas more consistent? two possible levels: a) updates to the same doc in the same order. b) all updates in the same order. 5) true conflict resolution. allow a revision to have multiple parents, so revision tree narrows. B. > On 12 Mar 2022, at 09:26, Jan Lehnardt wrote: > > Thanks Bob for passing this along. > > I’m looking forward to renewed interest in the 3.x codebase :) > > For our 4.x plans, we’ll have to discuss here what we want to do with it and > I’m looking at everyone for input here. Even if you’ve never spoken up on > this list before, I’d lie to hear from you. > > * * * > > First off, as a project, CouchDB is not obliged to follow IBMs lead and > abandon the FDB-CouchDB effort. At the same time, it is not obliged to take > what they leave behind and finish it. > > I know for some the 4.x release is highly anticipated and we as a project > hoped to make a generational jump for our underlying storage and distribution > technologies. During initial discussions about FDB-Couch and during its > development, we anticipated certain developments on the FDB side (especially > allowing longer transactions for consistent _changes responses with their new > Redwood storage engine). It is my understanding that these developments have > not materialised in the way we would like them. The consequence is that there > are certain API guarantees that 3.x CouchDB gives (consistent full-database > snapshots in _changes) are not possible to build with native FDB features. — > I can’t speak to the very specifics of this, and I hope we can dig into all > this together in this thread, but my takeaway from this is that *if* we > continue with FDB-Couch, I think we will have to reevaluate its compatibility > story, as we had hoped to make it mainly a seamless (but better) API upgrade > from 3.x. > > We also learned that operating a FDB cluster is a significant effort that > somewhat goes against CouchDB’s mostly “just works” nature. We had asked the > IBM team to share their operational FDB learnings with the CoucHDB project, > so we can build up community knowledge around this, but this has not > materialised either. > > I’m personally still excited about the opportunities we have with FDB-Couch, > but as a project, we might have to come up with a more realistic positioning > of FDB-CouchDB. Less a “new and improved drop-in replacement” and maybe more > a “if you exceed the scale/capacity of 3.x CouchDB, you can upgrade to > FDB-CouchDB at the expense of a few API differences and higher operational > cost”. This might be worth a trade-off for large users of CouchDB and thus it > might be worth having both of these codebases live alongside each other. > > However, that comes with a number of consequences: > > - The 3.x/4.x naming doesn’t quite work if these are meant to continue > alongside each other. > > - Maybe FDB-Couch gets its own separate project name and versioning, with a > clear delineation between them. > > - We would have to maintain two projects complete with release management, > vulnerability management, the lot. At the moment, CouchDB has just about > enough folks contributing to move forward at a reasonable pace. Doubling that > effort might be tricky. While we
Re: Important update on couchdb's foundationdb work
Thanks Bob for passing this along. I’m looking forward to renewed interest in the 3.x codebase :) For our 4.x plans, we’ll have to discuss here what we want to do with it and I’m looking at everyone for input here. Even if you’ve never spoken up on this list before, I’d lie to hear from you. * * * First off, as a project, CouchDB is not obliged to follow IBMs lead and abandon the FDB-CouchDB effort. At the same time, it is not obliged to take what they leave behind and finish it. I know for some the 4.x release is highly anticipated and we as a project hoped to make a generational jump for our underlying storage and distribution technologies. During initial discussions about FDB-Couch and during its development, we anticipated certain developments on the FDB side (especially allowing longer transactions for consistent _changes responses with their new Redwood storage engine). It is my understanding that these developments have not materialised in the way we would like them. The consequence is that there are certain API guarantees that 3.x CouchDB gives (consistent full-database snapshots in _changes) are not possible to build with native FDB features. — I can’t speak to the very specifics of this, and I hope we can dig into all this together in this thread, but my takeaway from this is that *if* we continue with FDB-Couch, I think we will have to reevaluate its compatibility story, as we had hoped to make it mainly a seamless (but better) API upgrade from 3.x. We also learned that operating a FDB cluster is a significant effort that somewhat goes against CouchDB’s mostly “just works” nature. We had asked the IBM team to share their operational FDB learnings with the CoucHDB project, so we can build up community knowledge around this, but this has not materialised either. I’m personally still excited about the opportunities we have with FDB-Couch, but as a project, we might have to come up with a more realistic positioning of FDB-CouchDB. Less a “new and improved drop-in replacement” and maybe more a “if you exceed the scale/capacity of 3.x CouchDB, you can upgrade to FDB-CouchDB at the expense of a few API differences and higher operational cost”. This might be worth a trade-off for large users of CouchDB and thus it might be worth having both of these codebases live alongside each other. However, that comes with a number of consequences: - The 3.x/4.x naming doesn’t quite work if these are meant to continue alongside each other. - Maybe FDB-Couch gets its own separate project name and versioning, with a clear delineation between them. - We would have to maintain two projects complete with release management, vulnerability management, the lot. At the moment, CouchDB has just about enough folks contributing to move forward at a reasonable pace. Doubling that effort might be tricky. While we had an influx of contributors recently, this would probably need more dedicated planning and outreach. - New API features would have to be implemented twice, if we want to keep a majority API overlap. This is not a fun proposition for folks who add features, which is hard enough, but now they have to do it twice, onto two different subsystems. Some features (say multi-doc-transactions) would only be possible in one of the projects (FDB-Couch), what would our policy be for deliberate API feature divergence? - probably more that elude me at the moment. While there are non-trivial points among these, they are not impossible tasks *if* we find enough and the right folks to carry the work forward. * * * For myself, I still see a lot of potential in the 3.x codebase and I’m looking forward to renewed roadmap discussions there. I know I have a long list of things I’d like to see added. From my professional observation, the thing that our (Neighbourhoodie) customers tend to run into the most is the scaling limits of the database-per-user pattern. We have a proposal for per-doc-authentication that helps mitigate a subset of those use-cases, which would be a great help overall. I have worked on a draft PR of this over the years, but it mostly stalled out during the pandemic. I’m planning to restart work on this shortly. If anyone wants to contribute with time and/or money, please do get in touch. The other major issue with 3.x as reported by IBM is _changes feed rewinds when nodes are rotated in and out of clusters. We already fixed a number of changes rewind bugs relatively recently. I don’t know if we got them all now, or if there are theoretical limits to how far we can take this given our consistency model, but it’d be worth spending some time on at least getting rid of all rewind-to-zero cases. * * * I’m also looking forward to all your input on the discussion here. I’m sure this will explode into a lot of detailed discussions quickly, so maybe as a guide to come back to when get closer to having to make a decision, here are three ways forward that I
Important update on couchdb's foundationdb work
Hi, For those that are following closely, and particularly those that build or use CouchDB from our git repo, you'll be aware that CouchDB embarked on an attempt to build a next-generation version of CouchDB using the FoundationDB database engine as its new base. The principal sponsors of this work, the Cloudant team at IBM, have informed us that, unfortunately, they will not be continuing to fund the development of this version and are refocusing their efforts on CouchDB 3.x. Cloudant developers will continue to contribute as they always have done and the CouchDB PMC thanks them for their efforts. As the Project Management Committee for the CouchDB project, we are now asking the developer community how we’d like to proceed in light of this new information. Regards, Robert Newson Apache CouchDB PMC
