Hi everyone, this is one of the threads we promised. This one covers the roadmap discussion.
This thread aims to answer: - which version would the FDB work land in? - what happens with non-FDB CouchDB until then? - specifically, how are the current roadmap items affected? - how are the two different versions being managed in the future. Out of scope: - whether how how we name things publicly. The below contains my current thinking for an ideal way to handle this and it is presented as a strawperson argument to be picked apart and refined, designed to solicit input form everybody. Please comment on the details of The Proposal (below) as well as the categorisation and comments in The Tickets (also below). There are a lot of details in here, feel free to comment on just one aspect of this rather than feeling compelled to make a comment on everything. * * * Naming prelude: to make this thread easier to follow, I’ll call the current `master` version of CouchDB “Erlang CouchDB” (despite there being non-Erlang bits in there) and the version that would be based on FoundationDB “FBD CouchDB”. Note: these aren’t suggestions for marketing names or anything, just shorthand for this particular discussion. * * * ## Guiding Principles: - Getting to a position where FDB CouchDB is production ready is going take a while. In the meantime, Erlang CouchDB will continue to receive feature-, bugfix-, and security-updates. This is *at least* until an FDB CouchDB verion is available, plus a grace period for a transition that is to be discussed (gut feel, this is anything between 1, 2 and 3 years). - As a result of a multi-year dev-lifespan of Erlang CouchDB, and extrapolating from experience in existing CouchDB production timespans, we can easily imagine folks running Erlang CouchDB in production for *at least* the next 10 years (At Neighbourhoodie, we are still supporting CouchDB 1.2.1 customers e.g.) - A direct consequence of this, and in light of major feature work *at some point* transitioning to FDB CouchDB, we should take this opportunity and rally to make Erlang CouchDB “The Best Erlang CouchDB we can make”™. That means we’ll re-evaluate the current roadmap against what’s essential to make that happen, while leaving things that might be easier to do on FDB CouchDB for the new setup. * * * ## The Proposal In very broad strokes, I imagine a CouchDB 3.x series that is “The Best Erlang CouchDB we can make”™ that has all the big ticket items we planned on doing modulo some that we leave for FDB CouchDB because they become possible or significantly easier. The main reason for calling this 3.x would be a set of deprecations as well as the addition of the _access field to docs (where we would want a forward compatible version of the 2.x series as well, so upgrades become possible) and the imposing of additional limits to document, attachment and HTTP request sizes to be within recommended best practices and to be forward compatible with FDB CouchDB, but which can be brought back to 2.x settings for folks who need it. FDB CouchDB would then become the 4.x series. Even though we make major changes under the hood, we would want for most CouchDB 2.x/3.x users to be able to upgrade as seamlessly as possible, so we’ll aim for a high degree of API compatibility like we’ve done in the 1.x -> 2.x transition and which worked very well for us. * * * ## The Annotated Roadmap The rest of this email is a list of all tickets that currently have `roadmap` label with a comment on how to proceed with it plus a few bits and bobs that are relevant in the context of an FDB future. Broadly, there are these categories: 1. we should add this to “The Best Erlang CouchDB we can make”™, category YES 2. this is a ticket that can be worked on independently of either CouchDB variant (say improvements to Mango, which can happen at any time), category INDIE 3. we should hold off until FDB CouchDB to tackle this feature, category WAIT ## The Tickets ### Category: YES Build and ship view/secondary index warmer https://github.com/apache/couchdb/issues/1825 Comment: n/a Update SpiderMonkey version (was: move to Chakra Core) https://github.com/apache/couchdb/issues/1875 Comment: we should get on this, really. Migrate to rebar3: https://github.com/apache/couchdb/issues/1428 Comment: n/a Improve plugin architecture / Erlang release building https://github.com/apache/couchdb/issues/1515 Comment: I believe there is a separate proposal for this coming. 1.x/2.x Deprecations https://github.com/apache/couchdb/issues/1534 Comment: to make upgrading to FDB CouchDB easier, we should be actively addressing deprecations as well as system limitations. Retire the node-local interface (port 5986) https://github.com/apache/couchdb/issues/1523 Comment: Joan is already working on this. Automate incrementally growing a cluster https://github.com/apache/couchdb/issues/1517 Comment: Nick is already working on this, see other threads on dev@ Better support for db-per-user https://github.com/apache/couchdb/issues/1524 Comment: I’m working on this ### Category: INDIE Add aggregation functions to Mango: https://github.com/apache/couchdb/issues/1254 Comment: this is an important feature that we should have sooner than later, but Mango improvements can happen (mostly) independently of Erlang vs. FDB CouchDB and can appear in either or both versions. Improve CouchDB "plugin" architecture for geo/search https://github.com/apache/couchdb/issues/1269 Comment: this only affects Erlang CouchDB for now, a similar discussion should happen for FDB CouchDB, but that’s not on the roadmap just yet. Simpler API: https://github.com/apache/couchdb/issues/1496 Comment: FDB CouchDB will re-use chttpd to make sure to be as close to API compatible as possible. Any new/versioned APIs would be treated similarly. Skip deletes / tombstone eviction https://github.com/apache/couchdb/issues/1505 Comments: this needs review in light of clustered purge being available now. Groping a few things for efficiency: - Support server side / auto conflict resolution https://github.com/apache/couchdb/issues/1506 - First class conflict handling https://github.com/apache/couchdb/issues/1507 - Selective database syncing https://github.com/apache/couchdb/issues/1508 - Support JOINs in Mango https://github.com/apache/couchdb/issues/1510 - Support arbitrary result sorting in views and/or mango https://github.com/apache/couchdb/issues/1511 - Mango: Add Bitwise Operators https://github.com/apache/couchdb/issues/1512 - NoJS Mode https://github.com/apache/couchdb/issues/1513 - Query Server Protocol v2 https://github.com/apache/couchdb/issues/1514https://github.com/apache/couchdb/issues/1514 - Support a Windows CI environment https://github.com/apache/couchdb/issues/1535 - Support CouchDB as hex / Erlang dep https://github.com/apache/couchdb/issues/1536 - Additional Mango-based update handler / VDU functionality https://github.com/apache/couchdb/issues/1554 - Mobile First replication protocol https://github.com/apache/couchdb/issues/1503 - Redesign CouchDB security system https://github.com/apache/couchdb/issues/1504 - Support GraphQL https://github.com/apache/couchdb/issues/1499 - Synthetic load test suite https://github.com/apache/couchdb/issues/1521 - Streaming API for attachment data https://github.com/apache/couchdb/issues/1540 - Improve config subsystem https://github.com/apache/couchdb/issues/1595 - Externalise Erlang Term things (_revs, replication _ids) https://github.com/apache/couchdb/issues/1530 Comment: n/a Cluster-aware clients / API https://github.com/apache/couchdb/issues/1519 Comment: I don’t know if this would even be possible in FDB CouchDB, but it would be a nice addition to Erlang CouchDB, but I don’t think it’s a critical feature. Schema extraction https://github.com/apache/couchdb/issues/1525 Comment: might be easier to do in FDB CouchDB Database corruption detection & repair https://github.com/apache/couchdb/issues/1526 Comment: only applies to Erlang CouchDB really, commercial solutions exist. Do not shipping records over the inter-node wire https://github.com/apache/couchdb/issues/1538 Comment: FDB CouchDB would not have this particular issue Multi-tenancy support https://github.com/apache/couchdb/issues/1539 Comment: n/a RFC: Sub-Document Operations https://github.com/apache/couchdb/issues/1559 Comment: Erlang CouchDB version is in the works, FDB CouchDB version would be even more efficient internally due to per-field storage. Improve time series data storage options https://github.com/apache/couchdb/issues/1531 Comment: this would look different in either variant, but can happen independently ### Category: WAIT Support renaming a database https://github.com/apache/couchdb/issues/1502 Comment: this will become a lot easier in FDB-land HTTP/2 Support https://github.com/apache/couchdb/issues/1497 Comment: FDB CouchDB will re-use chttpd to make sure to be as close to API compatible as possible, which won’t get us towards HTTP/2 just yet. Any discussion about updating the HTTP layer would happen after the FDB CouchDB transition. Plus HAProxy now ships decent HTTP/2 support, so at least cluster clients can get the benefits today already. Support "very wide” clusters https://github.com/apache/couchdb/issues/1527 Comment: this is one of the things we want FDB for in the first place, afaict. Server-side mass update function https://github.com/apache/couchdb/issues/1532 Comment: This becomes a lot easier in FDB CouchDB, maybe we just wait for that. Auto-record data on document create / update https://github.com/apache/couchdb/issues/1533 Comment: this is extremely gnarly in an eventually consistent scenario and in a permanently consistent FDB CouchDB world becomes much easier to do. * * * Best Jan -- -- Professional Support for Apache CouchDB: https://neighbourhood.ie/couchdb-support/
