I received some notes privately from Gregor Martynus, which I’m reproducing
here in email thread form. This email is all Gregor’s notes, my next email is
my replies to them.
> On 10. Mar 2019, at 15:51, Jan Lehnardt <j...@apache.org> wrote:
>
> Hey all,
>
> after mulling this over some more, I’d like to tackle the detailed API and
> behaviour for this. Especially how _access work in conjunction with existing
> access control features.
>
> My guiding principles so far are:
>
> 1. Make the API intuitive, things should work like they look like they should
> work like.
> 2. The default should never be that a resources is accidentally left
> accessible to the public.
> 3. This should work as a natural extension to the existing security features*.
>
> * I’d be up for reworking the whole lot, too, but that might be a better
> discussion for > 4.0.
>
>
> ## Database Creation and Default Behaviours
>
> Creating a database with _access features is, as mentioned before done via a
> flag to PUT /database?access=true
>
> In a 3.0 world where this would land, we already agreed that databases should
> be admin-only by default (instead of world read/writeable today). This is a
> sensible default, but that leaves us with an _access enabled database that
> can’t be used by anyone by server or db admins. Not very useful.
>
> To allow arbitrary users to use the db, I suggest we use the existing
> _security system: i.e. if a user or a group a user belongs to is mentioned in
> either `admins` or `members` inside of _security, they can proceed and create
> documents on the db. This puts a second step burden on the application
> developer, but it slots cleanly into the existing security mechanisms, and
> doesn’t require special case handling. Alternatively, we could define that
> _security isn’t available in _access enabled databases, but that’s something
> I’d like to avoid if at all possible.
>
> In order to make it easy to specify that “everyone in _users” should be able
> to use the db, I suggest we add a new role `_users` that is valid inside
> _security, which means “everyone in /_users” (this only excludes server
> admins which have full access anyway).
>
> * * *
>
>
> ## Document Creation and Access Control
>
> Next, one of our non-admin users creates a doc. There are multiple options as
> to how we store the _access information.
>
> 1. Automatically translate the userCtx.name of a doc creation (not an update)
> into the first element of the _access array. E.g. user_a PUT /db/doc {"a":1}
> creates this doc: {"a":1,"_access":["user_a"]}. This is a little bit
> counter-intuitive.
>
> 2. We require that a user puts "_access":["user_a"] in themselves. This is an
> explicit granting of access permissions on doc creation and I think is
> preferable.
I prefer being explicit.
>
> This leaves the edge case of docs that have no _access member: so far I
> thought those docs are admin-only, with maybe a db-wide option to swap the
> default to public access, but I think given the explicitness of 2. we can do
> better: require _access for all new doc creations in access-enabled
> databases. A user can not create a new document without an _access field that
> is an array that has at least one member. For public documents, we could
> invent a new role _public, and admin-only docs could use the existing role
> _admin.
>
> The one downside to this approach is that we won’t be able to replicate
> existing databases into an access-enabled database without modifying all
> documents. This might be a worthwhile trade-off, but we should make that
> decision consciously and document it well.
We could also provide tooling for migrations?
> We could allow for a special case where an _admin user can create docs that
> have no _access field, and those docs are treated as having only the _admin
> role in _access. So at least we could replicate all data in, but then require
> a manual step to update all docs to say, migrate an existing db-per-user app,
> while not accidentally exposing any docs to folks that shouldn’t read them.
>
> For the rest of cRUD, the existing document must store one of the RUD-ing
> user’s name or role in its _access field.
>
> For both creations and updates, a user MUST supply at least one role they
> belong to or their own username.
>
> * * *
>
>
> ## _revs_diff
>
> /db/_revs_diff can answer the question of which revisions of a document do
> NOT exist on a replication target:
> http://docs.couchdb.org/en/stable/api/database/misc.html#db-revs-diff
>
> This would allow users to specify ids and rev(s) for docs they don’t have
> access too (anymore), so the result schema should be expanded to handle id:
> unauthorized or somesuch, something the replicator needs to know what to do
> with, if it encounters it (say a user got removed from the _access list
> inbetween the replicator opening _changes and requesting the doc).
>
> The _revs_diff implementation would have to altered to send an unauthorized
> token for each doc the requesting userCtx has no access to. If we can re-use
> some of our existing indexes, or any other performance optimisation, that’d
> be great. I haven’t looked at that code at all, yet.
>
> An important side-effect of this is, once a user has been added to a doc’s
> _access list, they get access to “the full history of the doc”, even before
> they had access. Of course, in CouchDB this means only getting access to the
> rev ids, and not the content, but since they are content-addressable hashes,
> a user could brute-force themselves into revealing certain real values from
> earlier incarnations of the doc. I’d rather not track _access per document
> revision in perpetuity, so this is something we have to be very up-front
> about.
>
> * * *
>
>
> ## Partitioned Databases
>
> I mentioned partitioned databases in my previous mail, and I think it is
> something we can document that end-users can opt into, but doesn’t require
> any special casing on the _access proposal. That is, if users start prefixing
> their doc ids with a user name or id and enable both _access and partitions,
> then they get all the benefits of a partitioned database, and if they choose
> not to, they don’t, but things keep working. There are enough use-cases to
> warrant both behaviours.
>
> * * *
>
>
> ## Scenarios that _access should help with.
>
> Overall, we developed _access to allow users to stop using the db-per-user
> architecture, but once we have per-doc-access control, folks might start
> using this for all manner of things. We should be clear about which scenarios
> we support and which we don’t.
>
>
> ### Scenario 1: db-per-user
>
> In this scenario, _access enabled databases, the only way to allow mutually
> untrusting users to store data in a part of CouchDB that only they (and
> admins) have access to was giving each user their own database.
>
> In an _access enabled database, users can CRUD/_changes/_all_docs/_revs_diff
> their own docs knowing no other user (aside from admins) can access those
> docs.
>
> This is the simplest scenario, as all we’d have to track the owner of a
> document and produce by-access-id/seq indexes based on that owner.
>
> The current prototype implementation mostly reflects this stage. Not saying
> this is what we should ship, but it is the easiest do implement and explain.
>
> Aside, I might be able to be persuaded to ship this as a 2.x feature, to help
> those folks who don’t need anything else.
>
>
> ### Scenario 2: db-per-user + Sharing
One scenario we should address is how stopping to share would work when
documents are continuously replicated, e.g. to a client for offline usage. My
understanding is that for the person who’s access to documents got revoked does
not get _changes update telling them that their access got removed, it would be
up to the application developer to implement some kind of "notification" meta
documents. Unless you have a better idea?
>
> The second we allow per doc auth, users will want to share those docs with
> other users. That’s why we initially suggested the _access field be an array,
> so other users and groups can be specified to have access. There are multiple
> scenarios in this one alone:
>
> #### 2.1: The Todo List
>
> In this scenario, a user has a reasonable amount of ”personal data” that they
> want to selectively share with one or more other users.
>
> #### 2.2: The Chat/Forum/Newsgroup
>
> In this scenario, a user wants to share any number of documents with a
> reasonable number of groups. However, since we need to limit the number of
> groups a user belongs to (currently 10, see below for details), this might
> actually not be a great solution. Or folks couldn’t be in more than 10 chat
> groups at a time.
>
> #### 2.3: The Corporate Hierarchy
>
> In this scenario, users want to share any number of docs with a reasonable
> number of groups in a top-down/bottom-up fashion. Think CEO shares with
> executives, execs share with divisions, divisions report up to their one
> executive, etc.
>
>
> ### 3: Multiple Apps
>
> The preceding scenarios all assume that a single application is responsible
> for everything. However, once we allow mutually distrusting users into a
> single database *and* make each per-user slice work (almost) like a full
> standalone CouchDB database, what would stop users from using this for a
> multi-homing feature, where different applications are used for each user in
> the same database?
>
> I’ll be referring to these scenarios down the line.
>
> * * *
>
>
> ## Design Docs
>
> ### Admin
>
> One of the downsides of db-per-user is managing design docs in the face of a
> changing application, that is, how to distribute new design docs across 10s
> of 1000+s of user dbs? It’s not impossible, but tedious. In all scenarios
> above but scenario 3., we could simplify this significantly. Say an admin
> creates a design doc, and gives all users in the db access to this design doc
> (this could be with the _users role, or yet another new role _members, if we
> need it), requesting the result of a view defined in that design doc will
> produce an index that is powered by the requesting user’s by-access-seq index
> section(s).
>
> N.B., this would require us to change a fundamental assumption when doing the
> association between a design doc’s definition and index: normally, there is
> only the `views` member that is hashed and that hash is used as the index’s
> filename. Because there is only by-seq to power a view, that all works. But
> now that we have an arbitrary set of sections on by-access-seq, any view
> index built will have to take a user’s name and roles into account. When a
> user leaves a group, or gains a group, all indexes for that user will no
> longer be valid and need rebuilding.
>
>
> ### User
>
> In any of the scenarios above, but especially 3., there could be legitimate
> per-user design docs, so how should those be treated in an _access enabled
> database?
>
> The significant fields in a design doc are `views`, `validate_doc_update` and
> `filters` (I’ll skip over the deprecated _show, _list, and _update).
>
> The easiest to handle is a `filters`: if a user specifies a filter for a
> _changes request or replication that lives in a design doc they don’t have
> access to, they get an error, similar to if they specify a non-existent
> design doc, just with `unauthorized` instead of `not_found`.
>
> Next `views` is also not very hard to imagine working: just like globally
> defined views for that db, the index is built for each user based on the
> user’s name and roles.
>
> More troubling are `validate_doc_update` functions: One, they are already
> troubling in that they slow down any document updates. Two, if we now import
> an existing db-per-user scenario where each user has their own design docs,
I can’t think of a db-per-user scenario where each user DB would have a
different validate_doc_update method? It would be the same method with access
to the user context, the DBs security setting and the document, so it would act
differently for different users, but using the same code.
> how should we apply validate_doc_update functions? 10s of 1000s of VDUs are
> impractical to apply on each doc update, let alone just the management of
> VDUs that are active on a database. One option would be to ignore VDUs if
> they are not defined globally (say with a _members role). But especially in
> scenario 3. this becomes problematic, but even without that specific
> scenario, this violates the no surprises best practice.
>
> We could say:
>
> a) we don’t support scenario 3.
+1, I think it would make our lives easier in general if we don’t recommend to
share the same CouchDB for multiple apps. At least I don’t see a reason to do
that at this point
> b) we find a complicated but efficient way to apply only those VDUs that are
> defined in design docs the writing user has access to plus any global ones
> (this would be neat but rather complicated and potentially still impractical
> from a performance perspective for N users).
> c) we could store all per-user design docs, but ignore them completely, VDUs,
> views and filters.
>
> I think I currently fall on the side of not supporting scenario 3. and asking
> folks who migrate db-per-user to de-duplicate design docs and keep them
> per-app. I believe that is a good trade-off between the most common scenarios
> for db-per-user while keeping the implementation manageable. Globally
> accessible design docs would show up in a user’s changes feed and would
> replicate down to say a PouchDB application which might be the exclusive user
> of those design docs.
>
> In practice this would mean, a document that has an _id that starts with
> _design/ will have to be produced by a database admin. Luckily, that’s
> already the case. We should just make sure that folks don’t give db-admin
> access to all users habitually.
>
>
> ## Read and Write Access
>
> Speaking of validate_doc_update, it is used for two things: checking document
> schema and doc update authorisation.
>
> Once we allow access to a document with an _access field, we need to decide
> what kind of access this gives to a doc: read-only or read-write (I’m not
> considering write-only because for anything but doc creations this is not
> useful as you need access to the current _rev).
>
> However, when we look at implementing an application on top of our existing
> API, it is already weird that read access can be controlled globally (or with
> _access on a per doc level), but write access requires writing JavaScript
> code. I think it would be a reasonable expectation for users to expect a
> per-doc read/write permission granting.
Yes!
>
> So we could have all of the above, but with two extra fields: _access_read
> and _access_write, or _access: {read: [], write: []}
I prefer this API for its compactness, thinking about offline synchronization.
The smaller the docs, the better.
Best
“Gregor”
—
> or we overload user and group names: _access: [user_a:read, user_b:write] (or
> any permutation thereof). Overloading can cause trouble with naturally
> occurring characters in group names.
>
> The former seems more explicit, but from an API perspective that’s a little
> more awkward: remember that we currently have an arbitrary limit of 10
> members in a user’s role array, to avoid excessive fan out on
> cluster-internal operations. Partitioned dbs could get away with more, more
> easily however. If we allow the specification of access control in two lists,
> and one of the lists implies membership in the other, we have a total limit
> of 10 members across both arrays. Or we limit 5 + 5, but that seems
> excessive, while 10 total seems weird, but doable. Anyway, good bikeshed.
>
>
> * * *
>
>
> So far. I think all of the problems outlined are solvable, if with a clear
> definition of what use-cases we do not support with access. If you have more
> scenarios than the ones I outlined, please add them and we can see if they
> cause any additional trouble.
>
> Thanks for reading this far and I’m looking forward to your feedback.
>
>
> Best,
> Jan “_access” Lehnardt
> —
>
>
>
>
>> On 17. Feb 2019, at 15:25, Jan Lehnardt <j...@apache.org> wrote:
>>
>> Hi Everyone,
>>
>> I’m happy to share my work in progress attempt to implement the per-doc
>> access control feature we discussed a good while ago:
>>
>> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E
>>
>> <https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E>
>>
>> You can check out my branch here:
>>
>> https://github.com/apache/couchdb/compare/access?expand=1
>> <https://github.com/apache/couchdb/compare/access?expand=1>
>>
>> It is very much work in progress, but it is far enough along to warrant
>> discussion.
>>
>> The main point of this branch is to show all the places that we would need
>> to change to support the proposal.
>>
>> Things I’ve left for later:
>>
>> - currently only the first element in the _access array is used. Our and/or
>> syntax can be added later.
>> - building per-access views has not been implemented yet, couch_index would
>> have to be taught about the new per-access-id index.
>> - pretty HTTP error handling
>> - tests except for a tiny shell script 😇
>>
>> Implementation notes:
>>
>> You create a database with the _access feature turned on like so: PUT
>> /db?access=true
>>
>> I started out with storing _access in the document body, as that would allow
>> for a minimal change set, however, on doc updates, we try hard not to load
>> the old doc body from the database, and forcing us to do so for EVERY doc
>> update under _access seemed prohibitive, so I extended the #doc, #doc_info
>> and #full_doc_info records with a new `access` attribute that is stored in
>> both by-id and by-seq. I will need guidance on how extending these records
>> impact multi-version cluster interop. And especially whether this is an
>> acceptable approach.
>>
>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-904ab7473ff8ddd07ea44aca414e3a36
>>
>> * * *
>>
>> The main addition is a new native query server called
>> couch_access_native_proc, which implements two new indexes by-access-id and
>> by-access-seq which do what you’d expect, pass in a userCtx and retrieve the
>> equivalent of _all_docs or _changes, but only including those docs that
>> match the username and roles in their _access property. The existing
>> handlers for _all_docs and _changes have been augmented to use the new
>> indexes instead of the default ones, unless the user is an admin.
>>
>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-fbb53323f07579be5e46ba63cb6701c4
>>
>> * * *
>>
>> The rest of the diff is concerned with making document CRUD behave as you’d
>> expect it. See this little demonstration for what things look like:
>>
>> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497
>> <https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497> (I’m just
>> noticing that there might be something wonky with DELETE, but you’ll get the
>> gist #rimshot)
>>
>> * * *
>>
>> Open questions:
>>
>> - The aim of this is to get as close to regular CouchDB behaviour as
>> possible. One thing that is new however which would require all apps to be
>> changed is that for an _access enabled database to include an _access field
>> in their docs (docs with no _access are admin-only for now). We might want
>> to consider on new document writes to auto-insert the authenticated user’s
>> name as the first element in the _access array, so existing apps “just work”.
>>
>> - Interplay with partitioned dbs: eschewing db-per-user is already a large
>> boon if you have a lot of users, but making those per-user requests inside
>> an _access enabled database efficient would be doubly nice, so why not use
>> the username from the first question above and use that as the partition
>> key? This would work nicely for natural users with their own docs that want
>> to share them with others later, but I can easily imagine a pipelined use of
>> CouchDB, where a “collector” user creates all new docs, an “analyser” takes
>> them over and hand them to a “result” user for viewing. In that case, we’d
>> violate the high-cardinality rule of partitions (have a lot of small ones),
>> instead all docs go through all three users. I’d be okay with treating the
>> later scenario as a minor use-case, but for that use-case, we should be able
>> to disable auto-partitioning on db creation.
>>
>> - building access view indexes for docs that have frequent _access changes,
>> lead to many orphaned view indexes, we should look at an auto-cleanup
>> solution here (maybe keep 1-N indexes in case folks just swap back and
>> forth).
>>
>> * * *
>>
>> I’ll leave this here for now, I’m sure there are a few more things to
>> consider.
>>
>> I’d love to hear any and all feedback you might have. Especially if anything
>> is unclear.
>>
>> Best
>> Jan
>> —
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
