Hey all,
after mulling this over some more, I’d like to tackle the detailed API and
behaviour for this. Especially how _access work in conjunction with existing
access control features.
My guiding principles so far are:
1. Make the API intuitive, things should work like they look like they should
work like.
2. The default should never be that a resources is accidentally left accessible
to the public.
3. This should work as a natural extension to the existing security features*.
* I’d be up for reworking the whole lot, too, but that might be a better
discussion for > 4.0.
## Database Creation and Default Behaviours
Creating a database with _access features is, as mentioned before done via a
flag to PUT /database?access=true
In a 3.0 world where this would land, we already agreed that databases should
be admin-only by default (instead of world read/writeable today). This is a
sensible default, but that leaves us with an _access enabled database that
can’t be used by anyone by server or db admins. Not very useful.
To allow arbitrary users to use the db, I suggest we use the existing _security
system: i.e. if a user or a group a user belongs to is mentioned in either
`admins` or `members` inside of _security, they can proceed and create
documents on the db. This puts a second step burden on the application
developer, but it slots cleanly into the existing security mechanisms, and
doesn’t require special case handling. Alternatively, we could define that
_security isn’t available in _access enabled databases, but that’s something
I’d like to avoid if at all possible.
In order to make it easy to specify that “everyone in _users” should be able to
use the db, I suggest we add a new role `_users` that is valid inside
_security, which means “everyone in /_users” (this only excludes server admins
which have full access anyway).
* * *
## Document Creation and Access Control
Next, one of our non-admin users creates a doc. There are multiple options as
to how we store the _access information.
1. Automatically translate the userCtx.name of a doc creation (not an update)
into the first element of the _access array. E.g. user_a PUT /db/doc {"a":1}
creates this doc: {"a":1,"_access":["user_a"]}. This is a little bit
counter-intuitive.
2. We require that a user puts "_access":["user_a"] in themselves. This is an
explicit granting of access permissions on doc creation and I think is
preferable.
This leaves the edge case of docs that have no _access member: so far I thought
those docs are admin-only, with maybe a db-wide option to swap the default to
public access, but I think given the explicitness of 2. we can do better:
require _access for all new doc creations in access-enabled databases. A user
can not create a new document without an _access field that is an array that
has at least one member. For public documents, we could invent a new role
_public, and admin-only docs could use the existing role _admin.
The one downside to this approach is that we won’t be able to replicate
existing databases into an access-enabled database without modifying all
documents. This might be a worthwhile trade-off, but we should make that
decision consciously and document it well. We could allow for a special case
where an _admin user can create docs that have no _access field, and those docs
are treated as having only the _admin role in _access. So at least we could
replicate all data in, but then require a manual step to update all docs to
say, migrate an existing db-per-user app, while not accidentally exposing any
docs to folks that shouldn’t read them.
For the rest of cRUD, the existing document must store one of the RUD-ing
user’s name or role in its _access field.
For both creations and updates, a user MUST supply at least one role they
belong to or their own username.
* * *
## _revs_diff
/db/_revs_diff can answer the question of which revisions of a document do NOT
exist on a replication target:
http://docs.couchdb.org/en/stable/api/database/misc.html#db-revs-diff
This would allow users to specify ids and rev(s) for docs they don’t have
access too (anymore), so the result schema should be expanded to handle id:
unauthorized or somesuch, something the replicator needs to know what to do
with, if it encounters it (say a user got removed from the _access list
inbetween the replicator opening _changes and requesting the doc).
The _revs_diff implementation would have to altered to send an unauthorized
token for each doc the requesting userCtx has no access to. If we can re-use
some of our existing indexes, or any other performance optimisation, that’d be
great. I haven’t looked at that code at all, yet.
An important side-effect of this is, once a user has been added to a doc’s
_access list, they get access to “the full history of the doc”, even before
they had access. Of course, in CouchDB this means only getting access to the
rev ids, and not the content, but since they are content-addressable hashes, a
user could brute-force themselves into revealing certain real values from
earlier incarnations of the doc. I’d rather not track _access per document
revision in perpetuity, so this is something we have to be very up-front about.
* * *
## Partitioned Databases
I mentioned partitioned databases in my previous mail, and I think it is
something we can document that end-users can opt into, but doesn’t require any
special casing on the _access proposal. That is, if users start prefixing their
doc ids with a user name or id and enable both _access and partitions, then
they get all the benefits of a partitioned database, and if they choose not to,
they don’t, but things keep working. There are enough use-cases to warrant both
behaviours.
* * *
## Scenarios that _access should help with.
Overall, we developed _access to allow users to stop using the db-per-user
architecture, but once we have per-doc-access control, folks might start using
this for all manner of things. We should be clear about which scenarios we
support and which we don’t.
### Scenario 1: db-per-user
In this scenario, _access enabled databases, the only way to allow mutually
untrusting users to store data in a part of CouchDB that only they (and admins)
have access to was giving each user their own database.
In an _access enabled database, users can CRUD/_changes/_all_docs/_revs_diff
their own docs knowing no other user (aside from admins) can access those docs.
This is the simplest scenario, as all we’d have to track the owner of a
document and produce by-access-id/seq indexes based on that owner.
The current prototype implementation mostly reflects this stage. Not saying
this is what we should ship, but it is the easiest do implement and explain.
Aside, I might be able to be persuaded to ship this as a 2.x feature, to help
those folks who don’t need anything else.
### Scenario 2: db-per-user + Sharing
The second we allow per doc auth, users will want to share those docs with
other users. That’s why we initially suggested the _access field be an array,
so other users and groups can be specified to have access. There are multiple
scenarios in this one alone:
#### 2.1: The Todo List
In this scenario, a user has a reasonable amount of ”personal data” that they
want to selectively share with one or more other users.
#### 2.2: The Chat/Forum/Newsgroup
In this scenario, a user wants to share any number of documents with a
reasonable number of groups. However, since we need to limit the number of
groups a user belongs to (currently 10, see below for details), this might
actually not be a great solution. Or folks couldn’t be in more than 10 chat
groups at a time.
#### 2.3: The Corporate Hierarchy
In this scenario, users want to share any number of docs with a reasonable
number of groups in a top-down/bottom-up fashion. Think CEO shares with
executives, execs share with divisions, divisions report up to their one
executive, etc.
### 3: Multiple Apps
The preceding scenarios all assume that a single application is responsible for
everything. However, once we allow mutually distrusting users into a single
database *and* make each per-user slice work (almost) like a full standalone
CouchDB database, what would stop users from using this for a multi-homing
feature, where different applications are used for each user in the same
database?
I’ll be referring to these scenarios down the line.
* * *
## Design Docs
### Admin
One of the downsides of db-per-user is managing design docs in the face of a
changing application, that is, how to distribute new design docs across 10s of
1000+s of user dbs? It’s not impossible, but tedious. In all scenarios above
but scenario 3., we could simplify this significantly. Say an admin creates a
design doc, and gives all users in the db access to this design doc (this could
be with the _users role, or yet another new role _members, if we need it),
requesting the result of a view defined in that design doc will produce an
index that is powered by the requesting user’s by-access-seq index section(s).
N.B., this would require us to change a fundamental assumption when doing the
association between a design doc’s definition and index: normally, there is
only the `views` member that is hashed and that hash is used as the index’s
filename. Because there is only by-seq to power a view, that all works. But now
that we have an arbitrary set of sections on by-access-seq, any view index
built will have to take a user’s name and roles into account. When a user
leaves a group, or gains a group, all indexes for that user will no longer be
valid and need rebuilding.
### User
In any of the scenarios above, but especially 3., there could be legitimate
per-user design docs, so how should those be treated in an _access enabled
database?
The significant fields in a design doc are `views`, `validate_doc_update` and
`filters` (I’ll skip over the deprecated _show, _list, and _update).
The easiest to handle is a `filters`: if a user specifies a filter for a
_changes request or replication that lives in a design doc they don’t have
access to, they get an error, similar to if they specify a non-existent design
doc, just with `unauthorized` instead of `not_found`.
Next `views` is also not very hard to imagine working: just like globally
defined views for that db, the index is built for each user based on the user’s
name and roles.
More troubling are `validate_doc_update` functions: One, they are already
troubling in that they slow down any document updates. Two, if we now import an
existing db-per-user scenario where each user has their own design docs, how
should we apply validate_doc_update functions? 10s of 1000s of VDUs are
impractical to apply on each doc update, let alone just the management of VDUs
that are active on a database. One option would be to ignore VDUs if they are
not defined globally (say with a _members role). But especially in scenario 3.
this becomes problematic, but even without that specific scenario, this
violates the no surprises best practice.
We could say:
a) we don’t support scenario 3.
b) we find a complicated but efficient way to apply only those VDUs that are
defined in design docs the writing user has access to plus any global ones
(this would be neat but rather complicated and potentially still impractical
from a performance perspective for N users).
c) we could store all per-user design docs, but ignore them completely, VDUs,
views and filters.
I think I currently fall on the side of not supporting scenario 3. and asking
folks who migrate db-per-user to de-duplicate design docs and keep them
per-app. I believe that is a good trade-off between the most common scenarios
for db-per-user while keeping the implementation manageable. Globally
accessible design docs would show up in a user’s changes feed and would
replicate down to say a PouchDB application which might be the exclusive user
of those design docs.
In practice this would mean, a document that has an _id that starts with
_design/ will have to be produced by a database admin. Luckily, that’s already
the case. We should just make sure that folks don’t give db-admin access to all
users habitually.
## Read and Write Access
Speaking of validate_doc_update, it is used for two things: checking document
schema and doc update authorisation.
Once we allow access to a document with an _access field, we need to decide
what kind of access this gives to a doc: read-only or read-write (I’m not
considering write-only because for anything but doc creations this is not
useful as you need access to the current _rev).
However, when we look at implementing an application on top of our existing
API, it is already weird that read access can be controlled globally (or with
_access on a per doc level), but write access requires writing JavaScript code.
I think it would be a reasonable expectation for users to expect a per-doc
read/write permission granting.
So we could have all of the above, but with two extra fields: _access_read and
_access_write, or _access: {read: [], write: []} or we overload user and group
names: _access: [user_a:read, user_b:write] (or any permutation thereof).
Overloading can cause trouble with naturally occurring characters in group
names.
The former seems more explicit, but from an API perspective that’s a little
more awkward: remember that we currently have an arbitrary limit of 10 members
in a user’s role array, to avoid excessive fan out on cluster-internal
operations. Partitioned dbs could get away with more, more easily however. If
we allow the specification of access control in two lists, and one of the lists
implies membership in the other, we have a total limit of 10 members across
both arrays. Or we limit 5 + 5, but that seems excessive, while 10 total seems
weird, but doable. Anyway, good bikeshed.
* * *
So far. I think all of the problems outlined are solvable, if with a clear
definition of what use-cases we do not support with access. If you have more
scenarios than the ones I outlined, please add them and we can see if they
cause any additional trouble.
Thanks for reading this far and I’m looking forward to your feedback.
Best,
Jan “_access” Lehnardt
—
> On 17. Feb 2019, at 15:25, Jan Lehnardt <j...@apache.org> wrote:
>
> Hi Everyone,
>
> I’m happy to share my work in progress attempt to implement the per-doc
> access control feature we discussed a good while ago:
>
> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E
>
> <https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E>
>
> You can check out my branch here:
>
> https://github.com/apache/couchdb/compare/access?expand=1
> <https://github.com/apache/couchdb/compare/access?expand=1>
>
> It is very much work in progress, but it is far enough along to warrant
> discussion.
>
> The main point of this branch is to show all the places that we would need to
> change to support the proposal.
>
> Things I’ve left for later:
>
> - currently only the first element in the _access array is used. Our and/or
> syntax can be added later.
> - building per-access views has not been implemented yet, couch_index would
> have to be taught about the new per-access-id index.
> - pretty HTTP error handling
> - tests except for a tiny shell script 😇
>
> Implementation notes:
>
> You create a database with the _access feature turned on like so: PUT
> /db?access=true
>
> I started out with storing _access in the document body, as that would allow
> for a minimal change set, however, on doc updates, we try hard not to load
> the old doc body from the database, and forcing us to do so for EVERY doc
> update under _access seemed prohibitive, so I extended the #doc, #doc_info
> and #full_doc_info records with a new `access` attribute that is stored in
> both by-id and by-seq. I will need guidance on how extending these records
> impact multi-version cluster interop. And especially whether this is an
> acceptable approach.
>
> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-904ab7473ff8ddd07ea44aca414e3a36
>
> * * *
>
> The main addition is a new native query server called
> couch_access_native_proc, which implements two new indexes by-access-id and
> by-access-seq which do what you’d expect, pass in a userCtx and retrieve the
> equivalent of _all_docs or _changes, but only including those docs that match
> the username and roles in their _access property. The existing handlers for
> _all_docs and _changes have been augmented to use the new indexes instead of
> the default ones, unless the user is an admin.
>
> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-fbb53323f07579be5e46ba63cb6701c4
>
> * * *
>
> The rest of the diff is concerned with making document CRUD behave as you’d
> expect it. See this little demonstration for what things look like:
>
> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497
> <https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497> (I’m just
> noticing that there might be something wonky with DELETE, but you’ll get the
> gist #rimshot)
>
> * * *
>
> Open questions:
>
> - The aim of this is to get as close to regular CouchDB behaviour as
> possible. One thing that is new however which would require all apps to be
> changed is that for an _access enabled database to include an _access field
> in their docs (docs with no _access are admin-only for now). We might want to
> consider on new document writes to auto-insert the authenticated user’s name
> as the first element in the _access array, so existing apps “just work”.
>
> - Interplay with partitioned dbs: eschewing db-per-user is already a large
> boon if you have a lot of users, but making those per-user requests inside an
> _access enabled database efficient would be doubly nice, so why not use the
> username from the first question above and use that as the partition key?
> This would work nicely for natural users with their own docs that want to
> share them with others later, but I can easily imagine a pipelined use of
> CouchDB, where a “collector” user creates all new docs, an “analyser” takes
> them over and hand them to a “result” user for viewing. In that case, we’d
> violate the high-cardinality rule of partitions (have a lot of small ones),
> instead all docs go through all three users. I’d be okay with treating the
> later scenario as a minor use-case, but for that use-case, we should be able
> to disable auto-partitioning on db creation.
>
> - building access view indexes for docs that have frequent _access changes,
> lead to many orphaned view indexes, we should look at an auto-cleanup
> solution here (maybe keep 1-N indexes in case folks just swap back and forth).
>
> * * *
>
> I’ll leave this here for now, I’m sure there are a few more things to
> consider.
>
> I’d love to hear any and all feedback you might have. Especially if anything
> is unclear.
>
> Best
> Jan
> —
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
