Re: Shard level querying in CouchDB Proposal

2018-08-03 Thread Mike Rhodes
Joan, I'd like to reassure that the question has been considered in reasonable depth. My wording was looser than it should have been. However, the solution is non-obvious enough that a single "correct" way forward probably doesn't exist and reasonable people can argue in many directions. A per

Re: Shard level querying in CouchDB Proposal

2018-08-02 Thread Jan Lehnardt
> On 2. Aug 2018, at 15:37, Garren Smith wrote: > > There was a discussion on #couchdb-dev and the proposal is that instead of > using _shards we would rather have _nodes so we could requests like /_node/ > couchdb@192.168.0.3/shards%2f-1%2ffoo/docid Clarification: I was proposing to

Re: Shard level querying in CouchDB Proposal

2018-08-02 Thread Garren Smith
There was a discussion on #couchdb-dev and the proposal is that instead of using _shards we would rather have _nodes so we could requests like /_node/ couchdb@192.168.0.3/shards%2f-1%2ffoo/docid That then allows us to use _partition as an endpoint for the partition work without us confusin

Re: Shard level querying in CouchDB Proposal

2018-08-02 Thread Joan Touzet
> I do agree with the confusion aspect of shards and partitions, and > I'm unsure exactly the way forward here yet :( This is all I care about, and I'm cross that this hasn't been given more serious consideration, given CouchDB is already confusing for people coming from other databases. Make all

Re: Shard level querying in CouchDB Proposal

2018-08-02 Thread Mike Rhodes
Joan, I'm in agreement that this feature isn't "more important" -- but neither is it "less important". They're both vital for different sets of users (app devs vs. admins I think). Disclosure: this URL scheme was mostly my thinking and is partly based around a future that is more partition awa

Re: Shard level querying in CouchDB Proposal

2018-08-01 Thread Joan Touzet
Hi everyone, Recently, Garren and Robert started making progress on this proposal via a PR: https://github.com/apache/couchdb/pull/1480 and specifically: https://github.com/apache/couchdb/pull/1480#issuecomment-409565736 which has lead to me with a strong -0.75 on the proposed endpoint of:

Re: Shard level querying in CouchDB Proposal

2018-04-16 Thread Geoffrey Cox
I don’t think I have misunderstood the design. IMO, one of the most difficult pieces of dealing with db-per-user is that you have to replicate data between DBs. In the most simple example, if two users have access to the same doc then that doc needs to be in both user’s DBs. The shard key proposal

Re: Shard level querying in CouchDB Proposal

2018-04-16 Thread Mike Rhodes
> On 8 Feb 2018, at 17:22, Geoffrey Cox wrote: > > Thanks for the clarification, Mike. I figured that having multiple shard > keys per doc would be a big change, but I was hoping I was wrong ;). I > still think your proposed solution will add a lot of value to CouchDB. > Unfortunately, it isn't

Re: Shard level querying in CouchDB Proposal

2018-02-08 Thread Geoffrey Cox
Thanks for the clarification, Mike. I figured that having multiple shard keys per doc would be a big change, but I was hoping I was wrong ;). I still think your proposed solution will add a lot of value to CouchDB. Unfortunately, it isn't going to be part of the silver bullet that makes it feasible

Re: Shard level querying in CouchDB Proposal

2018-02-07 Thread Mike Rhodes
Geoff, Apologies for taking ages to reply. You can only have a single shard key per document, because the shard key directly affects where the document ends up being written to disk and, modulo shard replicas, a document is only in one place within the primary data shards. I think what you are

Re: Shard level querying in CouchDB Proposal

2018-01-23 Thread Adam Kocoloski
The way I understand the proposal, you could satisfy at most one of those requests (probably the *username* one) with a local query. The other one would have to be a global query, but the proposal does allow for a mix of local and global queries against the same dataset. Adam > On Jan 22, 2018

Re: Shard level querying in CouchDB Proposal

2018-01-22 Thread Geoffrey Cox
Hey Mike, I've been thinking more about your proposal above and when it is combined with the new access-per-db enhancement it should greatly reduce the need for db-per-user. One thing that I'm left wondering though is whether there is consideration for different shard keys per doc. From what I gat

Re: Shard level querying in CouchDB Proposal

2017-11-28 Thread Mike Rhodes
> On 25 Nov 2017, at 15:45, Adam Kocoloski wrote: > > Yes indeed Jan :) Thanks Mike for writing this up! A couple of comments on > the proposal: > >> • For databases where this is enabled, every document needs a shard key. > > What would happen if this constraint were relaxed, and docume

Re: Shard level querying in CouchDB Proposal

2017-11-28 Thread Mike Rhodes
On 24 Nov 2017, at 16:45, Jan Lehnardt wrote: > > Hi Mike, > > this is a great proposal, thanks a lot! I like this general idea, I guess > this was what Adam was hinting at in the _access draft review ;) Heh. But good to see the positive reaction! > I left one comments inline. > >> From this

Re: Shard level querying in CouchDB Proposal

2017-11-25 Thread Adam Kocoloski
Yes indeed Jan :) Thanks Mike for writing this up! A couple of comments on the proposal: > • For databases where this is enabled, every document needs a shard key. What would happen if this constraint were relaxed, and documents without a “:” in their ID simply used the full ID as the sha

Re: Shard level querying in CouchDB Proposal

2017-11-24 Thread Jan Lehnardt
Hi Mike, this is a great proposal, thanks a lot! I like this general idea, I guess this was what Adam was hinting at in the _access draft review ;) I left one comments inline. > On 22. Nov 2017, at 14:56, Mike Rhodes wrote: > > All, > > Just to be clear before starting: this is a proposal fr

Re: Shard level querying in CouchDB Proposal

2017-11-24 Thread Mike Rhodes
Johs, Good question. I think really it comes down to allowing the developer more options in how they access data, and how the database can help this be more efficient. I don't think it reduces the usefulness of clustering. We still have three replicas of each shard, which allows different node

Re: Shard level querying in CouchDB Proposal

2017-11-23 Thread Johs Ensby
Hi Mike and Geoff, forgive me if I am asking a really stupid question, but wouldn't restricting certain data to specific shards defy the very concept and core benefits of a clustered database? br Johs > On 23 Nov 2017, at 22:49, Geoffrey Cox wrote: > > Ah, yeah, this makes sense to me. I think

Re: Shard level querying in CouchDB Proposal

2017-11-23 Thread Geoffrey Cox
Ah, yeah, this makes sense to me. I think this has great potential! On Thu, Nov 23, 2017 at 4:56 AM Mike Rhodes wrote: > > > > On 22 Nov 2017, at 18:39, Geoffrey Cox wrote: > > > > Hi Mike, this sounds like a pretty cool enhancement. Just to clarify, > > you're also proposing modifying the PUT/

Re: Shard level querying in CouchDB Proposal

2017-11-23 Thread Mike Rhodes
> On 22 Nov 2017, at 18:39, Geoffrey Cox wrote: > > Hi Mike, this sounds like a pretty cool enhancement. Just to clarify, > you're also proposing modifying the PUT/POST doc, etc... so that you can > specify a shard key per doc so that the doc can be stored on a specific > shard? Yes, sort of.

Re: Shard level querying in CouchDB Proposal

2017-11-22 Thread Geoffrey Cox
Hi Mike, this sounds like a pretty cool enhancement. Just to clarify, you're also proposing modifying the PUT/POST doc, etc... so that you can specify a shard key per doc so that the doc can be stored on a specific shard? On Wed, Nov 22, 2017 at 5:57 AM Mike Rhodes wrote: > All, > > Just to be c

Shard level querying in CouchDB Proposal

2017-11-22 Thread Mike Rhodes
All, Just to be clear before starting: this is a proposal from Cloudant not just myself :D ## What? Introduce a user-specified shard key per document and a way for the user to scope queries to a single shard using this key, thereby reducing query latency by allowing the database to consult on