Re: Running 1.x-era JavaScript tests against master

[Will Holley]

It's probably easier to configure the cluster with n=1 rather than change
write quorum on a per-request basis. This was the approach we took with the
PouchDB tests and it worked well (as a first step, at least).

Will
On 8 Apr 2015 20:26, Sebastian Rothbucher 
sebastianrothbuc...@googlemail.com wrote:

 Hi,

 went a little further - and it could become a bumpy ride (see

 https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest
 )
 for how it got so far:
 - you have to use full write quorum (e.g. via query param), otherwise
 you'll end up with lots of Heisenbugs (because you might get the shard not
 yet up to date; waits are not the solution. Replication tests could become
 tricky)
 - correctly detected the Etags issue (which is not yet solved - there is a
 chttpd PR open)

 Cheers
  Sebastian

 On Wed, Apr 1, 2015 at 11:28 PM, Sebastian Rothbucher 
 sebastianrothbuc...@googlemail.com wrote:

  Hi Jan,
 
  I tried just that but there is lots of work to do. Maybe it's just minor,
  but the tests assume a lot which a normal user does not necessarily do,
  they depend on time, etc.
  First attempt for others 2 try also:
 
 https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest
 
  Let's see how far this can get...
 
  Cheers
  Sebastian
 
  On Wed, Mar 18, 2015 at 10:24 PM, Jan Lehnardt j...@apache.org wrote:
 
  Hi all,
 
  I’m concerned about people upgrading from 1.x to 2.0 and finding that
  a number of small things are subtly broken.
 
  I propose we figure out how to run the JavaScript test suite against
  the clustered port in master, as they are a decent approximation of
  what 1.x clients expect.
 
  There will be a few things that need disabling (tests for _config
  e.g.) and some need adjusting (expecting the seq_id to be a number),
  but overall, I think this is not a bad task.
 
  I know the JavaScript tests aren’t well liked here, but we’ve spent
  quite some time putting a lot of CouchDB API knowledge into them
  and I don’t want to miss an opportunity to make upgrades to 2.0 as
  seamless as possible.
 
  What do you think?
 
  Best
  Jan
  --
  Professional Support for Apache CouchDB:
  http://www.neighbourhood.ie/couchdb-support/
 
 
 

Re: Intent to ship: _bulk_get in PouchDB 5.0.0 and PouchDB Server 1.0.0

[Will Holley]

ermouth: _bulk_get is not available on Cloudant today but if it lands
in CouchDB and has a positive effect on replication then I expect
we'll look into supporting it (not my call).

On 15 September 2015 at 16:21, ermouth <ermo...@gmail.com> wrote:
> Wonderful feature, seems that it can reduce replication costs dramatically,
> thanks.
>
>  > I've gotten the +1 from Will Holley that our implementation is consistent
>> with the proposed CouchDB version
>
> Does Cloudant already has this endpoint onboard? I‘ve tried to, but failed
> – although not sure I did all the things right.
>
> ermouth
>
> 2015-09-15 17:09 GMT+03:00 Nolan Lawson <no...@nolanlawson.com>:
>
>> Hi friends,
>>
>> I've been working for some time with folks from CouchDB/Cloudant to get an
>> implementation of _bulk_get into CouchDB. The goal of the new endpoint is
>> to vastly speed up replication by avoiding unnecessary additional HTTP
>> requests [1].
>>
>> There is already a PR by Alexander Shorin out on CouchDB itself [2], based
>> on some discussion of prior art [3] including a similar endpoint in rcouch
>> and Couchbase Sync Gateway.
>>
>> In PouchDB, we implemented both a server-side version in PouchDB Server [4]
>> as well as a client-side version [5] which falls back to normal slow GETs
>> if it doesn't find a _bulk_get endpoint.
>>
>> I've gotten the +1 from Will Holley that our implementation is consistent
>> with the proposed CouchDB version, so it is very likely that PouchDB will
>> ship with support for the endpoint in the next release (5.0.0, sometime
>> around October 1st), at which time we'll also release PouchDB Server 1.0.0,
>> so that folks can try out the fast new sync.
>>
>> If you would like to test out the implementation yourself, it's just:
>>
>> npm install -g pouchdb-server@pouchdb
>> /pouchdb-server#use-pouchdb-master
>> npm install pouchdb@pouchdb/pouchdb
>>
>> Or to download the browserified version of the master branch, you can go to
>> http://pouchtest.com/nightly .
>>
>> Please try the proposed implementation and let us know at
>> http://slack.pouchdb.com, irc://freenode.net/#pouchdb, or
>> https://groups.google.com/forum/#!forum/pouchdb if you have any issues
>> with
>> it. I'm also going to take the time before the next release to test against
>> Couchbase Sync Gateway. Although it's passing in our CI [6] I would like to
>> confirm that we are using CSG's endpoint as intended.
>>
>> Cheers,
>> Nolan
>>
>> [1]: https://issues.apache.org/jira/browse/COUCHDB-2310
>> [2]: https://github.com/apache/couchdb-chttpd/pull/33
>> [3]: https://github.com/apache/couchdb-couch/pull/18
>> [4]: https://github.com/pouchdb/express-pouchdb/pull/208
>> [5]: https://github.com/pouchdb/pouchdb/pull/4335
>> [6]: https://travis-ci.org/pouchdb/pouchdb/jobs/80300066
>>
>> --
>> Nolan Lawson
>> nolanlawson.com
>> github.com/nolanlawson
>>

docs for 1.6.1

[Will Holley]

Hi all,

I noticed that the docs for 1.6.1 seem to be missing. If I visit
http://docs.couchdb.org/ I get redirected to
http://docs.couchdb.org/en/1.6.1/ which seems to be the WIP 2.0 docs.

Is this intentional (I assume not)?

Will

Re: Getting libraries to test RCs

[Will Holley]

Thanks Joan,

I think PouchDB can work around both issues if the decision is that
they represent expected behaviour in 2.0 - that's a call we need the
CouchDB dev team to make though.

On 2 September 2016 at 10:15, Joan Touzet <woh...@apache.org> wrote:
> Hi Will,
>
> Neither of these are currently tagged as blocking issues for CouchDB
> 2.0, only major priority. If you want to flag them as such, this is
> your last chance, and even still, there's no guarantee fixes for them
> will hit 2.0.
>
> Erlangers, is there any chance of at least triaging these today?
>
> -Joan
>
> ----- Original Message -
>> From: "Will Holley" <willhol...@gmail.com>
>> To: dev@couchdb.apache.org, "Joan Touzet" <woh...@apache.org>
>> Sent: Friday, September 2, 2016 4:43:48 AM
>> Subject: Re: Getting libraries to test RCs
>>
>> Assuming nothing's changed in the last few weeks, there are 2 issues
>> which cause the PouchDB tests to fail against master: COUCHDB-3017
>> and
>> COUCHDB-3034.
>>
>> Both could be addressed in the test suite by using different database
>> names for each test, but that's quite a disruptive change.
>>
>> On 2 September 2016 at 03:15, Joan Touzet <woh...@apache.org> wrote:
>> > Hi Nolan, you state that it's 'failing for known reasons.' Is that
>> > reasons in PouchDB or anything you need to push back on us? We'd
>> > like
>> > to know ASAP as we're very, very close to releasing 2.0 now.
>> >
>> > I have zero PouchDB knowledge so I'm hoping you can give us a short
>> > summary of what you think is wrong.
>> >
>> > All the best,
>> > Joan
>> >
>> > - Original Message -
>> >> From: "Nolan Lawson" <no...@nolanlawson.com>
>> >> To: dev@couchdb.apache.org
>> >> Sent: Thursday, September 1, 2016 7:56:42 PM
>> >> Subject: Re: Getting libraries to test RCs
>> >>
>> >> We have been testing CouchDB master in PouchDB for months now, but
>> >> as
>> >> an allowed failure because I believe it’s failing for known
>> >> reasons.
>> >> We test both using Node.js and the browser.
>> >>
>> >> Node: https://travis-ci.org/pouchdb/pouchdb/jobs/156198210
>> >> Browser: https://travis-ci.org/pouchdb/pouchdb/jobs/156198211
>> >>
>> >> For anyone who wants to run the Pouch test suite against CouchDB,
>> >> it’s just:
>> >>
>> >> git clone https://github.com/pouchdb/pouchdb.git
>> >> cd pouchdb
>> >> npm I
>> >> COUCH_HOST=http://localhost:5984 BAIL=0 npm t
>> >>
>> >> BAIL=0 will tell it to run the full test suite and not stop on any
>> >> failures. That way you can inspect the failures and see if they’re
>> >> serious or not.
>> >>
>> >> Cheers,
>> >> Nolan
>> >>
>> >> > On Aug 29, 2016, at 12:15 PM, Jan Lehnardt <j...@apache.org>
>> >> > wrote:
>> >> >
>> >> > Anyone on this list who could help with this? The work items are
>> >> > fairly self-explanatory and not very big individually <3
>> >> >
>> >> > Best
>> >> > Jan
>> >> > --
>> >> >
>> >> >> On 10 Aug 2016, at 09:37, Jan Lehnardt <j...@apache.org> wrote:
>> >> >>
>> >> >> Hey everyone,
>> >> >>
>> >> >> from Joan’s excellent blog post about testing Release
>> >> >> Candidates:
>> >> >>
>> >> >>> To our valued CouchDB application and library developers:
>> >> >>> please,
>> >> >>> please run your software against each of the options below.
>> >> >>
>> >> >> — https://blog.couchdb.org/2016/08/08/release-candidates/
>> >> >>
>> >> >> I think we can be a little more proactive about this for
>> >> >> CouchDB
>> >> >> client libraries: let’s open issues on all the
>> >> >> CouchDB-compatible
>> >> >> client software we care about to test an RC.
>> >> >>
>> >> >> Since there are a lot of projects, and we don’t necessarily
>> >> >> know
>> >> >> which one we “care” about, we should try to be clever about it.
>> >> >

Re: Getting libraries to test RCs

[Will Holley]

Assuming nothing's changed in the last few weeks, there are 2 issues
which cause the PouchDB tests to fail against master: COUCHDB-3017 and
COUCHDB-3034.

Both could be addressed in the test suite by using different database
names for each test, but that's quite a disruptive change.

On 2 September 2016 at 03:15, Joan Touzet  wrote:
> Hi Nolan, you state that it's 'failing for known reasons.' Is that
> reasons in PouchDB or anything you need to push back on us? We'd like
> to know ASAP as we're very, very close to releasing 2.0 now.
>
> I have zero PouchDB knowledge so I'm hoping you can give us a short
> summary of what you think is wrong.
>
> All the best,
> Joan
>
> - Original Message -
>> From: "Nolan Lawson" 
>> To: dev@couchdb.apache.org
>> Sent: Thursday, September 1, 2016 7:56:42 PM
>> Subject: Re: Getting libraries to test RCs
>>
>> We have been testing CouchDB master in PouchDB for months now, but as
>> an allowed failure because I believe it’s failing for known reasons.
>> We test both using Node.js and the browser.
>>
>> Node: https://travis-ci.org/pouchdb/pouchdb/jobs/156198210
>> Browser: https://travis-ci.org/pouchdb/pouchdb/jobs/156198211
>>
>> For anyone who wants to run the Pouch test suite against CouchDB,
>> it’s just:
>>
>> git clone https://github.com/pouchdb/pouchdb.git
>> cd pouchdb
>> npm I
>> COUCH_HOST=http://localhost:5984 BAIL=0 npm t
>>
>> BAIL=0 will tell it to run the full test suite and not stop on any
>> failures. That way you can inspect the failures and see if they’re
>> serious or not.
>>
>> Cheers,
>> Nolan
>>
>> > On Aug 29, 2016, at 12:15 PM, Jan Lehnardt  wrote:
>> >
>> > Anyone on this list who could help with this? The work items are
>> > fairly self-explanatory and not very big individually <3
>> >
>> > Best
>> > Jan
>> > --
>> >
>> >> On 10 Aug 2016, at 09:37, Jan Lehnardt  wrote:
>> >>
>> >> Hey everyone,
>> >>
>> >> from Joan’s excellent blog post about testing Release Candidates:
>> >>
>> >>> To our valued CouchDB application and library developers: please,
>> >>> please run your software against each of the options below.
>> >>
>> >> — https://blog.couchdb.org/2016/08/08/release-candidates/
>> >>
>> >> I think we can be a little more proactive about this for CouchDB
>> >> client libraries: let’s open issues on all the CouchDB-compatible
>> >> client software we care about to test an RC.
>> >>
>> >> Since there are a lot of projects, and we don’t necessarily know
>> >> which one we “care” about, we should try to be clever about it.
>> >>
>> >> Maybe something like this can work:
>> >>
>> >> 1. We prepare an issue text explaining the thing: Heya, CouchDB
>> >> team here, major new version coming up, you should test it like
>> >> so: > >> even provide a cluster to do this, or Cloudant can sponsor
>> >> something?
>> >>
>> >> 2. Post this message with a call to action on user@c.a.o, the
>> >> weekly news, and our other (social) media channels.
>> >>
>> >> 3. Ask people who submitted an issue to report back with a link.
>> >>
>> >> 4. Collect the link in an issue or JIRA (this could be done in 3.,
>> >> but then everybody needs to be added to the wiki write group, and
>> >> that’s just extra overhead we don’t need). Maybe we borrow a gist
>> >> for this, or a Google doc.
>> >>
>> >> That way we encourage client software to check out RCs and we can
>> >> keep track, while the community helps to select which software to
>> >> encourage to test 2.0 compat, and helps spread the word and the
>> >> burden is not left with just a few folks.
>> >>
>> >> What do you think?
>> >>
>> >> Best
>> >> Jan
>> >> --
>> >>
>> >
>> > --
>> > Professional Support for Apache CouchDB:
>> > https://neighbourhood.ie/couchdb-support/
>> >
>>
>>

Re: Getting libraries to test RCs

[Will Holley]

Jan - I can understand that being the case in a clustered setup with
distributed shard maps but shouldn't n=1 mitigate that?

On 2 September 2016 at 10:53, Jan Lehnardt <j...@apache.org> wrote:
>
>> On 02 Sep 2016, at 11:45, Dale Harvey <d...@arandomurl.com> wrote:
>>
>> In PouchDB we used to generate unique database names for tests, however we
>> removed it for serveral reasons, one large reason being it indicates a race
>> condition in critical code if we cannot reliably create -> delete -> create
>> the same database (we have uncovered and fixed a lot of bugs in PouchDB due
>> to this). While its not my call how to prioritise those bugs, I really do
>> not think we should be closing what are fairly serious bugs because it
>> wasnt inconvenient to workaround them in the couch test suite.
>
> It’s just that a CouchDB 2.0 cluster is an AP system, and recreating databases
> in quick succession reliably basically requires a CA system and that’s not 
> what can do easily.
>
> (I hope I got the CAP letters right, but I think it is clear what I mean)
>
> That is, maybe we skip those tests when run against a CouchDB 2.0 endpoint 
> and keep them for PouchDB?
>
> Best
> Jan
> --
>
>
>>
>> On 2 September 2016 at 10:31, Joan Touzet <woh...@apache.org> wrote:
>>
>>> Hi Nolan, Will:
>>>
>>> A further update from looking deeper with @janl. It appears that we
>>> have a pending fix for COUCHDB-3017 and we'll work on getting that
>>> merged before 2.0.
>>>
>>> COUCHDB-3034 is a WONTFIX. FYI in CouchDB itself we changed all of
>>> our tests to use unique database names. I'll update the bug myself
>>> shortly.
>>>
>>> -Joan
>>>
>>> - Original Message -
>>>> From: "Joan Touzet" <woh...@apache.org>
>>>> To: dev@couchdb.apache.org
>>>> Sent: Friday, September 2, 2016 5:15:00 AM
>>>> Subject: Re: Getting libraries to test RCs
>>>>
>>>> Hi Will,
>>>>
>>>> Neither of these are currently tagged as blocking issues for CouchDB
>>>> 2.0, only major priority. If you want to flag them as such, this is
>>>> your last chance, and even still, there's no guarantee fixes for them
>>>> will hit 2.0.
>>>>
>>>> Erlangers, is there any chance of at least triaging these today?
>>>>
>>>> -Joan
>>>>
>>>> - Original Message -
>>>>> From: "Will Holley" <willhol...@gmail.com>
>>>>> To: dev@couchdb.apache.org, "Joan Touzet" <woh...@apache.org>
>>>>> Sent: Friday, September 2, 2016 4:43:48 AM
>>>>> Subject: Re: Getting libraries to test RCs
>>>>>
>>>>> Assuming nothing's changed in the last few weeks, there are 2
>>>>> issues
>>>>> which cause the PouchDB tests to fail against master: COUCHDB-3017
>>>>> and
>>>>> COUCHDB-3034.
>>>>>
>>>>> Both could be addressed in the test suite by using different
>>>>> database
>>>>> names for each test, but that's quite a disruptive change.
>>>>>
>>>>> On 2 September 2016 at 03:15, Joan Touzet <woh...@apache.org>
>>>>> wrote:
>>>>>> Hi Nolan, you state that it's 'failing for known reasons.' Is
>>>>>> that
>>>>>> reasons in PouchDB or anything you need to push back on us? We'd
>>>>>> like
>>>>>> to know ASAP as we're very, very close to releasing 2.0 now.
>>>>>>
>>>>>> I have zero PouchDB knowledge so I'm hoping you can give us a
>>>>>> short
>>>>>> summary of what you think is wrong.
>>>>>>
>>>>>> All the best,
>>>>>> Joan
>>>>>>
>>>>>> - Original Message -
>>>>>>> From: "Nolan Lawson" <no...@nolanlawson.com>
>>>>>>> To: dev@couchdb.apache.org
>>>>>>> Sent: Thursday, September 1, 2016 7:56:42 PM
>>>>>>> Subject: Re: Getting libraries to test RCs
>>>>>>>
>>>>>>> We have been testing CouchDB master in PouchDB for months now,
>>>>>>> but
>>>>>>> as
>>>>>>> an allowed failure because I believe it’s failing for known
>>>>>>> reasons.
>>>>>>> We

Re: [GitHub] glonco commented on issue #1315: Different behavior "sorting" in the couchdb in version 2.1.0 and 2.1.1

[Will Holley]

Ah. 2.1.1 has a bug in /_explain when a sort is specified
(https://github.com/apache/couchdb/pull/1025).

That said, the 2.1.0 output looks broken - it's not using an index on
"test" so can't be sorting. Were the results actually sorted?

On 9 May 2018 at 09:27, GitBox  wrote:
> glonco commented on issue #1315: Different behavior "sorting" in the couchdb 
> in version 2.1.0 and 2.1.1
> URL: https://github.com/apache/couchdb/issues/1315#issuecomment-387662797
>
>
>I created index for field "test" in both version of couch db.
>Here you can see _explain from both versions of couch db:
>v2.1.0 ->
>`
>{
>"dbname": "test",
>"index": {
>"ddoc": null,
>"name": "_all_docs",
>"type": "special",
>"def": {
>"fields": [
>{
>"_id": "asc"
>}
>]
>}
>},
>"selector": {
>"name": {
>"$eq": "test1"
>}
>},
>"opts": {
>"use_index": [],
>"bookmark": "nil",
>"limit": 100,
>"skip": 0,
>"sort": {
>"test": "desc"
>},
>"fields": "all_fields",
>"r": [
>49
>],
>"conflicts": false,
>"stale": false,
>"update": true,
>"stable": false
>},
>"limit": 100,
>"skip": 0,
>"fields": "all_fields",
>"range": {
>"start_key": null,
>"end_key": "\ufffd"
>}
>}`
>v2.1.1 ->
>`{
>"error": "error",
>"reason": "{invalid_object,{<<\"\ufffd\ufffd\ufffd\ufffd\">>}}"
>}`
>Here it is error.
>
>So It is bug in the version 2.1.1 or it was bug in the version2.1.0 ?
>
> 
> This is an automated message from the Apache Git Service.
> To respond to the message, please log on GitHub and use the
> URL above to go to the specific comment.
>
> For queries about this service, please contact Infrastructure at:
> us...@infra.apache.org
>
>
> With regards,
> Apache Git Services

Re: [DISCUSS] Implementing Mango Indexes for FoundationDB

[Will Holley]

Thanks for kicking this off Garren! I have a few questions / thoughts:

1. Will these indexes continue to have associated design documents? I
assume this aspect wouldn't change, but it would be good to be explicit.
Aside from being required for replication, it's useful to be able to query
the /db/_design//_info endpoint to get the index size etc. Perhaps
this also addresses the issue around how to get the set of available
indexes?

2. Does the ICU sort key have a bounded length? Mostly I'm wondering
whether we can guarantee that the generated keys will fit within the
maximum FDB key length or if there needs to be some thought as to the
failure mode / workaround. As Adam mentioned, it seems fine to store an
encoded key given Mango (currently) always fetches the associated document
/ fields from the primary index to filter on anyway. It might even be
beneficial to have an additional layer of indirection and allow multiple
docs to be associated with each row so that we can maintain compact keys.

3. I don't immediately see how you clear previous values from the index
when a doc is updated, but I could easily be missing something obvious :)

4. Regarding "Index on write" behaviour, is there something in the existing
design (Mango overlaying mrview / lucene) that would prevent this? I can
see some benefit for certain workloads (and headaches for others) but I
don't see that it's necessarily coupled to the Mango design given
background indexing of new/changed indexes needs to be supported anyway.

5. I have a few concerns about the performance given we'll no longer be
able to push down filtering to the data storage tier. It sounds [1] like a
key thing will be the extent to which we can concurrently fetch/filter
non-overlapping key ranges from FDB - likely something that will come up in
relation to views as well. This gets a bit more fun when considering
multi-tenancy, and is perhaps something to stew on whilst getting something
functional together.

Will

[1] https://forums.foundationdb.org/t/secondary-indexing-approaches/792/4


On Thu, 28 Mar 2019 at 17:48, Adam Kocoloski  wrote:

> Hi Garren, cool, this is a good start.
>
> On the ICU side of things, Russell pointed out that sort keys are a
> one-way trip; i.e., there’s no way to recover the original string from a
> sort key. For the initial pass at Mango I think that’s OK, as we’re reading
> the indexed documents anyway. When we get to views I guess the design will
> need to store the original string in the value so that we can return it as
> the “key” field in the response.
>
> Adam
>
> > On Mar 28, 2019, at 7:01 AM, Garren Smith  wrote:
> >
> > Hi everyone,
> >
> >
> > I want to start a discussion, with the aim of an RFC, around implementing
> > Mango JSON indexes for FoundationDB. Currently Mango indexes are a layer
> > above CouchDB map/reduce indexes, but with FoundationDB we can make them
> > separate indexes in FoundationDB. This gives us the possibility of being
> > able to update the indexes in the same transaction that a document is
> being
> > saved in. Later we can look at adding specific mango like covering
> indexes.
> >
> >
> > Lets dive into the data model. Currently a user defines an index like
> this:
> >
> >
> > {
> >
> >  name: ‘view-name’ - optional will be auto-generated
> >
> >  index: {
> >
> >fields: [‘fieldA’, ‘fieldB’]
> >
> >  },
> >
> >  partial_filter_selector {} - optional
> >
> > }
> >
> >
> > For query planning we need to be able to access the list of available
> > indexes. So we would have a index_definitions subspace with the following
> > content:
> >
> >
> > (, …) = (,
> > )
> >
> >
> > Otherwise we could just store the index definitions as:
> >
> > (index_name) = ((fields), partial_filter_selector).
> >
> >
> > At this stage, I can’t think of a fancy way of storing the index
> > definitions so that when we need to select an index for a query there
> would
> > be a fast way to only fetch a subset of the indexes. I think the best is
> to
> > rather fetch them all like we currently do and process them. However, we
> > can look at caching these index definitions in the application layer, and
> > using FoundationDB watches[0] to notify us when a definition has changed
> so
> > we can update the cached definitions.
> >
> >
> > Then each index definition will have its own dedicated subspace for the
> > actual built index key/values. Keys in this subspace would be the fields
> > defined in the index with the doc id at the end of the tuple, e.g for an
> > index with fields name and age, it would be:
> >
> >
> > (“john”, 40, “doc-id-1) = null
> >
> > (“mary”, 30, “doc-id-2) = null
> >
> >
> > This follows the same key format that document layer[1] does for its
> > indexes. One point to make here is that the doc id is kept in the key
> part
> > so that we can avoid duplicate keys.
> >
> >
> > Then in terms of sorting the keys, current CouchDB uses ICU to sort all
> > secondary indexes. We would need to use ICU to sort the 

Re: [DISCUSS] Map Indexes

[Will Holley]

Thanks Garren,

As usual, a few questions :)

1. The data model suggests the idea of view groups gets carried over to
fdb. Are there API / behaviour reasons to keep them? Would an index update
transaction scope to a view group rather than a single view?
2. Regarding emitting doc as the value in a view function, this is so
common that I wonder if it's worth handling as a special case. It sounds
like there wouldn't be a solution for customers who use this technique to
ensure they can retrieve the version of the document that is consistent
with the emitted key?
3. When you say "Emitted keys will not be able to exceed 10 KB", do you
mean any single emitted key cannot exceed 10KB? The "id index" proposal
suggests there would also be a 100KB limit on the combined emitted key
length.

Cheers,

Will

On Mon, 15 Apr 2019 at 15:25, Garren Smith  wrote:

> Hi Everyone,
>
> I want to start a discussion around creating map/reduce view indexes. One
> way to get views indexes to work with FoundationDB is to break up a view
> index into indexes for the map functions and indexes for the reduce
> functions. Along those lines, I’m going to break the discussions into two,
> this discussion around map functions and indexes and then a another one on
> reduce functions and the indexes that go with those.
>
> ## Data model
> For a map function, we need to store the emitted keys and the emitted
> values:
>
> {?DATABASE, ?VIEWS, ?VIEW_SIGNATURE, ?VIEWS, , ?MAP, ,
> <_id>} -> 
>
> To briefly explain what the above means, it creates a views subspace in a
> database subspace, then every view defined on a design doc is grouped via
> the design doc’s view signature. The view_id is the name of the view in the
> design doc - we can look at ways to make that smaller to save some key
> space. The ?MAP groups the key/value into the view’s map index subspace,
> then we have the keys that were emitted for the map function and finally
> the _id field of the document used to create the keys for this row.
>
> ## Emitted Value
> The value stored for the row is the emitted value from the map function.
> Because we have a limitation on the size of the value field one caveat
> around this design is that a user will run into issues if they emit a
> document that exceeds 100KB. In CouchDB we don’t recommend users emitting
> the doc, but there are some nice speed optimisations you get by emitting
> the document as the value. With CouchDB on FDB that performance
> optimisation won’t be required and so we will have to actively discourage
> users from doing that.
>
> Just to note, a user would experience the same issue if they emit a value
> exceeding 100KB.
>
> ## Key ordering
> There are some changes to how we will manage keys emitted from a map
> function. For strings we will need to generate a ICU sort string upfront
> instead of using the ICU comparison. To maintain the way CouchDB currently
> does view collation [1], we need to prepend a type value to each key so
> that we get the correct sort order of null < boolean < numbers < strings <
> arrays < objects. CouchDB currently allows duplicate keys to be emitted for
> an index, to allow for that a counter value will be added to the end of the
> keys.
>
> ## Index Key Management
> For every document that needs to be processed for an index, we have to run
> the document through the javascript process to get the emitted keys and
> values. This means that it won’t be possible to update a map/reduce index
> in the same transaction that a document is updated. To account for this, we
> will need to keep an `id index` similar to the `id tree` we current keep.
> This index will hold the document id as the key and the value would be the
> keys that were emitted. We would then use this information to know which
> fields need to be updated or removed from the index when a document is
> changed.  A data model for this would be:
>
> {?DATABASE, ?VIEWS, ?VIEW_SIGNATURE, ?VIEWS, , ?ID_INDEX, <_id>,
> } -> [emitted keys]
>
> ## Updating an index
> To help in knowing which documents have changed since a view was last
> updated, we will need to keep the latest update sequence. This will change
> from the really long string we currently have, to using the sequence value
> defined in the _changes RFC [2].
>
> Based on all of that, a slightly expanded data model for map functions
> inside a database subspace would look like:
>
> * views
> * 
> * update_seq
> * idtree
> * (<_id>, ) -> [keys]
> * views
> * 
> * map
> * (, <_id>) -> 
>
> ## Size limits
> There are some size limits that are worth listing and keeping in mind.
>
> * Emitted keys will not be able to exceed 10 KB
> * Values cannot exceed 100 KB
> * Following from Alex’s email on how transaction sizes are calculated [3],
> there could be rare cases where the number of key-value pairs emitted for a
> map function could lead to a transaction either exceeding 10 MB which isn’t
> 

Re: IO queueing defaults

[Will Holley]

I defer to those with more operational experience of ken and smoosh but
wouldn't those new subsystems radically impact performance if IOQ is
completely bypassed (assuming ken/smoosh are enabled by default)?

On Wed, 11 Sep 2019 at 22:04, Adam Kocoloski  wrote:

> A few months ago a bunch of code landed on master around IO QoS and
> prioritization. I think we need to have a conversation about the defaults
> for that system and what we want to allow users to enable.
>
> First topic - there are actually two different generations of the IOQ
> system: IOQ and IOQ2. Only one can be active at a given time, and the
> configurations are not compatible. The best use case for this queueing
> system is to de-prioritize IO for bookkeeping tasks like internal
> replication and compaction in favor of IO to respond to client requests.
>
> The original and currently default IOQ system primarily works by
> classifying the IO based on whether it’s serving an interactive read or
> write request, an index build, a compaction job, etc. It builds queues for
> each of these IO classes and allows for relative prioritization of the
> different classes of IO. The main downside of this system is that it can
> only sustain a total throughput of about 20,000 operations/sec/node.
> Heavily-loaded systems frequently have to configure “bypasses” for certain
> classes of IO to keep latencies low.
>
> IOQ2 was conceived to deliver higher throughput without resorting to
> bypasses and thus defeating the QoS. It’s a significantly more complex
> system. Tenants are a first-class concept in IOQ2, but of course they’re
> not in the rest of the CouchDB, so some of the code in there that computes
> per-user priorities will not work correctly. As far as I can tell it will
> fail gracefully (i.e., it will bucket every database as belonging to the
> same “user”), but I doubt this has been tested. IOQ2 definitely can sustain
> higher throughputs, though it has been known to enqueue so many more IO
> requests than it can issue that it effectively led to an outage anyway. It
> is still a material overhead compared to bypassing the QoS entirely.
>
> I think there are a few possible paths forward:
>
> 1) Switch to IOQ2 and only document that one.
> 2) Document IOQ, installing bypasses across the board by default to avoid
> a big performance regression on upgrade
> 3) Just bypass the whole thing and don’t document it, to avoid introducing
> a big new admin capability in 3.0 and removing it in 4.0
>
> Personally I think I’m leaning towards 3) at this point, but could be
> convinced otherwise. Regards,
>
> Adam

Re: Proposed deprecations / removals (was: Re: Getting started on CouchDB 3.0, and an introduction)

[Will Holley]

>From an operational pov, the compaction daemon was removed in favour of
smoosh [1]. We owe some documentation around these new subsystems (smoosh,
ioq, ken) and some guides covering common cases (only compact during quiet
periods, disable compaction, etc) to aid migration.

[1] https://github.com/apache/couchdb/pull/1904

On Thu, 5 Sep 2019 at 03:17, Adam Kocoloski  wrote:

> I found a couple of other bits while grep’ing through the documentation
> for “deprecated”:
>
> > * The ``stale`` parameter for views and ``_find`` has been deprecated in
> favour
> >   of two new parameters: ``stable`` and ``update``. The old ``stale=ok``
> >   behaviour is equivalent to ``stable=true=false``, and the old
> >   ``stale=update_after`` behaviour is equivalent to
> ``stable=true=lazy``.
> >   The deprecated ``stale`` parameter will be removed in CouchDB 3.0.
>
> and
>
> > * :ghissue:`820`, :ghissue:`1032`: Multiple queries can now be made at
> the
> >   ``POST /{db}/_all_docs/queries``, ``POST /{db}/_design_docs/queries``
> and
> >   ``POST /{db}/_local_docs/queries`` endpoints. Also, a new endpoint
> >   ``POST /{db}/_design/{ddoc}/_view/{view}/queries`` has been introduced
> to replace
> >   the ``?queries`` parameter formerly provided for making multiple
> queries to a view.
> >   The old ``?queries`` parameter *is now deprecated and will be removed
> in a future
> >   release of CouchDB.*
>
> I’m not super-excited about following through on either of those
> deprecations :) I feel like the `stale` one in particular is still in heavy
> use. The queries one is probably OK.
>
> On the FDB topic, I believe the transaction logs are configured with the
> equivalent of delayed_commits=false (thus ensuring the durability of a
> transaction), while the storage servers only periodically fsync data that
> they pull off the transaction logs. I don’t think the fsync behavior can be
> configured using the normal configuration file, but if you go digging into
> the codebase there’s a Knobs file that has all kinds of settings that
> shouldn’t be messed with by mere mortals :) I wouldn’t be surprised if some
> of the durability settings can be tweaked there.
>
> Adam
>
> > On Sep 4, 2019, at 9:13 PM, Joan Touzet  wrote:
> >
> >
> >
> > On 2019-09-04 20:53, Adam Kocoloski wrote:
> >> Great list, Joan, thanks. I created a handful of GitHub issues for the
> first two sections and tossed them in the “Proposed for 3.x” column. I also
> submitted a PR to address one of them :)
> >
> > Cheers! I've also retitled that column "3.0 Release Tasks" since I think
> > that's how we're treating it. If it turns out anything needs to be
> > bumped to 3.x, we can create a new column for that.
> >
> >> Out of that list of already-deprecated functionality the only one that
> gives me pause is the delayed_commits one. We haven’t talked about it in
> years, and we ship a safe default. I’d be a little concerned about breaking
> folks who intentionally switched it on with full knowledge of the tradeoffs.
> >
> > Another option would be to just remove the setting from default.ini and
> > the documentation.
> >
> > Does FDB have anything similar?
> >
> > -Joan
> >
> >> Adam
> >>
> >>> On Sep 4, 2019, at 5:44 PM, Johs Ensby  wrote:
> >>>
> >>> Much appreciated, Joan
> >>> You're a hero
> >>> Johs
> >>>
>  On 4 Sep 2019, at 23:37, Joan Touzet  wrote:
> 
>  Hey Adam,
> 
> > When it comes to deprecating and/or removing functionality, I feel
> like I don’t know exactly where we stand today. We have occasionally
> described some of the CouchApp functionality as already being deprecated,
> but I’m having trouble finding any official record of that in our
> documentation.
>  Thanks for re-opening the deprecation discussion. I've reviewed [1]
> and
>  provide the following summary tables (Markdown format).
> 
>  **NOTE**: This is /not/ the vote for deprecation, nor a formal
>  announcement of such. This is a starting point for discussion. A vote
>  still needs to happen for this to move forward. Anything already
>  deprecated in 2.0 can be removed in 3.0 without a formal vote, but
> it'd
>  be nice if it got mentioned on the dev@ list before the PR lands on
>  master, please.
> 
>  As I'm going to be travelling for most of the rest of September, I'd
>  prefer if someone else (like Adam or Deni) can help drive this
> discussion.
> 
>  Once there is consensus from the community on these lists, we should
>  close #1534 and split it into 3 new tickets based on the tables below
>  (excepting the features already removed in 2.x).
> 
> > I guess let’s start with: does anyone believe we are in a position
> to be eliminating previously-deprecated functionality in 3.0?
> 
>  Yes, for the items in the 2nd table below, absolutely.
> 
>  -Joan "turning the tables" Touzet
> 
> 
>  
> 
> 
>  # Recently removed features in 2.x
> 
>  

[DISCUSS] Import CouchDB Helm Chart

[Will Holley]

Hi all,

CouchDB has a mature Helm chart at
https://github.com/helm/charts/tree/master/stable/couchdb to facilitate
deploying CouchDB clusters to Kubernetes. Historically, Helm charts have
been curated by the Helm team alongside a list of chart-specific
maintainers (@kocolosk in this case).

The Helm team have indicated that the current monolithic repository
approach is unsustainable for them and have started asking chart owners to
move to self-hosted repositories (most commonly GitHub pages -
https://github.com/helm/hub/blob/master/docs/github.md). These are then
aggregated at http://hub.helm.sh.

What do people think about moving the CouchDB chart from the helm/charts
Git repository into an Apache-owned repository (e.g.
github.com/apache/couchdb-helm) to complement the existing
apache/couchdb-docker repository?

If there's appetite for this I can move to a vote.

Thanks!

Will

p.s. for those interested, the workflow for deprecating the existing chart
is described at
https://github.com/helm/helm/blob/master/docs/charts.md#deprecating-charts.

Re: [DISCUSS] Import CouchDB Helm Chart

[Will Holley]

> Is the Helm team providing any CI assistance to self-hosted repositories?
The testing tools are freely available (
https://github.com/helm/chart-testing) but they don't provide any
infrastructure. We can potentially use something like Kind (
https://kind.sigs.k8s.io/) on Jenkins for deployment testing.

On Thu, 26 Sep 2019 at 17:31, Adam Kocoloski  wrote:

> It’s certainly a pain to get the Helm maintainers to approve Pull
> Requests, so I’m glad to see a strategy here.
>
> Is the Helm team providing any CI assistance to self-hosted repositories?
>
> Part of me wonders if ASF would want to host a repository to which all
> projects could contribute but I don’t personally feel like spending the
> cycles to find out :) I’m quite ok with CouchDB hosting its own.
>
> Adam
>
> > On Sep 26, 2019, at 12:18 PM, Will Holley  wrote:
> >
> > Hi all,
> >
> > CouchDB has a mature Helm chart at
> > https://github.com/helm/charts/tree/master/stable/couchdb to facilitate
> > deploying CouchDB clusters to Kubernetes. Historically, Helm charts have
> > been curated by the Helm team alongside a list of chart-specific
> > maintainers (@kocolosk in this case).
> >
> > The Helm team have indicated that the current monolithic repository
> > approach is unsustainable for them and have started asking chart owners
> to
> > move to self-hosted repositories (most commonly GitHub pages -
> > https://github.com/helm/hub/blob/master/docs/github.md). These are then
> > aggregated at http://hub.helm.sh.
> >
> > What do people think about moving the CouchDB chart from the helm/charts
> > Git repository into an Apache-owned repository (e.g.
> > github.com/apache/couchdb-helm) to complement the existing
> > apache/couchdb-docker repository?
> >
> > If there's appetite for this I can move to a vote.
> >
> > Thanks!
> >
> > Will
> >
> > p.s. for those interested, the workflow for deprecating the existing
> chart
> > is described at
> >
> https://github.com/helm/helm/blob/master/docs/charts.md#deprecating-charts
> .
>
>

[VOTE] Import CouchDB Helm Chart (was [DISCUSS] Import CouchDB Helm Chart)

[Will Holley]

>Hi all,
>
>CouchDB has a mature Helm chart at
https://github.com/helm/charts/tree/master/stable/couchdb >to facilitate
deploying CouchDB clusters to Kubernetes. Historically, Helm charts have
been >curated by the Helm team alongside a list of chart-specific
maintainers (@kocolosk in this case).
>
>The Helm team have indicated that the current monolithic repository
approach is unsustainable >for them and have started asking chart owners to
move to self-hosted repositories (most >commonly GitHub pages -
https://github.com/helm/hub/blob/master/docs/github.md). These are >then
aggregated at http://hub.helm.sh.
>
>What do people think about moving the CouchDB chart from the helm/charts
Git repository into >an Apache-owned repository (e.g.
github.com/apache/couchdb-helm) to complement the existing
>apache/couchdb-docker repository?
>
>If there's appetite for this I can move to a vote.
>

Not much discussion on this but also no objections raised so moving this to
a formal vote.

Shall we import the CouchDB Helm chart?

+1 from me.

All the best,

Will

Re: [VOTE] Import CouchDB Helm Chart (was [DISCUSS] Import CouchDB Helm Chart)

[Will Holley]

+1: 5
+0: 0
-1: 0

Thanks all - vote passes. I'll aim to start the migration this week.

Cheers,

Will

On Tue, 8 Oct 2019 at 18:25, Adam Kocoloski  wrote:

> +1 thanks Will
>
> > On Oct 8, 2019, at 9:31 AM, Will Holley  wrote:
> >
> > 
> >>
> >> Hi all,
> >>
> >> CouchDB has a mature Helm chart at
> > https://github.com/helm/charts/tree/master/stable/couchdb >to facilitate
> > deploying CouchDB clusters to Kubernetes. Historically, Helm charts have
> > been >curated by the Helm team alongside a list of chart-specific
> > maintainers (@kocolosk in this case).
> >>
> >> The Helm team have indicated that the current monolithic repository
> > approach is unsustainable >for them and have started asking chart owners
> to
> > move to self-hosted repositories (most >commonly GitHub pages -
> > https://github.com/helm/hub/blob/master/docs/github.md). These are >then
> > aggregated at http://hub.helm.sh.
> >>
> >> What do people think about moving the CouchDB chart from the helm/charts
> > Git repository into >an Apache-owned repository (e.g.
> > github.com/apache/couchdb-helm) to complement the existing
> >> apache/couchdb-docker repository?
> >>
> >> If there's appetite for this I can move to a vote.
> >>
> >
> > Not much discussion on this but also no objections raised so moving this
> to
> > a formal vote.
> >
> > Shall we import the CouchDB Helm chart?
> >
> > +1 from me.
> >
> > All the best,
> >
> > Will
>
>

Re: [VOTE] Release Apache CouchDB 3.0.0 (-rc1)

[Will Holley]

+1 make check passing on Centos 7, tested both amd64 and ppc64le. I'm going
to try on Centos 8 next...

One thing I noticed is that the Unix install instructions look outdated
with respect to Erlang / Node / Python dependencies though I'm not 100%
sure what the supported options are now.



On Thu, 13 Feb 2020 at 09:27, Peng Hui Jiang  wrote:

> +1 from "Debian 10 Buster on ppc64le" 's point of view.
>
> The test was performed against Debian 10 Buster docker image on Ubuntu
> ppc64le box.
>
> Eunit test can pass but just saw that t_stats_retained_on_job_removal
> might randomly fail due to timeout. Other test including `make javascript`
> and `make elixir` passed fine.
>
> jenkins@15706e15ee21:/tmp/couchdb/dist/apache-couchdb-3.0.0$ uname -a
> Linux 15706e15ee21 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:14:44
> UTC 2018 ppc64le GNU/Linux
> jenkins@15706e15ee21:/tmp/couchdb/dist/apache-couchdb-3.0.0$ lsb_release
> -sirc
> Debian
> 10
> buster
>
> ==
> Peng Hui Jiang
> Beijing
>
> [image: Inactive hide details for Joan Touzet ---2020/02/13 01:35:28
> PM---The Windows installer improvements are now live at the same U]Joan
> Touzet ---2020/02/13 01:35:28 PM---The Windows installer improvements are
> now live at the same URL below, the installer is called apac
>
> From: Joan Touzet 
> To: dev@couchdb.apache.org
> Date: 2020/02/13 01:35 PM
> Subject: [EXTERNAL] Re: [VOTE] Release Apache CouchDB 3.0.0 (-rc1)
> --
>
>
>
> The Windows installer improvements are now live at the same URL below,
> the installer is called apache-couchdb-3.0.0.1.msi.
>
> https://dist.apache.org/repos/dist/dev/couchdb/binary/win/3.0.0/rc.1/
>
> LOTS of improvements:
>
>   * Prompts for an admin user/password as CouchDB 3.0 requires
> * Will not overwrite existing credentials if in place
>   * No longer remove user-modified config files, closing
> apache/couchdb#1989
> * Also will not overwrite them on install, naturally :)
>   * Checkbox to disable installation of the Windows service
>   * Silent install support for all the above
>   * Friendly link to the online release notes in the exit dialog
>   * Higher resolution icon for HiDPI (500x500)
>
> The details are here:
>
>   https://github.com/apache/couchdb-glazier/pull/12
>
> The build runs fine on Windows, obviously, and `make check` passes
> entirely, so this is my official +1 on the RC.
>
> -Joan "If you can see what I can see/When I'm cleanin' windows" Touzet
>
>
> On 2020-02-10 14:36, I wrote:
> > Here are the Windows binaries:
> >
> >https://dist.apache.org/repos/dist/dev/couchdb/binary/win/3.0.0/rc.1/
>
> >
> > NOTE: You have to use Notepad (or similar) as Administrator and edit
> > your local.ini file before the service will start correctly.
> >
> > I've got a few minor things planned for the Windows installer to improve
> > on it, including prompting for an admin user on install, but this is
> > functional for now.
> >
> > Linux binaries of various flavours are being built right now.
> >
> > -Joan
> >
> > On 2020-02-10 13:32, Jan Lehnardt wrote:
> >> Hey all,
> >>
> >> Mac convenience binaries for your testing pleasure are available here:
> >>
> >>
> >> https://dist.apache.org/repos/dist/dev/couchdb/binary/mac/3.0.0/rc.1/
> >>
> >> Best
> >> Jan
> >> —
> >>
> >>> On 10. Feb 2020, at 17:33, Joan Touzet  wrote:
> >>>
> >>> Dear community,
> >>>
> >>> I would like to propose that we release Apache CouchDB 3.0.0.
> >>>
> >>> Candidate release notes:
> >>>
> >>> https://docs.couchdb.org/en/latest/whatsnew/3.0.html
> >>>
> >>> We encourage the whole community to download and test these release
> >>> artefacts so that any critical issues can be resolved before the
> >>> release is made. Everyone is free to vote on this release, so dig
> >>> right in! (Only PMC members have binding votes, but they depend on
> >>> community feedback to gauge if an official release is ready to be
> made.)
> >>>
> >>> The release artefacts we are voting on are available here:
> >>>
> >>> https://dist.apache.org/repos/dist/dev/couchdb/source/3.0.0/rc.1
> >>>
> >>> There, you will find a tarball, a GPG signature, and SHA256/SHA512
> >>> checksums.
> >>>
> >>> Please follow the test procedure here:
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
>
> >>>
> >>>
> >>> Please remember that "RC1" is an annotation. If the vote passes,
> >>> these artefacts will be released as Apache CouchDB 3.0.0.
> >>>
> >>> Please cast your votes now.
> >>>
> >>> Thanks,
> >>> Joan "once in a lifetime" Touzet
> >>
>
>
>
>
>

Re: [VOTE] Release Apache CouchDB 3.0.0 (-rc1)

[Will Holley]

also verified make check passes on CentOS 8 with SM60 using the standard
mozjs60/mozjs60-devel packages. Tested amd64 and ppc64le.

I also ran some basic clouseau tests which ran fine.

On Thu, 13 Feb 2020 at 10:47, Will Holley  wrote:

> +1 make check passing on Centos 7, tested both amd64 and ppc64le. I'm
> going to try on Centos 8 next...
>
> One thing I noticed is that the Unix install instructions look outdated
> with respect to Erlang / Node / Python dependencies though I'm not 100%
> sure what the supported options are now.
>
>
>
> On Thu, 13 Feb 2020 at 09:27, Peng Hui Jiang  wrote:
>
>> +1 from "Debian 10 Buster on ppc64le" 's point of view.
>>
>> The test was performed against Debian 10 Buster docker image on Ubuntu
>> ppc64le box.
>>
>> Eunit test can pass but just saw that t_stats_retained_on_job_removal
>> might randomly fail due to timeout. Other test including `make javascript`
>> and `make elixir` passed fine.
>>
>> jenkins@15706e15ee21:/tmp/couchdb/dist/apache-couchdb-3.0.0$ uname -a
>> Linux 15706e15ee21 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:14:44
>> UTC 2018 ppc64le GNU/Linux
>> jenkins@15706e15ee21:/tmp/couchdb/dist/apache-couchdb-3.0.0$ lsb_release
>> -sirc
>> Debian
>> 10
>> buster
>>
>> ==
>> Peng Hui Jiang
>> Beijing
>>
>> [image: Inactive hide details for Joan Touzet ---2020/02/13 01:35:28
>> PM---The Windows installer improvements are now live at the same U]Joan
>> Touzet ---2020/02/13 01:35:28 PM---The Windows installer improvements are
>> now live at the same URL below, the installer is called apac
>>
>> From: Joan Touzet 
>> To: dev@couchdb.apache.org
>> Date: 2020/02/13 01:35 PM
>> Subject: [EXTERNAL] Re: [VOTE] Release Apache CouchDB 3.0.0 (-rc1)
>> --
>>
>>
>>
>> The Windows installer improvements are now live at the same URL below,
>> the installer is called apache-couchdb-3.0.0.1.msi.
>>
>> https://dist.apache.org/repos/dist/dev/couchdb/binary/win/3.0.0/rc.1/
>>
>> LOTS of improvements:
>>
>>   * Prompts for an admin user/password as CouchDB 3.0 requires
>> * Will not overwrite existing credentials if in place
>>   * No longer remove user-modified config files, closing
>> apache/couchdb#1989
>> * Also will not overwrite them on install, naturally :)
>>   * Checkbox to disable installation of the Windows service
>>   * Silent install support for all the above
>>   * Friendly link to the online release notes in the exit dialog
>>   * Higher resolution icon for HiDPI (500x500)
>>
>> The details are here:
>>
>>   https://github.com/apache/couchdb-glazier/pull/12
>>
>> The build runs fine on Windows, obviously, and `make check` passes
>> entirely, so this is my official +1 on the RC.
>>
>> -Joan "If you can see what I can see/When I'm cleanin' windows" Touzet
>>
>>
>> On 2020-02-10 14:36, I wrote:
>> > Here are the Windows binaries:
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/couchdb/binary/win/3.0.0/rc.1/
>> >
>> > NOTE: You have to use Notepad (or similar) as Administrator and edit
>> > your local.ini file before the service will start correctly.
>> >
>> > I've got a few minor things planned for the Windows installer to
>> improve
>> > on it, including prompting for an admin user on install, but this is
>> > functional for now.
>> >
>> > Linux binaries of various flavours are being built right now.
>> >
>> > -Joan
>> >
>> > On 2020-02-10 13:32, Jan Lehnardt wrote:
>> >> Hey all,
>> >>
>> >> Mac convenience binaries for your testing pleasure are available here:
>> >>
>> >>
>> >> https://dist.apache.org/repos/dist/dev/couchdb/binary/mac/3.0.0/rc.1/
>> >>
>> >> Best
>> >> Jan
>> >> —
>> >>
>> >>> On 10. Feb 2020, at 17:33, Joan Touzet  wrote:
>> >>>
>> >>> Dear community,
>> >>>
>> >>> I would like to propose that we release Apache CouchDB 3.0.0.
>> >>>
>> >>> Candidate release notes:
>> >>>
>> >>> https://docs.couchdb.org/en/latest/whatsnew/3.0.html
>> >>>
>> >>> We encourage the whole community to download and test these release
>> >>> artefacts so that any critical issues can be resolved before the
>> >>> release is made. Everyone is free to vote on this release, so dig
>> >>> right in! (Only PMC members have binding votes, but they depend on
>> >>> community feedback to gauge if an official release is ready to be
>> made.)
>> >>>
>> >>> The release artefacts we are voting on are available here:
>> >>>
>> >>> https://dist.apache.org/repos/dist/dev/couchdb/source/3.0.0/rc.1
>> >>>
>> >>> There, you will find a tarball, a GPG signature, and SHA256/SHA512
>> >>> checksums.
>> >>>
>> >>> Please follow the test procedure here:
>> >>>
>> >>>
>> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
>>
>> >>>
>> >>>
>> >>> Please remember that "RC1" is an annotation. If the vote passes,
>> >>> these artefacts will be released as Apache CouchDB 3.0.0.
>> >>>
>> >>> Please cast your votes now.
>> >>>
>> >>> Thanks,
>> >>> Joan "once in a lifetime" Touzet
>> >>
>>
>>
>>
>>
>>

Re: Removal of ibmcom/couchdb3 preview image

[Will Holley]

Thanks Joan for highlighting the problem; the image has been removed from
DockerHub.

On Mon, 3 Feb 2020 at 17:46, Joan Touzet  wrote:

> On 2020-02-03 12:34, Jan Lehnardt wrote:
> > Thanks Joan for raising this,
> >
> > I’m throwing in an extra “this is bad” because of the version number.
> 3.0 is not a thing yet and we have brought that up with several IBMers that
> calling anything intermediate “CouchDB3” (this is isn’t the first time) is
> problematic in the CouchDB Slack.
> >
> > Y’all need to do better. Is suggest the removal of this image blocks the
> 3.0.0 release, to make sure nobody gets the wrong bits.
>
>
> I agree - I was going to fork 3.x and 3.0.x today but I'll hold off
> until this gets resolved. We still have 2 documentation issues that need
> to be resolved, and verification of the Windows build, before we can
> release anyway.
>
> -Joan
>
> >
> > Best
> > Jan
> > —
> >
> >> On 3. Feb 2020, at 18:08, Joan Touzet  wrote:
> >>
> >> Hi IBM people,
> >>
> >>   https://hub.docker.com/r/ibmcom/couchdb3
> >>
> >> This is a problem, especially because i'm seeing 50K+ pulls on this
> image already.
> >>
> >> We've not yet released CouchDB 3.0. Any use of this image outside of
> our immediate developer community is a direct violation of Apache release
> protocols, and could result in serious problems down the road.
> >>
> >> If you need a dev image for CouchDB to build Fauxton, that's one thing
> - but that image needs to be built each time, or hosted on a *private*
> repository with credentials.
> >>
> >> Please take this image offline immediately.
> >>
> >> -Joan "not messing around" Touzet
> >
>

Re: [VOTE] Release Apache CouchDB 3.0.0 (-rc1)

[Will Holley]

Some of our internal teams reported that the cluster seedlist function is
broken in RC1. I was able to reproduce and filed
https://github.com/apache/couchdb/issues/2559 to track.

On Sat, 15 Feb 2020 at 01:23, Joan Touzet  wrote:

> On 2020-02-14 5:49 p.m., Dave Cottlehuber wrote:
> > On Fri, 14 Feb 2020, at 20:13, Dave Cottlehuber wrote:
> >>> Please cast your votes now.
> >>
> >> +1
> >>
> >> - FreeBSD 13.0-CURRENT amd64
> >> - SM185 & SM60
> >> - OTP 22.2.6
> >
> > also a very excited double-plus-one for ... drumroll ...
> >
> > FreeBSD 13.0-CURRENT aarch64 / armv8 < aw
> > - SM60 with very latest patches 60.9.0_1
> > - OTP 21.3.8.11
> >
> > Finished in 58.4 seconds
> > 301 tests, 0 failures, 24 excluded
> >
> > Randomized with seed 566866
> > gmake[1]: Leaving directory '/tmp/apache-couchdb-3.0.0'
> > root@a01:/tmp/apache-couchdb-3.0.0 # uname -a
> > FreeBSD a01 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r357847: Thu Feb 13
> 04:15:31 UTC 2020 
> r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC
> arm64
> > root@a01:/tmp/apache-couchdb-3.0.0 #
> >
> > I've left both of these running in a loop, but so far I've had over an
> hour without any issue. Very very excited.
>
> Thanks Dave, that confirms my suspicions, that the packages in CentOS
> and Debian are just a bit too far behind the tip of 60esr. If you can
> isolate the problem to a very specific patch, we can both push that
> patch to the Linux distros as well as update our release notes / build
> guidance. With new distro packages we could even rebuild our releases to
> use those builds. That is, if you have the time ;)
>
> -Joan
>

Re: CI for prototype/fdb-layer branch

[Will Holley]

For running in a container, you could also try using the Red Hat UBI
instead of CentOS. There is a ubi-init variant which runs systemd (see
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/building_running_and_managing_containers/index#using_init_red_hat_base_images
and https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image
).

On Thu, 20 Feb 2020 at 10:30, Ronny Berndt  wrote:

> @Alex
>
> I have a running centos8 vm. Maybe i can help…
>
>
> > Am 20.02.2020 um 11:13 schrieb Alex Miller  >:
> >
> >
> >> On Feb 19, 2020, at 20:09, Joan Touzet  wrote:
> >> On 2020-02-19 23:00, Alex Miller wrote:
>  On Feb 19, 2020, at 16:07, Paul Davis 
> wrote:
> 
>  foundationdb does take a while to build though, so finding binaries
>  might short circuit everything to be even a single apt-get line or
>  w/e.
> >>> The build is both slow and quite memory hungry.
> >>> In addition to FROM + COPY in docker, foundationdb.org hosts
> downloads in both a tarball-of-binaries form and a .deb of the server.
>  Though that papers over CentOS support and the like. Dunno what that
>  story is like.
> >>> RPMs for RHEL6 and RHEL7 are also published (which I think should
> correlate to centos6 and centos7).
> >>
> >> Are there plans for a CentOS 8 RPM? CentOS 8 has been out since
> September 2019, and is the only CentOS that we support with SpiderMonkey 60
> today.
> >
> > I don't think anyone in FDB realized Centos 8 is out, so that's a good
> question.
> >
> > After digging through packaging code, the only difference between the EL6
> > and EL7 RPMs is that EL6 installs a sysv init script, and el7 installs a
> systemd
> > unit file.  The binaries in both cases are built on centos6 and the
> build system
> > jumps through all the hoops of statically linking a C++ binary, so that
> > fdbserver will run on anything centos6 or newer just fine.  This should
> > mean that EL7 RPMs are for EL7+, or at least, until Centos changes init
> > systems again.
> >
> > But, that's just theory, and doing a quick install on a centos8 VM
> sounded like
> > it'd be qick and simple...
> >
> > Except parallels doesn't support centos8 out of the box yet, and I broke
> a
> > centos7 install trying to do an (unsupported) upgrade to centos8.  So
> that's
> > out.
> >
> > Docker should save the day here, but it turns out that running systemd
> in a
> > docker container is nontrivial.  Even when I did get systemd running as
> PID 1,
> > FoundationDB didn't start automatically for me, and systemctl doesn't
> work,
> > because centos:8 gives you a half baked systemd install that somehow
> lacks dbus.
> >
> > So I'm out of easy options.  fdbserver still runs manually just fine,
> and all
> > the files _look_ like they got installed in the right place.  So if
> someone has
> > an actual running VM of Centos 8, it _seems_ like things should still
> start fine
> > when installing the EL7 RPM.
> >
> > This exercise did point out that centos8 intentionally doesn't provide a
> /usr/bin/python,
> > which FDB's RPM packages accidentally depend on, so I've posted
> > https://github.com/apple/foundationdb/pull/2700 to get rid of that.  One
> > will have to use `rpm -i —force foundationdb-server*.rpm` until the
> > next 6.2 release.
>
>

Re: [DISCUSS] Mango indexes on FDB

[Will Holley]

Broadly, I think it's a big step forward if we can prevent Mango from
automatically selecting extremely stale indexes.

I've been going back and forth on whether step 3 could lead to some
difficult-to-predict behaviour. If we assume that requests have a short
timeout - e.g. we can't return any result if it doesn't complete within the
FDB transaction timeout - then I think it's fine: queries that use
_all_docs and a large database will be timing out anyway.

If we were to allow long-running queries then it seems a bit sketchier
because adding an index to a large database could cause queries that
previously completed to start timing out whilst they block on the index
build. This is basically how Mango in CouchDB 2/3 behaves and has been a
big pain point for customers I've worked with, to the point where you
basically need to explicitly specify which index Mango uses in all cases if
you're to avoid surprise timeouts when somebody adds a new index.

As I understand it, we're not allowing queries to span FDB transactions so
this latter case is not something to worry about?

Cheers,

Will

On Wed, 25 Mar 2020 at 19:43, Garren Smith  wrote:

> On Wed, Mar 25, 2020 at 8:35 PM Paul Davis 
> wrote:
>
> > > It was therefore felt that having an immediate "Not ready" signal for
> > just _some_ calls to _find, based on the type of backing index, was a bad
> > and confusing api.
> > >
> > > We also discussed _find calls where the user does not specify an index,
> > and concluded that we would be free to choose between using the _all_docs
> > index (which is always up to date but rarely the best index for a given
> > selector) or blocking to update a better but stale index.
> > >
> > > Summary-ing my summarisation;
> > >
> > > 1) if you specify an index, we'll use it even if we have to update it,
> > no matter how long that takes.
> > > 2) if you don't specify an index, it's the dealers choice. The details
> > here may change in point releases.
> > >
> >
> > So it seems there's still a bit of confusion on what the consensus is
> > here. The way that I had thought this would work is that we'd do
> > something like such:
> >
> > 1. If user specifies and index, use it even if we have to wait
> > 2. If an index is built that can be used, use it
> > 3. If an index is building that can be used, wait for it
> > 4. As a last resort use _all_docs
> >
> > Discussing with Garren on the PR he's of the opinion that we should
> > skip step 3 and just go directly to using _all_docs if nothing is
> > built.
> >
>
> I just want to clarify step 3. I'm ok with using an index that still needs
> to be built as long as there is no other built index
> that can service the request.
>
> So the big thing for me is to always prefer a built index over a building
> index. In the situation where there is only 1 building index versus all
> docs I'm ok with using the building index.
>
>
>
>
> > My main assumption is that most cases where a user is creating an
> > index and then wanting to run a query with it are in the
> > design/exploration phase of learning the feature or designing an index
> > to use. In that scenario if we skip waiting it seems likely that a
> > user could easily be led to believe that an index creation "worked"
> > for their selector when in reality it was just backed by _all_docs.
> >
> > The other reason for preferring to wait for an index to finish
> > building is that the UI for the normal case of creating indexes is a
> > bit awkward. Having to run a polling loop around checking the index
> > status seems suboptimal in most cases.
> >
> > Am I missing other cases that would benefit from not waiting and just
> > using _all_docs?
> >
> > Paul
> >
>

Re: [DISCUSS] Mango indexes on FDB

[Will Holley]

Ah - in that case I think we should remove step 3, as it leads to a
confusing mental model. It's much simpler to explain that Mango will only
use fresh indexes and any new indexes will build in the background.

On Thu, 26 Mar 2020 at 10:15, Garren Smith  wrote:

> On Thu, Mar 26, 2020 at 11:04 AM Will Holley  wrote:
>
> > Broadly, I think it's a big step forward if we can prevent Mango from
> > automatically selecting extremely stale indexes.
> >
> > I've been going back and forth on whether step 3 could lead to some
> > difficult-to-predict behaviour. If we assume that requests have a short
> > timeout - e.g. we can't return any result if it doesn't complete within
> the
> > FDB transaction timeout - then I think it's fine: queries that use
> > _all_docs and a large database will be timing out anyway.
> >
> > If we were to allow long-running queries then it seems a bit sketchier
> > because adding an index to a large database could cause queries that
> > previously completed to start timing out whilst they block on the index
> > build. This is basically how Mango in CouchDB 2/3 behaves and has been a
> > big pain point for customers I've worked with, to the point where you
> > basically need to explicitly specify which index Mango uses in all cases
> if
> > you're to avoid surprise timeouts when somebody adds a new index.
> >
> > As I understand it, we're not allowing queries to span FDB transactions
> so
> > this latter case is not something to worry about?
>
>
> We are going to allow queries to span transactions. This is already
> implemented for views and will be for mango
>
>
> >
> > Cheers,
> >
> > Will
> >
> > On Wed, 25 Mar 2020 at 19:43, Garren Smith  wrote:
> >
> > > On Wed, Mar 25, 2020 at 8:35 PM Paul Davis <
> paul.joseph.da...@gmail.com>
> > > wrote:
> > >
> > > > > It was therefore felt that having an immediate "Not ready" signal
> for
> > > > just _some_ calls to _find, based on the type of backing index, was a
> > bad
> > > > and confusing api.
> > > > >
> > > > > We also discussed _find calls where the user does not specify an
> > index,
> > > > and concluded that we would be free to choose between using the
> > _all_docs
> > > > index (which is always up to date but rarely the best index for a
> given
> > > > selector) or blocking to update a better but stale index.
> > > > >
> > > > > Summary-ing my summarisation;
> > > > >
> > > > > 1) if you specify an index, we'll use it even if we have to update
> > it,
> > > > no matter how long that takes.
> > > > > 2) if you don't specify an index, it's the dealers choice. The
> > details
> > > > here may change in point releases.
> > > > >
> > > >
> > > > So it seems there's still a bit of confusion on what the consensus is
> > > > here. The way that I had thought this would work is that we'd do
> > > > something like such:
> > > >
> > > > 1. If user specifies and index, use it even if we have to wait
> > > > 2. If an index is built that can be used, use it
> > > > 3. If an index is building that can be used, wait for it
> > > > 4. As a last resort use _all_docs
> > > >
> > > > Discussing with Garren on the PR he's of the opinion that we should
> > > > skip step 3 and just go directly to using _all_docs if nothing is
> > > > built.
> > > >
> > >
> > > I just want to clarify step 3. I'm ok with using an index that still
> > needs
> > > to be built as long as there is no other built index
> > > that can service the request.
> > >
> > > So the big thing for me is to always prefer a built index over a
> building
> > > index. In the situation where there is only 1 building index versus all
> > > docs I'm ok with using the building index.
> > >
> > >
> > >
> > >
> > > > My main assumption is that most cases where a user is creating an
> > > > index and then wanting to run a query with it are in the
> > > > design/exploration phase of learning the feature or designing an
> index
> > > > to use. In that scenario if we skip waiting it seems likely that a
> > > > user could easily be led to believe that an index creation "worked"
> > > > for their selector when in reality it was just backed by _all_docs.
> > > >
> > > > The other reason for preferring to wait for an index to finish
> > > > building is that the UI for the normal case of creating indexes is a
> > > > bit awkward. Having to run a polling loop around checking the index
> > > > status seems suboptimal in most cases.
> > > >
> > > > Am I missing other cases that would benefit from not waiting and just
> > > > using _all_docs?
> > > >
> > > > Paul
> > > >
> > >
> >
>

Re: [VOTE] Release Apache CouchDB 3.1.1 (RC2)

[Will Holley]

Environment:
  RHEL 8
  Elixir: 1.9.1
  Erlang: 20.3.8.25

CPU Architectures: amd64, ppc64le, s390x

Sig: ok
Checksums: ok
Configure, make & make check: ok
Build release, add admin & start: ok
Used Fauxton to:
  - configure cluster: ok
  - verify install: ok
  - create dbs: ok
  - create docs: ok
  - create replications between dbs on same cluster: ok
  - links to docs: ok

+1

On Tue, 15 Sep 2020 at 05:20, Nick Vatamaniuc  wrote:

> Environment:
>   Ubuntu 18.04.5, x86_64
>   $ asdf current
> elixir 1.9.4-otp-22
> erlang 22.2.3
>  $ cat /etc/apt/sources.list.d/couchdb-bintray.list
> deb https://apache.bintray.com/couchdb-deb bionic main
>
> Sig: ok
> Checksums: ok
> Configure, make & make check: ok
> Build release, add admin & start: ok
> Used Fauxton to:
>   - configure cluster: ok
>   - verify install: ok
>   - create dbs: ok
>   - create docs: ok
>   - create replications between dbs on same cluster: ok
>   - links to docs: ok
>
> +1
>
> Thank you for creating the release, Joan!
>
> -Nick
>
> On Mon, Sep 14, 2020 at 10:15 PM Joan Touzet  wrote:
> >
> > Hi everyone,
> >
> > There have been no votes on this release. Are people available to try it
> > out? Tomorrow, I will be able to complete the binary builds and submit
> > my own vote.
> >
> > Remember, 3.1.1 will not be cut unless 3 +1 votes, minimum, arrive from
> > committers, with no blockers by convention. The Mac situation needs to
> > be investigated. It's not clear to me that there is consensus on
> > blocking the release on the new request header toggle, but if there is,
> > I could cut a 3.1.1-RC3 _without_ that change tomorrow if consensus
> > materialises overnight.
> >
> > Note that I will be on holiday starting end of the week for 2.5 weeks.
> > If 3.1.1 doesn't cut from this RC, for whatever reason, the next
> > opportunity I will have to turn the crank will be 5 October 2020.
> >
> > -Joan
> >
> > On 2020-09-11 6:53 p.m., Joan Touzet wrote:
> > > Dear community,
> > >
> > > I would like to propose that we release Apache CouchDB 3.1.1.
> > >
> > > Changes since the last round:
> > >
> > >  https://github.com/apache/couchdb/compare/3.1.1-RC1...3.1.1-RC2
> > >
> > > Candidate release notes:
> > >
> > >  https://docs.couchdb.org/en/latest/whatsnew/3.1.html
> > >
> > > We encourage the whole community to download and test these release
> > > artefacts so that any critical issues can be resolved before the
> release
> > > is made. Everyone is free to vote on this release, so dig right in!
> > > (Only PMC members have binding votes, but they depend on community
> > > feedback to gauge if an official release is ready to be made.)
> > >
> > > The release artefacts we are voting on are available here:
> > >
> > >  https://dist.apache.org/repos/dist/dev/couchdb/source/3.1.1/rc.2/
> > >
> > > There, you will find a tarball, a GPG signature, and SHA256/SHA512
> > > checksums.
> > >
> > > Please follow the test procedure here:
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
> > >
> > >
> > > Please remember that "RC2" is an annotation. If the vote passes, these
> > > artefacts will be released as Apache CouchDB 3.1.1.
> > >
> > > Because of the weekend, this vote will remain open until 5PM ET
> (UTC-4),
> > > Tuesday, 15 September 2020.
> > >
> > > Please cast your votes now.
> > >
> > > Thanks,
> > > Joan "once more unto the breech, dear friends" Touzet
>

Re: [ANNOUNCE] Glynn Bird joins the PMC

[Will Holley]

Congrats Glynn!

On Mon, 14 Sep 2020 at 17:29, Joshua Mintz  wrote:

> Congrats, Glynn!
>
> On Mon, Sep 14, 2020 at 12:23 PM Jan Lehnardt  wrote:
>
> > Congratulations Glynn! :)
> >
> > > On 14. Sep 2020, at 18:22, Michelle P  wrote:
> > >
> > >
> > > Dear community,
> > >
> > > I am delighted to announce that Glynn Bird joins the Apache CouchDB
> > Project Management Committee today.
> > >
> > > Glynn has made outstanding, sustained contributions to the project.
> This
> > appointment is an official acknowledgement of their position within the
> > community, and our trust in their ability to provide oversight for the
> > project.
> > >
> > > Everybody, please join me in congratulating Glynn!
> > >
> > > On behalf of the CouchDB PMC,
> > > Michelle
> > >
> >
> >
>

Re: [DISCUSS] Prometheus endpoint in CouchDB 4.x

[Will Holley]

Thanks Peng Hui, Garren. I can definitely see value in having metrics
exposed in Prometheus format.

One aspect I'm unclear about is what _active_tasks metrics would represent.
My expectation is that /_metrics would be scoped to a node (and then leave
the aggregation across nodes to Prometheus or similar), whereas
_active_tasks is scoped to a cluster. What are your thoughts?

On Wed, 23 Sep 2020 at 10:18, jiangph  wrote:

> Thanks Joan for your quick response and suggestion.  As we can see, JSON
> is not the standard Prometheus format for _metrics endpoint. In order to be
> compatible with many monitoring tools, what about going with Prometheus
> standard format first and we may add JSON format support later?
>
> Best regards,
> Peng Hui
>
> > On Sep 23, 2020, at 9:41 AM, Joan Touzet  wrote:
> >
> > I like this, but not at the expense of JSON output. It would be the only
> new API surface for CouchDB that isn't JSON-based, and there needs to be
> excellent justification for such. Prometheus is well-known enough to be
> supported, but we should continue to put out JSON stats for the foreseeable
> future.
> >
> > I know that Prometheus can't send a header, but sending an accepts
> application/json to /_metrics and having it send back the same data as
> Prometheus, but in JSON, would be lovely. If you feel up to it :)
> >
> > -Joan "on vacation" Touzet
> >
> > On 2020-09-22 8:55 a.m., jiangph wrote:
> >> Hey all,
> >> We would like to add a Prometheus metrics endpoint for CouchDB and
> wanted to see if the community would be interested in us contributing this
> to CouchDB 4.x.
> >> Prometheus is a CNCF open-source project and the Prometheus metrics
> endpoint format is supported by many monitoring tools. Its data model is
> based around having a metric name which then contains a label name and a
> label value:
> >> {=, ...}
> >> And it supports the Counter, Gauge, Histogram, and Summary metric types.
> >> The idea for the new Prometheus endpoint, /_metrics, would be that the
> endpoint is a consolidation of the _stats [1],  _system [2], and
> _active_tasks [3] endpoints.
> >> For _stats and _system, the conversion from JSON to Prometheus-based
> format seems to be straightforward.
> >> JSON format:
> >> {
> >>  "value": {
> >>   "min": 0,
> >>   "max": 0,
> >>   "arithmetic_mean": 0,
> >>   "geometric_mean": 0,
> >>   "harmonic_mean": 0,
> >>   "median": 0,
> >>   "variance": 0,
> >>   "standard_deviation": 0,
> >> ...
> >> "percentile": [
> >>[
> >> 50,
> >> 0
> >>],
> >>[
> >> 75,
> >> 0
> >>],
> >>[
> >> 90,
> >> 0
> >>],
> >>[
> >> 95,
> >> 0
> >>],
> >>[
> >> 99,
> >> 0
> >>],
> >>[
> >> 999,
> >> 0
> >>]
> >>   ],
> >>   "histogram": [
> >>[
> >> 0,
> >> 0
> >>]
> >>   ],
> >> }
> >> Prometheus-based format:
> >> couchdb_stats{value="min"} 0
> >> couchdb_stats{value="max"} 0
> >> couchdb_stats{value="percentile50"} 0
> >> couchdb_stats{value="percentile75"} 0
> >> couchdb_stats{value="percentile95"} 0
> >> For _active_tasks, the change will be a bit more complicated, and some
> fields will be added to labels and tags.
> >> JSON format:
> >> {
> >> "checkpointed_source_seq": 68585,
> >> "continuous": false,
> >> "doc_id": null,
> >> "doc_write_failures": 0,
> >> "docs_read": 4524,
> >> "docs_written": 4524,
> >> "missing_revisions_found": 4524,
> >> "pid": "<0.1538.5>",
> >> "progress": 44,
> >> "replication_id": "9bc1727d74d49d9e157e260bb8bbd1d5",
> >> "revisions_checked": 4524,
> >> "source": "mailbox",
> >> "source_seq": 154419,
> >> "started_on": 1376116644,
> >> "target": "http://mailsrv:5984/mailbox <
> http://mailsrv:5984/mailbox>  http://mailsrv:5984/mailbox>>",
> >> "type": "replication",
> >> "updated_on": 1376116651
> >> }
> >> Prometheus-based would look something like:
> >> format:couchdb_active_task{type="replication", source="mailbox",
> target="http://mailsrv:5984/mailbox  <
> http://mailsrv:5984/mailbox >", docs_count =
> "docs_read"} 4524
> >> couchdb_active_task{type="replication", source="mailbox", target="
> http://mailsrv:5984/mailbox  <
> http://mailsrv:5984/mailbox >", docs_count =
> "docs_written"} 4524
> >> couchdb_active_task{type="replication", source="mailbox", target="
> http://mailsrv:5984/mailbox  <
> http://mailsrv:5984/mailbox >", docs_count =
> "missing_revisions_found"} 4524
> >> Best regards,
> >> Garren Smith
> >> Peng Hui Jiang
> >> [1]
> https://docs.couchdb.org/en/latest/api/server/common.html#node-node-name-stats
> <
> https://docs.couchdb.org/en/latest/api/server/common.html#node-node-name-stats>
> <
> 

Re: [DISCUSS] Prometheus endpoint in CouchDB 4.x

[Will Holley]

It's a good point about querying every node for cluster-wide data such as
_active_tasks (assuming a direct translation of the current endpoint),
though I think it's better to lean on the monitoring tool to aggregate data
across nodes rather than deduplicate. Possibly the simplest thing for a v1
would be to only expose the existing node-level metrics (_stats, _system)?

Having also written an OpenMetrics exporter for CouchDB, one problem I
found is that the existing stats/system responses don't map cleanly to
OpenMetrics idioms. Some examples:

 * OpenMetrics uses a flat naming structure whereas _stats uses a
hierarchical structure, and automatically flattening the names using a
convention leads to nonsensical / duplicated names.
 * Some groupings of counters in _stats (e.g. the httpd_requests_methods
group) should be represented as a single counter with multiple label values
(method="PUT|POST|GET|DELETE|COPY|HEAD"). For others (e.g. the couchdb
group containing auth_cache_hits, database_reads, etc), each field
represents a different counter.
 * Histograms can't be translated to OpenMetrics format purely from the
stats response - you need to understand the CouchDB config/what the
boundaries of the histogram buckets are.

Even if it ends up being a plugin/sidecar, I think it would be useful to
have a reference implementation that could be used to inform these
decisions and make it simpler to share assets which build on the
OpenMetrics output (e.g. Grafana dashboards, PromQL alert definitions).




On Wed, 23 Sep 2020 at 22:10, Tobias Gesellchen  wrote:

> Hi,
>
> chiming in as maintainer of the already mentioned
> https://github.com/gesellix/couchdb-prometheus-exporter <
> https://github.com/gesellix/couchdb-prometheus-exporter>.
>
> My impression would be to consolidate the existing endpoints first (maybe
> at /_metrics, because /_info sounds too informal), which would make high
> frequent scrapes more efficient. The current approach of the
> CouchDB-Prometheus-Exporter doesn’t feel right, because every node * every
> stats/system/active_tasks endpoint needs to be queried on each scrape. That
> endpoint should certainly be able to provide JSON format by default, which
> would already help a lot to improve the existing Prometheus exporter.
>
> Content negotiation via Accept header would be nice to respond with the
> Prometheus specific format. I wouldn’t prefer the workaround with the
> request parameter, though. Without too much knowledge about CouchDB
> internals: I’d suggest yet another endpoint /_prometheus which would
> provide text/plain (prometheus formatted) content by default. That endpoint
> could internally “delegate” to the /_metrics endpoint.
>
> While I might be biased, and knowing that other frameworks/tools already
> provide Prometheus stats out of the box, I personally tend to keep things
> separated. From an operational perspective it would be great to _not_ have
> to co-locate CouchDB with a sidecar-exporter, but on the contrary it would
> also be great if I could perform upgrades or configuration separately.
>
> Best
> Tobias
>
>
>
> > On 23. Sep 2020, at 22:43, Robert Samuel Newson 
> wrote:
> >
> > Hi,
> >
> > I don't see why this can't be a new endpoint (emitting the normal
> Prometheus format) that couchdb administrators can choose to enable (and
> leave it disabled by default, returning a 404).
> >
> > I agree with the general view that content type negotiation doesn't
> really work well in practice, and I don't much like the suggested ?accept=
> hack.
> >
> > I am old and world-weary and have seen these sorts of things come and go
> many times. Prometheus seems a fine option for now, and perhaps for a
> while, but it feels like a plugin, not core, to me.
> >
> > B.
> >
> >> On 23 Sep 2020, at 17:25, Richard Ellis  wrote:
> >>
> >>> so we should absolutely make this info available in JSON
> >>
> >> This sounds like a good idea to me
> >>
> >>> we could fall back to a ?accept=prometheus option
> >>
> >> I'm opposed to adding endpoints that supply different content-type
> >> responses via non-standard means. The CouchDB API has some examples of
> >> this through history and it can make using those endpoints with
> standard
> >> tooling somewhat painful.
> >>
> >> A bit of quick searching seems to suggest that the format has its own
> >> project https://openmetrics.io/ - and this declares it's text
> >> representation linking back to
> >>
> https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format
> >> which declares a Content-Type of "text/plain; version=0.0.4" - so
> >> defaulting to that, but following Joan's suggestion and switching to
> JSON
> >> for a supplied Accept:application/json in the standard way seems a like
> >> good choice to me.
> >>
> >> Rich
> >>
> >>
> >>
> >> From:   Jan Lehnardt 
> >> To: dev@couchdb.apache.org
> >> Cc: "Gesellchen, Tobias" 
> >> Date:   23/09/2020 16:42
> >> Subject:[EXTERNAL] Re: [DISCUSS] Prometheus endpoint 

[jira] [Commented] (COUCHDB-1935) Fauxton should show API URLs

[Will Holley (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825160#comment-13825160
 ] 

Will Holley commented on COUCHDB-1935:
--

There is already a collapsable API URL section at the top of the interface. 
Can you elaborate on what is needed beyond this or, if you hadn't found it, how 
the usability might be improved?

 Fauxton should show API URLs
 

 Key: COUCHDB-1935
 URL: https://issues.apache.org/jira/browse/COUCHDB-1935
 Project: CouchDB
  Issue Type: Bug
  Components: Fauxton
Reporter: Noah Slater

 When playing around with Fauxton, it would be super handy to grab the API 
 URLs that Fauxton is using to request data from CouchDB. I could then plug 
 them into an app I am working on. This turns Fauxton into a bit of an 
 instructional API URL builder for beginners.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (COUCHDB-2188) Pagination is broken if map fun emits more rows than page size is

[Will Holley (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13922671#comment-13922671
 ] 

Will Holley commented on COUCHDB-2188:
--

This sounds like it is not fixable using the pagination algorithm that uses 
startkey / startkey_docid to iterate through the result set. I can only think 
that in this case we'd have to use skip as a fallback in the case where the 
returned results are exactly the same as the current page?

 Pagination is broken if map fun emits more rows than page size is
 -

 Key: COUCHDB-2188
 URL: https://issues.apache.org/jira/browse/COUCHDB-2188
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: Fauxton
Reporter: Alexander Shorin

 1. Create new database with single doc - his content doesn't matters
 2. Create the view that emits multiple rows for single doc, like:
 {code}
 function(doc){
   for(var i=0; i  20; i++){
 emit(doc._id, i);
   }
 }
 {code}
 Yes, too synthetic, but you got the idea. 
 3. Try to navigate thought results with page size 5 and 10 - you'll always 
 browse the same content
 Fauxton @ 
 [4e60f0b|https://git-wip-us.apache.org/repos/asf?p=couchdb.git;a=commit;h=4e60f0b]



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (COUCHDB-2249) Fauxton generates incorrect parameters when complex keys are specified

[Will Holley (JIRA)]

Will Holley created COUCHDB-2249:


 Summary: Fauxton generates incorrect parameters when complex keys 
are specified
 Key: COUCHDB-2249
 URL: https://issues.apache.org/jira/browse/COUCHDB-2249
 Project: CouchDB
  Issue Type: Bug
  Security Level: public (Regular issues)
  Components: Fauxton
Reporter: Will Holley


Fauxton is not stringifying user-defined query parameters correctly (added in 
the Query Options panel). This results in a query parameter being generated 
for each component of the complex key.

For example, specifying a startkey of {noformat}[a,b]{noformat} generates 
the URL 

{noformat}
?startkey[]=astartkey[]=b
{noformat}

instead of:

{noformat}
?startkey=[a,b]
{noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (COUCHDB-2249) Fauxton generates incorrect parameters when complex keys are specified

[Will Holley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Will Holley resolved COUCHDB-2249.
--

Resolution: Fixed

 Fauxton generates incorrect parameters when complex keys are specified
 --

 Key: COUCHDB-2249
 URL: https://issues.apache.org/jira/browse/COUCHDB-2249
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: Fauxton
Reporter: Will Holley

 Fauxton is not stringifying user-defined query parameters correctly (added in 
 the Query Options panel). This results in a query parameter being generated 
 for each component of the complex key.
 For example, specifying a startkey of {noformat}[a,b]{noformat} generates 
 the URL 
 {noformat}
 ?startkey[]=astartkey[]=b
 {noformat}
 instead of:
 {noformat}
 ?startkey=[a,b]
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (COUCHDB-2518) CouchDB 2.0 does not support conflicts=true on /_changes

[Will Holley (JIRA)]

Will Holley created COUCHDB-2518:


 Summary: CouchDB 2.0 does not support conflicts=true on /_changes
 Key: COUCHDB-2518
 URL: https://issues.apache.org/jira/browse/COUCHDB-2518
 Project: CouchDB
  Issue Type: Bug
  Security Level: public (Regular issues)
  Components: Database Core
Reporter: Will Holley


CouchDB 1.X supports the conflicts=true query parameter for all API endpints 
which accept the include_docs parameter (since version 1.0.3 according to the 
release notes). This appears to have regressed in CouchDB 2.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (COUCHDB-2518) CouchDB 2.0 does not support conflicts=true on /_changes

[Will Holley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Will Holley updated COUCHDB-2518:
-
Description: 
CouchDB 1.X supports the conflicts=true query parameter for all API endpints 
which accept the include_docs parameter (since version 1.0.3 according to the 
release notes). This appears to have regressed in CouchDB 2.0.

I've observed this in the PouchDB tests (which fail for this). Attempting to 
reproduce in curl:

$ curl 'http://127.0.0.1:15984/test' -XPUT
{ok:true}

$ curl 'http://127.0.0.1:15984/test/foo?new_edits=false' 
-H'Content-Type:application/json' -XPUT 
-d'{_id:foo,_rev:2-aa01552213fafa022e6167113ed01087,value:bar}'
{ok:true,id:foo,rev:2-aa01552213fafa022e6167113ed01087}

$ curl 'http://127.0.0.1:15984/test/foo?new_edits=false' 
-H'Content-Type:application/json' -XPUT 
-d'{_id:foo,_rev:3-aa01552213fafa022e6167113ed01087,value:baz}'
{ok:true,id:foo,rev:3-aa01552213fafa022e6167113ed01087}

$ curl 'http://127.0.0.1:15984/test/foo?conflicts=true' | jq .
{
  _id: foo,
  _rev: 3-aa01552213fafa022e6167113ed01087,
  value: baz,
  _conflicts: [
2-aa01552213fafa022e6167113ed01087
  ]
}

$curl 'http://127.0.0.1:15984/test/_changes?include_docs=trueconflicts=true' | 
jq .
{
  results: [
{
  seq: [
2,

g1GpeJzLYWBg4MhgTmHgz8tPSTV0MDQy1zMAQsMcoARTIkOS_P___7MygKxcoAC7kbGFSZKpOU4NSQpAMskeRU9acopJqrExbj0OID3xKHoMDI0MTM2NcOtJAOmpR9GTCrTIMNkQrscITU8eC5BkaABSQG3zszKYE5nA-kyTzFNNLc0xdcFNMsZq0gKISfsRLkg0M7I0ME0h4IIDEH33sxIZCKh8AFH5H6gyCwCZjmsY
  ],
  id: foo,
  changes: [
{
  rev: 3-aa01552213fafa022e6167113ed01087
}
  ],
  doc: {
_id: foo,
_rev: 3-aa01552213fafa022e6167113ed01087,
value: baz
  }
}
  ],
  last_seq: [
2,

g1GzeJyFz1EOgjAMBuAFTfTNI-gJzLpRB09yE6VshhDAI-hN9CZ6E73JLJKIGAlZ0jVtvvxpKYSY5xMrFvXROkhAmbXkByUvglTQ0ntf5NxVPJgpHYWEZhDQiitte-aQ2dBpPWySxux6RoKSaNSw2Tfm1DOOgyCDj1E_pp5yFWf-mF0aF7wdknEYdzfpv-7auluXl25ULNGO5N1b9-gcRia0EY24Z-u-7iNMyQEWL83XbI4
  ],
  pending: 0
}

  was:CouchDB 1.X supports the conflicts=true query parameter for all API 
endpints which accept the include_docs parameter (since version 1.0.3 according 
to the release notes). This appears to have regressed in CouchDB 2.0.


 CouchDB 2.0 does not support conflicts=true on /_changes
 

 Key: COUCHDB-2518
 URL: https://issues.apache.org/jira/browse/COUCHDB-2518
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: Database Core
Reporter: Will Holley

 CouchDB 1.X supports the conflicts=true query parameter for all API endpints 
 which accept the include_docs parameter (since version 1.0.3 according to the 
 release notes). This appears to have regressed in CouchDB 2.0.
 I've observed this in the PouchDB tests (which fail for this). Attempting to 
 reproduce in curl:
 $ curl 'http://127.0.0.1:15984/test' -XPUT
 {ok:true}
 $ curl 'http://127.0.0.1:15984/test/foo?new_edits=false' 
 -H'Content-Type:application/json' -XPUT 
 -d'{_id:foo,_rev:2-aa01552213fafa022e6167113ed01087,value:bar}'
 {ok:true,id:foo,rev:2-aa01552213fafa022e6167113ed01087}
 $ curl 'http://127.0.0.1:15984/test/foo?new_edits=false' 
 -H'Content-Type:application/json' -XPUT 
 -d'{_id:foo,_rev:3-aa01552213fafa022e6167113ed01087,value:baz}'
 {ok:true,id:foo,rev:3-aa01552213fafa022e6167113ed01087}
 $ curl 'http://127.0.0.1:15984/test/foo?conflicts=true' | jq .
 {
   _id: foo,
   _rev: 3-aa01552213fafa022e6167113ed01087,
   value: baz,
   _conflicts: [
 2-aa01552213fafa022e6167113ed01087
   ]
 }
 $curl 'http://127.0.0.1:15984/test/_changes?include_docs=trueconflicts=true' 
 | jq .
 {
   results: [
 {
   seq: [
 2,
 
 g1GpeJzLYWBg4MhgTmHgz8tPSTV0MDQy1zMAQsMcoARTIkOS_P___7MygKxcoAC7kbGFSZKpOU4NSQpAMskeRU9acopJqrExbj0OID3xKHoMDI0MTM2NcOtJAOmpR9GTCrTIMNkQrscITU8eC5BkaABSQG3zszKYE5nA-kyTzFNNLc0xdcFNMsZq0gKISfsRLkg0M7I0ME0h4IIDEH33sxIZCKh8AFH5H6gyCwCZjmsY
   ],
   id: foo,
   changes: [
 {
   rev: 3-aa01552213fafa022e6167113ed01087
 }
   ],
   doc: {
 _id: foo,
 _rev: 3-aa01552213fafa022e6167113ed01087,
 value: baz
   }
 }
   ],
   last_seq: [
 2,
 
 g1GzeJyFz1EOgjAMBuAFTfTNI-gJzLpRB09yE6VshhDAI-hN9CZ6E73JLJKIGAlZ0jVtvvxpKYSY5xMrFvXROkhAmbXkByUvglTQ0ntf5NxVPJgpHYWEZhDQiitte-aQ2dBpPWySxux6RoKSaNSw2Tfm1DOOgyCDj1E_pp5yFWf-mF0aF7wdknEYdzfpv-7auluXl25ULNGO5N1b9-gcRia0EY24Z-u-7iNMyQEWL83XbI4
   ],
   pending: 0
 }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (COUCHDB-2518) CouchDB 2.0 does not support conflicts=true on /_changes

[Will Holley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Will Holley updated COUCHDB-2518:
-
Description: 
CouchDB 1.X supports the conflicts=true query parameter for all API endpints 
which accept the include_docs parameter (since version 1.0.3 according to the 
release notes). This appears to have regressed in CouchDB 2.0.

I've observed this in the PouchDB tests (which fail for this). Attempting to 
reproduce in curl:

{code}
$ curl 'http://127.0.0.1:15984/test' -XPUT
{ok:true}

$ curl 'http://127.0.0.1:15984/test/foo?new_edits=false' 
-H'Content-Type:application/json' -XPUT 
-d'{_id:foo,_rev:2-aa01552213fafa022e6167113ed01087,value:bar}'
{ok:true,id:foo,rev:2-aa01552213fafa022e6167113ed01087}

$ curl 'http://127.0.0.1:15984/test/foo?new_edits=false' 
-H'Content-Type:application/json' -XPUT 
-d'{_id:foo,_rev:3-aa01552213fafa022e6167113ed01087,value:baz}'
{ok:true,id:foo,rev:3-aa01552213fafa022e6167113ed01087}

$ curl 'http://127.0.0.1:15984/test/foo?conflicts=true' | jq .
{
  _id: foo,
  _rev: 3-aa01552213fafa022e6167113ed01087,
  value: baz,
  _conflicts: [
2-aa01552213fafa022e6167113ed01087
  ]
}

$curl 'http://127.0.0.1:15984/test/_changes?include_docs=trueconflicts=true' | 
jq .
{
  results: [
{
  seq: [
2,

g1GpeJzLYWBg4MhgTmHgz8tPSTV0MDQy1zMAQsMcoARTIkOS_P___7MygKxcoAC7kbGFSZKpOU4NSQpAMskeRU9acopJqrExbj0OID3xKHoMDI0MTM2NcOtJAOmpR9GTCrTIMNkQrscITU8eC5BkaABSQG3zszKYE5nA-kyTzFNNLc0xdcFNMsZq0gKISfsRLkg0M7I0ME0h4IIDEH33sxIZCKh8AFH5H6gyCwCZjmsY
  ],
  id: foo,
  changes: [
{
  rev: 3-aa01552213fafa022e6167113ed01087
}
  ],
  doc: {
_id: foo,
_rev: 3-aa01552213fafa022e6167113ed01087,
value: baz
  }
}
  ],
  last_seq: [
2,

g1GzeJyFz1EOgjAMBuAFTfTNI-gJzLpRB09yE6VshhDAI-hN9CZ6E73JLJKIGAlZ0jVtvvxpKYSY5xMrFvXROkhAmbXkByUvglTQ0ntf5NxVPJgpHYWEZhDQiitte-aQ2dBpPWySxux6RoKSaNSw2Tfm1DOOgyCDj1E_pp5yFWf-mF0aF7wdknEYdzfpv-7auluXl25ULNGO5N1b9-gcRia0EY24Z-u-7iNMyQEWL83XbI4
  ],
  pending: 0
}
{code}

  was:
CouchDB 1.X supports the conflicts=true query parameter for all API endpints 
which accept the include_docs parameter (since version 1.0.3 according to the 
release notes). This appears to have regressed in CouchDB 2.0.

I've observed this in the PouchDB tests (which fail for this). Attempting to 
reproduce in curl:

$ curl 'http://127.0.0.1:15984/test' -XPUT
{ok:true}

$ curl 'http://127.0.0.1:15984/test/foo?new_edits=false' 
-H'Content-Type:application/json' -XPUT 
-d'{_id:foo,_rev:2-aa01552213fafa022e6167113ed01087,value:bar}'
{ok:true,id:foo,rev:2-aa01552213fafa022e6167113ed01087}

$ curl 'http://127.0.0.1:15984/test/foo?new_edits=false' 
-H'Content-Type:application/json' -XPUT 
-d'{_id:foo,_rev:3-aa01552213fafa022e6167113ed01087,value:baz}'
{ok:true,id:foo,rev:3-aa01552213fafa022e6167113ed01087}

$ curl 'http://127.0.0.1:15984/test/foo?conflicts=true' | jq .
{
  _id: foo,
  _rev: 3-aa01552213fafa022e6167113ed01087,
  value: baz,
  _conflicts: [
2-aa01552213fafa022e6167113ed01087
  ]
}

$curl 'http://127.0.0.1:15984/test/_changes?include_docs=trueconflicts=true' | 
jq .
{
  results: [
{
  seq: [
2,

g1GpeJzLYWBg4MhgTmHgz8tPSTV0MDQy1zMAQsMcoARTIkOS_P___7MygKxcoAC7kbGFSZKpOU4NSQpAMskeRU9acopJqrExbj0OID3xKHoMDI0MTM2NcOtJAOmpR9GTCrTIMNkQrscITU8eC5BkaABSQG3zszKYE5nA-kyTzFNNLc0xdcFNMsZq0gKISfsRLkg0M7I0ME0h4IIDEH33sxIZCKh8AFH5H6gyCwCZjmsY
  ],
  id: foo,
  changes: [
{
  rev: 3-aa01552213fafa022e6167113ed01087
}
  ],
  doc: {
_id: foo,
_rev: 3-aa01552213fafa022e6167113ed01087,
value: baz
  }
}
  ],
  last_seq: [
2,

g1GzeJyFz1EOgjAMBuAFTfTNI-gJzLpRB09yE6VshhDAI-hN9CZ6E73JLJKIGAlZ0jVtvvxpKYSY5xMrFvXROkhAmbXkByUvglTQ0ntf5NxVPJgpHYWEZhDQiitte-aQ2dBpPWySxux6RoKSaNSw2Tfm1DOOgyCDj1E_pp5yFWf-mF0aF7wdknEYdzfpv-7auluXl25ULNGO5N1b9-gcRia0EY24Z-u-7iNMyQEWL83XbI4
  ],
  pending: 0
}


 CouchDB 2.0 does not support conflicts=true on /_changes
 

 Key: COUCHDB-2518
 URL: https://issues.apache.org/jira/browse/COUCHDB-2518
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: Database Core
Reporter: Will Holley

 CouchDB 1.X supports the conflicts=true query parameter for all API endpints 
 which accept the include_docs parameter (since version 1.0.3 according to the 
 release notes). This appears to have regressed in CouchDB 2.0.
 I've observed this in the PouchDB tests (which fail for this). Attempting to 
 reproduce in curl:
 {code}
 $ curl 'http://127.0.0.1:15984/test' -XPUT
 {ok:true}
 $ curl 'http://127.0.0.1:15984/test/foo?new_edits=false' 
 -H'Content-Type:application/json' -XPUT 
 -d'{_id:foo,_rev:2

[jira] [Created] (COUCHDB-2519) CouchDB 2.0 does not support attachments=true on /_changes

[Will Holley (JIRA)]

Will Holley created COUCHDB-2519:


 Summary: CouchDB 2.0 does not support attachments=true on /_changes
 Key: COUCHDB-2519
 URL: https://issues.apache.org/jira/browse/COUCHDB-2519
 Project: CouchDB
  Issue Type: Bug
  Security Level: public (Regular issues)
  Components: Database Core
Reporter: Will Holley


In CouchDB 1.X, querying /_changes?include_docs=trueattachments=true returns 
the binary attachment data in the changes feed.

In CouchDB 2.0, the attachments=true parameter appears to be ignored - the 
changes feed includes stubs only.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (COUCHDB-2523) CouchDB 2.0: Specifying startkey/endkey parameters alongside a keys parameter when querying /_all_docs should be invalid

[Will Holley (JIRA)]

Will Holley created COUCHDB-2523:


 Summary: CouchDB 2.0: Specifying startkey/endkey parameters 
alongside a keys parameter when querying /_all_docs should be invalid
 Key: COUCHDB-2523
 URL: https://issues.apache.org/jira/browse/COUCHDB-2523
 Project: CouchDB
  Issue Type: Bug
  Security Level: public (Regular issues)
  Components: Database Core
Reporter: Will Holley


CouchDB 1.X validates that the combination of query parameters to _all_docs is 
valid. For instance, you cannot specify keys and also startkey/endkey:

{code}
curl -g -XGET 'http://127.0.0.1:5984/testdb/_all_docs?keys=[a]startkey=a;' 
{error:query_parse_error,reason:`keys` is incompatible with `key`, 
`start_key` and `end_key`}
{code}

In CouchDB 2.0, there is no such validation:
{code}
curl -g -XGET 'http://127.0.0.1:15984/testdb/_all_docs?keys=[a]startkey=a;'
{total_rows:5,rows:[
{id:a,key:a,value:{rev:1-4c6114c65e295552ab1019e2b046b10e}}
]}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (COUCHDB-2530) CouchDB 2.0 does not support POST requests to _changes

[Will Holley (JIRA)]

Will Holley created COUCHDB-2530:


 Summary: CouchDB 2.0 does not support POST requests to _changes
 Key: COUCHDB-2530
 URL: https://issues.apache.org/jira/browse/COUCHDB-2530
 Project: CouchDB
  Issue Type: Bug
  Security Level: public (Regular issues)
Reporter: Will Holley


In CouchDB 1.6, the _changes feed accepts a POST request (e.g. to submit a 
large doc_ids parameter):

{code}
curl http://127.0.0.1:5984/testdb/_changes -XPOST 
-HContent-Type:application/json -d '{doc_ids:[1]}'
{results:[
{seq:1,id:1,changes:[{rev:1-967a00dff5e02add41819138abb3284d}]}
],
last_seq:1}
{code}

In CouchDB 2.0, this fails with a 405 Method Not Allowed response:

{code}
curl http://127.0.0.1:15984/testdb/_changes -XPOST 
-HContent-Type:application/json -d '{doc_ids:[1]}'
{error:method_not_allowed,reason:Only GET,HEAD allowed}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (COUCHDB-2530) CouchDB 2.0 does not support POST requests to _changes

[Will Holley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Will Holley updated COUCHDB-2530:
-
Description: 
In CouchDB 1.6, the _changes feed accepts a POST request (e.g. to submit a 
large doc_ids parameter):

{code}
curl http://127.0.0.1:5984/testdb/_changes?filter=_doc_ids -XPOST 
-HContent-Type:application/json -d '{doc_ids:[1]}'
{results:[
{seq:1,id:1,changes:[{rev:1-967a00dff5e02add41819138abb3284d}]}
],
last_seq:1}
{code}

In CouchDB 2.0, this fails with a 405 Method Not Allowed response:

{code}
curl http://127.0.0.1:15984/testdb/_changes?filter=_doc_ids -XPOST 
-HContent-Type:application/json -d '{doc_ids:[1]}'
{error:method_not_allowed,reason:Only GET,HEAD allowed}
{code}

  was:
In CouchDB 1.6, the _changes feed accepts a POST request (e.g. to submit a 
large doc_ids parameter):

{code}
curl http://127.0.0.1:5984/testdb/_changes -XPOST 
-HContent-Type:application/json -d '{doc_ids:[1]}'
{results:[
{seq:1,id:1,changes:[{rev:1-967a00dff5e02add41819138abb3284d}]}
],
last_seq:1}
{code}

In CouchDB 2.0, this fails with a 405 Method Not Allowed response:

{code}
curl http://127.0.0.1:15984/testdb/_changes -XPOST 
-HContent-Type:application/json -d '{doc_ids:[1]}'
{error:method_not_allowed,reason:Only GET,HEAD allowed}
{code}


 CouchDB 2.0 does not support POST requests to _changes
 --

 Key: COUCHDB-2530
 URL: https://issues.apache.org/jira/browse/COUCHDB-2530
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
Reporter: Will Holley

 In CouchDB 1.6, the _changes feed accepts a POST request (e.g. to submit a 
 large doc_ids parameter):
 {code}
 curl http://127.0.0.1:5984/testdb/_changes?filter=_doc_ids -XPOST 
 -HContent-Type:application/json -d '{doc_ids:[1]}'
 {results:[
 {seq:1,id:1,changes:[{rev:1-967a00dff5e02add41819138abb3284d}]}
 ],
 last_seq:1}
 {code}
 In CouchDB 2.0, this fails with a 405 Method Not Allowed response:
 {code}
 curl http://127.0.0.1:15984/testdb/_changes?filter=_doc_ids -XPOST 
 -HContent-Type:application/json -d '{doc_ids:[1]}'
 {error:method_not_allowed,reason:Only GET,HEAD allowed}
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (COUCHDB-2531) CouchDB 2.0: POST to /_revs_diff with no revisions times out

[Will Holley (JIRA)]

Will Holley created COUCHDB-2531:


 Summary: CouchDB 2.0: POST to /_revs_diff with no revisions times 
out
 Key: COUCHDB-2531
 URL: https://issues.apache.org/jira/browse/COUCHDB-2531
 Project: CouchDB
  Issue Type: Bug
  Security Level: public (Regular issues)
Reporter: Will Holley


In CouchDB 1.6, posting an empty object to the _revs_diff endpoint returns an 
empty object:
{code}
$ curl http://127.0.0.1:5984/revsdifftest -XPUT 

{ok:true}
$ curl http://127.0.0.1:5984/revsdifftest/_revs_diff -XPOST 
-HContent-Type:application/json -d{}
{}
{code}

In CouchDB 2.0, the same request results in a timeout:
{code}
$ curl http://127.0.0.1:15984/revsdifftest -XPUT
 
{ok:true}
$ curl http://127.0.0.1:15984/revsdifftest/_revs_diff -XPOST 
-HContent-Type:application/json -d{}
{error:badmatch,reason:{error,timeout},ref:1478668763}
{code}

This currently breaks the PouchDB test suite when running against CouchDB 
master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2537) Propose removal of ?local_seq=true from the GET /db/doc API for CouchDB 2.0

[Will Holley (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303676#comment-14303676
 ] 

Will Holley commented on COUCHDB-2537:
--

I'm happy to be shot down but I wouldn't really expect CouchDB to work 
correctly if documents are submitted with _revs that don't conform to the spec. 
If two documents have exactly the same _id/_rev then they should have the same 
content / lineage (because CouchDB relies on hash histories).

 Propose removal of ?local_seq=true from the GET /db/doc API for CouchDB 2.0
 ---

 Key: COUCHDB-2537
 URL: https://issues.apache.org/jira/browse/COUCHDB-2537
 Project: CouchDB
  Issue Type: Improvement
  Security Level: public(Regular issues) 
  Components: Database Core
Reporter: Glynn Bird

 Prior to CouchDB 2.0, a document could be fetched thus:
 /db/8E795956?local_seq=true
 {Code}
 { _id: '8E795956',
   _rev: '1-4fffae881c4d89048cf9319c2ae021a1',
   test: 'somestuff',
   _local_seq: 1 }
 {Code}
 with the _local_seq being returned indicating 'Document’s sequence number in 
 current database'.
 Post CouchDB2.0, this quantity makes little sense as it represents the local 
 sequence number within the shard.
 I propose that
 * ?local_seq=true is deprecated
 * _local_seq is no longer returned in the response
 * the documentation is updated accordingly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (COUCHDB-2568) CouchDB 2.0 /_all_docs does not return full attachment data with attachments=true

[Will Holley (JIRA)]

Will Holley created COUCHDB-2568:


 Summary: CouchDB 2.0 /_all_docs does not return full attachment 
data with attachments=true
 Key: COUCHDB-2568
 URL: https://issues.apache.org/jira/browse/COUCHDB-2568
 Project: CouchDB
  Issue Type: Bug
  Security Level: public (Regular issues)
  Components: Database Core
Reporter: Will Holley


Similar to case COUCHDB-2522, in CouchDB 1.6, 
'/_all_docs'?include_docs=trueattachments=true' returns the full attachment 
data. In the clustered CouchDB 2.0 interface, the attachments query parameter 
is ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (COUCHDB-2568) CouchDB 2.0 /_all_docs does not return full attachment data with keys=[somekey]attachments=true

[Will Holley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Will Holley updated COUCHDB-2568:
-
Description: Similar to case COUCHDB-2522, in CouchDB 1.6, 
'/_all_docs'?keys=[somekey]include_docs=trueattachments=true' returns the 
full attachment data. In the clustered CouchDB 2.0 interface, the attachments 
query parameter is ignored.  (was: Similar to case COUCHDB-2522, in CouchDB 
1.6, '/_all_docs'?include_docs=trueattachments=true' returns the full 
attachment data. In the clustered CouchDB 2.0 interface, the attachments query 
parameter is ignored.)

 CouchDB 2.0 /_all_docs does not return full attachment data with 
 keys=[somekey]attachments=true
 --

 Key: COUCHDB-2568
 URL: https://issues.apache.org/jira/browse/COUCHDB-2568
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: Database Core
Reporter: Will Holley

 Similar to case COUCHDB-2522, in CouchDB 1.6, 
 '/_all_docs'?keys=[somekey]include_docs=trueattachments=true' returns the 
 full attachment data. In the clustered CouchDB 2.0 interface, the attachments 
 query parameter is ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (COUCHDB-2568) CouchDB 2.0 /_all_docs does not return full attachment data with keys=[somekey]attachments=true

[Will Holley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Will Holley updated COUCHDB-2568:
-
Summary: CouchDB 2.0 /_all_docs does not return full attachment data with 
keys=[somekey]attachments=true  (was: CouchDB 2.0 /_all_docs does not return 
full attachment data with attachments=true)

 CouchDB 2.0 /_all_docs does not return full attachment data with 
 keys=[somekey]attachments=true
 --

 Key: COUCHDB-2568
 URL: https://issues.apache.org/jira/browse/COUCHDB-2568
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: Database Core
Reporter: Will Holley

 Similar to case COUCHDB-2522, in CouchDB 1.6, 
 '/_all_docs'?include_docs=trueattachments=true' returns the full attachment 
 data. In the clustered CouchDB 2.0 interface, the attachments query parameter 
 is ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (COUCHDB-2568) CouchDB 2.0 /_all_docs does not return full attachment data with keys=[somekey]attachments=true

[Will Holley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Will Holley updated COUCHDB-2568:
-
Description: 
Similar to case COUCHDB-2522, in CouchDB 1.6, 
'/_all_docs'?keys=[somekey]include_docs=trueattachments=true' returns the 
full attachment data. In the clustered CouchDB 2.0 interface, the attachments 
query parameter is ignored.

In CouchDB 1.6:
{code}
~$ curl -X PUT http://127.0.0.1:5984/test
{ok:true}

~$ curl -X POST http://127.0.0.1:5984/test -H 'Content-type:application/json' 
-d '{ _id: foo, _attachments: { bar.txt: {content_type: 
text/plain,data: holy moly} } }'
{ok:true,id:foo,rev:1-b4adf85456c3026765ddee2d36c3814f}

~$ curl -g 
'http://127.0.0.1:5984/test/_all_docs?keys=[foo]include_docs=trueattachments=true'
{total_rows:1,offset:0,rows:[
{id:foo,key:foo,value:{rev:1-b4adf85456c3026765ddee2d36c3814f},doc:{_id:foo,_rev:1-b4adf85456c3026765ddee2d36c3814f,_attachments:{bar.txt:{content_type:text/plain,revpos:1,digest:md5-iRNdzYnZ/dk8u44XAJXaAQ==,data:holymoly
]}
{code}


In CouchDB 2.0:
{code}
~$ curl -X PUT http://127.0.0.1:15984/test
{ok:true}

~$ curl -X POST http://127.0.0.1:15984/test -H 'Content-type:application/json' 
-d '{ _id: foo, _attachments: { bar.txt: {content_type: 
text/plain,data: holy moly} } }'
{ok:true,id:foo,rev:1-b4adf85456c3026765ddee2d36c3814f}

~$ curl -g 
'http://127.0.0.1:15984/test/_all_docs?keys=[foo]include_docs=trueattachments=true'
{total_rows:1,rows:[
{id:foo,key:foo,value:{rev:1-b4adf85456c3026765ddee2d36c3814f},doc:{_id:foo,_rev:1-b4adf85456c3026765ddee2d36c3814f,_attachments:{bar.txt:{content_type:text/plain,revpos:1,digest:md5-iRNdzYnZ/dk8u44XAJXaAQ==,length:6,stub:true
]}
{code}

  was:Similar to case COUCHDB-2522, in CouchDB 1.6, 
'/_all_docs'?keys=[somekey]include_docs=trueattachments=true' returns the 
full attachment data. In the clustered CouchDB 2.0 interface, the attachments 
query parameter is ignored.


 CouchDB 2.0 /_all_docs does not return full attachment data with 
 keys=[somekey]attachments=true
 --

 Key: COUCHDB-2568
 URL: https://issues.apache.org/jira/browse/COUCHDB-2568
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: Database Core
Reporter: Will Holley

 Similar to case COUCHDB-2522, in CouchDB 1.6, 
 '/_all_docs'?keys=[somekey]include_docs=trueattachments=true' returns the 
 full attachment data. In the clustered CouchDB 2.0 interface, the attachments 
 query parameter is ignored.
 In CouchDB 1.6:
 {code}
 ~$ curl -X PUT http://127.0.0.1:5984/test
 {ok:true}
 ~$ curl -X POST http://127.0.0.1:5984/test -H 'Content-type:application/json' 
 -d '{ _id: foo, _attachments: { bar.txt: {content_type: 
 text/plain,data: holy moly} } }'
 {ok:true,id:foo,rev:1-b4adf85456c3026765ddee2d36c3814f}
 ~$ curl -g 
 'http://127.0.0.1:5984/test/_all_docs?keys=[foo]include_docs=trueattachments=true'
 {total_rows:1,offset:0,rows:[
 {id:foo,key:foo,value:{rev:1-b4adf85456c3026765ddee2d36c3814f},doc:{_id:foo,_rev:1-b4adf85456c3026765ddee2d36c3814f,_attachments:{bar.txt:{content_type:text/plain,revpos:1,digest:md5-iRNdzYnZ/dk8u44XAJXaAQ==,data:holymoly
 ]}
 {code}
 In CouchDB 2.0:
 {code}
 ~$ curl -X PUT http://127.0.0.1:15984/test
 {ok:true}
 ~$ curl -X POST http://127.0.0.1:15984/test -H 
 'Content-type:application/json' -d '{ _id: foo, _attachments: { 
 bar.txt: {content_type: text/plain,data: holy moly} } }'
 {ok:true,id:foo,rev:1-b4adf85456c3026765ddee2d36c3814f}
 ~$ curl -g 
 'http://127.0.0.1:15984/test/_all_docs?keys=[foo]include_docs=trueattachments=true'
 {total_rows:1,rows:[
 {id:foo,key:foo,value:{rev:1-b4adf85456c3026765ddee2d36c3814f},doc:{_id:foo,_rev:1-b4adf85456c3026765ddee2d36c3814f,_attachments:{bar.txt:{content_type:text/plain,revpos:1,digest:md5-iRNdzYnZ/dk8u44XAJXaAQ==,length:6,stub:true
 ]}
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (COUCHDB-2674) Inconsistent handling of URL encoded design doc name between clustered and non-clustered interfaces

[Will Holley (JIRA)]

Will Holley created COUCHDB-2674:


 Summary: Inconsistent handling of URL encoded design doc name 
between clustered and non-clustered interfaces
 Key: COUCHDB-2674
 URL: https://issues.apache.org/jira/browse/COUCHDB-2674
 Project: CouchDB
  Issue Type: Bug
  Security Level: public (Regular issues)
  Components: Database Core
Reporter: Will Holley


In CouchDB 1.6 (and non-clustered 2.X), a request to GET a design document 
where the full design document name is URL encoded results in a 301 redirect:

{code}
$ curl -v http://127.0.0.1:15986/test/_design%2Ffoo
 GET /test/_design%2Ffoo HTTP/1.1
 User-Agent: curl/7.37.1
 Host: 127.0.0.1:15986
 Accept: */*
 
 HTTP/1.1 301 Moved Permanently
* Server CouchDB/44fe5b6 (Erlang OTP/17) is not blacklisted
 Server: CouchDB/44fe5b6 (Erlang OTP/17)
 Location: http://127.0.0.1:15986/test/_design/foo
 Date: Sun, 26 Apr 2015 18:18:30 GMT
 Content-Length: 0
 
* Connection #0 to host 127.0.0.1 left intact
{code}

However, the same request through the clustered interface is handled directly:

{code}
$ curl -v http://127.0.0.1:15984/test/_design%2Ffoo
 GET /test/_design%2Ffoo HTTP/1.1
 User-Agent: curl/7.37.1
 Host: 127.0.0.1:15984
 Accept: */*
 
 HTTP/1.1 200 OK
 X-CouchDB-Body-Time: 0
 X-Couch-Request-ID: f9a8af50
* Server CouchDB/44fe5b6 (Erlang OTP/17) is not blacklisted
 Server: CouchDB/44fe5b6 (Erlang OTP/17)
 Etag: 1-670d087e023b4320c148c8e73ba82129
 Date: Sun, 26 Apr 2015 18:16:51 GMT
 Content-Type: text/plain; charset=utf-8
 Content-Length: 160
 Cache-Control: must-revalidate
 
{_id:_design/foo,_rev:1-670d087e023b4320c148c8e73ba82129,views:{newView:{map:function(doc)
 {\n  emit(doc._id, 1);\n}}},language:javascript}
* Connection #0 to host 127.0.0.1 left intact
{code}

I think the behaviour in the clustered interface is preferable but we should be 
consistent. The difference seems to be at 
https://github.com/apache/couchdb-chttpd/blob/ab80f3131e244af967e2d162925ee45008d54a50/src/chttpd_db.erl#L512.

This came to light due to the .NET HTTP client not sending authentication 
headers when following the redirect (see 
https://github.com/danielwertheim/mycouch/issues/75).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)