Re: 3.3.3 packages update

[Jan Lehnardt]

I’ve also uploaded the Mac and Win packages. Thanks to Ronny for providing the 
Win build.

Best
Jan
— 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

24/7 Observation for your CouchDB Instances:
https://opservatory.app

SQL Queries for CouchDB:
https://neighbourhood.ie/products-and-services/structured-query-server

> On 3. Jan 2024, at 21:49, Nick Vatamaniuc  wrote:
> 
> Hi everyone,
> 
> Debian and RPM 3.3.3 packages were rebuilt and updated with the Erlang
> version 24.3.4.15. That version fixes a memory leak
> https://github.com/erlang/otp/issues/7834
> 
> There were no other CouchDB source changes. The updated deb package
> has version 3.3.3-1 and the rpm one has version 3.3.3.1-1
> 
> Cheers,
> -Nick

Re: Older versions for OSX are not available for download

[Jan Lehnardt]

That’s a bug on my end it seems, I’ll get it seen to. Thanks for reporting.

Best
Jan


> On 25. Dec 2023, at 17:59, Miroslav Rodic  wrote:
> 
> Hello,
> 
> I’m trying to download CouchDB 3.2.0 for OSX 
> , but without 
> success. Only older OSX versions are unavailable. All older Windows versions 
> are available.
> 
> Is there a reason why this is specific only to older OSX versions or it is 
> simply a bug?
> 
> Regards,
> Miroslav
> 

[ANNOUNCE] Apache CouchDB 3.3.3 released

[Jan Lehnardt]

Dear community,

Apache CouchDB® 3.3.3 has been released and is available for download.

https://couchdb.apache.org/#download

Pre-built packages for Windows, macOS, Debian/Ubuntu and RHEL/CentOS are 
available.

CouchDB 3.3.3 is a maintenance release, and was originally published on 
2023-12-05.

The community would like to thank all contributors for their part in making 
this release, from the smallest bug report or patch to major contributions in 
code, design, or marketing, we couldn’t have done it without you!

See the official release notes document for an exhaustive list of all changes:

http://docs.couchdb.org/en/stable/whatsnew/3.3.html

Release Notes highlights:

  - Fix for CVE-2023-45725. The details will be released in seven days. We 
recommend all CouchDB users upgrade.

  - Require authentication for the _replicate endpoint. This is a continuation 
of the 3.x “closed-by-default” security policy.

  - Fix multipart parser "attachment longer than expected” error. Previously 
there was a small chance attachments were not replicated.

  - Various bug fixes related to shard splitting, purging, and replication.

Apache CouchDB® lets you access your data where you need it. The Couch 
Replication Protocol is implemented in a variety of projects and products that 
span every imaginable computing environment from globally distributed 
server-clusters, over mobile phones to web browsers.

Store your data safely, on your own servers, or with any leading cloud 
provider. Your web- and native applications love CouchDB, because it speaks 
JSON natively and supports binary data for all your data storage needs.

The Couch Replication Protocol lets your data flow seamlessly between server 
clusters to mobile phones and web browsers, enabling a compelling offline-first 
user-experience while maintaining high performance and strong reliability. 
CouchDB comes with a developer-friendly query language, and optionally 
MapReduce for simple, efficient, and comprehensive data retrieval.

On behalf of the CouchDB PMC,
Jan Lehnardt
—

Introducing Structured Query Server, SQL Queries for CouchDB

[Jan Lehnardt]

Dear CouchDB user community,

My company Neighbourhoodie is happy to announce Structured Query Server,
a CouchDB companion application that gives you full-fidelity SQL query
abilities for your CouchDB installations.

See all infos, technical details and benchmarks on our product page:

https://neighbourhood.ie/products-and-services/structured-query-server


Do let me know off-list, if you have any questions.

Best
Jan
—

Re: High CPU at startup and wake

[Jan Lehnardt]

Heya,

The last time we fixed something along these lines was in 2019:

https://github.com/apache/couchdb/pull/2195/files

Which predates 3.1.0: 

https://blog.couchdb.org/2020/05/06/3-0-1-3-1-0/

Are you 100% sure about your version?

A quick look does not suggest we have introduced this pattern again
since.

But CouchDB should behave fine in your scenario.

Can you try and remsh into the node and see which erlang processes
are hogging the CPU?

Also we should move this to the bug tracker if you can reproduce this
on 3.3.2.

Best
Jan
—
> On 25. Apr 2023, at 05:53, Andrew E  wrote:
> 
> Hi all,
> 
> Firstly, sorry for posting this previously on the dev channel.
> 
> Our issue...
> 
> We use CouchDB (3.1.0) on Windows and Linux.
> 
> We have noticed that on Windows startup and wake from hibernate, Couch
> sometimes "goes bananas" for a period of time.
> 
> By that I mean it shows a CPU load of >30% for 10-20 minutes on a
> reasonably spec'd Windows developer laptop. After that period, erl.exe goes
> to a normal "almost invisible" CPU level.
> 
> The active task list is shown as empty in Fauxton.
> 
> In Production the Windows instances are on vehicles that are not reachable
> on the network for hours at a time while collecting data. They replicate to
> the Linux servers when the network is present. Those computers are
> underpowered compared to development laptops, which means they are barely
> usable when Couch grinds like this.
> 
> Any ideas what could be going on or how to find out?
> 
> Thank you!
> Andrew

[ANNOUNCE] Apache CouchDB 3.3.2 released

[Jan Lehnardt]

Dear community,

Apache CouchDB® 3.3.2 has been released and is available for download.

Apache CouchDB® lets you access your data where you need it. The Couch 
Replication Protocol is implemented in a variety of projects and products that 
span every imaginable computing environment from globally distributed 
server-clusters, over mobile phones to web browsers.

Store your data safely, on your own servers, or with any leading cloud 
provider. Your web- and native applications love CouchDB, because it speaks 
JSON natively and supports binary data for all your data storage needs.

The Couch Replication Protocol lets your data flow seamlessly between server 
clusters to mobile phones and web browsers, enabling a compelling offline-first 
user-experience while maintaining high performance and strong reliability. 
CouchDB comes with a developer-friendly query language, and optionally 
MapReduce for simple, efficient, and comprehensive data retrieval.

https://couchdb.apache.org/#download

Pre-built packages for Windows, macOS, Debian/Ubuntu and RHEL/CentOS are 
available.

CouchDB 3.3.2 is a maintenance release, and was originally published on 
2023-04-25.

The community would like to thank all contributors for their part in making 
this release, from the smallest bug report or patch to major contributions in 
code, design, or marketing, we couldn’t have done it without you!

See the official release notes document for an exhaustive list of all changes:

http://docs.couchdb.org/en/stable/whatsnew/3.3.html

Release Notes highlights:

- Fix for CVE-2023-26268, the details of which will be released in 7 days

- Improve JavaScript process management

- Fix quoting issue with `remsh`

- Allow configurable timeouts for _view and _search

- Proxy auth can now use one of the configured hash algorithms from 
chttpd_auth/hash_algorithms to decode authentication tokens

- Bump recon to 2.5.3

On behalf of the CouchDB PMC,
Jan Lehnardt
—

[ANNOUNCE] Apache CouchDB 3.2.3 released

[Jan Lehnardt]

Dear community,

Apache CouchDB® 3.2.3 has been released and is available for download.

Apache CouchDB® lets you access your data where you need it. The Couch 
Replication Protocol is implemented in a variety of projects and products that 
span every imaginable computing environment from globally distributed 
server-clusters, over mobile phones to web browsers.

Store your data safely, on your own servers, or with any leading cloud 
provider. Your web- and native applications love CouchDB, because it speaks 
JSON natively and supports binary data for all your data storage needs.

The Couch Replication Protocol lets your data flow seamlessly between server 
clusters to mobile phones and web browsers, enabling a compelling offline-first 
user-experience while maintaining high performance and strong reliability. 
CouchDB comes with a developer-friendly query language, and optionally 
MapReduce for simple, efficient, and comprehensive data retrieval.

https://couchdb.apache.org/#download

Pre-built packages for Windows, macOS, Debian/Ubuntu and RHEL/CentOS are 
available.

CouchDB 3.2.3 is a maintenance release, and was originally published on 
2023-04-25.

The community would like to thank all contributors for their part in making 
this release, from the smallest bug report or patch to major contributions in 
code, design, or marketing, we couldn’t have done it without you!

See the official release notes document for an exhaustive list of all changes:

http://docs.couchdb.org/en/stable/whatsnew/3.2.html

Release Notes highlights:

- Fix for CVE-2023-26268, the details of which will be released in 7 days

- Improve JavaScript process management

- Fix quoting issue with `remsh`

On behalf of the CouchDB PMC,
Jan Lehnardt
—

[ANNOUNCE] CouchDB Berlin User Group Meetup Wednesday March 15 (online)

[Jan Lehnardt]

Hey all,

we're happy to invite you to our next online CouchDB Berlin
Meetup on March 15th (this week!),
2022, 5:30 PM – 7:00 PM (CET)

Talks by Alex Feyerke, Jan Johannes & your’s truly:

Talk #1: Type safe access to CouchDB with PouchDB
Talk #2: Zero Boilerplate Data Binding for Reactive and Offline
 UIs with CouchDB and PouchDB
Talk #3: CouchDB and… SQL?

Register here: https://vi.to/hubs/couchdb-berlin

Best
Jan
— 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

24/7 Observation for your CouchDB Instances:
https://opservatory.app

Re: Some random user questions

[Jan Lehnardt]

Heya Vincent,

apologies for taking so long to respond :)

> On 14. Jan 2023, at 23:08, Vincent van der Leun  wrote:
> 
> Hello, 
> 
> I am a Dutch user of CouchDB for quite some time. I know I won't make 
> headlines, given that I only run a single-node CouchDB instance that runs on 
> a small VPS that also runs my small web-applications as well (my DB's are 
> only up to dozens of MBs in size), but that wouldn't stop me from mailing the 
> mailing list :)

Any size is a good size for CouchDB :)

> Recently upgraded my CouchDB 2 instance to the latest 3 version, it was about 
> time... It was an easy process and v3 seemingly fixed some bugs that were 
> plaguing me, so I am happy with my choice.

\o/

> I was mostly wondering how others are using their CouchDB setup, so that's 
> why I am writing. I have some questions to the community in general:
> 
> Coming late to the CouchDB party (around the v2 days), Mango querying is one 
> of my favorite features. Tutorials/videos always seem to talk about 
> JavaScript filters functions and dismiss Mango as a minor, unimportant 
> feature (IF they mention Mango at all). Was just wondering how long-time 
> users are querying their JSON documents mostly nowadays?

From my observation, most folks still use JavaScript MapReduce Views for 
querying, but as Mango provides a substantial indexing performance increase, 
and some additional easy of use, it is not unpopular, even if not as 
feature-rich as JS views. Mango also receives updates[1] like the rest of the 
system.

> I saw somewhere in the v3 release notes that the Javascript "update" 
> functions were deprecated in v3, although I didn't see a Deprecation warning 
> in the https://docs.couchdb.org/en/stable/ddocs/ddocs.html#updatefun section. 
> Is it indeed deprecated? I kinda liked that feature, for example to update 
> creation/update timestamp fields in documents automatically. 

We have deprecated show, list & rewrite functions, but not filters or update or 
validate_doc_update functions. Update functions were considered for 
deprecation, but we left them in for now.

> Am I correct in thinking that CouchDB is moving away from the idea that web 
> applications should run on top of CouchDB (since I use Nginx, I personally 
> never even considered it) and logic like updating timestamps, should, 
> according to the CouchDB devs, ideally be implemented in the client 
> applications themselves? Also, are "Validate Document Update Functions" also 
> part of the  "Update functions" that are deprecated?

You are correct. CouchDB and its JS functions was conceived in a world before 
Node.js, which is now the de-facto standard for running server-side JS and had 
it been around, we’d likely have made use of it then. These days, using a 
Node.js process alongside your CouchDB installation can be indefinitely more 
useful and a lot more efficient than the deprecated functions could ever be.

Update functions and validate_doc_update functions are different things:

- update functions receive any HTTP requests and transform them into CouchDB 
docs
- validate_doc_update functions receive a single document update and allow or 
deny it being written in the database

The latter is also used for access control and validation purposes and they are 
not deprecated. Although we are (slowly) designing more efficient 
alternatives[2].

> Are members of this mailinglist using custom Query Servers in practice and if 
> so, what are your use-cases? It seems so interesting and I wonder what people 
> are doing with it.

The only ones I’m aware of are some folks that run a Python query server, so 
they have an all-python development environment.

> Finally, one of the limitations that I know of, that deleted documents stay 
> in the CouchDB database "forever", is one that stops me from choosing CouchDB 
> for some applications, that have to deal with temporary data. Is that 
> limitation still in place (now and/or for the foreseeable future?).  

We are discussing an auto-prune features on the dev@ mailing list, here is a 
summary[3] of our current thinking.

> To the CouchDB team members, past and present: many thanks for your efforts! 
> I wish I was more familiar with Erlang and could contribute somehow.

Thank you! [4]

[1]: https://github.com/apache/couchdb/pull/4410
[2]: https://github.com/apache/couchdb/issues/1554 

[3]: 
https://docs.google.com/document/d/1ZkJs3Lrk8YOmPHeVYSe_yTp5rkqqAQyAnSQ-VqZ0R-0/edit
[4]: https://learnyousomeerlang.com/content

Best
Jan
—
> 
> Kind regards,
> Vincent
> 

[ANNOUNCE] Apache CouchDB 3.3.1 released

[Jan Lehnardt]

Dear community,

Apache CouchDB® 3.3.1 has been released and is available for download.

See the official release notes document for an exhaustive list of all changes:

https://docs.couchdb.org/en/stable/whatsnew/3.3.html

Release Notes highlights:

  - Fix issue that prevented more than one replication running at a time.
  - Fix issue with _replicator docs that have a `user_cdx` field.


https://couchdb.apache.org/#download

Pre-built packages for Windows, macOS, Debian/Ubuntu and RHEL/CentOS are 
available.

CouchDB 3.3.1 is a maintenance release, and was originally published on 
2023-01-11.

* * *

Apache CouchDB® lets you access your data where you need it. The Couch 
Replication Protocol is implemented in a variety of projects and products that 
span every imaginable computing environment from globally distributed 
server-clusters, over mobile phones to web browsers.

Store your data safely, on your own servers, or with any leading cloud 
provider. Your web- and native applications love CouchDB, because it speaks 
JSON natively and supports binary data for all your data storage needs.

The Couch Replication Protocol lets your data flow seamlessly between server 
clusters to mobile phones and web browsers, enabling a compelling offline-first 
user-experience while maintaining high performance and strong reliability. 
CouchDB comes with a developer-friendly query language, and optionally 
MapReduce for simple, efficient, and comprehensive data retrieval.


The community would like to thank all contributors for their part in making 
this release, from the smallest bug report or patch to major contributions in 
code, design, or marketing, we couldn’t have done it without you!


On behalf of the CouchDB PMC,
Jan Lehnardt
—

Re: [RELEASE] Apache CouchDB 3.3.0 released

[Jan Lehnardt]

Wip here: https://github.com/apache/couchdb/pull/4291

But no timelines, as per usual :) — The initial focus will be on feature parity 
with the existing search, while allowing for more powerful features later on. 
How many of these advanced features will be available when will depend on folks 
contributing to this.

If you wanna help, do give that branch a spin and report back on the PR.

Best
Jan
—

> On 3. Jan 2023, at 13:31, Rick Jarvis  wrote:
> 
> Thanks Jan
> 
> Did I read somewhere that there is a new more powerful search coming to 
> either this version or perhaps 3.4.0? Any info on this / timescales if so?
> 
> R
> 
>> On 3 Jan 2023, at 10:42, Jan Lehnardt  wrote:
>> 
>> Dear community,
>> 
>> Apache CouchDB® 3.3.0 has been released and is available for download. It is 
>> a feature release, and was originally published on 2023-01-03.
>> 
>> Release Notes highlights:
>>   • Improve replication performance by at least 3x for certain workloads by…
>>   • …speeding up the _bulk_get & _revs_diff endpoints
>>   • …making use of the faster _bulk_get endpoint in the replicator
>>   • …statistically skip calling the _revs_diff endpoint if it is not 
>> needed
>>   • this speeds up replications into empty databases significantly
>>   • …more efficiently encoding all occurrences of _rev values
>>   • A new winning_revs_only replicator option to create a database copy 
>> without any conflicts occurring in the source database
>>   • Start using SHA256 for session cookie HMAC calculation
>>   • Support Erlang 25 with its improved JIT support for ARM64
>>   • For a more in-depth discussion see this recording of the November Berlin 
>> CouchDB User Group online meetup: 
>> https://vi.to/hubs/couchdb-berlin/pages/state-of-the-couch-november-2022-jan-lehnardt-nov-16-2022?v=%2Fvideos%2F5904
>>  (free signup required, no spam)
>>   • CouchDB is on Mastodon now: https://fosstodon.org/@couchdb
>> 
>> See the official release notes document for an exhaustive list of all 
>> changes:
>> http://docs.couchdb.org/en/stable/whatsnew/3.3.html
>> 
>> Pre-built packages for Windows, macOS, Debian/Ubuntu and RHEL/CentOS are 
>> available alongside the source code distribution: 
>> https://couchdb.apache.org/#download
>> 
>> Apache CouchDB® lets you access your data where you need it. The Couch 
>> Replication Protocol is implemented in a variety of projects and products 
>> that span every imaginable computing environment from globally distributed 
>> server-clusters, over mobile phones to web browsers.
>> 
>> Store your data safely, on your own servers, or with any leading cloud 
>> provider. Your web- and native applications love CouchDB, because it speaks 
>> JSON natively and supports binary data for all your data storage needs.
>> 
>> The Couch Replication Protocol lets your data flow seamlessly between server 
>> clusters to mobile phones and web browsers, enabling a compelling 
>> offline-first user-experience while maintaining high performance and strong 
>> reliability. CouchDB comes with a developer-friendly query language, and 
>> optionally MapReduce for simple, efficient, and comprehensive data retrieval.
>> 
>> The community would like to thank all contributors for their part in making 
>> this release, from the smallest bug report or patch to major contributions 
>> in code, design, or marketing, we couldn’t have done it without you!
>> 
>> On behalf of the CouchDB PMC,
>> Jan Lehnardt
>> —
> 

[RELEASE] Apache CouchDB 3.3.0 released

[Jan Lehnardt]

Dear community,

Apache CouchDB® 3.3.0 has been released and is available for download. It is a 
feature release, and was originally published on 2023-01-03.

Release Notes highlights:
• Improve replication performance by at least 3x for certain workloads by…
• …speeding up the _bulk_get & _revs_diff endpoints
• …making use of the faster _bulk_get endpoint in the replicator
• …statistically skip calling the _revs_diff endpoint if it is not 
needed
• this speeds up replications into empty databases significantly
• …more efficiently encoding all occurrences of _rev values
• A new winning_revs_only replicator option to create a database copy 
without any conflicts occurring in the source database
• Start using SHA256 for session cookie HMAC calculation
• Support Erlang 25 with its improved JIT support for ARM64
• For a more in-depth discussion see this recording of the November Berlin 
CouchDB User Group online meetup: 
https://vi.to/hubs/couchdb-berlin/pages/state-of-the-couch-november-2022-jan-lehnardt-nov-16-2022?v=%2Fvideos%2F5904
 (free signup required, no spam)
• CouchDB is on Mastodon now: https://fosstodon.org/@couchdb

See the official release notes document for an exhaustive list of all changes:
http://docs.couchdb.org/en/stable/whatsnew/3.3.html

Pre-built packages for Windows, macOS, Debian/Ubuntu and RHEL/CentOS are 
available alongside the source code distribution: 
https://couchdb.apache.org/#download

Apache CouchDB® lets you access your data where you need it. The Couch 
Replication Protocol is implemented in a variety of projects and products that 
span every imaginable computing environment from globally distributed 
server-clusters, over mobile phones to web browsers.

Store your data safely, on your own servers, or with any leading cloud 
provider. Your web- and native applications love CouchDB, because it speaks 
JSON natively and supports binary data for all your data storage needs.

The Couch Replication Protocol lets your data flow seamlessly between server 
clusters to mobile phones and web browsers, enabling a compelling offline-first 
user-experience while maintaining high performance and strong reliability. 
CouchDB comes with a developer-friendly query language, and optionally 
MapReduce for simple, efficient, and comprehensive data retrieval.

The community would like to thank all contributors for their part in making 
this release, from the smallest bug report or patch to major contributions in 
code, design, or marketing, we couldn’t have done it without you!

On behalf of the CouchDB PMC,
Jan Lehnardt
—

[RELEASE] PouchDB 8.0.0

[Jan Lehnardt]

Hey all.

PouchDB 8.0.0 has been released: 
https://fosstodon.org/@pouchdb/109517906610488395

Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

*24/7 Observation for your CouchDB Instances:
https://opservatory.app 

[ANNOUNCE] (Virtual) CouchDB Usergroup Meetup on Wednesday, Nov 16, 5:30 CET)

[Jan Lehnardt]

Heya folks,


We are running a @CouchDB u meetup next week (Wed 16th). It’s the Berlin user 
group organising, but it is entirely virtual. We have three nice talks lined 
up, you can sign up here for free, anybody is welcome: 
https://vi.to/hubs/couchdb-berlin

Best
Jan
— 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

24/7 Observation for your CouchDB Instances:
https://opservatory.app

CVE-2022-24706: Apache CouchDB Remote Privilege Escalation

[Jan Lehnardt]

Description
===

An attacker can access an improperly secured default installation without
authenticating and gain admin privileges.

1. CouchDB opens a random network port, bound to all available interfaces
   in anticipation of clustered operation and/or runtime introspection. A
   utility process called `epmd` advertises that random port to the network.
   `epmd` itself listens on a fixed port.
2. CouchDB packaging previously chose a default `cookie` value for single-node
   as well as clustered installations. That cookie authenticates any
   communication between Erlang nodes.

The CouchDB documentation[1] has always made recommendations for properly
securing an installation, but not all users follow the advice.

We recommend a firewall in front of all CouchDB installations. The full
CouchDB api is available on registered port `5984` and this is the only
port that needs to be exposed for a single-node install. Installations
that do not expose the separate distribution port to external access are
not vulnerable.

Mitigation
==

CouchDB 3.2.2 and onwards will refuse to start with the former default
Erlang cookie value of `monster`. Installations that upgrade to this
versions are forced to choose a different value.

In addition, all binary packages have been updated to bind `epmd` as
well as the CouchDB distribution port to `127.0.0.1` and/or `::1`
respectively.

Credit
==

This issue was identified by Alex Vandiver .

[1]: https://docs.couchdb.org/en/stable/setup/cluster.html

[ANNOUNCE] Apache CouchDB 3.2.2 released

[Jan Lehnardt]

Dear community,

Apache CouchDB® 3.2.2 has been released and is available for download.

Apache CouchDB® lets you access your data where you need it. The Couch 
Replication Protocol is implemented in a variety of projects and products that 
span every imaginable computing environment from globally distributed 
server-clusters, over mobile phones to web browsers.

Store your data safely, on your own servers, or with any leading cloud 
provider. Your web- and native applications love CouchDB, because it speaks 
JSON natively and supports binary data for all your data storage needs.

The Couch Replication Protocol lets your data flow seamlessly between server 
clusters to mobile phones and web browsers, enabling a compelling offline-first 
user-experience while maintaining high performance and strong reliability. 
CouchDB comes with a developer-friendly query language, and optionally 
MapReduce for simple, efficient, and comprehensive data retrieval.

https://couchdb.apache.org/#download

Pre-built packages for Windows, macOS, Debian/Ubuntu and RHEL/CentOS are 
available.

CouchDB 3.2.2 is a maintenance release, and was originally published on 
2022-04-19.

The community would like to thank all contributors for their part in making 
this release, from the smallest bug report or patch to major contributions in 
code, design, or marketing, we couldn’t have done it without you!

See the official release notes document for an exhaustive list of all changes:

http://docs.couchdb.org/en/stable/whatsnew/3.2.html

Release Notes highlights:

  - This releases includes a fix for CVE-2022-24706, the nature of which will 
be announced a week after the release.

  - Optimize compaction and doc updates for conflicted documents on Erlang 
versions higher than 21.

  - Add support for SpiderMonkey 91esr.

On behalf of the CouchDB PMC,
Jan Lehnardt
—

Re: CouchDB 3.1.1 high disk utilization and degradation over time

[Jan Lehnardt]

Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

*24/7 Observation for your CouchDB Instances:
https://opservatory.app

> On 4. Apr 2022, at 18:21, Roberto Iglesias  wrote:
> 
> Hello.
> 
> About 1 year ago, we had two CouchDB 2.3.1 instances running inside Docker 
> containers and pull-replicating one each other. This way, we could read from 
> and write to any of these servers, although we generally choose one as the 
> "active" server and write to it. The second server would act as a spare or 
> backup.
> 
> At this point (1y ago) we decided to migrate from CouchDB version 2.3.1 to 
> 3.1.1. Instead of upgrading our existing databases, we added two extra 
> instances and configure pull replications in all of them until we get the 
> following scenario:
> 
> 2.3.1-A <===> 2.3.1-B <===> 3.1.1-A <===> 3.1.1-B
> 
> where <===> represents two pull replications, one configured on each side. 
> i.e: 2.3.1-A pulls from 2.3.1-B and vice versa.
> 
> If a write is made at 2.3.1-A, it has to make it through all servers until it 
> reaches 3.1.1-B.
> 
> All of them have an exclusive HDD which is not shared with any other service.
> 
> We have not a single problem with 2.3.1.
> 
> After pointing our services to 3.1.1-A, it gradually started to increase Read 
> I/O wait times over weeks until it reached peaks of 600ms (totally 
> unworkable). So we stopped making write requests (http POST) to it and 
> pointed all applications to 3.1.1-B. 3.1.1-A was still receiving writes but 
> only by replication protocol, as I explained before.
> 
> At 3.1.1-A server, disk stats decreased to acceptable values, so a few weeks 
> after we pointed applications back to it in order to confirm whether the 
> problem is related to write requests sent from our application or not. Read 
> I/O times did not increase this time. Instead, 3.1.1-B (which handled 
> application traffic for a few weeks), started to show the same behaviour, 
> despite it was not handling requests from applications.
> 
> It feels like some fragmentation is occurring, but filesystem (ext4) shows 
> none.
> 
> Some changes we've made since problem started:
>   • Upgraded kernel from 4.15.0-55-generic to 5.4.0-88-generic
>   • Upgraded ubuntu from 18.04 to 20.04
>   • Deleted _global_changes database from couchdb3.1.1-A
> 
> More info:
>   • Couchdb is using docker local-persist 
> (https://github.com/MatchbookLab/local-persist) volumes.
>   • Disks are WD Purple for 2.3.1 couchdbs and WD Black for 3.1.1 
> couchdbs.
>   • We have only one database of 88GiB and 2 views: one of 22GB and a 
> little one of 30MB (highly updated)
>   • docker stats shows that couchdb3.1.1 uses lot of memory compared to 
> 2.3.1:
>   • 2.5GiB for couchdb3.1.1-A (not receiving direct write requests)
>   • 5.0GiB for couchdb3.1.1-B (receiving both read and write requests)
>   • 900MiB for 2.3.1-A
>   • 800MiB for 2.3.1-B
>   • Database compaction is run at night. Problem only occurs over day, 
> when most of the writes are made.

Did you account for the completely rewritten compaction daemon (smoosh) that 
has a different configuration from the one in 2.x?

https://docs.couchdb.org/en/stable/maintenance/compaction.html#compact-auto

Otherwise you might see compaction going on at all times (what we recommend, 
usually), rather than what you expect: just at night.

And in general, at this point, we strongly recommend running on SSDs for the 
obvious speed benefits :)

And finally: which Erlang version are you running? There are a few odd ones out 
there that might affect what you’re doing.

Best
Jan
—
>   • Most of the config is default.
>   • Latency graph from munin monitoring attached (at the peak, there is 
> an outage of the server caused by a kernel upgrade that went wrong)
> 
> Any help is appreciated.
> 
> -- 
> --
> 
> Roberto E. Iglesias

Re: Post/Comment DB design: Postgresql v/s CouchDB

[Jan Lehnardt]

Hi Meeka,

we don’t generally recommend a per-user database system unless you deal with 
replication with e.g. PouchDB where each database needs distinct user access.

For just a regular blog app, you can use a single database. The CouchDB docs 
explain just that example in fact. You’ll be working with different types of 
documents in the same database and use views to build 1:N relations like you 
know them from Postgres:

https://docs.couchdb.org/en/stable/ddocs/views/intro.html
https://docs.couchdb.org/en/stable/ddocs/views/intro.html#the-view-to-get-comments-for-posts
https://docs.couchdb.org/en/stable/ddocs/views/joins.html

Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

*24/7 Observation for your CouchDB Instances:
https://opservatory.app

> On 1. Apr 2022, at 10:38, Meekaa Saangoo  wrote:
> 
> I am comparing DB design for a simple "Post and Comment" system using 
> Postgres and CouchDB.
> With Postgres I can design the following tables:
> 
> user_info {email, pass_hash, pass_salt, ...}
> post_info {post_id, creator_email, title, text, ...}
> comment_info {comment_id, creator_email, post_id, parent_comment_id, text, 
> ...}
> 
> But if I use CouchDB, there is a concept of creating per-user tables. So I 
> was thinking of the following design:
> 
> user_table {email, table_id}
> user_ {email, pass_hash, pass_salt, ...}
> post_ {post_id, _creator_email, title, text, ...}
> comment_ {comment_id, _creator_email, _post_id, 
> _parent_comment_id, text, ...}
> 
> I am in no way expert in Postgres and CouchDB, so my question is, is this the 
> correct way to design per-user CouchDB tables? What is the better way? And 
> what is the efficient way to create/use CRUD queries?
> 
> Thank you!
> 

Re: CouchDB High Availablility deployment and ChangeFeed

[Jan Lehnardt]

Hi A,

for your setup, we recommend using CouchDB Clustering to achieve high 
availability.

But note that neither a single node nor a cluster _changes feed guarantees 
exactly-once-delivery. You have to solve that in your Spring-boot API.

Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

*24/7 Observation for your CouchDB Instances:
https://opservatory.app

> On 27. Mar 2022, at 13:45, Al Z.  wrote:
> 
> Hello.
> 
> I have a single node running couchDB 3 deployed live.
> I have a DB-per-user setup with about 2k databases on a single node.
> A single Spring-boot API talks to couchDB vis HAProxy ...
> 
> 1) I would like to have multiple 2 to 3 nodes in the same DC so that in
> case one node goes down, the system is still operational.
> Each node will have an instance of the Spring-boot API, ...
> I am not sure whether I should be looking into clustering or replication.
> I do understand that with replication, the nodes can be in different DB.
> At this point, the plan is a single DC only. In the future, I may look into
> a cross DC setup.
> 
> Any hint will me most appreciated.
> 
> 
> 2) I use the couchDB change feed to update some internal systems and send
> notification emails.
> Each change feed needs to be processed exactly once.
> The change feed seems to be a major point of failure for my setup ...( I
> also noticed that if couchDB is restarted, I need to restart the
> spring-boot app, otherwise, it does not process the feed).
> Given that I am planning to move to a multiple nodes setup, I am not very
> sure how I would be able to process each feed only once without having to
> manually handle duplicate.
> Any experience or hint or alternative would be most appreciated.
> 
> Thank you very much.
> 
> Best regards.
> 
> A.

Re: Off-line sync by exchanging .couch files

[Jan Lehnardt]

Oh, and note that for 3.2.0 and CouchApps to work, you will have to change the 
default CSP headers: https://docs.couchdb.org/en/stable/cve/2021-38295.html / 
https://github.com/apache/couchdb/pull/3724

Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

*24/7 Observation for your CouchDB Instances:
https://opservatory.app

> On 20. Oct 2021, at 14:56, Miroslav Rodic  wrote:
> 
> For an application, developed 10 years ago as couchapp with CouchDB 1.1.1, an 
> OFF-LINE data exchange (synchronization) is implemented by adding the local 
> .couch file, with randomly generated name, in a .zip file, which is then 
> sent, by email, to another user and then that user extracts and copies the 
> received .couch file into the local CouchDB instance. The imported .couch 
> file is then synchronized with the appropriate local database using 
> replication. All export/import steps, including the final replication, were 
> implemented within the application, so there were no problems with non-IT 
> users understanding how to use it. Since the application is implemented as a 
> couchapp type, the necessary Erlang functions have been developed that allow 
> archiving/copying/etc.
> 
> The question follows: is it possible to keep the same approach in off-line 
> synchronization with the latest version of CouchDB, which is 3.2.0, when the 
> installed CouchDB is configured as a single node? If this is not possible, 
> then which solutions are recommended for off-line synchronization?
> 
> Copying of .couch files is not clearly explained in the backup documentation. 
> A backup assumes, if I understood correctly, that .couch files can be copied 
> to a secure location, and then, when needed, can be restored by overwriting 
> the existing .couch files in the data directory. If, for any reason, the 
> corresponding DB is deleted by using the Fauxton before the .couch files are 
> reverted back, then reverted .couch files will not be accepted and displayed 
> in Fauxton. This is the behavior I had on my comp testing the backup/restore 
> procedure.
> 
> Backup/restore procedure tested on OS X 10.15.7 with CouchDB 3.2.0, where 
> path to data directory is as follows: ~/Library/Application 
> Support/CouchDB2/var/lib/couchdb.
> 
> Regards,
> Miroslav
> 

Re: Off-line sync by exchanging .couch files

[Jan Lehnardt]

Hi Miroslav,

that’s a nice application :)

You should be able to do the same thing with CouchDB 3.2.0, but you need to do 
an extra step.

In CouchDB 1.x, when opening a a database `foo`, it would open a file at 
`$database_dir/foo.couch`, very straightforward.

In CouchDB 2.x and 3.x, the process has one more step:

1. request to open database `foo`
2. CouchDB looks in its internal database `/_node/_local/dbs` for the document 
`foo` (/_node/_local/dbs/foo)
  - The structure of that document is explained in 
https://docs.couchdb.org/en/stable/cluster/sharding.html#updating-cluster-metadata-to-reflect-the-new-target-shard-s
3. read the shard suffix and the shard range from the document (a single node 
database created with q=1 will only have one shard range -).
  - the suffix is in bytes, but they are a regular UNIX timestamp, you’ll have 
to convert it for use on the file system.
4. open this file now: `$database_dir/shards/-/foo.$suffix.couch

So to follow your procedure you need to:
1. create a document in the internal _dbs database with the same _id as the 
database name
2. put the .couch file in the place specified by the shard range and `shards/` 
subdirectory and match the shard suffix.

Voilá


Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

*24/7 Observation for your CouchDB Instances:
https://opservatory.app

> On 20. Oct 2021, at 14:56, Miroslav Rodic  wrote:
> 
> For an application, developed 10 years ago as couchapp with CouchDB 1.1.1, an 
> OFF-LINE data exchange (synchronization) is implemented by adding the local 
> .couch file, with randomly generated name, in a .zip file, which is then 
> sent, by email, to another user and then that user extracts and copies the 
> received .couch file into the local CouchDB instance. The imported .couch 
> file is then synchronized with the appropriate local database using 
> replication. All export/import steps, including the final replication, were 
> implemented within the application, so there were no problems with non-IT 
> users understanding how to use it. Since the application is implemented as a 
> couchapp type, the necessary Erlang functions have been developed that allow 
> archiving/copying/etc.
> 
> The question follows: is it possible to keep the same approach in off-line 
> synchronization with the latest version of CouchDB, which is 3.2.0, when the 
> installed CouchDB is configured as a single node? If this is not possible, 
> then which solutions are recommended for off-line synchronization?
> 
> Copying of .couch files is not clearly explained in the backup documentation. 
> A backup assumes, if I understood correctly, that .couch files can be copied 
> to a secure location, and then, when needed, can be restored by overwriting 
> the existing .couch files in the data directory. If, for any reason, the 
> corresponding DB is deleted by using the Fauxton before the .couch files are 
> reverted back, then reverted .couch files will not be accepted and displayed 
> in Fauxton. This is the behavior I had on my comp testing the backup/restore 
> procedure.
> 
> Backup/restore procedure tested on OS X 10.15.7 with CouchDB 3.2.0, where 
> path to data directory is as follows: ~/Library/Application 
> Support/CouchDB2/var/lib/couchdb.
> 
> Regards,
> Miroslav
> 

Re: Upgrade CouchDB on Mac

[Jan Lehnardt]

You can quit the existing Apache CouchDB app by using the menu bar item and 
selecting “Quit Apache CouchDB”. Then you can overwritee the application and 
start it again. No other task required.

Best
Jan
— 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

24/7 Observation for your CouchDB Instances:
https://opservatory.app

> On 12. Oct 2021, at 17:45, Vincent Lextrait  
> wrote:
> 
> Hello there,
> 
> I am trying to upgrade from CouchDB 3.1.1 to 3.2.0 on macOS. Drag & drop of 
> the unzipped app does not work, because CouchDB needs to be stopped first 
> (otherwise error because it is “in use”).
> 
> I did not find any resource on couchdb.apache.org to explain the process to 
> perform that clean stop. Did I miss it somewhere? There is an Upgrade entry 
> in the documentation, but it does not explain how to do that.
> 
> Thanks!
> 
> Vincent

[RELEASE] CouchDB 3.2.0

[Jan Lehnardt]

Dear community,

Apache CouchDB 3.2.0 has been released and is available for download.

https://couchdb.apache.org/#download

* * *

CouchDB 3.2.0 is a feature release and was originally published on 2021-10-12.

See the official release notes document for an exhaustive list of all changes:

http://docs.couchdb.org/en/stable/whatsnew/3.2.html#version-3.2.0

The community would like to thank all contributors for their part in making 
this release, from the smallest bug report or patch to major contributions in 
code, design, or marketing, we couldn’t have done it without you!

Release highlights:

- The couch_sever module is now sharded. Despite following a high-concurrency 
process architecture generally, the couch_server module used in previous 
versions is a single Erlang process that, on busy nodes, could become a 
bottleneck. CouchDB 3.2.0 introduces a couch_server_N module per CPU core, 
effectively removing the bottleneck.
- The replication scheduler manages which replications run at any one time. It 
is important for setups where there are more total replications than are 
configured to run concurrently. Previously, the replication scheduler would 
iterate over all replications in a round-robin fashion and give them all equal 
time to do their work. CouchDB 3.2.0 introduces a fair-share option that allows 
you to use multiple _replicator databases each with a different relative 
priority, so your important replications get more time to do their job vs. your 
less important replications. See the _replicator DB docs for more details: 
https://docs.couchdb.org/en/stable/replication/replicator.html#replicator
- Support for Erlang versions 23 and 24, dropped support for version 19.
- Support for SpiderMonkey versions 78 and 86.
- Addresses CVE-2021-2838295
- Reduced occurrence of the “No DB shards can be opened”
- Support specifying password requirements via regex
- Logs no longer include credentials in almost all cases
- More fine-grained CSP configuration
- Easier development setups via .devcontainer for the 3.x series.
- Makes auto-compaction less aggressive, saves CPU and I/O on busy clusters.
- Includes weatherreport module for advanced diagnostics:
- 
https://docs.couchdb.org/en/latest/cluster/troubleshooting.html#cluster-troubleshooting
- Includes a dedicated prometheus endpoint for stats and metrics:
- 
https://docs.couchdb.org/en/latest/api/server/common.html#api-server-prometheus
- All JS tests have been migrated Elixir and the JS test suite has been 
retired. 

Find the full list of changes here: 
https://docs.couchdb.org/en/latest/whatsnew/3.2.html

* * *

Apache CouchDB™ lets you access your data where you need it. The Couch 
Replication Protocol is implemented in a variety of projects and products that 
span every imaginable computing environment from globally distributed 
server-clusters, over mobile phones to web browsers.

Store your data safely, on your own servers, or with any leading cloud 
provider. Your web- and native applications love CouchDB, because it speaks 
JSON natively and supports binary data for all your data storage needs.

The Couch Replication Protocol lets your data flow seamlessly between server 
clusters to mobile phones and web browsers, enabling a compelling offline-first 
user-experience while maintaining high performance and strong reliability. 
CouchDB comes with a developer-friendly query language, and optionally 
MapReduce for simple, efficient, and comprehensive data retrieval.

Pre-built packages for Windows, macOS, Debian/Ubuntu and RHEL/CentOS are 
available.

https://couchdb.apache.org/#download

[RELEASE] CouchDB 3.1.2

[Jan Lehnardt]

Dear CouchDB community,

Apache CouchDB® 3.1.2 has been released and is available for download.

CouchDB 3.1.2 is a security release for a low severity security issue, and was 
originally published on 2021-10-05.

Details for the security issue will be published one week after this release. 
See the CVE database[1] for updates at that time.

Apache CouchDB® lets you access your data where you need it. The Couch 
Replication Protocol is implemented in a variety of projects and products that 
span every imaginable computing environment from globally distributed 
server-clusters, over mobile phones to web browsers.

Store your data safely, on your own servers, or with any leading cloud 
provider. Your web- and native applications love CouchDB, because it speaks 
JSON natively and supports binary data for all your data storage needs.

The Couch Replication Protocol lets your data flow seamlessly between server 
clusters to mobile phones and web browsers, enabling a compelling offline-first 
user-experience while maintaining high performance and strong reliability. 
CouchDB comes with a developer-friendly query language, and optionally 
MapReduce for simple, efficient, and comprehensive data retrieval.

https://couchdb.apache.org/#download

Pre-built packages for Windows, macOS, Debian/Ubuntu and RHEL/CentOS are 
available, along with the apache/couchdb and couchdb Docker containers.

https://docs.couchdb.org/en/3.1.2/whatsnew/3.1.html

The community would like to thank all contributors for their part in making 
this release, from the smallest bug report or patch to major contributions in 
code, design, or marketing, we couldn’t have done it without you!

[1]: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-38295

—

Re: Per subdomain DBs with Node

[Jan Lehnardt]

Hi Rick,

> On 5. Jul 2021, at 18:14, Rick Jarvis  wrote:
> 
> That looks like a good addition. Couple of quick questions on that:
> 
> Does that keep the socket open indefinitely whilst the Node process is 
> running, and if so does it create much benefit?
> 
> Is ’nano.db.use’ actually opening the socket to the DB then? And how many 
> sockets is a reasonable expectation (I guess that’s how long is a piece of 
> string though?!)?


The exact socket handling I’m not sure about, you’d have to look at how nano is 
actually implemented. All this will also be affected by your keepalive 
settings, so the answer is probably something you’ll have to look for yourself, 
but you should have all pointers now :)


> I guess I could make it ‘global.dbs’ and access the DBs easily through other 
> modules etc then?

Yeah that works too. In our case, all database instances are created in the 
same module that all other modules then use to get their instances from that 
one central module, so the “cache” can be module-global for us. Maybe that 
pattern also works for you. I hear people have strong opinions about global 
variables.

Best
Jan
—

> 
> Thanks!
> 
> 
> 
> From: Jan Lehnardt 
> Reply: user@couchdb.apache.org 
> Date: 5 July 2021 at 10:34:29
> To: user@couchdb.apache.org 
> Subject:  Re: Per subdomain DBs with Node  
> 
> Hi Rick,  
> 
> your original implementation strikes me as better than the other one.  
> 
> Since storing something in Redis means serialising it, you won’t get the 
> benefit of any sockets being kept open, and I’m not even sure you can make 
> that work properly.  
> 
> If you are worried about having a db instance open per request vs 
> once-per-server-per-user, how about this slight modification to your original 
> pattern. This is something we’re using in Opservatory* for the same reason:  
> 
> let dbs = {}  
> 
> function getDb(orgname) {  
> if (!dbs[orgname] {  
> // cache db instance if not already cached  
> dbs[orgname] = nano.db.use(orgname);  
> }  
> // return cached db instance  
> return dbs[orgname]  
> }  
> 
> app.use(function(req, res, next) {  
> const orgname = req.headers.host.split('.')[0].toLowerCase();  
> req.couch = getDb(orgname)  
> next();  
> }  
> 
> If you have very long running servers and high user churn, you also want to 
> make sure you can drop cached instances from `dbs` when a user gets deleted. 
> Or you add a max length check to this and drop the least recently used 
> instance, but you get the idea.  
> 
> Best  
> Jan  
> —  
> 
> Professional Support for Apache CouchDB:  
> https://neighbourhood.ie/couchdb-support/  
> 
> *24/7 Observation for your CouchDB Instances:  
> https://opservatory.app  
> 
>> On 4. Jul 2021, at 19:02, Rick Jarvis  wrote:  
>> 
>> Hey all  
>> 
>> I have an app which serves multiple orgs with individual databases, by way 
>> of subdomain, ORGNAME.myapp.foo . My current method of attaching the 
>> database to the ExpressJS requests via middleware is like this:  
>> 
>> app.use(function(req, res, next) {  
>> const orgname = req.headers.host.split('.')[0].toLowerCase();  
>> req.couch = nano.db.use(orgname);  
>> next();  
>> }  
>> 
>> I often wonder if this is the best way of doing it. Is there a better way?  
>> 
>> I could for instance, attach it just once, to the session (stored in Redis): 
>>  
>> 
>> app.use(function(req, res, next) {  
>> if(!req.session.couch) {  
>> const orgname = req.headers.host.split('.')[0].toLowerCase();  
>> req.session.couch = nano.db.use(orgname);  
>> }  
>> next();  
>> }  
>> 
>> But I guess that depends on size of the Couch instance (is it just metadata, 
>> or something more tangible?), and other aspects which I perhaps haven’t 
>> considered.  
>> 
>> I’d love to hear some thoughts on this?  
>> 
>> Thank you!  
>> R  

Re: Per subdomain DBs with Node

[Jan Lehnardt]

Hi Rick,

your original implementation strikes me as better than the other one.

Since storing something in Redis means serialising it, you won’t get the 
benefit of any sockets being kept open, and I’m not even sure you can make that 
work properly.

If you are worried about having a db instance open per request vs 
once-per-server-per-user, how about this slight modification to your original 
pattern. This is something we’re using in Opservatory* for the same reason:

let dbs = {}

function getDb(orgname) {
  if (!dbs[orgname] {
// cache db instance if not already cached
dbs[orgname] = nano.db.use(orgname);
  }
  // return cached db instance
  return dbs[orgname]
}

app.use(function(req, res, next) {
  const orgname = req.headers.host.split('.')[0].toLowerCase();
  req.couch = getDb(orgname)
  next();
}

If you have very long running servers and high user churn, you also want to 
make sure you can drop cached instances from `dbs` when a user gets deleted. Or 
you add a max length check to this and drop the least recently used instance, 
but you get the idea.

Best
Jan
—

Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

*24/7 Observation for your CouchDB Instances:
 https://opservatory.app

> On 4. Jul 2021, at 19:02, Rick Jarvis  wrote:
> 
> Hey all
> 
> I have an app which serves multiple orgs with individual databases, by way of 
> subdomain, ORGNAME.myapp.foo . My current method of attaching the database to 
> the ExpressJS requests via middleware is like this:
> 
> app.use(function(req, res, next) {
>   const orgname = req.headers.host.split('.')[0].toLowerCase();
>   req.couch = nano.db.use(orgname);
>   next();
> }
> 
> I often wonder if this is the best way of doing it. Is there a better way?
> 
> I could for instance, attach it just once, to the session (stored in Redis):
> 
> app.use(function(req, res, next) {
>   if(!req.session.couch) {
>   const orgname = req.headers.host.split('.')[0].toLowerCase();
>   req.session.couch = nano.db.use(orgname);
>   }
>   next();
> }
> 
> But I guess that depends on size of the Couch instance (is it just metadata, 
> or something more tangible?), and other aspects which I perhaps haven’t 
> considered.
> 
> I’d love to hear some thoughts on this?
> 
> Thank you!
> R

Re: Proxying document updates with update handlers

[Jan Lehnardt]

Hi Aurélien,

we generally recommend doing this kind of stuff outside of CouchDB,
these days.

A Node.js proxy that does this completely and reliably is maybe 50
lines of code and not a huge operational overhead, while granted not
as neat as doing all this inside of CouchDB.

As for getting the new _rev in the function: the rev will be generated
from the result that the function returns, so there is no way to get
that other than calculating it yourself (it is deterministic), but
that requires knowledge of erlang term encoding and such things. I’ve
done it in JS (for other things), but it is not pretty.

Best
Jan
— 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

24/7 Observation for your CouchDB Instances:
https://opservatory.app

> On 27. May 2021, at 22:08, Aurélien Bénel  wrote:
> 
> Dear all,
> 
> I have known update handlers for quite long but I never used them "for real" 
> myself... My current idea, which must be very common, is to proxy updates of 
> whole documents in order to add some accountability of who contributed to the 
> document and when.
> 
># rewrites.json
>[{
>   "from": "",
>   "to": "elsewhere",
>   "method": "GET"
> }, {
>  "from": "",
>  "to": "_update/accounting"
>}, {
>   "from": ":object",
>  "to": "../../:object",
>   "method": "GET"
>}, {
>   "from": ":object",
>   "to": "_update/accounting/:object"
>}]
> 
># accounting.js
>function(doc, req) {
>  var o = JSON.parse(req.body);
>  o._id = o._id || req.id || req.uuid;
>  var h = doc && doc.history || [];
>  h.push({
>user: req.userCtx.name,
>timestamp: new Date()
>  });
>  o.history = h;
>  return [o, {json: {ok: true, id: o._id }}];
>}
> 
> Tested on CouchDB 2.3.1, it *nearly* emulates the direct update of a document 
> and adds contributions accounting, however I face two problems ;
> 
> 1. In the update handler, I see no way to get the new `_rev` value  (which 
> should be returned either in the JSON body or as an ETag for compatibility 
> with normal update of an object). Is there a secret builtin function that 
> could be used to get (or set) this? Or is it set afterwards and then cannot 
> be get or set at this stage of the process?
> 
> 2. In the update handler, when used with POST (with the `_id` in the body but 
> not in the URI),  it seems that `doc` is always null (even when the ID refers 
> to an existing document)... Is this behaviour intended? I feel that the 
> documentation could be interpreted both ways... 
> Of course, we can still use PUT. But I wanted a complete emulation of normal 
> updates (with both methods)...
> 
> Any help or insight would be appreciated.
> 
> 
> Regards,
> 
> Aurélien
> 
> 

Re: Changes to 'this' in update function are visible in validate_doc_update. Bug or Feature?

[Jan Lehnardt]

Hi Stefan,

you absolutely should and can not rely on this behaviour, it might even
not be available in setups with other (more modern) JS interpreters.

Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

24/7 Observation for your CouchDB Instances:
https://opservatory.app

> On 26. May 2021, at 10:17, Stefan Klein  wrote:
> 
> Hi all,
> 
> i'm trying to prevent unintended changes to some documents.
> 
> While toying around with update & validate_doc_update functions, I found
> that, if i add a property to the `this` context of an update function, the
> `this` context of the validate_doc_update running after the update contains
> that property.
> 
> If I understand the couchdb & query server architecture correct I can not
> rely on this behavior. Or can I?
> 
> Thanks & regards,
> Stefan Klein

Re: Data backup in CouchDB cluster

[Jan Lehnardt]

Hi Simon,

there are multiple aspects to backup, data safety and
time to recovery (TTR).

If your only goal is to have a separate copy of your
data, backing up only one node of your database does
the trick.

However, if you want a short TTR, so your cluster is
complete again as soon as possible, you want a backup of
each of your nodes, so you can replay your backup at
any time. Each node stores the data slightly differently
on a physical level, which means restoring data from
node A’s backup to node B is not trivial. While logically,
this means you have three backups when you might only
want one.

If TTR is not a concern, you can just back up a single
node to make sure you can restore that if needed, but rely
on CouchDB internal backfill, if a node is replaced after a
failure without data being added from a backup. This will
take longer than restoring a node from its own backup.

Best
Jan
— 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

24/7 Observation for your CouchDB Instances:
https://opservatory.app


> On 7. May 2021, at 11:27, Simon Schwichtenberg  
> wrote:
> 
> Hi,
> 
> I wonder how you'd do backups of your data in a CouchDB cluster. The 
> documentation does not mention backups of clusters explicitly 
> (https://docs.couchdb.org/en/latest/maintenance/backups.html#database-backups).
> 
> When you have a cluster of three nodes and the nodes are set to n=3 and q=2 
> (see https://docs.couchdb.org/en/latest/cluster/sharding.html), I'd expect 
> that every single node in the cluster has all the data and you can copy the 
> .couch files from any of these three nodes. When you have 6 nodes with n=3 
> and q=2 this approach does not work anymore because every node has just a 
> single shard. Please correct me if I am wrong.
> 
> What is best practice to backup a cluster?
> 
> This message is a follow-up from here: 
> https://github.com/cloudant/couchbackup/issues/349
> 
> Thanks,
> Simon

Re: New CouchDB CLI tool

[Jan Lehnardt]

Heya Jonathan,

this is really cool, congrats on getting this together!

And thank you for sharing :)

Best
Jan
—

> On 27. Apr 2021, at 22:25, Jonathan Hall  wrote:
> 
> Good day everyone!
> 
> I'd like to announce the "alpha" release of a new CLI tool for interacting 
> with CouchDB.  The tool is designed to replace `curl` as a tool for 
> interacting with the CouchDB API for administrative and debugging tasks.  
> Further, it adds offers the ability to replicate between CouchDB servers and 
> local filesystem directories, thus facilitating bootstrapping of CouchDB 
> servers.
> 
> Read the full announcement here: http://kivik.io/kivik-cli-pre-release
> 
> Download binaries for common architectures here: 
> https://github.com/go-kivik/xkivik/releases
> 
> There's still some work to be done, and there are no doubt some rough edges 
> and bugs. I welcome any feeback!
> 
> Jonathan
> 
> 

Re: Upgrade 2.3.1 on Mac osx

[Jan Lehnardt]

Yup, that will only delete the binary code that makes up CouchDB, but none of 
your data or configuration.

> On 14. Apr 2021, at 11:43, Jonathan Aquilina 
>  wrote:
> 
> Hi Jan,
> 
> So to uninstall its just a matter of dragging couchdb icon from applications 
> to trash bin and that is it?
> 
> 
> 
> -----Original Message-
> From: Jan Lehnardt  
> Sent: 14 April 2021 11:38
> To: user@couchdb.apache.org
> Subject: Re: Upgrade 2.3.1 on Mac osx
> 
> The 3.x binary will just load your existing 
> ~/Library/Preferences/couchdb2-local.ini and inherit all data file paths and 
> configurations from there. So you data will just show up and no replication 
> or other migration is necessary.
> 
> Best
> Jan
> --
> 
> 
>> On 14. Apr 2021, at 10:56, Jonathan Aquilina 
>>  wrote:
>> 
>> Hi Jan,
>> 
>> Forward is fine this was for local testing only in terms of a script at the 
>> time. Now I am trying to do some development with .net core and blazor apps 
>> and want to take advantage of couch 
>> 
>> So if I delete 2.3.1 what happens with the data 3.1.1 will auto find it or 
>> should I first replicate the data to 3.1.1 prior to decommissioning 2.3.1
>> 
>> Regards,
>> Jonathan
>> 
>> 
>> 
>> -Original Message-
>> From: Jan Lehnardt  
>> Sent: 14 April 2021 10:45
>> To: user@couchdb.apache.org
>> Subject: Re: Upgrade 2.3.1 on Mac osx
>> 
>> Hi Jonathan,
>> 
>> you can just install 3.1.1 and delete your 2.3.1 app. The 3.1.1 will 
>> automatically use your existing data.
>> 
>> Note that you can’t just go back, only forward.
>> 
>> Best
>> Jan
>> —
>> 
>>> On 13. Apr 2021, at 21:48, Jonathan Aquilina 
>>>  wrote:
>>> 
>>> Good Evening,
>>> 
>>> I have a quick question how does one go about upgrading from 2.3.1 to the 
>>> latest version 3.1.1 do I need to uninstall the current version I have 
>>> installed on my mac or can I install 3.1.1 along side 2.3.1 and replicate 
>>> the databases to it.
>>> 
>>> Regards,
>>> Jonathan
>> 
> 

Re: Upgrade 2.3.1 on Mac osx

[Jan Lehnardt]

The 3.x binary will just load your existing 
~/Library/Preferences/couchdb2-local.ini and inherit all data file paths and 
configurations from there. So you data will just show up and no replication or 
other migration is necessary.

Best
Jan
--


> On 14. Apr 2021, at 10:56, Jonathan Aquilina 
>  wrote:
> 
> Hi Jan,
> 
> Forward is fine this was for local testing only in terms of a script at the 
> time. Now I am trying to do some development with .net core and blazor apps 
> and want to take advantage of couch 
> 
> So if I delete 2.3.1 what happens with the data 3.1.1 will auto find it or 
> should I first replicate the data to 3.1.1 prior to decommissioning 2.3.1
> 
> Regards,
> Jonathan
> 
> 
> 
> -Original Message-
> From: Jan Lehnardt  
> Sent: 14 April 2021 10:45
> To: user@couchdb.apache.org
> Subject: Re: Upgrade 2.3.1 on Mac osx
> 
> Hi Jonathan,
> 
> you can just install 3.1.1 and delete your 2.3.1 app. The 3.1.1 will 
> automatically use your existing data.
> 
> Note that you can’t just go back, only forward.
> 
> Best
> Jan
> —
> 
>> On 13. Apr 2021, at 21:48, Jonathan Aquilina 
>>  wrote:
>> 
>> Good Evening,
>> 
>> I have a quick question how does one go about upgrading from 2.3.1 to the 
>> latest version 3.1.1 do I need to uninstall the current version I have 
>> installed on my mac or can I install 3.1.1 along side 2.3.1 and replicate 
>> the databases to it.
>> 
>> Regards,
>> Jonathan
> 

Re: Upgrade 2.3.1 on Mac osx

[Jan Lehnardt]

Hi Jonathan,

you can just install 3.1.1 and delete your 2.3.1 app. The 3.1.1 will 
automatically use your existing data.

Note that you can’t just go back, only forward.

Best
Jan
—

> On 13. Apr 2021, at 21:48, Jonathan Aquilina 
>  wrote:
> 
> Good Evening,
> 
> I have a quick question how does one go about upgrading from 2.3.1 to the 
> latest version 3.1.1 do I need to uninstall the current version I have 
> installed on my mac or can I install 3.1.1 along side 2.3.1 and replicate the 
> databases to it.
> 
> Regards,
> Jonathan

Re: Single vs user db's

[Jan Lehnardt]

> On 8. Apr 2021, at 19:53, Olaf Krueger  wrote:
> 
> 
>> Also, I don’t quite get what “re-creating” a database would entail.
> 
> Sorry, it seems to me that doesn't makes sense.
> While I wrote this I had the docs in my mind [1]:
> 
>> If your use case creates lots of deleted documents (for example, if you are 
>> storing short-term data like >log entries, message queues, etc), you might 
>> want to periodically switch to a new database and >delete the old one (once 
>> the entries in it have all expired).
> 
> But re-reading this it turns out to me that this only makes sense if we don't 
> need to keep the remaining docs.
> 
>> Deleting a doc creates a tombstone, that’s where just deleting the user db 
>> is easier, you don’t want to collect too many tombstones.
> 
> Just out of curiosity:
> Would be the purge feature [2] a way out here?

Yes, but it comes with other caveats. It is best used for “I accidentally 
committed a credit card number to CouchDB, let’s undo that” and less “clean up 
1000s of old records every day” type of operations.

It’ll work in a pinch, but if you are starting out, db-per-time allows you to 
avoid getting into a tight spot to begin with :)

Best
Jan
—
> 
> Thanks,
> Olaf
> 
> 
> [1] 
> https://docs.couchdb.org/en/stable/maintenance/performance.html#delete-operation
> [2] https://docs.couchdb.org/en/latest/api/database/misc.html?highlight=purge

Re: Single vs user db's

[Jan Lehnardt]

> On 8. Apr 2021, at 13:56, Olaf Krueger  wrote:
> 
> Hi, 
> 
>> ... I will be required to be able to delete all data for a departing 
>> customer
> 
> Wouldn't it be a valid option to remove all documents which are related to a 
> particular customer from a single database and (if needed) re-create the 
> database afterwards in order to get rid of the revisions or "tombstones”?

Deleting a doc creates a tombstone, that’s where just deleting the user db is 
easier, you don’t want to collect too many tombstones.

Also, I don’t quite get what “re-creating” a database would entail.


> Using meaningful _ids instead of anonymous uuids could be helpful here, e.g.
> 
> customer:1
> customer:1.address
> customer.1.meeting.1.scheduling
> customer.1.meeting.1.result
> customer.1.meeting.2.result
> 
> customer:2
> customer:2.address
> customer.2.meeting.1.scheduling
> customer.2.meeting.1.result
> customer.2.meeting.2.result
> 
> By using the leading string of the _id, e.g. "customer:1", you could fetch or 
> delete all relations for a particular customer.
> Moreover, with the knowledge of the customer-id and meeting-id, you can 
> easily compose the _id of each document in order to directly fetch it by its 
> _id.

Good example, that’s what I meant with:

> - you can get all data per user in one request
>  - either via _all_docs if your doc _ids are prefixed with the user id

Best
Jan

Re: Single vs user db's

[Jan Lehnardt]

Hi Sam,


> On 8. Apr 2021, at 12:24, Sam Lown  wrote:
> 
> - create databases for data which will grow infinitely like calendar
> entries or anything that has time as a primary key.

Excellent additions, and one more point on this in particular:

- database-per-time-unit (say one per month), especially in offline-replication 
scenarios, but also useful otherwise.

This allows you do just delete or stop replicating databases older than needed, 
and make, say, an archive feature an online-only feature in the app, while 
“current” data stays available for offline replication.

And if you have loads of data and accumulating it fast, some of that data might 
be less important than other data, so you can easily move “archive” data to a 
server or cluster that has less performance but higher data storage capacity, 
while your “current” data cluster skews towards performance.

But all that incurs management cost, and is usually a “next level” 
optimisation, but if you start with, say, database-per-month now, it is easy to 
implement when you need it.

Best
Jan
—

Re: Single vs user db's

[Jan Lehnardt]

Hi Kiril,

first, this is a good question :)

I’d like to add, that you are missing one more approach to consider: use a 
single database.

Let’s first look at the other approaches, and I’ll start with database-per-type:

- there is no real benefit of doing this, other than theoretical purity
- it won’t allow you to query across types, say: give me all or a part of types 
for a single user, which is, I assume, going to be a very common thing
- to get to all types for a user, you have to make a request to each database
- the more types you get, the more queries you have to make
- reducing the number of queries, and reducing the number of databases you need 
to talk to to get your most common operation is beneficial

So I would recommend against that. I know of some apps using this pattern 
successfully, but that’s because they have very strong separation of types, so 
they will never have to query more than one type at a time. If someone else 
reading this who is happy with their setup this way, I won’t begrudge them, but 
I would *not* recommend anyone starts with this pattern.

Next, database-per-user:

- you always clearly know where a user’s data is. Wanna delete a user: delete 
their db. Easy for making sure data is separated from other users, easy to 
comply with legislation about data deletion, etcpp.
- if you plan to replicate the data onto a per-user device (like an offline web 
app, or native app), access control in CouchDB is such that it is per database. 
So if that’s a requirement for you, then you are already locked into 
database-per-user. If not, it is still neat from a data organisation point of 
view.
- downsides include not being able to query across multiple user (how many 
address book entries are there in total, what’s the median per user, etc). 
Sometimes this doesn’t matter, sometimes it is important.
- there are workarounds to query across all databases (replicate all individual 
databases into a central big database for querying). Again, this is usually the 
case for setups where there needs to be per-user replication. But it is a 
workaround.
- having many small database requires a bit more work on CouchDB’s side when 
dealing with many concurrent users being active

Finally: single database:

- you can query across all users
- you can get all data per user in one request
  - either via _all_docs if your doc _ids are prefixed with the user id
  - or a view/mango by user-id
- you can’t authenticate/replicate by-user (but if you don’t have that 
requirement, that’s not a downside)
- database sharding scales up linearly with your data usage
- concurrent users can be handled very efficiently
- partitioned databases with a partition key of the user id gives you 
additional benefits at larger scales (likely matters less with 1000 users), but 
ymmv, and you know the option is there for later

* * *

Given the situation you described, I’d strongly recommend a single database for 
this app, because the benefits are overwhelming.

I’d strongly recommend against using db-per-type, just because it doesn’t gain 
you anything practically.

If you need per-user replication, db-per-user is your only choice. It has 
drawbacks, but no show stoppers at the data sizes you quote.

HTH,
Jan
—

> On 8. Apr 2021, at 10:07, Kiril Stankov  wrote:
> 
> Hi all,
> 
> I have a design question.
> Lets say that my solution will serve hundreds (or even more than 1000)
> customers.
> For each customer I will need 4 to 6 sets of objects, say: assets,
> address book, meetings and meeting results.
> I wonder what would be best approach here:
> - have userDB's for each customer with documents identified by type
> (this means potentially thousands of db's), or:
> - have 6 db's, one per document type and filter by customer ID.
> 
> There might be thousands of documents of each type per customer,
> eventually bringing the total number of documents to millions.
> 
> How each of the approaches above will affect clustering and what will be
> the best cluster setup.
> In terms of performance and indexing, does one of the designs
> outperforms the other?
> 
> Thanks in advance,
> Kiril.

Re: [Eventual consitency] Is the uniqness of the _id guaranteed across nodes within a cluster

[Jan Lehnardt]

Hi Olaf,

> On 13. Dec 2020, at 11:13, Olaf Krueger  wrote:
> 
> Hi again,
> 
> we're working on a booking app.
> In order to prevent over booking of a particular slot (A slot can booked only 
> once), it's crucial to know if a slot is already booked or even not.
> 
> We're using a cluster of 3 CouchDB nodes, so having the eventual consistency 
> issue in mind the question is, if it's possible to basically achieve the 
> above requirement.
> 
> The idea is to create an own document for each booking by using a custom _id, 
> e.g.
> {
>   _id: = "slot1:booked 
> }
> 
> But that would probably only work if CouchDB guarantees the uniqness across 
> all nodes.
> Is that the case?
> Or do we have to accept that consistency is sacrificed in favour of 
> availability?

Yes.

> Or do we need to think about using another DB which sacrifies availability in 
> favor of consistency?

Unless you can resolve a double-booking after the fact, yes. A memcached or 
redis instance is often used in conjunction with CouchDB, but other 
unique-id-register mechanisms exists.

Best
Jan
—
> 
> Many thanks!
> Olaf
> 
> 

Re: Mulltiple gets on same databases

[Jan Lehnardt]

Hi Paul,

this is usually a networking issue, not a CouchDB issue.

What OS are you running this on?

What are your open file ulimits for the requesting and the CouchDB process?

Try running a netstat to see the state of open sockets during the benchmark 
progress

Best
Jan
—

> On 22. Sep 2020, at 10:45, Paul Milner  wrote:
> 
> Hello
> 
> I make a limited number of get requests simultaneously on my database and
> everything runs fine. I see the requests in the couchdb log. However when I
> increase the number of requests to a larger number say 1‘000 couchdb stops
> and I have to restart it. I see no entries in the log.
> 
> Can you tell me if there‘s an appropriate setting in the setup to increase
> to make this work please?
> 
> Thanks a lot for your help
> Paul


-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

24/7 Observation for your CouchDB Instances:
https://opservatory.app

Announcing Opservatory: 24/7 CouchDB Observation and Analysis

[Jan Lehnardt]

Dear CouchDB Community,

I’m happy to announce Opservatory https://opservatory.app, my company
Neighbourhoodie’s latest product for monitoring your CouchDB instances.

Opservatory knows CouchDB better than any single human could. We’ve put
our combined multi-decade experience with supporting CouchDB
installations of all shapes into one tool that makes sure you are not
running into any trouble, ever.

It continuously monitors your CouchDB installations and runs a number
of checks to ensure your CouchDBs are running correctly, fast and
securely. It also checks for conditions and configurations that have
the potential to cause production issues in the future.

It compares metrics against settings to make real-time recommendations
for improving your CouchDB instance or cluster. It evaluates
configuration, metrics, and database information to ensure smooth
operation. It even evaluates your JavaScript code in design documents
for common mistakes and performance pitfalls.

Here are just a few examples of the checks that Opservatory runs periodically:

  - If you are just starting out with CouchDB, Opservatory checks if
you’ve got any quirks in your setup and configuration and gives you
recommendations for well-established best practices for running CouchDB
correctly, fast and securely.

  - If your production cluster ran into trouble and you don’t know
where to start looking for things to fix, Opservatory will give you the
top list of issues that you can resolve to make your problem(s) go away.

  - If your intern uploaded a design doc with an inefficient map
function (again), Opservatory will let you know, but it also catches
more common patterns used by seasoned CouchDB developers that can be
improved.

  - If you create a new database and forget to set the correct
_security object, Opservatory will warn you about publicly accessible
data that you might want to make private as soon as you get the
notification.

Opservatory has over 80 checks that cover the five main areas of every
CouchDB installation: setup, configuration, metrics, databases and
queries/design docs. We have at least as many more in the backlog, and
we are constantly adding more as we find more issues during our
professional support service work.

You can configure Opservatory to receive alerts via email or Slack, and
it covers all CouchDB versions: 1.x, 2.x and 3.x. It even works with
Cloudant!

And while we are very happy with where we are already, we have big
plans to make maintaining CouchDB in production even easier with
features round capacity planning and seamless growth as well as helping
to automate common maintenance tasks safely.

If you have any questions, I’m happy to answer off-list.

If you think this is interesting, we welcome you to sign up for a free
trial at https://opservatory.app.

Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

24/7 Observation for your CouchDB Instances:
https://opservatory.app

Re: Local-First apps and CouchDB

[Jan Lehnardt]

I came to similar conclusions while reading the JSON CRDT paper by the same
author. It closes with essentially “garbage collection is TBD”, which makes
it also not feasible in practice for the same reasons ermouth outlines.

Best
Jan
—

> On 7. Aug 2020, at 03:08, ermouth  wrote:
> 
> Both CRDT and OT (operational transform) are implementable over CouchDB.
> CRDT approach is ok for Couch/Pouch built-in MapReduce mechanics, esp if
> complicated combining reducers are executed client-side (ie no 64Mb for JS
> instance limitation). Best results however are achieved if a finalizing
> part of a reducer is put out of Couch/Pouch.
> 
> As anything ‘magic’, CRDT collab scheme, esp if there’s a goal to implement
> a text editor, has its price.
> 
> The entire text should be split into lower-level atomic blocks, say words,
> or even characters, and each should be stored as a separate entity inside
> JSON obj or array. Each action (insert/delete of an entity) is then stored
> as a separate doc, then distributed over collaborators using _changes, and
> then re-combined at each client. So, half a kilobyte text easily bloats
> into hundreds of docs and hundreds of kilobytes of JSON data.
> 
> Since we store each modification as a separate doc, there’s absolutely no
> way we can get conflicting doc revisions, so the problem of conflicting
> docs just never happens.
> 
> I’ve tested the CRDT collab approach over CouchDB, found it viable, however
> dropped it for enormous storage overhead. So, excessive storage is the
> problem, not conflicts or speed; however, bloating is inevitable with CRDT.
> 
> ermouth
> 
> 
> чт, 6 авг. 2020 г. в 01:09, Bill Stephenson :
> 
>> There have been 3 submissions of a paper about "Local-First" apps (
>> https://www.inkandswitch.com/local-first.html <
>> https://www.inkandswitch.com/local-first.html>) to Hacker News in the
>> past week. Twice it made it to the front page and the most recent was there
>> most all day and was still on the 2nd page the next morning.
>> 
>> The paper mentions and discusses CouchDB but I think it drifts quickly
>> from the subject of "Local-First” apps to conflict resolution and
>> overlooked using CouchDB as a data storage layer for Local-First apps.
>> 
>> I thought it’s worth mentioning here because CouchDB makes it easy for
>> both devs and users to create and install a Local-First app. A couple
>> months ago I made a very simple demo app that demonstrates one way to do
>> that. I posted a link to it in the comments on those HN posts and it seemed
>> to generate a little interest so I thought I’d share it here too...
>> 
>> https://cherrypc.com/app/editor/setup.html <
>> https://cherrypc.com/app/editor/setup.html>
>> 
>> --
>> 
>> Bill Stephenson
>> 
>> 
>> 
>> 
>> 
>> 

Re: Local-First apps and CouchDB

[Jan Lehnardt]

The author of this sadly has demonstrated multiple times that they don’t 
understand CouchDB very well, and despite my outreach, wasn’t open to learning 
any more. I’m not sure if this is a personal choice, or a reflection of few 
scientific papers existing around CouchDB, but it’s worth noting.

Best
Jan
—

> On 6. Aug 2020, at 00:09, Bill Stephenson  wrote:
> 
> There have been 3 submissions of a paper about "Local-First" apps 
> (https://www.inkandswitch.com/local-first.html 
> ) to Hacker News in the past 
> week. Twice it made it to the front page and the most recent was there most 
> all day and was still on the 2nd page the next morning. 
> 
> The paper mentions and discusses CouchDB but I think it drifts quickly from 
> the subject of "Local-First” apps to conflict resolution and overlooked using 
> CouchDB as a data storage layer for Local-First apps.
> 
> I thought it’s worth mentioning here because CouchDB makes it easy for both 
> devs and users to create and install a Local-First app. A couple months ago I 
> made a very simple demo app that demonstrates one way to do that. I posted a 
> link to it in the comments on those HN posts and it seemed to generate a 
> little interest so I thought I’d share it here too...
> 
> https://cherrypc.com/app/editor/setup.html 
> 
> 
> --
> 
> Bill Stephenson
> 
> 
> 
> 
> 

Re: Is this mailing list obsolete now?

[Jan Lehnardt]

> On 12. Jul 2020, at 22:07, Miles Fidelman  wrote:
> 
> Well, as a user, I kind of find that the move to a github discussion is not 
> at all helpful.  Email lists are a time-proven mechanism - why break what 
> ain't broke.

This mailing list isn’t going anywhere.

Best
Jan
—
> 
> Miles Fidelman
> 
> On 7/12/20 3:43 PM, Kiril Stankov wrote:
>> Hi,
>> 
>> I see that some topics on the list are not in the github discussions and
>> vice versa?
>> 
>> Shall we all consider the mailing list obsolete and move to github?
>> 
>> Is there a way to get summaries from github or other kind of
>> notifications by email?
>> 
>> Cheers,
>> 
>> Kiril.
> 
> -- 
> In theory, there is no difference between theory and practice.
> In practice, there is.   Yogi Berra
> 
> Theory is when you know everything but nothing works.
> Practice is when everything works but no one knows why.
> In our lab, theory and practice are combined:
> nothing works and no one knows why.  ... unknown
> 

Re: Fastest way to get a doc _rev

[Jan Lehnardt]

Heya Garren,

great point, I’ve adjusted things to measure from last row sent in the request
to the end of the response[1], and update the results (not much of a difference
overall), numbers go up a little, but the doc size seems to make little 
difference.

Note that this is a rather speed SSD system here (recent Mac mini) and fast 
CPUs.

I’m running the tests many times to avoid caching artefacts, but that means all
access should be cached. So you’d likely see more differences on slower disk and
for uncached requests.

[1]: behold the bashism: RESP=`curl -sv --trace-time 
http://admin:admin@127.0.0.1:5984/benchbulk-1/_all_docs?key=\"0050\;
 2>&1`; SENT_TS=$(echo "$RESP" | grep -Eo \(.\{15\}\)\ \>\ Accept | sed -e 's/ 
> Accept//' | sed -e 's/[:.]//g'); END_TS=$(echo "$RESP" | grep -Eo 
\(.\{17\}\)\ ?\ Connection | sed -e 's/\ \*\ Connection//' | sed -e 
's/[.:]//g'); echo $(($END_TS-$SENT_TS))

> On 12. Jul 2020, at 16:58, Garren Smith  wrote:
> 
> Hi Jan,
> 
> That is a really interesting experiment. I was trying to benchmark
> _all_docs recently and I've noticed is that it will stream the results, so
> it returns the header and the start of the body before its done any actual
> work. I'm not 100% sure if that is the case when `key=` is used. You might
> have to adjust your benchmark to check for the first `{` to signify the
> start of a document.
> 
> Cheers
> Garren
> 
> On Sun, Jul 12, 2020 at 3:37 PM Jan Lehnardt  wrote:
> 
>> Hey all,
>> 
>> based on a question in our new GitHub Discussion board, I got interested
>> in what is faster: retrieve a doc _rev with a HEAD request or with an
>> _all_docs?key=docid request. The results might be interesting for folks:
>> 
>> 
>> https://github.com/apache/couchdb/discussions/2996#discussioncomment-36190
>> 
>> Best
>> Jan
>> —

Fastest way to get a doc _rev

[Jan Lehnardt]

Hey all,

based on a question in our new GitHub Discussion board, I got interested in 
what is faster: retrieve a doc _rev with a HEAD request or with an 
_all_docs?key=docid request. The results might be interesting for folks:

https://github.com/apache/couchdb/discussions/2996#discussioncomment-36190

Best
Jan
—

Re: Restriction of autocompaction to a time window not working

[Jan Lehnardt]

For posterity, this got resolved here: 
https://github.com/apache/couchdb/discussions/2997

Best
Jan

> On 10. Jul 2020, at 20:15, Cluxter  wrote:
> 
> Hi everyone
> 
> I am using the official Docker image of CouchDB, v3.1.0.
> 
> I created a couchdb.ini file in which I put my custom CouchDB parameters. I
> know this file is taken into account by CouchDB because I checked in
> Fauxton/Configuration and my custom settings are shown.
> 
> The configuration is default except that I added this:
> 
> [couchdb]
> single_node = true
> 
> Now, I want to limit autocompaction to a time window from 04:00 am to 5:30
> am, and I want this to be as strict as possible. So I added these
> parameters in the same file:
> 
> [smoosh.overnight_channel]
> from = 04:00
> to = 05:30
> strict_window = true
> 
> 
> Then I started to insert 3 millions of documents in one database one by one
> using ApacheBench like this:
> 
> ab -r -n 300 -c 1000 -p data.json -T "application/json" -A
> admin:password "http://localhost:5984/benchmark;
> 
> and "data.json" contains:
> 
> {
>"text": "Some text"
> }
> 
> 
> It works very well, I get about 10,000 inserts/s.
> But very quickly an autocompaction task is run and slows everything down to
> about 2,000 inserts/s. I checked that in Fauxton/Active tasks/All tasks,
> there are clearly a couple of compaction jobs running that I never asked
> for. The thing is: I'm not doing all this between 4am and 5:30am and the
> time of my computer is right.
> 
> So it looks like the restriction to the time window doesn't work.
> 
> Am I doing something wrong or missing something?
> 
> Thanks.
> 
> Kind regards,
> 
> Baptiste Rebillard

Re: Don’t rely on CouchDB’s auto-UUID generation?

[Jan Lehnardt]

Hi Olaf,

> On 10. Jul 2020, at 14:03, Olaf Krüger  wrote:
> 
> Hi guys,
> 
> before reading the docs I was afraid of creating custom _ids.
> But the docs [1] even says that we shouldn't rely on the auto generated UUID.
> 
> Our primary goal is to create more human readable _ids.
> So, in order to avoid conflicts, the idea is to introduce scoped custom _ids 
> which starts e.g. with a leading namespace followed by a short id, e.g.:
> 
> INV-HG6FU
> 
> (The length of the short id depends on the expected number of documents for a 
> certain scope/doc type)
> 
> However, before going this way I would like to know if there are any 
> drawbacks with such approach and how others achieve this.

This is a fine approach. The only drawback is when your expectation ends up not 
holding, but then you could just lengthen the id one by one to get more space.


> Or should we rather rely on the auto-UUID and move custom ids into its own 
> document property, something like e.g. "publicId”?

This also works, bur your original idea is also fine.


—

I think what the docs want to convey is that avoiding accidental duplicates on 
network-retries is the main reason for this advice. You can also retrieve UUIDs 
from CouchDB’s to assign to docs on the client if you don’t have your own.

In addition, CouchDB’s default UUID scheme produces UUIDs in sequential 
batches[1], that make inserts more efficient. If you can reproduce this in your 
external IDs, your get the same benefits.

[1]: 
https://github.com/apache/couchdb/blob/master/rel/overlay/etc/default.ini#L372-L375

Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
> 
> Many thanks in advance!
> 
> Olaf
> 
> [1]
> https://docs.couchdb.org/en/stable/best-practices/documents.html#don-t-rely-on-couchdb-s-auto-uuid-generation
> 
> 

Re: couchdb 3.X partitioning

[Jan Lehnardt]

> On 7. Jul 2020, at 16:42, Nick Vatamaniuc  wrote:
> 
> Hi Sharath,
> 
> You can use {"source": ..., "target": ..., "create_target": true,
> "create_target_params": {"partitioned": true}, ...} to create
> partitioned target dbs.
> 
> The option was there since version 2.2.0
> https://docs.couchdb.org/en/stable/whatsnew/2.2.html?highlight=create_target_params#features
> but unfortunately it was not documented yet.


lol, thanks for digging this out. That explains why I couldn’t find it in the 
docs 

PRs welcome ;)

Best
Jan
—
> 
> Cheers,
> -Nick
> 
> On Tue, Jul 7, 2020 at 9:23 AM Jan Lehnardt  wrote:
>> 
>> 
>> 
>>> On 7. Jul 2020, at 15:19, Sharath  wrote:
>>> 
>>> Thanks - thats what I thought.
>> 
>> Ah yes, I was assuming your _ids were already set up correctly.
>> 
>> Best
>> Jan
>> —
>>> 
>>> Have to write an ETL job for my hugeish database - would be cool if
>>> replication protocol could be used to achieve the same thing.
>>> 
>>> Sharath
>>> 
>>> 
>>> On Tue, Jul 7, 2020 at 11:15 PM Adam Kocoloski  wrote:
>>> 
>>>> The tricky part is that partitioned databases have a hard requirement on
>>>> document IDs to have a “:” in them to demarcate between the partition and
>>>> rest of the document ID.  Replication can’t change document ID, but if the
>>>> source database happens to fulfill that requirement for all of its
>>>> documents (excluding _design documents), then you could create a
>>>> partitioned database on the target and replicate into it. But that’s a
>>>> pretty unlikely coincidence.
>>>> 
>>>> Switching to partitioned databases is unfortunately more likely to require
>>>> an external ETL job.
>>>> 
>>>> Adam
>>>> 
>>>>> On Jul 7, 2020, at 8:30 AM, Jan Lehnardt  wrote:
>>>>> 
>>>>> Hi Sharath,
>>>>> 
>>>>>> On 7. Jul 2020, at 14:17, Sharath  wrote:
>>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> Got couchdb 3.1 running and migrated my database (replicated) over.
>>>>>> 
>>>>>> Read about partitioning and have the following questions:
>>>>>> 
>>>>>> Can a partitioned database be created when replicating from another
>>>> couchdb
>>>>>> instance?
>>>>> 
>>>>> Do you mean with the `create_target: true` option? Probably not, but you
>>>> can
>>>>> create the database yourself as partitioned and then replicate over.
>>>>> 
>>>>> Best
>>>>> Jan
>>>>> —
>>>>> 
>>>>>> 
>>>>>> [I think not but have to ask]
>>>>>> 
>>>>>> thanks
>>>>>> Sharath
>>>> 
>>>> 
>> 

Re: couchdb 3.X partitioning

[Jan Lehnardt]

> On 7. Jul 2020, at 15:19, Sharath  wrote:
> 
> Thanks - thats what I thought.

Ah yes, I was assuming your _ids were already set up correctly.

Best
Jan
—
> 
> Have to write an ETL job for my hugeish database - would be cool if
> replication protocol could be used to achieve the same thing.
> 
> Sharath
> 
> 
> On Tue, Jul 7, 2020 at 11:15 PM Adam Kocoloski  wrote:
> 
>> The tricky part is that partitioned databases have a hard requirement on
>> document IDs to have a “:” in them to demarcate between the partition and
>> rest of the document ID.  Replication can’t change document ID, but if the
>> source database happens to fulfill that requirement for all of its
>> documents (excluding _design documents), then you could create a
>> partitioned database on the target and replicate into it. But that’s a
>> pretty unlikely coincidence.
>> 
>> Switching to partitioned databases is unfortunately more likely to require
>> an external ETL job.
>> 
>> Adam
>> 
>>> On Jul 7, 2020, at 8:30 AM, Jan Lehnardt  wrote:
>>> 
>>> Hi Sharath,
>>> 
>>>> On 7. Jul 2020, at 14:17, Sharath  wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> Got couchdb 3.1 running and migrated my database (replicated) over.
>>>> 
>>>> Read about partitioning and have the following questions:
>>>> 
>>>> Can a partitioned database be created when replicating from another
>> couchdb
>>>> instance?
>>> 
>>> Do you mean with the `create_target: true` option? Probably not, but you
>> can
>>> create the database yourself as partitioned and then replicate over.
>>> 
>>> Best
>>> Jan
>>> —
>>> 
>>>> 
>>>> [I think not but have to ask]
>>>> 
>>>> thanks
>>>> Sharath
>> 
>> 

Re: couchdb 3.X partitioning

[Jan Lehnardt]

Hi Sharath,

> On 7. Jul 2020, at 14:17, Sharath  wrote:
> 
> Hi,
> 
> Got couchdb 3.1 running and migrated my database (replicated) over.
> 
> Read about partitioning and have the following questions:
> 
> Can a partitioned database be created when replicating from another couchdb
> instance?

Do you mean with the `create_target: true` option? Probably not, but you can
create the database yourself as partitioned and then replicate over.

Best
Jan
—

> 
> [I think not but have to ask]
> 
> thanks
> Sharath

Re: Compaction file still behind main file

[Jan Lehnardt]

On 6. Jul 2020, at 10:26, Alan Malta  wrote:
> 
> Dear experts,
> 
> apologies if I'm asking something that has already been asked here, if
> so, I'm happy to have just a link to a previous discussion.
> 
> Coming to the problem, over the last week, one of my couch instances
> went twice from ?~20GB to ~300GB. The problem is that the compaction
> never manages to switch the databases because there are always a few
> thousand changes to catch up on (~30k), e.g.:
> """
> Compaction file still behind main file (update seq=199559009. compact
> update seq=199532061). Retrying.
> """
> 
> By the time it catches up, there is again another 20k or 30k changes to 
> process.
> 
> I have this setup for many years now (CentOS7, CouchDB 1.6.1, probably
> a normal load), so I wonder if this is actually a symptom that
> something else is not fine in the system? Besides stopping database
> writes, is there anything else that I could do to avoid it?

Stopping database writes is the way to do this in 1.x (which we’ve long
stopped supporting). If you have the option to increase IOPS on your
system, that might also help.

I’d say this is normal and you have reached database sizes where the
existing underlying hardware doesn’t suffice to support normal operation.

It doesn’t look like something else is not fine.

> 
> Thank you in advance.
> Alan.
> 
> PS.: Out of curiosity, is it handled in a different/better way in the
> latest CouchDB versions (like 3.1)?

From 2.x onwards, databases can be sharded, that is split up into multiple
database files (which individually are the same as a single database from
1.x). So for example with four shards, each shard will only receive 1/4th
the write load, making it easier to catch up with.

Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

Re: X-Content-Type-Options and strict-transport-security

[Jan Lehnardt]

> On 3. Jul 2020, at 08:53, Sebastien  wrote:
> 
> Given that CouchDB exposes its functionality over HTTP through a RESTful
> API, IMHO it should allow to define such important http headers for
> security directly.

This is a fair point and a patch/PR to that effect is going to be
uncontroversial.

> Only being able to rely on additional infrastructure to secure the system
> is problematic. Indeed many production deployments will have such
> infrastructure in place, but it will not always be the case. Even if it is,
> then it would also require mTLS to ensure a good level of security.
> Moreover, SSL termination is indeed one way, but it's based on the "old
> way", considering internal traffic as trusted, which is not in line with
> current security practices. Defense in depth also considers internal
> traffic as requiring secure communications.

CouchDB does support native TLS.

Best
Jan
—

> 
> kr,
> Sébastien
> 
> On Thu, Jul 2, 2020 at 7:17 PM Joan Touzet  wrote:
> 
>> Best option: use a reverse proxy like haproxy or nginx to inject these.
>> You can also terminate SSL at this layer for better SSL support and
>> performance.
>> 
>> -Joan
>> 
>> On 02/07/2020 05:01, Mody, Darshan Arvindkumar (Darshan) wrote:
>>> Hi
>>> 
>>> In our project we would like to set the header X-Content-Type-Options
>> and strict-transport-security whenever CouchDB responds to an request
>>> 
>>> How can we set the headers?
>>> 
>>> Thanks in advance
>>> 
>>> Regards
>>> Darshan
>>> 
>> 

Re: Way to disable the management GUI

[Jan Lehnardt]

Hi Darshan,

Fauxton, the management GUI is just a web app that uses the CouchDB API that 
your application uses as well.

The way to secure CouchDB is to secure who has access to the API. Whether or 
not the management GUI is present makes no difference.

For example, if you have an CouchDB API endpoint with hypothetically Fauxton 
removed, I can just run Fauxton on my machine and point it at your API endpoint.

So securing that API endpoint is the one and only correct way of securing 
access to CouchDB.

Best
Jan
—

> On 2. Jul 2020, at 09:09, Mody, Darshan Arvindkumar (Darshan) 
>  wrote:
> 
> Hi
> 
> We are using CouchDB as the database in our project. One of the concerns from 
> the Security team is the management GUI which can lead to vulnerabilities
> .
> Is there a way to disable the management GUI
> 
> Thanks
> Darshan

Re: CouchDB and Rust blogs

[Jan Lehnardt]

Congrats Garren, this is really cool! :)

One related question: have you pondered embedding a JS engine into Erlang 
itself as well?

Best
Jan
—

> On 24. Jun 2020, at 16:21, Garren Smith  wrote:
> 
> Hi All,
> 
> I've been playing around with the rust language quite a bit recently and
> using it to write some rust related side projects. I've recently finished a
> CouchDB View Server written in Rust using V8. Here is a blog post about
> that details the new View Server protocol for CouchDB 4.x and my Rust
> implementation
> https://www.garrensmith.com/blogs/fortuna-rs-couchdb-view-server
> 
> A few weeks back I also wrote about a miniCouchDB implementation I wrote in
> Rust during a Cloudant Hack week.
> https://www.garrensmith.com/blogs/mini-couch-hack-week
> 
> Cheers
> Garren

Re: Newsfeed IFRAME in Fauxton and IP collection

[Jan Lehnardt]

Thanks ermouth,

I’m surprised my proposal made it through without discussion. I have the
same question ;D

FWIW, this “leaks” the browser connection to the internet, not necessarily
CouchDB instance data.

For a production version of this, I would at least expect an opt-in button
on that page, before loading remote content.

My PR was meant to start this discussion :)

Best
Jan
—

> On 24. Jun 2020, at 10:33, ermouth  wrote:
> 
> Since I hadn’t received any answer at Github, I’d like to raise an
> important CouchDB Fauxton security question publicly.
> 
> One of the latest Fauxton PRs (
> https://github.com/apache/couchdb-fauxton/pull/1284) adds a remote newsfeed
> to Fauxton. Emitting a newsfeed in the admin panel in that way may lead to
> IP collection of CouchDB instances (or subnets, that is even worse)
> somewhere.
> 
> Where is this ‘somewhere’ located? Pinging blog.couchdb.org shows it points
> to lb.wordpress.com, which seems a bit ridiculous. CouchDB instances are
> not uncommon for very critical parts of infrastructure and security
> projects, and I doubt anyone wants to expose node IPs to _whatever_ logs,
> esp wordpress.com.
> 
> So I’d like to ask devs and users: does anyone think adding news to the
> admin panel worth creating such a security hole?
> 
> ermouth

Re: Attempting to update legacy view in logs

[Jan Lehnardt]

Hi Milko,

there is nothing broken here so there is nothing to fix. CouchDB
occasionally updates internal data formats and when that happens,
it migrates any existing data from the old format to the new format.

That’s what’s happening here and there is nothing to worry about.

Best
Jan
—
-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

> On 23. Jun 2020, at 17:29, Milko Petrov  wrote:
> 
> Hello everyone,
> 
> After upgrade from CouchDB 2.3.1 to 3.1.0 messages like these listed
> below appear in cluster logs at level [notice], I am curious if I can do
> something to "fix" this.
> It seems that this is related somehow to a view, which is filtering
> conflicting docs, but I was unable to find any useful info so far. For
> both versions of CouchDB I am using official docker images.
> 
> 
> Attempting to update legacy view index file from
> "./data/.shards/a000-bfff/userdb-milko.1584610755_design/mrview/ae859be327047ad369e22b6d979cb10b.view"
> to
> ./data/.shards/a000-bfff/userdb-milko.1584610755_design/mrview/9965f2c5b05311e90d3c99140ad4a3a2.view
> 
>  Successfully updated legacy view index file
> ./data/.shards/a000-bfff/userdb-milko.1584610755_design/mrview/ae859be327047ad369e22b6d979cb10b.view
> 
> Best Regards,
> Milko
> 
> 
> 
> 
> 

Re: design document with no name

[Jan Lehnardt]

Heya Andrea,

Which version of CouchDB is this?

Best
Jan
_

> On 15. Jun 2020, at 14:54, Andrea Brancatelli 
>  wrote:
> 
> Sorry the attachment got lost along the way: 
> 
> http://storage.gcloud.schema31.it/obj/9d6870890a00-1608-ae11-70fa-57153e15
> 
> ---
> 
> Andrea Brancatelli
> Schema31 S.p.a.
> 
> On 2020-06-15 14:44, Andrea Brancatelli wrote:
> 
>> I really don't know how but one of my developers succeeded, by using Photon, 
>> in creating a design document with an empty ID. 
>> 
>> Needless to say this broke almost everything like any Post to this database 
>> with: 
>> 
>> [error] 2020-06-15T12:35:00.511174Z couchdb@127.0.0.1 <0.24049.1262> 
>>  could not load validation funs {{nocatch,{illegal_docid,<<"Illegal 
>> document id 
>> `_design/`">>}},[{couch_doc,validate_docid,1,[{file,"src/couch_doc.erl"},{line,213}]},{couch_doc,transfer_fields,3,[{file,"src/couch_doc.erl"},{line,252}]},{couch_doc,get_validate_doc_fun,1,[{file,"src/couch_doc.erl"},{line,399}]},{ddoc_cache_entry_validation_funs,'-recover/1-fun-0-',1,[{file,"src/ddoc_cache_entry_validation_funs.erl"},{line,35}]},{lists,flatmap,2,[{file,"lists.erl"},{line,1250}]},{ddoc_cache_entry_validation_funs,recover,1,[{file,"src/ddoc_cache_entry_validation_funs.erl"},{line,34}]},{ddoc_cache_entry,do_open,1,[{file,"src/ddoc_cache_entry.erl"},{line,297}]}]}
>>  
>> 
>> I can't edit the document, I can't delete the document because the server 
>> says that _design/ is an illegal id name.  
>> 
>> I strongly agree with him :-) 
>> 
>> Luckily this is a Development Database so I guess I'll drop the whole thing 
>> and start from scratch, but maybe there's a bug in the validation code for 
>> new design documents? 
>> 
>> I tried to reproduce the thing but I wasn't able to do it... 
>> 
>> -- 
>> 
>> Andrea Brancatelli
>> Schema31 S.p.a.

Re: Announcing GitHub Discussions for Apache CouchDB [Beta]

[Jan Lehnardt]

> On 10. Jun 2020, at 00:36, Joan Touzet  wrote:
> 
> Hi Piotr,
> 
> We learned of the feature based on friends who are testing the feature in 
> private beta, in places like the JavaScript React community:
> 
>  https://github.com/vercel/next.js/discussions
> 
> That page shows some of the upcoming features in Discussions, currently hard 
> coded for that community.
> 
> You'll have to talk to ASF Infrastructure if you want to experiment with this 
> feature. Open a JIRA ticket with them; they are the gatekeepers.

You *also* need a contact on the GitHub side afaict. I’m happy to make an 
introduction.

Best
Jan
—

> 
> -Joan "now in the correct Channel" Touzet
> 
> On 2020-06-09 5:22 p.m., Piotr Zarzycki wrote:
>> Hi Guys,
>> I'm PMC in a different Apache project and I was wondering - what required
>> actually to switch ON Discussion on GH? How idea started?
>> Thanks,
>> Piotr
>> On Tue, Jun 9, 2020, 10:48 PM Alessio 'Blaster' Biancalana <
>> dottorblas...@apache.org> wrote:
>>> Woh! This is huuuge!
>>> Thanks for making this happen :-)
>>> 
>>> Alessio
>>> 
>>> On Tue, Jun 9, 2020 at 8:14 PM Joan Touzet  wrote:
>>> 
>>>> Hi everyone,
>>>> 
>>>> Thanks to some personal connections and the support of ASF Infra, Apache
>>>> CouchDB now has GitHub Discussions enabled on our repository:
>>>> 
>>>>https://github.com/apache/couchdb/discussions
>>>> 
>>>> Right now, we're beta testing this (in conjunction with MS/GitHub and
>>>> the ASF) as a new user support channel. You may have already seen us
>>>> move a few Issues over to Discussions, where they now belong.
>>>> 
>>>> We're hoping that this gives people a friendlier way to build community
>>>> around CouchDB, one that doesn't require mailing list membership or
>>>> joining a possibly-intimidating Slack channel. Please feel free to use
>>>> the Discussions to ask for help with CouchDB, share ideas, show us what
>>>> you've done with CouchDB, offer thanks to the team, or post things you
>>>> learned and want to share with others.
>>>> 
>>>> One key feature is that we hope to give more recognition and affordance
>>>> to our non-committer contributors through the Discussions platform. Stay
>>>> tuned for more info.
>>>> 
>>>> As more Discussions features become available, we'll make use of them to
>>>> customize our space there, including linking to our project
>>>> "bylaws"/rules and code of conduct.
>>>> 
>>>> Two caveats: Right now, there is no webhook availability for
>>>> Discussions, which means we can't mirror activity there to our user@
>>>> mailing lists. We've been told this will arrive around the time
>>>> Discussions leaves private beta.
>>>> 
>>>> The other: Please remember that all discussion of project *direction*
>>>> and *decision making* must occur on the dev@ mailing list. It's fine to
>>>> link to threads in Discussions to help start the decision making
>>>> process, just the same as we do with Slack, Stack Overflow, etc. today.
>>>> 
>>>> I hope this comes as welcome news to everyone. I'm pretty excited about
>>>> it! Special thanks to Jan Lehnardt, who helped get this enabled for us,
>>>> and Garren Smith, who started the discussion months ago about moving out
>>>> of the twentieth century for community building.
>>>> 
>>>> -Joan "I'm a woman. I can change. If I have to. I guess." Touzet
>>>> 
>>> 

Re: couchdb file corruption

[Jan Lehnardt]

Hi Sharath,

the error you are seeing originates in CouchDB’s file checksumming
feature[1]. When writing a block of data to disk, in either a
database or a view index, CouchDB also stores a checksum of the
block. When reading the block later, it checks if the stored
checksum and the newly read block still match. This is to prevent
hard drive errors sneaking into your database.

In your case, a block in a view index doesn’t match its checksum
anymore. The reasons for this could be manifold, but all of them
will have resulted in a bit or more on your storage device having
changed values behind CouchDB’s back.

To get back to working views, you can delete the index files and
rebuild them.

But more importantly:

1. do triple check what could have caused that block to go bad.
You don’t want to have this happen to your database files.

2. triple check your backups are up to date and restorable.

[1]: 
https://github.com/apache/couchdb/blob/3505281559513e2922484ebf0996a8846dcc0a34/src/couch/src/couch_file.erl#L173-L188

Best
Jan
—

> On 9. Jun 2020, at 02:44, Sharath  wrote:
> 
> Hi,
> 
> I'm encountering a file corruption in CouchDb 2.3.1 running on Ubuntu
> 18.04.1 LTS.
> 
> The disk store is an ext4 SSD.
> 
> I'm unable to access the view and couch logs shows the error below.
> 
> Is there a way of knowing which view is corrupt (I have a few databases).
> 
> I'm thinking of deleting all the views and recreating them - would that
> work?
> 
> thanks!
> 
> error 1:
> [emergency] 2020-06-09T00:33:10.122474Z couchdb@127.0.0.1 <0.1427.0>
>  File corruption in <0.951.0> at position 2668802729
> [error] 2020-06-09T00:33:10.123271Z couchdb@127.0.0.1 <0.1392.0> 9f5e486e35
> rexi_server: from: couchdb@127.0.0.1(<0.758.0>) mfa: fabric_rpc:map_view/5
> throw:{file_corruption,<<"file corruption">>}
> [{couch_mrview_util,get_view_index_state,5,[{file,"src/couch_mrview_util.erl"},{line,137}]},{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,81}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,247}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
> [error] 2020-06-09T00:33:10.123712Z couchdb@127.0.0.1 <0.758.0> 9f5e486e35
> req_err(4089041121) file_corruption : file corruption
>[<<"couch_mrview_util:get_view_index_state/5
> L137">>,<<"couch_mrview_util:get_view/4 L81">>,<<"couch_mrview:query_view/6
> L247">>,<<"rexi_server:init_p/3 L140">>]
> [notice] 2020-06-09T00:33:10.124065Z couchdb@127.0.0.1 <0.758.0> 9f5e486e35
> 192.168.0.13:5984 192.168.0.8 admin GET
> /stock/_design/company/_view/getallamexcompanies?reduce=false=0=101
> 500 ok 
> [error] 2020-06-09T00:33:11.129465Z couchdb@127.0.0.1 <0.1419.0> 
> gen_server <0.1419.0> terminated with reason: no match of right hand value
> eof at couch_file:read_raw_iolist_int/3(line:627) <=
> couch_file:handle_call/3(line:449) <=
> gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
> <= proc_lib:init_p_do_apply/3(line:247)
>  last msg: {pread_iolist,1190244510}
> state:
> [{data,[{"State",{file,{file_descriptor,prim_file,{#Port<0.7921>,106}},false,1767501252,undefined,infinity}},{"InitialFilePath","/data/couchdb/.shards/8000-9fff/stock.1584663325_design/mrview/856ffe4a2101b41233877c86e8e3f8e6.view"}]}]
>extra: []
> [error] 2020-06-09T00:33:11.132250Z couchdb@127.0.0.1 <0.1419.0> 
> CRASH REPORT Process  (<0.1419.0>) with 1 neighbors exited with reason: no
> match of right hand value eof at couch_file:read_raw_iolist_int/3(line:627)
> <= couch_file:handle_call/3(line:449) <=
> gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
> <= proc_lib:init_p_do_apply/3(line:247) at gen_server:terminate/7(line:812)
> <= proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_file,init,['Argument__1']}, ancestors: [<0.1408.0>,<0.1407.0>],
> messages: [], links: [<0.1408.0>], dictionary:
> [{couch_file_fd,{{file_descriptor,prim_file,{#Port<0.7921>,106}},"/dat..."}},...],
> trap_exit: false, status: running, heap_size: 6772, stack_size: 27,
> reductions: 573398
> [error] 2020-06-09T00:33:11.132607Z couchdb@127.0.0.1 <0.1432.0> 
> gen_server <0.1432.0> terminated with reason: no match of right hand value
> eof at couch_file:read_raw_iolist_int/3(line:627) <=
> couch_file:handle_call/3(line:449) <=
> gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
> <= proc_lib:init_p_do_apply/3(line:247)
>  last msg:
> {'EXIT',<0.1408.0>,{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}
> state: {st,<0.1408.0>,couch_mrview_index,undefined}
>extra: []
> 
> 
> error 2:
> [error] 2020-06-09T00:40:25.580012Z 

Re: Option request for user access to the /_all_dbs endpoint (old behavior)

[Jan Lehnardt]

Hi Erik,

there is: 
https://github.com/apache/couchdb/blob/master/rel/overlay/etc/default.ini#L140-L141
 


It is also mentioned in the accompanying blog post: 
https://blog.couchdb.org/2020/02/26/the-road-to-couchdb-3-0-security/ 


But good call that the release notes could do with a mention. Would you be up 
for submitting a PR?

Best
Jan
—

> On 28. May 2020, at 11:45, Erik Verheul  wrote:
> 
> Hi,
> 
> From the upgrade notes: #2576: CouchDB 3.0 now requires admin-level access 
> for the /_all_dbs endpoint. See 
> https://docs.couchdb.org/en/3.1.0/whatsnew/3.0.html
> This change means a major overhaul of my onebacklog.net application. And the 
> temptation to assign _admin rights to the admin role.
> Can there be an option in the local.ini file to reestablish the old behavior?
> 
> kind regards,
> Erik

Re: scaling?

[Jan Lehnardt]

> On 24. May 2020, at 17:32, Miles Fidelman  wrote:
> 
> Thanks Jan, and a follow-up, below:
> 
> On 5/24/20 4:51 AM, Jan Lehnardt wrote:
>> 
>>> On 23. May 2020, at 18:57, Miles Fidelman  
>>> wrote:
>>> 
>>> On 5/23/20 12:29 PM, Jan Lehnardt wrote:
>>> 
>>>>> On 23. May 2020, at 16:28, Miles Fidelman  
>>>>> wrote:
>>>>> 
>>>>> 
>>>>> On 5/22/20 11:13 AM, Jan Lehnardt wrote:
>>>>> 
>>>>>>> On 22. May 2020, at 15:06, Miles Fidelman  
>>>>>>> wrote:
>>>>>>> 
>>>>>>> Hi Jan,
>>>>>>> 
>>>>>>> On 5/22/20 6:17 AM, Jan Lehnardt wrote:
>>>>>>>> Hi Miles,
>>>>>>>> 
>>>>>>>> I wanted to reply for a while, but struggled to find a good angle. I 
>>>>>>>> think I finally figured out what I missed. I’m not sure I understand 
>>>>>>>> your deployment scenario.
>>>>>>>> 
>>>>>>>> When I think conference app, I think folks having that on their mobile 
>>>>>>>> phones, or tablets. Given that, you’d be using PouchDB (for web apps) 
>>>>>>>> or one of Cloudant mobile sync libraries (if it is a native app). 
>>>>>>>> Either way, to get the data to the devices, it’d have to come from 
>>>>>>>> somewhere, but then you say there is no central server. Where does the 
>>>>>>>> data come from?
>>>>>>> I was using that as an example - but I'm really thinking more the case 
>>>>>>> of "smart documents." Think of a plan (business, mission) or briefing 
>>>>>>> package - and the whole notion of "living documents.
>>>>>> not sure what these are and how they compare to CouchDB documents.
>>>>> "Living Document" as in writerspeak - a binder full of documentation that 
>>>>> is continually being kept up to date, the latest version of a proposal, a 
>>>>> book draft, a theatrical script that's being marked up with commentary 
>>>>> during rehearsal, etc.
>>>>> 
>>>>> Or think of marrying a wiki to a DCVS.
>>>>> 
>>>>>>>> It sounds like you imagine the devices talking to each other in a 
>>>>>>>> replication mesh kind of way. While technically possible from a 
>>>>>>>> CouchDB replication protocol point of view, this approach isn’t very 
>>>>>>>> practical. For one, you won’t be able to guarantee that all devices 
>>>>>>>> can talk to each other, especially when you don’t control the wifi.
>>>>>>>> 
>>>>>>>> What’s more likely is that you’d have central CouchDB server that is 
>>>>>>>> connected to the wifi network, either on site, or in a datacenter 
>>>>>>>> somewhere that all devices connect to.
>>>>>>>> 
>>>>>>>> Having that many devices talk to a central database to get all the 
>>>>>>>> relatively static data of the conference (schedule, info, etc), 
>>>>>>>> doesn’t sound like much of a problem. Group interaction is a little 
>>>>>>>> more interesting. You could model this a 1-db-per-group, and it would 
>>>>>>>> work okay, but there are some devil-in-the-details-details to work out 
>>>>>>>> with access control and joining and leaving a group, etc.
>>>>>>>> 
>>>>>>>> So overall:
>>>>>>>> 
>>>>>>>> - what architecture do you envision?
>>>>>>>> - I think this can be made to work.
>>>>>>>> - CouchDB definitely can handle 5000 intermittent clients. Depending 
>>>>>>>> on your use-case, you may need more or less computing resources for 
>>>>>>>> this, but there aren’t any fundamental blockers in CouchDB’s design 
>>>>>>>> that would prevent this from being a success.
>>>>>>> I keep coming back to the model of replicated copies, stored in 
>>>>>>> something like a distributed version control system, linked by a 
>>>>>>> publish-subscribe network. (Not that much different than the way 
>>>>>&g

Re: scaling?

[Jan Lehnardt]

> On 23. May 2020, at 18:57, Miles Fidelman  wrote:
> 
> On 5/23/20 12:29 PM, Jan Lehnardt wrote:
> 
>> 
>>> On 23. May 2020, at 16:28, Miles Fidelman  
>>> wrote:
>>> 
>>> 
>>> On 5/22/20 11:13 AM, Jan Lehnardt wrote:
>>> 
>>>>> On 22. May 2020, at 15:06, Miles Fidelman  
>>>>> wrote:
>>>>> 
>>>>> Hi Jan,
>>>>> 
>>>>> On 5/22/20 6:17 AM, Jan Lehnardt wrote:
>>>>>> Hi Miles,
>>>>>> 
>>>>>> I wanted to reply for a while, but struggled to find a good angle. I 
>>>>>> think I finally figured out what I missed. I’m not sure I understand 
>>>>>> your deployment scenario.
>>>>>> 
>>>>>> When I think conference app, I think folks having that on their mobile 
>>>>>> phones, or tablets. Given that, you’d be using PouchDB (for web apps) or 
>>>>>> one of Cloudant mobile sync libraries (if it is a native app). Either 
>>>>>> way, to get the data to the devices, it’d have to come from somewhere, 
>>>>>> but then you say there is no central server. Where does the data come 
>>>>>> from?
>>>>> I was using that as an example - but I'm really thinking more the case of 
>>>>> "smart documents." Think of a plan (business, mission) or briefing 
>>>>> package - and the whole notion of "living documents.
>>>> not sure what these are and how they compare to CouchDB documents.
>>> "Living Document" as in writerspeak - a binder full of documentation that 
>>> is continually being kept up to date, the latest version of a proposal, a 
>>> book draft, a theatrical script that's being marked up with commentary 
>>> during rehearsal, etc.
>>> 
>>> Or think of marrying a wiki to a DCVS.
>>> 
>>>>>> It sounds like you imagine the devices talking to each other in a 
>>>>>> replication mesh kind of way. While technically possible from a CouchDB 
>>>>>> replication protocol point of view, this approach isn’t very practical. 
>>>>>> For one, you won’t be able to guarantee that all devices can talk to 
>>>>>> each other, especially when you don’t control the wifi.
>>>>>> 
>>>>>> What’s more likely is that you’d have central CouchDB server that is 
>>>>>> connected to the wifi network, either on site, or in a datacenter 
>>>>>> somewhere that all devices connect to.
>>>>>> 
>>>>>> Having that many devices talk to a central database to get all the 
>>>>>> relatively static data of the conference (schedule, info, etc), doesn’t 
>>>>>> sound like much of a problem. Group interaction is a little more 
>>>>>> interesting. You could model this a 1-db-per-group, and it would work 
>>>>>> okay, but there are some devil-in-the-details-details to work out with 
>>>>>> access control and joining and leaving a group, etc.
>>>>>> 
>>>>>> So overall:
>>>>>> 
>>>>>> - what architecture do you envision?
>>>>>> - I think this can be made to work.
>>>>>> - CouchDB definitely can handle 5000 intermittent clients. Depending on 
>>>>>> your use-case, you may need more or less computing resources for this, 
>>>>>> but there aren’t any fundamental blockers in CouchDB’s design that would 
>>>>>> prevent this from being a success.
>>>>> I keep coming back to the model of replicated copies, stored in something 
>>>>> like a distributed version control system, linked by a publish-subscribe 
>>>>> network. (Not that much different than the way large-scale modeling & 
>>>>> sims are done - local "world models" linked by a protocol like DIS or 
>>>>> HLA.)
>>>>> 
>>>>> I've been wondering if Couch/Pouch might make a good platform - coupled 
>>>>> with something like NNTP for epidemic style distribution of changes.
>>>> If you squint a little, CouchDB fits your buzzword bill here, but without 
>>>> more concrete descriptions of what you need (rather than less), we can’t 
>>>> help much.
>>>> 
>>>> And you didn’t address the network connectivity issue. If you plan to have 
>>>> device-to-device communication, an

Re: scaling?

[Jan Lehnardt]

> On 23. May 2020, at 16:28, Miles Fidelman  wrote:
> 
> 
> On 5/22/20 11:13 AM, Jan Lehnardt wrote:
> 
>> 
>>> On 22. May 2020, at 15:06, Miles Fidelman  
>>> wrote:
>>> 
>>> Hi Jan,
>>> 
>>> On 5/22/20 6:17 AM, Jan Lehnardt wrote:
>>>> Hi Miles,
>>>> 
>>>> I wanted to reply for a while, but struggled to find a good angle. I think 
>>>> I finally figured out what I missed. I’m not sure I understand your 
>>>> deployment scenario.
>>>> 
>>>> When I think conference app, I think folks having that on their mobile 
>>>> phones, or tablets. Given that, you’d be using PouchDB (for web apps) or 
>>>> one of Cloudant mobile sync libraries (if it is a native app). Either way, 
>>>> to get the data to the devices, it’d have to come from somewhere, but then 
>>>> you say there is no central server. Where does the data come from?
>>> I was using that as an example - but I'm really thinking more the case of 
>>> "smart documents." Think of a plan (business, mission) or briefing package 
>>> - and the whole notion of "living documents.
>> not sure what these are and how they compare to CouchDB documents.
> 
> "Living Document" as in writerspeak - a binder full of documentation that is 
> continually being kept up to date, the latest version of a proposal, a book 
> draft, a theatrical script that's being marked up with commentary during 
> rehearsal, etc.
> 
> Or think of marrying a wiki to a DCVS.
> 
>> 
>>> 
>>>> It sounds like you imagine the devices talking to each other in a 
>>>> replication mesh kind of way. While technically possible from a CouchDB 
>>>> replication protocol point of view, this approach isn’t very practical. 
>>>> For one, you won’t be able to guarantee that all devices can talk to each 
>>>> other, especially when you don’t control the wifi.
>>>> 
>>>> What’s more likely is that you’d have central CouchDB server that is 
>>>> connected to the wifi network, either on site, or in a datacenter 
>>>> somewhere that all devices connect to.
>>>> 
>>>> Having that many devices talk to a central database to get all the 
>>>> relatively static data of the conference (schedule, info, etc), doesn’t 
>>>> sound like much of a problem. Group interaction is a little more 
>>>> interesting. You could model this a 1-db-per-group, and it would work 
>>>> okay, but there are some devil-in-the-details-details to work out with 
>>>> access control and joining and leaving a group, etc.
>>>> 
>>>> So overall:
>>>> 
>>>> - what architecture do you envision?
>>>> - I think this can be made to work.
>>>> - CouchDB definitely can handle 5000 intermittent clients. Depending on 
>>>> your use-case, you may need more or less computing resources for this, but 
>>>> there aren’t any fundamental blockers in CouchDB’s design that would 
>>>> prevent this from being a success.
>>> I keep coming back to the model of replicated copies, stored in something 
>>> like a distributed version control system, linked by a publish-subscribe 
>>> network. (Not that much different than the way large-scale modeling & sims 
>>> are done - local "world models" linked by a protocol like DIS or HLA.)
>>> 
>>> I've been wondering if Couch/Pouch might make a good platform - coupled 
>>> with something like NNTP for epidemic style distribution of changes.
>> If you squint a little, CouchDB fits your buzzword bill here, but without 
>> more concrete descriptions of what you need (rather than less), we can’t 
>> help much.
>> 
>> And you didn’t address the network connectivity issue. If you plan to have 
>> device-to-device communication, and you are not Apple or Google making those 
>> devices and/or mobile operating systems, you won’t be having much luck, 
>> especially on the web platform.
> 
> I'm thinking tactical environments & mesh networks, with intermittent 
> connectivity.  Updates propagate slowly, but with either automatic 
> fork/merge, or with eventual consistency.

Right, so far the theory, which the CouchDB replication protocol supports. Now 
in practice, say you are using PouchDB, how do you make one device’s browser 
talk to another device’s browser?

Best
Jan
—

> 
> Cheers,
> 
> Miles
> 
> 
>> 
>> Best
>> Jan
>> —
>> 
>>> 

Re: scaling?

[Jan Lehnardt]

> On 22. May 2020, at 15:06, Miles Fidelman  wrote:
> 
> Hi Jan,
> 
> On 5/22/20 6:17 AM, Jan Lehnardt wrote:
>> Hi Miles,
>> 
>> I wanted to reply for a while, but struggled to find a good angle. I think I 
>> finally figured out what I missed. I’m not sure I understand your deployment 
>> scenario.
>> 
>> When I think conference app, I think folks having that on their mobile 
>> phones, or tablets. Given that, you’d be using PouchDB (for web apps) or one 
>> of Cloudant mobile sync libraries (if it is a native app). Either way, to 
>> get the data to the devices, it’d have to come from somewhere, but then you 
>> say there is no central server. Where does the data come from?
> 
> I was using that as an example - but I'm really thinking more the case of 
> "smart documents."  Think of a plan (business, mission) or briefing package - 
> and the whole notion of "living documents.

not sure what these are and how they compare to CouchDB documents.

> 
> 
>> It sounds like you imagine the devices talking to each other in a 
>> replication mesh kind of way. While technically possible from a CouchDB 
>> replication protocol point of view, this approach isn’t very practical. For 
>> one, you won’t be able to guarantee that all devices can talk to each other, 
>> especially when you don’t control the wifi.
>> 
>> What’s more likely is that you’d have central CouchDB server that is 
>> connected to the wifi network, either on site, or in a datacenter somewhere 
>> that all devices connect to.
>> 
>> Having that many devices talk to a central database to get all the 
>> relatively static data of the conference (schedule, info, etc), doesn’t 
>> sound like much of a problem. Group interaction is a little more 
>> interesting. You could model this a 1-db-per-group, and it would work okay, 
>> but there are some devil-in-the-details-details to work out with access 
>> control and joining and leaving a group, etc.
>> 
>> So overall:
>> 
>> - what architecture do you envision?
>> - I think this can be made to work.
>> - CouchDB definitely can handle 5000 intermittent clients. Depending on your 
>> use-case, you may need more or less computing resources for this, but there 
>> aren’t any fundamental blockers in CouchDB’s design that would prevent this 
>> from being a success.
> 
> I keep coming back to the model of replicated copies, stored in something 
> like a distributed version control system, linked by a publish-subscribe 
> network.  (Not that much different than the way large-scale modeling & sims 
> are done - local "world models" linked by a protocol like DIS or HLA.)
> 
> I've been wondering if Couch/Pouch might make a good platform - coupled with 
> something like NNTP for epidemic style distribution of changes.

If you squint a little, CouchDB fits your buzzword bill here, but without more 
concrete descriptions of what you need (rather than less), we can’t help much.

And you didn’t address the network connectivity issue. If you plan to have 
device-to-device communication, and you are not Apple or Google making those 
devices and/or mobile operating systems, you won’t be having much luck, 
especially on the web platform.

Best
Jan
—

> 
> Miles
> 
> 
>> 
>> Best
>> Jan
>> —
>> 
>>> On 20. May 2020, at 14:23, Miles Fidelman  
>>> wrote:
>>> 
>>> Hi Folks,
>>> 
>>> I'm thinking of using Couch (or Couch plus Pouch) as the platform for a 
>>> conference program app - everyone gets their copy of the program, plus 
>>> handouts/proceedings, with the added capabilities to set up one's personal 
>>> schedule, hold side conversations with various groups, schedule BOF 
>>> meetings, etc.
>>> 
>>> Which leads me to wonder what experience folks have had with scaling - 
>>> particularly without using a central server.  Can, say, 5000 instances, 
>>> linked intermittently, stay reasonably consistent?
>>> 
>>> What's the largest collection of Couch instances anybody has played with?
>>> 
>>> Thanks,
>>> 
>>> Miles Fidelman
>>> 
>>> -- 
>>> In theory, there is no difference between theory and practice.
>>> In practice, there is.   Yogi Berra
>>> 
>>> Theory is when you know everything but nothing works.
>>> Practice is when everything works but no one knows why.
>>> In our lab, theory and practice are combined:
>>> nothing works and no one knows why.  ... unknown
>>> 
> -- 
> In theory, there is no difference between theory and practice.
> In practice, there is.   Yogi Berra
> 
> Theory is when you know everything but nothing works.
> Practice is when everything works but no one knows why.
> In our lab, theory and practice are combined:
> nothing works and no one knows why.  ... unknown

Re: scaling?

[Jan Lehnardt]

Hi Miles,

I wanted to reply for a while, but struggled to find a good angle. I think I 
finally figured out what I missed. I’m not sure I understand your deployment 
scenario.

When I think conference app, I think folks having that on their mobile phones, 
or tablets. Given that, you’d be using PouchDB (for web apps) or one of 
Cloudant mobile sync libraries (if it is a native app). Either way, to get the 
data to the devices, it’d have to come from somewhere, but then you say there 
is no central server. Where does the data come from?

It sounds like you imagine the devices talking to each other in a replication 
mesh kind of way. While technically possible from a CouchDB replication 
protocol point of view, this approach isn’t very practical. For one, you won’t 
be able to guarantee that all devices can talk to each other, especially when 
you don’t control the wifi.

What’s more likely is that you’d have central CouchDB server that is connected 
to the wifi network, either on site, or in a datacenter somewhere that all 
devices connect to.

Having that many devices talk to a central database to get all the relatively 
static data of the conference (schedule, info, etc), doesn’t sound like much of 
a problem. Group interaction is a little more interesting. You could model this 
a 1-db-per-group, and it would work okay, but there are some 
devil-in-the-details-details to work out with access control and joining and 
leaving a group, etc.

So overall:

- what architecture do you envision?
- I think this can be made to work.
- CouchDB definitely can handle 5000 intermittent clients. Depending on your 
use-case, you may need more or less computing resources for this, but there 
aren’t any fundamental blockers in CouchDB’s design that would prevent this 
from being a success.

Best
Jan
—

> On 20. May 2020, at 14:23, Miles Fidelman  wrote:
> 
> Hi Folks,
> 
> I'm thinking of using Couch (or Couch plus Pouch) as the platform for a 
> conference program app - everyone gets their copy of the program, plus 
> handouts/proceedings, with the added capabilities to set up one's personal 
> schedule, hold side conversations with various groups, schedule BOF meetings, 
> etc.
> 
> Which leads me to wonder what experience folks have had with scaling - 
> particularly without using a central server.  Can, say, 5000 instances, 
> linked intermittently, stay reasonably consistent?
> 
> What's the largest collection of Couch instances anybody has played with?
> 
> Thanks,
> 
> Miles Fidelman
> 
> -- 
> In theory, there is no difference between theory and practice.
> In practice, there is.   Yogi Berra
> 
> Theory is when you know everything but nothing works.
> Practice is when everything works but no one knows why.
> In our lab, theory and practice are combined:
> nothing works and no one knows why.  ... unknown
> 

[CVE-2020-1955] Apache CouchDB Remote Privilege Escalation

[Jan Lehnardt]

Description
===

CouchDB version 3.0.0 shipped with a new configuration setting that
governs access control to the entire database server called
`require_valid_user_except_for_up`. It was meant as an extension to the
long-standing setting `require_valid_user`, which in turn requires that
any and all requests to CouchDB will have to be made with valid
credentials, effectively forbidding any anonymous requests.

The new `require_valid_user_except_for_up` is an off-by-default setting
that was meant to allow requiring valid credentials for all endpoints
except for the `/_up` endpoint.

However, the implementation of this made an error that lead to not
enforcing credentials on any endpoint, when enabled.

CouchDB versions 3.0.1[1] and 3.1.0[2] fix this issue.

Mitigation
==

Users who have not enabled `require_valid_user_except_for_up` are not
affected.

Users who have it enabled can either disable it again, or upgrade to
CouchDB versions 3.0.1[1] and 3.1.0[2].

[1]: https://docs.couchdb.org/en/stable/whatsnew/3.0.html#version-3-0-1
[2]: https://docs.couchdb.org/en/stable/whatsnew/3.1.html#version-3-1-0

On behalf of the CouchDB Security Team,
Jan Lehnardt
—

Re: [External] CouchDB 2.1.1 - Unable to setup replication

[Jan Lehnardt]

FWIW, I filed a quick issue on Fauxton. This would make a nice first 
contribution for someone :)

https://github.com/apache/couchdb-fauxton/issues/1278

Best
Jan
—

> On 16. May 2020, at 10:28, Jan Lehnardt  wrote:
> 
> One of the docs in the test is a design document[1], those can only be 
> written by admin users. It looks like you are running verify install by a 
> non-admin user.
> 
> And just FYI: attachments are stripped on this mailing list. You can upload 
> your resources somewhere and link to them in your mail.
> 
> Best
> Jan
> —
> 
> 
> [1]: 
> https://github.com/apache/couchdb-fauxton/blob/master/app/addons/verifyinstall/resources.js#L102-L116
> 
>> On 15. May 2020, at 21:00, Gone, Sajan  wrote:
>> 
>> Thanks Jan. Appreciate your response on this. 
>> 
>> By local I meant I was attempting to replicate from a database named "cust"  
>> to "cust2" on the same local CouchDB instance. In fact I have tried the full 
>> URL's as well attempting to replicate from a remote source but that does not 
>> work either. 
>> 
>> Also I tried verifying the installation and it failing on  Replication check 
>> with an error saying "Replication failed, expected 4 docs got 3". Attached 
>> is the screen shot. 
>> 
>> 
>> Thank You,
>> Sajan Gone
>> Senior Database Engineer
>> LBrands | Mast Global Technologies
>> Mobile #:517-990-5282
>> Office #:   614-577-7622 
>> 
>> 
>> 
>> On 5/15/20, 2:43 PM, "Jan Lehnardt"  wrote:
>> 
>>   Do you mean local as in `dbname` or as in 
>> `https://urldefense.proofpoint.com/v2/url?u=https-3A__thelocalcluster_dbname-2560-3F=DwIFaQ=spYp1tZ3AQD6dfuI6rqaeg=7n1bLwXDzepgjKSNziw-ng=TAv05Wolj8UR5V3rj52qa0GyDRl_ULRPm4mlv_nwO6w=n7NNLsWhVbDFx_6zIKZZHJqAO5I86XgK_Lu65LR4F5M=
>>  
>> 
>>   If it is the former, that just doesn’t work. You need full URLs for source 
>> and target.
>> 
>>   Best
>>   Jan
>>   —
>> 
>>> On 15. May 2020, at 17:28, Gone, Sajan  wrote:
>>> 
>>> Hi All,
>>> 
>>> I am relatively new to CouchDB, we just started doing a POC on it for our 
>>> Customer Grid like application. We installed it on a CentOS machine using 
>>> yum repositories and Everything worked well for some time and I was able to 
>>> replicate from one database to another locally and also to a remote 
>>> database.
>>> 
>>> All of a sudden the replication setup stopped working. Whenever I try to 
>>> set a new replication channel I am getting an "undefined" message on the 
>>> browser. This is happening even when Source/Destination are specified to be 
>>> local databases. I have tried uninstalling, wiping off the data/log 
>>> directories and did a fresh install of couchdb 2.1.1 but the issue still 
>>> persists. I can't wrap my head around on what is going on here. Any help, 
>>> thoughts or suggestions is greatly appreciated. Many thanks in advance.
>>> 
>>> Attaching the log file. 
>>> 
>>> Thanks,
>>> Sajan Gone.
>> 
>> 
> 

Re: [External] Re: CouchDB 2.1.1 - Unable to setup replication

[Jan Lehnardt]

One of the docs in the test is a design document[1], those can only be written 
by admin users. It looks like you are running verify install by a non-admin 
user.

And just FYI: attachments are stripped on this mailing list. You can upload 
your resources somewhere and link to them in your mail.

Best
Jan
—


[1]: 
https://github.com/apache/couchdb-fauxton/blob/master/app/addons/verifyinstall/resources.js#L102-L116

> On 15. May 2020, at 21:00, Gone, Sajan  wrote:
> 
> Thanks Jan. Appreciate your response on this. 
> 
> By local I meant I was attempting to replicate from a database named "cust"  
> to "cust2" on the same local CouchDB instance. In fact I have tried the full 
> URL's as well attempting to replicate from a remote source but that does not 
> work either. 
> 
> Also I tried verifying the installation and it failing on  Replication check 
> with an error saying "Replication failed, expected 4 docs got 3". Attached is 
> the screen shot. 
> 
> 
> Thank You,
> Sajan Gone
> Senior Database Engineer
> LBrands | Mast Global Technologies
> Mobile #:517-990-5282
> Office #:   614-577-7622 
> 
> 
> 
> On 5/15/20, 2:43 PM, "Jan Lehnardt"  wrote:
> 
>Do you mean local as in `dbname` or as in 
> `https://urldefense.proofpoint.com/v2/url?u=https-3A__thelocalcluster_dbname-2560-3F=DwIFaQ=spYp1tZ3AQD6dfuI6rqaeg=7n1bLwXDzepgjKSNziw-ng=TAv05Wolj8UR5V3rj52qa0GyDRl_ULRPm4mlv_nwO6w=n7NNLsWhVbDFx_6zIKZZHJqAO5I86XgK_Lu65LR4F5M=
>  
> 
>If it is the former, that just doesn’t work. You need full URLs for source 
> and target.
> 
>Best
>Jan
>—
> 
>> On 15. May 2020, at 17:28, Gone, Sajan  wrote:
>> 
>> Hi All,
>> 
>> I am relatively new to CouchDB, we just started doing a POC on it for our 
>> Customer Grid like application. We installed it on a CentOS machine using 
>> yum repositories and Everything worked well for some time and I was able to 
>> replicate from one database to another locally and also to a remote database.
>> 
>> All of a sudden the replication setup stopped working. Whenever I try to set 
>> a new replication channel I am getting an "undefined" message on the 
>> browser. This is happening even when Source/Destination are specified to be 
>> local databases. I have tried uninstalling, wiping off the data/log 
>> directories and did a fresh install of couchdb 2.1.1 but the issue still 
>> persists. I can't wrap my head around on what is going on here. Any help, 
>> thoughts or suggestions is greatly appreciated. Many thanks in advance.
>> 
>> Attaching the log file. 
>> 
>> Thanks,
>> Sajan Gone.
> 
> 

Re: CouchDB 2.1.1 - Unable to setup replication

[Jan Lehnardt]

Do you mean local as in `dbname` or as in `https://thelocalcluster/dbname`?

If it is the former, that just doesn’t work. You need full URLs for source and 
target.

Best
Jan
—

> On 15. May 2020, at 17:28, Gone, Sajan  wrote:
> 
> Hi All,
>  
> I am relatively new to CouchDB, we just started doing a POC on it for our 
> Customer Grid like application. We installed it on a CentOS machine using yum 
> repositories and Everything worked well for some time and I was able to 
> replicate from one database to another locally and also to a remote database.
> 
> All of a sudden the replication setup stopped working. Whenever I try to set 
> a new replication channel I am getting an "undefined" message on the browser. 
> This is happening even when Source/Destination are specified to be local 
> databases. I have tried uninstalling, wiping off the data/log directories and 
> did a fresh install of couchdb 2.1.1 but the issue still persists. I can't 
> wrap my head around on what is going on here. Any help, thoughts or 
> suggestions is greatly appreciated. Many thanks in advance.
>  
> Attaching the log file. 
>  
> Thanks,
> Sajan Gone.

Re: [ANNOUNCE] Apache CouchDB 3.0.1 and 3.1.0 released

[Jan Lehnardt]

> On 6. May 2020, at 20:15, Damjan Georgievski  wrote:
> 
> On Wed, 6 May 2020 at 16:40, Robert Samuel Newson  wrote:
>> 
>> Make an issue (https://github.com/apache/couchdb/issues)
>> 
>> At first blush, I don't see why not, though I thought there was value in
>> Authorization: Bearer  from _not_ being a cookie. I guess those 
>> benefits
>> are not coupled with the token itself though.
> 
> I think the reasoning was that cookies can be long lived, and can
> persist in different places,
> and for JWTs that's generally undesirable. the Authorization: header
> is explicitly set by code, and shouldn't be persisted.

Plus browsers auto-sending cookies in many occasions when you don’t
want that. I forget the correct web security acronym this comes out to,
but it’s somewhere in the family of XSRF and XSS.

Best
Jan
—

Re: [Ticket ID: 975818] [ANNOUNCE] Apache CouchDB 3.0.1 and 3.1.0 released

[Jan Lehnardt]

Apparently I didn’t ;D

It looks like I’m not a moderator for this list. Can any one of you help?

Best
Jan
—

> On 6. May 2020, at 13:05, Jan Lehnardt  wrote:
> 
> FYI: I have removed this email address.
> 
>> On 6. May 2020, at 13:03, IdeaStack Support  wrote:
>> 
>> user@couchdb.apache.org,
>> 
>> Thank you for contacting our support team. A support ticket has now been 
>> opened for your request. You will be notified when a response is made by 
>> email. The details of your ticket are shown below.
>> 
>> Subject: Re: [ANNOUNCE] Apache CouchDB 3.0.1 and 3.1.0 released
>> Priority: Medium
>> Status: Open
>> 
>> You can view the ticket at any time at 
>> https://ideastack.com/order/viewticket.php?tid=975818=R6QER8vO
>> 
>> ideastack
> 

Re: [Ticket ID: 975818] [ANNOUNCE] Apache CouchDB 3.0.1 and 3.1.0 released

[Jan Lehnardt]

> On 6. May 2020, at 14:31, Joel Jucá  wrote:
> 
> QQ: it mentions support for Java Web Tokens, but what I understand of JWT
> is that it means JSON Web Tokens. Is it a typo or I'm confusing two
> conflicting acronyms?

It’s a running gag :)

Best
Jan
—
> 
> On Wed, May 6, 2020 at 8:05 AM Jan Lehnardt  wrote:
> 
>> FYI: I have removed this email address.
>> 
>>> On 6. May 2020, at 13:03, IdeaStack Support 
>> wrote:
>>> 
>>> user@couchdb.apache.org,
>>> 
>>> Thank you for contacting our support team. A support ticket has now been
>> opened for your request. You will be notified when a response is made by
>> email. The details of your ticket are shown below.
>>> 
>>> Subject: Re: [ANNOUNCE] Apache CouchDB 3.0.1 and 3.1.0 released
>>> Priority: Medium
>>> Status: Open
>>> 
>>> You can view the ticket at any time at
>> https://ideastack.com/order/viewticket.php?tid=975818=R6QER8vO
>>> 
>>> ideastack
>> 
>> 
> 
> -- 
> Joel Jucá
> joelwallis.com

Re: [Ticket ID: 975818] Re: [ANNOUNCE] Apache CouchDB 3.0.1 and 3.1.0 released

[Jan Lehnardt]

FYI: I have removed this email address.

> On 6. May 2020, at 13:03, IdeaStack Support  wrote:
> 
> user@couchdb.apache.org,
> 
> Thank you for contacting our support team. A support ticket has now been 
> opened for your request. You will be notified when a response is made by 
> email. The details of your ticket are shown below.
> 
> Subject: Re: [ANNOUNCE] Apache CouchDB 3.0.1 and 3.1.0 released
> Priority: Medium
> Status: Open
> 
> You can view the ticket at any time at 
> https://ideastack.com/order/viewticket.php?tid=975818=R6QER8vO
> 
> ideastack

Re: [ANNOUNCE] Apache CouchDB 3.0.1 and 3.1.0 released

[Jan Lehnardt]

> On 6. May 2020, at 12:55, Sebastien  wrote:
> 
> Awesome news, thanks to everyone involved!
> 
> A quick question about the JWT authentication support (greatest news in
> this release for my project). Are there plans for Couch to support
> extracting JWT tokens from a cookie?
> In some scenarios it can be easier/safer to pass JWT tokens to clients
> through cookies (secure, sameSite, httpOnly, etc) so that they don't store
> those in localStorage/have to add those themselves.
> 
> If you want I can create a ticket with this request (where?).

Tickets go here: https://github.com/apache/couchdb/issues

Best
Jan
—
> 
> kr,
> Sébastien D.
> 
> On Wed, May 6, 2020 at 9:59 AM Joan Touzet  wrote:
> 
>> Dear community,
>> 
>> Apache CouchDB® 3.0.1 and 3.1.0 have been released and are available for
>> download.
>> 
>> Apache CouchDB® lets you access your data where you need it. The Couch
>> Replication Protocol is implemented in a variety of projects and products
>> that span every imaginable computing environment from globally distributed
>> server-clusters, over mobile phones to web browsers.
>> 
>> Store your data safely, on your own servers, or with any leading cloud
>> provider. Your web- and native applications love CouchDB, because it speaks
>> JSON natively and supports binary data for all your data storage needs.
>> 
>> The Couch Replication Protocol lets your data flow seamlessly between
>> server clusters to mobile phones and web browsers, enabling a compelling
>> offline-first user-experience while maintaining high performance and strong
>> reliability. CouchDB comes with a developer-friendly query language, and
>> optionally MapReduce for simple, efficient, and comprehensive data
>> retrieval.
>> 
>> https://couchdb.apache.org/#download
>> 
>> Pre-built packages for Windows, macOS, Debian/Ubuntu and RHEL/CentOS are
>> available.
>> 
>> CouchDB 3.0.1 is a maintenance release, and was originally published on
>> 2020-05-05.
>> 
>> CouchDB 3.1.0 is a feature release, and was originally published on
>> 2020-05-05.
>> 
>> The community would like to thank all contributors for their part in
>> making this release, from the smallest bug report or patch to major
>> contributions in code, design, or marketing, we couldn’t have done it
>> without you!
>> 
>> See the official release notes document for an exhaustive list of all
>> changes:
>> 
>> http://docs.couchdb.org/en/stable/whatsnew/3.1.html
>> 
>> Release Notes highlights from 3.0.1:
>> 
>>   - A memory leak when encoding large binary content was patched
>> 
>>   - Improvements in documentation and defaults
>> 
>>   - JavaScript will no longer corrupt UTF-8 strings in various JS
>> functions
>> 
>> Release Notes highlights from 3.1.0:
>> 
>> Everything from 3.0.1, plus...
>> 
>>   - Support for Java Web Tokens
>> 
>>   - Support for SpiderMonkey 68, including binaries for Ubuntu 20.04
>> (Focal Fossa)
>> 
>>   - Up to a 40% performance improvement in the database compactor
>> 
>>   -
>> On behalf of the CouchDB PMC,
>> Joan Touzet
>> 
>> 

Re: What happens when I combine the view query params sorted=false with startkey & endkey?

[Jan Lehnardt]

Heya Thilo,

startkey and endkey are applied first, so you are guaranteed to get all results 
within your specified range, but you are not guaranteed to get it in the sorted 
order defined by the second link you sent.

Imagine a sharded database. When making a view query, CouchDB makes an internal 
query with the same options (including startkey/endkey) to each shard. To 
ensure a sorted result (the default), the results from each shard get streamed 
to the node that handles the query, and that node does an in-flight merge of 
the incoming results.

The sorted=false option just returns all the result rows as they come in from 
the shards.

The potential performance gain is that some shards might reply faster than 
others, but to ensure an in-order result, those shard-results have to wait 
before they can be streamed to the client because a row from a slower shard 
might have logical precedence.

If you can sort on the client, or if the order doesn’t matter (just the 
startkey/endkey selection), then sorted=false can give you some potential 
speedup on a sharded database. You won’t go faster than your slowest shard, 
however, you might just get other shards’ results sooner.

Best
Jan
—

> On 9. Apr 2020, at 17:40, Thilo from Cobot  wrote:
> 
> Hi there,
> 
> I was looking into speeding up queries to the database. I found under
> https://docs.couchdb.org/en/stable/api/ddoc/views.html#db-design-design-doc-view-view-name
> that I can disable sorting of results with sorted=false. I wonder what 
> happens if I combine that with the startkey/endkey parameters.
> 
> What I understood from 
> https://docs.couchdb.org/en/stable/api/ddoc/views.html#api-ddoc-view-sorting
> sorting is applied before filtering. I assume the result is undetermined, it 
> might include some docs that are between start and endkey but not all or 
> none, so its not wise to combine sorted=false with any filters. Is this 
> correct?
> 
> Thanks for any insights.
> 
> Cheers
> Thilo
> 
> --
> more time for your coworkers: https://cobot.me

Re: SpiderMonkey 60 in CouchDB 3.x

[Jan Lehnardt]

Hi Joel,

> On 25. Mar 2020, at 18:29, Joel Jucá  wrote:
> 
> Hello community!
> 
> I saw the announcement of CouchDB 3 with great enthusiasm when there was a
> mention of the updated JavaScript engine, SpiderMonkey version 60. It seems
> to fully support ES6, so it would be possible to write queries and/or
> design documents using JavaScript modules (eg: ESM, CommonJS, etc.)?

You can already use CommonJS modules in CouchDB. This continues to work:

   https://docs.couchdb.org/en/stable/query-server/javascript.html#commonjs

We didn’t build any support for ESM and that likely won’t happen, unless
someone volunteers to champion that.


> The thing that I always wondered would be how a professional flow could be
> set up for a code that's supposed to live inside CouchDB. How could I write
> code that's submitted to tests on a CI pipeline, and deployed to a CouchDB
> instance on successful executions? Or, how could I leverage JavaScript/npm
> ecosystem to build such functionalities? Also, how could I provision a
> database from a blank-slate CouchDB (eg: a Docker image) and inject all
> configurations *and* documents on it automatically? Is it something that
> SpiderMonkey 60 could be helpful with?

I recommend doing all this tooling outside of CouchDB. My favourite tool for
this is https://github.com/jo/couchdb-bootstrap which allows significant
automation of many things you might want to do.

To add a layer of CI testing, I built https://github.com/janl/couchdb-ddoc-test 
some time ago. It is very basic, but does the job.

Best
Jan
—

> PS: I'm a basic user of CouchDB. I currently do not run services using it,
> but I've been interested in using it for a while - hence my questions.
> 
> -- 
> Joel Jucá
> about.me/joelwallis

Re: CouchDB 2.3.1 - Really Big Document Breaks View

[Jan Lehnardt]

You also need to raise the os_process_timeout

Cheers
Jan
—

> On 9. Apr 2020, at 22:29, Mantell, Christopher Arthur 
>  wrote:
> 
> Hi,
> 
> One document in our database is really big (1.2 million lines) and it causes 
> an OS_process_error when we access it.  I've tried increasing the 
> os_process_timeout to 12 but it didn't fix it.  I don't know if this 
> should be increased further.  Deleting the document causes the view to work 
> again.
> 
> Unfortunately there is no ideal way around not using this document or 
> splitting it up into smaller documents.  Is there anything I can do to 
> Couch's configurations so this document won't break the view?
> 
> Here is the output in the couch.log:
> 
> [error] 2020-04-09T17:29:05.046811Z couchdb@127.0.0.1 <0.2218.0> 4e36c78fcd 
> rexi_server: from: couchdb@127.0.0.1(<0.745.0>) mfa: fabric_rpc:map_view/5 
> exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.e\
> rl"},{line,265}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,205}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,462}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,526}]},\
> {couch_mrview_util,fold_fun,4,[{file,"src/couch_mrview_util.erl"},{line,437}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_bt\
> ree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,219}]}]
> 
> Any help would be greatly appreciated!
> 
> Thank you,
> 
> 
> Chris Mantell
> 
> Programmer
> 
> Athinoula A Martinos Center for Biomedical Imaging
> Massachusetts General Hospital
> Building 149 - South Central
> Charlestown, MA 02129
> cmant...@mgh.harvard.edu
> 
> 
> The information in this e-mail is intended only for the person to whom it is
> addressed. If you believe this e-mail was sent to you in error and the e-mail
> contains patient information, please contact the Partners Compliance HelpLine 
> at
> http://www.partners.org/complianceline . If the e-mail was sent to you in 
> error
> but does not contain patient information, please contact the sender and 
> properly
> dispose of the e-mail.

Re: [DISCUSS] moving email lists to Discourse

[Jan Lehnardt]

> On 12. Mar 2020, at 16:21, Paul Davis  wrote:
> 
> I'm not against anything of that nature, but if memory serves the
> email lists are dictated by ASF policy.

If you remember when we did the GitHub transition, as long as we can make
sure messages end up on a mailing list, we should be fine wrt the requirements.

Best
Jan
—
> 
> On Thu, Mar 12, 2020 at 9:32 AM Garren Smith  wrote:
>> 
>> Hi All,
>> 
>> The CouchDB slack channel has been a real success with lots of people
>> asking for help and getting involved. The main issue is that it is not
>> searchable so we often get people asking the same questions over and over.
>> The user mailing list is great in that sense that if you have subscribed to
>> it you have a searchable list of questions and answers. However, it's
>> really not user-friendly and judging by the fact that it has very low user
>> participation I'm guessing most people prefer to use slack to ask questions.
>> 
>> I've been really impressed with how the FoundationDB forum[1] and the rust
>> internal forum work [2]. I find them easy to use and really encourage
>> participation. I would like to propose that we move our user and dev
>> discussion to Discourse or a forum that works as well as Discourse. I think
>> that would make it really easy for users of CouchDB to look up answers to
>> questions and get involved in the development discussion.
>> 
>> I haven't checked yet, but I'm sure we could get all discourse threads to
>> automatically email back to the user and dev mailing list so that we still
>> fulfill our Apache requirements.
>> 
>> I know its a big step away from what we're used to with our mailing lists,
>> but I think it would definitely open up our community.
>> 
>> Cheers
>> Garren
>> 
>> 
>> [1] https://forums.foundationdb.org/
>> [2] https://internals.rust-lang.org/

Re: Problem with unauthorized CouchDb 3.0 and ssl

[Jan Lehnardt]

Heya Marcus,

Newly created databases are admin-only by default in 3.0 (which is different 
from 2.x):

https://blog.couchdb.org/2020/02/26/the-road-to-couchdb-3-0-security/ 


So you’ll have to make an authenticated request against /dbname/_all_docs or 
change that database’s _security object, if you want to make the database 
publicly readable.

SSL should not play into this, as you’re getting an error message transmitted 
via SSL.

I just note that you showed a `http://`  and not a `https://` 
 request.

Best
Jan
—

> On 10. Mar 2020, at 11:39, Marcus Bieber  wrote:
> 
> Hi everyone,
> 
> I have a dev environment with CouchDB 2.3.1 which is running well.
> I have an admin and no ssl.
> So calls like http://1xx.1xx.xxx.xx:5984/dbname/_all_docs are no problem.
> 
> So now I wanted to setup the live server. CouchDB 3.0 with ssl and an admin.
> Running fine.
> Replication worked.
> Now I get on the same call:
> 
> {
> "error": "unauthorized",
> "reason": "You are not authorized to access this db."
> }
> 
> Did I miss / forget something.
> Is this a problem caused by changes in 3.0 or could this be a problem of the 
> ssl?
> 
> Can someone help me please.
> 
> Thanks in advance and kind regards
> Marcus
> 

-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

[ANNOUNCE] Apache CouchDB 3.0.0 released

[Jan Lehnardt]

Dear community,

Apache CouchDB® 3.0.0 has been released and is available for download.

Apache CouchDB® lets you access your data where you need it. The Couch 
Replication Protocol is implemented in a variety of projects and products that 
span every imaginable computing environment from globally distributed 
server-clusters, over mobile phones to web browsers.

Store your data safely, on your own servers, or with any leading cloud 
provider. Your web- and native applications love CouchDB, because it speaks 
JSON natively and supports binary data for all your data storage needs.

The Couch Replication Protocol lets your data flow seamlessly between server 
clusters to mobile phones and web browsers, enabling a compelling offline-first 
user-experience while maintaining high performance and strong reliability. 
CouchDB comes with a developer-friendly query language, and optionally 
MapReduce for simple, efficient, and comprehensive data retrieval.

https://couchdb.apache.org/#download

Pre-built packages for Windows, macOS, Debian/Ubuntu and RHEL/CentOS are 
available. Docker images have been submitted to Docker Hub for review and will 
be available as soon as that  process is done.

CouchDB 3.0.0 is a major release, and was originally published on 2020-02-26.

The community would like to thank all contributors for their part in making 
this release, from the smallest bug report or patch to major contributions in 
code, design, or marketing, we couldn’t have done it without you!

See the official release notes document for an exhaustive list of all changes:

http://docs.couchdb.org/en/stable/whatsnew/3.0.html

Release Notes highlights:

  - Default installations are now secure and locked down.

  - User-defined partitioned databases for faster querying

  - Live Shard Splitting for incremental scale-out

  - Updated to modern JavaScript engine SpiderMonkey 60

  - Official support for ARM and PPC 32bit and 64bit systems

  - Many large and small performance improvements

  - Automatic view index warmer

  - Smarter Compaction Daemon

  - Smarter I/O Queue

  - Much improved installers for Windows

  - macOS binaries are now Notarized for full future Catalina support

  - Extremely simplified setup of Lucene search

See the “Road to CouchDB 3.0” blog post series for many more details: 
http://blog.couchdb.org/2020/02/25/the-road-to-couchdb-3-0/

On behalf of the CouchDB PMC,
Jan Lehnardt
—

Re: Re: Re: Backup Question on Version 2.2.0

[Jan Lehnardt]

curl http://127.0.0.1:5984/_node/_local/_config/couchdb/view_index_dir

> On 18. Nov 2019, at 16:26, Krawetzky, Peter J  
> wrote:
> 
> I have views but I don't find that sub-directory you mentioned in my 
> directory tree.  What is the default of view_index_dir?
> 
> 
> 
> Peter J Krawetzky | Senior Technical Infrastructure Specialist, Automation 
> Engineering
> P 860-808-6112
> 151 Farmington Avenue, Mail Code AN14, Hartford, CT 06156
> 
> CONFIDENTIALITY NOTICE: This communication and any attachments may contain 
> confidential and/or privileged information for the use of the designated 
> recipients named above. If you are not the intended recipient, you are hereby 
> notified that you have received this communication in error and that any 
> review, disclosure, dissemination, distribution or copying of it or its 
> contents is prohibited. If you have received this communication in error, 
> please notify the sender immediately by email or telephone and destroy all 
> copies of this communication and any attachments.
> 
> 
> Proprietary
> 
> -Original Message-
> From: Jan Lehnardt  
> Sent: Monday, November 18, 2019 10:21 AM
> To: user@couchdb.apache.org
> Subject: [EXTERNAL] Re: Re: Backup Question on Version 2.2.0
> 
>  External Email - Use Caution 
> 
> That looks all normal. Your database dir is /u1/couchdb and you have a 
> /u1/couchdb/shards dir, that’s where all your couch files live, separated by 
> one more dir level that represents the shard range.
> 
> The .shards dir exists in the view_index_dir (not database_dir) 
> configuration, and it includes all view indexes. if you don’t have any views 
> at all, you might not have a .shards dir. If you have views, your view 
> indexes live in view_index_dir/.shards
> 
> Best
> Jan
> —
> 
>> On 18. Nov 2019, at 16:18, Krawetzky, Peter J 
>>  wrote:
>> 
>> Thanks and that is the document I'm referencing which is confusing me.  I 
>> don't have a data/.shards directory.  Here is my structure.  Take note that 
>> databases in my shards directory do no appear under the database_dir 
>> location specified in my local.ini file.
>> 
>> $ cd /u01/couchdb
>> $ ls -al
>> total 592
>> drwxr-xr-x  4 couchdb couchdb233 Oct 31  2018 .
>> drwxr-xr-x  4 rootroot43 Oct 15  2018 ..
>> -rw-r--r--  1 couchdb couchdb 106703 Oct 31  2018 build_queue.couch
>> -rw-r--r--  1 couchdb couchdb  82111 Feb 14  2019 _dbs.couch
>> drwxr-xr-x  2 couchdb couchdb  6 Nov 18 10:09 .delete
>> -rw-r--r--  1 couchdb couchdb 118960 Oct 31  2018 dev.couch
>> -rw-r--r--  1 couchdb couchdb  45260 Oct 31  2018 hieradb.couch
>> -rw-r--r--  1 couchdb couchdb   4272 Oct 15  2018 _nodes.couch
>> -rw-r--r--  1 couchdb couchdb  73901 Oct 31  2018 prod.couch
>> -rw-r--r--  1 couchdb couchdb  16582 Oct 31  2018 qa.couch
>> -rw-r--r--  1 couchdb couchdb   4278 Oct 15  2018 _replicator.couch
>> drwxr-xr-x 10 couchdb couchdb206 Oct 15  2018 shards
>> -rw-r--r--  1 couchdb couchdb 110765 Oct 31  2018 test.couch
>> -rw-r--r--  1 couchdb couchdb   4278 Oct 15  2018 _users.couch
>> 
>> In my /opt/couchdb/etc/local.ini file I have:
>> database_dir = /u01/couchdb
>> 
>> When I list all of the shards sub-directories I see all of my databases:
>> $ pwd
>> /u01/couchdb/shards
>> $ ls -alR
>> .:
>> total 32
>> drwxr-xr-x 10 couchdb couchdb  206 Oct 12  2018 .
>> drwxr-xr-x  4 couchdb couchdb  297 Jun 20 11:39 ..
>> drwxr-xr-x  2 couchdb couchdb 4096 Oct 28 15:13 -1fff 
>> drwxr-xr-x  2 couchdb couchdb 4096 Oct 28 15:11 2000-3fff 
>> drwxr-xr-x  2 couchdb couchdb 4096 Oct 28 15:10 4000-5fff 
>> drwxr-xr-x  2 couchdb couchdb 4096 Oct 28 15:10 6000-7fff 
>> drwxr-xr-x  2 couchdb couchdb 4096 Oct 28 15:08 8000-9fff 
>> drwxr-xr-x  2 couchdb couchdb 4096 Nov  5 14:05 a000-bfff 
>> drwxr-xr-x  2 couchdb couchdb 4096 Oct 31 11:21 c000-dfff 
>> drwxr-xr-x  2 couchdb couchdb 4096 Oct 28 15:05 e000-
>> 
>> ./-1fff:
>> total 4448
>> drwxr-xr-x  2 couchdb couchdb4096 Oct 28 15:13 .
>> drwxr-xr-x 10 couchdb couchdb 206 Oct 12  2018 ..
>> -rw-r--r--  1 couchdb couchdb   90310 Jul  9 08:46 
>> build_queue.1539344519.couch
>> -rw-r--r--  1 couchdb couchdb   28864 Jul  9 08:51 
>> cloudforms.1539344527.couch
>> -rw-r--r--  1 couchdb couchdb   45244 Oct 28 15:13 
>> _global_changes.1539344313.couch
>> -rw-r--r--  1 couchdb couchdb  118982 Sep 16 09:34 hieradb.1539344528.couch
>> -rw-r--r--  1 couchdb couchdb   41107 Mar 

Re: Re: Backup Question on Version 2.2.0

[Jan Lehnardt]

chdb  127170 Oct 31 11:23 
> _global_changes.1539344313.couch
> -rw-r--r--  1 couchdb couchdb  123078 Jul  9 08:59 hieradb.1539344528.couch
> -rw-r--r--  1 couchdb couchdb   41107 Mar 21  2019 
> hieramanager_test.1540907701.couch
> -rw-r--r--  1 couchdb couchdb   82124 Oct 31 11:24 
> iac_birth_certificate.1542119804.couch
> -rw-r--r--  1 couchdb couchdb   12435 Nov 13  2018 
> iac_hostname_generator.1542119640.couch
> -rw-r--r--  1 couchdb couchdb  939951 Nov 18 10:16 
> iac_metadata_manager.1561041628.couch
> -rw-r--r--  1 couchdb couchdb   94399 Jul  9 09:01 
> iac_reference_data.1542135843.couch
> -rw-r--r--  1 couchdb couchdb   16575 Nov  5  2018 jdsandbox.1539344533.couch
> -rw-r--r--  1 couchdb couchdb   24759 Jul  9 09:01 labalpha.1539344538.couch
> -rw-r--r--  1 couchdb couchdb   20665 Jul  9 09:01 labbeta.1539344543.couch
> -rw-r--r--  1 couchdb couchdb   37049 Jul  9 09:02 labrand.1539344548.couch
> -rw-r--r--  1 couchdb couchdb4240 Oct 12  2018 metadata.1539344552.couch
> -rw-r--r--  1 couchdb couchdb  221381 Oct 28 15:07 
> metadata_dictionary.1539885019.couch
> -rw-r--r--  1 couchdb couchdb   37061 Oct 28 15:07 
> metadata_dictionary_classes.1547583298.couch
> -rw-r--r--  1 couchdb couchdb 1976599 Nov 18 10:15 
> metadata_dictionary_faq.1550175781.couch
> -rw-r--r--  1 couchdb couchdb   12432 Jan 30  2019 
> pjk_rep_test.1539891783.couch
> -rw-r--r--  1 couchdb couchdb   90300 Jul  9 09:02 
> _replicator.1539344313.couch
> -rw-r--r--  1 couchdb couchdb   12467 Jan 29  2019 _users.1539344313.couch
> 
> ./e000-:
> total 3848
> drwxr-xr-x  2 couchdb couchdb4096 Oct 28 15:05 .
> drwxr-xr-x 10 couchdb couchdb 206 Oct 12  2018 ..
> -rw-r--r--  1 couchdb couchdb   53439 Jul  9 07:39 
> build_queue.1539344519.couch
> -rw-r--r--  1 couchdb couchdb4240 Oct 12  2018 cloudforms.1539344527.couch
> -rw-r--r--  1 couchdb couchdb   65727 Sep 16 09:34 
> _global_changes.1539344313.couch
> -rw-r--r--  1 couchdb couchdb   65727 Jul  9 08:59 hieradb.1539344528.couch
> -rw-r--r--  1 couchdb couchdb   24752 Jul  9 09:00 
> hieramanager_test.1540907701.couch
> -rw-r--r--  1 couchdb couchdb   37061 Aug 21 15:39 
> iac_birth_certificate.1542119804.couch
> -rw-r--r--  1 couchdb couchdb   16538 Nov 13  2018 
> iac_hostname_generator.1542119640.couch
> -rw-r--r--  1 couchdb couchdb  939312 Nov 18 10:14 
> iac_metadata_manager.1561041628.couch
> -rw-r--r--  1 couchdb couchdb   24767 Sep 20 17:42 
> iac_reference_data.1542135843.couch
> -rw-r--r--  1 couchdb couchdb   45247 Dec 12  2018 jdsandbox.1539344533.couch
> -rw-r--r--  1 couchdb couchdb   41158 Jul  9 09:02 labalpha.1539344538.couch
> -rw-r--r--  1 couchdb couchdb   98495 Oct 30 10:33 labbeta.1539344543.couch
> -rw-r--r--  1 couchdb couchdb  102598 Oct 30 10:33 labrand.1539344548.couch
> -rw-r--r--  1 couchdb couchdb   20656 Jul  9 09:03 metadata.1539344552.couch
> -rw-r--r--  1 couchdb couchdb  200901 Oct 28 15:05 
> metadata_dictionary.1539885019.couch
> -rw-r--r--  1 couchdb couchdb   28869 Oct 28 15:05 
> metadata_dictionary_classes.1547583298.couch
> -rw-r--r--  1 couchdb couchdb 1978382 Nov 18 10:16 
> metadata_dictionary_faq.1550175781.couch
> -rw-r--r--  1 couchdb couchdb   12473 Jan 30  2019 
> pjk_rep_test.1539891783.couch
> -rw-r--r--  1 couchdb couchdb   49340 Jul  9 09:22 
> _replicator.1539344313.couch
> -rw-r--r--  1 couchdb couchdb   37049 Jan 29  2019 _users.1539344313.couch
> 
> 
> 
> Peter J Krawetzky | Senior Technical Infrastructure Specialist, Automation 
> Engineering
> P 860-808-6112
> 151 Farmington Avenue, Mail Code AN14, Hartford, CT 06156
> 
> CONFIDENTIALITY NOTICE: This communication and any attachments may contain 
> confidential and/or privileged information for the use of the designated 
> recipients named above. If you are not the intended recipient, you are hereby 
> notified that you have received this communication in error and that any 
> review, disclosure, dissemination, distribution or copying of it or its 
> contents is prohibited. If you have received this communication in error, 
> please notify the sender immediately by email or telephone and destroy all 
> copies of this communication and any attachments.
> 
> 
> Proprietary
> 
> -Original Message-
> From: Jan Lehnardt  
> Sent: Monday, November 18, 2019 10:05 AM
> To: CouchDB Users 
> Subject: [EXTERNAL] Re: Backup Question on Version 2.2.0
> 
>  External Email - Use Caution 
> 
> Hi Peter,
> 
> The documentation page on Backups explains how to backup sharded database 
> files: 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.couchdb.org_en_stable_maintenance_backups.html=DwIFAg=wluqKIiwffOpZ6k5sqMWMBOn0vyYnlulRJmmvOXCFpM=Uls

Re: Backup Question on Version 2.2.0

[Jan Lehnardt]

Hi Peter,

The documentation page on Backups explains how to backup sharded database 
files: http://docs.couchdb.org/en/stable/maintenance/backups.html

Best
Jan
-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/


> On 18. Nov 2019, at 15:50, Krawetzky, Peter J  
> wrote:
> 
> I know the document recommendation is to use replication for backups but I 
> want to make sure I provide a recovery solution in case someone totally 
> deletes a document or data within a document by accident and I can recover 
> that.  Replication doesn't provide for this situation.
> 
> The documentation references backing up .couch files but my implementation 
> doesn't have those.  All of my database files are broken up under the shards 
> directory.  I have no single point of  a database name based on the 
> documentation reference.  Do I just copy the shards directory now?
> 
> Any advice would be great.
> 
> 
> 
> CONFIDENTIALITY NOTICE: This communication and any attachments may contain 
> confidential and/or privileged information for the use of the designated 
> recipients named above. If you are not the intended recipient, you are hereby 
> notified that you have received this communication in error and that any 
> review, disclosure, dissemination, distribution or copying of it or its 
> contents is prohibited. If you have received this communication in error, 
> please notify the sender immediately by email or telephone and destroy all 
> copies of this communication and any attachments.
> 
> 
> 
> Proprietary
> 
> NOTICE TO RECIPIENT OF INFORMATION:
> This e-mail may contain confidential or privileged information. If you think 
> you have received this e-mail in error, please advise the sender by reply 
> e-mail and then delete this e-mail immediately.  
> This e-mail may also contain protected health information (PHI) with 
> information about sensitive medical conditions, including, but not limited 
> to, treatment for substance use disorders, behavioral health, HIV/AIDS, or 
> pregnancy. This type of information may be protected by various federal 
> and/or state laws which prohibit any further disclosure without the express 
> written consent of the person to whom it pertains or as otherwise permitted 
> by law. Any unauthorized further disclosure may be considered a violation of 
> federal and/or state law. A general authorization for the release of medical 
> or other information may NOT be sufficient consent for release of this type 
> of information.
> Thank you. Aetna

Apache CouchDB at Kubecon and FDB Summit: Free FoundationDB Summit Tickets Available

[Jan Lehnardt]

Hey all,

IBM are giving away free tickets to the FoundationDB Summit during Kubecon San 
Diego: 

https://blog.couchdb.org/2019/10/10/apache-couchdb-at-kubecon-and-fdb-summit-free-foundationdb-summit-tickets-available/

Best
Jan
-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

[ANNOUNCE] CouchDB Meetup in Berlin around ApacheCon EU, October 21st.

[Jan Lehnardt]

Dear CouchDB users and developers,

I’m happy to announce our CouchDB meetup in Berlin just before ApacheCon EU:

https://www.meetup.com/couchberlin/events/263966219/


There’s going to be CouchDB Committers, PMC members as well as a number of 
users form Berlin and all over Europe.

We’re running a small Call for Speakers for the next two weeks. You can find 
all the relevant info on the meetup page. We’d love to hear your CouchDB story.


Please RSVP, and I”m excited to see you all in Berlin!

If you have any questions, do let me know :)


Best
Jan
-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

Re: CouchDB 3.0 features?

[Jan Lehnardt]

Hi Brian,

this reflects our current thinking:

  https://github.com/apache/couchdb/projects/1

It is not too late to pitch in ;)

Best
Jan
-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/


> On 2. Aug 2019, at 20:51, Brian Cottingham  wrote:
> 
> I read the interview[0] about CouchDB 3.0 and I'm excited to see what's it 
> it! Is there a roadmap and a (very) rough ETA for it?
> 
> [0]: https://www.ibm.com/cloud/blog/new-builders/database-deep-dives-couchdb
> 
> -Brian

Re: Bind to localhost without require valid user

[Jan Lehnardt]

Only other admins can see admin's credentials in relocation docs.

Regular users can only see their own replication docs. 

Cheers
Jan
—

> On 29. Jul 2019, at 15:11, Dilushan Delgoda  wrote:
> 
> Hi.
> 
> Thank you very much for your reply.
> 
> I'm trying to replicate a database by putting a document to _replicator.
> And when i set source and target "http://admin:password@localhost:5984/db;
> like this replication happening without any issue. But when i try without
> putting username and password "http://localhost:5984/db; replication not
> happening. Is there workaround to solve this? I don't want to put password
> in replication doc. And source, target and _replicator in same couchdb
> instance.
> 
> This is because even a db user with view access can then log in and see the
> admin password which would compromise security.
> 
> Best regards
> 
> On Mon, Jul 29, 2019 at 3:19 PM Matteo Guadrini 
> wrote:
> 
>> I don't think we can do ... because, network traffic at a much lower level
>> than the application.
>> The CouchDB server (frontend) sees GET / PUT / POST coming from a certain
>> ip. At this point, he asks the database specified in the url for the
>> request and the server itself will verify the security.
>> But there is no TCP / IP control at this level. Rather, you could make two
>> servers, the first one responding only to localhost in AdminParty and the
>> second with the user and password and in bind on a private / public ip. The
>> second will receive the db in reply from the first.
>> It could be an idea...
>> 
>> Matteo Guadrini
>> 
>> 
>> Da: Dilushan Delgoda 
>> Inviato: lunedì 29 luglio 2019 10:29
>> A: user@couchdb.apache.org 
>> Oggetto: Bind to localhost without require valid user
>> 
>> Is it possible to bind Couchdb to localhost without require valid user and
>> all other interfaces with require valid user setting. So when application
>> access through localhost that applications doesn't require username
>> password to Couchdb and applications access from external interfaces
>> require username password.
>> 
>> --
>> Regards,
>> Dilushan Delgoda
>> 
> 
> 
> -- 
> Regards,
> Dilushan Delgoda

Re: can't get couchdb installed (on a fresh 18.04 ubuntu)

[Jan Lehnardt]

Jut a quick note that the official install instructions are here:

http://docs.couchdb.org/en/stable/install/index.html

And I saw this issue fly by which might be related?

https://github.com/apache/couchdb-pkg/issues/44

Best
Jan
—

> On 24. Jun 2019, at 10:55, Rene Veerman  wrote:
> 
> i followed https://linuxize.com/post/how-to-install-couchdb-on-ubuntu-18-04/
> 
> 
> but ended up with :
> 
> root@crow:~# apt upgrade
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Calculating upgrade... Done
> The following packages were automatically installed and are no longer
> required:
>  libargon2-0 libcryptsetup12 libdevmapper1.02.1 libidn11 libip4tc0
> libjson-c3
>  libkmod2 systemd
> Use 'apt autoremove' to remove them.
> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
> 16 not fully installed or removed.
> After this operation, 0 B of additional disk space will be used.
> Do you want to continue? [Y/n]
> Processing triggers for mime-support (3.60ubuntu1) ...
> Setting up python3 (3.6.7-1~18.04) ...
> running python rtupdate hooks for python3.6...
> dpkg-query: package 'hplip-data' is not installed
> Use dpkg --info (= dpkg-deb --info) to examine archive files,
> and dpkg --contents (= dpkg-deb --contents) to list their contents.
> Traceback (most recent call last):
>  File "/usr/bin/py3clean", line 210, in 
>main()
>  File "/usr/bin/py3clean", line 196, in main
>pfiles = set(dpf.from_package(options.package))
>  File "/usr/share/python3/debpython/files.py", line 53, in from_package
>raise Exception("cannot get content of %s" % package_name)
> Exception: cannot get content of hplip-data
> error running python rtupdate hook hplip-data
> dpkg-query: package 'ibus-hangul' is not installed
> Use dpkg --info (= dpkg-deb --info) to examine archive files,
> and dpkg --contents (= dpkg-deb --contents) to list their contents.
> Traceback (most recent call last):
>  File "/usr/bin/py3clean", line 210, in 
>main()
>  File "/usr/bin/py3clean", line 196, in main
>pfiles = set(dpf.from_package(options.package))
>  File "/usr/share/python3/debpython/files.py", line 53, in from_package
>raise Exception("cannot get content of %s" % package_name)
> Exception: cannot get content of ibus-hangul
> error running python rtupdate hook ibus-hangul
> dpkg-query: package 'ibus-libpinyin' is not installed
> Use dpkg --info (= dpkg-deb --info) to examine archive files,
> and dpkg --contents (= dpkg-deb --contents) to list their contents.
> Traceback (most recent call last):
>  File "/usr/bin/py3clean", line 210, in 
>main()
>  File "/usr/bin/py3clean", line 196, in main
>pfiles = set(dpf.from_package(options.package))
>  File "/usr/share/python3/debpython/files.py", line 53, in from_package
>raise Exception("cannot get content of %s" % package_name)
> Exception: cannot get content of ibus-libpinyin
> error running python rtupdate hook ibus-libpinyin
> dpkg-query: package 'ibus-table' is not installed
> Use dpkg --info (= dpkg-deb --info) to examine archive files,
> and dpkg --contents (= dpkg-deb --contents) to list their contents.
> Traceback (most recent call last):
>  File "/usr/bin/py3clean", line 210, in 
>main()
>  File "/usr/bin/py3clean", line 196, in main
>pfiles = set(dpf.from_package(options.package))
>  File "/usr/share/python3/debpython/files.py", line 53, in from_package
>raise Exception("cannot get content of %s" % package_name)
> Exception: cannot get content of ibus-table
> error running python rtupdate hook ibus-table
> dpkg-query: package 'ibus' is not installed
> Use dpkg --info (= dpkg-deb --info) to examine archive files,
> and dpkg --contents (= dpkg-deb --contents) to list their contents.
> Traceback (most recent call last):
>  File "/usr/bin/py3clean", line 210, in 
>main()
>  File "/usr/bin/py3clean", line 196, in main
>pfiles = set(dpf.from_package(options.package))
>  File "/usr/share/python3/debpython/files.py", line 53, in from_package
>raise Exception("cannot get content of %s" % package_name)
> Exception: cannot get content of ibus
> error running python rtupdate hook ibus
> dpkg-query: package 'python3-uno' is not installed
> Use dpkg --info (= dpkg-deb --info) to examine archive files,
> and dpkg --contents (= dpkg-deb --contents) to list their contents.
> Traceback (most recent call last):
>  File "/usr/bin/py3clean", line 210, in 
>main()
>  File "/usr/bin/py3clean", line 196, in main
>pfiles = set(dpf.from_package(options.package))
>  File "/usr/share/python3/debpython/files.py", line 53, in from_package
>raise Exception("cannot get content of %s" % package_name)
> Exception: cannot get content of python3-uno
> error running python rtupdate hook python3-uno
> dpkg-query: package 'rhythmbox-plugin-alternative-toolbar' is not installed
> Use dpkg --info (= dpkg-deb --info) to examine archive files,
> and dpkg --contents (= dpkg-deb --contents) to list their contents.
> 

Re: Need help with replication

[Jan Lehnardt]

And don’t replicate _global_changes! :)

> On 17. May 2019, at 00:35, Mark Richter  wrote:
> 
> Third time - the image is here: 
> https://drive.google.com/open?id=1pU3GL9cbhtLuFdqfl9kyEw5lVAwPhCjJ
> 
> Thanks (in advance).
> 
> From: Mark Richter
> Sent: Thursday, May 16, 2019 3:30 PM
> To: user@couchdb.apache.org
> Subject: RE: Need help with replication
> 
> Ok, I tried to attach the image and that doesn't appear to work.  I also 
> tried to create a gist containing the image and that also did not work.
> 
> Suggestions please?
> 
> 
> From: Mark Richter mailto:mrich...@solarflare.com>>
> Sent: Thursday, May 16, 2019 3:22 PM
> To: user@couchdb.apache.org
> Subject: Need help with replication
> 
> My current project is attempting to set up a couchdb cluster, but so far that 
> has failed miserably.
> 
> As a fallback, we're trying to set up couchdb replication from one server to 
> a backup server, but this also does not seem to work.  I have couchdb 
> installed on both machines, one with a fair number of active databases, the 
> other just a plain fresh installation.  No replication is working.
> 
> Here is a screenshot of the current state of the replications:
> 
> [cid:image001.png@01D50BFB.2968D290]
> 
> Can someone tell me what I'm doing wrong?
> 
> Thanks.
> 
> 
> Mark Richter | Senior Staff Engineer
> SolarFlare Communications, Inc. | 
> www.Solarflare.com
> 9444 Waples Street, #170, San Diego, CA  92121
> Mobile: +1 949-632-8403
> [Description: Description: cid:EC628FDE-ACA6-4F34-A8AE-E1F672D4E395]
> 
> The information contained in this message is confidential and is intended for 
> the addressee(s) only. If you have received this message in error, please 
> notify the sender immediately and delete the message. Unless you are an 
> addressee (or authorized to receive for an addressee), you may not use, copy 
> or disclose to anyone this message or any information contained in this 
> message. The unauthorized use, disclosure, copying or alteration of this 
> message is strictly prohibited.
> The information contained in this message is confidential and is intended for 
> the addressee(s) only. If you have received this message in error, please 
> notify the sender immediately and delete the message. Unless you are an 
> addressee (or authorized to receive for an addressee), you may not use, copy 
> or disclose to anyone this message or any information contained in this 
> message. The unauthorized use, disclosure, copying or alteration of this 
> message is strictly prohibited.

-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

Re: can´t install couchdb?

[Jan Lehnardt]

now you have missed this particular sentence:

> and replace {distribution} with the appropriate choice for your OS version:



> On 16. May 2019, at 15:56, Rene Veerman  wrote:
> 
> thanks.
> 
> but it still doesnt work :
> root@albatross:~# sudo apt update
> Hit:1 http://nl.archive.ubuntu.com/ubuntu bionic InRelease
> Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88,7
> kB]
> Hit:3 http://nl.archive.ubuntu.com/ubuntu bionic-updates
> InRelease
> Hit:4 http://nl.archive.ubuntu.com/ubuntu bionic-backports
> InRelease
> Ign:5 https://apache.bintray.com/couchdb-deb {distribution}
> InRelease
> Err:6 https://apache.bintray.com/couchdb-deb {distribution}
> Release
>  404  Not Found [IP: 54.93.138.218 443]
> Reading package lists... Done
> E: The repository 'https://apache.bintray.com/couchdb-deb {distribution}
> Release' does not have a Release file.
> N: Updating from such a repository can't be done securely, and is therefore
> disabled by default.
> N: See apt-secure(8) manpage for repository creation and user configuration
> details.
> 
> 
> On Thu, May 16, 2019 at 3:44 PM Jan Lehnardt  wrote:
> 
>> looks like you skipped this step:
>> 
>> 
>> http://docs.couchdb.org/en/stable/install/unix.html#enabling-the-apache-couchdb-package-repository
>> 
>> Best
>> Jan
>> —
>> 
>>> On 16. May 2019, at 15:39, Rene Veerman  wrote:
>>> 
>>> ubuntu 18.04, installed, updated, then :
>>> 
>>> root@albatross:~# curl -L
>> https://couchdb.apache.org/repo/bintray-pubkey.asc
>>> | sudo apt-key add -
>>> % Total% Received % Xferd  Average Speed   TimeTime Time
>>> Current
>>>Dload  Upload   Total   SpentLeft
>>> Speed
>>> 100  3100  100  31000 0  20129  0 --:--:-- --:--:-- --:--:--
>>> 20129
>>> OK
>>> root@albatross:~# sudo apt update
>>> Hit:1 http://nl.archive.ubuntu.com/ubuntu bionic InRelease
>>> Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease
>>> Hit:3 http://nl.archive.ubuntu.com/ubuntu bionic-updates InRelease
>>> Hit:4 http://nl.archive.ubuntu.com/ubuntu bionic-backports InRelease
>>> Reading package lists... Done
>>> Building dependency tree
>>> Reading state information... Done
>>> All packages are up to date.
>>> 
>>> root@albatross:~# apt install couchdb
>>> Reading package lists... Done
>>> Building dependency tree
>>> Reading state information... Done
>>> Package couchdb is not available, but is referred to by another package.
>>> This may mean that the package is missing, has been obsoleted, or
>>> is only available from another source
>>> 
>>> E: Package 'couchdb' has no installation candidate
>>> 
>>> --
>>> i wonder whatś going on here..
>> 
>> --
>> Professional Support for Apache CouchDB:
>> https://neighbourhood.ie/couchdb-support/
>> 
>> 

-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

Re: can´t install couchdb?

[Jan Lehnardt]

looks like you skipped this step:

http://docs.couchdb.org/en/stable/install/unix.html#enabling-the-apache-couchdb-package-repository

Best
Jan
—

> On 16. May 2019, at 15:39, Rene Veerman  wrote:
> 
> ubuntu 18.04, installed, updated, then :
> 
> root@albatross:~# curl -L https://couchdb.apache.org/repo/bintray-pubkey.asc
> | sudo apt-key add -
>  % Total% Received % Xferd  Average Speed   TimeTime Time
> Current
> Dload  Upload   Total   SpentLeft
> Speed
> 100  3100  100  31000 0  20129  0 --:--:-- --:--:-- --:--:--
> 20129
> OK
> root@albatross:~# sudo apt update
> Hit:1 http://nl.archive.ubuntu.com/ubuntu bionic InRelease
> Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease
> Hit:3 http://nl.archive.ubuntu.com/ubuntu bionic-updates InRelease
> Hit:4 http://nl.archive.ubuntu.com/ubuntu bionic-backports InRelease
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> All packages are up to date.
> 
> root@albatross:~# apt install couchdb
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Package couchdb is not available, but is referred to by another package.
> This may mean that the package is missing, has been obsoleted, or
> is only available from another source
> 
> E: Package 'couchdb' has no installation candidate
> 
> --
> i wonder whatś going on here..

-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

Re: Shards-directory still big after deleting big database in Fauxton

[Jan Lehnardt]

Glad this worked out. Quick tip then, unless you run this on an 8-core (or 
more) machine, you might want to look into reducing your q for this database. 
q=2 or $num_cores is a good rule of thumb. You can use our couch-continuum tool 
to migrate an existing db: https://npmjs.com/couch-continuum

Cheers
Jan
—

> On 2. May 2019, at 14:17, Frank Röhm  wrote:
> 
> OK, I found it.
> In the 8 shards subdirectories (from -1fff to e000-) 
> there was still 8 frwiki directories (frwiki.1510609658.couch) with each 5 GB.
> I deleted them with:
> 
> find . -name frwiki.1510609658.couch -delete
> 
> from the shards dir and gone they are.
> Hopefully it won’t affect my CouchDB, but as I heard this is very robust ;)
> 
> I think I can stick to the v2.x now, no need to downgrade now, ouff.
> 
> frank
> 
>> Am 02.05.2019 um 07:25 schrieb Joan Touzet :
>> 
>> Look for a .deleted directory under your data/ directory. The files may not 
>> have been deleted but moved aside due to the enable_database_recovery 
>> setting, or because the DB was still in use when you restarted CouchDB.
>> 
>> Another useful command is:
>> 
>> $ du -sh /opt/couchdb/data/*
>> 
>> which should tell you where the storage is being used. Does this show 
>> anything useful to you?
>> 
>> -Joan
>> 
>>> On 2019-05-01 2:22 p.m., Frank Walter wrote:
>>> Hello
>>> I have CouchDB v2.3.1 (on Ubuntu 14.04) and I use it only for creating
>>> Wikipedia databases with mwscrape.
>>> My shards folder was too big, over 50 GB big, so I deleted one big db
>>> (frwiki) which had 32 GB in Fauxton. That db is gone now.
>>> After this, I thought now my shards folder should be about 20 GB but it
>>> is still 52 GB.
>>> I don't find any documentation about that in the CouchDB Doc.
>>> I restarted CouchDB (/etc/init.d/couchdb restart) but nothing changes.
>>> How can I reduce the size of shards? How can I get rid of this ghost-db?
>>> My next step would be, if I cannot solve this issue, to uninstall
>>> CouchDB 2.x and reinstall 1.x, because I dont need that feature of
>>> cluster server anyway. I see only inconvenience for my use.
>>> Thanks
>>> frank
> 

Re: CouchDB & PouchDB Meetup #14, Thursday, May 9, 2019 Amsterdam

[Jan Lehnardt]

I’ll be there, thanks for putting this on :)

Best
Jan
—

> On 16. Apr 2019, at 14:34, Andy Wenk  wrote:
> 
> Dear CouchDB Community,
> 
> I would like to invite you to the CouchDB & PouchDB Meetup #14 in Amsterdam. 
> It will be at Thursday, May 9 this year. Please RSVP for the event here:
> 
>   https://www.meetup.com/CouchDB-Meetup-Europe/events/260249490/
> 
> A very big thank you goes out to Martin Broerse who is the organiser and who 
> has pushed the event after we had a fantastic Meetup in Hamburg last year. 
> 
> I hope to see you there.
> 
> All the best!
> 
> Andy
> -- 
> Andy Wenk
> Hamburg - Germany
> RockIt!
> 
> GPG public key: 
> https://goo.gl/mhw74A
> 
> 
> 

-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

Re: Leaking memory in logger process couch 2.3.1

[Jan Lehnardt]

In particular, of you have views, each design doc will cause a full database 
scan to be dumped into the logs. 

Cheers
Jan
—

> On 21. Mar 2019, at 19:40, Robert Newson  wrote:
> 
> Hi,
> 
> Eek. This queue should never get this big, it indicates that there is far too 
> much logging traffic generated and your target (file or syslog server) can't 
> take it. It looks like you have 'debug' level set which goes a long way to 
> explaining it. I would return to the default level of 'notice' for a 
> significant reduction in logging volume.
> 
> -- 
>  Robert Samuel Newson
>  rnew...@apache.org
> 
>> On Thu, 21 Mar 2019, at 18:34, Vladimir Ralev wrote:
>> Hello,
>> 
>> I am testing couch 2.3.1 in various configurations and while loading high
>> number of test DBs I notice a ton of memory being eaten at some point and
>> never recovered More than 20 gigs and going into swap at which point i kill
>> the machine.
>> 
>> So went into the remsh to see where the memory goes and it is the logging
>> process. Take a look at the message queue len 4671185:
>> 
>> (couc...@couch01.int.test)65> MQSizes2 = lists:map(fun(A) -> {_,B} =
>> process_info(A,message_queue_len), {B,A} end, processes()).
>> (couc...@couch01.int.test)66> {_,BadProcess} =
>> hd(lists:reverse(lists:sort(MQSizes2))).
>> (couc...@couch01.int.test)67> process_info(BadProcess).
>> [{registered_name,couch_log_server},
>> {current_function,{prim_file,drv_get_response,1}},
>> {initial_call,{proc_lib,init_p,5}},
>> {status,running},
>> {message_queue_len,4671185},
>> {messages,[{'$gen_cast',{log,{log_entry,debug,<0.8973.15>,
>> 
>> [79,83,32,80,114,111,99,101,115,115,32,[...]|...],
>> "",
>> 
>> ["2019",45,["0",51],45,"21",84,["0",50],58,"40",58|...]}}},
>>{'$gen_cast',{log,{log_entry,debug,<0.8973.15>,
>> 
>> [79,83,32,80,114,111,99,101,115,115,32|...],
>> "",
>> 
>> ["2019",45,["0",51],45,"21",84,["0",50],58,[...]|...]}}},
>>{'$gen_cast',{log,{log_entry,debug,<0.15949.9>,
>> 
>> [79,83,32,80,114,111,99,101,115,115|...],
>> "",
>> 
>> ["2019",45,["0",51],45,"21",84,[[...]|...],58|...]}}},
>>{'$gen_cast',{log,{log_entry,debug,<0.8971.15>,
>> 
>> [79,83,32,80,114,111,99,101,115|...],
>> "",
>> 
>> ["2019",45,["0",51],45,"21",84,[...]|...]}}},
>>{'$gen_cast',{log,{log_entry,debug,<0.9015.15>,
>> [79,83,32,80,114,111,99,101|...],
>> "",
>> 
>> ["2019",45,["0",51],45,"21",84|...]}}},
>>{'$gen_cast',{log,{log_entry,debug,<0.9015.15>,
>> [79,83,32,80,114,111,99|...],
>> "",
>> 
>> ["2019",45,["0",51],45,[...]|...]}}},
>>{'$gen_cast',{log,{log_entry,debug,<0.8973.15>,
>> [79,83,32,80,114,111|...],
>> "",
>> ["2019",45,[[...]|...],45|...]}}},
>>{'$gen_cast',{log,{log_entry,debug,<0.15949.9>,
>> [79,83,32,80,114|...],
>> "",
>> ["2019",45,[...]|...]}}},
>>{'$gen_cast',{log,{log_entry,debug,<0.8971.15>,
>> [79,83,32,80|...],
>> "",
>> ["2019",45|...]}}},
>>{'$gen_cast',{log,{log_entry,debug,<0.8973.15>,
>> [79,83,32|...],
>> "",
>> [[...]|...]}}},
>>{'$gen_cast',{log,{log_entry,debug,<0.15949.9>,
>> [79,83|...],
>> "",
>> [...]}}},
>>{'$gen_cast',{log,{log_entry,debug,<0.9015.15>,
>> [79|...],
>> [...],...}}},
>>{'$gen_cast',{log,{log_entry,debug,<0.8971.15>,[...],...}}},
>>{'$gen_cast',{log,{log_entry,debug,<0.8973.15>,...}}},
>>{'$gen_cast',{log,{log_entry,debug,...}}},
>>{'$gen_cast',{log,{log_entry,...}}},
>>{'$gen_cast',{log,{...}}},
>>{'$gen_cast',{log,...}},
>>{'$gen_cast',{...}},
>>{'$gen_cast',...},
>>{...}|...]},
>> {links,[<0.122.0>,#Port<0.2149>]},
>> {dictionary,[{'$initial_call',{couch_log_server,init,1}},
>>  {'$ancestors',[couch_log_sup,<0.121.0>]}]},
>> {trap_exit,true},
>> {error_handler,error_handler},
>> {priority,normal},
>> 

Re: Getting an error "all_dbs_active" running a CouchDB 2.3 cluster

[Jan Lehnardt]

Heya Jake,

what is your q value for your databases on the source and target clusters?

The default is 8, so 8*90 gives us 720 potential open dbs.

It could just be that under normal operation, your 2.0 cluster never has to 
open all shards at the same time, but now that you are running a migration, the 
target cluster has to.

IIRC there were no semantic changes in this handling between 2.0 and 2.3.

Best
Jan
-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/


> On 14. Mar 2019, at 11:05, Jake Kroon  
> wrote:
> 
> Hi,
>  
> I’m in the process of trying to migrate a CouchDB cluster from 2.0 to 2.3, by 
> creating a new cluster and replicating the databases over to it, and 
> eventually plan to switch over to the new one. Generally this process is 
> going fine, but I’m getting errors similar to the following when running my 
> applications against the new cluster:
>  
> [error] 2019-02-21T07:04:51.213276Z couchdb@ip <0.17397.4590> f346ddb688 
> rexi_server: from: couchdb@ip (<0.32026.4592>) mfa: fabric_rpc:map_view/5 
> error:{badmatch,{error,all_dbs_active}} 
> [{fabric_rpc,map_view,5,[{file,"src/fabric_rpc.erl"},{line,148}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
>  
> I’m using the default max_dbs_open value of 500 (which is preset in the 
> default.ini file). As far as I understand it, this should be plenty, and it’s 
> what I’m successfully using on my current 2.0 cluster with no errors. I may 
> be misunderstanding how this setting works though.
>  
> I have about 90 databases in the cluster, and all I’m currently running is a 
> couple of scripts:
>  
>   • A “build views” script that runs every hour, that goes through each 
> database and queries each of the views (in series).
>   • A “conflict resolver” script that runs every 15 minutes, that queries 
> all databases for conflicts and then performs custom logic to deal with 
> conflicts (though there won’t be any conflicts on our new server at this 
> time, so it’s just querying the conflicts view on each database)
>  
> I also previously had continuous bidirectional replication set up between the 
> new cluster and the old one, and the “all_dbs_active” error was happening 
> quite often (a couple of times per hour). I’ve cancelled all the replication 
> jobs and the error has reduced to about 1 or 2 instances per day.
>  
> I haven’t yet tried increasing the max_dbs_open value (which seems to be a 
> common suggestion for dealing with the “all_dbs_active” error), because the 
> live 2.0 cluster is working fine with the default value of 500, and has 
> higher load on it than the new 2.3 cluster.
>  
> I was wondering if anyone has any suggestions on what I should look at to try 
> to solve this issue?
>  
> I’m running the cluster on Ubuntu 18.04 LTS.
>  
> Thanks!
> Jake Kroon
> Software Engineer
>  
> D: +61 8 9318 6949
> E: jkr...@immersivetechnologies.com
>  
> 
>  
> YouTube  |  LinkedIn  |  Google+

-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

[ANNOUNCE] CouchDB 2.3.1

[Jan Lehnardt]

Dear community,

Apache CouchDB 2.3.1 has been released and is available for download.

Apache CouchDB™ lets you access your data where you need it. The Couch 
Replication Protocol is implemented in a variety of projects and products that 
span every imaginable computing environment from globally distributed 
server-clusters, over mobile phones to web browsers.

Store your data safely, on your own servers, or with any leading cloud 
provider. Your web- and native applications love CouchDB, because it speaks 
JSON natively and supports binary data for all your data storage needs.

The Couch Replication Protocol lets your data flow seamlessly between server 
clusters to mobile phones and web browsers, enabling a compelling offline-first 
user-experience while maintaining high performance and strong reliability. 
CouchDB comes with a developer-friendly query language, and optionally 
MapReduce for simple, efficient, and comprehensive data retrieval.

* * *

https://couchdb.apache.org/#download

Pre-built packages for Windows, macOS, Debian/Ubuntu and RHEL/CentOS are 
available.

* * *

CouchDB 2.3.1 is a bugfix release, and was originally published on 2019-03-12.

The community would like to thank all contributors for their part in making 
this release, from the smallest bug report or patch to major contributions in 
code, design, or marketing, we couldn’t have done it without you!

See the official release notes document for an exhaustive list of all changes:

http://docs.couchdb.org/en/stable/whatsnew/2.3.html#version-2-3-1

Release highlights:

#1811: Add new /{db}/_sync_shards endpoint (admin-only).

#1870: Update to mochiweb 2.19.0. See also #1875.

#1875: Refuse building with known bad versions of Erlang.

#1880: Compaction: Add snooze_period_ms for finer tuning.

#1799: Restrict _purge to server admin.

#1803: Use the same salt for admin passwords on cluster setup.

#1053: Fix python2 compatibility for couchup.

#1905: Fix python3 compatibility for couchup.


On behalf of the CouchDB PMC,
Jan Lehnardt
—

Re: r and w parameters in couch2.x

[Jan Lehnardt]

Specifically, n is the number of copies of your data, not the number of nodes 
in the system. You can tweak read concurrency performance by increasing a 
database’s number of shards (q) and adding more nodes for those shards to live 
on, at the expense of view, all_docs and changes requests becoming more 
expensive.

> On 12. Mar 2019, at 08:08, Vladimir Ralev  wrote:
> 
> OK, I see. Thank you.
> 
> On Mon, Mar 11, 2019 at 8:48 PM Robert Newson  wrote:
> 
>> Hi,
>> 
>> Yes, you will have 4 copies of your data, your nodes will be mirrors of
>> each other in effect.
>> 
>> R and W only control one thing; the number of replies we wait for before
>> returning your response. All N requests are made, in parallel,  no matter
>> what setting for R or W you use. You're not saving I/O by changing it, you
>> are just modifying your latency (lower values of R and W will lower request
>> latency) and consistency (higher values of R and W will improve
>> consistency, though nothing delivers strong consistency in CouchDB).
>> 
>> Your understanding is not quite right, and so there neither are the
>> inferences made from that base.
>> 
>> B.
>> 
>> --
>>  Robert Samuel Newson
>>  rnew...@apache.org
>> 
>> On Mon, 11 Mar 2019, at 15:25, Vladimir Ralev wrote:
>>> Ah thanks a lot for the reply.
>>> 
>>> The idea for n = 4 is both fault tolerance and performance. Since I have
>>> very few writes, I expect replication IO and view indexing IO to be
>> minimal
>>> and I have no issues with temporary inconsistencies and conflicts.
>>> 
>>> My understanding is that since there are very few writes, the 4 nodes
>> will
>>> behave almost like 4 independent single nodes and will be able to serve
>> the
>>> read requests independently without having to proxy to cluster peers and
>>> thus avoiding a great deal of extra network and disk IO.
>>> 
>>> R=3 to me means 3 times the IO and thus 3 machines will be busy for the
>>> same read request instead of serving other requests. Which I understand
>> is
>>> 3 times less performance from the cluster as a whole.
>>> 
>>> If my understanding is correct, I imagine this would be a common use-case
>>> for couch?
>>> 
>>> On Mon, Mar 11, 2019 at 4:58 PM Robert Newson 
>> wrote:
>>> 
 r and w are no longer configurable from the config file by design. The
 default is n/2+1 (so 3 in your case) unless you specify r or w as
>> request
 parameters.
 
 setting n = 4 for a 4 node cluster is very unusual, do you really need
>> 4
 full copies of your data?
 
 couchdb will also automatically lower both r and w if nodes are
>> offline.
 
 The default of n=3, r=w=2 is appropriate in almost all cases as the
>> right
 balance between data safety and availability. Nothing you've said so
>> far
 suggests it would be good to deviate from those settings.
 
 --
  Robert Samuel Newson
  rnew...@apache.org
 
 On Mon, 11 Mar 2019, at 14:52, Vladimir Ralev wrote:
> Hi all,
> 
> I am looking into running a 4-node couchdb 2.3 with this config in
> default.ini and I made sure no other config file override them:
> [cluster]
> q = 8
> n = 4
> r = 1
> w = 1
> 
> But when i create a test DB and check the settings I get:
> curl -s couch01:5984/mytest1234 |jq .   "cluster": { "q": 8,
 "n": 4,
> "w": 3, "r": 3 },
> 
> r and w settings are not respected and seem stuck to be the defaults.
> 
> When I kill 3 of the machine and test reads and writes, they still
>> work
> fine so it doesn't seem like the r and w are actually 3 either. I
>> checked
> if the debug logs printed out the r and w anywhere to confirm what is
 being
> configured or executed but there is nothing.
> 
> It is unclear if r and w are active in this version of couch. I can
>> see
> the
> they have been partially removed from the documentation
> https://docs.couchdb.org/en/master/cluster/theory.html as opposed to
> couchdb 2.0.0 original doc
> 
 
>> https://web.archive.org/web/20160109122310/https://docs.couchdb.org/en/stable/cluster/theory.html
> 
> Additionally curl -s couch01:5984/mytest1234/doc?r=3
> still works even if 3 out of the 4 nodes are dead which is
>> unexpected per
> the quorum documentation here
> https://docs.couchdb.org/en/master/cluster/sharding.html#quorum
> 
> My specific concern with r and w is that if r is 3 this means 3 times
 more
> network and disk IO since it will have to read 3 times from remote
> machines. My use case really doesn't need this and performance will
 suffer.
> This is a little hard to test so I was hopinh someone can shed some
>> light
> on the current situation with r and w in couch 2.3.
> 
> Thanks
> 
 
>>> 
>> 

-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/

Re: nano: UnhandledPromiseRejectionWarning: Error: connect ENOBUFS

[Jan Lehnardt]

Hi Stephan,

welcome to CouchDB and asynchronous programming :)

Let’s annotate your code to see what’s happening:

const nano = require('nano')("http://localhost:5984”); # get a a nano instance, 
all good
const larch = nano.db.use('larch’);# tell nano which db to 
use, all good

larch.list().then((allDocs) => {   # get a list of all 
docs, asynchronously
   # this makes one request 
to CouchDB and
   # CouchDB returns a 
large JSON object with
   # all documents in an 
array structure.
   # when this is read into 
nano, the callback
   # in `then()` is called 
with that result.


   var len = allDocs.rows.length;  # get the array length, 
all good
   console.log('total # of docs -> ' + len);   # console.log, all good

   allDocs.rows.forEach((document) => {# now you are iterating 
over the result set
   # from above. For each 
id of a document in
   # the result set, you 
send a single request
   # to CouchDB 
asynchronously, and wait for the
   # result.
   # The clincher is here: 
the forEach loop is
   # synchronous, and the 
.get() is asynchronous.
   # You might think the 
code runs as it reads:
   # Iterate over my list, 
fetch one doc, iterate
   # to next item in the 
list, fetch one doc and so
   # on until the list is 
all iterated over.
   #
   # What really happens is 
this:
   # Iterate over my list, 
start the fetching of one doc,
   # and register a 
`then()` callback to print the result
   # when it comes back. 
Then iterate to the next item
   #
   # Now, ’starting the 
fetching of one doc’ is a near
   # instant operation that 
just sets everything up to
   # make an HTTP request, 
so iterating over the whole
   # list is really quick.
   # In your case, you are 
starting 8 requests to
   # CouchDB in a VERY 
short amount of time, and you
   # are quickly running 
out of operating system resources
   # (ENOBUFS, or no more 
buffers to do networking with)
   # to make HTTP requests.

   larch.get(document.id).then((body) => {
   console.log("id: " + body._id);
   });
   });
});

There are multiple ways to solve this in asynchronous programming. One would be 
a queue with a maximum parallel jobs setting that keeps concurrent operations 
to a nice minimum, but eventually gets you all the results. I’ve used 
https://www.npmjs.com/package/promise-queue in the past for this.

To get the same result in CouchDB, you can pass in the `include_docs` parameter 
set to `true`, then CouchDB will fetch the doc bodies for you in the original 
`larch.list()` request and include the doc bodies inside the result set.

Best
Jan
—

> On 12. Feb 2019, at 21:09, Stephan Mühlstrasser 
>  wrote:
> 
> Hi,
> 
> I'm new to CouchDB/nano and asynchronous JavaScript, and I hope this is
> the right place to ask questions about how to use the nano API.
> 
> I'm trying to process all documents in a local CouchDB database with the
> following code using nano (in the real program I want to get access to
> all fields of each document):
> 
> const nano = require('nano')("http://localhost:5984;);
> const larch = nano.db.use('larch');
> 
> larch.list().then((allDocs) => {
>var len = allDocs.rows.length;
>console.log('total # of docs -> ' + len);
> 
>allDocs.rows.forEach((document) => {
>larch.get(document.id).then((body) => {
>console.log("id: " + body._id);
>

CouchDB Inspection Raffle

[Jan Lehnardt]

Heya Everyone,

you may or may not know that company Neighbourhoodie offers professional 
CouchDB services.

One of these services is what we call a CouchDB Inspection, which gives you a 
detailed analysis of all the things that might be wrong with your CouchDB 
installation. See https://neighbourhood.ie/couchdb-support/inspection/ for more 
details.

A CouchDB Inspection retails currently at 999 EUR. To promote the service we 
are offering the following deal:

• You tell us about your success story with Apache CouchDB that we can 
use in our marketing materials to promote CouchDB and our services.

• In return, we’re giving away ten (10!) Inspections (one each ;) to 
the folks who submit their story. (T apply): 
https://docs.google.com/forms/d/e/1FAIpQLSd9BRnSFRMyFtQFUb5g91K2q2lYYlj-JI8uP0ypV5HxZVpiug/viewform

Please email me privately, if you have any questions.

Best
Jan
—

Re: Upgrade from 2.1.1 to 2.3.0 now PUTing an attachment hangs

[Jan Lehnardt]

Hi Bill,

bugs are best reported on https://github.com/apache/couchdb/issues

but while we are at it, how big is your mp3?

Can you show a full curl -v log, and the corresponding CouchDB debug log, 
ideally even your mp3?

Best
Jan
—

> On 4. Feb 2019, at 23:08, Compu Net  wrote:
> 
> FYI, the reason for wanting to downgrade is that after I upgraded from 2.1.1 
> to 2.3.0 if I do : curl -vX PUT 
> http://127.0.0.1:5984/my_database/001/test.mp3 --data-binary @test.mp3 -H 
> "ContentType:audio/mp3" the upload hangs.
> 
> This is on a 2.3.0 cluster with 2 nodes, previously on 2.1.1 it worked and on 
> a single 1 node system the upload works fine.
> 
> Bill

-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/