CouchDB and RabbitMQ clusters
Hello everybody, I have a general Erlang question but I think you could help me with that... I need to run CouchDB and RabbitMQ on the same set of (three) nodes, all clustered together. What happens with epmd? Erlang's documentation (https://erlang.org/doc/man/epmd.html) is pretty vague: "The daemon is started automatically by command erl(1) [1] if the node is to be distributed and no running instance is present."... So what happens? The first one between Couch and Rabbit who starts opens epmd and the second one just hooks to the already running copy? Thanks. -- Andrea Brancatelli Links: -- [1] https://erlang.org/doc/man/erl.html
Re: Data backup in CouchDB cluster
I don't think so, but never tried it. --- Andrea Brancatelli On 2021-05-07 12:38, Willem van der Westhuizen wrote: > This looks very interesting, is it possible to do incremental backups with > this approach, from a given seqence number in the db only? > > Willem > > On 2021/05/07 11:52, Andrea Brancatelli wrote: We're using this: > > https://github.com/danielebailo/couchdb-dump > > since a few years. > > It almost always worked flawlessly. It's fast, and, to me, it's better > than backing up the .couch files for various reasons: > > * you can restore datas on a cluster with a different N/Q layout > * you can restore datas on a different machine with a different > cluster name / different IP / different whatever ... .couch files > include references to vm.args parameters. > * when you restore the DB you get a clean db without the tombstones. > * you can backup the db without having local access to the machine, > passing trough the standard HTTP port. > > Hope it helps you. > > --- > > Andrea Brancatelli > > On 2021-05-07 11:27, Simon Schwichtenberg wrote: > > Hi, > > I wonder how you'd do backups of your data in a CouchDB cluster. The > documentation does not mention backups of clusters explicitly > (https://docs.couchdb.org/en/latest/maintenance/backups.html#database-backups). > > When you have a cluster of three nodes and the nodes are set to n=3 and q=2 > (see https://docs.couchdb.org/en/latest/cluster/sharding.html), I'd expect > that every single node in the cluster has all the data and you can copy the > .couch files from any of these three nodes. When you have 6 nodes with n=3 > and q=2 this approach does not work anymore because every node has just a > single shard. Please correct me if I am wrong. > > What is best practice to backup a cluster? > > This message is a follow-up from here: > https://github.com/cloudant/couchbackup/issues/349 > > Thanks, > Simon
Re: Data backup in CouchDB cluster
We're using this: https://github.com/danielebailo/couchdb-dump since a few years. It almost always worked flawlessly. It's fast, and, to me, it's better than backing up the .couch files for various reasons: * you can restore datas on a cluster with a different N/Q layout * you can restore datas on a different machine with a different cluster name / different IP / different whatever ... .couch files include references to vm.args parameters. * when you restore the DB you get a clean db without the tombstones. * you can backup the db without having local access to the machine, passing trough the standard HTTP port. Hope it helps you. --- Andrea Brancatelli On 2021-05-07 11:27, Simon Schwichtenberg wrote: > Hi, > > I wonder how you'd do backups of your data in a CouchDB cluster. The > documentation does not mention backups of clusters explicitly > (https://docs.couchdb.org/en/latest/maintenance/backups.html#database-backups). > > When you have a cluster of three nodes and the nodes are set to n=3 and q=2 > (see https://docs.couchdb.org/en/latest/cluster/sharding.html), I'd expect > that every single node in the cluster has all the data and you can copy the > .couch files from any of these three nodes. When you have 6 nodes with n=3 > and q=2 this approach does not work anymore because every node has just a > single shard. Please correct me if I am wrong. > > What is best practice to backup a cluster? > > This message is a follow-up from here: > https://github.com/cloudant/couchbackup/issues/349 > > Thanks, > Simon
Re: R: FYI : a new library for accessing couchdb through PHP, which i'm willing to upgrade, maintain and bugfix.
I tried to signup but got really confused by the following steps that started talking about Jira and whatever... If you could update the link for SAG to our fork (and maybe put it higher in the list since I think this is the only library actively maintained) that would be super cool. https://github.com/skeyby/sag --- Andrea Brancatelli On 2021-03-04 09:14, Jonathan Hall wrote: > On 3/2/21 5:35 PM, Matteo Guadrini wrote: > >> @Jonathan: How do I get my powershell module into that list? > > Can you sign up for a wiki account there? I'm not sure how access is > controlled. If not, or if it's too long/boring for you to bother, send me the > details and I can add it. :) > > Jonathan
Re: FYI : a new library for accessing couchdb through PHP, which i'm willing to upgrade, maintain and bugfix.
The "old" SAG was. The "new" now (our fork) probably isn't as I would know what you mean by "couchdb introduction or help pages". https://couchdb.apache.org/ used to have a page for client libraries, but I don't seem to find it anymore... --- Andrea Brancatelli On 2021-03-02 16:05, Rene Veerman wrote: > Hi Andrea.. > > i wasn't aware of the existence of SAG when i wrote > github.com/nicerapp/php-couchdb [1]. > > i can and will change my business-logic code to be using SAG, and i am also > willing to become a contributor for it, when a situation calls for that of > course. > > is SAG listed on the couchdb introduction or help pages already? > > With regards, > Rene > > On Mon, Mar 1, 2021 at 10:18 AM Andrea Brancatelli > wrote: > > Hello Rene, > > really interesting, I'll take a look at it later. > > In the meantime, please, take a look to our fork of the old-but-gold SAG we > kept improving: > > https://github.com/skeyby/sag > > fragmentation is never a good idea so maybe we could merge forces. > > Have a good day. > > --- > > Andrea Brancatelli > > On 2021-02-28 15:29, Rene Veerman wrote: > Hi all. > > I've made a new small PHP library up at > https://github.com/nicerapp/php-couchdb , which aims to provide simple yet > robust access to couchdb from PHP. > > I'm open to feature requests, bugfix support, etc. > > Enjoy :) > > Oh, you'll have to run and examine unitTest.php to get started. Links: -- [1] http://github.com/nicerapp/php-couchdb
Re: FYI : a new library for accessing couchdb through PHP, which i'm willing to upgrade, maintain and bugfix.
Hello Rene, really interesting, I'll take a look at it later. In the meantime, please, take a look to our fork of the old-but-gold SAG we kept improving: https://github.com/skeyby/sag fragmentation is never a good idea so maybe we could merge forces. Have a good day. --- Andrea Brancatelli On 2021-02-28 15:29, Rene Veerman wrote: > Hi all. > > I've made a new small PHP library up at > https://github.com/nicerapp/php-couchdb , which aims to provide simple yet > robust access to couchdb from PHP. > > I'm open to feature requests, bugfix support, etc. > > Enjoy :) > > Oh, you'll have to run and examine unitTest.php to get started.
Re: No DB Shards
You're pretty vague. The insert was parallel or sequential? The final document counts shows 500.000 docs or 499.994? Is it a single instance CouchDB Server? All the docs where in the same database? How was the disk load during the test? "No DB Shards could be opened" is a pretty vague message that may be generated by many different causes. If it's randomly appearing on a "normally working" instance, in my experience it may be related to slow IO: writes to disk pile up and when the write queue is too long some writes start to timeout giving the No DB Shards could be opened. What where the N and Q settings of your test database? --- Andrea Brancatelli On 2020-09-28 13:11, Paul Milner wrote: > Hello > > I am testing my DB installation and I have an insert of a document that has > been done ~500'000 times in one process. This should multiply significantly > as more users come onboard. > > Everything ran to completion, but in the log I got the following error: > > No DB shards could be opened.","stack":"Error: No DB shards could be > opened.\nat Request._callback - about 6 times > > Can someone point me in the right direction as to how I can debug this > please? > > Thanks a lot > Best regards > Paul
Re: How to deal with updates of denormalized, duplicated data?
It absolutely makes sense and it's actually even closer to what we did
than my previous solution.
Glad you liked the idea.
---
Andrea Brancatelli
On 2020-09-03 10:59, Olaf Krueger wrote:
> Hi Andrea,
>
> in order to handle enums, I end up with a variant of your solution:
> I rather would like to keep the translation separated from the enum
> implementation.
> So I moved the translation responsability to whatever i18n solution, e.g.
> https://www.i18next.com.
> One benefit is that all translations are at the same place and we don't need
> to duplicate same translations for same enumeral values across other enums.
>
> Moreover, I decided to store each single enumeral of an enumeration in its
> own document.
> A namespace could be used if needed:
> {
> _id: "issue.status.closed",
> docType: "enumeral",
> namespace: "common"
> value: "closed"
> }
>
> {
> _id: "issue.status.open",
> docType: "enumeral",
> namespace: "common"
> value: "open"
> }
>
> In order to fetch all enumerals of an enumeration we just have to query the
> _all_docs view by using a proper start- and end key.
>
> The enumeral document is nested within the parent:
> {
> _id: "issue.23423",
> title: "How to handle enums",
> status: {
> _id: "issue.status.open",
> docType: "enumeral",
> namespace: "common"
> value: "open"
> }
> }
>
> Then, at presentation level, I am am going to use a i18n solution in order to
> map the enumeral value with its associated text/translation.
> E.g.:
> text = i18next.t(doc.status.namespace + ':' + doc.staus.value);
>
> Hope that makes sense! ;-)
> Maybe it's helpful for others.
>
> Thanks,
> Olaf
Re: How to deal with updates of denormalized, duplicated data?
Humans are the only ones who need to know that "ASDF0001" means "Completed"/"Completato"/"Whatever", so, yes, our solution was to do it at presentation level. --- Andrea Brancatelli On 2020-09-01 17:07, Olaf Krueger wrote: >> Then all the logic of the app uses the native keyword (completed) while >> always presented the translated one to the user according to his locale >> (Completed/Completato). > > Does that mean that your solution is to always join/merge the enumeral at > application level? > > Thanks! > > Olaf
Re: How to deal with updates of denormalized, duplicated data?
After a long long trial and guess we now have "enumeral" docs in the
database, that we also use for translation/localization and in the real
docs (and view outputs) we put out the key for the enumbered doc.
Something like this:
// Issue doc
{
_id: "issue.foo",
type: "issue",
title: "No joins with NoSQL",
status: {
value: "completed"
}
}
// Issues statuses doc
{
"_id": "issue.statuses",
"new": {
"en": "New",
"it": "Nuovo"
},
"completed": {
"en": "Completed",
"it": "Completato"
}
}
Then all the logic of the app uses the native keyword (completed) while
always presented the translated one to the user according to his locale
(Completed/Completato).
If the user is searching for "Completed" stuff then a find is run using
"completed" behind the scene.
I cannot guarantee this is the best method but we've been using it for
at least one year and a half with mild satisfaction.
---
Andrea Brancatelli
On 2020-09-01 15:51, Olaf Krüger wrote:
> Hi guys,
>
> I am still a CouchDB beginner but I read a lot about it in the meanwhile and
> also about common NOSQL stuff.
> But I still struggle with the question of how to handle duplicate data,
> especially with the update of duplicated data which are "distributed" across
> the entire database.
>
> Following scenario, just as an example:
>
> Imagine there is an issue document which has a relation to an enumeral (A
> particular item of an enumeration).
>
> // Issue doc
> {
> _id: "issue.foo",
> type: "issue",
> title: "No joins with NoSQL",
> // Duplicated "enumeral":
> status: {
> _id: "enumeral.status.in-progress",
> type: "enumeral",
> title: "In progres"
> }
> }
>
> // Origin "enumeral" doc
> {
> _id: "enumeral.status.in-progress",
> type: "enumeral"
> title: "In progres"
> }
>
> Now, we detect a typo at our enumeral:
> The title needs to be updated from "In progres" to "In progress" afterwards.
> And of course, all duplicates needs to be updated also (A few thousend docs
> could be affected).
>
> In order to achive this, I would create a background task at application
> level like this:
> - Search for all affected documents.
> - Replace/update the title of each particular doc
> - Bulk update all docs
>
> This might work, but now we created a new revision for each affected document.
> If a user works on the previous revision, he will ran into a conflict.
> Basically, this is pretty ok, but in this case, a background task has changed
> the document
> and in this special case, the user shouldn't take care of this little change.
>
> As an alternative, instead of duplicating the "enumeral" doc, I tried to
> "join" it by using "linked documents" or other kind of views.
> But then, I got at least two different docs instead of one clean formatted
> JSON doc.
> The result needs to be formatted at application level which doesn't feel
> right to me.
> Moreover, it also doesn't feel right to me to complicate the schema more than
> needed.
>
> So, I would like to ask you experienced guys how do you deal with such things.
> There might be not "the one any only" solution, but I would like to ensure if
> I am totally on the wrong path or even not. Maybe I have to change my
> thinking, not sure.
>
> Please also confirm if there's no way out of the dilemma, this would be
> really helpful too ;-)
>
> Many thanks in advance!
>
> Olaf
Re: Copying Contents of /Data Directory from a 3.0 Instance of CouchDB(Windows 10) to the /Data directory(/var/lib/couchdb)(data) Directory of Ubuntu 20.04
My totally wild guess is the node name in the vm.args across the two
systems is not matching.
---
Andrea Brancatelli
On 2020-08-25 15:05, John Le Brasseur wrote:
> Hi.
> I copied the entire Couchdb 3.0 Installation from my Windows 10
> installation and saved it.
> I have a new Ubuntu 20.04 installation.
> After correctly installing CouchDB 3.1 on Ubuntu I deleted all files in the
> /var/lib/couchdb(data) directory and copied the files from my Windows /data
> directory into it.
> I changed ownership to couchdb:couchdb and directory modes to 755 and file
> modes to 644.
> Using curl I can access couchdb and _all_dbs. but when accessing a
> particular database I get
> {
> "error": "internal_server_error",
> "reason": "No DB shards could be opened.",
> "ref": 2822102114
> }
> Also, the databases are greyed out in fauxton.
>
> Obviously this is not best practice, but is there a way to make this work?
Re: design document with no name
CouchDB 2.3.1 Vendor FreeBSD Welcome msg Welcome Aux features pluggable-storage-engines, scheduler Photon 1.9 Build 36, 2020-04-20 Photon addons cw.Sys.JSON 2.7.17 cw.Couch.Monitor 2.1.58 --- Andrea Brancatelli Schema31 S.p.a. Chief Technology Officier Tel: +39-06-98358472 Cell: +39-3312488468 Società del Gruppo OVIDIO TECH S.R.L. On 2020-06-15 22:39, ermouth wrote: >> creating a design document with an empty ID > > This CouchDB bug was filed long ago > https://github.com/apache/couchdb/issues/1472, dunno was it fixed. > > Btw, please provide your version of CouchDB and Photon. > > ermouth
Re: design document with no name
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"a9c59fa2023151b05f880669de8e31ed","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"FreeBSD"}}
---
Andrea Brancatelli
Schema31 S.p.a.
Chief Technology Officier
Tel: +39-06-98358472
Cell: +39-3312488468
Società del Gruppo OVIDIO TECH S.R.L.
On 2020-06-16 10:18, Jan Lehnardt wrote:
> Heya Andrea,
>
> Which version of CouchDB is this?
>
> Best
> Jan
> _
>
> On 15. Jun 2020, at 14:54, Andrea Brancatelli
> wrote:
>
> Sorry the attachment got lost along the way:
>
> http://storage.gcloud.schema31.it/obj/9d6870890a00-1608-ae11-70fa-57153e15
>
> ---
>
> Andrea Brancatelli
> Schema31 S.p.a.
>
> On 2020-06-15 14:44, Andrea Brancatelli wrote:
>
> I really don't know how but one of my developers succeeded, by using Photon,
> in creating a design document with an empty ID.
>
> Needless to say this broke almost everything like any Post to this database
> with:
>
> [error] 2020-06-15T12:35:00.511174Z couchdb@127.0.0.1 <0.24049.1262>
> could not load validation funs {{nocatch,{illegal_docid,<<"Illegal document
> id
> `_design/`">>}},[{couch_doc,validate_docid,1,[{file,"src/couch_doc.erl"},{line,213}]},{couch_doc,transfer_fields,3,[{file,"src/couch_doc.erl"},{line,252}]},{couch_doc,get_validate_doc_fun,1,[{file,"src/couch_doc.erl"},{line,399}]},{ddoc_cache_entry_validation_funs,'-recover/1-fun-0-',1,[{file,"src/ddoc_cache_entry_validation_funs.erl"},{line,35}]},{lists,flatmap,2,[{file,"lists.erl"},{line,1250}]},{ddoc_cache_entry_validation_funs,recover,1,[{file,"src/ddoc_cache_entry_validation_funs.erl"},{line,34}]},{ddoc_cache_entry,do_open,1,[{file,"src/ddoc_cache_entry.erl"},{line,297}]}]}
>
>
> I can't edit the document, I can't delete the document because the server
> says that _design/ is an illegal id name.
>
> I strongly agree with him :-)
>
> Luckily this is a Development Database so I guess I'll drop the whole thing
> and start from scratch, but maybe there's a bug in the validation code for
> new design documents?
>
> I tried to reproduce the thing but I wasn't able to do it...
>
> --
>
> Andrea Brancatelli
> Schema31 S.p.a.
Re: design document with no name
Sorry the attachment got lost along the way:
http://storage.gcloud.schema31.it/obj/9d6870890a00-1608-ae11-70fa-57153e15
---
Andrea Brancatelli
Schema31 S.p.a.
On 2020-06-15 14:44, Andrea Brancatelli wrote:
> I really don't know how but one of my developers succeeded, by using Photon,
> in creating a design document with an empty ID.
>
> Needless to say this broke almost everything like any Post to this database
> with:
>
> [error] 2020-06-15T12:35:00.511174Z couchdb@127.0.0.1 <0.24049.1262>
> could not load validation funs {{nocatch,{illegal_docid,<<"Illegal document
> id
> `_design/`">>}},[{couch_doc,validate_docid,1,[{file,"src/couch_doc.erl"},{line,213}]},{couch_doc,transfer_fields,3,[{file,"src/couch_doc.erl"},{line,252}]},{couch_doc,get_validate_doc_fun,1,[{file,"src/couch_doc.erl"},{line,399}]},{ddoc_cache_entry_validation_funs,'-recover/1-fun-0-',1,[{file,"src/ddoc_cache_entry_validation_funs.erl"},{line,35}]},{lists,flatmap,2,[{file,"lists.erl"},{line,1250}]},{ddoc_cache_entry_validation_funs,recover,1,[{file,"src/ddoc_cache_entry_validation_funs.erl"},{line,34}]},{ddoc_cache_entry,do_open,1,[{file,"src/ddoc_cache_entry.erl"},{line,297}]}]}
>
>
> I can't edit the document, I can't delete the document because the server
> says that _design/ is an illegal id name.
>
> I strongly agree with him :-)
>
> Luckily this is a Development Database so I guess I'll drop the whole thing
> and start from scratch, but maybe there's a bug in the validation code for
> new design documents?
>
> I tried to reproduce the thing but I wasn't able to do it...
>
> --
>
> Andrea Brancatelli
> Schema31 S.p.a.
design document with no name
I really don't know how but one of my developers succeeded, by using
Photon, in creating a design document with an empty ID.
Needless to say this broke almost everything like any Post to this
database with:
[error] 2020-06-15T12:35:00.511174Z couchdb@127.0.0.1 <0.24049.1262>
could not load validation funs
{{nocatch,{illegal_docid,<<"Illegal document id
`_design/`">>}},[{couch_doc,validate_docid,1,[{file,"src/couch_doc.erl"},{line,213}]},{couch_doc,transfer_fields,3,[{file,"src/couch_doc.erl"},{line,252}]},{couch_doc,get_validate_doc_fun,1,[{file,"src/couch_doc.erl"},{line,399}]},{ddoc_cache_entry_validation_funs,'-recover/1-fun-0-',1,[{file,"src/ddoc_cache_entry_validation_funs.erl"},{line,35}]},{lists,flatmap,2,[{file,"lists.erl"},{line,1250}]},{ddoc_cache_entry_validation_funs,recover,1,[{file,"src/ddoc_cache_entry_validation_funs.erl"},{line,34}]},{ddoc_cache_entry,do_open,1,[{file,"src/ddoc_cache_entry.erl"},{line,297}]}]}
I can't edit the document, I can't delete the document because the
server says that _design/ is an illegal id name.
I strongly agree with him :-)
Luckily this is a Development Database so I guess I'll drop the whole
thing and start from scratch, but maybe there's a bug in the validation
code for new design documents?
I tried to reproduce the thing but I wasn't able to do it...
--
Andrea Brancatelli
Schema31 S.p.a.
Re: How to work with Indexes
As ermouth said the best approach is to use the explain feature of
Fauxton / Photon.
Couch tries to guess what's the best index to use for any case so you
don't necessarily need to build complex indexes, if you ask for more
selectors than the indexed ones usually it picks an index and then scan
the documents to apply the subselectors.
Yet I must admit I feel the logic behind it a bit entropic and
cumbersome.
Just remember to ask the selector in the same order as they are indexed,
even for partial indexes.
Like a select for { fieldA, fieldB, fieldC } and an index for { fieldA,
fieldB } would work as you expect, but if the index was on { fieldA,
fieldC } it won't get used.
On the contrary I don't think Couch is (yet?) able to to Union between
indexes, so if you have index { fieldA } index { fieldB } and ask for
select { fieldA, fieldB } only index { fieldA } will be used and then
all the documents will be processed to find the subset.
That's my empiric deduction from some usage, but someone with the hands
on the code can probably correct me.
---
Andrea Brancatelli
On 2020-06-09 13:21, Piotr Zarzycki wrote:
> Hi Aurélien,
>
> It is not actually about any specific case. My questions are just helpers
> to truly understand what approach we should take in our database. I just
> have hope that I will get more understanding to get answer to some cases.
>
> Answer to your questions inline.
>
> wt., 9 cze 2020 o 13:04 Aurélien Bénel napisał(a):
>
> Hi Piotr,
>
> Let's say that I have created two indexes for fields "docType" and "age":
> (...)
> My question is - How CouchDb will behave in case when I'm searching my db
> using both fields when there are two indexes which contains the same
> fields?
> To answer your question (at least with js views, I don't use Mango but I
> suppose it to be similar),
> we need more details:
>
> - Are both fields variable parameters ? Or is one of them a constant?
I think both cases I would consider. However the first one would be that
I'm searching using both fields as a variable parameters. I'm not sure
about constant question. I think non of those fields are constant - I
mean
from the database point of view docType won't change in the document
that's
for sure.
> - Will both parameters be always set ? If not, which one might be unset?
>
> Let's assume that it always will, but like I said what if I'm looking only
and searching/querying using docType only ?
> Regards,
>
> Aurélien
Re: CouchDB vulnerability
Thanks Joan, You’re accurate as usual. Do you think it’s worth writing to exploit-db to correct those misleading reports? Inviato da iPhone > Il giorno 20 mag 2020, alle ore 19:29, Joan Touzet ha > scritto: > > Hi Andrea, > >> On 2020-05-20 9:37, Andrea Brancatelli wrote: >> A client sent us a link about a supposed security problem with one of >> our couchdb 2.3.1 instances. >> He related to this https://www.exploit-db.com/exploits/46595 which, to >> me, seems a quite confused report that, I guess, can be related to a >> "out of the box" couchdb setup in admin party. > > I agree. > > The first 3 things are just showing that, in admin party, you can create a > DB, delete a DB, and create a document. This is nothing new. > > #4 is showing you can create an admin on a new install if there is no admin > there already. Same thing. > > #5 and #6 are nonsense entries, in that they are adding nonsense config > settings through the admin config API. Not only are these not possible once > you leave admin party, junk in the config file like this will be ignored. > > There is no new exploit or CVE here. > >> Am I wrong? Do a correctly setup couchdb with a local admin and correct >> grants to the dbs suffer of that issue? > > Nope! In short, none of this is possible once you disable admin party - > except for #3 in 2.x, and that's fixable by tightening up each DB's _security. > >> Thanks. > > -Joan "open by default is confusing in 2020" Touzet
CouchDB vulnerability
A client sent us a link about a supposed security problem with one of our couchdb 2.3.1 instances. He related to this https://www.exploit-db.com/exploits/46595 which, to me, seems a quite confused report that, I guess, can be related to a "out of the box" couchdb setup in admin party. Am I wrong? Do a correctly setup couchdb with a local admin and correct grants to the dbs suffer of that issue? Thanks. -- Andrea Brancatelli
Re: Disable Compaction for a single database
Thanks, I appreciate it a lot.
I'm forwarding your code to our pouch-eous guys :-)
---
Andrea Brancatelli
Schema31 S.p.a.
On 2020-04-28 15:30, Joel Jucá wrote:
> Andrea,
>
> Just like Robert and Garren said, CouchDB is not designed to work this way.
> But I'm thinking of an architecture on top of it:
>
> You could, for instance, create document versions that point to their
> ancestors. This way you would create a tree of versions (it's necessary if
> you want to preserve the eventual consistency capabilities).
>
> The document layout would look like this:
>
> type DocumentVersion = {
> // CouchDB's own ID system
> _id: string;
>
> // Your document history ID - the same across all versions of a document
> historyId: string;
>
> // The _id of the ancestor document (undefined if first version)
> parentId: string;
>
> // Indicates if this version is archived or not
> archived: boolean;
>
> // Utility flags. Might be useful in future implementations of
> reconciliation
> // algorithms, or the construction of a changelog feature
> createdAt: Date;
> };
>
> You would have to handle the versioning manually - something like this:
>
> const saveDocumentVersion = async ({ _id, doc }) => {
> // Avoid saving a document version that's already archived
> delete doc.archived;
>
> const newDoc = {
> ...doc,
> createdAt: new Date().toISOString(),
> };
>
> let parentDoc;
>
> if (doc.parentId) {
> parentDoc = await loadDocument(doc.parentId);
> }
>
> if (parentDoc) {
> // Rely on parentDoc to get historyId and parentId
> newDoc.historyId = parentDoc.historyId;
> newDoc.parentId = parentDoc._id;
> } else {
> newDoc.historyId = await generateHistoryId();
> }
>
> const savedDoc = await saveDocument(newDoc);
>
> // Archives the parent document only after new version is saved
> // (sends `parentDoc.archived = true` to CouchDB)
> if (parentDoc) {
> await archiveDocument(parentDoc);
> }
>
> return savedDoc;
> };
>
> It could serve you for simpler use cases, but not for complex ones (you
> would need a stronger reconciliation algorithm to perform reconciliation in
> updates across nodes).
>
> I made a Gist with this code, so you can read it more comfortably:
> https://gist.github.com/joelwallis/67bfbdcf0aa42418eed4c0246f5fe2a4
>
> I hope it helps you in some way. Stay safe!
>
> On Tue, Apr 28, 2020 at 7:46 AM Garren Smith wrote:
>
> I think it would be better to create a daily or hourly snapshot of your
> database instead of relying on a database that doesn't run compaction.
> Depending on the versioning history of a CouchDB database is a really bad
> idea.
> As Bob said, rather create new docs than one document with lots of
> revisions. PouchDB is slow to replicate documents with lots of revisions
> versus lots of new documents.
>
> Cheers
> Garren
>
> On Tue, Apr 28, 2020 at 9:06 AM Andrea Brancatelli
> wrote:
>
> Hello Robert,
>
> I see your point and mostly understand it. The plan was not to "use"
> this secondary database as an active one, but as a passively replicated
> database from a main instance, so performances of the secondary database
> weren't a big priority - the idea is to keep the whole "journal" of the
> main database.
>
> We thought of having multiple copies of the documents as well, but the
> "client" is a React/Pouch application and that would become a pita.
>
> My plan was to have a main database with a very aggressive compaction
> rule, so that pouch replication would be as fast as possibile and the
> local storage be as little as possible (also because pouch isn't blazing
> fast with local views and indexes when you have a lot of documents) and
> a secondary replicated database with a more relaxed compaction rule (as
> I was saying maybe disabled at all) to run backups on or to do
> post-mortem analysis of any problem that may rise on business logic.
>
> ---
>
> Andrea Brancatelli
>
> On 2020-04-27 20:34, Robert Samuel Newson wrote:
>
> Hi,
>
> This is the most common mistake made with CouchDB, that it provides (or could
> provide) a full history of document changes.
> Compaction is essential, it's the only time that the b+tree's are rebalanced
> and obsolete version of b+tree nodes are removed from disk.
>
> If the old revisions of your documents really matter, make new
documents
> instead of updating them, and use some scheme of your choice to group
them
> (you could use a view on some property common to all revisions of the same
> logical document).
>
> B.
>
> On 27 Apr 2020, at 17:10, Andrea Brancatelli <
abrancate...@schema31.it.INVALID>
> wrote:
> Let's say I'd like to keep the whole revision history for documents
in a
> specific database (but maybe drop old views, if it's possible).
>
> What compaction setting would do that overriding the more-reasonable
> default we usually have?
>
> --
>
> Andrea Brancatelli
Re: Disable Compaction for a single database
Hello Robert, I see your point and mostly understand it. The plan was not to "use" this secondary database as an active one, but as a passively replicated database from a main instance, so performances of the secondary database weren't a big priority - the idea is to keep the whole "journal" of the main database. We thought of having multiple copies of the documents as well, but the "client" is a React/Pouch application and that would become a pita. My plan was to have a main database with a very aggressive compaction rule, so that pouch replication would be as fast as possibile and the local storage be as little as possible (also because pouch isn't blazing fast with local views and indexes when you have a lot of documents) and a secondary replicated database with a more relaxed compaction rule (as I was saying maybe disabled at all) to run backups on or to do post-mortem analysis of any problem that may rise on business logic. --- Andrea Brancatelli On 2020-04-27 20:34, Robert Samuel Newson wrote: > Hi, > > This is the most common mistake made with CouchDB, that it provides (or could > provide) a full history of document changes. > > Compaction is essential, it's the only time that the b+tree's are rebalanced > and obsolete version of b+tree > nodes are removed from disk. > > If the old revisions of your documents really matter, make new documents > instead of updating them, and use some scheme of your choice to group them > (you could use a view on some property common to all revisions of > the same logical document). > > B. > >> On 27 Apr 2020, at 17:10, Andrea Brancatelli >> wrote: >> >> Let's say I'd like to keep the whole revision history for documents in a >> specific database (but maybe drop old views, if it's possible). >> >> What compaction setting would do that overriding the more-reasonable >> default we usually have? >> >> -- >> >> Andrea Brancatelli
Disable Compaction for a single database
Let's say I'd like to keep the whole revision history for documents in a specific database (but maybe drop old views, if it's possible). What compaction setting would do that overriding the more-reasonable default we usually have? -- Andrea Brancatelli
Re: CouchDB 2.3.1 - Really Big Document Breaks View
We had a similar problem we fixed by having couchjs allocate more RAM.
We used the Environment Variable to configure the Query Servers like
this:
COUCHDB_QUERY_SERVER_JAVASCRIPT="/usr/local/libexec/couchdb2/bin/couchjs
-S 536870912 /usr/local/libexec/couchdb2/share/server/main.js"
Obviously feel free to tune the memory size (-S) and the path.
Consider that Couch can run many couchjs at the same time, so plan the
-S parameter accordingly.
---
Andrea Brancatelli
On 2020-04-09 22:29, Mantell, Christopher Arthur wrote:
> Hi,
>
> One document in our database is really big (1.2 million lines) and it causes
> an OS_process_error when we access it. I've tried increasing the
> os_process_timeout to 12 but it didn't fix it. I don't know if this
> should be increased further. Deleting the document causes the view to work
> again.
>
> Unfortunately there is no ideal way around not using this document or
> splitting it up into smaller documents. Is there anything I can do to
> Couch's configurations so this document won't break the view?
>
> Here is the output in the couch.log:
>
> [error] 2020-04-09T17:29:05.046811Z couchdb@127.0.0.1 <0.2218.0> 4e36c78fcd
> rexi_server: from: couchdb@127.0.0.1(<0.745.0>) mfa: fabric_rpc:map_view/5
> exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.e\
> rl"},{line,265}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,205}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,462}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,526}]},\
> {couch_mrview_util,fold_fun,4,[{file,"src/couch_mrview_util.erl"},{line,437}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_bt\
> ree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,219}]}]
>
> Any help would be greatly appreciated!
>
> Thank you,
>
> Chris Mantell
>
> Programmer
>
> Athinoula A Martinos Center for Biomedical Imaging
> Massachusetts General Hospital
> Building 149 - South Central
> Charlestown, MA 02129
> cmant...@mgh.harvard.edu
>
> The information in this e-mail is intended only for the person to whom it is
> addressed. If you believe this e-mail was sent to you in error and the e-mail
> contains patient information, please contact the Partners Compliance HelpLine
> at
> http://www.partners.org/complianceline . If the e-mail was sent to you in
> error
> but does not contain patient information, please contact the sender and
> properly
> dispose of the e-mail.
Re: CouchDB Sessions
OK, I see. I was confused by the DELETE /_session endpoint. BTW, what I'm trying to do is just the possibility to have an user logout another session of his own. I clearly understand that with something else sitting in front of couch and hiding couch's session I can do the same, but then I don't exactly grasp the coolness of the /_session endpoint, one could just always inject the Basic Auth and have the same result (without the hassle of the expiration). I guess I'm misunderstanding something, please shed a light if you can ;-) --- Andrea Brancatelli On 2020-01-21 16:44, Jonathan Hall wrote: > No, there's not. I've previously answered this same question on > StackOverflow: https://stackoverflow.com/a/43354080/13860 Answer pasted > below: > > Is it possible to view a list of active user sessions on a couchdb > server? > > Short answer: No. > > Long answer: There's no such thing, really, as user sessions in CouchDB. > > CouchDB's "user session" cookies are just an HMAC of the user's password > salt, the server secret, and the time the cookie was created (so it can tell > when it expires). > > This means that an "active session" is any cookie that contains a valid HMAC > composed from a valid user salt, the valid user cookie, and any timestamp > that is less than N minutes in the past (where N is the expiration time). > > These sessions don't even have to be created on the CouchDB server, so even > logging auth requests is not sufficient. It's a common practice in some > situations to create these cookies in an app external to CouchDB. > > As a followup question: > > Why are you interested in listing active sessions? Maybe there's an > alternative approach to accomplish whatever you're aiming for. > > On 1/21/20 3:54 PM, Andrea Brancatelli wrote: > >> Hello everybody, >> >> speaking of the _session endpoint, is there any way to have a list of >> active sessions by _user? >> >> I don't seem to find one in the docs but maybe it's me... :-)
CouchDB Sessions
Hello everybody, speaking of the _session endpoint, is there any way to have a list of active sessions by _user? I don't seem to find one in the docs but maybe it's me... :-) -- Andrea Brancatelli
Help in understanding an error message
>From time to time I get these errors in my CouchDB log.
Can anyone help to understand what it's complaining about?
[error] 2019-06-05T14:48:57.184907Z couchdb@10.133.79.176 <0.18022.1113>
rexi_server: from: couchdb@10.133.79.175(<12084.22463.1153>)
mfa: fabric_rpc:all_docs/3 exit:timeout
[{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,265}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,205}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,462}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,526}]},{couch_bt_engine,include_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,1074}]},{couch_bt_engine,skip_deleted,4,[{file,"src/couch_bt_engine.erl"},{line,1069}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]}]
It's a CouchDB 2.3.1 on FreeBSD 11.2 release p9, with Erlang 21.3.8.1
Thanks.
--
Andrea Brancatelli
Re: How many replications per server?
OK. Thanks a lot. I'll investigate a bit on how they work and then will try to update the documentation accordingly. I'm not very fluent in Erlang, if I have doubts I'll have to ask for your help again. --- Andrea Brancatelli On 2018-11-07 00:06, Adam Kocoloski wrote: > Hmph, that seems to be an oversight in the documentation! Here's the set of > performance-related configuration options that can be specified as top-level > fields in the replication specification: > > worker_batch_size > worker_processes > http_connections > connection_timeout > retries_per_request > socket_options > checkpoint_interval > use_checkpoints > > These are all documented in the [replicator] section of the configuration > docs, which is where you'd go to set the defaults for all replications > mediated by that server: > > http://docs.couchdb.org/en/stable/config/replicator.html#replicator > <http://docs.couchdb.org/en/stable/config/replicator.html#replicator> > > Configuring one of those fields in the replication doc will always override > the default for the server. There are several other additional fields that > are meaningful in a replication document -- I haven't checked to see if every > one is documented. The code that validates them all is here: > > https://github.com/apache/couchdb/blob/2.2.0/src/couch_replicator/src/couch_replicator_docs.erl#L469-L529 > > <https://github.com/apache/couchdb/blob/2.2.0/src/couch_replicator/src/couch_replicator_docs.erl#L469-L529> > > Looks like we have a bit of homework to do here ... Cheers, > > Adam > > On Nov 6, 2018, at 2:15 AM, Andrea Brancatelli > wrote: > > Hi Adam, > > can you elaborate a bit on the "It's also possible to override resource > settings on a per-replication basis" topic? > > I can't seem to find anything here: > http://docs.couchdb.org/en/stable/replication/replicator.html > > Neither here: > http://docs.couchdb.org/en/stable/api/server/common.html#replicate > > --- > > Andrea Brancatelli > > On 2018-10-30 17:17, Adam Kocoloski wrote: > > The `worker_processes` and `http_connections` in particular can have a > significant impact on the resource consumption of each replication job. If > your goal is to host a large number of lightweight replications you could > reduce those settings, and then configure the scheduler to keep a large > `max_jobs` running. It's also possible to override resource settings on a > per-replication basis.
Re: How many replications per server?
Hi Adam, can you elaborate a bit on the "It's also possible to override resource settings on a per-replication basis" topic? I can't seem to find anything here: http://docs.couchdb.org/en/stable/replication/replicator.html Neither here: http://docs.couchdb.org/en/stable/api/server/common.html#replicate --- Andrea Brancatelli On 2018-10-30 17:17, Adam Kocoloski wrote: > The `worker_processes` and `http_connections` in particular can have a > significant impact on the resource consumption of each replication job. If > your goal is to host a large number of lightweight replications you could > reduce those settings, and then configure the scheduler to keep a large > `max_jobs` running. It's also possible to override resource settings on a > per-replication basis.
Re: How many replications per server?
See if you like this:
https://andrea.brancatelli.it/2018/10/31/couchdb-replication-scheduler-tweak-and-tunes/
---
Andrea Brancatelli
On 2018-10-30 18:53, Adam Kocoloski wrote:
> Precisely :)
>
> I agree the settings can be difficult to grok. I'll bet a few examples in a
> blog post would go a long way towards illustrating the interplay between
> them. Cheers,
>
> Adam
>
> On Oct 30, 2018, at 1:03 PM, Andrea Brancatelli
> wrote:
>
> Thanks Adam, that's what I thought as well, but believe me I'm having a
> really hard time understanding the explanation of max_jobs and max_churns
> from the docs.
>
> I don't exactly get the difference between those two values. My first guess
> was that max_jobs was a systemwide max value while max_churn would define how
> many jobs would run at the same time.
>
> I tried it and it wasn't working as expected.
>
> Now I just reread it and I'm guessing that
>
> while true {
>
> if (jobs > max_jobs) {
>
> for (x = 1 to max_churn) {
>
> kill_or_start(something)
>
> }
>
> }
>
> sleep(interval)
>
> }
>
> Is this correct?
>
> ---
>
> Andrea Brancatelli
>
> On 2018-10-30 17:17, Adam Kocoloski wrote:
> Hi Andrea, your numbers don't sound crazy for an out-of-the-box setup.
>
> Worth noting that in CouchDB 2.1 and above there is a replication scheduler
> which can cycle through an ~unlimited number of continuous replications
> within a defined resource envelope. The scheduler is documented here:
>
> http://docs.couchdb.org/en/stable/replication/replicator.html#replication-scheduler
>
> <http://docs.couchdb.org/en/stable/replication/replicator.html#replication-scheduler>
>
> There are a number of configuration properties that govern the behavior of
> the scheduler and also the default resources allocated to any particular
> replication. These are clustered in the [replicator] configuration block:
>
> http://docs.couchdb.org/en/stable/config/replicator.html#replicator
> <http://docs.couchdb.org/en/stable/config/replicator.html#replicator>
>
> The `worker_processes` and `http_connections` in particular can have a
> significant impact on the resource consumption of each replication job. If
> your goal is to host a large number of lightweight replications you could
> reduce those settings, and then configure the scheduler to keep a large
> `max_jobs` running. It's also possible to override resource settings on a
> per-replication basis.
>
> Cheers, Adam
>
> On Oct 30, 2018, at 11:52 AM, Stefan Klein wrote:
>
> Hi,
>
> can't comment on the behavior of recent, 2.x, versions of couchdb.
>
> Long time ago, with couchdb 1.4 or so I ran a similar test.
> Our solution was to:
> * keep a list of "active" users (by our application specific definition)
> * listen to _db_changes
> * run one-shot replications for the changed documents to the per-user dbs
> of the users who got access to the documents and are "active"
> When a users becomes "active" - again determined by application logic - a
> one-shot replication is run to bring the per-user db up to date.
>
> Sadly this logic is deeply integrated in our application code and can't be
> easily extracted to a module (we're using nodejs).
> It's also basically unchanged since then and we have to adapt to couchdb
> 2.x.
>
> regards,
> Stefan
>
> Am Di., 30. Okt. 2018 um 16:22 Uhr schrieb Andrea Brancatelli <
> abrancate...@schema31.it>:
>
> Sorry the attachment got stripped - here it is:
> https://pasteboard.co/HKRwOFy.png
>
> ---
>
> Andrea Brancatelli
>
> On 2018-10-30 15:51, Andrea Brancatelli wrote:
>
> Hi,
>
> I have a bare curiosity - I know it's a pretty vague question, but how many
> continuous replication jobs one can expect to run on a single "common"
> machine?
> With common I'd say a quad/octa core with ~16GB RAM...
>
> I don't need an exact number, just the order of it... 1? 10? 100? 1000?
>
> I've read a lot about the per-user approach, the filtered replication and all
> that stuff, but on a test server with 64 replication jobs (1
> central user and 32 test users) the machine is totally bent on its knees:
> root@bigdata-free-rm-01:~/asd # uptime
> 3:50PM up 5 days, 4:55, 3 users, load averages: 9.28, 9.84, 9.39
>
> I'm attaching a screenshot of current htop output (filtered for CouchDB user,
> but it's the only thing running on the machine)...
> --
>
> Andrea Brancatelli
Re: How many replications per server?
OK.
This my first attempt so please be gentle with me :-)
https://github.com/apache/couchdb-documentation/pull/345
Plus I'm not native english so a double check would be great.
---
Andrea Brancatelli
On 2018-10-30 22:10, Joan Touzet wrote:
> Hi Mark,
>
> Thanks for the suggestion! Are you volunteering? :) Please remember that
> CouchDB is a volunteer-run project. We're always on the lookout for new
> contributors.
>
> If you know your way around a text editor and GitHub, we'd love a pull
> request here:
>
> https://github.com/apache/couchdb-documentation/
>
> All the best,
> Joan
>
> - Original Message -
> From: "Mark Richter"
> To: user@couchdb.apache.org
> Sent: Tuesday, October 30, 2018 3:46:05 PM
> Subject: Re: How many replications per server?
>
> It would be better to put examples like that in the official documentation so
> we don't have to search the web to find what needs to be documented in the
> first place.
>
> Just my $0.02.
>
> Mark
>
> ________
> From: Adam Kocoloski
> Sent: Tuesday, October 30, 2018 10:53 AM
> To: Andrea Brancatelli
> Cc: user@couchdb.apache.org
> Subject: Re: How many replications per server?
>
> Precisely :)
>
> I agree the settings can be difficult to grok. I'll bet a few examples in a
> blog post would go a long way towards illustrating the interplay between
> them. Cheers,
>
> Adam
>
> On Oct 30, 2018, at 1:03 PM, Andrea Brancatelli
> wrote:
>
> Thanks Adam, that's what I thought as well, but believe me I'm having a
> really hard time understanding the explanation of max_jobs and max_churns
> from the docs.
>
> I don't exactly get the difference between those two values. My first guess
> was that max_jobs was a systemwide max value while max_churn would define how
> many jobs would run at the same time.
>
> I tried it and it wasn't working as expected.
>
> Now I just reread it and I'm guessing that
>
> while true {
>
> if (jobs > max_jobs) {
>
> for (x = 1 to max_churn) {
>
> kill_or_start(something)
>
> }
>
> }
>
> sleep(interval)
>
> }
>
> Is this correct?
>
> ---
> Andrea Brancatelli
>
> On 2018-10-30 17:17, Adam Kocoloski wrote:
>
> Hi Andrea, your numbers don't sound crazy for an out-of-the-box setup.
>
> Worth noting that in CouchDB 2.1 and above there is a replication scheduler
> which can cycle through an ~unlimited number of continuous replications
> within a defined resource envelope. The scheduler is documented here:
>
> http://docs.couchdb.org/en/stable/replication/replicator.html#replication-scheduler
>
> <http://docs.couchdb.org/en/stable/replication/replicator.html#replication-scheduler>
>
> <http://docs.couchdb.org/en/stable/replication/replicator.html#replication-scheduler
>
> <http://docs.couchdb.org/en/stable/replication/replicator.html#replication-scheduler>>
>
> There are a number of configuration properties that govern the behavior of
> the scheduler and also the default resources allocated to any particular
> replication. These are clustered in the [replicator] configuration block:
>
> http://docs.couchdb.org/en/stable/config/replicator.html#replicator
> <http://docs.couchdb.org/en/stable/config/replicator.html#replicator>
> <http://docs.couchdb.org/en/stable/config/replicator.html#replicator
> <http://docs.couchdb.org/en/stable/config/replicator.html#replicator>>
>
> The `worker_processes` and `http_connections` in particular can have a
> significant impact on the resource consumption of each replication job. If
> your goal is to host a large number of lightweight replications you could
> reduce those settings, and then configure the scheduler to keep a large
> `max_jobs` running. It's also possible to override resource settings on a
> per-replication basis.
>
> Cheers, Adam
>
> On Oct 30, 2018, at 11:52 AM, Stefan Klein <mailto:st.fankl...@gmail.com>> wrote:
>
> Hi,
>
> can't comment on the behavior of recent, 2.x, versions of couchdb.
>
> Long time ago, with couchdb 1.4 or so I ran a similar test.
> Our solution was to:
> * keep a list of "active" users (by our application specific definition)
> * listen to _db_changes
> * run one-shot replications for the changed documents to the per-user dbs
> of the users who got access to the documents and are "active"
> When a users becomes "active" - again determined by application logic - a
> one-shot replication is run to bring the per-user db up to date.
>
> Sadly this logic is deeply integrated in our ap
Re: How many replications per server?
Thanks Adam, that's what I thought as well, but believe me I'm having a
really hard time understanding the explanation of max_jobs and
max_churns from the docs.
I don't exactly get the difference between those two values. My first
guess was that max_jobs was a systemwide max value while max_churn would
define how many jobs would run at the same time.
I tried it and it wasn't working as expected.
Now I just reread it and I'm guessing that
while true {
if (jobs > max_jobs) {
for (x = 1 to max_churn) {
kill_or_start(something)
}
}
sleep(interval)
}
Is this correct?
---
Andrea Brancatelli
On 2018-10-30 17:17, Adam Kocoloski wrote:
> Hi Andrea, your numbers don't sound crazy for an out-of-the-box setup.
>
> Worth noting that in CouchDB 2.1 and above there is a replication scheduler
> which can cycle through an ~unlimited number of continuous replications
> within a defined resource envelope. The scheduler is documented here:
>
> http://docs.couchdb.org/en/stable/replication/replicator.html#replication-scheduler
>
> <http://docs.couchdb.org/en/stable/replication/replicator.html#replication-scheduler>
>
> There are a number of configuration properties that govern the behavior of
> the scheduler and also the default resources allocated to any particular
> replication. These are clustered in the [replicator] configuration block:
>
> http://docs.couchdb.org/en/stable/config/replicator.html#replicator
> <http://docs.couchdb.org/en/stable/config/replicator.html#replicator>
>
> The `worker_processes` and `http_connections` in particular can have a
> significant impact on the resource consumption of each replication job. If
> your goal is to host a large number of lightweight replications you could
> reduce those settings, and then configure the scheduler to keep a large
> `max_jobs` running. It's also possible to override resource settings on a
> per-replication basis.
>
> Cheers, Adam
>
> On Oct 30, 2018, at 11:52 AM, Stefan Klein wrote:
>
> Hi,
>
> can't comment on the behavior of recent, 2.x, versions of couchdb.
>
> Long time ago, with couchdb 1.4 or so I ran a similar test.
> Our solution was to:
> * keep a list of "active" users (by our application specific definition)
> * listen to _db_changes
> * run one-shot replications for the changed documents to the per-user dbs
> of the users who got access to the documents and are "active"
> When a users becomes "active" - again determined by application logic - a
> one-shot replication is run to bring the per-user db up to date.
>
> Sadly this logic is deeply integrated in our application code and can't be
> easily extracted to a module (we're using nodejs).
> It's also basically unchanged since then and we have to adapt to couchdb
> 2.x.
>
> regards,
> Stefan
>
> Am Di., 30. Okt. 2018 um 16:22 Uhr schrieb Andrea Brancatelli <
> abrancate...@schema31.it>:
>
> Sorry the attachment got stripped - here it is:
> https://pasteboard.co/HKRwOFy.png
>
> ---
>
> Andrea Brancatelli
>
> On 2018-10-30 15:51, Andrea Brancatelli wrote:
>
> Hi,
>
> I have a bare curiosity - I know it's a pretty vague question, but how many
> continuous replication jobs one can expect to run on a single "common"
> machine?
> With common I'd say a quad/octa core with ~16GB RAM...
>
> I don't need an exact number, just the order of it... 1? 10? 100? 1000?
>
> I've read a lot about the per-user approach, the filtered replication and all
> that stuff, but on a test server with 64 replication jobs (1
> central user and 32 test users) the machine is totally bent on its knees:
> root@bigdata-free-rm-01:~/asd # uptime
> 3:50PM up 5 days, 4:55, 3 users, load averages: 9.28, 9.84, 9.39
>
> I'm attaching a screenshot of current htop output (filtered for CouchDB user,
> but it's the only thing running on the machine)...
> --
>
> Andrea Brancatelli
Re: How many replications per server?
Sorry the attachment got stripped - here it is: https://pasteboard.co/HKRwOFy.png --- Andrea Brancatelli On 2018-10-30 15:51, Andrea Brancatelli wrote: > Hi, > > I have a bare curiosity - I know it's a pretty vague question, but how many > continuous replication jobs one can expect to run on a single "common" > machine? > > With common I'd say a quad/octa core with ~16GB RAM... > > I don't need an exact number, just the order of it... 1? 10? 100? 1000? > > I've read a lot about the per-user approach, the filtered replication and all > that stuff, but on a test server with 64 replication jobs (1 central user and > 32 test users) the machine is totally bent on its knees: > > root@bigdata-free-rm-01:~/asd # uptime > 3:50PM up 5 days, 4:55, 3 users, load averages: 9.28, 9.84, 9.39 > > I'm attaching a screenshot of current htop output (filtered for CouchDB user, > but it's the only thing running on the machine)... > > -- > > Andrea Brancatelli
Re: CouchDB 1.x to CouchDB 2.x
Even on the new machine I just updated I'm having a lot of issues with Replication. The same _replicator docs that used to work on 1.7 don't work anymore since the "local" database (I mean any source/target without an url) doesn't seem to work with the very same scenario of the mail I sent a few days ago. In the same way I could not get any kind of replication to work by using a FQDN in the Source or Target with a lot of complains about the _session database. The only way to get it to work was using the machine IP in the source and target url - this way the replication started but I'm receiving a constant bunch of those errors in the log: [error] 2018-10-01T15:55:57.119670Z couchdb@127.0.0.1 <0.2287.13> couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid [error] 2018-10-01T15:56:05.683295Z couchdb@127.0.0.1 <0.2698.13> couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid [error] 2018-10-01T15:56:58.559221Z couchdb@127.0.0.1 <0.4899.13> couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid [error] 2018-10-01T15:57:08.167212Z couchdb@127.0.0.1 <0.5191.13> couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid [error] 2018-10-01T15:58:01.143074Z couchdb@127.0.0.1 <0.7405.13> couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid [error] 2018-10-01T15:58:11.539561Z couchdb@127.0.0.1 <0.7875.13> couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid Given that this is happening with Couch talking directly to itself I'm pretty puzzled by the reason for this header cookie_format_invalid. Does anybody have any suggestion? --- Andrea Brancatelli On 2018-10-01 13:13, Andrea Brancatelli wrote: > Hello everybody. > > Today I upgraded the first CouchDB 1.7 to CouchDB 2.2 in production with > the help of the marvellous couchup, by just coping the old .couch files > in the new locations. > > Everything went smoothly apart from some points: > > * The Documentation doesn't mention the need for the -i switch when > invoking couchup to migrate the _users and _replication databases. Maybe > a note on this would be useful in scenarios when one is migrating an > entire machine. > * I had to manually reapply all the permission on the databases. Seems > a quite obvious conseguence of the replication-process not replicating > the security document. Yet I don't think that replicating a server into > a totally open one makes a lot of sense. Did I miss something? > > Since I have to do a couple more Production servers with more complex > permissions, do you have any brilliant suggestions on how to replicate > permissions too? > > Quite frankly it sounds like an easy improvement to couchup as well, but > personally I'm a bit python-adverse eheheh > > Thanks
CouchDB 1.x to CouchDB 2.x
Hello everybody. Today I upgraded the first CouchDB 1.7 to CouchDB 2.2 in production with the help of the marvellous couchup, by just coping the old .couch files in the new locations. Everything went smoothly apart from some points: * The Documentation doesn't mention the need for the -i switch when invoking couchup to migrate the _users and _replication databases. Maybe a note on this would be useful in scenarios when one is migrating an entire machine. * I had to manually reapply all the permission on the databases. Seems a quite obvious conseguence of the replication-process not replicating the security document. Yet I don't think that replicating a server into a totally open one makes a lot of sense. Did I miss something? Since I have to do a couple more Production servers with more complex permissions, do you have any brilliant suggestions on how to replicate permissions too? Quite frankly it sounds like an easy improvement to couchup as well, but personally I'm a bit python-adverse eheheh Thanks -- Andrea Brancatelli
Re: Problem with Local Replication on CouchDB 2.2
It's the server contacting itself.
No firewall, nothing fancy on the tcpip side, as you can see in the curl
output on the bottom.
---
Andrea Brancatelli
On 2018-09-25 08:48, Rene Veerman wrote:
> maybe an obvious question but did you check that the ports are
> forwarded all the way to rubidio.roma.schema31.it ?
>
> On Mon, Sep 24, 2018 at 4:48 PM Andrea Brancatelli
> wrote:
>
> Just a side note: the very same replication document started working
> when I changed rubidio.roma.schema31.it in 10.33.102.30... but I don't
> know if it's because there's no DNS involved or because Couch may be
> treating the database as Remote - if it makes sense.
>
> ---
>
> Andrea Brancatelli
>
> On 2018-09-24 15:58, Andrea Brancatelli wrote:
>
> Hello everybody,
>
> I'm finally able to play a little with CouchDB 2.2 on FreeBSD.
>
> I was trying a very basic replication from a remote 1.7 couch instance
> to a local one, but I'm not able to get it working.
>
> I tried almost any kind of combination and I've found that not even a
> simple local to local works.
>
> I'm clearly missing something.
>
> This is the error for it:
>
> [error] 2018-09-24T13:54:08.877102Z couchdb@127.0.0.1 <0.5049.0>
> couch_replicator_httpc: auth plugin initialization failed
> "http://rubidio.roma.schema31.it:5984/snapsafe_dev/;
> {session_request_failed,"http://rubidio.roma.schema31.it:5984/_session
> ","snapsafe_dev",{conn_failed,{error,econnrefused}}} [error]
> 2018-09-24T13:54:08.877318Z couchdb@127.0.0.1 <0.5049.0>
>
> throw:{replication_auth_error,{session_request_failed,"
> http://rubidio.roma.schema31.it:5984/_session
> ","snapsafe_dev",{conn_failed,{error,econnrefused: Replication
> 876b75d67ec4fa4b6def5cf07ac0a5f3 failed to start
> "http://rubidio.roma.schema31.it:5984/snapsafe_dev/; ->
> "http://rubidio.roma.schema31.it:5984/userdb-736e6170736166655f646576/;
> doc
> <<"shards/8000-9fff/_replicator.1537794057">>:<<"05769e4d0221c5@rubidio
> ">>stack:[{couch_replicator_httpc,setup,1,[{file,"src/couch_replicator_httpc.erl"},{line,63}]},{couch_replicator_api_wrap,db_open,4,[{file,"src/couch_replicator_api_wrap.erl"},{line,74}]}]
>
> rubidio.roma.schema31.it is the local machine. 5984 is open and
> listening. the name resolves...
>
> root@rubidio:/usr/local/etc/couchdb2 # curl -v
> http://rubidio.roma.schema31.it:5984/_session
> * Trying 10.33.102.30...
> * TCP_NODELAY set
> * Connected to rubidio.roma.schema31.it (10.33.102.30) port 5984 (#0)
>
> GET /_session HTTP/1.1
> Host: rubidio.roma.schema31.it:5984
> User-Agent: curl/7.61.1
> Accept: */* < HTTP/1.1 200 OK
> < Cache-Control: must-revalidate
> < Content-Length: 132
> < Content-Type: application/json
> < Date: Mon, 24 Sep 2018 13:57:25 GMT
> < Server: CouchDB/2.2.0 (Erlang OTP/20)
> <
{"ok":true,"userCtx":{"name":null,"roles":[]},"info":{"authentication_db":"_users","authentication_handlers":["cookie","default"]}}
> * Connection #0 to host rubidio.roma.schema31.it left intact
>
> The password for the user in the replication page is correct.
>
> Note: I was able to get the replication going by setting it as a remote
> to remote replication (with same hostnames and so on)...
>
> I'm very confused... any hint?
Re: Problem with Local Replication on CouchDB 2.2
Just a side note: the very same replication document started working
when I changed rubidio.roma.schema31.it in 10.33.102.30... but I don't
know if it's because there's no DNS involved or because Couch may be
treating the database as Remote - if it makes sense.
---
Andrea Brancatelli
On 2018-09-24 15:58, Andrea Brancatelli wrote:
> Hello everybody,
>
> I'm finally able to play a little with CouchDB 2.2 on FreeBSD.
>
> I was trying a very basic replication from a remote 1.7 couch instance
> to a local one, but I'm not able to get it working.
>
> I tried almost any kind of combination and I've found that not even a
> simple local to local works.
>
> I'm clearly missing something.
>
> This is the error for it:
>
> [error] 2018-09-24T13:54:08.877102Z couchdb@127.0.0.1 <0.5049.0>
> couch_replicator_httpc: auth plugin initialization failed
> "http://rubidio.roma.schema31.it:5984/snapsafe_dev/;
> {session_request_failed,"http://rubidio.roma.schema31.it:5984/_session","snapsafe_dev",{conn_failed,{error,econnrefused}}}
> [error] 2018-09-24T13:54:08.877318Z couchdb@127.0.0.1 <0.5049.0>
>
> throw:{replication_auth_error,{session_request_failed,"http://rubidio.roma.schema31.it:5984/_session","snapsafe_dev",{conn_failed,{error,econnrefused:
> Replication 876b75d67ec4fa4b6def5cf07ac0a5f3 failed to start
> "http://rubidio.roma.schema31.it:5984/snapsafe_dev/; ->
> "http://rubidio.roma.schema31.it:5984/userdb-736e6170736166655f646576/;
> doc
> <<"shards/8000-9fff/_replicator.1537794057">>:<<"05769e4d0221c5@rubidio">>
> stack:[{couch_replicator_httpc,setup,1,[{file,"src/couch_replicator_httpc.erl"},{line,63}]},{couch_replicator_api_wrap,db_open,4,[{file,"src/couch_replicator_api_wrap.erl"},{line,74}]}]
>
> rubidio.roma.schema31.it is the local machine. 5984 is open and
> listening. the name resolves...
>
> root@rubidio:/usr/local/etc/couchdb2 # curl -v
> http://rubidio.roma.schema31.it:5984/_session
> * Trying 10.33.102.30...
> * TCP_NODELAY set
> * Connected to rubidio.roma.schema31.it (10.33.102.30) port 5984 (#0)
>
>> GET /_session HTTP/1.1
>> Host: rubidio.roma.schema31.it:5984
>> User-Agent: curl/7.61.1
>> Accept: */*
> < HTTP/1.1 200 OK
> < Cache-Control: must-revalidate
> < Content-Length: 132
> < Content-Type: application/json
> < Date: Mon, 24 Sep 2018 13:57:25 GMT
> < Server: CouchDB/2.2.0 (Erlang OTP/20)
> <
> {"ok":true,"userCtx":{"name":null,"roles":[]},"info":{"authentication_db":"_users","authentication_handlers":["cookie","default"]}}
> * Connection #0 to host rubidio.roma.schema31.it left intact
>
> The password for the user in the replication page is correct.
>
> Note: I was able to get the replication going by setting it as a remote
> to remote replication (with same hostnames and so on)...
>
> I'm very confused... any hint?
Problem with Local Replication on CouchDB 2.2
Hello everybody,
I'm finally able to play a little with CouchDB 2.2 on FreeBSD.
I was trying a very basic replication from a remote 1.7 couch instance
to a local one, but I'm not able to get it working.
I tried almost any kind of combination and I've found that not even a
simple local to local works.
I'm clearly missing something.
This is the error for it:
[error] 2018-09-24T13:54:08.877102Z couchdb@127.0.0.1 <0.5049.0>
couch_replicator_httpc: auth plugin initialization failed
"http://rubidio.roma.schema31.it:5984/snapsafe_dev/;
{session_request_failed,"http://rubidio.roma.schema31.it:5984/_session","snapsafe_dev",{conn_failed,{error,econnrefused}}}
[error] 2018-09-24T13:54:08.877318Z couchdb@127.0.0.1 <0.5049.0>
throw:{replication_auth_error,{session_request_failed,"http://rubidio.roma.schema31.it:5984/_session","snapsafe_dev",{conn_failed,{error,econnrefused:
Replication 876b75d67ec4fa4b6def5cf07ac0a5f3 failed to start
"http://rubidio.roma.schema31.it:5984/snapsafe_dev/; ->
"http://rubidio.roma.schema31.it:5984/userdb-736e6170736166655f646576/;
doc
<<"shards/8000-9fff/_replicator.1537794057">>:<<"05769e4d0221c5@rubidio">>
stack:[{couch_replicator_httpc,setup,1,[{file,"src/couch_replicator_httpc.erl"},{line,63}]},{couch_replicator_api_wrap,db_open,4,[{file,"src/couch_replicator_api_wrap.erl"},{line,74}]}]
rubidio.roma.schema31.it is the local machine. 5984 is open and
listening. the name resolves...
root@rubidio:/usr/local/etc/couchdb2 # curl -v
http://rubidio.roma.schema31.it:5984/_session
* Trying 10.33.102.30...
* TCP_NODELAY set
* Connected to rubidio.roma.schema31.it (10.33.102.30) port 5984 (#0)
> GET /_session HTTP/1.1
> Host: rubidio.roma.schema31.it:5984
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: must-revalidate
< Content-Length: 132
< Content-Type: application/json
< Date: Mon, 24 Sep 2018 13:57:25 GMT
< Server: CouchDB/2.2.0 (Erlang OTP/20)
<
{"ok":true,"userCtx":{"name":null,"roles":[]},"info":{"authentication_db":"_users","authentication_handlers":["cookie","default"]}}
* Connection #0 to host rubidio.roma.schema31.it left intact
The password for the user in the replication page is correct.
Note: I was able to get the replication going by setting it as a remote
to remote replication (with same hostnames and so on)...
I'm very confused... any hint?
--
Andrea Brancatelli
Binding both IPv4 and IPv6
Hello, I'm trying to have CouchDB bind Both to IPv6 and IPv4 but I don't seem to able to do it properly. I google ad bit and found the magic "any" keyword to be used in bind_address (the documentation just report ::1 to enable ipv6 or 0.0.0.0 to enable ipv4 but not the combination of both), yet even with "any" I only get a IPv4 Binding. I'm on FreeBSD and still on Couch 1.7.2. Anyone has any hint on this? Thanks -- Andrea Brancatelli
Re: Dynamic Filtered Replication
OK, case 1 sounds satisfying to me.
And I suppose that, since the Slave DB has every “duplicate”/“already
replicated” document with a Lower or Equal revision level the replication would
just skip it.
Sounds cool.
Thanks.
---
Andrea Brancatelli
> On 28 Aug 2018, at 15:12, James Dingwall wrote:
>
> On 28/08/18 13:51, Andrea Brancatelli wrote:
>> Hello everybody.
>>
>> I have a pretty academic question about CouchDB replication…
>>
>> I’m working on creating a setup to test my idea but I thought asking could
>> save me some headaches.
>>
>> This is the scenario.
>>
>> Let’s suppose I have two database: Master and Slave with a Filtered
>> Replication that uses some information stored in the _user database to
>> determine wether to replicate the document or not …
>>
>> Now we put 100 docs in the Master DB, so Master Update SEQ reaches, let’s
>> say, 100.
>>
>> The filter has filtered the replicated documents and on the Slave DB we
>> have, let’s say 40 documents.
>>
>> The condition in the _user database changes and the new condition would mach
>> 60 documents of the Master.
>>
>> Is there an “easy” way to refresh the sincronization between the two
>> database and have the 20 missing docs synced?
>>
>> Maybe resetting the replication would automagically fix this (like the
>> replication skipping the docs that are already in the Slave db?)
>>
>> How would you handle such a situation? Especially when the amount of docs is
>> not 100... :-)
>
> My experience with 1.x are that this can be done by:
>
> 1. recreate the replication with a new _id
> 2. recreate the replication with the same _id but add some query
> parameter to make it unique from the previous
> 3. recreate the replication with the same _id but remove the _local
> document relating to the replication from master and slave dbs. (Use
> the document name is _local/ where replication_id can be
> found from the related task and is processed with (python)
> task["replication_id"].split("+")[0])
> 4. if there is no unique content in the slave and you can afford to miss
> it, just delete it and it will be re-created if your replication is
> allowed to create_target.
>
> By observation the status is maintained in a _local document using a
> checksum of the replication JSON (perhaps the same calculation used to
> generate the _rev) so unless you change the JSON defining the
> replication it will resume from the last sequence recorded in the _local
> document.
>
> My usual approach is 3 although poking at the _local document probably
> isn't supported. The sequence I use is:
> - delete replication definition (_deleted: true, not an HTTP DELETE
> otherwise replication processes may not be correctly terminated)
> - remove _local documents from master and slave (which may not be
> present depending on the status of the existing replication)
> - re-create the replication with the same JSON content as before
>
> The main issue is when your condition for replication changes such that
> documents already present in the slave would no longer be replicated
> according to your new criteria, in this case 4 is the only solution.
>
>>
>> I hope my question is clear enough.
>>
>> Thanks a lot.
>>
>> ---
>> Andrea Brancatelli
>>
>>
>
> James
> Zynstra is a private limited company registered in England and Wales
> (registered number 07864369). Our registered office and Headquarters are at
> The Innovation Centre, Broad Quay, Bath, BA1 1UD. This email, its contents
> and any attachments are confidential. If you have received this message in
> error please delete it from your system and advise the sender immediately.
Dynamic Filtered Replication
Hello everybody. I have a pretty academic question about CouchDB replication… I’m working on creating a setup to test my idea but I thought asking could save me some headaches. This is the scenario. Let’s suppose I have two database: Master and Slave with a Filtered Replication that uses some information stored in the _user database to determine wether to replicate the document or not … Now we put 100 docs in the Master DB, so Master Update SEQ reaches, let’s say, 100. The filter has filtered the replicated documents and on the Slave DB we have, let’s say 40 documents. The condition in the _user database changes and the new condition would mach 60 documents of the Master. Is there an “easy” way to refresh the sincronization between the two database and have the 20 missing docs synced? Maybe resetting the replication would automagically fix this (like the replication skipping the docs that are already in the Slave db?) How would you handle such a situation? Especially when the amount of docs is not 100... :-) I hope my question is clear enough. Thanks a lot. --- Andrea Brancatelli
Re: FreeBSD Port of CouchDB 2.x
Thanks, thanks, thanks, and then again thanks. --- Andrea Brancatelli On 2018-08-24 22:29, Dave Cottlehuber wrote: > Just following up, 2.2.0 is fine with OTP20 and the FreeBSD port should > be committed next week. > > -- > Dave Cottlehuber > +43 67 67 22 44 78 > Managing Director > Skunkwerks, GmbH > http://skunkwerks.at/ > ATU70126204 > Firmenbuch 410811i
Re: PHP couchdb?
Here we use heavily SAG, with generally good satisfaction. https://github.com/sbisbee/sag --- Andrea Brancatelli On 2018-08-01 14:43, Rene Veerman wrote: > found it! :) > > https://github.com/ibm-watson-data-lab/php-couchdb > > https://stackoverflow.com/questions/3684749/creating-regular-users-in-couchdb > > now that allows me to create users and set their roles, but i haven't found > yet how to edit the permissions on a given table (from PHP, using curl or > php-couchdb). i'd much appreciate some tips here. > > On Wed, Aug 1, 2018 at 2:04 PM Rene Veerman wrote: > >> What's a good way to connect to a couchdb from PHP? >> >> I've looked at PHP-on-Couch but it's GPL, and i want something MIT/LGPL >> licensed instead. >> >> I also need something that allows me to add users to the couchdb itself, >> and set permissions on new tables to specific users, >> or my app is never going to be secure.
Re: Couch Document with Fancy ID
Ok later i’ll try.
But if it works, it looks like a bug, isn’t?
Andrea Brancatelli
> Il giorno 27 lug 2018, alle ore 13:13, Jan Lehnardt ha scritto:
>
> Try %20 instead of the +
>
> Cheers
> Jan
> —
>
>> On 27. Jul 2018, at 12:23, Andrea Brancatelli
>> wrote:
>>
>> Hi all.
>>
>> One of our developers, in a drunkiness moment, created a document called
>> "user_Fri Jul 20 2018 15:30:19 GMT+0200 (CEST)".
>>
>> This is giving us a enormous headache to delete it.
>>
>> I guess the problem is with the "+" sign in the ID Any creative
>> suggestion?
>>
>> Some Debug:
>>
>> curl -v https://xxx/snapsafe_dev/_all_docs
>>
>> {"total_rows":6,"offset":0,"rows":[
>>
>> ...cut cut cut
>>
>> {"id":"user_Fri Jul 20 2018 15:30:19 GMT+0200 (CEST)","key":"user_Fri
>> Jul 20 2018 15:30:19 GMT+0200
>> (CEST)","value":{"rev":"1-4c75286f18e14e76a1ade89d02fa95b7"}}
>> ]}
>>
>> curl -v -X DELETE
>> 'https://xx/snapsafe_dev/user_Fri+Jul+20+2018+15%3A30%3A19+GMT%2B0200+%28CEST%29?rev=1-4c75286f18e14e76a1ade89d02fa95b7'
>>
>>
>>> DELETE
>>> /snapsafe_dev/user_Fri+Jul+20+2018+15%3A30%3A19+GMT%2B0200+%28CEST%29?rev=1-4c75286f18e14e76a1ade89d02fa95b7
>>> HTTP/1.1
>>> Host: x
>>> Authorization: Basic x
>>> User-Agent: curl/7.54.0
>>> Accept: */*
>>>
>> < HTTP/1.1 404 Object Not Found
>> < Date: Fri, 27 Jul 2018 12:20:35 GMT
>> < Server: CouchDB/1.7.1 (Erlang OTP/19)
>> < Content-Type: text/plain; charset=utf-8
>> < Content-Length: 41
>> < Cache-Control: must-revalidate
>> <
>> {"error":"not_found","reason":"missing"}
>>
>> --
>>
>> Andrea Brancatelli
>
Re: Couch Document with Fancy ID
It’s couch 1.7 so se have futon.
Futon refuses to handle the document with the same errors.
Andrea Brancatelli
> Il giorno 29 lug 2018, alle ore 09:48, Rene Veerman
> ha scritto:
>
> oh, sorry.. i missed the fact you're already trying a %2B..
>
> try deleting it in fauxton?
>
> On Sun, Jul 29, 2018 at 9:45 AM Rene Veerman
> wrote:
>
>> i think you need a %2B for the +, (according to
>> https://www.urlencoder.org/) instead of a %20.
>>
>> %20 is a space,
>> %2B is the + sign
>>
>> let us know if this fixes it for you ok.. i'd like to know myself.
>>
>>> On Fri, Jul 27, 2018 at 1:13 PM Jan Lehnardt wrote:
>>>
>>> Try %20 instead of the +
>>>
>>> Cheers
>>> Jan
>>> —
>>>
>>>> On 27. Jul 2018, at 12:23, Andrea Brancatelli
>>> wrote:
>>>>
>>>> Hi all.
>>>>
>>>> One of our developers, in a drunkiness moment, created a document called
>>>> "user_Fri Jul 20 2018 15:30:19 GMT+0200 (CEST)".
>>>>
>>>> This is giving us a enormous headache to delete it.
>>>>
>>>> I guess the problem is with the "+" sign in the ID Any creative
>>>> suggestion?
>>>>
>>>> Some Debug:
>>>>
>>>> curl -v https://xxx/snapsafe_dev/_all_docs
>>>>
>>>> {"total_rows":6,"offset":0,"rows":[
>>>>
>>>> ...cut cut cut
>>>>
>>>> {"id":"user_Fri Jul 20 2018 15:30:19 GMT+0200 (CEST)","key":"user_Fri
>>>> Jul 20 2018 15:30:19 GMT+0200
>>>> (CEST)","value":{"rev":"1-4c75286f18e14e76a1ade89d02fa95b7"}}
>>>> ]}
>>>>
>>>> curl -v -X DELETE
>>>> '
>>> https://xx/snapsafe_dev/user_Fri+Jul+20+2018+15%3A30%3A19+GMT%2B0200+%28CEST%29?rev=1-4c75286f18e14e76a1ade89d02fa95b7
>>> '
>>>>
>>>>
>>>>> DELETE
>>> /snapsafe_dev/user_Fri+Jul+20+2018+15%3A30%3A19+GMT%2B0200+%28CEST%29?rev=1-4c75286f18e14e76a1ade89d02fa95b7
>>> HTTP/1.1
>>>>> Host: x
>>>>> Authorization: Basic x
>>>>> User-Agent: curl/7.54.0
>>>>> Accept: */*
>>>>>
>>>> < HTTP/1.1 404 Object Not Found
>>>> < Date: Fri, 27 Jul 2018 12:20:35 GMT
>>>> < Server: CouchDB/1.7.1 (Erlang OTP/19)
>>>> < Content-Type: text/plain; charset=utf-8
>>>> < Content-Length: 41
>>>> < Cache-Control: must-revalidate
>>>> <
>>>> {"error":"not_found","reason":"missing"}
>>>>
>>>> --
>>>>
>>>> Andrea Brancatelli
>>>
>>>
Couch Document with Fancy ID
Hi all.
One of our developers, in a drunkiness moment, created a document called
"user_Fri Jul 20 2018 15:30:19 GMT+0200 (CEST)".
This is giving us a enormous headache to delete it.
I guess the problem is with the "+" sign in the ID Any creative
suggestion?
Some Debug:
curl -v https://xxx/snapsafe_dev/_all_docs
{"total_rows":6,"offset":0,"rows":[
...cut cut cut
{"id":"user_Fri Jul 20 2018 15:30:19 GMT+0200 (CEST)","key":"user_Fri
Jul 20 2018 15:30:19 GMT+0200
(CEST)","value":{"rev":"1-4c75286f18e14e76a1ade89d02fa95b7"}}
]}
curl -v -X DELETE
'https://xx/snapsafe_dev/user_Fri+Jul+20+2018+15%3A30%3A19+GMT%2B0200+%28CEST%29?rev=1-4c75286f18e14e76a1ade89d02fa95b7'
> DELETE
> /snapsafe_dev/user_Fri+Jul+20+2018+15%3A30%3A19+GMT%2B0200+%28CEST%29?rev=1-4c75286f18e14e76a1ade89d02fa95b7
> HTTP/1.1
> Host: x
> Authorization: Basic x
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 404 Object Not Found
< Date: Fri, 27 Jul 2018 12:20:35 GMT
< Server: CouchDB/1.7.1 (Erlang OTP/19)
< Content-Type: text/plain; charset=utf-8
< Content-Length: 41
< Cache-Control: must-revalidate
<
{"error":"not_found","reason":"missing"}
--
Andrea Brancatelli
FreeBSD Port of CouchDB 2.x
Hi, I've recently "awaken" the approval process for CouchDB 2 port in FreeBSD. The porter may have some CouchDB specific questions... maybe someone with more CouchDB 2 experience can give it a look? https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=218844 Thanks. -- Andrea Brancatelli
CouchDB, mod_proxy and Authentication
Hello everybody, I have a problem trying to pass the authentication to couchdb trought apache's 2.4 mod_proxy. I've read various forum/stackoverflow/whatever and many people are reporting that apache is stripping the Authentication headers... I don't exactly understand why that should happens, but the symptom I have is that randomly couch reports 401 and the browser just stay there without speaking a word. The "consumer" is some jQuery stuff but the problem seems to happen with basic requests as well... Do someone has some kind of magic recipe to make it work... -- Andrea Brancatelli
Re: Couchdb 2.1.1 FreeBSD binary packages ?
There is an open request on the FreeBSD ticketing system. It seems stuck but it's not really clear why. References: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=218844 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212846 As soon as the port will be approved it will automatically show in PKG as usual... --- Andrea Brancatelli On 2017-11-20 17:58, Frédéric Alix wrote: > Hello, > > First, a big thank you to the community to dev CouchDB. > I used it everyday and it is a real pleasure to make a production service > with it. > Now, we can download CouchDB binary packages for Debian and Ubuntu. > The same for MacOS X and MS Windows. > Do you plan to make it for FreeBSD 11 too ? > > Best regards, Frédéric
Content Encoding gzip
Hello everybody. Today I realized CouchDB (1.6) is not sending gzip-compressed replies. I googled a bit and found various discussions and even a git repository with the implementation... I was wondering if it ever made it to the main branch... ? Maybe in 2.0? Thanks. -- Andrea Brancatelli
