Re: Intercepting HTTP Requests
> The aim is to implement a least privilege model To achieve that with _rewrite you very likely still need at least a simple proxy, which will redirect most or all inbound requests to a particular _rewrite endpoint. This case R.Newson’s argument doesn’t stand, rewrite becomes mandatory. As for creating DBs – that’s right, user can’t create DBs without being an _admin. As for deprecation – about 80% of QS methods are deprecated, anyway they are useful and there’s no reason not to use them. Porting _rewrite to node.js is pretty straightforward if needed. Hope it helps. ermouth
Re: Intercepting HTTP Requests
Consider using _rewrite as a function, see https://docs.couchdb.org/en/stable/api/ddoc/rewrites.html#using-a-stringified-function-for-rewrites for more details. ermouth On Sat, Jul 8, 2023 at 10:28 PM Ronnie Royston wrote: > I am a CouchDB user. I need more granularity in terms of DB authorization, > e.g. limit who can read a document in a shared database. > > It appears that show functions do get passed the request object, (doc, > req), however it looks like this is discouraged via a deprecation warning. > Update validation documents pass (newDoc, oldDoc, userCtx, secObj) to the > query server, however I need the request object, and for *all* HTTP > methods. > > src/chttpd/src/chttpd_node.erl seems to handle HTTP requests but I do not > know Erlang well enough to pipe all requests out. I would really like to > allow clients/browsers to communicate directly with couch (albeit via > recommended reverse proxy) and not force all db requests through, for > example, Node.js. > > It seems like the query server architecture is 99% there in terms of what I > need - it's just that I need the full request object and need my validation > to get called for every HTTP method. > > How can I restrict access to a document in a shared database based on > userID? I believe I need to intercept HTTP requests and validate them, > right? > > -- >
Re: Couchdb and Nginx Proxies
> http://[::1]:5984/a?limit=1 You likely have 127.0.0.1 denoted as localhost in nginx config. Make it 127.0.0.1 back, it will disable IPv6 connection upstream. ermouth On Sat, Jun 10, 2023 at 3:31 PM Paul Milner wrote: > Hello there, > > further to my issues with couchdb and nginx. I get the following error in > my nginx error log: > > 2023/06/10 11:55:36 [error] 87270#87270: *115 connect() failed (111: > Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, > server: _, request: "GET /a?limit=1 HTTP/1.1", upstream: > "http://[::1]:5984/a?limit=1;, > host: "12.12.12.12" > > The request actually always succeeds but I always get this message. > > My bind in local.ini and default.ini is: > > port = 5984 > bind_address = 127.0.0.1;0.0.0.0;:: > > Would anyone know why please? > > Thanks > Paul >
Re: Photon 1.24: SQL queries for CouchDB
> but still not present, like REGEXP/RLIKE and BETWEEN
Since 1.24.11 Photon supports MySQL style REGEXP/RLIKE and ILIKE, as well
as classic LIKE. BETWEEN and NOT BETWEEN are also now supported.
To address questions about /i modifier for RegExps: both in Mango and in
the Photon version of SQL one should start a regex with (?i) to make it
case insensitive. So RLIKE '(?i)^a' (or {$regex:'(?i)^a'} in Mango) matches
both 'a' and 'A'.
https://github.com/ermouth/couch-photon
ermouth
On Tue, Apr 4, 2023 at 12:44 AM ermouth wrote:
> Hi,
>
> happy to announce Photon 1.24.2, a new version of an alternative CouchDB
> admin panel.
>
> Photon now incorporates glynnbird/sqltomango which allows querying CouchDB
> using a subset of SQL syntax. The feature is available in the dropdown
> where all DB view indices and feeds live.
>
> https://github.com/ermouth/couch-photon
>
> Enjoy!
>
> Cheers,
> ermouth
>
Photon 1.24: SQL queries for CouchDB
Hi, happy to announce Photon 1.24.2, a new version of an alternative CouchDB admin panel. Photon now incorporates glynnbird/sqltomango which allows querying CouchDB using a subset of SQL syntax. The feature is available in the dropdown where all DB view indices and feeds live. Features like joins, unions, aliases, virtual tables are not present – they can’t be implemented using Mango only. Also there are several features that can be implemented, but still not present, like REGEXP/RLIKE and BETWEEN. Hope I’ll add them later. Anyway, for simple requests SQL syntax looks much more concise and handy than Mango, even in the present state. Also you can store SQL (and Mango) queries in localStorage of your browser for later re-use. https://github.com/ermouth/couch-photon Enjoy! Cheers, ermouth
Re: Some random user questions
Hi Vincent, I have somewhat different long standing opinion on several topics touched by Jan, so I decided to add some notes ) > small VPS that also runs my small web-applications as well I also have a lot of small instances, however we also use Couch for much larger projects. Couch scales well not only as a cluster, but also as a mesh, i.e. a set of nodes of very different perf and workload, more or less independent, and only syncing small portions of data or design docs. > how long-time users are querying their JSON documents mostly nowadays Mostly map/reduce, mostly JS. Mango is attractive if your docs are small and are in a way ‘atomized’ (docs with a single data record, not an aggregate of records). A single doc can only emit a single key into a given Mango index, but map function can emit several index keys with different values for each item of a complex doc. Couch is great for large complex docs of flexible structure, map/reduce approach fits them remarkably well. Couch is ok for small docs of predefined structure if performance and storage isn’t an issue. Otherwise SQL-like DBs are generally much faster. > are "Validate Document Update Functions" also part of the "Update functions" >> update functions receive any HTTP requests and transform them into CouchDB docs Actually, rewrite + update functions provide that feature. Update function alone, without rewrite, only accepts requests of predefined syntax, which limits it immensely in terms of transforming any HTTP request into CouchDB doc. > since I use Nginx, I personally never even considered it Well, if you build small webapps using nginx and Couch, consider looking at rewrite as a function. Moving from long nginx configs to JS code was one of the incentives to implement rewrite-as-js-function feature. See https://docs.couchdb.org/en/stable/api/ddoc/rewrites.html#using-a-stringified-function-for-rewrites for more details. Rewrite functions are for making simple APIs right on top of Couch, so they can be useful for established Mango queries – you can have a much simpler API for known requests in your webapp. Cheers, ermouth
Photon 1.23.9
Hi, happy to announce Photon 1.23.9, a new version of an alternative CouchDB admin panel. New features since 1.22: 1. Deleted docs opened from _changes feed are now handled much better by JSON editor: the _deleted property is at the top, and switching it to false allows both Save and Clone actions. This feature is especially helpful for un-deleting docs with retained content. 2. Table view for indices now supports 2 new column modifiers, /int and /curr, so typing smth like “price /curr” at column header shows price properties casted to 123.00 fixed point format. Little reminder: there’s a lot of other modifiers, ie /dt which converts whatever to readable datetime, /list which shows array values line by line, and so on. 3. Added shortcut to _node/_local/_versions into About tab (Couch 3.3+ only). 4. JSON editor now folds/unfolds all top level branches on doc root click. https://github.com/ermouth/couch-photon Enjoy! Cheers, ermouth
Photon 1.22
Hi, happy to announce Photon 1.22, a new release of an alternative CouchDB admin panel. New features since 1.20: — list of docs with conflicts for a DB (simple Mango qry under the hood); — 'Download doc as ZIP' button, which packs a doc and all its attachments as separate files into a zip archive; — 'Add view' dialog. https://github.com/ermouth/couch-photon Enjoy! Cheers, ermouth
Re: CouchDB Memory Usage Unexpectedly Low
Sometimes QS can be the bottleneck. Check os_process_limit config, it may heavily impact db-per-user scenarios. https://docs.couchdb.org/en/2.3.1/config/query-servers.html#query_server_config/os_process_limit ermouth On Tue, Jul 26, 2022 at 7:10 PM Josh Kuestersteffen wrote: > Thanks for the suggestion Wendall! Am I correct in thinking that might be > related to the "Maximum open file descriptors" documentation in the Couch > docs? > https://docs.couchdb.org/en/stable/maintenance/performance.html#maximum-open-file-descriptors-ulimit > Either way, it seems like this is not the cause of our behavior since we > are only seeing 1% of the inodes being used... > Josh > On Jul 25 2022, at 11:18 am, Wendall Cada wrote: > > You may have run out of inodes, or inodes limit is being reached. I've > seen > > this before. Just something to check, not saying it is the issue. Given > how > > many resources you are handling, this is the first thing that came to > mind. > > > > Wendall > > On Mon, Jul 25, 2022 at 8:36 AM Josh Kuestersteffen > > wrote: > > > > > I am trying to figure out a weird situation where we have a decent > sized > > > CouchDB instance (~75GB of data, 2000+ databases) deployed on a host > with > > > 31GB of memory. The issue I am running into is that under heavy load, > Couch > > > starts timing out and throwing errors like it has run out of system > > > resources to use (e.g. req_timedout, exit:timeout), but it is not > maxing > > > out on anything. CPU usage rises as expected but still only peaks > around > > > 70%. > > > > > > It is the memory usage, though, that is very strange to me. Regardless > of > > > how much load is on the server, the memory usage remains essentially > > > constant at ~2.5GB used and ~28GB in "buff/cache". I had expected the > > > memory usage to be considerably higher (given there seems to be plenty > of > > > free memory on the server). > > > Has anyone seen this behavior before? It feels like there is some > > > bottleneck that I am missing here that is preventing Couch from > actually > > > utilizing the available resources... > > > Also FWIW, this is a Couch 2.x instance with these config options > (among > > > others): > > > [couchdb] > > > os_process_timeout = 6 > > > max_dbs_open = 5000 > > > > > > Thank you, > > > Josh Kuestersteffen > > > > > > >
Re: Using CouchApps in Couch 3.2.2
Hi,
CouchDB hosted apps generally work ok in 3.2.2, however small config
tune ups required after upgrade from 3.1.
First you need to carefully copy 3.1 config, considering [couch_httpd_auth]
members moved to [chttpd_auth] since 3.2. Read more here
https://docs.couchdb.org/en/3.2.2/config/auth.html#chttpd_auth.
Also you likely need to set those Couch config keys:
— csp / attachments_enable = false,
— csp / showlist_enable = false
— chttpd_auth / same_site = lax
Also, if you use proxy auth, you might need to tune [chttpd] config
section: add {chttpd_auth, proxy_authentication_handler} into chttpd /
authentication_handlers and restart Couch.
Hope it helps.
ermouth
On Sat, Jul 23, 2022 at 5:51 AM Alan Malta wrote:
>
> keep CouchApps functional (same behavior of 3.1.2?) in 3.2.2? Or is moving
>
>
Photon 1.20
Hi, happy to announce Photon 1.20, a new release of an alternative CouchDB admin panel. New features: — highly anticipated dark theme — last request duration/traffic info for viewindex and Mango requests. Also fixed Chrome-related issues with undumping large ZIPs. https://github.com/ermouth/couch-photon Enjoy! Cheers, ermouth
Photon 1.19
Hi, New Photon 1.19, an alternative CouchDB admin panel, finally has collapsible groups in the DB list. This feature makes Photon comfortable for managing Couch instances with thousands of DBs: collapsed groups not only take much less UI space, are easy to navigate (no paging), but also they make no excessive info and security requests unless expanded. Also since v1.9 Photon received table view, in-browser DB backup/restore to zip, attachments editor, doc cloning feature, and a lot of small improvements here and there. https://github.com/ermouth/couch-photon ermouth
Re: How to clone a document with attachments?
Consider using Photon https://github.com/ermouth/couch-photon, it has explicit button Clone in doc editor, and clones correctly. ermouth ср, 29 сент. 2021 г. в 22:09, Erik Verheul : > Hi all, > > I tried do clone multiple documents, some with attachments, by creating a > new _id and a copy of the fields from the original including "_attachments". > The _bulk_docs post request to save the cloned documents failed with > status code 412. If no document has an attachment all is fine. > > How can I avoid this error? > > Any help would be very much appreciated. > Thanks, > Erik > > >
Re: Proxying document updates with update handlers
Consider using JS _rewrite for this, it can do whatever you want with your
request.
https://docs.couchdb.org/en/2.3.1/api/ddoc/rewrites.html?highlight=_rewrite#using-a-stringified-function-for-rewrites
Please note, that JS _rewrite in v 2.3.1 has a bug which may cause inbound
attachments to drop (https://github.com/apache/couchdb/issues/1878), it was
fixed in 3.0 afaik.
ermouth
чт, 27 мая 2021 г. в 23:09, Aurélien Bénel :
> Dear all,
>
> I have known update handlers for quite long but I never used them "for
> real" myself... My current idea, which must be very common, is to proxy
> updates of whole documents in order to add some accountability of who
> contributed to the document and when.
>
> # rewrites.json
> [{
>"from": "",
>"to": "elsewhere",
>"method": "GET"
> }, {
> "from": "",
> "to": "_update/accounting"
> }, {
>"from": ":object",
> "to": "../../:object",
>"method": "GET"
> }, {
>"from": ":object",
>"to": "_update/accounting/:object"
> }]
>
> # accounting.js
> function(doc, req) {
> var o = JSON.parse(req.body);
> o._id = o._id || req.id || req.uuid;
> var h = doc && doc.history || [];
> h.push({
> user: req.userCtx.name,
> timestamp: new Date()
> });
> o.history = h;
> return [o, {json: {ok: true, id: o._id }}];
> }
>
> Tested on CouchDB 2.3.1, it *nearly* emulates the direct update of a
> document and adds contributions accounting, however I face two problems ;
>
> 1. In the update handler, I see no way to get the new `_rev` value (which
> should be returned either in the JSON body or as an ETag for compatibility
> with normal update of an object). Is there a secret builtin function that
> could be used to get (or set) this? Or is it set afterwards and then cannot
> be get or set at this stage of the process?
>
> 2. In the update handler, when used with POST (with the `_id` in the body
> but not in the URI), it seems that `doc` is always null (even when the ID
> refers to an existing document)... Is this behaviour intended? I feel that
> the documentation could be interpreted both ways...
> Of course, we can still use PUT. But I wanted a complete emulation of
> normal updates (with both methods)...
>
> Any help or insight would be appreciated.
>
>
> Regards,
>
> Aurélien
>
>
>
Re: Changes to 'this' in update function are visible in validate_doc_update. Bug or Feature?
> Or can I? You better not unless you clearly understand how it works and what it depends on. However if you do, this bizzare feature can be invaluable for ie a sort of volatile cache, or (with a certain config) for altering params of a running filtered replication. Anyway, those tricks are very far from being conventional: they are nice to play with, but are no good for production. ermouth ср, 26 мая 2021 г. в 11:17, Stefan Klein : > Hi all, > > i'm trying to prevent unintended changes to some documents. > > While toying around with update & validate_doc_update functions, I found > that, if i add a property to the `this` context of an update function, the > `this` context of the validate_doc_update running after the update contains > that property. > > If I understand the couchdb & query server architecture correct I can not > rely on this behavior. Or can I? > > Thanks & regards, > Stefan Klein >
Dump DB to zip from admin panel
Hi, Just released a new version of Photon admin panel with Dump to zip feature ( https://github.com/ermouth/couch-photon#dump-to-zip-and-restore). Photon can now stream several DBs into a single zip, with _security descriptors, right from the browser; also it can restore DBs from zip. Performance is ok for large docs, however there’s a field for improvements for DBs with zillions of docs smaller than several kb. Also storing conflicts and _deleted docs is not yet implemented. The tool was created mostly for deployment tasks, however it’s ok for non-critical backup of not very large (up to Gb is tested, and ok) DBs I suppose. ermouth
Re: How to deal with updates of denormalized, duplicated data?
You might not need a background task. Since your doc _id:"issue.foo" knows
the _id of related doc exactly, you can emit dependency in your view
response, ie:
views:{foo:{
map: function(doc){
if (doc.type == 'issue') {
emit (doc._id, {_id: doc.status._id}); // this emits dependency doc
}
}
}
Querying this view with include_docs=true will deliver body of a dependency
of type "enumeral" instead of parent doc of type "issue". Read more here
https://docs.couchdb.org/en/3.1.0/ddocs/views/joins.html#linked-documents.
This method shouldn’t be expected any fast, it reads actual docs when it
runs.
There also exists another method, more limited and tricky but sometimes
more fast. Numerals may emit in the same index with issues, but under
special range of keys. Also the view should have a reducer, which will
expand issues with enumerals. Since your reducer is no more a reducer, but
an expander, and bloats data serverside, you need to set reduce_limit=false
at CouchDB config.
When you query that kind of issues-related view, you can append
enumerals keyrange, and reducer joins all things together. Or do reduce at
client side.
Also there is another method employing _list function, which re-combines
view response in a manner of previously stated reducer. That sort of _list
is easier to implement (no rereduce complexities), and unlike reducer it
can pipe data. Note that lists may imply some speed penalties, and they are
deprecated.
Best regards,
ermouth
вт, 1 сент. 2020 г. в 17:51, Olaf Krüger :
> Hi guys,
>
> I am still a CouchDB beginner but I read a lot about it in the meanwhile
> and also about common NOSQL stuff.
> But I still struggle with the question of how to handle duplicate data,
> especially with the update of duplicated data which are "distributed"
> across the entire database.
>
> Following scenario, just as an example:
>
> Imagine there is an issue document which has a relation to an enumeral (A
> particular item of an enumeration).
>
> // Issue doc
> {
> _id: "issue.foo",
> type: "issue",
> title: "No joins with NoSQL",
> // Duplicated "enumeral":
> status: {
> _id: "enumeral.status.in-progress",
> type: "enumeral",
> title: "In progres"
> }
> }
>
> // Origin "enumeral" doc
> {
> _id: "enumeral.status.in-progress",
> type: "enumeral"
> title: "In progres"
> }
>
> Now, we detect a typo at our enumeral:
> The title needs to be updated from "In progres" to "In progress"
> afterwards.
> And of course, all duplicates needs to be updated also (A few thousend
> docs could be affected).
>
> In order to achive this, I would create a background task at application
> level like this:
> - Search for all affected documents.
> - Replace/update the title of each particular doc
> - Bulk update all docs
>
> This might work, but now we created a new revision for each affected
> document.
> If a user works on the previous revision, he will ran into a conflict.
> Basically, this is pretty ok, but in this case, a background task has
> changed the document
> and in this special case, the user shouldn't take care of this little
> change.
>
> As an alternative, instead of duplicating the "enumeral" doc, I tried to
> "join" it by using "linked documents" or other kind of views.
> But then, I got at least two different docs instead of one clean formatted
> JSON doc.
> The result needs to be formatted at application level which doesn't feel
> right to me.
> Moreover, it also doesn't feel right to me to complicate the schema more
> than needed.
>
> So, I would like to ask you experienced guys how do you deal with such
> things.
> There might be not "the one any only" solution, but I would like to ensure
> if I am totally on the wrong path or even not. Maybe I have to change my
> thinking, not sure.
>
> Please also confirm if there's no way out of the dilemma, this would be
> really helpful too ;-)
>
> Many thanks in advance!
>
> Olaf
>
>
>
>
Re: Local-First apps and CouchDB
> it also not feasible in practice for the same reasons ermouth outlines If we shift the scope a bit, CRDT is ok for distributed collaborative allocation/deallocation of quantitative resources or commutative numerical entities of other kinds (ie likes or even spectral vectors). Storing *changes* of tagged numbers or vectors instead of abs values, I mean smth like +1, or [+0.2, -0.5, +0.1, ...], is more or less ok, esp if we aggregate changes received from a single source (ie user or sensor) into a single doc. The reducer for this case is obvious – built-in _sum for instance. Garbage collection is also more or less obvious – collapsing timeseries of a single doc into final result time to time, and then compacting a bucket. However, this solution is already well known and only deserves to be named CRDT for hype or marketing. ermouth
Re: Local-First apps and CouchDB
Both CRDT and OT (operational transform) are implementable over CouchDB. CRDT approach is ok for Couch/Pouch built-in MapReduce mechanics, esp if complicated combining reducers are executed client-side (ie no 64Mb for JS instance limitation). Best results however are achieved if a finalizing part of a reducer is put out of Couch/Pouch. As anything ‘magic’, CRDT collab scheme, esp if there’s a goal to implement a text editor, has its price. The entire text should be split into lower-level atomic blocks, say words, or even characters, and each should be stored as a separate entity inside JSON obj or array. Each action (insert/delete of an entity) is then stored as a separate doc, then distributed over collaborators using _changes, and then re-combined at each client. So, half a kilobyte text easily bloats into hundreds of docs and hundreds of kilobytes of JSON data. Since we store each modification as a separate doc, there’s absolutely no way we can get conflicting doc revisions, so the problem of conflicting docs just never happens. I’ve tested the CRDT collab approach over CouchDB, found it viable, however dropped it for enormous storage overhead. So, excessive storage is the problem, not conflicts or speed; however, bloating is inevitable with CRDT. ermouth чт, 6 авг. 2020 г. в 01:09, Bill Stephenson : > There have been 3 submissions of a paper about "Local-First" apps ( > https://www.inkandswitch.com/local-first.html < > https://www.inkandswitch.com/local-first.html>) to Hacker News in the > past week. Twice it made it to the front page and the most recent was there > most all day and was still on the 2nd page the next morning. > > The paper mentions and discusses CouchDB but I think it drifts quickly > from the subject of "Local-First” apps to conflict resolution and > overlooked using CouchDB as a data storage layer for Local-First apps. > > I thought it’s worth mentioning here because CouchDB makes it easy for > both devs and users to create and install a Local-First app. A couple > months ago I made a very simple demo app that demonstrates one way to do > that. I posted a link to it in the comments on those HN posts and it seemed > to generate a little interest so I thought I’d share it here too... > > https://cherrypc.com/app/editor/setup.html < > https://cherrypc.com/app/editor/setup.html> > > -- > > Bill Stephenson > > > > > >
Newsfeed IFRAME in Fauxton and IP collection
Since I hadn’t received any answer at Github, I’d like to raise an important CouchDB Fauxton security question publicly. One of the latest Fauxton PRs ( https://github.com/apache/couchdb-fauxton/pull/1284) adds a remote newsfeed to Fauxton. Emitting a newsfeed in the admin panel in that way may lead to IP collection of CouchDB instances (or subnets, that is even worse) somewhere. Where is this ‘somewhere’ located? Pinging blog.couchdb.org shows it points to lb.wordpress.com, which seems a bit ridiculous. CouchDB instances are not uncommon for very critical parts of infrastructure and security projects, and I doubt anyone wants to expose node IPs to _whatever_ logs, esp wordpress.com. So I’d like to ask devs and users: does anyone think adding news to the admin panel worth creating such a security hole? ermouth
Re: design document with no name
Reproduced in CouchDB 2.x and 3.x. Photon blocks saving those kind of ddocs from Mango index editor since v1.10.10 that was released just now. See https://github.com/ermouth/couch-photon/issues/26 for details. For those who use Photon for creating Mango indices: do update. Andrea, thanks for reporting. ermouth
Re: design document with no name
> creating a design document with an empty ID This CouchDB bug was filed long ago https://github.com/apache/couchdb/issues/1472, dunno was it fixed. Btw, please provide your version of CouchDB and Photon. ermouth
Re: Publicly available usage/download statistics?
> some online training related to CouchDB Regardless of CouchDB popularity you can count on me if you need any help in this particular task. ermouth
Re: How to work with Indexes
> Just remember to ask the selector in the same order as they are indexed
It doesn’t guarantee appropriate index will be used, however providing
use_index param might help.
https://docs.couchdb.org/en/2.3.1/api/database/find.html?highlight=use_index
ermouth
вт, 9 июн. 2020 г. в 15:18, Andrea Brancatelli
:
> As ermouth said the best approach is to use the explain feature of
> Fauxton / Photon.
>
> Couch tries to guess what's the best index to use for any case so you
> don't necessarily need to build complex indexes, if you ask for more
> selectors than the indexed ones usually it picks an index and then scan
> the documents to apply the subselectors.
>
> Yet I must admit I feel the logic behind it a bit entropic and
> cumbersome.
>
> Just remember to ask the selector in the same order as they are indexed,
> even for partial indexes.
>
> Like a select for { fieldA, fieldB, fieldC } and an index for { fieldA,
> fieldB } would work as you expect, but if the index was on { fieldA,
> fieldC } it won't get used.
>
> On the contrary I don't think Couch is (yet?) able to to Union between
> indexes, so if you have index { fieldA } index { fieldB } and ask for
> select { fieldA, fieldB } only index { fieldA } will be used and then
> all the documents will be processed to find the subset.
>
> That's my empiric deduction from some usage, but someone with the hands
> on the code can probably correct me.
>
> ---
>
> Andrea Brancatelli
>
> On 2020-06-09 13:21, Piotr Zarzycki wrote:
>
> > Hi Aurélien,
> >
> > It is not actually about any specific case. My questions are just helpers
> > to truly understand what approach we should take in our database. I just
> > have hope that I will get more understanding to get answer to some cases.
> >
> > Answer to your questions inline.
> >
> > wt., 9 cze 2020 o 13:04 Aurélien Bénel
> napisał(a):
> >
> > Hi Piotr,
> >
> > Let's say that I have created two indexes for fields "docType" and "age":
> > (...)
> > My question is - How CouchDb will behave in case when I'm searching my db
> > using both fields when there are two indexes which contains the same
> > fields?
> > To answer your question (at least with js views, I don't use Mango but I
> > suppose it to be similar),
> > we need more details:
> >
> > - Are both fields variable parameters ? Or is one of them a constant?
>
> I think both cases I would consider. However the first one would be that
> I'm searching using both fields as a variable parameters. I'm not sure
> about constant question. I think non of those fields are constant - I
> mean
> from the database point of view docType won't change in the document
> that's
> for sure.
>
> > - Will both parameters be always set ? If not, which one might be unset?
> >
> > Let's assume that it always will, but like I said what if I'm looking
> only
> and searching/querying using docType only ?
>
> > Regards,
> >
> > Aurélien
Re: How to work with Indexes
> How CouchDb will behave in case when I'm searching my db > using both fields when there are two indexes which contains the same Consider using Explain button, it exists both in Fauxton and Photon and located in Mango qry panel. Very likely you will discover that CouchDB by default does not use any of indexes. ermouth
Re: scaling?
Miles, mesh-like architecture which uses CouchDB replication protocol to distribute/aggregate data is perfectly viable. As well as other members of the ML I also have several projects employing this approach. And like others I also can not disclose any details, even qty of nodes involved; you better be ready to see this attitude for any mesh-related project, because they often involve processing sensitive personal, industry or infrastructure data in vast amounts. However, I have a lot of observations, main are: - whenever possible use CouchDB, not Pouch, because failure rate of Pouch instances on whatever platform is much higher overall - CouchDB is able to run for years without getting down, Pouch is not so sturdy, there exist a bunch of ways to knock down Pouch node with a single request - on mobile devices you better avoid constantly growing DBs, you may loose data and DB integrity eventually - a single doc rejected by Couch during upstream replication from Pouch can block entire process of replication, you can easily get the situation when sync just can’t restart – so carefully check what you store in Pouch, especially attachments size - get ready to spend a lot of time debugging Pouch—Couch routes having enormous latencies (like sync through satellites or 14400 GPRS connections) - mesh CouchDB networks tend to have very fast growing logs, reason is the number of repeated connections from different peers, so take care of it - do not distribute mesh topology across nodes using CouchDB replication, this approach looks inviting but brings a lot of risks. Best regards. ermouth вс, 24 мая 2020 г. в 18:54, Miles Fidelman : > > On 5/24/20 11:42 AM, Jan Lehnardt wrote: > > > >> On 24. May 2020, at 17:32, Miles Fidelman > wrote: > >> > >> Thanks Jan, and a follow-up, below: > >> > >> On 5/24/20 4:51 AM, Jan Lehnardt wrote: > >>>> On 23. May 2020, at 18:57, Miles Fidelman > wrote: > >>>> > >>>> On 5/23/20 12:29 PM, Jan Lehnardt wrote: > >>>> > >>>>>> On 23. May 2020, at 16:28, Miles Fidelman < > mfidel...@meetinghouse.net> wrote: > >>>>>> > >>>>>> > >>>>>> On 5/22/20 11:13 AM, Jan Lehnardt wrote: > >>>>>> > >>>>>>>> On 22. May 2020, at 15:06, Miles Fidelman < > mfidel...@meetinghouse.net> wrote: > >>>>>>>> > >>>>>>>> Hi Jan, > >>>>>>>> > >>>>>>>> On 5/22/20 6:17 AM, Jan Lehnardt wrote: > >>>>>>>>> Hi Miles, > >>>>>>>>> > >>>>>>>>> I wanted to reply for a while, but struggled to find a good > angle. I think I finally figured out what I missed. I’m not sure I > understand your deployment scenario. > >>>>>>>>> > >>>>>>>>> When I think conference app, I think folks having that on their > mobile phones, or tablets. Given that, you’d be using PouchDB (for web > apps) or one of Cloudant mobile sync libraries (if it is a native app). > Either way, to get the data to the devices, it’d have to come from > somewhere, but then you say there is no central server. Where does the data > come from? > >>>>>>>> I was using that as an example - but I'm really thinking more the > case of "smart documents." Think of a plan (business, mission) or briefing > package - and the whole notion of "living documents. > >>>>>>> not sure what these are and how they compare to CouchDB documents. > >>>>>> "Living Document" as in writerspeak - a binder full of > documentation that is continually being kept up to date, the latest version > of a proposal, a book draft, a theatrical script that's being marked up > with commentary during rehearsal, etc. > >>>>>> > >>>>>> Or think of marrying a wiki to a DCVS. > >>>>>> > >>>>>>>>> It sounds like you imagine the devices talking to each other in > a replication mesh kind of way . While technically possible from a CouchDB > replication protocol point of view, this approach isn’t very practical. For > one, you won’t be able to guarantee that all devices can talk to each > other, especially when you don’t control the wifi. > >>>>>>>>> > >>>>>>>>> What’s more likely is that you’d have central CouchDB server > that is connected to the wifi network, either on site, or in a datacenter > somewhere that all devices connect to.
Re: Sudden very slow indexing of the views
Alan, latest version of Photon, an alternative admin panel for Couch, shows list of ddocs with info about data size and disk size. When ddoc indices files size is much larger than data size, you probably need to compact viewindices of the ddoc. You can select several design docs at a time, and compact their indices in a turn just clicking [Compact...] btn. Hope it might help. Btw, Photon UX is very close to Futon. https://github.com/ermouth/couch-photon ermouth ср, 20 мая 2020 г. в 17:03, Alan Malta : > Hi everyone, > > it's been more than a week that I have been debugging a strange > performance problem with CouchDB; mainly affecting couchdb views. > > About my Couch setup, I have one central couch instance and around 5 > to 15 other instances replicating documents to it. In addition to the > replication, each of those other couch instances are also running a > service that posts documents to the central one via '_bulk_docs' API. > It's important to note that this model is deployed in production for > many years now. > > What started to happen is that the indexing of the views became very > very slow, like < 1k changes within 10min. Making GET calls to the > views (either with or without reduce function), I also see a poor > response rate (a few tens kilobytes, either remotely or localhost). > > Has anyone ever faced such slowness with CouchDB (views)? Would you > have any recommendations on where I should start looking and tests to > be performed? I have already ruled out problems with the virtual > machine and the hypervisor (load is normal for months). I have also > already recreated the views from scratch; recreated the database from > scratch (dumping deleted docs). I have also created a view to see > whether there were any large documents, and the biggest one is only > .5MB. > > When I replicated the database from scratch today, CouchDB indexed > around 1.5M docs in an hour or so; while it's been the last 2h > indexing 26k changes to the database... > > Any help or pointers would be very much appreciated here. > Thanks, > Alan. >
Re: Downgrading to CouchDB 1.6 on Ubuntu 18.04
> to downgrade on a Ubuntu 18.04 server to the old CouchDB 1.6? There’s instructions for 1.7 https://cloudwall.me/setup_couch#h-15vwa7rs ermouth
Photon 1.9.39
Photon admin panel now supports creating partitioned DBs in Couch 3. https://github.com/ermouth/couch-photon Btw, I highly recommend official documentation about partitioning, it’s well written and includes simple but illustrative examples. https://docs.couchdb.org/en/latest/partitioned-dbs/index.html#partitioned-dbs ermouth
Re: CloudStation free CouchDB host now running CouchDB 3.0
New Photon version (1.9.36) fixes several issues and provides more or less useful roundtrips related to CouchDB 3 restrictions for non-admin users. https://github.com/ermouth/couch-photon ermouth чт, 16 апр. 2020 г. в 17:59, ermouth : > NB: until the abovementioned fix is shipped, one can type desired DB name > in URL right after #, and press Enter. This will open the DB doc list. > > ermouth >
Re: CloudStation free CouchDB host now running CouchDB 3.0
NB: until the abovementioned fix is shipped, one can type desired DB name in URL right after #, and press Enter. This will open the DB doc list. ermouth чт, 16 апр. 2020 г. в 17:25, Martin Broerse : > Hi Garren, > > We just added better direct links to databases for Fauxton and Photon. > > [image: image.png] > > Photon will be fixed for the CouchDB 3.0 401 bug too. See > https://github.com/ermouth/couch-photon/issues/22 > >> >>
Re: Maximum number of databases?
In terms of file system, a DB inside Couch is, basically, several files. Number of files depends on number of the DB shards stored at particular node, and also depends on the number of design documents with indices for the DB. Opening a file is costly for OS – so having a lot of sharded DBs is bad in terms of opening/closing a lot of files. Concurrent writing into many files is generally bad practice for spindle devices, it may be also not so optimal for some SSDs. Also, since CouchDB uses ‘append only’ strategy inserting/updating data, having consistent inbound feed multiplexed into several DBs may cause excessive storage fragmentation. So – do test. Also please read this at SO: https://bit.ly/2tJIJnN ermouth
Re: No way to edit a design doc?
> Sometimes I miss Fauton Consider use https://github.com/ermouth/couch-photon, which is admin panel organized similar to Futon, but it works on any Couch. ermouth
Re: How much cost views over index?
A mango index is, basically, a more or less standard view, defined using special syntax. Mango index rules live in design docs and can be created not only using POST db/_index endpoint, but also saving a syntactically valid ddoc directly. Unlike most views, indices emit only keys, no values – so doc parts are included into keys. Since indices are views, they behave as views, at least as I observed (never dug into Mango code). So unless you fetch an index using direct /_view or indirect /_find, no indexing occurs. So – no background indexers and no index update on upsert. ermouth пн, 23 сент. 2019 г. в 09:55, Jose Juan Montiel : > hi ermouth, > > i think it's clear enough, last questions, 1º simple index work in the same > way (in order to trigger the work) i.e. it's the call to the query that > match the index (in the case of partial index the query that use the index) > the trigger that update the index with the latest updates? Or in this case > there is something in background? 2º reggarding to this, in case of create > index with a created database, when does trigger the indexing job, in any > insert o call to database? > > Could i try to find a good place in doc (if not there yet) to accommodate > this Q and made a PR? I would like to contribute. > > Thanks a lot. > > > El lun., 23 sept. 2019 a las 1:16, ermouth () escribió: > > > No penalty until first request to a view. > > > > When a view is accessed, all documents updated since last request to the > > view are processed by the view code. More precisely, all views from the > > parent design document are applied to each updated doc, and their results > > are persisted. > > > > So unless you use at least one of a design doc views at least once, the > > design doc‘s view cost is ~zero. > > > > However, first accessing a view in a DB with a lot of docs may seriously > > impact performance. > > > > ermouth > > > > > > пн, 23 сент. 2019 г. в 01:30, Jose Juan Montiel < > > josejuan.mont...@gmail.com > > >: > > > > > Hi, > > > > > > the extended/explained title is: how much cost/increase views (in terms > > of > > > cpu, memory, storage) over only index, in the life of > > database/application. > > > > > > I know there are different things, for different targets, but i would > > like > > > to know how many things need to do the database, to keep its update > with > > > new inserts. > > > > > > Some people propose in https://jira.hyperledger.org/browse/FAB-5528 to > > > include views in the smartcontract, but the mantainers decide to > dismiss. > > > Now is possible create index, to accelerate the queries, but not > possible > > > to aggregate info directly with view queries. > > > > > > The question its to know how much performance cold the blockchain loose > > if > > > views are created in couchdb. > > > > > > Thanks for all. > > > > > > -- > > > This is your badness level. > > > It's unusually high for someone your size. > > > We have to fix that. > > > > > > > > -- > This is your badness level. > It's unusually high for someone your size. > We have to fix that. >
Re: How much cost views over index?
No penalty until first request to a view. When a view is accessed, all documents updated since last request to the view are processed by the view code. More precisely, all views from the parent design document are applied to each updated doc, and their results are persisted. So unless you use at least one of a design doc views at least once, the design doc‘s view cost is ~zero. However, first accessing a view in a DB with a lot of docs may seriously impact performance. ermouth пн, 23 сент. 2019 г. в 01:30, Jose Juan Montiel : > Hi, > > the extended/explained title is: how much cost/increase views (in terms of > cpu, memory, storage) over only index, in the life of database/application. > > I know there are different things, for different targets, but i would like > to know how many things need to do the database, to keep its update with > new inserts. > > Some people propose in https://jira.hyperledger.org/browse/FAB-5528 to > include views in the smartcontract, but the mantainers decide to dismiss. > Now is possible create index, to accelerate the queries, but not possible > to aggregate info directly with view queries. > > The question its to know how much performance cold the blockchain loose if > views are created in couchdb. > > Thanks for all. > > -- > This is your badness level. > It's unusually high for someone your size. > We have to fix that. >
Photon 1.9
Happy to announce Photon 1.9, new version of alternative CouchDB admin panel. Photon recently celebrated 2 years of development, and just received several unique features: massive DB backup, _stats graphs, and structured revisions diff. Photon provides comfortable view editing, both for JS and Erlang, and also memoizes Mango queries in localStorage for re-use. All those features make Photon especially comfortable for learning CouchDB views of different kinds. Photon distribution is a single design document, so it installs on any CouchDB in just several browser clicks. GitHub repo/readme: https://github.com/ermouth/couch-photon YouTube screencast: https://youtu.be/M9ptWXfwMN8 Please enjoy ) I’m very open to suggestions and issue reports. ermouth
Re: couchdb on multiple domains, mixing http and https?
> multiple https > domains on the same machine for couchdb Consider using nginx or other web server in front of CouchDB. Very basic example of nginx config and certbot install you can find here https://cloudwall.me/setup_couch#h-15vwa7rs. ermouth
