Patches welcome :) — Professional Support for Apache CouchDB: https://neighbourhood.ie/couchdb-support/
24/7 Observation for your CouchDB Instances: https://opservatory.app SQL Queries for CouchDB: https://neighbourhood.ie/products-and-services/structured-query-server > On 14. Jun 2024, at 13:59, Markus Doppelbauer <doppelba...@gmx.net.INVALID> > wrote: > > Perhaps it wouldn't be too hard to make CouchDB scalable: > > a) When creating a view: add an option to a "key-range map": > Maybe via callback(): "byte-range -> shard" > Something like: from "aa" to "bb" -> node-0; and so on > > b) During "indexing": Store the emitted view-index on the shard > according to the "range-map" > Similar to a partition key, but scalable > > c) During "query": e.g.: startkey=fooAAA & endkey=fooBBB & limit=10 > Map-reduce to query "fooAAA" from shard-8 and "fooBBB" from shard-9 > > CouchDB scales out to 10000+ nodes. > The query only asks 1, maybe 2 shards > > > > Am Mittwoch, dem 12.06.2024 um 16:47 -0400 schrieb Nick Vatamaniuc: >> On Wed, Jun 12, 2024 at 3:54 PM Markus Doppelbauer >> < >> doppelba...@gmx.net.invalid >>> wrote: >> >>> Dear Nick, >>> >>> This means: >>> If Q+N are large enough to distribute the data to all >>> nodes, e.g. 120 nodes Q=40, N=3 >>> Then the view-query: startkey=foobar&endkey=foobaz >>> has to ask all 120 nodes? >>> >> >> In the initial startup phase it could query all 120 nodes, then it >> will >> pick 40 shard workers and will stream from those only. >> >> You do have some control where each of the 3 copies of shards end up: >> https://docs.couchdb.org/en/stable/cluster/databases.html#placing-a-database-on-specific-nodes >> >> >> >>> If I use a partion-key, is it possible to distribute the partioned- >>> view >>> among multiple nodes? >>> The docs say it should stay under 10GB. >>> >> >> A partition cannot be split across multiple shard ranges. A single >> shard >> range can contain multiple partitions. So, for a given partition key, >> it >> would pick a particular shard range from Q, and then it would pick >> one of >> the N copies (usually 3) to stream from. >> >> Cheers, >>> Markus >>> >>> >>> >>> Am Mittwoch, dem 12.06.2024 um 13:54 -0400 schrieb Nick Vatamaniuc: >>>> Another feature related to efficient view querying are >>>> partitioneddatabases: >>>> https://docs.couchdb.org/en/stable/partitioned-dbs/index.html.It's >>>> a >>>> bit of a niche, as you'd need to have a good partition key, >>>> butaside >>>> from that, it can speed up your queries as responses would be >>>> comingfrom a single shard only instead of Q shards. >>>> >>>> >>>> On Wed, Jun 12, 2024 at 1:30 PM Markus Doppelbauer< >>>> doppelba...@gmx.net.invalid >>>>> wrote: >>>>> Hi Nick,Thank you very much for your reply.This is exactly what >>>>> we >>>>> are lookingfor.There are so many DBs that store the secondary >>>>> indexlocally(Cassandra, Aerospike, SyllaDB, ...)Thanks again >>>>> for >>>>> the answerMarcus >>>>> >>>>> Am Mittwoch, dem 12.06.2024 um 13:23 -0400 schrieb Nick >>>>> Vatamaniuc: >>>>>> Hi Marcus,The node handling the request only queries the >>>>>> nodes >>>>>> with shardcopies ofthat database. In a 100 node cluster the >>>>>> shards for thatparticulardatabase might be present on only 6 >>>>>> nodes, depending on theQ and Nsharding factors, so it will >>>>>> query >>>>>> 6 out 100 nodes. Forinstance, for N=3and Q=2 sharding >>>>>> factors, it >>>>>> will first send N*Q=6requests, and wait untilit gets at least >>>>>> one >>>>>> response for each of theQ=2 shard ranges. Thishappens very >>>>>> quickly. Then, for the duration ofthe response, it will >>>>>> onlystream responses from those Q=2 workers.So, to summarize >>>>>> for >>>>>> a Q=2database, it will be a streaming responsefrom 2 workers. >>>>>> For >>>>>> Q=4, from 4workers, etc...Cheers,-Nick >>>>>> On Wed, Jun 12, 2024 at 1:00 PM Markus Doppelbauer< >>>>>> doppelba...@gmx.net.invalid >>>>>>> wrote: >>>>>>> Hello,Is the CouchDB-view a "global" or "local" index?For >>>>>>> example, if acluster has 100 nodes, would the query askfor >>>>>>> a >>>>>>> single node - or100 >>>>>>> nodes?/.../_view/posts?startkey="foobar"&endkey="foobaz"Bes >>>>>>> t >>>>>>> wishesMarcus >
