[Couchdb Wiki] Update of Views_for_SQL_Jockeys by JoanTouzet
Dear Wiki user,
You have subscribed to a wiki page or wiki category on Couchdb Wiki for
change notification.
The Views_for_SQL_Jockeys page has been changed by JoanTouzet:
http://wiki.apache.org/couchdb/Views_for_SQL_Jockeys?action=diffrev1=4rev2=5
Comment:
Removing outdated incomplete content in favour of the superior reference
Include(EditTheWiki)
- The latest version is maintained at
http://books.couchdb.org/relax/reference/views-for-sql-jockeys
+ This page now resides
athttp://books.couchdb.org/relax/reference/views-for-sql-jockeys instead.
-
- = View Cookbook for SQL Jockeys =
-
- This is a collection of some common SQL queries and how to get the same
result in CouchDB. The key to remember here is that CouchDB does not work like
an SQL database at all and that best practices from the SQL world do not
translate well or at all to CouchDB. This cookbook assumes that you are
familiar with the CouchDB basics like creating and updating databases and
documents.
-
- == Using Views (CREATE / ALTER TABLE) ==
-
- Using views is a two step process. First you ''define'' a view, then you
''query'' it. This is analogous to defining a table structure (with indexes)
using `CREATE TABLE` or `ALTER TABLE` and querying it using an SQL query.
-
- === Defining a View ===
-
- Defining a view is done by creating a special document in a CouchDB database.
The only actual speciality is the `_id` of the document: it starts with
`_design/`, for example `_design/application`. Other than that, it is just a
regular CouchDB document. To make sure CouchDB understands that you are
defining a view, you need to prepare the contents of that design document in a
special format. Here is an example:
-
- {{{
- {
- _id: _design/application,
- _rev: 1-C1687D17,
- views: {
- viewname: {
- map: function(doc) { ... },
- reduce: function(keys, values) { ... }
- }
- }
- }
- }}}
-
- We are defining a view `viewname`. The definition of the view consists of two
functions. The ''map function'' and the ''reduce function''. Specifying a
reduce function is optional. We'll look at the nature of the functions later.
Note that `viewname` can be whatever you like; `users`, `by-name`, or `by date`
are just some examples.
-
- A single design document can also include multiple view definitions, each
identified by a unique name:
-
- {{{
- {
- _id: _design/application,
- _rev: 1-C1687D17,
- views: {
- viewname: {
- map: function(doc) { ... },
- reduce: function(keys, values) { ... }
- },
- anotherview: {
- map: function(doc) { ... },
- reduce: function(keys, values) { ... }
- }
- }
- }
- }}}
-
- === Querying a View ===
-
- The name of the design document and the name of the view are significant for
querying the view. To query the view `viewname` you perform a HTTP `GET`
request to the following URI:
-
- {{{
- /database/_design/application/_view/viewname
- }}}
-
- `database` is the name of the database you created your design document in.
Next up is the design document name and then the view name prefixed with
`_view/`. To query `anotherview` replace `viewname` in that URI with
`anotherview`. If you want to query a view in a different design document
adjust the design document name.
-
-
- === Map Reduce Functions ===
-
- Map/Reduce is a concept that solves problems by applying a two-step process;
aptly named the ''map'' phase and the 'reduce' phase. The map phase looks at
all documents in CouchDB separately one after the other and creates a ''map
result''. The map result is an ordered list of key-value pairs. Both key and
value can be specified by the user writing the map function. A map function may
call the built-in `emit(key, value)` function 0 to N times per document,
creating a row in the map result per invocation.
-
- CouchDB is smart enough to only run a map function once for every document,
even on subsequent queries on a view. Only changes to documents, or new
documents need to be processed anew.
-
- Map Functions
-
- Map functions run in isolation for every document. They can't modify the
document and they can't talk to the outside world; they can't have
''side-effects''. This is required so CouchDB can guarantee correct results
without having to recalculate a complete result when only one document gets
changed.
-
- The map result looks like this:
-
- {{{
- {total_rows:3,offset:0,rows:[
- {id:fc2636bf50556346f1ce46b4bc01fe30,key:Lena,value:5},
- {id:1fb2449f9b9d4e466dbfa47ebe675063,key:Lisa,value:4},
- {id:8ede09f6f6aeb35d948485624b28f149,key:Sarah,value:6}
- }
- }}}
-
- It is a list of rows sorted by the value of `key`. The `id` is added
automatically and refers back to the document that created this row. The
`value` is the data you're looking for. For example purposes, it's the girl's
age.
-
- The map function that produces this result is:
-
- {{{
- function(doc) {
- if(doc.name
[Couchdb Wiki] Trivial Update of Views_for_SQL_Jockeys by JoanTouzet
Dear Wiki user, You have subscribed to a wiki page or wiki category on Couchdb Wiki for change notification. The Views_for_SQL_Jockeys page has been changed by JoanTouzet: http://wiki.apache.org/couchdb/Views_for_SQL_Jockeys?action=diffrev1=5rev2=6 Include(EditTheWiki) - This page now resides athttp://books.couchdb.org/relax/reference/views-for-sql-jockeys instead. + This page now resides at http://books.couchdb.org/relax/reference/views-for-sql-jockeys instead.
[Couchdb Wiki] Update of Using_Views by JoanTouzet
Dear Wiki user, You have subscribed to a wiki page or wiki category on Couchdb Wiki for change notification. The Using_Views page has been changed by JoanTouzet: http://wiki.apache.org/couchdb/Using_Views?action=diffrev1=7rev2=8 Comment: Updated link to guide which is more complete * [[Introduction_to_CouchDB_views|Introduction to CouchDB Views]] An introduction to views * [[HTTP_view_API|HTTP view API]] How to use Views * [[View_Snippets|View Snippets]] Look at how others solved particular problems using views. - * [[Views_for_SQL_Jockeys|Views for SQL Jockeys]] How to translate SQL knowledge to CouchDB views + * --([[Views_for_SQL_Jockeys|Views for SQL Jockeys]])-- [[http://guide.couchdb.org/draft/cookbook.html|View Cookbook for SQL Jockeys]] How to translate SQL knowledge to CouchDB views * [[Views_for_Lotus_Geeks|Views for Lotus Geeks]] How to translate Lotus Notes views to CouchDB views * [[EnableErlangViews|Enable Erlang Views]] How to enable the native Erlang view server (since CouchDB 0.10.0)
[Couchdb Wiki] Update of View_Snippets by JoanTouzet
Dear Wiki user,
You have subscribed to a wiki page or wiki category on Couchdb Wiki for
change notification.
The View_Snippets page has been changed by JoanTouzet:
http://wiki.apache.org/couchdb/View_Snippets?action=diffrev1=45rev2=46
= View Snippets =
TableOfContents()
- This page collects code snippets to be used in your [[Views]]. They are
mainly meant to help get your head around the map/reduce approach to accessing
database content.
+ This page collects code snippets to be used in your [[Views]]. They are
mainly meant to help get your head around the map/reduce approach to accessing
database content.
'''Remember''': the Futon web client silently adds ''group=true'' to your
views.
== Common mistakes ==
-
When creating a reduce function, a re-reduce should behave in the same way as
the regular reduce. The reason is that CouchDB doesn't necessarily call
re-reduce on your map results.
Think about it this way: If you have a bunch of values ''V1 V2 V3'' for key
''K'', then you can get the combined result either by calling
''reduce([K,K,K],[V1,V2,V3],0)'' or by re-reducing the individual results:
''reduce(null,[R1,R2,R3],1)''. This depends on what your view results look like
internally.
-
== Get docs with a particular user id ==
-
{{{#!highlight javascript
// map
function(doc) {
@@ -26, +23 @@
}
}
}}}
-
Then query with ''key=USER_ID'' to get all the rows that match that user.
-
== Get all documents which have an attachment ==
-
This lists only the documents which have an attachment.
{{{#!highlight javascript
@@ -42, +36 @@
}
}
}}}
-
In SQL this would be something like {{{SELECT id FROM table WHERE attachment
IS NOT NULL}}}.
-
== Count documents with and without an attachment ==
-
Call this with ''group=true'' or you only get the combined number of
documents with and without attachments.
{{{#!highlight javascript
@@ -57, +48 @@
emit(with attachment, 1);
}
else {
- emit(without attachment, 1);
+ emit(without attachment, 1);
}
}
}}}
-
{{{#!highlight javascript
// reduce
function(keys, values) {
return sum(values);
}
}}}
-
Using curl you can call it like this:
{{{
- curl -s -i -X POST -H 'Content-Type: application/json'
+ curl -s -i -X POST -H 'Content-Type: application/json'
- -d '{map: function(doc){if(doc._attachments) {emit(\with\,1);} else
{emit(\without\,1);}},
+ -d '{map: function(doc){if(doc._attachments) {emit(\with\,1);} else
{emit(\without\,1);}},
- reduce: function(keys, values) {return sum(values);}}'
+ reduce: function(keys, values) {return sum(values);}}'
'http://localhost:5984/somedb/_temp_view?group=true'
}}}
-
In SQL this would be something along the lines of {{{SELECT num_attachments
FROM table GROUP BY num_attachments}}} (but this would give extra output for
rows containing more than one attachment).
-
== Generating a list of unique values ==
-
Here we use the fact that the key for a view result can be an array. Suppose
you have a map that generates (key, value) pairs with many duplicates and you
want to remove the duplicates. To do so, use {{{emit([key, value], null)}}} as
the map output.
Call this with ''group=true'' or you only get ''null''.
@@ -95, +81 @@
}
}
}}}
-
{{{#!highlight javascript
// reduce
function(keys, values) {
return null;
}
}}}
+ This will give you results like
- This will give you results like
{{{#!highlight javascript
{rows:[
{key:[thisparent,thatlink],value:null},
{key:[thisparent,thatotherlink],value:null}
]}
}}}
-
- You can then get all the rows for the key ''thisparent'' with the view
parameters
''startkey=[''thisparent]endkey=[thisparent,{}]inclusive_end=false''
+ You can then get all the rows for the key ''thisparent'' with the view
parameters
''startkey=[thisparent]endkey=[thisparent,{}]inclusive_end=false''
Note that the trick here is using the key for what you want to make unique.
You can combine this with the counting above to get a count of duplicate values:
@@ -123, +107 @@
}
}
}}}
-
{{{#!highlight javascript
// reduce
function(keys, values) {
return sum(values);
}
}}}
+ If you then want to know the total count for each parent, you can use the
''group_level'' view parameter:
''startkey=[thisparent]endkey=[thisparent,{}]inclusive_end=falsegroup_level=1''
-
- If you then want to know the total count for each parent, you can use the
''group_level'' view parameter:
-
''startkey=[''thisparent]endkey=[thisparent,{}]inclusive_end=falsegroup_level=1''
== Get contents of an object with specific attributes e.g.
doc.objects.[0].attribute ==
-
{{{#!highlight javascript
// map
function(doc) {
@@ -144, +124 @@
}
}
}}}
+ == Arbitrarily query against any document property (uniview) ==
+ '''Note: This is not a generally recommended approach, as your view size
[1/2] git commit: remove obsolete entry in .gitignore
Updated Branches: refs/heads/1.2.x 32c702572 - c7765f657 refs/heads/master fb3f2ae8b - 6282b5d03 remove obsolete entry in .gitignore Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/c7765f65 Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/c7765f65 Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/c7765f65 Branch: refs/heads/1.2.x Commit: c7765f6574768caf2be2c98331263ab91182cda9 Parents: 32c7025 Author: Randall Leeds rand...@apache.org Authored: Tue Jan 24 13:29:02 2012 -0800 Committer: Randall Leeds rand...@apache.org Committed: Tue Jan 24 13:29:29 2012 -0800 -- .gitignore |1 - 1 files changed, 0 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/couchdb/blob/c7765f65/.gitignore -- diff --git a/.gitignore b/.gitignore index 365d963..cbf2b42 100644 --- a/.gitignore +++ b/.gitignore @@ -45,7 +45,6 @@ var/Makefile bin/couchdb bin/couchdb.1 -bin/couchjs bin/couch-config bin/couch-config_dev etc/couchdb/default.ini
[2/2] git commit: remove obsolete entry in .gitignore
remove obsolete entry in .gitignore Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/6282b5d0 Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/6282b5d0 Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/6282b5d0 Branch: refs/heads/master Commit: 6282b5d030ac74274fa1628a5a0b0e66293297b1 Parents: fb3f2ae Author: Randall Leeds rand...@apache.org Authored: Tue Jan 24 13:29:02 2012 -0800 Committer: Randall Leeds rand...@apache.org Committed: Tue Jan 24 13:29:02 2012 -0800 -- .gitignore |1 - 1 files changed, 0 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/couchdb/blob/6282b5d0/.gitignore -- diff --git a/.gitignore b/.gitignore index a24f0a1..f3bdc52 100644 --- a/.gitignore +++ b/.gitignore @@ -48,7 +48,6 @@ var/Makefile bin/couchdb bin/couchdb.1 -bin/couchjs bin/couch-config bin/couch-config_dev etc/couchdb/default.ini
[2/4] git commit: Prevent multiple updates to a single _local doc
Prevent multiple updates to a single _local doc
If a user uploads a two _local docs with the same docid in a _bulk_docs
request the updater will insert multiple entries into the local docs
btree. This patch avoids that by throwing an error if a user attempts
it.
This is an old bug that I caught and just added a fix for. I split it
out as its own patch to point out that its a bug fix that differs from
the old behvior.
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/9d0db1b9
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/9d0db1b9
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/9d0db1b9
Branch: refs/heads/new-security-object
Commit: 9d0db1b923dba01b665674dc55ada47a4fe6146c
Parents: eb4138f
Author: Paul Joseph Davis dav...@apache.org
Authored: Mon Jan 23 17:56:18 2012 -0600
Committer: Paul Joseph Davis dav...@apache.org
Committed: Wed Jan 25 01:14:07 2012 -0600
--
src/couchdb/couch_db.erl | 10 ++
1 files changed, 10 insertions(+), 0 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/9d0db1b9/src/couchdb/couch_db.erl
--
diff --git a/src/couchdb/couch_db.erl b/src/couchdb/couch_db.erl
index a6903c4..be65f53 100644
--- a/src/couchdb/couch_db.erl
+++ b/src/couchdb/couch_db.erl
@@ -727,6 +727,16 @@ update_docs(Db, Docs, Options, interactive_edit) -
end
end, {[], []}, Docs2),
+% Prevent updating multiple _local docs in a single update
+% request. This relies on couch_db_updater not collecting
+% more than one update that contains _local docs but this
+% is still trigerable with a _bulk_docs request.
+UniqNRIds = lists:usort([Id || #doc{id=Id} - NonRepDocs0]),
+case length(UniqNRIds) == length(NonRepDocs0) of
+true - ok;
+false - throw({update_error, repeated_local_docs})
+end,
+
{NonRepDocs, _} = new_revs(NonRepDocs0, [], []),
DocBuckets = before_docs_update(Db, group_alike_docs(Docs3)),
[4/4] git commit: Implement _security as a _local doc
Implement _security as a _local doc
As per mailing list discussion on enabling the syncing of _security
objects this builds upon the previous commits to enable the agreed upon
strategy of giving a revision tree to the _security object.
The old API is maintained with identical in the normal single node case.
Though it is also now possible to update the _local/_security doc
directly using the normal HTTP document and _bulk_docs API's to edit the
doc directly.
This document is special cased like _design docs to disallow non-admins
from editing it.
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/76fff259
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/76fff259
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/76fff259
Branch: refs/heads/new-security-object
Commit: 76fff259a62e1c2f443d625756d3f8d6c88ee6a8
Parents: 9d0db1b
Author: Paul Joseph Davis dav...@apache.org
Authored: Mon Jan 23 17:59:57 2012 -0600
Committer: Paul Joseph Davis dav...@apache.org
Committed: Wed Jan 25 01:14:07 2012 -0600
--
src/couchdb/couch_db.erl | 22 ++
src/couchdb/couch_db.hrl |1 +
src/couchdb/couch_db_updater.erl | 76 ++---
3 files changed, 65 insertions(+), 34 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/76fff259/src/couchdb/couch_db.erl
--
diff --git a/src/couchdb/couch_db.erl b/src/couchdb/couch_db.erl
index be65f53..987e8f7 100644
--- a/src/couchdb/couch_db.erl
+++ b/src/couchdb/couch_db.erl
@@ -359,21 +359,23 @@
check_is_member(#db{user_ctx=#user_ctx{name=Name,roles=Roles}=UserCtx}=Db) -
end
end.
-get_admins(#db{security=SecProps}) -
+get_admins(#db{security=Doc}) -
+#doc{body={SecProps}} = Doc,
couch_util:get_value(admins, SecProps, {[]}).
-get_members(#db{security=SecProps}) -
+get_members(#db{security=Doc}) -
+#doc{body={SecProps}} = Doc,
% we fallback to readers here for backwards compatibility
couch_util:get_value(members, SecProps,
couch_util:get_value(readers, SecProps, {[]})).
-get_security(#db{security=SecProps}) -
-{SecProps}.
+get_security(#db{security=SecDoc}) -
+SecDoc#doc.body.
-set_security(#db{update_pid=Pid}=Db, {NewSecProps}) when is_list(NewSecProps)
-
+set_security(#db{security=Doc}=Db, {NewSecProps}) when is_list(NewSecProps) -
check_is_admin(Db),
ok = validate_security_object(NewSecProps),
-ok = gen_server:call(Pid, {set_security, NewSecProps}, infinity),
+{ok, _} = update_doc(Db, Doc#doc{body={NewSecProps}}, []),
{ok, _} = ensure_full_commit(Db),
ok;
set_security(_, _) -
@@ -459,10 +461,12 @@ group_alike_docs([{Doc,Ref}|Rest], [Bucket|RestBuckets])
-
validate_doc_update(#db{}=Db, #doc{id= _design/,_/binary},
_GetDiskDocFun) -
catch check_is_admin(Db);
-validate_doc_update(#db{validate_doc_funs=[]}, _Doc, _GetDiskDocFun) -
-ok;
+validate_doc_update(Db, #doc{id= ?SECURITY_ID}, _GetDiskDocFun) -
+catch check_is_admin(Db);
validate_doc_update(_Db, #doc{id= _local/,_/binary}, _GetDiskDocFun) -
ok;
+validate_doc_update(#db{validate_doc_funs=[]}, _Doc, _GetDiskDocFun) -
+ok;
validate_doc_update(Db, Doc, GetDiskDocFun) -
DiskDoc = GetDiskDocFun(),
JsonCtx = couch_util:json_user_ctx(Db),
@@ -731,7 +735,7 @@ update_docs(Db, Docs, Options, interactive_edit) -
% request. This relies on couch_db_updater not collecting
% more than one update that contains _local docs but this
% is still trigerable with a _bulk_docs request.
-UniqNRIds = lists:usort([Id || #doc{id=Id} - NonRepDocs0]),
+UniqNRIds = lists:usort([Id || [{#doc{id=Id}, _}] - NonRepDocs0]),
case length(UniqNRIds) == length(NonRepDocs0) of
true - ok;
false - throw({update_error, repeated_local_docs})
http://git-wip-us.apache.org/repos/asf/couchdb/blob/76fff259/src/couchdb/couch_db.hrl
--
diff --git a/src/couchdb/couch_db.hrl b/src/couchdb/couch_db.hrl
index 65eb7f0..26ffe0d 100644
--- a/src/couchdb/couch_db.hrl
+++ b/src/couchdb/couch_db.hrl
@@ -11,6 +11,7 @@
% the License.
-define(LOCAL_DOC_PREFIX, _local/).
+-define(SECURITY_ID, _local/_security).
-define(DESIGN_DOC_PREFIX0, _design).
-define(DESIGN_DOC_PREFIX, _design/).
-define(DEFAULT_COMPRESSION, snappy).
http://git-wip-us.apache.org/repos/asf/couchdb/blob/76fff259/src/couchdb/couch_db_updater.erl
--
diff --git a/src/couchdb/couch_db_updater.erl b/src/couchdb/couch_db_updater.erl
index 862c48a..42b6460 100644
--- a/src/couchdb/couch_db_updater.erl
+++ b/src/couchdb/couch_db_updater.erl
@@ -67,14 +67,6
[3/4] git commit: Replace _local doc sequence with revision trees
Replace _local doc sequence with revision trees
There are a number of cases that _local docs might need to be merged
between nodes. One motivating case is for clusters that might wish to
move _local docs across nodes to maintain replication checkpoints with
external CouchDB instances. The previous _local docs strategy of a
single linear sequence breaks down in this situation.
This new behavior should be indistinguishable from the previous behavior
assuming clients did not try and introspect the _rev value for _local
documents. It should be impossible for normal HTTP clients to introduce
different behavior than previously existed because there's no support
for non-linear updates at the HTTP level. This update is merely and
internal refactoring for special cases like clusters.
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/eb4138f1
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/eb4138f1
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/eb4138f1
Branch: refs/heads/new-security-object
Commit: eb4138f196f15364e293128cacc2c5011bb28b69
Parents: 0ab5ebd
Author: Paul Joseph Davis dav...@apache.org
Authored: Thu Jan 19 18:10:05 2012 -0600
Committer: Paul Joseph Davis dav...@apache.org
Committed: Wed Jan 25 01:14:07 2012 -0600
--
src/couch_replicator/src/couch_replicator.erl |9 +-
src/couchdb/couch_db.erl | 21 +++-
src/couchdb/couch_db_updater.erl | 116 ++--
3 files changed, 81 insertions(+), 65 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/eb4138f1/src/couch_replicator/src/couch_replicator.erl
--
diff --git a/src/couch_replicator/src/couch_replicator.erl
b/src/couch_replicator/src/couch_replicator.erl
index 1f7c08a..53acfd5 100644
--- a/src/couch_replicator/src/couch_replicator.erl
+++ b/src/couch_replicator/src/couch_replicator.erl
@@ -580,15 +580,18 @@ fold_replication_logs([Db | Rest] = Dbs, Vsn, LogId,
NewId, Rep, Acc) -
fold_replication_logs(Dbs, Vsn - 1,
?l2b(?LOCAL_DOC_PREFIX ++ OldRepId), NewId, Rep, Acc);
{error, not_found} -
+Doc0 = #doc{id = NewId},
+Doc1 = Doc0#doc{revs = {1, [couch_db:new_revid(Doc0)]}},
fold_replication_logs(
-Rest, ?REP_ID_VERSION, NewId, NewId, Rep, [#doc{id = NewId} |
Acc]);
+Rest, ?REP_ID_VERSION, NewId, NewId, Rep, [Doc1 | Acc]);
{ok, Doc} when LogId =:= NewId -
fold_replication_logs(
Rest, ?REP_ID_VERSION, NewId, NewId, Rep, [Doc | Acc]);
{ok, Doc} -
-MigratedLog = #doc{id = NewId, body = Doc#doc.body},
+Log0 = #doc{id = NewId, body = Doc#doc.body},
+Log1 = Log0#doc{revs = {1, [couch_db:new_revid(Log0)]}},
fold_replication_logs(
-Rest, ?REP_ID_VERSION, NewId, NewId, Rep, [MigratedLog | Acc])
+Rest, ?REP_ID_VERSION, NewId, NewId, Rep, [Log1 | Acc])
end.
http://git-wip-us.apache.org/repos/asf/couchdb/blob/eb4138f1/src/couchdb/couch_db.erl
--
diff --git a/src/couchdb/couch_db.erl b/src/couchdb/couch_db.erl
index ae21bfa..a6903c4 100644
--- a/src/couchdb/couch_db.erl
+++ b/src/couchdb/couch_db.erl
@@ -717,16 +717,18 @@ update_docs(Db, Docs, Options, interactive_edit) -
% associate reference with each doc in order to track duplicates
Docs2 = lists:map(fun(Doc) - {Doc, make_ref()} end,Docs),
-{Docs3, NonRepDocs} = lists:foldl(
+{Docs3, NonRepDocs0} = lists:foldl(
fun({#doc{id=Id},_Ref}=Doc, {DocsAcc, NonRepDocsAcc}) -
case Id of
?LOCAL_DOC_PREFIX, _/binary -
-{DocsAcc, [Doc | NonRepDocsAcc]};
+{DocsAcc, [[Doc] | NonRepDocsAcc]};
Id-
{[Doc | DocsAcc], NonRepDocsAcc}
end
end, {[], []}, Docs2),
+{NonRepDocs, _} = new_revs(NonRepDocs0, [], []),
+
DocBuckets = before_docs_update(Db, group_alike_docs(Docs3)),
case (Db#db.validate_doc_funs /= []) orelse
@@ -826,8 +828,9 @@ collect_results(UpdatePid, MRef, ResultsAcc) -
end.
write_and_commit(#db{update_pid=UpdatePid}=Db, DocBuckets1,
-NonRepDocs, Options0) -
+NonRepDocs1, Options0) -
DocBuckets = prepare_doc_summaries(Db, DocBuckets1),
+NonRepDocs = prepare_doc_summaries(Db, NonRepDocs1),
Options = set_commit_option(Options0),
MergeConflicts = lists:member(merge_conflicts, Options),
FullCommit = lists:member(full_commit, Options),
@@ -1182,9 +1185,15 @@ open_doc_revs_int(Db, IdRevs, Options) -
open_doc_int(Db, ?LOCAL_DOC_PREFIX, _/binary = Id, Options) -
case couch_btree:lookup(local_btree(Db), [Id]) of
-
[1/3] git commit: Implement _security as a _local doc
Updated Branches:
refs/heads/new-security-object 76fff259a - e058fff59 (forced update)
Implement _security as a _local doc
As per mailing list discussion on enabling the syncing of _security
objects this builds upon the previous commits to enable the agreed upon
strategy of giving a revision tree to the _security object.
The old API is maintained with identical in the normal single node case.
Though it is also now possible to update the _local/_security doc
directly using the normal HTTP document and _bulk_docs API's to edit the
doc directly.
This document is special cased like _design docs to disallow non-admins
from editing it.
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/e058fff5
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/e058fff5
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/e058fff5
Branch: refs/heads/new-security-object
Commit: e058fff599fb88e029155315072d47fe419984c8
Parents: c6ed253
Author: Paul Joseph Davis dav...@apache.org
Authored: Mon Jan 23 17:59:57 2012 -0600
Committer: Paul Joseph Davis dav...@apache.org
Committed: Wed Jan 25 01:25:06 2012 -0600
--
src/couchdb/couch_db.erl | 22 ++
src/couchdb/couch_db.hrl |1 +
src/couchdb/couch_db_updater.erl | 76 ++---
3 files changed, 65 insertions(+), 34 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/e058fff5/src/couchdb/couch_db.erl
--
diff --git a/src/couchdb/couch_db.erl b/src/couchdb/couch_db.erl
index be65f53..987e8f7 100644
--- a/src/couchdb/couch_db.erl
+++ b/src/couchdb/couch_db.erl
@@ -359,21 +359,23 @@
check_is_member(#db{user_ctx=#user_ctx{name=Name,roles=Roles}=UserCtx}=Db) -
end
end.
-get_admins(#db{security=SecProps}) -
+get_admins(#db{security=Doc}) -
+#doc{body={SecProps}} = Doc,
couch_util:get_value(admins, SecProps, {[]}).
-get_members(#db{security=SecProps}) -
+get_members(#db{security=Doc}) -
+#doc{body={SecProps}} = Doc,
% we fallback to readers here for backwards compatibility
couch_util:get_value(members, SecProps,
couch_util:get_value(readers, SecProps, {[]})).
-get_security(#db{security=SecProps}) -
-{SecProps}.
+get_security(#db{security=SecDoc}) -
+SecDoc#doc.body.
-set_security(#db{update_pid=Pid}=Db, {NewSecProps}) when is_list(NewSecProps)
-
+set_security(#db{security=Doc}=Db, {NewSecProps}) when is_list(NewSecProps) -
check_is_admin(Db),
ok = validate_security_object(NewSecProps),
-ok = gen_server:call(Pid, {set_security, NewSecProps}, infinity),
+{ok, _} = update_doc(Db, Doc#doc{body={NewSecProps}}, []),
{ok, _} = ensure_full_commit(Db),
ok;
set_security(_, _) -
@@ -459,10 +461,12 @@ group_alike_docs([{Doc,Ref}|Rest], [Bucket|RestBuckets])
-
validate_doc_update(#db{}=Db, #doc{id= _design/,_/binary},
_GetDiskDocFun) -
catch check_is_admin(Db);
-validate_doc_update(#db{validate_doc_funs=[]}, _Doc, _GetDiskDocFun) -
-ok;
+validate_doc_update(Db, #doc{id= ?SECURITY_ID}, _GetDiskDocFun) -
+catch check_is_admin(Db);
validate_doc_update(_Db, #doc{id= _local/,_/binary}, _GetDiskDocFun) -
ok;
+validate_doc_update(#db{validate_doc_funs=[]}, _Doc, _GetDiskDocFun) -
+ok;
validate_doc_update(Db, Doc, GetDiskDocFun) -
DiskDoc = GetDiskDocFun(),
JsonCtx = couch_util:json_user_ctx(Db),
@@ -731,7 +735,7 @@ update_docs(Db, Docs, Options, interactive_edit) -
% request. This relies on couch_db_updater not collecting
% more than one update that contains _local docs but this
% is still trigerable with a _bulk_docs request.
-UniqNRIds = lists:usort([Id || #doc{id=Id} - NonRepDocs0]),
+UniqNRIds = lists:usort([Id || [{#doc{id=Id}, _}] - NonRepDocs0]),
case length(UniqNRIds) == length(NonRepDocs0) of
true - ok;
false - throw({update_error, repeated_local_docs})
http://git-wip-us.apache.org/repos/asf/couchdb/blob/e058fff5/src/couchdb/couch_db.hrl
--
diff --git a/src/couchdb/couch_db.hrl b/src/couchdb/couch_db.hrl
index 65eb7f0..26ffe0d 100644
--- a/src/couchdb/couch_db.hrl
+++ b/src/couchdb/couch_db.hrl
@@ -11,6 +11,7 @@
% the License.
-define(LOCAL_DOC_PREFIX, _local/).
+-define(SECURITY_ID, _local/_security).
-define(DESIGN_DOC_PREFIX0, _design).
-define(DESIGN_DOC_PREFIX, _design/).
-define(DEFAULT_COMPRESSION, snappy).
http://git-wip-us.apache.org/repos/asf/couchdb/blob/e058fff5/src/couchdb/couch_db_updater.erl
--
diff --git a/src/couchdb/couch_db_updater.erl b/src/couchdb/couch_db_updater.erl
index 862c48a..42b6460 100644
---
[2/3] git commit: Prevent multiple updates to a single _local doc
Prevent multiple updates to a single _local doc
If a user uploads a two _local docs with the same docid in a _bulk_docs
request the updater will insert multiple entries into the local docs
btree. This patch avoids that by throwing an error if a user attempts
it.
This is an old bug that I caught and just added a fix for. I split it
out as its own patch to point out that its a bug fix that differs from
the old behvior.
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/c6ed2538
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/c6ed2538
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/c6ed2538
Branch: refs/heads/new-security-object
Commit: c6ed2538749697bbbedfbf7ff32afa8a0c940e94
Parents: b96bea4
Author: Paul Joseph Davis dav...@apache.org
Authored: Mon Jan 23 17:56:18 2012 -0600
Committer: Paul Joseph Davis dav...@apache.org
Committed: Wed Jan 25 01:25:06 2012 -0600
--
src/couchdb/couch_db.erl | 10 ++
1 files changed, 10 insertions(+), 0 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/c6ed2538/src/couchdb/couch_db.erl
--
diff --git a/src/couchdb/couch_db.erl b/src/couchdb/couch_db.erl
index a6903c4..be65f53 100644
--- a/src/couchdb/couch_db.erl
+++ b/src/couchdb/couch_db.erl
@@ -727,6 +727,16 @@ update_docs(Db, Docs, Options, interactive_edit) -
end
end, {[], []}, Docs2),
+% Prevent updating multiple _local docs in a single update
+% request. This relies on couch_db_updater not collecting
+% more than one update that contains _local docs but this
+% is still trigerable with a _bulk_docs request.
+UniqNRIds = lists:usort([Id || #doc{id=Id} - NonRepDocs0]),
+case length(UniqNRIds) == length(NonRepDocs0) of
+true - ok;
+false - throw({update_error, repeated_local_docs})
+end,
+
{NonRepDocs, _} = new_revs(NonRepDocs0, [], []),
DocBuckets = before_docs_update(Db, group_alike_docs(Docs3)),
[3/3] git commit: Update _local docs to use revision trees
Update _local docs to use revision trees
There are a number of cases that _local docs might need to be merged
between nodes. One motivating case is for clusters that might wish to
move _local docs across nodes to maintain replication checkpoints with
external CouchDB instances. The previous _local docs strategy of a
single linear sequence breaks down in this situation.
This new behavior should be indistinguishable from the previous behavior
assuming clients did not try and introspect the _rev value for _local
documents. It should be impossible for normal HTTP clients to introduce
different behavior than previously existed because there's no support
for non-linear updates at the HTTP level. This update is merely and
internal refactoring for special cases like clusters.
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/b96bea42
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/b96bea42
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/b96bea42
Branch: refs/heads/new-security-object
Commit: b96bea429a38655cb3efeb07f30f5925416a6524
Parents: 0ab5ebd
Author: Paul Joseph Davis dav...@apache.org
Authored: Thu Jan 19 18:10:05 2012 -0600
Committer: Paul Joseph Davis dav...@apache.org
Committed: Wed Jan 25 01:24:48 2012 -0600
--
src/couch_replicator/src/couch_replicator.erl |9 +-
src/couchdb/couch_db.erl | 21 +++-
src/couchdb/couch_db_updater.erl | 116 ++--
3 files changed, 81 insertions(+), 65 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/b96bea42/src/couch_replicator/src/couch_replicator.erl
--
diff --git a/src/couch_replicator/src/couch_replicator.erl
b/src/couch_replicator/src/couch_replicator.erl
index 1f7c08a..53acfd5 100644
--- a/src/couch_replicator/src/couch_replicator.erl
+++ b/src/couch_replicator/src/couch_replicator.erl
@@ -580,15 +580,18 @@ fold_replication_logs([Db | Rest] = Dbs, Vsn, LogId,
NewId, Rep, Acc) -
fold_replication_logs(Dbs, Vsn - 1,
?l2b(?LOCAL_DOC_PREFIX ++ OldRepId), NewId, Rep, Acc);
{error, not_found} -
+Doc0 = #doc{id = NewId},
+Doc1 = Doc0#doc{revs = {1, [couch_db:new_revid(Doc0)]}},
fold_replication_logs(
-Rest, ?REP_ID_VERSION, NewId, NewId, Rep, [#doc{id = NewId} |
Acc]);
+Rest, ?REP_ID_VERSION, NewId, NewId, Rep, [Doc1 | Acc]);
{ok, Doc} when LogId =:= NewId -
fold_replication_logs(
Rest, ?REP_ID_VERSION, NewId, NewId, Rep, [Doc | Acc]);
{ok, Doc} -
-MigratedLog = #doc{id = NewId, body = Doc#doc.body},
+Log0 = #doc{id = NewId, body = Doc#doc.body},
+Log1 = Log0#doc{revs = {1, [couch_db:new_revid(Log0)]}},
fold_replication_logs(
-Rest, ?REP_ID_VERSION, NewId, NewId, Rep, [MigratedLog | Acc])
+Rest, ?REP_ID_VERSION, NewId, NewId, Rep, [Log1 | Acc])
end.
http://git-wip-us.apache.org/repos/asf/couchdb/blob/b96bea42/src/couchdb/couch_db.erl
--
diff --git a/src/couchdb/couch_db.erl b/src/couchdb/couch_db.erl
index ae21bfa..a6903c4 100644
--- a/src/couchdb/couch_db.erl
+++ b/src/couchdb/couch_db.erl
@@ -717,16 +717,18 @@ update_docs(Db, Docs, Options, interactive_edit) -
% associate reference with each doc in order to track duplicates
Docs2 = lists:map(fun(Doc) - {Doc, make_ref()} end,Docs),
-{Docs3, NonRepDocs} = lists:foldl(
+{Docs3, NonRepDocs0} = lists:foldl(
fun({#doc{id=Id},_Ref}=Doc, {DocsAcc, NonRepDocsAcc}) -
case Id of
?LOCAL_DOC_PREFIX, _/binary -
-{DocsAcc, [Doc | NonRepDocsAcc]};
+{DocsAcc, [[Doc] | NonRepDocsAcc]};
Id-
{[Doc | DocsAcc], NonRepDocsAcc}
end
end, {[], []}, Docs2),
+{NonRepDocs, _} = new_revs(NonRepDocs0, [], []),
+
DocBuckets = before_docs_update(Db, group_alike_docs(Docs3)),
case (Db#db.validate_doc_funs /= []) orelse
@@ -826,8 +828,9 @@ collect_results(UpdatePid, MRef, ResultsAcc) -
end.
write_and_commit(#db{update_pid=UpdatePid}=Db, DocBuckets1,
-NonRepDocs, Options0) -
+NonRepDocs1, Options0) -
DocBuckets = prepare_doc_summaries(Db, DocBuckets1),
+NonRepDocs = prepare_doc_summaries(Db, NonRepDocs1),
Options = set_commit_option(Options0),
MergeConflicts = lists:member(merge_conflicts, Options),
FullCommit = lists:member(full_commit, Options),
@@ -1182,9 +1185,15 @@ open_doc_revs_int(Db, IdRevs, Options) -
open_doc_int(Db, ?LOCAL_DOC_PREFIX, _/binary = Id, Options) -
case couch_btree:lookup(local_btree(Db), [Id]) of
-[{ok,
