GitHub user sagelywizard opened a pull request:
https://github.com/apache/couchdb-couch-mrview/pull/2
Add view changes functionality
This PR adds functionality for a new _changes-like feed for views. It also
adds some optimizations for _changes feed filtering by views.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sagelywizard/couchdb-couch-mrview
view-changes-windsor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/couchdb-couch-mrview/pull/2.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2
----
commit 78c58e114a7d11caed33b9f2d16cec1c2884f5f0
Author: benoitc <[email protected]>
Date: 2014-01-26T22:56:09Z
add couch_mrview:view_changes_since/{6,7} function
This function add the possibility to get changes in a view since the
last upddated sequence. You can all changes in a view since a sequence
or all changes for a key or a range in a view.
The following new secondaries are created to allows this feature:
- a generic log index to log the latest changes in views for a docid :
{DocId, [{ViewId, {Key, Seq, OP}}]} where OP can be del or add. This
index allows us to mark a key as removed if needed. It will be useful
later to help us to chain map/reduces operations or such things.
- a seq index associated to a view id : {ViewId, [{Seq, Key}, {DocId,
Val}]} to look for all changes in a view
- an index indexing keys by seq: {ViewId, [{[Key, Seq], DocId}, Val}]},
to looks for changes associated to a key or a ranhge
Note: all deleted keys are marked as deleted in the log index and their
value is {[{<<"_removed">>, true}]}.
To start to index changes you need to pass the options {seq_indexed:
true} to the design document.
Caveat: when the changes are indexed the size of the index is significantly
higher.
Example of usage: https://www.friendpaste.com/5Y6gihQReaxd8ERqbDom3y
Conflicts:
src/couch_mrview_updater.erl
src/couch_mrview_util.erl
commit 6947da6985525b625b5e4ae5773cd4088462c3b9
Author: benoitc <[email protected]>
Date: 2014-01-27T09:14:59Z
couch_mrview: fix purge when seq_indexed=true
This change makes tsure to also purge the log btree and related btrees
in views when seqs_indexed is set to true.
commit 89e79cf142657a3112639339a6783fcd7650a217
Author: benoitc <[email protected]>
Date: 2014-01-27T09:24:25Z
couch_mrview: fix changes log
This changes make sure we pass correct keys to remove to the seq and
key_byseq btrees in views.
commit e43f7c5c52d28bc8c96fa990de63feea89fb35f6
Author: benoitc <[email protected]>
Date: 2014-01-27T10:51:11Z
couch_mrview: add couch_mrview:count_view_changes_since/{4,5}
Add function to couch changes in a view index, similar to
couch_db:count_changes_since/2 .
Conflicts:
src/couch_mrview.erl
commit 9d4651fb86021092c623409ea1fdd1bc07a792b4
Author: benoitc <[email protected]>
Date: 2014-01-27T16:24:54Z
couch_mrview: add view changes test
test view changes and fix errors.
commit a3e4a71b2fb8435216b7256677ec2690aa52cf42
Author: benoitc <[email protected]>
Date: 2014-01-27T20:03:28Z
couch_mrview: check removed keys from the index.
commit 19fe652c93a8cd30563662930d79808c12d66ac2
Author: benoitc <[email protected]>
Date: 2014-01-28T22:23:37Z
couch_mrview: add couch_mrview:refresh/2 function
function to refresh a view index.
Conflicts:
src/couch_mrview.erl
commit 477eec24253ccfbd94c55338cca8a0f510c0118c
Author: benoitc <[email protected]>
Date: 2014-01-30T08:21:06Z
add index update events notifications
Conflicts:
src/couch_mrview_updater.erl
commit 53efc1e4600239cfe93ae400a5c7851531d8bd3d
Author: benoitc <[email protected]>
Date: 2014-01-31T12:13:23Z
couch_mrview: couch_mrview_changes:handle_changes
Similar to couch_changes:handle_changes but for view changes. It add
support for longpolling, normal and continuous stream The API differs
from the one for doc by beeing independant from the transport: the
support of HTTP will be added on top for example.
This API will be also used to replace the view filter in the current
_changes
API.
Also add unittests.
commit 7e70678f571cb5664bb03728be4b864f86979ad1
Author: benoitc <[email protected]>
Date: 2014-02-07T14:38:34Z
add supports of view changes in the _changes API
Now when the option `seq_indexed=true` is set in the design doc, the
view filter in _changes will use it to retrieve the results. Compared to
the current way, using a view index will be faster to retrieve changes.
It also gives the possibility to filter changes by key or get changes in
a key range. All the view options can be used.
Note 1: if someone is trying to filter a changes with view options when
the views are not indexed by sequence, a 400 error will be returned.
Note 2: The changes will only be returned when the view is updated if
seq_indexed=true
Conflicts:
src/couch_mrview_http.erl
commit 2a79690b6365f4829c102ce2f1960e1b642bb7de
Author: benoitc <[email protected]>
Date: 2014-02-08T18:55:40Z
couch_index: add background indexing facility
This change add the possibility to trigger a view indexation in
background. The indexation can only work in background if at least one
process acquired it using the `couch_index_server:acquire_index/3`
function. If all the process that acquired it are down or released it
using `couch_index_server:release_indexer/3` then the background task is
stopped.
By default the background indexation will happen every 1s or when 200
docs has been saved in the database. These parameters can be changed
using the options `threshold` and `refresh_interval` in the couch_index
section.
To use it with couch_mrview a new option {refresh, true} has been added
to couch_mrview_changes:handle_changes Also the query parameter
refresh=true is passsed in t the HTTP changes API.
commit c59af49c709a9fea0106f05ac2d19865e3b7ea2e
Author: benoitc <[email protected]>
Date: 2014-02-08T19:20:47Z
couch_mrview: fix 10-index-changes.t
proper test to check the indexer.
commit 35105aa7d506f89cf28312dc93476d0adb346b49
Author: benoitc <[email protected]>
Date: 2014-02-08T21:49:42Z
couch_mrview: add API to retrieve view info.
Add couch_mrview:get_view_info/3 to retrieve the internal informations
of a view like the last update seq in this view or the number of rows.
Conflicts:
src/couch_mrview.erl
test/04-index-info.t
commit bd67f905ed03f211c2dd62245f15518cae7b1469
Author: benoitc <[email protected]>
Date: 2014-02-08T22:15:28Z
couch_mrview: add HTTP handler to retrieve the infos on a view.
commit d03d64e125c249d1f8be336adced1bf218bda69d
Author: benoitc <[email protected]>
Date: 2014-02-08T23:43:23Z
couch_replicator: add replication using changes in a view
Instead of a database, the replicator can now filter the documents using
a view index. All documents having a key emitted in the view can be
replicated.
View parameters can be used. Which means that you can replicate results
corresponding to a key in a view or a range.
commit b1c8e8100e87bedee7335d8083007ba095deaa87
Author: benoitc <[email protected]>
Date: 2014-02-13T21:59:47Z
use ERL_FLAGS instead of a shebang to launch tests
Conflicts:
test/03-red-views.t
commit e57fbc28ac418774857d53f3dfdc7e7315d8f69c
Author: benoitc <[email protected]>
Date: 2014-02-27T20:55:46Z
fix case clause
commit 1b3978d66ddfd04298ddfe4683e8a14e526643d5
Author: benoitc <[email protected]>
Date: 2014-03-04T12:38:55Z
add /<db>/_design/<ddocname>/_reindex handler
When you 'POST' on the _reindex handle all the views of the group will
be reindexed.
close #RCOUCH-36
commit 687aa3ee71cf2297b12f4f4381f2dbf931fa409f
Author: benoitc <[email protected]>
Date: 2014-01-27T14:29:10Z
couch_mrview: fix compaction with seqs_indexed=true
This change smakes sure to compact the view indexes when using the view
changes.
Conflicts:
src/couch_mrview_compactor.erl
commit 79e276af14bc28b38205b8e3a611737c3a143604
Author: benoitc <[email protected]>
Date: 2014-02-02T18:54:01Z
extract couch_httpd changes API in its own module
Conflicts:
src/couch_mrview_updater.erl
commit 9a4e79ac8853f2617995e899420513375623bd71
Author: Benjamin Bastian <[email protected]>
Date: 2014-08-22T08:52:31Z
Add preliminary version of view changes
commit 1d306c0aa4679a436a4edb6f613f145d3a111283
Author: Benjamin Bastian <[email protected]>
Date: 2014-08-22T10:46:13Z
Make feed=continuous work for view changes
commit 218264dc107ef3f704d62be7ff0a804362faf352
Author: Benjamin Bastian <[email protected]>
Date: 2014-08-22T13:28:44Z
Throw 400 on _view_changes when by-seq indexing is disabled
commit c7726c82a30d7f2acccc5a22a984e8ea4e292407
Author: Benjamin Bastian <[email protected]>
Date: 2014-08-22T16:57:48Z
Add view filtering optimization to changes feeds. Remove unnecessary old
stuff.
commit ea7cf81dc6603985d220d8cfe4d91da9d53f9778
Author: Benjamin Bastian <[email protected]>
Date: 2014-08-25T09:49:07Z
Add rev to view changes respose
commit 45485bf0d9ede07e51841e281be96573ffa65bc8
Author: Benjamin Bastian <[email protected]>
Date: 2014-08-26T08:28:56Z
Make they key-by-key-and-seq view btree optional independently of the
key-by-seq view btree
commit 9ba439b8da3ca314111d46608cfe89fa74ef36b2
Author: Benjamin Bastian <[email protected]>
Date: 2014-08-27T17:02:48Z
Change function names to match changes in couchdb-couch
commit 4c2a977abf72219fd3289ce52e5711214dbb50b4
Author: Benjamin Bastian <[email protected]>
Date: 2014-08-28T08:29:22Z
Clean up update_log function for readability
commit 9c50a94e8a92d5dc1c829c61532824d9a946388b
Author: Benjamin Bastian <[email protected]>
Date: 2014-08-28T14:22:55Z
Fix removal of keys from view seq index
commit 5a365913986fd5bcab0efcc7e5b6367ae39e2e3f
Author: Benjamin Bastian <[email protected]>
Date: 2014-08-28T16:40:59Z
Fix stuff which I broke in a rebase. I'll fix the git history later
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
