[jira] [Commented] (COUCHDB-1230) Replication slows down over time
[ https://issues.apache.org/jira/browse/COUCHDB-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13069505#comment-13069505 ] Paul Hirst commented on COUCHDB-1230: - It's quite likely I could borrow some hardware in the next couple of weeks to try this out. I'll see what I can do. I don't have anywhere immediately to test this. Thanks. Replication slows down over time Key: COUCHDB-1230 URL: https://issues.apache.org/jira/browse/COUCHDB-1230 Project: CouchDB Issue Type: Bug Components: Replication Affects Versions: 1.0.2, 1.1 Environment: Ubuntu 10.04, Reporter: Paul Hirst Attachments: sequence_number.png I have two databases which were replicated in the past, one is running 1.0.2. I shall call this the source database. The other is running 1.1.0, I shall call this the target database. The source and target are bidirectionally replicated using a push and pull replication from the target (using a couple of documents in the new _replicator database). The source database is in production and is getting changes applied to it from live systems. The target is only participating in replication and isn't being used directly by any production systems. The database has about 50 million documents many of these will have been updated a handful of times. The database is about 500G after compaction, but the source database is currently at about 900G as it hasn't been compacted for a while. The databases were replicated in the past however this replication was torn down when the target was upgraded from 1.0.2 to 1.1.0. When replication was reenabled the system wasn't able to pick up were it left off and had to reenumerate all the documents again. This process initially started quickly but after a while ground to a halt such that the target actually stopped making progress against the source database. I found that restarting replication starts the process running again at a decent speed for a while. I did this by deleting and recreating the appropriate document in the _replicator database on the target. I have graphed the last_seq of the target database against time for about a day, noting when replication was manually restarted. I shall try to attach the graph if possible. It shows a clear improvement in replication speed after restarting replication. I previously witnessed this behaviour between 1.0.2 databases but didn't grab any stats at the time but I don't think it's a new problem. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
create_target isn't working with the _replicator db
Hi all,
While testing the _replicator db I encounter an error when trying to
create the target db . Before opening a ticket I wanted to make sure
this wasn't expected or if its an error.
The error:
$ curl -XPUT -HContent-Type: application/json localhost:5984/a
{ok:true}
$ curl -XPUT -d'{}' localhost:5984/a/doca
{ok:true,id:doca,rev:1-967a00dff5e02add41819138abb3284d}
curl -XPUT -d'{}' localhost:5984/a/docb
{ok:true,id:docb,rev:1-967a00dff5e02add41819138abb3284d}
$ curl -XPOST -d'{
source: a,
target: b,
create_target: true
}' -HContent-Type: application/json localhost:5984/_replicator
{ok:true,id:2f6df84e4a02978cf80424bcef1c,rev:1-4b8c482da7ce243353493fb87e766428}
enki:~ benoitc$ curl -XGET
localhost:5984/_replicator/2f6df84e4a02978cf80424bcef1c
{_id:2f6df84e4a02978cf80424bcef1c,_rev:2-b557bd698ecebc359bd5f51b2691a2fd,source:a,target:b,create_target:true,_replication_state:error,_replication_state_time:2011-07-22T13:49:21+02:00,_replication_id:01e205795e3f5ad9968f51e6e1e61335}
enki:~ benoitc$
Error:
unauthorized to access database b}.
Retrying in 10 seconds
[error] [0.867.0] {error_report,0.30.0,
{0.867.0,crash_report,
[[{initial_call,{couch_rep,init,['Argument__1']}},
{pid,0.867.0},
{registered_name,[]},
{error_info,
{exit,
{unauthorized,b},
[{gen_server,init_it,6},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_rep_sup,couch_primary_services,
couch_server_sup,0.31.0]},
{messages,[]},
{links,[0.81.0]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,610},
{stack_size,24},
{reductions,199}],
[]]}}
Let me know.
- benoît
[jira] [Updated] (COUCHDB-1202) Function 'require' undefined in reduce functions, cannot require CommonJS module
[
https://issues.apache.org/jira/browse/COUCHDB-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dipesh Patel updated COUCHDB-1202:
--
Attachment: reduce_lib.patch
Hello, I've attempted a patch for this. Please find attached.
You can also find it on github here:
https://github.com/dipthegeezer/couchdb/tree/1202-require-undefined-in-reduce
Let me know if there are any issues with it.
Dip
Function 'require' undefined in reduce functions, cannot require CommonJS
module
Key: COUCHDB-1202
URL: https://issues.apache.org/jira/browse/COUCHDB-1202
Project: CouchDB
Issue Type: Bug
Components: JavaScript View Server
Affects Versions: 1.1
Environment: Mac OS X 10.6.8, CouchDB 1.1.0 (homebrew), Erlang R14B03
Reporter: Gabor Ratky
Priority: Minor
Labels: commonjs, reduce, views
Attachments: reduce_lib.patch
CouchDB 1.1.0 views can require CommonJS modules according to the Modules
1.1.1 specification when defined in the design document under views/lib. In
CouchDB 1.1.0 this works in map functions, but the require is undefined in
reduce functions. See the shell transcript along with the design document and
log output. Also available at [1].
ddoc.json:
{
views: {
lib: {
foo: exports.bar = 42;
},
test: {
map: function(doc) { emit(doc._id, require('views/lib/foo').bar);
},
reduce: function (keys, values, rereduce) { return
require('views/lib/foo').bar; }
}
}
}
require_test.sh:
$ curl http://localhost:5984
{couchdb:Welcome,version:1.1.0}
$ curl -X PUT http://localhost:5984/reduce_test
{ok:true}
$ curl -X PUT http://localhost:5984/reduce_test/_design/test --data @ddoc.json
{ok:true,id:_design/test,rev:1-5a587aeb82027657db334265037a4692}
$ curl -X POST http://localhost:5984/reduce_test -H Content-Type:
application/json -d '{}'
{ok:true,id:cd5869297d813d45d3e9f8016f001834,rev:1-967a00dff5e02add41819138abb3284d}
$ curl
http://localhost:5984/reduce_test/_design/test/_view/test?reduce=false;
{total_rows:1,offset:0,rows:[
{id:cd5869297d813d45d3e9f8016f001834,key:cd5869297d813d45d3e9f8016f001834,value:42}
]}
$ curl http://localhost:5984/reduce_test/_design/test/_view/test;
{rows:[
{key:null,value:null}
]}
^^^ Expected {key:null,value:42}
couch.log:
[info] [0.313.0] OS Process #Port0.2368 Log :: function raised exception
(new ReferenceError(require is not defined, ))
[info] [0.14096.0] 127.0.0.1 - - 'GET' /reduce_test/_design/test/_view/test
200
[1] https://gist.github.com/1049231
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: create_target isn't working with the _replicator db
If you have credentials and roles defined on the target DB, you need
to set the the user_ctx in the _replicator doc. Something like this: {
source ..., user_ctx: { roles: [ admin ] } }.
K.
---
http://blitz.io
@pcapr
On Fri, Jul 22, 2011 at 4:56 AM, Benoit Chesneau bchesn...@gmail.com wrote:
Hi all,
While testing the _replicator db I encounter an error when trying to
create the target db . Before opening a ticket I wanted to make sure
this wasn't expected or if its an error.
The error:
$ curl -XPUT -HContent-Type: application/json localhost:5984/a
{ok:true}
$ curl -XPUT -d'{}' localhost:5984/a/doca
{ok:true,id:doca,rev:1-967a00dff5e02add41819138abb3284d}
curl -XPUT -d'{}' localhost:5984/a/docb
{ok:true,id:docb,rev:1-967a00dff5e02add41819138abb3284d}
$ curl -XPOST -d'{
source: a,
target: b,
create_target: true
}' -HContent-Type: application/json localhost:5984/_replicator
{ok:true,id:2f6df84e4a02978cf80424bcef1c,rev:1-4b8c482da7ce243353493fb87e766428}
enki:~ benoitc$ curl -XGET
localhost:5984/_replicator/2f6df84e4a02978cf80424bcef1c
{_id:2f6df84e4a02978cf80424bcef1c,_rev:2-b557bd698ecebc359bd5f51b2691a2fd,source:a,target:b,create_target:true,_replication_state:error,_replication_state_time:2011-07-22T13:49:21+02:00,_replication_id:01e205795e3f5ad9968f51e6e1e61335}
enki:~ benoitc$
Error:
unauthorized to access database b}.
Retrying in 10 seconds
[error] [0.867.0] {error_report,0.30.0,
{0.867.0,crash_report,
[[{initial_call,{couch_rep,init,['Argument__1']}},
{pid,0.867.0},
{registered_name,[]},
{error_info,
{exit,
{unauthorized,b},
[{gen_server,init_it,6},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_rep_sup,couch_primary_services,
couch_server_sup,0.31.0]},
{messages,[]},
{links,[0.81.0]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,610},
{stack_size,24},
{reductions,199}],
[]]}}
Let me know.
- benoît
Re: create_target isn't working with the _replicator db
On Fri, Jul 22, 2011 at 4:59 PM, kowsik kow...@gmail.com wrote:
If you have credentials and roles defined on the target DB, you need
to set the the user_ctx in the _replicator doc. Something like this: {
source ..., user_ctx: { roles: [ admin ] } }.
K.
---
http://blitz.io
@pcapr
Yup, but I think that's a bug then. I shouldn't have to set any
userctx imo. If no admin has been set, every user is an admin except
if we change the default behavior and then it's not consistent.
- benoit
- benoît
Re: create_target isn't working with the _replicator db
On Fri, Jul 22, 2011 at 8:43 AM, Benoit Chesneau bchesn...@gmail.com wrote: Yup, but I think that's a bug then. I shouldn't have to set any userctx imo. If no admin has been set, every user is an admin except if we change the default behavior and then it's not consistent. This was discussed sometime before the 1.1.0 release in the security list. And it's a principle of the least privileges by default (roles is an empty list). - benoit - benoît -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: create_target isn't working with the _replicator db
On Fri, Jul 22, 2011 at 5:54 PM, Filipe David Manana fdman...@apache.org wrote: On Fri, Jul 22, 2011 at 8:43 AM, Benoit Chesneau bchesn...@gmail.com wrote: Yup, but I think that's a bug then. I shouldn't have to set any userctx imo. If no admin has been set, every user is an admin except if we change the default behavior and then it's not consistent. This was discussed sometime before the 1.1.0 release in the security list. And it's a principle of the least privileges by default (roles is an empty list). I've no problem with that, it's even good. But other part of the API aren't consistent then. While _replicator is ok, I can still do this operation on _replicate. I propose to port the same behavior _replicate.OK for that? - benoît
Re: create_target isn't working with the _replicator db
On Fri, Jul 22, 2011 at 09:40, Benoit Chesneau bchesn...@gmail.com wrote: On Fri, Jul 22, 2011 at 5:54 PM, Filipe David Manana fdman...@apache.org wrote: On Fri, Jul 22, 2011 at 8:43 AM, Benoit Chesneau bchesn...@gmail.com wrote: Yup, but I think that's a bug then. I shouldn't have to set any userctx imo. If no admin has been set, every user is an admin except if we change the default behavior and then it's not consistent. This was discussed sometime before the 1.1.0 release in the security list. And it's a principle of the least privileges by default (roles is an empty list). I've no problem with that, it's even good. But other part of the API aren't consistent then. While _replicator is ok, I can still do this operation on _replicate. I propose to port the same behavior _replicate.OK for that? I'd definitely prefer they be consistent. In fact, I've been arguing quietly for POST to _replicator to be exactly the _replicate API and to deprecate the latter. Isn't this possible?
Re: create_target isn't working with the _replicator db
On Fri, Jul 22, 2011 at 11:26, Randall Leeds randall.le...@gmail.com wrote:
On Fri, Jul 22, 2011 at 09:40, Benoit Chesneau bchesn...@gmail.com wrote:
On Fri, Jul 22, 2011 at 5:54 PM, Filipe David Manana
fdman...@apache.org wrote:
On Fri, Jul 22, 2011 at 8:43 AM, Benoit Chesneau bchesn...@gmail.com
wrote:
Yup, but I think that's a bug then. I shouldn't have to set any
userctx imo. If no admin has been set, every user is an admin except
if we change the default behavior and then it's not consistent.
This was discussed sometime before the 1.1.0 release in the security list.
And it's a principle of the least privileges by default (roles is an
empty list).
I've no problem with that, it's even good. But other part of the API
aren't consistent then. While _replicator is ok, I can still do this
operation on _replicate. I propose to port the same behavior
_replicate.OK for that?
I'd definitely prefer they be consistent.
In fact, I've been arguing quietly for POST to _replicator to be
exactly the _replicate API and to deprecate the latter.
Isn't this possible?
What I mean to say is that I think it's absolute craziness to have two
replication APIs.
{persist: true} or something would have made more sense to me.
?block=true perhaps if we want the old one-shot, blocking. Persist on
PUT or with {id: replication_name}, otherwise make it one-shot.
I hate to be complaining like this after we've already released it
with a different API, but I raised this a few times before 1.1 went
out. I think two APIs for replication is ugly and confusing. A new
CouchDB user has enough to digest without having to remember that
_replicate is different from _replicator. I would have preferred we
papered over the differences as described above and made _replicate
use a database, rather than create a brand new path.
/rant
Scheduled Compaction
Hi all, Just working on a system that is generating a considerable number of revisions, and would like to know whether the list has suggestions on best practice for regular compaction. Thanks Vish
