Re: git commit: COUCHDB-1424 Fix etap to not consume any message
On Wed, Oct 31, 2012 at 4:32 PM, Paul Davis paul.joseph.da...@gmail.com wrote:
Nice find
Say thanks to Damien :)
Fixed in an older etap version, I just ported it to latest etap.
Robert (Dionne), wanna test this to see if it fixes the hangs you used
to have with OTP R15Bx?
thanks
On Wed, Oct 31, 2012 at 11:29 AM, fdman...@apache.org wrote:
Updated Branches:
refs/heads/1424-fix-etap-consuming-any-test-message [created] 67d531b02
COUCHDB-1424 Fix etap to not consume any message
Turns out that etap consumes any message in the mailbox in
some cases. This can make some tests that use message passing
hang, as etap itself consumes the messages.
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/67d531b0
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/67d531b0
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/67d531b0
Branch: refs/heads/1424-fix-etap-consuming-any-test-message
Commit: 67d531b028503b316c9ff8842b8fb34ea75db29d
Parents: 8ccf696
Author: Filipe David Borba Manana fdman...@apache.org
Authored: Wed Oct 31 16:19:48 2012 +0100
Committer: Filipe David Borba Manana fdman...@apache.org
Committed: Wed Oct 31 16:19:48 2012 +0100
--
src/etap/etap.erl | 21 +++--
1 files changed, 11 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/67d531b0/src/etap/etap.erl
--
diff --git a/src/etap/etap.erl b/src/etap/etap.erl
index 7380013..ae3896c 100644
--- a/src/etap/etap.erl
+++ b/src/etap/etap.erl
@@ -120,14 +120,14 @@ plan(N) when is_integer(N), N 0 -
%% @doc End the current test plan and output test results.
%% @todo This should probably be done in the test_server process.
end_tests() -
-timer:sleep(100),
+Ref = make_ref(),
case whereis(etap_server) of
-undefined - self() ! true;
-_ - etap_server ! {self(), state}
+undefined - self() ! {Ref, true};
+_ - etap_server ! {self(), state, Ref}
end,
-State = receive X - X end,
+State = receive {Ref, X} - X end,
if
-State#test_state.planned == -1 -
+is_record(State, test_state) andalso State#test_state.planned == -1
-
io:format(1..~p~n, [State#test_state.count]);
true -
ok
@@ -564,8 +564,8 @@ test_server(State) -
count = State#test_state.count + 1,
fail = State#test_state.fail + 1
};
-{From, state} -
-From ! State,
+{From, state, Ref} -
+From ! {Ref, State},
State;
{_From, diag, Message} -
io:format(~s~n, [Message]),
@@ -573,8 +573,8 @@ test_server(State) -
{From, count} -
From ! State#test_state.count,
State;
-{From, is_skip} -
-From ! State#test_state.skip,
+{From, is_skip, Ref} -
+From ! {Ref, State#test_state.skip},
State;
done -
exit(normal)
@@ -584,7 +584,8 @@ test_server(State) -
%% @private
%% @doc Process the result of a test and send it to the etap_server process.
mk_tap(Result, Desc) -
-IsSkip = lib:sendw(etap_server, is_skip),
+etap_server ! {self(), is_skip, Ref = make_ref()} ,
+receive {Ref, IsSkip} - ok end,
case [IsSkip, Result] of
[_, true] -
etap_server ! {self(), pass, Desc},
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: git commit: COUCHDB-1424 Fix etap to not consume any message
On Wed, Oct 31, 2012 at 8:01 PM, Bob Dionne
dio...@dionne-associates.com wrote:
Hi Filipe,
I tested this new branch and it seems to have issues (at least on my machine)
:
ok 44 reduce_false
ok 45 reduce_false_temp
not ok 46 replication
Reason: expected 'tony', got 'null'
Trace back (most recent call first):
0:
Error(expected '\tony\', got 'null')
46: /Users/bitdiddle/emacs/foo/couchdb/test/javascript/cli_runner.js
T(false,expected '\tony\', got 'null',undefined)
324: /Users/bitdiddle/emacs/foo/couchdb/share/www/script/couch_test_runner.js
TEquals(tony,null)
I also retested the original patch with and without the sleep(100) and
without still fails, though it's move around a bit in the etaps, so this
could be other issues.
Bob, that failure is irrelevant I think. It's a js test, I only
modified etap.erl, therefore only etap tests matter.
thanks for testing
Cheers,
Bob
On Oct 31, 2012, at 11:35 AM, Filipe David Manana fdman...@gmail.com wrote:
On Wed, Oct 31, 2012 at 4:32 PM, Paul Davis paul.joseph.da...@gmail.com
wrote:
Nice find
Say thanks to Damien :)
Fixed in an older etap version, I just ported it to latest etap.
Robert (Dionne), wanna test this to see if it fixes the hangs you used
to have with OTP R15Bx?
thanks
On Wed, Oct 31, 2012 at 11:29 AM, fdman...@apache.org wrote:
Updated Branches:
refs/heads/1424-fix-etap-consuming-any-test-message [created] 67d531b02
COUCHDB-1424 Fix etap to not consume any message
Turns out that etap consumes any message in the mailbox in
some cases. This can make some tests that use message passing
hang, as etap itself consumes the messages.
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/67d531b0
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/67d531b0
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/67d531b0
Branch: refs/heads/1424-fix-etap-consuming-any-test-message
Commit: 67d531b028503b316c9ff8842b8fb34ea75db29d
Parents: 8ccf696
Author: Filipe David Borba Manana fdman...@apache.org
Authored: Wed Oct 31 16:19:48 2012 +0100
Committer: Filipe David Borba Manana fdman...@apache.org
Committed: Wed Oct 31 16:19:48 2012 +0100
--
src/etap/etap.erl | 21 +++--
1 files changed, 11 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/67d531b0/src/etap/etap.erl
--
diff --git a/src/etap/etap.erl b/src/etap/etap.erl
index 7380013..ae3896c 100644
--- a/src/etap/etap.erl
+++ b/src/etap/etap.erl
@@ -120,14 +120,14 @@ plan(N) when is_integer(N), N 0 -
%% @doc End the current test plan and output test results.
%% @todo This should probably be done in the test_server process.
end_tests() -
-timer:sleep(100),
+Ref = make_ref(),
case whereis(etap_server) of
-undefined - self() ! true;
-_ - etap_server ! {self(), state}
+undefined - self() ! {Ref, true};
+_ - etap_server ! {self(), state, Ref}
end,
-State = receive X - X end,
+State = receive {Ref, X} - X end,
if
-State#test_state.planned == -1 -
+is_record(State, test_state) andalso State#test_state.planned ==
-1 -
io:format(1..~p~n, [State#test_state.count]);
true -
ok
@@ -564,8 +564,8 @@ test_server(State) -
count = State#test_state.count + 1,
fail = State#test_state.fail + 1
};
-{From, state} -
-From ! State,
+{From, state, Ref} -
+From ! {Ref, State},
State;
{_From, diag, Message} -
io:format(~s~n, [Message]),
@@ -573,8 +573,8 @@ test_server(State) -
{From, count} -
From ! State#test_state.count,
State;
-{From, is_skip} -
-From ! State#test_state.skip,
+{From, is_skip, Ref} -
+From ! {Ref, State#test_state.skip},
State;
done -
exit(normal)
@@ -584,7 +584,8 @@ test_server(State) -
%% @private
%% @doc Process the result of a test and send it to the etap_server
process.
mk_tap(Result, Desc) -
-IsSkip = lib:sendw(etap_server, is_skip),
+etap_server ! {self(), is_skip, Ref = make_ref()} ,
+receive {Ref, IsSkip} - ok end,
case [IsSkip, Result] of
[_, true] -
etap_server ! {self(), pass, Desc},
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world
Re: [2/8] git commit: Fixing the replicator_db JS test
On Thu, Apr 26, 2012 at 12:26 PM, Benoit Chesneau bchesn...@gmail.com wrote:
On Thu, Apr 26, 2012 at 12:18 AM, dav...@apache.org wrote:
Fixing the replicator_db JS test
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/6c976bd9
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/6c976bd9
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/6c976bd9
Branch: refs/heads/master
Commit: 6c976bd948565305c006746f449dc8cd21a749e1
Parents: 7d2fe95
Author: Paul Joseph Davis dav...@apache.org
Authored: Mon Mar 26 15:17:02 2012 -0500
Committer: Paul Joseph Davis dav...@apache.org
Committed: Wed Apr 25 16:54:08 2012 -0500
--
share/www/script/test/replicator_db.js | 4
src/couch_replicator/src/couch_replicator.erl | 5 -
2 files changed, 8 insertions(+), 1 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/6c976bd9/share/www/script/test/replicator_db.js
--
diff --git a/share/www/script/test/replicator_db.js
b/share/www/script/test/replicator_db.js
index 48ca341..edc85f4 100644
--- a/share/www/script/test/replicator_db.js
+++ b/share/www/script/test/replicator_db.js
@@ -1076,6 +1076,10 @@ couchTests.replicator_db = function(debug) {
});
TEquals(200, xhr.status);
+ // Temporary band-aid, give the replicator db some
+ // time to make the switch
+ wait(500);
+
new_doc = {
_id: foo666,
value: 666
http://git-wip-us.apache.org/repos/asf/couchdb/blob/6c976bd9/src/couch_replicator/src/couch_replicator.erl
--
diff --git a/src/couch_replicator/src/couch_replicator.erl
b/src/couch_replicator/src/couch_replicator.erl
index 1f7c08a..e91e1ae 100644
--- a/src/couch_replicator/src/couch_replicator.erl
+++ b/src/couch_replicator/src/couch_replicator.erl
@@ -112,7 +112,7 @@ async_replicate(#rep{id = {BaseId, Ext}, source = Src,
target = Tgt} = Rep) -
RepChildId,
{gen_server, start_link, [?MODULE, Rep, [{timeout, Timeout}]]},
temporary,
- 1,
+ 250,
worker,
[?MODULE]
},
@@ -333,6 +333,9 @@ do_init(#rep{options = Options, id = {BaseId, Ext}} =
Rep) -
}.
+handle_info(shutdown, St) -
+ {stop, shutdown, St};
+
handle_info({'DOWN', Ref, _, _, Why}, #rep_state{source_monitor = Ref} =
St) -
?LOG_ERROR(Source database is down. Reason: ~p, [Why]),
{stop, source_db_down, St};
Never had an issue there. What was the problem? I have the feeling it
may be related to
https://issues.apache.org/jira/browse/COUCHDB-1461
Why?
Afaik Paul's changes are to increase the shutdown time (from 1ms to
250ms) and to avoid the noisy OTP error messages when a running
replication is cancelled (shutdown handle_info clause).
However I would be highly surprised if it fixes the issue you reported.
- benoît
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: [2/8] git commit: Fixing the replicator_db JS test
On Thu, Apr 26, 2012 at 1:10 PM, Benoit Chesneau bchesn...@gmail.com wrote:
On Thu, Apr 26, 2012 at 2:05 PM, Filipe David Manana fdman...@gmail.com
wrote:
On Thu, Apr 26, 2012 at 12:26 PM, Benoit Chesneau bchesn...@gmail.com
wrote:
On Thu, Apr 26, 2012 at 12:18 AM, dav...@apache.org wrote:
Fixing the replicator_db JS test
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/6c976bd9
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/6c976bd9
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/6c976bd9
Branch: refs/heads/master
Commit: 6c976bd948565305c006746f449dc8cd21a749e1
Parents: 7d2fe95
Author: Paul Joseph Davis dav...@apache.org
Authored: Mon Mar 26 15:17:02 2012 -0500
Committer: Paul Joseph Davis dav...@apache.org
Committed: Wed Apr 25 16:54:08 2012 -0500
--
share/www/script/test/replicator_db.js | 4
src/couch_replicator/src/couch_replicator.erl | 5 -
2 files changed, 8 insertions(+), 1 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/6c976bd9/share/www/script/test/replicator_db.js
--
diff --git a/share/www/script/test/replicator_db.js
b/share/www/script/test/replicator_db.js
index 48ca341..edc85f4 100644
--- a/share/www/script/test/replicator_db.js
+++ b/share/www/script/test/replicator_db.js
@@ -1076,6 +1076,10 @@ couchTests.replicator_db = function(debug) {
});
TEquals(200, xhr.status);
+ // Temporary band-aid, give the replicator db some
+ // time to make the switch
+ wait(500);
+
new_doc = {
_id: foo666,
value: 666
http://git-wip-us.apache.org/repos/asf/couchdb/blob/6c976bd9/src/couch_replicator/src/couch_replicator.erl
--
diff --git a/src/couch_replicator/src/couch_replicator.erl
b/src/couch_replicator/src/couch_replicator.erl
index 1f7c08a..e91e1ae 100644
--- a/src/couch_replicator/src/couch_replicator.erl
+++ b/src/couch_replicator/src/couch_replicator.erl
@@ -112,7 +112,7 @@ async_replicate(#rep{id = {BaseId, Ext}, source = Src,
target = Tgt} = Rep) -
RepChildId,
{gen_server, start_link, [?MODULE, Rep, [{timeout, Timeout}]]},
temporary,
- 1,
+ 250,
worker,
[?MODULE]
},
@@ -333,6 +333,9 @@ do_init(#rep{options = Options, id = {BaseId, Ext}} =
Rep) -
}.
+handle_info(shutdown, St) -
+ {stop, shutdown, St};
+
handle_info({'DOWN', Ref, _, _, Why}, #rep_state{source_monitor = Ref} =
St) -
?LOG_ERROR(Source database is down. Reason: ~p, [Why]),
{stop, source_db_down, St};
Never had an issue there. What was the problem? I have the feeling it
may be related to
https://issues.apache.org/jira/browse/COUCHDB-1461
Why?
Afaik Paul's changes are to increase the shutdown time (from 1ms to
250ms) and to avoid the noisy OTP error messages when a running
replication is cancelled (shutdown handle_info clause).
However I would be highly surprised if it fixes the issue you reported.
I understand the change, but I don't know what is the problem . I may
have misinterpreted what was the switch in question. But this is quite
similar since stopping (deleting a doc) or canceling are both calling
the same function, aren't they?
Right. But the issue you reported before is unrelated. The problem
there is that 2 processes are calling 2 functions of the same
supervisor (start_child and delete_child). Those supervisor functions
are synchronous, meaning that one has to wait for the other to finish.
- benoît
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: [jira] [Created] (COUCHDB-1453) Replicator fails with use_users_db = false
On Tuesday, April 3, 2012, Wendall Cada wrote:
I read this gist before creating the ticket. The documentation states that
it *can contain user_ctx. No indication that this is required if I have
use_users_db = false. I'm not sure that this isn't a bug as, it requires a
component to use the baked in _users auth stuff, even though I've
intentionally configured my database to not use this. This is a change in
behavior from what I've expected with couchdb in the past. So now, even if
I have use_users_db = false, I still have to be aware of how _users works
to understand implications of permissions failures.
The parameter use_users_db is related to the oauth authentication handler
(and new in 1.2). It's under the section couch_httpd_oauth. Doesn't apply
to anything else except oauth handler.
Wendall
On 04/02/2012 05:39 PM, Filipe Manana (Commented) (JIRA) wrote:
[ https://issues.apache.org/**jira/browse/COUCHDB-1453?page=**
com.atlassian.jira.plugin.**system.issuetabpanels:comment-**
tabpanelfocusedCommentId=**13244879#comment-13244879https://issues.apache.org/jira/browse/COUCHDB-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244879#comment-13244879]
Filipe Manana commented on COUCHDB-1453:
--**--
Wendall, it's documented in a gist mentioned at:
http://wiki.apache.org/**couchdb/Replication#**Replicator_databasehttp://wiki.apache.org/couchdb/Replication#Replicator_database
Gist: https://gist.github.com/832610 (section 8)
Replicator fails with use_users_db = false
--**
Key: COUCHDB-1453
URL: https://issues.apache.org/**
jira/browse/COUCHDB-1453https://issues.apache.org/jira/browse/COUCHDB-1453
Project: CouchDB
Issue Type: Bug
Components: Replication
Affects Versions: 1.2
Environment: Centos 6 32bit, Erlang R14B, Spidermonkey 1.8.5
Reporter: Wendall Cada
If I create a new replication document in _replicate like this:
{
source: http://localhost:5990/users;,
target: users_backup,
create_target: true,
continuous: true
}
Creation of DB fails with:
unauthorized to access or create database users_backup
If I manually create this database, and set create_target to false,
replication completes, but generates errors while processing the
update_sequence like this:
Replicator: couldn't write document `_design/lck`, revision `2-**
8edc91dec975f893efdc6f440286c7**9e`, to target database `users_backup`.
Error: `unauthorized`, reason: `You are not a db or server admin.`.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA
administrators: https://issues.apache.org/**jira/secure/**
ContactAdministrators!default.**jspahttps://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/**
software/jira http://www.atlassian.com/software/jira
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: [VOTE] Apache CouchDB 1.2.0 release, third round
On Wed, Mar 28, 2012 at 6:51 PM, Stefan Kögl koeglste...@gmail.com wrote: Hi everybody, I just wanted to raise some attention to the DB compaction bug discovered today https://issues.apache.org/jira/browse/COUCHDB-1451 While I initially discovered the bug with a 1.1.2 instance, this could also affect 1.2. I think this issue should be resolved before closing the vote. Filipe already provided a patch, so it shouldn't take too long. Stefan, the patch is only for 1.2.x (and master). That exact issue can't happen on 1.1.x releases, as it's related to code added in 1.2.x. I'm voting -1 on this round just because of this issue. -- Stefan -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [VOTE] Apache CouchDB 1.2.0 release, third round
On Wed, Mar 28, 2012 at 7:03 PM, Jan Lehnardt j...@apache.org wrote: On Mar 28, 2012, at 19:58 , Filipe David Manana wrote: On Wed, Mar 28, 2012 at 6:51 PM, Stefan Kögl koeglste...@gmail.com wrote: Hi everybody, I just wanted to raise some attention to the DB compaction bug discovered today https://issues.apache.org/jira/browse/COUCHDB-1451 While I initially discovered the bug with a 1.1.2 instance, this could also affect 1.2. I think this issue should be resolved before closing the vote. Filipe already provided a patch, so it shouldn't take too long. Stefan, the patch is only for 1.2.x (and master). That exact issue can't happen on 1.1.x releases, as it's related to code added in 1.2.x. I'm voting -1 on this round just because of this issue. Filipe, can you explain how frequent this issue would be? I can't quite discover the circumstances from the patch commit message. Very frequent on some environments / work loads. It's highly timing sensitive. Happens if the compactor inserts the last batch of btree records in less than 500ms after the previous batch. Cheers Jan -- -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: git commit: restore some log messages after 04c4a10
Perfect. Thanks Randall
On Mon, Mar 19, 2012 at 10:56 PM, rand...@apache.org wrote:
Updated Branches:
refs/heads/1.2.x eaa9c4aee - ede9482fc
restore some log messages after 04c4a10
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/ede9482f
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/ede9482f
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/ede9482f
Branch: refs/heads/1.2.x
Commit: ede9482fc3c9e629572ab1376d88f4499c4c8beb
Parents: eaa9c4a
Author: Randall Leeds rand...@apache.org
Authored: Mon Mar 19 15:54:08 2012 -0700
Committer: Randall Leeds rand...@apache.org
Committed: Mon Mar 19 15:54:08 2012 -0700
--
src/couchdb/couch_file.erl | 7 ++-
src/couchdb/couch_log.erl | 6 --
2 files changed, 10 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/ede9482f/src/couchdb/couch_file.erl
--
diff --git a/src/couchdb/couch_file.erl b/src/couchdb/couch_file.erl
index 5e476af..7725ddc 100644
--- a/src/couchdb/couch_file.erl
+++ b/src/couchdb/couch_file.erl
@@ -53,14 +53,19 @@ open(Filepath, Options) -
ignore -
% get the error
receive
- {Ref, Pid, Error} -
+ {Ref, Pid, {error, Reason} = Error} -
case process_info(self(), trap_exit) of
{trap_exit, true} - receive {'EXIT', Pid, _} - ok end;
{trap_exit, false} - ok
end,
+ ?LOG_ERROR(Error opening file ~s: ~s,
+ [Filepath, file:format_error(Reason)]),
Error
end;
Error -
+ % We can't say much here, because it could be any kind of error.
+ % Just let it bubble and an encapsulating subcomponent can perhaps
+ % be more informative. It will likely appear in the SASL log, anyway.
Error
end.
http://git-wip-us.apache.org/repos/asf/couchdb/blob/ede9482f/src/couchdb/couch_log.erl
--
diff --git a/src/couchdb/couch_log.erl b/src/couchdb/couch_log.erl
index 7fb95a7..f299ba7 100644
--- a/src/couchdb/couch_log.erl
+++ b/src/couchdb/couch_log.erl
@@ -89,8 +89,10 @@ init([]) -
case file:open(Filename, [append]) of
{ok, Fd} -
{ok, #state{fd = Fd, level = Level, sasl = Sasl}};
- Error -
- {stop, Error}
+ {error, Reason} -
+ ReasonStr = file:format_error(Reason),
+ io:format(Error opening log file ~s: ~s, [Filename, ReasonStr]),
+ {stop, {error, ReasonStr, Filename}}
end.
debug_on() -
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: [3/3] git commit: remove unnecessary eaccess special casing
--
diff --git a/src/couchdb/couch_server_sup.erl
b/src/couchdb/couch_server_sup.erl
index 7baede3..be3c3a3 100644
--- a/src/couchdb/couch_server_sup.erl
+++ b/src/couchdb/couch_server_sup.erl
@@ -46,7 +46,9 @@ start_server(IniFiles) -
{ok, [PidFile]} -
case file:write_file(PidFile, os:getpid()) of
ok - ok;
- Error - io:format(Failed to write PID file ~s, error: ~p,
[PidFile, Error])
+ {error, Reason} -
+ io:format(Failed to write PID file ~s: ~s,
+ [PidFile, file:format_error(Reason)])
end;
_ - ok
end,
@@ -121,12 +123,10 @@ start_server(IniFiles) -
end end || Uri - Uris],
case file:write_file(UriFile, Lines) of
ok - ok;
- {error, eacces} -
- ?LOG_ERROR(Permission error when writing to URI file ~s,
[UriFile]),
- throw({file_permission_error, UriFile});
- Error2 -
- ?LOG_ERROR(Failed to write to URI file ~s: ~p~n, [UriFile,
Error2]),
- throw(Error2)
+ {error, Reason2} = Error -
+ ?LOG_ERROR(Failed to write to URI file ~s: ~s,
+ [UriFile, file:format_error(Reason2)]),
+ throw(Error)
end
end,
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: [3/3] git commit: remove unnecessary eaccess special casing
On Mon, Mar 19, 2012 at 6:33 PM, Randall Leeds randall.le...@gmail.com wrote:
On Mon, Mar 19, 2012 at 04:48, Filipe David Manana fdman...@apache.orgwrote:
Any special reason why couch_log and couch_file don't get the same
treatment (properly formatted error message) as couch_config and
couch_config_writer?
Hmm. I think maybe I should add it to couch_file because that makes a ton
of sense. At the time, I was thinking I would just let it bubble and it
would be caught above, as it certainly is in many cases.
For couch_log, how can we log anything? It felt futile to try to invoke log
calls when couch_log fails to start.
For couch log just throw an exception/error containing the results of
file:format_error/1 and/or io:format it to stdout/stderr like it's
done in couch_server_sup.erl. Either approach will produce a much more
user friendly error.
Seems like a small (incidental) regression to me.
thanks
On Mon, Mar 19, 2012 at 3:56 AM, rand...@apache.org wrote:
remove unnecessary eaccess special casing
It's better to let these errors bubble and/or not give them special
treatment when file:format_error/1 can do a better job of describing
the failure.
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/48371335
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/48371335
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/48371335
Branch: refs/heads/COUCHDB-1445
Commit: 483713352cb397510111798c4b076afad83f6c46
Parents: 29eac04
Author: Randall Leeds rand...@apache.org
Authored: Sun Mar 18 20:19:27 2012 -0700
Committer: Randall Leeds rand...@apache.org
Committed: Sun Mar 18 20:41:42 2012 -0700
--
src/couchdb/couch_config.erl | 11 ---
src/couchdb/couch_config_writer.erl | 8 +---
src/couchdb/couch_file.erl | 5 +
src/couchdb/couch_log.erl | 2 --
src/couchdb/couch_server_sup.erl | 14 +++---
5 files changed, 17 insertions(+), 23 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/48371335/src/couchdb/couch_config.erl
--
diff --git a/src/couchdb/couch_config.erl b/src/couchdb/couch_config.erl
index f669853..44a102d 100644
--- a/src/couchdb/couch_config.erl
+++ b/src/couchdb/couch_config.erl
@@ -187,13 +187,10 @@ parse_ini_file(IniFile) -
case file:read_file(IniFilename) of
{ok, IniBin0} -
IniBin0;
- {error, eacces} -
- throw({file_permission_error, IniFile});
- {error, enoent} -
- Fmt = Couldn't find server configuration file ~s.,
- Msg = ?l2b(io_lib:format(Fmt, [IniFilename])),
- ?LOG_ERROR(~s~n, [Msg]),
- throw({startup_error, Msg})
+ {error, Reason} = Error -
+ ?LOG_ERROR(Couldn't read server configuration file ~s: ~s,
+ [IniFilename, file:format_error(Reason)]),
+ throw(Error)
end,
Lines = re:split(IniBin, \r\n|\n|\r|\032, [{return, list}]),
http://git-wip-us.apache.org/repos/asf/couchdb/blob/48371335/src/couchdb/couch_config_writer.erl
--
diff --git a/src/couchdb/couch_config_writer.erl
b/src/couchdb/couch_config_writer.erl
index decd269..3a65c37 100644
--- a/src/couchdb/couch_config_writer.erl
+++ b/src/couchdb/couch_config_writer.erl
@@ -22,6 +22,8 @@
-export([save_to_file/2]).
+-include(couch_db.hrl).
+
%% @spec save_to_file(
%% Config::{{Section::string(), Option::string()},
Value::string()},
%% File::filename()) - ok
@@ -38,9 +40,9 @@ save_to_file({{Section, Key}, Value}, File) -
case file:write_file(File, NewFileContents) of
ok -
ok;
- {error, eacces} -
- {file_permission_error, File};
- Error -
+ {error, Reason} = Error -
+ ?LOG_ERROR(Couldn't write config file ~s: ~s,
+ [File, file:format_error(Reason)]),
Error
end.
http://git-wip-us.apache.org/repos/asf/couchdb/blob/48371335/src/couchdb/couch_file.erl
--
diff --git a/src/couchdb/couch_file.erl b/src/couchdb/couch_file.erl
index e195db0..5e476af 100644
--- a/src/couchdb/couch_file.erl
+++ b/src/couchdb/couch_file.erl
@@ -58,10 +58,7 @@ open(Filepath, Options) -
{trap_exit, true} - receive {'EXIT', Pid, _} - ok end;
{trap_exit, false} - ok
end,
- case Error of
- {error, eacces} - {file_permission_error, Filepath};
- _ - Error
- end
Re: Please report your indexing speed
On Mon, Mar 5, 2012 at 2:15 AM, Robert Newson rnew...@apache.org wrote:
Filipe,
Thanks for the explanation. I agree on the way forward and will apply
your patch to the relevant branches with attribution.
Thanks. I'll do that myself soon unless someone in the meanwhile shows
worse results with it.
B.
On 5 March 2012 07:41, Filipe David Manana fdman...@apache.org wrote:
On Sun, Mar 4, 2012 at 9:45 AM, Bob Dionne dio...@dionne-associates.com
wrote:
yes, I was surprised by the 30% claim as my numbers showed it only getting
back to where we were with 1.1.x
I think Bob's suggestion to get to the root code change that caused this
regression is important as it will help us assess all the other cases this
testing hasn't even touched yet
The explanation I gave in the 1.2.0 second round vote identifies the
reason, which is that the updater is (depending on timings) collecting
smaller batches of map results, which makes the btree updates less
efficient (besides higher number of btree updates). The patch
addresses this by queuing a batch of map results instead of queuing
map results one by one. Jan's tests and mine are evidence that this is
valid in practice and not just theory.
The original main goal of COUCHDB-1186 was to make the indexing of
views that emit reasonably large (or complex in structure) map values
more efficient.
Here's an example using Jason's slow_couchdb script with wow.tpl and
map function of function(doc) {emit([doc.type, doc.category],
doc);}:
1.1.x:
fdmanana 07:04:12 ~/git/hub/slow_couchdb (master) docs=20
batch=5000 ./bench.sh wow.tpl
Server: CouchDB/1.1.2a785d32f-git (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.1.2a785d32f-git}
[INFO] Created DB named `db1'
[INFO] Uploaded 5000 documents via _bulk_docs
()
[INFO] Uploaded 5000 documents via _bulk_docs
Building view.
{total_rows:20,offset:0,rows:[
{id:00144af5-9f07-448e-af88-026674e3e3d0,key:[dwarf,assassin],value:{_id:00144af5-9f07-448e-af88-026674e3e3d0,_rev:1-785fbf5e641f3d10fa083501ad82a9fe,data3:Vl6BftQEWY6imvNs0FasOj2CrPCptP70z5d,ratio:1.6,integers:[48028,3170,54066,95547,70643,23763,25804,33180,89061,35274,48244,91792,37936,11855],category:assassin,nested:{dict:{3XGVdTTF:31490,SDxKa54e:40,XIzUloRH:7,5Mj9F4bp:192,1sXfjgYf:1203,XP5YSqhX:25461,QJr0Xhxn:9941},string1:3Q4tvmhHwKvedKiRnoL6xUz,string2:dWI1mrrAypRh,values:[33712,57371,88567,88361,70873,6327,17326,91004,41840,86257],string3:i7OGysnXvynz41VMQJ,coords:[{x:65350.46,y:103881.18},{x:24180.14,y:8474.9},{x:88326.66,y:43151.76},{x:120199.77,y:102270.29},{x:191924.18,y:74479.75}]},level:21,type:dwarf,data1:Vpkplo80LshlcjBE0ySJNNpfgDy2bu8byWrmZ44B,data2:GnyNbos75Wxm1C5MLdOeXNniHamBjld70vHqoJnEtnlfekuPXJ}}
]}
real 2m49.227s
user 0m0.006s
sys 0m0.005s
1.2.x:
fdmanana 07:13:30 ~/git/hub/slow_couchdb (master) docs=20
batch=5000 ./bench.sh wow.tpl
Server: CouchDB/1.2.0 (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.2.0}
[INFO] Created DB named `db1'
[INFO] Uploaded 5000 documents via _bulk_docs
()
[INFO] Uploaded 5000 documents via _bulk_docs
Building view.
{total_rows:20,offset:0,rows:[
{id:0005cd07-49f4-4a99-b506-acef948f2acc,key:[dwarf,assassin],value:{_id:0005cd07-49f4-4a99-b506-acef948f2acc,_rev:1-4b418e69618bf11124a03e1a3845f071,data3:T0W2JBUET9yzRXHfUqcUBwFhYGKh14YFVxk,ratio:1.6999556,integers:[25658,7573,47779,43217,49586,57992,13549,90984,45253,49560,1643,64085,38381,62544],category:assassin,nested:{dict:{oWhW4jJ6:199,EPSVtKtS:5638,8WpzvD5x:73714,stD9Ynfh:8924,0qh5Nc1g:5994,pBa5vJyy:18,s25oAkRc:165270},string1:fNNHb8lxtcy7GpwSU3yRyaC,string2:rilbiZM7yAaK,values:[49632,93665,73258,75675,4229,15742,16965,76825,22049,79829],string3:IwX09SiOLMSSyxffMB,coords:[{x:179620.451164,y:11539.9899782},{x:68483.8206985,y:110559.199709},{x:67197.9402328,y:96702.2106403},{x:25469.8698981,y:79049.4905239},{x:157059.899418,y:34963.4103492}]},level:6,type:dwarf,data1:njpz38JSfz00p2Lc2Jv0dON7UfTljRgz0J2B7w7K,data2:4hpsT2szDrssbUCHEirTzHOIhSxTd83i1FO5aNXgoGAfO2srH1}}
]}
real 1m51.989s
user 0m0.006s
sys 0m0.004s
1.2.x + patch:
fdmanana 07:29:11 ~/git/hub/slow_couchdb (master) docs=20
batch=5000 ./bench.sh wow.tpl
Server: CouchDB/1.2.0 (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.2.0}
[INFO] Created DB named `db1'
[INFO] Uploaded 5000 documents via _bulk_docs
()
[INFO] Uploaded 5000 documents via _bulk_docs
Building view.
{total_rows:20,offset:0,rows:[
{id:0005cd07-49f4-4a99-b506-acef948f2acc,key:[dwarf,assassin],value:{_id:0005cd07-49f4-4a99-b506-acef948f2acc,_rev:1-4b418e69618bf11124a03e1a3845f071,data3:T0W2JBUET9yzRXHfUqcUBwFhYGKh14YFVxk,ratio:1.6999556,integers:[25658,7573,47779,43217,49586,57992,13549,90984,45253,49560,1643,64085,38381,62544],category:assassin,nested:{dict:{oWhW4jJ6:199,EPSVtKtS:5638,8WpzvD5x:73714,stD9Ynfh:8924,0qh5Nc1g:5994
Re: Please report your indexing speed
,type:dwarf,data1:njpz38JSfz00p2Lc2Jv0dON7UfTljRgz0J2B7w7K,data2:4hpsT2szDrssbUCHEirTzHOIhSxTd83i1FO5aNXgoGAfO2srH1}} ]} real1m45.573s user0m0.006s sys 0m0.004s Unless someone comes up with scenarios where 1.2.x with the patch is significantly slower than 1.1.x, I think we should close this and move to release 1.2.0. Thanks all for the testing. On Mar 3, 2012, at 5:25 PM, Bob Dionne wrote: I ran some tests, using Bob's latest script. The first versus the second clearly show the regression. The third is the 1.2.x with the patch to couch_os_process reverted and it seems to have no impact. The last has Filipe's latest patch to couch_view_updater discussed in the other thread and it brings the performance back to the 1.1.x level. I'd say that patch is a +1 1.2.x real 3m3.093s user 0m0.028s sys 0m0.008s == 1.1.x real 2m16.609s user 0m0.026s sys 0m0.007s = 1.2.x with patch to couch_os_process reverted real 3m7.012s user 0m0.029s sys 0m0.008s = 1.2.x with Filipe's katest patch to couch_view_updater real 2m11.038s user 0m0.028s sys 0m0.007s On Feb 28, 2012, at 8:17 AM, Jason Smith wrote: Forgive the clean new thread. Hopefully it will not remain so. If you can, would you please clone https://github.com/jhs/slow_couchdb And build whatever Erlangs and CouchDB checkouts you see fit, and run the test. For example: docs=50 ./bench.sh small_doc.tpl That should run the test and, God willing, upload the results to a couch in the cloud. We should be able to use that information to identify who you are, whether you are on SSD, what Erlang and Couch build, and how fast it ran. Modulo bugs. -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [VOTE] Apache CouchDB 1.2.0 release, second round
Jason, made some more tests with larger documents (template is
https://gist.github.com/1930804) and a different map function:
function(doc) {
emit([doc.type, doc.category], doc.nested.coords);
}
(patch http://friendpaste.com/5C99aqXocN6N6H1BAYIigs)
Here's the results I got ( https://gist.github.com/1930807 )
Before COUCHDB-1186
fdmanana 23:21:05 ~/git/hub/slow_couchdb (master) docs=50
batch=5000 ./bench.sh wow.tpl
Server: CouchDB/1.2.0a-a68a792-git (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.2.0a-a68a792-git}
[INFO] Created DB named `db1'
[INFO] Uploaded 5000 documents via _bulk_docs
()
[INFO] Uploaded 5000 documents via _bulk_docs
Building view.
{total_rows:50,offset:0,rows:[
{id:00051ef7-d735-48d7-9ba8-5a21a86e8d57,key:[dwarf,assassin],value:[{x:31227.35,y:31529.73},{x:116667.85,y:82008.25},{x:224.11,y:36652.41},{x:128565.95,y:6780.2},{x:165230.43,y:176208.63}]}
]}
real5m6.676s
user0m0.009s
sys 0m0.010s
After COUCHDB-1186
fdmanana 23:50:07 ~/git/hub/slow_couchdb (master) docs=50
batch=5000 ./bench.sh wow.tpl
Server: CouchDB/1.2.0a-f023052-git (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.2.0a-f023052-git}
[INFO] Created DB named `db1'
[INFO] Uploaded 5000 documents via _bulk_docs
()
[INFO] Uploaded 5000 documents via _bulk_docs
Building view.
{total_rows:50,offset:0,rows:[
{id:00051ef7-d735-48d7-9ba8-5a21a86e8d57,key:[dwarf,assassin],value:[{x:31227.35,y:31529.73},{x:116667.85,y:82008.25},{x:224.11,y:36652.41},{x:128565.95,y:6780.2},{x:165230.43,y:176208.63}]}
]}
real5m1.395s
user0m0.008s
sys 0m0.010s
After COUCHDB-1186 + better queueing patch (
http://friendpaste.com/178nPFgfyyeGf2vtNRpL0w )
fdmanana 00:14:25 ~/git/hub/slow_couchdb (master) docs=50
batch=5000 ./bench.sh wow.tpl
Server: CouchDB/1.2.0a-f023052-git (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.2.0a-f023052-git}
[INFO] Created DB named `db1'
[INFO] Uploaded 5000 documents via _bulk_docs
()
[INFO] Uploaded 5000 documents via _bulk_docs
Building view.
{total_rows:50,offset:0,rows:[
{id:00051ef7-d735-48d7-9ba8-5a21a86e8d57,key:[dwarf,assassin],value:[{x:31227.35,y:31529.73},{x:116667.85,y:82008.25},{x:224.11,y:36652.41},{x:128565.95,y:6780.2},{x:165230.43,y:176208.63}]}
]}
real4m48.175s
user0m0.008s
sys 0m0.009s
Disk model is APPLE SSD TS128C, quad core machine, 8Gb of ram.
Unfortunately I don't have access to the machine I used for the tests
in COUCHDB-1186 (spinning disk, Linux) before next week.
On Mon, Feb 27, 2012 at 7:49 PM, Paul Davis paul.joseph.da...@gmail.com wrote:
On Mon, Feb 27, 2012 at 7:18 PM, Filipe David Manana
fdman...@apache.org wrote:
Jason, can't reproduce those results, not even close:
http://friendpaste.com/1L4pHH8WQchaLIMCWhKX9Z
Before COUCHDB-1186
fdmanana 16:58:02 ~/git/hub/slow_couchdb (master) docs=50
batch=5 ./bench.sh small_doc.tpl
Server: CouchDB/1.2.0a-a68a792-git (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.2.0a-a68a792-git}
[INFO] Created DB named `db1'
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
Building view.
{total_rows:50,offset:0,rows:[
{id:doc1,key:1,value:1}
]}
real 0m56.241s
user 0m0.006s
sys 0m0.005s
After COUCHDB-1186
fdmanana 17:02:02 ~/git/hub/slow_couchdb (master) docs=50
batch=5 ./bench.sh small_doc.tpl
Server: CouchDB/1.2.0a-f023052-git (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.2.0a-f023052-git}
[INFO] Created DB named `db1'
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
Building view.
{total_rows:50,offset:0,rows:[
{id:doc1,key:1,value:1}
]}
real 1m11.694s
user 0m0.006s
sys 0m0.005s
fdmanana 17:06:01 ~/git/hub/slow_couchdb (master)
1.2.0a-f023052-git with patch
http://friendpaste.com/178nPFgfyyeGf2vtNRpL0w applied on top
fdmanana 17:06:53 ~/git/hub/slow_couchdb (master) docs=50
batch=5 ./bench.sh small_doc.tpl
Server: CouchDB/1.2.0a-f023052-git (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.2.0a-f023052-git}
[INFO] Created DB named `db1'
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via
Re: Please report your indexing speed
Jason, repeated my last test with the 1Kb docs (
https://gist.github.com/1930804, map function
http://friendpaste.com/5C99aqXocN6N6H1BAYIigs ) to cover branch 1.1.x
as well. Here are the full results (also in
https://gist.github.com/1930807):
Before COUCHDB-1186
fdmanana 23:21:05 ~/git/hub/slow_couchdb (master) docs=50
batch=5000 ./bench.sh wow.tpl
Server: CouchDB/1.2.0a-a68a792-git (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.2.0a-a68a792-git}
[INFO] Created DB named `db1'
[INFO] Uploaded 5000 documents via _bulk_docs
()
[INFO] Uploaded 5000 documents via _bulk_docs
Building view.
{total_rows:50,offset:0,rows:[
{id:00051ef7-d735-48d7-9ba8-5a21a86e8d57,key:[dwarf,assassin],value:[{x:31227.35,y:31529.73},{x:116667.85,y:82008.25},{x:224.11,y:36652.41},{x:128565.95,y:6780.2},{x:165230.43,y:176208.63}]}
]}
real5m6.676s
user0m0.009s
sys 0m0.010s
After COUCHDB-1186
fdmanana 23:50:07 ~/git/hub/slow_couchdb (master) docs=50
batch=5000 ./bench.sh wow.tpl
Server: CouchDB/1.2.0a-f023052-git (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.2.0a-f023052-git}
[INFO] Created DB named `db1'
[INFO] Uploaded 5000 documents via _bulk_docs
()
[INFO] Uploaded 5000 documents via _bulk_docs
Building view.
{total_rows:50,offset:0,rows:[
{id:00051ef7-d735-48d7-9ba8-5a21a86e8d57,key:[dwarf,assassin],value:[{x:31227.35,y:31529.73},{x:116667.85,y:82008.25},{x:224.11,y:36652.41},{x:128565.95,y:6780.2},{x:165230.43,y:176208.63}]}
]}
real5m1.395s
user0m0.008s
sys 0m0.010s
After COUCHDB-1186 + better queueing patch
(http://friendpaste.com/178nPFgfyyeGf2vtNRpL0w)
fdmanana 00:14:25 ~/git/hub/slow_couchdb (master) docs=50
batch=5000 ./bench.sh wow.tpl
Server: CouchDB/1.2.0a-f023052-git (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.2.0a-f023052-git}
[INFO] Created DB named `db1'
[INFO] Uploaded 5000 documents via _bulk_docs
()
[INFO] Uploaded 5000 documents via _bulk_docs
Building view.
{total_rows:50,offset:0,rows:[
{id:00051ef7-d735-48d7-9ba8-5a21a86e8d57,key:[dwarf,assassin],value:[{x:31227.35,y:31529.73},{x:116667.85,y:82008.25},{x:224.11,y:36652.41},{x:128565.95,y:6780.2},{x:165230.43,y:176208.63}]}
]}
real4m48.175s
user0m0.008s
sys 0m0.009s
CouchDB branch 1.1.x
fdmanana 08:16:58 ~/git/hub/slow_couchdb (master) docs=50
batch=5000 ./bench.sh wow.tpl
Server: CouchDB/1.1.2a785d32f-git (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.1.2a785d32f-git}
[INFO] Created DB named `db1'
[INFO] Uploaded 5000 documents via _bulk_docs
()
[INFO] Uploaded 5000 documents via _bulk_docs
Building view.
{total_rows:50,offset:0,rows:[
{id:0001c0a1-edcb-4dbc-aa9d-533c73d980cb,key:[dwarf,assassin],value:[{x:62038.32,y:105825.29},{x:90713.13,y:128570.97},{x:43836.37,y:80517.12},{x:71610.97,y:143739.99},{x:86038.39,y:84731.8}]}
]}
real5m44.374s
user0m0.008s
sys 0m0.010s
Disk model APPLE SSD TS128C, quad core machine, 8Gb of ram.
On Tue, Feb 28, 2012 at 5:17 AM, Jason Smith j...@apache.org wrote:
Forgive the clean new thread. Hopefully it will not remain so.
If you can, would you please clone https://github.com/jhs/slow_couchdb
And build whatever Erlangs and CouchDB checkouts you see fit, and run
the test. For example:
docs=50 ./bench.sh small_doc.tpl
That should run the test and, God willing, upload the results to a
couch in the cloud. We should be able to use that information to
identify who you are, whether you are on SSD, what Erlang and Couch
build, and how fast it ran. Modulo bugs.
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: [VOTE] Apache CouchDB 1.2.0 release, second round
we can ignore that, it's something flaky in that fellow's environment. I tend to agree but I'm conservative On Feb 23, 2012, at 1:23 PM, Noah Slater wrote: Can someone convince me this bus error stuff and segfaults is not a blocking issue. Bob tells me that he's followed the steps above and he's still experiencing the issues. Bob, you did follow the steps to install your own SSL right? On Thu, Feb 23, 2012 at 5:09 PM, Jan Lehnardt j...@apache.org wrote: On Feb 23, 2012, at 00:28 , Noah Slater wrote: Hello, I would like call a vote for the Apache CouchDB 1.2.0 release, second round. We encourage the whole community to download and test these release artifacts so that any critical issues can be resolved before the release is made. Everyone is free to vote on this release, so get stuck in! We are voting on the following release artifacts: http://people.apache.org/~nslater/dist/1.2.0/ These artifacts have been built from the following tree-ish in Git: 4cd60f3d1683a3445c3248f48ae064fb573db2a1 Please follow the test procedure before voting: http://wiki.apache.org/couchdb/Test_procedure Thank you. Happy voting, Signature and hashes check out. Mac OS X 10.7.3, 64bit, SpiderMonkey 1.8.0, Erlang R14B04: make check works fine, browser tests in Safari work fine. Mac OS X 10.7.3, 64bit, SpiderMonkey 1.8.5, Erlang R14B04: make check works fine, browser tests in Safari work fine. FreeBSD 9.0, 64bit, SpiderMonkey 1.7.0, Erlang R14B04: make check works fine, browser tests in Safari work fine. CentOS 6.2, 64bit, SpiderMonkey 1.8.5, Erlang R14B04: make check works fine, browser tests in Firefox work fine. Ubuntu 11.4, 64bit, SpiderMonkey 1.8.5, Erlang R14B02: make check works fine, browser tests in Firefox work fine. Ubuntu 10.4, 32bit, SpiderMonkey 1.8.0, Erlang R13B03: make check fails in - 076-file-compression.t: https://gist.github.com/1893373 - 220-compaction-daemon.t: https://gist.github.com/1893387 This on runs in a VM and is 32bit, so I don't know if there's anything in the tests that rely on 64bittyness or the R14B03. Filipe, I think you worked on both features, do you have an idea? I tried running it all through Erlang R15B on Mac OS X 1.7.3, but a good way into `make check` the tests would just stop and hang. The last time, repeatedly in 160-vhosts.t, but when run alone, that test finished in under five seconds. I'm not sure what the issue is here. Despite the things above, I'm happy to give this a +1 if we put a warning about R15B on the download page. Great work all! Cheers Jan -- -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: Welcome Jason Smith
Congratulations Jason! On Mon, Feb 27, 2012 at 9:04 AM, Noah Slater nsla...@tumbolia.org wrote: Hey, I'd like to announce that the Apache CouchDB PMC has voted for Jason Smith to become the 14th committer. Jason has been doing great work providing CouchDB hosting with IrisCouch, is active within the project, and has contributed back to the community through a number of complimentary projects. Welcome Jason! Relax, N -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [VOTE] Apache CouchDB 1.2.0 release, second round
Jason, can't reproduce those results, not even close:
http://friendpaste.com/1L4pHH8WQchaLIMCWhKX9Z
Before COUCHDB-1186
fdmanana 16:58:02 ~/git/hub/slow_couchdb (master) docs=50
batch=5 ./bench.sh small_doc.tpl
Server: CouchDB/1.2.0a-a68a792-git (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.2.0a-a68a792-git}
[INFO] Created DB named `db1'
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
Building view.
{total_rows:50,offset:0,rows:[
{id:doc1,key:1,value:1}
]}
real0m56.241s
user0m0.006s
sys 0m0.005s
After COUCHDB-1186
fdmanana 17:02:02 ~/git/hub/slow_couchdb (master) docs=50
batch=5 ./bench.sh small_doc.tpl
Server: CouchDB/1.2.0a-f023052-git (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.2.0a-f023052-git}
[INFO] Created DB named `db1'
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
Building view.
{total_rows:50,offset:0,rows:[
{id:doc1,key:1,value:1}
]}
real1m11.694s
user0m0.006s
sys 0m0.005s
fdmanana 17:06:01 ~/git/hub/slow_couchdb (master)
1.2.0a-f023052-git with patch
http://friendpaste.com/178nPFgfyyeGf2vtNRpL0w applied on top
fdmanana 17:06:53 ~/git/hub/slow_couchdb (master) docs=50
batch=5 ./bench.sh small_doc.tpl
Server: CouchDB/1.2.0a-f023052-git (Erlang OTP/R14B03)
{couchdb:Welcome,version:1.2.0a-f023052-git}
[INFO] Created DB named `db1'
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
[INFO] Uploaded 5 documents via _bulk_docs
Building view.
{total_rows:50,offset:0,rows:[
{id:doc1,key:1,value:1}
]}
real0m51.089s
user0m0.006s
sys 0m0.004s
fdmanana 17:10:29 ~/git/hub/slow_couchdb (master)
Can you try with R14B0x and also with the patch
http://friendpaste.com/178nPFgfyyeGf2vtNRpL0w ?
Back then I made all testing on a machine with a spinning disk, so the
writer process was slower and likely dequeing more KV pairs from the
work queue on each dequeue operation. The tests I did just now are on
a machine with a ssd disk.
On Mon, Feb 27, 2012 at 4:40 PM, Jason Smith j...@apache.org wrote:
Hi, Filipe. Most people seem to be holding their OTP build constant
for these tests.
If you have the time, would you please check out
https://github.com/jhs/slow_couchdb
It uses seatoncouch mixed with Bob's script to run a basic benchmark.
I expect more template types to grow to help create different data
profiles.
Anyway, here are my results with 500k documents. Note that I built
from your optimization commit, then its parent.
https://gist.github.com/1928169
tl;dr = 2:50 before your commit; 4:13 after.
On Mon, Feb 27, 2012 at 11:33 PM, Filipe David Manana
fdman...@apache.org wrote:
I just tried Jason's script (modified it to use 500 000 docs instead
of 50 000) against 1.2.x and 1.1.1, using OTP R14B03. Here's my
results:
1.2.x:
$ port=5984 ./test.sh
none
Filling db.
done
HTTP/1.1 200 OK
Server: CouchDB/1.2.0 (Erlang OTP/R14B03)
Date: Mon, 27 Feb 2012 16:08:43 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 252
Cache-Control: must-revalidate
{db_name:db1,doc_count:51,doc_del_count:0,update_seq:51,purge_seq:0,compact_running:false,disk_size:130494577,data_size:130490673,instance_start_time:1330358830830086,disk_format_version:6,committed_update_seq:51}
Building view.
real 1m5.725s
user 0m0.006s
sys 0m0.005s
done
1.1.1:
$ port=5984 ./test.sh
Filling db.
done
HTTP/1.1 200 OK
Server: CouchDB/1.1.2a785d32f-git (Erlang OTP/R14B03)
Date: Mon, 27 Feb 2012 16:15:33 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 230
Cache-Control: must-revalidate
{db_name:db1,doc_count:51,doc_del_count:0,update_seq:51,purge_seq:0,compact_running:false,disk_size:122142818,instance_start_time:1330359233327316,disk_format_version:5
Re: [VOTE] Apache CouchDB 1.2.0 release, second round
On Thu, Feb 23, 2012 at 9:09 AM, Jan Lehnardt j...@apache.org wrote: On Feb 23, 2012, at 00:28 , Noah Slater wrote: Hello, I would like call a vote for the Apache CouchDB 1.2.0 release, second round. We encourage the whole community to download and test these release artifacts so that any critical issues can be resolved before the release is made. Everyone is free to vote on this release, so get stuck in! We are voting on the following release artifacts: http://people.apache.org/~nslater/dist/1.2.0/ These artifacts have been built from the following tree-ish in Git: 4cd60f3d1683a3445c3248f48ae064fb573db2a1 Please follow the test procedure before voting: http://wiki.apache.org/couchdb/Test_procedure Thank you. Happy voting, Signature and hashes check out. Mac OS X 10.7.3, 64bit, SpiderMonkey 1.8.0, Erlang R14B04: make check works fine, browser tests in Safari work fine. Mac OS X 10.7.3, 64bit, SpiderMonkey 1.8.5, Erlang R14B04: make check works fine, browser tests in Safari work fine. FreeBSD 9.0, 64bit, SpiderMonkey 1.7.0, Erlang R14B04: make check works fine, browser tests in Safari work fine. CentOS 6.2, 64bit, SpiderMonkey 1.8.5, Erlang R14B04: make check works fine, browser tests in Firefox work fine. Ubuntu 11.4, 64bit, SpiderMonkey 1.8.5, Erlang R14B02: make check works fine, browser tests in Firefox work fine. Ubuntu 10.4, 32bit, SpiderMonkey 1.8.0, Erlang R13B03: make check fails in - 076-file-compression.t: https://gist.github.com/1893373 - 220-compaction-daemon.t: https://gist.github.com/1893387 This on runs in a VM and is 32bit, so I don't know if there's anything in the tests that rely on 64bittyness or the R14B03. Filipe, I think you worked on both features, do you have an idea? The compression fails because snappy NIF needs R13B04+ (forgot to update the test to work with R13B03 and below). For the compaction one, on ubuntu/debian you need to install the package 'erlang-os-mon' I tried running it all through Erlang R15B on Mac OS X 1.7.3, but a good way into `make check` the tests would just stop and hang. The last time, repeatedly in 160-vhosts.t, but when run alone, that test finished in under five seconds. I'm not sure what the issue is here. Despite the things above, I'm happy to give this a +1 if we put a warning about R15B on the download page. Great work all! Cheers Jan -- -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: replicator
On Tue, Feb 21, 2012 at 7:30 PM, Bob Dionne dio...@dionne-associates.com wrote: Hi Filipe, I'm seeing an issue in the replicator. This anonymous function [1] is throwing function_clause errors as it doesn't handle codes 400-500 that might occur in [2,3]. Right, like in some other places we don't deal with all possible http status codes, only a subset that are expected (success or otherwise). Which http status code are you receiving? How to reproduce that? I have a rough idea how to fix it but I'm not sure whether I should handle it in replicator_httpc or up in the callback. Does this ring a bell? Opinions? Cheers, Bob [1] https://github.com/bdionne/couchdb/blob/master/src/couch_replicator/src/couch_replicator_api_wrap.erl#L335 [2] https://github.com/bdionne/couchdb/blob/master/src/couch_replicator/src/couch_replicator_httpc.erl#L90 [3] https://github.com/bdionne/couchdb/blob/master/src/couch_replicator/src/couch_replicator_httpc.erl#L111 -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: Issues blocking the 1.2.0 release
I think COUCHDB-1413 wouldn't hurt to have for 1.2.0, after all it's about correct query results. 1.2.1 is also aceptable. If no objections, I'll push the fix to 1.2.x as well. On Tue, Feb 21, 2012 at 6:32 PM, Jason Smith j...@iriscouch.com wrote: My reading of the JIRA ticket (FWIW) is that Paul explained pretty convincingly why this is only a minor bug if at all. For this release, Paul had a simple fix; although I do not see it in 1.2.x nor JIRA and don't recall offhand what it was exactly. On Tue, Feb 21, 2012 at 10:50 PM, Robert Newson rnew...@apache.org wrote: heh, actually I don't think we did. On 21 February 2012 22:41, Paul Davis paul.joseph.da...@gmail.com wrote: Did we fix the original JSON thing that started this whole broughaha? On Tue, Feb 21, 2012 at 3:57 PM, Noah Slater nsla...@tumbolia.org wrote: Thanks. On Tue, Feb 21, 2012 at 9:46 PM, Jan Lehnardt j...@apache.org wrote: On 21.02.2012, at 22:38, Robert Newson rnew...@apache.org wrote: I resolved the ipv6 ticket as 'cannot reproduce' given that two committers have verified ipv6 replication with 1.2.x. Time for round 2? +1 On 21 February 2012 21:11, Noah Slater nsla...@tumbolia.org wrote: Are we blocked on anything else? Are we good to go? On Tue, Feb 21, 2012 at 7:21 PM, Jan Lehnardt j...@apache.org wrote: Thanks guys, committed. Noah, 1.2.0 is unblocked on this one. On Feb 21, 2012, at 20:13 , Paul Davis wrote: +1 on the patch to require admin for _changes. On Tue, Feb 21, 2012 at 3:36 AM, Jan Lehnardt j...@apache.org wrote: *nudge* I don't feel very confident with a single opinion (thanks Robert), and would love your input on this one. Cheers Jan -- On Feb 16, 2012, at 16:12 , Jan Lehnardt wrote: On Feb 14, 2012, at 13:14 , Noah Slater wrote: Devs, Please outline: - What remains to be fixed for regression purposes I want to bring up one more thing (sorry :). /_users/_changes is currently end-user readable. While /_users/_changes?include_docs=true will not fetch docs the requesting user doesn't have access to, it still gets all doc ids in the /_users db and thus easily can generate a list of all users. I'd like to propose to make /_user/_changes also admin-only before we ship 1.2.0. Again, I'm happy to revisit and make things configurable down the road. Note that the information that a particular user is registered is leaked (a user can't sign up with a username that is already taken, from that it can be deduced that that particular username is already registered). This is in line with most signup systems. Making /_users/_changes admin-only doesn't prevent all leakage of what users have signed up, but it stops bulk-leakage of *all* users in one swoop. What do you think? Cheers Jan -- -- Iris Couch -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [VOTE] Apache CouchDB 1.2.0 release, first round
On Mon, Feb 13, 2012 at 10:55 AM, Randall Leeds randall.le...@gmail.com wrote:
A very rigorous approach might be to work with the tokenized, but not
decoded, JSON so that numeric field values could be passed verbatim. For
1.2 I'd just like to see the integer regression patched.
Catching up on this thread, that seems to be what I had in mind as well.
The following simple patch would do it:
http://friendpaste.com/1H4zOcciVjajGuYTRyrUUB
For every JSON document coming from the http interface, it doesn't
decode numbers, only validates them, and have them represented by the
term {json, RawJsonNumberIoList} (this term is something already used
by mochijson2, but not by ejson before).
For JSON coming from the view server, like the 1 coming from a
validate_doc_update request, we would keep doing as we do now, except
for map/reduce values where we would use the approach mentioned above.
The above patch is just a quick experiment, I didn't attempt to verify
all possible scenarios. Only verified that the test suite passes.
With the patch:
$ curl -X PUT http://localhost:5984/test/doc1 -d '{_id: doc1,
integer: 1, float: 1.0}'
{ok:true,id:doc1,rev:1-67bc2600ba5ce41db1961e410f79a611}
$ curl http://localhost:5984/test/doc1
{_id:doc1,_rev:1-67bc2600ba5ce41db1961e410f79a611,integer:1,float:1.0}
Without (current behaviour):
$ curl -X PUT http://localhost:5984/test/doc1 -d '{_id: doc1,
integer: 1, float: 1.0}'
{ok:true,id:doc1,rev:1-308b01bbabea74f0cea2af7d0e270fb8}
$ curl http://localhost:5984/test/doc1
{_id:doc1,_rev:1-308b01bbabea74f0cea2af7d0e270fb8,integer:1,float:1}
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: [VOTE] Apache CouchDB 1.2.0 release, first round
On Mon, Feb 13, 2012 at 4:04 PM, Jeroen Janssen
jeroen.jans...@gmail.com wrote:
Hi,
Just a quick check (since I am not entirely familiar with this).
Does this also imply that the couchdb 'on disk' format needs a version
increase?
(since the term representing numeric fields changes?)
I don't think it's needed.
The biggest issue here I see at the moment is view collation, as for
it to work properly it would imply to actually decode numbers that are
part of keys emitted by the map functions in order for proper view
collation to happen.
Best regards,
Jeroen Janssen
On Mon, Feb 13, 2012 at 4:46 PM, Filipe David Manana
fdman...@apache.org wrote:
On Mon, Feb 13, 2012 at 10:55 AM, Randall Leeds randall.le...@gmail.com
wrote:
A very rigorous approach might be to work with the tokenized, but not
decoded, JSON so that numeric field values could be passed verbatim. For
1.2 I'd just like to see the integer regression patched.
Catching up on this thread, that seems to be what I had in mind as well.
The following simple patch would do it:
http://friendpaste.com/1H4zOcciVjajGuYTRyrUUB
For every JSON document coming from the http interface, it doesn't
decode numbers, only validates them, and have them represented by the
term {json, RawJsonNumberIoList} (this term is something already used
by mochijson2, but not by ejson before).
For JSON coming from the view server, like the 1 coming from a
validate_doc_update request, we would keep doing as we do now, except
for map/reduce values where we would use the approach mentioned above.
The above patch is just a quick experiment, I didn't attempt to verify
all possible scenarios. Only verified that the test suite passes.
With the patch:
$ curl -X PUT http://localhost:5984/test/doc1 -d '{_id: doc1,
integer: 1, float: 1.0}'
{ok:true,id:doc1,rev:1-67bc2600ba5ce41db1961e410f79a611}
$ curl http://localhost:5984/test/doc1
{_id:doc1,_rev:1-67bc2600ba5ce41db1961e410f79a611,integer:1,float:1.0}
Without (current behaviour):
$ curl -X PUT http://localhost:5984/test/doc1 -d '{_id: doc1,
integer: 1, float: 1.0}'
{ok:true,id:doc1,rev:1-308b01bbabea74f0cea2af7d0e270fb8}
$ curl http://localhost:5984/test/doc1
{_id:doc1,_rev:1-308b01bbabea74f0cea2af7d0e270fb8,integer:1,float:1}
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: [jira] [Commented] (COUCHDB-1407) JSON encoding of number changes
On Mon, Feb 13, 2012 at 6:52 PM, Kevin R. Coombes kevin.r.coom...@gmail.com
If an application depends on the distinction between integers and floating
point values, then it is up to the person writing the application to make
sure this distinction survives. As has already been pointed out, they can
accomplish that goal by storing all numbers as (JSON) strings and using
their application to decode/eval them. This fix requires no changes to the
CouchDB code.
I pretty much agree with that, the fix here is to do nothing other
than document the behaviour.
My point with this patch was not to get it merged, it was more a proof
of concept and to see what problems it posed - i mentioned in the
1.2.0 release first round thread the view collation problem, which
alone by itself it shows it's not a good idea.
I would not even change the encoder to deal with decimal points and
precision. I would advocate just making sure that the documentation is
clear on this point. In particular, it is probably necessary to document
(as a breaking change that may require people to rewrite some of their
applications) the fact that 1.2 may drop trailing zeros after the decimal
point.
You cannot really promise to support different types of numbers without
radically changing the CouchDB code. You would then have to continually
fight with JSON to get it to support something that is beyond its
capabilities. Maintenance would become a nightmare. Let's try to avoid that
road
Kevin
On 2/13/2012 12:07 PM, Paul Joseph Davis (Commented) (JIRA) wrote:
[
https://issues.apache.org/jira/browse/COUCHDB-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207022#comment-13207022
]
Paul Joseph Davis commented on COUCHDB-1407:
As mentioned on the dev@ thread, I'm pretty dead set against this
approach. While there seems to be some sort of general consensus that
storing numbers as uninterpreted bytes and repeating them back is the way to
go it really misses the entirety of the issue.
First, CouchDB has never claimed to pass numbers around while keeping byte
identical representations. This patch attempts to change that drastically
with a very large number of consequences that we haven't begun to
investigate.
Secondly, if we were to actually consider going this route then we'd also
be obliged to start looking at every other place where we change
representations internally as well.
Thirdly, if we were to do that then we'd also have to get into all of the
cases where we're stricter than JSON specifically allows and then try and
address all of those issues as well.
Basically, how about we just fix the encoder to spit out a decimal point
and an appropriate amount of precision and then start documenting our round
tripping limitations.
JSON encoding of number changes
---
Key: COUCHDB-1407
URL: https://issues.apache.org/jira/browse/COUCHDB-1407
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Affects Versions: 1.2
Environment: Ubuntu 12.04 (alpha)
Reporter: Adam Lofts
Attachments: 0001-Only-validate-numbers-on-JSON-decoding.patch
JSON encoding of Number has changed from 1.0.2 to 1.2. JSON only defines
Number but this change causes issues in my app because python decodes the
number as an int in 1.2.
Test case:
PORT=5985
curl -X DELETE http://localhost:$PORT/test-floats/
curl -X PUT http://localhost:$PORT/test-floats/
curl -X PUT http://localhost:$PORT/test-floats/doc1 -H Content-Type:
application/json -d { \a\: 1.0 }
curl http://localhost:$PORT/test-floats/doc1
Run against 1.0.2:
{ok:true}
{ok:true}
{ok:true,id:doc1,rev:1-78e61304147429d3d500aee7806fd26d}
{_id:doc1,_rev:1-78e61304147429d3d500aee7806fd26d,a:1.0}
Run against 1.2:
{ok:true}
{ok:true}
{ok:true,id:doc1,rev:1-78e61304147429d3d500aee7806fd26d}
{_id:doc1,_rev:1-78e61304147429d3d500aee7806fd26d,a:1}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA
administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: possible replication issue
On Wed, Feb 8, 2012 at 9:31 PM, Bob Dionne dio...@dionne-associates.com wrote: Hi Filipe, I ran into a possible issue replicating from BC 0.4 to CouchDB 1.2.x (someone needs to verify this is a problem and not my local setup). The issue seems to be the treatment of sequence numbers like [2,foo]. The latest in 1.2.x of couch_replicator_httpc:send_req runs ?b2l(iolist_to_binary(...)) over that seq and eventually full_url trundles on down to mochiweb_util:quote_plus which loses the brackets and converts the 2 to a %02 Hi Bob, I'm assuming this is related to _changes feed request since parameter. In general, couch_replicator_httpc doesn't do query parameter value encoding (other then percent encoding), that task is left to couch_replicator_api_wrap.erl - for e.g. some requests require JSON encoded values, like open_revs for doc GETs, and others not, like the feed parameter for _changes. If it applies only for that request, I would go for something less abstract like (untested) for example: http://friendpaste.com/6aWgbQaBjMeR9pIshWr5rl And you can't replace the list comprehension in couch_replicator_httpc because there the goal is only to make sure the values are lists/strings - when OAuth auth is used, the OAuth signature must be computed with the query parameter values unencoded and as strings/lists - percent encoding must happen after computing the OAuth signature. regards, I can get around this [1] by converting [2,foo] to 2-foo which is the old format, but this is clearly a workaround. Perhaps this should be fixed in mochiweb_util ? As an aside I noticed mochiweb_util:urlencode does exactly what you have in query_to_str_args so those funs can probably go away. Cheers, Bob [1] https://gist.github.com/1772911 -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [Windows] proposed binary build changes
On Thu, Jan 26, 2012 at 1:09 PM, Dave Cottlehuber d...@muse.net.nz wrote: On Wednesday, 25 January 2012, Filipe David Manana fdman...@apache.org wrote: On Wed, Jan 25, 2012 at 7:24 PM, Jeroen Janssen jeroen.jans...@gmail.com wrote: Hi, Is the issue Filipe mentioned (where a not compressed version 6 database doesn't compress on compaction) going to be fixed before the actual 1.2.0 release? I would say so. Here's a jira ticket I just created with a patch: https://issues.apache.org/jira/browse/COUCHDB-1394 this works fine on mac Jeroen. Thanks for testing Dave. It's now included in 1.2.x. if you apply the patch attached you can check it directly on windows before i get around to rolling a new snapshot. I assume you're ok with erlang? +1 for having this committed for 1.2 please. Thanks Filipe! Dave -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [Windows] proposed binary build changes
On Wed, Jan 25, 2012 at 12:12 AM, Dave Cottlehuber d...@muse.net.nz wrote: On 23 January 2012 08:00, Jeroen Janssen jeroen.jans...@gmail.com wrote: Hi, It seems to be ok, since I get: All 2 tests passed. I also tried compaction again, but it didn't compress the 800Mb .couch file, so I guess something else might be needed aswell? Best regards, Jeroen Hi Jeroen, Thanks heaps, missing DLL is sorted - https://www.dropbox.com/s/jeifcxpbtpo78ak/snapshots/20120124/SDK7.1?v=l Either of these will work but in my testing R15B seems both faster and more memory efficient. NB I am testing compaction by going from 1.1.1 - 1.2.x here, as the on-disk format changes in between. I don't get compression either using your procedure above even with a POST /_restart in between. Hopefully Filipe can clarify what's expected here. If the disk version of a database is already at version 6 (version 1.2) and the database doesn't have compression, compaction will not compress it. I'll correct this soon. If you grab a database file from CouchDB 1.1 (or older), it will be compressed on compaction. 01/23/2012 03:25 PM 246,907,000 fatty_no_snappy.couch 01/23/2012 03:19 PM 38,830,200 fatty_snappy.couch 01/23/2012 01:57 PM 3,104,881 sluggy_no_snappy.couch 01/23/2012 01:58 PM 3,072,113 sluggy_snappy.couch - sluggy is a set of small docs, all around 4KiB size, generated via http://json-generator.appspot.com/ - fatty is a set of large docs, all around ~400KiB size, artificially full of http://baconipsum.com/ These json blobs are in https://www.dropbox.com/s/jeifcxpbtpo78ak/testing/ and are formatted for _bulk_docs. I'm testing your approach using http://friendpaste.com/6t4KwKYWKofHvVzF3kDMLQ A+ Dave -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: Test suite problems blocking 1.2.0 release
(LogLevel=info) is starting. Apache CouchDB has started. Time to relax. [info] [0.263.0] Apache CouchDB has started on http://127.0.0.1:5984/ [info] [0.358.0] 127.0.0.1 - - GET / 200 [info] [0.358.0] 127.0.0.1 - - GET /test_suite_db/0?rev=3-cac36b6fd7e2c7347b3147626c8cfdaf 200 [info] [0.358.0] 127.0.0.1 - - POST /test_suite_db 201 [info] [0.358.0] 127.0.0.1 - - POST /test_suite_db 201 [info] [0.358.0] 127.0.0.1 - - GET /test_suite_db/oppossum 200 [info] [0.358.0] 127.0.0.1 - - PUT /test_suite_db/newdoc 201 [info] [0.358.0] 127.0.0.1 - - DELETE /test_suite_db/doc-does-not-exist 404 [info] [0.358.0] 127.0.0.1 - - PUT /test_suite_db/goldfish 500 -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [Windows] proposed binary build changes
On Wed, Jan 25, 2012 at 7:24 PM, Jeroen Janssen jeroen.jans...@gmail.com wrote: Hi, I can confirm that the new windows binaries work for the following: *) version 5 database, run compaction, reduced disksize (due to compression working) *) create new database, add 200k document, results in ~33k database (also due to compression) So that is good news! :) Is the issue Filipe mentioned (where a not compressed version 6 database doesn't compress on compaction) going to be fixed before the actual 1.2.0 release? I would say so. Here's a jira ticket I just created with a patch: https://issues.apache.org/jira/browse/COUCHDB-1394 Best regards, Jeroen On Wed, Jan 25, 2012 at 1:12 AM, Dave Cottlehuber d...@muse.net.nz wrote: On 23 January 2012 08:00, Jeroen Janssen jeroen.jans...@gmail.com wrote: Hi, It seems to be ok, since I get: All 2 tests passed. I also tried compaction again, but it didn't compress the 800Mb .couch file, so I guess something else might be needed aswell? Best regards, Jeroen Hi Jeroen, Thanks heaps, missing DLL is sorted - https://www.dropbox.com/s/jeifcxpbtpo78ak/snapshots/20120124/SDK7.1?v=l Either of these will work but in my testing R15B seems both faster and more memory efficient. NB I am testing compaction by going from 1.1.1 - 1.2.x here, as the on-disk format changes in between. I don't get compression either using your procedure above even with a POST /_restart in between. Hopefully Filipe can clarify what's expected here. 01/23/2012 03:25 PM 246,907,000 fatty_no_snappy.couch 01/23/2012 03:19 PM 38,830,200 fatty_snappy.couch 01/23/2012 01:57 PM 3,104,881 sluggy_no_snappy.couch 01/23/2012 01:58 PM 3,072,113 sluggy_snappy.couch - sluggy is a set of small docs, all around 4KiB size, generated via http://json-generator.appspot.com/ - fatty is a set of large docs, all around ~400KiB size, artificially full of http://baconipsum.com/ These json blobs are in https://www.dropbox.com/s/jeifcxpbtpo78ak/testing/ and are formatted for _bulk_docs. I'm testing your approach using http://friendpaste.com/6t4KwKYWKofHvVzF3kDMLQ A+ Dave -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: Issues blocking 1.2.0 release
On Mon, Jan 23, 2012 at 5:54 AM, Noah Slater nsla...@tumbolia.org wrote: Yep, that seems reasonable. Let me know when I can test again. :) Done. Can you test again?
Re: Issues blocking 1.2.0 release
Noah, does it fail occasionally or every time for you?
I'm assuming you're with a slow machine or the machine is a bit overloaded.
Can you try with the following patch?
diff --git a/test/etap/242-replication-many-leaves.t
b/test/etap/242-replication-many-leaves.t
index d8d3eb9..ad9d180 100755
--- a/test/etap/242-replication-many-leaves.t
+++ b/test/etap/242-replication-many-leaves.t
@@ -77,6 +77,7 @@ test() -
couch_server_sup:start_link(test_util:config_files()),
ibrowse:start(),
crypto:start(),
+couch_config:set(replicator, worker_processes, 1, false),
Pairs = [
{source_db_name(), target_db_name()},
On Sun, Jan 22, 2012 at 5:41 PM, Noah Slater nsla...@tumbolia.org wrote:
Hello,
The following test fails:
./test/etap/run test/etap/242-replication-many-leaves.t
Log is:
ok 38 - Document revisions updated with 2 attachments
# Triggering replication again
[info] [0.3567.0] 127.0.0.1 - - HEAD /couch_test_rep_db_b/ 200
[info] [0.3567.0] 127.0.0.1 - - GET /couch_test_rep_db_b/ 200
[info] [0.3567.0] 127.0.0.1 - - GET
/couch_test_rep_db_b/_local/4296cc7705ed9d0c0cf63593b42a10b7 200
[info] [0.6460.0] Replication `4296cc7705ed9d0c0cf63593b42a10b7` is
using:
4 worker processes
a worker batch size of 500
20 HTTP connections
a connection timeout of 3 milliseconds
10 retries per request
socket options are: [{keepalive,true},{nodelay,false}]
source start sequence 6
[info] [0.2.0] starting new replication
`4296cc7705ed9d0c0cf63593b42a10b7` at 0.6460.0 (`couch_test_rep_db_a` - `
http://127.0.0.1:53314/couch_test_rep_db_b/`)
[info] [0.3573.0] 127.0.0.1 - - POST /couch_test_rep_db_b/_revs_diff 200
[info] [0.3569.0] 127.0.0.1 - - POST /couch_test_rep_db_b/_revs_diff 200
[info] [0.3573.0] 127.0.0.1 - - POST /couch_test_rep_db_b/_bulk_docs 201
[info] [0.3567.0] 127.0.0.1 - - POST /couch_test_rep_db_b/_revs_diff 200
[info] [0.3567.0] 127.0.0.1 - - POST /couch_test_rep_db_b/_bulk_docs 201
[info] [0.6468.0] Retrying POST request to
http://127.0.0.1:53314/couch_test_rep_db_b/_bulk_docs in 0.25 seconds due
to error req_timedout
[info] [0.3573.0] 127.0.0.1 - - POST /couch_test_rep_db_b/_bulk_docs 201
[info] [0.6468.0] Retrying POST request to
http://127.0.0.1:53314/couch_test_rep_db_b/_bulk_docs in 0.5 seconds due to
error req_timedout
[info] [0.3568.0] 127.0.0.1 - - POST /couch_test_rep_db_b/_bulk_docs 201
[info] [0.6468.0] Retrying POST request to
http://127.0.0.1:53314/couch_test_rep_db_b/_bulk_docs in 1.0 seconds due to
error req_timedout
[info] [0.3571.0] 127.0.0.1 - - POST /couch_test_rep_db_b/_bulk_docs 201
[info] [0.6468.0] Retrying POST request to
http://127.0.0.1:53314/couch_test_rep_db_b/_bulk_docs in 2.0 seconds due to
error req_timedout
[info] [0.6463.0] 127.0.0.1 - - POST /couch_test_rep_db_b/_bulk_docs 201
[info] [0.6468.0] Retrying POST request to
http://127.0.0.1:53314/couch_test_rep_db_b/_bulk_docs in 4.0 seconds due to
error req_timedout
[info] [0.8537.0] 127.0.0.1 - - POST /couch_test_rep_db_b/_bulk_docs 201
[info] [0.6468.0] Retrying POST request to
http://127.0.0.1:53314/couch_test_rep_db_b/_bulk_docs in 8.0 seconds due to
error req_timedout
[info] [0.9151.0] 127.0.0.1 - - POST /couch_test_rep_db_b/_bulk_docs 201
[info] [0.6468.0] Retrying POST request to
http://127.0.0.1:53314/couch_test_rep_db_b/_bulk_docs in 16.0 seconds due
to error req_timedout
[info] [0.9768.0] 127.0.0.1 - - POST /couch_test_rep_db_b/_bulk_docs 201
[info] [0.6468.0] Retrying POST request to
http://127.0.0.1:53314/couch_test_rep_db_b/_bulk_docs in 32.0 seconds due
to error req_timedout
[info] [0.10390.0] 127.0.0.1 - - POST /couch_test_rep_db_b/_bulk_docs 201
Bail out! Timeout waiting for replication to finish
escript: exception error: bad argument
in function etap:diag/1
in call from erl_eval:do_apply/5
in call from erl_eval:exprs/5
in call from escript:eval_exprs/5
in call from erl_eval:local_func/5
in call from escript:interpret/4
in call from escript:start/1
in call from init:start_it/1
Please advise.
Thanks,
N
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: git commit: Fix SpiderMonkey version detection
Paul, after this change I'm no longer able to build master (haven't
tried other branches however).
configure can't find js/jsapi.h, this is because my jsapi.h doesn't
live inside a js directory.
Ubuntu 10.10
jsapi.h full path: /usr/include/xulrunner-1.9.2.24/jsapi.h
flags passed to configure:
configure --with-js-include=/usr/include/xulrunner-1.9.2.24
--with-js-lib=/usr/lib/xulrunner-1.9.2.24
It's the only spidermonkey version I have installed.
Without this commit, configure doesn't complain and everything works fine.
Is this a local issue or something missing in the autotools config?
On Sat, Jan 21, 2012 at 10:10 PM, dav...@apache.org wrote:
Updated Branches:
refs/heads/master da33e3447 - 10047e759
Fix SpiderMonkey version detection
Randall's last patch to only test for JSOPTION_ANONFUNFIX ended up
reordering the test before the headers were located. This ran into
errors in version detection. This patch reorders the header location as
well as adds a few more default search paths when no --with-js-include
option is specified to account for newer SpiderMonkeys that puth their
headers into $PREFIX/include/js.
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/10047e75
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/10047e75
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/10047e75
Branch: refs/heads/master
Commit: 10047e75935818e0421bdd9ac96dc21334f90e95
Parents: da33e34
Author: Paul Joseph Davis dav...@apache.org
Authored: Sat Jan 21 16:08:58 2012 -0600
Committer: Paul Joseph Davis dav...@apache.org
Committed: Sat Jan 21 16:08:58 2012 -0600
--
configure.ac | 41 ++---
1 files changed, 22 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/10047e75/configure.ac
--
diff --git a/configure.ac b/configure.ac
index c6d564a..adfd740 100644
--- a/configure.ac
+++ b/configure.ac
@@ -177,8 +177,11 @@ AS_CASE([$(uname -s)],
[CYGWIN*], [] ,
[*], [
CPPFLAGS=$CPPFLAGS -I/opt/local/include
+ CPPFLAGS=$CPPFLAGS -I/opt/local/include/js
CPPFLAGS=$CPPFLAGS -I/usr/local/include
+ CPPFLAGS=$CPPFLAGS -I/usr/local/include/js
CPPFLAGS=$CPPFLAGS -I/usr/include
+ CPPFLAGS=$CPPFLAGS -I/usr/include/js
LDFLAGS=$LDFLAGS -L/opt/local/lib
LDFLAGS=$LDFLAGS -L/usr/local/lib
])
@@ -203,6 +206,17 @@ AS_CASE([$(uname -s)],
AM_CONDITIONAL([WINDOWS], [test x$IS_WINDOWS = xTRUE])
+AC_CHECK_HEADER([jsapi.h], [], [
+ AC_CHECK_HEADER([js/jsapi.h],
+ [
+ CPPFLAGS=$CPPFLAGS -I$JS_INCLUDE/js
+ ],
+ [
+ AC_MSG_ERROR([Could not find the jsapi header.
+
+Are the Mozilla SpiderMonkey headers installed?])
+ ])])
+
OLD_LIBS=$LIBS
LIBS=$JS_LIBS $LIBS
OLD_CPPFLAGS=$CPPFLAGS
@@ -247,6 +261,14 @@ AC_CHECK_LIB([$JS_LIB_BASE],
[JS_GetStringCharsAndLength],
# Else, hope that 1.7.0 works
+# Deal with JSScript - JSObject - JSScript switcheroo
+
+AC_CHECK_TYPE([JSScript*],
+ [AC_DEFINE([JSSCRIPT_TYPE], [JSScript*], [Use JSObject* for scripts])],
+ [AC_DEFINE([JSSCRIPT_TYPE], [JSObject*], [Use JSScript* for scripts])],
+ [[#include jsapi.h]]
+)
+
AC_DEFINE([COUCHJS_NAME], [couchjs], [CouchJS executable name.])
if test x${IS_WINDOWS} = xTRUE; then
@@ -298,25 +320,6 @@ fi
JS_LIBS=-l$JS_LIB_BASE -lm $JS_LIBS
AC_SUBST(JS_LIBS)
-AC_CHECK_HEADER([jsapi.h], [], [
- AC_CHECK_HEADER([js/jsapi.h],
- [
- CPPFLAGS=$CPPFLAGS -I$JS_INCLUDE/js
- ],
- [
- AC_MSG_ERROR([Could not find the jsapi header.
-
-Are the Mozilla SpiderMonkey headers installed?])
- ])])
-
-# Deal with JSScript - JSObject - JSScript switcheroo
-
-AC_CHECK_TYPE([JSScript*],
- [AC_DEFINE([JSSCRIPT_TYPE], [JSScript*], [Use JSObject* for scripts])],
- [AC_DEFINE([JSSCRIPT_TYPE], [JSObject*], [Use JSScript* for scripts])],
- [[#include jsapi.h]]
-)
-
LIBS=$OLD_LIBS
CPPFLAGS=$OLD_CPPFLAGS
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: Issues blocking 1.2.0 release
On Sun, Jan 22, 2012 at 6:47 PM, Noah Slater nsla...@tumbolia.org wrote:
No change, still fails.
Noah, to try to find out if it's due to slowness of the machine or
some other issue, do you think you can try to increase the following
timeout in the test?
diff --git a/test/etap/242-replication-many-leaves.t
b/test/etap/242-replication-many-leaves.t
index d8d3eb9..737cd31 100755
--- a/test/etap/242-replication-many-leaves.t
+++ b/test/etap/242-replication-many-leaves.t
@@ -287,6 +287,6 @@ replicate(Source, Target) -
receive
{'DOWN', MonRef, process, Pid, Reason} -
etap:is(Reason, normal, Replication finished successfully)
-after 30 -
+after 90 -
etap:bail(Timeout waiting for replication to finish)
end.
On Sun, Jan 22, 2012 at 6:08 PM, Noah Slater nsla...@tumbolia.org wrote:
On Sun, Jan 22, 2012 at 6:01 PM, Filipe David Manana
fdman...@apache.orgwrote:
Noah, does it fail occasionally or every time for you?
Fails every time.
I'm assuming you're with a slow machine or the machine is a bit
overloaded.
Shouldn't be, I'm not doing anything else right now, and this is a new MBA.
Can you try with the following patch?
Yes. Will report back.
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: Issues blocking 1.2.0 release
On Sun, Jan 22, 2012 at 7:20 PM, Noah Slater nsla...@tumbolia.org wrote:
Works. How do we proceed?
For how much time does the test runs? On 2 different physical
machines, it takes about 1 minute and 10 seconds for me.
Perhaps some manual replication tests could confirm if there's
something wrong with the codebase, your environment or if simply
increasing the timeout is not alarming.
On Sun, Jan 22, 2012 at 7:05 PM, Noah Slater nsla...@tumbolia.org wrote:
OVAR 9000! (Testing now...)
On Sun, Jan 22, 2012 at 6:56 PM, Filipe David Manana
fdman...@apache.orgwrote:
On Sun, Jan 22, 2012 at 6:47 PM, Noah Slater nsla...@tumbolia.org
wrote:
No change, still fails.
Noah, to try to find out if it's due to slowness of the machine or
some other issue, do you think you can try to increase the following
timeout in the test?
diff --git a/test/etap/242-replication-many-leaves.t
b/test/etap/242-replication-many-leaves.t
index d8d3eb9..737cd31 100755
--- a/test/etap/242-replication-many-leaves.t
+++ b/test/etap/242-replication-many-leaves.t
@@ -287,6 +287,6 @@ replicate(Source, Target) -
receive
{'DOWN', MonRef, process, Pid, Reason} -
etap:is(Reason, normal, Replication finished successfully)
- after 30 -
+ after 90 -
etap:bail(Timeout waiting for replication to finish)
end.
On Sun, Jan 22, 2012 at 6:08 PM, Noah Slater nsla...@tumbolia.org
wrote:
On Sun, Jan 22, 2012 at 6:01 PM, Filipe David Manana
fdman...@apache.orgwrote:
Noah, does it fail occasionally or every time for you?
Fails every time.
I'm assuming you're with a slow machine or the machine is a bit
overloaded.
Shouldn't be, I'm not doing anything else right now, and this is a new
MBA.
Can you try with the following patch?
Yes. Will report back.
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: Issues blocking 1.2.0 release
Noah, was able to reproduce your issue by tweaking the test to create
more leaf revisions for a document:
diff --git a/test/etap/242-replication-many-leaves.t
b/test/etap/242-replication-many-leaves.t
index d8d3eb9..4eb4765 100755
--- a/test/etap/242-replication-many-leaves.t
+++ b/test/etap/242-replication-many-leaves.t
@@ -56,7 +56,7 @@ doc_ids() -
doc_num_conflicts(doc1) - 10;
doc_num_conflicts(doc2) - 100;
-doc_num_conflicts(doc3) - 286.
+doc_num_conflicts(doc3) - 500.
main(_) -
With that change, I get exactly the same timeout as you get when the
test runs the push replication. It turns out that some _bulk_docs
requests are taking more than 30 seconds (default replication
connection timeout) therefore the replication request retry messages.
Verified this by timing the _bulk_docs handler to log the time it
takes:
diff --git a/src/couchdb/couch_httpd_db.erl b/src/couchdb/couch_httpd_db.erl
index d7ecb4a..442571d 100644
--- a/src/couchdb/couch_httpd_db.erl
+++ b/src/couchdb/couch_httpd_db.erl
@@ -297,6 +297,7 @@
db_req(#httpd{path_parts=[_,_ensure_full_commit]}=Req, _Db) -
send_method_not_allowed(Req, POST);
db_req(#httpd{method='POST',path_parts=[_,_bulk_docs]}=Req, Db) -
+T0 = now(),
couch_stats_collector:increment({httpd, bulk_requests}),
couch_httpd:validate_ctype(Req, application/json),
{JsonProps} = couch_httpd:json_body_obj(Req),
@@ -357,7 +358,9 @@
db_req(#httpd{method='POST',path_parts=[_,_bulk_docs]}=Req, Db)
-
{ok, Errors} = couch_db:update_docs(Db, Docs, Options,
replicated_changes),
ErrorsJson =
lists:map(fun update_doc_result_to_json/1, Errors),
-send_json(Req, 201, ErrorsJson)
+Rr = send_json(Req, 201, ErrorsJson),
+?LOG_ERROR(BULK DOCS took ~p ms~n,
[timer:now_diff(now(), T0) / 1000]),
+Rr
end
end;
db_req(#httpd{path_parts=[_,_bulk_docs]}=Req, _Db) -
I was getting _bulk_docs response times after 50 seconds.
This convinces me there's nothing wrong with the codebase, the
timeouts just needs to be increased:
diff --git a/test/etap/242-replication-many-leaves.t
b/test/etap/242-replication-many-leaves.t
index d8d3eb9..6508112 100755
--- a/test/etap/242-replication-many-leaves.t
+++ b/test/etap/242-replication-many-leaves.t
@@ -77,6 +77,7 @@ test() -
couch_server_sup:start_link(test_util:config_files()),
ibrowse:start(),
crypto:start(),
+couch_config:set(replicator, connection_timeout, 9, false),
Pairs = [
{source_db_name(), target_db_name()},
@@ -287,6 +288,6 @@ replicate(Source, Target) -
receive
{'DOWN', MonRef, process, Pid, Reason} -
etap:is(Reason, normal, Replication finished successfully)
-after 30 -
+after 90 -
etap:bail(Timeout waiting for replication to finish)
end.
Alternatively the test can be updated to create less revisions for the
document doc3. The current revisions # is 286 but for the tests'
purpose 205+ is enough, which should make it faster - 7000 (max url
length) / length(DocRevision) = 205
If it's ok for you, updating the timeouts plus reducing the # from 286
to 210 is fine for me.
On Mon, Jan 23, 2012 at 12:00 AM, Noah Slater nsla...@tumbolia.org wrote:
I'm just the dumb QA guy.
If you have some diagnostics you want me to run on my machine, I am happy
to.
On Sun, Jan 22, 2012 at 11:31 PM, Filipe David Manana
fdman...@apache.orgwrote:
On Sun, Jan 22, 2012 at 7:20 PM, Noah Slater nsla...@tumbolia.org wrote:
Works. How do we proceed?
For how much time does the test runs? On 2 different physical
machines, it takes about 1 minute and 10 seconds for me.
Perhaps some manual replication tests could confirm if there's
something wrong with the codebase, your environment or if simply
increasing the timeout is not alarming.
On Sun, Jan 22, 2012 at 7:05 PM, Noah Slater nsla...@tumbolia.org
wrote:
OVAR 9000! (Testing now...)
On Sun, Jan 22, 2012 at 6:56 PM, Filipe David Manana
fdman...@apache.orgwrote:
On Sun, Jan 22, 2012 at 6:47 PM, Noah Slater nsla...@tumbolia.org
wrote:
No change, still fails.
Noah, to try to find out if it's due to slowness of the machine or
some other issue, do you think you can try to increase the following
timeout in the test?
diff --git a/test/etap/242-replication-many-leaves.t
b/test/etap/242-replication-many-leaves.t
index d8d3eb9..737cd31 100755
--- a/test/etap/242-replication-many-leaves.t
+++ b/test/etap/242-replication-many-leaves.t
@@ -287,6 +287,6 @@ replicate(Source, Target) -
receive
{'DOWN', MonRef, process, Pid, Reason} -
etap:is(Reason, normal, Replication finished successfully)
- after 30 -
+ after 90 -
etap:bail(Timeout waiting for replication to finish)
end.
On Sun, Jan 22, 2012 at 6:08 PM, Noah Slater nsla...@tumbolia.org
wrote:
On Sun, Jan 22, 2012 at 6:01 PM, Filipe
Re: 1.2.0 bug fix question
On Sunday, January 15, 2012, Randall Leeds randall.le...@gmail.com wrote: On Sun, Jan 15, 2012 at 13:55, Randall Leeds randall.le...@gmail.com wrote: On Sun, Jan 15, 2012 at 13:47, Dustin Sallings dus...@spy.net wrote: I don't want to divert the RFC question for what's really a Q, but I'd like to ask as it directly affects usability for me. When going through the Couchbase Single development, I hit tons of bugs in the replicator DB, most of which Filipe fixed for me pretty quickly. As this was before the git integration and was done in our git repos, it's hard to know whether all those contributions made it back. I don't know if actual Apache bugs were fixed, because of the workflow. I'd hit something, tell Filipe, he'd give me a fix and I'd be happy. Do we know if all those things made it back into replicator DB? All are already present in the 1.1.1 release. Thanks for the heads up Dustin. There has been a pretty constant stream of commits to the replicator in the 1.2.x branch and many closed JIRA tickets. Filipe is probably the only person who can really answer that. $ git log --pretty=oneline --author=Filipe 1.1.1.. | wc -l 295 $ git log --pretty=oneline --author=Filipe 1.1.1.. | grep -i replic | wc -l 103 That was on master, but the numbers aren't much different for 1.2.x. :-D -- dustin sallings -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: RFC: Releasing 1.2.0
On Sunday, January 15, 2012, Robert Newson rnew...@apache.org wrote: I'd say it's too late for the ibrowse fix but we should pick it up for 1.3 Can you point us to what that ibrowse fix is (e.g. Github commit link)? As far as I can see there are no fixes upstream that we don't have in 1.2.x Thanks B. On 15 January 2012 21:31, Noah Slater nsla...@tumbolia.org wrote: I await your feedback. On Sun, Jan 15, 2012 at 7:46 PM, Robert Newson rnew...@apache.org wrote: If it's a regression, before. If not, after. B. On 15 January 2012 19:40, Noah Slater nsla...@tumbolia.org wrote: Are we going to punt on a Windows binary, Dave? On Sun, Jan 15, 2012 at 7:05 PM, Dave Cottlehuber d...@muse.net.nz wrote: On 15 January 2012 19:05, Noah Slater nsla...@tumbolia.org wrote: Bump. Dave? Gonna move without if you don't speak up. :) Sorry!! Literally *just* finished looking at this. TL;DR - let's roll 1.2.0. I don't see any *functional* issues in the failures from the test suite - attachments are written, and read, correctly. For some as yet unknown reason the MD5 is different on Windows vs Linux Mac OS, but this has been present for some time. It's only turned up as a result of additional tests applied in COUCHDB-1337. The underlying crypto:md5 values are the same, and so is the raw HTTP data. I am still digging through mochi to where this comes from. I don't see any issues for replication - can anybody confirm? This looks like a dev issue rather than user impacting. from 1.1.1_js-1.8.0: 12 Digest = base64:encode(Digest_Binary). jeLnIuUvK7d+6gya044lVA== from 1.2.x: 8 Digest = base64:encode(Digest_Binary). jeLnIuUvK7d+6gya044lVA== A+ Dave -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: RFC: Releasing 1.2.0
On Tuesday, January 10, 2012, Randall Leeds randall.le...@gmail.com wrote: On Tue, Jan 10, 2012 at 00:31, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Mon, Jan 9, 2012 at 23:19, Filipe David Manana fdman...@apache.org wrote: 2) Can anyone mention how snappy/zlib impacts performance? * https://issues.apache.org/jira/browse/COUCHDB-1120 So does 1.2 have both the new JSON implementation and the snappy compression? It seems like all the benchmarks in the ticket are either compression or JSON NIF, but not both. Dirkjan, the sections that say Results with the snappy branch mean that the respective branch had the C JSON encoder/decoder and the compression. The reason to compare against with and without the C JSON parser was to verify that all performance gains were not obtained only by the JSON parser. Both. They are not alternatives. Compression occurs after JSON decoding when storing, decompression before JSON encoding on the way out. You'll have to test the impact on your workload with each and decide on the settings that are right for you. Cheers, Dirkjan -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: 1.2.x builds on Iris Couch
On Mon, Jan 9, 2012 at 11:58 AM, Jason Smith j...@iriscouch.com wrote: I merged and deployed a 1.2.x branch on Iris Couch, so if you need a couch in space for testing prior to the next release, let me know and I'll reboot you. Perhaps Filipe? Sure. If you can create me an account based on 1.2.x, it would be great. Thanks Jason. :) -- Iris Couch -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: RFC: Releasing 1.2.0
On Mon, Jan 9, 2012 at 9:40 PM, till t...@php.net wrote: Just looked at CHANGES, a few questions: Hi Till, answers below. 1) Can anyone go into detail on the replicator? (E.g. where are the configuration options.) * https://issues.apache.org/jira/browse/COUCHDB-1024 * http://s.apache.org/KsY * http://wiki.apache.org/couchdb/Replication#New_features_introduced_in_CouchDB_1.2.0 There's more stuff out there I don't recall at the moment. 2) Can anyone mention how snappy/zlib impacts performance? * https://issues.apache.org/jira/browse/COUCHDB-1120 Most of the stuff is a little vague for my taste – maybe the Jira issues explain the additions better. If so, should they be linked always? Personally, I would like to link a list of jira tickets to each release. I don't recall if this possibility was ever discussed before. Till On Mon, Jan 9, 2012 at 10:34 PM, till t...@php.net wrote: Can haz release? :) On Fri, Jan 6, 2012 at 11:34 PM, Randall Leeds randall.le...@gmail.comwrote: On Fri, Jan 6, 2012 at 13:22, Robert Newson rnew...@apache.org wrote: I'm checking it now. Jan did his JIRA dance, so I think we're clean on that end. I'm looking into whether all the fixes got backported, etc, and I'll also read over the full 1.1.x-1.2.x diff. Just checked over my commits and everything appropriate is back-ported. Good to go from my end. +1 on calling the vote. B. On 6 January 2012 21:19, Noah Slater nsla...@tumbolia.org wrote: Hey chaps, Let's pull our finger out and get 1.2.0 shipped. Do we have everything in trunk that we want in 1.2.0, or are there any patches y'all are sitting on? Also, can you all check NEWS and CHANGES to make sure they are up to date? Thanks all, N -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: javascript ./run
I don't get the same error as you, but the changes made to master (by Jan) to make the JavaScript CLI tests work, were not applied to 1.2.x. On Sat, Jan 7, 2012 at 11:28 AM, Bob Dionne b...@cloudant.com wrote: I see the same behavior, it's built in to make check now also. I thought maybe it was SM because I'm running 1.7.x On Jan 7, 2012, at 6:19 AM, Benoit Chesneau wrote: Does javascript console testts work for you on 1.2 ? I tried after a clean -fdx : $ cd test/javascript/ $ ./run [couchjs] SyntaxError: missing } after property list Failed to compile script. - benoît -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: git commit: Add heartbeat parameter to all _changes requests
On Sat, Jan 7, 2012 at 8:55 PM, Randall Leeds randall.le...@gmail.com wrote:
I'm cherry-picking this for 1.2. Especially with a brand new
replicator, it would be very nice if no-one gets bitten by this doing
filtered replication.
Applies cleanly.
Fine for me.
On Sat, Jan 7, 2012 at 05:48, fdman...@apache.org wrote:
Updated Branches:
refs/heads/master 975ea08c7 - a2d57c8bb
Add heartbeat parameter to all _changes requests
Even non-continuous _changes requests, particularly filtered
ones, can cause long periods of inactivity.
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/a2d57c8b
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/a2d57c8b
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/a2d57c8b
Branch: refs/heads/master
Commit: a2d57c8bb34f4621fed00f4e65191a1531bc3110
Parents: 975ea08
Author: Filipe David Borba Manana fdman...@apache.org
Authored: Sat Jan 7 13:44:45 2012 +
Committer: Filipe David Borba Manana fdman...@apache.org
Committed: Sat Jan 7 13:44:45 2012 +
--
.../src/couch_replicator_api_wrap.erl | 6 --
1 files changed, 4 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/a2d57c8b/src/couch_replicator/src/couch_replicator_api_wrap.erl
--
diff --git a/src/couch_replicator/src/couch_replicator_api_wrap.erl
b/src/couch_replicator/src/couch_replicator_api_wrap.erl
index a29fe94..d287fcb 100644
--- a/src/couch_replicator/src/couch_replicator_api_wrap.erl
+++ b/src/couch_replicator/src/couch_replicator_api_wrap.erl
@@ -306,13 +306,15 @@ update_docs(Db, DocList, Options, UpdateType) -
changes_since(#httpdb{headers = Headers1} = HttpDb, Style, StartSeq,
UserFun, Options) -
+ HeartBeat = erlang:max(1000, HttpDb#httpdb.timeout div 3),
BaseQArgs = case get_value(continuous, Options, false) of
false -
[{feed, normal}];
true -
- [{feed, continuous}, {heartbeat, 1}]
+ [{feed, continuous}]
end ++ [
- {style, atom_to_list(Style)}, {since,
couch_util:to_list(StartSeq)}
+ {style, atom_to_list(Style)}, {since,
couch_util:to_list(StartSeq)},
+ {heartbeat, integer_to_list(HeartBeat)}
],
DocIds = get_value(doc_ids, Options),
{QArgs, Method, Body, Headers} = case DocIds of
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: [ANN]: Dave Cottlehuber elected CouchDB committer
Welcome Dave! On Fri, Jan 6, 2012 at 9:22 PM, Bob Dionne b...@cloudant.com wrote: Congrats Dave! On Jan 6, 2012, at 4:15 PM, Jan Lehnardt wrote: Hey all, I am happy to report that the CouchDB PMC has elected Dave Cottlehuber to become a CouchDB committer. His election recognises his outstanding contributions to the Windows build system and other parts of the code as well as his community engagement. Dave, it is an honour to have you on board :) Everybody, please give a warm welcome to to Dave! Cheers Jan -- -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: proposal: Using the db hooks in other databases
While the use case for the system databases is clear, I wonder what are the possible motivations to do it for regular databases. Does anyone has real world scenarios/examples where this level of customization is useful and can't be done with the existing infrastructure? On Wed, Jan 4, 2012 at 10:31 PM, Benoit Chesneau bchesn...@gmail.com wrote: On Wed, Jan 4, 2012 at 11:11 PM, Randall Leeds randall.le...@gmail.com wrote: On Wed, Jan 4, 2012 at 05:52, Benoit Chesneau bchesn...@gmail.com wrote: I've put a more readable and updated version here : https://github.com/refuge/refuge/wiki/DB-Security-hooks On Wed, Jan 4, 2012 at 10:52 AM, Benoit Chesneau bchesn...@gmail.com wrote: Hi all, Following is a quick proposal to implement support of hooks to other databases. Let me know what you think about it. I would be be happy to start the work on that. Thanks for getting this conversation going. I support the idea. You and I discussed a database erlang behaviour and IIRC you did some work on it. Did you go further? I wonder if a default behaviour is to delegate to design docs and if any common behaviour module could simplify code for _users, _replicator, and _design-based. I am already getting too into implementation, though. +1 on the overall idea. -Randall Thanks for the support. I've tried this idea of having an erlang behaviour for that indeed. To summarize, the idea was about having a module behaviour (in erlang meaning) exposing 4 erlang functions: - before_doc_update - after_doc_read - on_all_docs - on_changes The main problem after that was solving the association of this module or functions to a database. Main idea was having this association done in the ini file. Something like: [security] dbname = modulename After trying it I don't like so much this idea. The main advantage is the speed right now. But this doesn't give any power for the design doc writer and it may not be scalable when we speak about N dbs where N1000 . On this other hand if such config is found in a db we have the following advantages: - replication of such rules - rules are now dynamic and can be changed easily, you just have to upload a new version of the design doc. The actual problem of such solution may be the speed of the JS processing if the functions are made in JS (same problem in validate_update though) . This problem can be solved later anyway by improving the JS engine. One other solution would be creating some kind of DSL evaluated in ERLANG like we do for the rewriter: Imo the idea of having functions in the design docs may be the simpler actually. - benoît -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: proposal: Using the db hooks in other databases
On Wed, Jan 4, 2012 at 10:38 PM, Benoit Chesneau bchesn...@gmail.com wrote: On Wed, Jan 4, 2012 at 11:35 PM, Filipe David Manana fdman...@apache.org wrote: While the use case for the system databases is clear, I wonder what are the possible motivations to do it for regular databases. Does anyone has real world scenarios/examples where this level of customization is useful and can't be done with the existing infrastructure? Yes, read my example. Ok, I see. I wonder if adding on_changes and on_all_docs isn't a bit too much complicated from a user's point of view. Following that reasoning, there should be a on_view_with_include_docs_true. The after_read_doc callback should work for these 3 cases when ?include_docs=true is specified. - benoît -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: git commit: add support of erlang R15B in oauth.
Any reason to not apply this change to branches 1.2.x and 1.1.x?
I see this small patch was applied upstream as well.
On Mon, Jan 2, 2012 at 4:39 PM, beno...@apache.org wrote:
Updated Branches:
refs/heads/master 389142e0e - adb62ce3a
add support of erlang R15B in oauth.
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/adb62ce3
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/adb62ce3
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/adb62ce3
Branch: refs/heads/master
Commit: adb62ce3a8cff25af7d0329e1a3790c389ad74e1
Parents: 389142e
Author: benoitc beno...@apache.org
Authored: Mon Jan 2 17:39:01 2012 +0100
Committer: benoitc beno...@apache.org
Committed: Mon Jan 2 17:39:01 2012 +0100
--
src/erlang-oauth/oauth_uri.erl | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/adb62ce3/src/erlang-oauth/oauth_uri.erl
--
diff --git a/src/erlang-oauth/oauth_uri.erl b/src/erlang-oauth/oauth_uri.erl
index 5023f98..aebf093 100644
--- a/src/erlang-oauth/oauth_uri.erl
+++ b/src/erlang-oauth/oauth_uri.erl
@@ -9,6 +9,8 @@
normalize(URI) -
case http_uri:parse(URI) of
+ {ok, {Scheme, UserInfo, Host, Port, Path, _Query}} - % R15B
+ normalize(Scheme, UserInfo, string:to_lower(Host), Port, [Path]);
{Scheme, UserInfo, Host, Port, Path, _Query} -
normalize(Scheme, UserInfo, string:to_lower(Host), Port, [Path]);
Else -
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: Understanding the CouchDB file format
On Tue, Dec 20, 2011 at 6:24 PM, Riyad Kalla rka...@gmail.com wrote: I've been reading everything I can find on the CouchDB file format[1] and am getting bits and pieces here and there, but not a great, concrete, step-by-step explanation of the process. I'm clear on the use of B+ trees and after reading a few papers on the benefits of log-structured file formats, I understand the benefits of inlining the B+ tree indices directly into the data file as well (locality + sequential I/O)... what I'm flummoxed about is how much of the B+ tree's index is rewritten after every modified document. Consider a CouchDB file that looks more or less like this: [idx/header][doc1, rev1][idx/header][doc1, rev2] After each revised doc is written and the b-tree root is rewritten after that, is that just a modified root node of the B+ tree or the entire B+ tree? The reason I ask is because regardless of the answer to my previous question, for a *huge* database will millions of records, that seems like an enormous amount of data to rewrite after every modification. Say the root node had a fanning factor of 133; that would still be alot of data to rewrite. Hi Riyad, Have you observed that in practice? Typically the depth of database btrees is not that high even for millions of documents. For example I have one around with about 10 million documents which doesn't have more than 5 or 6 levels if I recall correctly. So updating a doc, for that particular case, means rewriting 5 or 6 new nodes plus the document itself. Each node is normally not much bigger than 1.2Kb. I've written once a tool to analyze database files which reports btree depths, however it's not updated to work with recent changes on master/1.2.x such as snappy compression and btree sizes: https://github.com/fdmanana/couchfoo It should work with CouchDB 1.1 (and older) database files. I am certain I am missing the boat on this; if anyone can pull me out of the water and point me to dry land I'd appreciate it. Best, R [1] -- http://jchrisa.net/drl/_design/sofa/_list/post/post-page?startkey=%5B%22CouchDB-Implements-a-Fundamental-Algorithm%22%5D -- http://horicky.blogspot.com/2008/10/couchdb-implementation.html -- http://blog.kodekabuki.com/post/132952897/couchdb-naked -- http://guide.couchdb.org/editions/1/en/btree.html -- http://ayende.com/blog/* (Over my head) -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: Understanding the CouchDB file format
On Tue, Dec 20, 2011 at 8:27 PM, Riyad Kalla rka...@gmail.com wrote: Filipe, Thank you for the reply. Maybe I am misunderstanding exactly what couch is writing out; the docs I've read say that it rewrites the root node -- I can't tell if the docs mean the parent node of the child doc that was changed (as one of the b+ leaves) or if it means the direct path, from the root, to that individual doc... or if it means the *entire* index... In the case of even rewriting the single parent, with such a shallow tree, each internal leaf will have a huge fan of nodes; let's say 1-10k in a decent sized data set. The leaf nodes don't contain the documents, but rather pointers (file offsets) to where the documents are stored in the file. Non-leaf nodes contain a list of pointers to child nodes. Updating a document means changing a value in its leaf node, writing the modified version to disk, than update its parent node to refer to the offset of the new leaf node, the parent of the parent, etc until you write a new root node. Maybe I misunderstood your doubt, but that doesn't seem a lot to me (remember each node is typically about 1.2K). However a doc update implies updating 2 btrees (seq and id trees). If you are seeing a few K of extra written out after each changed doc then that cannot be write... I almost assumed my understanding was wrong because the sheer volume of data would make performance abysmal if it was true. Given that... is it just the changed path, from the root to the new leaf that is rewritten? That makes me all sorts of curious as to how Couch updates/searches the new modified index with the small diff that is written out. Any pointers to reading that will help me dig down on this (even early bugs in JIRA?) would be appreciated. I've tried skimming back in 2007/08 on Damien's blog to see if it wrote about it in depth and so far haven't found anything as detailed as I am hoping for on this architecture. Just read couch_btree.erl, couch_file.erl and couch_db_updater.erl. Best, Riyad On Tue, Dec 20, 2011 at 1:07 PM, Filipe David Manana fdman...@apache.orgwrote: On Tue, Dec 20, 2011 at 6:24 PM, Riyad Kalla rka...@gmail.com wrote: I've been reading everything I can find on the CouchDB file format[1] and am getting bits and pieces here and there, but not a great, concrete, step-by-step explanation of the process. I'm clear on the use of B+ trees and after reading a few papers on the benefits of log-structured file formats, I understand the benefits of inlining the B+ tree indices directly into the data file as well (locality + sequential I/O)... what I'm flummoxed about is how much of the B+ tree's index is rewritten after every modified document. Consider a CouchDB file that looks more or less like this: [idx/header][doc1, rev1][idx/header][doc1, rev2] After each revised doc is written and the b-tree root is rewritten after that, is that just a modified root node of the B+ tree or the entire B+ tree? The reason I ask is because regardless of the answer to my previous question, for a *huge* database will millions of records, that seems like an enormous amount of data to rewrite after every modification. Say the root node had a fanning factor of 133; that would still be alot of data to rewrite. Hi Riyad, Have you observed that in practice? Typically the depth of database btrees is not that high even for millions of documents. For example I have one around with about 10 million documents which doesn't have more than 5 or 6 levels if I recall correctly. So updating a doc, for that particular case, means rewriting 5 or 6 new nodes plus the document itself. Each node is normally not much bigger than 1.2Kb. I've written once a tool to analyze database files which reports btree depths, however it's not updated to work with recent changes on master/1.2.x such as snappy compression and btree sizes: https://github.com/fdmanana/couchfoo It should work with CouchDB 1.1 (and older) database files. I am certain I am missing the boat on this; if anyone can pull me out of the water and point me to dry land I'd appreciate it. Best, R [1] -- http://jchrisa.net/drl/_design/sofa/_list/post/post-page?startkey=%5B%22CouchDB-Implements-a-Fundamental-Algorithm%22%5D -- http://horicky.blogspot.com/2008/10/couchdb-implementation.html -- http://blog.kodekabuki.com/post/132952897/couchdb-naked -- http://guide.couchdb.org/editions/1/en/btree.html -- http://ayende.com/blog/* (Over my head) -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men. -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [2/2] git commit: Fix OAuth authentication with VHosts + URL rewriting
On Monday, December 12, 2011, Randall Leeds randall.le...@gmail.com wrote: I'm missing something. Is there any place where pre_rewrite_user_ctx is read? I only see put and erase calls. Yes. You miss the fact that erase returns the current value (or undefined). On Sat, Dec 10, 2011 at 12:11, Filipe David Manana fdman...@apache.org wrote: On Sat, Dec 10, 2011 at 8:08 PM, Benoit Chesneau bchesn...@gmail.com wrote: what is the point of wariting in the process registry? To make it simple, not adding a new handle_request_int clause to couch_httpd or a new entry point. On Sat, Dec 10, 2011 at 9:03 PM, fdman...@apache.org wrote: Fix OAuth authentication with VHosts + URL rewriting The OAuth handler was not getting the right path (the one the client used to compute its OAuth signature) to verify the client's signature. The right path is the one from before doing the VHost dispatch. Secondly, after the OAuth handler succeeds, the rewriter kicks in and calls couch_httpd:handle_request_int/5 with a new mochiweb request which contains the rewritten patch. This will cause all the authentication handlers to run again, which makes the OAuth handler fail this second time because it gets a rewritten patch. COUCHDB-1320 Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/b86fa1f6 Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/b86fa1f6 Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/b86fa1f6 Branch: refs/heads/1.2.x Commit: b86fa1f6bedee9d441bf4cac53c2794a60c69216 Parents: 25754ac Author: Filipe David Borba Manana fdman...@apache.org Authored: Sat Dec 10 19:05:52 2011 + Committer: Filipe David Borba Manana fdman...@apache.org Committed: Sat Dec 10 19:40:37 2011 + -- src/couchdb/couch_httpd.erl |3 +- src/couchdb/couch_httpd_oauth.erl | 11 +++- src/couchdb/couch_httpd_rewrite.erl |4 +- test/etap/160-vhosts.t | 89 +- 4 files changed, 102 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/couchdb/blob/b86fa1f6/src/couchdb/couch_httpd.erl -- diff --git a/src/couchdb/couch_httpd.erl b/src/couchdb/couch_httpd.erl index 11b0bca..2d4c38d 100644 --- a/src/couchdb/couch_httpd.erl +++ b/src/couchdb/couch_httpd.erl @@ -298,7 +298,8 @@ handle_request_int(MochiReq, DefaultFun, db_url_handlers = DbUrlHandlers, design_url_handlers = DesignUrlHandlers, default_fun = DefaultFun, -url_handlers = UrlHandlers +url_handlers = UrlHandlers, +user_ctx = erlang:erase(pre_rewrite_user_ctx) }, HandlerFun = couch_util:dict_find(HandlerKey, UrlHandlers, DefaultFun),
Re: [2/2] git commit: Fix OAuth authentication with VHosts + URL rewriting
]}}.
main(_) -
test_util:init_code_path(),
- etap:plan(15),
+ etap:plan(18),
case (catch test()) of
ok -
etap:end_tests();
@@ -135,9 +135,11 @@ test() -
test_vhost_request_path2(),
test_vhost_request_path3(),
test_vhost_request_to_root(),
+ test_vhost_request_with_oauth(Db),
%% restart boilerplate
couch_db:close(Db),
+ ok = couch_server:delete(couch_db:name(Db), [admin_user_ctx()]),
timer:sleep(3000),
couch_server_sup:stop(),
@@ -301,3 +303,88 @@ test_vhost_request_to_root() -
etap:is(HasCouchDBWelcome, true, should allow redirect to /);
_Else - etap:is(false, true, ibrowse fail)
end.
+
+test_vhost_request_with_oauth(Db) -
+ {ok, AuthDb} = couch_db:create(
+ tap_test_sec_db, [admin_user_ctx(), overwrite]),
+ PrevAuthDbName = couch_config:get(couch_httpd_auth,
authentication_db),
+ couch_config:set(couch_httpd_auth, authentication_db,
tap_test_sec_db, false),
+ couch_config:set(oauth_token_users, otoksec1, joe, false),
+ couch_config:set(oauth_consumer_secrets, consec1, foo, false),
+ couch_config:set(oauth_token_secrets, otoksec1, foobar, false),
+ couch_config:set(couch_httpd_auth, require_valid_user, true,
false),
+
+ DDoc = couch_doc:from_json_obj({[
+ {_id, _design/test},
+ {language, javascript},
+ {rewrites, [
+ {[
+ {from, foobar},
+ {to, _info}
+ ]}
+ ]}
+ ]}),
+ {ok, _} = couch_db:update_doc(Db, DDoc, []),
+
+ RewritePath = /etap-test-db/_design/test/_rewrite/foobar,
+ ok = couch_config:set(vhosts, oauth-example.com, RewritePath,
false),
+ couch_httpd_vhost:reload(),
+
+ case ibrowse:send_req(server(), [], get, [], [{host_header,
oauth-example.com}]) of
+ {ok, 401, _, Body} -
+ {JsonBody} = ejson:decode(Body),
+ etap:is(
+ couch_util:get_value(error, JsonBody),
+ unauthorized,
+ Request without OAuth credentials failed);
+ Error -
+ etap:bail(Request without OAuth credentials did not fail: ++
+ couch_util:to_list(Error))
+ end,
+
+ JoeDoc = couch_doc:from_json_obj({[
+ {_id, org.couchdb.user:joe},
+ {type, user},
+ {name, joe},
+ {roles, []},
+ {password_sha,
fe95df1ca59a9b567bdca5cbaf8412abd6e06121},
+ {salt, 4e170ffeb6f34daecfd814dfb4001a73}
+ ]}),
+ {ok, _} = couch_db:update_doc(AuthDb, JoeDoc, []),
+
+ Url = http://oauth-example.com/;,
+ Consumer = {consec1, foo, hmac_sha1},
+ SignedParams = oauth:signed_params(
+ GET, Url, [], Consumer, otoksec1, foobar),
+ OAuthUrl = oauth:uri(server(), SignedParams),
+
+ case ibrowse:send_req(OAuthUrl, [], get, [], [{host_header,
oauth-example.com}]) of
+ {ok, 200, _, Body2} -
+ {JsonBody2} = ejson:decode(Body2),
+ etap:is(couch_util:get_value(name, JsonBody2), test,
+ should return ddoc info with OAuth credentials);
+ Error2 -
+ etap:bail(Failed to access vhost with OAuth credentials: ++
+ couch_util:to_list(Error2))
+ end,
+
+ Consumer2 = {consec1, bad_secret, hmac_sha1},
+ SignedParams2 = oauth:signed_params(
+ GET, Url, [], Consumer2, otoksec1, foobar),
+ OAuthUrl2 = oauth:uri(server(), SignedParams2),
+
+ case ibrowse:send_req(OAuthUrl2, [], get, [], [{host_header,
oauth-example.com}]) of
+ {ok, 401, _, Body3} -
+ {JsonBody3} = ejson:decode(Body3),
+ etap:is(
+ couch_util:get_value(error, JsonBody3),
+ unauthorized,
+ Request with bad OAuth credentials failed);
+ Error3 -
+ etap:bail(Failed to access vhost with bad OAuth credentials:
++
+ couch_util:to_list(Error3))
+ end,
+
+ couch_config:set(couch_httpd_auth, authentication_db,
PrevAuthDbName, false),
+ couch_config:set(couch_httpd_auth, require_valid_user, false,
false),
+ ok = couch_server:delete(couch_db:name(AuthDb), [admin_user_ctx()]).
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: git commit: Fix distcheck make target
On Tue, Dec 6, 2011 at 1:27 AM, Noah Slater nsla...@tumbolia.org wrote: Can you prefix these paths with $(top_builddir)/ please? Done. Thanks for the top Noah. On Mon, Dec 5, 2011 at 4:58 PM, fdman...@apache.org wrote: Updated Branches: refs/heads/master b68edd1bb - 8f38cbed2 Fix distcheck make target The check target now starts the utils/run script to execute the JavaScript test suite. This script leaves the files couchdb.stdout and couchdb.stderr which caused the distcheck target to fail. Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/8f38cbed Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/8f38cbed Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/8f38cbed Branch: refs/heads/master Commit: 8f38cbed2b952e3eb27e36deeb59c15abdad344a Parents: b68edd1 Author: Filipe David Borba Manana fdman...@apache.org Authored: Mon Dec 5 16:55:46 2011 + Committer: Filipe David Borba Manana fdman...@apache.org Committed: Mon Dec 5 16:55:46 2011 + -- Makefile.am | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) -- http://git-wip-us.apache.org/repos/asf/couchdb/blob/8f38cbed/Makefile.am -- diff --git a/Makefile.am b/Makefile.am index 01879dc..98c7be9 100644 --- a/Makefile.am +++ b/Makefile.am @@ -113,6 +113,8 @@ distclean-local: rm -fr $(top_builddir)/etc/couchdb/default.d rm -fr $(top_builddir)/etc/couchdb/local.d rm -fr $(top_builddir)/tmp + rm -f couchdb.stdout + rm -f couchdb.stderr .PHONY: local-clean local-clean: maintainer-clean -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: all_dbs_active error and continuous replications
On Tue, Dec 6, 2011 at 9:29 AM, Gregor Martynus gre...@martynus.net wrote: What I really don't understand is, _why_ the continuous replication keep the db files open? I'm not an Erlang developer at all, sorry if this is a dump question/thought, but shouldn't the continuous replication be triggered only if a change happens in the db, instead of keeping it alive all the time? Is this a bug or probably something that might be improved in future? Actually it doesn't keep the source and database open when there's no changes to transfer from the source to the target. What happens is that when started, the replication records the databases' instance start times - they're timestamps that tell when the databases where open by the couchdb server. Whenever the replication commits a checkpoint (it is committed against the source and the target), it verifies if the endpoints' current instance start times are the same as when it started. If they're not the same, then you'll get that max_dbs_open error message from the replicator. When the system has max_dbs_open and a request to open another database X comes, it will close the least recently used database Y in order to open the new one. If later on a request to open the database X that was previously closed (because we reached max_dbs_open) arrives, the new least recently used database will be closed, and X will be reopened - at this point it gets a new instance start timestamp. The reason to abort if the instance start timestamps changed, it to prevent you from getting into an inconsistent state. For example, during an idle period (when there's no changes to transfer from source to target) someone could swap database files - if the replicator ignored the timestamps, it could mean in the end your target would be missing all the changes transferred before the file swap. One thing I have been thinking for sometime is to have the server remember the timestamps and header md5 of each database opened since it started. When it reopens a database that was previously closed by the LRU system (due to max_dbs_open), it would use the cached timestamp if the database's header md5 is the same as the one that's cached along with the timestamp. I haven't convinced myself this is a flawless/good idea however and never made extensive tests. Thanks × Gregor On Tue, Dec 6, 2011 at 10:16 AM, Benoit Chesneau bchesn...@gmail.comwrote: On Tue, Dec 6, 2011 at 10:11 AM, Gregor Martynus gre...@martynus.net wrote: Thanks Benoit, I know about this setting. But at some point, this won't work anymore. My question is if there is any way to stop continuous replications to keep the db files active changes have been replicated? Does that make sense? other way would be to queue changes events and when replicated dbs max dbs open a new task if a change happened on this . Something like it. Otherwise to answer to your question, if you use the _replicator db you can interrupt a task by just deleting the doc. - benoît -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: builbot failure
On Mon, Dec 5, 2011 at 4:19 PM, Paul Davis paul.joseph.da...@gmail.com wrote: This means we're missing them in a CLEAN_FILES directive in Makefile.am Yeah, just realized that make distcheck is in fact broken since the cli tests were added to the make check target. /utils/run is what is producing those 2 files. This seems to do it: diff --git a/Makefile.am b/Makefile.am index 01879dc..98c7be9 100644 --- a/Makefile.am +++ b/Makefile.am @@ -113,6 +113,8 @@ distclean-local: rm -fr $(top_builddir)/etc/couchdb/default.d rm -fr $(top_builddir)/etc/couchdb/local.d rm -fr $(top_builddir)/tmp + rm -f couchdb.stdout + rm -f couchdb.stderr .PHONY: local-clean local-clean: maintainer-clean On Mon, Dec 5, 2011 at 10:49 AM, Filipe David Manana fdman...@apache.org wrote: Just fixed a misspelled filename in Makefile.am. However still not enough for make distcheck: ERROR: files left in build directory after distclean: ./couchdb.stderr ./couchdb.stdout make[1]: *** [distcleancheck] Error 1 make: *** [distcheck] Error 2 It doesn't give me any other more helpful error. Something seems to be missing in the new Makefile.am (just a guess). On Mon, Dec 5, 2011 at 10:55 AM, Benoit Chesneau bchesn...@gmail.com wrote: Hi, I've pushed couch_replicator code online. Tests are running well locally but the bot fail. Any idea what's going on ? The logs in buildbot don't help at all btw. Maybe there should be a way to launch tests in verbose mode ? - benoît -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men. -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [2/5] git commit: use couch_db calls instead of couch_btree
Randall, I have to disagree on this one.
The reason to not call couch_db:get_design_docs/1 is to avoid reading
the body of the documents, which is not needed.
Plus, couch_db:get_design_docs/1 skips deleted documents, which will
causes _changes rows to be skipped.
On Wed, Nov 9, 2011 at 10:07 AM, rand...@apache.org wrote:
use couch_db calls instead of couch_btree
couch_changes can use the interface from couch_db to access documents,
keeping the logic of the btree storage better encapsulated.
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/e41d226d
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/e41d226d
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/e41d226d
Branch: refs/heads/master
Commit: e41d226dca11dc8b24d6c011cc3014e6c553db42
Parents: 13f704b
Author: Randall Leeds rand...@apache.org
Authored: Wed Nov 9 01:31:18 2011 -0800
Committer: Randall Leeds rand...@apache.org
Committed: Wed Nov 9 01:42:40 2011 -0800
--
src/couchdb/couch_changes.erl | 9 ++---
1 files changed, 2 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/e41d226d/src/couchdb/couch_changes.erl
--
diff --git a/src/couchdb/couch_changes.erl b/src/couchdb/couch_changes.erl
index 267f3d7..30f1d88 100644
--- a/src/couchdb/couch_changes.erl
+++ b/src/couchdb/couch_changes.erl
@@ -298,7 +298,7 @@ send_changes(Args, Callback, UserAcc, Db, StartSeq,
Prepend, FirstRound) -
send_changes_doc_ids(DocIds, Db, StartSeq, Dir, Fun, Acc0) -
- Lookups = couch_btree:lookup(Db#db.fulldocinfo_by_id_btree, DocIds),
+ Lookups = couch_db:get_full_doc_infos(Db, DocIds),
FullDocInfos = lists:foldl(
fun({ok, FDI}, Acc) -
[FDI | Acc];
@@ -310,12 +310,7 @@ send_changes_doc_ids(DocIds, Db, StartSeq, Dir, Fun,
Acc0) -
send_changes_design_docs(Db, StartSeq, Dir, Fun, Acc0) -
- FoldFun = fun(FullDocInfo, _, Acc) -
- {ok, [FullDocInfo | Acc]}
- end,
- KeyOpts = [{start_key, _design/}, {end_key_gt, _design0}],
- {ok, _, FullDocInfos} = couch_btree:fold(
- Db#db.fulldocinfo_by_id_btree, FoldFun, [], KeyOpts),
+ FullDocInfos = couch_db:get_design_docs(Db),
send_lookup_changes(FullDocInfos, StartSeq, Dir, Db, Fun, Acc0).
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: [5/5] git commit: fix improper comparison on filtered changes
Randall, which test exactly was failing in changes.js? I've never had such failure locally. If it's due to browser caching, it would be a good idea to add a test to test/etap/073-changes.t On Wed, Nov 9, 2011 at 10:07 AM, rand...@apache.org wrote: fix improper comparison on filtered changes Regression caught by the changes js test. Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/6213d064 Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/6213d064 Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/6213d064 Branch: refs/heads/master Commit: 6213d064ef2467be9e6673df0fa71ffa9821ba4a Parents: 866769f Author: Randall Leeds rand...@apache.org Authored: Wed Nov 9 01:16:36 2011 -0800 Committer: Randall Leeds rand...@apache.org Committed: Wed Nov 9 01:40:56 2011 -0800 -- src/couchdb/couch_changes.erl | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/couchdb/blob/6213d064/src/couchdb/couch_changes.erl -- diff --git a/src/couchdb/couch_changes.erl b/src/couchdb/couch_changes.erl index ab95481..267f3d7 100644 --- a/src/couchdb/couch_changes.erl +++ b/src/couchdb/couch_changes.erl @@ -328,7 +328,7 @@ send_lookup_changes(FullDocInfos, StartSeq, Dir, Db, Fun, Acc0) - end, GreaterFun = case Dir of fwd - - fun(A, B) - A = B end; + fun(A, B) - A B end; rev - fun(A, B) - A = B end end, -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [2/5] git commit: use couch_db calls instead of couch_btree
On Wed, Nov 9, 2011 at 7:00 PM, Randall Leeds randall.le...@gmail.com wrote: On Wed, Nov 9, 2011 at 02:48, Robert Newson rnew...@apache.org wrote: Randall, Did we miss the chance to review this work before it landed? Yes. Sorry. I thought it was more trivial than it was. I owe it to Filipe for catching the mistake. While I changed get_design_docs not to get the body by default (I like this change), he's correct that it skips deleted docs and that would break _changes. No sweat. I noticed after getting back home that there's a subsequent commit that change couch_db:get_design_docs/1 which makes it return only the full_doc_info records. Avoiding doing this directly in couch_changes is fine for me. The intention is good. Agree with Benoit that the couch_db function could take parameters to tell if deleted docs are to be returned or not and opt for docs or just full_doc_infos. thanks My apologies, Bob. I've just reverted it. B. On 9 November 2011 10:36, Benoit Chesneau bchesn...@gmail.com wrote: On Wed, Nov 9, 2011 at 11:27 AM, Filipe David Manana fdman...@apache.org wrote: Randall, I have to disagree on this one. The reason to not call couch_db:get_design_docs/1 is to avoid reading the body of the documents, which is not needed. Plus, couch_db:get_design_docs/1 skips deleted documents, which will causes _changes rows to be skipped. maybe couch_db:get_design_docs could take some options to handle the case. I think it's a good idea to use couch_db as abstraction to the deep level. - benoit -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [5/5] git commit: fix improper comparison on filtered changes
On Wed, Nov 9, 2011 at 6:43 PM, Randall Leeds randall.le...@gmail.com wrote: On Wed, Nov 9, 2011 at 02:51, Filipe David Manana fdman...@apache.org wrote: Randall, which test exactly was failing in changes.js? It was the andmore-only section near the end: T(line.seq == 8) T(line.id == andmore) The problem is that seq _should_ be strictly greater than since. If you look at couch_db:changes_since/5 and related functions, that module adds one to the sequence number in order to get this behavior. Only when using filters was this broken and became =. I think you're right. Looking at the test it seems it should always fail, maybe I add a browser cache issue. I've never had such failure locally. If it's due to browser caching, it would be a good idea to add a test to test/etap/073-changes.t I can add an etap test. I'm positive it's not browser caching because I identified and fixed it. Also, because I just committed a change to improve Futon's browser caching :). Great! :) On Wed, Nov 9, 2011 at 10:07 AM, rand...@apache.org wrote: fix improper comparison on filtered changes Regression caught by the changes js test. Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/6213d064 Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/6213d064 Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/6213d064 Branch: refs/heads/master Commit: 6213d064ef2467be9e6673df0fa71ffa9821ba4a Parents: 866769f Author: Randall Leeds rand...@apache.org Authored: Wed Nov 9 01:16:36 2011 -0800 Committer: Randall Leeds rand...@apache.org Committed: Wed Nov 9 01:40:56 2011 -0800 -- src/couchdb/couch_changes.erl | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/couchdb/blob/6213d064/src/couchdb/couch_changes.erl -- diff --git a/src/couchdb/couch_changes.erl b/src/couchdb/couch_changes.erl index ab95481..267f3d7 100644 --- a/src/couchdb/couch_changes.erl +++ b/src/couchdb/couch_changes.erl @@ -328,7 +328,7 @@ send_lookup_changes(FullDocInfos, StartSeq, Dir, Db, Fun, Acc0) - end, GreaterFun = case Dir of fwd - - fun(A, B) - A = B end; + fun(A, B) - A B end; rev - fun(A, B) - A = B end end, -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men. -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [VOTE] Apache CouchDB 1.1.1 Release, Round 3
+1 All tests pass, signatures match. Mac OS X Lion, OTP R14B04, Chrome browser. On Mon, Oct 24, 2011 at 11:11 AM, Robert Newson rnew...@apache.org wrote: This is the third release vote for Apache CouchDB 1.1.1 Changes since round 2; * Remove basename() dependency, fixing Windows build. Git Note: While there is no current 1.1.1 tag in our git repository, there was such a tag during round 2, which you may still have locally. It will be created again when a round is successful, and a 'git pull --tags' will correctly update your tag. It is advisable, however, to simply delete this tag locally with 'git tag -d 1.1.1'. Future releases will not create any interim tags, so this is a one-time procedure. We encourage the whole community to download and test these release artifacts so that any critical issues can be resolved before the release is made. Everyone is free to vote on this release. Please report your results and vote to this thread. We are voting on the following release artifacts: http://people.apache.org/~rnewson/dist/1.1.1/ Instructions for validating the release tarball can be found here: http://people.apache.org/~rnewson/dist/ Instructions for testing the build artefacts can be found here: http://wiki.apache.org/couchdb/Test_procedure These artifacts have been built from: af7a6ae52890c6c158615068f2a48d40c52ec538 apache-couchdb-1.1.1.tar.gz apache-couchdb-1.1.1.tar.gz.md5 apache-couchdb-1.1.1.tar.gz.asc apache-couchdb-1.1.1.tar.gz.sha Test ALL the things. -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [VOTE] Apache CouchDB 1.1.1 Release, Round 3
On Fri, Oct 28, 2011 at 12:54 PM, Benoit Chesneau bchesn...@gmail.com wrote: On Fri, Oct 28, 2011 at 12:41 PM, Filipe David Manana fdman...@apache.org wrote: +1 All tests pass, signatures match. Mac OS X Lion, OTP R14B04, Chrome browser. Which ssl version? Still the bug? OpenSSL 0.9.8 built from source, exactly like it is described here: https://gist.github.com/1199903 - benoît -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [VOTE] Apache CouchDB 1.1.1 Release
On Thu, Oct 20, 2011 at 2:12 PM, Benoit Chesneau bchesn...@gmail.com wrote: On Wed, Oct 19, 2011 at 4:27 PM, Robert Newson rnew...@apache.org wrote: This is the release vote for Apache CouchDB 1.1.1 Changes in this release: * Support SpiderMonkey 1.8.5 * Add configurable maximum to the number of bytes returned by _log. * Allow CommonJS modules to be an empty string. * Bump minimum Erlang version to R13B02. * Do not run deleted validate_doc_update functions. * ETags for views include current sequence if include_docs=true. * Fix bug where duplicates can appear in _changes feed. * Fix bug where update handlers break after conflict resolution. * Fix bug with _replicator where include filter could crash couch. * Fix crashes when compacting large views. * Fix file descriptor leak in _log * Fix missing revisions in _changes?style=all_docs. * Improve handling of compaction at max_dbs_open limit. * JSONP responses now send text/javascript for Content-Type. * Link to ICU 4.2 on Windows. * Permit forward slashes in path to update functions. * Reap couchjs processes that hit reduce_overflow error. * Status code can be specified in update handlers. * Support provides() in show functions. * _view_cleanup when ddoc has no views now removes all index files. * max_replication_retry_count now supports infinity. * Fix replication crash when source database has a document with empty ID. * Fix deadlock when assigning couchjs processes to serve requests. * Fixes to the document multipart PUT API. We encourage the whole community to download and test these release artifacts so that any critical issues can be resolved before the release is made. Everyone is free to vote on this release. Please report your results and vote to this thread. We are voting on the following release artifacts: http://people.apache.org/~rnewson/dist/1.1.1/ Instructions for validating the release tarball can be found here: http://people.apache.org/~rnewson/dist/ Instructions for testing the build artefacts can be found here: http://wiki.apache.org/couchdb/Test_procedure These artifacts have been built from the 1.1.1 tag in Git: apache-couchdb-1.1.1.tar.gz apache-couchdb-1.1.1.tar.gz.md5 apache-couchdb-1.1.1.tar.gz.asc apache-couchdb-1.1.1.tar.gz.sha Since you have read this far, you MUST vote. make check pass, but when running js tests I got the following error (reproducible from time to time) : [info] [0.1915.0] Stopping all ongoing replications because the replicator database was deleted or changed Apache CouchDB 1.1.1 (LogLevel=info) is starting. Segmentation fault: 11 configuration : rb1404, osx lion (last update) That's likely the OpenSSL issue on Lion. If init:restart/0 is invoked and the crypto application was loaded (CouchDB's case) the VM crashes with either a bus error, segmentation fault or no error message at all. Have you tried building OTP like in https://gist.github.com/1199903 ? I had exactly the same issue. - benoit -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [VOTE] Apache CouchDB 1.1.1 Release
On Thu, Oct 20, 2011 at 2:28 PM, Benoit Chesneau bchesn...@gmail.com wrote: On Thu, Oct 20, 2011 at 3:17 PM, Filipe David Manana fdman...@apache.org wrote: On Thu, Oct 20, 2011 at 2:12 PM, Benoit Chesneau bchesn...@gmail.com wrote: On Wed, Oct 19, 2011 at 4:27 PM, Robert Newson rnew...@apache.org wrote: This is the release vote for Apache CouchDB 1.1.1 Changes in this release: * Support SpiderMonkey 1.8.5 * Add configurable maximum to the number of bytes returned by _log. * Allow CommonJS modules to be an empty string. * Bump minimum Erlang version to R13B02. * Do not run deleted validate_doc_update functions. * ETags for views include current sequence if include_docs=true. * Fix bug where duplicates can appear in _changes feed. * Fix bug where update handlers break after conflict resolution. * Fix bug with _replicator where include filter could crash couch. * Fix crashes when compacting large views. * Fix file descriptor leak in _log * Fix missing revisions in _changes?style=all_docs. * Improve handling of compaction at max_dbs_open limit. * JSONP responses now send text/javascript for Content-Type. * Link to ICU 4.2 on Windows. * Permit forward slashes in path to update functions. * Reap couchjs processes that hit reduce_overflow error. * Status code can be specified in update handlers. * Support provides() in show functions. * _view_cleanup when ddoc has no views now removes all index files. * max_replication_retry_count now supports infinity. * Fix replication crash when source database has a document with empty ID. * Fix deadlock when assigning couchjs processes to serve requests. * Fixes to the document multipart PUT API. We encourage the whole community to download and test these release artifacts so that any critical issues can be resolved before the release is made. Everyone is free to vote on this release. Please report your results and vote to this thread. We are voting on the following release artifacts: http://people.apache.org/~rnewson/dist/1.1.1/ Instructions for validating the release tarball can be found here: http://people.apache.org/~rnewson/dist/ Instructions for testing the build artefacts can be found here: http://wiki.apache.org/couchdb/Test_procedure These artifacts have been built from the 1.1.1 tag in Git: apache-couchdb-1.1.1.tar.gz apache-couchdb-1.1.1.tar.gz.md5 apache-couchdb-1.1.1.tar.gz.asc apache-couchdb-1.1.1.tar.gz.sha Since you have read this far, you MUST vote. make check pass, but when running js tests I got the following error (reproducible from time to time) : [info] [0.1915.0] Stopping all ongoing replications because the replicator database was deleted or changed Apache CouchDB 1.1.1 (LogLevel=info) is starting. Segmentation fault: 11 configuration : rb1404, osx lion (last update) That's likely the OpenSSL issue on Lion. If init:restart/0 is invoked and the crypto application was loaded (CouchDB's case) the VM crashes with either a bus error, segmentation fault or no error message at all. Have you tried building OTP like in https://gist.github.com/1199903 ? I had exactly the same issue. - benoit Shouldn't it be fixed in r144b04? Nop, not fixed in r14b04 unfortunately. And it's independent of the couch codebase, since all start the crypto application. ALso I don't reproduce it at all on refuge when running tests, s it may be due to the way we compile couch. - benoit -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [VOTE] Apache CouchDB 1.1.1 Release
On Thu, Oct 20, 2011 at 2:57 PM, Benoit Chesneau bchesn...@gmail.com wrote: On Thu, Oct 20, 2011 at 3:47 PM, Filipe David Manana fdman...@apache.org wrote: On Thu, Oct 20, 2011 at 2:28 PM, Benoit Chesneau bchesn...@gmail.com wrote: On Thu, Oct 20, 2011 at 3:17 PM, Filipe David Manana fdman...@apache.org wrote: On Thu, Oct 20, 2011 at 2:12 PM, Benoit Chesneau bchesn...@gmail.com wrote: On Wed, Oct 19, 2011 at 4:27 PM, Robert Newson rnew...@apache.org wrote: This is the release vote for Apache CouchDB 1.1.1 Changes in this release: * Support SpiderMonkey 1.8.5 * Add configurable maximum to the number of bytes returned by _log. * Allow CommonJS modules to be an empty string. * Bump minimum Erlang version to R13B02. * Do not run deleted validate_doc_update functions. * ETags for views include current sequence if include_docs=true. * Fix bug where duplicates can appear in _changes feed. * Fix bug where update handlers break after conflict resolution. * Fix bug with _replicator where include filter could crash couch. * Fix crashes when compacting large views. * Fix file descriptor leak in _log * Fix missing revisions in _changes?style=all_docs. * Improve handling of compaction at max_dbs_open limit. * JSONP responses now send text/javascript for Content-Type. * Link to ICU 4.2 on Windows. * Permit forward slashes in path to update functions. * Reap couchjs processes that hit reduce_overflow error. * Status code can be specified in update handlers. * Support provides() in show functions. * _view_cleanup when ddoc has no views now removes all index files. * max_replication_retry_count now supports infinity. * Fix replication crash when source database has a document with empty ID. * Fix deadlock when assigning couchjs processes to serve requests. * Fixes to the document multipart PUT API. We encourage the whole community to download and test these release artifacts so that any critical issues can be resolved before the release is made. Everyone is free to vote on this release. Please report your results and vote to this thread. We are voting on the following release artifacts: http://people.apache.org/~rnewson/dist/1.1.1/ Instructions for validating the release tarball can be found here: http://people.apache.org/~rnewson/dist/ Instructions for testing the build artefacts can be found here: http://wiki.apache.org/couchdb/Test_procedure These artifacts have been built from the 1.1.1 tag in Git: apache-couchdb-1.1.1.tar.gz apache-couchdb-1.1.1.tar.gz.md5 apache-couchdb-1.1.1.tar.gz.asc apache-couchdb-1.1.1.tar.gz.sha Since you have read this far, you MUST vote. make check pass, but when running js tests I got the following error (reproducible from time to time) : [info] [0.1915.0] Stopping all ongoing replications because the replicator database was deleted or changed Apache CouchDB 1.1.1 (LogLevel=info) is starting. Segmentation fault: 11 configuration : rb1404, osx lion (last update) That's likely the OpenSSL issue on Lion. If init:restart/0 is invoked and the crypto application was loaded (CouchDB's case) the VM crashes with either a bus error, segmentation fault or no error message at all. Have you tried building OTP like in https://gist.github.com/1199903 ? I had exactly the same issue. - benoit Shouldn't it be fixed in r144b04? Nop, not fixed in r14b04 unfortunately. And it's independent of the couch codebase, since all start the crypto application. ALso I don't reproduce it at all on refuge when running tests, s it may be due to the way we compile couch. - benoit -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men. Like I said i've no problem on refuge though. Yes all apps starts crypt but here: 2 crypto:start(). ok 3 init:restart(). ok 4 Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.8.5 (abort with ^G) 1 works. So it maybe not related to crypto or openssl. The openssl issue didn't happen always, only sometimes. Try running the following bash loop against trunk/refuge: $ for i in `seq 1 1000`; do curl -s -H 'Content-Type: application/json' -X POST http://localhost:5984/_restart ; sleep 1; done Se then if you get a bus error, seg fault or the vm simply doesn't restart and no error gets sent to the console. - benoit -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [VOTE] Apache CouchDB 1.1.1 Release
On Thu, Oct 20, 2011 at 2:28 PM, Benoit Chesneau bchesn...@gmail.com wrote: On Thu, Oct 20, 2011 at 3:17 PM, Filipe David Manana fdman...@apache.org wrote: On Thu, Oct 20, 2011 at 2:12 PM, Benoit Chesneau bchesn...@gmail.com wrote: On Wed, Oct 19, 2011 at 4:27 PM, Robert Newson rnew...@apache.org wrote: This is the release vote for Apache CouchDB 1.1.1 Changes in this release: * Support SpiderMonkey 1.8.5 * Add configurable maximum to the number of bytes returned by _log. * Allow CommonJS modules to be an empty string. * Bump minimum Erlang version to R13B02. * Do not run deleted validate_doc_update functions. * ETags for views include current sequence if include_docs=true. * Fix bug where duplicates can appear in _changes feed. * Fix bug where update handlers break after conflict resolution. * Fix bug with _replicator where include filter could crash couch. * Fix crashes when compacting large views. * Fix file descriptor leak in _log * Fix missing revisions in _changes?style=all_docs. * Improve handling of compaction at max_dbs_open limit. * JSONP responses now send text/javascript for Content-Type. * Link to ICU 4.2 on Windows. * Permit forward slashes in path to update functions. * Reap couchjs processes that hit reduce_overflow error. * Status code can be specified in update handlers. * Support provides() in show functions. * _view_cleanup when ddoc has no views now removes all index files. * max_replication_retry_count now supports infinity. * Fix replication crash when source database has a document with empty ID. * Fix deadlock when assigning couchjs processes to serve requests. * Fixes to the document multipart PUT API. We encourage the whole community to download and test these release artifacts so that any critical issues can be resolved before the release is made. Everyone is free to vote on this release. Please report your results and vote to this thread. We are voting on the following release artifacts: http://people.apache.org/~rnewson/dist/1.1.1/ Instructions for validating the release tarball can be found here: http://people.apache.org/~rnewson/dist/ Instructions for testing the build artefacts can be found here: http://wiki.apache.org/couchdb/Test_procedure These artifacts have been built from the 1.1.1 tag in Git: apache-couchdb-1.1.1.tar.gz apache-couchdb-1.1.1.tar.gz.md5 apache-couchdb-1.1.1.tar.gz.asc apache-couchdb-1.1.1.tar.gz.sha Since you have read this far, you MUST vote. make check pass, but when running js tests I got the following error (reproducible from time to time) : [info] [0.1915.0] Stopping all ongoing replications because the replicator database was deleted or changed Apache CouchDB 1.1.1 (LogLevel=info) is starting. Segmentation fault: 11 configuration : rb1404, osx lion (last update) That's likely the OpenSSL issue on Lion. If init:restart/0 is invoked and the crypto application was loaded (CouchDB's case) the VM crashes with either a bus error, segmentation fault or no error message at all. Have you tried building OTP like in https://gist.github.com/1199903 ? I had exactly the same issue. - benoit Shouldn't it be fixed in r144b04? \ Nop, not fixed in r14b04 unfortunately. And it's independent of the couch codebase, since all start the crypto application. ALso I don't reproduce it at all on refuge when running tests, s it may be due to the way we compile couch. - benoit
Re: CouchDB 1.1.1
On Mon, Oct 10, 2011 at 9:48 PM, Robert Newson rnew...@apache.org wrote: It's not just oauth_rsa_sha1.erl; ./oauth_http.erl:13: Warning: http:request/4 is deprecated and will be removed in R15B; use httpc:request/4 Yep, mentioned it before, that one is apparently harmless since we don't trigger that codepath in CouchDB. B. On 7 October 2011 18:42, Filipe David Manana fdman...@apache.org wrote: Regarding the erlang-oauth and R15 compatibility, The compilation warnings only happen in the 1.1.x because src/erlang-oauth/Makefile.am is compiling oauth_rsa_sha1.erl while trunk and 1.2.x are not. Our OAuth handler is not supporting rsa-sha1 signatures, so it can be safely ignored. I think the right thing would be to exclude that file from the build list. It was added in the following 1.1.x commit: https://github.com/apache/couchdb/commit/ea57780780730eb5f2d98f30697e6a8c2b3cf7f7 That said, upgrading to the same erlang-oauth we have in trunk/1.2.x won't help. On Thu, Oct 6, 2011 at 7:10 PM, Filipe David Manana fdman...@apache.org wrote: On Thu, Oct 6, 2011 at 6:59 PM, Robert Newson rnew...@apache.org wrote: does the newer erlang-oauth break anything? Not that I know of. On 6 October 2011 18:52, Filipe David Manana fdman...@apache.org wrote: On Thu, Oct 6, 2011 at 6:23 PM, Robert Newson rnew...@apache.org wrote: If there are no objections, I'm going to build the first 1.1.1 candidate in the morning and start a new thread for the release. As pointed out in 2 other related threads, we have some compilation warnings about functions that will no longer exist in OTP R15 (to be released by the end of this year): /opt/r14b03/bin/erlc +debug_info oauth_http.erl ./oauth_http.erl:13: Warning: http:request/4 is deprecated and will be removed in R15B; use httpc:request/4 /opt/r14b03/bin/erlc +debug_info oauth_plaintext.erl /opt/r14b03/bin/erlc +debug_info oauth_rsa_sha1.erl ./oauth_rsa_sha1.erl:9: Warning: public_key:pem_to_der/1: deprecated (will be removed in R15A); use file:read_file/1 and public_key:pem_decode/1 ./oauth_rsa_sha1.erl:10: Warning: public_key:decode_private_key/1 is deprecated and will be removed in R15A; use public_key:pem_entry_decode/1 ./oauth_rsa_sha1.erl:22: Warning: public_key:pem_to_der/1: deprecated (will be removed in R15A); use file:read_file/1 and public_key:pem_decode/1 /opt/r14b03/bin/erlc +debug_info oauth_unix.erl The ones regarding public_key concern me, as it will break the OAuth authentication handler. I see 2 solutions: 1) upgrade erlang-oauth to the same version we have in trunk/1.2.x (doesn't produce these warnings) 2) modify the erlang-oauth in 1.1.x and use try catches where the catch would call the equivalent versions in R14/R15 (these new functions don't exist in R13B03 for example) Naturally, I would prefer 1) The warnings about http:request can be ignored I think, as in Couch we don't use any codepath that will execute the deprecated function B. On 6 October 2011 12:05, Robert Newson rnew...@apache.org wrote: Thanks all, I've pushed the change to 1.1.x. make check and futon all pass; review would still be nice. :) I simply reverted the two commits. B. On 6 October 2011 12:02, Jan Lehnardt j...@apache.org wrote: On Oct 6, 2011, at 10:30 , Robert Newson wrote: There is no build of 1.1.1 on Ubuntu 11.x that will work as well as 1.1.0, so I think it's correct that it cannot build under those conditions. Let's get 1.1.1 out, with the many useful bug fixes and tweaks, and then focus on getting 1.2 out with 1.8.5 support (and BREAKING CHANGES). I vote +1 to removing 1.8.5 support and the paren hack from 1.1.x. +1 Cheers Jan -- B. On 6 October 2011 09:25, Paul Davis paul.joseph.da...@gmail.com wrote: On Thu, Oct 6, 2011 at 3:23 AM, Robert Newson rnew...@apache.org wrote: All, Paul Davis has researched the issue and it seems intractable. I would like to remove 1.8.5 support from 1.1.1. It was not present in 1.1.0 so will not be (officially) missed. The place for a breaking change of this magnitude is 1.2, not a minor bug fix release. Thoughts? B. +1 on removing the paren hack for sure. Not sure about removing 1.8.5 support completely. On the one hand, it would prevent breakage because people couldn't link against the breaking SM. On the other hand, it prevents people from linking against 1.8.5 which means it won't build on Ubuntu 11.x. Unless someone comes up with a magic option I'd say put it to an informal vote so that I can blame someone else. On 5 October 2011 18:25, Paul Davis paul.joseph.da...@gmail.com wrote: Yes, its release blocking. On Wed, Oct 5, 2011 at 7:56 AM, Robert Newson rnew...@apache.org wrote: All, I went through JIRA and updated CHANGES and NEWS on origin/1.1.x to include everything that was missing (Sidenote: Can we all keep this file up to date when commit bugfixes or add features?). I'd appreciate
Re: CouchDB Replication lacking resilience for many database
On Tue, Oct 11, 2011 at 12:03 AM, Chris Stockton chrisstockto...@gmail.com wrote: A couple posts about my struggles thus far are located here [1] and here [2], I have heard from these threads that a individual has been working on major changes to replication, to use a small pool of TCP connections instead of a 1 to 1 ration. I think this would improve our situation drastically, but I was unable to find said work anywhere in SVN or on the web to test or look into it's design concepts. If said work is not actually under development yet, I am willing to put in some development time on the weekends to learn erlang and improve this portion of couchdb, but I do not want to volunteer this time if the work has already been complete. Chris, That said work is in the'1.2.x' branch (and master). CouchDB recently migrated from SVN to GIT, see: http://couchdb.apache.org/community/code.html -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: CouchDB 1.1.1
Regarding the erlang-oauth and R15 compatibility, The compilation warnings only happen in the 1.1.x because src/erlang-oauth/Makefile.am is compiling oauth_rsa_sha1.erl while trunk and 1.2.x are not. Our OAuth handler is not supporting rsa-sha1 signatures, so it can be safely ignored. I think the right thing would be to exclude that file from the build list. It was added in the following 1.1.x commit: https://github.com/apache/couchdb/commit/ea57780780730eb5f2d98f30697e6a8c2b3cf7f7 That said, upgrading to the same erlang-oauth we have in trunk/1.2.x won't help. On Thu, Oct 6, 2011 at 7:10 PM, Filipe David Manana fdman...@apache.org wrote: On Thu, Oct 6, 2011 at 6:59 PM, Robert Newson rnew...@apache.org wrote: does the newer erlang-oauth break anything? Not that I know of. On 6 October 2011 18:52, Filipe David Manana fdman...@apache.org wrote: On Thu, Oct 6, 2011 at 6:23 PM, Robert Newson rnew...@apache.org wrote: If there are no objections, I'm going to build the first 1.1.1 candidate in the morning and start a new thread for the release. As pointed out in 2 other related threads, we have some compilation warnings about functions that will no longer exist in OTP R15 (to be released by the end of this year): /opt/r14b03/bin/erlc +debug_info oauth_http.erl ./oauth_http.erl:13: Warning: http:request/4 is deprecated and will be removed in R15B; use httpc:request/4 /opt/r14b03/bin/erlc +debug_info oauth_plaintext.erl /opt/r14b03/bin/erlc +debug_info oauth_rsa_sha1.erl ./oauth_rsa_sha1.erl:9: Warning: public_key:pem_to_der/1: deprecated (will be removed in R15A); use file:read_file/1 and public_key:pem_decode/1 ./oauth_rsa_sha1.erl:10: Warning: public_key:decode_private_key/1 is deprecated and will be removed in R15A; use public_key:pem_entry_decode/1 ./oauth_rsa_sha1.erl:22: Warning: public_key:pem_to_der/1: deprecated (will be removed in R15A); use file:read_file/1 and public_key:pem_decode/1 /opt/r14b03/bin/erlc +debug_info oauth_unix.erl The ones regarding public_key concern me, as it will break the OAuth authentication handler. I see 2 solutions: 1) upgrade erlang-oauth to the same version we have in trunk/1.2.x (doesn't produce these warnings) 2) modify the erlang-oauth in 1.1.x and use try catches where the catch would call the equivalent versions in R14/R15 (these new functions don't exist in R13B03 for example) Naturally, I would prefer 1) The warnings about http:request can be ignored I think, as in Couch we don't use any codepath that will execute the deprecated function B. On 6 October 2011 12:05, Robert Newson rnew...@apache.org wrote: Thanks all, I've pushed the change to 1.1.x. make check and futon all pass; review would still be nice. :) I simply reverted the two commits. B. On 6 October 2011 12:02, Jan Lehnardt j...@apache.org wrote: On Oct 6, 2011, at 10:30 , Robert Newson wrote: There is no build of 1.1.1 on Ubuntu 11.x that will work as well as 1.1.0, so I think it's correct that it cannot build under those conditions. Let's get 1.1.1 out, with the many useful bug fixes and tweaks, and then focus on getting 1.2 out with 1.8.5 support (and BREAKING CHANGES). I vote +1 to removing 1.8.5 support and the paren hack from 1.1.x. +1 Cheers Jan -- B. On 6 October 2011 09:25, Paul Davis paul.joseph.da...@gmail.com wrote: On Thu, Oct 6, 2011 at 3:23 AM, Robert Newson rnew...@apache.org wrote: All, Paul Davis has researched the issue and it seems intractable. I would like to remove 1.8.5 support from 1.1.1. It was not present in 1.1.0 so will not be (officially) missed. The place for a breaking change of this magnitude is 1.2, not a minor bug fix release. Thoughts? B. +1 on removing the paren hack for sure. Not sure about removing 1.8.5 support completely. On the one hand, it would prevent breakage because people couldn't link against the breaking SM. On the other hand, it prevents people from linking against 1.8.5 which means it won't build on Ubuntu 11.x. Unless someone comes up with a magic option I'd say put it to an informal vote so that I can blame someone else. On 5 October 2011 18:25, Paul Davis paul.joseph.da...@gmail.com wrote: Yes, its release blocking. On Wed, Oct 5, 2011 at 7:56 AM, Robert Newson rnew...@apache.org wrote: All, I went through JIRA and updated CHANGES and NEWS on origin/1.1.x to include everything that was missing (Sidenote: Can we all keep this file up to date when commit bugfixes or add features?). I'd appreciate everyone giving it a look over before I start to build the release artifact. I believe there's an outstanding issue (not present in JIRA) around javascript function evaluation? Can someone confirm that it's release blocking? Thanks, B. -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men
Re: CouchDB 1.1.1
On Thu, Oct 6, 2011 at 6:23 PM, Robert Newson rnew...@apache.org wrote: If there are no objections, I'm going to build the first 1.1.1 candidate in the morning and start a new thread for the release. As pointed out in 2 other related threads, we have some compilation warnings about functions that will no longer exist in OTP R15 (to be released by the end of this year): /opt/r14b03/bin/erlc +debug_info oauth_http.erl ./oauth_http.erl:13: Warning: http:request/4 is deprecated and will be removed in R15B; use httpc:request/4 /opt/r14b03/bin/erlc +debug_info oauth_plaintext.erl /opt/r14b03/bin/erlc +debug_info oauth_rsa_sha1.erl ./oauth_rsa_sha1.erl:9: Warning: public_key:pem_to_der/1: deprecated (will be removed in R15A); use file:read_file/1 and public_key:pem_decode/1 ./oauth_rsa_sha1.erl:10: Warning: public_key:decode_private_key/1 is deprecated and will be removed in R15A; use public_key:pem_entry_decode/1 ./oauth_rsa_sha1.erl:22: Warning: public_key:pem_to_der/1: deprecated (will be removed in R15A); use file:read_file/1 and public_key:pem_decode/1 /opt/r14b03/bin/erlc +debug_info oauth_unix.erl The ones regarding public_key concern me, as it will break the OAuth authentication handler. I see 2 solutions: 1) upgrade erlang-oauth to the same version we have in trunk/1.2.x (doesn't produce these warnings) 2) modify the erlang-oauth in 1.1.x and use try catches where the catch would call the equivalent versions in R14/R15 (these new functions don't exist in R13B03 for example) Naturally, I would prefer 1) The warnings about http:request can be ignored I think, as in Couch we don't use any codepath that will execute the deprecated function B. On 6 October 2011 12:05, Robert Newson rnew...@apache.org wrote: Thanks all, I've pushed the change to 1.1.x. make check and futon all pass; review would still be nice. :) I simply reverted the two commits. B. On 6 October 2011 12:02, Jan Lehnardt j...@apache.org wrote: On Oct 6, 2011, at 10:30 , Robert Newson wrote: There is no build of 1.1.1 on Ubuntu 11.x that will work as well as 1.1.0, so I think it's correct that it cannot build under those conditions. Let's get 1.1.1 out, with the many useful bug fixes and tweaks, and then focus on getting 1.2 out with 1.8.5 support (and BREAKING CHANGES). I vote +1 to removing 1.8.5 support and the paren hack from 1.1.x. +1 Cheers Jan -- B. On 6 October 2011 09:25, Paul Davis paul.joseph.da...@gmail.com wrote: On Thu, Oct 6, 2011 at 3:23 AM, Robert Newson rnew...@apache.org wrote: All, Paul Davis has researched the issue and it seems intractable. I would like to remove 1.8.5 support from 1.1.1. It was not present in 1.1.0 so will not be (officially) missed. The place for a breaking change of this magnitude is 1.2, not a minor bug fix release. Thoughts? B. +1 on removing the paren hack for sure. Not sure about removing 1.8.5 support completely. On the one hand, it would prevent breakage because people couldn't link against the breaking SM. On the other hand, it prevents people from linking against 1.8.5 which means it won't build on Ubuntu 11.x. Unless someone comes up with a magic option I'd say put it to an informal vote so that I can blame someone else. On 5 October 2011 18:25, Paul Davis paul.joseph.da...@gmail.com wrote: Yes, its release blocking. On Wed, Oct 5, 2011 at 7:56 AM, Robert Newson rnew...@apache.org wrote: All, I went through JIRA and updated CHANGES and NEWS on origin/1.1.x to include everything that was missing (Sidenote: Can we all keep this file up to date when commit bugfixes or add features?). I'd appreciate everyone giving it a look over before I start to build the release artifact. I believe there's an outstanding issue (not present in JIRA) around javascript function evaluation? Can someone confirm that it's release blocking? Thanks, B. -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: CouchDB 1.1.1
On Thu, Oct 6, 2011 at 6:59 PM, Robert Newson rnew...@apache.org wrote: does the newer erlang-oauth break anything? Not that I know of. On 6 October 2011 18:52, Filipe David Manana fdman...@apache.org wrote: On Thu, Oct 6, 2011 at 6:23 PM, Robert Newson rnew...@apache.org wrote: If there are no objections, I'm going to build the first 1.1.1 candidate in the morning and start a new thread for the release. As pointed out in 2 other related threads, we have some compilation warnings about functions that will no longer exist in OTP R15 (to be released by the end of this year): /opt/r14b03/bin/erlc +debug_info oauth_http.erl ./oauth_http.erl:13: Warning: http:request/4 is deprecated and will be removed in R15B; use httpc:request/4 /opt/r14b03/bin/erlc +debug_info oauth_plaintext.erl /opt/r14b03/bin/erlc +debug_info oauth_rsa_sha1.erl ./oauth_rsa_sha1.erl:9: Warning: public_key:pem_to_der/1: deprecated (will be removed in R15A); use file:read_file/1 and public_key:pem_decode/1 ./oauth_rsa_sha1.erl:10: Warning: public_key:decode_private_key/1 is deprecated and will be removed in R15A; use public_key:pem_entry_decode/1 ./oauth_rsa_sha1.erl:22: Warning: public_key:pem_to_der/1: deprecated (will be removed in R15A); use file:read_file/1 and public_key:pem_decode/1 /opt/r14b03/bin/erlc +debug_info oauth_unix.erl The ones regarding public_key concern me, as it will break the OAuth authentication handler. I see 2 solutions: 1) upgrade erlang-oauth to the same version we have in trunk/1.2.x (doesn't produce these warnings) 2) modify the erlang-oauth in 1.1.x and use try catches where the catch would call the equivalent versions in R14/R15 (these new functions don't exist in R13B03 for example) Naturally, I would prefer 1) The warnings about http:request can be ignored I think, as in Couch we don't use any codepath that will execute the deprecated function B. On 6 October 2011 12:05, Robert Newson rnew...@apache.org wrote: Thanks all, I've pushed the change to 1.1.x. make check and futon all pass; review would still be nice. :) I simply reverted the two commits. B. On 6 October 2011 12:02, Jan Lehnardt j...@apache.org wrote: On Oct 6, 2011, at 10:30 , Robert Newson wrote: There is no build of 1.1.1 on Ubuntu 11.x that will work as well as 1.1.0, so I think it's correct that it cannot build under those conditions. Let's get 1.1.1 out, with the many useful bug fixes and tweaks, and then focus on getting 1.2 out with 1.8.5 support (and BREAKING CHANGES). I vote +1 to removing 1.8.5 support and the paren hack from 1.1.x. +1 Cheers Jan -- B. On 6 October 2011 09:25, Paul Davis paul.joseph.da...@gmail.com wrote: On Thu, Oct 6, 2011 at 3:23 AM, Robert Newson rnew...@apache.org wrote: All, Paul Davis has researched the issue and it seems intractable. I would like to remove 1.8.5 support from 1.1.1. It was not present in 1.1.0 so will not be (officially) missed. The place for a breaking change of this magnitude is 1.2, not a minor bug fix release. Thoughts? B. +1 on removing the paren hack for sure. Not sure about removing 1.8.5 support completely. On the one hand, it would prevent breakage because people couldn't link against the breaking SM. On the other hand, it prevents people from linking against 1.8.5 which means it won't build on Ubuntu 11.x. Unless someone comes up with a magic option I'd say put it to an informal vote so that I can blame someone else. On 5 October 2011 18:25, Paul Davis paul.joseph.da...@gmail.com wrote: Yes, its release blocking. On Wed, Oct 5, 2011 at 7:56 AM, Robert Newson rnew...@apache.org wrote: All, I went through JIRA and updated CHANGES and NEWS on origin/1.1.x to include everything that was missing (Sidenote: Can we all keep this file up to date when commit bugfixes or add features?). I'd appreciate everyone giving it a look over before I start to build the release artifact. I believe there's an outstanding issue (not present in JIRA) around javascript function evaluation? Can someone confirm that it's release blocking? Thanks, B. -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men. -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: git commit: Open replicator database in child process
On Wednesday, October 5, 2011, Benoit Chesneau bchesn...@gmail.com wrote:
On Wed, Oct 5, 2011 at 2:12 AM, fdman...@apache.org wrote:
Updated Branches:
refs/heads/master 3c24a94dd - 0c7fbd3da
Open replicator database in child process
This is more correct and less confusing.
Project: http://git-wip-us.apache.org/repos/asf/couchdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/couchdb/commit/0c7fbd3d
Tree: http://git-wip-us.apache.org/repos/asf/couchdb/tree/0c7fbd3d
Diff: http://git-wip-us.apache.org/repos/asf/couchdb/diff/0c7fbd3d
Branch: refs/heads/master
Commit: 0c7fbd3dab5bf13215def4349dbba7f689a22ac6
Parents: 3c24a94
Author: Filipe David Manana fdman...@apache.org
Authored: Wed Oct 5 01:10:54 2011 +0100
Committer: Filipe David Manana fdman...@apache.org
Committed: Wed Oct 5 01:12:45 2011 +0100
--
src/couchdb/couch_replication_manager.erl | 11 +++
1 files changed, 7 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/couchdb/blob/0c7fbd3d/src/couchdb/couch_replication_manager.erl
--
diff --git a/src/couchdb/couch_replication_manager.erl
b/src/couchdb/couch_replication_manager.erl
index 5885732..7d1860e 100644
--- a/src/couchdb/couch_replication_manager.erl
+++ b/src/couchdb/couch_replication_manager.erl
@@ -223,9 +223,12 @@ code_change(_OldVsn, State, _Extra) -
changes_feed_loop() -
{ok, RepDb} = ensure_rep_db_exists(),
+RepDbName = couch_db:name(RepDb),
+couch_db:close(RepDb),
Server = self(),
Pid = spawn_link(
fun() -
+{ok, Db} = couch_db:open_int(RepDbName, [sys_db]),
ChangesFeedFun = couch_changes:handle_changes(
#changes_args{
include_docs = true,
@@ -234,7 +237,7 @@ changes_feed_loop() -
db_open_options = [sys_db]
},
{json_req, null},
-RepDb
+Db
),
ChangesFeedFun(
fun({change, Change, _}, _) -
@@ -248,11 +251,11 @@ changes_feed_loop() -
(_, _) -
ok
end
-)
+),
+couch_db:close(Db)
end
),
-couch_db:close(RepDb),
-{Pid, couch_db:name(RepDb)}.
+{Pid, RepDbName}.
has_valid_rep_id({Change}) -
Can't you ensure it in the process too ? What is the reason to have it
externally ?
To return the name to the parent.
- benoit
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: CouchDB 1.1.1
I have one concern I would like to be solved for 1.1.1, which I wrote about sometime ago but got no replies so far: http://mail-archives.apache.org/mod_mbox/couchdb-dev/201109.mbox/%3ccal3q7h5aseiwrj+cggzzzr82mpfbgucg_5oe8nkwell06vt...@mail.gmail.com%3E I have it basically finished on a local branch and will try to finalize it tonight. On Wed, Sep 28, 2011 at 2:19 PM, Robert Newson rnew...@apache.org wrote: Thanks Randall, would be good to get them in if poss. B. On 28 September 2011 21:55, Randall Leeds randall.le...@gmail.com wrote: I updated three issues I had in my backlog to be 1.1.1. https://issues.apache.org/jira/browse/COUCHDB/fixforversion/12316395 If I can't get it done in the next 48 hours, we can probably punt, but they're all pretty straightforward. I either have patches mostly finished or the ticket already contains the fix. On Wed, Sep 28, 2011 at 05:39, Jan Lehnardt j...@apache.org wrote: On Sep 28, 2011, at 12:24 , Robert Newson wrote: All, I've been cleaning up JIRA tickets for 1.1.1 as I'd like to start the release process soon; there are several valuable fixes over 1.1.0. If anyone has anything that should be in 1.1.1 but is not marked as such in JIRA, please let me know (or fix JIRA :P). Also if there's anything that you think should be backported, now's the time to say it. I will go through the commits on trunk looking for missing backported fixes myself, but your efforts are greatly appreciated. Thanks Robert, I have no outstanding issues for 1.1.1. Cheers Jan -- -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: Starting the Git Experiment
+1 as well Paul On Fri, Sep 23, 2011 at 4:18 PM, Robert Newson rnew...@apache.org wrote: +1 On 23 September 2011 21:18, Randall Leeds randall.le...@gmail.com wrote: +1 Thank you! On Fri, Sep 23, 2011 at 13:52, Paul J. Davis paul.joseph.da...@gmail.comwrote: Dear committers, We now have a green light from infrastructure to switch to using Git as our writable VCS. This is to be considered a live experiment. If something breaks its possible we'll have to revert back to SVN. But nothing will break and everyone will forgive me for any bugs that may crop up. If there are no objections I would like to switch over soonish. Normally I would say Monday to give people a chance to respond to this email but we've had quite a few discussions on switching to Git already and no one has voiced opposition. Seeing as that's the case if I get a majority of +1's from the committers I'll start disabling SVN access as soon as I see the majority vote. Paul Davis -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
etap test suite and OTP module http
Hi all,
The module 'http', which is used by some etap tests, has been
deprecated in the OTP R14 series and will no longer exist in OTP R15,
soon to be released (actually it's a rename to 'httpc').
I started replacing all call to 'http' with the ibrowse equivalent
calls. However, specially for branch 1.1.x, the tests will fail often
with a badmatch exception like this:
[info] [0.74.0] 127.0.0.1 - - 'PUT'
/couch_test_atts_compression/doc_att_deflate/readme.txt 415
ok 76 - Couldn't create an already compressed attachment using the
unsupported encoding 'deflate'
# Test died abnormally: {'EXIT',
{{badmatch,
{error,
{'EXIT',
{normal,
{gen_server,call,
[0.105.0,
{send_req,
{{url,
http://127.0.0.1:64038/couch_test_atts_compression/doc_att_compress/readme.txt;,
127.0.0.1,64038,undefined,undefined,
/couch_test_atts_compression/doc_att_compress/readme.txt,
http,ipv4_address},
[{Content-Encoding,compress},
{Content-Type,text/plain}],
This is something that can be avoided by retrying the request when
this specific error happens. For example, it's how the new replicator
deals with it:
https://github.com/apache/couchdb/blob/trunk/src/couchdb/couch_api_wrap_httpc.erl#L71
I'm thinking about doing the equivalent in the test suite by adding a
new function to test/etap/test_util.erl that will do this retry logic.
Another alternative is for that new function to use 'httpc' if 'http'
doesn't exists.
There are also a few places in our bundled etap which use 'http'.
Basically, as things are now, it means we won't be able to build couch
(because of the bundled etap) and run the etap tests on R15, which is
supposed to be released by the end of the year or early next year.
For 1.1.x, here's the current patch:
https://github.com/fdmanana/couchdb/commit/22fcbccb275885292ff1d349afbec3c7a543ff73
Anyone has a simpler idea in mind?
--
Filipe David Manana,
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: Unofficial Windows build
Needless is to say how important and useful Dave's work on CouchDB for Windows has been. Good luck with the move Dave, and thanks for all your work. On Wed, Sep 14, 2011 at 9:02 AM, Noah Slater nsla...@apache.org wrote: On 14 Sep 2011, at 15:41, Adam Kocoloski wrote: Wow Dave, awesome stuff. Enjoy your holiday and good luck with the move! Best, Seconded, this is terrific work. Take care! N -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: Branching 1.2.x
Completely agree it's time to start thinking about 1.2.0. There a few things I would like to see in 1.2.0: 1) https://issues.apache.org/jira/browse/COUCHDB-1238 (use _users database to store OAuth tokens) 2) https://issues.apache.org/jira/browse/COUCHDB-1266 (richer _active_tasks) 3) https://issues.apache.org/jira/browse/COUCHDB-1271 (replication cancelation fixes and simplification) On Mon, Sep 12, 2011 at 9:14 AM, Jan Lehnardt j...@apache.org wrote: On Sep 12, 2011, at 18:07 , Noah Slater wrote: On 12 Sep 2011, at 16:35, Jan Lehnardt wrote: The fact that we branch 1.2.x won't mean we can't get more tickets in there, it's just to unblock trunk for post-1.2.x commits. I hope this makes sense and I hope you all agree. How are we going to stop a repeat of the 1.0 release branch kitchen sink problems? I'd like everybody to suggest their wish for 1.2.x and then agree with this group on how much of the resulting list we can actually get into the branch in a reasonable amount of time :) Cheers Jan -- -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: Request to correct a comment typo
Done. Thanks Jason. On Sun, Sep 11, 2011 at 11:50 AM, Jason Smith j...@iriscouch.com wrote: I believe there is a typo in a comment which makes it difficult to grep for _update handler code. My usual searches for '_update' or '\b_update' did not match in src/couchdb because of this typo (and because this string is found in the .ini file templates). Would a committer mind reviewing this one-liner and committing it? https://github.com/jhs/couchdb/commit/29336bb60b27679260fd936696521712b36d2cdd P.S. Do GitHub pull requests work yet? To whom should the rabble like me send them? Thanks! -- Iris Couch -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: The _security object should be versioned
On Fri, Aug 26, 2011 at 8:01 PM, Jason Smith j...@iriscouch.com wrote: 1. Does this require updating the replicator to update _local docs correctly? Yes 2. Only admins can change _security. But anybody with read access can change _local/*. Does couch special-case _local/security? My preference: _security would become a regular document (just a special id, which starts with underscore). We can still cache the latest revision in the db header, db updater state, whatever. This _security document (or perhaps any other starting with underscore in the future), would only be replicable if the replication is triggered by some special user with some special role (_admin, _server_admin, whatever). Does it sound simple and satisfies people's needs? -- Iris Couch -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: The _security object should be versioned
On Fri, Aug 26, 2011 at 8:46 PM, Jason Smith j...@iriscouch.com wrote: Does it sound simple and satisfies people's needs? AFAIK, nobody wants security to ever replicate. Some people want to manually sync them as an application feature. Ok, catching this thread late, my understanding was that people wanted to replicate it. Going back to Adam's first message, just promoting to _security to a #full_doc_info seems very reasonable. -- Iris Couch -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: The _security object should be versioned
On Fri, Aug 26, 2011 at 10:36 PM, Paul Davis paul.joseph.da...@gmail.com wrote: Imagine you have a phone with a CouchDB. And your friend says, Just replicate this photo album. But he's inserted a _security doc that gives him permission to touch your private data. If someone said the obvious answer is have a validate_doc_update function, I would obviously slap that person. That's why only a very special role could replicate it. I agree allowing it to replicate is very dangerous. Never in no way ever should it be remotely possible to unknowingly change authorization settings because your db accidentally slurped up a _security doc. -- Iris Couch -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men. -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [jira] [Commented] (COUCHDB-1153) Database and view index compaction daemon
On Tue, Aug 16, 2011 at 2:58 AM, Benoit Chesneau bchesn...@gmail.com wrote: Could be a local docs, But why didn't we took this path for this _security object ? Also since they are really meta informations, i've the feeling it should be solved as a special member in the db file, just like the _security object. I don't know why _security is like it is now, that predates me, and it's another topic :) Anyway what I really dislike is saving per db configuration in an ini file. Per db configuration should be done on the db. What if you more than 100 dbs. Having 100 lines in an ini file to parse is awkward. I don't think the common case is to have a separate compact config for every single database. The fragmentation parameter, which is likely the most useful, you're likely to not set a different value for 100 databases (neither the period for e.g.). For other things like the oauth tokens/secrets, the .ini system doesn't scale. But that's again another topic. This is just as simple as this line, creating a db create an entry in a db index (or db file) that you can use later. I suspect what you think is something like rather than scanning periodically, to let the daemon be notified when a db (or view) can be compacted? At some point I considered reacting to db_updated events but this was pretty much flooding the the event handler (daemon). Was this your idea? Using db events is my idea yes. If t actually flood the db event handler (not sure why), then maybe we should fix it first? The problem is when you have many dbs in the system and under a reasonable write load, the daemon (which is the receiver of db_updated events) receives too many messages. To know if you need to compact the db after such message, you need to open it, and opening it on every message is a big burden as well. I tried this on a system with 1024 databases being updated constantly. It also doesn't deal with the case on startup where if a db with a high fragmentation is not updated for a long period, it won't have compaction started. If someone can measure the current solution's impact and present another working alternative with a lower impact (and practical tests, not just theory) I would be the first one wanting to make the change asap. - benoit -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: compaction plugin, auth handler, foo plugin couchdb core [resent]
On Tue, Aug 16, 2011 at 2:59 PM, Randall Leeds As an experiment in JIRA usage I created an umbrella task for this. Please place tickets under this umbrella and we can start to break down the sub-tasks we need to actually get this work done. https://issues.apache.org/jira/browse/COUCHDB-1251 I set the due date as the 21st of December. Holiday season. This should give us enough time to get 1.2 out the door and make some real progress on these goals. Sounds like a good idea Randall. Thanks for it. Again, this is an experiment. Sorry for those of you who hate process, but I thought maybe injecting a bit of here would stop the flow of e-mails and focus us all collectively. -Randall -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Bringing automatic compaction into trunk
Developers, users, It's been a while now since I opened a Jira ticket for it ( https://issues.apache.org/jira/browse/COUCHDB-1153 ). I won't describe it here with detail since it's already done in the Jira ticket. Unless there are objections, I would like to get this moving soon. Thanks -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: Trouble replicating
On Fri, Aug 12, 2011 at 1:52 PM, Zachary Zolton zachary.zol...@gmail.com wrote: So, my replication task has again stalled, with my logs filling up with thousands of that aforementioned error message, again today at around 1:25 CDT. I'll also note that this error state seems to adversely affect couch's responsiveness... I'm still unsure how to troubleshoot this. Is there any more information about my setup that would help? I never experienced that SSL issue with R14B03 yet. Maybe you can try with R14B01 for example? FYI, I deleted the _replicator doc and have gone back to maintaining this continuous replication with a cron script. You shouldn't need that anymore in the next minor release (1.1.1). With 1.1.0, if it ends in the error state you have to manually start it again (delete the doc and add it again or just remove its _replication_state field) On Thu, Aug 11, 2011 at 11:40 PM, Dustin Sallings dus...@spy.net wrote: On Aug 11, 2011, at 11:59 AM, Zachary Zolton wrote: By the way, after the _replicator doc went into the error state, I removed all the _replication* fields and save it. The replication task resumed and it seems to be keeping up-to-date for the past few hours. Guess I'll keep my fingers crossed for now... (^_^ You only need to remove the _replicator_state. There are a few bugs that prevent it from automatically restarting as you might expect. I'm sure I've hit all of them. :) -- dustin sallings -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: svn commit: r1156381 - /couchdb/trunk/src/couchdb/couch_server.erl
Forgot to mention, this issue was actually not happening in any Apache
CouchDB release. Apologies for the confusion. I'll update the comment.
On Wed, Aug 10, 2011 at 2:49 PM, fdman...@apache.org wrote:
Author: fdmanana
Date: Wed Aug 10 21:49:20 2011
New Revision: 1156381
URL: http://svn.apache.org/viewvc?rev=1156381view=rev
Log:
Revert couch_server changes added in revision 1156360
Leaving the tests however since it doesn't hurt.
Modified:
couchdb/trunk/src/couchdb/couch_server.erl
Modified: couchdb/trunk/src/couchdb/couch_server.erl
URL:
http://svn.apache.org/viewvc/couchdb/trunk/src/couchdb/couch_server.erl?rev=1156381r1=1156380r2=1156381view=diff
==
--- couchdb/trunk/src/couchdb/couch_server.erl (original)
+++ couchdb/trunk/src/couchdb/couch_server.erl Wed Aug 10 21:49:20 2011
@@ -316,13 +316,11 @@ handle_call({open, DbName, Options}, {Fr
{reply, couch_db:open_ref_counted(MainPid, FromPid), Server}
end;
handle_call({create, DbName, Options}, From, Server) -
- FileName = get_full_filename(Server, ?b2l(DbName)),
- case file:open(FileName, [read]) of
- {ok, Fd} -
- ok = file:close(Fd),
- {reply, file_exists, Server};
- Error -
- open_db(DbName, Server, [create | Options], From)
+ case ets:lookup(couch_dbs_by_name, DbName) of
+ [] -
+ open_db(DbName, Server, [create | Options], From);
+ [_AlreadyRunningDb] -
+ {reply, file_exists, Server}
end;
handle_call({delete, DbName, _Options}, _From, Server) -
DbNameList = binary_to_list(DbName),
--
Filipe David Manana,
fdman...@gmail.com, fdman...@apache.org
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: Futon Test Suite
);
- wait(2000);
+ waitForRepEnd(rep_id);
copy = targetDb.open(doc._id);
TEquals(null, copy);
}
@@ -1359,7 +1375,7 @@ couchTests.replication = function(debug) {
var tasksAfter = JSON.parse(xhr.responseText);
TEquals(tasks.length, tasksAfter.length);
- waitForSeq(sourceDb, targetDb);
+ waitForSeq(sourceDb, targetDb, rep_id);
T(sourceDb.open(30) !== null);
// cancel replication
--
Filipe David Manana,
fdman...@gmail.com, fdman...@apache.org
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: Futon Test Suite
On Mon, Aug 8, 2011 at 11:43 PM, Paul Davis paul.joseph.da...@gmail.com wrote: The entire test suite takes about 4 minutes to run on a semi recent MBA. Most of the tests are fairly speedy, but four of them stick out quite drastically: delayed_commits: 15s design_docs: 25s replication: 90s replcaitor_db: 60s The replication.js test grew a lot after the new replicator was introduced. Basically it covers a lot more scenarios then the old replication.js test, and tests with larger amounts of documents and continuous replications. I think this is a good thing and inevitable (due to bug fixes, new features, etc). The replicator_db.js does several server restart calls, which are necessary to test this feature. After Jan's patch to add a verify installation feature to Futon, I don't think individual tests taking 1, 2 or 5 minutes are an issue, as long as they succeed. For a database management system, having much more comprehensive tests (which mean that they take longer to run) is a good thing. I agree with everything said in this thread. I haven't dug into these too much yet. The only thing I've noticed is that replication and relplicator_db seem to spend a lot of their time polling various URLs waiting for task completion. Perhaps I'm just being impatient but it seems that each poll lasts an uncessarily long time for a unit tests (3-5s) so I was wondering if we were hitting a timeout or something. If anyone wants to dig into the test suite and see if they can't speed these up or even just split them apart so we know what's taking awhile that'd be super awesome. Also, I've been thinking more and more about beefing up the JavaScript test suite runner and moving more of our browser tests over to dedicated code in those tests. If anyone's interested in hacking on some C and JavaScript against an HTTP API, let me know. -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: create_target isn't working with the _replicator db
On Fri, Jul 22, 2011 at 8:43 AM, Benoit Chesneau bchesn...@gmail.com wrote: Yup, but I think that's a bug then. I shouldn't have to set any userctx imo. If no admin has been set, every user is an admin except if we change the default behavior and then it's not consistent. This was discussed sometime before the 1.1.0 release in the security list. And it's a principle of the least privileges by default (roles is an empty list). - benoit - benoît -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [VOTE] Apache CouchDB 1.0.3 Release
On Wed, Jul 13, 2011 at 9:05 PM, Robert Newson rnew...@apache.org wrote: bump. Please can PMC members vote on this release so we can restore couchdb replication compatibility between our two stable release branches? As far as I known it's possible to replicate between 1.0.2 and 1.1.0 (either pull or push, in any direction). Thanks, B. On 28 June 2011 20:05, Noah Slater nsla...@apache.org wrote: On 28 Jun 2011, at 19:24, Paul Davis wrote: Yours truly, Paul Davis Received, and understood. Ever handsome, Mr. Noah Slater Esq. -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: Improving password hashing.
Looks good to me as well. Minor nitpick, ideally it would respect our coding standard of not having lines longer than 80 characters. Good work Robert On Wed, Jul 6, 2011 at 2:10 PM, Robert Newson rnew...@apache.org wrote: Making it pluggable is probably not much more work but I have to point at that use sha256 is an inadequate description of a secure password hashing protocol. B. On 6 July 2011 14:05, Benoit Chesneau bchesn...@gmail.com wrote: On Wed, Jul 6, 2011 at 2:43 PM, Robert Newson rnew...@apache.org wrote: All, Our current password hashing scheme is weak. In fact, it's regarded as weak as plaintext. I'd like to change that. Some time ago I wrote some code to implement the PBKDF2 protocol. This is a cryptographically sound means of deriving a key from a password. The output is also usable as a password hash. An important part of the protocol is that the work factor can be increased by increasing the loop count. Additionally, it is not tied to a specific digest algorithm. All these points are not true of the sometimes proposed alternative called 'bcrypt' which I do not recommend. I would like this to go into CouchDB 1.2. New passwords, and updated passwords, from 1.2 onwards would use the new scheme, but 1.2 will, obviously, be able to verify the current style. This work will take place within couch_server where hash_admin_passwords currently lives. The PKBDF2 code is here: https://github.com/rnewson/couchdb/tree/pbkdf2. It passes all the test vectors. The ticket for this work is https://issues.apache.org/jira/browse/COUCHDB-1060 B. That sounds good. I would prefer however a customizable hashing method for passwords so we can change it easily depending the target. Some administrations for example require that you use some methods (like sha256 in europe) and it would be very useful. - benoît -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: Improving password hashing.
After looking at it more closely I have 1 observation. The user's password can come either from the .ini file or from JSON coming from an HTTP post (couch_httpd_auth:handle_session_req/1 for example). I see that the password is passed to the new module, couch_passwords, which will end up calling cryto:sha_mac(Password, Salt). This concerns me because when the Password comes from the .ini config, it is a string and when it comes from JSON it is an UTF-8 binary. Before passing the password to crypto:sha_mac/2, I think it needs to be converted to an UTF-8 binary, otherwise it doesn't produce the same result as if it had came from JSON, example: 1 H1 = crypto:sha_mac(àbc, 123). 149,17,150,46,220,128,255,66,209,115,186,23,167,63,117, 165,113,82,100,234 2 2 H2 = crypto:sha_mac(unicode:characters_to_binary(àbc), 123). 30,96,182,160,202,26,142,36,32,79,28,250,228,105,79,11, 73,48,52,197 3 3 H1 =:= H2. false On Wed, Jul 6, 2011 at 2:53 PM, Robert Newson rnew...@apache.org wrote: Patch will be tidied to community standards before submission. The upgrade code is not yet written but should be straightforward. B. On 6 July 2011 14:50, Filipe David Manana fdman...@apache.org wrote: Looks good to me as well. Minor nitpick, ideally it would respect our coding standard of not having lines longer than 80 characters. Good work Robert On Wed, Jul 6, 2011 at 2:10 PM, Robert Newson rnew...@apache.org wrote: Making it pluggable is probably not much more work but I have to point at that use sha256 is an inadequate description of a secure password hashing protocol. B. On 6 July 2011 14:05, Benoit Chesneau bchesn...@gmail.com wrote: On Wed, Jul 6, 2011 at 2:43 PM, Robert Newson rnew...@apache.org wrote: All, Our current password hashing scheme is weak. In fact, it's regarded as weak as plaintext. I'd like to change that. Some time ago I wrote some code to implement the PBKDF2 protocol. This is a cryptographically sound means of deriving a key from a password. The output is also usable as a password hash. An important part of the protocol is that the work factor can be increased by increasing the loop count. Additionally, it is not tied to a specific digest algorithm. All these points are not true of the sometimes proposed alternative called 'bcrypt' which I do not recommend. I would like this to go into CouchDB 1.2. New passwords, and updated passwords, from 1.2 onwards would use the new scheme, but 1.2 will, obviously, be able to verify the current style. This work will take place within couch_server where hash_admin_passwords currently lives. The PKBDF2 code is here: https://github.com/rnewson/couchdb/tree/pbkdf2. It passes all the test vectors. The ticket for this work is https://issues.apache.org/jira/browse/COUCHDB-1060 B. That sounds good. I would prefer however a customizable hashing method for passwords so we can change it easily depending the target. Some administrations for example require that you use some methods (like sha256 in europe) and it would be very useful. - benoît -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men. -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: New write performance optimizations coming
writers, 5 minutes run
trunk: 878 520 docs written
branch: 991 330 docs written
## batches of 100 1Kb docs, 50 writers, 5 minutes run
trunk: 1 627 600 docs written
branch: 1 865 800 docs written
## batches of 1 2.5Kb docs, 50 writers, 5 minutes run
trunk: 142 531 docs written
branch: 143 012 docs written
## batches of 10 2.5Kb docs, 50 writers, 5 minutes run
trunk: 724 880 docs written
branch: 780 690 docs written
## batches of 100 2.5Kb docs, 50 writers, 5 minutes run
trunk: 1 028 600 docs written
branch: 1 152 800 docs written
# bash bench tests, via the internal Erlang APIs
## batches of 100 2.5Kb docs, 50 writers, 5 minutes run
trunk: 3 170 100 docs written
branch: 3 359 900 docs written
# Relaximation tests
1Kb docs:
http://graphs.mikeal.couchone.com/#/graph/4843dbdf8fa104783870094b83002a1a
2.5Kb docs:
http://graphs.mikeal.couchone.com/#/graph/4843dbdf8fa104783870094b830022c0
4Kb docs:
http://graphs.mikeal.couchone.com/#/graph/4843dbdf8fa104783870094b8300330d
All the documents used for these tests can be found at:
https://github.com/fdmanana/basho_bench_couch/tree/master/couch_docs
Now some view indexing tests.
# indexer_test_2 database
(http://fdmanana.couchone.com/_utils/database.html?indexer_test_2)
## trunk
$ time curl
http://localhost:5984/indexer_test_2/_design/test/_view/view1?limit=1
{total_rows:1102400,offset:0,rows:[
{id:00d49881-7bcf-4c3d-a65d-e44435eeb513,key:[dwarf,assassin,2,1.1],value:[{x:174347.18,y:127272.8},{x:35179.93,y:41550.55},{x:157014.38,y:172052.63},{x:116185.83,y:69871.73},{x:153746.28,y:190006.59}]}
]}
real 20m51.388s
user 0m0.040s
sys 0m0.000s
## branch async writes
$ time curl
http://localhost:5984/indexer_test_2/_design/test/_view/view1?limit=1
{total_rows:1102400,offset:0,rows:[
{id:00d49881-7bcf-4c3d-a65d-e44435eeb513,key:[dwarf,assassin,2,1.1],value:[{x:174347.18,y:127272.8},{x:35179.93,y:41550.55},{x:157014.38,y:172052.63},{x:116185.83,y:69871.73},{x:153746.28,y:190006.59}]}
]}
real 15m17.908s
user 0m0.008s
sys 0m0.020s
# indexer_test_3_database
(http://fdmanana.couchone.com/_utils/database.html?indexer_test_3)
## trunk
$ time curl
http://localhost:5984/indexer_test_3/_design/test/_view/view1?limit=1
{total_rows:1102400,offset:0,rows:[
{id:00d49881-7bcf-4c3d-a65d-e44435eeb513,key:[dwarf,assassin,2,1.1],value:[{x:174347.18,y:127272.8},{x:35179.93,y:41550.55},{x:157014.38,y:172052.63},{x:116185.83,y:69871.73},{x:153746.28,y:190006.59}]}
]}
real 21m17.346s
user 0m0.012s
sys 0m0.028s
## branch async writes
$ time curl
http://localhost:5984/indexer_test_3/_design/test/_view/view1?limit=1
{total_rows:1102400,offset:0,rows:[
{id:00d49881-7bcf-4c3d-a65d-e44435eeb513,key:[dwarf,assassin,2,1.1],value:[{x:174347.18,y:127272.8},{x:35179.93,y:41550.55},{x:157014.38,y:172052.63},{x:116185.83,y:69871.73},{x:153746.28,y:190006.59}]}
]}
real 16m28.558s
user 0m0.012s
sys 0m0.020s
We don’t show nearly as big of improvements for single write per request
benchmarks as we do with bulk writes. This is due to the HTTP request
overhead and our own inefficiencies at that layer. We have lots of room yet
for optimizations at the networking layer.
We'd like to merge this code into trunk next week by next wednesday. Please
respond with any improvement, objections or comments by then. Thanks!
-Damien
[1] -
http://blog.couchbase.com/driving-performance-improvements-couchbase-single-server-two-dot-zero
[2] - https://github.com/fdmanana/couchdb/compare/async_file_writes_no_test
[3] - https://github.com/fdmanana/couchdb/compare/async_file_writes
[4] - https://issues.apache.org/jira/browse/COUCHDB-1084
[5] - https://github.com/fdmanana/basho_bench_couch
[6] - https://github.com/fdmanana/couchdb/blob/async_file_writes/gen_load.sh
[7] -
https://github.com/fdmanana/couchdb/blob/async_file_writes/src/couchdb/couch_internal_load_gen.erl
--
Filipe David Manana,
fdman...@gmail.com, fdman...@apache.org
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: Commits to 1.1.x not in 1.0.x
Paul, why do you need to backport that patch? It's related to a change that ended up only in 1.1.x and trunk (use of mochiweb's Request:accepts_content_type/1). On Wed, Jun 15, 2011 at 2:04 AM, Paul Davis paul.joseph.da...@gmail.com wrote: On Tue, Jun 14, 2011 at 8:55 AM, Jan Lehnardt j...@apache.org wrote: Good find Paul, On 14 Jun 2011, at 01:22, Paul Davis wrote: I was just going through what we back ported to 1.1.x to make sure that all the relevant commits are also on 1.0.x before rolling a release. There are a few that are mostly trivial but I've come across two that I'm a bit stumped on. The first is this one: * https://github.com/davisp/couchdb/commit/a0563ab7ce204cf82feb56259d7a52c0b84077b7 Large contiguous conflict in share/www/script/test/design_docs.js that I'm not sure if I can just choose this patched version or not. The patch to util.js applies cleanly, but the diff to design_docs.js results in a fairly large conflict. It may be a simple code move with indentation difference but if someone could eyeball that closer that'd help. It seems that these two are conflicting: commit 118b8906ba8199471265398f11296d756d62d992 Author: Filipe David Borba Manana fdman...@apache.org Date: Mon Nov 15 20:53:32 2010 + Refactoring of the design_docs.js test: fixed indentation and improved readability. git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1035447 13f79535-47bb-0310-9956-ffa450edef68 commit b49ac86e9ac820ff327d132e418f0df5e0f772c8 Author: Filipe David Borba Manana fdman...@apache.org Date: Sat Oct 30 11:28:32 2010 + Trivial change: add/move semi-collons to where they belong to. git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1029030 13f79535-47bb-0310-9956-ffa450edef68 Cheers Jan -- The second is: * https://github.com/davisp/couchdb/commit/d952ac01cb4cd4ae5ceb0c8cc079acf595ff9747 Appears to rely on mochiweb_request:accepts_content_type which does not appear to exist in 1.0.x I'm not even sure if this is relevant to the 1.0.x branch or was a fix for something introduced in the 1.1.x branch with the various content-type changes and what not. Again, if someone would eyeball this that is more familiar with the interaction with jquery/futon that'd be a help. So I managed to get the circular imports thing applied which was the first patch. My SVN rage only flared two or three times even. Unless someone can show me a way to backport this patch: * https://github.com/davisp/couchdb/commit/d952ac01cb4cd4ae5ceb0c8cc079acf595ff9747 Appears to rely on mochiweb_request:accepts_content_type which does not appear to exist in 1.0.x I'm going to ignore it and proceed rolling 1.0.3. -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: couch_server.erl - is couch_dbs_by_name table correclty used in terminate() function ?
It seems to me you're right about it.
Do you want to provide a small patch to fix it? (Ideally in a Jira ticket)
Thanks Andrey
On Wed, Jun 15, 2011 at 10:30 AM, Andrey Somov
trophyb...@googlemail.com wrote:
Hi all,
when I read the code in couch_server.terminate() function I see the
following in the trunk:
terminate(_Reason, _Srv) -
[couch_util:shutdown_sync(Pid) || {_, {Pid, _LruTime}} -
ets:tab2list(couch_dbs_by_name)],
ok.
It looks like the values in the couch_dbs_by_name table are tuples of 2
terms.
But when I read the rest of the code I see that the values are always
3-term-tuples:
{opened, Pid, LruTime} or {opening, Opener, [Froms]}
I must admit I do not understand how the code in the terminate() may work.
Is it just a bug ?
-
Andrey
--
Filipe David Manana,
fdman...@gmail.com, fdman...@apache.org
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: couch_server.erl - is couch_dbs_by_name table correclty used in terminate() function ?
Andrey, I would note that in general list comprehensions are a bit
dangerous besides mapping they also allow filtering.
If that code was written as:
lists:foreach(
fun({_, {Pid, _LruTime}}) - couch_util:shutdown_sync(Pid) end,
ets:tab2list(couch_dbs_by_name))
We would have noticed before the bug, as it would generate a
function_clause error/exception. Notice here we don't care about the
resulting list, so a lists:foreach is the way to go.
On Wed, Jun 15, 2011 at 10:47 AM, Filipe David Manana
fdman...@apache.org wrote:
It seems to me you're right about it.
Do you want to provide a small patch to fix it? (Ideally in a Jira ticket)
Thanks Andrey
On Wed, Jun 15, 2011 at 10:30 AM, Andrey Somov
trophyb...@googlemail.com wrote:
Hi all,
when I read the code in couch_server.terminate() function I see the
following in the trunk:
terminate(_Reason, _Srv) -
[couch_util:shutdown_sync(Pid) || {_, {Pid, _LruTime}} -
ets:tab2list(couch_dbs_by_name)],
ok.
It looks like the values in the couch_dbs_by_name table are tuples of 2
terms.
But when I read the rest of the code I see that the values are always
3-term-tuples:
{opened, Pid, LruTime} or {opening, Opener, [Froms]}
I must admit I do not understand how the code in the terminate() may work.
Is it just a bug ?
-
Andrey
--
Filipe David Manana,
fdman...@gmail.com, fdman...@apache.org
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
--
Filipe David Manana,
fdman...@gmail.com, fdman...@apache.org
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
Re: couch_server.erl - is couch_dbs_by_name table correclty used in terminate() function ?
On Wed, Jun 15, 2011 at 11:54 AM, Andrey Somov trophyb...@googlemail.com wrote: The bug has been introduced more then a year ago (02-Apr-2010) in revision 930363 with the comment Deterministic/synchronous shutdown code Since no one is complaining may be the code is redundant and it can be safely removed ? (the less code the better) I think it's better to do the explicit synchronous shutdown. It ensures any current database writes are finished before a restart (as triggered by POST to /_restart/) complete. Just try it out, run all the tests (Etap and JavaScript) and see if all pass. cheers - Andrey -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: couch_server.erl - is couch_dbs_by_name table correclty used in terminate() function ?
On Wed, Jun 15, 2011 at 2:08 PM, Andrey Somov trophyb...@googlemail.com wrote: My problem is that I never managed to pass ALL the tests. Some of the tests (both Etap and JavaScript) fail in trunk and in branches. (I even came to the conclusion that developers do not run all the tests before each and every commit.) That is why I cannot really trust the patch I can provide. Hum, that's odd. It's not very frequent to have the Etap tests failing. For the JavaScript tests, it's better to use Firefox 3.6 or Chrome, and eventually clear the browser's cache - very often running the failing tests individually results in success. In the past, I have already created a few JIRA issues with patches. None of them is accepted. Perhaps no one had the time yet to review them. If the tickets were not closed and you got no negative feedback, it doesn't mean they were refused. I'll try to look at them soon. Taking all this into account I think it is better if someone else can change the code. You're contributions are always welcome Andrey. Thank you for the help. Andrey -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: make error with latest trunk (ICU related autotools config)
On Mon, Jun 13, 2011 at 10:50 PM, Randall Leeds randall.le...@gmail.com wrote: I restored the default include paths we used to run with in r1135302. But I'm glad to endure this pain. It's really uncovered a lot of places where corners were cut on getting the minimal set of correct flags to each build step by just slapping everything everywhere. I still need confirmation from dch on the openssl linking for windows, but otherwise everything seem straightened out to people? Any more build failures please let me know asap. Thanks Randall. The missing icu headers errors are gone now. However there's something strange still going on. A \c parameter is being passed to the compiler: ... -Wwrite-strings -Wno-long-long \c -I/opt/otp14/lib/erlang/usr/include ... Full command line: /bin/sh ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/snappy/google-snappy -I/opt/local/include -I/usr/local/include -I/usr/include -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c -I/opt/otp14/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c -o couch_icu_driver_la-couch_icu_driver.lo `test -f 'icu_driver/couch_icu_driver.c' || echo './'`icu_driver/couch_icu_driver.c libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/snappy/google-snappy -I/opt/local/include -I/usr/local/include -I/usr/include -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c -I/opt/otp14/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c icu_driver/couch_icu_driver.c -fno-common -DPIC -o .libs/couch_icu_driver_la-couch_icu_driver.o i686-apple-darwin10-gcc-4.2.1: c: No such file or directory The same at friendpaste might be more readble: http://friendpaste.com/3ERusm7juiyJLY0grCBQ97 Any idea about this? Agreed. On Jun 12, 2011 4:07 PM, Filipe David Manana fdman...@apache.org wrote: Hi all, After all the recent changes in the autotools config, I'm able to build trunk on Linux, however I'm no longer able to do it on a 32bits Mac OS X (snow leopard). It seems the include path is not what it used to be. The relevant `make`/`make dev` output: /bin/sh ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../../../src/couchdb/priv -I../../.. -I../../../src/snappy/google-snappy -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c -I/Users/fdmanana/tmp/couchdb/apache-couchdb-1.2.0a-0d27a97-git/_inst/lib/erlang/usr/include -I/usr/lib/erlang/usr/include -I/usr/local/lib/erlang/usr/include -I/opt/local/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c -o couch_icu_driver_la-couch_icu_driver.lo `test -f 'icu_driver/couch_icu_driver.c' || echo '../../../../src/couchdb/priv/'`icu_driver/couch_icu_driver.c libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../../../src/couchdb/priv -I../../.. -I../../../src/snappy/google-snappy -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long c -I/Users/fdmanana/tmp/couchdb/apache-couchdb-1.2.0a-0d27a97-git/_inst/lib/erlang/usr/include -I/usr/lib/erlang/usr/include -I/usr/local/lib/erlang/usr/include -I/opt/local/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c ../../../../src/couchdb/priv/icu_driver/couch_icu_driver.c -fno-common -DPIC -o .libs/couch_icu_driver_la-couch_icu_driver.o i686-apple-darwin10-gcc-4.2.1: c: No such file or directory ../../../../src/couchdb/priv/icu_driver/couch_icu_driver.c:27:26: error: unicode/ucol.h: No such file or directory ../../../../src/couchdb/priv/icu_driver/couch_icu_driver.c:28:30: error: unicode/ucasemap.h: No such file or directory With trunk revision 1132947 (and before), on this same machine, couch_icu_driver compiles just fine. Relevant make output: /bin/sh ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/snappy/google-snappy -L/usr/local/lib -L/opt/local/lib -I/opt/otp14/lib/erlang/usr/include -I/usr/include -I/usr/include/js -I/usr/include/mozjs -I/usr/local/include -I/opt/local/include -I/usr/local/include/js -I/opt/local/include/js -DXP_UNIX -I/opt/local/include\c -L/opt/local/lib\c -g -O2 -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c -o couch_icu_driver_la-couch_icu_driver.lo `test -f 'icu_driver/couch_icu_driver.c' || echo './'`icu_driver/couch_icu_driver.c libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/snappy/google-snappy -L/usr/local/lib -L/opt/local/lib -I/opt/otp14/lib/erlang
Re: make error with latest trunk (ICU related autotools config)
Seems to come from the icu-config command itself: $ icu-config --shlib-c /usr/bin/gcc-4.2 -dynamiclib -dynamic -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long\c $ icu-config --shlib-cc /usr/bin/g++-4.2 -dynamiclib -dynamic -O2 -O2 -W -Wall -ansi -pedantic -Wpointer-arith -Wwrite-strings -Wno-long-long\c So, either something that can be added to http://svn.apache.org/viewvc?view=revisionrevision=1133287 or revert this? On Tue, Jun 14, 2011 at 6:57 PM, Filipe David Manana fdman...@apache.org wrote: On Tue, Jun 14, 2011 at 6:53 PM, Randall Leeds randall.le...@gmail.com wrote: ... -Wwrite-strings -Wno-long-long \c -I/opt/otp14/lib/erlang/usr/include ... Based on the settings that I have in my build, I think that's at the border of ICU_CFLAGS and ERLANG_FLAGS. Can you check src/couchdb/priv/Makefile for both of these variables and see if either has the \c? Forgot to mention it before. The \c's appear only in the top level Makefile, generted by configure: ICU_BIN = ICU_CFLAGS = -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c ICU_CONFIG = /opt/local/bin/icu-config ICU_CXXFLAGS = -O2 -O2 -W -Wall -ansi -pedantic -Wpointer-arith -Wwrite-strings -Wno-long-long \c ICU_LIBS = -lpthread -lm -L/opt/local/lib -licui18n -licuuc -licudata -lpthread -lm \c Funny, they only appear in the ICU related variables. On Linux these lines end with 4 spaces instead of a \c Emacs whitespace-mode doesn't seem to highlight any weird unprintable characters. Full command line: /bin/sh ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/snappy/google-snappy -I/opt/local/include -I/usr/local/include -I/usr/include -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c -I/opt/otp14/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c -o couch_icu_driver_la-couch_icu_driver.lo `test -f 'icu_driver/couch_icu_driver.c' || echo './'`icu_driver/couch_icu_driver.c libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/snappy/google-snappy -I/opt/local/include -I/usr/local/include -I/usr/include -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c -I/opt/otp14/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c icu_driver/couch_icu_driver.c -fno-common -DPIC -o .libs/couch_icu_driver_la-couch_icu_driver.o i686-apple-darwin10-gcc-4.2.1: c: No such file or directory The same at friendpaste might be more readble: http://friendpaste.com/3ERusm7juiyJLY0grCBQ97 Any idea about this? Agreed. On Jun 12, 2011 4:07 PM, Filipe David Manana fdman...@apache.org wrote: Hi all, After all the recent changes in the autotools config, I'm able to build trunk on Linux, however I'm no longer able to do it on a 32bits Mac OS X (snow leopard). It seems the include path is not what it used to be. The relevant `make`/`make dev` output: /bin/sh ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../../../src/couchdb/priv -I../../.. -I../../../src/snappy/google-snappy -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c -I/Users/fdmanana/tmp/couchdb/apache-couchdb-1.2.0a-0d27a97-git/_inst/lib/erlang/usr/include -I/usr/lib/erlang/usr/include -I/usr/local/lib/erlang/usr/include -I/opt/local/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c -o couch_icu_driver_la-couch_icu_driver.lo `test -f 'icu_driver/couch_icu_driver.c' || echo '../../../../src/couchdb/priv/'`icu_driver/couch_icu_driver.c libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../../../src/couchdb/priv -I../../.. -I../../../src/snappy/google-snappy -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long c -I/Users/fdmanana/tmp/couchdb/apache-couchdb-1.2.0a-0d27a97-git/_inst/lib/erlang/usr/include -I/usr/lib/erlang/usr/include -I/usr/local/lib/erlang/usr/include -I/opt/local/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c ../../../../src/couchdb/priv/icu_driver/couch_icu_driver.c -fno-common -DPIC -o .libs/couch_icu_driver_la-couch_icu_driver.o i686-apple-darwin10-gcc-4.2.1: c: No such file or directory ../../../../src/couchdb/priv/icu_driver/couch_icu_driver.c:27:26: error: unicode/ucol.h: No such file or directory ../../../../src/couchdb/priv/icu_driver/couch_icu_driver.c:28:30: error: unicode/ucasemap.h: No such file or directory With trunk revision 1132947
Re: make error with latest trunk (ICU related autotools config)
Funny thing, this only happens with iTerm. Using the standard Terminal that comes with OS X I don't get the \c from the icu-config command. *uckin weird. That said, with Terminal I'm able to build trunk successfully. On Tue, Jun 14, 2011 at 7:02 PM, Filipe David Manana fdman...@apache.org wrote: Seems to come from the icu-config command itself: $ icu-config --shlib-c /usr/bin/gcc-4.2 -dynamiclib -dynamic -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c $ icu-config --shlib-cc /usr/bin/g++-4.2 -dynamiclib -dynamic -O2 -O2 -W -Wall -ansi -pedantic -Wpointer-arith -Wwrite-strings -Wno-long-long \c So, either something that can be added to http://svn.apache.org/viewvc?view=revisionrevision=1133287 or revert this? On Tue, Jun 14, 2011 at 6:57 PM, Filipe David Manana fdman...@apache.org wrote: On Tue, Jun 14, 2011 at 6:53 PM, Randall Leeds randall.le...@gmail.com wrote: ... -Wwrite-strings -Wno-long-long \c -I/opt/otp14/lib/erlang/usr/include ... Based on the settings that I have in my build, I think that's at the border of ICU_CFLAGS and ERLANG_FLAGS. Can you check src/couchdb/priv/Makefile for both of these variables and see if either has the \c? Forgot to mention it before. The \c's appear only in the top level Makefile, generted by configure: ICU_BIN = ICU_CFLAGS = -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c ICU_CONFIG = /opt/local/bin/icu-config ICU_CXXFLAGS = -O2 -O2 -W -Wall -ansi -pedantic -Wpointer-arith -Wwrite-strings -Wno-long-long \c ICU_LIBS = -lpthread -lm -L/opt/local/lib -licui18n -licuuc -licudata -lpthread -lm \c Funny, they only appear in the ICU related variables. On Linux these lines end with 4 spaces instead of a \c Emacs whitespace-mode doesn't seem to highlight any weird unprintable characters. Full command line: /bin/sh ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/snappy/google-snappy -I/opt/local/include -I/usr/local/include -I/usr/include -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c -I/opt/otp14/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c -o couch_icu_driver_la-couch_icu_driver.lo `test -f 'icu_driver/couch_icu_driver.c' || echo './'`icu_driver/couch_icu_driver.c libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/snappy/google-snappy -I/opt/local/include -I/usr/local/include -I/usr/include -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c -I/opt/otp14/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c icu_driver/couch_icu_driver.c -fno-common -DPIC -o .libs/couch_icu_driver_la-couch_icu_driver.o i686-apple-darwin10-gcc-4.2.1: c: No such file or directory The same at friendpaste might be more readble: http://friendpaste.com/3ERusm7juiyJLY0grCBQ97 Any idea about this? Agreed. On Jun 12, 2011 4:07 PM, Filipe David Manana fdman...@apache.org wrote: Hi all, After all the recent changes in the autotools config, I'm able to build trunk on Linux, however I'm no longer able to do it on a 32bits Mac OS X (snow leopard). It seems the include path is not what it used to be. The relevant `make`/`make dev` output: /bin/sh ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../../../src/couchdb/priv -I../../.. -I../../../src/snappy/google-snappy -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c -I/Users/fdmanana/tmp/couchdb/apache-couchdb-1.2.0a-0d27a97-git/_inst/lib/erlang/usr/include -I/usr/lib/erlang/usr/include -I/usr/local/lib/erlang/usr/include -I/opt/local/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c -o couch_icu_driver_la-couch_icu_driver.lo `test -f 'icu_driver/couch_icu_driver.c' || echo '../../../../src/couchdb/priv/'`icu_driver/couch_icu_driver.c libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../../../src/couchdb/priv -I../../.. -I../../../src/snappy/google-snappy -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long c -I/Users/fdmanana/tmp/couchdb/apache-couchdb-1.2.0a-0d27a97-git/_inst/lib/erlang/usr/include -I/usr/lib/erlang/usr/include -I/usr/local/lib/erlang/usr/include -I/opt/local/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c ../../../../src/couchdb/priv/icu_driver/couch_icu_driver.c -fno-common -DPIC -o .libs/couch_icu_driver_la-couch_icu_driver.o i686-apple-darwin10-gcc
Re: make error with latest trunk (ICU related autotools config)
On Tue, Jun 14, 2011 at 7:41 PM, Randall Leeds randall.le...@gmail.com wrote: On Tue, Jun 14, 2011 at 11:29, Filipe David Manana fdman...@apache.org wrote: Funny thing, this only happens with iTerm. Using the standard Terminal that comes with OS X I don't get the \c from the icu-config command. *uckin weird. That said, with Terminal I'm able to build trunk successfully. Weird indeed. Are you comfortable calling this a bug in iTerm and leaving the commit alone? Yes I am. Thanks for looking into it On Tue, Jun 14, 2011 at 7:02 PM, Filipe David Manana fdman...@apache.org wrote: Seems to come from the icu-config command itself: $ icu-config --shlib-c /usr/bin/gcc-4.2 -dynamiclib -dynamic -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c $ icu-config --shlib-cc /usr/bin/g++-4.2 -dynamiclib -dynamic -O2 -O2 -W -Wall -ansi -pedantic -Wpointer-arith -Wwrite-strings -Wno-long-long \c So, either something that can be added to http://svn.apache.org/viewvc?view=revisionrevision=1133287 or revert this? On Tue, Jun 14, 2011 at 6:57 PM, Filipe David Manana fdman...@apache.org wrote: On Tue, Jun 14, 2011 at 6:53 PM, Randall Leeds randall.le...@gmail.com wrote: ... -Wwrite-strings -Wno-long-long \c -I/opt/otp14/lib/erlang/usr/include ... Based on the settings that I have in my build, I think that's at the border of ICU_CFLAGS and ERLANG_FLAGS. Can you check src/couchdb/priv/Makefile for both of these variables and see if either has the \c? Forgot to mention it before. The \c's appear only in the top level Makefile, generted by configure: ICU_BIN = ICU_CFLAGS = -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c ICU_CONFIG = /opt/local/bin/icu-config ICU_CXXFLAGS = -O2 -O2 -W -Wall -ansi -pedantic -Wpointer-arith -Wwrite-strings -Wno-long-long \c ICU_LIBS = -lpthread -lm -L/opt/local/lib -licui18n -licuuc -licudata -lpthread -lm \c Funny, they only appear in the ICU related variables. On Linux these lines end with 4 spaces instead of a \c Emacs whitespace-mode doesn't seem to highlight any weird unprintable characters. Full command line: /bin/sh ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/snappy/google-snappy -I/opt/local/include -I/usr/local/include -I/usr/include -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c -I/opt/otp14/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c -o couch_icu_driver_la-couch_icu_driver.lo `test -f 'icu_driver/couch_icu_driver.c' || echo './'`icu_driver/couch_icu_driver.c libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/snappy/google-snappy -I/opt/local/include -I/usr/local/include -I/usr/include -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c -I/opt/otp14/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c icu_driver/couch_icu_driver.c -fno-common -DPIC -o .libs/couch_icu_driver_la-couch_icu_driver.o i686-apple-darwin10-gcc-4.2.1: c: No such file or directory The same at friendpaste might be more readble: http://friendpaste.com/3ERusm7juiyJLY0grCBQ97 Any idea about this? Agreed. On Jun 12, 2011 4:07 PM, Filipe David Manana fdman...@apache.org wrote: Hi all, After all the recent changes in the autotools config, I'm able to build trunk on Linux, however I'm no longer able to do it on a 32bits Mac OS X (snow leopard). It seems the include path is not what it used to be. The relevant `make`/`make dev` output: /bin/sh ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../../../src/couchdb/priv -I../../.. -I../../../src/snappy/google-snappy -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c -I/Users/fdmanana/tmp/couchdb/apache-couchdb-1.2.0a-0d27a97-git/_inst/lib/erlang/usr/include -I/usr/lib/erlang/usr/include -I/usr/local/lib/erlang/usr/include -I/opt/local/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c -o couch_icu_driver_la-couch_icu_driver.lo `test -f 'icu_driver/couch_icu_driver.c' || echo '../../../../src/couchdb/priv/'`icu_driver/couch_icu_driver.c libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../../../src/couchdb/priv -I../../.. -I../../../src/snappy/google-snappy -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long c -I/Users/fdmanana/tmp/couchdb/apache-couchdb-1.2.0a-0d27a97-git/_inst/lib/erlang/usr/include -I/usr/lib/erlang/usr/include -I/usr/local/lib/erlang/usr
Re: svn commit: r1133561 - /couchdb/trunk/src/couchdb/couch_replicator.erl
Randall, I disagree with this change.
While it's ok for local databases, it causes noise for remote
databases, as the httpc pool is linked to the main replication
gen_server and it starts receiving EXIT messages from the later when
it finishes normally (without errors).
Look at the following gist, provided by Robert Dionne:
https://gist.github.com/1022516
This is making the replication.js test failing for a few folks at
least (Robert, Jan for e.g.)
I would suggest instead this patch:
http://friendpaste.com/2VZvViEvWugHt4TmHFep0M
On Wed, Jun 8, 2011 at 10:30 PM, rand...@apache.org wrote:
Author: randall
Date: Wed Jun 8 21:30:03 2011
New Revision: 1133561
URL: http://svn.apache.org/viewvc?rev=1133561view=rev
Log:
let dbs be closed implicitly when replication dies
We have the supervision tree and monitoring set up with ref counter.
There's really no need to call close here and it generates log noise
when a user DELETEs a local DB involved in replication unless we track
what happened and doing some nil-ing or special casing. Just letting it
all be implicit seems easiest. Closes COUCHDB-863.
Modified:
couchdb/trunk/src/couchdb/couch_replicator.erl
Modified: couchdb/trunk/src/couchdb/couch_replicator.erl
URL:
http://svn.apache.org/viewvc/couchdb/trunk/src/couchdb/couch_replicator.erl?rev=1133561r1=1133560r2=1133561view=diff
==
--- couchdb/trunk/src/couchdb/couch_replicator.erl (original)
+++ couchdb/trunk/src/couchdb/couch_replicator.erl Wed Jun 8 21:30:03 2011
@@ -478,9 +478,7 @@ terminate(Reason, State) -
terminate_cleanup(State) -
couch_task_status:update(Finishing),
stop_db_compaction_notifier(State#rep_state.source_db_compaction_notifier),
-
stop_db_compaction_notifier(State#rep_state.target_db_compaction_notifier),
- couch_api_wrap:db_close(State#rep_state.source),
- couch_api_wrap:db_close(State#rep_state.target).
+
stop_db_compaction_notifier(State#rep_state.target_db_compaction_notifier).
do_last_checkpoint(#rep_state{seqs_in_progress = [],
--
Filipe David Manana,
fdman...@gmail.com, fdman...@apache.org
Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men.
make error with latest trunk (ICU related autotools config)
Hi all, After all the recent changes in the autotools config, I'm able to build trunk on Linux, however I'm no longer able to do it on a 32bits Mac OS X (snow leopard). It seems the include path is not what it used to be. The relevant `make`/`make dev` output: /bin/sh ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../../../src/couchdb/priv -I../../.. -I../../../src/snappy/google-snappy-O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long \c -I/Users/fdmanana/tmp/couchdb/apache-couchdb-1.2.0a-0d27a97-git/_inst/lib/erlang/usr/include -I/usr/lib/erlang/usr/include -I/usr/local/lib/erlang/usr/include -I/opt/local/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c -o couch_icu_driver_la-couch_icu_driver.lo `test -f 'icu_driver/couch_icu_driver.c' || echo '../../../../src/couchdb/priv/'`icu_driver/couch_icu_driver.c libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../../../src/couchdb/priv -I../../.. -I../../../src/snappy/google-snappy -O2 -O2 -Wall -ansi -pedantic -Wshadow -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -Wno-long-long c -I/Users/fdmanana/tmp/couchdb/apache-couchdb-1.2.0a-0d27a97-git/_inst/lib/erlang/usr/include -I/usr/lib/erlang/usr/include -I/usr/local/lib/erlang/usr/include -I/opt/local/lib/erlang/usr/include -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c ../../../../src/couchdb/priv/icu_driver/couch_icu_driver.c -fno-common -DPIC -o .libs/couch_icu_driver_la-couch_icu_driver.o i686-apple-darwin10-gcc-4.2.1: c: No such file or directory ../../../../src/couchdb/priv/icu_driver/couch_icu_driver.c:27:26: error: unicode/ucol.h: No such file or directory ../../../../src/couchdb/priv/icu_driver/couch_icu_driver.c:28:30: error: unicode/ucasemap.h: No such file or directory With trunk revision 1132947 (and before), on this same machine, couch_icu_driver compiles just fine. Relevant make output: /bin/sh ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/snappy/google-snappy -L/usr/local/lib -L/opt/local/lib -I/opt/otp14/lib/erlang/usr/include -I/usr/include -I/usr/include/js -I/usr/include/mozjs -I/usr/local/include -I/opt/local/include -I/usr/local/include/js -I/opt/local/include/js -DXP_UNIX -I/opt/local/include\c -L/opt/local/lib\c -g -O2 -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c -o couch_icu_driver_la-couch_icu_driver.lo `test -f 'icu_driver/couch_icu_driver.c' || echo './'`icu_driver/couch_icu_driver.c libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/snappy/google-snappy -L/usr/local/lib -L/opt/local/lib -I/opt/otp14/lib/erlang/usr/include -I/usr/include -I/usr/include/js -I/usr/include/mozjs -I/usr/local/include -I/opt/local/include -I/usr/local/include/js -I/opt/local/include/js -DXP_UNIX -I/opt/local/includec -L/opt/local/libc -g -O2 -MT couch_icu_driver_la-couch_icu_driver.lo -MD -MP -MF .deps/couch_icu_driver_la-couch_icu_driver.Tpo -c icu_driver/couch_icu_driver.c -fno-common -DPIC -o .libs/couch_icu_driver_la-couch_icu_driver.o mv -f .deps/couch_icu_driver_la-couch_icu_driver.Tpo .deps/couch_icu_driver_la-couch_icu_driver.Plo /bin/sh ../../../libtool --tag=CC --mode=link gcc -I/opt/local/include\c -L/opt/local/lib\c -g -O2 -module -avoid-version -I/opt/local/include\c -L/opt/local/lib\c -L/usr/local/lib -L/opt/local/lib -I/opt/otp14/lib/erlang/usr/include -I/usr/include -I/usr/include/js -I/usr/include/mozjs -I/usr/local/include -I/opt/local/include -I/usr/local/include/js -I/opt/local/include/js -DXP_UNIX -lm -o couch_icu_driver.la -rpath /opt/couchdb/lib/couchdb/erlang/lib/couch-1.2.0a-99a4982-git/priv/lib couch_icu_driver_la-couch_icu_driver.lo -licuuc -licudata -licui18n -L/usr/local/lib -L/opt/local/lib libtool: link: gcc -Wl,-undefined -Wl,dynamic_lookup -o .libs/couch_icu_driver.so -bundle .libs/couch_icu_driver_la-couch_icu_driver.o -L/opt/local/libc -L/usr/local/lib -L/opt/local/lib -lm -licuuc -licudata -licui18n Does anyone else has this issue as well? This is happening to me even with a fresh git checkout. Or does one needs to specify some parameters to configure, etc? -- Filipe David Manana, fdman...@gmail.com, fdman...@apache.org Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
