[jira] [Created] (COUCHDB-3391) The _replicator Database Is Not Scalable or My Design Needs Tweaking
Geoffrey Cox created COUCHDB-3391: - Summary: The _replicator Database Is Not Scalable or My Design Needs Tweaking Key: COUCHDB-3391 URL: https://issues.apache.org/jira/browse/COUCHDB-3391 Project: CouchDB Issue Type: Question Components: Replication Reporter: Geoffrey Cox I think it is important that I elaborate on where I am coming from so that you can understand my use case, please bear with me. Background: I’m a big fan of CouchDB, its offline capabilities and the ecosystem surrounding it, specifically PouchDB. So much so, that I built the Quizster app (https://quizster.co) using CouchDB and PouchDB. Both are amazingly powerful, but they have some rough edges so I’ve had to create a significant amount of software on top of CouchDB/PouchDB and am in the process of open sourcing it. Before I do, I’m looking to migrate this technology from using CouchDB 1 to 2 and this migration is going to take a decent amount of work. I just want to double check that I’m not reinventing the wheel and make sure that there isn’t a better design to what I will elaborate on below, especially since CouchDB 2 appears to have some awesome new features. Consider the following use case for an app that allows students to submit quiz answers digitally. Each student should be able to submit her/his quiz answers and the teacher should be able to view all the answers. This design needs to work with PouchDB as PouchDB speaks directly to the DB and this saves us a lot of time as otherwise an elaborate set of APIs would need to be written. (This solution is similar to what was implemented for Quizster, but it is greatly simplified so that we can focus on just the root of the design). My chosen design consists of a database per student and a database per teacher, i.e. a database per user. Only the owner of the database can edit her/his database and this is enforced via CouchDB roles. When a student submits an answer, it is synced with her/his database via PouchDB. The answers are then replicated to the teacher’s database. This in turn allows the students to quickly load their answers in the app and the teachers to load all the answers for all their students. Of course, there are views in the teacher databases that segment the answers by class, quiz, etc… so that the teacher doesn’t have to load the answers for all their students at once. If we didn’t have the teacher database then a teacher would need access to all the students’ databases and would have to sync with all of the their student’s databases. First question: is this database-per-user design for both students and teachers the best solution or is there a better solution? At first glance, the _replicator database appears to be the the obvious way to replicate the data from the student databases to a single teacher database. The big gotcha is that when you use continuous replication, it consumes a file handle and a database connection which means that you can very quickly starve a database of its resources. For example, if we have say 10,000 students in our database then we need 10,000 concurrent file handles and database connections just for the replications. This is pretty crazy considering that it is unlikely that even say 100 of these 10,000 students would be using the app simultaneously. Instead, I developed a service that listens to the _db_updates feed and then only replicates a database when there is a change to that specific database. With this method, we only worry about consuming resources when there are changes and as a result we end up with plenty of free file handles and database connections. I’ve briefly experimented with CouchDB 2 and it appears that the _replicator database is just as greedy with resources as it was in CouchDB 1. Second question: is there a better way of replicating this data that doesn’t consume so many resources? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator
nickva commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112800117
##
File path: src/couch_replicator/src/couch_replicator.erl
##
@@ -191,847 +120,244 @@ wait_for_result(RepId) ->
end.
-cancel_replication({BaseId, Extension}) ->
-FullRepId = BaseId ++ Extension,
-couch_log:notice("Canceling replication `~s`...", [FullRepId]),
-case supervisor:terminate_child(couch_replicator_job_sup, FullRepId) of
-ok ->
-couch_log:notice("Replication `~s` canceled.", [FullRepId]),
-case supervisor:delete_child(couch_replicator_job_sup, FullRepId) of
-ok ->
-{ok, {cancelled, ?l2b(FullRepId)}};
-{error, not_found} ->
-{ok, {cancelled, ?l2b(FullRepId)}};
-Error ->
-Error
-end;
-Error ->
-couch_log:error("Error canceling replication `~s`: ~p", [FullRepId,
Error]),
-Error
-end.
-
-cancel_replication(RepId, #user_ctx{name = Name, roles = Roles}) ->
-case lists:member(<<"_admin">>, Roles) of
-true ->
-cancel_replication(RepId);
-false ->
-case find_replicator(RepId) of
-{ok, Pid} ->
-case details(Pid) of
-{ok, #rep{user_ctx = #user_ctx{name = Name}}} ->
-cancel_replication(RepId);
-{ok, _} ->
-throw({unauthorized,
-<<"Can't cancel a replication triggered by another
user">>});
-Error ->
-Error
-end;
-Error ->
-Error
-end
-end.
-
-find_replicator({BaseId, Ext} = _RepId) ->
-case lists:keysearch(
-BaseId ++ Ext, 1, supervisor:which_children(couch_replicator_job_sup))
of
-{value, {_, Pid, _, _}} when is_pid(Pid) ->
-{ok, Pid};
-_ ->
-{error, not_found}
-end.
-
-details(Pid) ->
-case (catch gen_server:call(Pid, get_details)) of
-{ok, Rep} ->
-{ok, Rep};
-{'EXIT', {noproc, {gen_server, call, _}}} ->
-{error, not_found};
-Error ->
-throw(Error)
+-spec cancel_replication(rep_id()) ->
+{ok, {cancelled, binary()}} | {error, not_found}.
+cancel_replication({BasedId, Extension} = RepId) ->
+FullRepId = BasedId ++ Extension,
+couch_log:notice("Canceling replication '~s' ...", [FullRepId]),
+case couch_replicator_scheduler:rep_state(RepId) of
+#rep{} ->
+ok = couch_replicator_scheduler:remove_job(RepId),
+couch_log:notice("Replication '~s' cancelled", [FullRepId]),
+{ok, {cancelled, ?l2b(FullRepId)}};
+nil ->
+couch_log:notice("Replication '~s' not found", [FullRepId]),
+{error, not_found}
end.
-init(InitArgs) ->
-{ok, InitArgs, 0}.
-do_init(#rep{options = Options, id = {BaseId, Ext}, user_ctx=UserCtx} = Rep) ->
-process_flag(trap_exit, true),
-
-random:seed(os:timestamp()),
-
-#rep_state{
-source = Source,
-target = Target,
-source_name = SourceName,
-target_name = TargetName,
-start_seq = {_Ts, StartSeq},
-committed_seq = {_, CommittedSeq},
-highest_seq_done = {_, HighestSeq},
-checkpoint_interval = CheckpointInterval
-} = State = init_state(Rep),
-
-NumWorkers = get_value(worker_processes, Options),
-BatchSize = get_value(worker_batch_size, Options),
-{ok, ChangesQueue} = couch_work_queue:new([
-{max_items, BatchSize * NumWorkers * 2},
-{max_size, 100 * 1024 * NumWorkers}
-]),
-% This starts the _changes reader process. It adds the changes from
-% the source db to the ChangesQueue.
-{ok, ChangesReader} = couch_replicator_changes_reader:start_link(
-StartSeq, Source, ChangesQueue, Options
-),
-% Changes manager - responsible for dequeing batches from the changes queue
-% and deliver them to the worker processes.
-ChangesManager = spawn_changes_manager(self(), ChangesQueue, BatchSize),
-% This starts the worker processes. They ask the changes queue manager for
a
-% a batch of _changes rows to process -> check which revs are missing in
the
-% target, and for the missing ones, it copies them from the source to the
target.
-MaxConns = get_value(http_connections, Options),
-Workers = lists:map(
-fun(_) ->
-couch_stats:increment_counter([couch_replicator, workers_started]),
-{ok, Pid} = couch_replicator_worker:start_link(
-self(), Source, Target, ChangesManager, MaxConns),
-Pid
-end,
-lists:seq(1, NumWorkers)),
+-spec replication_states() -> [atom()].
+replication_states() ->
+?REPLICATION_STATES.
-couch_task_status:add_task([
-{type, replication},
-{user, UserCtx#user_ctx.name},
-{replication_id, ?l2b(BaseId ++ Ext)},
-{database, Rep#rep.db_name},
-
[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator
nickva commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112799987
##
File path: src/couch_replicator/src/couch_replicator_connection.erl
##
@@ -0,0 +1,211 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_replicator_connection).
+
+-behavior(gen_server).
+-behavior(config_listener).
+
+-export([start_link/0]).
+-export([init/1, handle_call/3, handle_cast/2, handle_info/2]).
+-export([code_change/3, terminate/2]).
+
+-export([acquire/1, relinquish/1]).
+
+-export([handle_config_change/5, handle_config_terminate/3]).
+
+-define(DEFAULT_CLOSE_INTERVAL, 9).
+-define(RELISTEN_DELAY, 5000).
+
+-record(state, {
+close_interval,
+timer
+}).
+
+-record(connection, {
+worker,
+host,
+port,
+mref
+}).
+
+-include_lib("ibrowse/include/ibrowse.hrl").
+
+
+start_link() ->
+gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+
+init([]) ->
+process_flag(trap_exit, true),
+?MODULE = ets:new(?MODULE, [named_table, public, {keypos,
#connection.worker}]),
+ok = config:listen_for_changes(?MODULE, nil),
+Interval = config:get_integer("replicator", "connection_close_interval",
?DEFAULT_CLOSE_INTERVAL),
+{ok, Timer} = timer:send_after(Interval, close_idle_connections),
+ibrowse:add_config([{inactivity_timeout, Interval}]),
+{ok, #state{close_interval=Interval, timer=Timer}}.
+
+
+acquire(URL) when is_binary(URL) ->
+acquire(binary_to_list(URL));
+
+acquire(URL) ->
+case gen_server:call(?MODULE, {acquire, URL}) of
+{ok, Worker} ->
+link(Worker),
+{ok, Worker};
+{error, all_allocated} ->
+{ok, Pid} = ibrowse:spawn_link_worker_process(URL),
+ok = gen_server:call(?MODULE, {create, URL, Pid}),
+{ok, Pid};
+{error, Reason} ->
+{error, Reason}
+end.
+
+
+relinquish(Worker) ->
+unlink(Worker),
+gen_server:cast(?MODULE, {relinquish, Worker}).
+
+
+handle_call({acquire, URL}, From, State) ->
+{Pid, _Ref} = From,
+case ibrowse_lib:parse_url(URL) of
+#url{host=Host, port=Port} ->
Review comment:
Yap, definitely.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] jaydoane commented on a change in pull request #476: Couchdb 3376 fix mem3 shards
jaydoane commented on a change in pull request #476: Couchdb 3376 fix mem3
shards
URL: https://github.com/apache/couchdb/pull/476#discussion_r112798011
##
File path: src/mem3/src/mem3_shards.erl
##
@@ -417,3 +524,221 @@ filter_shards_by_name(Name, Matches,
[#shard{name=Name}=S|Ss]) ->
filter_shards_by_name(Name, [S|Matches], Ss);
filter_shards_by_name(Name, Matches, [_|Ss]) ->
filter_shards_by_name(Name, Matches, Ss).
+
+
+-ifdef(TEST).
+
+-include_lib("eunit/include/eunit.hrl").
+
+-define(DB, <<"eunit_db_name">>).
+-define(INFINITY, ).
+
+
+mem3_shards_test_() ->
+{
+foreach,
+fun setup/0,
+fun teardown/1,
+[
+t_maybe_spawn_shard_writer_already_exists(),
+t_maybe_spawn_shard_writer_new(),
+t_flush_writer_exists_normal(),
+t_flush_writer_times_out(),
+t_flush_writer_crashes(),
+t_writer_deletes_itself_when_done(),
+t_writer_does_not_delete_other_writers_for_same_shard(),
+t_spawn_writer_in_load_shards_from_db(),
+t_cache_insert_takes_new_update(),
+t_cache_insert_ignores_stale_update_and_kills_worker()
+]
+}.
+
+
+setup() ->
+ets:new(?SHARDS, [bag, public, named_table, {keypos, #shard.dbname}]),
+ets:new(?OPENERS, [bag, public, named_table]),
+ets:new(?DBS, [set, public, named_table]),
+ets:new(?ATIMES, [ordered_set, public, named_table]),
+meck:expect(config, get, ["mem3", "shards_db", '_'], "_dbs"),
+ok.
+
+
+teardown(_) ->
+meck:unload(),
+ets:delete(?ATIMES),
+ets:delete(?DBS),
+ets:delete(?OPENERS),
+ets:delete(?SHARDS).
+
+
+t_maybe_spawn_shard_writer_already_exists() ->
+?_test(begin
+ets:insert(?OPENERS, {?DB, self()}),
+Shards = mock_shards(),
+WRes = maybe_spawn_shard_writer(?DB, Shards, ?INFINITY),
+?assertEqual(ignore, WRes)
+end).
+
+
+t_maybe_spawn_shard_writer_new() ->
+?_test(begin
+Shards = mock_shards(),
+WPid = maybe_spawn_shard_writer(?DB, Shards, 1000),
+WRef = erlang:monitor(process, WPid),
+?assert(is_pid(WPid)),
+?assert(is_process_alive(WPid)),
+WPid ! write,
+?assertEqual(normal, wait_writer_result(WRef)),
+?assertEqual(Shards, ets:tab2list(?SHARDS))
+end).
+
+
+t_flush_writer_exists_normal() ->
+?_test(begin
+Shards = mock_shards(),
+WPid = spawn_link_mock_writer(?DB, Shards, ?INFINITY),
+?assertEqual(ok, flush_write(?DB, WPid, ?INFINITY)),
+?assertEqual(Shards, ets:tab2list(?SHARDS))
+end).
+
+
+t_flush_writer_times_out() ->
+?_test(begin
+WPid = spawn(fun() -> receive will_never_receive_this -> ok end end),
+Error = {mem3_shards_write_timeout, ?DB},
+?assertExit(Error, flush_write(?DB, WPid, 100)),
+exit(WPid, kill)
+end).
+
+
+t_flush_writer_crashes() ->
+?_test(begin
+WPid = spawn(fun() -> receive write -> exit('kapow!') end end),
+Error = {mem3_shards_bad_write, 'kapow!'},
+?assertExit(Error, flush_write(?DB, WPid, 1000))
+end).
+
+
+t_writer_deletes_itself_when_done() ->
+?_test(begin
+Shards = mock_shards(),
+WPid = spawn_link_mock_writer(?DB, Shards, ?INFINITY),
+WRef = erlang:monitor(process, WPid),
+ets:insert(?OPENERS, {?DB, WPid}),
+WPid ! write,
+?assertEqual(normal, wait_writer_result(WRef)),
+?assertEqual(Shards, ets:tab2list(?SHARDS)),
+?assertEqual([], ets:tab2list(?OPENERS))
+end).
+
+
+t_writer_does_not_delete_other_writers_for_same_shard() ->
+?_test(begin
+Shards = mock_shards(),
+WPid = spawn_link_mock_writer(?DB, Shards, ?INFINITY),
+WRef = erlang:monitor(process, WPid),
+ets:insert(?OPENERS, {?DB, WPid}),
+ets:insert(?OPENERS, {?DB, self()}), % should not be deleted
+WPid ! write,
+?assertEqual(normal, wait_writer_result(WRef)),
+?assertEqual(Shards, ets:tab2list(?SHARDS)),
+?assertEqual(1, ets:info(?OPENERS, size)),
+?assertEqual([{?DB, self()}], ets:tab2list(?OPENERS))
+end).
+
+
+t_spawn_writer_in_load_shards_from_db() ->
+?_test(begin
+meck:expect(couch_db, open_doc, 3, {ok, #doc{body = {[]}}}),
+meck:expect(couch_db, get_update_seq, 1, 1),
+meck:expect(mem3_util, build_ordered_shards, 2, mock_shards()),
+erlang:register(?MODULE, self()), % register to get cache_insert cast
+load_shards_from_db(#db{name = <<"testdb">>}, ?DB),
+meck:validate(couch_db),
+meck:validate(mem3_util),
+Cast = receive
+{'$gen_cast', Msg} -> Msg
+after 1000 ->
+timeout
+end,
+?assertMatch({cache_insert, ?DB, Pid, 1} when is_pid(Pid), Cast),
+{cache_insert, _, WPid, _} = Cast,
+exit(WPid, kill),
+
[GitHub] iilyak commented on issue #470: Scheduling Replicator
iilyak commented on issue #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#issuecomment-296315646
Tests pass locally
```
EUnit
module 'json_stream_parse'
module 'couch_replicator_worker'
module 'couch_replicator_utils'
module 'couch_replicator_test_helper'
module 'couch_replicator_sup'
module 'couch_replicator_stats'
module 'couch_replicator_small_max_request_size_target'
module 'couch_replicator_scheduler_sup'
module 'couch_replicator_scheduler_job'
module 'couch_replicator_scheduler'
module 'couch_replicator_rate_limiter_tables'
module 'couch_replicator_rate_limiter'
module 'couch_replicator_rate_limiter_tests'
module 'couch_replicator_notifier'
module 'couch_replicator_manager'
module 'couch_replicator_job_sup'
module 'couch_replicator_ids'
module 'couch_replicator_httpd_util'
module 'couch_replicator_httpd'
module 'couch_replicator_httpc_pool'
module 'couch_replicator_httpc_pool_tests'
module 'couch_replicator_httpc'
module 'couch_replicator_filters'
module 'couch_replicator_fabric_rpc'
module 'couch_replicator_fabric'
module 'couch_replicator_docs'
module 'couch_replicator_doc_processor_worker'
module 'couch_replicator_doc_processor'
module 'couch_replicator_db_changes'
module 'couch_replicator_connection'
module 'couch_replicator_connection_tests'
module 'couch_replicator_clustering'
module 'couch_replicator_changes_reader'
module 'couch_replicator_app'
module 'couch_replicator'
module 'couch_replicator_use_checkpoints_tests'
module 'couch_replicator_selector_tests'
module 'couch_replicator_proxy_tests'
module 'couch_replicator_modules_load_tests'
module 'couch_replicator_missing_stubs_tests'
module 'couch_replicator_many_leaves_tests'
module 'couch_replicator_large_atts_tests'
module 'couch_replicator_id_too_long_tests'
module 'couch_replicator_filtered_tests'
module 'couch_replicator_compact_tests'
===
All 300 tests passed.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (COUCHDB-3324) Scheduling Replicator
[ https://issues.apache.org/jira/browse/COUCHDB-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979442#comment-15979442 ] ASF subversion and git services commented on COUCHDB-3324: -- Commit 2d8d58bff41f7bfda8ab106e462d108990595615 in couchdb's branch refs/heads/63012-scheduler from [~vatamane] [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=2d8d58b ] Stitch scheduling replicator together. Glue together all the scheduling replicator pieces. Scheduler is the main component. It can run a large number of replication jobs by switching between them, stopping and starting some periodically. Jobs which fail are backed off exponentially. Normal (non-continuous) jobs will be allowed to run to completion to preserve their current semantics. Scheduler behavior can configured by these configuration options in `[replicator]` sections: * `max_jobs` : Number of actively running replications. Making this too high could cause performance issues. Making it too low could mean replications jobs might not have enough time to make progress before getting unscheduled again. This parameter can be adjusted at runtime and will take effect during next reschudling cycle. * `interval` : Scheduling interval in milliseconds. During each reschedule cycle scheduler might start or stop up to "max_churn" number of jobs. * `max_churn` : Maximum number of replications to start and stop during rescheduling. This parameter along with "interval" defines the rate of job replacement. During startup, however a much larger number of jobs could be started (up to max_jobs) in short period of time. Replication jobs are added to the scheduler by the document processor or from the `couch_replicator:replicate/2` function when called from `_replicate` HTTP endpoint handler. Document processor listens for updates via couch_mutlidb_changes module then tries to add replication jobs to the scheduler. Sometimes translating a document update to a replication job could fail, either permantly (if document is malformed and missing some expected fields for example) or temporarily if it is a filtered replication and filter cannot be fetched. A failed filter fetch will be retried with an exponential backoff. couch_replicator_clustering is in charge of monitoring cluster membership changes. When membership changes, after a configurable quiet period, a rescan will be initiated. Rescan will shufle replication jobs to make sure a replication job is running on only one node. A new set of stats were added to introspect scheduler and doc processor internals. The top replication supervisor structure is `rest_for_one`. This means if a child crashes, all children to the "right" of it will be restarted (if visualized supervisor hierarchy as an upside-down tree). Clustering, connection pool and rate limiter are towards the "left" as they are more fundamental, if clustering child crashes, most other components will be restart. Doc process or and multi-db changes children are towards the "right". If they crash, they can be safely restarted without affecting already running replication or components like clustering or connection pool. Jira: COUCHDB-3324 > Scheduling Replicator > - > > Key: COUCHDB-3324 > URL: https://issues.apache.org/jira/browse/COUCHDB-3324 > Project: CouchDB > Issue Type: New Feature >Reporter: Nick Vatamaniuc > > Improve CouchDB replicator > * Allow running a large number of replication jobs > * Improve API with a focus on ease of use and performance. Avoid updating > replication document with transient state updates. Instead create a proper > API for querying replication states. At the same time provide a compatibility > mode to let users keep existing behavior (of getting updates in documents). > * Improve network resource usage and performance. Multiple connection to the > same cluster could share socket connections > * Handle rate limiting on target and source HTTP endpoints. Let replication > request auto-discover rate limit capacity based on a proven algorithm such as > Additive Increase / Multiplicative Decrease feedback control loop. > * Improve performance by avoiding repeatedly retrying failing replication > jobs. Instead use exponential backoff. > * Improve recovery from long (but temporary) network failure. Currently if > replications jobs fail to start 10 times in a row they will not be retried > anymore. This is not always desirable. In case of a long enough DNS (or other > network) failure replication jobs will effectively stop until they are > manually restarted. > * Better handling of filtered replications: Failing to fetch filters could > block couch replicator manager, lead to message queue backups and memory > exhaustion. Also, when replication filter code changes update replication > accordingly (replication
[GitHub] iilyak commented on issue #470: Scheduling Replicator
iilyak commented on issue #470: Scheduling Replicator URL: https://github.com/apache/couchdb/pull/470#issuecomment-296312054 foundings -> findings This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator
nickva commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112781160
##
File path: src/couch_replicator/src/couch_replicator.erl
##
@@ -191,847 +120,244 @@ wait_for_result(RepId) ->
end.
-cancel_replication({BaseId, Extension}) ->
-FullRepId = BaseId ++ Extension,
-couch_log:notice("Canceling replication `~s`...", [FullRepId]),
-case supervisor:terminate_child(couch_replicator_job_sup, FullRepId) of
-ok ->
-couch_log:notice("Replication `~s` canceled.", [FullRepId]),
-case supervisor:delete_child(couch_replicator_job_sup, FullRepId) of
-ok ->
-{ok, {cancelled, ?l2b(FullRepId)}};
-{error, not_found} ->
-{ok, {cancelled, ?l2b(FullRepId)}};
-Error ->
-Error
-end;
-Error ->
-couch_log:error("Error canceling replication `~s`: ~p", [FullRepId,
Error]),
-Error
-end.
-
-cancel_replication(RepId, #user_ctx{name = Name, roles = Roles}) ->
-case lists:member(<<"_admin">>, Roles) of
-true ->
-cancel_replication(RepId);
-false ->
-case find_replicator(RepId) of
-{ok, Pid} ->
-case details(Pid) of
-{ok, #rep{user_ctx = #user_ctx{name = Name}}} ->
-cancel_replication(RepId);
-{ok, _} ->
-throw({unauthorized,
-<<"Can't cancel a replication triggered by another
user">>});
-Error ->
-Error
-end;
-Error ->
-Error
-end
-end.
-
-find_replicator({BaseId, Ext} = _RepId) ->
-case lists:keysearch(
-BaseId ++ Ext, 1, supervisor:which_children(couch_replicator_job_sup))
of
-{value, {_, Pid, _, _}} when is_pid(Pid) ->
-{ok, Pid};
-_ ->
-{error, not_found}
-end.
-
-details(Pid) ->
-case (catch gen_server:call(Pid, get_details)) of
-{ok, Rep} ->
-{ok, Rep};
-{'EXIT', {noproc, {gen_server, call, _}}} ->
-{error, not_found};
-Error ->
-throw(Error)
+-spec cancel_replication(rep_id()) ->
+{ok, {cancelled, binary()}} | {error, not_found}.
+cancel_replication({BasedId, Extension} = RepId) ->
+FullRepId = BasedId ++ Extension,
+couch_log:notice("Canceling replication '~s' ...", [FullRepId]),
+case couch_replicator_scheduler:rep_state(RepId) of
+#rep{} ->
+ok = couch_replicator_scheduler:remove_job(RepId),
+couch_log:notice("Replication '~s' cancelled", [FullRepId]),
+{ok, {cancelled, ?l2b(FullRepId)}};
+nil ->
+couch_log:notice("Replication '~s' not found", [FullRepId]),
+{error, not_found}
end.
-init(InitArgs) ->
-{ok, InitArgs, 0}.
-do_init(#rep{options = Options, id = {BaseId, Ext}, user_ctx=UserCtx} = Rep) ->
-process_flag(trap_exit, true),
-
-random:seed(os:timestamp()),
-
-#rep_state{
-source = Source,
-target = Target,
-source_name = SourceName,
-target_name = TargetName,
-start_seq = {_Ts, StartSeq},
-committed_seq = {_, CommittedSeq},
-highest_seq_done = {_, HighestSeq},
-checkpoint_interval = CheckpointInterval
-} = State = init_state(Rep),
-
-NumWorkers = get_value(worker_processes, Options),
-BatchSize = get_value(worker_batch_size, Options),
-{ok, ChangesQueue} = couch_work_queue:new([
-{max_items, BatchSize * NumWorkers * 2},
-{max_size, 100 * 1024 * NumWorkers}
-]),
-% This starts the _changes reader process. It adds the changes from
-% the source db to the ChangesQueue.
-{ok, ChangesReader} = couch_replicator_changes_reader:start_link(
-StartSeq, Source, ChangesQueue, Options
-),
-% Changes manager - responsible for dequeing batches from the changes queue
-% and deliver them to the worker processes.
-ChangesManager = spawn_changes_manager(self(), ChangesQueue, BatchSize),
-% This starts the worker processes. They ask the changes queue manager for
a
-% a batch of _changes rows to process -> check which revs are missing in
the
-% target, and for the missing ones, it copies them from the source to the
target.
-MaxConns = get_value(http_connections, Options),
-Workers = lists:map(
-fun(_) ->
-couch_stats:increment_counter([couch_replicator, workers_started]),
-{ok, Pid} = couch_replicator_worker:start_link(
-self(), Source, Target, ChangesManager, MaxConns),
-Pid
-end,
-lists:seq(1, NumWorkers)),
+-spec replication_states() -> [atom()].
+replication_states() ->
+?REPLICATION_STATES.
-couch_task_status:add_task([
-{type, replication},
-{user, UserCtx#user_ctx.name},
-{replication_id, ?l2b(BaseId ++ Ext)},
-{database, Rep#rep.db_name},
-
[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator
nickva commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112781160
##
File path: src/couch_replicator/src/couch_replicator.erl
##
@@ -191,847 +120,244 @@ wait_for_result(RepId) ->
end.
-cancel_replication({BaseId, Extension}) ->
-FullRepId = BaseId ++ Extension,
-couch_log:notice("Canceling replication `~s`...", [FullRepId]),
-case supervisor:terminate_child(couch_replicator_job_sup, FullRepId) of
-ok ->
-couch_log:notice("Replication `~s` canceled.", [FullRepId]),
-case supervisor:delete_child(couch_replicator_job_sup, FullRepId) of
-ok ->
-{ok, {cancelled, ?l2b(FullRepId)}};
-{error, not_found} ->
-{ok, {cancelled, ?l2b(FullRepId)}};
-Error ->
-Error
-end;
-Error ->
-couch_log:error("Error canceling replication `~s`: ~p", [FullRepId,
Error]),
-Error
-end.
-
-cancel_replication(RepId, #user_ctx{name = Name, roles = Roles}) ->
-case lists:member(<<"_admin">>, Roles) of
-true ->
-cancel_replication(RepId);
-false ->
-case find_replicator(RepId) of
-{ok, Pid} ->
-case details(Pid) of
-{ok, #rep{user_ctx = #user_ctx{name = Name}}} ->
-cancel_replication(RepId);
-{ok, _} ->
-throw({unauthorized,
-<<"Can't cancel a replication triggered by another
user">>});
-Error ->
-Error
-end;
-Error ->
-Error
-end
-end.
-
-find_replicator({BaseId, Ext} = _RepId) ->
-case lists:keysearch(
-BaseId ++ Ext, 1, supervisor:which_children(couch_replicator_job_sup))
of
-{value, {_, Pid, _, _}} when is_pid(Pid) ->
-{ok, Pid};
-_ ->
-{error, not_found}
-end.
-
-details(Pid) ->
-case (catch gen_server:call(Pid, get_details)) of
-{ok, Rep} ->
-{ok, Rep};
-{'EXIT', {noproc, {gen_server, call, _}}} ->
-{error, not_found};
-Error ->
-throw(Error)
+-spec cancel_replication(rep_id()) ->
+{ok, {cancelled, binary()}} | {error, not_found}.
+cancel_replication({BasedId, Extension} = RepId) ->
+FullRepId = BasedId ++ Extension,
+couch_log:notice("Canceling replication '~s' ...", [FullRepId]),
+case couch_replicator_scheduler:rep_state(RepId) of
+#rep{} ->
+ok = couch_replicator_scheduler:remove_job(RepId),
+couch_log:notice("Replication '~s' cancelled", [FullRepId]),
+{ok, {cancelled, ?l2b(FullRepId)}};
+nil ->
+couch_log:notice("Replication '~s' not found", [FullRepId]),
+{error, not_found}
end.
-init(InitArgs) ->
-{ok, InitArgs, 0}.
-do_init(#rep{options = Options, id = {BaseId, Ext}, user_ctx=UserCtx} = Rep) ->
-process_flag(trap_exit, true),
-
-random:seed(os:timestamp()),
-
-#rep_state{
-source = Source,
-target = Target,
-source_name = SourceName,
-target_name = TargetName,
-start_seq = {_Ts, StartSeq},
-committed_seq = {_, CommittedSeq},
-highest_seq_done = {_, HighestSeq},
-checkpoint_interval = CheckpointInterval
-} = State = init_state(Rep),
-
-NumWorkers = get_value(worker_processes, Options),
-BatchSize = get_value(worker_batch_size, Options),
-{ok, ChangesQueue} = couch_work_queue:new([
-{max_items, BatchSize * NumWorkers * 2},
-{max_size, 100 * 1024 * NumWorkers}
-]),
-% This starts the _changes reader process. It adds the changes from
-% the source db to the ChangesQueue.
-{ok, ChangesReader} = couch_replicator_changes_reader:start_link(
-StartSeq, Source, ChangesQueue, Options
-),
-% Changes manager - responsible for dequeing batches from the changes queue
-% and deliver them to the worker processes.
-ChangesManager = spawn_changes_manager(self(), ChangesQueue, BatchSize),
-% This starts the worker processes. They ask the changes queue manager for
a
-% a batch of _changes rows to process -> check which revs are missing in
the
-% target, and for the missing ones, it copies them from the source to the
target.
-MaxConns = get_value(http_connections, Options),
-Workers = lists:map(
-fun(_) ->
-couch_stats:increment_counter([couch_replicator, workers_started]),
-{ok, Pid} = couch_replicator_worker:start_link(
-self(), Source, Target, ChangesManager, MaxConns),
-Pid
-end,
-lists:seq(1, NumWorkers)),
+-spec replication_states() -> [atom()].
+replication_states() ->
+?REPLICATION_STATES.
-couch_task_status:add_task([
-{type, replication},
-{user, UserCtx#user_ctx.name},
-{replication_id, ?l2b(BaseId ++ Ext)},
-{database, Rep#rep.db_name},
-
[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator
nickva commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112781160
##
File path: src/couch_replicator/src/couch_replicator.erl
##
@@ -191,847 +120,244 @@ wait_for_result(RepId) ->
end.
-cancel_replication({BaseId, Extension}) ->
-FullRepId = BaseId ++ Extension,
-couch_log:notice("Canceling replication `~s`...", [FullRepId]),
-case supervisor:terminate_child(couch_replicator_job_sup, FullRepId) of
-ok ->
-couch_log:notice("Replication `~s` canceled.", [FullRepId]),
-case supervisor:delete_child(couch_replicator_job_sup, FullRepId) of
-ok ->
-{ok, {cancelled, ?l2b(FullRepId)}};
-{error, not_found} ->
-{ok, {cancelled, ?l2b(FullRepId)}};
-Error ->
-Error
-end;
-Error ->
-couch_log:error("Error canceling replication `~s`: ~p", [FullRepId,
Error]),
-Error
-end.
-
-cancel_replication(RepId, #user_ctx{name = Name, roles = Roles}) ->
-case lists:member(<<"_admin">>, Roles) of
-true ->
-cancel_replication(RepId);
-false ->
-case find_replicator(RepId) of
-{ok, Pid} ->
-case details(Pid) of
-{ok, #rep{user_ctx = #user_ctx{name = Name}}} ->
-cancel_replication(RepId);
-{ok, _} ->
-throw({unauthorized,
-<<"Can't cancel a replication triggered by another
user">>});
-Error ->
-Error
-end;
-Error ->
-Error
-end
-end.
-
-find_replicator({BaseId, Ext} = _RepId) ->
-case lists:keysearch(
-BaseId ++ Ext, 1, supervisor:which_children(couch_replicator_job_sup))
of
-{value, {_, Pid, _, _}} when is_pid(Pid) ->
-{ok, Pid};
-_ ->
-{error, not_found}
-end.
-
-details(Pid) ->
-case (catch gen_server:call(Pid, get_details)) of
-{ok, Rep} ->
-{ok, Rep};
-{'EXIT', {noproc, {gen_server, call, _}}} ->
-{error, not_found};
-Error ->
-throw(Error)
+-spec cancel_replication(rep_id()) ->
+{ok, {cancelled, binary()}} | {error, not_found}.
+cancel_replication({BasedId, Extension} = RepId) ->
+FullRepId = BasedId ++ Extension,
+couch_log:notice("Canceling replication '~s' ...", [FullRepId]),
+case couch_replicator_scheduler:rep_state(RepId) of
+#rep{} ->
+ok = couch_replicator_scheduler:remove_job(RepId),
+couch_log:notice("Replication '~s' cancelled", [FullRepId]),
+{ok, {cancelled, ?l2b(FullRepId)}};
+nil ->
+couch_log:notice("Replication '~s' not found", [FullRepId]),
+{error, not_found}
end.
-init(InitArgs) ->
-{ok, InitArgs, 0}.
-do_init(#rep{options = Options, id = {BaseId, Ext}, user_ctx=UserCtx} = Rep) ->
-process_flag(trap_exit, true),
-
-random:seed(os:timestamp()),
-
-#rep_state{
-source = Source,
-target = Target,
-source_name = SourceName,
-target_name = TargetName,
-start_seq = {_Ts, StartSeq},
-committed_seq = {_, CommittedSeq},
-highest_seq_done = {_, HighestSeq},
-checkpoint_interval = CheckpointInterval
-} = State = init_state(Rep),
-
-NumWorkers = get_value(worker_processes, Options),
-BatchSize = get_value(worker_batch_size, Options),
-{ok, ChangesQueue} = couch_work_queue:new([
-{max_items, BatchSize * NumWorkers * 2},
-{max_size, 100 * 1024 * NumWorkers}
-]),
-% This starts the _changes reader process. It adds the changes from
-% the source db to the ChangesQueue.
-{ok, ChangesReader} = couch_replicator_changes_reader:start_link(
-StartSeq, Source, ChangesQueue, Options
-),
-% Changes manager - responsible for dequeing batches from the changes queue
-% and deliver them to the worker processes.
-ChangesManager = spawn_changes_manager(self(), ChangesQueue, BatchSize),
-% This starts the worker processes. They ask the changes queue manager for
a
-% a batch of _changes rows to process -> check which revs are missing in
the
-% target, and for the missing ones, it copies them from the source to the
target.
-MaxConns = get_value(http_connections, Options),
-Workers = lists:map(
-fun(_) ->
-couch_stats:increment_counter([couch_replicator, workers_started]),
-{ok, Pid} = couch_replicator_worker:start_link(
-self(), Source, Target, ChangesManager, MaxConns),
-Pid
-end,
-lists:seq(1, NumWorkers)),
+-spec replication_states() -> [atom()].
+replication_states() ->
+?REPLICATION_STATES.
-couch_task_status:add_task([
-{type, replication},
-{user, UserCtx#user_ctx.name},
-{replication_id, ?l2b(BaseId ++ Ext)},
-{database, Rep#rep.db_name},
-
[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator
iilyak commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112780051
##
File path: src/couch_replicator/src/couch_replicator.erl
##
@@ -191,847 +120,244 @@ wait_for_result(RepId) ->
end.
-cancel_replication({BaseId, Extension}) ->
-FullRepId = BaseId ++ Extension,
-couch_log:notice("Canceling replication `~s`...", [FullRepId]),
-case supervisor:terminate_child(couch_replicator_job_sup, FullRepId) of
-ok ->
-couch_log:notice("Replication `~s` canceled.", [FullRepId]),
-case supervisor:delete_child(couch_replicator_job_sup, FullRepId) of
-ok ->
-{ok, {cancelled, ?l2b(FullRepId)}};
-{error, not_found} ->
-{ok, {cancelled, ?l2b(FullRepId)}};
-Error ->
-Error
-end;
-Error ->
-couch_log:error("Error canceling replication `~s`: ~p", [FullRepId,
Error]),
-Error
-end.
-
-cancel_replication(RepId, #user_ctx{name = Name, roles = Roles}) ->
-case lists:member(<<"_admin">>, Roles) of
-true ->
-cancel_replication(RepId);
-false ->
-case find_replicator(RepId) of
-{ok, Pid} ->
-case details(Pid) of
-{ok, #rep{user_ctx = #user_ctx{name = Name}}} ->
-cancel_replication(RepId);
-{ok, _} ->
-throw({unauthorized,
-<<"Can't cancel a replication triggered by another
user">>});
-Error ->
-Error
-end;
-Error ->
-Error
-end
-end.
-
-find_replicator({BaseId, Ext} = _RepId) ->
-case lists:keysearch(
-BaseId ++ Ext, 1, supervisor:which_children(couch_replicator_job_sup))
of
-{value, {_, Pid, _, _}} when is_pid(Pid) ->
-{ok, Pid};
-_ ->
-{error, not_found}
-end.
-
-details(Pid) ->
-case (catch gen_server:call(Pid, get_details)) of
-{ok, Rep} ->
-{ok, Rep};
-{'EXIT', {noproc, {gen_server, call, _}}} ->
-{error, not_found};
-Error ->
-throw(Error)
+-spec cancel_replication(rep_id()) ->
+{ok, {cancelled, binary()}} | {error, not_found}.
+cancel_replication({BasedId, Extension} = RepId) ->
+FullRepId = BasedId ++ Extension,
+couch_log:notice("Canceling replication '~s' ...", [FullRepId]),
+case couch_replicator_scheduler:rep_state(RepId) of
+#rep{} ->
+ok = couch_replicator_scheduler:remove_job(RepId),
+couch_log:notice("Replication '~s' cancelled", [FullRepId]),
+{ok, {cancelled, ?l2b(FullRepId)}};
+nil ->
+couch_log:notice("Replication '~s' not found", [FullRepId]),
+{error, not_found}
end.
-init(InitArgs) ->
-{ok, InitArgs, 0}.
-do_init(#rep{options = Options, id = {BaseId, Ext}, user_ctx=UserCtx} = Rep) ->
-process_flag(trap_exit, true),
-
-random:seed(os:timestamp()),
-
-#rep_state{
-source = Source,
-target = Target,
-source_name = SourceName,
-target_name = TargetName,
-start_seq = {_Ts, StartSeq},
-committed_seq = {_, CommittedSeq},
-highest_seq_done = {_, HighestSeq},
-checkpoint_interval = CheckpointInterval
-} = State = init_state(Rep),
-
-NumWorkers = get_value(worker_processes, Options),
-BatchSize = get_value(worker_batch_size, Options),
-{ok, ChangesQueue} = couch_work_queue:new([
-{max_items, BatchSize * NumWorkers * 2},
-{max_size, 100 * 1024 * NumWorkers}
-]),
-% This starts the _changes reader process. It adds the changes from
-% the source db to the ChangesQueue.
-{ok, ChangesReader} = couch_replicator_changes_reader:start_link(
-StartSeq, Source, ChangesQueue, Options
-),
-% Changes manager - responsible for dequeing batches from the changes queue
-% and deliver them to the worker processes.
-ChangesManager = spawn_changes_manager(self(), ChangesQueue, BatchSize),
-% This starts the worker processes. They ask the changes queue manager for
a
-% a batch of _changes rows to process -> check which revs are missing in
the
-% target, and for the missing ones, it copies them from the source to the
target.
-MaxConns = get_value(http_connections, Options),
-Workers = lists:map(
-fun(_) ->
-couch_stats:increment_counter([couch_replicator, workers_started]),
-{ok, Pid} = couch_replicator_worker:start_link(
-self(), Source, Target, ChangesManager, MaxConns),
-Pid
-end,
-lists:seq(1, NumWorkers)),
+-spec replication_states() -> [atom()].
+replication_states() ->
+?REPLICATION_STATES.
-couch_task_status:add_task([
-{type, replication},
-{user, UserCtx#user_ctx.name},
-{replication_id, ?l2b(BaseId ++ Ext)},
-{database, Rep#rep.db_name},
-
[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator
nickva commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112780172
##
File path: src/couch_replicator/src/couch_replicator_scheduler.erl
##
@@ -0,0 +1,1430 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_replicator_scheduler).
+
+-behaviour(gen_server).
+-behaviour(config_listener).
+
+-export([
+start_link/0
+]).
+
+-export([
+ init/1,
+ terminate/2,
+ handle_call/3,
+ handle_info/2,
+ handle_cast/2,
+ code_change/3,
+ format_status/2
+]).
+
+-export([
Review comment:
Here is what I have:
```
(node1@127.0.0.1)3>
couch_replicator:restart_job(<<"e327d79214831ca4c11550b4a453c9ba+continuous">>).
{ok,{[{id,<<"e327d79214831ca4c11550b4a453c9ba+continuous">>},
{pid,<<"<0.3805.0>">>},
{source,<<"http://adm:*@localhost:15984/cdyno-001/;>>},
{target,<<"http://adm:*@localhost:15984/cdyno-002/;>>},
{database,<<"shards/6000-7fff/_replicator.1492636029">>},
{user,null},
{doc_id,<<"cdyno-001-002">>},
{history,[{[{timestamp,<<"2017-04-21T21:27:03Z">>},
{type,started}]},
{[{timestamp,<<"2017-04-21T21:27:03Z">>},{type,added}]}]},
{node,'node2@127.0.0.1'},
{start_time,<<"2017-04-21T21:26:43Z">>}]}}
```
Will push changes soon.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator
nickva commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112780103
##
File path: src/couch_replicator/src/couch_replicator_scheduler.erl
##
@@ -0,0 +1,1430 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_replicator_scheduler).
+
+-behaviour(gen_server).
+-behaviour(config_listener).
+
+-export([
+start_link/0
+]).
+
+-export([
+ init/1,
+ terminate/2,
+ handle_call/3,
+ handle_info/2,
+ handle_cast/2,
+ code_change/3,
+ format_status/2
+]).
+
+-export([
+ add_job/1,
+ remove_job/1,
+ reschedule/0,
+ rep_state/1,
+ find_jobs_by_dbname/1,
+ find_jobs_by_doc/2,
+ job_summary/2,
+ health_threshold/0,
+ jobs/0,
+ job/1
+]).
+
+%% config_listener callbacks
+-export([
+handle_config_change/5,
+handle_config_terminate/3
+]).
+
+%% for status updater process to allow hot code loading
+-export([
+stats_updater_loop/1
+]).
+
+-include("couch_replicator_scheduler.hrl").
+-include("couch_replicator.hrl").
+-include("couch_replicator_api_wrap.hrl").
+-include_lib("couch/include/couch_db.hrl").
+
+%% types
+-type event_type() :: added | started | stopped | {crashed, any()}.
+-type event() :: {Type:: event_type(), When :: erlang:timestamp()}.
+-type history() :: nonempty_list(event()).
+
+%% definitions
+-define(MAX_BACKOFF_EXPONENT, 10).
+-define(BACKOFF_INTERVAL_MICROS, 30 * 1000 * 1000).
+-define(DEFAULT_HEALTH_THRESHOLD_SEC, 2 * 60).
+-define(RELISTEN_DELAY, 5000).
+-define(STATS_UPDATE_WAIT, 5000).
+
+-define(DEFAULT_MAX_JOBS, 500).
+-define(DEFAULT_MAX_CHURN, 20).
+-define(DEFAULT_MAX_HISTORY, 20).
+-define(DEFAULT_SCHEDULER_INTERVAL, 6).
+
+
+-record(state, {interval, timer, max_jobs, max_churn, max_history, stats_pid}).
+-record(job, {
+id :: job_id() | '$1' | '_',
+rep :: #rep{} | '_',
+pid :: undefined | pid() | '$1' | '_',
+monitor :: undefined | reference() | '_',
+history :: history() | '_'
+}).
+
+-record(stats_acc, {
+pending_n = 0 :: non_neg_integer(),
+running_n = 0 :: non_neg_integer(),
+crashed_n = 0 :: non_neg_integer()
+}).
+
+
+%% public functions
+
+-spec start_link() -> {ok, pid()} | ignore | {error, term()}.
+start_link() ->
+gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+
+-spec add_job(#rep{}) -> ok.
+add_job(#rep{} = Rep) when Rep#rep.id /= undefined ->
+Job = #job{
+id = Rep#rep.id,
+rep = Rep,
+history = [{added, os:timestamp()}]
+},
+gen_server:call(?MODULE, {add_job, Job}, infinity).
+
+
+-spec remove_job(job_id()) -> ok.
+remove_job(Id) ->
+gen_server:call(?MODULE, {remove_job, Id}, infinity).
+
+
+-spec reschedule() -> ok.
+% Trigger a manual reschedule. Used for testing and/or ops.
+reschedule() ->
+gen_server:call(?MODULE, reschedule, infinity).
+
+
+-spec rep_state(rep_id()) -> #rep{} | nil.
+rep_state(RepId) ->
+case (catch ets:lookup_element(?MODULE, RepId, #job.rep)) of
+{'EXIT',{badarg, _}} ->
+nil;
+Rep ->
+Rep
+end.
+
+
+-spec job_summary(job_id(), non_neg_integer()) -> [_] | nil.
+job_summary(JobId, HealthThreshold) ->
+case job_by_id(JobId) of
+{ok, #job{pid = Pid, history = History, rep = Rep}} ->
+ErrorCount = consecutive_crashes(History, HealthThreshold),
+{State, Info} = case {Pid, ErrorCount} of
+{undefined, 0} ->
+{pending, null};
+{undefined, ErrorCount} when ErrorCount > 0 ->
+ [{{crashed, Error}, _When} | _] = History,
+ ErrMsg =
couch_replicator_utils:rep_error_to_binary(Error),
+ {crashing, ErrMsg};
+{Pid, ErrorCount} when is_pid(Pid) ->
+ {running, null}
+end,
+[
+{source, iolist_to_binary(ejson_url(Rep#rep.source))},
+{target, iolist_to_binary(ejson_url(Rep#rep.target))},
+{state, State},
+{info, Info},
+{error_count, ErrorCount},
+{last_updated, last_updated(History)},
+{start_time,
+couch_replicator_utils:iso8601(Rep#rep.start_time)},
+{proxy, job_proxy_url(Rep#rep.source)}
+];
+{error, not_found} ->
+nil % Job might have just completed
+end.
+
+
[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator
iilyak commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112780051
##
File path: src/couch_replicator/src/couch_replicator.erl
##
@@ -191,847 +120,244 @@ wait_for_result(RepId) ->
end.
-cancel_replication({BaseId, Extension}) ->
-FullRepId = BaseId ++ Extension,
-couch_log:notice("Canceling replication `~s`...", [FullRepId]),
-case supervisor:terminate_child(couch_replicator_job_sup, FullRepId) of
-ok ->
-couch_log:notice("Replication `~s` canceled.", [FullRepId]),
-case supervisor:delete_child(couch_replicator_job_sup, FullRepId) of
-ok ->
-{ok, {cancelled, ?l2b(FullRepId)}};
-{error, not_found} ->
-{ok, {cancelled, ?l2b(FullRepId)}};
-Error ->
-Error
-end;
-Error ->
-couch_log:error("Error canceling replication `~s`: ~p", [FullRepId,
Error]),
-Error
-end.
-
-cancel_replication(RepId, #user_ctx{name = Name, roles = Roles}) ->
-case lists:member(<<"_admin">>, Roles) of
-true ->
-cancel_replication(RepId);
-false ->
-case find_replicator(RepId) of
-{ok, Pid} ->
-case details(Pid) of
-{ok, #rep{user_ctx = #user_ctx{name = Name}}} ->
-cancel_replication(RepId);
-{ok, _} ->
-throw({unauthorized,
-<<"Can't cancel a replication triggered by another
user">>});
-Error ->
-Error
-end;
-Error ->
-Error
-end
-end.
-
-find_replicator({BaseId, Ext} = _RepId) ->
-case lists:keysearch(
-BaseId ++ Ext, 1, supervisor:which_children(couch_replicator_job_sup))
of
-{value, {_, Pid, _, _}} when is_pid(Pid) ->
-{ok, Pid};
-_ ->
-{error, not_found}
-end.
-
-details(Pid) ->
-case (catch gen_server:call(Pid, get_details)) of
-{ok, Rep} ->
-{ok, Rep};
-{'EXIT', {noproc, {gen_server, call, _}}} ->
-{error, not_found};
-Error ->
-throw(Error)
+-spec cancel_replication(rep_id()) ->
+{ok, {cancelled, binary()}} | {error, not_found}.
+cancel_replication({BasedId, Extension} = RepId) ->
+FullRepId = BasedId ++ Extension,
+couch_log:notice("Canceling replication '~s' ...", [FullRepId]),
+case couch_replicator_scheduler:rep_state(RepId) of
+#rep{} ->
+ok = couch_replicator_scheduler:remove_job(RepId),
+couch_log:notice("Replication '~s' cancelled", [FullRepId]),
+{ok, {cancelled, ?l2b(FullRepId)}};
+nil ->
+couch_log:notice("Replication '~s' not found", [FullRepId]),
+{error, not_found}
end.
-init(InitArgs) ->
-{ok, InitArgs, 0}.
-do_init(#rep{options = Options, id = {BaseId, Ext}, user_ctx=UserCtx} = Rep) ->
-process_flag(trap_exit, true),
-
-random:seed(os:timestamp()),
-
-#rep_state{
-source = Source,
-target = Target,
-source_name = SourceName,
-target_name = TargetName,
-start_seq = {_Ts, StartSeq},
-committed_seq = {_, CommittedSeq},
-highest_seq_done = {_, HighestSeq},
-checkpoint_interval = CheckpointInterval
-} = State = init_state(Rep),
-
-NumWorkers = get_value(worker_processes, Options),
-BatchSize = get_value(worker_batch_size, Options),
-{ok, ChangesQueue} = couch_work_queue:new([
-{max_items, BatchSize * NumWorkers * 2},
-{max_size, 100 * 1024 * NumWorkers}
-]),
-% This starts the _changes reader process. It adds the changes from
-% the source db to the ChangesQueue.
-{ok, ChangesReader} = couch_replicator_changes_reader:start_link(
-StartSeq, Source, ChangesQueue, Options
-),
-% Changes manager - responsible for dequeing batches from the changes queue
-% and deliver them to the worker processes.
-ChangesManager = spawn_changes_manager(self(), ChangesQueue, BatchSize),
-% This starts the worker processes. They ask the changes queue manager for
a
-% a batch of _changes rows to process -> check which revs are missing in
the
-% target, and for the missing ones, it copies them from the source to the
target.
-MaxConns = get_value(http_connections, Options),
-Workers = lists:map(
-fun(_) ->
-couch_stats:increment_counter([couch_replicator, workers_started]),
-{ok, Pid} = couch_replicator_worker:start_link(
-self(), Source, Target, ChangesManager, MaxConns),
-Pid
-end,
-lists:seq(1, NumWorkers)),
+-spec replication_states() -> [atom()].
+replication_states() ->
+?REPLICATION_STATES.
-couch_task_status:add_task([
-{type, replication},
-{user, UserCtx#user_ctx.name},
-{replication_id, ?l2b(BaseId ++ Ext)},
-{database, Rep#rep.db_name},
-
[jira] [Commented] (COUCHDB-3341) EUnit: config listener unknown failure
[
https://issues.apache.org/jira/browse/COUCHDB-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979335#comment-15979335
]
Joan Touzet commented on COUCHDB-3341:
--
Fourth occurrence, Ubuntu 16.04, default Erlang.
> EUnit: config listener unknown failure
> --
>
> Key: COUCHDB-3341
> URL: https://issues.apache.org/jira/browse/COUCHDB-3341
> Project: CouchDB
> Issue Type: Test
> Components: Test Suite
>Reporter: Joan Touzet
>
> One occurrence in our Jenkins builds so far.
> {noformat}
> module 'couch_log_config_listener_test'
> couch_log_config_listener_test: couch_log_config_test_...*failed*
> in function
> couch_log_config_listener_test:'-check_restart_listener/0-fun-2-'/1
> (test/couch_log_config_listener_test.erl, line 38)
> in call from couch_log_config_listener_test:check_restart_listener/0
> (test/couch_log_config_listener_test.erl, line 38)
> **error:{assertEqual,[{module,couch_log_config_listener_test},
> {line,38},
> {expression,"get_handler ( )"},
> {expected,not_found},
> {value,{config_listener,{couch_log_sup,<0.3192.0>}}}]}
> output:<<"">>
> {noformat}
> No clue what's going on here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (COUCHDB-3390) clustered _users/_changes returns error when include_docs=true
[
https://issues.apache.org/jira/browse/COUCHDB-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joan Touzet resolved COUCHDB-3390.
--
Resolution: Fixed
Fix Version/s: 2.1.0
Hi there, thanks for the report. This has already been fixed on master.
> clustered _users/_changes returns error when include_docs=true
> --
>
> Key: COUCHDB-3390
> URL: https://issues.apache.org/jira/browse/COUCHDB-3390
> Project: CouchDB
> Issue Type: Bug
> Components: HTTP Interface
>Reporter: CJ Herman
> Fix For: 2.1.0
>
>
> When requesting changes feed for clustered _users DB with include_docs=true,
> an error is returned:
> curl -X GET http://localhost:5984/_users/_changes?include_docs=true
> {"error":"error","reason":"{not_found,nil,\n
> [{couch_users_db,after_doc_read,2,\n
> [{file,\"src/couch_users_db.erl\"},{line,123}]},\n
> {couch_db,open_doc_int,3,[{file,\"src/couch_db.erl\"},{line,1345}]},\n
> {couch_db,open_doc,3,[{file,\"src/couch_db.erl\"},{line,189}]},\n
>{fabric_rpc,doc_member,3,[{file,\"src/fabric_rpc.erl\"},{line,354}]},\n
> {fabric_rpc,changes_enumerator,2,\n
> [{file,\"src/fabric_rpc.erl\"},{line,347}]},\n
> {couch_btree,stream_kv_node2,8,\n
> [{file,\"src/couch_btree.erl\"},{line,783}]},\n
> {couch_btree,fold,4,[{file,\"src/couch_btree.erl\"},{line,220}]},\n
> {couch_db,changes_since,5,\n
> [{file,\"src/couch_db.erl\"},{line,1244}]}]}"}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] nickva opened a new pull request #484: Apply random jitter during initial _replicator shard discovery
nickva opened a new pull request #484: Apply random jitter during initial _replicator shard discovery URL: https://github.com/apache/couchdb/pull/484 This is bringing back previous code: https://github.com/apache/couchdb/blob/884cf3e55f77ab1a5f26dc7202ce21771062eae6/src/couch_replicator_manager.erl#L940-L946 The rationale is the following: during shard scanning a lot of `resume_scan` messages are sent back to back. This causes the replicator manager to open change feeds for all of those shards. By delaying `resume_scan` message by a jitter proportional to the number messages sent to far, it gives replicator manager a chance to open some change feeds, finish processing them and close them before newer resume_scan messages arrive. The random delay average starts 10 msec for first message, up to 1 min for 6000th and higher. Some sample values: * For 100 messages, average wait will be 1 second * For 1000 - 10 seconds * For 6000 and higher - 1 minute Jira: COUCHDB-3389 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator
nickva commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112765656
##
File path: src/couch_replicator/src/couch_replicator_scheduler_job.erl
##
@@ -0,0 +1,969 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_replicator_scheduler_job).
+
+-behaviour(gen_server).
+
+-export([
+ start_link/1
+]).
+
+-export([
+ init/1,
+ terminate/2,
+ handle_call/3,
+ handle_info/2,
+ handle_cast/2,
+ code_change/3,
+ format_status/2
+]).
+
+-include_lib("couch/include/couch_db.hrl").
+-include("couch_replicator_api_wrap.hrl").
+-include("couch_replicator_scheduler.hrl").
+-include("couch_replicator.hrl").
+
+-import(couch_util, [
+get_value/2,
+get_value/3,
+to_binary/1
+]).
+
+-import(couch_replicator_utils, [
+start_db_compaction_notifier/2,
+stop_db_compaction_notifier/1,
+pp_rep_id/1
+]).
+
+
+-define(LOWEST_SEQ, 0).
+-define(DEFAULT_CHECKPOINT_INTERVAL, 3).
+-define(STARTUP_JITTER_DEFAULT, 5000).
+
+-record(rep_state, {
+rep_details,
+source_name,
+target_name,
+source,
+target,
+history,
+checkpoint_history,
+start_seq,
+committed_seq,
+current_through_seq,
+seqs_in_progress = [],
+highest_seq_done = {0, ?LOWEST_SEQ},
+source_log,
+target_log,
+rep_starttime,
+src_starttime,
+tgt_starttime,
+timer, % checkpoint timer
+changes_queue,
+changes_manager,
+changes_reader,
+workers,
+stats = couch_replicator_stats:new(),
+session_id,
+source_db_compaction_notifier = nil,
+target_db_compaction_notifier = nil,
+source_monitor = nil,
+target_monitor = nil,
+source_seq = nil,
+use_checkpoints = true,
+checkpoint_interval = ?DEFAULT_CHECKPOINT_INTERVAL,
+type = db,
+view = nil
+}).
+
+
+start_link(#rep{id = {BaseId, Ext}, source = Src, target = Tgt} = Rep) ->
+RepChildId = BaseId ++ Ext,
+Source = couch_replicator_api_wrap:db_uri(Src),
+Target = couch_replicator_api_wrap:db_uri(Tgt),
+ServerName = {global, {?MODULE, Rep#rep.id}},
+
+case gen_server:start_link(ServerName, ?MODULE, Rep, []) of
+{ok, Pid} ->
+couch_log:notice("starting new replication `~s` at ~p (`~s` ->
`~s`)",
+ [RepChildId, Pid, Source, Target]),
+{ok, Pid};
+{error, Reason} ->
+couch_log:warning("failed to start replication `~s` (`~s` ->
`~s`)",
+ [RepChildId, Source, Target]),
+{error, Reason}
+end.
+
+
+init(InitArgs) ->
+{ok, InitArgs, 0}.
Review comment:
This was another direct copy from previous code:
https://github.com/apache/couchdb/blob/master/src/couch_replicator/src/couch_replicator.erl#L253-L256
The code for replication task used to live in couch_replicator.erl then just
moved largely unmodified `couch_replicator_scheduler_jobs.erl` so it's been
like this for a long time.
50/50 on this as well, as modifying this might cause some unintended
consequences. But I'll give it a try. Seems easy enough
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (COUCHDB-3390) clustered _users/_changes returns error when include_docs=true
CJ Herman created COUCHDB-3390:
--
Summary: clustered _users/_changes returns error when
include_docs=true
Key: COUCHDB-3390
URL: https://issues.apache.org/jira/browse/COUCHDB-3390
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Reporter: CJ Herman
When requesting changes feed for clustered _users DB with include_docs=true, an
error is returned:
curl -X GET http://localhost:5984/_users/_changes?include_docs=true
{"error":"error","reason":"{not_found,nil,\n
[{couch_users_db,after_doc_read,2,\n
[{file,\"src/couch_users_db.erl\"},{line,123}]},\n
{couch_db,open_doc_int,3,[{file,\"src/couch_db.erl\"},{line,1345}]},\n
{couch_db,open_doc,3,[{file,\"src/couch_db.erl\"},{line,189}]},\n
{fabric_rpc,doc_member,3,[{file,\"src/fabric_rpc.erl\"},{line,354}]},\n
{fabric_rpc,changes_enumerator,2,\n
[{file,\"src/fabric_rpc.erl\"},{line,347}]},\n
{couch_btree,stream_kv_node2,8,\n
[{file,\"src/couch_btree.erl\"},{line,783}]},\n
{couch_btree,fold,4,[{file,\"src/couch_btree.erl\"},{line,220}]},\n
{couch_db,changes_since,5,\n
[{file,\"src/couch_db.erl\"},{line,1244}]}]}"}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator
iilyak commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112761438
##
File path: src/couch_replicator/src/couch_replicator_scheduler_job.erl
##
@@ -0,0 +1,969 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_replicator_scheduler_job).
+
+-behaviour(gen_server).
+
+-export([
+ start_link/1
+]).
+
+-export([
+ init/1,
+ terminate/2,
+ handle_call/3,
+ handle_info/2,
+ handle_cast/2,
+ code_change/3,
+ format_status/2
+]).
+
+-include_lib("couch/include/couch_db.hrl").
+-include("couch_replicator_api_wrap.hrl").
+-include("couch_replicator_scheduler.hrl").
+-include("couch_replicator.hrl").
+
+-import(couch_util, [
+get_value/2,
+get_value/3,
+to_binary/1
+]).
+
+-import(couch_replicator_utils, [
+start_db_compaction_notifier/2,
+stop_db_compaction_notifier/1,
+pp_rep_id/1
+]).
+
+
+-define(LOWEST_SEQ, 0).
+-define(DEFAULT_CHECKPOINT_INTERVAL, 3).
+-define(STARTUP_JITTER_DEFAULT, 5000).
+
+-record(rep_state, {
+rep_details,
+source_name,
+target_name,
+source,
+target,
+history,
+checkpoint_history,
+start_seq,
+committed_seq,
+current_through_seq,
+seqs_in_progress = [],
+highest_seq_done = {0, ?LOWEST_SEQ},
+source_log,
+target_log,
+rep_starttime,
+src_starttime,
+tgt_starttime,
+timer, % checkpoint timer
+changes_queue,
+changes_manager,
+changes_reader,
+workers,
+stats = couch_replicator_stats:new(),
+session_id,
+source_db_compaction_notifier = nil,
+target_db_compaction_notifier = nil,
+source_monitor = nil,
+target_monitor = nil,
+source_seq = nil,
+use_checkpoints = true,
+checkpoint_interval = ?DEFAULT_CHECKPOINT_INTERVAL,
+type = db,
+view = nil
+}).
+
+
+start_link(#rep{id = {BaseId, Ext}, source = Src, target = Tgt} = Rep) ->
+RepChildId = BaseId ++ Ext,
+Source = couch_replicator_api_wrap:db_uri(Src),
+Target = couch_replicator_api_wrap:db_uri(Tgt),
+ServerName = {global, {?MODULE, Rep#rep.id}},
+
+case gen_server:start_link(ServerName, ?MODULE, Rep, []) of
+{ok, Pid} ->
+couch_log:notice("starting new replication `~s` at ~p (`~s` ->
`~s`)",
+ [RepChildId, Pid, Source, Target]),
+{ok, Pid};
+{error, Reason} ->
+couch_log:warning("failed to start replication `~s` (`~s` ->
`~s`)",
+ [RepChildId, Source, Target]),
+{error, Reason}
+end.
+
+
+init(InitArgs) ->
+{ok, InitArgs, 0}.
Review comment:
I noticed that it is lifted from couch_replicator.erl. Therefore you can
ignore it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator
iilyak commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112760882
##
File path: src/couch_replicator/src/couch_replicator_scheduler_job.erl
##
@@ -0,0 +1,969 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_replicator_scheduler_job).
+
+-behaviour(gen_server).
+
+-export([
+ start_link/1
+]).
+
+-export([
+ init/1,
+ terminate/2,
+ handle_call/3,
+ handle_info/2,
+ handle_cast/2,
+ code_change/3,
+ format_status/2
+]).
+
+-include_lib("couch/include/couch_db.hrl").
+-include("couch_replicator_api_wrap.hrl").
+-include("couch_replicator_scheduler.hrl").
+-include("couch_replicator.hrl").
+
+-import(couch_util, [
+get_value/2,
+get_value/3,
+to_binary/1
+]).
+
+-import(couch_replicator_utils, [
+start_db_compaction_notifier/2,
+stop_db_compaction_notifier/1,
+pp_rep_id/1
+]).
+
+
+-define(LOWEST_SEQ, 0).
+-define(DEFAULT_CHECKPOINT_INTERVAL, 3).
+-define(STARTUP_JITTER_DEFAULT, 5000).
+
+-record(rep_state, {
+rep_details,
+source_name,
+target_name,
+source,
+target,
+history,
+checkpoint_history,
+start_seq,
+committed_seq,
+current_through_seq,
+seqs_in_progress = [],
+highest_seq_done = {0, ?LOWEST_SEQ},
+source_log,
+target_log,
+rep_starttime,
+src_starttime,
+tgt_starttime,
+timer, % checkpoint timer
+changes_queue,
+changes_manager,
+changes_reader,
+workers,
+stats = couch_replicator_stats:new(),
+session_id,
+source_db_compaction_notifier = nil,
+target_db_compaction_notifier = nil,
+source_monitor = nil,
+target_monitor = nil,
+source_seq = nil,
+use_checkpoints = true,
+checkpoint_interval = ?DEFAULT_CHECKPOINT_INTERVAL,
+type = db,
+view = nil
+}).
+
+
+start_link(#rep{id = {BaseId, Ext}, source = Src, target = Tgt} = Rep) ->
+RepChildId = BaseId ++ Ext,
+Source = couch_replicator_api_wrap:db_uri(Src),
+Target = couch_replicator_api_wrap:db_uri(Tgt),
+ServerName = {global, {?MODULE, Rep#rep.id}},
+
+case gen_server:start_link(ServerName, ?MODULE, Rep, []) of
+{ok, Pid} ->
+couch_log:notice("starting new replication `~s` at ~p (`~s` ->
`~s`)",
+ [RepChildId, Pid, Source, Target]),
+{ok, Pid};
+{error, Reason} ->
+couch_log:warning("failed to start replication `~s` (`~s` ->
`~s`)",
+ [RepChildId, Source, Target]),
+{error, Reason}
+end.
+
+
+init(InitArgs) ->
+{ok, InitArgs, 0}.
Review comment:
This is not the safest way of doing delayed initialization. Here are two
problems with this method:
- You have to implement `handle_info(timeout, ...)` which is not very
descriptive since you don't have control over message construction
- There is a race condition since you return Pid to a caller prior to a call
to do_init
see
[gen_server:init_it](https://github.com/erlang/otp/blob/maint/lib/stdlib/src/gen_server.erl#L328)
```
case Mod:init(Args) of
{ok, State, Timeout} ->
proc_lib:init_ack(Starter, {ok, self()}), %%% Pid is
returned to caller
loop(Parent, Name, State, Mod, Timeout, Debug);
```
There is a safer way of doing it
```
init(InitArgs) ->
gen_server:cast(self(), {do_init, InitArgs}),
{ok, nil}.
```
This ensures that do_init message gets into mailbox of a process prior to
return of the pid to the caller. Since caller will not get Pid there is no one
who would be able to send a message before do_init (we don't consider malicious
code here).
This way also make it possible to use something more meaningful than atom
`timeout`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (COUCHDB-3389) Bring back jittered delay during replication shard scan
Nick Vatamaniuc created COUCHDB-3389: Summary: Bring back jittered delay during replication shard scan Key: COUCHDB-3389 URL: https://issues.apache.org/jira/browse/COUCHDB-3389 Project: CouchDB Issue Type: Bug Reporter: Nick Vatamaniuc When we switched to using mem3 db for shard discovery we dropped jittered delay during shard scan. On a large production system with thousands of replicator dbs, back to back shard notification, which spawn change feeds can cause performance issues. https://github.com/apache/couchdb/blob/884cf3e55f77ab1a5f26dc7202ce21771062eae6/src/couch_replicator_manager.erl#L940-L946 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator
nickva commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112758805
##
File path: src/couch_replicator/src/couch_replicator_scheduler.erl
##
@@ -0,0 +1,1430 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_replicator_scheduler).
+
+-behaviour(gen_server).
+-behaviour(config_listener).
+
+-export([
+start_link/0
+]).
+
+-export([
+ init/1,
+ terminate/2,
+ handle_call/3,
+ handle_info/2,
+ handle_cast/2,
+ code_change/3,
+ format_status/2
+]).
+
+-export([
+ add_job/1,
+ remove_job/1,
+ reschedule/0,
+ rep_state/1,
+ find_jobs_by_dbname/1,
+ find_jobs_by_doc/2,
+ job_summary/2,
+ health_threshold/0,
+ jobs/0,
+ job/1
+]).
+
+%% config_listener callbacks
+-export([
+handle_config_change/5,
+handle_config_terminate/3
+]).
+
+%% for status updater process to allow hot code loading
+-export([
+stats_updater_loop/1
+]).
+
+-include("couch_replicator_scheduler.hrl").
+-include("couch_replicator.hrl").
+-include("couch_replicator_api_wrap.hrl").
+-include_lib("couch/include/couch_db.hrl").
+
+%% types
+-type event_type() :: added | started | stopped | {crashed, any()}.
+-type event() :: {Type:: event_type(), When :: erlang:timestamp()}.
+-type history() :: nonempty_list(event()).
+
+%% definitions
+-define(MAX_BACKOFF_EXPONENT, 10).
+-define(BACKOFF_INTERVAL_MICROS, 30 * 1000 * 1000).
+-define(DEFAULT_HEALTH_THRESHOLD_SEC, 2 * 60).
+-define(RELISTEN_DELAY, 5000).
+-define(STATS_UPDATE_WAIT, 5000).
+
+-define(DEFAULT_MAX_JOBS, 500).
+-define(DEFAULT_MAX_CHURN, 20).
+-define(DEFAULT_MAX_HISTORY, 20).
+-define(DEFAULT_SCHEDULER_INTERVAL, 6).
+
+
+-record(state, {interval, timer, max_jobs, max_churn, max_history, stats_pid}).
+-record(job, {
+id :: job_id() | '$1' | '_',
+rep :: #rep{} | '_',
+pid :: undefined | pid() | '$1' | '_',
+monitor :: undefined | reference() | '_',
+history :: history() | '_'
+}).
+
+-record(stats_acc, {
+pending_n = 0 :: non_neg_integer(),
+running_n = 0 :: non_neg_integer(),
+crashed_n = 0 :: non_neg_integer()
+}).
+
+
+%% public functions
+
+-spec start_link() -> {ok, pid()} | ignore | {error, term()}.
+start_link() ->
+gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+
+-spec add_job(#rep{}) -> ok.
+add_job(#rep{} = Rep) when Rep#rep.id /= undefined ->
+Job = #job{
+id = Rep#rep.id,
+rep = Rep,
+history = [{added, os:timestamp()}]
+},
+gen_server:call(?MODULE, {add_job, Job}, infinity).
+
+
+-spec remove_job(job_id()) -> ok.
+remove_job(Id) ->
+gen_server:call(?MODULE, {remove_job, Id}, infinity).
+
+
+-spec reschedule() -> ok.
+% Trigger a manual reschedule. Used for testing and/or ops.
+reschedule() ->
+gen_server:call(?MODULE, reschedule, infinity).
+
+
+-spec rep_state(rep_id()) -> #rep{} | nil.
+rep_state(RepId) ->
+case (catch ets:lookup_element(?MODULE, RepId, #job.rep)) of
+{'EXIT',{badarg, _}} ->
+nil;
+Rep ->
+Rep
+end.
+
+
+-spec job_summary(job_id(), non_neg_integer()) -> [_] | nil.
+job_summary(JobId, HealthThreshold) ->
+case job_by_id(JobId) of
+{ok, #job{pid = Pid, history = History, rep = Rep}} ->
+ErrorCount = consecutive_crashes(History, HealthThreshold),
+{State, Info} = case {Pid, ErrorCount} of
+{undefined, 0} ->
+{pending, null};
+{undefined, ErrorCount} when ErrorCount > 0 ->
+ [{{crashed, Error}, _When} | _] = History,
+ ErrMsg =
couch_replicator_utils:rep_error_to_binary(Error),
+ {crashing, ErrMsg};
+{Pid, ErrorCount} when is_pid(Pid) ->
+ {running, null}
+end,
+[
+{source, iolist_to_binary(ejson_url(Rep#rep.source))},
+{target, iolist_to_binary(ejson_url(Rep#rep.target))},
+{state, State},
+{info, Info},
+{error_count, ErrorCount},
+{last_updated, last_updated(History)},
+{start_time,
+couch_replicator_utils:iso8601(Rep#rep.start_time)},
+{proxy, job_proxy_url(Rep#rep.source)}
+];
+{error, not_found} ->
+nil % Job might have just completed
+end.
+
+
[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator
nickva commented on a change in pull request #470: Scheduling Replicator URL: https://github.com/apache/couchdb/pull/470#discussion_r112758360 ## File path: src/couch_replicator/src/couch_replicator_scheduler.erl ## @@ -0,0 +1,1430 @@ +% Licensed under the Apache License, Version 2.0 (the "License"); you may not +% use this file except in compliance with the License. You may obtain a copy of +% the License at +% +% http://www.apache.org/licenses/LICENSE-2.0 +% +% Unless required by applicable law or agreed to in writing, software +% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +% License for the specific language governing permissions and limitations under +% the License. + +-module(couch_replicator_scheduler). + +-behaviour(gen_server). +-behaviour(config_listener). + +-export([ +start_link/0 +]). + +-export([ + init/1, + terminate/2, + handle_call/3, + handle_info/2, + handle_cast/2, + code_change/3, + format_status/2 +]). + +-export([ Review comment: Good point. That might be useful for operators. `replicator:restart_replication` would be cluster aware and `replicator_scheduler:restart_job` might be per-node (local only). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator
nickva commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112756667
##
File path: src/couch_replicator/src/couch_replicator_httpd.erl
##
@@ -24,13 +30,68 @@
to_binary/1
]).
--export([handle_req/1]).
+
+-define(DEFAULT_TASK_LIMIT, 100).
+-define(REPDB, <<"_replicator">>).
+
+
+handle_scheduler_req(#httpd{method='GET', path_parts=[_,<<"jobs">>]}=Req) ->
+Limit = couch_replicator_httpd_util:parse_int_param(Req, "limit",
+?DEFAULT_TASK_LIMIT, 0, infinity),
+Skip = couch_replicator_httpd_util:parse_int_param(Req, "skip", 0, 0,
+infinity),
+{Replies, _BadNodes} = rpc:multicall(couch_replicator_scheduler, jobs, []),
+Flatlist = lists:concat(Replies),
+% couch_replicator_scheduler:job_ejson/1 guarantees {id, Id} to be the
+% the first item in the list
+Sorted = lists:sort(fun({[{id,A}|_]},{[{id,B}|_]}) -> A =< B end,
Flatlist),
+Total = length(Sorted),
+Offset = min(Skip, Total),
+Sublist = lists:sublist(Sorted, Offset+1, Limit),
+Sublist1 = [couch_replicator_httpd_util:update_db_name(Task)
+|| Task <- Sublist],
+send_json(Req, {[{total_rows, Total}, {offset, Offset}, {jobs,
Sublist1}]});
+handle_scheduler_req(#httpd{method='GET',
path_parts=[_,<<"jobs">>,JobId]}=Req) ->
+case couch_replicator:job(JobId) of
+{ok, JobInfo} ->
+send_json(Req,
couch_replicator_httpd_util:update_db_name(JobInfo));
+{error, not_found} ->
+throw(not_found)
Review comment:
Ah I see. Agree in general it might be nice to have. Afraid in the context
of this PR there is a good chance it might destabilize the code at the last
minute. We could probably add this feature in general to other paths like doc
lookups/updates and such.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator
iilyak commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112756535
##
File path: src/couch_replicator/src/couch_replicator_scheduler.erl
##
@@ -0,0 +1,1430 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_replicator_scheduler).
+
+-behaviour(gen_server).
+-behaviour(config_listener).
+
+-export([
+start_link/0
+]).
+
+-export([
+ init/1,
+ terminate/2,
+ handle_call/3,
+ handle_info/2,
+ handle_cast/2,
+ code_change/3,
+ format_status/2
+]).
+
+-export([
+ add_job/1,
+ remove_job/1,
+ reschedule/0,
+ rep_state/1,
+ find_jobs_by_dbname/1,
+ find_jobs_by_doc/2,
+ job_summary/2,
+ health_threshold/0,
+ jobs/0,
+ job/1
+]).
+
+%% config_listener callbacks
+-export([
+handle_config_change/5,
+handle_config_terminate/3
+]).
+
+%% for status updater process to allow hot code loading
+-export([
+stats_updater_loop/1
+]).
+
+-include("couch_replicator_scheduler.hrl").
+-include("couch_replicator.hrl").
+-include("couch_replicator_api_wrap.hrl").
+-include_lib("couch/include/couch_db.hrl").
+
+%% types
+-type event_type() :: added | started | stopped | {crashed, any()}.
+-type event() :: {Type:: event_type(), When :: erlang:timestamp()}.
+-type history() :: nonempty_list(event()).
+
+%% definitions
+-define(MAX_BACKOFF_EXPONENT, 10).
+-define(BACKOFF_INTERVAL_MICROS, 30 * 1000 * 1000).
+-define(DEFAULT_HEALTH_THRESHOLD_SEC, 2 * 60).
+-define(RELISTEN_DELAY, 5000).
+-define(STATS_UPDATE_WAIT, 5000).
+
+-define(DEFAULT_MAX_JOBS, 500).
+-define(DEFAULT_MAX_CHURN, 20).
+-define(DEFAULT_MAX_HISTORY, 20).
+-define(DEFAULT_SCHEDULER_INTERVAL, 6).
+
+
+-record(state, {interval, timer, max_jobs, max_churn, max_history, stats_pid}).
+-record(job, {
+id :: job_id() | '$1' | '_',
+rep :: #rep{} | '_',
+pid :: undefined | pid() | '$1' | '_',
+monitor :: undefined | reference() | '_',
+history :: history() | '_'
+}).
+
+-record(stats_acc, {
+pending_n = 0 :: non_neg_integer(),
+running_n = 0 :: non_neg_integer(),
+crashed_n = 0 :: non_neg_integer()
+}).
+
+
+%% public functions
+
+-spec start_link() -> {ok, pid()} | ignore | {error, term()}.
+start_link() ->
+gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+
+-spec add_job(#rep{}) -> ok.
+add_job(#rep{} = Rep) when Rep#rep.id /= undefined ->
+Job = #job{
+id = Rep#rep.id,
+rep = Rep,
+history = [{added, os:timestamp()}]
+},
+gen_server:call(?MODULE, {add_job, Job}, infinity).
+
+
+-spec remove_job(job_id()) -> ok.
+remove_job(Id) ->
+gen_server:call(?MODULE, {remove_job, Id}, infinity).
+
+
+-spec reschedule() -> ok.
+% Trigger a manual reschedule. Used for testing and/or ops.
+reschedule() ->
+gen_server:call(?MODULE, reschedule, infinity).
+
+
+-spec rep_state(rep_id()) -> #rep{} | nil.
+rep_state(RepId) ->
+case (catch ets:lookup_element(?MODULE, RepId, #job.rep)) of
+{'EXIT',{badarg, _}} ->
+nil;
+Rep ->
+Rep
+end.
+
+
+-spec job_summary(job_id(), non_neg_integer()) -> [_] | nil.
+job_summary(JobId, HealthThreshold) ->
+case job_by_id(JobId) of
+{ok, #job{pid = Pid, history = History, rep = Rep}} ->
+ErrorCount = consecutive_crashes(History, HealthThreshold),
+{State, Info} = case {Pid, ErrorCount} of
+{undefined, 0} ->
+{pending, null};
+{undefined, ErrorCount} when ErrorCount > 0 ->
+ [{{crashed, Error}, _When} | _] = History,
+ ErrMsg =
couch_replicator_utils:rep_error_to_binary(Error),
+ {crashing, ErrMsg};
+{Pid, ErrorCount} when is_pid(Pid) ->
+ {running, null}
+end,
+[
+{source, iolist_to_binary(ejson_url(Rep#rep.source))},
+{target, iolist_to_binary(ejson_url(Rep#rep.target))},
+{state, State},
+{info, Info},
+{error_count, ErrorCount},
+{last_updated, last_updated(History)},
+{start_time,
+couch_replicator_utils:iso8601(Rep#rep.start_time)},
+{proxy, job_proxy_url(Rep#rep.source)}
+];
+{error, not_found} ->
+nil % Job might have just completed
+end.
+
+
[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator
iilyak commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112756535
##
File path: src/couch_replicator/src/couch_replicator_scheduler.erl
##
@@ -0,0 +1,1430 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_replicator_scheduler).
+
+-behaviour(gen_server).
+-behaviour(config_listener).
+
+-export([
+start_link/0
+]).
+
+-export([
+ init/1,
+ terminate/2,
+ handle_call/3,
+ handle_info/2,
+ handle_cast/2,
+ code_change/3,
+ format_status/2
+]).
+
+-export([
+ add_job/1,
+ remove_job/1,
+ reschedule/0,
+ rep_state/1,
+ find_jobs_by_dbname/1,
+ find_jobs_by_doc/2,
+ job_summary/2,
+ health_threshold/0,
+ jobs/0,
+ job/1
+]).
+
+%% config_listener callbacks
+-export([
+handle_config_change/5,
+handle_config_terminate/3
+]).
+
+%% for status updater process to allow hot code loading
+-export([
+stats_updater_loop/1
+]).
+
+-include("couch_replicator_scheduler.hrl").
+-include("couch_replicator.hrl").
+-include("couch_replicator_api_wrap.hrl").
+-include_lib("couch/include/couch_db.hrl").
+
+%% types
+-type event_type() :: added | started | stopped | {crashed, any()}.
+-type event() :: {Type:: event_type(), When :: erlang:timestamp()}.
+-type history() :: nonempty_list(event()).
+
+%% definitions
+-define(MAX_BACKOFF_EXPONENT, 10).
+-define(BACKOFF_INTERVAL_MICROS, 30 * 1000 * 1000).
+-define(DEFAULT_HEALTH_THRESHOLD_SEC, 2 * 60).
+-define(RELISTEN_DELAY, 5000).
+-define(STATS_UPDATE_WAIT, 5000).
+
+-define(DEFAULT_MAX_JOBS, 500).
+-define(DEFAULT_MAX_CHURN, 20).
+-define(DEFAULT_MAX_HISTORY, 20).
+-define(DEFAULT_SCHEDULER_INTERVAL, 6).
+
+
+-record(state, {interval, timer, max_jobs, max_churn, max_history, stats_pid}).
+-record(job, {
+id :: job_id() | '$1' | '_',
+rep :: #rep{} | '_',
+pid :: undefined | pid() | '$1' | '_',
+monitor :: undefined | reference() | '_',
+history :: history() | '_'
+}).
+
+-record(stats_acc, {
+pending_n = 0 :: non_neg_integer(),
+running_n = 0 :: non_neg_integer(),
+crashed_n = 0 :: non_neg_integer()
+}).
+
+
+%% public functions
+
+-spec start_link() -> {ok, pid()} | ignore | {error, term()}.
+start_link() ->
+gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+
+-spec add_job(#rep{}) -> ok.
+add_job(#rep{} = Rep) when Rep#rep.id /= undefined ->
+Job = #job{
+id = Rep#rep.id,
+rep = Rep,
+history = [{added, os:timestamp()}]
+},
+gen_server:call(?MODULE, {add_job, Job}, infinity).
+
+
+-spec remove_job(job_id()) -> ok.
+remove_job(Id) ->
+gen_server:call(?MODULE, {remove_job, Id}, infinity).
+
+
+-spec reschedule() -> ok.
+% Trigger a manual reschedule. Used for testing and/or ops.
+reschedule() ->
+gen_server:call(?MODULE, reschedule, infinity).
+
+
+-spec rep_state(rep_id()) -> #rep{} | nil.
+rep_state(RepId) ->
+case (catch ets:lookup_element(?MODULE, RepId, #job.rep)) of
+{'EXIT',{badarg, _}} ->
+nil;
+Rep ->
+Rep
+end.
+
+
+-spec job_summary(job_id(), non_neg_integer()) -> [_] | nil.
+job_summary(JobId, HealthThreshold) ->
+case job_by_id(JobId) of
+{ok, #job{pid = Pid, history = History, rep = Rep}} ->
+ErrorCount = consecutive_crashes(History, HealthThreshold),
+{State, Info} = case {Pid, ErrorCount} of
+{undefined, 0} ->
+{pending, null};
+{undefined, ErrorCount} when ErrorCount > 0 ->
+ [{{crashed, Error}, _When} | _] = History,
+ ErrMsg =
couch_replicator_utils:rep_error_to_binary(Error),
+ {crashing, ErrMsg};
+{Pid, ErrorCount} when is_pid(Pid) ->
+ {running, null}
+end,
+[
+{source, iolist_to_binary(ejson_url(Rep#rep.source))},
+{target, iolist_to_binary(ejson_url(Rep#rep.target))},
+{state, State},
+{info, Info},
+{error_count, ErrorCount},
+{last_updated, last_updated(History)},
+{start_time,
+couch_replicator_utils:iso8601(Rep#rep.start_time)},
+{proxy, job_proxy_url(Rep#rep.source)}
+];
+{error, not_found} ->
+nil % Job might have just completed
+end.
+
+
[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator
nickva commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112755764
##
File path: src/couch_replicator/src/couch_replicator_filters.erl
##
@@ -0,0 +1,214 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_replicator_filters).
+
+-export([
+parse/1,
+fetch/4,
+view_type/2,
+ejsort/1
+]).
+
+-include_lib("couch/include/couch_db.hrl").
+
+
+% Parse the filter from replication options proplist.
+% Return {ok, {FilterType,...}} | {error, ParseError}.
+% For `user` filter, i.e. filters specified as user code
+% in source database, this code doesn't fetch the filter
+% code, but only returns the name of the filter.
+-spec parse([_]) ->
+{ok, nil} |
+{ok, {view, binary(), {[_]}}} |
+{ok, {user, {binary(), binary()}, {[_]}}} |
+{ok, {docids, [_]}} |
+{ok, {mango, {[_]}}} |
+{error, binary()}.
+parse(Options) ->
+Filter = couch_util:get_value(filter, Options),
+DocIds = couch_util:get_value(doc_ids, Options),
+Selector = couch_util:get_value(selector, Options),
+case {Filter, DocIds, Selector} of
+{undefined, undefined, undefined} ->
+{ok, nil};
+{<<"_", _/binary>>, undefined, undefined} ->
+{ok, {view, Filter, query_params(Options)}};
+{_, undefined, undefined} ->
+case parse_user_filter(Filter) of
+{ok, {Doc, FilterName}} ->
+{ok, {user, {Doc, FilterName}, query_params(Options)}};
+{error, Error} ->
+{error, Error}
+end;
+{undefined, _, undefined} ->
+{ok, {docids, DocIds}};
+{undefined, undefined, _} ->
+{ok, {mango, ejsort(mango_selector:normalize(Selector))}};
+_ ->
+Err = "`selector`, `filter` and `doc_ids` are mutually exclusive",
+{error, list_to_binary(Err)}
+end.
+
+
+% Fetches body of filter function from source database. Guaranteed to either
+% return {ok, Body} or an {error, Reason}. Also assume this function might
+% block due to network / socket issues for an undeterminted amount of time.
+-spec fetch(binary(), binary(), binary(), #user_ctx{}) ->
+{ok, {[_]}} | {error, binary()}.
+fetch(DDocName, FilterName, Source, UserCtx) ->
+{Pid, Ref} = spawn_monitor(fun() ->
+try fetch_internal(DDocName, FilterName, Source, UserCtx) of
+Resp ->
+exit({exit_ok, Resp})
+catch
+throw:{fetch_error, Reason} ->
+exit({exit_fetch_error, Reason});
+_OtherTag:Reason ->
+exit({exit_other_error, Reason})
+end
+end),
+receive
+{'DOWN', Ref, process, Pid, {exit_ok, Resp}} ->
+{ok, Resp};
+{'DOWN', Ref, process, Pid, {exit_fetch_error, Reason}} ->
+{error, Reason};
+{'DOWN', Ref, process, Pid, {exit_other_error, Reason}} ->
+{error, couch_util:to_binary(Reason)}
+end.
+
+
+% Get replication type and view (if any) from replication document props
+-spec view_type([_], [_]) ->
+{view, {binary(), binary()}} | {db, nil} | {error, binary()}.
+view_type(Props, Options) ->
+case couch_util:get_value(<<"filter">>, Props) of
+<<"_view">> ->
+{QP} = couch_util:get_value(query_params, Options, {[]}),
Review comment:
Let's keep for now. It was copy from _utils module:
https://github.com/apache/couchdb/blob/f4c6113808d1809469df9c8be9d2f83ef399f064/src/couch_replicator/src/couch_replicator_utils.erl#L49-L56
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator
iilyak commented on a change in pull request #470: Scheduling Replicator URL: https://github.com/apache/couchdb/pull/470#discussion_r112750006 ## File path: src/couch_replicator/src/couch_replicator_scheduler.erl ## @@ -0,0 +1,1430 @@ +% Licensed under the Apache License, Version 2.0 (the "License"); you may not +% use this file except in compliance with the License. You may obtain a copy of +% the License at +% +% http://www.apache.org/licenses/LICENSE-2.0 +% +% Unless required by applicable law or agreed to in writing, software +% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +% License for the specific language governing permissions and limitations under +% the License. + +-module(couch_replicator_scheduler). + +-behaviour(gen_server). +-behaviour(config_listener). + +-export([ +start_link/0 +]). + +-export([ + init/1, + terminate/2, + handle_call/3, + handle_info/2, + handle_cast/2, + code_change/3, + format_status/2 +]). + +-export([ Review comment: I think we need a `restart_job(Id)`. Either here or in `replicator:restart_replication`. Currently in order to restart replication operators has to do crazy low-level thing like: `exit(list_to_pid(), restart_replication_task).` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator
iilyak commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112748558
##
File path: src/couch_replicator/src/couch_replicator_filters.erl
##
@@ -0,0 +1,214 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_replicator_filters).
+
+-export([
+parse/1,
+fetch/4,
+view_type/2,
+ejsort/1
+]).
+
+-include_lib("couch/include/couch_db.hrl").
+
+
+% Parse the filter from replication options proplist.
+% Return {ok, {FilterType,...}} | {error, ParseError}.
+% For `user` filter, i.e. filters specified as user code
+% in source database, this code doesn't fetch the filter
+% code, but only returns the name of the filter.
+-spec parse([_]) ->
+{ok, nil} |
+{ok, {view, binary(), {[_]}}} |
+{ok, {user, {binary(), binary()}, {[_]}}} |
+{ok, {docids, [_]}} |
+{ok, {mango, {[_]}}} |
+{error, binary()}.
+parse(Options) ->
+Filter = couch_util:get_value(filter, Options),
+DocIds = couch_util:get_value(doc_ids, Options),
+Selector = couch_util:get_value(selector, Options),
+case {Filter, DocIds, Selector} of
+{undefined, undefined, undefined} ->
+{ok, nil};
+{<<"_", _/binary>>, undefined, undefined} ->
+{ok, {view, Filter, query_params(Options)}};
+{_, undefined, undefined} ->
+case parse_user_filter(Filter) of
+{ok, {Doc, FilterName}} ->
+{ok, {user, {Doc, FilterName}, query_params(Options)}};
+{error, Error} ->
+{error, Error}
+end;
+{undefined, _, undefined} ->
+{ok, {docids, DocIds}};
+{undefined, undefined, _} ->
+{ok, {mango, ejsort(mango_selector:normalize(Selector))}};
+_ ->
+Err = "`selector`, `filter` and `doc_ids` are mutually exclusive",
+{error, list_to_binary(Err)}
+end.
+
+
+% Fetches body of filter function from source database. Guaranteed to either
+% return {ok, Body} or an {error, Reason}. Also assume this function might
+% block due to network / socket issues for an undeterminted amount of time.
+-spec fetch(binary(), binary(), binary(), #user_ctx{}) ->
+{ok, {[_]}} | {error, binary()}.
+fetch(DDocName, FilterName, Source, UserCtx) ->
+{Pid, Ref} = spawn_monitor(fun() ->
+try fetch_internal(DDocName, FilterName, Source, UserCtx) of
+Resp ->
+exit({exit_ok, Resp})
+catch
+throw:{fetch_error, Reason} ->
+exit({exit_fetch_error, Reason});
+_OtherTag:Reason ->
+exit({exit_other_error, Reason})
+end
+end),
+receive
+{'DOWN', Ref, process, Pid, {exit_ok, Resp}} ->
+{ok, Resp};
+{'DOWN', Ref, process, Pid, {exit_fetch_error, Reason}} ->
+{error, Reason};
+{'DOWN', Ref, process, Pid, {exit_other_error, Reason}} ->
+{error, couch_util:to_binary(Reason)}
+end.
+
+
+% Get replication type and view (if any) from replication document props
+-spec view_type([_], [_]) ->
+{view, {binary(), binary()}} | {db, nil} | {error, binary()}.
+view_type(Props, Options) ->
+case couch_util:get_value(<<"filter">>, Props) of
+<<"_view">> ->
+{QP} = couch_util:get_value(query_params, Options, {[]}),
+ViewParam = couch_util:get_value(<<"view">>, QP),
+case re:split(ViewParam, <<"/">>) of
Review comment:
Look very similar to `parse_user_filter/1`. Please, ignore.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator
iilyak commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112747929
##
File path: src/couch_replicator/src/couch_replicator_filters.erl
##
@@ -0,0 +1,214 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_replicator_filters).
+
+-export([
+parse/1,
+fetch/4,
+view_type/2,
+ejsort/1
+]).
+
+-include_lib("couch/include/couch_db.hrl").
+
+
+% Parse the filter from replication options proplist.
+% Return {ok, {FilterType,...}} | {error, ParseError}.
+% For `user` filter, i.e. filters specified as user code
+% in source database, this code doesn't fetch the filter
+% code, but only returns the name of the filter.
+-spec parse([_]) ->
+{ok, nil} |
+{ok, {view, binary(), {[_]}}} |
+{ok, {user, {binary(), binary()}, {[_]}}} |
+{ok, {docids, [_]}} |
+{ok, {mango, {[_]}}} |
+{error, binary()}.
+parse(Options) ->
+Filter = couch_util:get_value(filter, Options),
+DocIds = couch_util:get_value(doc_ids, Options),
+Selector = couch_util:get_value(selector, Options),
+case {Filter, DocIds, Selector} of
+{undefined, undefined, undefined} ->
+{ok, nil};
+{<<"_", _/binary>>, undefined, undefined} ->
+{ok, {view, Filter, query_params(Options)}};
+{_, undefined, undefined} ->
+case parse_user_filter(Filter) of
+{ok, {Doc, FilterName}} ->
+{ok, {user, {Doc, FilterName}, query_params(Options)}};
+{error, Error} ->
+{error, Error}
+end;
+{undefined, _, undefined} ->
+{ok, {docids, DocIds}};
+{undefined, undefined, _} ->
+{ok, {mango, ejsort(mango_selector:normalize(Selector))}};
+_ ->
+Err = "`selector`, `filter` and `doc_ids` are mutually exclusive",
+{error, list_to_binary(Err)}
+end.
+
+
+% Fetches body of filter function from source database. Guaranteed to either
+% return {ok, Body} or an {error, Reason}. Also assume this function might
+% block due to network / socket issues for an undeterminted amount of time.
+-spec fetch(binary(), binary(), binary(), #user_ctx{}) ->
+{ok, {[_]}} | {error, binary()}.
+fetch(DDocName, FilterName, Source, UserCtx) ->
+{Pid, Ref} = spawn_monitor(fun() ->
+try fetch_internal(DDocName, FilterName, Source, UserCtx) of
+Resp ->
+exit({exit_ok, Resp})
+catch
+throw:{fetch_error, Reason} ->
+exit({exit_fetch_error, Reason});
+_OtherTag:Reason ->
+exit({exit_other_error, Reason})
+end
+end),
+receive
+{'DOWN', Ref, process, Pid, {exit_ok, Resp}} ->
+{ok, Resp};
+{'DOWN', Ref, process, Pid, {exit_fetch_error, Reason}} ->
+{error, Reason};
+{'DOWN', Ref, process, Pid, {exit_other_error, Reason}} ->
+{error, couch_util:to_binary(Reason)}
+end.
+
+
+% Get replication type and view (if any) from replication document props
+-spec view_type([_], [_]) ->
+{view, {binary(), binary()}} | {db, nil} | {error, binary()}.
+view_type(Props, Options) ->
+case couch_util:get_value(<<"filter">>, Props) of
+<<"_view">> ->
+{QP} = couch_util:get_value(query_params, Options, {[]}),
Review comment:
`ViewParam = couch_util:get_nested_json_value(Options, [query_params,
<<"view">>]).` maybe. Feel free to ignore (cosmetic).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator
iilyak commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112746251
##
File path: src/couch_replicator/src/couch_replicator_filters.erl
##
@@ -0,0 +1,214 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_replicator_filters).
+
+-export([
+parse/1,
+fetch/4,
+view_type/2,
+ejsort/1
+]).
+
+-include_lib("couch/include/couch_db.hrl").
+
+
+% Parse the filter from replication options proplist.
+% Return {ok, {FilterType,...}} | {error, ParseError}.
+% For `user` filter, i.e. filters specified as user code
+% in source database, this code doesn't fetch the filter
+% code, but only returns the name of the filter.
+-spec parse([_]) ->
+{ok, nil} |
+{ok, {view, binary(), {[_]}}} |
+{ok, {user, {binary(), binary()}, {[_]}}} |
+{ok, {docids, [_]}} |
+{ok, {mango, {[_]}}} |
+{error, binary()}.
+parse(Options) ->
+Filter = couch_util:get_value(filter, Options),
+DocIds = couch_util:get_value(doc_ids, Options),
+Selector = couch_util:get_value(selector, Options),
+case {Filter, DocIds, Selector} of
+{undefined, undefined, undefined} ->
+{ok, nil};
+{<<"_", _/binary>>, undefined, undefined} ->
+{ok, {view, Filter, query_params(Options)}};
+{_, undefined, undefined} ->
+case parse_user_filter(Filter) of
+{ok, {Doc, FilterName}} ->
+{ok, {user, {Doc, FilterName}, query_params(Options)}};
+{error, Error} ->
+{error, Error}
+end;
+{undefined, _, undefined} ->
+{ok, {docids, DocIds}};
+{undefined, undefined, _} ->
+{ok, {mango, ejsort(mango_selector:normalize(Selector))}};
+_ ->
+Err = "`selector`, `filter` and `doc_ids` are mutually exclusive",
Review comment:
awesome. I wish we had more of this kind of validations.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator
iilyak commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112746251
##
File path: src/couch_replicator/src/couch_replicator_filters.erl
##
@@ -0,0 +1,214 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_replicator_filters).
+
+-export([
+parse/1,
+fetch/4,
+view_type/2,
+ejsort/1
+]).
+
+-include_lib("couch/include/couch_db.hrl").
+
+
+% Parse the filter from replication options proplist.
+% Return {ok, {FilterType,...}} | {error, ParseError}.
+% For `user` filter, i.e. filters specified as user code
+% in source database, this code doesn't fetch the filter
+% code, but only returns the name of the filter.
+-spec parse([_]) ->
+{ok, nil} |
+{ok, {view, binary(), {[_]}}} |
+{ok, {user, {binary(), binary()}, {[_]}}} |
+{ok, {docids, [_]}} |
+{ok, {mango, {[_]}}} |
+{error, binary()}.
+parse(Options) ->
+Filter = couch_util:get_value(filter, Options),
+DocIds = couch_util:get_value(doc_ids, Options),
+Selector = couch_util:get_value(selector, Options),
+case {Filter, DocIds, Selector} of
+{undefined, undefined, undefined} ->
+{ok, nil};
+{<<"_", _/binary>>, undefined, undefined} ->
+{ok, {view, Filter, query_params(Options)}};
+{_, undefined, undefined} ->
+case parse_user_filter(Filter) of
+{ok, {Doc, FilterName}} ->
+{ok, {user, {Doc, FilterName}, query_params(Options)}};
+{error, Error} ->
+{error, Error}
+end;
+{undefined, _, undefined} ->
+{ok, {docids, DocIds}};
+{undefined, undefined, _} ->
+{ok, {mango, ejsort(mango_selector:normalize(Selector))}};
+_ ->
+Err = "`selector`, `filter` and `doc_ids` are mutually exclusive",
Review comment:
awesome. I wish we had more of this kind of validations.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] janl commented on a change in pull request #483: New couchup 1.x -> 2.x database migration tool
janl commented on a change in pull request #483: New couchup 1.x -> 2.x
database migration tool
URL: https://github.com/apache/couchdb/pull/483#discussion_r112739905
##
File path: rel/overlay/bin/couchup
##
@@ -0,0 +1,474 @@
+#!/usr/bin/env python
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations under
+# the License.
+
+import argparse
+import base64
+import json
+import textwrap
+import threading
+import time
+import sys
+try:
+from urllib import quote
+except ImportError:
+from urllib.parse import quote
+import requests
+try:
+import progressbar
+HAVE_BAR = True
+except ImportError:
+HAVE_BAR = False
+
+def _tojson(req):
+if requests.__version__[0] == '0':
+return json.loads(req.content)
+return req.json()
+
+def _args(args):
+args = vars(args)
+if args['password']:
+args['creds'] = (args['login'], args['password'])
+else:
+args['creds'] = None
+return args
+
+def _do_list(args):
+port = str(args['local_port'])
+req = requests.get('http://127.0.0.1:' + port + '/_all_dbs',
+auth=args['creds'])
+req.raise_for_status()
+dbs = _tojson(req)
+local_dbs = [x for x in dbs if "shards" not in x
+and x not in ['_dbs', '_nodes']]
+clustered_dbs = list(set(
+[x.split('/')[2].split('.')[0] for x in dbs if "shards" in x]
+))
+if not args['include_system_dbs']:
+local_dbs = [x for x in local_dbs if x[0] != '_']
+clustered_dbs = [x for x in clustered_dbs if x[0] != '_']
+local_dbs.sort()
+clustered_dbs.sort()
+if args.get('clustered'):
+return clustered_dbs
+return local_dbs
+
+def _list(args):
+args = _args(args)
+ret = _do_list(args)
+print(", ".join(ret))
+
+def _watch_replication(db,
+local_port=5986,
+clustered_port=5984,
+creds=None,
+no_progress_bar=False,
+quiet=False,
+timeout=30):
+"""Watches replication, optionally with a progressbar."""
+time.sleep(1)
+if not quiet:
+print("Replication started.")
+url = "http://127.0.0.1:{}/{}".format(local_port, db)
+try:
+req = requests.get(url, auth=creds)
+req.raise_for_status()
+req = _tojson(req)
+local_docs = req['doc_count']
+local_size = req['data_size']
+except requests.exceptions.HTTPError:
+raise Exception('Cannot retrieve {} doc_count!'.format(db))
+if local_size == 0:
+return
+if HAVE_BAR and not no_progress_bar and not quiet:
+widgets = [
+db,
+' ', progressbar.Percentage(),
+' ', progressbar.Bar(marker=progressbar.RotatingMarker()),
+' ', progressbar.ETA(),
+' ', progressbar.FileTransferSpeed(),
+]
+progbar = progressbar.ProgressBar(widgets=widgets,
+maxval=local_size).start()
+count = 0
+stall_count = 0
+url = "http://127.0.0.1:{}/{}".format(clustered_port, db)
+while count < local_docs:
+try:
+req = requests.get(url, auth=creds)
+req.raise_for_status()
+req = _tojson(req)
+clus_count = req['doc_count']
+clus_size = req['data_size']
+except requests.exceptions.HTTPError as exc:
+if exc.response.status_code == 404:
+clus_count = 0
+clus_size = 0
+else:
+raise Exception('Cannot retrieve {} doc_count!'.format(db))
+if count == clus_count:
+stall_count += 1
+else:
+stall_count = 0
+if stall_count == timeout:
+print("Replication is stalled. Increase timeout or reduce load.")
+exit(1)
+if HAVE_BAR and not no_progress_bar and not quiet:
+if clus_size > local_size:
+clus_size = local_size
+progbar.update(clus_size)
+count = clus_count
+time.sleep(1)
+if HAVE_BAR and not no_progress_bar and not quiet:
+progbar.finish()
+return 0
+
+def _put_filter(args, db=None):
+"""Adds _design/repl_filters tombstone replication filter to DB."""
+ddoc = {
+'_id': '_design/repl_filters',
+'filters': {
+'no_deleted': 'function(doc,req){return !doc._deleted;};'
+}
+}
+try:
+req = requests.get(
+'http://127.0.0.1:{}/{}/_design/repl_filters'.format(
+args['local_port'], db),
+
[GitHub] janl commented on a change in pull request #483: New couchup 1.x -> 2.x database migration tool
janl commented on a change in pull request #483: New couchup 1.x -> 2.x
database migration tool
URL: https://github.com/apache/couchdb/pull/483#discussion_r112739684
##
File path: rel/overlay/bin/couchup
##
@@ -0,0 +1,474 @@
+#!/usr/bin/env python
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations under
+# the License.
+
+import argparse
+import base64
+import json
+import textwrap
+import threading
+import time
+import sys
+try:
+from urllib import quote
+except ImportError:
+from urllib.parse import quote
+import requests
+try:
+import progressbar
+HAVE_BAR = True
+except ImportError:
+HAVE_BAR = False
+
+def _tojson(req):
+if requests.__version__[0] == '0':
+return json.loads(req.content)
+return req.json()
+
+def _args(args):
+args = vars(args)
+if args['password']:
+args['creds'] = (args['login'], args['password'])
+else:
+args['creds'] = None
+return args
+
+def _do_list(args):
+port = str(args['local_port'])
+req = requests.get('http://127.0.0.1:' + port + '/_all_dbs',
+auth=args['creds'])
+req.raise_for_status()
+dbs = _tojson(req)
+local_dbs = [x for x in dbs if "shards" not in x
+and x not in ['_dbs', '_nodes']]
+clustered_dbs = list(set(
+[x.split('/')[2].split('.')[0] for x in dbs if "shards" in x]
+))
+if not args['include_system_dbs']:
+local_dbs = [x for x in local_dbs if x[0] != '_']
+clustered_dbs = [x for x in clustered_dbs if x[0] != '_']
+local_dbs.sort()
+clustered_dbs.sort()
+if args.get('clustered'):
+return clustered_dbs
+return local_dbs
+
+def _list(args):
+args = _args(args)
+ret = _do_list(args)
+print(", ".join(ret))
+
+def _watch_replication(db,
+local_port=5986,
+clustered_port=5984,
+creds=None,
+no_progress_bar=False,
+quiet=False,
+timeout=30):
+"""Watches replication, optionally with a progressbar."""
+time.sleep(1)
+if not quiet:
+print("Replication started.")
+url = "http://127.0.0.1:{}/{}".format(local_port, db)
+try:
+req = requests.get(url, auth=creds)
+req.raise_for_status()
+req = _tojson(req)
+local_docs = req['doc_count']
Review comment:
gotcha
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] janl commented on a change in pull request #483: New couchup 1.x -> 2.x database migration tool
janl commented on a change in pull request #483: New couchup 1.x -> 2.x
database migration tool
URL: https://github.com/apache/couchdb/pull/483#discussion_r112739730
##
File path: rel/overlay/bin/couchup
##
@@ -0,0 +1,474 @@
+#!/usr/bin/env python
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations under
+# the License.
+
+import argparse
+import base64
+import json
+import textwrap
+import threading
+import time
+import sys
+try:
+from urllib import quote
+except ImportError:
+from urllib.parse import quote
+import requests
+try:
+import progressbar
+HAVE_BAR = True
+except ImportError:
+HAVE_BAR = False
+
+def _tojson(req):
+if requests.__version__[0] == '0':
+return json.loads(req.content)
+return req.json()
+
+def _args(args):
+args = vars(args)
+if args['password']:
+args['creds'] = (args['login'], args['password'])
+else:
+args['creds'] = None
+return args
+
+def _do_list(args):
+port = str(args['local_port'])
+req = requests.get('http://127.0.0.1:' + port + '/_all_dbs',
+auth=args['creds'])
+req.raise_for_status()
+dbs = _tojson(req)
+local_dbs = [x for x in dbs if "shards" not in x
+and x not in ['_dbs', '_nodes']]
+clustered_dbs = list(set(
+[x.split('/')[2].split('.')[0] for x in dbs if "shards" in x]
+))
+if not args['include_system_dbs']:
+local_dbs = [x for x in local_dbs if x[0] != '_']
+clustered_dbs = [x for x in clustered_dbs if x[0] != '_']
+local_dbs.sort()
+clustered_dbs.sort()
+if args.get('clustered'):
+return clustered_dbs
+return local_dbs
+
+def _list(args):
+args = _args(args)
+ret = _do_list(args)
+print(", ".join(ret))
+
+def _watch_replication(db,
+local_port=5986,
+clustered_port=5984,
+creds=None,
+no_progress_bar=False,
+quiet=False,
+timeout=30):
+"""Watches replication, optionally with a progressbar."""
+time.sleep(1)
+if not quiet:
+print("Replication started.")
+url = "http://127.0.0.1:{}/{}".format(local_port, db)
+try:
+req = requests.get(url, auth=creds)
+req.raise_for_status()
+req = _tojson(req)
+local_docs = req['doc_count']
+local_size = req['data_size']
+except requests.exceptions.HTTPError:
+raise Exception('Cannot retrieve {} doc_count!'.format(db))
+if local_size == 0:
+return
+if HAVE_BAR and not no_progress_bar and not quiet:
+widgets = [
+db,
+' ', progressbar.Percentage(),
+' ', progressbar.Bar(marker=progressbar.RotatingMarker()),
+' ', progressbar.ETA(),
+' ', progressbar.FileTransferSpeed(),
+]
+progbar = progressbar.ProgressBar(widgets=widgets,
+maxval=local_size).start()
+count = 0
+stall_count = 0
+url = "http://127.0.0.1:{}/{}".format(clustered_port, db)
+while count < local_docs:
+try:
+req = requests.get(url, auth=creds)
+req.raise_for_status()
+req = _tojson(req)
+clus_count = req['doc_count']
+clus_size = req['data_size']
+except requests.exceptions.HTTPError as exc:
+if exc.response.status_code == 404:
+clus_count = 0
+clus_size = 0
+else:
+raise Exception('Cannot retrieve {} doc_count!'.format(db))
+if count == clus_count:
+stall_count += 1
+else:
+stall_count = 0
+if stall_count == timeout:
+print("Replication is stalled. Increase timeout or reduce load.")
+exit(1)
+if HAVE_BAR and not no_progress_bar and not quiet:
+if clus_size > local_size:
+clus_size = local_size
+progbar.update(clus_size)
+count = clus_count
+time.sleep(1)
Review comment:
aye.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] tonysun83 commented on issue #476: Couchdb 3376 fix mem3 shards
tonysun83 commented on issue #476: Couchdb 3376 fix mem3 shards URL: https://github.com/apache/couchdb/pull/476#issuecomment-296252888 tests pass for me now after rebase, +1, awesome analysis with the boxplots This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wohali commented on issue #483: New couchup 1.x -> 2.x database migration tool
wohali commented on issue #483: New couchup 1.x -> 2.x database migration tool URL: https://github.com/apache/couchdb/pull/483#issuecomment-296251199 Thanks @janl. I'll correct a few of the more straightforward tweaks today. I'd like to get feedback from at least a couple of people trying the script (other than me!) before merging, but if no one responds on user@ or dev@ by middle of next week, I'll merge anyway. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wohali commented on a change in pull request #483: New couchup 1.x -> 2.x database migration tool
wohali commented on a change in pull request #483: New couchup 1.x -> 2.x
database migration tool
URL: https://github.com/apache/couchdb/pull/483#discussion_r112735903
##
File path: rel/overlay/bin/couchup
##
@@ -0,0 +1,474 @@
+#!/usr/bin/env python
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations under
+# the License.
+
+import argparse
+import base64
+import json
+import textwrap
+import threading
+import time
+import sys
+try:
+from urllib import quote
+except ImportError:
+from urllib.parse import quote
+import requests
+try:
+import progressbar
+HAVE_BAR = True
+except ImportError:
+HAVE_BAR = False
+
+def _tojson(req):
+if requests.__version__[0] == '0':
+return json.loads(req.content)
+return req.json()
+
+def _args(args):
+args = vars(args)
+if args['password']:
+args['creds'] = (args['login'], args['password'])
+else:
+args['creds'] = None
+return args
+
+def _do_list(args):
+port = str(args['local_port'])
+req = requests.get('http://127.0.0.1:' + port + '/_all_dbs',
+auth=args['creds'])
+req.raise_for_status()
+dbs = _tojson(req)
+local_dbs = [x for x in dbs if "shards" not in x
+and x not in ['_dbs', '_nodes']]
+clustered_dbs = list(set(
+[x.split('/')[2].split('.')[0] for x in dbs if "shards" in x]
+))
+if not args['include_system_dbs']:
+local_dbs = [x for x in local_dbs if x[0] != '_']
+clustered_dbs = [x for x in clustered_dbs if x[0] != '_']
+local_dbs.sort()
+clustered_dbs.sort()
+if args.get('clustered'):
+return clustered_dbs
+return local_dbs
+
+def _list(args):
+args = _args(args)
+ret = _do_list(args)
+print(", ".join(ret))
+
+def _watch_replication(db,
+local_port=5986,
+clustered_port=5984,
+creds=None,
+no_progress_bar=False,
+quiet=False,
+timeout=30):
+"""Watches replication, optionally with a progressbar."""
+time.sleep(1)
+if not quiet:
+print("Replication started.")
+url = "http://127.0.0.1:{}/{}".format(local_port, db)
+try:
+req = requests.get(url, auth=creds)
+req.raise_for_status()
+req = _tojson(req)
+local_docs = req['doc_count']
+local_size = req['data_size']
+except requests.exceptions.HTTPError:
+raise Exception('Cannot retrieve {} doc_count!'.format(db))
+if local_size == 0:
+return
+if HAVE_BAR and not no_progress_bar and not quiet:
+widgets = [
+db,
+' ', progressbar.Percentage(),
+' ', progressbar.Bar(marker=progressbar.RotatingMarker()),
+' ', progressbar.ETA(),
+' ', progressbar.FileTransferSpeed(),
+]
+progbar = progressbar.ProgressBar(widgets=widgets,
+maxval=local_size).start()
+count = 0
+stall_count = 0
+url = "http://127.0.0.1:{}/{}".format(clustered_port, db)
+while count < local_docs:
+try:
+req = requests.get(url, auth=creds)
+req.raise_for_status()
+req = _tojson(req)
+clus_count = req['doc_count']
+clus_size = req['data_size']
+except requests.exceptions.HTTPError as exc:
+if exc.response.status_code == 404:
+clus_count = 0
+clus_size = 0
+else:
+raise Exception('Cannot retrieve {} doc_count!'.format(db))
+if count == clus_count:
+stall_count += 1
+else:
+stall_count = 0
+if stall_count == timeout:
+print("Replication is stalled. Increase timeout or reduce load.")
+exit(1)
+if HAVE_BAR and not no_progress_bar and not quiet:
+if clus_size > local_size:
+clus_size = local_size
+progbar.update(clus_size)
+count = clus_count
+time.sleep(1)
+if HAVE_BAR and not no_progress_bar and not quiet:
+progbar.finish()
+return 0
+
+def _put_filter(args, db=None):
+"""Adds _design/repl_filters tombstone replication filter to DB."""
+ddoc = {
+'_id': '_design/repl_filters',
+'filters': {
+'no_deleted': 'function(doc,req){return !doc._deleted;};'
+}
+}
+try:
+req = requests.get(
+'http://127.0.0.1:{}/{}/_design/repl_filters'.format(
+args['local_port'], db),
+
[GitHub] wohali commented on a change in pull request #483: New couchup 1.x -> 2.x database migration tool
wohali commented on a change in pull request #483: New couchup 1.x -> 2.x
database migration tool
URL: https://github.com/apache/couchdb/pull/483#discussion_r112735609
##
File path: rel/overlay/bin/couchup
##
@@ -0,0 +1,474 @@
+#!/usr/bin/env python
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations under
+# the License.
+
+import argparse
+import base64
+import json
+import textwrap
+import threading
+import time
+import sys
+try:
+from urllib import quote
+except ImportError:
+from urllib.parse import quote
+import requests
+try:
+import progressbar
+HAVE_BAR = True
+except ImportError:
+HAVE_BAR = False
+
+def _tojson(req):
+if requests.__version__[0] == '0':
+return json.loads(req.content)
+return req.json()
+
+def _args(args):
+args = vars(args)
+if args['password']:
+args['creds'] = (args['login'], args['password'])
+else:
+args['creds'] = None
+return args
+
+def _do_list(args):
+port = str(args['local_port'])
+req = requests.get('http://127.0.0.1:' + port + '/_all_dbs',
+auth=args['creds'])
+req.raise_for_status()
+dbs = _tojson(req)
+local_dbs = [x for x in dbs if "shards" not in x
+and x not in ['_dbs', '_nodes']]
+clustered_dbs = list(set(
+[x.split('/')[2].split('.')[0] for x in dbs if "shards" in x]
+))
+if not args['include_system_dbs']:
+local_dbs = [x for x in local_dbs if x[0] != '_']
+clustered_dbs = [x for x in clustered_dbs if x[0] != '_']
+local_dbs.sort()
+clustered_dbs.sort()
+if args.get('clustered'):
+return clustered_dbs
+return local_dbs
+
+def _list(args):
+args = _args(args)
+ret = _do_list(args)
+print(", ".join(ret))
+
+def _watch_replication(db,
+local_port=5986,
+clustered_port=5984,
+creds=None,
+no_progress_bar=False,
+quiet=False,
+timeout=30):
+"""Watches replication, optionally with a progressbar."""
+time.sleep(1)
+if not quiet:
+print("Replication started.")
+url = "http://127.0.0.1:{}/{}".format(local_port, db)
+try:
+req = requests.get(url, auth=creds)
+req.raise_for_status()
+req = _tojson(req)
+local_docs = req['doc_count']
+local_size = req['data_size']
+except requests.exceptions.HTTPError:
+raise Exception('Cannot retrieve {} doc_count!'.format(db))
+if local_size == 0:
+return
+if HAVE_BAR and not no_progress_bar and not quiet:
+widgets = [
+db,
+' ', progressbar.Percentage(),
+' ', progressbar.Bar(marker=progressbar.RotatingMarker()),
+' ', progressbar.ETA(),
+' ', progressbar.FileTransferSpeed(),
+]
+progbar = progressbar.ProgressBar(widgets=widgets,
+maxval=local_size).start()
+count = 0
+stall_count = 0
+url = "http://127.0.0.1:{}/{}".format(clustered_port, db)
+while count < local_docs:
+try:
+req = requests.get(url, auth=creds)
+req.raise_for_status()
+req = _tojson(req)
+clus_count = req['doc_count']
+clus_size = req['data_size']
+except requests.exceptions.HTTPError as exc:
+if exc.response.status_code == 404:
+clus_count = 0
+clus_size = 0
+else:
+raise Exception('Cannot retrieve {} doc_count!'.format(db))
+if count == clus_count:
+stall_count += 1
+else:
+stall_count = 0
+if stall_count == timeout:
+print("Replication is stalled. Increase timeout or reduce load.")
+exit(1)
+if HAVE_BAR and not no_progress_bar and not quiet:
+if clus_size > local_size:
+clus_size = local_size
+progbar.update(clus_size)
+count = clus_count
+time.sleep(1)
Review comment:
Disagree...this is only used for the --timeout=XX option, and needs to stay
at 1s granularity otherwise the timeout needs to be forced to be a multiple of
the per-loop value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] wohali commented on a change in pull request #483: New couchup 1.x -> 2.x database migration tool
wohali commented on a change in pull request #483: New couchup 1.x -> 2.x
database migration tool
URL: https://github.com/apache/couchdb/pull/483#discussion_r112735444
##
File path: rel/overlay/bin/couchup
##
@@ -0,0 +1,474 @@
+#!/usr/bin/env python
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations under
+# the License.
+
+import argparse
+import base64
+import json
+import textwrap
+import threading
+import time
+import sys
+try:
+from urllib import quote
+except ImportError:
+from urllib.parse import quote
+import requests
+try:
+import progressbar
+HAVE_BAR = True
+except ImportError:
+HAVE_BAR = False
+
+def _tojson(req):
+if requests.__version__[0] == '0':
+return json.loads(req.content)
+return req.json()
+
+def _args(args):
+args = vars(args)
+if args['password']:
+args['creds'] = (args['login'], args['password'])
+else:
+args['creds'] = None
+return args
+
+def _do_list(args):
+port = str(args['local_port'])
+req = requests.get('http://127.0.0.1:' + port + '/_all_dbs',
+auth=args['creds'])
+req.raise_for_status()
+dbs = _tojson(req)
+local_dbs = [x for x in dbs if "shards" not in x
+and x not in ['_dbs', '_nodes']]
+clustered_dbs = list(set(
+[x.split('/')[2].split('.')[0] for x in dbs if "shards" in x]
+))
+if not args['include_system_dbs']:
+local_dbs = [x for x in local_dbs if x[0] != '_']
+clustered_dbs = [x for x in clustered_dbs if x[0] != '_']
+local_dbs.sort()
+clustered_dbs.sort()
+if args.get('clustered'):
+return clustered_dbs
+return local_dbs
+
+def _list(args):
+args = _args(args)
+ret = _do_list(args)
+print(", ".join(ret))
+
+def _watch_replication(db,
+local_port=5986,
+clustered_port=5984,
+creds=None,
+no_progress_bar=False,
+quiet=False,
+timeout=30):
+"""Watches replication, optionally with a progressbar."""
+time.sleep(1)
+if not quiet:
+print("Replication started.")
+url = "http://127.0.0.1:{}/{}".format(local_port, db)
+try:
+req = requests.get(url, auth=creds)
+req.raise_for_status()
+req = _tojson(req)
+local_docs = req['doc_count']
Review comment:
Good point, will change. I meant "node-local" docs. Could just make this
more generic with source/target rather than local_/clus_.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] wohali commented on a change in pull request #483: New couchup 1.x -> 2.x database migration tool
wohali commented on a change in pull request #483: New couchup 1.x -> 2.x
database migration tool
URL: https://github.com/apache/couchdb/pull/483#discussion_r112735306
##
File path: rel/overlay/bin/couchup
##
@@ -0,0 +1,474 @@
+#!/usr/bin/env python
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations under
+# the License.
+
+import argparse
+import base64
+import json
+import textwrap
+import threading
+import time
+import sys
+try:
+from urllib import quote
+except ImportError:
+from urllib.parse import quote
+import requests
+try:
+import progressbar
+HAVE_BAR = True
+except ImportError:
+HAVE_BAR = False
+
+def _tojson(req):
+if requests.__version__[0] == '0':
+return json.loads(req.content)
+return req.json()
+
+def _args(args):
+args = vars(args)
+if args['password']:
+args['creds'] = (args['login'], args['password'])
+else:
+args['creds'] = None
+return args
+
+def _do_list(args):
+port = str(args['local_port'])
+req = requests.get('http://127.0.0.1:' + port + '/_all_dbs',
+auth=args['creds'])
+req.raise_for_status()
+dbs = _tojson(req)
+local_dbs = [x for x in dbs if "shards" not in x
+and x not in ['_dbs', '_nodes']]
+clustered_dbs = list(set(
+[x.split('/')[2].split('.')[0] for x in dbs if "shards" in x]
+))
+if not args['include_system_dbs']:
+local_dbs = [x for x in local_dbs if x[0] != '_']
+clustered_dbs = [x for x in clustered_dbs if x[0] != '_']
+local_dbs.sort()
+clustered_dbs.sort()
+if args.get('clustered'):
+return clustered_dbs
+return local_dbs
+
+def _list(args):
+args = _args(args)
+ret = _do_list(args)
+print(", ".join(ret))
+
+def _watch_replication(db,
+local_port=5986,
+clustered_port=5984,
+creds=None,
+no_progress_bar=False,
Review comment:
This is because of how argparse works; it's annoying to get it to handle
"--flag={true,false}" but very easy to get it to handle "--flag" (and store
either True or False depending) I want the default for the progress bar to be
on, so the CLI flag needs to turn it off.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] wohali commented on a change in pull request #483: New couchup 1.x -> 2.x database migration tool
wohali commented on a change in pull request #483: New couchup 1.x -> 2.x
database migration tool
URL: https://github.com/apache/couchdb/pull/483#discussion_r112734909
##
File path: rel/overlay/bin/couchup
##
@@ -0,0 +1,474 @@
+#!/usr/bin/env python
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations under
+# the License.
+
+import argparse
+import base64
+import json
+import textwrap
+import threading
+import time
+import sys
+try:
+from urllib import quote
+except ImportError:
+from urllib.parse import quote
+import requests
+try:
+import progressbar
+HAVE_BAR = True
+except ImportError:
+HAVE_BAR = False
+
+def _tojson(req):
+if requests.__version__[0] == '0':
+return json.loads(req.content)
+return req.json()
+
+def _args(args):
+args = vars(args)
+if args['password']:
+args['creds'] = (args['login'], args['password'])
+else:
+args['creds'] = None
+return args
+
+def _do_list(args):
+port = str(args['local_port'])
+req = requests.get('http://127.0.0.1:' + port + '/_all_dbs',
Review comment:
I presumed this might be one of the first requests, and agree that it'd be a
good thing for a first timer to tackle, though it's a trivial refactor.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] wohali commented on a change in pull request #483: New couchup 1.x -> 2.x database migration tool
wohali commented on a change in pull request #483: New couchup 1.x -> 2.x database migration tool URL: https://github.com/apache/couchdb/pull/483#discussion_r112734793 ## File path: rel/overlay/bin/couchup ## @@ -0,0 +1,474 @@ +#!/usr/bin/env python +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. + +import argparse +import base64 +import json +import textwrap +import threading +import time +import sys +try: +from urllib import quote +except ImportError: +from urllib.parse import quote +import requests +try: +import progressbar +HAVE_BAR = True +except ImportError: +HAVE_BAR = False + +def _tojson(req): Review comment: Yup, this is how I support python-requests 0.x. Will do. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator
iilyak commented on a change in pull request #470: Scheduling Replicator
URL: https://github.com/apache/couchdb/pull/470#discussion_r112732052
##
File path: src/couch_replicator/src/couch_replicator_httpd.erl
##
@@ -24,13 +30,68 @@
to_binary/1
]).
--export([handle_req/1]).
+
+-define(DEFAULT_TASK_LIMIT, 100).
+-define(REPDB, <<"_replicator">>).
+
+
+handle_scheduler_req(#httpd{method='GET', path_parts=[_,<<"jobs">>]}=Req) ->
+Limit = couch_replicator_httpd_util:parse_int_param(Req, "limit",
+?DEFAULT_TASK_LIMIT, 0, infinity),
+Skip = couch_replicator_httpd_util:parse_int_param(Req, "skip", 0, 0,
+infinity),
+{Replies, _BadNodes} = rpc:multicall(couch_replicator_scheduler, jobs, []),
+Flatlist = lists:concat(Replies),
+% couch_replicator_scheduler:job_ejson/1 guarantees {id, Id} to be the
+% the first item in the list
+Sorted = lists:sort(fun({[{id,A}|_]},{[{id,B}|_]}) -> A =< B end,
Flatlist),
+Total = length(Sorted),
+Offset = min(Skip, Total),
+Sublist = lists:sublist(Sorted, Offset+1, Limit),
+Sublist1 = [couch_replicator_httpd_util:update_db_name(Task)
+|| Task <- Sublist],
+send_json(Req, {[{total_rows, Total}, {offset, Offset}, {jobs,
Sublist1}]});
+handle_scheduler_req(#httpd{method='GET',
path_parts=[_,<<"jobs">>,JobId]}=Req) ->
+case couch_replicator:job(JobId) of
+{ok, JobInfo} ->
+send_json(Req,
couch_replicator_httpd_util:update_db_name(JobInfo));
+{error, not_found} ->
+throw(not_found)
Review comment:
Fine with me feel free to ignore. My rationale behind always returning the
breadcrumbs so we have sane log messages in the following cases:
- if the function would be called from other places in the codebase (i.e. in
the context when the reply is not delivered to the customer)
- if the caller would crash and the error would be logged using a generic
error logger (the one which handles 500)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (COUCHDB-3376) Fix mem3_shards under load
[ https://issues.apache.org/jira/browse/COUCHDB-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979032#comment-15979032 ] ASF subversion and git services commented on COUCHDB-3376: -- Commit 32e5006f8d7e08579dbe9fab0e7e7ac6d78d7d40 in couchdb's branch refs/heads/COUCHDB-3376-fix-mem3-shards from [~vatamane] [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=32e5006 ] Add unit tests for mem3_shards COUCHDB-3376 > Fix mem3_shards under load > -- > > Key: COUCHDB-3376 > URL: https://issues.apache.org/jira/browse/COUCHDB-3376 > Project: CouchDB > Issue Type: Bug >Reporter: Paul Joseph Davis > > There were two issues with mem3_shards that were fixed while I've been > testing the PSE code. > The first issue was found by [~jaydoane] where a database can have its shards > inserted into the cache after its been deleted. This can happen if a client > does a rapid CREATE/DELETE/GET cycle on a database. The fix for this is to > track the changes feed update sequence from the changes feed listener and > only insert shard maps that come from a client that has read as recent of an > update_seq as mem3_shards. > The second issue found during heavy benchmarking was that large shard maps > (in the Q>=128 range) can quite easily cause mem3_shards to backup when > there's a thundering herd attempting to open the database. There's no > coordination among workers trying to add a shard map to the cache so if a > bunch of independent clients all send the shard map at once (say, at the > beginning of a benchmark) then mem3_shards can get overwhelmed. The fix for > this was two fold. First, rather than send the shard map directly to > mem3_shards, we copy it into a spawned process and when/if mem3_shards wants > to write it, it tells this writer process to do its business. The second > optimization for this change is to create an ets table to track these > processes. Then independent clients can check if a shard map is already > enroute to mem3_shards by using ets:insert_new and canceling their writer if > that returns false. > PR incoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] davisp commented on issue #476: Couchdb 3376 fix mem3 shards
davisp commented on issue #476: Couchdb 3376 fix mem3 shards URL: https://github.com/apache/couchdb/pull/476#issuecomment-296228469 @tonysun83 After adding those config options I went ahead and rebased this on master so that failing test is fixed on this branch as well. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (COUCHDB-3376) Fix mem3_shards under load
[ https://issues.apache.org/jira/browse/COUCHDB-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978952#comment-15978952 ] ASF subversion and git services commented on COUCHDB-3376: -- Commit e4c3705def6021a6b801c0bc0ceaac4abbc7c0d8 in couchdb's branch refs/heads/COUCHDB-3376-fix-mem3-shards from [~paul.joseph.davis] [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=e4c3705 ] Fix stale shards cache There's a race condition in mem3_shards that can result in having shards in the cache for a database that's been deleted. This results in a confused cluster that thinks a database exists until you attempt to open it. The fix is to ignore any cache insert requests that come from an older version of the dbs db than mem3_shards cache knows about. Big thanks to @jdoane for the identification and original patch. COUCHDB-3376 > Fix mem3_shards under load > -- > > Key: COUCHDB-3376 > URL: https://issues.apache.org/jira/browse/COUCHDB-3376 > Project: CouchDB > Issue Type: Bug >Reporter: Paul Joseph Davis > > There were two issues with mem3_shards that were fixed while I've been > testing the PSE code. > The first issue was found by [~jaydoane] where a database can have its shards > inserted into the cache after its been deleted. This can happen if a client > does a rapid CREATE/DELETE/GET cycle on a database. The fix for this is to > track the changes feed update sequence from the changes feed listener and > only insert shard maps that come from a client that has read as recent of an > update_seq as mem3_shards. > The second issue found during heavy benchmarking was that large shard maps > (in the Q>=128 range) can quite easily cause mem3_shards to backup when > there's a thundering herd attempting to open the database. There's no > coordination among workers trying to add a shard map to the cache so if a > bunch of independent clients all send the shard map at once (say, at the > beginning of a benchmark) then mem3_shards can get overwhelmed. The fix for > this was two fold. First, rather than send the shard map directly to > mem3_shards, we copy it into a spawned process and when/if mem3_shards wants > to write it, it tells this writer process to do its business. The second > optimization for this change is to create an ets table to track these > processes. Then independent clients can check if a shard map is already > enroute to mem3_shards by using ets:insert_new and canceling their writer if > that returns false. > PR incoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (COUCHDB-3379) Fix couch_auth_cache reinitialization logic
[ https://issues.apache.org/jira/browse/COUCHDB-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978949#comment-15978949 ] ASF subversion and git services commented on COUCHDB-3379: -- Commit 3f0f806e903ac334a768210a9ac4a3c7947956c7 in couchdb's branch refs/heads/COUCHDB-3376-fix-mem3-shards from [~paul.joseph.davis] [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=3f0f806 ] Merge pull request #481 from apache/COUCHDB-3379-fix-couch-auth-cache-reinit Fix couch_auth_cache reinitialization logic > Fix couch_auth_cache reinitialization logic > --- > > Key: COUCHDB-3379 > URL: https://issues.apache.org/jira/browse/COUCHDB-3379 > Project: CouchDB > Issue Type: Bug > Components: Database Core >Reporter: Paul Joseph Davis > > The reinitialization logic is subtle and quite silly in hindsight. This > reacted badly with the PSE work that has a slight change to the order of > signals (which nothing should be relying on in an async system :). This > simplifies and fixes the reinitialization of couch_auth_cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (COUCHDB-3380) Fix mem3_sync_event_listener unit tests
[ https://issues.apache.org/jira/browse/COUCHDB-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978948#comment-15978948 ] ASF subversion and git services commented on COUCHDB-3380: -- Commit e367575c75dbf9ca5c325c146336b63223b5bb27 in couchdb's branch refs/heads/COUCHDB-3376-fix-mem3-shards from [~paul.joseph.davis] [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=e367575 ] Merge pull request #482 from apache/COUCHDB-3380-fix-mem3-sync-event-listener-unit-tests Fix mem3_sync_event_listener unit test > Fix mem3_sync_event_listener unit tests > --- > > Key: COUCHDB-3380 > URL: https://issues.apache.org/jira/browse/COUCHDB-3380 > Project: CouchDB > Issue Type: Bug > Components: Database Core >Reporter: Paul Joseph Davis > > The tests in mem3_sync_event_listener get skipped because of meck issues but > if you run the mem3 eunit tests directly (i.e., make eunit apps=mem3) you'll > see this failure. The change is pretty trivial. Just a matter of this test > never having run in CI because reasons. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (COUCHDB-3378) Fix mango full text detection
[ https://issues.apache.org/jira/browse/COUCHDB-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978951#comment-15978951 ] ASF subversion and git services commented on COUCHDB-3378: -- Commit db2f19b8b40c12420070612acc3ed7faed45b819 in couchdb's branch refs/heads/COUCHDB-3376-fix-mem3-shards from [~paul.joseph.davis] [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=db2f19b ] Fix mango full text detection The check for whether full text support in mango is available was broken in that the mango_cursor_text module is only compiled if dreyfus exists while the check to use the module was a runtime check. Thus is a user were to add dreyfus after mango had been compiled it would have led to a runtime error when we attempted to use a module that didn't exist. This changes the check to a compile time check in both places. And now that its a compile time define we can remove the need to rename source files around which causes problems with source control seeing that file change depending on local configuration. COUCHDB-3378 > Fix mango full text detection > - > > Key: COUCHDB-3378 > URL: https://issues.apache.org/jira/browse/COUCHDB-3378 > Project: CouchDB > Issue Type: Bug > Components: Mango >Reporter: Paul Joseph Davis > > The renaming of source files for mango's full text adapter was not super > awesome. So I fixed it to not do that. PR incoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (COUCHDB-3100) require_valid_user is not working
[ https://issues.apache.org/jira/browse/COUCHDB-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978946#comment-15978946 ] ASF subversion and git services commented on COUCHDB-3100: -- Commit a9dd4c4cc86634e4bd9c0ec7d947c12e467d1c7f in couchdb's branch refs/heads/COUCHDB-3376-fix-mem3-shards from [~wohali] [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=a9dd4c4 ] Merge pull request #478 from apache/COUCHDB-3100 Clarify & correct require_valid_user setting > require_valid_user is not working > - > > Key: COUCHDB-3100 > URL: https://issues.apache.org/jira/browse/COUCHDB-3100 > Project: CouchDB > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Tiago Pereira >Assignee: Joan Touzet > Fix For: 2.1.0 > > > When the configuration "require_valid_user = true" is added to the local.ini, > the server ignores it and the database is still kept public. This problem was > replicated in klaemo's docker image 2.0-single and 2.0-rc3 . -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (COUCHDB-3379) Fix couch_auth_cache reinitialization logic
[ https://issues.apache.org/jira/browse/COUCHDB-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978947#comment-15978947 ] ASF subversion and git services commented on COUCHDB-3379: -- Commit d754c8c7f7d43e6d526f800d329704e906f793f0 in couchdb's branch refs/heads/COUCHDB-3376-fix-mem3-shards from [~paul.joseph.davis] [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=d754c8c ] Fix couch_auth_cache reinitialization logic The couch_auth_cache process was attempting to track its various monitors and then based on its monitor history decide if a 'DOWN' message meant it should reinitialize or exit. With the new pluggable storage engine work there was a subtle change to monitors which ended up causing couch_auth_cache to receive a monitor from the file descriptor before it received the monitor from the database's main pid. This reordering was enough for couch_auth_cache to exit which ended up causing errors in the test suite with race conditions when the test tried to open an auth db before couch_auth_cache had restarted and created the database. However, if we step back and think harder, we'll never get a monitor message we didn't ask for so rather than attempt to track specific monitors we just reinitialize the cache everytime we see a 'DOWN' message. On the downside this means that we're clearing the cache multiple times per database exit, however normal operations won't have this issue since that database never changes and clearing the cache twice in the test suite is a non issue. COUCHDB-3379 > Fix couch_auth_cache reinitialization logic > --- > > Key: COUCHDB-3379 > URL: https://issues.apache.org/jira/browse/COUCHDB-3379 > Project: CouchDB > Issue Type: Bug > Components: Database Core >Reporter: Paul Joseph Davis > > The reinitialization logic is subtle and quite silly in hindsight. This > reacted badly with the PSE work that has a slight change to the order of > signals (which nothing should be relying on in an async system :). This > simplifies and fixes the reinitialization of couch_auth_cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (COUCHDB-3376) Fix mem3_shards under load
[ https://issues.apache.org/jira/browse/COUCHDB-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978953#comment-15978953 ] ASF subversion and git services commented on COUCHDB-3376: -- Commit a4b57a9e5d5b7a7147540a4d795eeca201fa4a1d in couchdb's branch refs/heads/COUCHDB-3376-fix-mem3-shards from [~vatamane] [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=a4b57a9 ] Add unit tests for mem3_shards COUCHDB-3376 > Fix mem3_shards under load > -- > > Key: COUCHDB-3376 > URL: https://issues.apache.org/jira/browse/COUCHDB-3376 > Project: CouchDB > Issue Type: Bug >Reporter: Paul Joseph Davis > > There were two issues with mem3_shards that were fixed while I've been > testing the PSE code. > The first issue was found by [~jaydoane] where a database can have its shards > inserted into the cache after its been deleted. This can happen if a client > does a rapid CREATE/DELETE/GET cycle on a database. The fix for this is to > track the changes feed update sequence from the changes feed listener and > only insert shard maps that come from a client that has read as recent of an > update_seq as mem3_shards. > The second issue found during heavy benchmarking was that large shard maps > (in the Q>=128 range) can quite easily cause mem3_shards to backup when > there's a thundering herd attempting to open the database. There's no > coordination among workers trying to add a shard map to the cache so if a > bunch of independent clients all send the shard map at once (say, at the > beginning of a benchmark) then mem3_shards can get overwhelmed. The fix for > this was two fold. First, rather than send the shard map directly to > mem3_shards, we copy it into a spawned process and when/if mem3_shards wants > to write it, it tells this writer process to do its business. The second > optimization for this change is to create an ets table to track these > processes. Then independent clients can check if a shard map is already > enroute to mem3_shards by using ets:insert_new and canceling their writer if > that returns false. > PR incoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] cboden opened a new issue #41: Greedy URL parser
cboden opened a new issue #41: Greedy URL parser
URL: https://github.com/apache/couchdb-nano/issues/41
When passing a URL to the parser (from couchdb-bootstrap) it's grabbing the
first `/` it finds and improperly returning a parsed URL when I have a `/` in
my password.
> 'http://admin:28vBiMbx3h/4=tRPdAfZ7PWrMrxq;BAa+Ru@localhost:5984'
```
config:
{ url: 'http://admin/:28vBiMbx3h',
db: '4=tRPdAfZ7PWrMrxq;BAa+Ru@localhost:5984',
defaultHeaders: { 'X-Couch-Full-Commit': 'true' } }
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] davisp commented on issue #476: Couchdb 3376 fix mem3 shards
davisp commented on issue #476: Couchdb 3376 fix mem3 shards URL: https://github.com/apache/couchdb/pull/476#issuecomment-296216602 @tonysun83 I assume you were running make check apps=mem3 or similar? That failing unit test should be fixed on master as of this commit: https://github.com/apache/couchdb/commit/c0fda54d3ed4f06f1e41db598859de23aaf0e3ac This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] davisp commented on a change in pull request #476: Couchdb 3376 fix mem3 shards
davisp commented on a change in pull request #476: Couchdb 3376 fix mem3 shards
URL: https://github.com/apache/couchdb/pull/476#discussion_r112708782
##
File path: src/mem3/src/mem3_shards.erl
##
@@ -249,6 +269,21 @@ code_change(_OldVsn, #st{}=St, _Extra) ->
%% internal functions
+start_changes_listener(SinceSeq) ->
Review comment:
Exactly right. @jaydoane noticed that if mem3_shards restarts we can end up
with two changes listeners sending updates to mem3_shards. This setup ensures
that the old one dies which is important now that we're tracking state from the
listener.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] davisp commented on a change in pull request #476: Couchdb 3376 fix mem3 shards
davisp commented on a change in pull request #476: Couchdb 3376 fix mem3 shards
URL: https://github.com/apache/couchdb/pull/476#discussion_r112708436
##
File path: src/mem3/src/mem3_shards.erl
##
@@ -210,12 +216,25 @@ handle_cast({cache_hit, DbName}, St) ->
couch_stats:increment_counter([mem3, shard_cache, hit]),
cache_hit(DbName),
{noreply, St};
-handle_cast({cache_insert, DbName, Shards}, St) ->
-couch_stats:increment_counter([mem3, shard_cache, miss]),
-{noreply, cache_free(cache_insert(St, DbName, Shards))};
+handle_cast({cache_insert, DbName, Writer, UpdateSeq}, St) ->
+NewSt = case UpdateSeq < St#st.update_seq of
Review comment:
I'll add a comment to explain that given that its caused more than one
developer to question it
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] davisp commented on a change in pull request #476: Couchdb 3376 fix mem3 shards
davisp commented on a change in pull request #476: Couchdb 3376 fix mem3 shards
URL: https://github.com/apache/couchdb/pull/476#discussion_r112708312
##
File path: src/mem3/src/mem3_shards.erl
##
@@ -210,12 +216,25 @@ handle_cast({cache_hit, DbName}, St) ->
couch_stats:increment_counter([mem3, shard_cache, hit]),
cache_hit(DbName),
{noreply, St};
-handle_cast({cache_insert, DbName, Shards}, St) ->
-couch_stats:increment_counter([mem3, shard_cache, miss]),
-{noreply, cache_free(cache_insert(St, DbName, Shards))};
+handle_cast({cache_insert, DbName, Writer, UpdateSeq}, St) ->
+NewSt = case UpdateSeq < St#st.update_seq of
Review comment:
Nope. Took me a bit of noodling over this question. In the end its easy
enough to see that it has to be < for when the dbs db isn't chaning. At steady
state if it were =< then nothing would ever be inserted into the cache.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] davisp commented on a change in pull request #476: Couchdb 3376 fix mem3 shards
davisp commented on a change in pull request #476: Couchdb 3376 fix mem3 shards URL: https://github.com/apache/couchdb/pull/476#discussion_r112708134 ## File path: src/mem3/src/mem3_shards.erl ## @@ -36,7 +37,10 @@ -define(DBS, mem3_dbs). -define(SHARDS, mem3_shards). -define(ATIMES, mem3_atimes). +-define(OPENERS, mem3_openers). -define(RELISTEN_DELAY, 5000). +-define(WRITE_TIMEOUT, 1000). +-define(WRITE_IDLE_TIMEOUT, 3). Review comment: I've been pondering that as well. I'll make them both configurable since that's easy enough. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] janl commented on issue #476: Couchdb 3376 fix mem3 shards
janl commented on issue #476: Couchdb 3376 fix mem3 shards URL: https://github.com/apache/couchdb/pull/476#issuecomment-296198850 This is very impressive, thanks for the hard work and diligent explanation! +1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (COUCHDB-3388) queryserver protocol violation - add_fun used incorrectly
Ondřej Novák created COUCHDB-3388:
-
Summary: queryserver protocol violation - add_fun used incorrectly
Key: COUCHDB-3388
URL: https://issues.apache.org/jira/browse/COUCHDB-3388
Project: CouchDB
Issue Type: Bug
Components: Database Core
Reporter: Ondřej Novák
In relation COUCHDB-3387, there is another serious issue with the query server
protocol violation.
Just look into the following log:
{noformat}
[debug] 2017-04-21T06:48:46.00Z couchdb@localhost <0.7374.0> OS
Process #Port<0.7565> Input :: ["reset",{"reduce_limit":true,"timeout":5000}]
[debug] 2017-04-21T06:48:46.001000Z couchdb@localhost <0.7374.0> OS
Process #Port<0.7565> Output :: true
[debug] 2017-04-21T06:48:46.001000Z couchdb@localhost <0.7374.0> OS
Process #Port<0.7565> Input :: ["add_fun","function(doc)
{emit(doc._id,doc._rev);}"]
[debug] 2017-04-21T06:48:46.003000Z couchdb@localhost <0.7374.0> OS
Process #Port<0.7565> Output :: true
[debug] 2017-04-21T06:48:46.004000Z couchdb@localhost <0.7374.0> OS
Process #Port<0.7565> Input :: ["add_fun","function(keys,values,rereduce) {if
(rereduce) return sum(values); else return values.length;}"]
[debug] 2017-04-21T06:48:46.007000Z couchdb@localhost <0.7374.0> OS
Process #Port<0.7565> Output :: true
{noformat}
As you can see, the third input line sends *reduce function* to *add_fun*.
This is against the rules. According to the documentation:
(http://docs.couchdb.org/en/2.0.0/query-server/protocol.html#add-fun), the
rgument of this command is: *Map function source code.*.
Why it is called with a *reduce function* ?
I can guess the reason. The couchDb core uses queryserver's *add_fun* to check,
whether the string contains a valid function. This is not ordinary bug in code,
this is very bad decision which may result to a lot of confusions.
I suggest to use the "ddoc/new" command instead the "add_fun"
(http://docs.couchdb.org/en/2.0.0/query-server/protocol.html#ddoc) to check
whether the design document is valid. The query server can eventually validate
every function in it including the shared code ("lib" section).
This is the serious bug for me now. Fixing this bug also resolves COUCHDB-3387
issue. However, leaving this unfixed prevent me to upgrade to the CouchDB 2.0.
Ondřej Novak
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (COUCHDB-3331) ETag not returned for GET against view
[
https://issues.apache.org/jira/browse/COUCHDB-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978272#comment-15978272
]
Ondřej Novák commented on COUCHDB-3331:
---
Unfortunately this is WAI. CouchDB 2.0 no longer support ETags for views. I
hope they find solution for this asap, because it degrades caching complete. As
workaround you can use list function to emulate this feature since the list
function can control what appears in the respone.
(And because it can be emulated, it should be easy to resolve this issue in
future version)
Ondrej Novak (I am not developer)
> ETag not returned for GET against view
> --
>
> Key: COUCHDB-3331
> URL: https://issues.apache.org/jira/browse/COUCHDB-3331
> Project: CouchDB
> Issue Type: Bug
>Reporter: Daniel Wertheim
>
> Upgrading client library to be compatible with 2.0. When running tests it
> fails since I no longer get {{ETag}} in the response headers when performing
> {{GET}} against a view index.
> Docs says it should:
> http://docs.couchdb.org/en/2.0.0/api/ddoc/views.html?highlight=etag
> Map function:
> {code}
> function(doc) {
> if (doc.$doctype !== 'artist') return;
> emit(doc.name, null);
> }
> {code}
> Request:
> {code}
> GET /mycouchtests_pri/_design/artists/_view/name_no_value?include_docs=true
> HTTP/1.1
> Host: dev01:5984
> Authorization: Basic ZGV2OjFxMnczZTRy
> Cache-Control: no-cache
> {code}
> Response headers:
> {code}
> Cache-Control →must-revalidate
> Content-Type →application/json
> Date →Sun, 19 Mar 2017 09:32:03 GMT
> Server →CouchDB/2.0.0 (Erlang OTP/17)
> Transfer-Encoding →chunked
> X-Couch-Request-ID →61ce66d480
> X-CouchDB-Body-Time →0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (COUCHDB-3387) add_fun without add_lib (query server protocol)
Ondřej Novák created COUCHDB-3387:
-
Summary: add_fun without add_lib (query server protocol)
Key: COUCHDB-3387
URL: https://issues.apache.org/jira/browse/COUCHDB-3387
Project: CouchDB
Issue Type: Bug
Components: Database Core
Reporter: Ondřej Novák
CouchDB 2.0 violates some query server protocol's rules. It is possible to
receive "add_fun" without preceding "add_lib". This doesn't happen in CouchDB
1.6
Design document (just example)
{noformat}
{
"_id": "_design/test",
"_rev": "2-e4b55d8d82fe023421624901deb9cf17",
"views": {
"lib": {
"utils": "exports.MAGIC = 42;"
},
"test": {
"map": "function(doc) {emit(doc._id,doc._rev);}",
"reduce": "function(keys,values,rereduce) {if (rereduce) return
sum(values); else return values.length;}"
}
}
}
{noformat}
Recorded log:
{noformat}
[debug] 2017-04-21T06:48:46.00Z couchdb@localhost <0.7374.0> OS
Process #Port<0.7565> Input :: ["reset",{"reduce_limit":true,"timeout":5000}]
[debug] 2017-04-21T06:48:46.001000Z couchdb@localhost <0.7374.0> OS
Process #Port<0.7565> Output :: true
[debug] 2017-04-21T06:48:46.001000Z couchdb@localhost <0.7374.0> OS
Process #Port<0.7565> Input :: ["add_fun","function(doc)
{emit(doc._id,doc._rev);}"]
[debug] 2017-04-21T06:48:46.003000Z couchdb@localhost <0.7374.0> OS
Process #Port<0.7565> Output :: true
[debug] 2017-04-21T06:48:46.004000Z couchdb@localhost <0.7374.0> OS
Process #Port<0.7565> Input :: ["add_fun","function(keys,values,rereduce) {if
(rereduce) return sum(values); else return values.length;}"]
[debug] 2017-04-21T06:48:46.007000Z couchdb@localhost <0.7374.0> OS
Process #Port<0.7565> Output :: true
{noformat}
However, later
{noformat}
[debug] 2017-04-21T06:49:01.525000Z couchdb@localhost <0.7372.0> OS
Process #Port<0.7564> Input :: ["reset",{"reduce_limit":true,"timeout":5000}]
[debug] 2017-04-21T06:49:01.525000Z couchdb@localhost <0.7372.0> OS
Process #Port<0.7564> Output :: true
[debug] 2017-04-21T06:49:01.525000Z couchdb@localhost <0.7372.0> OS
Process #Port<0.7564> Input :: ["add_lib",{"utils":"exports.MAGIC = 42;"}]
[debug] 2017-04-21T06:49:01.526000Z couchdb@localhost <0.7372.0> OS
Process #Port<0.7564> Output :: true
[debug] 2017-04-21T06:49:01.526000Z couchdb@localhost <0.7372.0> OS
Process #Port<0.7564> Input :: ["add_fun","function(doc)
{emit(doc._id,doc._rev);}"]
[debug] 2017-04-21T06:49:01.527000Z couchdb@localhost <0.7372.0> OS
Process #Port<0.7564> Output :: true
{noformat}
This behaviour is OK for javascript, but it leads to serious issues in custom
query servers. If the query server uses benefit of add_fun to prepare and
possibly to compile the function, The compiler require presence of add_lib to
complete the compilation of the code. Otherwise, the compiler has no way to
detect, whether add_lib is missing due this behaviour or due error in design
document.
This happen during PUT of the design document.
Ondrej Novak
PS: Working on couchcpp project - C++ query server protocil,
https://github.com/ondra-novak/couchcpp
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
