[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
, +ok = couch_db:close(Db), +scan_local_db(self(), <<"some_other_db">>), +receive + {'$gen_cast', Msg} -> +?assertNotEqual(Msg, {resume_scan, LocalDb}) +after 0 -> +?assert(true

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
epDoc) -> +% Parsing replication doc (but not calculating the id) could throw an +% exception which would indicate this document is malformed. This exception +% should propagate to db_change function and will be recorded as permanent +% failure in the document. User will hav

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
t;Replication id changed: ~p -> ~p", [ +OldRepId, RepId]), +Row0#rdoc{rid = RepId, info = couch_util:to_binary(Msg)}; +#rdoc{rid = nil} -> +% Calculated new replication id for non-filtered replication. +% Remove replication doc body, after th

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
a replication. Returns a process monitor reference. The worker is +% guaranteed to exit with rep_start_result() type only. +-spec maybe_start_worker(db_doc_id()) -> ok. +maybe_start_worker(Id) -> +case ets:lookup(?MODULE, Id) of +[] -> +ok; +[#rdoc{state = scheduled, fil

[GitHub] millayr opened a new pull request #896: Robustify Replication Unit Tests

2017-04-10 Thread git
nored. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
gt; +ok +end. + + +-spec maybe_update_doc_triggered(#rep{}, rep_id()) -> ok. +maybe_update_doc_triggered(Rep, RepId) -> +case update_docs() of +true -> +couch_replicator_docs:update_triggered(Rep, RepId); +false -> +ok +end. + + +-spec error_backoff(non_neg_integer()) -> seconds(). +error_backoff(ErrCnt) -> +Exp = min(ErrCnt, ?ERROR_MAX_BACKOFF_EXPONENT), +random:uniform(64 bsl Exp). Review comment: The math here could use a comment. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
t; +ok +end. + + +-spec maybe_update_doc_triggered(#rep{}, rep_id()) -> ok. +maybe_update_doc_triggered(Rep, RepId) -> +case update_docs() of +true -> +couch_replicator_docs:update_triggered(Rep, RepId); +false -> +ok +end.

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
is is a terminal state and it will not +% be tried again, so skip over it. +t_failed_change() -> +?_test(begin +?assertEqual(ok, process_change(?DB, change(<<"failed">>))), +?assert(did_not_remove_state_fields()), +?assertNot(ets:member(?MODU

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
ps1), +?assertMatch({start_time, BinVal2} when is_binary(BinVal2), StartTime), +ExpectedProps = [ +{database, ?DB}, +{doc_id, ?DOC1}, +{error_count, 0}, +{id, null}, +{info, null}, + {node, node()}, +{state, initializing} +], +?assertEqual(ExpectedProps, lists:usort(DocProps2)) +end). + + +% Check that when cluster membership changes records from doc processor and job +% scheduler get removed +t_cluster_membership_foldl() -> + ?_test(begin +mock_existing_jobs_lookup([test_rep(?R1)]), +?assertEqual(ok, process_change(?DB, change())), +meck:expect(couch_replicator_clustering, owner, 2, different_node), +?assert(ets:member(?MODULE, {?DB, ?DOC1})), +gen_server:cast(?MODULE, {cluster, stable}), +timer:sleep(100), Review comment: Rather than sleeping you could use meck:wait on `ets:delete/2` or `couch_replicator_scheduler:remove_job/1` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
y after +% being added during start. +% +% The end of the consecutive crashes ends when a crashed event is seen with +% the time delta between previous greater then the threshold. +-spec consecutive_crashes(history(), non_neg_integer()) -> non_neg_integer(). +consecutive_crashes(History, Healt

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
on id, then +% start a replication. Returns a process monitor reference. The worker is +% guaranteed to exit with rep_start_result() type only. +-spec maybe_start_worker(db_doc_id()) -> ok. +maybe_start_worker(Id) -> +case ets:lookup(?MODULE, Id) of +[] -> +ok; +[#rdoc{state = scheduled, fil

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
eshold) when is_list(History) -> +consecutive_crashes(History, HealthThreshold, 0). + + +-spec consecutive_crashes(history(), non_neg_integer(), non_neg_integer()) -> + non_neg_integer(). +consecutive_crashes([], _HealthThreashold, Count) -> +Count; + +consecutive_crashes([{{crashed, _}, CrashT}, {_, PrevT} = PrevEvent | Rest], +HealthThreshold, Count) -> +case timer:now_diff(CrashT, PrevT) > HealthThreshold * 100 of Review comment: Finally stumbled across what HealthThreshold means. Not sure where a good place would be to comment on its purpose but seems like somewhere would be good. Maybe on couch_replicator_scheduler:healt_threshold/0 and/or in default.ini This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
nt + 1) +end; + +consecutive_crashes([{stopped, _}, {started, _} | _], _HealthThreshold, +Count) -> +Count; + +consecutive_crashes([_ | Rest], HealthThreshold, Count) -> +consecutive_crashes(Rest, HealthThreshold, Count). + + +-spec latest_crash_timestamp(history()) -> er

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
ec rotate_jobs(#state{}, non_neg_integer(), non_neg_integer()) -> ok. +rotate_jobs(State, Running, Pending) -> +#state{max_jobs=MaxJobs, max_churn=MaxChurn} = State, +if Running == MaxJobs, Pending > 0 -> +RotateCount = lists:min([Pending, Running, MaxChurn]), +Stop

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
ern. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache

[GitHub] davisp commented on issue #470: Scheduling Replicator

2017-04-10 Thread git
ing though. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
eck:expect(couch_replicator_clustering, owner, 2, different_node), +?assertEqual(ok, process_change(?DB, change())), +?assert(did_not_spawn_worker()) + end). + + +% Change handled when cluster is unstable (nodes are added or removed), so +% job is not added. A rescan will be triggered soon and change will be

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
a replication. Returns a process monitor reference. The worker is +% guaranteed to exit with rep_start_result() type only. +-spec maybe_start_worker(db_doc_id()) -> ok. +maybe_start_worker(Id) -> +case ets:lookup(?MODULE, Id) of +[] -> +ok; +[#rdoc{state = scheduled,

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
g_jobs_lookup([]), +?assertEqual(ok, process_change(?DB, change(<<"error">>))), +?assert(removed_state_fields()), +?assert(ets:member(?MODULE, {?DB, ?DOC1})), +?assert(started_worker({?DB, ?DOC1})) +end). + + +% Change is in `failed` stat

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
eck:expect(couch_replicator_clustering, owner, 2, different_node), +?assertEqual(ok, process_change(?DB, change())), +?assert(did_not_spawn_worker()) + end). + + +% Change handled when cluster is unstable (nodes are added or removed), so +% job is not added. A rescan will be triggered soon and change will be

[GitHub] nickva commented on issue #470: Scheduling Replicator

2017-04-10 Thread git
mated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] nickva commented on issue #470: Scheduling Replicator

2017-04-10 Thread git
n to add more? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] nickva commented on issue #470: Scheduling Replicator

2017-04-10 Thread git
mated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
ately after +% being added during start. +% +% The end of the consecutive crashes ends when a crashed event is seen with +% the time delta between previous greater then the threshold. +-spec consecutive_crashes(history(), non_neg_integer()) -> non_neg_integer(). +consecutive_crashes(History, HealthThr

[GitHub] millayr opened a new pull request #897: Prevent navbar link underlines

2017-04-10 Thread git
events the temporary underline. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, plea

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
pattern. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Ap

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
ing_jobs(State, running_job_count(), pending_job_count()). + + +start_pending_jobs(State, Running, Pending) -> +#state{max_jobs=MaxJobs} = State, +if Running < MaxJobs, Pending > 0 -> +start_jobs(MaxJobs - Running, State); +true -> +ok +end. + + +-spec ro

[GitHub] millayr closed pull request #897: Prevent navbar link underlines

2017-04-10 Thread git
millayr closed pull request #897: Prevent navbar link underlines URL: https://github.com/apache/couchdb-fauxton/pull/897 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] millayr opened a new pull request #897: Prevent navbar link underlines

2017-04-10 Thread git
events the temporary underline. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, plea

[GitHub] justin-mcdavid-ibm commented on issue #897: Prevent navbar link underlines

2017-04-10 Thread git
justin-mcdavid-ibm commented on issue #897: Prevent navbar link underlines URL: https://github.com/apache/couchdb-fauxton/pull/897#issuecomment-293066170 +1 This is an automated message from the Apache Git Service. To

[GitHub] davisp commented on issue #470: Scheduling Replicator

2017-04-10 Thread git
to the module in case we ever move the module somewhere and make it more general to be re-used in other places cause it looks rather useful. This is an automated message from the Apache Git Service. To respond to the message

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
Shouldn't this be couch_replicator_multidb_changes? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about thi

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
th rep_start_result() type only. +-spec maybe_start_worker(db_doc_id()) -> ok. +maybe_start_worker(Id) -> +case ets:lookup(?MODULE, Id) of +[] -> +ok; +[#rdoc{state = scheduled, filter = Filter}] when Filter =/= user -> +ok; +[#rdoc{rep = Rep} = Doc]

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
replicator_scheduler:find_jobs_by_dbname(DbName), +lists:foreach(fun couch_replicator_scheduler:remove_job/1, RepIds). + + +% Spawn a worker process which will attempt to calculate a replication id, then +% start a replication. Returns a process monitor reference. The worker is +% guaranteed

[GitHub] tonysun83 commented on a change in pull request #471: Use efficient set storage for field names

2017-04-10 Thread git
build the return value. Do you think it's more efficient to build the gb_set without the value rather than just converting the whole thing? -------- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
t would be a generic module, in other words I didn't hard code suffix in there. o technically could monitor any suffix-ed set of local shards. This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
urs to me to ask why this isn't in couch_replicator_httpd.erl ---- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact In

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
t would be a generic module, in other words I didn't hard code suffix in there. o technically could monitor any suffix-ed set of local shards. This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
er to chttpd. The interesting thing, it doesn't seems like couch_replicator_httpd is used? Maybe I am looking in the wrong place. -------- This is an automated message from the Apache Git Service. To respond to the message, please log on Gi

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
er to chttpd. The interesting thing, it doesn't seems like couch_replicator_httpd is used? Maybe I am looking in the wrong place. -------- This is an automated message from the Apache Git Service. To respond to the message, please log on Gi

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
------ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] davisp commented on issue #470: Scheduling Replicator

2017-04-10 Thread git
quite a lot but I'll also look at coverage and consider any tests that should be added for anything that's lacking significant coverage and is non-trivial. Awesome work on this everyone! ---- This is an automated messa

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-10 Thread git
th rep_start_result() type only. +-spec maybe_start_worker(db_doc_id()) -> ok. +maybe_start_worker(Id) -> +case ets:lookup(?MODULE, Id) of +[] -> +ok; +[#rdoc{state = scheduled, filter = Filter}] when Filter =/= user -> +ok; +[#rdoc{rep = Rep} = Doc] ->

[GitHub] nickva commented on issue #470: Scheduling Replicator

2017-04-10 Thread git
This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] rnewson closed pull request #896: Robustify Replication Unit Tests

2017-04-11 Thread git
rnewson closed pull request #896: Robustify Replication Unit Tests URL: https://github.com/apache/couchdb-fauxton/pull/896 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] rnewson closed pull request #897: Prevent navbar link underlines

2017-04-11 Thread git
rnewson closed pull request #897: Prevent navbar link underlines URL: https://github.com/apache/couchdb-fauxton/pull/897 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] iilyak commented on a change in pull request #474: 3364 fix view compactor unknown info

2017-04-11 Thread git
on}), Review comment: I think the following would be better. Since it would allow us to hide the internal representation. `{ok, IdxState} = couch_index:handle_failure(Pid, Reason)` This is an automated message from the A

[GitHub] iilyak commented on a change in pull request #474: 3364 fix view compactor unknown info

2017-04-11 Thread git
re. I think it is acceptable behavior though since it is currently failing anyway. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the sp

[GitHub] eiri commented on a change in pull request #474: 3364 fix view compactor unknown info

2017-04-11 Thread git
ctor's flow, we want public API on triggering compaction, but not on interrupting its execution except from compaction process. This is an automated message from the Apache Git Service. To respond to the message, please log on Gi

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-11 Thread git
call to job_summary can fail with `badarg` if `ets` table doesn't exists. Which violates the spec. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-11 Thread git
_neg_integer()) -> [_] | nil. +job_summary(JobId, HealthThreshold) -> Review comment: It's called from doc processor. Doc processor is started later in the rest-for-one-supervisor https://github.com/apache/couchdb/pull/470/commits/9c22dc3276e5c46c9cc10347c57668b9f596e160 that guaran

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-11 Thread git
t: Is there a possibility it would receive shard file name instead of a shard (i.e. `shards//.couch`)? If so it is better to use `couch_db:normalize_dbname/1` -------- This is an automated message from the Apache Git Service. To resp

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-11 Thread git
"})`. ---- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-11 Thread git
ew comment: It looks like this part already merged into chttpd. ---- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-11 Thread git
+StrStates = [string:to_lower(S) || S <- string:tokens(States, ",")], +AtomStates = try +[list_to_existing_atom(S) || S <- StrStates] Review comment: I don't like this but it is already included in chttpd. ------

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-11 Thread git
t; Review comment: could be a local _replicator db -------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-11 Thread git
t; Review comment: could be a local _replicator db -------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-11 Thread git
quot;: "Unknown states [not_a_list]. Choose from: [initializing,error,running,pending,crashing,completed,failed]" } ``` ---- This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-11 Thread git
quot;: "Unknown states [not_a_list]. Choose from: [initializing,error,running,pending,crashing,completed,failed]" } ``` ---- This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-11 Thread git
ew comment: It looks like this part already merged into chttpd. ---- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-11 Thread git
+StrStates = [string:to_lower(S) || S <- string:tokens(States, ",")], +AtomStates = try +[list_to_existing_atom(S) || S <- StrStates] Review comment: I don't like this but it is already included in chttpd. ------

[GitHub] jaydoane commented on a change in pull request #474: 3364 fix view compactor unknown info

2017-04-11 Thread git
, Reason}), Review comment: Since there seems to be a disagreement, should we try to find a tiebreaker? In any case, I'm unable to merge this PR, so a committer will need to do the honors. This is an automated messa

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-11 Thread git
7;s designed to crash quicker before we even get to build sets and such from input strings. -------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to t

[GitHub] eiri commented on a change in pull request #474: 3364 fix view compactor unknown info

2017-04-12 Thread git
ndex works more challenging for future curious reader. ---- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastruc

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-12 Thread git
rops} = EJson -> +{state, DocState} = lists:keyfind(state, 1, Props), +case ejson_doc_state_filter(DocState, States) of + true -> +[EJson | Acc]; +false -> +Acc +

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-12 Thread git
couch_log:error(Msg, [DDoc, View, Db, Reason]), +couch_replicator_docs:ensure_cluster_rep_ddoc_exists(Db), +timer:sleep(1000), Review comment: Use `-define(` to clarify the meaning of a magic number. ---- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-12 Thread git
t; [{[_]}] | []. +docs(States) -> +HealthThreshold = couch_replicator_scheduler:health_threshold(), +ets:foldl(fun(RDoc, Acc) -> +case ejson_doc(RDoc, HealthThreshold) of +nil -> +Acc; % Could have been deleted if job just completed +

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-12 Thread git
e end. ``` Feel free to ignore since it is cosmetic. -------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries a

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-12 Thread git
lt;<"deleted">>, Change, false). ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-12 Thread git
uted but let's keep it the way it is for now. ---- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] davisp commented on a change in pull request #470: Scheduling Replicator

2017-04-12 Thread git
: Though this should be list_to_existing_atom, no? -------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-12 Thread git
is_deleted(Change) -> -case couch_util:get_value(<<"deleted">>, Change) of +case get_value(<<"deleted">>, Change) of Review comment: Good idea ---- This is an automated message

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-12 Thread git
ou could use ``` [HD | TL] = string:tokens("foo+bar+baz", "+"), {HD, string:join(TL, "+")}. ``` There are other options as well. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-12 Thread git
plicator manager. I'd be worried about changing it. Specifically worried about the general case of calling `get_json_value(<<"some_binary_key">>, ...)` then blowing up when key is not found and we try to translate it to an existing atom ------

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-12 Thread git
worried about changing it. Specifically since it deals with replication ID transformation. Afraid it might impact checkpoints and somehow lead to re-winding of replications. This is an automated message from the Apache Git Service. To re

[GitHub] jaydoane commented on a change in pull request #474: 3364 fix view compactor unknown info

2017-04-12 Thread git
, Reason}), Review comment: Thanks for the detailed explanation, Eric. Makes sense to me to leave it as you wrote it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and u

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-12 Thread git
n_url(Rep#rep.target), +History = lists:map(fun({Type, When}) -> +EventProps = case Type of +{crashed, Reason} -> +[{type, crashed}, {reason, crash_reason_json(Reason)}]; +Type -> +[{type, Type}] +end,

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-12 Thread git
+ok = gen_server:cast(?MODULE, {set_max_jobs, list_to_integer(V)}), +{ok, S}; + +handle_config_change("replicator", "max_churn", V, _, S) -> +ok = gen_server:cast(?MODULE, {set_max_churn, list_to_integer(V)}), +{ok, S}; + +handle_config_change("replicat

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-12 Thread git
ok = gen_server:cast(?MODULE, {set_max_jobs, list_to_integer(V)}), +{ok, S}; + +handle_config_change("replicator", "max_churn", V, _, S) -> +ok = gen_server:cast(?MODULE, {set_max_churn, list_to_integer(V)}), +{ok, S}; + +handle_config_change("replicator&q

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-12 Thread git
l(Rep#rep.source), +Target = ejson_url(Rep#rep.target), +History = lists:map(fun({Type, When}) -> + EventProps = case Type of +{crashed, Reason} -> +[{type, crashed}, {reason, crash_reason_json(Reason)}]; +Type -> +[{type, Type}] +end, +

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-13 Thread git
query_acc()) -> {ok, query_acc()}; +({error, any()}, query_acc()) -> {error, any()}; +({meta, any()}, query_acc()) -> {ok, query_acc()}; +(complete, query_acc()) -> {ok, query_acc()}. +handle_replicator_doc_query({row, Props}, {Db, Cb, UserAcc, States}) -> + DocId = g

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-13 Thread git
is +% used since the expected wait would then be 0.5 * Range so it is easier +% to see the average wait. `1 +` is used because random:uniform only +% accepts > 0 values and crashes otherwise. +Range = 1 + min(2 * (Total / 10), ?TS_DAY_SEC), +60 + random:uniform(round(Range)). + + +% Docu

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-13 Thread git
_jobs", ?DEFAULT_MAX_JOBS), +MaxChurn = config:get_integer("replicator", "max_churn", +?DEFAULT_MAX_CHURN), +MaxHistory = config:get_integer("replicator", "max_history", +?DEFAULT_MAX_HISTORY), +{ok, Timer} = timer:send_after(In

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-13 Thread git
shed(Job, Now, HealthThreshold), + gb_sets:size(Set) >= Count} of +{true, true} -> + % Job is healthy but already reached accumulated limit, so might + % have to replace one of the accumulated jobs + pending_maybe_replace(Job, Set); +{true, false} -> + % Job is healthy and we haven't reached the limit, so add job + % to accumulator + gb_sets:add_element({last_started(Job), Job}, Set); +{false, _} -> + % This jobs is not healthy (has crashed too recently), so skip it. Review comment: Typo `jobs` -> `job` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-13 Thread git
cutive_crashes([_ | Rest], HealthThreshold, Count) -> +consecutive_crashes(Rest, HealthThreshold, Count). + + +-spec latest_crash_timestamp(history()) -> erlang:timestamp(). +latest_crash_timestamp([]) -> +{0, 0, 0}; % Used to avoid special-casing "no crash" when doing now_

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-13 Thread git
CrashMsg = "Duplicate replication running on another node", +couch_log:notice("~p: Job ~p already running as ~p. Most likely" +" because a duplicate replication is running on another node", + [?MODULE, Job#job.id, OtherPid]), +ok = update_state_crashed(Job, CrashMsg, State); +{error, Reason} -> +couch_log:notice("~p: Job ~p failed to start for reason ~p", +[?MODULE, Job, Reason]), +ok = update_state_crashed(Job, Reason, State) +end. + + +-spec stop_job_int(#job{}, #state{}) -> ok | {error, term()}. +stop_job_int(#job{pid = undefined}, _State) -> +ok; + +stop_job_int(#job{} = Job, State) -> +ok = couch_replicator_scheduler_sup:terminate_child(Job#job.pid), Review comment: Do we want to crash if supervisor return an error? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] iilyak commented on a change in pull request #470: Scheduling Replicator

2017-04-13 Thread git
{max_jobs=MaxJobs, max_churn=MaxChurn} = State, +if Running == MaxJobs, Pending > 0 -> +RotateCount = lists:min([Pending, Running, MaxChurn]), +StopCount = stop_jobs(RotateCount, true, State), +start_jobs(StopCount, State); +true -> +ok +end. + + +-spec last_started(#job{}) -> erlang:timestamp(). +last_started(#job{} = Job) -> +case lists:keyfind(started, 1, Job#job.history) of +false -> +{0, 0, 0}; +{started, When} -> +When +end. + + +-spec update_history(#job{}, event_type(), erlang:timestamp(), #state{}) -> +#job{}. +update_history(Job, Type, When, State) -> +History0 = [{Type, When} | Job#job.history], +History1 = lists:sublist(History0, State#state.max_history), Review comment: This would remove very old events from the history. Which means that `started` time will be unavailable. Which also mean that `last_started` would return `{0, 0, 0}`. Would it be a problem for `stats_fold`? In my observation it shouldn't but wanted to double check. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] iilyak commented on a change in pull request #474: 3364 fix view compactor unknown info

2017-04-13 Thread git
Anyway I am fine with merging it as is. Since couchdb code base uses `gen_server:call` everywhere. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the UR

[GitHub] wqj2101 opened a new issue #40: Destroying An Undefined Doc Removes the Whole Database

2017-04-13 Thread git
ire( 'nano' )( CouchDB ); const dbName = nano.db.use( 'dbName' ); dbName.destroy( undefined, undefined, ( err ) => { **//DB is destroyed at this point** } ); This is an automated message from the

[GitHub] lornajane opened a new issue #126: Update published version of docs

2017-04-13 Thread git
from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-13 Thread git
d since the expected wait would then be 0.5 * Range so it is easier +% to see the average wait. `1 +` is used because random:uniform only +% accepts > 0 values and crashes otherwise. +Range = 1 + min(2 * (Total / 10), ?TS_DAY_SEC), +60 + random:uniform(round(Range)). + + +% Docu

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-13 Thread git
erPid}} when node(OtherPid) =/= node() -> +CrashMsg = "Duplicate replication running on another node", +couch_log:notice("~p: Job ~p already running as ~p. Most likely" +" because a duplicate replication is running on another node", +[?MODULE, Job#job.id, OtherPid]), +ok = update_state_crashed(Job, CrashMsg, State); +{error, Reason} -> +couch_log:notice("~p: Job ~p failed to start for reason ~p", +[?MODULE, Job, Reason]), +ok = update_state_crashed(Job, Reason, State) +end. + + +-spec stop_job_int(#job{}, #state{}) -> ok | {error, term()}. +stop_job_int(#job{pid = undefined}, _State) -> +ok; + +stop_job_int(#job{} = Job, State) -> +ok = couch_replicator_scheduler_sup:terminate_child(Job#job.pid), Review comment: Yes. We expect supervisor to work here otherwise a crash is acceptable. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-13 Thread git
et_integer("replicator", "max_jobs", ?DEFAULT_MAX_JOBS), +MaxChurn = config:get_integer("replicator", "max_churn", +?DEFAULT_MAX_CHURN), +MaxHistory = config:get_integer("replicator", "max_history", +?DEFAULT_MAX_HISTORY), +{ok, Timer} = timer:send_after(In

[GitHub] nickva commented on a change in pull request #470: Scheduling Replicator

2017-04-13 Thread git
tate{max_jobs=MaxJobs, max_churn=MaxChurn} = State, +if Running == MaxJobs, Pending > 0 -> +RotateCount = lists:min([Pending, Running, MaxChurn]), +StopCount = stop_jobs(RotateCount, true, State), +start_jobs(StopCount, State); +true -> +ok +end. +

[GitHub] wohali commented on issue #126: Update published version of docs

2017-04-13 Thread git
://docs.couchdb.org/en/latest/contributing.html When we release 2.1.0 the instructions will update. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the

[GitHub] wohali closed issue #126: Update published version of docs

2017-04-13 Thread git
wohali closed issue #126: Update published version of docs URL: https://github.com/apache/couchdb-documentation/issues/126 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] wohali closed issue #111: Requesting a CONTRIBUTING.md document

2017-04-13 Thread git
wohali closed issue #111: Requesting a CONTRIBUTING.md document URL: https://github.com/apache/couchdb-documentation/issues/111 This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] wohali commented on issue #122: Additions to _find and _index from original Mango docs PR

2017-04-13 Thread git
the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services

[GitHub] wohali closed pull request #122: Additions to _find and _index from original Mango docs PR

2017-04-13 Thread git
wohali closed pull request #122: Additions to _find and _index from original Mango docs PR URL: https://github.com/apache/couchdb-documentation/pull/122 This is an automated message from the Apache Git Service. To respond

[GitHub] wohali closed pull request #43: Removing _rev

2017-04-13 Thread git
wohali closed pull request #43: Removing _rev URL: https://github.com/apache/couchdb-documentation/pull/43 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use

  1   2   3   4   5   6   7   8   9   10   >