[jira] [Commented] (COUCHDB-2765) CouchDB sends 200 and invalid JSON if _changes view filter is invalid

[JIRA]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303837#comment-15303837
 ] 

Łukasz Biernot commented on COUCHDB-2765:
-

Happens also if the view is valid but throws exception on some document. In 
this case you will get changes in "results" list until a document causing 
exception is encountered. At that point the response is cut off, resulting in 
broken JSON and exactly the same message.

> CouchDB sends 200 and invalid JSON  if _changes view filter is invalid
> --
>
> Key: COUCHDB-2765
> URL: https://issues.apache.org/jira/browse/COUCHDB-2765
> Project: CouchDB
>  Issue Type: Bug
>  Components: HTTP Interface
>Reporter: Nolan Lawson
>
> If you provide a {{view}} with {{filter=_view}} in {{_changes}} and the 
> referenced {{view}} is a design doc with no {{map}} function, then the 
> database returns a 200 with invalid JSON.
> cURL to reproduce:
> {code}
> curl 'http://127.0.0.1:5984/testdb/' -X DELETE  -H 'Content-Type: 
> application/json' -H 'Accept: application/json' 
> curl 'http://127.0.0.1:5984/testdb/' -X PUT -H 'Content-Type: 
> application/json' -H 'Accept: application/json'
> curl 'http://127.0.0.1:5984/testdb/_design/name' -X PUT  -H 'Content-Type: 
> application/json' -H 'Accept: application/json' --data-binary 
> '{"_id":"_design/name","views":{"name":{"empty":"sad face"}}}' 
> curl 
> 'http://127.0.0.1:5984/testdb/_changes?timeout=25000=longpoll=_view=name%2Fname=0=25&_nonce=1438464395669'
>   -H 'Accept: application/json'
> {code}
> Result:
> {code}
> {"ok":true}
> {"ok":true}
> {"ok":true,"id":"_design/name","rev":"1-214291b1f3879ad7a5f408d3e1edb53b"}
> {"results":[
> curl: (56) Illegal or missing hexadecimal sequence in chunked-encoding
> {code}
> The final response from CouchDB is this invalid JSON:
> {code}
> {"results":[
> An error has occurred: {"bytesParsed":0,"code":"HPE_INVALID_CHUNK_SIZE"}
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] couchdb-couch pull request: Implement Mango selectors for change f...

[nickva]

Github user nickva commented on the pull request:

https://github.com/apache/couchdb-couch/pull/162#issuecomment-222195609
  
@colinskow that implies some kind of persistence. The selector passed in a 
change feed request is active only in the context of the POST request. There 
could be multiple such concurrent requests. After request is finished, there is 
no history preserved of previous request.

Or were you thinking about a long / continuous request, and while it is 
returning data back to the user,  a document which passed the filter and was 
returned before, was updated to no longer pass the selector. But there is 
nothing to tell the user about it. So perhaps have something in the change feed 
saying "this document passed the selector before but no longer does".

That is an interesting case then. If document is deleted we return it with 
`"_deleted":true`, is it reasonable to have something like 
`"_selector_failed":true`? I am not sure about this. You might want to take 
this up on the couchdb mailing list. This looks like an API change then and 
needs a better discussion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (COUCHDB-2991) userdb behaves less strict on cluster

[Sebastian Rothbucher (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304328#comment-15304328
 ] 

Sebastian Rothbucher commented on COUCHDB-2991:
---

just re-checked against master (then ./configure) the delete user doc problem 
persists (so does the 2nd one). 

> userdb behaves less strict on cluster
> -
>
> Key: COUCHDB-2991
> URL: https://issues.apache.org/jira/browse/COUCHDB-2991
> Project: CouchDB
>  Issue Type: Bug
>Reporter: Sebastian Rothbucher
>Priority: Blocker
>
> Topic #1: _design/auth does not get installed on custom users DB
> To reproduce: set both
> [{section: "couch_httpd_auth",
>   key: "authentication_db", value: usersDb.name},
>  {section: "chttpd_auth",
>   key: "authentication_db", value: usersDb.name}]
> to a usersDB that is not _users and check if the document is there (certainly 
> minor)
> Topic #2: conflicts in user docs don't seem to bother any longer
> To reproduce: produce a conflict in a user doc (the users_db.js test does 
> that also) and try to login (which works)
> Topic #3: any user can call _all_docs on _users via cluster port (not 
> backdoor)
> To reproduce: create a user doc and use it to get _users/_all_docs (once on 
> cluster, once on backdoor), you'll see the difference: cluster allows the 
> action while backdoor does not



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] couchdb-chttpd pull request: Fix _node/$node/_system endpoint

[iilyak]

Github user iilyak commented on the pull request:

https://github.com/apache/couchdb-chttpd/pull/123#issuecomment-21511
  
Other than a small nitpick it looks good to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-chttpd pull request: Fix _node/$node/_system endpoint

[iilyak]

Github user iilyak commented on a diff in the pull request:

https://github.com/apache/couchdb-chttpd/pull/123#discussion_r64948608
  
--- Diff: src/chttpd_misc.erl ---
@@ -294,8 +295,9 @@ handle_node_req(#httpd{path_parts=[_, _Node, 
<<"_stats">>]}=Req) ->
 send_method_not_allowed(Req, "GET");
 % GET /_node/$node/_system
 handle_node_req(#httpd{method='GET', path_parts=[_, Node, 
<<"_system">>]}=Req) ->
-Stats = call_node(Node, chttpd_misc, handle_system_req, [Req]),
-chttpd:send_json(Req, Stats);
+Stats = call_node(Node, chttpd_misc, get_stats, []),
+EJSON = couch_stats_httpd:to_ejson(Stats),
+chttpd:send_json(Req, EJSON);
--- End diff --

Nitpick. Here you specify module name while [on line 
341](https://github.com/apache/couchdb-chttpd/pull/123/files#diff-938aabd9888322523915339d0185cf7aR341)
 you don't specify module name. Both forms would work, but it is better to not 
mix them without a reason.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-chttpd pull request: Fix _node/$node/_system endpoint

[eiri]

GitHub user eiri opened a pull request:

https://github.com/apache/couchdb-chttpd/pull/123

Fix _node/$node/_system endpoint

The endpoint is using the same handler as a backdoor interface, which only 
works for the requests to a local $node. For rpc calls to the remote nodes, as 
it's done on `/_node` endpoints, `handle_node_req` handler is getting 
`{badrpc,{'EXIT',normal}}` response and fails with `invalid_ejson` exception.

The change splits stats gathering from stats responding, allowing to use 
former on both `/_node/$node/_system` and backend's `/_system`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cloudant/couchdb-chttpd 
65878-fix-_node-_system-endpoint

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/couchdb-chttpd/pull/123.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #123


commit f3209871eeedf9228c2ef1e35512a8128e0e8ba7
Author: Eric Avdey 
Date:   2016-05-27T17:59:23Z

Fix _node/$node/_system endpoint

The endpoint was using the same handler as a backdoor interface,
which only works for the requests to a local $node.

For rpc calls to the remote nodes, as it's done on /_node endpoints,
handle_node_req handler was getting {badrpc,{'EXIT',normal}} response
and failed with invalid_ejson exception as a result.

The change splits stats gathering from stats responding
allowing to use former on both /_node/$node/_system and
/_system on the backdoor interface.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (COUCHDB-3010) Handle 429 for replication

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-3010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304665#comment-15304665
 ] 

ASF subversion and git services commented on COUCHDB-3010:
--

Commit c6e891d26879bdaef9408196f436769e69e5e58f in couchdb-couch-replicator's 
branch refs/heads/3010-handle-429 from [~tonysun83]
[ 
https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=c6e891d
 ]

Add exponential backoff for 429 errors.

When we encounter a 429, we retry with a different set of retries and
timeout. This will theoretically reduce client replication overload.
When 429s have stopped, it's possible that a 500 error could occur.
Then the retry mechanism should go back to the original way for
backwards compatibility.

COUCHDB-3010


> Handle 429 for replication
> --
>
> Key: COUCHDB-3010
> URL: https://issues.apache.org/jira/browse/COUCHDB-3010
> Project: CouchDB
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Tony Sun
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] couchdb-couch-replicator pull request: Add Exponential Backoff for...

[nickva]

Github user nickva commented on a diff in the pull request:


https://github.com/apache/couchdb-couch-replicator/pull/40#discussion_r64961577
  
--- Diff: src/couch_replicator_httpc.erl ---
@@ -138,6 +139,8 @@ process_response({ibrowse_req_id, ReqId}, Worker, 
HttpDb, Params, Callback) ->
 
 process_response({ok, Code, Headers, Body}, Worker, HttpDb, Params, 
Callback) ->
 case list_to_integer(Code) of
+C when C =:= 429 ->
--- End diff --

`429 ->`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator pull request: Add New Rate Limiter For Re...

[tonysun83]

Github user tonysun83 commented on the pull request:


https://github.com/apache/couchdb-couch-replicator/pull/38#issuecomment-39121
  
Closing this PR since it's overly complicated for what is needed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator pull request: Add New Rate Limiter For Re...

[tonysun83]

Github user tonysun83 closed the pull request at:

https://github.com/apache/couchdb-couch-replicator/pull/38


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator pull request: Add Exponential Backoff for...

[tonysun83]

GitHub user tonysun83 opened a pull request:

https://github.com/apache/couchdb-couch-replicator/pull/40

Add Exponential Backoff for 429 errors.

When we encounter a 429, we retry with a different set of retries and
timeout. This will theoretically reduce client replication overload.
When 429s have stopped, it's possible that a 500 error could occur.
Then the retry mechanism should go back to the original way for
backwards compatibility.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/couchdb-couch-replicator 
3010-handle-429

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/couchdb-couch-replicator/pull/40.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #40


commit c6e891d26879bdaef9408196f436769e69e5e58f
Author: Tony Sun 
Date:   2016-05-27T19:47:08Z

Add exponential backoff for 429 errors.

When we encounter a 429, we retry with a different set of retries and
timeout. This will theoretically reduce client replication overload.
When 429s have stopped, it's possible that a 500 error could occur.
Then the retry mechanism should go back to the original way for
backwards compatibility.

COUCHDB-3010




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator pull request: Add Exponential Backoff for...

[nickva]

Github user nickva commented on a diff in the pull request:


https://github.com/apache/couchdb-couch-replicator/pull/40#discussion_r64961202
  
--- Diff: src/couch_replicator_httpc.erl ---
@@ -251,18 +257,42 @@ clean_mailbox(_, Count) when Count > 0 ->
 maybe_retry(Error, Worker, #httpdb{retries = 0} = HttpDb, Params) ->
 report_error(Worker, HttpDb, Params, {error, Error});
 
+%% For 429 errors, we perform an exponential backoff up to 250 * 2^15
+%% times, or roughly 2.17 hours. Since the #httpd.retries is initialized
+%% to 10 and we need 15, we use the Wait time as a timeout/failure end.
+maybe_retry(backoff, Worker, #httpdb{wait = Wait} = HttpDb, Params) ->
+ok = timer:sleep(random:uniform(Wait)),
+Wait2 = Wait*2,
+case Wait2 of
+W0 when W0 >= 512000 -> % Past 8 min, we log retries
--- End diff --

Let's make it a macro, something like ?MAX_492_BACKOFF_MSEC


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator pull request: Add Exponential Backoff for...

[nickva]

Github user nickva commented on a diff in the pull request:


https://github.com/apache/couchdb-couch-replicator/pull/40#discussion_r64961411
  
--- Diff: src/couch_replicator_httpc.erl ---
@@ -251,18 +257,42 @@ clean_mailbox(_, Count) when Count > 0 ->
 maybe_retry(Error, Worker, #httpdb{retries = 0} = HttpDb, Params) ->
 report_error(Worker, HttpDb, Params, {error, Error});
 
+%% For 429 errors, we perform an exponential backoff up to 250 * 2^15
+%% times, or roughly 2.17 hours. Since the #httpd.retries is initialized
+%% to 10 and we need 15, we use the Wait time as a timeout/failure end.
+maybe_retry(backoff, Worker, #httpdb{wait = Wait} = HttpDb, Params) ->
+ok = timer:sleep(random:uniform(Wait)),
+Wait2 = Wait*2,
+case Wait2 of
+W0 when W0 >= 512000 -> % Past 8 min, we log retries
+log_retry_error(Params, HttpDb, Wait, "429 Retry");
+W1 when W1 > ?MAX_BACKOFF_WAIT ->
+report_error(Worker, HttpDb, Params, {error,
+"429 Retry Timeout"});
+_ ->
+NewWait = erlang:min(Wait2, ?MAX_BACKOFF_WAIT),
+NewHttpDb = HttpDb#httpdb{wait = NewWait},
+throw({retry, NewHttpDb, Params})
+end;
+
 maybe_retry(Error, _Worker, #httpdb{retries = Retries, wait = Wait} = 
HttpDb,
 Params) ->
-Method = string:to_upper(atom_to_list(get_value(method, Params, get))),
-Url = couch_util:url_strip_password(full_url(HttpDb, Params)),
-couch_log:notice("Retrying ~s request to ~s in ~p seconds due to error 
~s",
-[Method, Url, Wait / 1000, error_cause(Error)]),
-ok = timer:sleep(Wait),
+log_retry_error(Params, HttpDb, Wait, Error),
+% This is so that a long backoff time is not used to ensure
+% backwards compatibility.
+ok = timer:sleep(erlang:min(Wait, ?MAX_WAIT)),
 Wait2 = erlang:min(Wait * 2, ?MAX_WAIT),
--- End diff --

Let's randomize this as well, it avoids global synchronization (having all 
replication back-off with the same schedule).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-index pull request: Adding couch_index_plugin:before...

[iilyak]

Github user iilyak commented on the pull request:

https://github.com/apache/couchdb-couch-index/pull/11#issuecomment-222169728
  
Already implemented


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-index pull request: Adding couch_index_plugin:before...

[iilyak]

Github user iilyak closed the pull request at:

https://github.com/apache/couchdb-couch-index/pull/11


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator pull request: Add Exponential Backoff for...

[nickva]

Github user nickva commented on a diff in the pull request:


https://github.com/apache/couchdb-couch-replicator/pull/40#discussion_r64970642
  
--- Diff: src/couch_replicator_httpc.erl ---
@@ -162,6 +165,9 @@ process_stream_response(ReqId, Worker, HttpDb, Params, 
Callback) ->
 receive
 {ibrowse_async_headers, ReqId, Code, Headers} ->
 case list_to_integer(Code) of
+C when C =:= 429 ->
+maybe_retry(back_off, Worker,
+HttpDb#httpdb{timeout = ?MAX_BACKOFF_WAIT}, Params);
--- End diff --

Does this change or update the timeout for the duration of this 
replication's life-cycle.  Wonder if that has any consequence. Or say what 
happens if 429 errors will start interleaving with other errors (socket 
timeouts, or authentication ones).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---