[jira] [Commented] (COUCHDB-2975) Automatically restart replication jobs if they crash

[Nick Vatamaniuc (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211101#comment-15211101
 ] 

Nick Vatamaniuc commented on COUCHDB-2975:
--

We might have to increase intensity threshold.  One common use case that will 
trigger is one source to multiple targets replications. Source fails, So all 
replications will fail as well. Tested it with 1 source to 200 targets. Then 
killed the source and noticed supervisors were restarted:

(node1@127.0.0.1)4> rpc:multicall(erlang, whereis, [couch_replicator_job_sup]).
{[<0.352.0>,<26873.355.0>,<26910.354.0>],[]} % before deleting source
(node1@127.0.0.1)5> rpc:multicall(erlang, whereis, [couch_replicator_job_sup]).
{[<0.5617.4>,<26873.7071.3>,<26910.8924.3>],[]} % after deleting source

Saw we already have some protection again failed repeated replication re-starts 
as the “max_replication_retry_count” parameter. By default it is 10. So 10 
failed replication starts for a particular replication will cancel that 
replication. Once it successfully starts once, the failed retries number gets 
reset back to max (10).

Another thing, noticed replications will restart even without {{transient}} 
supervisors if they are killed with an exit reason other than 'kill' (brutal 
kill). So if the goal is to just restart them, sending them exit(Pid, meh) 
should suffice. 

> Automatically restart replication jobs if they crash
> 
>
> Key: COUCHDB-2975
> URL: https://issues.apache.org/jira/browse/COUCHDB-2975
> Project: CouchDB
>  Issue Type: Improvement
>  Components: Replication
>Reporter: Robert Newson
>
> We currently use the temporary restart strategy for replication jobs, which 
> means if they crash they are not restarted.
> Instead, let's use the transient restart strategy, ensuring they are 
> restarted on abnormal termination, while still allowing these tasks to end 
> successfully on completion or cancellation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] couchdb-couch pull request: Refactor rename on delete

[eiri]

Github user eiri commented on the pull request:

https://github.com/apache/couchdb-couch/pull/141#issuecomment-201037949
  
I've redone `deleted_filename` with regexps to avoid magic filters. Granted 
it still not exactly obvious piece of code, but I hope it's more clear now 
what's going on in there.

I've added edge cases to the tests and made them more explicit on how file 
hierarchy organized for db and view files.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (COUCHDB-2975) Automatically restart replication jobs if they crash

[Robert Newson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Newson resolved COUCHDB-2975.

Resolution: Fixed

> Automatically restart replication jobs if they crash
> 
>
> Key: COUCHDB-2975
> URL: https://issues.apache.org/jira/browse/COUCHDB-2975
> Project: CouchDB
>  Issue Type: Improvement
>  Components: Replication
>Reporter: Robert Newson
>
> We currently use the temporary restart strategy for replication jobs, which 
> means if they crash they are not restarted.
> Instead, let's use the transient restart strategy, ensuring they are 
> restarted on abnormal termination, while still allowing these tasks to end 
> successfully on completion or cancellation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2975) Automatically restart replication jobs if they crash

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210728#comment-15210728
 ] 

ASF subversion and git services commented on COUCHDB-2975:
--

Commit eb93044ca4aa02ab4427ec7082df37bae6602973 in couchdb-couch-replicator's 
branch refs/heads/master from [~rnewson]
[ 
https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=eb93044
 ]

Use transient restart type for all replications

We want replication tasks to be restarted automatically if they crash
abnormally. Replication tasks that complete or are cancelled (by
deleting the backing _replicator doc or issuing an "cancel":true for
non-persistent jobs) should still exit, should not be restarted, and
should not have their child spec linger in the supervisor.

COUCHDB-2975


> Automatically restart replication jobs if they crash
> 
>
> Key: COUCHDB-2975
> URL: https://issues.apache.org/jira/browse/COUCHDB-2975
> Project: CouchDB
>  Issue Type: Improvement
>  Components: Replication
>Reporter: Robert Newson
>
> We currently use the temporary restart strategy for replication jobs, which 
> means if they crash they are not restarted.
> Instead, let's use the transient restart strategy, ensuring they are 
> restarted on abnormal termination, while still allowing these tasks to end 
> successfully on completion or cancellation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2975) Automatically restart replication jobs if they crash

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210727#comment-15210727
 ] 

ASF subversion and git services commented on COUCHDB-2975:
--

Commit 73afc584bd10f68626d2049442b5a6058ff002db in couchdb-couch-replicator's 
branch refs/heads/master from [~rnewson]
[ 
https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=73afc58
 ]

Remove obsoleted R14-era code

We no longer support R14 so we're dropping R14-specific complications
in the codebase.

COUCHDB-2975


> Automatically restart replication jobs if they crash
> 
>
> Key: COUCHDB-2975
> URL: https://issues.apache.org/jira/browse/COUCHDB-2975
> Project: CouchDB
>  Issue Type: Improvement
>  Components: Replication
>Reporter: Robert Newson
>
> We currently use the temporary restart strategy for replication jobs, which 
> means if they crash they are not restarted.
> Instead, let's use the transient restart strategy, ensuring they are 
> restarted on abnormal termination, while still allowing these tasks to end 
> successfully on completion or cancellation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2975) Automatically restart replication jobs if they crash

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210730#comment-15210730
 ] 

ASF GitHub Bot commented on COUCHDB-2975:
-

Github user asfgit closed the pull request at:

https://github.com/apache/couchdb-couch-replicator/pull/33


> Automatically restart replication jobs if they crash
> 
>
> Key: COUCHDB-2975
> URL: https://issues.apache.org/jira/browse/COUCHDB-2975
> Project: CouchDB
>  Issue Type: Improvement
>  Components: Replication
>Reporter: Robert Newson
>
> We currently use the temporary restart strategy for replication jobs, which 
> means if they crash they are not restarted.
> Instead, let's use the transient restart strategy, ensuring they are 
> restarted on abnormal termination, while still allowing these tasks to end 
> successfully on completion or cancellation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2975) Automatically restart replication jobs if they crash

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210729#comment-15210729
 ] 

ASF subversion and git services commented on COUCHDB-2975:
--

Commit 4cb517659c235c06a39ee7eb6b4150cdfded6116 in couchdb-couch-replicator's 
branch refs/heads/master from [~rnewson]
[ 
https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=4cb5176
 ]

Reduce likelihood of a bad replication job taking down the job supervisor

While we can't disable max_restart_intensity, we can make it unlikely
to happen. Ordinarily, we would want this behaviour, but replication
jobs involve human input. A bad password, or malformed url, etc, can
cause repeated and fast crashing.

For now, we require ten crashes within one second before we would
bounce the job supervisor. In future, we should manage replication
jobs with greater care.

COUCHDB-2975


> Automatically restart replication jobs if they crash
> 
>
> Key: COUCHDB-2975
> URL: https://issues.apache.org/jira/browse/COUCHDB-2975
> Project: CouchDB
>  Issue Type: Improvement
>  Components: Replication
>Reporter: Robert Newson
>
> We currently use the temporary restart strategy for replication jobs, which 
> means if they crash they are not restarted.
> Instead, let's use the transient restart strategy, ensuring they are 
> restarted on abnormal termination, while still allowing these tasks to end 
> successfully on completion or cancellation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] couchdb-couch-replicator pull request: restart replications on cra...

[davisp]

Github user davisp commented on the pull request:


https://github.com/apache/couchdb-couch-replicator/pull/33#issuecomment-200961456
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-fauxton pull request: Update NW to 0.8.18; asyncHookTimeou...

[benkeen]

Github user benkeen closed the pull request at:

https://github.com/apache/couchdb-fauxton/pull/671


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-fauxton pull request: Update NW to 0.8.18; asyncHookTimeou...

[benkeen]

Github user benkeen commented on the pull request:

https://github.com/apache/couchdb-fauxton/pull/671#issuecomment-200946027
  
Thanks! Merged as adaf1d30018a5aa4d4cc69262bb2e10f4b2d9a45


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-fauxton pull request: Update NW to 0.8.18; asyncHookTimeou...

[garrensmith]

Github user garrensmith commented on the pull request:

https://github.com/apache/couchdb-fauxton/pull/671#issuecomment-200943478
  
+1 nice


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-fauxton pull request: Webpack updates

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/couchdb-fauxton/pull/673


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-fauxton pull request: remove comment

[benkeen]

Github user benkeen closed the pull request at:

https://github.com/apache/couchdb-fauxton/pull/672


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-fauxton pull request: Webpack updates

[garrensmith]

GitHub user garrensmith opened a pull request:

https://github.com/apache/couchdb-fauxton/pull/673

Webpack updates

Some small fixes

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/garrensmith/couchdb-fauxton webpack-updates

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/couchdb-fauxton/pull/673.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #673


commit 620184b65bf7b751102c09ce86e4d01e72654d86
Author: Garren Smith 
Date:   2016-03-23T12:56:54Z

remove console.log

commit 87487ec891aa9cf8bd908b1692fe47a267b09bee
Author: Garren Smith 
Date:   2016-03-23T16:53:51Z

webpack server fixes for travis




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-fauxton pull request: remove comment

[garrensmith]

Github user garrensmith commented on the pull request:

https://github.com/apache/couchdb-fauxton/pull/672#issuecomment-200940673
  
Sorry I thought I removed that. +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-fauxton pull request: remove comment - causing lots of clu...

[benkeen]

GitHub user benkeen opened a pull request:

https://github.com/apache/couchdb-fauxton/pull/672

remove comment - causing lots of clutter

This just removes a console.log that's causing a lot of noise on the 
command line when running the dev server.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/benkeen/couchdb-fauxton remove-comment

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/couchdb-fauxton/pull/672.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #672


commit f0412a13de61ed501bcdde07b574f44615d3
Author: Ben Keen 
Date:   2016-03-24T16:59:47Z

remove comment - causing lots of clutter




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb pull request: Update couch_epi to fix test suite

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/couchdb/pull/397


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: Refactor rename on delete

[eiri]

Github user eiri commented on the pull request:

https://github.com/apache/couchdb-couch/pull/141#issuecomment-200896565
  
@iilyak username goes into db name and we do validate db names. try for 
kicks `http PUT :5984/bird%23bird%2Fsparrow`. Unencoded `#` will be treated as 
URL anchor, `http PUT :5984/bird#bird%2Fsparrow` will just create db `bird`, so 
we safe on that matter.

There are no `.deleted` suffix on files moved into `.delete` directory, 
i.e. on files removed on compaction. `.deleted` added only on 'really' deleted 
files that stay in a shards directory and I think we need to keep it to 
simplify periodic cleanup, or backup, or whatever it is an alternative to 
"delete on a spot" strategy.

Anyway, I'll got with `#` as a separator. I'm finishing with less 
confusing, more extended tests here, going to push update in a couple of hours.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator pull request: restart replications on cra...

[kxepal]

Github user kxepal commented on the pull request:


https://github.com/apache/couchdb-couch-replicator/pull/33#issuecomment-200870682
  
@rnewson Question? (sorry, I'm out of IRC). Well, it's hard to say for sure 
without giving a try to find the right values under different situations. I 
hope you did some research on this already and knows the best. I'm really not 
sure about this now, sorry ):


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator pull request: restart replications on cra...

[rnewson]

Github user rnewson commented on the pull request:


https://github.com/apache/couchdb-couch-replicator/pull/33#issuecomment-200867271
  
@kxepal any thoughts on the restart intensity question?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: Refactor rename on delete

[iilyak]

Github user iilyak commented on the pull request:

https://github.com/apache/couchdb-couch/pull/141#issuecomment-200865747
  
@eiri: `#` could work. Unfortunately we don't validate user so if username 
contains `#` it would be tricky. We could do 
`couch_util:url_encode("/foo/bar")`. [Unreserved 
characters](https://tools.ietf.org/html/rfc3986#section-2.3) look portable to 
me if used as part of a file name. I.e. we would have 
`.delete/...deleted`. We don't need 
`.deleted` suffix and can get rid of it probably.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator pull request: restart replications on cra...

[kxepal]

Github user kxepal commented on the pull request:


https://github.com/apache/couchdb-couch-replicator/pull/33#issuecomment-200846727
  
LGFM, +1. Good to see R14 bits away!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (COUCHDB-2975) Automatically restart replication jobs if they crash

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210238#comment-15210238
 ] 

ASF GitHub Bot commented on COUCHDB-2975:
-

GitHub user rnewson opened a pull request:

https://github.com/apache/couchdb-couch-replicator/pull/33

restart replications on crash

COUCHDB-2975 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cloudant/couchdb-couch-replicator 
2975-restart-replications-on-crash

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/couchdb-couch-replicator/pull/33.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #33


commit 73afc584bd10f68626d2049442b5a6058ff002db
Author: Robert Newson 
Date:   2016-03-24T11:26:08Z

Remove obsoleted R14-era code

We no longer support R14 so we're dropping R14-specific complications
in the codebase.

COUCHDB-2975

commit eb93044ca4aa02ab4427ec7082df37bae6602973
Author: Robert Newson 
Date:   2016-03-24T11:29:10Z

Use transient restart type for all replications

We want replication tasks to be restarted automatically if they crash
abnormally. Replication tasks that complete or are cancelled (by
deleting the backing _replicator doc or issuing an "cancel":true for
non-persistent jobs) should still exit, should not be restarted, and
should not have their child spec linger in the supervisor.

COUCHDB-2975

commit 4cb517659c235c06a39ee7eb6b4150cdfded6116
Author: Robert Newson 
Date:   2016-03-24T13:40:14Z

Reduce likelihood of a bad replication job taking down the job supervisor

While we can't disable max_restart_intensity, we can make it unlikely
to happen. Ordinarily, we would want this behaviour, but replication
jobs involve human input. A bad password, or malformed url, etc, can
cause repeated and fast crashing.

For now, we require ten crashes within one second before we would
bounce the job supervisor. In future, we should manage replication
jobs with greater care.

COUCHDB-2975




> Automatically restart replication jobs if they crash
> 
>
> Key: COUCHDB-2975
> URL: https://issues.apache.org/jira/browse/COUCHDB-2975
> Project: CouchDB
>  Issue Type: Improvement
>  Components: Replication
>Reporter: Robert Newson
>
> We currently use the temporary restart strategy for replication jobs, which 
> means if they crash they are not restarted.
> Instead, let's use the transient restart strategy, ensuring they are 
> restarted on abnormal termination, while still allowing these tasks to end 
> successfully on completion or cancellation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] couchdb-couch-replicator pull request: restart replications on cra...

[rnewson]

GitHub user rnewson opened a pull request:

https://github.com/apache/couchdb-couch-replicator/pull/33

restart replications on crash

COUCHDB-2975 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cloudant/couchdb-couch-replicator 
2975-restart-replications-on-crash

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/couchdb-couch-replicator/pull/33.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #33


commit 73afc584bd10f68626d2049442b5a6058ff002db
Author: Robert Newson 
Date:   2016-03-24T11:26:08Z

Remove obsoleted R14-era code

We no longer support R14 so we're dropping R14-specific complications
in the codebase.

COUCHDB-2975

commit eb93044ca4aa02ab4427ec7082df37bae6602973
Author: Robert Newson 
Date:   2016-03-24T11:29:10Z

Use transient restart type for all replications

We want replication tasks to be restarted automatically if they crash
abnormally. Replication tasks that complete or are cancelled (by
deleting the backing _replicator doc or issuing an "cancel":true for
non-persistent jobs) should still exit, should not be restarted, and
should not have their child spec linger in the supervisor.

COUCHDB-2975

commit 4cb517659c235c06a39ee7eb6b4150cdfded6116
Author: Robert Newson 
Date:   2016-03-24T13:40:14Z

Reduce likelihood of a bad replication job taking down the job supervisor

While we can't disable max_restart_intensity, we can make it unlikely
to happen. Ordinarily, we would want this behaviour, but replication
jobs involve human input. A bad password, or malformed url, etc, can
cause repeated and fast crashing.

For now, we require ten crashes within one second before we would
bounce the job supervisor. In future, we should manage replication
jobs with greater care.

COUCHDB-2975




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (COUCHDB-2975) Automatically restart replication jobs if they crash

[Robert Newson (JIRA)]

Robert Newson created COUCHDB-2975:
--

 Summary: Automatically restart replication jobs if they crash
 Key: COUCHDB-2975
 URL: https://issues.apache.org/jira/browse/COUCHDB-2975
 Project: CouchDB
  Issue Type: Improvement
  Components: Replication
Reporter: Robert Newson


We currently use the temporary restart strategy for replication jobs, which 
means if they crash they are not restarted.

Instead, let's use the transient restart strategy, ensuring they are restarted 
on abnormal termination, while still allowing these tasks to end successfully 
on completion or cancellation.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] couchdb-couch pull request: Explicity use the {minor_version, 1} o...

[MikeFair]

GitHub user MikeFair opened a pull request:

https://github.com/apache/couchdb-couch/pull/156

Explicity use the {minor_version, 1} option with term_to_binary when …

…calculating the revision id.

This uses the 64-bit IEEE format for floats, making the binary 
representation consistent between OTP versions before and after 17.0.
This also makes it easier for third parties to replicate the md5 portion of 
the revision id calculation.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MikeFair/couchdb-couch 
revId-minor_version_1-md5Calc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/couchdb-couch/pull/156.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #156


commit fc9acc2f35743fcb8077b71d7e292acf261143aa
Author: Mike Fair 
Date:   2016-03-24T06:20:01Z

Explicity use the {minor_version, 1} option with term_to_binary when 
calculating the revision id.
This uses the 64-bit IEEE format for floats, making the binary 
representation consistent between OTP versions before and after 17.0.
This also makes it easier for third parties to replicate the md5 portion of 
the revision id calculation.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-fabric pull request: Support raw view collation

[banjiewen]

Github user banjiewen commented on a diff in the pull request:

https://github.com/apache/couchdb-fabric/pull/43#discussion_r57278616
  
--- Diff: src/fabric_view_map.erl ---
@@ -135,31 +137,100 @@ handle_message(#view_row{} = Row, {_,From}, 
#collector{sorted=false} = St) ->
 
 handle_message(#view_row{} = Row, {Worker, From}, State) ->
 #collector{
-query_args = #mrargs{direction=Dir},
+query_args = #mrargs{direction = Dir},
 counters = Counters0,
 rows = Rows0,
-keys = KeyDict
+keys = KeyDict0,
+collation = Collation
 } = State,
-Rows = merge_row(Dir, KeyDict, Row#view_row{worker={Worker, From}}, 
Rows0),
+{Rows, KeyDict} = merge_row(
+Dir,
+Collation,
+KeyDict0,
+Row#view_row{worker={Worker, From}},
+Rows0
+),
 Counters1 = fabric_dict:update_counter(Worker, 1, Counters0),
-State1 = State#collector{rows=Rows, counters=Counters1},
+State1 = State#collector{rows=Rows, counters=Counters1, keys=KeyDict},
 fabric_view:maybe_send_row(State1);
 
 handle_message(complete, Worker, State) ->
 Counters = fabric_dict:update_counter(Worker, 1, 
State#collector.counters),
 fabric_view:maybe_send_row(State#collector{counters = Counters}).
 
-merge_row(fwd, undefined, Row, Rows) ->
-lists:merge(fun(#view_row{key=KeyA, id=IdA}, #view_row{key=KeyB, 
id=IdB}) ->
-couch_ejson_compare:less_json_ids({KeyA, IdA}, {KeyB, IdB})
-end, [Row], Rows);
-merge_row(rev, undefined, Row, Rows) ->
-lists:merge(fun(#view_row{key=KeyA, id=IdA}, #view_row{key=KeyB, 
id=IdB}) ->
-couch_ejson_compare:less_json_ids({KeyB, IdB}, {KeyA, IdA})
-end, [Row], Rows);
-merge_row(_, KeyDict, Row, Rows) ->
-lists:merge(fun(#view_row{key=A, id=IdA}, #view_row{key=B, id=IdB}) ->
-if A =:= B -> IdA < IdB; true ->
-dict:fetch(A, KeyDict) < dict:fetch(B, KeyDict)
-end
-end, [Row], Rows).
+merge_row(Dir, Collation, undefined, Row, Rows0) ->
+Rows1 = lists:merge(
+fun(#view_row{key=KeyA, id=IdA}, #view_row{key=KeyB, id=IdB}) ->
+compare(Dir, Collation, {KeyA, IdA}, {KeyB, IdB})
+end,
+[Row],
+Rows0
+),
+{Rows1, undefined};
+merge_row(Dir, Collation, KeyDict0, Row, Rows0) ->
+CmpFun = case Collation of
+<<"raw">> ->
+fun (A, A) -> 0;
+(A, B) -> case A < B of
+true -> -1;
+false -> 1
+end
+end;
+_ ->
+fun couch_ejson_compare:less/2
+end,
+case maybe_update_keydict(Row#view_row.key, KeyDict0, CmpFun) of
+undefined ->
+{Rows0, KeyDict0};
+KeyDict1 ->
+Rows1 = lists:merge(
+fun(#view_row{key=A, id=IdA}, #view_row{key=B, id=IdB}) ->
+case {Dir, CmpFun(A, B)} of
+{fwd, 0} ->
+IdA < IdB;
+{rev, 0} ->
+IdB < IdA;
+{fwd, _} ->
+dict:fetch(A, KeyDict1) < dict:fetch(B, 
KeyDict1);
+{rev, _} ->
+dict:fetch(B, KeyDict1) < dict:fetch(A, 
KeyDict1)
+end
+end,
+[Row],
+Rows0
+),
+{Rows1, KeyDict1}
+end.
+
+compare(_, _, A, A) -> true;
+compare(fwd, <<"raw">>, A, B) -> A < B;
+compare(rev, <<"raw">>, A, B) -> B < A;
+compare(fwd, _, A, B) -> couch_ejson_compare:less_json_ids(A, B);
+compare(rev, _, A, B) -> couch_ejson_compare:less_json_ids(B, A).
+
+% KeyDict captures the user-supplied ordering of keys POSTed by the user by
+% mapping to integers (see fabric_view:keydict/1). It's possible that 
these keys
+% do not compare equal (i.e., =:=, used by dict) to those returned by the 
view
+% but are in fact equal under ICU. In this case (assuming the view uses ICU
+% collation) we must update KeyDict with a mapping from the ICU-equal key 
to its
+% appropriate value.
+maybe_update_keydict(Key, KeyDict, CmpFun) ->
+case dict:find(Key, KeyDict) of
+{ok, _} ->
+KeyDict;
+error ->
+case key_index(Key, dict:to_list(KeyDict), CmpFun) of
+undefined ->
+undefined;
+Value ->
+dict:store(Key, Value, KeyDict)
+end
+end.
+
+key_index(_, [], _) ->

[GitHub] couchdb-fabric pull request: Support raw view collation

[banjiewen]

Github user banjiewen commented on the pull request:

https://github.com/apache/couchdb-fabric/pull/43#issuecomment-200693294
  
This PR has been substantially rewritten in light of [this 
discussion](https://github.com/apache/couchdb-fabric/pull/43#discussion_r57205987).
 The only fundamental behavior change is in 1be5506, and the previous HEAD is 
[here](https://github.com/banjiewen/couchdb-fabric/tree/e90569a4df2912c1fa1631c14f3973da51eca02f).
 It's ready for another pass when you folks have a chance. /cc @kocolosk, 
@mikewallace1979, @iilyak


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (COUCHDB-2932) View query crashes if the emited key contains hidden characters

[Benjamin Anderson (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209844#comment-15209844
 ] 

Benjamin Anderson commented on COUCHDB-2932:


I've updated the PR for COUCHDB-2900 with a fix for this.

> View query crashes if the emited key contains hidden characters
> ---
>
> Key: COUCHDB-2932
> URL: https://issues.apache.org/jira/browse/COUCHDB-2932
> Project: CouchDB
>  Issue Type: Bug
>  Components: View Server Support
>Reporter: ILYA
>Priority: Blocker
>  Labels: regression
> Fix For: 2.0.0
>
> Attachments: invisible_char.sh
>
>
> In the cases where map function of the view emits invisible characters such 
> as U+200B. Subsequent query of the view with the keys not containing 
> invisible characters returns `badarg`.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)