[jira] [Commented] (COUCHDB-2980) Replicator DB on 15984 replicates to backdoor ports

[Chris Foster (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468251#comment-15468251
 ] 

Chris Foster commented on COUCHDB-2980:
---

Hi everyone,

Just chiming in on this again. Sorry for the delay, don't check JIRA often. 
Sounds like you guys have summarized our parallel conclusions.

We also stumbled across using "http://localhost:5984/db; as a workaround. It's 
not perfect, because it essentially means we can not change the port without 
having to run some weird migration script. It does ensure we won't accidentally 
replicate to production all the time though.

I think part of the confusion for me is that I was coming from 1.X. This worked 
really well for us in 1.X because we just did "databaseA" to "databaseB" and 
never had to worry about URLs. In 2.X, even though we still aren't using 
clustering yet and would prefer to not worry about it right now, it appears 
that our databases are still clustered (on one host) and our method of just 
specifying the database name was failing in a really confusing way.

As mentioned, this workaround works for us for now so don't let us hold you 
back from 2.0, but it would be cool to eventually have a better approach for 
this. Although, I am not sure what that might look like :)

> Replicator DB on 15984 replicates to backdoor ports
> ---
>
> Key: COUCHDB-2980
> URL: https://issues.apache.org/jira/browse/COUCHDB-2980
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0
>Reporter: Robert Kowalski
>
> If you POST a doc into the replicator database a replication is kicked off 
> and finishes successfully (usual 5984 port which maps to 15984 via haproxy).
> The problem is that the DB is replicated to the backdoor ports (15986) and is 
> not visible on the other ports.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2980) Replicator DB on 15984 replicates to backdoor ports

[Nick Vatamaniuc (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15467500#comment-15467500
 ] 

Nick Vatamaniuc commented on COUCHDB-2980:
--

Wonder if it is worth at least preventing creating local replications like the 
original pr did? https://github.com/apache/couchdb-couch-replicator/pull/41

Otherwise behavior is surprising for someone with 1.x experience. And then 
later even if we add a local clustered support (say in 2.1), it will all of the 
sudden do something different.

In the meantime is using `http://localhost:5984/db` an alternative for users to 
get the equivalent behavior? In other words would that cover Chris's case of 
make replicator db work as expected if it is replicated to another cluster?

> Replicator DB on 15984 replicates to backdoor ports
> ---
>
> Key: COUCHDB-2980
> URL: https://issues.apache.org/jira/browse/COUCHDB-2980
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0
>Reporter: Robert Kowalski
>
> If you POST a doc into the replicator database a replication is kicked off 
> and finishes successfully (usual 5984 port which maps to 15984 via haproxy).
> The problem is that the DB is replicated to the backdoor ports (15986) and is 
> not visible on the other ports.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2980) Replicator DB on 15984 replicates to backdoor ports

[Robert Newson (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15467133#comment-15467133
 ] 

Robert Newson commented on COUCHDB-2980:


this needs to be added to the release notes.

> Replicator DB on 15984 replicates to backdoor ports
> ---
>
> Key: COUCHDB-2980
> URL: https://issues.apache.org/jira/browse/COUCHDB-2980
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0
>Reporter: Robert Kowalski
>
> If you POST a doc into the replicator database a replication is kicked off 
> and finishes successfully (usual 5984 port which maps to 15984 via haproxy).
> The problem is that the DB is replicated to the backdoor ports (15986) and is 
> not visible on the other ports.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2980) Replicator DB on 15984 replicates to backdoor ports

[Robert Newson (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15466795#comment-15466795
 ] 

Robert Newson commented on COUCHDB-2980:


I had a good stab at implementing what was needed; namely, supporting a third 
variant of database in couch_replicator_api_wrap that uses fabric. That work is 
on branch 2980-cluster-local-repl.

I did not finish and I think it cannot be done in the timeframe of 2.0 to the 
quality we need. I'd like to make this not a blocker for 2.0. Users _can_ use 
full remote urls for source/target to replicate clustered databases around. 2.0 
will not have an answer for Chris's use case, but it's a valid one, and we 
should come back to this.


> Replicator DB on 15984 replicates to backdoor ports
> ---
>
> Key: COUCHDB-2980
> URL: https://issues.apache.org/jira/browse/COUCHDB-2980
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0
>Reporter: Robert Kowalski
>Priority: Blocker
>
> If you POST a doc into the replicator database a replication is kicked off 
> and finishes successfully (usual 5984 port which maps to 15984 via haproxy).
> The problem is that the DB is replicated to the backdoor ports (15986) and is 
> not visible on the other ports.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2980) Replicator DB on 15984 replicates to backdoor ports

[Robert Newson (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455101#comment-15455101
 ] 

Robert Newson commented on COUCHDB-2980:


I'll look again but making "local" names work correctly depending on whether 
they are initiating from the cluster port or private port or clustered 
_replicator db or node-local _replicator db is a tall order, especially at the 
last minute.

That the _replicator database itself can be replicated is an especially ugly 
consequence of this regrettable api.

> Replicator DB on 15984 replicates to backdoor ports
> ---
>
> Key: COUCHDB-2980
> URL: https://issues.apache.org/jira/browse/COUCHDB-2980
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0
>Reporter: Robert Kowalski
>Priority: Blocker
>
> If you POST a doc into the replicator database a replication is kicked off 
> and finishes successfully (usual 5984 port which maps to 15984 via haproxy).
> The problem is that the DB is replicated to the backdoor ports (15986) and is 
> not visible on the other ports.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2980) Replicator DB on 15984 replicates to backdoor ports

[Joan Touzet (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453057#comment-15453057
 ] 

Joan Touzet commented on COUCHDB-2980:
--

Ping [~rnewson]. It'd be real swell to close this out before 2.0 is released. 
You seem to suggest above that your patch is inadvisable. Is another better at 
this point or are you comfortable with releasing the code "as-is"?

> Replicator DB on 15984 replicates to backdoor ports
> ---
>
> Key: COUCHDB-2980
> URL: https://issues.apache.org/jira/browse/COUCHDB-2980
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0
>Reporter: Robert Kowalski
>Priority: Blocker
>
> If you POST a doc into the replicator database a replication is kicked off 
> and finishes successfully (usual 5984 port which maps to 15984 via haproxy).
> The problem is that the DB is replicated to the backdoor ports (15986) and is 
> not visible on the other ports.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2980) Replicator DB on 15984 replicates to backdoor ports

[Nick Vatamaniuc (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405193#comment-15405193
 ] 

Nick Vatamaniuc commented on COUCHDB-2980:
--

[~chrisfosterelli] Interesting points.

Thinking more about this, it seems it is hard to for a node in a cluster to 
know the host of the cluster in general. Say a cluster is behind a proxy for 
fault tollerance, after the document is added to a replicator db, can't see how 
it would know what the external cluster host would be say database {{a}} means 
"https://user:p...@mycluster.com/a; or 
"http://user:p...@user.somecluster.net/a; for example. In case of {

> Replicator DB on 15984 replicates to backdoor ports
> ---
>
> Key: COUCHDB-2980
> URL: https://issues.apache.org/jira/browse/COUCHDB-2980
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0
>Reporter: Robert Kowalski
>Priority: Blocker
>
> If you POST a doc into the replicator database a replication is kicked off 
> and finishes successfully (usual 5984 port which maps to 15984 via haproxy).
> The problem is that the DB is replicated to the backdoor ports (15986) and is 
> not visible on the other ports.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2980) Replicator DB on 15984 replicates to backdoor ports

[Chris Foster (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364767#comment-15364767
 ] 

Chris Foster commented on COUCHDB-2980:
---

I think we really need a way of not specifying the full URL. We have a series 
of inter-cluster replications that are continuous and persistent (in the 
`_replicator` table), and full URL's make that a big pain.

If all of the replication tables are set up to use a full URL, then it becomes 
impossible to replicate a production database elsewhere. The destination 
database you replicate the `_replicator` table to immediately starts double 
replicating the production tables, not its own databases. Since the credentials 
are included there is no way to stop this without also stopping replication in 
the production cluster.

Even if there was just a way to say "this cluster", that would be significantly 
more ideal than hardcoded full URL database strings that include the username 
and password.

> Replicator DB on 15984 replicates to backdoor ports
> ---
>
> Key: COUCHDB-2980
> URL: https://issues.apache.org/jira/browse/COUCHDB-2980
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0
>Reporter: Robert Kowalski
>Priority: Blocker
>
> If you POST a doc into the replicator database a replication is kicked off 
> and finishes successfully (usual 5984 port which maps to 15984 via haproxy).
> The problem is that the DB is replicated to the backdoor ports (15986) and is 
> not visible on the other ports.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2980) Replicator DB on 15984 replicates to backdoor ports

[Robert Newson (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324498#comment-15324498
 ] 

Robert Newson commented on COUCHDB-2980:


I don't think this can be fixed to match < 2.0 behaviour. A local source or 
target is being honoured correctly, it's just (probably) not what the user 
intended. It doesn't "replicate to backdoor ports", it's reading/writing 
directly, not using http.

"foo" in the :5986/_replicator db works as expected and it's not entirely 
unreasonable that "foo" in the :5984/_replicator means exactly the same thing.

I don't think it's appropriate to prohibit local source/target unless we will 
do so for the node-local :5986/_replicator database as well as the clustered 
:5984/_replicator database.

The hack in chttpd.erl is actually quite bad. It uses http (not https, even if 
available) and uses the local nodes public IP address, so is not fault-tolerant.

Still, the behaviour between _replicate and _replicator is inconsistent. This 
has been true in the bigcouch codebase since forever so it's arguably not 
release blocking, but now is the time to decide what behaviour we desire. To 
that end, these are all the options I think we can actually deliver in a short 
timeframe;

1) remove fix_uri/possibly_hack from _replicate. This means "foo" always means 
a local db (and therefore unsharded and unreachable by default).

2) prohibit local source/target in all cases (_replicate will return a 400 Bad 
Request and _replicator will reject a document update that tries to insert it).




> Replicator DB on 15984 replicates to backdoor ports
> ---
>
> Key: COUCHDB-2980
> URL: https://issues.apache.org/jira/browse/COUCHDB-2980
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0
>Reporter: Robert Kowalski
>Priority: Blocker
>
> If you POST a doc into the replicator database a replication is kicked off 
> and finishes successfully (usual 5984 port which maps to 15984 via haproxy).
> The problem is that the DB is replicated to the backdoor ports (15986) and is 
> not visible on the other ports.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2980) Replicator DB on 15984 replicates to backdoor ports

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322628#comment-15322628
 ] 

ASF GitHub Bot commented on COUCHDB-2980:
-

GitHub user rnewson opened a pull request:

https://github.com/apache/couchdb-couch-replicator/pull/41

ban local endpoints

Using "local" names in source and target yields unexpected behaviour 
(creating unsharded dbs which are also, by default, unreachable). This patch 
insists that "source" and "target" are http or https URL's.

COUCHDB-2980

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cloudant/couchdb-couch-replicator 
2980-ban-local-endpoints

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/couchdb-couch-replicator/pull/41.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #41


commit 12a5e2ac0d47d942327133a996e6065292c4f213
Author: Robert Newson 
Date:   2016-06-09T14:39:04Z

Ensure _design/_replicator VDU is updated

commit 69558f31f52c14ea8bee510e69111b0e00f85fe8
Author: Robert Newson 
Date:   2016-06-09T14:39:19Z

Insist on http/https url's for source and target




> Replicator DB on 15984 replicates to backdoor ports
> ---
>
> Key: COUCHDB-2980
> URL: https://issues.apache.org/jira/browse/COUCHDB-2980
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0
>Reporter: Robert Kowalski
>Priority: Blocker
>
> If you POST a doc into the replicator database a replication is kicked off 
> and finishes successfully (usual 5984 port which maps to 15984 via haproxy).
> The problem is that the DB is replicated to the backdoor ports (15986) and is 
> not visible on the other ports.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2980) Replicator DB on 15984 replicates to backdoor ports

[Robert Kowalski (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322418#comment-15322418
 ] 

Robert Kowalski commented on COUCHDB-2980:
--

other example:


{
  "_id": "my_rep2",
  "_rev": "3-b66e82cfb790a314ee5f9278860a00a9",
  "source": 
"http://rockoartischocko:mypassw...@rockoartischocko.cloudant.com/animaldb;,
  "target": "animaldbfromcloudant",
  "continuous": false,
  "create_target": true,
  "user_ctx": {
"name": "YOU",
"roles": [
  "_admin"
]
  },
  "owner": null,
  "_replication_state": "completed",
  "_replication_state_time": "2016-06-09T14:21:49+02:00",
  "_replication_id": "25221306efcdeb84fdd16e7bcbe9438b",
  "_replication_stats": {
"revisions_checked": 16,
"missing_revisions_found": 16,
"docs_read": 16,
"docs_written": 16,
"changes_pending": null,
"doc_write_failures": 0,
"checkpointed_source_seq": 
"18-g1GjeJzLYWBgYMlgTmGQT0lKzi9KdUhJMtPLSs1LLUst0kvOyS9NScwr0ctLLckBKmRKZEiyf1YGcyJ3LlCAPck4zdggKY2wdlQrTHBbkeQAJJPqobYwgW0xNUszsrQwJWwC0R7JYwGSDA1ACmjRfoRN5oZJyUnmBqT6x4KQTQcgNoH9xAy2yczCOMUyOY2wKVkAq-mIRA"
  }
}


> Replicator DB on 15984 replicates to backdoor ports
> ---
>
> Key: COUCHDB-2980
> URL: https://issues.apache.org/jira/browse/COUCHDB-2980
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0
>Reporter: Robert Kowalski
>Priority: Blocker
>
> If you POST a doc into the replicator database a replication is kicked off 
> and finishes successfully (usual 5984 port which maps to 15984 via haproxy).
> The problem is that the DB is replicated to the backdoor ports (15986) and is 
> not visible on the other ports.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2980) Replicator DB on 15984 replicates to backdoor ports

[Nick Vatamaniuc (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227328#comment-15227328
 ] 

Nick Vatamaniuc commented on COUCHDB-2980:
--

We should probably disallow "local" replications from being accepted in source 
and target of replication doc. Those end up as "local" databases (like say 
_users, _nodes, _dbs) don't do what is expected.

To make things more interesting, for the _replicate http endpoint we do some 
hacks to turn a local db into a full url:

https://github.com/apache/couchdb-chttpd/blob/master/src/chttpd.erl#L389

But that is running inside the context of a http request so it easy to access 
to authorization headers and such.



> Replicator DB on 15984 replicates to backdoor ports
> ---
>
> Key: COUCHDB-2980
> URL: https://issues.apache.org/jira/browse/COUCHDB-2980
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Reporter: Robert Kowalski
>
> If you POST a doc into the replicator database a replication is kicked off 
> and finishes successfully (usual 5984 port which maps to 15984 via haproxy).
> The problem is that the DB is replicated to the backdoor ports (15986) and is 
> not visible on the other ports.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2980) Replicator DB on 15984 replicates to backdoor ports

[Robert Newson (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227242#comment-15227242
 ] 

Robert Newson commented on COUCHDB-2980:


so we have a fun called possibly_hack that fixes up local source/target for 
_replicate calls but not the equivalent for _replicator docs.

I'm inclined to prohibit "local" source/target in _replicator docs by tweaking 
the validate_doc_update function.

To fix this naturally would involve making _replicator document behave 
differently based on whether it came from a clustered or non-clustered 
database, which might be tricky.

> Replicator DB on 15984 replicates to backdoor ports
> ---
>
> Key: COUCHDB-2980
> URL: https://issues.apache.org/jira/browse/COUCHDB-2980
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Reporter: Robert Kowalski
>
> If you POST a doc into the replicator database a replication is kicked off 
> and finishes successfully (usual 5984 port which maps to 15984 via haproxy).
> The problem is that the DB is replicated to the backdoor ports (15986) and is 
> not visible on the other ports.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2980) Replicator DB on 15984 replicates to backdoor ports

[Robert Kowalski (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227032#comment-15227032
 ] 

Robert Kowalski commented on COUCHDB-2980:
--

POST to $DB/replicator with a simple replication to kick of a replication:

{
  "_id": "my_rep2",
  "_rev": "3-9ede1daaf9b0fed4ef92a7162b7162e0",
  "source": "http://localhost:5984/animaldb;,
  "target": "copyanimaldbrepbug",
  "continuous": false,
  "create_target": true,
  "user_ctx": {
"name": "YOU",
"roles": [
  "_admin"
]
  }
}

> Replicator DB on 15984 replicates to backdoor ports
> ---
>
> Key: COUCHDB-2980
> URL: https://issues.apache.org/jira/browse/COUCHDB-2980
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Reporter: Robert Kowalski
>
> If you POST a doc into the replicator database a replication is kicked off 
> and finishes successfully (usual 5984 port which maps to 15984 via haproxy).
> The problem is that the DB is replicated to the backdoor ports (15986) and is 
> not visible on the other ports.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2980) Replicator DB on 15984 replicates to backdoor ports

[Alexander Shorin (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227004#comment-15227004
 ] 

Alexander Shorin commented on COUCHDB-2980:
---

What kind of POST that should be?

> Replicator DB on 15984 replicates to backdoor ports
> ---
>
> Key: COUCHDB-2980
> URL: https://issues.apache.org/jira/browse/COUCHDB-2980
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Reporter: Robert Kowalski
>
> If you POST a doc into the replicator database a replication is kicked off 
> and finishes successfully (usual 5984 port which maps to 15984 via haproxy).
> The problem is that the DB is replicated to the backdoor ports (15986) and is 
> not visible on the other ports.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)