Hi Joan, Thanks for pushing this forward, replied inline.
On Mon, Oct 27, 2014 at 11:58 PM, Joan Touzet <[email protected]> wrote: > Today, someone came to the #couchdb channel asking about > _externals. For a long while it's been on my mind that perhaps > we should deprecate the entire _externals feature for a number > of reasons: I would like to add to _externals also os daemons, query servers and update_notification_handlers as node-level external processes which are hardly to setup among the cluster. > 1. Couch is not a great reverse proxy. Making it into one is > as hard as rewriting nginx or haproxy in erlang. It's a > distraction to our development team and far outside our > core competency. Actually, CouchDB doesn't works as a real reverse proxy for _externals - it still communicates with them via stdio by using JSON messages (cgi style) which is even worse than couch_httpd_proxy feature. > 2. In a clustered CouchDB (the default in 2.0), the > assumptions around externals change drastically. For an > _external to work, it must be stateless and not rely upon > multiple sequential requests to hit the same node (assuming > the standard n-node cluster + a load balancer/reverse proxy > at the front.) > > People who wrote a CouchDB 1.x external could reasonably > expect to write an old-school singleton app (i.e., the only > copy of that external process running, on a single machine). > > If they engaged in any of a number of bad behaviours for > distributed systems - storing content on local disk, locking > or blocking connections to other services/databases in a > "single-threaded" pattern, or even expecting CouchDB not to > possibly introduce a conflict or "read your writes" - they > will probably fail outright at best, or at worst introduce > subtle and confusing behaviour. The problems you described for 1.x state aren't very critical: - os_process_timeout will not let you serve request too long; - side effects as communication with other IO sources is really up to developers of these developers - they should know what they're doing; - conflicts aren't a problem, but matter of things; But you're right that "singleton" instance is a problem. Big one. If only CouchDB had pool of externals as like as it doing for query servers... > TL;DR: We're changing the contract we give to _externals in a > reverse-compatibility-breaking way. We either need to document it > straight up, along with all of the admonishments required for > people who expect it to operate the same as in 1.x, or we need to > remove it. > > My opinion is that now that the default CouchDB rollout will be > a cluster with a reverse proxy, that _externals should be exposed > through the load balancer, which can then reference 1 or more > processes distributed either on the same CouchDB nodes, or on > different hosts should compute needs demand it. > > The exception here would be a single-node CouchDB, which could > still use the same approach. However I don't see the issue with > deploying an haproxy on that same node and using the same approach > I describe above. > > Thoughts, comments, suggestions? So far we have two setups for CouchDB 2.0: 1. Standalone (single node) 2. Clustered In standalone mode there are no problems at all since it acts as CouchDB 1.x. - all the features may continue to work as they do before. Also, in this mode people don't usually need haproxy since there mostly will be nginx as reverse proxy. Problems starts to happen after we join our node into the cluster. Externals are the only part of whole problem. As I mentioned above, query servers, os daemons, update notification handlers...even couch_http_proxy features are heavily bounded to single node. I'll call them all as externals below. The first and biggest one is that external process is referenced to OS command or some file on disk which will be executed. If each node is on our server such a things wouldn't oblivious works - you need to setup and maintain the same environment on each node to make your OS process be executed without errors. There we comes to the second one - distribution (also actual for 1.x). Today we have a lot of provisioning tools, but it's still not a thing that will handled with "relax" moto; "complicated" - yes, that's a word. Partially we could solve both with one simple move: store externals not on disk, but as attachments for some special document. This will help to replicate externals among the cluster keeping them all the same on each node and start them with some magic command defined in the related document. Still, OS environment configuration is required, but at least now each node is able to serve our externals. Now we comes to "stateless" problem or tainting externals with side effect IO to other resources which aren't be available on each node, but only for a few. Well, a really problem, but it's depends on implementation and developers will. As like as for plugins. Btw, CouchDB Plugins feature is going to be worked for clustered environment, right? Then it's perfect replacement for some externals - fast, distributed, erlangish. And with some trick I believe we could turn custom query servers into CouchDB plugins via tiny Erlang wrapper above OS process. What's more important, this should also solve distribution between clusters via plugins registry. What we loose with removing externals: 1. Easy to extend CouchDB without need to know what is Erlang stands for. Python, Ruby, NodeJS - use whatever tool you know. 2. No need to setup runit / monit/ supervisord - they are complicated while all you need is plain stupid supervisor which will bring your process up when it falls. What CouchDB perfectly does. 3. If you need to update existed external, you just open terminal and sends SIGTERM to specific external - it brings down, CouchDB respawns it and it's done. 4. Easy to setup continuously background tasks via OS daemons. 5. External processes are good for extending API while you still don't need in some webframework at front side. 6. Custom query servers allows you to break away from javascript and use other languages with their batteries - language processing and math is much simpler and powerful with Python. All these customization are on your own, but they turns CouchDB from slow (in terms of communication protocol) and stupid (in terms of features) database into powerful application server which is ready to crack your tasks. Later, you may optimize your infra with runit, haproxy, web framework - all configured right, but until them all you need for start is CouchDB and few scripts to bootstrap. But enough, I wrote too much already (: So what's our options: 1. Remove externals since they don't behave as expected for clustered CouchDB. Broken expectation is worser than missed feature. 2. Let them run for single node mode only. However, this partially breaks the expectations and not clear how to disable them right during cluster setup. 3. Investigate into the possibilities of how we could fix this. We still have to figure out what to do with query servers. Could plugins help us? 4. Any other options we have? Like drop stdio protocol and use TCP-based to communicate with external services which may be hosted wherever they be? -- ,,,^..^,,,
