We should move the discussion to JIRA so that we don't lose all the inputs. Camille, could you open 2 JIRAs when you get a chance?
1. Node TTLs 2. Pluggable RPC endpoints, starting with HTTP On Mon, Apr 13, 2015 at 3:00 PM, Patrick Hunt <[email protected]> wrote: > That's really the easy part. Coming up with something that will work in our > context is the harder bit IMO. For example, if you have multiple REST > service endpoints how will that work, one per zk server? Are you going to > tie a particular client session to a particular server, how will sessions > be maintained. Would you want to put a http load balancer in between? > Typically you want to do this otw you need more logic on the client (like > we have today in the java client). Seems like there's a lot of these types > of issues that would need to be worked through. Same (but different) for > thrift support. > > Patrick > > On Mon, Apr 13, 2015 at 2:19 PM, Flavio Junqueira < > [email protected]> wrote: > >> I also like the plugin idea, it's a good one. >> >> -Flavio >> >> > On 13 Apr 2015, at 18:58, Patrick Hunt <[email protected]> wrote: >> > >> > Once something is generally available, embedded and easy to enable, it >> > probably won't take much for someone to extend. Once they see the >> benefits. >> > Although there are already folks interested in what we have today. >> > >> > We could do the same thing re thrift, or any other "plugin" for that >> > matter, (enable dynamic config of embedded endpoint) if there's interest. >> > >> > Patrick >> > >> > On Mon, Apr 13, 2015 at 10:48 AM, Camille Fournier <[email protected]> >> > wrote: >> > >> >> Yes ok agreed. I do personally think fully supporting TTL nodes (and >> >> long-poll based watches) for purposes of enabling the same types of >> >> high-level behaviors as etcd/consul would probably be useful, as per the >> >> original start of this epic thread. But even starting with an embedded >> >> proxy is a good start. >> >> >> >> C >> >> >> >> On Mon, Apr 13, 2015 at 1:42 PM, Patrick Hunt <[email protected]> wrote: >> >> >> >>> We already have a proxy. My thought was to take advantage of jetty now >> >>> being embedded in ZK (3.5+) so that a proxy wouldn't be necessary. >> >>> >> >>> Patrick >> >>> >> >>> On Mon, Apr 13, 2015 at 10:39 AM, Camille Fournier <[email protected] >> > >> >>> wrote: >> >>> >> >>>> I've lost the thread now. Are we talking about inside of the ZK >> servers >> >>>> themselves, or as a proxy in front of them? >> >>>> >> >>>> On Mon, Apr 13, 2015 at 1:35 PM, Jordan Zimmerman < >> >>>> [email protected]> wrote: >> >>>> >> >>>>> I agree that get/set kinds of things is useful - especially for OPs >> >>>>> groups. What if ZK had some sort of pluggable server support? Ops >> folks >> >>>>> could plug in a simple REST server, others could plug in a Thrift >> server, >> >>>>> etc. >> >>>>> >> >>>>> >> >>>>> >> >>>>> On April 13, 2015 at 12:31:51 PM, Patrick Hunt ([email protected]) >> >>>>> wrote: >> >>>>> >> >>>>> I think it depends on the use case. I've had a number of folks asking >> >>>>> for >> >>>>> the REST support, in those cases I think they are mainly interested >> in >> >>>>> active participation (get/set rather than watches/ephemerals). >> >>>>> >> >>>>> The SPEC page has all the details on what the REST proxy currently >> >>>>> supports. >> >>>>> >> >>>>> Patrick >> >>>>> >> >>>>> On Mon, Apr 13, 2015 at 8:35 AM, Jordan Zimmerman < >> >>>>> [email protected]> wrote: >> >>>>> >> >>>>>> How does the current REST code handle watches and background >> >>>>>> notifications? Are clients required to poll periodically? This is >> >>>>> really >> >>>>>> RPC and REST is not ideal for RPC. So, as I mentioned before, I >> think >> >>>>> the >> >>>>>> ZK team should consider a real RPC system like Thrift instead of >> REST. >> >>>>>> >> >>>>>> -Jordan >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> On April 11, 2015 at 1:33:03 PM, Patrick Hunt ([email protected]) >> >>>>> wrote: >> >>>>>> >> >>>>>> We pack quite a bit of functionality in our client side libraries. >> >>>>> That's >> >>>>>> probably one of the main things you'll noticed if you try to use >> >>>>> REST. But >> >>>>>> then if your use case is primarily configuration, or something that >> >>>>> doesn't >> >>>>>> require sessions then the current REST support should be more than >> >>>>>> sufficient. What are the other systems doing that we are currently >> not >> >>>>>> doing wrt API support? >> >>>>>> >> >>>>>> I'd think that taking our current rest contrib, updating the >> >>>>> dependencies, >> >>>>>> and deploying as part of the embedded jetty would work well with >> >>>>> minimal >> >>>>>> effort. >> >>>>>> >> >>>>>> Patrick >> >>>>>> >> >>>>>> On Sat, Apr 11, 2015 at 9:22 AM, Camille Fournier < >> [email protected] >> >>>>>> >> >>>>>> wrote: >> >>>>>> >> >>>>>>> I agree that they have to write clients that do work, but there is >> >>>>>> clearly >> >>>>>>> a desire and willingness out there to do that work in exchange for >> >>>>> a more >> >>>>>>> "obvious" way of interacting. So supporting it should be something >> >>>>> that >> >>>>>> we >> >>>>>>> consider if the users of such systems want the option. >> >>>>>>> >> >>>>>>> On Sat, Apr 11, 2015 at 12:07 PM, Jordan Zimmerman < >> >>>>>>> [email protected]> wrote: >> >>>>>>> >> >>>>>>>> REST has issues for end users. They will still have to write >> >>>>> clients >> >>>>>> and >> >>>>>>>> do a lot of extra work. REST support is good in most languages but >> >>>>>> having >> >>>>>>>> native support is superior. That's why I choose Thrift for the >> >>>>>> CuratorRPC >> >>>>>>>> Proxy. >> >>>>>>>> >> >>>>>>>> -Jordan >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On April 10, 2015 at 9:32:47 PM, Michi Mutsuzaki ( >> >>>>>> [email protected]) >> >>>>>>>> wrote: >> >>>>>>>> >> >>>>>>>> Yeah it's quite painful to manage another set of processes just >> >>>>> for >> >>>>>>>> proxying requests. I'd definitely use this if I can run it >> >>>>> embedded in >> >>>>>>>> the ZooKeeper process. I'm very excited about the idea of being >> >>>>> able >> >>>>>>>> to use curl to look at what's in ZooKeeper :) >> >>>>>>>> >> >>>>>>>> On Fri, Apr 10, 2015 at 6:03 PM, Patrick Hunt <[email protected]> >> >>>>>> wrote: >> >>>>>>>>> Here's the spec and readme from contrib/rest: >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> http://svn.apache.org/viewvc/zookeeper/trunk/src/contrib/rest/SPEC.txt?view=markup >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> http://svn.apache.org/viewvc/zookeeper/trunk/src/contrib/rest/README.txt?view=markup >> >>>>>>>>> >> >>>>>>>>> The current implementation is a standalone proxy. It's not >> >>>>> embedded >> >>>>>> in >> >>>>>>> zk >> >>>>>>>>> itself. That might be part of the reason. >> >>>>>>>>> >> >>>>>>>>> Patrick >> >>>>>>>>> >> >>>>>>>>> On Fri, Apr 10, 2015 at 4:43 PM, Camille Fournier < >> >>>>>> [email protected]> >> >>>>>>>>> wrote: >> >>>>>>>>> >> >>>>>>>>>> Forgive me for not reading the code, can you share in more >> >>>>> detail >> >>>>>> what >> >>>>>>>> the >> >>>>>>>>>> existing REST proxy provides? I'm curious why people are >> >>>>> jumping to >> >>>>>>> use >> >>>>>>>>>> etcd because of ease of use w/http access if we already have >> >>>>>> something >> >>>>>>>> that >> >>>>>>>>>> works? >> >>>>>>>>>> >> >>>>>>>>>> On Fri, Apr 10, 2015 at 7:28 PM, Patrick Hunt < >> >>>>> [email protected]> >> >>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> There's the REST work in contrib. Both Andrei and I worked >> >>>>> on that >> >>>>>>> - I >> >>>>>>>>>> did >> >>>>>>>>>>> the basic support and Andrei added sessions and heartbeating >> >>>>> among >> >>>>>>>> other >> >>>>>>>>>>> improvements. >> >>>>>>>>>>> >> >>>>>>>>>>> Now that 3.5 has embedded Jetty it should be much simpler to >> >>>>> run >> >>>>>>> REST >> >>>>>>>> as >> >>>>>>>>>>> part of the ZK service itself. When the original proxy work >> >>>>> was >> >>>>>> done >> >>>>>>>>>> Jetty >> >>>>>>>>>>> was not yet part of ZK. >> >>>>>>>>>>> >> >>>>>>>>>>> Patrick >> >>>>>>>>>>> >> >>>>>>>>>>> On Thu, Apr 9, 2015 at 10:20 AM, Jordan Zimmerman < >> >>>>>>>>>>> [email protected]> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>>> Since Curator is now Apache and I'm no longer at Netflix, I >> >>>>> don't >> >>>>>>>> follow >> >>>>>>>>>>>> Netflix messages very much. Sorry about that. >> >>>>>>>>>>>> >> >>>>>>>>>>>> -Jordan >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> On April 9, 2015 at 12:15:02 PM, Camille Fournier ( >> >>>>>>>> [email protected]) >> >>>>>>>>>>>> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>> Thanks Jordan! I actually asked on Twitter whether Netflix >> >>>>> had >> >>>>>>>> anything >> >>>>>>>>>>>> but >> >>>>>>>>>>>> didn't get a clear answer. >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Thu, Apr 9, 2015 at 11:22 AM, Jordan Zimmerman < >> >>>>>>>>>>>> [email protected]> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>>> FYI >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Curator now has a Thrift-based proxy that has all the ZK >> >>>>> APIs >> >>>>>>>> exposed >> >>>>>>>>>> as >> >>>>>>>>>>>>> well as Curator's added APIs and recipes: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> http://curator.apache.org/curator-x-rpc/index.html >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> -Jordan >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> On April 8, 2015 at 2:09:33 PM, Camille Fournier ( >> >>>>>>>> [email protected]) >> >>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> All, >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> I've been doing a bit of research on etcd as part of work >> >>>>> for >> >>>>>> an >> >>>>>>>>>>>> upcoming >> >>>>>>>>>>>>> talk, and it has gotten me thinking about what it would >> >>>>> take to >> >>>>>>>> create >> >>>>>>>>>>>> an >> >>>>>>>>>>>>> http version of ZK for certain operations. For many >> >>>>> operations >> >>>>>>> you >> >>>>>>>>>> could >> >>>>>>>>>>>>> put an http proxy in front of ZK to translate, even >> >>>>>> implementing >> >>>>>>>> the >> >>>>>>>>>>>>> "long-poll-style" watch operation to some extent. But it >> >>>>> would >> >>>>>> be >> >>>>>>>> very >> >>>>>>>>>>>>> hard >> >>>>>>>>>>>>> to do a temporary node via a proxy without a lot of proxy >> >>>>>>> failover >> >>>>>>>>>>>>> complexity. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> As a bit of background, if you want to do an "ephemeral" >> >>>>> node >> >>>>>> in >> >>>>>>>> etcd, >> >>>>>>>>>>>> you >> >>>>>>>>>>>>> basically create a key with a TTL. Unless the key is >> >>>>> updated >> >>>>>>> with a >> >>>>>>>>>> new >> >>>>>>>>>>>>> TTL, the key will auto-expire when the TTL is reached. >> >>>>> Now, I >> >>>>>>> have >> >>>>>>>> a >> >>>>>>>>>> lot >> >>>>>>>>>>>>> of >> >>>>>>>>>>>>> thoughts about this (seems like you have to implement >> >>>>>> heartbeats >> >>>>>>>> via >> >>>>>>>>>>>> http >> >>>>>>>>>>>>> to truly mimic ephemeral nodes which may not be as simple >> >>>>> as >> >>>>>> all >> >>>>>>>> this >> >>>>>>>>>>>> http >> >>>>>>>>>>>>> sounds), but I do think that if there is appetite for >> >>>>> easy http >> >>>>>>>> access >> >>>>>>>>>>>> for >> >>>>>>>>>>>>> consensus systems we should at least take the time to >> >>>>> think >> >>>>>> about >> >>>>>>>> what >> >>>>>>>>>>>> it >> >>>>>>>>>>>>> would take for us to provide this. In particular, I think >> >>>>> we'd >> >>>>>>>> have to >> >>>>>>>>>>>>> make >> >>>>>>>>>>>>> it possible to create a node with a TTL that is not tied >> >>>>> to a >> >>>>>>>>>> particular >> >>>>>>>>>>>>> session. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Curious to see if anyone has any thoughts on this. It >> >>>>> seems >> >>>>>> like >> >>>>>>> a >> >>>>>>>> bit >> >>>>>>>>>>>> of >> >>>>>>>>>>>>> a >> >>>>>>>>>>>>> shame that ZK, which is a good battle-tested system, is >> >>>>>>> frequently >> >>>>>>>>>> being >> >>>>>>>>>>>>> passed-over these days because of the complexity of >> >>>>> clients, >> >>>>>> and >> >>>>>>>> the >> >>>>>>>>>>>> fact >> >>>>>>>>>>>>> that it is really pretty damn hard to do a client impl in >> >>>>>> certain >> >>>>>>>>>>>>> languages >> >>>>>>>>>>>>> (Ruby is the notable one I've heard). >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Best, >> >>>>>>>>>>>>> C >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>>> >> >>> >> >> >> >>
