Here I'm back on the topic I have created this issue https://issues.apache.org/jira/browse/ZOOKEEPER-2755 Allow to subclass ClientCnxnSocketNetty and NettyServerCnxn in order to use Netty Local transport
and submitted a patch https://github.com/apache/zookeeper/pull/227 My idea is not to provide an official ZooKeeper network less version at this moment but to allow developers to easly extend/subclass ZooKeeper networking in a more future-proof way. As ZooKeeper 3.5 onwards uses Netty it is possibile to use it to have a network-less zookeeper setup The implementation will be very "easy" using Netty, look at this test case: https://github.com/eolivelli/zookeeper/blob/ZOOKEEPER-2755-localaddress/src/java/test/org/apache/zookeeper/test/NetworklessTest.java I hope that helps -- Enrico 2017-01-09 15:33 GMT+01:00 Enrico Olivelli <eolive...@gmail.com>: > Hi Matthias, > sorry for late reply. I've taken a look to your code. > Personally I would prefer to use Netty directly, but I think that if your > implementation is working you should try to file a PR in GitHub and create > a JIRA > > did you have any other answer from the ZK community ? > > -- Enrico > > > > > > > > 2016-10-04 16:22 GMT+02:00 Matthias Amberg <matthias.amb...@unblu.com>: > >> Hi Enrico, >> >> I set up a github repo with just the 'embedded' connection classes. >> They unfortunately are from an older (3.4.6) release and would some clean >> up to fit into the current dev branch of Zookeeper. >> >> However feel free to have a look. Unfortunately there is a lot of code in >> there (just like in the other implementations of that version, that aren't >> directly related to >> the connection (which later got cleaned up in ZK and would need clean up >> on >> my side too.) >> >> The biggest changes are in the ServerCnxn class starting here: >> https://github.com/matthiasamberg/ZookeeperEmbeddedConnectio >> n/blob/master/org/apache/zookeeper/server/EmbeddedServerCnxn.java#L416 >> >> The gist of it is: >> EmbeddedClientCnxnSocket and EmbeddedServerCnxn get instatiated together >> when an (internal) client tries to connect. It finds the >> EmbeddedServerCnxnFactory via a singleton instance of that class. >> The ByteBuffers that normally get sent over the network simply end up in a >> pair of Piped Input/Output streams that are shared between >> ClientCnxnSocket >> and ServerCnxn, that's already it. Everything around it just got adapted >> to fit into the ZK architecture. >> >> I happily port this to a current version of ZK if you (and/or others) >> think >> this implementation approach would actually get patched into ZK, but it >> would be great to hear from you guys whether it is worthwile for me to >> port >> this to the newest version first. >> >> Best regards, >> >> Matthias Amberg >> Developer >> Office: ++41 511 11 64 >> >> unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland >> >> unblu makes websites personal http://www.unblu.com >> >> Please consider the environment before printing this email. >> >> This e-mail contains including its attachments confidential protected >> information by law. If you are not the intended recipient, please contact >> the sender and delete this message. Any unauthorised copying of this >> message or unauthorised distribution of the information contained herein >> is >> prohibited. >> >> Diese E-Mail einschließlich Ihrer Anhänge enthält vertrauliche rechtlich >> geschützte Informationen. Wenn Sie nicht der beabsichtigte Empfänger sind, >> informieren Sie bitte sofort den Absender und löschen Sie diese E-Mail. >> Das >> unbefugte Kopieren oder die unbefugte Weitergabe der enthaltenen >> Informationen ist nicht gestattet. >> >> On 30 September 2016 at 14:30, Enrico Olivelli <eolive...@gmail.com> >> wrote: >> >> > Matthias, >> > I am sorry I so not have time to pack a good patch these days. >> > If you already have some working code maybe you can create a PR on >> GitHub >> > so that the community can review and comment about it. >> > >> > My actual idea is to make all Netty-aware classes extensible in order to >> > support "local" transport. >> > In 3.5 branch Netty is fully supported for both client and server sides >> of >> > the connection >> > >> > On BookKeeper I have already implemented such a feature >> > https://issues.apache.org/jira/browse/BOOKKEEPER-896 >> > https://github.com/apache/bookkeeper/pull/20 >> > >> > Regards >> > Enrico >> > >> > >> > >> > Il ven 30 set 2016, 14:14 Matthias Amberg <matthias.amb...@unblu.com> >> ha >> > scritto: >> > >> > > Hi Enrico, >> > > >> > > thanks for your response, sorry for the late reply. >> > > Would you mind sharing some details of your upcoming implementation, >> do >> > you >> > > think it will cover our use case of using it for an embedded Zookeeper >> > > instance for a production system? >> > > If yes, we'll continue to use our own implementation until this >> becomes >> > > available in Zookeeper, do you have an ETA? >> > > >> > > Best regards, >> > > >> > > Matthias >> > > >> > > >> > > Matthias Amberg >> > > Developer >> > > Office: ++41 511 11 64 >> > > >> > > unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland >> > > >> > > unblu makes websites personal http://www.unblu.com >> > > >> > > Please consider the environment before printing this email. >> > > >> > > This e-mail contains including its attachments confidential protected >> > > information by law. If you are not the intended recipient, please >> contact >> > > the sender and delete this message. Any unauthorised copying of this >> > > message or unauthorised distribution of the information contained >> herein >> > is >> > > prohibited. >> > > >> > > Diese E-Mail einschließlich Ihrer Anhänge enthält vertrauliche >> rechtlich >> > > geschützte Informationen. Wenn Sie nicht der beabsichtigte Empfänger >> > sind, >> > > informieren Sie bitte sofort den Absender und löschen Sie diese >> E-Mail. >> > Das >> > > unbefugte Kopieren oder die unbefugte Weitergabe der enthaltenen >> > > Informationen ist nicht gestattet. >> > > >> > > On 27 September 2016 at 12:40, Enrico Olivelli <eolive...@gmail.com> >> > > wrote: >> > > >> > > > Hi Matthias, >> > > > I am working on a proposal on this topic. My idea is to use local >> netty >> > > > features. >> > > > I will share some code. >> > > > Beware that this usage of zookeeper is to be adopted only if >> needed. My >> > > > case is for instance to run unit tests without a real network. >> > > > Maybe you should consider to abstract your zookkeper access layer >> > > > >> > > > Enrico >> > > > >> > > > Il mar 27 set 2016, 11:07 Matthias Amberg < >> matthias.amb...@unblu.com> >> > ha >> > > > scritto: >> > > > >> > > > > Dear Zookeeper developers, >> > > > > >> > > > > I have a feature request for allowing an embedded Zookeeper be run >> > and >> > > > > connected to without using any system networking stack. All >> > > > > client-server communications would be handled internally of a >> single >> > > > > runtime. >> > > > > We at Unblu offer our product as a cloud service (orchestrated by >> > > > > Zookeeper) as well as an on-premises installable product. >> > > > > In the latter scenario we use a single embedded version of >> Zookeeper >> > to >> > > > > orchestrate all the different embedded Version. >> > > > > This setup allows us to use the same mechanisms for service >> discovery >> > > and >> > > > > synchronization, whether we run in a distributed cloud setup or >> > > > > a single on-premises node. >> > > > > >> > > > > For this use case we have implemented versions of ServerCnxn, >> > > > > ServerCnxnFactory and ClientCnxnSocket that handle client-server >> > > > > connections internally without >> > > > > using any server networking stack. >> > > > > >> > > > > I would very much love to commit a patch that implements this >> feature >> > > or >> > > > > open a jira issue to discuss this further, if there is interest >> from >> > > the >> > > > > Zookeeper community for such a feature. >> > > > > >> > > > > We think this feature offers several advantages over running >> > Zookeeper >> > > > > locally or embedded via a networking stack: >> > > > > - less system resource usage (no need for additional networking >> > > overhead >> > > > in >> > > > > what could be systems that already have a high load of networking >> IO) >> > > > > - simpler configuration and potentially better security: no need >> to >> > > > > configure any ports and network addresses, firewalls, loopback >> > > interfaces >> > > > > etc. (Our software runs in banking environments where security >> > > > constraints >> > > > > often makes it >> > > > > difficult to (even) have loopback connections) >> > > > > - simpler Continuous Integration: Different Test Runs of our >> builds >> > > > > constantly interfere with each other due to ports being already in >> > use >> > > > and >> > > > > so forth. >> > > > > >> > > > > Additionally as this feature simply would be an extension to the >> > > current >> > > > > code base that wouldn't even be instantiated in the traditional >> > server >> > > > > setup there would be almost no interference with the current code >> > base >> > > > and >> > > > > hence no threat >> > > > > to Zookeeper stability in traditional setups. >> > > > > >> > > > > It would be great if you could provide your thoughts and feedback >> on >> > > this >> > > > > and whether this is a welcome feature, in which case I would >> create a >> > > > Jira >> > > > > Issue for further discussion and patch submission. >> > > > > >> > > > > Best regards, >> > > > > >> > > > > Matthias Amberg >> > > > > Developer >> > > > > Office: ++41 511 11 64 >> > > > > >> > > > > unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland >> > > > > >> > > > > unblu makes websites personal http://www.unblu.com >> > > > > >> > > > -- >> > > > >> > > > >> > > > -- Enrico Olivelli >> > > > >> > > >> > >> > >