On Wed, Apr 12, 2017 at 4:11 PM, Gary Dusbabek <[email protected]> wrote:
> I am leery of using springboot as is. It brings a lot of transitive > dependencies. In fact, without looking I'm willing to bet its version of > Jackson and the one used by Gossip differ (happy to be wrong though). > > Here's why: Choosing a technology like springboot can function as a lock-in > to downstream users. For example, downstream users would be prevented from > integrating Gossip into their project that uses Dropwizard or an > incompatible version of spring-web. The only remedy is to have a complex > shading setup. In my experience this is rather brittle. > > What would make sense to me is turning gossip into a multi-module maven > project, where we had modules for the different components. E.g.: > gossip-core, gossip-transport-udp, gossip-transport-springboot, etc. It > would probably make sense to do this earlier, rather than wait. A nice > side-effect is that it will force us to thinking about getting > component/layer encapsulation right. > > Gary. > > On Wed, Apr 12, 2017 at 1:25 PM, Edward Capriolo <[email protected]> > wrote: > > > > > > > On Wed, Apr 12, 2017 at 2:06 PM, Gary Dusbabek <[email protected]> > > wrote: > > > >> I have a few questions: > >> > >> 1. Interchangeable Transports: Is there support for adding transports > >> beyond UDP? I can see some utility for TCP-based transports. > >> > >> 2. Interchangeable Protocols: JSON is great for some things, but IME > >> nothing beats a good binary protocol. Also, would protocol versioning > be a > >> good idea. Consider the scenario of many thousands of nodes being > upgraded > >> in rolling fashion. > >> > >> 3. Layered Approach: This leads me to the final topic, which is to > propose > >> adopting an architecture that is more layered. Doing so will lead to > code > >> that is inherently more testable and usable as a library in other > >> projects. > >> I am mainly thinking in the context of OSI layers 5-7 but I'm open to > >> suggestions. > >> > >> If so, I'll work up a design, create JIRAs and start the work. > >> > >> Thx, > >> > >> Gary > >> > > > > Gary. Good to hear from you. > > > > Good timing. > > > > 1) I just started working on https://issues.apache.org/ > > jira/browse/GOSSIP-50. looking at the change briefly I noticed a few > > things. I am looking at spring boot. PassiveGossipThread needs to be > > modified slightly so that it can run as an http servlet. This is not a > huge > > issue. PassiveGossipThread is essentially blocking on standard in and > > calling GossipCore.recieve() > > > > The other side of this is GossipCore.send() needs to pick a client based > > on the URI of the partner. IE if talking to http use a http client. > > > > 2) The code had a piece that constructed a packet with a 4 byte header > and > > payload, I took that out because I found it was less byte copies to have > > Jackson read the object from the wire without having to read a byte[] and > > strip 4 characters, then decode the payload from the byte array. I am ok > > with adding the header (or some other header) back. The challenge I see > is > > that our payload could contain any java object so the binary protocol has > > to be able to deconstruct/reconstruct types. This takes some mucking > around > > with json modules at times to get what you want, and if you had one user > > supporting json and another kyro this would be a lot of work. But to wrap > > up yes we need something better then current. > > > > 3) In general I do not have any problem with this. One thing I found with > > GossipCore: GossipCore just became a place for the network stuff, and > also > > the per node and shared hash maps landed in there. I felt the abstraction > > was wrong, from gossipmanager it only meant chained method calls. It made > > funky OOD because features cross cut layers. I recently cut some of this > > out GossipService->GossipManager->GossipCore. Now it is just > > GossipManager->GossipCore. I probably want to rename GossipCore to > > GossipComm and move those data maps back into GossipManager. > > > > In terms of OSI layers I can only think of a few concrete examples, IE if > > I was gossiping over multicast pieces would be different. > > > > Again I am open to any refactors needed as features go forward. Even some > > not attached to a feature that makes things cleaner. For #1 maybe we can > > address that as part of gossip-50, for #2 you can tackle that now. For #3 > > that looks like a case by case. > > > > > Yes. The first thing I did with this was convert the project multi-module. Walking through this you can not really package like you would first think here is why: gossip-core, gossip-transport-udp, gossip-transport-springboot, Imagine: node1 is udp://10.0.0.1:5000 node2 is http://10.0.0.1:8080 Node 1 needs to be packaged up with a web-client that can talk to whatever is living on 8080. Going further: There is really no reason why a user would not want to change node2 from http to smtp so really the person on node1 needs essentially ALL the libraries so that they can react to whatever node 4 says it is running. Also theoretically we could start a server with more than one transport udp + http + irc. As for spring boot, I don't think we can get away from that problem. Even the slimmest thing, Jetty, has version lock in and changing APIs. How many times hadoop+ jetty burned me? Alot. I considered simply making the type a WAR but then the configuration kinda sucks. You dealing with web-xml for properties, having to make servlets with lifecycle methods and make sure <run-on-startup>1</run-on-startup>. I think the end product modules lay out like this gossip-base (vast majority of the current code) gossip-udp-client ( current GossipCore) gossip-udp-server (PassiveGossipThread child that listens on udp) gossip-http-client (commons-http version of GossipCore) gossip-boot-server (Servlet that extends/delegates PassiveGossipThread) gossip-uber (includes * and produces an uber release) If you were doing Gossip as a part of your project I think you would only directly link to the parts, not the gossip-uber. You would probably start Gossip like this: GossipBuilder().transport(o.a.g.boot.AsBootService).listen(1.1.1.1:8080) .build();
