I see how performance of certain components could benefit from switching its implementation to Go. Earlier this year I did performance measurements of Go vs Scala http servers / logging systems. I was able to find a combo of Go based components that outperformed Scala implementation by 3-4x times.
Thus **+1** to what Markus said with regards to the language choice. On Wed, 29 Aug 2018 at 04:31 Markus Thömmes <markusthoem...@apache.org> wrote: > Hi, > > okay, let's separate concerns: > > ### Golang vs. Scala vs. something else entirely: ### > > I'm okay doing it in Scala although I don't see an issue in swapping the > language for a component that needs to be rewritten from the ground up > anyway (which I'm suggesting at least the ContainerRouter is). > > The ContainerPool/ContainerProxy part of it comes to mind immediately. It > was built with a scale of maybe 100 containers max in mind. There are a few > performance problems with it at scale, that I can point out off the bat. > The changes needed to mitigate those are rather severe and while we **can** > implement those (and maybe even should if this proofs to make sense) I > believe it makes sense to at least experiment with a freshly written > component. In the new architecture, we can also greatly reduce the state > model to just paused/unpaused for the ContainerRouter, further simplifying > its implementation. > > The ContainerRouter's speed requirements are more close to those of > nginx/envoy than they are to those of the current ContainerPool/Proxy > implementation. As Dave mentioned, for the ContainerRouter it will be very > critical to know how many containers it can handle in reality, hence I'm > shooting for an as efficient implementation of its innerts as possible to > flesh out how far we can push the boundaries here. > > In any case: Yes we can stick to Scala here, but at least the > ContainerRouter might be 100% new code and we might not even be able to > rely on things like akka-http for its implementation for the reasons > mentioned above. > > ### Seperate repo vs. branch ### > > Towards Michaels points: The prototype repo is supposed to be used for > experimenting/prototyping and for that purpose only. As Dave pointed out > above, we'd want these experiments to happen with Apache visibility. There > will never be a completely working system available in that repository. The > goal of it is to find out if certain assumptions of a new design are > feasible or not. The goal is **not** to build productised components that > we can take over to the mainline. > > I always thought of that rearch as an incremental step as well and it > shouldn't need major changes to the API (or at least none that we are not > already introducing right now). I don't think we're going to see the true > potential of this though if we try to take the system's components as is > and reuse them in contexts they have never been built for in the first > place. > > As soon as we have clearance over which way we want to ultimately go with > the architecture, I agree we should come up with a nice migration plan and > start building it into the main repository through that migration plan. The > prototyping repository should vanish at some point. > > ### Summary ### > > I don't feel super strong about a choice of language, I just thought I'd > throw that in the ring as well. As sentiments seem quite strong against > doing that right now, which is perfectly fine, I'd be fine doing the > implementations in Scala. The disclaimer though is, that I'd like to write > critical components afresh to not have to fight with existing architectural > choices to much but to be able to think freely about the problems at hand. > > Which is why I do feel quite strongly about not doing this in a branch. The > structure of the project will be very different and I can easily see people > confused if we start having pull-requests against a "prototype" branch > mixed in with our usual pull-requests. > > I do however agree that we eventually will need to build this in an > incremental way, once we know what we want to build in the first place. For > me, it's off the table to attempt to rewrite the system from scratch, we'll > likely never get that done (I think that's version 2 syndrome Michael?). I > agree that might end up in a dead end and we shouldn't do that. > > Does that make the intention more clear? I hope I addressed all the points > asked. Opinions welcome :) > > Cheers, > Markus > > Am Mi., 29. Aug. 2018 um 00:55 Uhr schrieb Dascalita Dragos < > ddrag...@gmail.com>: > > > I also click on the clean and lean upgrade path, with gradual > improvements > > to the existing system (which is production-proof), as Michael and Rodric > > are suggesting. > > > > At the same time I see how Go would make sense from the Kubernetes and > > Knative impl POV, how the trends favor Go over Scala [1], [2], and the > > potential benefits from addressing a larger dev audience. @Markus, feel > > free to keep me honest if this is where you're coming from. > > > > Trying to keep our options open, I'm thinking: why not take Markus' > > challenge, and design these enhancements now to support polyglot > > implementations in the future ? This would be the important decision we > can > > take. BTW, for the data plane itself, we could go at even lower levels to > > Envoy or Nginx, to provide superior performance, should we choose to. > > > > [1] - > > > > > https://trends.google.com/trends/explore?geo=US&q=%2Fm%2F091hdj,%2Fm%2F09gbxjr > > > > [2] - https://octoverse.github.com/ > > > > On Tue, Aug 28, 2018 at 2:53 PM Rodric Rabbah <rod...@gmail.com> wrote: > > > > > Thanks Michael for raising these points. I share the same opinion and > > > sentiment and think a branch with a clean migration story is better and > > > makes more sense. I am not entirely convinced that the choice of > language > > > itself will make the difference vs the new architecture which is quite > > > different and should in itself be more efficient. > > > > > > -r > > > > > > On Tue, Aug 28, 2018 at 4:51 PM Michael Marth <mma...@adobe.com.invalid > > > > > wrote: > > > > > > > Hi Markus, > > > > > > > > IMHO what you propose below is a rather severe change in scope of > this > > > > discussion and effort. > > > > Up until so far this was about _evolving_ the OW architecture. We > have > > > not > > > > explicitly discussed it, but one could assume that it is at least > > > feasible > > > > to gradually adopt the new architecture. So there would be a smooth > > path > > > > between the current state of the code base and a future one. > > > > > > > > Your proposal below breaks this assumption somewhat (by proposing a > new > > > > repo instead of a branch - which will inevitably make the 2 code > bases > > > > drift apart) as well as explicitly by suggesting a new implementation > > > > language. Especially the latter would create a schism between OW-now > > and > > > > OW-future. > > > > This schism has implications like the perception of OW-now being > > > > deprecated, the _possibility_ of no clean upgrade path, the immediate > > > split > > > > of the community between *-now and *-future and of course carries the > > > risk > > > > of the version 2 syndrome. > > > > > > > > I would propose to implement the future architecture in a branch and > in > > > > Scala first. If it turns out to be good, then subsequent experiments > > can > > > > show or not-show if a switch of language is of additional value. That > > > would > > > > allow to make a decision based on data rather than anything else. > > > > > > > > My2c > > > > Michael > > > > > > > > > > > > On 28.08.18, 14:26, "Markus Thömmes" <markusthoem...@apache.org> > > wrote: > > > > > > > > Hi all, > > > > > > > > Am Mo., 27. Aug. 2018 um 20:04 Uhr schrieb David P Grove < > > > > gro...@us.ibm.com > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > "Markus Thömmes" <markusthoem...@apache.org> wrote on > 08/23/2018 > > > > 04:19:33 > > > > > PM: > > > > > > > > > > > > > > > > > Key point I want to make is: At some point we'll have to > start > > to > > > > > prototype > > > > > > things out and see if our assumptions actually hold water. > For > > > > example, > > > > > my > > > > > > assumption on a work-stealing backend is pretty much in the > > air. > > > > > > > > > > > > My proposal for going forward would be: > > > > > > 1. Create a playground for the implementation of some parts > of > > > the > > > > system > > > > > > (a new repository?) > > > > > > 2. Let's build some of the things that are uncontroversial > and > > > > absolutely > > > > > > needed in any case (ContainerRouter, ContainerManager). > > > > > > 3. Play around with them, hook them up in different ways, see > > > what > > > > works > > > > > > and what doesn't. > > > > > > > > > > > > Some things will need some testing out to see the scale that > > the > > > > > components > > > > > > can operate at. These things will narrow or widen the > solution > > > > space for > > > > > > the more controversial topics around how to distribute > > containers > > > > in the > > > > > > system, how to balance between the routers, work-stealing > > queue: > > > > yes/no > > > > > etc. > > > > > > > > > > > > Having some simple components fleshed out could encourage > > > > innovation and > > > > > > creates some facts that we need to focus things into a good > > > > direction. > > > > > > > > > > > > What do you think? Too early to start with this and/or the > > wrong > > > > way of > > > > > > doing it? > > > > > > > > > > > > > > > > +1 for starting to prototype. It's been a good discussion and > I > > > > think > > > > > we've identified some things that we know we don't know, so > time > > to > > > > > experiment and find out. > > > > > > > > > > > > > > > Not sure what the best logistics are for this. Would like the > > work > > > > to be > > > > > at Apache (community visibility). I'm not sure if the best way > > is > > > a > > > > new > > > > > repo or an experimental branch of the main repo (we could dial > > down > > > > the > > > > > level of testing on the branch to make it less cumbersome?). > The > > > > branch is > > > > > attractive to me because it might make it easier to keep in > synch > > > > with the > > > > > components we aren't changing. > > > > > > > > > > > > > I actually think we should generate a new repository for this. > > > Opening > > > > PRs > > > > etc. will then not clutter the "main" repository and we don't > need > > to > > > > worry > > > > about breaking anything. > > > > > > > > If nobody objects I'm going to get "incubator-openwhisk-protoype" > > > > generated > > > > and will create a rough outline in the repository (different > > folders > > > > for > > > > different parts of the system). > > > > > > > > One more basic question (and this is going to be controversial): > > Most > > > > of > > > > the componentry in the execution layer will have to be build > anew. > > > I'd > > > > like > > > > to switch the implementation language of at least the > > ContainerRouter > > > > to > > > > Golang. Why? Because scale/performance matters a lot for them. As > > > Dave > > > > mentioned on multiple occasions, it will greatly matter how many > > > > containers > > > > such a Router can handle under load and that scale will define > the > > > > implementation alternatives we will have at hand. I therefore > would > > > > like to > > > > optimise these Routers for minimal overhead. We could go even > lower > > > > level, > > > > but I guess that'd be at a kinda big maintenance cost. > > > > > > > > I can envision the ContainerManager to be simpler to implement in > > > > Golang as > > > > well, at least for some of the deployment alternatives > (Kubernetes > > > > comes to > > > > mind). The Golang based clients seemed superior to me vs. clients > > in > > > > any > > > > other language. > > > > > > > > As all of the communication will probably be HTTP only anyway, > > these > > > > implementations should be swappable anytime. > > > > > > > > Comments very welcome! Let me know your opinions. > > > > > > > > Cheers, > > > > Markus > > > > > > > > > > > > > > > > > >
