Sven, all,
That's a good call-out.
In consequence to that: how about we discuss larger changes with
a) a test script that simulates the types of load under discussion (and at the
same time makes it clear which types of load are ignored for now) and
b) a declaration of the KPIs one seeks to improve. Could be performance,
scalability, cold start latency, cost, etc. This also makes it clear which
other KPIs are disregarded or assumed to be ok to deteriorate in the proposal.
I think that might make it easier to discuss the relative merits of a change.
Cheers
Michael
On 18.07.19, 19:49, "Sven Lange-Last" <sven.lange-l...@de.ibm.com> wrote:
Hello Dominic and other discussion participants,
we are in the great situation that Dominic (style95) volunteers to create
and verify an alternative OpenWhisk architecture that may solve some of
the problems that adopters of OpenWhisk may observe when running
production systems with high utilization. I really appreciate that Dominic
already invested and is willing to invest so much time.
While I encourage Dominic to experiment with a different architecture, my
point of view is that it's in the interest of the project as well as all
adopters running OpenWhisk-based production systems to keep the project
stable. We need an approach where we can continue to evolve the existing
architecture and provide bug fixes without being slowed down or affected
by the work on or problems with the new architecture. At the same time, we
need a playground where we can work on the new architecture effectively
without negatively affecting the existing architecture.
For me, this means:
* We need feature flags / switches as suggested by Dominic and others that
keep the existing architecture enabled by default.
* The standard deployment only configures and installs what is needed for
the existing architecture. Installing extra components like etcd takes
more time so that development, build and test cycles take longer. In
addition, extra components consume more resources so that brittle system
tests for the existing architecture may fail more often.
* We need a good separation between tests for existing and new
architecture. System tests are often brittle and need special resources
like etcd. Such tests for the new architecture must not be part of the
existing gradle suites. I think that stable, fast-running unit tests can
be part of existing suites.
* The Travis checks for pull requests must not contain system tests or
other brittle / long-running tests of the new architecture. We already
have some brittle tests today that occasionally fail Travis. As Dominic
suggested, extra build / deployment / test jobs that run separately would
be a great idea.
I would like to add a different aspect...
My experience is that the observed problems of an OpenWhisk based system
really depend on utilization and the workload mix (memory limits, action
duration, action container image, number of distinct actions, ...). And I
would expect that people running a few activations will have totally
different observations than those people running a system with hundreds of
invokers and millions of activations with a diverse workload mix. And
based on the observed problems, different contributors will probably try
to solve different problems and set different priorities.
I had a great discussion with Dominic on these aspects. It turned out that
he observes different problems than I do when running a large production
environment. I'm very interested in Dominic's proposal - at the same time,
my impression is that it will not necessarily address the problems I see.
Still, we can learn a lot from Dominic's work.
At some point in time, the OpenWhisk community probably needs to decide
whether to keep the existing architecture and evolve it - or to replace it
with the suggested architecture if it proves useful. My impression is that
Dominic only received limited feedback on his proposal and I'm guilty of
providing my feedback much too late. We should discuss the problems we
observe with running OpenWhisk more openly so that contributors can align
their work with the need of others.
Mit freundlichen Grüßen / Regards,
Sven Lange-Last
Senior Software Engineer
IBM Cloud Functions
Apache OpenWhisk
E-mail: sven.lange-l...@de.ibm.com
Find me on:
Schoenaicher Str. 220
Boeblingen, 71032
Germany
IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294
