Re: detection of base64 encoded code fields

2020-04-20 Thread Markus Thömmes
I agree, I don't think it's possible to accurately tell base64 from some random string without input from the user. Am Sa., 18. Apr. 2020 um 23:46 Uhr schrieb Rodric Rabbah : > Actually my suggested decoding step doesn't work either. I don't think > there's a sound way to record the binary

Re: discuss whether need to consider prewarmedPool as consumed memory resources

2020-03-30 Thread Markus Thömmes
Hi, this does mean, that the prewarm pool's capacity cannot be reclaimed by other actions should the need arise, right? I think that's problematic if that notion isn't carried through into loadbalancing itself as AFAIK the controller currently does not take the prewarm pool into account at all.

Re: JDK 11

2020-01-30 Thread Markus Thömmes
FWIW, the UUID entropy workaround where not JVM specific IIRC but OS dependent, as we switched from random to urandom as a source of entropy. I don't feel strongly for either OpenJDK vs. OpenJ9, but it'd be nice to understand where the degradation comes from and thus make a deliberate decision to

Re: OpenWhisk project goodies

2019-12-21 Thread Markus Thömmes
Thanks for taking this on you Rodric. For new contributors it can really make all the difference to be welcomed into a community like this. Kudos to you! Happy holidays! Am Mi., 18. Dez. 2019 um 17:16 Uhr schrieb Rodric Rabbah : > As a project and community we have a lot to celebrate this year

Re: Road to Scala 2.13

2019-11-29 Thread Markus Thömmes
I to build the controller cluster. Am Do., 21. Nov. 2019 um 15:10 Uhr schrieb Carlos Santana < csantan...@gmail.com>: > Woot +1 > > - Carlos Santana > @csantanapr > > > On Nov 21, 2019, at 5:58 AM, David P Grove wrote: > > > >  > > > > &

Re: Pool slot was shut down

2019-11-27 Thread Markus Thömmes
The fix you shared seems to be only logging related so it seems that these logs are harmless? If we don't see any detrimental effects to actual usage I'd vote for waiting for the fix to be released to just shut down the noisy logging. If we see actual errors though, we of course need to look at

Re: Road to Scala 2.13

2019-11-20 Thread Markus Thömmes
Am Mi., 20. Nov. 2019 um 19:52 Uhr schrieb David P Grove : > > "Markus Thömmes" wrote on 11/20/2019 10:44:21 > AM: > > > > Hey fellow Openwhiskers, > > > > I somewhat fancy a little bit of Scala action and wanted to give updating > > Openwhisk to

Re: Extend the SPI entitlement interface

2019-11-03 Thread Markus Thömmes
Hi Steffen, the interface is already Future[Boolean], why is it not sufficient to fail the future if the network calls fail for any reason in this case? After all, that's what the Future's failure mode is all about. Or asked differently: What states does a failed Future represent today? Cheers,

Re: Action health checks

2019-10-30 Thread Markus Thömmes
ly for /run, will be handled by way of rescheduling back to > ContainerPool (/init should already be handled by retry for a time period). > > Thanks! > Tyson > > On 10/30/19, 7:03 AM, "Markus Thömmes" wrote: > > Increasing latency would be my biggest conce

Re: Action health checks

2019-10-30 Thread Markus Thömmes
Increasing latency would be my biggest concern here as well. With a health ping, we can't even be sure that a container is still healthy for the "real request". To guarantee that, I'd still propose to have a look at the possible failure modes and implement a retry mechanism on them. If you get a

Re: Action health checks

2019-10-28 Thread Markus Thömmes
Heya, thanks for the elaborate proposal. Do you have any more information on why these containers are dying off in the first place? In the case of Kubernetes/Mesos I could imagine we might want to keep the Invoker's state consistent by checking it against the respective API repeatedly. On

Re: A direction to take in a new component, "the scheduler"

2019-07-18 Thread Markus Thömmes
Hi, this sounds a lot like the "OpenWhisk 2.0" discussion we had in an older thread where I proposed an architectural overhaul. See this thread for context: https://lists.apache.org/thread.html/d19306dd976138a153f48d32c5a55f2853e4b8ff405fc46f7260e905@%3Cdev.openwhisk.apache.org%3E There have

Re: Execute OpenWhisk action via Ignite and Firecracker VM (pr #4556)

2019-07-18 Thread Markus Thömmes
Great stuff Chetan! The quick impl. also shows our abstractions kinda work. That makes me happy too :). Am Di., 16. Juli 2019 um 00:56 Uhr schrieb Carlos Santana < csantan...@gmail.com>: > Great work Chetan > > This must be a new record how fast to get a POC integrated in a few days > after

Re: Using Rust for the KnativeWhisk controller

2019-07-18 Thread Markus Thömmes
er provides enough isolation where having a jvm in a > container in a virtual machine is total nonsense... So there is a > comeback to "native" code > > Yes of course there is a library to make it dynamic, there are code > generator, there are tons of tools to overcome the l

Re: Using Rust for the KnativeWhisk controller

2019-07-18 Thread Markus Thömmes
e or > > modules needed. I haven't kept up to date with C++ since pre-'10, but > > I'd imagine that it's easier to learn Rust than the latest version of > > C++ at this point. > > > > On Wed, 17 Jul 2019 at 08:06, Michele Sciabarra > > wrote: > >

Re: Using Rust for the KnativeWhisk controller

2019-07-17 Thread Markus Thömmes
+1 to all arguments. Rust's barrier of entry is considerably higher than that of Scala even. As much as I like the language's design, from a community attraction point-of-view we should absolutely opt for go, especially for things that are built around Kubernetes. That's of course to be taken

Re: Maximum number of file descriptors available when running actions with managed runtimes

2019-07-10 Thread Markus Thömmes
Thanks as always for your very detailed writeup and analysis of the issue at hand! Great job Sven! It seems to me that we might want to look at analyzing our runtimes for the number of file-descriptors they open and make up for that for the user. The user should always get at least the amount of

Re: Allow temporal unused codes to include a big change.

2019-07-05 Thread Markus Thömmes
Continuous integration is a crucial aspect of a system used in production like OpenWhisk. As such, let's bite the bullet by merging things incrementally into master. That will also force you to think about migration on each step you take, which in the long run will pay off big time. Accumulating

Re: [VOTE] Apache OpenWhisk graduation to Top Level Project

2019-06-11 Thread Markus Thömmes
+1 Apache OpenWhisk should graduate. Thanks all for the great community and opportunity to work with such smart people! Am Mo., 10. Juni 2019 um 17:55 Uhr schrieb Justin Halsall < jus...@juice10.com>: > +1 OpenWhisk is ready to graduate! > > > On Jun 6, 2019, at 5:11 AM, Bertrand Delacretaz >

Re: A plan to (re) implement OpenWhisk on top of Knative

2019-05-20 Thread Markus Thömmes
ild API > documented in the website. Is Tekton "compatible" or it has a different API? > > > > > > -- > > Michele Sciabarra > > mich...@sciabarra.com > > > > - Original message - > > From: "Markus Thömmes" > > To:

Re: A plan to (re) implement OpenWhisk on top of Knative

2019-05-19 Thread Markus Thömmes
Hi Michele, thank you for the detailed writeup. A few thoughts inline: Am So., 19. Mai 2019 um 19:00 Uhr schrieb Michele Sciabarra < mich...@sciabarra.com>: > I have an idea for implementing a prototype of OpenWhisk on top of Knative. > > My basic ideas are: do not use any proxy, forwarding or

Re: [DISCUSS] graduation from the incubator

2019-03-19 Thread Markus Thömmes
+1. Make it happen! Am Di., 19. März 2019 um 07:02 Uhr schrieb Michele Sciabarra < mich...@sciabarra.com>: > Actually it looks to me pretty unusual that a project of this size has not > yet graduated. > > -- > Michele Sciabarra > mich...@sciabarra.com > > > > - Original message - >

Re: [Long] I implemented websocket support for the Golang and related runtimes:

2019-01-22 Thread Markus Thömmes
Hi, leaving a few remarks. Thanks as always for your hard work Michele, you're cranking out a lot of stuff! Great job! 1. On the initial problem statement (and for your book): Have you considered using an action with a very high concurrency setting for your problem? The usual issue with these

Re: Persistent Kafka Connections

2019-01-09 Thread Markus Thömmes
Hi, with Tyson's concurrency > 1 support we should be able to rehash the requirements here. A single action that supports a high concurrency on one shared connection should fulfill what's needed by the Kafka backend teams here. Outscaling a backend through a lot of functions is a common problem

Re: the behavior of batcher

2018-11-09 Thread Markus Thömmes
Hi Jiang, this is a fair question. The trade-off of using groupedWithin vs. batch is, that groupedWithin **always** adds some latency to the database commands, where batch only adds that latency on backpressure, as you've noticed. Cheers, Markus Am Fr., 9. Nov. 2018 um 04:39 Uhr schrieb 蒋鹏程 :

Re: Relieve CouchDB on high load

2018-10-23 Thread Markus Thömmes
Hi Christian, given the recent work towards getting logs to a separate store to relieve CouchDB and to make it possible to move it into a store that's more appropriate for logging load and some other bits that are already implemented (removing the DB based polling, being able to disable log

Re: Promises (or equivalent) in Python and GoLang

2018-09-17 Thread Markus Thömmes
Hi Michele, commenting only on Golang: While there are packages available that provide a similar API surface, they have very little adoption throughout the Golang community. In general, the mechanism to achieve concurrent processing in Golang is goroutines

Re: Notes+Video posted from today's Tech. Interchange call 2018-09-12

2018-09-13 Thread Markus Thömmes
Thanks Matt for putting this all together in lightning speed again! Very much appreciated! Am Mi., 12. Sep. 2018 um 20:07 Uhr schrieb Matt Rutkowski < mrutk...@us.ibm.com>: > Thanks Markus for moderating and special thanks to Tyson for leading us on > the interesting topic of concurrent actions

Re: How to pass size the max size of the request to the runtime?

2018-09-11 Thread Markus Thömmes
Heya, I tend to agree with Felix. Buffers are usually just used as a transport mechanism and they shouldn't need to fit the whole response at once (which doesn't imply we don't load the response in memory, but at least that input buffer doesn't need to be == content-length). Could you give a bit

Re: How to do basic validation of custom OpenWhisk deployment

2018-09-05 Thread Markus Thömmes
Huge +1 from my end! I always thought of the "system.*" suite to cover exactly what you're looking for, but we've been lax with regards to splitting test-suites and the properties needed as you're experiencing now. It'd be great to clean that up and put tests in appropriate packages to be able to

Re: Prototyping for a future architecture

2018-08-29 Thread Markus Thömmes
t; issues I've found in previous threads and may be solved in mind between > folks however.) > > [1] https://tz70s.github.io/posts/openwhisk-performance-improvement/ > > Thanks, > Tzu-Chiao > > On Wed, Aug 29, 2018 at 11:31 AM Markus Thömmes > > wrote: > >

Re: Prototyping for a future architecture

2018-08-28 Thread Markus Thömmes
gt; > > deprecated, the _possibility_ of no clean upgrade path, the immediate > > split > > > of the community between *-now and *-future and of course carries the > > risk > > > of the version 2 syndrome. > > > > > > I would propose to implement the fu

Re: Bi-weekly Tech Interchange call tomorrow

2018-08-28 Thread Markus Thömmes
Hi Tyson, shall we discuss the mechanics on how to start the prototype for the future-arch discussion? Cheers, Markus Am Di., 28. Aug. 2018 um 16:59 Uhr schrieb Tyson Norris : > Hi Whiskers! > > Please send any agenda items you would like to discuss at the Tech > Interchange call tomorrow. >

Re: Prototyping for a future architecture

2018-08-28 Thread Markus Thömmes
Hi all, Am Mo., 27. Aug. 2018 um 20:04 Uhr schrieb David P Grove : > > > > "Markus Thömmes" wrote on 08/23/2018 04:19:33 > PM: > > > > > Key point I want to make is: At some point we'll have to start to > prototype > > things out and see if our a

Re: [Discussion] Use TransactionID as ActivationID

2018-08-27 Thread Markus Thömmes
Thanks for bringing that up Rodric, it seems like we could have a one stone two birds thing going on here? Do you see an immediate reason why the proposal I made would not work? We'd certainly need to check if any client hardcodes a fixed length for our activation ids. Am Di., 28. Aug. 2018 um

Re: [Discussion] Use TransactionID as ActivationID

2018-08-27 Thread Markus Thömmes
enerate a child id for that invocation. If we can make that readable, we can easily correlate the parent with the child, giving us even more of a gain from this proposal. So from my PoV: Go for it! > > Thanks > Best regards > Dominic. > > > 2018년 8월 23일 (목) 오후 5:34, Markus

Re: Kafka and Proposal on a future architecture of OpenWhisk

2018-08-23 Thread Markus Thömmes
Hi, Am Fr., 24. Aug. 2018 um 00:07 Uhr schrieb Tyson Norris : > > Router is not pulling at queue for "specific actions", just for any > action > > that might replace idle containers - right? This is complicated with > > concurrency though since while a container is not idle (paused +

Prototyping for a future architecture

2018-08-23 Thread Markus Thömmes
Hi OpenWhiskers, We've been discussing a new direction for our architecture based on a proposal I made here: https://cwiki.apache.org/confluence/display/OPENWHISK/OpenWhisk+future+architecture Discussion threads: -

Re: Kafka and Proposal on a future architecture of OpenWhisk

2018-08-23 Thread Markus Thömmes
Hi Dave, I agree! I'll start another thread on a discussion of how/where we prototype things to hash out some of the unknowns. Cheers, Markus Am Do., 23. Aug. 2018 um 22:05 Uhr schrieb David P Grove : > > Related to the random vs. smart routing discussion. > > A key unknown that influences the

Re: Kafka and Proposal on a future architecture of OpenWhisk

2018-08-23 Thread Markus Thömmes
Hi Tyson, Am Do., 23. Aug. 2018 um 21:28 Uhr schrieb Tyson Norris : > > > > And each ContainerRouter has a queue consumer that presumably pulls > from > > the queue constantly? Or is consumption based on something else? If > all > > ContainerRouters are consuming at the same

Re: Simplifying configuring OpenWhisk Components (#3984)

2018-08-23 Thread Markus Thömmes
igure-pod-configmap/#populate-a-volume-with-data-stored-in-a-configmap > > On Thu, Aug 23, 2018 at 1:32 PM Markus Thömmes > wrote: > > > Hi Chetan, > > > > good idea! > > > > A bit of background on why it is how it is: When I implemented the > approach

Re: [Discussion] Use TransactionID as ActivationID

2018-08-23 Thread Markus Thömmes
Hi Dominic, this is certainly a good thought and would simplify debugging production systems as well. A rough and dirty idea for the uniqueness issue you mentioned: What if we just append a suffix for each of the child ids (essentially they are in a parent/child relationship). A simple example:

Re: Kafka and Proposal on a future architecture of OpenWhisk

2018-08-23 Thread Markus Thömmes
Hi Tyson, Am Do., 23. Aug. 2018 um 00:33 Uhr schrieb Tyson Norris : > Hi - thanks for the discussion! More inline... > > On 8/22/18, 2:55 PM, "Markus Thömmes" wrote: > > Hi Tyson, > > Am Mi., 22. Aug. 2018 um 23:37 Uhr schrieb Ty

Re: Simplifying configuring OpenWhisk Components (#3984)

2018-08-23 Thread Markus Thömmes
Hi Chetan, good idea! A bit of background on why it is how it is: When I implemented the approach we're having today, the basic thought was to be able to detect on a quite granular level what changes are needed to which components, even when they share values. For example: In the Kubernetes

Re: Kafka and Proposal on a future architecture of OpenWhisk

2018-08-22 Thread Markus Thömmes
Hi Tyson, Am Mi., 22. Aug. 2018 um 23:37 Uhr schrieb Tyson Norris : > Hi - > > > > When exactly is the case that a ContainerRouter should put a blocking > > activation to a queue for stealing? Since a) it is not spawning > containers > > and b) it is not parsing request/response

Re: Kafka and Proposal on a future architecture of OpenWhisk

2018-08-22 Thread Markus Thömmes
nces to the Routers it gave Containers to the Routers that have none. (This is the edge-case described in the proposal). The work-stealing queue though is used to rebalance work in case one of the Routers get overloaded. > > Thanks > Tyson > > On 8/21/18, 1:16 AM, "Markus Th

Re: Guide to conduct performance benchmark against OpenWhisk

2018-08-21 Thread Markus Thömmes
Hi Dominic, it seems like you haven't published your writeup? The page seems empty. Cheers, Markus Am Di., 21. Aug. 2018 um 12:44 Uhr schrieb Dominic Kim : > Dear whiskers. > > I just wrote the guide to perform a throughput test against Openwhisk in > cwiki. >

Re: Kafka and Proposal on a future architecture of OpenWhisk

2018-08-21 Thread Markus Thömmes
ontainerRouters that receive all async activations, and others that > receive all sync activations, but the end result is the same, I think. > > > > On Aug 19, 2018, at 4:29 AM, Markus Thömmes > wrote: > > > > Hi Tyson, Carlos, > > > > FWIW I should change th

Re: Proposal on a future architecture of OpenWhisk

2018-08-21 Thread Markus Thömmes
Hi Tyson, Am Di., 21. Aug. 2018 um 00:20 Uhr schrieb Tyson Norris : > > > On Aug 19, 2018, at 3:59 AM, Markus Thömmes <mailto:markusthoem...@apache.org>> wrote: > > Hi Tyson, > > Am Fr., 17. Aug. 2018 um 23:45 Uhr schrieb Tyson Norris > mailto:tnor...@adobe.com

Re: Proposal on a future architecture of OpenWhisk

2018-08-20 Thread Markus Thömmes
Am So., 19. Aug. 2018 um 18:59 Uhr schrieb TzuChiao Yeh < su3g4284zo...@gmail.com>: > On Sun, Aug 19, 2018 at 7:13 PM Markus Thömmes > wrote: > > > Hi Tzu-Chiao, > > > > Am Sa., 18. Aug. 2018 um 06:56 Uhr schrieb TzuChiao Yeh < > > su3g4284zo...@gmail.c

Re: Kafka and Proposal on a future architecture of OpenWhisk

2018-08-20 Thread Markus Thömmes
becoming optional, while opening the door for > other connectors like AWS Kinesis, Azure Event Hub, and others (see the > link at [1] for a more complete list of connectors ) > > [1] - https://developer.lightbend.com/docs/alpakka/current/ > On Sun, Aug 19, 2018 at 7:30 AM Markus T

Re: Concurrency PR

2018-08-19 Thread Markus Thömmes
Hi Tyson, thanks for pushing forward on this! I'll try to get a review in on it soon. Am Fr., 17. Aug. 2018 um 19:04 Uhr schrieb Tyson Norris : > Hi - > I have been noodling with a few tests and the akka http client and gotten > the concurrency PR [1] to a good place, I think, so if anyone can

Re: Kafka and Proposal on a future architecture of OpenWhisk

2018-08-19 Thread Markus Thömmes
Hi Tyson, Carlos, FWIW I should change that to no longer say "Kafka" but "buffer" or "message queue". I see two use-cases for a queue here: 1. What you two are alluding to: Buffering asynchronous requests because of a different notion of "latency sensitivity" if the system is in an overload

Re: Proposal on a future architecture of OpenWhisk

2018-08-19 Thread Markus Thömmes
Hi Tzu-Chiao, Am Sa., 18. Aug. 2018 um 06:56 Uhr schrieb TzuChiao Yeh < su3g4284zo...@gmail.com>: > Hi Markus, > > Nice thoughts on separating logics in this revision! I'm not sure this > question has already been clarified, sorry if duplicate. > > Same question on cluster singleton: > > I think

Re: Proposal on a future architecture of OpenWhisk

2018-08-19 Thread Markus Thömmes
Hi Tyson, Am Fr., 17. Aug. 2018 um 23:45 Uhr schrieb Tyson Norris : > > If the failover of the singleton is too long (I think it will be based on > cluster size, oldest node becomes the singleton host iirc), I think we need > to consider how containers can launch in the meantime. A first step

Re: Proposal on a future architecture of OpenWhisk

2018-08-17 Thread Markus Thömmes
Hi Tyson, thanks for the great input! Am Do., 16. Aug. 2018 um 23:14 Uhr schrieb Tyson Norris : > Thinking more about the singleton aspect, I guess this is mostly an issue > for blackbox containers, where manifest/managed containers will mitigate at > least some of the singleton failure delays

Re: Proposal on a future architecture of OpenWhisk

2018-08-15 Thread Markus Thömmes
also for > operational concerns). > What are your thoughts about losing a shard (planned or crashed) or adding > a shard? > > Michael > > > On 15.08.18, 09:58, "Markus Thömmes" wrote: > > Hi Dragos, > > thanks for your questions, good discussi

Re: Proposal on a future architecture of OpenWhisk

2018-08-15 Thread Markus Thömmes
is encapsulated in the ContainerManager. Cheers, Markus Am Mi., 15. Aug. 2018 um 10:35 Uhr schrieb Bertrand Delacretaz < bdelacre...@apache.org>: > Hi, > > On Tue, Aug 14, 2018 at 4:07 PM Markus Thömmes > wrote: > ... > > > https://cwiki.apache.org/confluence/disp

Re: Proposal on a future architecture of OpenWhisk

2018-08-15 Thread Markus Thömmes
ocs/akka/2.5/distributed-data.html > > > ____ > From: David P Grove > Sent: Tuesday, August 14, 2018 2:15:13 PM > To: dev@openwhisk.apache.org > Subject: Re: Proposal on a future architecture of OpenWhisk > > > > > "Markus Thöm

Re: Proposal on a future architecture of OpenWhisk

2018-08-15 Thread Markus Thömmes
Hi Dave, thanks a lot for your input! Greatly appreciated. Am Di., 14. Aug. 2018 um 23:15 Uhr schrieb David P Grove : > > > > "Markus Thömmes" wrote on 08/14/2018 10:06:49 > AM: > > > > I just published a revision on the initial proposal I made. I sti

Proposal on a future architecture of OpenWhisk

2018-08-14 Thread Markus Thömmes
Hey OpenWhiskers, I just published a revision on the initial proposal I made. I still owe a lot of sequence diagrams for the container distribution, sorry for taking so long on that, I'm working on it. I did include a clear seperation of concerns into the proposal, where user-facing abstractions

Re: BDD Test Cases Contribution to complement existing test cases and coverage

2018-08-13 Thread Markus Thömmes
Hi Rahul, thanks a lot for the contribution and the work you've put into this! Very much appreciated. I'm not familiar with Karate or BDD at all. Could you go into a little bit more detail on what the benefit of this approach are and why we should consider doubling our test-coverage (and thus

Re: System env vars in user containers

2018-08-06 Thread Markus Thömmes
I guess this is the famous "context" object, that some other platforms use in their function signature? @Chetan is that what you're referring to in option 2? Am Mo., 6. Aug. 2018 um 10:58 Uhr schrieb Chetan Mehrotra : > > > What are you thinking as an alternarive? > > There can be 2 ways > > 1.

Re: System env vars in user containers

2018-08-06 Thread Markus Thömmes
Ah yeah! Dividing /init and /run time provided parameters makes a lot of sense to minimize potential performance impact! +1 on that. Am Mo., 6. Aug. 2018 um 10:37 Uhr schrieb Rodric Rabbah : > > +1 for backward compatibility. > > I would also provide the env vars at init time. (Currently they’re

Re: System env vars in user containers

2018-08-06 Thread Markus Thömmes
Hi Vadim, thanks for bringing this up! If I were to design a new system from the ground up, I'd definitely introduce a new structure to have a good distinction between environment and parameters. Taking into consideration that we need to be backwards compatible, I'd definitely choose option 1.

Re: Update Kafka to 2.0.0

2018-07-31 Thread Markus Thömmes
ou know if the Kafka client > library is able to handle both versions ? > > Thanks, > Dragos > On Mon, Jul 30, 2018 at 12:57 PM Markus Thömmes > wrote: > > > Dear OpenWhiskers, > > > > James Dubee, Sven Lange-Last and I have been on a Kafka-related bug > &g

Re: Tech Interchange meeting Wednesday

2018-07-30 Thread Markus Thömmes
Thanks Justin, Yes you can put me on the agenda on knative. Consider it a shared "Ben Browning/Markus Thömmes - Update on Knative" please! Cheers, Markus Am Mo., 30. Juli 2018 um 23:39 Uhr schrieb Justin Halsall : > > Hi everyone, > > > Reminder that the bi-weekly ca

Update Kafka to 2.0.0

2018-07-30 Thread Markus Thömmes
Dear OpenWhiskers, James Dubee, Sven Lange-Last and I have been on a Kafka-related bug hunt throughout the past days and found several issues in our current KafkaConsumerConnector. Several PRs have been opened since and several are yet to come. Currently, I'm trying to make the Kafka 2.0.0

Re: Proposal on a future architecture of OpenWhisk

2018-07-25 Thread Markus Thömmes
Hi David, the problem is, that if you have N controllers in the system and M containers but N > M and all controllers manage their containers exclusively, you'll end up with controllers not having a container to manage at all. There's valid, very slow workload that needs to create only 1

Re: Knative, Kubernetes, and Apache OpenWhisk

2018-07-25 Thread Markus Thömmes
Hi Ben, thanks for the write-up! For full disclosure to the rest of the dev-list: I've been involved with Knative and have contributed some bits especially to their autoscaling strategy. The usual word of caution: Knative is in its very early stages (which is actually a good thing, because we

Re: Invoker activation queueing proposal

2017-10-05 Thread Markus Thömmes
The overflow queue is not intended to be drained by a large scale of invokers but by a relatively low scale of loadbalancers! I don‘t think rebalancing is a problem then. That loadbalancer basically has 2 states: 1. capacity is there: schedule everything you get and put it on immediate

Re: Distributed Tracing support PR

2017-09-26 Thread Markus Thömmes
Hi, I think OpenTracing will be a great way to implement this, so we don't need to write SPIs for every tracing backend there is but OpenTracing already handles that for us. +1 Cheers, Markus Am 26. September 2017 um 08:20 schrieb "sandy...@gmail.com" : Hi, sometime

Re: PR feedback for SPIs

2017-09-13 Thread Markus Thömmes
Hi Tyson, first of all: Thanks for the great effort put into all of these SPI implementations. Here are some general comments from myself on SPIs The ContainerFactory SPI feels natural and makes a lot of sense to support different deployment models. There is no doubt about that one from my

Re: Moving out runtime images and tests

2017-09-12 Thread Markus Thömmes
Sounds all sensible to me. To 1: Sounds like the right thing to do, if versions are common enough? Like major bumps could probably use different repos but I guess we can just try and find out what works. As a starter, the list you posted sounds sensible to me! 2. By Integration Tests I meant:

Moving out runtime images and tests

2017-09-12 Thread Markus Thömmes
Hey all, again we're hitting the Travis build time limit of 50 minutes and we need to disable more tests again to bring us in good shape there. This is not really acceptable and it's really painful for me to turn down testcoverage. A thought has been lingering for some time now and with the

Re: Warm containers

2017-09-06 Thread Markus Thömmes
which would involve a request payload passing through a sequence of actions. So I have a mixed bag of heterogeneous micro services I want to mimic as actions. I had already gone through your blog earlier. On Wed, Sep 6, 2017 at 11:27 AM, Markus Thömmes <markusthoem...@me.com> wrote:

Re: Warm containers

2017-09-05 Thread Markus Thömmes
Hi Mandeep, OpenWhisk relies on container reuse (i.e. using warm containers) heavily to reach its performance goals. Currently, containers are kept around for a maximum of 10 minutes if they are not replaced by the need of some other container. You can refer to this article written by me

Today - Join us for the OpenWhisk “Tech. Interchange” (bi-weekly) Zoom Meeting

2017-08-02 Thread Markus Thömmes
. Introduction/New faces on the call? 2. What's new? Short status updates depending on who's joining the call. 3. Topics:   3.1: Nick Mitchell - OpenWhisk Shell   3.2: Markus Thömmes - Invoker Reactive: What's the fuzz and how does it work?   3.3: Anything else you are interested in! 4. Confirm

Deprecation of the old containerpool

2017-07-26 Thread Markus Thömmes
Hi all, development of the reactive containerpool is finished and it's been tested thoroughly in our (IBM's) pipeline. It is used in the Bluemix hosted OpenWhisk deployment for a while now which makes me believe we should start the deprecation process for the "old" containerpool since it's

Re: Improving support for UI driven use cases

2017-07-02 Thread Markus Thömmes
Right, I think the UI workflows are just an example of apps that are latency sensitive in general. I had a discussion with Stephen Fink on the matter of detecting ourselves that an action is latency sensitive by using the blocking parameter or as mentioned the user's configuration in terms of

Re: Improving support for UI driven use cases

2017-07-01 Thread Markus Thömmes
Thanks for the vry detailed writeup! Great job! One thing I'd note: As Rodric pointed out we should break the issues apart and address them one by one. For instance the proposed loadbalancer changes (Controllers knows of Containers downstream) is desirable for each workload and not

Loadbalancer Improvements

2017-06-12 Thread Markus Thömmes
Hey folks, it's me again with the latest news on performance :). As some of you probably now: Our current loadbalancer strategy is quite "simple" and doesn't take load in the system into account at all. It hops to the next available invoker after you've invoked an action X times (where X is a

Re: Red Hat and Apache OpenWhisk

2017-05-31 Thread Markus Thömmes
Hi Ben, That sounds awesome :). Welcome to the family and looking forward to working with all of you. Cheers, Markus Von meinem iPhone gesendet > Am 31.05.2017 um 22:59 schrieb Ben Browning : > > My name is Ben Browning and I lead the Serverless and >

Re: performance tests

2017-05-09 Thread Markus Thömmes
Hi Tyson, awesome, we had the same train of thought :). I looked into Gatling as well, mainly because it's in Scala so it'd be easy for us to read and build new scenarios. Gatling also has a Jenkins plugin (we use Jenkins) which makes it very easy to integrate test results. I also liked how

Re: concurrent requests on actions

2017-05-01 Thread Markus Thömmes
Hi Tyson, Sounds like you did a lot of investigation here, thanks a lot for that :) Seeing the numbers, 4 RPS in the "off" case seem very odd. The Travis build that runs the current system as is also reaches 40+ RPS. So we'd need to look at a mismatch here. Other than that I'd indeed suspect

Re: Performance tests for OpenWhisk

2017-04-26 Thread Markus Thömmes
Hi Michael, yeah that sounds pretty much spot on. I'd like to have at least 2 VMs with 4+ cores and 8GB memory. One VM would host the management stack while one would be dedicated to an Invoker only. That way we could assert single-invoker performance the easiest. Thanks for helping!

Performance tests for OpenWhisk

2017-04-25 Thread Markus Thömmes
Hi all, Since many have asked I today started an effort to bring a comprehensive performance test suite for OpenWhisk out in the open. Without much talking, here's the repository:  https://github.com/markusthoemmes/openwhisk-performance My initial idea was to have the tests run on Travis to

Re: factoring out "runtimes" into deployment configuration

2017-03-05 Thread Markus Thömmes
Good one! Another vector where this will improve the system in general is the ability to run a minimized set of tests for our core components and thus reducing feedback time. - mt Von meinem iPhone gesendet > Am 05.03.2017 um 17:39 schrieb Rodric Rabbah : > > I've been

Re: Allow actions to be accessible from a web browser

2017-01-13 Thread Markus Thömmes
ts use may be somewhat constrained (until we have streaming?) by any payload limitations a whisk installation might have in place. On Thu, Jan 12, 2017 at 5:51 PM, Markus Thömmes <markusthoem...@me.com> wrote: Haven't looked at the implementation yet but I really dig the idea! Are query pa