2020-01-03 05:22:42 UTC - chetanm: Off late finding quite a bit of reference for Apache Pulsar [1] with most recent [2] providing a perspective from an actual user. This may be a good option to try given its pub-sub support along with support for large number of topics. That would allow us to support a mode similar to what Dominic proposed [4] a while back.
Compared to Kafka it has though more moving parts (brokers + book keeper for storage) but looks like it is designed for k8s kind of deployment which can relieve some of the deployment complexity. May be using this along with moving to a pull model by Action containers (like used in AWS Lambda [3]) using HTTP api may allow better throughput for some workloads and remove invoker from the critical path specially for web actions Something to explore in 2020!! [1]: <https://pulsar.apache.org/> [2]: <https://news.ycombinator.com/item?id=21936252> [3]: <https://docs.aws.amazon.com/lambda/latest/dg/runtimes-api.html> [4]: <https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduler+v2> +1 : Dominic Kim, Bill Zong, Neeraj Mangal https://openwhisk-team.slack.com/archives/C3UDXSFA6/p1578028962005700?thread_ts=1578028962.005700&cid=C3UDXSFA6 ---- 2020-01-03 15:03:53 UTC - Dominic Kim: FTR, we here implemented our own queue to solve the issue. https://openwhisk-team.slack.com/archives/C3UDXSFA6/p1578063833006600?thread_ts=1578028962.005700&cid=C3UDXSFA6 ----
