First of all, discussion is welcome in this PR (ok to move to mailing list as well). My thoughts are not firm yet but I do believe we need it.
Here are my thoughts: Streamlet is not really the abstraction. My feeling is that Streamlet is good at the DAG layer but not flexible enough in the low level (operators). I would think it is like Scala vs Java(not the same, just some idea). Scala has the nice functional API but it is pretty useless in real life if procedural code is not allowed/supported. Two reasons: 1. Migration is one major reason. There are quite some existing topologies written in low level API (for heron and storm). Streamlet is only friendly to new users, existing code such as KafkaSpout (it is spout, but same issue) in storm and some ML bolts has to be rewritten to take the readability/maintainability advantages. 2. Bolt/Spout are more flexible. They can do a lot more than a function provided by Streamlet API (initialization, config, checkpoint, etc). For examples, the stateful processing and component configs, they are not supported currently by Streamlet and if we add the features, it is likely user has to provide extra functions as parameters and the Streamlet API would became more and more complicated. Streamlet API will evolve but supporting Bolt/Spout could give us a lot room to design a clean API. [ Full content available at: https://github.com/apache/incubator-heron/pull/3029 ] This message was relayed via gitbox.apache.org for [email protected]
