It really depends on the skill level of the developer. Using low-level API requires to think about many details (eg. state handling etc.) that could be done wrong.
As Flink gets a broader community, more people will use it who might not have the required skill level to deal with low-level API. For more trained uses, it is of course a powerful tool! I guess it boils down to the question, what type of developer Flink targets, if low-level API should be offensive advertised or not. Also keep in mind, that many people criticized Storm's low-level API as hard to program etc. -Matthias On 08/15/2016 07:46 AM, Gyula Fóra wrote: > Hi Jamie, > > I agree that it is often much easier to work on the lower level APIs if you > know what you are doing. > > I think it would be nice to have very clean abstractions on that level so > we could teach this to the users first but currently I thinm its not easy > enough to be good starting point. > > The user needs to understand a lot about the system if the dont want to > hurt other parts of the pipeline. For insance working with the > streamrecords, propagating watermarks, working with state internals > > This all might be overwhelming at the first glance. But maybe we can slim > some abstractions down to the point where this becomes kind of the > extension of the RichFunctions. > > Cheers, > Gyula > > On Sat, Aug 13, 2016, 17:48 Jamie Grier <ja...@data-artisans.com> wrote: > >> Hey all, >> >> I've noticed a few times now when trying to help users implement particular >> things in the Flink API that it can be complicated to map what they know >> they are trying to do onto higher-level Flink concepts such as windowing or >> Connect/CoFlatMap/ValueState, etc. >> >> At some point it just becomes easier to think about writing a Flink >> operator yourself that is integrated into the pipeline with a transform() >> call. >> >> It can just be easier to think at a more basic level. For example I can >> write an operator that can consume one or two input streams (should >> probably be N), update state which is managed for me fault tolerantly, and >> output elements or setup timers/triggers that give me callbacks from which >> I can also update state or emit elements. >> >> When you think at this level you realize you can program just about >> anything you want. You can create whatever fault-tolerant data structures >> you want, and easily execute robust stateful computation over data streams >> at scale. This is the real technology and power of Flink IMO. >> >> Also, at this level I don't have to think about the complexities of >> windowing semantics, learn as much API, etc. I can easily have some inputs >> that are broadcast, others that are keyed, manage my own state in whatever >> data structure makes sense, etc. If I know exactly what I actually want to >> do I can just do it with the full power of my chosen language, data >> structures, etc. I'm not "restricted" to trying to map everything onto >> higher-level Flink constructs which is sometimes actually more complicated. >> >> Programming at this level is actually fairly easy to do but people seem a >> bit afraid of this level of the API. They think of it as low-level or >> custom hacking.. >> >> Anyway, I guess my thought is this.. Should we explain Flink to people at >> this level *first*? Show that you have nearly unlimited power and >> flexibility to build what you want *and only then* from there explain the >> higher level APIs they can use *if* those match their use cases well. >> >> Would this better demonstrate to people the power of Flink and maybe >> *liberate* them a bit from feeling they have to map their problem onto a >> more complex set of higher level primitives? I see people trying to >> shoe-horn what they are really trying to do, which is simple to explain in >> english, onto windows, triggers, CoFlatMaps, etc, and this get's >> complicated sometimes. It's like an impedance mismatch. You could just >> solve the problem very easily programmed in straight Java/Scala. >> >> Anyway, it's very easy to drop down a level in the API and program whatever >> you want but users don't seem to *perceive* it that way. >> >> Just some thoughts... Any feedback? Have any of you had similar >> experiences when working with newer Flink users or as a newer Flink user >> yourself? Can/should we do anything to make the *lower* level API more >> accessible/visible to users? >> >> -Jamie >> >
signature.asc
Description: OpenPGP digital signature
