Mathieu, Damian, thanks a lot for your feedback. It's very valuable to see what, how and why people are using certain methods right now.
We don't want to lock people out (that's why we put this KIP on users list, too) and we want to keep the ability to mix-and-match DSL and Processor API. Furthermore, we actually plan another KIP to clean-up DSL. If we see that we put some limitation with this KIP, we can add missing features in this follow up KIP. We want to get both done before 0.10.3 so only people working on trunk would be affected for some time. - updated both #build() methods -- I also added one overload for each that was missing - adding KStreamBuilder#topologyBuilder() seems like be a good idea to address any concern with limited access to TopologyBuilder and DSL/PAPI mix-and-match approach. However, we should try to cover as much as possible with #process(), #transform() etc. - about TopologyBuilder#addInternalTopic(): I am not sure about this -- internal topics are a DSL concept to simplify the user code by avoiding manual calls to #through() -- thus, I would prefer to remove it from TopologyBuilder. Many methods are "potentially useful" -- as long as removing a method does not limit things that can be done, I rather remove them. TopologyBuilder API should give basic and complete API -- but no need for syntactic sugar IMHO. And Mathieu also stated "This could be worked around fairly easily." - about TopologyBuilder.nodeGroups & TopologyBuilder.build: not sure what analysis you do -- there is also KafkaStreams#toString() that describes the topology/DAG of the job. @Mathieu: Could you use this for your analysis? - KStreamBuilder.newName: similar argument as above -- if you use TopologyBuilder, you have to create your own names (there is no need that KStreamBuilder to provided helper methods for TopologyBuilder usage). And Mathieu also stated "Lacking this method would be fairly easy to work around". -Matthias On 2/6/17 8:07 AM, Damian Guy wrote: > Hi Matthias, > > Thanks for the KIP. Should TopologyBuilder#build() and > KStreamBuilder#build accept a StreamsConfig as an argument? > > Should we add a KStreamBuilder#topologyBuilder() for cases where people > want to mix and match DSL & PAPI? Or do you think we already provide enough > support for that via the DSL methods, i.e, KStream#process(...), > KStream.transform(...) etc? > > I wonder if we could leave TopologyBuilder#addInternalTopic() ? It seems > like it could be potentially useful? Perhaps the method could be renamed. > > Thanks, > Damian > > On Mon, 6 Feb 2017 at 15:39 Mathieu Fenniak <mathieu.fenn...@replicon.com> > wrote: > >> Hi Matthias, >> >> I use a few of the methods that you're pointing out that will be deprecated >> and don't have an apparent alternative, so I wanted to just let you know >> what they are and what my use-cases are for them. >> >> First of all, I use a combination of DSL and PAPI in the same application >> very happily. It looks like this API would not permit that at all... it >> looks like the intent here is to block that kind of usage, but as the DSL >> is not extensible (short of building on "internal" API classes and using >> reflection), this creates some real limitations in the type of applications >> that can be built. >> >> TopologyBuilder.addInternalTopic -- as we've discussed before, I have some >> reusable join components that build multiple processors w/ internal >> repartition topics. We use addInternalTopic to create application-id >> prefixed repartition topics. This could be worked around fairly easily. >> >> TopologyBuilder.nodeGroups & TopologyBuilder.build -- I have an option in >> my KS app to run once & output a graphviz document with my entire topology >> for debugging and analysis purposes; I use these methods to >> create ProcessorTopology instances to inspect the topology and create this >> output. I don't really see any alternative approach with this new API. >> >> KStreamBuilder.newName -- Similar to addInternalTopic, I use this to create >> processor names in reusable components. Lacking this method would be >> fairly easy to work around. >> >> Mathieu >> >> >> On Fri, Feb 3, 2017 at 4:33 PM, Matthias J. Sax <matth...@confluent.io> >> wrote: >> >>> Hi All, >>> >>> I did prepare a KIP to do some cleanup some of Kafka's Streaming API. >>> >>> Please have a look here: >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>> 120%3A+Cleanup+Kafka+Streams+builder+API >>> >>> Looking forward to your feedback! >>> >>> >>> -Matthias >>> >>> >> >
signature.asc
Description: OpenPGP digital signature
