Regarding (2), I think it would be nice to expand the "Trident spouts" section of the manual ( http://storm.incubator.apache.org/documentation/Trident-spouts.html), since Trident internals are practically undocumented (no Javadocs, etc. compared to Storm itself). For example, there's no documentation on the Coordinator interface. Also, the new Storm website under Incubator doesn't seem to have an publicly editable wiki, it would be very useful to have one.
On Mon, Mar 31, 2014 at 2:08 PM, David Smith <[email protected]>wrote: > Maybe this email got lost among a bunch of commits. But I hope that you > maybe able to give us some insight where you intend to take Storm. > > Thank you, > Peter > > > On Wed, Mar 26, 2014 at 4:30 PM, David Smith <[email protected] > >wrote: > > > Hello, > > > > I would like some clarity from committers about certain features and your > > intentions about their future direction going forward. > > > > 1. Transactional topologie / Linear DRPC Topologies. For the longest > time, > > this has been informally described as deprecated. But the codebase for > this > > is still not marked as deprecated. Can we have some guidance on whether > > this code will live on and if for how long and when it should be > preferred > > over Trident (what Trident cannot accomplish that this can)? If it's not > > intended to live on, then may I suggested that you deprecate the code > > initially and mark it for removal during subsequent release? > > > > 2. Trident. There's a lot of good documentation on how to write a Trident > > topology but there's hardly any documentation on how to write a Trident > > spout. People waste a lot of time attempting to write a Trident spout for > > their data sources and give up because it's unclear and all their > > assumptions they will bring forward from their knowledge of standard > Storm > > Spouts are not valid for Trident Spouts. Bringing the Kafka spout into > > storm project will address some of these concerns as Trident is fully > > supported under Kafka. But it will be useful to know the operational > > characteristics that need to be supported by the data source in order to > > support a Trident spout. It would be quite helpful to see a simple > > implementation of a Trident Spout to serve as guidance for writing a > > Trident spout. > > > > 3. Trident currently supports building Trident spout based on an existing > > IRichSpout's with the understanding that they will only be > > non-transactional within Trident. But lot of IRichSpouts cannot be used > in > > this manner as they expect to receive ack's and fail's. That IRichSpout > > contract is not enforced by Trident (understandably) but it would be much > > easier for developers to grok this if there was a common API contract > that > > IRichSpouts that work with Trident can implement (even if only in a > > non-transactional manner). > > > > 3. DRPC support in regular Storm without Trident. From what I understand > > this is not possible right now without using the deprecated > > LinearDRPCTopologyBuilder. > > > > 4. Tick Tuple support in Trident > > > > 5. Storm internals commonly being used by users to develop features. Like > > CoordinatedBolt, RotatingMap, etc. These can be highly valuable and we > > might want to consider making them a part of the core API for advanced > > users and document how they should be used. > > > > 6. IBatchBolt. Can this be used without Transactional Topologies? > > > > 7. Different stream groupings other than shuffle, field may require > better > > documentation before users can understand how to use them. > > > > Once again, this is not a complaint but a plea for clarity about future > > direction. I understand that some of these are not currently supported > and > > it may be while before they will be (if ever). That is fine, but now that > > Storm is incubating under Apache, it would be nice to have a roadmap that > > indicates the direction that Storm is heading towards. > > > > Thank you, > > David > > > -- Danijel Schiavuzzi E: [email protected] W: www.schiavuzzi.com T: +385989035562 Skype: danijels7
