Being relatively new to Storm before I came to jwplayer, SQE has made it extremely easy for me to pick up and hit the ground running. The code is robust, and I look forward to contributing to the project to ensure its continued success.
On Fri, Sep 2, 2016 at 9:49 AM, Lee Morris <l...@jwplayer.com> wrote: > Hi, Storm Dev! > > I wanted to chime in to show support for SQE and show how committed we are > to SQE. *StormSQL looks awesome and has some real potential! * > > We use SQE in production. It has been tested, code reviewed, load tested, > maintained, and processing an average of 8 million tuples per minute or > more for over a year now. The investment into this code base has been > significant. > > Please take a look at the code itself. The production quality code is ready > to go. Developers with no experience with Storm or even streaming > successfully launch robust topologies using SQE. Our productivity in this > area went up by orders of magnitude. > > Based on this experience we realized the value of querying storm, and we > decided to give that value back to the storm community. > > Our data pipelines and real-time processing are very important to the > success of JW Player. SQE has been a foundation for that. We will continue > to invest into this technology for years to come. Unfortunately we wouldn't > be able to adopt StormSQL as is until it has been put through the crucible > of production level usage and has had the same rigor applied. It seems much > of the development has been over the last couple of weeks. > > *Quick Gap Analysis (Not Exhaustive)* > *States* > - SQE supports Redis and MongoDB as states in addition to Kafka. (Soon > adding a Test/Monitor State) > - SQE supports non-static field names for Redis state > - Storm SQL supports Kafka > - SQE supports replay filtering for Kafka > > *Aggregations* > - SQE supports stateful, exactly-once aggregations for states that > support it > - Storm SQL supports aggregations within each micro batch > > *SQL* > - StormSQL supports SQL > - SQE supports SQL "like" JSON > > *Scaling* > - SQE has a mechanism for controlling parallelism or scaling > - Could not find parallelism or scaling controls within StormSQL (May > need to look harder) > > *Support for SQE* > So far the SQE / JW Player developers have been watching this thread > without knowing if we should chime in. I call upon the devs at JW to chime > in because we are dedicated to the success of this SQL in Storm. > > (Noticed I said "chime" three times in this email... well now four times) > > Thanks for reading, > > Lee Morris, Sr Principal Engineer, Data | JWPLAYER > > O: 212.244.0140 <212.244.0140%20x999> | M: 215.920.1331 > > 2 Park Avenue, 10th Floor North, New York NY 10016 > > jwplayer.com | @jwplayer <http://twitter.com/jwplayer> > > On Tue, Aug 30, 2016 at 5:46 PM, Jungtaek Lim <kabh...@gmail.com> wrote: > > > Hi Morrigan, > > > > Thanks for joining discussion. I thought we need to hear your goal to > > donate SQE code, and opinion for how to apply SQE to Storm SQL and > working > > on further improvements. > > > > Not sure when you took a look at the feature set of Storm SQL, but if you > > haven't recently, you may want to do that. > > I started working on improving Storm SQL several weeks ago, and many > things > > are addressed in recent weeks. > > > > * STORM-1435 <https://issues.apache.org/jira/browse/STORM-1435>: You can > > easily launch Storm SQL runner without concerning dependencies for Storm > > SQL core and runtime. It wasn't easy to run before STORM-2016 > > <http://issues.apache.org/jira/browse/STORM-2016> is introduced. > > * Refactored Storm SQL code for Trident to fit to Trident operations. > Storm > > SQL parsed SQL and generated topology code but it was not easy to know > how > > topology code is generated, and also hard to determine how Trident > > optimizations are applied. > > * STORM-1434 <https://issues.apache.org/jira/browse/STORM-1434>, > > STORM-2050 > > <https://issues.apache.org/jira/browse/STORM-2050>: Addressed GROUP BY > > with > > UDAF (User Defined Aggregate Function) on Trident mode. Storm SQL already > > supported UDF on Trident mode. > > * STORM-2057 <https://issues.apache.org/jira/browse/STORM-2057>: JOIN > > (inner, left outer, right outer, full outer) feature is now on reviewing. > > Note that only equi-join is supported. > > > > The changes are not included to official release yet, but I expect Storm > > 1.1.0 will include them which are worth to try out for early adopters. > > > > You can also refer STORM-1433 > > <https://issues.apache.org/jira/browse/STORM-1433> for current phase of > > Storm SQL. Might need to have another phases (epics) for resolving other > > issues as well. > > > > I only had a look at SQE wiki so don't know the detailed features of SQE, > > but my feeling is that recent changes fills the gap between SQE and Storm > > SQL, and even addressing some TODOs of SQE. We might need to cross check > > feature set of each project to make clear on pros and cons for each > > project. > > > > Btw, while Storm SQL has been implemented its missing features, the > > difficult part for Storm SQL is SQL optimizations. There seems lots of > SQL > > optimizations (like filter pushdown) but I'm not expert on that and it > > apparently needs more deep understanding of Calcite. Other parts also > need > > contributors but we strongly need contributors in this area. > > > > Thanks, > > Jungtaek Lim (HeartSaVioR) > > > > 2016년 8월 31일 (수) 오전 12:47, Morrigan Jones <morri...@jwplayer.com>님이 작성: > > > > > Hi, I'm the original creator and primary developer of SQE. Sorry for > > > the radio silence on my part, I was out on vacation the past two > > > weeks. > > > > > > I'm glad to see the Storm SQL project chugging along. I started SQE > > > because I wanted better tools on top of Storm, particularly the > > > ability to query streams and build topologies using SQL. Our > > > philosophy is to quickly iterate on our production systems and provide > > > immediate value. We've been able to do this with SQE, which powers our > > > streaming systems. Work on SQE and adding functions is driven by our > > > current use cases. The big near term item on our road map is to add > > > SQL parsing. Calcite is very promising there and brings lots of > > > additional features, as I'm sure you know. Additionally, we're going > > > to improve our function, stream, and state support. > > > > > > The difficulty I can see for us with Storm SQL is the amount of work > > > necessary to get from where we are now with SQE to integrating any > > > functionality and making sure Storm SQL can provide the functionality > > > we have now, assuming that is the path we would all go. We're super > > > excited to see support for Storm grow and mature, and we'd like to be > > > a part of that. But we also have to maintain our ability to iterate > > > quickly and provide immediate value. > > > > > > > > > > > > -- > > > Morrigan Jones > > > Principal Engineer > > > JWPLAYER | Your Way to Play > > > morri...@jwplayer.com | jwplayer.com > > > > > > -- Kelvin Shek Software Development Engineer in Test | JW PLAYER m: (917) 548-8949 kel...@jwplayer.com
