Our data science efforts rely on SQE to power our recommendations engine. I am also excited to contribute to it especially as we continue to implement predictive models at larger scales.
On Fri, Sep 2, 2016 at 10:57 AM, Sahil Shah <sa...@jwplayer.com> wrote: > I would like to throw my support behind SQE. Having working with it in a > production environment, I have seen the many benefits in testing new > topologies and quickly understanding what a topology is doing. As our data > needs have grown, we have only increased our reliance on SQE and it stands > the test repeatedly. I am excited at the opportunity to contribute to this > wonderful open source community. > > On Fri, Sep 2, 2016 at 10:31 AM, Alex Halter <a...@jwplayer.com> wrote: > > > I too want to voice my support for SQE and our commitment to the > initiative > > going forward. We've been working on adapting Storm to our needs for most > > of two years. It was thoughtfully designed and supports our production > > needs. We have a long list of features we want to build out and we'd love > > to work with the community. > > > > > > On Fri, Sep 2, 2016 at 10:19 AM, Rohit Garg <rohit.gar...@gmail.com> > > wrote: > > > > > I am one of the developers who has been working on SQE for past 1.5 > > years. > > > Over time, we have made it more stable and production ready. > > > > > > As of now, one can easily scale SQE for more production data with easy > > > config changes and re-deploy, aggregate across different dimensions by > > > writing json like sql, write to different state stores and most > > > importantly, address new feature requirements really quick.(Since it's > > just > > > writing a sql like json file and sqe handles everything for you ! ) > > > > > > I think SQE can really help companies who want to setup a production > > ready > > > and well tested framework within weeks (instead of months) for large > > scale > > > event stream processing and with minimum risks and limited resources. > We > > > are actively working on SQE to make it more awesome and are committed > to > > > make the experience of developing a highly scalable and fault tolerant > > > stream processing framework more seamless and less stressful !!!! > > > > > > On Fri, Sep 2, 2016 at 9:49 AM, Lee Morris <l...@jwplayer.com> wrote: > > > > > > > Hi, Storm Dev! > > > > > > > > I wanted to chime in to show support for SQE and show how committed > we > > > are > > > > to SQE. *StormSQL looks awesome and has some real potential! * > > > > > > > > We use SQE in production. It has been tested, code reviewed, load > > tested, > > > > maintained, and processing an average of 8 million tuples per minute > or > > > > more for over a year now. The investment into this code base has been > > > > significant. > > > > > > > > Please take a look at the code itself. The production quality code is > > > ready > > > > to go. Developers with no experience with Storm or even streaming > > > > successfully launch robust topologies using SQE. Our productivity in > > > this > > > > area went up by orders of magnitude. > > > > > > > > Based on this experience we realized the value of querying storm, and > > we > > > > decided to give that value back to the storm community. > > > > > > > > Our data pipelines and real-time processing are very important to the > > > > success of JW Player. SQE has been a foundation for that. We will > > > continue > > > > to invest into this technology for years to come. Unfortunately we > > > wouldn't > > > > be able to adopt StormSQL as is until it has been put through the > > > crucible > > > > of production level usage and has had the same rigor applied. It > seems > > > much > > > > of the development has been over the last couple of weeks. > > > > > > > > *Quick Gap Analysis (Not Exhaustive)* > > > > *States* > > > > - SQE supports Redis and MongoDB as states in addition to Kafka. > > (Soon > > > > adding a Test/Monitor State) > > > > - SQE supports non-static field names for Redis state > > > > - Storm SQL supports Kafka > > > > - SQE supports replay filtering for Kafka > > > > > > > > *Aggregations* > > > > - SQE supports stateful, exactly-once aggregations for states that > > > > support it > > > > - Storm SQL supports aggregations within each micro batch > > > > > > > > *SQL* > > > > - StormSQL supports SQL > > > > - SQE supports SQL "like" JSON > > > > > > > > *Scaling* > > > > - SQE has a mechanism for controlling parallelism or scaling > > > > - Could not find parallelism or scaling controls within StormSQL > (May > > > > need to look harder) > > > > > > > > *Support for SQE* > > > > So far the SQE / JW Player developers have been watching this thread > > > > without knowing if we should chime in. I call upon the devs at JW to > > > chime > > > > in because we are dedicated to the success of this SQL in Storm. > > > > > > > > (Noticed I said "chime" three times in this email... well now four > > times) > > > > > > > > Thanks for reading, > > > > > > > > Lee Morris, Sr Principal Engineer, Data | JWPLAYER > > > > > > > > O: 212.244.0140 <212.244.0140%20x999> | M: 215.920.1331 > > > > > > > > 2 Park Avenue, 10th Floor North, New York NY 10016 > > > > > > > > jwplayer.com | @jwplayer <http://twitter.com/jwplayer> > > > > > > > > On Tue, Aug 30, 2016 at 5:46 PM, Jungtaek Lim <kabh...@gmail.com> > > wrote: > > > > > > > > > Hi Morrigan, > > > > > > > > > > Thanks for joining discussion. I thought we need to hear your goal > to > > > > > donate SQE code, and opinion for how to apply SQE to Storm SQL and > > > > working > > > > > on further improvements. > > > > > > > > > > Not sure when you took a look at the feature set of Storm SQL, but > if > > > you > > > > > haven't recently, you may want to do that. > > > > > I started working on improving Storm SQL several weeks ago, and > many > > > > things > > > > > are addressed in recent weeks. > > > > > > > > > > * STORM-1435 <https://issues.apache.org/jira/browse/STORM-1435>: > You > > > can > > > > > easily launch Storm SQL runner without concerning dependencies for > > > Storm > > > > > SQL core and runtime. It wasn't easy to run before STORM-2016 > > > > > <http://issues.apache.org/jira/browse/STORM-2016> is introduced. > > > > > * Refactored Storm SQL code for Trident to fit to Trident > operations. > > > > Storm > > > > > SQL parsed SQL and generated topology code but it was not easy to > > know > > > > how > > > > > topology code is generated, and also hard to determine how Trident > > > > > optimizations are applied. > > > > > * STORM-1434 <https://issues.apache.org/jira/browse/STORM-1434>, > > > > > STORM-2050 > > > > > <https://issues.apache.org/jira/browse/STORM-2050>: Addressed > GROUP > > BY > > > > > with > > > > > UDAF (User Defined Aggregate Function) on Trident mode. Storm SQL > > > already > > > > > supported UDF on Trident mode. > > > > > * STORM-2057 <https://issues.apache.org/jira/browse/STORM-2057>: > > JOIN > > > > > (inner, left outer, right outer, full outer) feature is now on > > > reviewing. > > > > > Note that only equi-join is supported. > > > > > > > > > > The changes are not included to official release yet, but I expect > > > Storm > > > > > 1.1.0 will include them which are worth to try out for early > > adopters. > > > > > > > > > > You can also refer STORM-1433 > > > > > <https://issues.apache.org/jira/browse/STORM-1433> for current > phase > > > of > > > > > Storm SQL. Might need to have another phases (epics) for resolving > > > other > > > > > issues as well. > > > > > > > > > > I only had a look at SQE wiki so don't know the detailed features > of > > > SQE, > > > > > but my feeling is that recent changes fills the gap between SQE and > > > Storm > > > > > SQL, and even addressing some TODOs of SQE. We might need to cross > > > check > > > > > feature set of each project to make clear on pros and cons for each > > > > > project. > > > > > > > > > > Btw, while Storm SQL has been implemented its missing features, the > > > > > difficult part for Storm SQL is SQL optimizations. There seems lots > > of > > > > SQL > > > > > optimizations (like filter pushdown) but I'm not expert on that and > > it > > > > > apparently needs more deep understanding of Calcite. Other parts > also > > > > need > > > > > contributors but we strongly need contributors in this area. > > > > > > > > > > Thanks, > > > > > Jungtaek Lim (HeartSaVioR) > > > > > > > > > > 2016년 8월 31일 (수) 오전 12:47, Morrigan Jones <morri...@jwplayer.com > >님이 > > > 작성: > > > > > > > > > > > Hi, I'm the original creator and primary developer of SQE. Sorry > > for > > > > > > the radio silence on my part, I was out on vacation the past two > > > > > > weeks. > > > > > > > > > > > > I'm glad to see the Storm SQL project chugging along. I started > SQE > > > > > > because I wanted better tools on top of Storm, particularly the > > > > > > ability to query streams and build topologies using SQL. Our > > > > > > philosophy is to quickly iterate on our production systems and > > > provide > > > > > > immediate value. We've been able to do this with SQE, which > powers > > > our > > > > > > streaming systems. Work on SQE and adding functions is driven by > > our > > > > > > current use cases. The big near term item on our road map is to > add > > > > > > SQL parsing. Calcite is very promising there and brings lots of > > > > > > additional features, as I'm sure you know. Additionally, we're > > going > > > > > > to improve our function, stream, and state support. > > > > > > > > > > > > The difficulty I can see for us with Storm SQL is the amount of > > work > > > > > > necessary to get from where we are now with SQE to integrating > any > > > > > > functionality and making sure Storm SQL can provide the > > functionality > > > > > > we have now, assuming that is the path we would all go. We're > super > > > > > > excited to see support for Storm grow and mature, and we'd like > to > > be > > > > > > a part of that. But we also have to maintain our ability to > iterate > > > > > > quickly and provide immediate value. > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Morrigan Jones > > > > > > Principal Engineer > > > > > > JWPLAYER | Your Way to Play > > > > > > morri...@jwplayer.com | jwplayer.com > > > > > > > > > > > > > > > > > > > > > > > > -- > *Sahil Shah,* Data Engineer > *JW*PLAYER | Your Way to Play > P: 240.595.1169 | jwplayer.com >
