Thanks to all the JW Player folks who joined this thread, and welcome to the Storm community! It’s good to see that the code donation comes with a community ready and willing to help out.
I’m +1 for accepting the code donation. -Taylor > On Sep 2, 2016, at 12:07 PM, Abhishek Agarwal <abhishc...@gmail.com> wrote: > > +1 for the merge. Apart from code contributions, helping users run and > troubleshoot SQE is equally important and it seems that quite a number of > folks are ready to help. > > On Fri, Sep 2, 2016 at 8:46 PM, Kamil Sindi <ka...@jwplayer.com> wrote: > >> Our data science efforts rely on SQE to power our recommendations engine. I >> am also excited to contribute to it especially as we continue to implement >> predictive models at larger scales. >> >> On Fri, Sep 2, 2016 at 10:57 AM, Sahil Shah <sa...@jwplayer.com> wrote: >> >>> I would like to throw my support behind SQE. Having working with it in a >>> production environment, I have seen the many benefits in testing new >>> topologies and quickly understanding what a topology is doing. As our >> data >>> needs have grown, we have only increased our reliance on SQE and it >> stands >>> the test repeatedly. I am excited at the opportunity to contribute to >> this >>> wonderful open source community. >>> >>> On Fri, Sep 2, 2016 at 10:31 AM, Alex Halter <a...@jwplayer.com> wrote: >>> >>>> I too want to voice my support for SQE and our commitment to the >>> initiative >>>> going forward. We've been working on adapting Storm to our needs for >> most >>>> of two years. It was thoughtfully designed and supports our production >>>> needs. We have a long list of features we want to build out and we'd >> love >>>> to work with the community. >>>> >>>> >>>> On Fri, Sep 2, 2016 at 10:19 AM, Rohit Garg <rohit.gar...@gmail.com> >>>> wrote: >>>> >>>>> I am one of the developers who has been working on SQE for past 1.5 >>>> years. >>>>> Over time, we have made it more stable and production ready. >>>>> >>>>> As of now, one can easily scale SQE for more production data with >> easy >>>>> config changes and re-deploy, aggregate across different dimensions >> by >>>>> writing json like sql, write to different state stores and most >>>>> importantly, address new feature requirements really quick.(Since >> it's >>>> just >>>>> writing a sql like json file and sqe handles everything for you ! ) >>>>> >>>>> I think SQE can really help companies who want to setup a production >>>> ready >>>>> and well tested framework within weeks (instead of months) for large >>>> scale >>>>> event stream processing and with minimum risks and limited resources. >>> We >>>>> are actively working on SQE to make it more awesome and are committed >>> to >>>>> make the experience of developing a highly scalable and fault >> tolerant >>>>> stream processing framework more seamless and less stressful !!!! >>>>> >>>>> On Fri, Sep 2, 2016 at 9:49 AM, Lee Morris <l...@jwplayer.com> wrote: >>>>> >>>>>> Hi, Storm Dev! >>>>>> >>>>>> I wanted to chime in to show support for SQE and show how committed >>> we >>>>> are >>>>>> to SQE. *StormSQL looks awesome and has some real potential! * >>>>>> >>>>>> We use SQE in production. It has been tested, code reviewed, load >>>> tested, >>>>>> maintained, and processing an average of 8 million tuples per >> minute >>> or >>>>>> more for over a year now. The investment into this code base has >> been >>>>>> significant. >>>>>> >>>>>> Please take a look at the code itself. The production quality code >> is >>>>> ready >>>>>> to go. Developers with no experience with Storm or even streaming >>>>>> successfully launch robust topologies using SQE. Our productivity >> in >>>>> this >>>>>> area went up by orders of magnitude. >>>>>> >>>>>> Based on this experience we realized the value of querying storm, >> and >>>> we >>>>>> decided to give that value back to the storm community. >>>>>> >>>>>> Our data pipelines and real-time processing are very important to >> the >>>>>> success of JW Player. SQE has been a foundation for that. We will >>>>> continue >>>>>> to invest into this technology for years to come. Unfortunately we >>>>> wouldn't >>>>>> be able to adopt StormSQL as is until it has been put through the >>>>> crucible >>>>>> of production level usage and has had the same rigor applied. It >>> seems >>>>> much >>>>>> of the development has been over the last couple of weeks. >>>>>> >>>>>> *Quick Gap Analysis (Not Exhaustive)* >>>>>> *States* >>>>>> - SQE supports Redis and MongoDB as states in addition to Kafka. >>>> (Soon >>>>>> adding a Test/Monitor State) >>>>>> - SQE supports non-static field names for Redis state >>>>>> - Storm SQL supports Kafka >>>>>> - SQE supports replay filtering for Kafka >>>>>> >>>>>> *Aggregations* >>>>>> - SQE supports stateful, exactly-once aggregations for states >> that >>>>>> support it >>>>>> - Storm SQL supports aggregations within each micro batch >>>>>> >>>>>> *SQL* >>>>>> - StormSQL supports SQL >>>>>> - SQE supports SQL "like" JSON >>>>>> >>>>>> *Scaling* >>>>>> - SQE has a mechanism for controlling parallelism or scaling >>>>>> - Could not find parallelism or scaling controls within StormSQL >>> (May >>>>>> need to look harder) >>>>>> >>>>>> *Support for SQE* >>>>>> So far the SQE / JW Player developers have been watching this >> thread >>>>>> without knowing if we should chime in. I call upon the devs at JW >> to >>>>> chime >>>>>> in because we are dedicated to the success of this SQL in Storm. >>>>>> >>>>>> (Noticed I said "chime" three times in this email... well now four >>>> times) >>>>>> >>>>>> Thanks for reading, >>>>>> >>>>>> Lee Morris, Sr Principal Engineer, Data | JWPLAYER >>>>>> >>>>>> O: 212.244.0140 <212.244.0140%20x999> | M: 215.920.1331 >>>>>> >>>>>> 2 Park Avenue, 10th Floor North, New York NY 10016 >>>>>> >>>>>> jwplayer.com | @jwplayer <http://twitter.com/jwplayer> >>>>>> >>>>>> On Tue, Aug 30, 2016 at 5:46 PM, Jungtaek Lim <kabh...@gmail.com> >>>> wrote: >>>>>> >>>>>>> Hi Morrigan, >>>>>>> >>>>>>> Thanks for joining discussion. I thought we need to hear your >> goal >>> to >>>>>>> donate SQE code, and opinion for how to apply SQE to Storm SQL >> and >>>>>> working >>>>>>> on further improvements. >>>>>>> >>>>>>> Not sure when you took a look at the feature set of Storm SQL, >> but >>> if >>>>> you >>>>>>> haven't recently, you may want to do that. >>>>>>> I started working on improving Storm SQL several weeks ago, and >>> many >>>>>> things >>>>>>> are addressed in recent weeks. >>>>>>> >>>>>>> * STORM-1435 <https://issues.apache.org/jira/browse/STORM-1435>: >>> You >>>>> can >>>>>>> easily launch Storm SQL runner without concerning dependencies >> for >>>>> Storm >>>>>>> SQL core and runtime. It wasn't easy to run before STORM-2016 >>>>>>> <http://issues.apache.org/jira/browse/STORM-2016> is introduced. >>>>>>> * Refactored Storm SQL code for Trident to fit to Trident >>> operations. >>>>>> Storm >>>>>>> SQL parsed SQL and generated topology code but it was not easy to >>>> know >>>>>> how >>>>>>> topology code is generated, and also hard to determine how >> Trident >>>>>>> optimizations are applied. >>>>>>> * STORM-1434 <https://issues.apache.org/jira/browse/STORM-1434>, >>>>>>> STORM-2050 >>>>>>> <https://issues.apache.org/jira/browse/STORM-2050>: Addressed >>> GROUP >>>> BY >>>>>>> with >>>>>>> UDAF (User Defined Aggregate Function) on Trident mode. Storm SQL >>>>> already >>>>>>> supported UDF on Trident mode. >>>>>>> * STORM-2057 <https://issues.apache.org/jira/browse/STORM-2057>: >>>> JOIN >>>>>>> (inner, left outer, right outer, full outer) feature is now on >>>>> reviewing. >>>>>>> Note that only equi-join is supported. >>>>>>> >>>>>>> The changes are not included to official release yet, but I >> expect >>>>> Storm >>>>>>> 1.1.0 will include them which are worth to try out for early >>>> adopters. >>>>>>> >>>>>>> You can also refer STORM-1433 >>>>>>> <https://issues.apache.org/jira/browse/STORM-1433> for current >>> phase >>>>> of >>>>>>> Storm SQL. Might need to have another phases (epics) for >> resolving >>>>> other >>>>>>> issues as well. >>>>>>> >>>>>>> I only had a look at SQE wiki so don't know the detailed features >>> of >>>>> SQE, >>>>>>> but my feeling is that recent changes fills the gap between SQE >> and >>>>> Storm >>>>>>> SQL, and even addressing some TODOs of SQE. We might need to >> cross >>>>> check >>>>>>> feature set of each project to make clear on pros and cons for >> each >>>>>>> project. >>>>>>> >>>>>>> Btw, while Storm SQL has been implemented its missing features, >> the >>>>>>> difficult part for Storm SQL is SQL optimizations. There seems >> lots >>>> of >>>>>> SQL >>>>>>> optimizations (like filter pushdown) but I'm not expert on that >> and >>>> it >>>>>>> apparently needs more deep understanding of Calcite. Other parts >>> also >>>>>> need >>>>>>> contributors but we strongly need contributors in this area. >>>>>>> >>>>>>> Thanks, >>>>>>> Jungtaek Lim (HeartSaVioR) >>>>>>> >>>>>>> 2016년 8월 31일 (수) 오전 12:47, Morrigan Jones <morri...@jwplayer.com >>>> 님이 >>>>> 작성: >>>>>>> >>>>>>>> Hi, I'm the original creator and primary developer of SQE. >> Sorry >>>> for >>>>>>>> the radio silence on my part, I was out on vacation the past >> two >>>>>>>> weeks. >>>>>>>> >>>>>>>> I'm glad to see the Storm SQL project chugging along. I started >>> SQE >>>>>>>> because I wanted better tools on top of Storm, particularly the >>>>>>>> ability to query streams and build topologies using SQL. Our >>>>>>>> philosophy is to quickly iterate on our production systems and >>>>> provide >>>>>>>> immediate value. We've been able to do this with SQE, which >>> powers >>>>> our >>>>>>>> streaming systems. Work on SQE and adding functions is driven >> by >>>> our >>>>>>>> current use cases. The big near term item on our road map is to >>> add >>>>>>>> SQL parsing. Calcite is very promising there and brings lots of >>>>>>>> additional features, as I'm sure you know. Additionally, we're >>>> going >>>>>>>> to improve our function, stream, and state support. >>>>>>>> >>>>>>>> The difficulty I can see for us with Storm SQL is the amount of >>>> work >>>>>>>> necessary to get from where we are now with SQE to integrating >>> any >>>>>>>> functionality and making sure Storm SQL can provide the >>>> functionality >>>>>>>> we have now, assuming that is the path we would all go. We're >>> super >>>>>>>> excited to see support for Storm grow and mature, and we'd like >>> to >>>> be >>>>>>>> a part of that. But we also have to maintain our ability to >>> iterate >>>>>>>> quickly and provide immediate value. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Morrigan Jones >>>>>>>> Principal Engineer >>>>>>>> JWPLAYER | Your Way to Play >>>>>>>> morri...@jwplayer.com | jwplayer.com >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> >>> >>> -- >>> *Sahil Shah,* Data Engineer >>> *JW*PLAYER | Your Way to Play >>> P: 240.595.1169 | jwplayer.com >>> >> > > > > -- > Regards, > Abhishek Agarwal
signature.asc
Description: Message signed with OpenPGP using GPGMail
