Thanks Jingsong for answering, and the Streamscope ref, I am going to check the paper, the concept of non-global-checkpointing sounds super interesting.
It is nice that you guys are also trying to promote the move to a unified model. Regards, Ismaël On Sun, Apr 2, 2017 at 3:40 PM, JingsongLee <lzljs3620...@aliyun.com> wrote: > Hi Ismaël, > We have a streaming computing platform in Alibaba. > Galaxy is an internal system, so you can't find some information from Google. > It is becoming more like StreamScope (you can search it for the paper). > Non-global-checkpoint makes failure recovery quickly and makes streaming > applications easier to develop and debug. > > > But as far as I know, each engine has its own tradeoffs, has its own good > cases. > So we also developed an open source platform, which has Spark, Flink and so > on. > We hope we can use Apache Beam to unify the user program model. This will > make > the user learning costs are low, the application migration costs are low. > (Not only from batch to streaming, but also conducive to migration from the > streaming to the streaming.) > > > ------------------------------------------------------------------From:Ismaël > Mejía <ieme...@gmail.com>Time:2017 Apr 2 (Sun) 03:18To:dev > <dev@beam.apache.org>Subject:Re: Update of Pei in Alibaba > Excellent news, > > Pei it would be great to have a new runner. I am curious about how > different are the implementations of storm among them considering that > there are already three 'versions': Storm, Jstorm and Heron, I wonder > if one runner could traduce to an API that would cover all of them (of > course maybe I am super naive I really don't know much about JStorm or > Heron and how much they differ from the original storm). > > Jingson, I am super curious about this Galaxy project, it is there any > public information about it? is this related to the previous blink ali > baba project? I already looked a bit but searching "Ali baba galaxy" > is a recipe for a myriad of telephone sellers :) > > Nice to see that you are going to keep contributing to the project Pei. > > Regards, > Ismaël > > > > On Sat, Apr 1, 2017 at 4:59 PM, Tibor Kiss <tibor.k...@gmail.com> wrote: >> Exciting times, looking forward to try it out! >> >> I shall mention that Taylor Goetz also started creating a BEAM runner using >> Storm. >> His work is available in the storm repo: >> https://github.com/apache/storm/commits/beam-runner >> Maybe it's worth while to take a peek and see if something is reusable from >> there. >> >> - Tibor >> >> On Sat, Apr 1, 2017 at 4:37 AM, JingsongLee <lzljs3620...@aliyun.com> wrote: >> >>> Wow, very glad to see JStorm also started building BeamRunner. >>> I am working in Galaxy (Another streaming process engine in Alibaba). >>> I hope that we can work together to promote the use of Apache Beam >>> in Alibaba and China. >>> >>> best, >>> JingsongLee >>> ------------------------------------------------------------------From:Pei >>> HE <pei...@gmail.com>Time:2017 Apr 1 (Sat) 09:24To:dev < >>> dev@beam.apache.org>Subject:Update of Pei in Alibaba >>> Hi all, >>> On February, I moved from Seattle to Hangzhou, China, and joined Alibaba. >>> And, I want to give an update of things in here. >>> >>> A colleague and I have been working on JStorm >>> <https://github.com/alibaba/jstorm> runner. We have a prototype that works >>> with WordCount and PAssert. (I am going to start a separate email thread >>> about how to get it reviewed and merged in Apache Beam.) >>> We also have Spark clusters, and are very interested in >>> using Spark runner. >>> >>> Last Saturday, I went to China Hadoop Summit, and gave a talk about Apache >>> Beam model. While many companies gave talks of their >>> in-house solutions for >>> unified batch&streaming and unified SQL, there are also lots of interests >>> and enthusiasts of Beam. >>> >>> Looking forward to chat more. >>> -- >>> Pei >>> >>> >> >> >> -- >> Kiss Tibor >> >> +36 70 275 9863 >> tibor.k...@gmail.com
