Hi Ismaël, We have a streaming computing platform in Alibaba. Galaxy is an internal system, so you can't find some information from Google. It is becoming more like StreamScope (you can search it for the paper). Non-global-checkpoint makes failure recovery quickly and makes streaming applications easier to develop and debug.
But as far as I know, each engine has its own tradeoffs, has its own good cases. So we also developed an open source platform, which has Spark, Flink and so on. We hope we can use Apache Beam to unify the user program model. This will make the user learning costs are low, the application migration costs are low. (Not only from batch to streaming, but also conducive to migration from the streaming to the streaming.) ------------------------------------------------------------------From:Ismaël Mejía <ieme...@gmail.com>Time:2017 Apr 2 (Sun) 03:18To:dev <dev@beam.apache.org>Subject:Re: Update of Pei in Alibaba Excellent news, Pei it would be great to have a new runner. I am curious about how different are the implementations of storm among them considering that there are already three 'versions': Storm, Jstorm and Heron, I wonder if one runner could traduce to an API that would cover all of them (of course maybe I am super naive I really don't know much about JStorm or Heron and how much they differ from the original storm). Jingson, I am super curious about this Galaxy project, it is there any public information about it? is this related to the previous blink ali baba project? I already looked a bit but searching "Ali baba galaxy" is a recipe for a myriad of telephone sellers :) Nice to see that you are going to keep contributing to the project Pei. Regards, Ismaël On Sat, Apr 1, 2017 at 4:59 PM, Tibor Kiss <tibor.k...@gmail.com> wrote: > Exciting times, looking forward to try it out! > > I shall mention that Taylor Goetz also started creating a BEAM runner using > Storm. > His work is available in the storm repo: > https://github.com/apache/storm/commits/beam-runner > Maybe it's worth while to take a peek and see if something is reusable from > there. > > - Tibor > > On Sat, Apr 1, 2017 at 4:37 AM, JingsongLee <lzljs3620...@aliyun.com> wrote: > >> Wow, very glad to see JStorm also started building BeamRunner. >> I am working in Galaxy (Another streaming process engine in Alibaba). >> I hope that we can work together to promote the use of Apache Beam >> in Alibaba and China. >> >> best, >> JingsongLee >> ------------------------------------------------------------------From:Pei >> HE <pei...@gmail.com>Time:2017 Apr 1 (Sat) 09:24To:dev < >> dev@beam.apache.org>Subject:Update of Pei in Alibaba >> Hi all, >> On February, I moved from Seattle to Hangzhou, China, and joined Alibaba. >> And, I want to give an update of things in here. >> >> A colleague and I have been working on JStorm >> <https://github.com/alibaba/jstorm> runner. We have a prototype that works >> with WordCount and PAssert. (I am going to start a separate email thread >> about how to get it reviewed and merged in Apache Beam.) >> We also have Spark clusters, and are very interested in >> using Spark runner. >> >> Last Saturday, I went to China Hadoop Summit, and gave a talk about Apache >> Beam model. While many companies gave talks of their >> in-house solutions for >> unified batch&streaming and unified SQL, there are also lots of interests >> and enthusiasts of Beam. >> >> Looking forward to chat more. >> -- >> Pei >> >> > > > -- > Kiss Tibor > > +36 70 275 9863 > tibor.k...@gmail.com
