Re: [PROPOSAL] External Join with KV Stores

ideInput. > It's not a one-fit-all rule as Jingsong mentioned, the batch data may be > too large, and it would be changed periodically. A userland PTransform > sounds a more straight-forward option, as it doesn't require support in > runner level. > > Mingmin > &g

Re: [ANNOUNCEMENT] New committers, August 2017 edition!

Thank you all! And congratulations to other new committers. Best, Jingsong Lee--From:Mark Liu Time:2017 Aug 16 (Wed) 02:18To:dev Subject:Re: [ANNOUNCEMENT] New committers, August 2017 edition! Congrats! Excellent works! On Mon, A

Re: [PROPOSAL] External Join with KV Stores

Sorry for so long to reply. Hi, Aljoscha, I think Async I/O operator and Batch the same, and Async is a better interface. All IO-related operations may be more appropriate for  asynchronous use. Just like you said, the beginning is like no any special  support by the Runners. I really like Luke's 

[PROPOSAL] External Join with KV Stores

Hi all: In some scenarios, the user needs to query some information from external kv store in the pipeline.I think we can have a good abstraction that allows users  to get as little as possible with the underlying details.Here is a docs of this proposal, would like to receive your feedback. https

Re: [DISCUSS] Apache Beam 2.1.0 release next week ?

Very nice to see this release. Include the merge of DSL_SQL? Pleased to see BEAM-1612 can be completed. (Not blocker)https://issues.apache.org/jira/browse/BEAM-1612 Best, JingsongLee --From:Kenneth Knowles Time:2017 Jun 22 (Thu) 10

Re: Reduced Availability from 17.6. - 24.6

Have a good time~ Best, JingsongLee--From:Jean-Baptiste Onofré Time:2017 Jun 22 (Thu) 10:21To:dev Subject:Re: Reduced Availability from 17.6. - 24.6 Enjoy !! Well deserved ! Regards JB On 06/21/2017 11:37 PM, Davor Bonaci wrote

Re: Fwd: [Report] Eagle - June 2017

I'd like participate in doing some work. Best, JingsongLee --From:Davor Bonaci Time:2017 Jun 22 (Thu) 06:10To:dev Subject:Fwd: [Report] Eagle - June 2017 From this month's Eagle report: "Community is working on Ale

Re: [DISCUSS] Bundle in Flink Runner

Thanks Aljoscha,  I will add these related links to the docs. Best, JingsongLee -- From:Aljoscha Krettek Time:2017 Jun 14 (Wed) 20:56 To:dev ; JingsongLee Subject:Re: [DISCUSS] Bundle in Flink Runner Hi, Thanks for summarising

[DISCUSS] Bundle in Flink Runner

Hi everyone, I take a discussion to the implement of real bundle in Flink Runner. https://docs.google.com/document/d/1UzELM4nFu8SIeu-QJkbs0sv7Uzd1Ux4aXXM3cw4s7po/edit?usp=sharing Feel free to comment/edit it. Best, JingsongLee

Re: [DISCUSS] Source Watermark Metrics

Hi @Ben Chambers @Aljoscha Krettek @Aviem Zur and other all,  I've written this up as a proposal found here:  https://docs.google.com/document/d/1ykjjG97DjVQP73jGbotGRbtK38hGvFbokNEOuNO4DAo/edit?usp=sharing Feel free to comment/edit it. Best, Jingso

Re: [DISCUSS] Source Watermark Metrics

d the  metric's type(Counter, Guage, Distribution) can not express it. Best,JingsongLee -- From:Ben Chambers Time:2017 Jun 2 (Fri) 21:46 To:dev ; JingsongLee Cc:Aviem Zur ; Ben Chambers Subject:Re: [DISCUSS] Source Watermark

Re: [DISCUSS] Source Watermark Metrics

@Aviem Zur @Ben Chambers What do you think about the value of METRIC_MAX_SPLITS? --From:JingsongLee Time:2017 May 11 (Thu) 16:37To:dev@beam.apache.org Subject:[DISCUSS] Source Watermark Metrics Hi everyone, The source watermark met

[DISCUSS] Source Watermark Metrics

Hi everyone, The source watermark metrics show the consumer latency of Source.  It allows the user to know the health of the job, or it can be used to monitor and alarm. We should have the runner report the watermark metricsrather than having the source report it using metrics. This addresses th

Re: [DISCUSS] Remove TimerInternals.deleteTimer(*) and Timer.cancel()

+1 to remove this, I have not encountered such a strong case. best, JingsongLee -- From:Kenneth Knowles Time:2017 May 9 (Tue) 05:45 To:dev Subject:Re: [DISCUSS] Remove TimerInternals.deleteTimer(*) and Timer.cancel() Interesting

Re: Congratulations Davor!

Congratulations! -- From:Jesse Anderson Time:2017 May 4 (Thu) 21:36 To:dev Subject:Re: Congratulations Davor! Congrats! On Thu, May 4, 2017, 6:20 AM Aljoscha Krettek  wrote: > Congrats! :-) > > On 4. May 2017, at 14:34, Kenneth Kno

Re: Community hackathon

+1 best, Jingsonglee --From:Ted Yu Time:2017 Apr 24 (Mon) 17:29To:dev Subject:Re: Community hackathon +1 > On Apr 24, 2017, at 12:51 AM, Jean-Baptiste Onofré  wrote: >  > That's a wonderful idea ! >  > I thin

Join to external table

tract interface is called ExternalState. There are HBaseState, JDBCState, RedisState. It is accessed by key and namespace. Provides bulk access to the external table for performance. Is there a better way to implement it? Can we make some abstracts in Beam Model?  What do you think? Best, JingsongLee

Re: Renaming SideOutput

strong +1 best, JingsongLee--From:Tang Jijun(上海_技术部_数据平台_唐觊隽) Time:2017 Apr 12 (Wed) 10:39To:dev@beam.apache.org Subject:答复: Renaming SideOutput +1 more clearer -邮件原件- 发件人: Ankur Chauhan [mailto:an...@malloc64.com]  发送时间

Re: StatefulDoFnRunner

There is no suitable way to get the CurrentKey. I think using StepContext.timerInternals() and StepContext.stateInternals() is better. best, JingsongLee --From:Thomas Weise Time:2017 Apr 6 (Thu) 12:45To:dev

Re: Update of Pei in Alibaba

 is reusable from > there. > > - Tibor > > On Sat, Apr 1, 2017 at 4:37 AM, JingsongLee  wrote: > >> Wow, very glad to see JStorm also started building BeamRunner. >> I am working in Galaxy (Another streaming process engine in Alibaba). >> I hope that we can work to

Re: Update of Pei in Alibaba

Wow, very glad to see JStorm also started building BeamRunner. I am working in Galaxy (Another streaming process engine in Alibaba).  I hope that we can work together to promote the use of Apache Beam in Alibaba and China. best, JingsongLee

Re: [PROPOSAL] @OnWindowExpiration

If user have a WordCount StatefulDoFn, the result of counts is always changing before the expiration of window. Maybe the user want a signal to know the count is the final value and then  archive the value to the timing database or somewhere else. best, JingsongLee

Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runners

Hi Aljoscha, I would like to work on the Flink runner with you. Best,JingsongLee--From:Jean-Baptiste Onofré Time:2017 Mar 28 (Tue) 14:04To:dev Subject:Re: Call for help: let's add Splittable DoFn to Spark, Flink and Apex runne

回复： [ANNOUNCEMENT] New committers, March 2017 edition!

Congratulations to all! --原始邮件 --发件人：Stas Levin 日期：2017-03-18 23:09:55收件人：dev@beam.apache.org 主题：Re: [ANNOUNCEMENT] New committers, March 2017 edition!Congrats to the new committers! On Sat, Mar 18, 2017 at 3:44 PM Aviem Zur wrote: Thanks all! Very excited to

回复：Interest in a (virtual) contributor meeting?

+1 来自阿里邮箱 iPhone版 --原始邮件 --发件人：Davor Bonaci 日期：2017-02-22 11:19:12收件人：dev@beam.apache.org 主题：Interest in a (virtual) contributor meeting?In the early days of the project, we have held a few meetings for the initial community to get to know each other. Since then

回复：How to implement Timer in runner

@小多 Thank you! I'll taka a look.--发件人：小多 发送时间：2017年1月27日(星期五) 23:22收件人：dev ; JingsongLee 抄　送：Kenneth Knowles ; Aljoscha Krettek 主　题：Re: How to implement Timer in runner Hi Jingsong, Take a look at Blink (a fork of Flink at Al

回复：How to implement Timer in runner

Thanks for the reply.Maybe we need a external priority queue.Happy Chinese New Year! --发件人：Aljoscha Krettek 发送时间：2017年1月25日(星期三) 18:38收件人：dev ; lzljs3620320 ; Kenneth Knowles 主　题：Re: How to implement Timer in runner Hi Jingsong,you