Re: [DISCUSS] Shall we make SpillableSubpartition repeatedly readable to support fine grained recovery

2019-01-28 Thread Tao Yangyu
olve all your concerns based on this architecuture. > > Best, > Zhijiang > > > -- > From:Bo WANG > Send Time:2019年1月28日(星期一) 15:11 > To:dev > Cc:wangzhijiang999 ; Kurt Young < > k...@apache.org> > Subject:Re: [DISCUSS] Shall we make SpillableSubparti

Re: [DISCUSS] Shall we make SpillableSubpartition repeatedly readable to support fine grained recovery

2019-01-28 Thread zhijiang
hen it is natural to solve > > > the above problems based on this architecuture. You can refer to the flip > > > [1] if interested. > > > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-31%3A+Pluggable+Shuffle+Manager > >

Re: [DISCUSS] Shall we make SpillableSubpartition repeatedly readable to support fine grained recovery

2019-01-27 Thread Bo WANG
the above problems based on this architecuture. You can refer to the flip > > > [1] if interested. > > > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-31%3A+Pluggable+Shuffle+Manager > > > > > > Best, > > > Zhijiang > >

Re: [DISCUSS] Shall we make SpillableSubpartition repeatedly readable to support fine grained recovery

2019-01-26 Thread Stephan Ewen
gt; > ------------------ > > From:Stephan Ewen > > Send Time:2019年1月24日(星期四) 22:17 > > To:dev ; Kurt Young > > Subject:Re: [DISCUSS] Shall we make SpillableSubpartition repeatedly > > readable to support fine grained re

Re: [DISCUSS] Shall we make SpillableSubpartition repeatedly readable to support fine grained recovery

2019-01-24 Thread Guowei Ma
-- > From:Stephan Ewen > Send Time:2019年1月24日(星期四) 22:17 > To:dev ; Kurt Young > Subject:Re: [DISCUSS] Shall we make SpillableSubpartition repeatedly > readable to support fine grained recovery > > The Spillabl

Re: [DISCUSS] Shall we make SpillableSubpartition repeatedly readable to support fine grained recovery

2019-01-24 Thread zhijiang
-- From:Stephan Ewen Send Time:2019年1月24日(星期四) 22:17 To:dev ; Kurt Young Subject:Re: [DISCUSS] Shall we make SpillableSubpartition repeatedly readable to support fine grained recovery The SpillableSubpartition can also be used during the execution of bounded DataStreams programs. I think this is largely

Re: [DISCUSS] Shall we make SpillableSubpartition repeatedly readable to support fine grained recovery

2019-01-24 Thread Stephan Ewen
The SpillableSubpartition can also be used during the execution of bounded DataStreams programs. I think this is largely independent from deprecating the DataSet API. I am wondering if this particular issue is one that has been addressed in the Blink code already (we are looking to merge much of t

Re: [DISCUSS] Shall we make SpillableSubpartition repeatedly readable to support fine grained recovery

2019-01-24 Thread Piotr Nowojski
Hi, I’m not sure how much effort we will be willing to invest in the existing batch stack. We are currently focusing on the support of bounded DataStreams (already done in Blink and will be merged to Flink soon) and unifing batch & stream under DataStream API. Piotrek > On 23 Jan 2019, at 04:

[DISCUSS] Shall we make SpillableSubpartition repeatedly readable to support fine grained recovery

2019-01-22 Thread Bo WANG
Hi all, When running the batch WordCount example, I configured the job execution mode as BATCH_FORCED, and failover-strategy as region, I manually injected some errors to let the execution fail in different phases. In some cases, the job could recovery from failover and became succeed, but in som