Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
This makes me want to dig into the streaming API. About time perhaps. :') Cheers! Sachin -- Sachin Goel Computer Science, IIT Delhi m. +91-9871457685 On Aug 6, 2015 2:39 AM, "Theodore Vasiloudis" < theodoros.vasilou...@gmail.com> wrote: > Great post Stephan! A small note: the code for Google Dataflow does display > correctly for me, I'm getting < and > instead of <> > > On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen wrote: > > > Hi all! > > > > We just published a blog post about how streaming fault tolerance > > mechanisms evolved, and what kind of performance Flink gets with its > > checkpointing mechanism. > > > > I think it is a pretty interesting read for people that are interested in > > Flink or data streaming in general. > > > > The blog post talks about: > > > > - Fault tolerance techniques, starting from acknowledgements, over > micro > > batches, to transactional updates and distributed snapshots. > > > > - Performance of Flink, throughput, latency, and tradeoffs. > > > > - A "chaos monkey" experiment where computation continues strongly > > consistent even when periodically killing workers. > > > > > > Comments welcome! > > > > Greetings, > > Stephan > > >
Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
Great post Stephan! A small note: the code for Google Dataflow does display correctly for me, I'm getting < and > instead of <> On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen wrote: > Hi all! > > We just published a blog post about how streaming fault tolerance > mechanisms evolved, and what kind of performance Flink gets with its > checkpointing mechanism. > > I think it is a pretty interesting read for people that are interested in > Flink or data streaming in general. > > The blog post talks about: > > - Fault tolerance techniques, starting from acknowledgements, over micro > batches, to transactional updates and distributed snapshots. > > - Performance of Flink, throughput, latency, and tradeoffs. > > - A "chaos monkey" experiment where computation continues strongly > consistent even when periodically killing workers. > > > Comments welcome! > > Greetings, > Stephan >
[jira] [Created] (FLINK-2489) Improve the streaming window join
Stephan Ewen created FLINK-2489: --- Summary: Improve the streaming window join Key: FLINK-2489 URL: https://issues.apache.org/jira/browse/FLINK-2489 Project: Flink Issue Type: Improvement Components: Streaming Affects Versions: 0.10 Reporter: Stephan Ewen This issue tracks the ideas and discussion to improve the current window join. Here is an initial list of desiderata for an improved window join: - Joins should initially run only on time windows - Support for operator time and event time - Event time version must support multiple windows being in progress (or buffered) while waiting for watermarks - Support for simple Java heap and Flink-managed memory - Flink-managed memory variant should support out-of-core operation - Join buffers or hashtables need to be checkpointed. - Join buffers or hashtables need to support incremental checkpointing - Join buffers or hashtables should support asynchronous checkpointing - For sliding time windows, we should consider variants to reuse the join candidates across multiple windows they are contained in. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
Pretty awesome piece. Sent from my iPhone > On Aug 5, 2015, at 10:10, Hawin Jiang wrote: > > Great job, Guys > > Let me read it carefully. > > > > > > > >> On Wed, Aug 5, 2015 at 7:25 AM, Stephan Ewen wrote: >> I forgot the link ;-) >> >> http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/ >> >>> On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen wrote: >>> Hi all! >>> >>> We just published a blog post about how streaming fault tolerance >>> mechanisms evolved, and what kind of performance Flink gets with its >>> checkpointing mechanism. >>> >>> I think it is a pretty interesting read for people that are interested in >>> Flink or data streaming in general. >>> >>> The blog post talks about: >>> >>> - Fault tolerance techniques, starting from acknowledgements, over micro >>> batches, to transactional updates and distributed snapshots. >>> >>> - Performance of Flink, throughput, latency, and tradeoffs. >>> >>> - A "chaos monkey" experiment where computation continues strongly >>> consistent even when periodically killing workers. >>> >>> >>> Comments welcome! >>> >>> Greetings, >>> Stephan >
Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
Great job, Guys Let me read it carefully. On Wed, Aug 5, 2015 at 7:25 AM, Stephan Ewen wrote: > I forgot the link ;-) > > > http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/ > > On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen wrote: > >> Hi all! >> >> We just published a blog post about how streaming fault tolerance >> mechanisms evolved, and what kind of performance Flink gets with its >> checkpointing mechanism. >> >> I think it is a pretty interesting read for people that are interested in >> Flink or data streaming in general. >> >> The blog post talks about: >> >> - Fault tolerance techniques, starting from acknowledgements, over >> micro batches, to transactional updates and distributed snapshots. >> >> - Performance of Flink, throughput, latency, and tradeoffs. >> >> - A "chaos monkey" experiment where computation continues strongly >> consistent even when periodically killing workers. >> >> >> Comments welcome! >> >> Greetings, >> Stephan >> >> >> >
Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
I forgot the link ;-) http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/ On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen wrote: > Hi all! > > We just published a blog post about how streaming fault tolerance > mechanisms evolved, and what kind of performance Flink gets with its > checkpointing mechanism. > > I think it is a pretty interesting read for people that are interested in > Flink or data streaming in general. > > The blog post talks about: > > - Fault tolerance techniques, starting from acknowledgements, over micro > batches, to transactional updates and distributed snapshots. > > - Performance of Flink, throughput, latency, and tradeoffs. > > - A "chaos monkey" experiment where computation continues strongly > consistent even when periodically killing workers. > > > Comments welcome! > > Greetings, > Stephan > > >
FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance
Hi all! We just published a blog post about how streaming fault tolerance mechanisms evolved, and what kind of performance Flink gets with its checkpointing mechanism. I think it is a pretty interesting read for people that are interested in Flink or data streaming in general. The blog post talks about: - Fault tolerance techniques, starting from acknowledgements, over micro batches, to transactional updates and distributed snapshots. - Performance of Flink, throughput, latency, and tradeoffs. - A "chaos monkey" experiment where computation continues strongly consistent even when periodically killing workers. Comments welcome! Greetings, Stephan
[jira] [Created] (FLINK-2488) Expose attemptNumber in RuntimeContext
Robert Metzger created FLINK-2488:
-
Summary: Expose attemptNumber in RuntimeContext
Key: FLINK-2488
URL: https://issues.apache.org/jira/browse/FLINK-2488
Project: Flink
Issue Type: Improvement
Components: JobManager, TaskManager
Affects Versions: 0.10
Reporter: Robert Metzger
Priority: Minor
It would be nice to expose the attemptNumber of a task in the
{{RuntimeContext}}.
This would allow user code to behave differently in restart scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: 答复: Need help with CI problem
That is interesting, never seen that before. Travis support would be best, I guess. On Wed, Aug 5, 2015 at 10:04 AM, huangwei (G) wrote: > Thank you! > I`ll try supp...@travis-ci.com. > It`s just a line printing "Do you have a question?" and nothing more when > I click the link. > > -邮件原件- > 发件人: Robert Metzger [mailto:rmetz...@apache.org] > 发送时间: 2015年8月5日 15:56 > 收件人: dev@flink.apache.org > 主题: Re: Need help with CI problem > > Hi, > I think the Apache Mailing lists don't support sending image attachments. > But I guess you are asking because of a message like this: > https://github.com/apache/flink/pull/977 > > What happens when you click the link? > Can you write to a message to supp...@travis-ci.com, maybe they can help > you. > > > On Wed, Aug 5, 2015 at 8:36 AM, huangwei (G) > wrote: > > > Hi, I cannot see the CI detail as I click the link. > > > > It was like: > > > > [image: cid:image001.png@01D0CF8B.60774F60] > > > > > > > > Does anyone help me? > > > > > > >
答复: Need help with CI problem
Thank you! I`ll try supp...@travis-ci.com. It`s just a line printing "Do you have a question?" and nothing more when I click the link. -邮件原件- 发件人: Robert Metzger [mailto:rmetz...@apache.org] 发送时间: 2015年8月5日 15:56 收件人: dev@flink.apache.org 主题: Re: Need help with CI problem Hi, I think the Apache Mailing lists don't support sending image attachments. But I guess you are asking because of a message like this: https://github.com/apache/flink/pull/977 What happens when you click the link? Can you write to a message to supp...@travis-ci.com, maybe they can help you. On Wed, Aug 5, 2015 at 8:36 AM, huangwei (G) wrote: > Hi, I cannot see the CI detail as I click the link. > > It was like: > > [image: cid:image001.png@01D0CF8B.60774F60] > > > > Does anyone help me? > > >
Re: Need help with CI problem
Hi, I think the Apache Mailing lists don't support sending image attachments. But I guess you are asking because of a message like this: https://github.com/apache/flink/pull/977 What happens when you click the link? Can you write to a message to supp...@travis-ci.com, maybe they can help you. On Wed, Aug 5, 2015 at 8:36 AM, huangwei (G) wrote: > Hi, I cannot see the CI detail as I click the link. > > It was like: > > [image: cid:image001.png@01D0CF8B.60774F60] > > > > Does anyone help me? > > >
[jira] [Created] (FLINK-2487) the array has out of bounds
zhangrucong created FLINK-2487: -- Summary: the array has out of bounds Key: FLINK-2487 URL: https://issues.apache.org/jira/browse/FLINK-2487 Project: Flink Issue Type: Bug Components: Streaming Affects Versions: 0.8.1 Reporter: zhangrucong Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
