Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Hey Ismaël, That totally makes sense, I will work on this on the weekend and send a PR. Cheers, Matthias On Thu, 31 May 2018 at 09:45 Ismaël Mejía wrote: > Great ! > > Matthias I think it makes sense to have these guidelines in the beam > site better than a google doc. Can you please submit a

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Great ! Matthias I think it makes sense to have these guidelines in the beam site better than a google doc. Can you please submit a PR for this? On Thu, May 31, 2018 at 8:03 AM Matthias Baetens wrote: > > Hey Eugene, hi all! > > Happy to say your talk is now on the Beam YouTube channel and can

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Hey Eugene, hi all! Happy to say your talk is now on the Beam YouTube channel and can be watched here . It'd be great to see more of these on the channel so we can start sharing this on meetups, conferences and other places and see this grow, so don't hesitate to

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Hi Matthias, Thank you. Here is the raw file on Google Drive: https://drive.google.com/file/d/1gUoe6UrpNNO3ijYSgTvD5GBy-14Zx6dT/view?usp=sharing And yes, I have permission from O'Reilly/Strata to use this file whichever way I want, so it's ok to share on YouTube. On Fri, May 11, 2018 at 12:13

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Hey Eugene, Apologies for picking this up so late, but I could help uploading your video to the Beam channel. Are you able to send me the raw file and do you have sign-off to go ahead with sharing it on YouTube? Thanks. Matthias On Sat, 14 Apr 2018 at 21:45 Eugene Kirpichov

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Hi all, The video is now available. I got it from my Strata account and I have permission to use and share it freely, so I published it on my own YouTube page (where there's nothing else...). Perhaps it makes sense to add to the Beam YouTube channel, but AFAIK only a PMC member can do that.

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Very informative, thanks! On Fri, Mar 9, 2018 at 4:49 PM Etienne Chauchot wrote: > Great ! > > Thanks for sharing. > > Etienne > > Le jeudi 08 mars 2018 à 19:49 +, Eugene Kirpichov a écrit : > > Hey all, > > The slides for my yesterday's talk at Strata San Jose >

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Great ! Thanks for sharing. Etienne Le jeudi 08 mars 2018 à 19:49 +, Eugene Kirpichov a écrit : > Hey all, > > The slides for my yesterday's talk at Strata San Jose  > https://conferences.oreilly.com/strata/strata-ca/public/schedule > /detail/63696 have been posted on the talk page. They may

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Great talk, Eugene. Ted, will share more info on Kafka IO for Python soon :) - Cham On Thu, Mar 8, 2018 at 4:55 PM Ted Yu wrote: > I see. > > I have added myself as watcher on BEAM-3788. > > Thanks > > On Thu, Mar 8, 2018 at 4:51 PM, Eugene Kirpichov

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

I see. I have added myself as watcher on BEAM-3788. Thanks On Thu, Mar 8, 2018 at 4:51 PM, Eugene Kirpichov wrote: > Hi Ted - KafkaIO is not yet implemented using Splittable DoFn's (it was > implemented before SDFs existed and hasn't been rewritten yet), but it will >

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Hi Ted - KafkaIO is not yet implemented using Splittable DoFn's (it was implemented before SDFs existed and hasn't been rewritten yet), but it will be, once more runners catch up with the support: currently we have Dataflow and Flink. +Chamikara Jayalath is currently working

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Eugene: Very informative talk. I looked at: sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/OffsetRangeTrackerTest.java Is there some example showing how OffsetRangeTracker works with Kafka partition(s) ? Thanks On Thu, Mar 8, 2018 at 3:58 PM, Eugene Kirpichov

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Hi Thomas! In case of tailing a Kafka partition, the restriction would be [start_offset, infinity), and it would keep being split by checkpointing into [start_offset, end_offset) and [end_offset, infinity) On Thu, Mar 8, 2018 at 3:52 PM Thomas Weise wrote: > Eugene, > > I

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Eugene, I actually had one question regarding the application of SDF for the Kafka consumer. Reading through a topic partition can be parallel by splitting a partition into multiple restrictions (for use cases where order does not matter). But how would the tail read be managed? I assume there

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Terrific! Thanks Eugene. Just the slides themselves are so good, can't wait for the video. Do you know when the video might be available? On Thu, Mar 8, 2018 at 12:16 PM Eugene Kirpichov wrote: > Oops that's just the template I used. Thanks for noticing, will regenerate >

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Love it. Great flashy title, too :-) On Thu, Mar 8, 2018 at 12:16 PM Eugene Kirpichov wrote: > Oops that's just the template I used. Thanks for noticing, will regenerate > the PDF and reupload when I get to it. > > On Thu, Mar 8, 2018, 11:59 AM Dan Halperin

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Oops that's just the template I used. Thanks for noticing, will regenerate the PDF and reupload when I get to it. On Thu, Mar 8, 2018, 11:59 AM Dan Halperin wrote: > Looks like it was a good talk! Why is it Google Confidential & > Proprietary, though? > > Dan > > On Thu,

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

I really like slide 19: Author: "I made a bigdata programming model" Reader: "Cool, how does data get in and out?" Author: "Brb" On Thu, Mar 8, 2018 at 11:49 AM, Eugene Kirpichov wrote: > Hey all, > > The slides for my yesterday's talk at Strata San Jose

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Looks like it was a good talk! Why is it Google Confidential & Proprietary, though? Dan On Thu, Mar 8, 2018 at 11:49 AM, Eugene Kirpichov wrote: > Hey all, > > The slides for my yesterday's talk at Strata San Jose https://conferences. >

"Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

Hey all, The slides for my yesterday's talk at Strata San Jose https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/63696 have been posted on the talk page. They may be of interest both to users and IO authors. Thanks.