Survey/Pulse: Beam on Flink/Spark/Samza/etc

Curious who all is using Beam on runners other than dataflow. Please respond either on-list or to me directly ... Mostly just curious the extent of whether Beam is fulfilling its promise of runner agnosticism. Getting good data on that is hard, so anecdotes would be very welcomed!

Re: Can apache beam be used for control flow (ETL workflow)

https://beamsummit.org/sessions/event-driven-movie-magic/ ^^ the question made me think of that use case. Though, unclear how close it is to what you're thinking about. Cheers - On Fri, Dec 15, 2023 at 7:01 AM Byron Ellis via user wrote: > As Jan says, theoretically possible? Sure. That

Re: [Request for Feedback] Swift SDK Prototype

This is great that is coming together, and am glad for the messages along the way to understand process, choices, ...! On Fri, Aug 25, 2023, 2:04 PM Byron Ellis via user wrote: > Okay, after a brief detour through "get this working in the Flink Portable > Runner" I think I have something

Re: Missing Beam Katas in Intellij >=2023.3

me in that GH issue to discuss [ @brucearctor ]. Cheers, Austin On Mon, Jul 31, 2023 at 12:10 PM Ahmet Altay wrote: > Hi Bartosz, > > Thanks for flagging this. > > Adding @Austin Bennett and @Israel Herraiz > -- They were the two people who maintained beam katas or > helped with

Re: PubSub Lite IO & Python?

ite >>> >>> On Thu, Aug 4, 2022 at 6:46 PM Chamikara Jayalath >>> wrote: >>> >>>> I believe this should be fully working. I'm not familiar with PyBeam >>>> though. Is the execution mechanism the same as running a regular Beam >>>> pipeline ?

PubSub Lite IO & Python?

Hi Users/Devs, Drew, copied, reported having troubles with PubSub Lite: "we just weren’t able to get PubSub Lite working with PyBeam. It’s been a few weeks since we last tried, but we were just trying to use `apache_beam.io.gcp.pubsublite.ReadFromPubSubLite` (here

Re: Apache Beam London meetup 9: recordings

Great! On Sun, Jun 19, 2022 at 11:18 AM Matthias Baetens wrote: > Hi all > > The recordings from last year's Apache Beam meetup London are now > available on the YouTube channel (apologies for the delay): > - Apache Beam meetup 9: BBC's journey with Apache Beam >

Re: [PROPOSAL] Stop Spark 2 support in Spark Runner

https://spark.apache.org/releases/spark-release-3-0-0.html Since Spark 3 has been out almost 2 years, this seems increasingly reasonable. On Fri, Apr 29, 2022 at 4:04 AM Jean-Baptiste Onofré wrote: > +1, it makes sense to me. Users wanting "old" spark version can take > previous Beam releases.

Re: JdbcIO

Without getting into the super specifics of your use-case, it sounds like you might want to checkout the DebeziumIO for CDC ( Change Data Capture ). I think DebeziumIO can generally handle even much more complex use cases than it sounds like you are trying for. Some pointers/talks from last

Re: Beam Summit is looking for speakers!

I would also add -- if in doubt/concerned, don't hesitate to reachout to me, I'd be happy to talk through potential submission ideas if that'd be helpful for anyone. On Thu, Mar 10, 2022 at 11:25 AM Pablo Estrada wrote: > Thanks for sharing Danielle! > > And to our users - please do submit your

Re: Spark Structured Streaming runner migrated to Spark 3

Hooray! Thanks, Etienne! On Thu, Aug 5, 2021 at 3:11 AM Etienne Chauchot wrote: > Hi all, > > Just to let you know that Spark Structured Streaming runner was migrated > to Spark 3. > > Enjoy ! > > Etienne > >

Re: Allyship workshops for open source contributors

+1, assuming timing can work. On Wed, Jun 2, 2021 at 2:07 PM Aizhamal Nurmamat kyzy wrote: > If we have a good number of people who express interest in this thread, I >> will set up training for the Airflow community. >> > > I meant Beam ^^' I am organizing it for the Airflow community as well.

Re: UX Research Findings Readout for Apache Beam Community

Is it possible to writeup/share results for those not able to attend and/or to digest ahead of attending? On Thu, Jan 28, 2021, 10:46 AM Carlos Camacho Frausto < carlos.cama...@wizeline.com> wrote: > Hello, > Some weeks ago, our firm conducted a User Experience Research Study for > Google

BeamSQL and Beam equivalent -- examples?

Hi All, For something I am currently writing -- I am seeking any examples of BeamSQL and Beam that take the same input and produce the same output. I can't recall, off head, any examples/slides/writeups. Do any exist? I would like to show: (a) that BeamSQL is a real thing :-) (b) that Beam

Re: Ability to link to "latest" of python docs

+dev Lynn, Seems totally doable. If others don't speak up with a good way to do this (or in opposition), I'm sure we can sort something out to accomplish this (will dig into intersphinx mapping tomorrow). Cheers, Austin On Tue, Sep 8, 2020, 5:19 PM Lynn Root wrote: > Hey folks - > > I'm

Intro to Beam and Contributing Workshops

Hi All, I'm a huge fan of HOPE . In the virtual edition this year, I am giving 2 talks. * a 2hr introduction to Beam. * a 1hr introduction to contributing to open source (with specific examples from Beam). These to occur on 30/31 July, schedule found:

Re: Can SpannerIO read data from different GCP project?

I havent tried yet, but looks like the connection string asks for the project to be specified. Based on that (and cross project working for other circumstances), I would imagine it will work, but...? Give it a try! One tricky place might be ensuring proper permissions, in both projects (and

Re: How to safely update jobs in-flight using Apache Beam on AWS EMR?

Hi Dan, AWS EMR generally runs Flink and/or Spark as supported Beam Runners. For EMR, you might want to check compatibility for versions of Beam/Flink can run, and the status of beam pipelines using either of those runners. On running Beam in AWS, had you seen:

Re: Writing pipeline output to google sheet in google drive

@OrielResearch Eila Arich-Landkof Depending on your needs, I wonder about establishing a sheet (or sheets, as needed) that has a BQ connector for the datasource of it. If you use Dataflow to write/create a BQ table, that would then hydrate the sheet (not sure the ordering -- maybe you'd need to

Beam First Steps Workshop - 9 June

Hi Beam Users, Wanted to share the Workshop that I'll give at Berlin Buzzword's next week: https://berlinbuzzwords.de/session/first-steps-apache-beam-writing-portable-pipelines-using-java-python-go Do consider joining if you are able and interested (if you're here and already using, then the

Re: Try Beam Katas Today

It looks like there are instructions online for writing exercises/Katas: https://www.jetbrains.com/help/education/educator-start-guide.html Do we have a guide for contributing and publication/releases occur (publishing to Stepik)? Although the code lives in the main repo (therefore subject to

Beam Digital Summit 2020 -- JUNE 2020!

Hi All, We are excited to announce the Beam Digital Summit 2020! This will occur for partial days during the week of 15-19 June. CfP is open and found: https://sessionize.com/beam-digital-summit-2020/ CfP closes on 20 May 2020. Do not hesitate to reach out to the organizers with any

Meetups

Seems we won't be convening in-person in about any city anytime soon. Seems like a chance to come together virtually. WHO CAN SHARE? Seeking: * Use Cases * Developing Beam/Components * Other If anything particular, also, what would you like to hear -- can see if we can track such speakers

Bay Area Beam Meetup 19 Feb (Last Wednesday).

Hi All, We had a meetup @Sentry.io on Wednesday -- with a solid 40+ engaged attendees. Thanks for those that joined in person, and for those that were unable, talks can be found online --> Syd's talk (real time data warehouse): https://youtu.be/rFK6drAWN40 Mike's talk (beam in production):

Re: Help needed on a problem statement

I'd disentangle Dataflow from Beam. Beam can help you. Dataflow might be useful, though, yes, for batch jobs the spin up cost might be a lot for small file sizes. There are potentially lots of ways to do this. An idea (that I haven't seen used anywhere). Have a streaming Beam pipeline (that

Re: Link to Flink on K8S Webinar

Cool; @aniket and @dagang, As someone who hasn't dug into the code of either (will go through your recording) -- might you share any thoughts on differences between: https://github.com/googlecloudplatform/flink-on-k8s-operator and https://github.com/lyft/flinkk8soperator ?? Also, for those in

Beam Meetup LA -- KICKOFF (10 March)

Come join the community kicking off in LA (in person) on 10 March: https://www.meetup.com/Los-Angeles-Apache-Beam/events/268207085/

Re: NYC ? (or more generally East Coast)

We did get 2 awesome speakers for an event at Spotify in NYC on 25 Feb. For interested, come join! https://www.meetup.com/New-York-Apache-Beam/events/268153356/ On Thu, Jan 23, 2020 at 8:08 AM Austin Bennett wrote: > > Hi Jennifer, > > I'd defer to your local expertis

Los Angeles Beam Meetup kickoff on 27 January

We're kicking off the Beam Community at community member Chad Dombrova's place (Luma Pictures) in Santa Monica. Come join us! https://www.meetup.com/Los-Angeles-Apache-Beam/events/267812648/

Bangalore / Bengaluru Meetup

Hi Dev and Users, Also we hope to kickoff a meetup in India this year. https://www.meetup.com/Bangalore-Apache-Beam/ Please let us know if you'd like to get involved, speaking, hosting, etc. Reply to me, private or on thread, and/or use this survey link: https://forms.gle/cud39eh3FA1em7EU7

Re: NYC ? (or more generally East Coast)

2020 at 4:02 PM Austin Bennett > wrote: >> >> Hi Devs and Users, >> >> We are looking for speakers for future Meetups and Events. Who is >> building cool things with Beam? We are looking at hosting a Meetup at >> Spotify in February, and ideally keep some me

NYC ? (or more generally East Coast)

Hi Devs and Users, We are looking for speakers for future Meetups and Events. Who is building cool things with Beam? We are looking at hosting a Meetup at Spotify in February, and ideally keep some meetups going throughout the year. For this to occur, we need to hear about what people are

Re: proto in pubsub

Bradshaw wrote: > > This should work. An example of what you're trying to do and what > errors/unexpected behavior you're getting would be helpful. > > On Sun, Dec 15, 2019 at 10:13 PM Austin Bennett > wrote: > > > > Hi All, > > > > Struggling with readi

proto in pubsub

Hi All, Struggling with reading a proto message from pubsub and writing to BigQuery in Beam (Direct Runner -- though will then use Dataflow runner. Hoping that distinction doesn't make a difference). Probably something I'm doing wrong (or not doing) with the proto coder. The output (BigQuery)

slides?

Hi Dev and User, Wondering if people would find a benefit from collecting slides from Meetups/Talks? Seems that this could be appropriate on the website, for instance. Not sure whether this has been asked previously, so bringing it to the group. Cheers, Austin

Kicking off Beam Meetup NYC

On the heels of the new Seattle Meetup (yesterday's event), announcing the kickoff of the first event in NYC. https://www.meetup.com/New-York-Apache-Beam/events/265128669/ We'll have Tyler Akidau sharing on Streaming SQL, and some talks from Oden Technologies (a fantastic example of Beam, using

Re: Beam/flink/kubernetes/minikube/wordcount example

I got hung up on that issue earlier this week. Was using Flink 1.7. V2.15 of Beam. Wasn't using Kubernetes. Then gave up, so don't have a solution :-/ I don't understand the job server enough, but think I was getting error when I did not have it running (I still don't understand portability

Re: Hackathon @BeamSummit @ApacheCon

+user@beam.apache.org On Fri, Sep 6, 2019 at 5:24 PM Austin Bennett wrote: > Ah, yes. We'll definitely be in Hackathon space 2-3p on Monday and > Tuesday (and can stay longer if needed). We aren't scheduling anything > official on Wed and Thurs, given the multiple Be

Re: Hackathon @BeamSummit @ApacheCon

And, for clarity, especially focused on Hackathon times on Monday and/or Tuesday of ApacheCon, to not conflict with BeamSummit sessions. On Thu, Aug 22, 2019 at 9:47 AM Austin Bennett wrote: > Less than 3 weeks till Beam Summit @ApacheCon! > > We are to be in Vegas for BeamSummit and

Hackathon @BeamSummit @ApacheCon

Less than 3 weeks till Beam Summit @ApacheCon! We are to be in Vegas for BeamSummit and ApacheCon in a few weeks. Likely to reserve space in the Hackathon Room to accomplish some tasks: * Help Users * Build Beam * Collaborate with other projects * etc If you're to be around (or not) let us know

Re: Live fixing of a Beam bug on July 25 at 3:30pm-4:30pm PST

Pablo, Assigned https://issues.apache.org/jira/browse/BEAM-7607 to you, to make even more likely that it is still around on the 25th :-) Cheers, Austin On Tue, Jul 23, 2019 at 11:24 AM Pablo Estrada wrote: > Hi all, > I've just realized that https://issues.apache.org/jira/browse/BEAM-7607 is

Re: Beam Summit at ApacheCon

concerns, etc. >> >> Cheers, >> Austin >> >> >> >> On Tue, Apr 30, 2019 at 7:59 AM Austin Bennett < >> whatwouldausti...@gmail.com> wrote: >> >>> Hi Users and Devs, >>> >>> The CfP deadline approaches. Do submit

Re: Beam Summit at ApacheCon

Hi All, Deadline for CfP is the morning of 13 May (this Monday) Pacific Time, as decided by ApacheCon. Please submit if you have anything. Also, do write if you have questions/concerns, etc. Cheers, Austin On Tue, Apr 30, 2019 at 7:59 AM Austin Bennett wrote: > Hi Users and D

Re: Apache BEAM on Flink in production

On the Beam YouTube channel: https://www.youtube.com/channel/UChNnb_YO_7B0HlW6FhAXZZQ you can see two talks from people at Lyft; they use Beam on Flink. Other users can also chime in as to how they are running. Would also suggest coming to BeamSummit.org in Berlin in June and/or sharing

Re: Beam Summit at ApacheCon

just want to highlight that this is a great > chance for Beam. There will be a _dedicated_ Beam track which means that > there is potential for lots of new people to learn about Beam. Of > course, there will also be many people already involved in Beam. > > -Max > > On 23.04.19 02:47,

Beam Summit at ApacheCon

Beam Summit will be at ApacheCon this year -- please consider submitting! Dates for Beam Summit 11 and 12 September 2019. There are other tracks at ApacheCon during this and on other dates too. https://www.apachecon.com/acna19/cfp.html

Re: kafka 0.9 support

gt;>> >>>> On the other hand, we warn user in Javadoc of KafkaIO (which is >>>> Unstable, btw) by the following: >>>> *“KafkaIO relies on kafka-clients for all its interactions with the >>>> Kafka cluster.**kafka-clients versions 0.10.1 and newer a

Re: kafka 0.9 support

FWIW -- On my (desired, not explicitly job-function) roadmap is to tap into a bunch of our corporate Kafka queues to ingest that data to places I can use. Those are 'stuck' 0.9, with no upgrade in sight (am told the upgrade path isn't trivial, is very critical flows, and they are scared for it to

Re: Beam Meetups Feb 2019

Hi, > > Can you please provide any video recordings if they are available? > > Thanks, > Teja > > On Mon, Feb 11, 2019, 4:51 PM Austin Bennett wrote: > >> The slides from Tyler's presentation found: >> http://s.apache.org/beam-intro-feb-2019 >> >> I'l

Re: Beam Summits!

Austin Bennett wrote: > Hi All, > > Other projects/Summits like Kafka and Spark offer add-on days to summits > for training. I'm wondering the appetite/interest for hands-on sessions > for working with Beam, and whether we think that'd be helpful. Are there > people that

Re: Beam Summits!

On Wed, Dec 19, 2018 at 3:05 PM Austin Bennett wrote: > Hi All, > > I really enjoyed Beam Summit in London (Thanks Matthias!), and there was > much enthusiasm for continuations. We had selected that location in a > large part due to the growing community there, and we have users

Re: Beam courses

Hi Alex, I'm certainly interested in helping more people use beam (and beyond beginner level). I believe there are people that can help as have already been mentioned in this thread, I am also happy to help create training materials for people as we identify areas that are in need. Have

Re: Beam Summits!

Flink >>> Forward Berlin on the previous 2 days. >>> >>> Same may be for Asia also following Flink Forward Asia where and >>> whenever it happens. >>> >>> On Wed, Dec 19, 2018 at 6:06 PM Austin Bennett < >>> whatwouldausti.

Re: 2019 Beam Events

Already got that process kicked off with the NY and LA meet ups, now that SF is about to be inagurated goal will be to get these moving as well. For anyone that is in (or goes to) those areas: https://www.meetup.com/New-York-Apache-Beam/ https://www.meetup.com/Los-Angeles-Apache-Beam/ Please

Bay Area Apache Beam Kickoff!

We have our first meetup scheduled for December 12th in San Francisco. Andrew Pilloud, a software engineer at Google and Beam committer, will demo the latest feature in Beam SQL: a standalone SQL shell. The talk cover why SQL is a good fit for streaming data processing, the technical details of

Re: FlinkRunner JAAS verify failed in Flink cluster

Related to another thread: Is there a value in posting issues that get put here (with follow up solutions, like this thread, which indeed was excellent to have shared the solution with the list) in Stack Overflow? Again, for ease of discoverability, for those that face similar issues. Not sure

Growing Beam -- A call for ideas? What is missing? What would be good to see?

Hi Beam Devs and Users, Trying to get a sense from the community on the sorts of things we think would be useful to build the community (I am thinking not from an angle of specific code/implementation/functionality, but from a user/usability -- I want to dive in and make real contributions with

Re: SF Meetup(s)

update here. >> >> On Fri, Oct 5, 2018 at 1:02 PM, Thomas Weise wrote: >> >>> Thanks for the initiative. >>> >>> Lyft may be able to help with hosting and I can help with talks. I will >>> check and circle back. >>> >>>

Re: SF Meetup(s)

Ah, and link to the meet up so can join or be aware: https://www.meetup.com/San-Francisco-Apache-Beam/ On Fri, Oct 5, 2018 at 8:47 AM Austin Bennett wrote: > Hi All, > > Looking to start organizing events for Beam around San Francisco. On the > lookout for space -- anyone work f

SF Meetup(s)

Hi All, Looking to start organizing events for Beam around San Francisco. On the lookout for space -- anyone work for a company that could offer space around the city (my company offices are in Foster City, a wholly undesirable Meetup location)? Also, anyone Beam using that would be happy to

Re: [ACTION REQUESTED] What do you use Beam for?

Hi All, In honor of Today's Beam Summit (where we'll gather feedback in-person, too), looking for input. How can we make Beam more usable? Are you struggling with anything? Did you struggle with anything previously that you could share to make things clearer/easier for people in the future?

Re: [Discuss] Upgrade story for Beam's execution engines

Do we currently maintain a finer grained list of compatibility between execution/runner versions and beam versions? Is this only really a concern with recent Flink (sounded like at least Spark jump, too)? I see the capability matrix:

Re: delayed emit (timer) in py-beam?

apache.org%3E > > Best, > Charles > > On Mon, Jul 30, 2018 at 3:34 PM Austin Bennett < > whatwouldausti...@gmail.com> wrote: > >> What's going on with timers and python? >> >> Am looking at building a pipeline (assuming another group in my company >> w

delayed emit (timer) in py-beam?

What's going on with timers and python? Am looking at building a pipeline (assuming another group in my company will grant access to the Kafka topic): Kafka -> beam -> have beam wait 24 hours -> do transform(s) and emit a record. If I read things correctly that's not currently possible in

Re: Cloud Next 2018 : Catch Up

Hi Guarav, Yes, I'll be around there, happy to meet up. You can follow up with me directly once we get closer, or perhaps we'll figure out a larger thing if more people chime in on this thread that they'll be around. Sessions: didn't see many sessions on Beam/DataFlow at this conference. Best,

Re: CSVSplitter - Splittable DoFn

Hi Beam Users/Dev, How are people handling currently handling CSVs as input to Beam (or not really doing so)? I see the things listed at the start of this thread -- any others? I have many batch workflows involve getting multi-GB CSV files from third party data aggregators (ex: hourly) and

Beam Cookbook?

I'm looking at assembling a physical book along the lines of "Apache Beam Cookbook", though might take a different approach to topic (if realize there is a better hole to fill or something that needs more attention before that). I believe many could benefit from more substantive write-ups and

Re: Initial contributor experience

at 1:49 PM Griselda Cuevas wrote: > >> +user@ in case someone has had similar experiences. >> >> Thanks for documenting this Austin & Pablo! >> >> If any other folks would like to participate in improving the "First >> contribution experience"

Re: Regarding Beam Slack Channel

It'd be easier to follow along there On Jan 8, 2018 9:32 PM, "Shashank Prabhakara" wrote: > I'd also like to be added, please. > > Thanks. > > On 2018-01-04 11:58, Jean-Baptiste Onofr� wrote: > > Hi,> > > > > you should have received the invite.> > > >