Re: [ANNOUNCE] New committer: Ahmed Abualsaud

Congrats Ahmed! On Fri, Aug 25, 2023 at 2:34 PM John Casey via dev wrote: > Congrats Ahmed! > > On Fri, Aug 25, 2023 at 10:43 AM Bjorn Pedersen via dev < > dev@beam.apache.org> wrote: > >> Congrats Ahmed! Well deserved! >> >> On Fri, Aug 25, 2023 at 10:36 AM Yi Hu via dev >> wrote: >> >>>

Re: [ANNOUNCE] New committer: Damon Douglas

Congratulations! On Tue, Apr 25, 2023 at 6:52 AM Bjorn Pedersen via dev wrote: > Congrats Damon! Well deserved! > > On Tue, Apr 25, 2023 at 9:41 AM John Casey via dev > wrote: > >> Congrats Damon! >> >> On Tue, Apr 25, 2023 at 9:36 AM Yi Hu via dev >> wrote: >> >>> Congrats Damon! >>> >>> On

Re: [ANNOUNCE] New committer: Anand Inguva

Congratulations! On Tue, Apr 25, 2023 at 3:48 AM Alexey Romanenko wrote: > Congratulations, Anand! Well deserved! > > On 25 Apr 2023, at 06:02, Byron Ellis via dev wrote: > > Congrats Anand! > > On Mon, Apr 24, 2023 at 9:54 AM Ahmet Altay via dev > wrote: > >> Congratulations Anand! >> >> On

Re: [ANNOUNCE] New PMC Member: Jan Lukavský

Congratulations! On Thu, Feb 16, 2023 at 7:47 AM Robert Burke wrote: > Congratulations! > > On Thu, Feb 16, 2023, 7:44 AM Danielle Syse via dev > wrote: > >> Congrats, Jan! That's awesome news. Thank you for your continued >> contributions! >> >> On Thu, Feb 16, 2023 at 10:42 AM Alexey

Re: [ANNOUNCE] New committer: Ning Kang

Congratulations! On Wed, Mar 24, 2021 at 12:53 PM Kenneth Knowles wrote: > Congratulations and thanks for all your contributions, Ning! > > Kenn > > On Tue, Mar 23, 2021 at 4:10 PM Valentyn Tymofieiev > wrote: > >> Congratulations, Ning, very well deserved! >> >> On Tue, Mar 23, 2021 at 2:01

Re: [ANNOUNCE] New committer: Piotr Szuberski

Congrats! On Tue, Jan 26, 2021 at 4:25 AM Piotr Szuberski wrote: > Thank you everyone! I really don't know what to say. I'm truly honoured. I > do hope I will be able to keep up with the contributions. > > On 2021/01/22 16:32:45, Alexey Romanenko > wrote: > > Hi everyone, > > > > Please join

Re: [ANNOUNCE] New PMC Member: Chamikara Jayalath

Congratulations! On Fri, Jan 22, 2021 at 6:58 AM Ankur Goenka wrote: > Congrats Cham! > > On Thu, Jan 21, 2021 at 2:57 PM Ahmet Altay wrote: > >> Hi all, >> >> Please join me and the rest of Beam PMC in welcoming Chamikara Jayalath >> as our >> newest PMC member. >> >> Cham has been part of

Re: Farewell mail

Thank you Piotr! On Thu, Dec 17, 2020 at 5:30 AM Rui Wang wrote: > Thank you Piotr for all your contributions so far! Wish all the best! > > > -Rui > > On Wed, Dec 16, 2020 at 12:36 PM Ismaël Mejía wrote: > >> Thanks Piotr, >> >> You made an impact on Beam! Best wishes in the future projects

Re: [PROPOSAL] Preparing for Beam 2.26.0 release

+1 Thanx! On Wed, Oct 28, 2020 at 7:14 AM Valentyn Tymofieiev wrote: > +1, thanks and good luck! > > On Tue, Oct 27, 2020 at 1:59 PM Tyson Hamilton wrote: > >> Thanks Rebo! SGTM. >> >> On Tue, Oct 27, 2020 at 11:11 AM Udi Meiri wrote: >> >>> +1 sg! >>> >>> On Tue, Oct 27, 2020 at 10:02 AM

Re: Cleaning up Approximate Algorithms in Beam

com/apache/beam/pull/12973/files#diff-8b3de19b55328b9ca265ec25d57c86e2c59a9c505195146df4af5f9eb73e5fc8> . Thanx Reza On Wed, Nov 27, 2019 at 2:29 AM Robert Bradshaw wrote: > I think this thread is sufficient. > > On Mon, Nov 25, 2019 at 5:59 PM Reza Rokni wrote: > >> Hi, &g

Re: [BEAM-10587] Support Maps in BigQuery #12389

+1 on configuration parameter for enable this and leave current behaviour as default. On Sat, Oct 10, 2020 at 12:35 AM Andrew Pilloud wrote: > BigQuery has no native support for Map types, but I agree that we should > be consistent with how other tools import maps into BigQuery. Is this >

Re: [ANNOUNCE] New committer: Reza Ardeshir Rokni

Thanx everyone! Looking forward to being able to contribute more :-) On Sat, Sep 12, 2020 at 4:33 AM Valentyn Tymofieiev wrote: > Congrats! > > On Thu, Sep 10, 2020 at 8:08 PM Connell O'Callaghan > wrote: > >> Excellent- well done Reza!!! >> >> On Thu, Sep 10, 2020 at 7:35 PM Austin Bennett <

Re: [ANNOUNCE] New committer: Heejong Lee

Congratulations! On Thu, Sep 10, 2020 at 12:20 AM Alexey Romanenko wrote: > Congratulations! > > On 9 Sep 2020, at 09:50, Jan Lukavský wrote: > > Congratulations! > On 9/9/20 1:00 AM, Chamikara Jayalath wrote: > > Congrats Heejong! > > On Tue, Sep 8, 2020 at 1:55 PM Yichi Zhang wrote: > >>

Re: [PROPOSAL] Preparing for Beam 2.25.0 release

Thanx Robin! On Thu, Sep 10, 2020 at 2:30 AM Ahmet Altay wrote: > Thank you Robin! > > On Wed, Sep 9, 2020 at 10:23 AM Rui Wang wrote: > >> Thanks Robin for working on this! >> >> >> -Rui >> >> On Wed, Sep 9, 2020 at 10:11 AM Robin Qiu wrote: >> >>> Hello everyone, >>> >>> The next Beam

Re: [VOTE] Release 2.23.0, release candidate #1

Hi, Are there strong objections to the ability to do patches? Cheers Reza On Fri, Jul 17, 2020 at 9:16 AM Valentyn Tymofieiev wrote: > Thanks for the feedback, help with release validation, and for reaching > out on dev@ regarding a cherry-pick request. > > BEAM-10397

Re: Streaming pipeline "most-recent" join

Hya, I never got a chance to finish this one, maybe I will get some time in the summer break... but I think it will help with your use case...

Re: [ANNOUNCE] New committer: Aizhamal Nurmamat kyzy

Congratulations ! On Tue, Jun 30, 2020 at 2:46 PM Michał Walenia wrote: > Congratulations, Aizhamal! :) > > On Tue, Jun 30, 2020 at 8:41 AM Tobiasz Kędzierski < > tobiasz.kedzier...@polidea.com> wrote: > >> Congratulations Aizhamal! :) >> >> On Mon, Jun 29, 2020 at 11:50 PM Austin Bennett < >>

Re: Request to be Added as Contributor

Welcome ! On Thu, 25 Jun 2020, 20:08 Alexey Romanenko, wrote: > Hi, > > I added you to Contributors list. > > Welcome! > > > On 25 Jun 2020, at 13:36, Abhishek Yadav wrote: > > > > Hi! > > > > I am Abhishek Yadav and I am working as a SWE intern at Google this > summer. > > I would like to be

Re: Jira Board to track most Beam's tickets

Very nice! On Thu, 18 Jun 2020, 05:27 Pablo Estrada, wrote: > I was not aware that you could track by number of watchers. That seems > like a great option. > It looks great Gris, thanks. > Best > -P. > > On Wed, Jun 17, 2020 at 10:44 AM Tyson Hamilton > wrote: > >> This is very cool! I love

Re: [ANNOUNCE] New PMC Member: Alexey Romanenko

Congratulations! On Wed, Jun 17, 2020 at 2:48 PM Michał Walenia wrote: > Congratulations! > > On Tue, Jun 16, 2020 at 11:45 PM Rui Wang wrote: > >> Congrats! >> >> >> -Rui >> >> On Tue, Jun 16, 2020 at 2:42 PM Ankur Goenka wrote: >> >>> Congratulations Alexey! >>> >>> On Tue, Jun 16, 2020 at

Re: [ANNOUNCE] New committer: Robin Qiu

Congratulations! On Tue, May 19, 2020 at 10:06 AM Ahmet Altay wrote: > Hi everyone, > > Please join me and the rest of the Beam PMC in welcoming a new committer: > Robin Qiu . > > Robin has been active in the community for close to 2 years, worked > on HyperLogLog++ [1], SQL [2], improved

Re: [VOTE] Accept the Firefly design donation as Beam Mascot - Deadline Mon April 6

+1(non-binding) On Mon, Apr 6, 2020 at 5:24 PM Alexey Romanenko wrote: > +1 (non-binding). > > > On 3 Apr 2020, at 14:53, Maximilian Michels wrote: > > > > +1 (binding) > > > > On 03.04.20 10:33, Jan Lukavský wrote: > >> +1 (non-binding). > >> > >> On 4/2/20 9:24 PM, Austin Bennett wrote: >

Re: [Interactive Runner] now available on master

Awesome ! On Thu, 19 Mar 2020, 05:38 Sam Rohde, wrote: > Hi All! > > > > I am happy to announce that an improved Interactive Runner is now > available on master. This Python runner allows for the interactive > development of Beam pipelines in a notebook (and IPython) environment. > > > > The

Re: Hello Beam Community!

Welcome! On Sat, 14 Mar 2020, 01:27 Tomo Suzuki, wrote: > Welcome. > > On Fri, Mar 13, 2020 at 1:20 PM Udi Meiri wrote: > >> Welcome! >> >> >> On Fri, Mar 13, 2020 at 9:47 AM Yichi Zhang wrote: >> >>> Welcome! >>> >>> On Fri, Mar 13, 2020 at 9:40 AM Ahmet Altay wrote: >>> Welcome

Re: GroupIntoBatches not Working properly for Direct Runner Java

Hi, There are some known issues with the DirectRunner PubSubIO, which are not present with Dataflow Runner. One of them is around watermarks: https://issues.apache.org/jira/browse/BEAM-7322 Not sure if this is part of the issue here, but worth exploring.. When testing are you sending a small

Re: [ANNOUNCE] New Committer: Kamil Wasilewski

Congratilation Kamil On Sat, 29 Feb 2020, 06:18 Udi Meiri, wrote: > Welcome Kamil! > > On Fri, Feb 28, 2020 at 12:53 PM Mark Liu wrote: > >> Congrats, Kamil! >> >> On Fri, Feb 28, 2020 at 12:23 PM Ismaël Mejía wrote: >> >>> Congratulations Kamil! >>> >>> On Fri, Feb 28, 2020 at 7:09 PM Yichi

Re: [ANNOUNCE] New committer: Chad Dombrova

Congratulations! :-) On Tue, Feb 25, 2020 at 6:41 AM Chad Dombrova wrote: > Thanks, folks! I'm very excited to "retest this" :) > > Especially big thanks to Robert and Udi for all their hard work reviewing > my PRs. > > -chad > > > On Mon, Feb 24, 2020 at 1:44 PM Brian Hulette wrote: > >>

Re: [ANNOUNCE] New committer: Jincheng Sun

Congrats! On Mon, Feb 24, 2020 at 7:15 PM Jan Lukavský wrote: > Congrats Jincheng! > > Jan > > On 2/24/20 11:55 AM, Maximilian Michels wrote: > > Hi everyone, > > > > Please join me and the rest of the Beam PMC in welcoming a new > > committer: Jincheng Sun > > > > Jincheng has worked on

Re: [VOTE][BIP-1] Beam Schema Options

+1 (non-binding) actually working on some pipelines where this would be super useful to have now :-) On Thu, Feb 20, 2020 at 10:09 PM Robert Burke wrote: > +1 (non-binding) it feels like a reasonable extension, and portability is > considered. I see no issues with eventually supporting it in

Re: [ANNOUNCE] New committer: Alex Van Boxel

Fantastic news! Congratulations :-) On Wed, 19 Feb 2020 at 07:54, jincheng sun wrote: > Congratulations！ > Best， > Jincheng > > > Robin Qiu 于2020年2月19日 周三05:52写道： > >> Congratulations, Alex! >> >> On Tue, Feb 18, 2020 at 1:48 PM Valentyn Tymofieiev >> wrote: >> >>> Congratulations! >>> >>> On

Re: Cleaning up Approximate Algorithms in Beam

Hi, I will be making time for completing this in March, with completion and reviews planned for April. Cheers Reza On Wed, 27 Nov 2019 at 02:29, Robert Bradshaw wrote: > I think this thread is sufficient. > > On Mon, Nov 25, 2019 at 5:59 PM Reza Rokni wrote: > >> Hi, &

Re: [ANNOUNCE] Beam 2.18.0 Released

Thank you Udi! On Wed, 29 Jan 2020 at 06:34, Ahmet Altay wrote: > Thank you Udi! > > On Tue, Jan 28, 2020 at 2:13 PM kant kodali wrote: > >> Looks like >> https://beam.apache.org/documentation/runners/capability-matrix/ needs >> to be updated? since there seems to be support for spark

Re: [ANNOUNCE] New committer: Hannah Jiang

Congratz! On Wed, 29 Jan 2020 at 09:52, Valentyn Tymofieiev wrote: > Congratulations, Hannah! > > On Tue, Jan 28, 2020 at 5:46 PM Udi Meiri wrote: > >> Welcome and congrats Hannah! >> >> On Tue, Jan 28, 2020 at 4:52 PM Robin Qiu wrote: >> >>> Congratulations, Hannah! >>> >>> On Tue, Jan 28,

Re: [ANNOUNCE] New committer: Michał Walenia

Congratulations buddy! On Tue, 28 Jan 2020, 06:52 Valentyn Tymofieiev, wrote: > Congratulations, Michał! > > On Mon, Jan 27, 2020 at 2:24 PM Austin Bennett < > whatwouldausti...@gmail.com> wrote: > >> Nice -- keep up the good work! >> >> On Mon, Jan 27, 2020 at 2:02 PM Mikhail Gryzykhin >>

Re: Dynamic timers now supported!

Very cool ! Thank you Rehman! On Fri, 24 Jan 2020, 06:12 Maximilian Michels, wrote: > Great work! That makes timers so much easier to use and also adds new > use cases. Thank you Rehman. > > On 23.01.20 22:54, Robert Burke wrote: > > Fascinating and great work! *makes notes for an eventual Go

Re: [DISCUSS] Integrate Google Cloud AI functionalities

+1 for using cross language transforms. On Thu, 16 Jan 2020 at 01:23, Ahmet Altay wrote: > > > On Wed, Jan 15, 2020 at 8:12 AM Kamil Wasilewski < > kamil.wasilew...@polidea.com> wrote: > >> Based on your feedback, I think it'd be fine to deal with the problem as >> follows: >> * for Python: put

Re: [Proposal] Slowly Changing Dimensions and Distributed Map Side Inputs (in Dataflow)

+1 To this proposal, this is a very common pattern requirement from users. With the following current workaround having seen a lot of traction: https://beam.apache.org/documentation/patterns/side-inputs/#slowly-updating-global-window-side-inputs Making this process simpler for users and Out Of

Unset / delete Timers

Hi, Was exploring the ability to add unset / reset option for Timer, would this be an expensive operation for runners to support? More complex State and Timer use cases can require this operation and while its possible to do today using a separate State Object, its heavy on boiler plate and

Re: Beam Testing Tools FAQ

Thanx! On Wed, 27 Nov 2019, 02:31 Pablo Estrada, wrote: > Very cool. Thanks Lukasz! > > On Tue, Nov 26, 2019 at 9:41 AM Alan Myrvold wrote: > >> Nice, thanks! >> >> On Tue, Nov 26, 2019 at 8:04 AM Robert Bradshaw >> wrote: >> >>> Thanks! >>> >>> On Tue, Nov 26, 2019 at 7:43 AM Łukasz Gajowy

Re: Full stream-stream join semantics

il they can no longer be >>>>>>> matched >>>>>>> in the join, and you still need to query that buffer for elements that >>>>>>> might match. The general "bi-temporal join" (without sorting) requires >>>>>>>

Re: [DISCUSS] AWS IOs V1 Deprecation Plan

Hi Alexey, With regards to @Experimental there are a couple of discussions around its usage ( or rather over usage! ) on dev@. It is something that we need to clean up ( some of those IO are now being used on production env for years!). Cheers Reza On Wed, 27 Nov 2019 at 04:50, Luke Cwik

Re: Cleaning up Approximate Algorithms in Beam

Hi, So do we need a vote for the final list of actions? Or is this thread enough to go ahead and raise the PR's? Cheers Reza On Tue, 26 Nov 2019 at 06:01, Ahmet Altay wrote: > > > On Mon, Nov 18, 2019 at 10:57 AM Robert Bradshaw > wrote: > >> On Sun, Nov 17, 2019

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[ ] Beaver [ ] Hedgehog [ ] Lemur [ ] Owl [X] Salmon [ ] Trout [ ] Robot dinosaur [ ] Firefly [ ] Cuttlefish [X] Dumbo Octopus [X] Angler fish On Wed, 20 Nov 2019 at 10:43, Kenneth Knowles wrote: > Please cast your votes of approval [1] for animals you would support as > Beam mascot. The animal

Re: Cleaning up Approximate Algorithms in Beam

ludes zetasketch, perhaps. >> 2. Deprecate the others, at least. And remove them from e.g. Javadoc. >> > > +1 > > >> On Wed, Nov 13, 2019 at 10:01 AM Reuven Lax wrote: >> >>> >>> >>> On Wed, Nov 13, 2019 at 9:58 AM Ahmet Altay wrote:

Re: [PROPOSAL] Add support for writing flattened schemas to pubsub

+1 to reduced boiler plate for basic things folks want to do with SQL. I like Alex use of Option for more advanced use cases. On Sun, 17 Nov 2019 at 20:17, Gleb Kanterov wrote: > Expanding on what Kenn said regarding having fewer dependencies on SQL. > Can the whole thing be seen as extending

Re: [ANNOUNCE] New committer: Brian Hulette

Great news! On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov wrote: > Congratulations! > > On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev > wrote: > >> Congratulations, Brian! >> >> On Thu, Nov 14, 2019 at 6:25 PM jincheng sun >> wrote: >> >>> Congratulation Brian! >>> >>> Best, >>> Jincheng

Cleaning up Approximate Algorithms in Beam

Hi everyone; TL/DR : Discussion on Beam's various Approximate Distinct Count algorithms. Today there are several options for Approximate Algorithms in Apache Beam 2.16 with HLLCount being the most recently added. Would like to canvas opinions here on the possibility of rationalizing these API's

Re: [Discuss] Beam Summit 2020 Dates & locations

Hi, I think doing something in Asia later in the year would be very cool, I think given there has not yet been a lot of formal activity there, we should give plenty of time to ensure good local sponsorship etc... Cheers Reza On Fri, 8 Nov 2019 at 19:10, jincheng sun wrote: > +1 for extend

Re: [Discuss] Beam mascot

Salmon... they love streams? :-) On Fri, 8 Nov 2019 at 12:00, Kenneth Knowles wrote: > Agree with Aizhamal that it doesn't matter if they are taken if they are > not too close in space to Beam: Apache projects, big data, log processing, > stream processing. Not a legal opinion, but an aesthetic

Re: Apache Pulsar connector for Beam

Hi Taher, You can see the list of current and wip IO's here: https://beam.apache.org/documentation/io/built-in/ Cheers Reza On Thu, 24 Oct 2019 at 13:56, Taher Koitawala wrote: > Hi All, > Been wanting to know if we have a Pulsar connector for Beam. > Pulsar is another messaging

Re: Proposal: Dynamic timer support (BEAM-6857)

+1 on this, having the ability to create timers based on data would make a bunch of use cases easier to write. Any thoughts on having a isSet() / read() / setMinimum(timeStamp) type ability? On Wed, 23 Oct 2019 at 00:52, Reuven Lax wrote: > Kenn: > +1 to using TimerFamily instead of TimerId

Re: Introduction + Support in Comms for Beam!

Welcome! On Tue, 1 Oct 2019 at 11:18, Lukasz Cwik wrote: > Welcome to the community. > > On Mon, Sep 30, 2019 at 3:15 PM María Cruz wrote: > >> Hi everyone, >> my name is María Cruz, I am from Buenos Aires but I live in the Bay Area. >> I recently became acquainted with Apache Beam project,

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

+1 On Tue, 1 Oct 2019 at 13:54, Tanay Tummalapalli wrote: > +1 > > On Tue, Oct 1, 2019 at 8:19 AM Suneel Marthi wrote: > >> +1 >> >> On Mon, Sep 30, 2019 at 10:33 PM Manu Zhang >> wrote: >> >>> +1 >>> >>> On Tue, Oct 1, 2019 at 9:44 AM Austin Bennett < >>> whatwouldausti...@gmail.com> wrote:

Re: [ANNOUNCE] New committer: Valentyn Tymofieiev

Thanks Valentin! On Tue, 27 Aug 2019, 05:32 Pablo Estrada, wrote: > Thanks Valentyn! > > On Mon, Aug 26, 2019 at 2:29 PM Robin Qiu wrote: > >> Thank you Valentyn! Congratulations! >> >> On Mon, Aug 26, 2019 at 2:28 PM Robert Bradshaw >> wrote: >> >>> Hi, >>> >>> Please join me and the rest of

Re: SqlTransform Metadata

:06, Reza Rokni wrote: > Hi, > > Coming back to this do we have enough of a consensus to say that in > principle this is a good idea? If yes I will raise a Jira for this. > > Cheers > > Reza > > On Thu, 16 May 2019 at 02:58, Robert Bradshaw wrote: > >>

Re: Dataflow worker overview graphs

Very nice resource, thanx Mikhail. On Fri, 9 Aug 2019 at 05:22, Mikhail Gryzykhin wrote: > Unfortunately no, I don't have those for streaming explicitly. > > However most of code is shared between streaming and batch with main > difference in initialization. Same goes for boilerplate parts of

Re: [VOTE] Support ZetaSQL as another SQL dialect for BeamSQL in Beam repo

+1 On Tue, 13 Aug 2019 at 09:28, Ahmet Altay wrote: > +1 > > On Mon, Aug 12, 2019 at 6:27 PM Kenneth Knowles wrote: > >> +1 >> >> On Mon, Aug 12, 2019 at 4:43 PM Rui Wang wrote: >> >>> Hi Community, >>> >>> I am using this separate thread to collect votes on contributing Beam >>> ZetaSQL(my

Re: [ANNOUNCE] New committer: Kyle Weaver

Congratz! On Wed, 7 Aug 2019 at 06:40, Chamikara Jayalath wrote: > Congrats!! > > On Tue, Aug 6, 2019 at 3:33 PM Udi Meiri wrote: > >> Congrats Kyle! >> >> On Tue, Aug 6, 2019 at 2:00 PM Melissa Pashniak >> wrote: >> >>> Congratulations Kyle! >>> >>> On Tue, Aug 6, 2019 at 1:36 PM Yichi Zhang

Re: [ANNOUNCE] New committer: Jan Lukavský

Congratulations , awesome stuff ! On Thu, 1 Aug 2019, 12:11 Maximilian Michels, wrote: > Congrats, Jan! Good to see you become a committer :) > > On 01.08.19 12:37, Łukasz Gajowy wrote: > > Congratulations! > > > > czw., 1 sie 2019 o 11:16 Robert Bradshaw > >

Re: Slowly changing lookup cache as a Table in BeamSql

tensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java#L186> >> *. * >> >> Create a JIRA to track this idea: >> https://jira.apache.org/jira/browse/BEAM-7758 >> >> >> -Rui >> >> >> On Tue, Jul 16, 2019 at 7:12 A

Re: Slowly changing lookup cache as a Table in BeamSql

blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java#L186> > *. * > > Create a JIRA to track this idea: > https://jira.apache.org/jira/browse/BEAM-7758 > > > -Rui > > > On Tue, Jul 16, 2019 at 7:12 AM Reza Rokni wro

Re: [ANNOUNCE] New committer: Robert Burke

Congratulations ! On Tue, 16 Jul 2019, 18:24 Ahmet Altay, wrote: > Hi, > > Please join me and the rest of the Beam PMC in welcoming a new committer: > Robert > Burke. > > Robert has been contributing to Beam and actively involved in the > community for over a year. He has been actively working

Re: Slowly changing lookup cache as a Table in BeamSql

Hi Rahul, FYI, that patterns is also available in the Beam docs ( with updated code example ) https://beam.apache.org/documentation/patterns/side-input-patterns/. Please note in the DoFn that feeds the View.asSingleton() you will need to manually call BigQuery using the BigQuery client.

Re: Looping timer blog

On Tue, 25 Jun 2019 at 21:20, Jan Lukavský wrote: > > On 6/25/19 1:43 PM, Reza Rokni wrote: > > > > On Tue, 25 Jun 2019 at 18:12, Jan Lukavský wrote: > >> > The TTL check would be in the same Timer rather than a separate Timer. >> The max value processed

Re: Return types of Write transforms (aka best way to signal)

The use case of a transform waiting for a SInk or Sinks to complete is very interesting indeed! Curious, if a sink internally makes use of a Global Window with processing time triggers to push its writes, what mechanism could be used to release a transform waiting for a signal from the Sink(s)

Re: Blogpost Beam Summit 2019

Thank you for putting this together! On Wed, 26 Jun 2019 at 01:23, Ahmet Altay wrote: > Thank you for writing and sharing this. I enjoyed reading it :) I think it > is worth sharing it as a tweet [1] as well. > > [1] s.apache.org/beam-tweets > > On Tue, Jun 25, 2019 at 10:16 AM Valentyn

Re: Looping timer blog

late data is a sharp edge, one solution for late data is to branch it away before GlobalWindow + State DoFn. This can then be output from the pipeline into a sink with a marker to initiate a manual procedure for correction. Essentially a manual redaction. Or? Am I missing something? > > Jan

Re: Looping timer blog

of this. > > Cheers, > > Jan > > [1] https://issues.apache.org/jira/browse/BEAM-7520 > > [2] > https://lists.apache.org/thread.html/1a3a0dd9da682e159f78f131d335782fd92b047895001455ff659613@%3Cdev.beam.apache.org%3E > > [3] > https://docs.google.com/document/d/1ObLVUFsf1NcG8ZuIZE4a

Re: Testing code in extensions against runner

tension, not tests of the runner. So that is another reason they > should not be @ValidatesRunner. > > It would be snazzy if it were a one-liner to add all this testing in the > gradle config for extensions. > > Kenn > > On Tue, Jun 18, 2019 at 12:05 AM Reza Rokni wrote: > >> Th

Re: [ANNOUNCE] New committer: Mikhail Gryzykhin

Congratulations! On Fri, 21 Jun 2019, 12:37 Robert Burke, wrote: > Congrats > > On Fri, Jun 21, 2019, 12:29 PM Thomas Weise wrote: > >> Hi, >> >> Please join me and the rest of the Beam PMC in welcoming a new committer: >> Mikhail Gryzykhin. >> >> Mikhail has been contributing to Beam and

Re: Looping timer blog

p. > Manu > > On Wed, Jun 19, 2019 at 12:15 PM Reza Rokni wrote: > >> Hi folks, >> >> Just wanted to drop a note here on a new pattern that folks may find >> interesting, called Looping Timers. It allows for default values to be >> created in interval w

Looping timer blog

Hi folks, Just wanted to drop a note here on a new pattern that folks may find interesting, called Looping Timers. It allows for default values to be created in interval windows in the absence of any external data coming into the pipeline. The details are in this blog below:

Re: Testing code in extensions against runner

ut how well a feature is > supported by each runner automatically. > > > > On Thu, Jun 6, 2019 at 8:36 PM Reza Rokni wrote: > >> Hi, >> >> I would like to validate some code that I am building under >> extensions against different runners. It makes use of

Re: [INPUT PLEASE] Introduction to Committing Workshop @BeamSummit

For patterns the "pipline-patterns" label would be good to add as well: https://beam.apache.org/documentation/dsls/sql/overview/ On Mon, 10 Jun 2019 at 11:22, Henry Suryawirawan wrote: > Adding new Kata might also be an option for people to contribute. > > > > > On Mon, Jun 10, 2019 at 7:00

Re: @RequireTimeSortedInput design draft

[2] > https://github.com/apache/beam/blob/master/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/Join.java > On 6/7/19 7:06 AM, Reza Rokni wrote: > > Hi Jan, > > I have been working on a timeseries extension which makes use of

Re: [ANNOUNCEMENT] Common Pipeline Patterns - new section in the documentation + contributions welcome

from >> StackOverflow, and from talking to users of Beam. >> >> It would be great if this section could grow, and be useful to many Beam >> users. For that reason, we invite anyone to share patterns, and pipeline >> examples that they have used in the past. If you a

Re: @RequireTimeSortedInput design draft

Hi Jan, I have been working on a timeseries extension which makes use of many of these techniques for joining two temporal streams, it's almost ready for the PR, will ping it here when it is as it might be useful for you. In general, I borrowed a lot of techniques from CoGroupBy code. *1) need

Testing code in extensions against runner

Hi, I would like to validate some code that I am building under extensions against different runners. It makes use of some caches in a DoFn which are a little off the beaten path. I have added @ValidatesRunner to the class and by adding the right values to the gradle file in flink_runner have

Re: Timer support in Flink

udes/capability-matrix.md#L18 > > [2] > https://github.com/apache/beam/blob/master/website/src/_sass/capability-matrix.scss#L130 > > > > > > On Mon, Jun 3, 2019 at 2:15 AM Maximilian Michels > wrote: > >> > >> Good point. I think I discovered th

Re: [DISCUSS] Cookbooks for users with knowledge in other frameworks

For layer 1, what about working through this link as a starting point : https://spark.apache.org/docs/latest/rdd-programming-guide.html#transformations ? On Sat, 1 Jun 2019 at 09:21, Ahmet Altay wrote: > Thank you Reza. That separation makes sense to me. > > On Wed, May 29, 2019 at 6:2

Re: Timer support in Flink

:-) https://issues.apache.org/jira/browse/BEAM-7456 On Thu, 30 May 2019 at 18:41, Alex Van Boxel wrote: > Oh... you can expand the matrix. Never saw that, this could indeed be > better. So it isn't you. > > _/ > _/ Alex Van Boxel > > > On Thu, May 30, 2019 at 12

Re: Shuffling on apache beam

Milli(); > > > context.output( > > > new PubsubMessage( > > > payload.getBytes(), > > > ImmutableMap.of(TRANSACTION_MESSAGE_ID_FIELD_NAME, > > > payload.split(",")[6],TRANSAC

Re: Timer support in Flink

PS, until it was just pointed out to me by Max, I had missed the (expand details) clickable link in the capability matrix. Probably just me, but do others think it's also easy to miss? If yes I will raise a Jira for it On Wed, 29 May 2019 at 19:52, Reza Rokni wrote: > Thanx Max! >

Re: [DISCUSS] Cookbooks for users with knowledge in other frameworks

+1 I think there will be at least two layers of this; Layer 1 - Using primitives : I do join, GBK, Aggregation... with system x this way, what is the canonical equivalent in Beam. Layer 2 - Patterns : I read and join Unbounded and Bounded Data in system x this way, what is the canonical

Re: Hazelcast Jet Runner

wrote: > > > On Mon, May 27, 2019 at 4:05 PM Reza Rokni wrote: > >> "Many APIs that have been in place for years and are used by most Beam >> users are still marked Experimental." >> >> Should there be a formal process in place to start 'graduating'

Re: Timer support in Flink

ng fine. > > > It makes very heavy use of Event.Time timers and has to do some manual > DoFn cache work to get around some O(heavy) issues. > > If you are running on Flink 1.5, timer deletion suffers from O(n) > complexity which has been fixed in newer versions. > > Cheers

Timer support in Flink

Hi Flink experts, I am getting ready to push a PR around a utility class for timeseries join left.timestamp match to closest right.timestamp where right.timestamp <= left.timestamp. It makes very heavy use of Event.Time timers and has to do some manual DoFn cache work to get around some

Re: Hazelcast Jet Runner

"Many APIs that have been in place for years and are used by most Beam users are still marked Experimental." Should there be a formal process in place to start 'graduating' features out of @Experimental? Perhaps even target an up coming release with a PR to remove the annotation from well

Re: Shuffling on apache beam

PS You can also make use of the GlobalWindow with a stateful DoFn. On Fri, 24 May 2019 at 15:13, Reza Rokni wrote: > Hi, > > Have you explored the use of triggers with your use case? > > > https://beam.apache.org/releases/javadoc/2.12.0/org/apache/beam/sdk/transforms/wind

Re: Shuffling on apache beam

Hi, Have you explored the use of triggers with your use case? https://beam.apache.org/releases/javadoc/2.12.0/org/apache/beam/sdk/transforms/windowing/Trigger.html Cheers Reza On Fri, 24 May 2019 at 14:14, pasquale.bon...@gmail.com < pasquale.bon...@gmail.com> wrote: > Hi Reuven, > I would

Re: DISCUSS: Sorted MapState API

+100 As someone who has just spent a lot of time coding all the "GC, caching of BagState fetches, etc" this would make life a lot easier! Its also generally valuable for a lot of timeseries work. On Fri, 24 May 2019 at 04:59, Lukasz Cwik wrote: > > > On Thu, May 23, 2019 at 1:53 PM Ahmet Altay

Re: Better naming for runner specific options

Its options are populated by querying the >> job service. Any portable runner is able to provide a list of options that >> is runner specific through that mechanism. >> >> *From: *Reza Rokni >> *Date: *Mon, May 6, 2019 at 2:57 PM >> *To: * >> >> So t

Re: SqlTransform Metadata

t, because SQL already has TIMESTAMP > columns and it is very natural to use SQL on unbounded relations where the > timestamp is just part of the data. > > That's why I was suggesting a single element_metadata() rather than > exploding each one out. > > Do you have a pointer to w

Re: Definition of Unified model (WAS: Semantics of PCollection.isBounded)

should support > these use cases more "natively". > > Cheers, > > Jan > On 5/21/19 11:03 AM, Reza Rokni wrote: > > In a lot of cases the initial combiner can dramatically reduce the amount > of data in this last phase making it tractable for a lot of use cases.

Re: Definition of Unified model (WAS: Semantics of PCollection.isBounded)

Hi Reza, thanks for reaction, comments inline. > On 5/21/19 1:02 AM, Reza Rokni wrote: > > Hi, > > If I have understood the use case correctly, your output is an ordered > counter of state changes. > > One approach which might be worth exploring is outlined below, haven't &

Re: Definition of Unified model (WAS: Semantics of PCollection.isBounded)

Hi, If I have understood the use case correctly, your output is an ordered counter of state changes. One approach which might be worth exploring is outlined below, haven't had a chance to test it so could be missing pieces or be plane old wrong ( will try and come up with a test example later

Re: [ANNOUNCE] New PMC Member: Pablo Estrada

Awesome news :-) *From: *Kenneth Knowles *Date: *Wed, 15 May 2019, 11:25 *To: *dev Hi all, > > Please join me and the rest of the Beam PMC in welcoming Pablo Estrada to > join the PMC. > > Pablo first picked up BEAM-722 in October of 2016 and has been a steady > part of the Beam community since

Re: SqlTransform Metadata

e whether it makes sense to build >> something extra that probably looks and functions like an existing feature. >> >> Regards, >> Anton >> >> >> >> *From: *Andrew Pilloud >> *Date: *Tue, May 14, 2019 at 7:29 AM >> *To: *dev >> >&g

Re: Contributing Beam Kata (Java & Python)

+1 :-) *From: *Lukasz Cwik *Date: *Wed, 15 May 2019 at 04:29 *To: *dev *Cc: *Lars Francke +1 > > *From: *Pablo Estrada > *Date: *Tue, May 14, 2019 at 1:27 PM > *To: *dev > *Cc: *Lars Francke > > +1 on merging. >> >> *From: *Reuven Lax >> *Date: *Tue, May 14, 2019 at 1:23 PM >> *To: *dev >>

SqlTransform Metadata

Hi, What are folks thoughts about adding something like SqlTransform.withMetadata().query(...)to enable users to be able to access things like Timestamp information from within the query without having to refiy the information into the element itself? Cheers Reza -- This email may be

  1   2   >