ElasticsearchIOTest failed during gradle build

Hey all, I'm working on debugging the process of release process and when running ./gradlew -PisRelease clean build, I got several tests failed. Here is one build scan: https://scans.gradle.com/s/t4ryx7y3jhdeo/console-log?task=:beam-sdks-java-io-elasticsearch-tests-5:test#L3. Any idea about why

Re: Missing copyright notices for shaded packages

Hey all, Release blocker https://issues.apache.org/jira/browse/BEAM-4393 marked as Resolved. Boyuan On Wed, May 23, 2018 at 1:52 PM Scott Wegner wrote: > FYI, I've opened https://issues.apache.org/jira/browse/BEAM-4393 to track > this work and marked it as a 2.5.0 release

Re: Java sdk tests are broken

ik <lc...@google.com> wrote: > Last successful postcommit was from about an hour ago at commit > 1e698901336b3499ad48d1bd2110963e45f364b7: > > https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/426/ > > What commit are you at? > > On Wed, May 16

Re: Java sdk tests are broken

ending PR with > https://github.com/apache/beam/commit/72b6c8f6d0923da641b5344eb6a1fa8c260d596c > > On Tue, May 15, 2018 at 3:41 PM Boyuan Zhang <boyu...@google.com> wrote: > >> Hey all, >> >> There are several Java tests failure now. Would anyone like taking a look

Java sdk tests are broken

Hey all, There are several Java tests failure now. Would anyone like taking a look at them? I collected 2 build failure links here: - https://scans.gradle.com/s/7dw7t34jxtqnm/ - https://scans.gradle.com/s/xialvtd3wqu6u Thanks so much for your attention! Boyuan

Re: [PROPOSAL] Preparing 2.5.0 release next week

nks, yeah I saw that but I'm planning to add some additional notes. > > Regards > JB > Le 5 juin 2018, à 19:02, Boyuan Zhang a écrit: >> >> Hey JB, >> >> We have some updates in : >> https://github.com/pabloem/beam-site/blob/372c93ecbafbf3a1440492df1e1

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

Hey JB, Could you please also build python wheels for RC2 and stage them in https://dist.apache.org/repos/dist/dev/beam/2.5.0/? There are some instructions in this PR: https://github.com/apache/beam-site/pull/467. Hope it can be helpful. Thanks. Boyuan On Sun, Jun 17, 2018 at 9:22 PM

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

at 6:08 PM Boyuan Zhang wrote: > Hey JB, > > I built python wheels and you can get them here: > https://storage.googleapis.com/python_wheels/python_wheels.zip. You > probably need to sign them and upload them into > https://dist.apache.org/repos/dist/dev/beam/2.5.0/ like what

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

> On Mon, Jun 18, 2018 at 10:08 AM Jean-Baptiste Onofré > wrote: > >> Hi, >> >> I thought I did it, let me double check. >> >> Regards >> JB >> >> On 18/06/2018 19:01, Boyuan Zhang wrote: >> > Hey JB, >> > >> >

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

on Python > > > source build). And we should try to incorporate a fix > > if we happen > > > to cut another release candidate for some reason. > > > > > > Thanks, > >

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

ost. The >>>>>> issue is actually quite small: >>>>>> >>>>>> - The JAR from the non-LGPL findbugs does not contain the >>>>>> SuppressFBWarnings annotation. This means that when building, ByteBuddy >>>>>&

Suggestions for Building Python Wheels

Hey all, I'm Boyuan and I'm currently working on release process. We began to release python wheels since last release so we wanna make this process as easy as possible. This doc describes how we built wheels for

Re: Suggestions for Building Python Wheels

> On Tue, Jun 19, 2018 at 3:45 PM, Boyuan Zhang wrote: > >> Hey all, >> >> I'm Boyuan and I'm currently working on release process. We began to >> release python wheels since last release so we wanna make this process >> as easy as possible.

Re: [PROPOSAL] Preparing 2.5.0 release next week

Hey JB, We have some updates in : https://github.com/pabloem/beam-site/blob/372c93ecbafbf3a1440492df1e12050dfe939e91/src/contribute/release-guide.md and https://github.com/apache/beam-site/pull/424/files

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

Hey JB, Could you please share your apache-beam-wheels repo and your build log? Maybe we can help debug together. Boyuan On Fri, Jun 22, 2018 at 7:51 AM Jean-Baptiste Onofré wrote: > Gently reminder for the PMC member. > > Right now, if I'm not wrong, we only have two concrete binding votes >

Apache Beam Python Wheels Repository

Hey all, I'm Boyuan Zhang from Google Dataflow Team, currently helping Release Manager(Pablo Estrada) with 2.6.0 release. Since Beam decided to release python wheels since 2.5.0, we need to create a wrapper repository(sample repo <https://github.com/boyuanzz/apache-beam-wheels>) under

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #1

+1 Tested Dataflow related items in: https://s.apache.org/beam-release-validation On Wed, Aug 1, 2018 at 11:40 AM Yifan Zou wrote: > +1 > Tested Python quickstarts and mobile gaming examples against tar and wheel > versions. > https://builds.apache.org/job/beam_PostRelease_Python_Candidate/123/

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #2

+1 non-binding Verified dataflow items listed in https://s.apache.org/beam-release-validation On Sun, Aug 5, 2018 at 6:37 AM Suneel Marthi wrote: > +1 non-binding > > 1. verified Sigs and Hashes of artifacts > 2. tested with my sample applications with local Runner > > On Sun, Aug 5, 2018 at

Re: Apache Beam Python Wheels Repository

gt; Best > -P. > > On Tue, Aug 7, 2018 at 2:04 PM Boyuan Zhang wrote: > >> Thanks for Davor's suggestions. Following content discusses more about >> why we need a new repository. Hope they would be helpful! >> >> Purpose of this proposal >> - Facilitate buil

Re: Apache Beam Python Wheels Repository

m/matthew-brett/multibuild/blob/devel/README.rst>, we need to use a repo wrapper. Alternatives - Unfortunately, I didn't find any alternatives at this moment. Boyuan Zhang On Sat, Aug 4, 2018 at 9:50 AM Davor Bonaci wrote: > New repository is not a ticket, it is a self-serve thing. > &

Re: [GitHub] aaltay commented on issue #1: Add .gitmodules file

Hey all, beam-wheels has sent messages to commits@. Thanks you dealing with the INFRA ticket, Ahmet! Boyuan Zhang On Tue, Aug 14, 2018 at 3:55 PM Jason Kuster wrote: > It's probably an INFRA ticket to change the mailing list preferences. > > On Tue, Aug 14, 2018 at 8:47 AM Boyuan Zha

[Build Python Wheels] New Repository for Building Python Wheels

am-wheels#apache-beam-wheels> to get more info if you are interested. Boyuan Zhang

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #2

t's harmless. > > I also noticed we are not building wheels for Windows - is that expected? > > Thanks, > Valentyn > > On Tue, Aug 7, 2018 at 7:12 AM Ahmet Altay wrote: > >> +1 (binding) >> >> Thank you. >> >> On Mon, Aug 6, 2018 at 4:21 PM, B

Re: [Vote] Create apache-beam-wheels repo like beam-site

Hey all, Thanks for everyone's votes and participations. We have decided to create this wrapper repo and will follow up with the detail guide about how to use this one asap. Also we will close this vote thread at this time. Appreciate all of your help! Best, Boyuan Zhang On Fri, Aug 10, 2018

Re: [GitHub] aaltay commented on issue #1: Add .gitmodules file

Hey Jason, I'm going to figure out this problem today. Do you have idea what should we do or where should we start? Boyuan On Tue, Aug 14, 2018 at 1:15 AM Jason Kuster wrote: > Looks like these are getting sent to dev. Is there a per-project config we > need to tweak or a ticket we need to

[Vote] Create apache-beam-wheels repo like beam-site

eels on multi-os. Please vote for this decision or raise your concerns. Appreciate your help! Boyuan Zhang

Re: Travis apache credentials

Hey Robert, I think your idea would be possible if following things are possible: 1. Link beam-wheels into travis-cli. I'm not sure who has the right permission to perform this operation. 2. We have a common svn credential or one of the beam committers would like to put his(or her) credential

Re: [PROPOSAL] Prepare Beam 2.7.0 release

+1 Thanks for volunteering, Charles! On Mon, Aug 20, 2018 at 3:22 PM Rafael Fernandez wrote: > +1, thanks for volunteering, Charles! > > On Mon, Aug 20, 2018 at 12:09 PM Charles Chen wrote: > >> Thank you Andrew for pointing out my mistake. We should follow the >> calendar and aim to cut on

Design Proposal for python wheels build process

Hey all, After 2.4.0, we also publish python wheels as part of python artifacts. In order to make release process easier, we need a standardized way to build and stage python wheels as s part of release process. In this deign proposal[1], I debriefed how we built python wheels for 2.4.0 & 2.5.0,

builds.apache.org refused connections since last night

Hey all, It seems like that builds.apache.org cannot be reached from last night. Does anyone have the sam e connection problem? Any idea what can we do? Boyuan Zhang

Re: builds.apache.org refused connections since last night

ite > changes? > > > On Wed, Aug 29, 2018 at 9:19 AM Boyuan Zhang wrote: > >> Thank you Andrew! >> >> On Wed, Aug 29, 2018 at 9:17 AM Andrew Pilloud >> wrote: >> >>> Down for me too. It sounds like the disk failed and it will be down for >>>

Re: builds.apache.org refused connections since last night

Thank you Andrew! On Wed, Aug 29, 2018 at 9:17 AM Andrew Pilloud wrote: > Down for me too. It sounds like the disk failed and it will be down for a > while: https://status.apache.org/ > > Andrew > > On Wed, Aug 29, 2018 at 9:13 AM Boyuan Zhang wrote: > >> H

Re: Permission to get JIRAs assigned to me

Hey Kenn, Thanks so much for your help! Boyuan Zhang On Wed, Apr 11, 2018 at 3:54 PM Kenneth Knowles <k...@google.com> wrote: > You need to be given the "Contributors" role in JIRA. I have done so - > give it a try. > > Kenn > > On Wed, Apr 11, 2018 at 3:24 P

Permission to get JIRAs assigned to me

Hey all, I created several JIRAs in beam and wanna assign them to myself but it looks like I don't have right permissions yet. Could you please help me to figure out how to get right permissions? By the way, I have filed ICLA. Thanks so much for all your help! Boyuan Zhang

Re: python post-commit failures

Seems like tests failed: test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) -> Bigquery table not found test_game_stats_it (apache_beam.examples.complete.game.game_stats_it_test.GameStatsIT) -> Bigquery table not found streaming related tests -> Assertion

Re: Java PreCommit and PostCommit cannot be triggered from PR

be related to instability of Github: >> https://status.github.com/messages. >> >> -Rui >> >> On Mon, Oct 22, 2018 at 11:22 AM Boyuan Zhang wrote: >> >>> Hey, >>> >>> It seems like the Java PreCommit and Java PostCommit cannot be triggere

Java PreCommit and PostCommit cannot be triggered from PR

Hey, It seems like the Java PreCommit and Java PostCommit cannot be triggered from PR from this morning. Does anyone have the same issue or have any idea what happened? Boyuan Zhang

Please ignore the 'Java FnApi PreCommit' and 'Java FnApi PostCommit' failures

Hey all, I'm working on adding 2 more Jenkins Jobs to run java PreCommit and PostCommit with fn-api worker and stabilizing the job status. Please ignore failures from these 2 jobs. Once they are ready, there will be another email to follow up. Sorry for the inconvenience! Best, Boyuan Zhang

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

Hey all, We are tracking the dataflow worker donating process here: https://issues.apache.org/jira/browse/BEAM-5634 . Boyuan Zhang On Mon, Sep 17, 2018 at 5:05 PM Lukasz Cwik wrote: > Thanks all, closing the vote with 18 +1s, 5 of which are binding. > > I'll try to get this

Re: org.apache.beam.sdk.io.FileIOTest.testMatchWatchForNewFiles failing

The BEAM-6352 is tracking this failure. On Fri, Jan 18, 2019 at 8:48 PM Reuven Lax wrote: > This test is consistently failing for me. Has anyone else seen this? > > Reuven >

Re: What is Jenkins job "Portable_Python" in PreCommit?

I believe it's in this pending PR: https://github.com/apache/beam/pull/7157. On Thu, Nov 29, 2018 at 8:36 AM Mark Liu wrote: > Hi guys, > > I made some changes to Python > PreCommit Gradle and then Portable_Python is invoked as a PrecCommit test > and

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 On Fri, Sep 14, 2018 at 3:15 PM Henning Rohde wrote: > +1 > > On Fri, Sep 14, 2018 at 2:40 PM Ahmet Altay wrote: > >> +1 (binding) >> >> On Fri, Sep 14, 2018 at 2:35 PM, Lukasz Cwik wrote: >> >>> +1 (binding) >>> >>> On Fri, Sep 14, 2018 at 2:34 PM Pablo Estrada >>> wrote: >>> +1

Re: SNAPSHOTS have not been updated since february

+Daniel Oliveira On Tue, Mar 26, 2019 at 9:57 AM Boyuan Zhang wrote: > Sorry for the typo. Ideally, the snapshot publish is *independent* from > postrelease_snapshot. > > On Tue, Mar 26, 2019 at 9:55 AM Boyuan Zhang wrote: > >> Hey, >> >> I'm trying to publish

Re: SNAPSHOTS have not been updated since february

Sorry for the typo. Ideally, the snapshot publish is *independent* from postrelease_snapshot. On Tue, Mar 26, 2019 at 9:55 AM Boyuan Zhang wrote: > Hey, > > I'm trying to publish the artifacts by commenting "Run Gradle Publish" in > my PR, but there are several error

Re: SNAPSHOTS have not been updated since february

ve this is related to > https://issues.apache.org/jira/browse/BEAM-6840 and +Boyuan Zhang > has a fix in progress > https://github.com/apache/beam/pull/8132 > > On Tue, Mar 26, 2019 at 7:09 AM Ismaël Mejía wrote: > >> I was trying to validate a fix on the Spark runner and realiz

Frequent failures on beam8

Hey all, Could anyone help take a look at beam8 ? Seems like there are many tests failed on beam8 owing to infra problems. Thanks!

Re: s3 filesystem for Python good for GSoC?

I believe the Boto3 lib should be helpful with right credential configuration when creating a client: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html#configuration On Thu, Feb 21, 2019 at 6:15 PM Suneel Marthi wrote: > Couldn't u just use Boto python package for

Re: SNAPSHOTS have not been updated since february

port I also noticed +Yifan > Zou has an experimental PR testing a fix: > https://github.com/apache/beam/pull/8148 > > > > > > On Tue, Mar 26, 2019 at 11:42 AM Boyuan Zhang > wrote: > > >> > > >> +Daniel Oliveira > > >> > >

Re: [ANNOUNCE] New PMC Member: Pablo Estrada

Congratulations Pablo! So well deserved! *From: *Kenneth Knowles *Date: *Tue, May 14, 2019 at 8:25 PM *To: *dev Hi all, > > Please join me and the rest of the Beam PMC in welcoming Pablo Estrada to > join the PMC. > > Pablo first picked up BEAM-722 in October of 2016 and has been a steady >

Re: Removing Java Reference Runner code

Another concern from me is, will it be difficult for a Java person (who developing Java SDK) to figure out what's going on in Python ULR when debugging? On Fri, Apr 26, 2019 at 12:05 PM Kenneth Knowles wrote: > Good points. Distilling one single item: can I, today, run the Java SDK's > suite of

Re: Integration of python/portable runner tests for Samza runner

Hi Daniel, If you are interested in portable python pipeline validation, I think fn_api_runner_test would also help. On Tue, Apr 23, 2019 at 10:19 AM Pablo Estrada wrote: > This is

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congratulations, Yifan~ On Mon, Apr 22, 2019 at 9:29 AM Connell O'Callaghan wrote: > Well done Yifan!!! > > Thank you for sharing Kenn!!! > > On Mon, Apr 22, 2019 at 9:00 AM Ahmet Altay wrote: > >> Congratulations, Yifan! >> >> On Mon, Apr 22, 2019 at 8:46 AM Tim Robertson >> wrote: >> >>>

Re: How to test a new precommit in PR?

And after you run seed job, other PR's commit may also trigger your percommit test. You can set your precommit task as only triggered by phrase for test purpose.

Re: [ANNOUNCE] New committer: Valentyn Tymofieiev

Congratulations! On Tue, Aug 27, 2019 at 10:44 AM Udi Meiri wrote: > Congrats! > > On Tue, Aug 27, 2019 at 9:50 AM Yichi Zhang wrote: > >> Congrats Valentyn! >> >> On Tue, Aug 27, 2019 at 7:55 AM Valentyn Tymofieiev >> wrote: >> >>> Thank you everyone! >>> >>> On Tue, Aug 27, 2019 at 2:57 AM

Re: [VOTE] Beam's Mascot will be the Firefly (Lampyridae)

+1 (non binding) On Thu, Dec 12, 2019 at 9:14 PM Robert Burke wrote: > +1 (non binding) > > On Thu, Dec 12, 2019, 8:58 PM Kenneth Knowles wrote: > >> Please vote on the proposal for Beam's mascot to be the Firefly. This >> encompasses the Lampyridae family of insects, without specifying a

beam_PostCommit_Java_Nexmark_Flink are failing continuously

Hey team, beam_PostCommit_Java_Nexmark_Flink test target has failed for a long time. Is this something expected? If not, any insights on these failures? Boyuan

Re: Jenkins jobs not running for my PR 10438

Same here. Even the phrase trigger doesn't work. On Fri, Dec 20, 2019 at 10:16 AM Luke Cwik wrote: > I'm also affected by this. > > On Fri, Dec 20, 2019 at 10:13 AM Tomo Suzuki wrote: > >> Hi Beam developers, >> >> Does anybody know why my PR does not trigger Jenkins jobs today? >>

Re: Can any contributor trigger PreCommit tests?

Hey Wenbing, Which tests you are interested in? Most non-dataflow tests can be ran locally for testing purposes. On Tue, Feb 25, 2020 at 11:15 AM Hannah Jiang wrote: > Committers can trigger the tests. Can you share your PRs here so people > who have permission can trigger tests for your PRs?

Re: Jenkins jobs not running for my PR 10438

Done : ) On Wed, Jan 29, 2020 at 7:52 PM Tomo Suzuki wrote: > HI Beam committers: > (Thanks, Luke!) > > Can somebody retrigger the following 2 failed checks for > https://github.com/apache/beam/pull/10714 ? > Run Java PreCommit > Run Java_Examples_Dataflow PreCommit > > On Wed, Jan 29, 2020 at

Release 2.19.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 2.19.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1],

[RELEASE VOTE RESULT] Release 2.19.0, release candidate #1

I'm happy to announce that we have unanimously approved this release. There are 5 approving votes, 4 of which are binging: * Ahmet Altay * Ismaël Mejía * Jean-Baptiste Onofré * Robert Bradshaw There are no disapproving votes. Thanks for everyone's help! I'm going to finalize the release and

Re: [ANNOUNCE] New committer: Hannah Jiang

Thanks for all your contributions! Congratulations~ On Tue, Jan 28, 2020 at 3:44 PM Pablo Estrada wrote: > yoooho : D > > On Tue, Jan 28, 2020 at 3:21 PM Luke Cwik wrote: > >> Hi everyone, >> >> Please join me and the rest of the Beam PMC in welcoming a new committer: >> Hannah Jiang >> >>

[ANNOUNCE] Beam 2.19.0 Released

. -- Boyuan Zhang, on behalf of The Apache Beam team

Re: [RELEASE VOTE RESULT] Release 2.19.0, release candidate #1

t;> >>> On Mon, Feb 3, 2020 at 3:40 PM Ahmet Altay wrote: >>> >>>> On Mon, Feb 3, 2020 at 1:22 PM Thomas Weise wrote: >>>> >>>>> Impressive, probably the fastest/smoothest Beam release so far. >>>>> >>>> >>>

[PROPOSAL] Preparing for Beam 2.19 release

Hi all, The next (2.19) release branch cut is scheduled forJan 15th, according to the calendar . I would like to volunteer myself to do this release. The plan is to cut the branch on that date,

Re: Failing Java PostCommit for Dataflow runner

This problem is addressed by PR10564. Now all affected tests are back to green. On Mon, Jan 13, 2020 at 1:11 PM Luke Cwik wrote: > This is being tracked in BEAM-9083 > > On Mon, Jan 13, 2020 at 11:23 AM Boyuan Zhang wrote: > >> Thanks Kirill! I'm going to look into it. &g

Re: Is Jenkins stuck?

Same here: https://github.com/apache/beam/pull/10375, at least has been pended for 15 hrs On Tue, Jan 14, 2020 at 12:31 PM Reuven Lax wrote: > For https://github.com/apache/beam/pull/10534 , all the tests have been > yellow "pending" for the past 15 hours. The same was true before when I >

Please add boyuanzz as an owner/container of apache-beam in PyPi

Hey, I'm Boyuan, currently working on beam 2.19.0 release. Can anyone add me as a owner of apache-beam package in PyPi for later pushing artifacts? My id is boyuanzz. Thanks for your help!

Re: beam_PreCommit_Java_Examples_Dataflow_Phrase is failing

Are you running the tests against 2.19.0 release branch? If so, it fails for tests misconfiguration and we have a fix for that: https://github.com/apache/beam/pull/10635. On Mon, Jan 20, 2020 at 7:20 AM Alexey Romanenko wrote: > Hi all, > > Seems like

Re: [Python] Setting a timer from a timer callback

I'll take a look. Thanks Max! On Wed, Apr 22, 2020 at 8:20 AM Maximilian Michels wrote: > Attempting to fix this here, if somebody could have a look: > https://github.com/apache/beam/pull/11492 > > On 22.04.20 17:10, Maximilian Michels wrote: > > Hi, > > > > I'm trying to set a timer from a

Re: Writing a new IO on beam, should I use the source API or SDF?

Hi Steve, Yes that's correct. On Fri, May 15, 2020 at 2:11 PM Steve Niemitz wrote: > ah! ok awesome, I think that was the piece I was misunderstanding. So I > _can_ use a SDF to split the work initially (like I was manually doing in > #1), but it just won't be further split dynamically on

Re: [ANNOUNCE] New committer: Robin Qiu

Congrats~~ On Mon, May 18, 2020 at 7:17 PM Reza Rokni wrote: > Congratulations! > > On Tue, May 19, 2020 at 10:06 AM Ahmet Altay wrote: > >> Hi everyone, >> >> Please join me and the rest of the Beam PMC in welcoming a new committer: >> Robin Qiu . >> >> Robin has been active in the community

Re: Clear Timer in Java SDK

remove timers. > > On Wed, Sep 2, 2020 at 11:00 AM Boyuan Zhang wrote: > >> Hi team, >> >> I'm looking for something similar to timer.clear() from Python SDK[1] in >> Java SDK but it seems like we haven't exposed clearing timer API from Java >> Timer. Does J

Clear Timer in Java SDK

Hi team, I'm looking for something similar to timer.clear() from Python SDK[1] in Java SDK but it seems like we haven't exposed clearing timer API from Java Timer. Does Java SDK have another way to clear a timer or we just haven't worked on this API? [1]

Re: Self-checkpoint Support on Portable Flink

applicable and could be used to implement >> dynamic splitting. >> >> -Max >> >> On 05.10.20 23:13, Luke Cwik wrote: >> > Thanks Boyuan, I left a few comments. >> > >> > On Mon, Oct 5, 2020 at 11:12 AM Boyuan Zhang > > <mailto:bo

Create External Transform with WindowFn

Hi team, I'm trying to create an External transform in Java SDK, which expands into several ParDo and a Window.into(FixWindow). When I use this transform in Python SDK, I get an pipeline construction error: apache_beam/utils/urns.py", line 186, in from_runner_api parameter_type, constructor

Re: Create External Transform with WindowFn

wing operations (e.g. global, fixed, sliding, sessions) in an > external transform. You should also be able to window into an > arbitrary WindowFn as long as it produces standards window types, but > if there's a bug here you could possibly work around it by windowing > into a more st

Re: Create External Transform with WindowFn

hat use external WindowingStrategies might be failing during >> proto -> object -> proto conversion we do today. This limitation will go >> away once Dataflow directly starts reading Beam protos. We are working on >> this now. >> >> Thanks, >> Cham &g

Re: [DISCUSS] Clearing timers (https://github.com/apache/beam/pull/12836)

Hi Reuven, Would you like to share the links to potential fixes? We can figure out what we can do there. On Fri, Sep 18, 2020 at 4:21 PM Reuven Lax wrote: > > > On Fri, Sep 18, 2020 at 3:14 PM Luke Cwik wrote: > >> PR 12836[1] is adding support for clearing timers and there is a >> discussion

Re: [Discuss] Build Kafka read transform on top of SplittableDoFn

On Thu, May 28, 2020 at 8:54 PM Reuven Lax wrote: > Which WatermarkEstimator do you think should be used? > > On Thu, May 28, 2020 at 7:17 PM Boyuan Zhang wrote: > >> Hi team, >> >> I'm Boyuan, currently working on building a Kafka read PTransform on top >>

[Discuss] Build Kafka read transform on top of SplittableDoFn

Hi team, I'm Boyuan, currently working on building a Kafka read PTransform on top of SplittableDoFn[1][2][3]. There are two questions about Kafka usage I want to discuss with you: 1. Compared to the KafkaIO.Read

Re: [Discuss] Build Kafka read transform on top of SplittableDoFn

e the SDF transform via KafkaIO.Read, we need to expose new APIs to KafkaIO.Read to accept the timestamp function and the type of policy. On Fri, May 29, 2020 at 10:58 AM Reuven Lax wrote: > Are you able to support CustomTimestampPolicyWithLimitedDelay? > > On Fri, May 29, 2020 at 9:58 AM B

Re: [Discuss] Build Kafka read transform on top of SplittableDoFn

to either ask the customer to write such a kind of function or just support built-in types. On Thu, May 28, 2020 at 9:12 PM Reuven Lax wrote: > This is per-partition, right? In that case I assume it will match the > current Kafka watermark. > > On Thu, May 28, 2020 at 9:03 PM Boyuan

Re: Self-checkpoint Support on Portable Flink

e > > > program the user needs to set a special flag when enabling > > > checkpointing:|env.enableCheckpointing(interval, > > > CheckpointingMode.EXACTLY_ONCE, force = true)|. > > > > > > Please note that records in flight in the loop edges (and the > &

Re: Output timestamp for Python event timers

Hi Maximilian, It makes sense to set hold_timestamp as fire_timestamp when the fire_timestamp is in the event time domain. Otherwise, the system may advance the watermark incorrectly. I think we can do something similar to Java FnApiRunner[1]: - Expose set_output_timestamp API to python

Re: Output timestamp for Python event timers

t; > > On Tue, Aug 11, 2020 at 12:36 PM Yichi Zhang > <mailto:zyi...@google.com>> wrote: > > > > +1 to expose set_output_timestamp and enrich python set timer api. > > > > On Tue, Aug 11, 2020 at 12:01 PM Boyuan Zhang > <mailto:boyu...@google.com>

Re: Needed help identifying a error in running a SDF

Hi Mayank, Which runner do you want to run your pipeline? You should add 'beam_fn_api' when you launch the pipeline --experiments=beam_fn_api. In your code: class TestDoFn(beam.DoFn): def process( self, element, restriction_tracker=beam.DoFn.RestrictionParam(

Re: Unknown accumulator coder error when running cross-language SpannerIO Write

Hi Piotr, Are you using the beam master head to dev? Can you share your code? The x-lang transform can be tested with Flink runner, where SDF is also supported, such as https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/flink_runner_test.py#L205-L261 On Tue,

Re: Python Cross-language wrappers for Java IOs

Thanks Cham. Standard coder is a good point. Does it mean non-standard coder doesn't work when crossing language boundaries even if it is implemented in both Java and Python sdk? On Mon, Jun 15, 2020 at 10:08 AM Chamikara Jayalath wrote: > Thanks. +1 for using RowCoder. We should try to use

Re: Python Cross-language wrappers for Java IOs

eam/coders/row_coder.py#L177-L189 >>> [2] >>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java#L341-L356 >>> [3] >>> https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/bea

Re: [DISCUSS] ReadAll pattern and consistent use in IO connectors

Hi Ismael, I think the ReadAll in the IO connector refers to the IO with SDF implementation despite the type of input, where Read refers to UnboundedSource. One major pushback of using KafkaIO.Read as source description is that not all configurations of KafkaIO.Read are meaningful to populate

Re: [DISCUSS] ReadAll pattern and consistent use in IO connectors

duplication and error-prone approach related to this. It > makes much sense since usually we have all needed configuration set in Read > objects and, as Ismaeil mentioned, ReadAll will consist mostly of only > Split-Shuffle-Read stages. So this case usually can be unified by using >

Re: [DISCUSS] ReadAll pattern and consistent use in IO connectors

Sorry for the typo. I mean I think we can go with *(3)* and (4): use the data type that is schema-aware as the input of ReadAll. On Wed, Jun 24, 2020 at 7:42 PM Boyuan Zhang wrote: > Thanks for the summary, Cham! > > I think we can go with (2) and (4): use the data type that is sch

Re: [DISCUSS] ReadAll pattern and consistent use in IO connectors

whenever a PTransform is used >>> as a data object. >>> Also, agree with Boyuan that using simple Java objects that are >>> convertible to Beam Rows allow us to make these transforms available to >>> other SDKs through the cross-language transforms. Using transforms or &

Re: [DISCUSS] ReadAll pattern and consistent use in IO connectors

language transforms is a slight tangent >>>> here. But I think, in general, it's great if we can use existing transforms >>>> (for example, IO connectors) as cross-language transforms without having to >>>> build more composites (irrespective of whether in ExternalTr

Re: [Discuss] Build Kafka read transform on top of SplittableDoFn

rce / available at >> this moment? >> >> Trying to implement here an IO to a custom source here using >> SplittableDoFn, and it would be helpful to see more examples :) >> >> Thanks, >> Pedro >> >> >> On 2020/05/29 02:16:49,

Re: Python Cross-language wrappers for Java IOs

odel/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml#L344-L364 > > On Fri, Jun 12, 2020 at 3:32 PM Luke Cwik wrote: > >> +Boyuan Zhang >> >> On Fri, Jun 12, 2020 at 3:32 PM Luke Cwik wrote: >> >>> What is the update

Re: KafkaIO sending KafkaRecords in CrossLanguage - where is the coder registered?

Hi Piotr, X-Lang uses TypedWithoutMetadata , which outputs the KV directly instead of KafkaRecord: see here

Re: please add me in beam project

Hi Keisuke, Welcome! For creating a PR, there is no additional permission required. You may want to create a JIRA account and ask for permission for task tracking. You may also want to check this out to get familiar with how to do contribution: https://beam.apache.org/contribute/ On Mon, Jan 11,

Re: Understanding KafkaIO.Read In Batch Mode

+dev Hi Shrikant, If you look into the expansion of BoundedReadfromUnboundedSource[1], you will notice that it will expand into Create single shard -> Split into multiple shard -> read from one shard. The number of records from one shard will not be larger than 1 and the number of shards

  1   2   >