Re: SNAPSHOTS have not been updated since february

The Infra is reviewing the security on our new Jenkins agents. In the meanwhile, I assigned the publishing job to the 'ubuntu' labelled nodes as a workaround. The snapshots were successfully published in beam_Release_NightlySnapshot#421

Beam Summit at ApacheCon

Beam Summit will be at ApacheCon this year -- please consider submitting! Dates for Beam Summit 11 and 12 September 2019. There are other tracks at ApacheCon during this and on other dates too. https://www.apachecon.com/acna19/cfp.html

Re: Integration of python/portable runner tests for Samza runner

Hi Daniel, We use flinkCompatibilityMatrix [1] to check the Flink compatibility with python. This is python equivalent to validatesRunner tests in java for portable runners. I think we can reuse it for Samza Portable runner with minor refactoring. [1]

Re: Artifact staging in cross-language pipelines

One more suggestion: It would be nice to be able to select the environment for the external transforms. For example, I would like to be able to use EMBEDDED for Flink. That's implicit for sources which are runner native unbounded read translations, but it should also be possible for writes. That

Integration of python/portable runner tests for Samza runner

Hi everyone, I'm working on improving the validation of the Python portable Samza runner. For java, we have the gradle task ( :validatesRunner) that runs the runner validation tests. I am looking for pointers on how to similarly integrate/enable the portability and Python tests for the Samza

Re: Projects Can Apply Individually for Google Season of Docs

Hello all, thanks to everyone for your participation. I have submitted the application on behalf of Beam, and requested one technical writer. Let's see how it goes : ) Best -P. On Wed, Apr 17, 2019 at 10:09 PM Ahmet Altay wrote: > Thanks Aizhamal, I completed the forms. > > On Wed, Apr 17, 2019

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congrats Yifan! On Mon, Apr 22, 2019 at 1:33 PM Ankur Goenka wrote: > Congratulations Yifan! > > On Mon, Apr 22, 2019 at 11:42 AM Thomas Weise wrote: > >> Congrats Yifan! >> >> >> On Mon, Apr 22, 2019 at 10:02 AM Maximilian Michels >> wrote: >> >>> Congrats! Great work. >>> >>> -Max >>> >>>

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congratulations Yifan! On Mon, Apr 22, 2019 at 11:42 AM Thomas Weise wrote: > Congrats Yifan! > > > On Mon, Apr 22, 2019 at 10:02 AM Maximilian Michels > wrote: > >> Congrats! Great work. >> >> -Max >> >> On 22.04.19 19:00, Rui Wang wrote: >> > Congratulations! Thanks for your contribution!!

Re: Artifact staging in cross-language pipelines

Great discussion! I have a few points around the structure of proto but that is less important as it can evolve. However, I think that artifact compatibility is another important aspect to look at. Example: TransformA uses Guava 1.6>< 1.7, TransformB uses 1.8><1.9 and TransformC uses 1.6><1.8. As

Re: [VOTE] Release 2.12.0, release candidate #4

I signed the wheels files and updated the build process to not require giving travis apache credentials. (You should probably change your password if you haven't already.) Andrew On Mon, Apr 22, 2019 at 12:18 PM Ahmet Altay wrote: > +1 (binding) > > Verified the python 2 wheel files with quick

Re: [VOTE] Release 2.12.0, release candidate #4

+1 (binding) Verified the python 2 wheel files with quick start examples. On Mon, Apr 22, 2019 at 11:26 AM Ahmet Altay wrote: > I built the wheel files. They are in the usual place along with other > python artifacts. I will test them a bit and update here. Could someone > else please try the

Meetups in London, Stockholm + recordings

Hi everyone, Happy to announce 2 meetups for the next week(s): - *London *is having its second meetup of the year on *01/05/2019 in Level39, Canary Wharf

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congrats Yifan! On Mon, Apr 22, 2019 at 10:02 AM Maximilian Michels wrote: > Congrats! Great work. > > -Max > > On 22.04.19 19:00, Rui Wang wrote: > > Congratulations! Thanks for your contribution!! > > > > -Rui > > > > On Mon, Apr 22, 2019 at 9:57 AM Ruoyun Huang >

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congrats Yifan! On Mon, Apr 22, 2019 at 11:04 AM Valentyn Tymofieiev wrote: > Congratulations, Yifan! Thanks a lot for your continued contributions to > Beam. > > On Mon, Apr 22, 2019 at 10:24 AM Robin Qiu wrote: > >> Congratulations Yifan! >> >> On Mon, Apr 22, 2019 at 10:17 AM Chamikara

Re: [VOTE] Release 2.12.0, release candidate #4

I built the wheel files. They are in the usual place along with other python artifacts. I will test them a bit and update here. Could someone else please try the wheel files as well? Andrew, could you sign and hash the wheel files? On Mon, Apr 22, 2019 at 10:11 AM Ahmet Altay wrote: > I

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congratulations, Yifan! Thanks a lot for your continued contributions to Beam. On Mon, Apr 22, 2019 at 10:24 AM Robin Qiu wrote: > Congratulations Yifan! > > On Mon, Apr 22, 2019 at 10:17 AM Chamikara Jayalath > wrote: > >> Congrats Yifan! >> >> On Mon, Apr 22, 2019 at 10:02 AM Maximilian

Re: Go SDK status

Nice summary, Robert! I really like the transparency on the state of the Go SDK and how it's being used. It would be great to see the streaming mode improve because only then we have a full-blown SDK. It looks like we will need a few more resources on the SDK to bring it up to par with

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congrats, Yifan!!! On Mon, Apr 22, 2019 at 10:24 AM Robin Qiu wrote: > Congratulations Yifan! > > On Mon, Apr 22, 2019 at 10:17 AM Chamikara Jayalath > wrote: > >> Congrats Yifan! >> >> On Mon, Apr 22, 2019 at 10:02 AM Maximilian Michels >> wrote: >> >>> Congrats! Great work. >>> >>> -Max >>>

Re: AvroUtils converting generic record to Beam Row causes class cast exception

I see. I created this PR [1] to ask feedback from the reviewer who knows better on Avro in Beam. -Rui [1]: https://github.com/apache/beam/pull/8376 On Sun, Apr 21, 2019 at 11:19 PM Vishwas Bm wrote: > Hi Rui, > > I checked the AvroUtils code. There is a static intializer block basically >

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congratulations Yifan! On Mon, Apr 22, 2019 at 10:17 AM Chamikara Jayalath wrote: > Congrats Yifan! > > On Mon, Apr 22, 2019 at 10:02 AM Maximilian Michels > wrote: > >> Congrats! Great work. >> >> -Max >> >> On 22.04.19 19:00, Rui Wang wrote: >> > Congratulations! Thanks for your

Re: Hazelcast Jet Runner

Hi Jozsef, If the Runner support the complete set of ValidatesRunner tests and the Nexmark suite, it is already in a very good state. Like Kenn already suggested, we can definitely add it to the capability matrix then. Thanks, Max On 19.04.19 22:52, Kenneth Knowles wrote: The

Re: [VOTE] Release 2.12.0, release candidate #4

I verified - signatures and hashes. - python streaming quickstart guide I would like to verify the wheel files before voting. Please let us know when they are ready. Also, if you need help with building wheel files I can help/build. Ahmet On Mon, Apr 22, 2019 at 3:33 AM Maximilian Michels

Re: Artifact staging in cross-language pipelines

Thanks for the summary Cham. All makes sense. I agree that we want to keep the option to manually specify artifacts. There are few unanswered questions though. (1) In what form will a transform author specify dependencies ? For example, URL to a Maven repo, URL to a local file, blob ? Going

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congrats Yifan! On Mon, Apr 22, 2019 at 10:02 AM Maximilian Michels wrote: > Congrats! Great work. > > -Max > > On 22.04.19 19:00, Rui Wang wrote: > > Congratulations! Thanks for your contribution!! > > > > -Rui > > > > On Mon, Apr 22, 2019 at 9:57 AM Ruoyun Huang > >

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congrats! Great work. -Max On 22.04.19 19:00, Rui Wang wrote: Congratulations! Thanks for your contribution!! -Rui On Mon, Apr 22, 2019 at 9:57 AM Ruoyun Huang > wrote: Congratulations, Yifan! On Mon, Apr 22, 2019 at 9:48 AM Boyuan Zhang

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congratulations, Yifan! On Mon, Apr 22, 2019 at 9:48 AM Boyuan Zhang wrote: > Congratulations, Yifan~ > > On Mon, Apr 22, 2019 at 9:29 AM Connell O'Callaghan > wrote: > >> Well done Yifan!!! >> >> Thank you for sharing Kenn!!! >> >> On Mon, Apr 22, 2019 at 9:00 AM Ahmet Altay wrote: >> >>>

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congratulations, Yifan~ On Mon, Apr 22, 2019 at 9:29 AM Connell O'Callaghan wrote: > Well done Yifan!!! > > Thank you for sharing Kenn!!! > > On Mon, Apr 22, 2019 at 9:00 AM Ahmet Altay wrote: > >> Congratulations, Yifan! >> >> On Mon, Apr 22, 2019 at 8:46 AM Tim Robertson >> wrote: >> >>>

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Well done Yifan!!! Thank you for sharing Kenn!!! On Mon, Apr 22, 2019 at 9:00 AM Ahmet Altay wrote: > Congratulations, Yifan! > > On Mon, Apr 22, 2019 at 8:46 AM Tim Robertson > wrote: > >> Congratulations Yifan! >> >> On Mon, Apr 22, 2019 at 5:39 PM Cyrus Maden wrote: >> >>> Congratulations

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congratulations, Yifan! On Mon, Apr 22, 2019 at 8:46 AM Tim Robertson wrote: > Congratulations Yifan! > > On Mon, Apr 22, 2019 at 5:39 PM Cyrus Maden wrote: > >> Congratulations Yifan!! >> >> On Mon, Apr 22, 2019 at 11:26 AM Kenneth Knowles wrote: >> >>> Hi all, >>> >>> Please join me and the

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congratulations Yifan! On Mon, Apr 22, 2019 at 5:39 PM Cyrus Maden wrote: > Congratulations Yifan!! > > On Mon, Apr 22, 2019 at 11:26 AM Kenneth Knowles wrote: > >> Hi all, >> >> Please join me and the rest of the Beam PMC in welcoming a new committer: >> Yifan Zou. >> >> Yifan has been

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congratulations Yifan!! On Mon, Apr 22, 2019 at 11:26 AM Kenneth Knowles wrote: > Hi all, > > Please join me and the rest of the Beam PMC in welcoming a new committer: > Yifan Zou. > > Yifan has been contributing to Beam since early 2018. He has proposed 70+ > pull requests, adding dependency

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congratulations, Yifan! Thank you for all your work on Beam  On Mon, Apr 22, 2019 at 08:26 Kenneth Knowles wrote: > Hi all, > > Please join me and the rest of the Beam PMC in welcoming a new committer: > Yifan Zou. > > Yifan has been contributing to Beam since early 2018. He has proposed 70+ >

[ANNOUNCE] New committer announcement: Yifan Zou

Hi all, Please join me and the rest of the Beam PMC in welcoming a new committer: Yifan Zou. Yifan has been contributing to Beam since early 2018. He has proposed 70+ pull requests, adding dependency checking and improving test infrastructure. But something the numbers cannot show adequately is

Re: [DISCUSS] Turn `WindowedValue` into `T` in the FnDataService and BeamFnDataClient interface definition

Makes sense to me. I don't see any problem. Kenn On Mon, Apr 22, 2019 at 12:08 AM jincheng sun wrote: > Hi Kenn, > > Thanks for your reply, and explained the design of WindowValue clearly! > > At present, the definitions of `FnDataService` and `BeamFnDataClient` in > Data Plane are very clear

Beam Dependency Check Report (2019-04-22)

High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release JIRA Issue future 0.16.0 0.17.1 2016-10-27

Re: [VOTE] Release 2.12.0, release candidate #4

+1 (binding) Found a minor bug while testing, but not a blocker: https://jira.apache.org/jira/browse/BEAM-7128 Thanks, Max On 20.04.19 23:02, Pablo Estrada wrote: +1 Ran SQL postcommit, and Dataflow Portability Java validatesrunner tests. -P. On Wed, Apr 17, 2019 at 1:38 AM Jean-Baptiste

Re: [DISCUSS] Turn `WindowedValue` into `T` in the FnDataService and BeamFnDataClient interface definition

Hi Kenn, Thanks for your reply, and explained the design of WindowValue clearly! At present, the definitions of `FnDataService` and `BeamFnDataClient` in Data Plane are very clear and universal, such as: send(...)/receive(...). If it is only applied in the project of Beam, it is already very

Re: AvroUtils converting generic record to Beam Row causes class cast exception

Hi Rui, I checked the AvroUtils code. There is a static intializer block basically it registers Avro Timestamp Conversion functions for logical type timestamp-millis. *// Code Snippet below* static { // This works around a bug in the Avro library (AVRO-1891) around SpecificRecord's handling //