Re: [VOTE] Vendored Dependencies Release

Thanks Pablo for jumping in for help. Now the sources are moved to [1]. Please let me know if it is ok. [1]: https://dist.apache.org/repos/dist/release/beam/vendor/calcite/1_20_0/ -Rui On Wed, Sep 4, 2019 at 4:15 PM Pablo Estrada wrote: > I can help. > > On Wed, Sep 4, 2019 at 1:09 PM Rui

Re: [VOTE] Vendored Dependencies Release

I can help. On Wed, Sep 4, 2019 at 1:09 PM Rui Wang wrote: > There is a step of releasing requires PMC permission: > > """ > > Copy the source release from the dev repository to the release repository > at dist.apache.org using Subversion. > Move last release artifacts from dist.apache.org to

Re: Improve container support

This will greatly simplify trying out portable runners: https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster Can't wait for following to disappear from the instructions page: ./gradlew :sdks:python:container:docker On Wed, Sep 4, 2019 at 3:35 PM

Re: Improve container support

Awesome, thank you! On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang wrote: > Hi Thomas > > I created snapshot images from head as of around 2PM today. > You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot. > > Thanks, > Hannah > > On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise

Re: Improve container support

Hi Thomas I created snapshot images from head as of around 2PM today. You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot. Thanks, Hannah On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise wrote: > Hi Hannah, > > Thank you, I know how to build the containers locally, but not how

Re: Improve container support

Hi Hannah, Thank you, I know how to build the containers locally, but not how to publish them! The cwiki says "Publishing images to gcr.io/beam requires permissions in apache-beam-testing project." Can I get access to the testing project (at least temporarily) and what would I need to setup to

Re: [VOTE] Vendored Dependencies Release

There is a step of releasing requires PMC permission: """ Copy the source release from the dev repository to the release repository at dist.apache.org using Subversion. Move last release artifacts from dist.apache.org to archive.apache.org using Subversion. """ Is there a PMC member could help

Re: Beam at Google Summer of Code 2019

Thank you Tanay! I hope you had a truly wonderful GSoC experience, and will stay involved in the community for years to come :) On Wed, Sep 4, 2019 at 10:42 AM Ahmet Altay wrote: > Thank you Tanay for all your contributions during summer and looking > forward to more of it :) > > On Wed, Sep 4,

Re: [discuss] Auto-close issues with a PR associated?

Hey Pablo, I think we should add the workflow to the contributor's guide[1] since most new contributors start from there. Best, - Tanay [1] https://beam.apache.org/contribute/ On Wed, Sep 4, 2019 at 11:03 PM Pablo Estrada wrote: > Hello all, > this has been discussed before, and I believe we

Re: Improve container support

Hi Thomas I haven't uploaded any snapshot images yet. Here is how you can create one from head. > cd [...]/beam/ # For Python > ./gradlew :sdks:python:container:py{version}:docker *where version is {2,35,36,37}* # For Java > ./gradlew -p sdks/java/container docker # For Go > ./gradlew -p

Re: Improve container support

I actually found something in [1], but it is 2.15 unfortunately. [1] https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30 On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise wrote: > Thanks for working on this. Do you happen to have

Re: Beam at Google Summer of Code 2019

Thank you Tanay for all your contributions during summer and looking forward to more of it :) On Wed, Sep 4, 2019 at 10:38 AM Tanay Tummalapalli wrote: > Hi everyone, > > I've completed Google Summer of Code '19[1]. > I had fun working on Beam for the past 3 months and learning about Beam >

Re: Beam at Google Summer of Code 2019

Hi everyone, I've completed Google Summer of Code '19[1]. I had fun working on Beam for the past 3 months and learning about Beam internals. Thank you Pablo for everything! None of it would have been possible without you. I'd also like to thank the Beam community for the code reviews and being

Re: Improve container support

Thanks for working on this. Do you happen to have publicly accessible snapshots published for your testing currently (even when the final location isn't sorted out)? I would like to use a 2.16 based Python SDK image for working on my downstream project, but could not find anything in

[discuss] Auto-close issues with a PR associated?

Hello all, this has been discussed before, and I believe we concluded that we did not want to auto-close JIRA issues referenced by a PR. I wanted us to revisit this decision, because I don't think everyone has adopted the correct workflow: Workflow A: 1. Send a PR to fix BEAM-X 2. Go to JIRA, and

Re: [VOTE] Vendored Dependencies Release

I'm happy to announce that we have unanimously approved this release. There are 5 approving votes, 3 of which are binding: * Lukasz Cwik * Kenneth Knowles * Ahmet Altay There are no disapproving votes. Thanks everyone! On Tue, Sep 3, 2019 at 1:29 PM Lukasz Cwik wrote: > +1 > > On Tue, Sep

Need help reviewing Protobuf enhancement

Hi all, I need some help getting the following PR's in Beam. Both of the next PR's are tested on the DirectRunner and the Dataflow runner... I'm currently using my own build of BEAM, because I need the support on the next product we're building, so it's been extensively battle tested in my own

Re: Cassandra flaky on Jenkins?

Thanks David, it makes sense, it gives me time to investigate and fix. Regards JB On 04/09/2019 15:01, David Morávek wrote: > Hi, temporarily disabling the test > , until BEAM-8025 > is resolved (marking

Re: Cassandra flaky on Jenkins?

Thanks, got my PR green now, _/ _/ Alex Van Boxel On Wed, Sep 4, 2019 at 3:02 PM David Morávek wrote: > Hi, temporarily disabling the test > , until BEAM-8025 > is resolved (marking it > as blocker for

Re: Cassandra flaky on Jenkins?

Hi, temporarily disabling the test , until BEAM-8025 is resolved (marking it as blocker for 2.16), so we can unblock ongoing pull requests. Best, D. On Tue, Sep 3, 2019 at 3:57 PM Jean-Baptiste Onofré

Problems with KafkaIOIT in Beam

Hi all, recently I've been struggling to adapt Java's KafkaIOIT to work with a large dataset generated by a SyntheticSource. I want to push 100M records through a Kafka topic and verify data correctness and at the same time check the performance of KafkaIO.Write and KafkaIO.Read. To perform the