Re: Python2.7 Beam End-of-Life Date

2020-06-18 Thread Ahmet Altay
Good to hear from you and good to hear the news. Release branch cut date for 2.24 is 8/12. On Thu, Jun 18, 2020 at 5:01 PM Chad Dombrova wrote: > Hi all, > Sorry I've been AWOL. I've been pulled in a number of different > directions. > > We are still increasing our use of Beam, and I have it

Re: Python2.7 Beam End-of-Life Date

2020-06-18 Thread Chad Dombrova
Hi all, Sorry I've been AWOL. I've been pulled in a number of different directions. We are still increasing our use of Beam, and I have it on my todo list to reach out to Kenneth about how Beam could be expanded into the VFX / Animation industries. Our industry uses a number of specialized

Re: Python2.7 Beam End-of-Life Date

2020-06-18 Thread Valentyn Tymofieiev
Thank you! Shared on Slack as well. On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay wrote: > OK, tweeted the message. Could you share on Slack? > > On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev > wrote: > >> Alright, let's publish the message on Twitter and echo on Slack once >> that's done.

Re: Python2.7 Beam End-of-Life Date

2020-06-18 Thread Ahmet Altay
OK, tweeted the message. Could you share on Slack? On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev wrote: > Alright, let's publish the message on Twitter and echo on Slack once > that's done. > Thank you! > > On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay wrote: > >> That sounds reasonable to

Re: Python2.7 Beam End-of-Life Date

2020-06-18 Thread Valentyn Tymofieiev
Alright, let's publish the message on Twitter and echo on Slack once that's done. Thank you! On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay wrote: > That sounds reasonable to me. I agree, it is better to get specific > reasons rather than a yes/no answer. > > On Tue, Jun 16, 2020 at 1:50 PM

Re: Beam Jenkins Migration

2020-06-18 Thread Tyson Hamilton
Currently no. We're already experiencing a backlog of builds so the additional load would be a problem. I've opened two related issues that I think need completion before allowing non-committers to trigger tests: Load sharing improvements: https://issues.apache.org/jira/browse/BEAM-10281 Admin

Re: Beam Jenkins Migration

2020-06-18 Thread Luke Cwik
Was about to ask the same question, so can non-committers trigger the tests now? On Thu, Jun 18, 2020 at 11:54 AM Heejong Lee wrote: > This is awesome. Could non-committers also trigger the test now? > > On Wed, Jun 17, 2020 at 6:12 AM Damian Gadomski < > damian.gadom...@polidea.com> wrote: >

Re: JIRA contributor permission

2020-06-18 Thread Yuwei Fu
Sorry I didn't mention my username. My JIRA username is: fuyuwei On Thu, Jun 18, 2020 at 2:21 PM Yuwei Fu wrote: > Hello, > > I would like to request JIRA contributor permission. >

Re: Python SDK ReadFromKafka: Timeout expired while fetching topic metadata

2020-06-18 Thread Piotr Filipiuk
Thank you for clarifying. Would you mind clarifying whether the issues that I experience running Kafka IO on Flink (or DirectRunner for testing) specific to my setup etc. or this setup is not yet fully functional (for Python SDK)? On Thu, Jun 18, 2020 at 12:03 PM Chamikara Jayalath wrote: >

JIRA contributor permission

2020-06-18 Thread Yuwei Fu
Hello, I would like to request JIRA contributor permission.

Request for support regarding FileIO write to s3 using assumeRole

2020-06-18 Thread Almeida, Julius
Hi Apache Beam Support Team, I am currently using FileIO with ParquetIO.sink() to write parquet files to s3 and need to assume role in-order to access s3 written files by multiple aws accounts. But currently I am facing some AWS related issue. I have created stackoverflow question with

cwiki edit access

2020-06-18 Thread Tyson Hamilton
Hello, Would someone kindly provide me (tysonjh) with edit access to the wiki? I don't have any updates planned now but as I go through pages I'd like the ability to fix broken links, add information, or update stale content. Thanks, -Tyson

Re: DISCUSS: Sorted MapState API

2020-06-18 Thread Jan Lukavský
My questions were just an example. I fully agree there is a fundamental need for a sorted state (of some form, and I also think this links to efficient implementation of retrations) - I was reacting to Kenn's question about BIP. This one would be pretty nice example why it would be good to

Re: Python SDK ReadFromKafka: Timeout expired while fetching topic metadata

2020-06-18 Thread Chamikara Jayalath
Beam does not have a concept of general availability. It's released with Beam so available. Some of the APIs used by Kafka are experimental so are subject to change (but less likely at this point). Various runners may offer their own levels of availability for cross-language transforms. Thanks,

Re: Beam Jenkins Migration

2020-06-18 Thread Heejong Lee
This is awesome. Could non-committers also trigger the test now? On Wed, Jun 17, 2020 at 6:12 AM Damian Gadomski wrote: > Hello, > > Good news, we've just migrated to the new CI: https://ci-beam.apache.org. > As from now beam projects at builds.apache.org are disabled. > > If you experience any

Re: Python SDK ReadFromKafka: Timeout expired while fetching topic metadata

2020-06-18 Thread Piotr Filipiuk
I also wanted to clarify whether Kafka IO for Python SDK is general availability or is it still experimental? On Fri, Jun 12, 2020 at 2:52 PM Piotr Filipiuk wrote: > For completeness I am also attaching task manager logs. > > On Fri, Jun 12, 2020 at 2:32 PM Piotr Filipiuk > wrote: > >> Thank

Re: Request to be added as a contributor

2020-06-18 Thread Luke Cwik
Welcome, I have added you as a contributor. On Thu, Jun 18, 2020 at 10:13 AM Jiadai Xia wrote: > > Hi! > > I am Jiadai Xia and I am working as a SWE intern at Google this summer and > I would like to be added as a contributor in the Jira. > > My ASF Jira username is danielxjd. > > Thanks, >

Request to be added as a contributor

2020-06-18 Thread Jiadai Xia
Hi! I am Jiadai Xia and I am working as a SWE intern at Google this summer and I would like to be added as a contributor in the Jira. My ASF Jira username is danielxjd. Thanks, Jiadai *Jiadai Xia* SWE Intern 1 (646) 413 8071 daniel...@google.com

Re: cwiki edit access

2020-06-18 Thread Kenneth Knowles
Done. Thanks! On Thu, Jun 18, 2020 at 9:30 AM Tyson Hamilton wrote: > Hello, > > Would someone kindly provide me (tysonjh) with edit access to the wiki? I > don't have any updates planned now but as I go through pages I'd like the > ability to fix broken links, add information, or update stale

Re: DISCUSS: Sorted MapState API

2020-06-18 Thread Reuven Lax
Jan - my proposal is exactly TimeSortedBagState (more accurately - TimeSortedListState), though I went a bit further and also proposed a way to have a dynamic number of tagged TimeSortedBagStates. You are correct that the runner doesn't really have to store the data time sorted - what's actually

Re: Snowflake connector

2020-06-18 Thread Shashanka Balakuntala
Hi Katarzyna, I have linked the Python snowflake connector with the Cross platform one. And I agree that we can , if needed in future reopen the issue based on users feedback. *Regards* Shashanka Balakuntala Srinivasa On Thu, Jun 18, 2020 at 7:54 PM Katarzyna Kucharczyk <

Re: DISCUSS: Sorted MapState API

2020-06-18 Thread Jan Lukavský
Big +1 for a BIP, as this might really help clarify all the pros and cons of all possibilities. There seem to be questions that need answering and motivating use cases - do we need sorted map state or can we solve our use cases by something simpler - e.g. the mentioned TimeSortedBagState? Does

Re: Snowflake connector

2020-06-18 Thread Katarzyna Kucharczyk
@Shashanka Balakuntala We are waiting for the Java Snowflake connector to be reviewed [1] and when it will be available we will publish code for a cross-language connector (it should be rather soon). We tested this code on Flink runner and the only issue we found for now is that it's impossible

Re: [Discuss] Build Kafka read transform on top of SplittableDoFn

2020-06-18 Thread Pedro H S Teixeira
Thanks, appreciated the links. Ended up using `Read.from` using an UnboundedSource due to some challenges with the actual splitting part of SDF. Perhaps it's trivial knowledge, but just sharing here that when testing (in direct runner) with unbounded sources, one has to test with more elements