Re: [VOTE] Release 2.23.0, release candidate #1

On Thu, Jul 16, 2020 at 7:46 PM Chamikara Jayalath wrote: > > > On Thu, Jul 16, 2020 at 7:28 PM Valentyn Tymofieiev > wrote: > >> >> >> On Thu, Jul 16, 2020, 19:07 Chamikara Jayalath >> wrote: >> >>> >>> >>> On Thu, Jul 16, 2020 at 6:16 PM Valentyn Tymofieiev >>> wrote: >>> Thanks for the

Re: [VOTE] Release 2.23.0, release candidate #1

On Thu, Jul 16, 2020 at 7:28 PM Valentyn Tymofieiev wrote: > > > On Thu, Jul 16, 2020, 19:07 Chamikara Jayalath > wrote: > >> >> >> On Thu, Jul 16, 2020 at 6:16 PM Valentyn Tymofieiev >> wrote: >> >>> Thanks for the feedback, help with release validation, and for reaching >>> out on dev@ regard

Re: [VOTE] Release 2.23.0, release candidate #1

On Thu, Jul 16, 2020, 19:07 Chamikara Jayalath wrote: > > > On Thu, Jul 16, 2020 at 6:16 PM Valentyn Tymofieiev > wrote: > >> Thanks for the feedback, help with release validation, and for reaching >> out on dev@ regarding a cherry-pick request. >> >> BEAM-10397

Re: [VOTE] Release 2.23.0, release candidate #1

On Thu, Jul 16, 2020 at 6:16 PM Valentyn Tymofieiev wrote: > Thanks for the feedback, help with release validation, and for reaching > out on dev@ regarding a cherry-pick request. > > BEAM-10397 pertains to > new functionality (xlang support on D

Re: [VOTE] Release 2.23.0, release candidate #1

Hi, Are there strong objections to the ability to do patches? Cheers Reza On Fri, Jul 17, 2020 at 9:16 AM Valentyn Tymofieiev wrote: > Thanks for the feedback, help with release validation, and for reaching > out on dev@ regarding a cherry-pick request. > > BEAM-10397

Re: [VOTE] Release 2.23.0, release candidate #1

Thanks for the feedback, help with release validation, and for reaching out on dev@ regarding a cherry-pick request. BEAM-10397 pertains to new functionality (xlang support on Dataflow). Are there any reasons that this fix cannot wait until 2.24.0

Re: Chronically flaky tests

What do other Apache projects do to address this issue? On Thu, Jul 16, 2020 at 5:51 PM Ahmet Altay wrote: > I agree with the comments in this thread. > - If we are not re-enabling tests back again or we do not have a plan to > re-enable them again, disabling tests only provides us temporary rel

Re: Chronically flaky tests

I agree with the comments in this thread. - If we are not re-enabling tests back again or we do not have a plan to re-enable them again, disabling tests only provides us temporary relief until eventually users find issues instead of disabled tests. - I feel similarly about retries. It is reasonable

Re: KafkaIO sending KafkaRecords in CrossLanguage - where is the coder registered?

If you want to send across a "rich" data record, consider defining a schema and using a row coder since row coder is XLang compatible. On Thu, Jul 16, 2020 at 9:28 AM Robert Bradshaw wrote: > Note also that once you get the Bytes in Python, you can use whatever > coder (or Map) to decode them th

Re: [PROPOSAL] Azure Filesystem for Beam Java SDK

Hi Ashwin, Thanks for the suggestion, I hadn't considered that. For now I plan to go ahead using the Azure client. Etta On Thu, Jul 16, 2020 at 3:36 PM Ashwin Ramaswami wrote: > Hi Etta, > > Have you thought about reusing the HadoopFileSystem to access Azure Blob > Storage instead? It appears

Re: [VOTE] Extension name of Interactive Beam Side Panel in JupyterLab

+1 for 3. On Thu, Jul 16, 2020 at 12:35 PM Pablo Estrada wrote: > +1 for 3. Thanks Ning. > > On Thu, Jul 16, 2020 at 10:54 AM Kenneth Knowles wrote: > >> +1 for [3] >> >> On Wed, Jul 15, 2020 at 5:47 PM Robert Bradshaw >> wrote: >> >>> +1 for [3] as well. >>> >>> On Wed, Jul 15, 2020 at 5:40 P

Re: Contributor permission

Hi Mani! I've added you as contributor. Welcome, and thanks for finding and fixing these bugs : ) Best -P. On Thu, Jul 16, 2020 at 12:27 PM Sunny, Mani Kolbe wrote: > Hello, > > > > My name is Mani. I am based in Dublin, Ireland and work for Dun & > Bradstreet. We have a product which process bi

Re: [VOTE] Extension name of Interactive Beam Side Panel in JupyterLab

if specific to jupyterlab, then [3] makes sense. Am wondering whether that gets confusing, if it will also be used in datalab, and/or other (similar/same underlying tech but going by different names)? On Thu, Jul 16, 2020 at 12:35 PM Pablo Estrada wrote: > +1 for 3. Thanks Ning. > > On Thu, Jul

Re: [PROPOSAL] Azure Filesystem for Beam Java SDK

Hi Etta, Have you thought about reusing the HadoopFileSystem to access Azure Blob Storage instead? It appears that Azure Blob Storage comes with a hdfs-compatible API with the wasb:// protocol. See https://issues.apache.org/jira/browse/BEAM-10103 Ashwin Ramaswami Student *Find me on my:* LinkedIn

Re: [VOTE] Extension name of Interactive Beam Side Panel in JupyterLab

+1 for 3. Thanks Ning. On Thu, Jul 16, 2020 at 10:54 AM Kenneth Knowles wrote: > +1 for [3] > > On Wed, Jul 15, 2020 at 5:47 PM Robert Bradshaw > wrote: > >> +1 for [3] as well. >> >> On Wed, Jul 15, 2020 at 5:40 PM Ahmet Altay wrote: >> > >> > I agree with Kyle. [3] sounds more accurate. >> >

[PROPOSAL] Azure Filesystem for Beam Java SDK

Hi, I am working on a project adding Azure Blobstore IO to Apache Beam. The design document is available at http://s.apache.org/beam-azfs-java and the JIRA issue is at https://issues.apache.org/jira/browse/BEAM-10378. Can you please provide any feedback or suggestions? Thank you, Etta Rapp

Contributor permission

Hello, My name is Mani. I am based in Dublin, Ireland and work for Dun & Bradstreet. We have a product which process billions of records per day. This is Hadoop batch job and we are in the process of migrating it into a streaming application based on Kinesis + Beam SparkRunner. As part of the e

Re: Chronically flaky tests

I think the original discussion[1] on introducing tenacity might answer that question. [1] https://lists.apache.org/thread.html/16060fd7f4d408857a5e4a2598cc96ebac0f744b65bf4699001350af%40%3Cdev.beam.apache.org%3E On Thu, Jul 16, 2020 at 10:48 AM Rui Wang wrote: > Is there an observation that en

Re: [VOTE] Extension name of Interactive Beam Side Panel in JupyterLab

+1 for [3] On Wed, Jul 15, 2020 at 5:47 PM Robert Bradshaw wrote: > +1 for [3] as well. > > On Wed, Jul 15, 2020 at 5:40 PM Ahmet Altay wrote: > > > > I agree with Kyle. [3] sounds more accurate. > > > > On Wed, Jul 15, 2020 at 3:00 PM Kyle Weaver wrote: > >> > >> I prefer [3]. > >> > >> On Tu

Re: Chronically flaky tests

Is there an observation that enabling tenacity improves the development experience on Python SDK? E.g. less wait time to get PR pass and merged? Or it might be a matter of a right number of retry to align with the "flakiness" of a test? -Rui On Thu, Jul 16, 2020 at 10:38 AM Valentyn Tymofieiev

Re: Chronically flaky tests

We used tenacity[1] to retry some unit tests for which we understood the nature of flakiness. [1] https://github.com/apache/beam/blob/3b9aae2bcaeb48ab43a77368ae496edc73634c91/sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner_test.py#L1156 On Thu, Jul 16, 2020 at 10:25 AM Kenneth

Re: Chronically flaky tests

Didn't we use something like that flaky retry plugin for Python tests at some point? Adding retries may be preferable to disabling the test. We need a process to remove the retries ASAP though. As Luke says that is not so easy to make happen. Having a way to make P1 bugs more visible in an ongoing

Re: KafkaIO sending KafkaRecords in CrossLanguage - where is the coder registered?

Note also that once you get the Bytes in Python, you can use whatever coder (or Map) to decode them that you want. On Thu, Jul 16, 2020 at 9:21 AM Boyuan Zhang wrote: > > Hi Piotr, > > X-Lang uses TypedWithoutMetadata, which outputs the KV directly instead of > KafkaRecord: see here. Given the l

Re: KafkaIO sending KafkaRecords in CrossLanguage - where is the coder registered?

Hi Piotr, X-Lang uses TypedWithoutMetadata , which outputs the KV directly instead of KafkaRecord: see here

Re: Chronically flaky tests

I don't think I have seen tests that were previously disabled become re-enabled. It seems as though we have about ~60 disabled tests in Java and ~15 in Python. Half of the Java ones seem to be in ZetaSQL/SQL due to missing features so unrelated to being a flake. On Thu, Jul 16, 2020 at 8:49 AM Gl

Re: Chronically flaky tests

There is something called test-retry-gradle-plugin [1]. It retries tests if they fail, and have different modes to handle flaky tests. Did we ever try or consider using it? [1]: https://github.com/gradle/test-retry-gradle-plugin On Thu, Jul 16, 2020 at 1:15 PM Gleb Kanterov wrote: > I agree wit

KafkaIO sending KafkaRecords in CrossLanguage - where is the coder registered?

I'm writing a python wrappers for KinesisIO and I encountered a problem that Read transform creates a PCollection with KinesisRecord class which's coder by default is assigned as 'beam:coders:javasdk:0.1'. I managed to register this coder using CoderTranslatorRegistrar which adds the coder to th

Re: Chronically flaky tests

I agree with what Ahmet is saying. I can share my perspective, recently I had to retrigger build 6 times due to flaky tests, and each retrigger took one hour of waiting time. I've seen examples of automatic tracking of flaky tests, where a test is considered flaky if both fails and succeeds for th