Re: Cassandra IO issues and contributing

On Thu, Dec 12, 2019 at 8:43 PM Kenneth Knowles wrote: > On Thu, Dec 12, 2019 at 3:30 PM Vincent Marquez > wrote: > >> Hello, as I've mentioned in previous emails, I've found the CassandraIO >> connector lacking some essential features for efficient batch processing in >> real world scenarios.

Re: [BEAM-9000] Java Test Assertions without relying on toString

Thank you for the response. I like that JSONassert approach. Added that idea into the ticket. https://issues.apache.org/jira/browse/BEAM-9000 On Thu, Dec 19, 2019 at 5:26 PM Luke Cwik wrote: > > What about using JSONassert or hamcrest-json or some other JSON matcher > library? > > On Thu, Dec

Re: [PROPOSAL] python precommit timeouts

This sounds reasonable. Would this be configurable per-test if needed? On Thu, Dec 19, 2019 at 5:52 PM Udi Meiri wrote: > Looking at this console log > , > it seems that some

[PROPOSAL] python precommit timeouts

Looking at this console log , it seems that some pytests got stuck (or slowed down considerably). I'd like to put a 10 minute default timeout on all unit tests, using the

External transform API in Java SDK

I wanted to know if anybody has any comment on external transform API for Java SDK. `External.of()` can create external transform for Java SDK. Depending on input and output types, two additional methods are provided: `withMultiOutputs()` which specifies the type of PCollection and

Re: Is org.apache.beam.sdk.transforms.FlattenTest.testFlattenMultipleCoders supposed to be supported ?

I'm pretty sure that Flatten with different coders is well defined. input: List> output: PCollection When flatten is executed using T vs encoded(T), transcoding can be optimized because the coder for the output PCollection is assumed to be able to encode all T's. The DirectRunner specifically

[BEAM-9000] Java Test Assertions without relying on toString

Hi Beam developers, There are many Java tests relying on toString() methods for assertions [1]. This style is prone to unnecessary maintenance of the test code when upgrading dependencies. For example, BEAM-8695 encountered ~10 comparison failures due to change in toString implementation when I

Re: Unifying Build/contributing instructions

+1 for website focus On Thu, Dec 19, 2019 at 10:22 AM Elliotte Rusty Harold wrote: > That's two votes for > https://beam.apache.org/contribute/contribution-guide/ and a lot of > abstentions. I'll update the PR to move content to > https://beam.apache.org/contribute/contribution-guide/ > > On

Re: BEAM-8989 fix for 2.18.0 release

Thanks. I've reassigned the bug to Reuven and pushed the fix back to 2.19.0 On Thu, Dec 19, 2019 at 11:11 AM Luke Cwik wrote: > Either Salman Raza who developed the PR or Reuven Lax who reviewed it > would have the most context. I don't know Salman's contact information > though. > > On Thu,

Re: BEAM-8989 fix for 2.18.0 release

Either Salman Raza who developed the PR or Reuven Lax who reviewed it would have the most context. I don't know Salman's contact information though. On Thu, Dec 19, 2019 at 10:18 AM Udi Meiri wrote: > The JIRA issue was assigned to me, but I have no background in the issue. > Who would be the

Re: Unifying Build/contributing instructions

That's two votes for https://beam.apache.org/contribute/contribution-guide/ and a lot of abstentions. I'll update the PR to move content to https://beam.apache.org/contribute/contribution-guide/ On Thu, Dec 19, 2019 at 12:29 PM Luke Cwik wrote: > > +1 on Kenn's suggestion. > > On Thu, Dec 12,

Re: Unifying Build/contributing instructions

+1 on Kenn's suggestion. On Thu, Dec 12, 2019 at 8:17 PM Kenneth Knowles wrote: > Thanks for taking this on! My preference would be to have CONTRIBUTING.md > link to https://beam.apache.org/contribute/contribution-guide/ and focus > work on the latter. > > Kenn > > On Thu, Dec 12, 2019 at 12:38

Re: Apache beam Python Error runners-spark-job-server-2.19.0-SNAPSHOT.jar not found

Hi Dhiren, Running via the Spark CLI doesn't work. You need to execute your Python pipeline directly. The Beam job server will then submit to the Spark cluster. The Jar can't be found because you are working with the development version, for which the jars haven't been released on Maven

Re: Testing Apache Beam with JDK 14 EA builds

Hi Kenn, Apologies for the delay, just back in the office today. On 16/12/2019 23:15, Kenneth Knowles wrote: Hi Rory, Here at Beam we are still in a major long-term push to support Java 11 for pipeline authoring and JRE 11 for execution. Many subtasks are filed under