Re: sdk.io.gcp.pubsublite.SubscriptionPartitionLoaderTest failing

Yes, thank you so much for fixing this! On Tue, May 25, 2021 at 4:48 PM Brian Hulette wrote: > > Would someone be willing to review and merge > https://github.com/apache/beam/pull/14878 which should fix this? > > Done! Thanks for writing the fix. > > On Mon, May 24, 2021 at 8:22 PM Daniel

Re: sdk.io.gcp.pubsublite.SubscriptionPartitionLoaderTest failing

> Would someone be willing to review and merge https://github.com/apache/beam/pull/14878 which should fix this? Done! Thanks for writing the fix. On Mon, May 24, 2021 at 8:22 PM Daniel Collins wrote: > Looks to me like this is flaky because the Watch class doesn't provide a > way to ensure a

BEAM-8787 Contribution Guide and WIKI Improvements

Hello, As a complement to the BEAM-8787 issue, I updated some of the pages at the Developer Guides wiki. You can see that most of the changes were: updates to setup info, organization of information, edit of steps. I can collect

Re: Out of band pickling in Python (pickle5)

Hm this would definitely be of interest for the DataFrame API, which is shuffling pandas objects. This issue [1] confirms what you suggested above, that pandas supports out-of-band pickling since DataFrames are mostly just collections of numpy arrays. Brian [1]

Re: Out of band pickling in Python (pickle5)

Beam's PickleCoder would need to be updated to pass the "buffer_callback" argument into pickle.dumps() and the "buffers" argument into pickle.loads(). I expect this would be relatively straightforward. Then it should "just work", assuming that data is stored in objects (like NumPy arrays or

Re: Out of band pickling in Python (pickle5)

I'm not aware of anyone looking at it. Will out-of-band pickling "just work" in Beam for types that implement the correct interface in Python 3.8? On Tue, May 25, 2021 at 2:43 PM Evan Galpin wrote: > +1 > > FWIW I recently ran into the exact case you described (high serialization > cost). The

Re: One Pager - Test Command Line Discoverability in Beam

Cool; will be good to have and make things clearer! On Tue, May 25, 2021 at 2:39 PM Kyle Weaver wrote: > I left some comments. In summary, I think this is mostly a documentation > problem. If running a test isn't as easy as "./gradlew > $MODULE:integrationTest", there should be instructions in

Re: Out of band pickling in Python (pickle5)

+1 FWIW I recently ran into the exact case you described (high serialization cost). The solution was to implement some not-so-intuitive alternative transforms in my case, but I would have very much appreciated faster serialization performance. Thanks, Evan On Tue, May 25, 2021 at 15:26 Stephan

Re: One Pager - Test Command Line Discoverability in Beam

I left some comments. In summary, I think this is mostly a documentation problem. If running a test isn't as easy as "./gradlew $MODULE:integrationTest", there should be instructions in the test class's javadoc. On Tue, May 25, 2021 at 2:05 PM Udi Meiri wrote: > My first place to go would be

Re: One Pager - Test Command Line Discoverability in Beam

My first place to go would be here: https://cwiki.apache.org/confluence/display/BEAM/Java+Tips (although it doesn't document your use-case) You are right that finding the correct gradle task or jenkins job is not straightforward. On Tue, May 25, 2021 at 12:48 PM Alex Amato wrote: > Friendly

Fwd: One Pager - Test Command Line Discoverability in Beam

Friendly ping. I'll wait for more suggestions by the end of the week. Then close it out. -- Forwarded message - From: Alex Amato Date: Fri, May 21, 2021 at 2:54 PM Subject: One Pager - Test Command Line Discoverability in Beam To: dev Hi, I have had some issues determining how

Out of band pickling in Python (pickle5)

Has anyone looked into out of band pickling for Beam's Python SDK, i.e., Pickle protocol version 5? https://www.python.org/dev/peps/pep-0574/ https://docs.python.org/3/library/pickle.html#out-of-band-buffers For Beam pipelines passing around NumPy arrays (or collections of NumPy arrays, like

Flaky test issue report (40)

This is your daily summary of Beam's current flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake) These are P1 issues because they have a major negative impact on the community and make it hard to

P1 issues report (40)

This is your daily summary of Beam's current P1 issues, not including flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake). See

Re: Oracle JDBC driver with expansion service

Are you using Maven? The JDBC jar should be added to pom.xml as dependency. If not using Maven, JDBC jar should be in classpath.  On Tuesday, May 25, 2021, 09:33:25 a.m. EDT, Rafael Ribeiro wrote: Hi, I'm trying to read and write on Oracle database using the JDBC driver of Beam but

Re: [DISCUSSION] Docker based development environment issue

I checked it again and I have failed Docker-dependent tests: > Task :sdks:java:io:clickhouse:test org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod FAILED java.lang.IllegalStateException at DockerClientProviderStrategy.java:215

Re: Oracle JDBC driver with expansion service

Hi, This question looks more as a user-related question, so let's continue this conversation on user@ — Alexey > On 25 May 2021, at 15:32, Rafael Ribeiro wrote: > > Hi, > > I'm trying to read and write on Oracle database using the JDBC driver of Beam > > but I'm having some problems,

Re: contributor permission for Beam Jira tickets

Hi Paresh, Thank you for working on this! Could you precise, what is your Apache Jira ID? — Alexey > On 25 May 2021, at 12:02, Paresh Saraf wrote: > > Hi, > > This is Paresh Saraf from MongoDB. I'm working on MongoDBIO Connector for > apache beam. Can someone add me as a contributor for

contributor permission for Beam Jira tickets

Hi, This is Paresh Saraf from MongoDB. I'm working on MongoDBIO Connector for apache beam. Can someone add me as a contributor for Beam's Jira issue tracker? I would like to create/assign tickets for my work. Thanks, Paresh -- { Name : "Paresh Saraf", Title: "Senior Solutions

Re: Missing copyright notices due to LICENSE change

The owners at pkg go.dev say they can't properly recognize the python license (see https://github.com/golang/go/issues/45095) due to the license being somewhat domineering (a go project could *only* have that license if it had that license, apparently). We also can't do a directory specific

Oracle JDBC driver with expansion service

Hi, I'm trying to read and write on Oracle database using the JDBC driver of Beam but I'm having some problems, specially on Dataflow that does not find the jar dependency could anyone help to solve this problem? PS: I have to compile JDBC class with jar dependency on it and create a expansion