Re: BigTable reader for Python?

2022-12-29 Thread Luke Cwik via dev
AutoService relies on Java's compiler annotation processor. https://github.com/google/auto/tree/main/service#getting-started shows that you need to configure Java's compiler to use the annotation processors within AutoService. I saw this public gist that seemed to enable using the AutoService anno

Re: BigTable reader for Python?

2022-12-29 Thread Lina Mårtensson via dev
That's good news about the direct runner, thanks! On Thu, Dec 29, 2022 at 2:02 PM Robert Bradshaw wrote: > On Thu, Jul 28, 2022 at 5:37 PM Chamikara Jayalath via dev > wrote: > > > > On Thu, Jul 28, 2022 at 4:51 PM Lina Mårtensson > wrote: > >> > >> Thanks for the detailed answers! > >> > >> I

Re: BigTable reader for Python?

2022-12-29 Thread Lina Mårtensson via dev
Thanks Luke! That makes sense. Are there any instructions/do you have any further pointers on how to build this JAR? Not working with Java normally, I don't know anything about what to expect here. I'll have to figure out how to build it with Bazel eventually of course, but instructions for how to

Re: BigTable reader for Python?

2022-12-29 Thread Robert Bradshaw via dev
On Thu, Jul 28, 2022 at 5:37 PM Chamikara Jayalath via dev wrote: > > On Thu, Jul 28, 2022 at 4:51 PM Lina Mårtensson wrote: >> >> Thanks for the detailed answers! >> >> I totally get the points about development & maintenance cost, and, >> from a user perspective, about getting the performance r

Re: BigTable reader for Python?

2022-12-29 Thread Luke Cwik via dev
I would have expected a META-INF/services/org.apache.beam.sdk.expansion.ExternalTransformRegistrar file in the jar containing the fully qualified class name of BigtableRegistrar in it. See https://repo1.maven.org/maven2/org/apache/beam/beam-sdks-java-io-kafka/2.43.0/beam-sdks-java-io-kafka-2.43.0.j

Beam Dependency Check Report (2022-12-29)

2022-12-29 Thread Apache Jenkins Server
<<< text/html; charset=UTF-8: Unrecognized >>>

Re: SparkRunner - ensure SDF output does not need to fit in memory

2022-12-29 Thread Jozef Vilcek
I am surprised to hear that Dataflow runner ( which I never used ) would have this kind oflimitation. I see that the `OutputManager` interface is implemented to write to `Receiver` [1] which follows the push model. Do you have a reference I can take a look to review the must fit memory limitation?

Beam High Priority Issue Report (43)

2022-12-29 Thread beamactions
This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/24787 [Failing Test]: sk