>
> Is it actually necessary for a PTransform that is configured via the
> Schema mechanism to also be one that uses RowCoder? Those strike me as two
> separate concerns and unnecessarily limiting
>
+1 to this, I have a wip branch somewhere with a coder for
https://github.com/apache/incubator-fury
In the same vein of reducing boilerplate to make a schema-aware transform,
I've opened a small PR to relax the requirement of 3 methods:
https://github.com/apache/beam/pull/30560
Would appreciate someone taking a look!
On Thu, Jun 29, 2023 at 9:43 AM Ahmed Abualsaud
wrote:
> Can someone take a
Can someone take a quick look at https://github.com/apache/beam/pull/27202?
If things look good, let's try getting it in before the release cut as I'm
also updating our cross-language documentation and would like to include
these changes.
Thank you,
Ahmed
On Thu, Jun 22, 2023 at 8:06 PM Reuven La
The goal was to make schema transforms as efficient as hand-written coders.
We found the avro encoding/decoding to often be quite inefficient, which is
one reason we didn't use it for schemas.
Our schema encoding is internal to Beam though, and not suggested for use
external to a pipeline. For sou
On Thu, Jun 22, 2023 at 2:19 PM Ahmed Abualsaud
wrote:
> Thank you all for your input. I have a PR for the changes I mentioned in
> my initial email: https://github.com/apache/beam/pull/27202. Please
> review when you get a chance!
>
> > perhaps we should consider just going to something Avro for
Thank you all for your input. I have a PR for the changes I mentioned in my
initial email: https://github.com/apache/beam/pull/27202. Please review
when you get a chance!
> perhaps we should consider just going to something Avro for portable
coding rather than something custom
Did you mean using
Sure, I get that… though perhaps we should consider just going to something
Avro for portable coding rather than something custom.
On Tue, May 30, 2023 at 2:22 PM Chamikara Jayalath
wrote:
> Input/output PCollection types at least have to be portable Beam types [1]
> for cross-language to work.
Input/output PCollection types at least have to be portable Beam types [1]
for cross-language to work.
I think we restricted schema-aware transforms to PCollection since Row
was expected to be an efficient replacement for arbitrary portable Beam
types (not sure how true that is in practice current
Is it actually necessary for a PTransform that is configured via the Schema
mechanism to also be one that uses RowCoder? Those strike me as two
separate concerns and unnecessarily limiting.
On Tue, May 30, 2023 at 1:29 PM Chamikara Jayalath
wrote:
> +1 for the simplification.
>
> On Tue, May 30,
+1 for the simplification.
On Tue, May 30, 2023 at 12:33 PM Robert Bradshaw
wrote:
> Yeah. Essentially one needs do (1) name the arguments and (2) implement
> the transform. Hopefully (1) could be done in a concise way that allows for
> easy consumption from both Java and cross-language.
>
+1 b
Yeah. Essentially one needs do (1) name the arguments and (2) implement the
transform. Hopefully (1) could be done in a concise way that allows for
easy consumption from both Java and cross-language.
On Tue, May 30, 2023 at 12:25 PM Byron Ellis wrote:
> Or perhaps the other way around? If you ha
Or perhaps the other way around? If you have a Schema we can auto-generate
the associated builder on the PTransform? Either way, more DRY.
On Tue, May 30, 2023 at 10:59 AM Robert Bradshaw via dev <
dev@beam.apache.org> wrote:
> +1 to this simplification, it's a historical artifact that provides n
+1 to this simplification, it's a historical artifact that provides no
value.
I would love it if we also looked into ways to auto-generate the
SchemaTransformProvider (e.g. via introspection if a transform takes a
small number of arguments, or uses the standard builder pattern...),
ideally with so
+1 in general (I will dig into details too, but later)
I will add a couple more steps that I dislike, which exist only for
historical obsolete reasons:
The whole existence of runners/core-construction-java and everything in it.
The main reason it exists is to make it so that main SDK has no depen
14 matches
Mail list logo
