Re: Python 3: final step

2019-01-04 Thread Manu Zhang
Guys, Happy New Year !!! I haven't got much time to contribute to Python 3 support. What is the progress now ? It seems there are quite a few open issues under https://issues.apache.org/jira/browse/BEAM-1251. People have kept asking about Python 3 support in tf.transform (

Beam Contribution

2019-01-04 Thread David Rieber
Hello, My name is David Rieber. I work on Google Cloud Dataflow Service. I would like to be added as a contributor to Beam. My Jira user name is drieber. Thanks!

Re: Schemas in the Go SDK

2019-01-04 Thread Robert Burke
Having slept on it here are my thoughts, but granted, AFAICT there is no spec for schema's so my understanding is based on what I've learned in the last 18-ish hours. If there is a spec, I'd love to see it. *1.* Default behavior to support Schema's in some way doesn't remove the need for certain

Re: [Go SDK] User Defined Coders

2019-01-04 Thread Robert Burke
I think you're right Kenn. Reuven alluded to the difficulty in inference of what to use between AtomicType and the rest, in particular Struct. Go has the additional concerns around Pointer vs Non Pointer types which isn't a concern either Python or Java have, but has implications on pipeline

Re: [Go SDK] User Defined Coders

2019-01-04 Thread Reuven Lax
Maybe a good first step would be to write a doc explaining how this would work in the Go SDK and share with the dev list. It's possible we will decide to just implement Coders first, however that way this will be done with everyone fully understanding the design tradeoffs. Reuven On Fri, Jan 4,

Re: [Go SDK] User Defined Coders

2019-01-04 Thread Kenneth Knowles
On Thu, Jan 3, 2019 at 4:33 PM Reuven Lax wrote: > If a user wants custom encoding for a primitive type, they can create a > byte-array field and wrap that field with a Coder > This is the crux of the issue, right? Roughly, today, we've got: Schema ::= [ (fieldname, Type) ]

Re: excessive java precommit logging

2019-01-04 Thread Udi Meiri
To follow up, I did some research yesterday on removing --info and my findings are: - Gradle Test tasks generate HTML and Junit XML reports. Both contain a stacktrace, STDOUT, and STDERR of the failed test (example

Re: [Go SDK] User Defined Coders

2019-01-04 Thread Robert Burke
That's an interesting idea. I must confess I don't rightly know the difference between a schema and coder, but here's what I've got with a bit of searching through memory and the mailing list. Please let me know if I'm off track. As near as I can tell, a schema, as far as Beam takes it