Re: [DISCUSS] Portability representation of schemas

>From a Python type hints perspective, how do schemas fit? Type hints are currently used to determine which coder to use. It seems that given a schema field, it would be useful to be able to convert it to a coder (using URNs?), and to convert the coder into a typing type. This would allow for

Re: [DISCUSS] Portability representation of schemas

Are you suggesting that schemas become an explicit field on PCollection or that the coder on PCollections has a well known schema coder type that has a payload that has field names, ids, type, ...? I'm much more for the latter since it allows for versioning schema representations over time without

Re: [discuss] Reducing log verbosity for Python failures?

In that case compaction makes much more sense. My experience was when a single test in a suite fails, I end up seeing lots of logs (usually proto dumps) from unrelated tests. *From: *Pablo Estrada *Date: *Wed, May 8, 2019 at 3:48 PM *To: *dev My impression - and I might be wrong [see meme at

Re: [discuss] Reducing log verbosity for Python failures?

My impression - and I might be wrong [see meme at 1], is that these are logged in debug mode, but when the test fails, the test runner dumps them all to stdout. Best -P. [1]

Re: [discuss] Reducing log verbosity for Python failures?

+1 It is hard to debug with lots logs messages. And if anybody is using them for development we can make those logs debug logs and hide them by default. *From: *Robert Bradshaw *Date: *Wed, May 8, 2019 at 3:01 PM *To: *dev +1 to making them significantly more compact in most cases. > > From:

Re: [discuss] Reducing log verbosity for Python failures?

+1 to making them significantly more compact in most cases. From: Pablo Estrada Date: Wed, May 8, 2019 at 11:35 PM To: dev > Hello all, > Some tests in Python have the problem that when they fail, lots of internal > logging is dumped onto stdout, and we end up having to scroll way up to find

Re: Python SDK timestamp precision

From: Kenneth Knowles Date: Wed, May 8, 2019 at 6:50 PM To: dev > This got pretty long, but I don't yet want to end it, because there's not > quite yet a solution that will allow a user to treat timestamps from most > systems as Beam timestamps. +1, it'd be really nice to find a solution to

[discuss] Reducing log verbosity for Python failures?

Hello all, Some tests in Python have the problem that when they fail, lots of internal logging is dumped onto stdout, and we end up having to scroll way up to find the actual stack trace for the failed test. This logging, as far as i can tell, is dumping of fn api protos. Does anyone use these

Re: [discuss] A tweak to the Python API for SDF?

Hello all, The API has been updated for Python (See https://github.com/apache/beam/pull/8430). Please, if you catch any documentation that needs updating, flag to me or just propose the change : ) As for Java - we didn't end up determining whether it makes sense to update the API as well.

Re: Requesting contributor permission for Beam JIRA tickets

Welcome, I have assigned BEAM-7240 to you. On Wed, May 8, 2019 at 10:30 AM Ajo Thomas wrote: > Hello, > > I am Ajo Thomas and I was hoping to work on making some improvements to > the beam KinesisIO java SDK. > I have created a ticket for it [ > https://issues.apache.org/jira/browse/BEAM-7240 ]

Re: [DISCUSS] Backwards compatibility of @Experimental features

On Wed, May 8, 2019 at 9:29 AM Ahmet Altay wrote: > > > *From: *Kenneth Knowles > *Date: *Wed, May 8, 2019 at 9:24 AM > *To: *dev > > >> >> On Fri, Apr 19, 2019 at 3:09 AM Ismaël Mejía wrote: >> >>> It seems we mostly agree that @Experimental is important, and that API >>> changes (removals)

Re: [DISCUSS] Portability representation of schemas

On Wed, May 8, 2019 at 1:23 PM Robert Bradshaw wrote: > Very excited to see this. In particular, I think this will be very > useful for cross-language pipelines (not just SQL, but also for > describing non-trivial data (e.g. for source and sink reuse). > > The proto specification makes sense to

Re: [DISCUSS] Portability representation of schemas

Very excited to see this. In particular, I think this will be very useful for cross-language pipelines (not just SQL, but also for describing non-trivial data (e.g. for source and sink reuse). The proto specification makes sense to me. The only thing that looks like it's missing (other than

Re: Kotlin iterator error

Does Kotlin have a different Iterable than the standard Java one? On Wed, May 8, 2019 at 10:47 AM Kenneth Knowles wrote: > Filed https://issues.apache.org/jira/browse/BEAM-7247 and assigned > initially to +Reuven Lax who is knowledgeable about > the @Element annotation and analysis. > > Kenn >

Re: Coder Evolution

There was a thread about coder update in the past here[1]. Also, Reuven sent out a doc[2] about pipeline drain and update which was discussed in this thread[3]. I believe there have been more references to pipeline update in other threads when people tried to change coder encodings in the past as

Re: [DISCUSS] Portability representation of schemas

On Wed, May 8, 2019 at 10:57 AM Rui Wang wrote: > Regarding to DATETIME, I totally agree it should be removed as > primitive type to avoid that each language has to find their time libraries > (and if they could not find any, they will likely go to logical type and > use int64 from Schema). > >

Coder Evolution

Hi, I'm looking into updating the Flink Runner to Flink version 1.8. Since version 1.7 Flink has a new optional interface for Coder evolution*. When a Flink pipeline is checkpointed, CoderSnapshots are written out alongside with the checkpointed data. When the pipeline is restored from that

Re: [DISCUSS] Portability representation of schemas

Regarding to DATETIME, I totally agree it should be removed as primitive type to avoid that each language has to find their time libraries (and if they could not find any, they will likely go to logical type and use int64 from Schema). I have two questions regarding to the representation: 1.

Re: request for beam minor release

For the benefit of the thread, I will also call out our incubating LTS (Long-Term Support) policy. For critical fixes, we will issue patch releases on the 2.7 branch. We are currently gathering proposals for cherry-picks to 2.7.1 to do that release. Other than that, releases are a lot of work so

Re: Kotlin iterator error

Filed https://issues.apache.org/jira/browse/BEAM-7247 and assigned initially to +Reuven Lax who is knowledgeable about the @Element annotation and analysis. Kenn On Wed, May 8, 2019 at 2:40 AM Maximilian Michels wrote: > Hi Ankur, > > I've left a comment. Looking at the stack trace [1], this

Requesting contributor permission for Beam JIRA tickets

Hello, I am Ajo Thomas and I was hoping to work on making some improvements to the beam KinesisIO java SDK. I have created a ticket for it [ https://issues.apache.org/jira/browse/BEAM-7240 ] and was hoping to assign it to myself. Requesting the admins to please add me as a contributor for Beam's

Re: Artifact staging in cross-language pipelines

On Tue, May 7, 2019 at 10:21 AM Maximilian Michels wrote: > Here's the first draft: > > https://docs.google.com/document/d/1XaiNekAY2sptuQRIXpjGAyaYdSc-wlJ-VKjl04c8N48/edit?usp=sharing > > It's rather high-level. We may want to add more details once we have > finalized the design. Feel free to

[DISCUSS] Portability representation of schemas

Beam Java's support for schemas is just about done: we infer schemas from a variety of types, we have a variety of utility transforms (join, aggregate, etc.) for schemas, and schemas are integrated with the ParDo machinery. The big remaining task I'm working on is writing documentation and

Re: Python SDK timestamp precision

This got pretty long, but I don't yet want to end it, because there's not quite yet a solution that will allow a user to treat timestamps from most systems as Beam timestamps. I'm cutting pieces just to make inline replies easier to read. On Tue, Apr 23, 2019 at 9:03 AM Robert Bradshaw wrote:

Re: [DISCUSS] Backwards compatibility of @Experimental features

*From: *Kenneth Knowles *Date: *Wed, May 8, 2019 at 9:24 AM *To: *dev > > On Fri, Apr 19, 2019 at 3:09 AM Ismaël Mejía wrote: > >> It seems we mostly agree that @Experimental is important, and that API >> changes (removals) on experimental features should happen quickly but still >> give some

Plans for Python type hints

Hi, I've written a document, with input from robertwb@, detailing the direction forward I want to take type hints in Python 3. The document contains background, a survey of existing type tools, and example usage. The summary of proposed changes is: 1. Update Beam's type hinting support to

Re: [DISCUSS] Backwards compatibility of @Experimental features

On Fri, Apr 19, 2019 at 3:09 AM Ismaël Mejía wrote: > It seems we mostly agree that @Experimental is important, and that API > changes (removals) on experimental features should happen quickly but still > give some time to users so the Experimental purpose is not lost. > > Ahmet proposal given

Re: request for beam minor release

Assuming 2.13 will include or otherwise be supported by flink-runner-1.7 then this should not be an issue. From: Jean-Baptiste Onofré Sent: Wednesday, May 8, 2019 10:09 AM To: dev@beam.apache.org Subject: Re: request for beam minor release I second Max here. If

Re: request for beam minor release

I second Max here. If you are just looking for this specific commit, you can take a next release that will include it. Regards JB On 08/05/2019 16:27, Maximilian Michels wrote: > Hi Richard, > > Would it be an option to use the upcoming 2.13.0 release? The commit > will be part of that release.

Re: request for beam minor release

Hi Richard, Would it be an option to use the upcoming 2.13.0 release? The commit will be part of that release. Thanks, Max On 08.05.19 15:43, Jean-Baptiste Onofré wrote: Hi, Any release are tagging. We create a branch based on a master commit. Are you requesting 2.10.1 maintenance release

Re: request for beam minor release

Hi, Any release are tagging. We create a branch based on a master commit. Are you requesting 2.10.1 maintenance release ? Regards JB On 08/05/2019 15:10, Moorhead,Richard wrote: > Is there a process for tagging a commit in master for a minor release? > > I am trying to get this >

request for beam minor release

Is there a process for tagging a commit in master for a minor release? I am trying to get this commit released into 2.10.1 CONFIDENTIALITY NOTICE This message and any included attachments are from

Re: Kotlin iterator error

Hi Ankur, I've left a comment. Looking at the stack trace [1], this looks like a problem with our Reflection analysis. -Max [1] https://gist.github.com/marcoslin/e1e19afdbacac9757f6974592cfd8d7f#file-stack-trace-txt On 04.05.19 00:56, Ankur Goenka wrote: Hi, A beam user on stackoverflow