Contributor permission request for Apache Beam Jira

2019-06-10 Thread Andy Wang
Hello, My name is Andy and I'd like to contribute where I can to the project. Been using the tool for over a year now and would like to start giving back to the community. Could someone give me the proper permissions to assign Jira tickets to myself? Thanks, Andy

Re: [DISCUSS] Portability representation of schemas

2019-06-10 Thread Robert Bradshaw
On Mon, Jun 10, 2019 at 11:53 PM Kenneth Knowles wrote: > Good points. At a high level it doesn't sound like anything is blocking, > right? > It doesn't sound like we've settled on an actual proto definition yet. which may be influenced by the questions below. > On Mon, Jun 10, 2019 at 2:14

Re: [DISCUSS] Portability representation of schemas

2019-06-10 Thread Kenneth Knowles
Good points. At a high level it doesn't sound like anything is blocking, right? On Mon, Jun 10, 2019 at 2:14 AM Robert Bradshaw wrote: > On Sat, Jun 8, 2019 at 9:25 PM Kenneth Knowles wrote: > >> On Fri, Jun 7, 2019 at 4:35 AM Robert Burke wrote: >> >>> Wouldn't SDK specific types always be

Re: [ANNOUNCE] Apache Beam 2.13.0 released!

2019-06-10 Thread Chad Dombrova
> > > @Chad Thanks for the feedback. I agree that we can improve our release > notes. The particular issue you were looking for was part of the detailed > list [1] linked in the blog post: > https://jira.apache.org/jira/browse/BEAM-7029 Just to be clear, I had no idea about the feature ahead of

Re: DirectRunner timers are not strictly time ordered

2019-06-10 Thread Jan Lukavský
Yes, there is no difference between GC and user timers in this case. I think the problem is simply that when watermark moves from time T1 to T2, DirectRunner fires all timers that fire until T2, but that can create new timers for time between T1 and T2, and these will be fired later, although

Re: DirectRunner timers are not strictly time ordered

2019-06-10 Thread Jan Lukavský
I would think, that runners would actually fire correctly (at least FlinkRunner I think does). The problem is IMHO only in DirectRunner. Here is draft of PR [1]. As I said, I'm hunting a bug which causes tests not to finish, because watermark is stuck due to watermark hold in

Re: DirectRunner timers are not strictly time ordered

2019-06-10 Thread Lukasz Cwik
Jan are you editing the implementation of how timers work within the DirectRunner or are trying to build support for time sorted input on top of the Beam model for timers? Because I think you will need to do the former. On Mon, Jun 10, 2019 at 8:41 AM Jan Lukavský wrote: > Hm, that would

Re: DirectRunner timers are not strictly time ordered

2019-06-10 Thread Kenneth Knowles
GC timer should fire at a time after all legal user timers, though. Kenn On Mon, Jun 10, 2019 at 8:33 AM Lukasz Cwik wrote: > We hit an instance of this problem before and solved it rescheduling the > GC timer again if there was a conflicting timer that was also meant to fire. > > On Mon, Jun

Re: DirectRunner timers are not strictly time ordered

2019-06-10 Thread Jan Lukavský
For a single key. I'm getting into collision of timerId `__StatefulParDoGcTimerId` (StatefulDoFnRunner) and my timerId for flushing sorted elements in implementation of @RequiresTimeSortedInput. The timers are being swapped at the end of input (but it can happen anywhere near end of window),

Re: DirectRunner timers are not strictly time ordered

2019-06-10 Thread Reuven Lax
Do you mean for a single key or across keys? On Mon, Jun 10, 2019, 5:11 AM Jan Lukavský wrote: > Hi, > > I have come across issue [1], where I'm not sure how to solve this in > most elegant way. > > Any suggestions? > > Thanks, > > Jan > > [1] https://issues.apache.org/jira/browse/BEAM-7520 >

Re: [ANNOUNCE] New PMC Member: Pablo Estrada

2019-06-10 Thread Gleb Kanterov
Congratulations! On Fri, May 24, 2019 at 9:50 PM Joana Filipa Bernardo Carrasqueira < joanafil...@google.com> wrote: > Congratulations Pablo! Well deserved :D > > On Fri, May 17, 2019 at 3:14 PM Hannah Jiang > wrote: > >> Congratulations, Pablo, you deserve it! >> >> *From: *Mark Liu >> *Date:

DirectRunner timers are not strictly time ordered

2019-06-10 Thread Jan Lukavský
Hi, I have come across issue [1], where I'm not sure how to solve this in most elegant way. Any suggestions? Thanks,  Jan [1] https://issues.apache.org/jira/browse/BEAM-7520

Beam Dependency Check Report (2019-06-10)

2019-06-10 Thread Apache Jenkins Server
High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release JIRA Issue google-cloud-bigquery 1.6.1 1.14.0

Re: [ANNOUNCE] Apache Beam 2.13.0 released!

2019-06-10 Thread Maximilian Michels
Thanks for managing the release, Ankur! @Chad Thanks for the feedback. I agree that we can improve our release notes. The particular issue you were looking for was part of the detailed list [1] linked in the blog post: https://jira.apache.org/jira/browse/BEAM-7029 Cheers, Max [1]

Re: [DISCUSS] Portability representation of schemas

2019-06-10 Thread Robert Bradshaw
On Sat, Jun 8, 2019 at 9:25 PM Kenneth Knowles wrote: > On Fri, Jun 7, 2019 at 4:35 AM Robert Burke wrote: > >> Wouldn't SDK specific types always be under the "coders" component >> instead of the logical type listing? >> >> Offhand, having a separate normalized listing of logical schema types

Re: I'm thinking about new features, what do you think?

2019-06-10 Thread i.am.moai
Hi all. It was necessary to normalize the identity of instances with multiple identities in my work, as I thought why I needed such a function. It is a name identification of identity. I first thought about doing this by customizing the "equals" method of the object that is the key of

Re: @RequireTimeSortedInput design draft

2019-06-10 Thread Jan Lukavský
Hi Reza, > if you are interested and have the bandwidth would be great to have you as a reviewer for the PR. I'd be happy to. Cheers,  Jan On 6/10/19 3:52 AM, Reza Rokni wrote: Hi, Interesting reading on the issue 143 :-) My example is more specific in its scope but the general pattern