Re: Python 3: final step

2019-01-07 Thread Ahmet Altay
+Matthias Feys +Valentyn Tymofieiev +Mark Liu could add more details here since they are working on Python 3 for a while now. Hopeful state is that we might have python 3 working with DirectRunner in the release after this one (2.12). Mark is also working on getting python 3 working on a

Re: Enforce javadoc comments in public methods?

2019-01-07 Thread Connell O'Callaghan
+1 On Mon, Jan 7, 2019 at 5:22 PM Udi Meiri wrote: > +1 > > On Mon, Jan 7, 2019 at 4:49 PM Daniel Oliveira > wrote: > >> +1 >> >> I like this idea, especially with the line number requirement. The exact >> number of lines is debatable, but you could go as low as 10 lines and that >> would

Re: Enforce javadoc comments in public methods?

2019-01-07 Thread Udi Meiri
+1 On Mon, Jan 7, 2019 at 4:49 PM Daniel Oliveira wrote: > +1 > > I like this idea, especially with the line number requirement. The exact > number of lines is debatable, but you could go as low as 10 lines and that > would exclude any trivial setters and getters. Even better might be if it's >

Re: Enforce javadoc comments in public methods?

2019-01-07 Thread Daniel Oliveira
+1 I like this idea, especially with the line number requirement. The exact number of lines is debatable, but you could go as low as 10 lines and that would exclude any trivial setters and getters. Even better might be if it's possible to configure checkstyle to ignore this for getters and

Re: [Go SDK] User Defined Coders

2019-01-07 Thread Andrew Pilloud
+1 on this. I think we are somewhat lacking on a written design for Schemas in Java. This would be really useful in driving adoption and expanding to other languages. Andrew On Mon, Jan 7, 2019 at 3:43 PM Robert Burke wrote: > Might I see the design doc (not code) for how they're supposed to

Re: Add all tests to release validation

2019-01-07 Thread Ahmet Altay
This is a a good idea. Some suggestions: - It would be nicer if we can figure out process to act on flaky test more frequently than releases. - Another improvement in the process would be having actual owners of issues rather than auto assigned component owners. A few folks have 100+ assigned

Re: Add all tests to release validation

2019-01-07 Thread Kenneth Knowles
I love this idea. It can easily feel like bugs filed for Jenkins flakes/failures just get lost if there is no process for looking them over regularly. I would suggest that test failures / flakes all get filed with Fix Version = whatever release is next. Then at release time we can triage the

Re: [Go SDK] User Defined Coders

2019-01-07 Thread Robert Burke
Might I see the design doc (not code) for how they're supposed to look and work in Java first? I'd rather not write a document based on a speculative understanding of Schemas based on the littany of assumptions I'm making about them. On Mon, Jan 7, 2019, 2:35 PM Reuven Lax wrote: > I suggest

Add all tests to release validation

2019-01-07 Thread Sam Rohde
Hi All, There are a number of tests in our system that are either flaky or permanently red. I am suggesting to add, if not all, then most of the tests (style, unit, integration, etc) to the release validation step. In this way, we will add a regular cadence to ensuring greenness and no flaky

Re: [Go SDK] User Defined Coders

2019-01-07 Thread Reuven Lax
I suggest that we write out a design of what schemas in go would look like and how it would interact with coders. We'll then be in a much better position to decide what the right short-term path forward is. Even if we decide it makes more sense to build up the coder support first, I think this

Re: Query expressions for schema fields

2019-01-07 Thread Reuven Lax
Some parts of schemas need a spec that is standardized cross language (in particular, the definition of what a schema is). Other things can be language specific. The next step with schemas will need to be formalizing those answer. On Mon, Jan 7, 2019 at 2:13 PM Robert Burke wrote: > In the

Re: Query expressions for schema fields

2019-01-07 Thread Robert Burke
In the eventual future where the Go SDK supports schemas, it should be possible to use struct Field Tags to specify paths for extraction from schema data, for usage similar to what Java uses parameter Annotations. eg. type MyKey struct { K string `jsonpath:userid` } type MyValue struct{ K

Re: [Go SDK] User Defined Coders

2019-01-07 Thread Robert Burke
Kenn has pointed out to me that Coders are not likely going to vanish in the next while, in particular over the FnAPI, so having a coder registry does remain useful, as described by an early adopter in another thread. On Fri, Jan 4, 2019, 10:51 AM Robert Burke wrote: > I think you're right

Re: Beam Contribution

2019-01-07 Thread Robert Burke
I like that idea. Even something as simple as "I'm working on X,Y,Z and here's my github profile." It would also be handy for allowing users tagging in specific folks if there's a particular aspect they want to discuss. On Mon, Jan 7, 2019, 7:01 AM Maximilian Michels wrote: > Hi David, > >

Re: Query expressions for schema fields

2019-01-07 Thread Reuven Lax
I'll take a look. Honestly though, if we leave out features such as array slices this is a dirt-simple path syntax, that pretty much matches what SQL does. It's basically just field1.field2, or field1.*. JMESPath along with JsonPath also supports various aggregations, which I think is beyond

Re: Enforce javadoc comments in public methods?

2019-01-07 Thread Ankur Goenka
I think it makes sense. Having an annotation to suppress this check for a method/class instead of adding trivial comment would be useful. On Mon, Jan 7, 2019 at 9:53 AM Ruoyun Huang wrote: > Yeah. Agree there is no reason to enforce anything for trivial methods > like setter/getter. > > What I

Re: Query expressions for schema fields

2019-01-07 Thread Jeff Klukas
There is also JMESPath (http://jmespath.org/) which is quite similar to JsonPath, but does have a spec and lacks the leading $ character. The AWS CLI uses JMESPath for defining queries. On Mon, Jan 7, 2019 at 1:05 PM Reuven Lax wrote: > > > On Mon, Jan 7, 2019 at 1:44 AM Robert Bradshaw >

Re: Query expressions for schema fields

2019-01-07 Thread Reuven Lax
On Mon, Jan 7, 2019 at 1:44 AM Robert Bradshaw wrote: > On Sun, Jan 6, 2019 at 12:46 PM Reuven Lax wrote: > > > > Some time ago, @Jean-Baptiste Onofré made the excellent suggestion that > we look into using JsonPath as a selector format for schema fields. This > provides a simple and natural

Re: Enforce javadoc comments in public methods?

2019-01-07 Thread Ruoyun Huang
Yeah. Agree there is no reason to enforce anything for trivial methods like setter/getter. What I meant is to enforce only for a method that is *BOTH* 1) public method 2) has longer than N lines. sorry for not making the proposal clear enough in the original message, it should've better titled

Re: Add code quality checks to pre-commits.

2019-01-07 Thread Mikhail Gryzykhin
@Jason I believe that we want to get some coverage report as a PR pre-commit check, so that we can utilize it as part of code review. Removing parallelization increases pre-commit duration greatly (see the big drop on this graph

Re: Add code quality checks to pre-commits.

2019-01-07 Thread Jason Kuster
Michael, could we solve that problem by adding an additional build with those features disabled specifically for generating coverage data over time? Any tradeoffs to doing something like that? On Mon, Jan 7, 2019 at 8:23 AM Michael Luckey wrote: > Unfortunately I am a bit worried, that enabling

Re: Add code quality checks to pre-commits.

2019-01-07 Thread Michael Luckey
Unfortunately I am a bit worried, that enabling code coverage will not come for free at the moment. As far as I understand the current state of the build system, we would either have to disable parallel builds or the build cache, which I assume both to be enabled currently on precommit runs. I am

Re: Python 3: final step

2019-01-07 Thread Maximilian Michels
Also curious because I see Python 3 requests quite often. I always say, we're close, but how close are we? :) Thanks, Max On 05.01.19 00:03, Manu Zhang wrote: Guys, Happy New Year !!! I haven't got much time to contribute to Python 3 support. What is the progress now ? It seems there are

Re: Add code quality checks to pre-commits.

2019-01-07 Thread Maximilian Michels
Code coverage would be very useful, regardless of what tool we use to measure it. Especially when reviewing PRs, because it provides a quantitative measure which can help reviewer and contributor to decide if testing is sufficient. I'd prefer to use specific tools instead of one-fits-it-all

Re: Beam Contribution

2019-01-07 Thread Maximilian Michels
Hi David, Welcome! I see you're already added by now. Thanks for introducing yourself. I'm Max, working on the Flink Runner. I wonder if it made sense to have an intro Wiki page with a little bit of information on committers and contributors? I think that would make it easier to get into

Beam Dependency Check Report (2019-01-07)

2019-01-07 Thread Apache Jenkins Server
High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release JIRA Issue future 0.16.0 0.17.1 2016-10-27

Re: Query expressions for schema fields

2019-01-07 Thread Robert Bradshaw
On Sun, Jan 6, 2019 at 12:46 PM Reuven Lax wrote: > > Some time ago, @Jean-Baptiste Onofré made the excellent suggestion that we > look into using JsonPath as a selector format for schema fields. This > provides a simple and natural way for users to select nested schema fields, > as well as

Re: Enforce javadoc comments in public methods?

2019-01-07 Thread Robert Bradshaw
IMHO, requiring comments on trivial methods like setters and getters is often a net negative, but setting some standard could be useful. On Mon, Jan 7, 2019 at 7:35 AM Jean-Baptiste Onofré wrote: > > Hi, > > for the presence of a comment on public method, it's a good idea. Now, > about the