Re: workspace cleanups needed on jenkins master

Alan, Mikhail, feel free to merge https://github.com/apache/beam/pull/7410 when ready. On Thu, Jan 3, 2019 at 5:39 PM Mikhail Gryzykhin wrote: > It is not required for https://s.apache.org/beam-community-metrics . > > I believe that's main dash we have atm. > > @Alan Myrvold Can you confirm? >

Schemas in the Go SDK

At this point I feel like the schema discussion should be a separate thread from having a Coder Registry in Go, which was the original topic, so I'm forking it. It does sounds like adding Schemas to the Go SDK would be a much larger extension than the registry. I'm not convinced that not having

Re: workspace cleanups needed on jenkins master

It is not required for https://s.apache.org/beam-community-metrics . I believe that's main dash we have atm. @Alan Myrvold Can you confirm? --Mikhail Have feedback ? On Thu, Jan 3, 2019 at 1:59 PM Udi Meiri wrote: > > > On Thu, Dec 27, 2018 at 11:02 AM Ismaël

Re: Add code quality checks to pre-commits.

I think we should also consider false positive ratio of the tool. Oftentimes deeper analysis easily produces tons of false positives which make people less interested in static analysis results because of triaging overheads. On Thu, Jan 3, 2019 at 4:18 PM Scott Wegner wrote: > Discussion on

Re: [Go SDK] User Defined Coders

On Fri, Jan 4, 2019 at 1:19 AM Robert Burke wrote: > Very interesting Reuven! > > That would be a huge readability improvement, but it would also be a > significant investment over my time budget to implement them on the Go side > correctly. I would certainly want to read your documentation

Re: [Go SDK] User Defined Coders

Very interesting Reuven! That would be a huge readability improvement, but it would also be a significant investment over my time budget to implement them on the Go side correctly. I would certainly want to read your documentation before going ahead. Will the Portability FnAPI have dedicated

Re: Add code quality checks to pre-commits.

Discussion on software engineering practices and tools tends to gather many opinions :) I suggest breaking this out into a doc to keep the discussion organized. I appreciate that you've started with a list of requirements. I would add a few: 6. Analysis results should be integrated into the code

Re: [Go SDK] User Defined Coders

I looked at Apache Arrow as a potential serialization format for Row coders. At the time it didn't seem a perfect fit - Beam's programming model is record-at-a-time, and Arrow is optimized for large batches of records (while Beam has a concept of "bundles" they are completely non deterministic,

Re: Add code quality checks to pre-commits.

Let me summarize and answer main question that I see: 1. Seems that we do want to have some statistics on coverage and integrate automatic requirements into our build system. 2. Implementation is still to be discussed. Lets talk about implementation further. My requirements for choice are: 1.

Re: [Go SDK] User Defined Coders

Reuven, it sounds great. I see there is a similar thing to Row coders happening in Apache Arrow , and there is a similarity between Apache Arrow Flight and data exchange service in

Re: [Go SDK] User Defined Coders

The biggest advantage is actually readability and usability. A secondary advantage is that it means that Go will be able to interact seamlessly with BeamSQL, which would be a big win for Go. A schema is basically a way of saying that a record has a specific set of (possibly nested, possibly

Re: Add code quality checks to pre-commits.

It would be very useful to have line and/or branch coverage visible. These are both very weak proxies for quality or reliability, so IMO strict thresholds are not helpful. One thing that is super useful is to integrate line coverage into code review, like this:

Re: workspace cleanups needed on jenkins master

On Thu, Dec 27, 2018 at 11:02 AM Ismaël Mejía wrote: > Bringing this subject for awareness to dev@ > We are sadly part of this top. > Does somebody know what this data is? And if we can clean it periodically? > Can somebody with more sysadmin super powers take a look and act on this. > >

Re: Add code quality checks to pre-commits.

I don't have any experience of using SonarQube but Coverity worked well for me. Looks like it already has beam repo: https://scan.coverity.com/projects/11881 On Thu, Jan 3, 2019 at 1:27 PM Reuven Lax wrote: > checkstyle and findbugs are already run as precommit checks, are they not? > > On Thu,

Re: Why does Beam not use the google-api-client libraries?

Cham is absolutely correct. The google-cloud-pubsub higher-level library didn't exist when the Beam connector were written, and nobody has gotten around to rewriting that connector. On Wed, Jan 2, 2019 at 11:29 PM Chamikara Jayalath wrote: > Thanks Jeff for the interest in this. > > I think

Re: [Go SDK] User Defined Coders

I'll make a different suggestion. There's been some chatter that schemas are a better tool than coders, and that in Beam 3.0 we should make schemas the basic semantics instead of coders. Schemas provide everything a coder provides, but also allows for far more readable code. We can't make such a

Re: Add code quality checks to pre-commits.

checkstyle and findbugs are already run as precommit checks, are they not? On Thu, Jan 3, 2019 at 7:19 PM Mikhail Gryzykhin wrote: > Hi everyone, > > In our current builds we (can) run multiple code quality checks tools like > checkstyle, findbugs, code test coverage via cubertura. However we

Re: Add code quality checks to pre-commits.

+1 to enabling warnings first. This will allow us to evaluate how much value we get and the frequency of false-positives / noise. I'd be curious to hear from those that've worked in other projects on experiences with SonarQube or other tools. At Google, code coverage is integrated into the code

Re: Beam Summits!

Hi Matthias, etc, Trying to get thoughts on formalizing a process for getting proposals together. I look forward to the potential day that there are many people that want (rather than just willing) to host a summit in a given region in a given year. Perhaps too forward looking. Also, you

[Go SDK] User Defined Coders

One area that the Go SDK currently lacks: is the ability for users to specify their own coders for types. I've written a proposal document, and while I'm confident about the core, there are certainly some edge

Re: Add code quality checks to pre-commits.

I had the same question, and tt supports many more than we do: https://www.sonarqube.org/features/multi-languages/ All the various rules checks have clear explanations and justifications for why they're doing what they do. It would be quite handy as part of the precommits I think, if at least as

Re: Add code quality checks to pre-commits.

+1 for adding more code quality signals. Could we add them in an advisory-only mode at first? (a warning and not an error) I'm curious how the "technical debt" metric is determined. I'm not familiar with SonarQube. What languages does it support? On Thu, Jan 3, 2019 at 10:19 AM Mikhail

Add code quality checks to pre-commits.

Hi everyone, In our current builds we (can) run multiple code quality checks tools like checkstyle, findbugs, code test coverage via cubertura. However we do not utilize many of those signals. I suggest to add requirements to code based on those tools. Specifically, I suggest to add pre-commit

Re: [PROPOSAL] Prepare Beam 2.10.0 release

Thanks for driving this Kenn! I'm in favor of a strict cut off, but I'd like to propose a week for cherry-picking relevant changes to the release branch. It looks like many people are returning from holidays or are still off. Cheers, Max On 02.01.19 17:20, Kenneth Knowles wrote: Done. I've

Re: Beam snapshots broken

Thank you for fixing this, Scott. I also can confirm that it works for me now. > On 2 Jan 2019, at 22:36, Ismaël Mejía wrote: > > Thanks Scott, I just tested and the errors I found are fixed and > everything seems to work now. Snapshots are being correctly updated > too. > > On Wed, Jan 2,