Re: SQL shaded jars don't work. How to test?

Yep, I added the issue as a blocker. https://issues.apache.org/jira/projects/BEAM/issues/BEAM-4357 On Thu, May 17, 2018, 6:05 PM Kenneth Knowles wrote: > This sounds like a release blocker. Can you add it to the list? (Assign > fix version on jira) > > Kenn > > On Thu, May 17,

Re: [DISCUSS] Remove findbugs from sdks/java

Thank you all. I think this is clear. Removing findbugs can happen at a future point. @Scott - I've been working through the java IO error prone issues (some already merged, some with open PRs now) so will take those IO Jiras. I will enable failOnWarning, address dependency issues for findbugs

Re: [PROPOSAL] Preparing 2.5.0 release next week

Hi, The build was OK yesterday but the maven-metadata is still missing. That's the point to fix before being able to move forward on the release. I gonna tackle this later today. Regards JB On 05/18/2018 02:41 AM, Ahmet Altay wrote: > Hi JB and all, > > I wanted to follow up on my

Re: Java PreCommit seems broken

I noticed that tests tests simply run "mvn clean install" on the archetype project. But I don't see any dependent task which installs built artifacts into the local Maven repo. Is that an oversight? If that's the case, perhaps the tests are failing sporadically when there are no previously

Re: I'm back and ready to help grow our community!

Congrats! On Thu, May 17, 2018, 6:44 PM Robert Burke wrote: > Congrats & welcome back! > > On Thu, May 17, 2018, 5:44 PM Huygaa Batsaikhan wrote: > >> Welcome back, Gris! Congratulations! >> >> On Thu, May 17, 2018 at 4:24 PM Robert Bradshaw

Re: Current progress on Portable runners

Hi Eugene, Thanks for putting this together, this is a very nice update and brings much needed visibility to those hoping to make use of the portability features or contribute to them. Since the P1 (MVP) milestone is "wordcount" and some of the next things listed are more contributor oriented,

Re: [DISCUSS] Remove findbugs from sdks/java

Thanks Tim. I think we will be able to remove findbugs after some run/check using ErrorProne and see the gaps. Regards JB Le 18 mai 2018 à 07:49, à 07:49, Tim Robertson a écrit: >Thank you all. > >I think this is clear. Removing findbugs can happen at a future

Build failed in Jenkins: beam_SeedJob #1723

See -- GitHub pull request #5180 of commit c89c4aba92e38f6e7b7adca7ce0165e679b76690, no merge conflicts. Setting status of c89c4aba92e38f6e7b7adca7ce0165e679b76690 to PENDING with url

Build failed in Jenkins: beam_SeedJob_Standalone #1013

See -- Started by timer [EnvInject] - Loading node environment variables. Building remotely on beam2 (beam) in workspace

Build failed in Jenkins: beam_SeedJob #1725

See -- GitHub pull request #5242 of commit eb00d20f65e394b263fed9a9f7958ea58fb782ca, no merge conflicts. Setting status of eb00d20f65e394b263fed9a9f7958ea58fb782ca to PENDING with url

Re: ElasticsearchIOTest failed during gradle build

Hey folks, I am new to gradle, but Boyuan and I had a chat on the slack beam late last night (CEST) on this. Here are my notes I've collected from my build attempts but I haven't yet isolated the problem: - seemingly only happens with -PisRelease - need --info and --stacktrace or else you

Re: Performance Testing Dashboard - which results should be official?

That is correct - I asked for purely organizational purposes. Please keep in mind that there is still some work to do in terms of getting rid of some test flakiness, properly building the test code before running the tests and detecting the anomalies/regressions that happen in IOs. We're working

Build failed in Jenkins: beam_SeedJob #1724

See -- Started by timer [EnvInject] - Loading node environment variables. Building remotely on beam12 (beam) in workspace > git rev-parse

Re: ElasticsearchIOTest failed during gradle build

Hey, Thanks for pointing out ! I'll take a look. Very strange ZipException Etienne Le mercredi 16 mai 2018 à 11:50 -0700, Boyuan Zhang a écrit : > Hey all, > > I'm working on debugging the process of release process and when running > ./gradlew -PisRelease clean build, I got > several tests

Build failed in Jenkins: beam_SeedJob_Standalone #1014

See -- Started by timer [EnvInject] - Loading node environment variables. Building remotely on beam2 (beam) in workspace

Re: com.google.api.services.clouddebugger.v2.CloudDebugger ???

Thanks Cham, forgot to mention about the recent migration to 1.23.0. On Wed, May 16, 2018 at 5:56 PM Chamikara Jayalath wrote: > Are you running using 2.4.0 or HEAD ? We upgraded google-api-client > dependencies of HEAD to 1.23 last month: >

Re: Performance Testing - request for comments

Hi, a small update on this: I improved the command in Perfkit. If you're interested, below you can find the link to the PR. I also noticed that the task used for running integration tests sometimes gets cached (locally, this doesn't happen on Jenkins). The PR to this issue is also below[2].

Re: ElasticsearchIOTest failed during gradle build

Thanks for the tests and the details Tim ! Etienne  Le jeudi 17 mai 2018 à 15:29 +0200, Tim Robertson a écrit : > Hey folks, > > I am new to gradle, but Boyuan and I had a chat on the slack beam late last > night (CEST) on this. > > Here are my notes I've collected from my build attempts but I

Jenkins build is back to normal : beam_SeedJob #1726

See

Re: JDBC support for Beam SQL

I hear some reasonable concerns around locking us into Calcite JDBC. I agree that there are quite a few unknowns around if it would work at all. I didn't think of option three, thanks for the suggestion Anton! I think building JDBC from scratch would be a large, tedious project. We should

Re: I'm back and ready to help grow our community!

Congrats & welcome back! On Thu, May 17, 2018, 5:44 PM Huygaa Batsaikhan wrote: > Welcome back, Gris! Congratulations! > > On Thu, May 17, 2018 at 4:24 PM Robert Bradshaw > wrote: > >> Congratulations, Gris! And welcome back! >> On Thu, May 17, 2018 at

Proposal: keeping precommit times fast

HI, I have a proposal to improve contributor experience by keeping precommit times low. I'm looking to get community consensus and approval about: 1. How long should precommits take. 2 hours @95th percentile over the past 4 weeks is the current proposal. 2. The process for dealing with slowness.

[DISCUSS] Remove findbugs from sdks/java

Hi all, [bringing a side thread discussion from slack to here] We're tackling error-prone warnings now and we aim to fail the build on warnings raised [1]. Enabling failOnWarning also fails the build on findbugs warnings. Currently I see places where these arise from missing a dependency on

Re: Beam high level directions (was "Graal instead of docker?")

All runners just provide translations so it is easy to build features on top of primitives, ie basic translations, instead of requiring runners to use the same lib which is not yet done and will likely not be done when adding new reusable parts - keep in mind beam starts to have runners not hosted

Re: I'm back and ready to help grow our community!

Welcome back and congrats again! On Thu, May 17, 2018 at 2:23 PM Dmitry Demeshchuk wrote: > While this may be a bit off topic, I still want to say this. > > Congratulations on your graduation, Gris! > > On Thu, May 17, 2018 at 2:19 PM, Griselda Cuevas

Re: I'm back and ready to help grow our community!

Congratulations!! On Thu, May 17, 2018 at 2:21 PM Griselda Cuevas wrote: > Hi Everyone, > > > I was absent from the mailing list, slack channel and our Beam community > for the past six weeks, the reason was that I took a leave to focus on > finishing my Masters Degree, which I

Re: I'm back and ready to help grow our community!

Wonderful Gris; warmest congratulations on the milestone and glad to have you back. :D On Thu, May 17, 2018 at 2:36 PM Kenneth Knowles wrote: > Congratulations!! > > On Thu, May 17, 2018 at 2:21 PM Griselda Cuevas wrote: > >> Hi Everyone, >> >> >> I was absent

Java PreCommit seems broken

I'm seeing failures on Maven Archetype-related tests. Build Scan of a sample run: https://scans.gradle.com/s/kr23q43mh6fmk And the failure is here specifically: https://scans.gradle.com/s/kr23q43mh6fmk/console-log?task=:beam-sdks-java-maven-archetypes-examples:generateAndBuildArchetypeTest#L116

Re: I'm back and ready to help grow our community!

While this may be a bit off topic, I still want to say this. Congratulations on your graduation, Gris! On Thu, May 17, 2018 at 2:19 PM, Griselda Cuevas wrote: > Hi Everyone, > > > I was absent from the mailing list, slack channel and our Beam community > for the past six

Re: I'm back and ready to help grow our community!

Congratulations On Thu, May 17, 2018 at 5:19 PM, Griselda Cuevas wrote: > Hi Everyone, > > > I was absent from the mailing list, slack channel and our Beam community > for the past six weeks, the reason was that I took a leave to focus on > finishing my Masters Degree,

Current progress on Portable runners

Hi all, A little over a month ago, a large group of Beam community members has been working a prototype of a portable Flink runner - that is, a runner that can execute Beam pipelines on Flink via the Portability API . The prototype was developed in a separate

Re: I'm back and ready to help grow our community!

Congratulations! Good to see you back! Reuven On Thu, May 17, 2018 at 2:24 PM Griselda Cuevas wrote: > Hi Everyone, > > > I was absent from the mailing list, slack channel and our Beam community > for the past six weeks, the reason was that I took a leave to focus on >

Re: Java code under main depends on junit?

My fault, I'll fix the maven issue. I added this file and it is not in test intentionally. The purpose of this class is similar to TestPipeline, in that other packages which depend on GCP IO can use this class in tests, including integration tests. For example, right now Beam SQL project depends

Re: I'm back and ready to help grow our community!

Congratulations! Welcome back! On Thu, May 17, 2018 at 3:23 PM Reuven Lax wrote: > Congratulations! Good to see you back! > > Reuven > > On Thu, May 17, 2018 at 2:24 PM Griselda Cuevas wrote: > >> Hi Everyone, >> >> >> I was absent from the mailing list,

SQL shaded jars don't work. How to test?

I decided to try our new JDBC support with sqlline and discovered that our SQL shaded jar is completely broken. As in java.lang.NoClassDefFoundError all over the place. How are we testing the output jars from other beam packages? Is there an example I can follow to make our integration tests run

Java code under main depends on junit?

Hi, Is the following dependency intended or an oversight? https://github.com/apache/beam/blob/06c70bdf871c5da8a115011b43f8072916cd79e8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java#L32 It appears that dependent code is in test scope. Should

Re: [DISCUSS] Remove findbugs from sdks/java

+0.02173913 I'm happy to replace FindBugs with ErrorProne, but we need to first upgrade ErrorProne analyzer warnings to errors. Currently the codebase is full of warning spam, and there's no enforcement preventing future violations from being added. I've done the work for enforcing ErrorProne

Re: [DISCUSS] Remove findbugs from sdks/java

+0.7 I think we should work to remove findbugs. Errorprone covers most of the same stuff but better and faster. The one thing I'm not sure about is nullness analysis. Findbugs has some serious limitations there but it really improves code quality and prevents blunders. I'm not sure errorprone

Re: Wait.on() - "Do this, then that" transform

Thanks Kenn, forwarding to user@ is a good idea; just did that. JB - this is orthogonal to SDF, because I'd expect this transform to be primarily used for waiting on the results of SomethingIO.write(), whereas SDF is primarily useful for implementing SomethingIO.read(). On Mon, May 14, 2018 at

Re: Beam high level directions (was "Graal instead of docker?")

If all engines were identical, having a shared optimizer would be useful. Having a proxy runner that performance optimizations before submission to an actual engine-specific runner has downsides in both directions: - obscures the ability of engine-specific runners to optimize the Beam primitives

Jenkins build is back to normal : beam_SeedJob_Standalone #1015

See

Re: Wait.on() - "Do this, then that" transform

This sounds super interesting and useful ! Eugene can you please elaborate on this phrase 'has to return a result that can be waited on'. It is not clear for me what this means and I would like to understand this to evaluate what other IOs could potentially support this. On Thu, May 17, 2018 at

Re: [DISCUSS] Remove findbugs from sdks/java

+0.7 also. Findbugs support for more recent versions of Java is lacking and the maintenance seems frozen in time. As a possible plan b can we identify the missing important validations to identify how much we lose and if it is considerable, maybe we can create a minimal configuration for those,

Re: Beam high level directions (was "Graal instead of docker?")

The approach you're looking for sounds like the user's Runner of Choice, would use a user side version of the runner core, without changing the Runner of Choice? So a user would update their version of the SDK, and the runner would have to pull the core component from the user pipeline? That

Re: Wait.on() - "Do this, then that" transform

I mean it has to return a PCollection of something, that contains elements representing the result of completing processing of the respective window. E.g. FileIO.write() returns a PCollection of filenames; SpannerIO.write() returns simply a PCollection of Void. However, connectors such as

Build failed in Jenkins: beam_SeedJob #1722

See Changes: [kmj] Bugfix: Read BQ bytes processed from correct field. [apilloud] [BEAM-3983] Add utils for converting to BigQuery types [apilloud] [BEAM-3983][SQL] Add BigQuery table provider [apilloud]

Re: Java code under main depends on junit?

This seems correct. Test jars are for tests. Utilities to be used for tests need to be in main jars. (If for no other reason, this is how transitive deps work) We've considered putting these things in a separate package (still in main). Just no one has done it. Kenn On Thu, May 17, 2018, 16:04

Re: Java code under main depends on junit?

Opened PR to fix the current build issue, opened BEAM-4358 to extract test dependencies. Should we keep maven precommits running for now if we have to fix the issues like these? In the PR I had to fix

Re: Java code under main depends on junit?

Thanks! IMO we should at least run "mvn verify -DskipTests" in precommit until the maven build can be retired (== deleted from master). On Thu, May 17, 2018 at 5:00 PM, Anton Kedin wrote: > Opened PR to fix the current > build

Re: SQL shaded jars don't work. How to test?

Typically we have a test block which uses a configuration that has the shadow/shadowTest configurations on the classpath instead of the compile/testCompile configurations. The most common examples are validates runner/integration tests for example:

Re: [PROPOSAL] Preparing 2.5.0 release next week

Hi JB and all, I wanted to follow up on my previous email. The python streaming issue I mentioned is resolved and removed from the blocker list. Blocker list is empty now. You can go ahead with the release branch cut when you are ready. Thank you, Ahmet On Sun, May 13, 2018 at 8:43 AM,

Re: [PROPOSAL] Preparing 2.5.0 release next week

In case you didn't see the other thread, Andrew just discovered a problem in SQL's jar build. It may be a release blocker. Just an FYI. Since the fix is likely small fixes to build file it seems ok to cut the branch and cherry pick. Kenn On Thu, May 17, 2018, 17:41 Ahmet Altay

Re: I'm back and ready to help grow our community!

Congratulations, Gris! And welcome back! On Thu, May 17, 2018 at 3:30 PM Robin Qiu wrote: > Congratulations! Welcome back! > On Thu, May 17, 2018 at 3:23 PM Reuven Lax wrote: >> Congratulations! Good to see you back! >> Reuven >> On Thu, May 17, 2018 at

Re: Java code under main depends on junit?

It is possible to depend on a test artifact to achieve the same, but unfortunately not transitively. Mixing test utilities into the main artifacts seems undesirable, since they are only needed for tests. It may give more food to the shading monster also.. So it is probably better to create a

Re: Java code under main depends on junit?

I know what you mean. But indeed, test artifacts are unsuitable to depend on since transitive deps don't work correctly. I think it makes sense to have a separate test utility. For the core, one reason we didn't was to have PAssert available in main. But now that we have Gradle we actually can do

Re: I'm back and ready to help grow our community!

Welcome back, Gris! Congratulations! On Thu, May 17, 2018 at 4:24 PM Robert Bradshaw wrote: > Congratulations, Gris! And welcome back! > On Thu, May 17, 2018 at 3:30 PM Robin Qiu wrote: > > > Congratulations! Welcome back! > > > On Thu, May 17, 2018 at

Re: SQL shaded jars don't work. How to test?

This sounds like a release blocker. Can you add it to the list? (Assign fix version on jira) Kenn On Thu, May 17, 2018, 17:30 Lukasz Cwik wrote: > Typically we have a test block which uses a configuration that has the > shadow/shadowTest configurations on the classpath

Re: [PROPOSAL] Preparing 2.5.0 release next week

On Thu, May 17, 2018 at 6:08 PM, Kenneth Knowles wrote: > In case you didn't see the other thread, Andrew just discovered a problem > in SQL's jar build. It may be a release blocker. > I missed Andrew's email. I only looked at the release blocking list. If it might be a release