Re: SQL shaded jars don't work. How to test?

For the shadowJar issue see https://issues.apache.org/jira/browse/BEAM-4402. There are 280 test failures, but many of them are the same issue. I believe there is significant work on both, but I think there is significant value as well. I suppose there should be a vote on applying these rules to

Re: SQL shaded jars don't work. How to test?

Spotless is slightly off topic and less serious than test coverage problems, but see https://issues.apache.org/jira/browse/BEAM-4394. On Fri, Jun 1, 2018 at 9:30 AM Scott Wegner wrote: > Andrew, it sounds like you're suggesting testShadowJar and enableSpotless > should also be the default for

Re: SQL shaded jars don't work. How to test?

Andrew, it sounds like you're suggesting testShadowJar and enableSpotless should also be the default for all modules? Do you have an idea of how much effort is required for migrating? For failOnWarning / ErrorProne, the migration is pretty straightforward, so I created starter bugs for each module

Re: SQL shaded jars don't work. How to test?

There is now a testShadowJar option on applyJavaNature, which allows you to run your tests against the shaded jar. Everyone should turn it on for their module along with failOnWarning and enableSpotless for maximum testing. On Thu, May 24, 2018 at 1:24 PM Andrew Pilloud wrote: > I've make the

Re: SQL shaded jars don't work. How to test?

I've make the SQL PR able to be merged standalone so I can get back to work. I've also opened issues to track the things I found and dumped my work in progress into a PR. https://issues.apache.org/jira/browse/BEAM-4402 https://issues.apache.org/jira/browse/BEAM-4403

Re: SQL shaded jars don't work. How to test?

If it is taking days of work we should definitely put it behind a flag and do it incrementally, find a way to share the work. If our tests aren't running on the actual artifacts, then we don't really have assurance that they work. That should interest just about everyone. (though it is also

Re: SQL shaded jars don't work. How to test?

I think I'm down to 11 packages with failures, some of which might be precommits that don't run outside of Jenkins. Its not an issue of figuring them out, it is an issue of time spent doing so. On Thu, May 24, 2018 at 9:31 AM Lukasz Cwik wrote: > Were you able to figure out

Re: SQL shaded jars don't work. How to test?

Were you able to figure out how to fix them or still having problems? On Thu, May 24, 2018 at 9:27 AM Andrew Pilloud wrote: > I spent all of yesterday investigating and fixing dependency issues > outside of SQL. I really regret the decision to write a test for this. > Would

Re: SQL shaded jars don't work. How to test?

I spent all of yesterday investigating and fixing dependency issues outside of SQL. I really regret the decision to write a test for this. Would it be acceptable for us to put testing with the output jar behind a flag like we do for failOnWarning? On Wed, May 23, 2018 at 5:21 PM Kenneth Knowles

Re: SQL shaded jars don't work. How to test?

What's the status of moving it forward? Is it a ton of work / too much to do quickly? On Wed, May 23, 2018 at 9:11 AM Andrew Pilloud wrote: > To loop the list in on discussions going on in > https://github.com/apache/beam/pull/5443: our normal tests don't run > against the

Re: SQL shaded jars don't work. How to test?

To loop the list in on discussions going on in https://github.com/apache/beam/pull/5443: our normal tests don't run against the shaded jars. Gradle can run the tests against the shaded jars, but a bunch fail due to dependency issues. It's not just SQL. Andrew On Mon, May 21, 2018 at 11:35 AM

Re: SQL shaded jars don't work. How to test?

Shading requires two pieces of information: 1) Which dependencies should be part of the shaded jar (controlled by includes/excludes) 2) How to relocate code within those dependencies (controlled by relocations) The reason why the exclude(".*") exists is because typically it is an error to produce

Re: SQL shaded jars don't work. How to test?

The issue SQL is seeing is caused by a default dependency of exclude(".*") added in build_rules.gradle. This breaks the normal method of building shadow jars as everything must be explicitly included. SQL explicitly added calcite to the jar, but not calcite's dependencies. I've been told this is

Re: SQL shaded jars don't work. How to test?

Yep, I added the issue as a blocker. https://issues.apache.org/jira/projects/BEAM/issues/BEAM-4357 On Thu, May 17, 2018, 6:05 PM Kenneth Knowles wrote: > This sounds like a release blocker. Can you add it to the list? (Assign > fix version on jira) > > Kenn > > On Thu, May 17,

Re: SQL shaded jars don't work. How to test?

This sounds like a release blocker. Can you add it to the list? (Assign fix version on jira) Kenn On Thu, May 17, 2018, 17:30 Lukasz Cwik wrote: > Typically we have a test block which uses a configuration that has the > shadow/shadowTest configurations on the classpath

Re: SQL shaded jars don't work. How to test?

Typically we have a test block which uses a configuration that has the shadow/shadowTest configurations on the classpath instead of the compile/testCompile configurations. The most common examples are validates runner/integration tests for example:

SQL shaded jars don't work. How to test?

I decided to try our new JDBC support with sqlline and discovered that our SQL shaded jar is completely broken. As in java.lang.NoClassDefFoundError all over the place. How are we testing the output jars from other beam packages? Is there an example I can follow to make our integration tests run