[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15553 Closing PR given pushback to commit'ing it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15553 Thanks for the comments. Specifically for our usecase, this is for automated full builds, where there is a container teardown after each (so that builds dont pollute each other : new $HOME/.m2, full recompilation each time, etc). In that context, current build takes 25+ minutes without test execution and about 10+ mins with skipping test compilation and doc generation. mvn package, install, later phases will always result in test compilation as well - compile by itself is useful for spark developers, not users/release engineers imo. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15553 `mvn compile` takes about 5 minutes on my laptop, and `mvn compile test-compile` takes about 9 minutes. I don't use either of these during development, because incremental compiles are tens of seconds. Even 9 minutes just doesn't seem like a big deal for period builds; the test suite takes 3 hours. But... maybe I miss something from the beginning, but, why not just run `mvn compile`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15553 Since I don't have much experience with the build system, I will defer to your opinion about it. Having said that, I want to know what can be done to add support for avoiding test compilation - the skip option you mentioned for docs is already helping us. For regular automated builds from tagged releases, test compilation in spark doubles the compilation time - same applies for end user (not spark dev) compilation. Anything to reduce that time would be great. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15553 I do not support this change for reasons above. The default is certainly to not merge if there is no additional support. Please do not make this change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15553 I will leave it open for a couple of days more to solicit suggestions /comments : if there is a way to reduce build complexity of the change please do let me know : would be great if there was a way to handle it better. If none, will merge it in after. Thx --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15553 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67436/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15553 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15553 **[Test build #67436 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67436/consoleFull)** for PR 15553 at commit [`ec49e8b`](https://github.com/apache/spark/commit/ec49e8bfa3d4a43106b418afb14aa7a5d2c3aff4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15553 **[Test build #67436 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67436/consoleFull)** for PR 15553 at commit [`ec49e8b`](https://github.com/apache/spark/commit/ec49e8bfa3d4a43106b418afb14aa7a5d2c3aff4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15553 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15553 @srowen This is explicitly supports maven.test.skip=true - the rationale for using it can be project specific (generating clean builds for various profile combinations, internal release builds from an apache tagged released version, etc for example). We are seeing close to halving the build time for clean builds (use of zinc, batched build, etc is already enabled). Regarding complexity of the build - if there are any suggestions on how to reduce that, please do let me know. The pr primarily reorganizes all test scope dependencies into a profile of its own which is activated by default unless maven.test.skip=true; which seems to be the only recommended way to do it from maven docs/discussions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15553 For development time, use an IDE or maybe SBT to do incremental compilation, which is even faster. I don't otherwise see a strong use case for not compiling tests. Yes it's slower to compile tests, but not slow compared to running them, and the downside is missing compile errors and the extra complexity of an already creaking build. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15553 @srowen -Dmaven.test.skip=true is the documented way in maven to skip test compilation (http://maven.apache.org/surefire/maven-surefire-plugin/examples/skipping-test.html) which we are not honoring in spark build. The more practical reason for supporting it is that the build time almost doubles with test and doc compilation included even if we dont care for them to be generated. Unfortunately, use of profile is the only way to resolve this problem - atleast what I could find in maven documentation/discussions : if there are alternatives, please do let me know. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15553 I think we should fix the jetty thing separately. I don't think I would support this profile to disable test compilation as it's a sort of cheap form of testing and don't think it's expensive. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15553 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15553 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67345/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15553 **[Test build #67345 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67345/consoleFull)** for PR 15553 at commit [`ec49e8b`](https://github.com/apache/spark/commit/ec49e8bfa3d4a43106b418afb14aa7a5d2c3aff4). * This patch **fails from timeout after a configured wait of \`250m\`**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15553 **[Test build #67345 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67345/consoleFull)** for PR 15553 at commit [`ec49e8b`](https://github.com/apache/spark/commit/ec49e8bfa3d4a43106b418afb14aa7a5d2c3aff4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15553 Changes done, please do let me know of any other comments, thanks ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15553 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15553 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67289/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15553 **[Test build #67289 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67289/consoleFull)** for PR 15553 at commit [`305b68c`](https://github.com/apache/spark/commit/305b68c93b56f9bc08c158f6bea3889f94a02a56). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15553 **[Test build #67289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67289/consoleFull)** for PR 15553 at commit [`305b68c`](https://github.com/apache/spark/commit/305b68c93b56f9bc08c158f6bea3889f94a02a56). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15553 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67196/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15553 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15553 **[Test build #67196 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67196/consoleFull)** for PR 15553 at commit [`16056f7`](https://github.com/apache/spark/commit/16056f7b46f556464bc4643d2b94e35b4b691078). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15553 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67192/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15553 **[Test build #67192 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67192/consoleFull)** for PR 15553 at commit [`e7d6490`](https://github.com/apache/spark/commit/e7d6490ce31b8b3b8f0050f8f7d46ac7e514c38e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15553 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15553 **[Test build #67196 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67196/consoleFull)** for PR 15553 at commit [`16056f7`](https://github.com/apache/spark/commit/16056f7b46f556464bc4643d2b94e35b4b691078). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15553 Please note that this applies only the spark artifacts - not any of the others (which, as you mentioned, will be a simple disk lookup). Since I was moving it for spark generated artifacts anyway, I moved the other test dependencies also for consistency sake - I can revert that part if it makes the change too cumbersome. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15553 Oh I see your point. Yeah, that would be a complication for a change like this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15553
Thanks for the link to 'skip' - will test it out !
About test compilation :
For example, in common/network-shuffle/pom.xml, we have
```
org.apache.spark
spark-network-common_${scala.binary.version}
${project.version}
test-jar
test
```
Since we are skipping compilation of test classes via maven's
maven.test.skip=true , the testjar is never build - which causes the subsequent
resolution in network-shuffle to fail since, the resolution of artifacts is not
related to these flags (so even if test compile is disabled, the artifacts will
still be resolved).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15553 Does resolution of the artifacts take much time? there aren't many, and they'd always resolve instantly off disk after an initial download. As you note you can selectively skip tests, or not compile tests, if you like, with existing Maven flags. I think suppressing test compilation isn't something that one would do normally, and wouldn't be done for Spark builds, so I am not sure it's worth putting in the Spark build. You can put it in your build of Spark if desired. I'm not sure it's worth the extra complexity. If you mean that you're looking for a flag that directly controls the scala-maven-plugin doc-jar mojo, it looks like it has its own skip flag: http://davidb.github.io/scala-maven-plugin/doc-jar-mojo.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15553 @srowen You are right; for normal build, this should be a no-op. Currently, there is no way to suppress compilation of tests (we can suppress running test via -DskipTests but it will still compile), same for generation of documentation. Both of these together take a non trivial amount of time for the build (particularly when not on ssd's). In maven, -Dmaven.test.skip=true is to be used to suppress building of test classes. Unfortunately, this does not apply to dependency resolution and in our builds, we reference test artifacts with 'test' scope. Since test compilation is suppressed via the flag, the corresponding dependencies will be missing causing build failures. The only way around this in maven is to move all relevant test dependencies into a profile which is triggered by appropriate condition (-Dmaven.test.skip != true). For uniformity, I moved all dependencies and not just spark test dependencies. -Dmaven.javadoc.skip=true is used in maven to suppress javadoc generation, but our documentation generation is directly associated with the verify phase and bypasses maven.javadoc.skip support from maven. Note that the profiles docBuild and testCompilation are not expected to be directly set via -P, but inferred via the corresponding maven flags. If there is a better way to achieve suppressing test class compilation and documentation generation, please do let me know. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15553 Why, what problem does the dependency change solve? test scope doesn't affect anything but tests to begin with. Likewise the scaladoc plugin attaches to a phase that's only active when javadoc is being built anyway, right? that seems like a no-op. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15553 **[Test build #67192 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67192/consoleFull)** for PR 15553 at commit [`e7d6490`](https://github.com/apache/spark/commit/e7d6490ce31b8b3b8f0050f8f7d46ac7e514c38e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15553: [SPARK-18008] [build] Add support for -Dmaven.test.skip=...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15553 +CC @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
