GitHub user sarahgerweck opened a pull request:
https://github.com/apache/spark/pull/1752
Optionally parallelize the Spark build.
This introduces a new environment variable, `SPARK_BUILD_THREADS`,
that controls the level of Maven parallelization (if set). This uses
Maven 3's syntax for the number of threads: either a simple integer or
something like `1.5C` to indicate 1.5 times the number of cores on the
build server.
On my hardware, (Intel Xeon E5-1620 v2, roughly equivalent to a fast
i7), a setting of `1.5C` speeds up the build by about 15%.
This will trigger some warnings that the Scala plugins are not marked as
threadsafe. Several repeated builds show no differences in the compiled
distribution (save timestamps), but this isn't proof that it can never
fail.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/AtScaleInc/spark parBuild
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1752.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1752
----
commit 3cac93065df7c5620c1f4024193c1081a418cdbb
Author: Sarah Gerweck <[email protected]>
Date: 2014-08-03T09:40:59Z
Optionally parallelize the Spark build.
This introduces a new environment variable, `SPARK_BUILD_THREADS`,
that controls the level of Maven parallelization (if set). This uses
Maven 3's syntax for the number of threads: either a simple integer or
something like `1.5C` to indicate 1.5 times the number of cores on the
build server.
On my hardware, (Intel Xeon E5-1620 v2, roughly equivalent to a fast
i7), a setting of `1.5C` speeds up the build by about 15%.
This will trigger some warnings that the Scala plugins are not marked as
threadsafe. Several repeated builds show no differences in the compiled
distribution (save timestamps), but this isn't proof that it can never
fail.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
