[
https://issues.apache.org/jira/browse/SPARK-15821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-15821:
------------------------------------
Assignee: (was: Apache Spark)
> Should we use mvn -T for multithreaded Spark builds?
> ----------------------------------------------------
>
> Key: SPARK-15821
> URL: https://issues.apache.org/jira/browse/SPARK-15821
> Project: Spark
> Issue Type: Question
> Components: Build
> Reporter: Adam Roberts
> Priority: Minor
>
> With Maven we can build Spark in a multithreaded way and benefit from
> increased build time performance as a result.
> On a machine with eight cores, I noticed the build time reduced from 20-25
> minutes to five minutes; this is by building with
> mvn -T 1C -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -DskipTests clean
> package
> -T 1C says that we'll use one extra thread for each core available, I've
> never experienced a problem with using this option (ranging from a single
> cored box to one with 192 cores available)
> Should we use this for building Spark quicker or is the Jenkins job
> deliberately set up such that each "executor" is needed for each pull request
> and we wouldn't see an improvement anyway?
> This can be discovered by checking core utilization across the farm and can
> potentially reduce our build times.
> Here's more information on the feature:
> https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
> If this isn't suitable for the current farm then I think we should document
> it for those building Spark from source
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
