spark 'master' branch (i.e. v1.4.0) builds successfully on windows 8.1 intel i7
64-bit with oracle jdk8_45.with maven opts without the flag
"-XX:ReservedCodeCacheSize=1g".
takes about 33 minutes.
Thanking you.
With Regards
Sree
On Thursday, April 16, 2015 9:07 PM, Arun Lists
wrote:
Here is what I got from the engineer who worked on building Spark and using it
on Windows:
1) Hadoop winutils.exe is needed on Windows, even for local files – and you
have to set the Hadoop.home.dir in the spark-class2.cmd (for the two lines with
$RUNNER near the end, by adding “-Dhadoop.home.dir=” file after
downloading Hadoop binaries + winutils. 2) Java/Spark cannot delete the spark
temporary files and it throws an exception (program still works though).
Manual clean-up works just fine, and it is not a permissions issue as it has
rights to create the file (I have also tried using my own directory rather than
the default, same error).3) tried building Spark again, and have attached the
log – I don’t get any errors, just warnings. However when I try to use that
JAR I just get the error message “Error: Could not find or load main class
org.apache.spark.deploy.SparkSubmit”.
On Thu, Apr 16, 2015 at 12:19 PM, Arun Lists wrote:
Thanks, Matei! We'll try that and let you know if it works. You are correct in
inferring that some of the problems we had were with dependencies.
We also had problems with the spark-submit scripts. I will get the details from
the engineer who worked on the Windows builds and provide them to you.
arun
On Thu, Apr 16, 2015 at 10:44 AM, Matei Zaharia wrote:
You could build Spark with Scala 2.11 on Mac / Linux and transfer it over to
Windows. AFAIK it should build on Windows too, the only problem is that Maven
might take a long time to download dependencies. What errors are you seeing?
Matei
> On Apr 16, 2015, at 9:23 AM, Arun Lists wrote:
>
> We run Spark on Mac and Linux but also need to run it on Windows 8.1 and
> Windows Server. We ran into problems with the Scala 2.10 binary bundle for
> Spark 1.3.0 but managed to get it working. However, on Mac/Linux, we are on
> Scala 2.11.6 (we built Spark from the sources). On Windows, however despite
> our best efforts we cannot get Spark 1.3.0 as built from sources working for
> Scala 2.11.6. Spark has too many moving parts and dependencies!
>
> When can we expect to see a binary bundle for Spark 1.3.0 that is built for
> Scala 2.11.6? I read somewhere that the only reason that Spark 1.3.0 is
> still built for Scala 2.10 is because Kafka is still on Scala 2.10. For those
> of us who don't use Kafka, can we have a Scala 2.10 bundle.
>
> If there isn't an official bundle arriving any time soon, can someone who has
> built it for Windows 8.1 successfully please share with the group?
>
> Thanks,
> arun
>
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
