Thanks Dean…
I was building based on the information found on the Spark 1.4.1 documentation.
So I have to ask the following:
Shouldn’t the examples be updated to reflect Hadoop 2.6 or are the vendors’
distro not up to 2.6 and that’s why its still showing 2.4?
Also I’m trying to build with support for Scala 2.11
Are there any known issues between Scala 2.11 and Hive and hive-thrift server?
Dean, the reason I asked about needed to specify the Hive and Hive-Thriftserver
options is that at the end of the build I see the following:
“
[INFO] Spark Project SQL .. SUCCESS [02:06 min]
[INFO] Spark Project ML Library ... SUCCESS [02:23 min]
[INFO] Spark Project Tools SUCCESS [ 13.305 s]
[INFO] Spark Project Hive . SUCCESS [01:55 min]
[INFO] Spark Project REPL . SUCCESS [ 40.488 s]
[INFO] Spark Project YARN . SUCCESS [ 38.793 s]
[INFO] Spark Project Assembly . SUCCESS [01:10 min]
[INFO] Spark Project External Twitter . SUCCESS [ 14.907 s]
[INFO] Spark Project External Flume Sink .. SUCCESS [ 21.748 s]
[INFO] Spark Project External Flume ... SUCCESS [ 31.754 s]
[INFO] Spark Project External MQTT SUCCESS [ 17.921 s]
[INFO] Spark Project External ZeroMQ .. SUCCESS [ 18.037 s]
[INFO] Spark Project External Kafka ... SUCCESS [ 41.941 s]
[INFO] Spark Project Examples . SUCCESS [01:56 min]
[INFO] Spark Project External Kafka Assembly .. SUCCESS [ 24.806 s]
[INFO] Spark Project YARN Shuffle Service . SUCCESS [ 5.204 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 22:40 min
[INFO] Finished at: 2015-07-20T12:54:23-07:00
[INFO] Final Memory: 109M/2332M
[INFO]
“
Granted this may be something completely different which is why the next time I
do a build, I’m going to capture the stderr/stdout to a file.
Thx for the quick response.
On Jul 20, 2015, at 1:11 PM, Ted Yu yuzhih...@gmail.com wrote:
In master (as well as 1.4.1) I don't see hive profile in pom.xml
I do find hive-provided profile, though.
FYI
On Mon, Jul 20, 2015 at 1:05 PM, Dean Wampler deanwamp...@gmail.com
mailto:deanwamp...@gmail.com wrote:
hadoop-2.6 is supported (look for profile XML in the pom.xml file).
For Hive, add -Phive -Phive-thriftserver (See
http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables
http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables)
for more details.
dean
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
http://shop.oreilly.com/product/0636920033073.do (O'Reilly)
Typesafe http://typesafe.com/
@deanwampler http://twitter.com/deanwampler
http://polyglotprogramming.com http://polyglotprogramming.com/
On Mon, Jul 20, 2015 at 2:55 PM, Michael Segel msegel_had...@hotmail.com
mailto:msegel_had...@hotmail.com wrote:
Sorry,
Should have sent this to user…
However… it looks like the docs page may need some editing?
Thx
-Mike
Begin forwarded message:
From: Michael Segel msegel_had...@hotmail.com
mailto:msegel_had...@hotmail.com
Subject: Silly question about building Spark 1.4.1
Date: July 20, 2015 at 12:26:40 PM MST
To: d...@spark.apache.org mailto:d...@spark.apache.org
Hi,
I’m looking at the online docs for building spark 1.4.1 …
http://spark.apache.org/docs/latest/building-spark.html
http://spark.apache.org/docs/latest/building-spark.html
I was interested in building spark for Scala 2.11 (latest scala) and also
for Hive and JDBC support.
The docs say:
“
To produce a Spark package compiled with Scala 2.11, use the -Dscala-2.11
property:
dev/change-version-to-2.11.sh http://change-version-to-2.11.sh/
mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
“
So…
Is there a reason I shouldn’t build against hadoop-2.6 ?
If I want to add the Thirft and Hive support, is it possible?
Looking at the Scala build, it looks like hive support is being built?
(Looking at the stdout messages…)
Should the docs be updated? Am I missing something?
(Dean W. can confirm, I am completely brain dead. ;-)
Thx
-Mike
PS. Yes I can probably download a prebuilt image, but I’m a glutton for
punishment. ;-)
