Re: Silly question about building Spark 1.4.1
Thanks Dean… I was building based on the information found on the Spark 1.4.1 documentation. So I have to ask the following: Shouldn’t the examples be updated to reflect Hadoop 2.6 or are the vendors’ distro not up to 2.6 and that’s why its still showing 2.4? Also I’m trying to build with support for Scala 2.11 Are there any known issues between Scala 2.11 and Hive and hive-thrift server? Dean, the reason I asked about needed to specify the Hive and Hive-Thriftserver options is that at the end of the build I see the following: “ [INFO] Spark Project SQL .. SUCCESS [02:06 min] [INFO] Spark Project ML Library ... SUCCESS [02:23 min] [INFO] Spark Project Tools SUCCESS [ 13.305 s] [INFO] Spark Project Hive . SUCCESS [01:55 min] [INFO] Spark Project REPL . SUCCESS [ 40.488 s] [INFO] Spark Project YARN . SUCCESS [ 38.793 s] [INFO] Spark Project Assembly . SUCCESS [01:10 min] [INFO] Spark Project External Twitter . SUCCESS [ 14.907 s] [INFO] Spark Project External Flume Sink .. SUCCESS [ 21.748 s] [INFO] Spark Project External Flume ... SUCCESS [ 31.754 s] [INFO] Spark Project External MQTT SUCCESS [ 17.921 s] [INFO] Spark Project External ZeroMQ .. SUCCESS [ 18.037 s] [INFO] Spark Project External Kafka ... SUCCESS [ 41.941 s] [INFO] Spark Project Examples . SUCCESS [01:56 min] [INFO] Spark Project External Kafka Assembly .. SUCCESS [ 24.806 s] [INFO] Spark Project YARN Shuffle Service . SUCCESS [ 5.204 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 22:40 min [INFO] Finished at: 2015-07-20T12:54:23-07:00 [INFO] Final Memory: 109M/2332M [INFO] “ Granted this may be something completely different which is why the next time I do a build, I’m going to capture the stderr/stdout to a file. Thx for the quick response. > On Jul 20, 2015, at 1:11 PM, Ted Yu wrote: > > In master (as well as 1.4.1) I don't see hive profile in pom.xml > > I do find hive-provided profile, though. > > FYI > > On Mon, Jul 20, 2015 at 1:05 PM, Dean Wampler <mailto:deanwamp...@gmail.com>> wrote: > hadoop-2.6 is supported (look for "profile" XML in the pom.xml file). > > For Hive, add "-Phive -Phive-thriftserver" (See > http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables > <http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables>) > for more details. > > dean > > Dean Wampler, Ph.D. > Author: Programming Scala, 2nd Edition > <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) > Typesafe <http://typesafe.com/> > @deanwampler <http://twitter.com/deanwampler> > http://polyglotprogramming.com <http://polyglotprogramming.com/> > > On Mon, Jul 20, 2015 at 2:55 PM, Michael Segel <mailto:msegel_had...@hotmail.com>> wrote: > Sorry, > > Should have sent this to user… > > However… it looks like the docs page may need some editing? > > Thx > > -Mike > > >> Begin forwarded message: >> >> From: Michael Segel > <mailto:msegel_had...@hotmail.com>> >> Subject: Silly question about building Spark 1.4.1 >> Date: July 20, 2015 at 12:26:40 PM MST >> To: d...@spark.apache.org <mailto:d...@spark.apache.org> >> >> Hi, >> >> I’m looking at the online docs for building spark 1.4.1 … >> >> http://spark.apache.org/docs/latest/building-spark.html >> <http://spark.apache.org/docs/latest/building-spark.html> >> >> I was interested in building spark for Scala 2.11 (latest scala) and also >> for Hive and JDBC support. >> >> The docs say: >> “ >> To produce a Spark package compiled with Scala 2.11, use the -Dscala-2.11 >> property: >> dev/change-version-to-2.11.sh <http://change-version-to-2.11.sh/> >> mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package >> “ >> So… >> Is there a reason I shouldn’t build against hadoop-2.6 ? >> >> If I want to add the Thirft and Hive support, is it possible? >> Looking at the Scala build, it looks like hive support is being built? >> (Looking at the stdout messages…) >> Should the docs be updated? Am I missing something? >> (Dean W. can confirm, I am completely brain dead. ;-) >> >> Thx >> >> -Mike >> PS. Yes I can probably download a prebuilt image, but I’m a glutton for >> punishment. ;-) >> > > >
Re: Silly question about building Spark 1.4.1
In master (as well as 1.4.1) I don't see hive profile in pom.xml I do find hive-provided profile, though. FYI On Mon, Jul 20, 2015 at 1:05 PM, Dean Wampler wrote: > hadoop-2.6 is supported (look for "profile" XML in the pom.xml file). > > For Hive, add "-Phive -Phive-thriftserver" (See > http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables) > for more details. > > dean > > Dean Wampler, Ph.D. > Author: Programming Scala, 2nd Edition > <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) > Typesafe <http://typesafe.com> > @deanwampler <http://twitter.com/deanwampler> > http://polyglotprogramming.com > > On Mon, Jul 20, 2015 at 2:55 PM, Michael Segel > wrote: > >> Sorry, >> >> Should have sent this to user… >> >> However… it looks like the docs page may need some editing? >> >> Thx >> >> -Mike >> >> >> Begin forwarded message: >> >> *From: *Michael Segel >> *Subject: **Silly question about building Spark 1.4.1* >> *Date: *July 20, 2015 at 12:26:40 PM MST >> *To: *d...@spark.apache.org >> >> Hi, >> >> I’m looking at the online docs for building spark 1.4.1 … >> >> http://spark.apache.org/docs/latest/building-spark.html >> >> I was interested in building spark for Scala 2.11 (latest scala) and also >> for Hive and JDBC support. >> >> The docs say: >> “ >> To produce a Spark package compiled with Scala 2.11, use the -Dscala-2.11 >> property: >> >> dev/change-version-to-2.11.sh >> mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package >> >> “ >> >> So… >> >> Is there a reason I shouldn’t build against hadoop-2.6 ? >> >> >> If I want to add the Thirft and Hive support, is it possible? >> >> Looking at the Scala build, it looks like hive support is being built? >> >> (Looking at the stdout messages…) >> >> Should the docs be updated? Am I missing something? >> >> (Dean W. can confirm, I am completely brain dead. ;-) >> >> >> Thx >> >> >> -Mike >> >> PS. Yes I can probably download a prebuilt image, but I’m a glutton for >> punishment. ;-) >> >> >> >> >
Re: Silly question about building Spark 1.4.1
hadoop-2.6 is supported (look for "profile" XML in the pom.xml file). For Hive, add "-Phive -Phive-thriftserver" (See http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables) for more details. dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typesafe <http://typesafe.com> @deanwampler <http://twitter.com/deanwampler> http://polyglotprogramming.com On Mon, Jul 20, 2015 at 2:55 PM, Michael Segel wrote: > Sorry, > > Should have sent this to user… > > However… it looks like the docs page may need some editing? > > Thx > > -Mike > > > Begin forwarded message: > > *From: *Michael Segel > *Subject: **Silly question about building Spark 1.4.1* > *Date: *July 20, 2015 at 12:26:40 PM MST > *To: *d...@spark.apache.org > > Hi, > > I’m looking at the online docs for building spark 1.4.1 … > > http://spark.apache.org/docs/latest/building-spark.html > > I was interested in building spark for Scala 2.11 (latest scala) and also > for Hive and JDBC support. > > The docs say: > “ > To produce a Spark package compiled with Scala 2.11, use the -Dscala-2.11 > property: > > dev/change-version-to-2.11.sh > mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package > > “ > > So… > > Is there a reason I shouldn’t build against hadoop-2.6 ? > > > If I want to add the Thirft and Hive support, is it possible? > > Looking at the Scala build, it looks like hive support is being built? > > (Looking at the stdout messages…) > > Should the docs be updated? Am I missing something? > > (Dean W. can confirm, I am completely brain dead. ;-) > > > Thx > > > -Mike > > PS. Yes I can probably download a prebuilt image, but I’m a glutton for > punishment. ;-) > > > >
Fwd: Silly question about building Spark 1.4.1
Sorry, Should have sent this to user… However… it looks like the docs page may need some editing? Thx -Mike > Begin forwarded message: > > From: Michael Segel > Subject: Silly question about building Spark 1.4.1 > Date: July 20, 2015 at 12:26:40 PM MST > To: d...@spark.apache.org > > Hi, > > I’m looking at the online docs for building spark 1.4.1 … > > http://spark.apache.org/docs/latest/building-spark.html > <http://spark.apache.org/docs/latest/building-spark.html> > > I was interested in building spark for Scala 2.11 (latest scala) and also for > Hive and JDBC support. > > The docs say: > “ > To produce a Spark package compiled with Scala 2.11, use the -Dscala-2.11 > property: > dev/change-version-to-2.11.sh > mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package > “ > So… > Is there a reason I shouldn’t build against hadoop-2.6 ? > > If I want to add the Thirft and Hive support, is it possible? > Looking at the Scala build, it looks like hive support is being built? > (Looking at the stdout messages…) > Should the docs be updated? Am I missing something? > (Dean W. can confirm, I am completely brain dead. ;-) > > Thx > > -Mike > PS. Yes I can probably download a prebuilt image, but I’m a glutton for > punishment. ;-) >
