Re: Re: RE: Error not found value sqlContext
Thanks for all the support. It was a code issue which I overlooked it Regards, Satish Chandra On Mon, Nov 23, 2015 at 3:49 PM, satish chandra j wrote: > Sorry, just to understand my issue.if Eclipse could not understand > Scala syntax properly than it should error for the other Spark Job which is > fetching data from a RDBM and printing the output in console. I think there > is some dependencies which are missing due to which "import > sqlContext.implicits._" is not recognized during compile time > > Please let me know if any further inputs needed to fix the same > > Regards, > Satish Chandra > > On Mon, Nov 23, 2015 at 3:29 PM, prosp4300 wrote: > >> >> >> So it is actually a compile time error in Eclipse, instead of jar >> generation from Eclipse, you can try to use sbt to assembly your jar, looks >> like your Eclipse does not recognize the Scala syntax properly. >> >> >> >> At 2015-11-20 21:36:55, "satish chandra j" >> wrote: >> >> HI All, >> I am getting this error while generating executable Jar file itself in >> Eclipse, if the Spark Application code has "import sqlContext.implicits._" >> line in it. Spark Applicaiton code works fine if the above mentioned line >> does not exist as I have tested by fetching data from an RDBMS by >> implementing JDBCRDD >> >> I tried couple of DataFrame related methods for which most of them errors >> stating that method has been overloaded >> >> Please let me know if any further inputs needed to analyze it >> >> Regards, >> Satish Chandra >> >> On Fri, Nov 20, 2015 at 5:46 PM, prosp4300 wrote: >> >>> >>> Looks like a classpath problem, if you can provide the command you used >>> to run your application and environment variable SPARK_HOME, it will help >>> others to identify the root problem >>> >>> >>> 在2015年11月20日 18:59，Satish 写道: >>> >>> Hi Michael, >>> As my current Spark version is 1.4.0 than why it error out as "error: >>> not found: value sqlContext" when I have "import sqlContext.implicits._" in >>> my Spark Job >>> >>> Regards >>> Satish Chandra >>> -- >>> From: Michael Armbrust >>> Sent: 20-11-2015 01:36 >>> To: satish chandra j >>> Cc: user ; hari krishna >>> Subject: Re: Error not found value sqlContext >>> >>> >>> http://spark.apache.org/docs/latest/sql-programming-guide.html#upgrading-from-spark-sql-10-12-to-13 >>> >>> On Thu, Nov 19, 2015 at 4:19 AM, satish chandra j < >>> jsatishchan...@gmail.com> wrote: >>> >>>> HI All, >>>> we have recently migrated from Spark 1.2.1 to Spark 1.4.0, I am >>>> fetching data from an RDBMS using JDBCRDD and register it as temp table to >>>> perform SQL query >>>> >>>> Below approach is working fine in Spark 1.2.1: >>>> >>>> JDBCRDD --> apply map using Case Class --> apply createSchemaRDD --> >>>> registerTempTable --> perform SQL Query >>>> >>>> but now as createSchemaRDD is not supported in Spark 1.4.0 >>>> >>>> JDBCRDD --> apply map using Case Class with* .toDF()* --> >>>> registerTempTable --> perform SQL query on temptable >>>> >>>> >>>> JDBCRDD --> apply map using Case Class --> RDD*.toDF()*.registerTempTable >>>> --> perform SQL query on temptable >>>> >>>> Only solution I get everywhere is to use "import >>>> sqlContext.implicits._" after val SQLContext = new >>>> org.apache.spark.sql.SQLContext(sc) >>>> >>>> But it errors with the two generic errors >>>> >>>> *1. error: not found: value sqlContext* >>>> >>>> *2. value toDF is not a member of org.apache.spark.rdd.RDD* >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> >> >> >> >> > >
Re: Re: RE: Error not found value sqlContext
Sorry, just to understand my issue.if Eclipse could not understand Scala syntax properly than it should error for the other Spark Job which is fetching data from a RDBM and printing the output in console. I think there is some dependencies which are missing due to which "import sqlContext.implicits._" is not recognized during compile time Please let me know if any further inputs needed to fix the same Regards, Satish Chandra On Mon, Nov 23, 2015 at 3:29 PM, prosp4300 wrote: > > > So it is actually a compile time error in Eclipse, instead of jar > generation from Eclipse, you can try to use sbt to assembly your jar, looks > like your Eclipse does not recognize the Scala syntax properly. > > > > At 2015-11-20 21:36:55, "satish chandra j" > wrote: > > HI All, > I am getting this error while generating executable Jar file itself in > Eclipse, if the Spark Application code has "import sqlContext.implicits._" > line in it. Spark Applicaiton code works fine if the above mentioned line > does not exist as I have tested by fetching data from an RDBMS by > implementing JDBCRDD > > I tried couple of DataFrame related methods for which most of them errors > stating that method has been overloaded > > Please let me know if any further inputs needed to analyze it > > Regards, > Satish Chandra > > On Fri, Nov 20, 2015 at 5:46 PM, prosp4300 wrote: > >> >> Looks like a classpath problem, if you can provide the command you used >> to run your application and environment variable SPARK_HOME, it will help >> others to identify the root problem >> >> >> 在2015年11月20日 18:59，Satish 写道: >> >> Hi Michael, >> As my current Spark version is 1.4.0 than why it error out as "error: not >> found: value sqlContext" when I have "import sqlContext.implicits._" in my >> Spark Job >> >> Regards >> Satish Chandra >> -- >> From: Michael Armbrust >> Sent: 20-11-2015 01:36 >> To: satish chandra j >> Cc: user ; hari krishna >> Subject: Re: Error not found value sqlContext >> >> >> http://spark.apache.org/docs/latest/sql-programming-guide.html#upgrading-from-spark-sql-10-12-to-13 >> >> On Thu, Nov 19, 2015 at 4:19 AM, satish chandra j < >> jsatishchan...@gmail.com> wrote: >> >>> HI All, >>> we have recently migrated from Spark 1.2.1 to Spark 1.4.0, I am fetching >>> data from an RDBMS using JDBCRDD and register it as temp table to perform >>> SQL query >>> >>> Below approach is working fine in Spark 1.2.1: >>> >>> JDBCRDD --> apply map using Case Class --> apply createSchemaRDD --> >>> registerTempTable --> perform SQL Query >>> >>> but now as createSchemaRDD is not supported in Spark 1.4.0 >>> >>> JDBCRDD --> apply map using Case Class with* .toDF()* --> >>> registerTempTable --> perform SQL query on temptable >>> >>> >>> JDBCRDD --> apply map using Case Class --> RDD*.toDF()*.registerTempTable >>> --> perform SQL query on temptable >>> >>> Only solution I get everywhere is to use "import >>> sqlContext.implicits._" after val SQLContext = new >>> org.apache.spark.sql.SQLContext(sc) >>> >>> But it errors with the two generic errors >>> >>> *1. error: not found: value sqlContext* >>> >>> *2. value toDF is not a member of org.apache.spark.rdd.RDD* >>> >>> >>> >>> >>> >>> >> >> >> > > > >
Re:Re: RE: Error not found value sqlContext
So it is actually a compile time error in Eclipse, instead of jar generation from Eclipse, you can try to use sbt to assembly your jar, looks like your Eclipse does not recognize the Scala syntax properly. At 2015-11-20 21:36:55, "satish chandra j" wrote: HI All, I am getting this error while generating executable Jar file itself in Eclipse, if the Spark Application code has "import sqlContext.implicits._" line in it. Spark Applicaiton code works fine if the above mentioned line does not exist as I have tested by fetching data from an RDBMS by implementing JDBCRDD I tried couple of DataFrame related methods for which most of them errors stating that method has been overloaded Please let me know if any further inputs needed to analyze it Regards, Satish Chandra On Fri, Nov 20, 2015 at 5:46 PM, prosp4300 wrote: Looks like a classpath problem, if you can provide the command you used to run your application and environment variable SPARK_HOME, it will help others to identify the root problem 在2015年11月20日 18:59，Satish 写道: Hi Michael, As my current Spark version is 1.4.0 than why it error out as "error: not found: value sqlContext" when I have "import sqlContext.implicits._" in my Spark Job Regards Satish Chandra From: Michael Armbrust Sent: 20-11-2015 01:36 To: satish chandra j Cc: user; hari krishna Subject: Re: Error not found value sqlContext http://spark.apache.org/docs/latest/sql-programming-guide.html#upgrading-from-spark-sql-10-12-to-13 On Thu, Nov 19, 2015 at 4:19 AM, satish chandra j wrote: HI All, we have recently migrated from Spark 1.2.1 to Spark 1.4.0, I am fetching data from an RDBMS using JDBCRDD and register it as temp table to perform SQL query Below approach is working fine in Spark 1.2.1: JDBCRDD --> apply map using Case Class --> apply createSchemaRDD --> registerTempTable --> perform SQL Query but now as createSchemaRDD is not supported in Spark 1.4.0 JDBCRDD --> apply map using Case Class with .toDF() --> registerTempTable --> perform SQL query on temptable JDBCRDD --> apply map using Case Class --> RDD.toDF().registerTempTable --> perform SQL query on temptable Only solution I get everywhere is to use "import sqlContext.implicits._" after val SQLContext = new org.apache.spark.sql.SQLContext(sc) But it errors with the two generic errors 1. error: not found: value sqlContext 2. value toDF is not a member of org.apache.spark.rdd.RDD
Re: RE: Error not found value sqlContext
HI All, I am getting this error while generating executable Jar file itself in Eclipse, if the Spark Application code has "import sqlContext.implicits._" line in it. Spark Applicaiton code works fine if the above mentioned line does not exist as I have tested by fetching data from an RDBMS by implementing JDBCRDD I tried couple of DataFrame related methods for which most of them errors stating that method has been overloaded Please let me know if any further inputs needed to analyze it Regards, Satish Chandra On Fri, Nov 20, 2015 at 5:46 PM, prosp4300 wrote: > > Looks like a classpath problem, if you can provide the command you used to > run your application and environment variable SPARK_HOME, it will help > others to identify the root problem > > > 在2015年11月20日 18:59，Satish 写道: > > Hi Michael, > As my current Spark version is 1.4.0 than why it error out as "error: not > found: value sqlContext" when I have "import sqlContext.implicits._" in my > Spark Job > > Regards > Satish Chandra > -- > From: Michael Armbrust > Sent: 20-11-2015 01:36 > To: satish chandra j > Cc: user ; hari krishna > Subject: Re: Error not found value sqlContext > > > http://spark.apache.org/docs/latest/sql-programming-guide.html#upgrading-from-spark-sql-10-12-to-13 > > On Thu, Nov 19, 2015 at 4:19 AM, satish chandra j < > jsatishchan...@gmail.com> wrote: > >> HI All, >> we have recently migrated from Spark 1.2.1 to Spark 1.4.0, I am fetching >> data from an RDBMS using JDBCRDD and register it as temp table to perform >> SQL query >> >> Below approach is working fine in Spark 1.2.1: >> >> JDBCRDD --> apply map using Case Class --> apply createSchemaRDD --> >> registerTempTable --> perform SQL Query >> >> but now as createSchemaRDD is not supported in Spark 1.4.0 >> >> JDBCRDD --> apply map using Case Class with* .toDF()* --> >> registerTempTable --> perform SQL query on temptable >> >> >> JDBCRDD --> apply map using Case Class --> RDD*.toDF()*.registerTempTable >> --> perform SQL query on temptable >> >> Only solution I get everywhere is to use "import sqlContext.implicits._" >> after val SQLContext = new org.apache.spark.sql.SQLContext(sc) >> >> But it errors with the two generic errors >> >> *1. error: not found: value sqlContext* >> >> *2. value toDF is not a member of org.apache.spark.rdd.RDD* >> >> >> >> >> >> > > >
回复：RE: Error not found value sqlContext
Looks like a classpath problem, if you can provide the command you used to run your application and environment variable SPARK_HOME, it will help others to identify the root problem 在2015年11月20日 18:59，Satish 写道: Hi Michael, As my current Spark version is 1.4.0 than why it error out as "error: not found: value sqlContext" when I have "import sqlContext.implicits._" in my Spark Job Regards Satish Chandra From: Michael Armbrust Sent: 20-11-2015 01:36 To: satish chandra j Cc: user; hari krishna Subject: Re: Error not found value sqlContext http://spark.apache.org/docs/latest/sql-programming-guide.html#upgrading-from-spark-sql-10-12-to-13 On Thu, Nov 19, 2015 at 4:19 AM, satish chandra j wrote: HI All, we have recently migrated from Spark 1.2.1 to Spark 1.4.0, I am fetching data from an RDBMS using JDBCRDD and register it as temp table to perform SQL query Below approach is working fine in Spark 1.2.1: JDBCRDD --> apply map using Case Class --> apply createSchemaRDD --> registerTempTable --> perform SQL Query but now as createSchemaRDD is not supported in Spark 1.4.0 JDBCRDD --> apply map using Case Class with .toDF() --> registerTempTable --> perform SQL query on temptable JDBCRDD --> apply map using Case Class --> RDD.toDF().registerTempTable --> perform SQL query on temptable Only solution I get everywhere is to use "import sqlContext.implicits._" after val SQLContext = new org.apache.spark.sql.SQLContext(sc) But it errors with the two generic errors 1. error: not found: value sqlContext 2. value toDF is not a member of org.apache.spark.rdd.RDD
RE: Error not found value sqlContext
Hi Michael, As my current Spark version is 1.4.0 than why it error out as "error: not found: value sqlContext" when I have "import sqlContext.implicits._" in my Spark Job Regards Satish Chandra -Original Message- From: "Michael Armbrust" Sent: 20-11-2015 01:36 To: "satish chandra j" Cc: "user" ; "hari krishna" Subject: Re: Error not found value sqlContext http://spark.apache.org/docs/latest/sql-programming-guide.html#upgrading-from-spark-sql-10-12-to-13 On Thu, Nov 19, 2015 at 4:19 AM, satish chandra j wrote: HI All, we have recently migrated from Spark 1.2.1 to Spark 1.4.0, I am fetching data from an RDBMS using JDBCRDD and register it as temp table to perform SQL query Below approach is working fine in Spark 1.2.1: JDBCRDD --> apply map using Case Class --> apply createSchemaRDD --> registerTempTable --> perform SQL Query but now as createSchemaRDD is not supported in Spark 1.4.0 JDBCRDD --> apply map using Case Class with .toDF() --> registerTempTable --> perform SQL query on temptable JDBCRDD --> apply map using Case Class --> RDD.toDF().registerTempTable --> perform SQL query on temptable Only solution I get everywhere is to use "import sqlContext.implicits._" after val SQLContext = new org.apache.spark.sql.SQLContext(sc) But it errors with the two generic errors 1. error: not found: value sqlContext 2. value toDF is not a member of org.apache.spark.rdd.RDD
Re: Error not found value sqlContext
http://spark.apache.org/docs/latest/sql-programming-guide.html#upgrading-from-spark-sql-10-12-to-13 On Thu, Nov 19, 2015 at 4:19 AM, satish chandra j wrote: > HI All, > we have recently migrated from Spark 1.2.1 to Spark 1.4.0, I am fetching > data from an RDBMS using JDBCRDD and register it as temp table to perform > SQL query > > Below approach is working fine in Spark 1.2.1: > > JDBCRDD --> apply map using Case Class --> apply createSchemaRDD --> > registerTempTable --> perform SQL Query > > but now as createSchemaRDD is not supported in Spark 1.4.0 > > JDBCRDD --> apply map using Case Class with* .toDF()* --> > registerTempTable --> perform SQL query on temptable > > > JDBCRDD --> apply map using Case Class --> RDD*.toDF()*.registerTempTable > --> perform SQL query on temptable > > Only solution I get everywhere is to use "import sqlContext.implicits._" > after val SQLContext = new org.apache.spark.sql.SQLContext(sc) > > But it errors with the two generic errors > > *1. error: not found: value sqlContext* > > *2. value toDF is not a member of org.apache.spark.rdd.RDD* > > > > > >
Error not found value sqlContext
HI All, we have recently migrated from Spark 1.2.1 to Spark 1.4.0, I am fetching data from an RDBMS using JDBCRDD and register it as temp table to perform SQL query Below approach is working fine in Spark 1.2.1: JDBCRDD --> apply map using Case Class --> apply createSchemaRDD --> registerTempTable --> perform SQL Query but now as createSchemaRDD is not supported in Spark 1.4.0 JDBCRDD --> apply map using Case Class with* .toDF()* --> registerTempTable --> perform SQL query on temptable JDBCRDD --> apply map using Case Class --> RDD*.toDF()*.registerTempTable --> perform SQL query on temptable Only solution I get everywhere is to use "import sqlContext.implicits._" after val SQLContext = new org.apache.spark.sql.SQLContext(sc) But it errors with the two generic errors *1. error: not found: value sqlContext* *2. value toDF is not a member of org.apache.spark.rdd.RDD*
