Re: Cannot saveAsParquetFile from a RDD of case class
More info on why toDF is required:
http://spark.apache.org/docs/latest/sql-programming-guide.html#upgrading-from-spark-sql-10-12-to-13
On Tue, Apr 14, 2015 at 6:55 AM, pishen tsai wrote:
> I've changed it to
>
> import sqlContext.implicits._
>
> but it still doesn't work. (I've updated the gist)
>
> BTW, using ".toDF()" do work, thanks for this information.
>
> Regards,
> pishen
>
> 2015-04-14 20:35 GMT+08:00 Todd Nist :
>
>> I think docs are correct. If you follow the example from the docs and
>> add this import shown below, I believe you will get what your looking for:
>>
>> // This is used to implicitly convert an RDD to a DataFrame.import
>> sqlContext.implicits._
>>
>> You could also simply take your rdd and do the following:
>>
>> logs.toDF.saveAsParquetFile("s3n://xxx/xxx")
>>
>>
>> -Todd
>>
>> On Tue, Apr 14, 2015 at 3:50 AM, pishen tsai wrote:
>>
>>> OK, it do work.
>>> Maybe it will be better to update this usage in the official Spark SQL
>>> tutorial:
>>> http://spark.apache.org/docs/latest/sql-programming-guide.html
>>>
>>> Thanks,
>>> pishen
>>>
>>>
>>> 2015-04-14 15:30 GMT+08:00 fightf...@163.com :
>>>
>>>> Hi，there
>>>>
>>>> If you want to use the saveAsParquetFile, you may want to use
>>>> val log_df = sqlContext.createDataFrame(logs)
>>>>
>>>> And then you can issue log_df.saveAsParquetFile (path)
>>>>
>>>> Best,
>>>> Sun.
>>>>
>>>> --
>>>> fightf...@163.com
>>>>
>>>>
>>>> *From:* pishen
>>>> *Date:* 2015-04-14 15:18
>>>> *To:* user
>>>> *Subject:* Cannot saveAsParquetFile from a RDD of case class
>>>> Hello,
>>>>
>>>> I tried to follow the tutorial of Spark SQL, but is not able to
>>>> saveAsParquetFile from a RDD of case class.
>>>> Here is my Main.scala and build.sbt
>>>> https://gist.github.com/pishen/939cad3da612ec03249f
>>>>
>>>> At line 34, compiler said that "value saveAsParquetFile is not a member
>>>> of org.apache.spark.rdd.RDD[core.Log]"
>>>>
>>>> Any suggestion on how to solve this?
>>>>
>>>> Thanks,
>>>> pishen
>>>>
>>>> --
>>>> View this message in context: Cannot saveAsParquetFile from a RDD of
>>>> case class
>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/Cannot-saveAsParquetFile-from-a-RDD-of-case-class-tp22488.html>
>>>> Sent from the Apache Spark User List mailing list archive
>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>>>
>>>>
>>>
>>
>
Re: Cannot saveAsParquetFile from a RDD of case class
I've changed it to
import sqlContext.implicits._
but it still doesn't work. (I've updated the gist)
BTW, using ".toDF()" do work, thanks for this information.
Regards,
pishen
2015-04-14 20:35 GMT+08:00 Todd Nist :
> I think docs are correct. If you follow the example from the docs and add
> this import shown below, I believe you will get what your looking for:
>
> // This is used to implicitly convert an RDD to a DataFrame.import
> sqlContext.implicits._
>
> You could also simply take your rdd and do the following:
>
> logs.toDF.saveAsParquetFile("s3n://xxx/xxx")
>
>
> -Todd
>
> On Tue, Apr 14, 2015 at 3:50 AM, pishen tsai wrote:
>
>> OK, it do work.
>> Maybe it will be better to update this usage in the official Spark SQL
>> tutorial:
>> http://spark.apache.org/docs/latest/sql-programming-guide.html
>>
>> Thanks,
>> pishen
>>
>>
>> 2015-04-14 15:30 GMT+08:00 fightf...@163.com :
>>
>>> Hi，there
>>>
>>> If you want to use the saveAsParquetFile, you may want to use
>>> val log_df = sqlContext.createDataFrame(logs)
>>>
>>> And then you can issue log_df.saveAsParquetFile (path)
>>>
>>> Best,
>>> Sun.
>>>
>>> --
>>> fightf...@163.com
>>>
>>>
>>> *From:* pishen
>>> *Date:* 2015-04-14 15:18
>>> *To:* user
>>> *Subject:* Cannot saveAsParquetFile from a RDD of case class
>>> Hello,
>>>
>>> I tried to follow the tutorial of Spark SQL, but is not able to
>>> saveAsParquetFile from a RDD of case class.
>>> Here is my Main.scala and build.sbt
>>> https://gist.github.com/pishen/939cad3da612ec03249f
>>>
>>> At line 34, compiler said that "value saveAsParquetFile is not a member
>>> of org.apache.spark.rdd.RDD[core.Log]"
>>>
>>> Any suggestion on how to solve this?
>>>
>>> Thanks,
>>> pishen
>>>
>>> --
>>> View this message in context: Cannot saveAsParquetFile from a RDD of
>>> case class
>>> <http://apache-spark-user-list.1001560.n3.nabble.com/Cannot-saveAsParquetFile-from-a-RDD-of-case-class-tp22488.html>
>>> Sent from the Apache Spark User List mailing list archive
>>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>>
>>>
>>
>
Re: Cannot saveAsParquetFile from a RDD of case class
I think docs are correct. If you follow the example from the docs and add
this import shown below, I believe you will get what your looking for:
// This is used to implicitly convert an RDD to a DataFrame.import
sqlContext.implicits._
You could also simply take your rdd and do the following:
logs.toDF.saveAsParquetFile("s3n://xxx/xxx")
-Todd
On Tue, Apr 14, 2015 at 3:50 AM, pishen tsai wrote:
> OK, it do work.
> Maybe it will be better to update this usage in the official Spark SQL
> tutorial:
> http://spark.apache.org/docs/latest/sql-programming-guide.html
>
> Thanks,
> pishen
>
>
> 2015-04-14 15:30 GMT+08:00 fightf...@163.com :
>
>> Hi，there
>>
>> If you want to use the saveAsParquetFile, you may want to use
>> val log_df = sqlContext.createDataFrame(logs)
>>
>> And then you can issue log_df.saveAsParquetFile (path)
>>
>> Best,
>> Sun.
>>
>> --
>> fightf...@163.com
>>
>>
>> *From:* pishen
>> *Date:* 2015-04-14 15:18
>> *To:* user
>> *Subject:* Cannot saveAsParquetFile from a RDD of case class
>> Hello,
>>
>> I tried to follow the tutorial of Spark SQL, but is not able to
>> saveAsParquetFile from a RDD of case class.
>> Here is my Main.scala and build.sbt
>> https://gist.github.com/pishen/939cad3da612ec03249f
>>
>> At line 34, compiler said that "value saveAsParquetFile is not a member
>> of org.apache.spark.rdd.RDD[core.Log]"
>>
>> Any suggestion on how to solve this?
>>
>> Thanks,
>> pishen
>>
>> --
>> View this message in context: Cannot saveAsParquetFile from a RDD of
>> case class
>> <http://apache-spark-user-list.1001560.n3.nabble.com/Cannot-saveAsParquetFile-from-a-RDD-of-case-class-tp22488.html>
>> Sent from the Apache Spark User List mailing list archive
>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>
>>
>
Re: Cannot saveAsParquetFile from a RDD of case class
OK, it do work. Maybe it will be better to update this usage in the official Spark SQL tutorial: http://spark.apache.org/docs/latest/sql-programming-guide.html Thanks, pishen 2015-04-14 15:30 GMT+08:00 fightf...@163.com : > Hi，there > > If you want to use the saveAsParquetFile, you may want to use > val log_df = sqlContext.createDataFrame(logs) > > And then you can issue log_df.saveAsParquetFile (path) > > Best, > Sun. > > -- > fightf...@163.com > > > *From:* pishen > *Date:* 2015-04-14 15:18 > *To:* user > *Subject:* Cannot saveAsParquetFile from a RDD of case class > Hello, > > I tried to follow the tutorial of Spark SQL, but is not able to > saveAsParquetFile from a RDD of case class. > Here is my Main.scala and build.sbt > https://gist.github.com/pishen/939cad3da612ec03249f > > At line 34, compiler said that "value saveAsParquetFile is not a member of > org.apache.spark.rdd.RDD[core.Log]" > > Any suggestion on how to solve this? > > Thanks, > pishen > > -- > View this message in context: Cannot saveAsParquetFile from a RDD of case > class > <http://apache-spark-user-list.1001560.n3.nabble.com/Cannot-saveAsParquetFile-from-a-RDD-of-case-class-tp22488.html> > Sent from the Apache Spark User List mailing list archive > <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com. > >
Cannot saveAsParquetFile from a RDD of case class
Hello, I tried to follow the tutorial of Spark SQL, but is not able to saveAsParquetFile from a RDD of case class. Here is my Main.scala and build.sbt https://gist.github.com/pishen/939cad3da612ec03249f At line 34, compiler said that "value saveAsParquetFile is not a member of org.apache.spark.rdd.RDD[core.Log]" Any suggestion on how to solve this? Thanks, pishen -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Cannot-saveAsParquetFile-from-a-RDD-of-case-class-tp22488.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
