咦 ？？？ I will have a try.
Thanks,
Minglei.
发件人: Yanbo Liang [mailto:[email protected]]
发送时间: 2015年12月25日 16:07
收件人: Jeff Zhang
抄送: zml张明磊; [email protected]
主题: Re: 答复: How can I get the column data based on specific column name and
then stored these data in array or list ?
Actually you can call df.collect_list("a").
2015-12-25 16:00 GMT+08:00 Jeff Zhang
<[email protected]<mailto:[email protected]>>:
You can use udf to convert one column for array type. Here's one sample
val conf = new SparkConf().setMaster("local[4]").setAppName("test")
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
import sqlContext.implicits._
import sqlContext._
sqlContext.udf.register("f", (a:String) => Array(a,a))
val df1 = Seq(
(1, "jeff", 12),
(2, "andy", 34),
(3, "pony", 23),
(4, "jeff", 14)
).toDF("id", "name", "age")
val df2=df1.withColumn("name", expr("f(name)"))
df2.printSchema()
df2.show()
On Fri, Dec 25, 2015 at 3:44 PM, zml张明磊
<[email protected]<mailto:[email protected]>> wrote:
Thanks, Jeff. It’s not choose some columns of a Row. It’s just choose all data
in a column and convert it to an Array. Do you understand my mean ?
In Chinese
我是想基于这个列名把这一列中的所有数据都选出来，然后放到数组里面去。
发件人: Jeff Zhang [mailto:[email protected]<mailto:[email protected]>]
发送时间: 2015年12月25日 15:39
收件人: zml张明磊
抄送: [email protected]<mailto:[email protected]>
主题: Re: How can I get the column data based on specific column name and then
stored these data in array or list ?
Not sure what you mean. Do you want to choose some columns of a Row and convert
it to an Arrray ?
On Fri, Dec 25, 2015 at 3:35 PM, zml张明磊
<[email protected]<mailto:[email protected]>> wrote:
Hi,
I am a new to Scala and Spark and trying to find relative API in
DataFrame to solve my problem as title described. However, I just only find
this API DataFrame.col(colName : String) : Column which returns an object of
Column. Not the content. If only DataFrame support such API which like
Column.toArray : Type is enough for me. But now, it doesn’t. How can I do can
achieve this function ?
Thanks,
Minglei.
--
Best Regards
Jeff Zhang
--
Best Regards
Jeff Zhang
