[
https://issues.apache.org/jira/browse/SPARK-8304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-8304:
-----------------------------
Component/s: SQL
Please review
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
I don't think this is a valid issue report since this doesn't really describe a
specific problem other than "it's slow for me".
> Table with a large number of columns
> ------------------------------------
>
> Key: SPARK-8304
> URL: https://issues.apache.org/jira/browse/SPARK-8304
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.3.1
> Reporter: jaeboo jung
>
> SQLContext can't handle any table with a large number of columns. Making
> dataframe is ok but when a user try to execute query on it, spark doesn't
> respond. To test, run below code from spark-shell.
> {code:java}
> import org.apache.spark.sql._
> import org.apache.spark.sql.types._
> val arr = (1 to 500000)
> val columns = StructType(arr.map(x => StructField("columnNum_"+x ,
> StringType, true)))
> val data = arr.map(x => arr)
> val rdd = sc.parallelize(data , 1000).map(Row.fromSeq(_))
> val df = sqlContext.createDataFrame(rdd,columns)
> //select few columns among 500,000 columns
> def select1() = {
> val t1 = System.currentTimeMillis
> df.select("columnNum_1")
> println( System.currentTimeMillis - t1 )
> }
> def select2() = {
> val t1 = System.currentTimeMillis
> df.select("columnNum_1","columnNum_2")
> println( System.currentTimeMillis - t1 )
> }
> def select3() = {
> val t1 = System.currentTimeMillis
> df.select("columnNum_1","columnNum_2","columnNum_3")
> println( System.currentTimeMillis - t1 )
> }
> def select4() = {
> val t1 = System.currentTimeMillis
> df.select("columnNum_1","columnNum_2","columnNum_3","columnNum_4")
> println( System.currentTimeMillis - t1 )
> }
> def select5() = {
> val t1 = System.currentTimeMillis
> df.select("columnNum_1","columnNum_2","columnNum_3","columnNum_4","columnNum_5")
>
> println( System.currentTimeMillis - t1 )
> }
> def select6() = {
> val t1 = System.currentTimeMillis
> df.select("columnNum_1","columnNum_2","columnNum_3","columnNum_4","columnNum_5","columnNum_6")
>
> println( System.currentTimeMillis - t1 )
> }
> def select7() = {
> val t1 = System.currentTimeMillis
> df.select("columnNum_1","columnNum_2","columnNum_3","columnNum_4","columnNum_5","columnNum_6","columnNum_7")
>
> println( System.currentTimeMillis - t1 )
> }
> {code}
> And the result is,
> {code}
> select1
> 20552
> select2
> 25391
> select3
> 29619
> select4
> 33695
> select5
> 42220
> select6
> 44790
> select7
> 49101
> {code}
> Elapsed time for selecting columns is increased about 4000ms after each
> addition.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
