Many operations in spark are lazy -- most likely your collect() statement
is actually forcing evaluation of severals steps earlier in the pipeline.
The logs the UI might give you some info about all the stages that are
being run when you get to collect().
I think collect() is just fine if you are trying to pull just one record to
the driver, that shouldn't be a bottleneck.
On Wed, Feb 4, 2015 at 1:32 AM, jguliani jasminkguli...@gmail.com wrote:
I have 3 text files in hdfs which I am reading using spark sql and
registering them as table. After that I am doing almost 5-6 operations -
including joins , group by etc.. And this whole process is taking hardly
6-7
secs. ( Source File size - 3 GB with almost 20 million rows ).
As a final step of my computation, I am expecting only 1 record in my final
rdd - named as acctNPIScr in below code snippet.
My question here is that when I am trying to print this rdd either by
registering as table and printing records from table or by this method -
acctNPIScr.map(t = Score: + t(1)).collect().foreach(println). It is
taking very long time - almost 1.5 minute to print 1 record.
Can someone pls help me if I am doing something wrong in printing. What is
the best way to print final result from schemardd.
.
val acctNPIScr = sqlContext.sql(SELECT party_id,
sum(npi_int)/sum(device_priority_new) as npi_score FROM AcctNPIScoreTemp
group by party_id )
acctNPIScr.registerTempTable(AcctNPIScore)
val endtime = System.currentTimeMillis()
logger.info(Total sql Time : + (endtime - st)) // this time is
hardly 5 secs
println(start printing)
val result = sqlContext.sql(SELECT * FROM
AcctNPIScore).collect().foreach(println)
//acctNPIScr.map(t = Score: + t(1)).collect().foreach(println)
logger.info(Total printing Time : + (System.currentTimeMillis() -
endtime)) // print one record is taking almost 1.5 minute
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-taking-long-time-to-print-records-from-a-table-tp21493.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
