Re: Performance of loading parquet files into case classes in Spark

[Maciej Bryński]

2016-08-27 15:27 GMT+02:00 Julien Dumazert <julien.dumaz...@gmail.com>:

> df.map(row => row.getAs[Long]("fieldToSum")).reduce(_ + _)


I think reduce and sum has very different performance.
Did you try sql.functions.sum ?
Or of you want to benchmark access to Row object then  count() function
will be better idea.

Regards,
-- 
Maciek Bryński