Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/10335#issuecomment-165888818
Regarding benchmarking, we typically measure this kind of stuff using
[ForeachResults](https://github.com/databricks/spark-sql-perf/blob/master/src/main/scala/com/databricks/spark/sql/perf/DatasetPerformance.scala#L27)
in spark-sql-perf (which measures just the [time to pull the rows out of the
iterator](https://github.com/databricks/spark-sql-perf/blob/master/src/main/scala/com/databricks/spark/sql/perf/Benchmark.scala#L616)
+ conversion to the external format.) We should probably add an internal
version as well that avoids the conversion cost.
Regarding @hvanhovell simplified implementation, I thought about proposing
it like this. The only question is if we will ever want to add optimizations
on top of this (i.e. we could do a count(*) on this kind of plan really
quickly). Since its already implemented I skew towards the more logically
transparent implementation. However, it might be nice to reuse the code in RDD
as he proposes in the physical operator.
Super minor point: We should probably use
[UnsafeRow.createFromByteArray](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java#L549)
instead of `pointTo`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
