Josh Rosen created SPARK-14966:
----------------------------------
Summary: SizeEstimator should ignore classes in the scala.reflect
package
Key: SPARK-14966
URL: https://issues.apache.org/jira/browse/SPARK-14966
Project: Spark
Issue Type: Improvement
Components: Spark Core
Reporter: Josh Rosen
Assignee: Josh Rosen
In local benchmarks, I noticed SizeEstimator spending tons of time estimating
the size of objects which contain TypeTag or ClassTag fields. The problem with
these tags is that they reference global Scala reflection objects, which, in
turn, reference many singletons, such as TestHive. This throws off the size
estimation and wastes tons of time traversing the huge object graph.
As a result, I think that SizeEstimator should ignore any classes in the
{{scala.reflect}} package.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
