GitHub user yhuai opened a pull request:
https://github.com/apache/spark/pull/9813
[SPARK-11792] [SQL] [FOLLOW-UP] Change SizeEstimation to
KnownSizeEstimation and make estimatedSize return Long instead of Option[Long]
https://issues.apache.org/jira/browse/SPARK-11792
The main changes include:
* Renaming `SizeEstimation` to `KnownSizeEstimation`. Hopefully this new
name has more information.
* Making `estimatedSize` return `Long` instead of `Option[Long]`.
* In `UnsaveHashedRelation`, `estimatedSize` will delegate the work to
`SizeEstimator` if we have not created a `BytesToBytesMap`.
Since we will put `UnsaveHashedRelation` to `BlockManager`, it is generally
good to let it provide a more accurate size estimation. Also, if we do not put
`BytesToBytesMap` directly into `BlockerManager`, I feel it is not really
necessary to make `BytesToBytesMap` extends `KnownSizeEstimation`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yhuai/spark SPARK-11792-followup
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9813.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9813
----
commit 329c72fbe7d0a49416edde5ac14cc5b9551d9e69
Author: Yin Huai <[email protected]>
Date: 2015-11-18T15:34:15Z
Change SizeEstimation to KnownSizeEstimation and make estimatedSize return
Long instead of Option[Long].
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
