Github user mvhlong commented on the pull request:
https://github.com/apache/tajo/pull/200#issuecomment-59454847
I'd like to explain again about the sampling and precision.
It is best to create a histogram from the full column data. However,
because it will take too long time, people often take a sample of the data and
create a histogram based on this sample. When the sampling ratio is low, the
histogram based on this sample will have low selectivity estimation accuracy.
In contrast, when the sampling ratio is high, the histogram will have high
estimation accuracy (samplingRatio = 100% means that we use the full column
data). So, we will want to build a histogram based on a big sample (i.e., high
sampling ratio). But, this may lead to a significant increase in histogram
construction time. Consequently, we will want to reduce the time by using
simple data structures instead of complex ones.
I checked the classes Long and Double. Double cannot represent the entire
value range of Long, thus a histogram built on Double cannot replace another
one built for Long. This is my mistake in making a careless assumption.
After reading your comments, I have thought a lot more about the supporting
of other data types in histograms. Now, I think that I should use Datum since
it is the only way to make a unified and clean implementation, although it
takes longer processing time. In HistogramBucketProto, min and max will be
changed from "double" to "bytes". ( @jihoonson please note that I changed my
opinion about the use of Datum )
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
