[
https://issues.apache.org/jira/browse/TAJO-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174594#comment-14174594
]
ASF GitHub Bot commented on TAJO-1112:
--------------------------------------
Github user mvhlong commented on the pull request:
https://github.com/apache/tajo/pull/200#issuecomment-59454847
I'd like to explain again about the sampling and precision.
It is best to create a histogram from the full column data. However,
because it will take too long time, people often take a sample of the data and
create a histogram based on this sample. When the sampling ratio is low, the
histogram based on this sample will have low selectivity estimation accuracy.
In contrast, when the sampling ratio is high, the histogram will have high
estimation accuracy (samplingRatio = 100% means that we use the full column
data). So, we will want to build a histogram based on a big sample (i.e., high
sampling ratio). But, this may lead to a significant increase in histogram
construction time. Consequently, we will want to reduce the time by using
simple data structures instead of complex ones.
I checked the classes Long and Double. Double cannot represent the entire
value range of Long, thus a histogram built on Double cannot replace another
one built for Long. This is my mistake in making a careless assumption.
After reading your comments, I have thought a lot more about the supporting
of other data types in histograms. Now, I think that I should use Datum since
it is the only way to make a unified and clean implementation, although it
takes longer processing time. In HistogramBucketProto, min and max will be
changed from "double" to "bytes". ( @jihoonson please note that I changed my
opinion about the use of Datum )
> Implement histogram interface and a candidate histogram
> -------------------------------------------------------
>
> Key: TAJO-1112
> URL: https://issues.apache.org/jira/browse/TAJO-1112
> Project: Tajo
> Issue Type: Sub-task
> Reporter: Long Mvh
> Assignee: Long Mvh
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
