Github user mvhlong commented on the pull request:
https://github.com/apache/tajo/pull/200#issuecomment-59173282
I'm glad to discuss with you, guys.
@jihoonson I am a little confusing. Your advice is to use NumericDatum (not
Datum) instead of Double. In Tajo, NumericDatum represents INTx and FLOATx. I
think that Double can cover a broader range of numeric values, not only int and
float, but also boolean, bit, datetime, and char. Because the data is big, the
sample data is big, too (if you take too small samples, the accuracy will be
too low). Meanwhile, a table may contain hundreds of columns, each of which
needs to construct a separate histogram. Hence, histogram construction time can
be very long and we should alleviate this burden by using simple data type,
such as Double instead of NumericDatum (or Datum). Data type conversion from
Datum to Double can be done by a utility function.
equi-width and equi-depth are simple histograms, thus obtain not-so-great
estimation accuracy. When you want to improve the accuracy of selectivity
estimation by implementing more complex histograms in the future (for example,
multidimensional histograms - to catch the dependencies of data between
different columns), you will see that histogram construction time is a real
problem. So, it'd be better to keep it simple.
For a fast extension, TEXT can be approximately mapped to a numerical
value, too. For more complex types (including TEXT if you wish), in my opinion,
Tajo needs a special treatment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
