Github user ron8hu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19783#discussion_r154223705
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/EstimationUtils.scala
---
@@ -114,4 +114,197 @@ object EstimationUtils {
}
}
+ /**
+ * Returns the number of the first bin into which a column values falls
for a specified
+ * numeric equi-height histogram.
+ *
+ * @param value a literal value of a column
+ * @param histogram a numeric equi-height histogram
+ * @return the number of the first bin into which a column values falls.
+ */
+
+ def findFirstBinForValue(value: Double, histogram: Histogram): Int = {
+ var binId = 0
--- End diff --
I hesitate to add an assert statement here. This is because an assert such
as this may cause Spark system to crash if a user does not fresh his data
statistics quickly. In real world, a user may load data, collect statistics,
and then add more incremental data, but does not collect statistics
immediately. He may issue a SQL query against his newly added data such as
"WHERE column=xxx", where xxx is a new value in his incremental load. After
all, statistics are auxiliary, a query should still run even the statistics are
not up to date.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
