alexeykudinkin commented on a change in pull request #4026:
URL: https://github.com/apache/hudi/pull/4026#discussion_r756313149
##########
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/DataSkippingUtils.scala
##########
@@ -36,120 +36,153 @@ import scala.collection.JavaConverters._
object DataSkippingUtils {
/**
- * create z_index filter and push those filters to index table to filter
all candidate scan files.
- * @param condition origin filter from query.
- * @param indexSchema schema from index table.
- * @return filters for index table.
- */
- def createZindexFilter(condition: Expression, indexSchema: StructType):
Expression = {
- def buildExpressionInternal(colName: Seq[String], statisticValue: String):
Expression = {
- val appendColName = UnresolvedAttribute(colName).name + statisticValue
- col(appendColName).expr
- }
-
- def reWriteCondition(colName: Seq[String], conditionExpress: Expression):
Expression = {
- val appendColName = UnresolvedAttribute(colName).name + "_minValue"
- if (indexSchema.exists(p => p.name == appendColName)) {
+ * Translates provided {@link filterExpr} into corresponding
filter-expression for Z-index index table
+ * to filter out candidate files that would hold records matching the
original filter
+ *
+ * @param filterExpr original filter from query
+ * @param indexSchema index table schema
+ * @return filter for Z-index table
+ */
+ def createZIndexLookupFilter(filterExpr: Expression, indexSchema:
StructType): Expression = {
Review comment:
I was thinking that we do this re-framing when we add Hilbert curve (in
#3952) -- there will be a few more places we will need to adjust naming,
re-shuffle some abstractions a bit, so i would rather suggest to take this up
as part of that PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
