lukasz-antoniak commented on code in PR #254:
URL:
https://github.com/apache/cassandra-analytics/pull/254#discussion_r4404051075
##########
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/data/DataLayer.java:
##########
@@ -348,6 +368,219 @@ public Filter[] unsupportedPushDownFilters(Filter[]
filters)
return partitionKeys.size() > 0 ? filters :
unsupportedFilters.toArray(new Filter[0]);
}
+ /**
+ * @return SAI definitions available for this table.
+ */
+ @NotNull
+ public List<SaiIndex> saiIndexes()
+ {
+ return Collections.emptyList();
+ }
+
+ /**
+ * Extracts Spark predicates that can safely be used as SAI pruning hints.
+ *
+ * These predicates remain in {@link
#unsupportedPushDownFilters(Filter[])} so Spark evaluates them after the
+ * Cassandra reader has reconciled all candidate partitions. SAI therefore
narrows physical I/O without becoming
+ * part of the correctness boundary.
+ */
+ @NotNull
+ public List<SaiFilter> saiFilters(@NotNull Filter[] filters)
+ {
+ if (saiIndexes().isEmpty())
+ {
+ return Collections.emptyList();
+ }
+
+ List<SaiFilter> result = new ArrayList<>();
+ for (Filter filter : filters)
+ {
+ addSaiFilters(filter, result);
+ }
+ return result;
+ }
+
+ /**
+ * Adds SAI-capable predicates from a conjunctive Spark filter tree.
+ *
+ * A partially-indexable AND can still use its indexable branch because
every final match must satisfy that
+ * branch. OR is not supported by Cassandra 5.0 SAI, so an OR subtree is
never used for SAI pruning.
+ */
+ private void addSaiFilters(@NotNull Filter filter, @NotNull
List<SaiFilter> result)
+ {
+ if (filter instanceof And)
+ {
+ And and = (And) filter;
+ addSaiFilters(and.left(), result);
+ addSaiFilters(and.right(), result);
+ return;
+ }
+ if (filter instanceof Or)
+ {
+ return;
+ }
+
+ SaiFilter saiFilter = toSaiFilter(filter);
+ if (saiFilter != null)
+ {
+ result.add(saiFilter);
+ }
+ }
+
+ @Nullable
+ private SaiFilter toSaiFilter(@NotNull Filter filter)
+ {
+ String attribute = filterAttribute(filter);
+ Object value = filterValue(filter);
+ SaiFilter.Operator operator = saiOperator(filter);
+ if (attribute == null || value == null || operator == null)
+ {
+ return null;
+ }
+
+ SaiIndex index = saiIndexForAttribute(attribute);
+ return index == null ? null : new SaiFilter(index, operator,
String.valueOf(value));
Review Comment:
Spark SQL built-in functions like `array_contains()` are not passed down to
push-down filters. Let us document lack of support for complex types and UDTs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
