flashJd commented on code in PR #8102:
URL: https://github.com/apache/hudi/pull/8102#discussion_r1203545727
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/ExpressionUtils.java:
##########
@@ -177,4 +183,85 @@ public static Object
getValueFromLiteral(ValueLiteralExpression expr) {
throw new UnsupportedOperationException("Unsupported type: " +
logicalType);
}
}
+
+ public static List<ResolvedExpression>
filterSimpleCallExpression(List<ResolvedExpression> exprs) {
+ return exprs.stream()
+ .filter(ExpressionUtils::isSimpleCallExpression)
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * Extracts partition predicate from filter condition.
+ *
+ * <p>NOTE: the {@code expressions} should be simple call expressions.
+ *
+ * @return A tuple of partition predicates and non-partition predicates.
+ */
+ public static Tuple2<List<ResolvedExpression>, List<ResolvedExpression>>
splitExprByPartitionCall(
+ List<ResolvedExpression> expressions,
+ List<String> partitionKeys,
+ RowType tableRowType) {
+ if (partitionKeys.isEmpty()) {
+ return Tuple2.of(expressions, Collections.emptyList());
+ } else {
+ List<ResolvedExpression> partitionFilters = new ArrayList<>();
+ List<ResolvedExpression> nonPartitionFilters = new ArrayList<>();
+ final List<String> fieldNames = tableRowType.getFieldNames();
+ Set<Integer> parFieldPos =
partitionKeys.stream().map(fieldNames::indexOf).collect(Collectors.toSet());
+ for (ResolvedExpression expr : expressions) {
+ for (CallExpression e : splitByAnd(expr)) {
+ if (isPartitionCallExpr(e, parFieldPos)) {
+ partitionFilters.add(expr);
Review Comment:
should it be partitionFilters.add(e)?
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/ExpressionUtils.java:
##########
@@ -177,4 +183,85 @@ public static Object
getValueFromLiteral(ValueLiteralExpression expr) {
throw new UnsupportedOperationException("Unsupported type: " +
logicalType);
}
}
+
+ public static List<ResolvedExpression>
filterSimpleCallExpression(List<ResolvedExpression> exprs) {
+ return exprs.stream()
+ .filter(ExpressionUtils::isSimpleCallExpression)
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * Extracts partition predicate from filter condition.
+ *
+ * <p>NOTE: the {@code expressions} should be simple call expressions.
+ *
+ * @return A tuple of partition predicates and non-partition predicates.
+ */
+ public static Tuple2<List<ResolvedExpression>, List<ResolvedExpression>>
splitExprByPartitionCall(
+ List<ResolvedExpression> expressions,
+ List<String> partitionKeys,
+ RowType tableRowType) {
+ if (partitionKeys.isEmpty()) {
+ return Tuple2.of(expressions, Collections.emptyList());
+ } else {
+ List<ResolvedExpression> partitionFilters = new ArrayList<>();
+ List<ResolvedExpression> nonPartitionFilters = new ArrayList<>();
+ final List<String> fieldNames = tableRowType.getFieldNames();
+ Set<Integer> parFieldPos =
partitionKeys.stream().map(fieldNames::indexOf).collect(Collectors.toSet());
+ for (ResolvedExpression expr : expressions) {
+ for (CallExpression e : splitByAnd(expr)) {
Review Comment:
I use the predicate "where age > 5 and `partition` = 1", flink framework
split the “and”, call applyFilters(List<ResolvedExpression> filters) with a
Two-element list(greatThan(age,5),equals(partition,1);
so I wonder in what condition we should call splitByAnd(expr), as expr
already be split by flink @beyond1920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
