AngersZhuuuu commented on a change in pull request #24973: [SPARK-28169] Fix
Partition table partition PushDown failed by "OR" expression
URL: https://github.com/apache/spark/pull/24973#discussion_r300249848
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala
##########
@@ -310,3 +310,97 @@ object PhysicalWindow {
case _ => None
}
}
+
+/**
+ * Extract partition push down condition from ExpressionSet
+ * Since origin judge condition is
+ * {
+ * !expression.references.isEmpty &&
+ * expression.references.subsetOf(partitionKeyIds)
+ * }
+ *
+ * This can only push down simple condition expression.
+ * Such as table:
+ * CREATE TABLE DEFAULT.PARTITION_TABLE(
+ * A STRING,
+ * B STRING)
+ * PARTITIONED BY(DT STRING)
+ *
+ * With SQL:
+ * SELECT A, B
+ * FROM DEFAULT.PARTITION_TABLE
+ * WHERE DT = 20190601 OR (DT = 20190602 AND C = "TEST")
+ *
+ * Where condition "DT = 20190601 OR (DT = 20190602 AND C = "TEST")"
Review comment:
@cloud-fan
In my code. coming predicate Set[Expression] has a potential **AND** logical.
For one Expression, it will be restricted by other same level Expression.
and :
- if it is a combine of **AND** each side can be a constraint to others, so
it one side is tenable, it can return a tenable condition.
- if it is a combine of **OR**, if one side is out of control(such as have
no condition about partition cols) this whole **OR** Expression should return
NONE. Only when both side of **OR** 's child is reasonable, it can return a
tenable combine of **OR**.
- if it 's a multilayer nested Expression combined by BinaryOperator. It
will visit the lowest level, if it found one level's **OR** Expression is
untenable, it will break this Expression totally and return null.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
