wanggx commented on code in PR #339:
URL:
https://github.com/apache/doris-spark-connector/pull/339#discussion_r2377528535
##########
spark-doris-connector/spark-doris-connector-spark-3.5/src/main/scala/org/apache/doris/spark/read/DorisScanV2.scala:
##########
@@ -27,7 +27,7 @@ class DorisScanV2(config: DorisConfig, schema: StructType,
filters: Array[Predic
override protected def compiledFilters(): Array[String] = {
val inValueLengthLimit =
config.getValue(DorisOptions.DORIS_FILTER_QUERY_IN_MAX_COUNT)
val v2ExpressionBuilder = new V2ExpressionBuilder(inValueLengthLimit)
- filters.map(e =>
Option[String](v2ExpressionBuilder.build(e))).filter(_.isDefined).map(_.get)
Review Comment:
Yes，Some Expression can be pushed down ,for example 'where c1 = 2'，then the
sql of get query plan from doris is select * from table where c1 = 2，and spark
not process c1 = 2 filter。if Expression could not pushed down , for example,
'where coalesce(c1, 'c') = 's' ' V2ExpressionBuilder should return none，and
the sql get query plan from doris is select * from table。the process of
coalesce(c1, 'c') = 's' should run in spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
