JingsongLi commented on code in PR #9891:
URL: https://github.com/apache/paimon/pull/9891#discussion_r4025567308
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/util/SparkExpressionConverter.scala:
##########
@@ -77,12 +87,26 @@ object SparkExpressionConverter {
i => new TrimTransform(i, TrimTransform.Flag.TRAILING))
case _ => None
}
- case c: Cast =>
- c.expression() match {
- case n: NamedReference =>
- CastTransform.tryCreate(
- toPaimonFieldRef(n, rowType),
- SparkTypeUtils.toPaimonType(c.dataType()))
+ // The connector `Extract` expression was added in Spark 3.4 and does
not exist on
+ // Spark 3.2/3.3 runtimes, so its type test must stay behind this
version gate to avoid
+ // a NoClassDefFoundError when linking the class there.
+ case e if org.apache.spark.SPARK_VERSION >= "3.4" =>
+ e match {
+ case extract: Extract =>
+ extract.source() match {
+ case n: NamedReference =>
+ val fieldRef = toPaimonFieldRef(n, rowType)
+ extract.field() match {
+ case EXTRACT_YEAR => YearTransform.tryCreate(fieldRef)
+ case EXTRACT_MONTH => MonthTransform.tryCreate(fieldRef)
+ case EXTRACT_DAY => DayTransform.tryCreate(fieldRef)
+ case EXTRACT_HOUR => HourTransform.tryCreate(fieldRef)
Review Comment:
Fixed in 345260b08a. `Extract` conversion now declines Paimon
`TIMESTAMP_WITHOUT_TIME_ZONE` when legacy timestamp mapping exposes it as Spark
`TIMESTAMP`, so Spark retains the time-zone-sensitive predicate. DATE
extraction and normal TIMESTAMP_NTZ extraction remain pushable. Added a
partitioned-table regression with JVM UTC and Spark session
America/Los_Angeles; the Spark 3.5 converter suite now passes all 37 tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
