leaves12138 commented on code in PR #9891:
URL: https://github.com/apache/paimon/pull/9891#discussion_r4025186150
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/util/SparkExpressionConverter.scala:
##########
@@ -77,12 +87,26 @@ object SparkExpressionConverter {
i => new TrimTransform(i, TrimTransform.Flag.TRAILING))
case _ => None
}
- case c: Cast =>
- c.expression() match {
- case n: NamedReference =>
- CastTransform.tryCreate(
- toPaimonFieldRef(n, rowType),
- SparkTypeUtils.toPaimonType(c.dataType()))
+ // The connector `Extract` expression was added in Spark 3.4 and does
not exist on
+ // Spark 3.2/3.3 runtimes, so its type test must stay behind this
version gate to avoid
+ // a NoClassDefFoundError when linking the class there.
+ case e if org.apache.spark.SPARK_VERSION >= "3.4" =>
+ e match {
+ case extract: Extract =>
+ extract.source() match {
+ case n: NamedReference =>
+ val fieldRef = toPaimonFieldRef(n, rowType)
+ extract.field() match {
+ case EXTRACT_YEAR => YearTransform.tryCreate(fieldRef)
+ case EXTRACT_MONTH => MonthTransform.tryCreate(fieldRef)
+ case EXTRACT_DAY => DayTransform.tryCreate(fieldRef)
+ case EXTRACT_HOUR => HourTransform.tryCreate(fieldRef)
Review Comment:
[P1] Keep legacy-mapped timestamp extraction in Spark
Checking only the Paimon field type is insufficient here. With
`spark.paimon.legacy-timestamp-mapping.enabled=true`, a Paimon
`TIMESTAMP_WITHOUT_TIME_ZONE` is exposed as Spark `TIMESTAMP`, so `hour(ts)` is
evaluated in the Spark session time zone. `HourTransform` instead extracts the
stored local timestamp directly. These differ when the Spark session and JVM
time zones differ. For a timestamp partition column, Spark considers this
predicate fully handled and removes its residual filter, while Paimon prunes
the matching partition.
Reproduced on this head with Spark 3.5.8, JVM time zone UTC, and both ORC
and Parquet:
```sql
SET spark.paimon.legacy-timestamp-mapping.enabled=true;
SET spark.sql.session.timeZone=America/Los_Angeles;
CREATE TABLE legacy_extract (id INT, ts TIMESTAMP)
USING paimon PARTITIONED BY (ts);
INSERT INTO legacy_extract VALUES (1, timestamp'2025-01-15 01:02:03');
SELECT hour(ts) FROM legacy_extract;
-- Returns 1.
SELECT id FROM legacy_extract WHERE hour(ts) = 1;
-- Expected [1], actual empty result.
```
The physical plan contains `PartitionFilters: [And([IsNotNull(ts),
Equal(HOUR(ts), 1)])]` with no Spark residual filter. Please decline timestamp
extraction when `treatPaimonTimestampTypeAsSparkTimestampType()` is enabled,
unless the transform explicitly reproduces Spark's time-zone semantics. A
temporary local guard for this case fixed both reproductions; all 42
converter/boundary tests passed with that guard. Please also add a
partitioned-table regression with different JVM and session time zones; the
existing LTZ fallback test only covers the default type mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
