JingsongLi opened a new pull request, #9905: URL: https://github.com/apache/paimon/pull/9905
### Purpose Spark's V2 expression builder exposes more deterministic string and date expressions than `SparkExpressionConverter` currently translates. This PR adds Paimon transforms and conversion support for: - `BIT_LENGTH`, `TRANSLATE`, and `OVERLAY` - `LPAD` and `RPAD` on Spark 4.1+ - `DATE_ADD`, `DATE_DIFF`, and `TRUNC` - `QUARTER`, ISO `DAY_OF_WEEK`, `DAY_OF_YEAR`, `WEEK`, and `YEAR_OF_WEEK` - Spark's arithmetic encodings for `dayofweek` and `weekday` The implementations preserve Spark's null, Unicode/code-point, ISO-week, input-type, and timestamp time-zone semantics. The new transforms are also registered for predicate JSON serialization. ### Tests - `SparkScalarTransformTest`, `DateExtractTransformTest`, and `TransformJsonSerdeTest`: 217 tests passed - Spark 3.5 `SparkV2FilterConverterTest`: 41 tests passed - Spark 4.1 `SparkV2FilterConverterTest`: 41 tests passed - Spark 3.2 `PaimonPushDownTest`: 14 tests passed and 8 version-gated tests canceled - Spotless check passed for the affected common and Spark modules -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
