polsinello opened a new pull request, #27927:
URL: https://github.com/apache/flink/pull/27927
## What is the purpose of the change
In embedded (thread) mode, Pemja's JNI bridge auto-converts Python
`datetime`/`date`/`time` objects to `java.sql.Timestamp`/`Date`/`Time`, but
Flink's `ExternalSerializer` and `DataFormatConverters.RowConverter` expect the
modern `java.time.*` bridge classes (`LocalDateTime`, `LocalDate`, `LocalTime`,
`Instant`). This mismatch causes `ClassCastException` at serialization
boundaries for any UDF that returns or receives temporal types. Process mode is
unaffected because its Beam-based `runnerOutputTypeSerializer` resolves to the
correct `java.time.*` serializers via `LegacyTypeInfoDataTypeConverter`.
This patch fixes several related type-conversion bugs in the embedded Python
execution path so that all Flink-supported types work correctly in thread mode.
## Brief change log
- Add bridge-aware `DataConverter` implementations for all temporal types
(`TIMESTAMP`, `DATE`, `TIME`, `TIMESTAMP_LTZ`) on both the Table API
(`PythonTypeUtils` in `flink-table-runtime`) and DataStream (`PythonTypeUtils`
in `flink-streaming-java`) paths
- Replace `IdentityDataConverter` for temporal types where it silently
passed `java.sql.*` objects through without conversion
- Fix `ROW`/`TUPLE` buffer reuse in `ArrayDataConverter` that caused
silent data corruption in `ARRAY<ROW<...>>` results
- Widen numeric `DataConverter` generics from `Long`/`Double` to `Number`
to avoid bridge-method `ClassCastException` when Pemja returns
`Integer`/`Float` for small values
- Cache the original Java `ExternalTypeInfo` in the Python
`ExternalTypeInfo` wrapper to prevent lossy `DECIMAL` precision round-trips
through legacy `TypeInfo` conversion
## Verifying this change
This change is already covered by existing tests — the existing PyFlink
Table API and DataStream test suites exercise all affected type paths.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no (fixes type conversion *before* values reach the
serializer)
- The runtime per-record code paths (performance sensitive): yes — the
embedded Python UDF execution path now performs explicit type conversions for
temporal/numeric types instead of relying on identity pass-through. The
overhead is minimal (one `java.time.*` factory call per value).
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
