[
https://issues.apache.org/jira/browse/FLINK-40300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Liu updated FLINK-40300:
----------------------------
Description:
{{pyflink.table.expressions.lit(value, data_type)}} and {{lit(value)}} both
pass Python values across Py4J before invoking the Java Table API. For
explicitly typed literals, Py4J converts Python values to general Java types
that may not match the declared data type’s conversion class, such as
{{java.lang.Integer}} instead of {{{}java.lang.Short{}}}. For inferred
literals, Python temporal values cannot be serialized directly, while Python
sequences arrive as generic Java containers that do not preserve the concrete
array types required by Java literal inference. Raw constructed value literals
may also be created but cannot be planned.
For example:
{{from pyflink.table import DataTypes}}
{{from pyflink.table.expressions import lit}}
{{lit(1, DataTypes.SMALLINT(False))}}
This fails with:
{{org.apache.flink.table.api.ValidationException:
Data type 'SMALLINT NOT NULL' with conversion class 'java.lang.Short'
does not support a value literal of class 'java.lang.Integer'.}}
For explicitly typed literals, the issue affects at least {{{}TINYINT{}}},
{{{}SMALLINT{}}}, {{BIGINT}} for 32-bit values, {{{}FLOAT{}}}, {{{}DATE{}}},
{{{}TIME{}}}, {{{}TIMESTAMP{}}}, {{{}TIMESTAMP_LTZ{}}}, intervals, and
constructed types. Without an explicit data type, it affects Python temporal
values, lists, tuples, {{{}array.array{}}}, and nested arrays. Constructed
{{{}ARRAY{}}}, {{{}MAP{}}}, and {{ROW}} values also require recursive
conversion and plannable constructor expressions. The DataFrame API is affected
because its {{lit}} method delegates to the Table API implementation.
was:
{{pyflink.table.expressions.lit(value, data_type)}} passes the Python value
directly to Java. Py4J converts Python values to general Java types (e.g.,
Python {{int}} to {{{}java.lang.Integer{}}}) while Flink requires the value to
match the declared data type’s conversion class exactly, such as
{{java.lang.Short}} for {{{}SMALLINT{}}}. No conversion is performed between
these representations.
For example:
{{from pyflink.table import DataTypes}}
{{from pyflink.table.expressions import lit}}
{{lit(1, DataTypes.SMALLINT(False))}}
This fails with:
{{org.apache.flink.table.api.ValidationException:}}
{{Data type 'SMALLINT NOT NULL' with conversion class 'java.lang.Short'}}
{{does not support a value literal of class 'java.lang.Integer'.}}
The issue affects at least {{{}TINYINT{}}}, {{{}SMALLINT{}}}, {{BIGINT}} for
32-bit values, {{{}FLOAT{}}}, {{{}DATE{}}}, {{{}TIME{}}}, {{{}TIMESTAMP{}}},
{{{}TIMESTAMP_LTZ{}}}, intervals, and some constructed types. It also affects
the DataFrame API because its {{lit}} method delegates to the Table API
implementation.
> PyFlink Table API lit fails for some literal types
> --------------------------------------------------
>
> Key: FLINK-40300
> URL: https://issues.apache.org/jira/browse/FLINK-40300
> Project: Flink
> Issue Type: Bug
> Components: API / Python
> Reporter: Liu Liu
> Assignee: Liu Liu
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.4.0
>
>
> {{pyflink.table.expressions.lit(value, data_type)}} and {{lit(value)}} both
> pass Python values across Py4J before invoking the Java Table API. For
> explicitly typed literals, Py4J converts Python values to general Java types
> that may not match the declared data type’s conversion class, such as
> {{java.lang.Integer}} instead of {{{}java.lang.Short{}}}. For inferred
> literals, Python temporal values cannot be serialized directly, while Python
> sequences arrive as generic Java containers that do not preserve the concrete
> array types required by Java literal inference. Raw constructed value
> literals may also be created but cannot be planned.
> For example:
> {{from pyflink.table import DataTypes}}
> {{from pyflink.table.expressions import lit}}
> {{lit(1, DataTypes.SMALLINT(False))}}
> This fails with:
> {{org.apache.flink.table.api.ValidationException:
> Data type 'SMALLINT NOT NULL' with conversion class 'java.lang.Short'
> does not support a value literal of class 'java.lang.Integer'.}}
> For explicitly typed literals, the issue affects at least {{{}TINYINT{}}},
> {{{}SMALLINT{}}}, {{BIGINT}} for 32-bit values, {{{}FLOAT{}}}, {{{}DATE{}}},
> {{{}TIME{}}}, {{{}TIMESTAMP{}}}, {{{}TIMESTAMP_LTZ{}}}, intervals, and
> constructed types. Without an explicit data type, it affects Python temporal
> values, lists, tuples, {{{}array.array{}}}, and nested arrays. Constructed
> {{{}ARRAY{}}}, {{{}MAP{}}}, and {{ROW}} values also require recursive
> conversion and plannable constructor expressions. The DataFrame API is
> affected because its {{lit}} method delegates to the Table API implementation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
