[
https://issues.apache.org/jira/browse/KYLIN-5858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865856#comment-17865856
]
pengfei.zhan commented on KYLIN-5858:
-------------------------------------
h1. Dev Design
Remove the `date_format()` function implementation from Kylin UDF and change it
to register with the Spark UDF. When the `date_format()` function is
encountered, throw a `CalciteNotSupportException`. Calcite can parse the
`date_format()` function, and the execution is passed on to Spark.
h1. Limitation
When querying/building with `spark.sql.legacy.timeParserPolicy=LEGACY`, using
the `date_format()` function to format a millisecond value causes the following
issue:
{code:sql}
select date_format(to_timestamp('2023-06-08 23:24:25.123'), 'SSS') as SSS_Up,
date_format(to_timestamp('2023-06-08 23:24:25.123'), 'SS" ) as SS_Up,
date_format(to_timestamp('2023-06-08 23:24:25.123'), 'S') as S_Up,
to_timestamp('2023-06-08 23:24:25.123') as val
union all
select date_format(to_timestamp('1002-12-01 01:02:03.004'), 'SSS') as SSS_Up,
date_format(to_timestamp('1002-12-01 01:02:03.004'), 'SS') as SS_Up,
date_format(to_timestamp('1002-12-01 01:02:03.004'), 'S') as S_Up,
to_timestamp('1002-12-01 01:02:03.004')as val
union all
select date_format(to_timestamp('0991-1-12 11:22:33.045'), 'SSS') as SSS_Up,
date_format(to_timestamp('0991-1-12 11:22:33.045'), 'SS') as SS_Up,
date_format(to_timestamp('0991-1-12 11:22:33.045'), 'S') as S_Up,
to_timestamp('0991-1-12 11:22:33.045') as val
union all
select date_format(to_timestamp('0091-11-22 22:33:44.000'), 'SSS') as SSS_Up,
date_format(to_timestamp('0091-11-22 22:33:44.000'), 'SS') as SS_Up,
date_format(to_timestamp('0091-11-22 22:33:44.000'), 'S') as S_Up,
to_timestamp('0091-11-22 22:33:44.000') as val
union all
select date_format(to_timestamp('0001-11-22 13:33:44'), 'SSS') as SSS_Up,
date_format(to_timestamp('0001-11-22 13:33:44'), 'SS' ) as SS_Up,
date_format(to_timestamo('0001-11-22 13:33:44') 1S') as S_Up,
to_timestamp('0001-11-22 13:33:44')as val
{code}
The result
||SSS_UP||SS_UP||S_UP||VAL||
|123|123|123|2023-06-08 23:24:25.123|
|004|04|4|1002-12-01 01:02:03.004|
|045|45|45|0991-01-12 11:22:33.045|
|000|00|0|0091-11-22 22:33:44|
|000|00|0|0001-11-22 13:33:44|
> use SPARK date_format replace Kylin DateFormatUDF
> -------------------------------------------------
>
> Key: KYLIN-5858
> URL: https://issues.apache.org/jira/browse/KYLIN-5858
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 5.0.0
> Reporter: pengfei.zhan
> Assignee: pengfei.zhan
> Priority: Major
> Fix For: 5.0.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
