[
https://issues.apache.org/jira/browse/SPARK-60547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60547:
-----------------------------------
Labels: correctness pull-request-available (was: correctness)
> createDataFrame with TimeType(p) keeps more than p fractional digits, so
> equal TIME(p) values compare different
> ---------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60547
> URL: https://issues.apache.org/jira/browse/SPARK-60547
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 4.2.0, 4.1.1
> Reporter: Yaoxuan Wu
> Priority: Major
> Labels: correctness, pull-request-available
>
> {code:python}
> import datetime
> from pyspark.sql.types import StructType, StructField, TimeType
> spark.conf.set("spark.sql.timeType.enabled", "true")
> df = spark.createDataFrame([(datetime.time(12, 0, 0, 500000),),
> (datetime.time(12, 0, 0),)],
> StructType([StructField("c", TimeType(0))]))
> df.collect()
> # expected: [Row(c=12:00:00), Row(c=12:00:00)]
> # actual: [Row(c=12:00:00.500000), Row(c=12:00:00)]
> df.distinct().count()
> # expected: 1, actual: 2
> {code}
> A TIME(0) column then holds two values that are the same TIME(0) (and that
> {{second(c)}} or a cast to TIME(6) and back treat as equal), but {{=}},
> {{GROUP BY}}, {{DISTINCT}} and joins treat as different.
> Cause: {{TimeType.toInternal}} in {{python/pyspark/sql/types.py}} converts
> the {{datetime.time}} to nanoseconds of the day without truncating to
> {{self.precision}} (same on master).
> On 4.1 / 4.2 the same happens for a cast from STRING: {{CAST('12:00:00.5' AS
> TIME(0))}} returns 12:00:00.5, while {{CAST(TIME'12:00:00.5' AS TIME(0))}}
> returns 12:00:00. In {{Cast.castToTime}} only the {{TimeType}} case calls
> {{truncateTimeToPrecision}}. Master already truncates the STRING case too
> (SPARK-57551 / SPARK-57588 work for 4.3.0); a backport to branch-4.1 /
> branch-4.2 would fix that part.
> Reproduces on 4.1.1 and 4.2.0.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
