Yaoxuan Wu created SPARK-60547:
----------------------------------
Summary: createDataFrame with TimeType(p) keeps more than p
fractional digits, so equal TIME(p) values compare different
Key: SPARK-60547
URL: https://issues.apache.org/jira/browse/SPARK-60547
Project: Spark
Issue Type: Bug
Components: PySpark, SQL
Affects Versions: 4.1.1, 4.2.0
Reporter: Yaoxuan Wu
{code:python}
import datetime
from pyspark.sql.types import StructType, StructField, TimeType
spark.conf.set("spark.sql.timeType.enabled", "true")
df = spark.createDataFrame([(datetime.time(12, 0, 0, 500000),),
(datetime.time(12, 0, 0),)],
StructType([StructField("c", TimeType(0))]))
df.collect()
# expected: [Row(c=12:00:00), Row(c=12:00:00)]
# actual: [Row(c=12:00:00.500000), Row(c=12:00:00)]
df.distinct().count()
# expected: 1, actual: 2
{code}
A TIME(0) column then holds two values that are the same TIME(0) (and that
{{second(c)}} or a cast to TIME(6) and back treat as equal), but {{=}}, {{GROUP
BY}}, {{DISTINCT}} and joins treat as different.
Cause: {{TimeType.toInternal}} in {{python/pyspark/sql/types.py}} converts the
{{datetime.time}} to nanoseconds of the day without truncating to
{{self.precision}} (same on master).
On 4.1 / 4.2 the same happens for a cast from STRING: {{CAST('12:00:00.5' AS
TIME(0))}} returns 12:00:00.5, while {{CAST(TIME'12:00:00.5' AS TIME(0))}}
returns 12:00:00. In {{Cast.castToTime}} only the {{TimeType}} case calls
{{truncateTimeToPrecision}}. Master already truncates the STRING case too
(SPARK-57551 / SPARK-57588 work for 4.3.0); a backport to branch-4.1 /
branch-4.2 would fix that part.
Reproduces on 4.1.1 and 4.2.0.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
