sepuri sai krishna created SPARK-60324:
------------------------------------------
Summary: Partitioned write fails with FileAlreadyExistsException
when a FLOAT or DOUBLE partition column holds 0.0 and -0.0
Key: SPARK-60324
URL: https://issues.apache.org/jira/browse/SPARK-60324
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.3, 4.2.0, 3.5.3
Reporter: sepuri sai krishna
A partitioned write fails when a FLOAT or DOUBLE partition column holds both
0.0 and -0.0 in the same task, and nothing is written. With the default
configuration:
{code:sql}
CREATE TABLE t (id INT, d DOUBLE) USING parquet PARTITIONED BY (d);
INSERT INTO t SELECT /*+ COALESCE(1) */ * FROM VALUES
(1, 0.0D), (2, -0.0D), (3, 0.0D), (4, 1.5D) AS v(id, d);
-- [TASK_WRITE_FAILED] Task failed while writing rows to file:/.../t.
-- Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: File already
exists:
--
file:/.../t/_temporary/0/_temporary/attempt_.../d=0.0/part-00000-<uuid>.c000.snappy.parquet
SELECT count(*) FROM t; -- 0
{code}
Same on 3.5.3, 4.0.3 and 4.2.0 (official binaries) and on master, for Parquet,
ORC, CSV and JSON, {{INSERT INTO}}, {{INSERT OVERWRITE}}, CTAS and
{{df.write.partitionBy}}, FLOAT and DOUBLE, with other partition columns before
it, and Hive tables. NaNs with different bits, e.g. {{NaN}} and {{-NaN}}, fail
the same way, on {{d=NaN}}.
Cause: the write sorts the rows of a task by the dynamic partition columns,
where -0.0 = 0.0 and all NaNs are equal, so the sort can leave them
interleaved: 0.0, -0.0, 0.0. {{DynamicPartitionDataSingleWriter}} starts a new
partition directory whenever the bytes of the partition values change, so it
writes d=0.0, then d=-0.0, then d=0.0 again, and the second file of d=0.0 has
the same name as the first ({{c000}}), which exists. With concurrent writers
({{spark.sql.maxConcurrentOutputFileWriters}}) the writers are keyed by those
bytes, so NaNs with different bits open the same {{d=NaN}} file, and when the
writers fall back to sorting, the zeros fail as above.
When the zeros are in different tasks the write succeeds, but creates both
d=0.0 and d=-0.0, so where a row ends up depends on how the rows are split into
tasks. The file stream sink names every file with a random UUID, so it does not
fail, but it writes both d=0.0 and d=-0.0 even within one task.
Spark already normalizes -0.0 to 0.0 and NaNs to one NaN for grouping, join and
window partition keys (SPARK-26021, NormalizeFloatingNumbers), and for array
set operations (SPARK-54918). Dynamic partition values are grouping keys as
well. SPARK-26012 fixed the same failure for null and empty strings, which both
go to the default partition, by converting empty strings to null before the
write.
Workaround for the zeros: write {{d + 0.0D}} instead of {{d}}, which turns -0.0
into 0.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
