SEPURI-SAI-KRISHNA commented on code in PR #59583:
URL: https://github.com/apache/spark/pull/59583#discussion_r4311460355
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/V1WriteCommandSuite.scala:
##########
@@ -227,6 +227,25 @@ class V1WriteCommandSuite extends SharedSparkSession with
V1WriteCommandSuiteBas
}
}
+ test("v1 write with already sorted plan - floating-point partition column") {
+ withPlannedWrite { enabled =>
+ withTable("t") {
+ sql(
+ """
+ |CREATE TABLE t(i INT, k STRING) USING PARQUET
+ |PARTITIONED BY (j DOUBLE)
+ |""".stripMargin)
+ // Normalizing the partition values keeps them in order, so they are
not sorted again.
+ executeAndCheckOrderingAndCustomValidate(
+ hasLogicalSort = true, orderingMatched = Some(true)) {
+ sql("INSERT INTO t SELECT i, k, CAST(j AS DOUBLE) AS j FROM t0 ORDER
BY j")
Review Comment:
Thanks, good catch. The value test now also writes the rows sorted by `p`
(`sortWithinPartitions`), and asserts that the input ordering matched, so that
the writer takes that branch with 0.0 and -0.0 still interleaved. With
normalization removed only in that branch of `FileFormatWriter`, the old test
still passed and the new one fails with the `FileAlreadyExistsException`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
