andygrove commented on code in PR #6943:
URL: https://github.com/apache/datafusion-comet/pull/6943#discussion_r4272653210
##########
spark/src/test/scala/org/apache/comet/exec/CometNativeShuffleSuite.scala:
##########
@@ -1563,6 +1563,30 @@ class CometNativeShuffleSuite extends CometTestBase with
AdaptiveSparkPlanHelper
}
}
+ test("native shuffle stores offsets as lengths only with zstd") {
Review Comment:
Done in 057f7a57ae, and I also did the one-off run.
Both tests now take their rows from a `rowsWithOffsets` helper in
`CometTestBase`, which has offsets in a string, a binary, an array, a map and a
struct column, with nulls, empty strings, empty binary values and null array
elements among them. Each test also checks the answer once more with spilling.
The native test sets `spark.comet.shuffle.native.maxBufferBytes` to 64k and
asserts that the writer spilled, and the columnar test runs at the suite's
threshold of 10 rows through both the bypass-merge and the sort-based writer.
The byte-count comparisons keep the higher threshold. I also ran just the spill
checks against a reader that ignores the lengths flag. Each of them fails with
`First offset ... of Utf8 is larger than values length`, so they do read blocks
that hold lengths.
For the one-off run I made zstd the default codec in `CometTestBase`. All
210 tests in `CometNativeShuffleSuite`, `CometShuffleSuite`,
`DisableAQECometShuffleSuite`, `CometShuffleEncryptionSuite`,
`CometShuffleInputConversionSuite`, `SpillSorterSuite` and
`CometDiskBlockWriterSuite` pass. With the reader that ignores the flag, 107 of
them fail, so that run went through flagged blocks well beyond these two tests.
##########
docs/source/contributor-guide/native_shuffle.md:
##########
Review Comment:
Added in b41b08d5e3, next to `ipc.rs`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
