felipepessoto commented on code in PR #13036:
URL: https://github.com/apache/gluten/pull/13036#discussion_r4032776678
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -172,6 +172,59 @@ cherry_pick_delta_fix
46bd45d57eadd7e528002a0ae7bd36ce5a456eca "#7104 (ScanRepor
cherry_pick_delta_fix 959e00e15f41f56afc1c9bb95d160c55c6dc7068 "#7105 (9 more
test suites)"
echo "::endgroup::"
+echo "::group::Capping DeltaParquetFileFormat fixture row groups by row count"
+# DeltaParquetFileFormatSuite generates one 20,000-row Parquet file and sets a
+# 50 KiB block size to ensure that it contains multiple row groups. Velox sizes
+# row groups by buffered bytes after writing each input batch. Because this
+# fixture arrives in one batch, lowering the byte threshold cannot split it.
+# Scope Gluten's native row-count limit around the fixture write so Arrow
splits
+# the 20,000 rows deterministically while keeping the native write path
enabled.
+DPFFS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala"
+if [ ! -f "$DPFFS" ]; then
+ echo "Expected file not found in Delta clone: $DPFFS" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
>&2
+ exit 1
+fi
+if ! sed 's/^__BLANK_CONTEXT__$/ /' <<'PATCH' | git -C "$DELTA_DIR" apply -
+diff --git
a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
+---
a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
++++
b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
+@@ -68,9 +68,11 @@ trait DeltaParquetFileFormatSuiteBase
+ protected def generateData(tablePath: String): Unit = {
+ // This is to generate a Parquet file with two row groups
+ hadoopConf().set("parquet.block.size", (1024 * 50).toString)
+__BLANK_CONTEXT__
+ // Keep the number of partitions to 1 to generate a single Parquet data
file
+ val df = Seq.range(0, 20000).toDF().repartition(1)
+- df.write.format("delta").mode("append").save(tablePath)
++ withSQLConf("spark.gluten.sql.native.parquet.write.blockRows" -> "10000")
{
++ df.write.format("delta").mode("append").save(tablePath)
++ }
+__BLANK_CONTEXT__
+ // Set DFS block size to be less than Parquet rowgroup size, to allow
+PATCH
+then
+ echo "ERROR: DeltaParquetFileFormat fixture patch did not apply." >&2
+ echo "The patch expects the Delta v4.2.0 generateData fixture shape;" \
+ "ref '${DELTA_REF}' must remain source-compatible." >&2
+ exit 1
+fi
+ROW_CAP_SCOPES=$(
+ grep -Fxc \
+ ' withSQLConf("spark.gluten.sql.native.parquet.write.blockRows" ->
"10000") {' \
+ "$DPFFS" || true
+)
Review Comment:
The preceding `git apply` already enforces this complete sequence
atomically: the `generateData` method, 50 KiB setting, 20,000-row DataFrame,
fixture write, and following DFS comment are all context in one hunk. If that
fixture is rearranged, the patch fails and setup exits before this
postcondition. This count only verifies the applied result; an unrelated
occurrence cannot make a failed `git apply` succeed, and an additional
occurrence would make the count 2 and fail. Repeating the full-block validation
after an already successful contextual apply would therefore add redundant
parsing without strengthening the guard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
