Copilot commented on code in PR #13036:
URL: https://github.com/apache/gluten/pull/13036#discussion_r4023948108
##########
.github/workflows/util/delta-spark-ut/setup-delta.sh:
##########
@@ -172,6 +172,33 @@ cherry_pick_delta_fix
46bd45d57eadd7e528002a0ae7bd36ce5a456eca "#7104 (ScanRepor
cherry_pick_delta_fix 959e00e15f41f56afc1c9bb95d160c55c6dc7068 "#7105 (9 more
test suites)"
echo "::endgroup::"
+echo "::group::Capping DeltaParquetFileFormat fixture row groups by row count"
+# DeltaParquetFileFormatSuite generates one 20,000-row Parquet file and sets a
+# 50 KiB block size to ensure that it contains multiple row groups. Velox sizes
+# row groups by compressed buffered bytes, so the highly compressible integer
+# fixture remains a single row group and the DV read tests fail before reaching
+# their assertions. Gluten's native writer also supports parquet.block.rows;
+# cap the fixture at 10,000 rows to create at least two row groups regardless
of
+# compression while keeping the native write path enabled.
+DPFFS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala"
+if [ ! -f "$DPFFS" ]; then
+ echo "Expected file not found in Delta clone: $DPFFS" >&2
+ echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
>&2
+ exit 1
+fi
+sed -i '/hadoopConf().set("parquet.block.size", (1024 \* 50).toString)/a\
+ hadoopConf().set("parquet.block.rows", "10000")' "$DPFFS"
+ROW_CAPS=$(grep -c 'hadoopConf().set("parquet.block.rows", "10000")' "$DPFFS"
|| true)
Review Comment:
This guard counts the row-cap text anywhere in the file, not whether it was
inserted immediately after the expected block-size setting. If Delta moves that
setting while an unrelated occurrence of the same row-cap line remains
elsewhere, `sed` matches nothing and setup still succeeds without capping
`generateData`, silently restoring the single-row-group failure. Validate the
exact adjacent code lines (and their line shape) instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
