ashokraminedi commented on code in PR #20193:
URL: https://github.com/apache/hudi/pull/20193#discussion_r4214161042
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantShreddingMixedLayouts.scala:
##########
@@ -942,11 +942,9 @@ class TestVariantShreddingMixedLayouts extends
HoodieSparkSqlTestBase with Varia
checkNestedExceptionContains(
() => spark.sql(s"select id, cast(v as string), note from
$tableName").collect())(
"pushVariantIntoScan")
- // The guard's empty-projection carve-out: count(*) reads no column
data, and the query
- // schema it would be checked against is the UNPRUNED table schema,
variant included, so
- // the guard must not run at all. Drop the requiredSchema.nonEmpty
gate at any of its five
- // sites and this count fails on the shredded file.
- checkAnswer(s"select count(*) from $tableName")(Seq(1))
+ // count(*) is deliberately not pinned: it fails on any schema-on-read
variant table once
+ // an internal schema is committed (#20139). Previously this stayed
green only because #20140
+ // leaked the query-local vectorization fallback into the Spark
session.
Review Comment:
Thanks for catching this. I've restored the `count(*)` assertion under the
row-based reader configuration and updated the comments as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
