HyukjinKwon commented on code in PR #59269:
URL: https://github.com/apache/spark/pull/59269#discussion_r4226608221
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala:
##########
@@ -2051,8 +2468,9 @@ class ParquetStorageFilterSuite extends QueryTest with
SharedSparkSession
.flatMap(_.getEncodings.asScala).toSet
// Reads every batch, projecting each row through `extract`. `storageFilter`
may be null, which
- // builds the plain vectorized reader rather than the late-materialization
one. Every read helper
- // in this suite goes through here.
+ // builds the plain vectorized reader rather than the late-materialization
one. Every helper that
+ // builds a reader by hand goes through here, except `readChecked`. That one
needs the Hadoop
Review Comment:
**Nit (P3):** `readChecked` no longer builds a reader by hand. Since the
round-1 refactor it calls `drain(formatReader(schema, Nil,
storageFilters)(file))`, which goes through
`ParquetFileFormat.buildReaderWithStorageFilters` the same way `readUnderCap`
does, and the `hadoopSplit`/Hadoop-`initialize` setup this sentence describes
is gone. Naming it as the one hand-built exception points readers at a setup
that no longer exists. Could this drop the `readChecked` carve-out? Or it could
say that `readChecked`, like the other format-based helpers, reads through
`formatReader`, because its requested schema holds a column the file does not
have.
**Verification:**
- **Inspection:** The comment's split between helpers that build readers by
hand and helpers that build through the format matches the helper definitions
in the suite, and `readChecked` is classed as format-based.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
