peter-toth commented on code in PR #59639:
URL: https://github.com/apache/spark/pull/59639#discussion_r4339162609
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:
##########
@@ -513,6 +568,16 @@ protected boolean loadNextRowGroup() throws IOException {
continue;
}
+ // A spill has the rest of a spliced row group read its keys again, in
part. That needs their
+ // offset indexes. Parquet builds a block's store once, over the columns
requested then, so it
+ // is built here over every projected column. Only a row group phase 2
reads in part can be
+ // spilled. Phase 2 reads its store anyway, and gives the filter up on
an unreadable index.
+ spillableRowGroup =
+ spliceCurrentRowGroup && accountKeys && finalRowCount <
blockRowCount;
+ if (spillableRowGroup) {
Review Comment:
Gone with the spill in
[778c14954a9](https://github.com/apache/spark/pull/59639/commits/778c14954a9ede1af0fc6357b0b5546322e69a23).
Phase 2 builds the store over its own columns again, as before this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
