viirya commented on code in PR #59117:
URL: https://github.com/apache/spark/pull/59117#discussion_r4254077566
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java:
##########
@@ -381,11 +385,8 @@ private void initDataReader(
this.isCurrentPageDictionaryEncoded = false;
}
- try {
- dataColumn.initFromPage(pageValueCount, in);
- } catch (IOException e) {
- throw new IOException("could not read page in col " + descriptor, e);
- }
+ // readPageV1 and readPageV2 add the page and the column to any error
thrown here.
Review Comment:
Fixed in d18a880: the comment now says that `readPageV1` and `readPageV2`
add the page and column to an `IOException` or a `ParquetDecodingException`.
Turning the `checkArgument` checks in
`VectorizedDeltaBinaryPackedReader.initFromPage` into
`ParquetDecodingException`s is left for SPARK-59854.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java:
##########
@@ -455,6 +456,9 @@ private int readPageV1(DataPageV1 page) throws IOException {
return pageValueCount;
} catch (IOException e) {
throw new IOException("could not read page " + page + " in col " +
descriptor, e);
Review Comment:
Done, the inner message is now "the dictionary was missing for encoding
...", so the page and column are added once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
