HyukjinKwon commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548861926
##########
File path:
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit,
TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
+ "rowGroupOffsets in ParquetInputSplit should be null.");
this.file = split.getPath();
- long[] rowGroupOffsets = split.getRowGroupOffsets();
-
- ParquetMetadata footer;
- List<BlockMetaData> blocks;
-
- // if task.side.metadata is set, rowGroupOffsets is null
- if (rowGroupOffsets == null) {
Review comment:
Yeah, just simply adding a comment because this is planned to move to
Parquet. It's stuck but I think we should port it in long run anyway.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
