HyukjinKwon commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548374525
##########
File path:
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit,
TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
+ "rowGroupOffsets in ParquetInputSplit should be null.");
this.file = split.getPath();
- long[] rowGroupOffsets = split.getRowGroupOffsets();
-
- ParquetMetadata footer;
- List<BlockMetaData> blocks;
-
- // if task.side.metadata is set, rowGroupOffsets is null
- if (rowGroupOffsets == null) {
Review comment:
After rethinking @LuciferYang, actually this is aimed to be ported to
Apache Parquet, see the comments above:
```
* TODO: move this to the parquet-mr project. There are performance benefits
of doing it
* this way, albeit at a higher cost to implement. This base class is
reusable.
```
It's being tracked at
[PARQUET-131](https://issues.apache.org/jira/browse/PARQUET-131). Shall we just
simply add a comment that this code is not used in Apache Spark for now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
