tanishq-chugh commented on code in PR #6506:
URL: https://github.com/apache/hive/pull/6506#discussion_r3545371320
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java:
##########
@@ -164,18 +164,29 @@ protected ParquetInputSplit getSplit(
legacyConversionEnabled =
DataWritableReadSupport.getZoneConversionLegacy(fileMetaData.getKeyValueMetaData(),
conf);
- split = new ParquetInputSplit(finalPath,
- splitStart,
- splitLength,
- fileSplit.getLocations(),
- filteredBlocks,
- readContext.getRequestedSchema().toString(),
- fileMetaData.getSchema().toString(),
- fileMetaData.getKeyValueMetaData(),
- readContext.getReadSupportMetadata());
+ split = buildParquetInputSplit(finalPath,
readContext.getRequestedSchema());
return split;
}
+ private ParquetInputSplit buildParquetInputSplit(final Path finalPath, final
MessageType requestedSchema)
Review Comment:
The [old
constructor](https://github.com/apache/parquet-java/blob/apache-parquet-1.16.0/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetInputSplit.java#L80-L91)
expected the schema in string, which was internally parsed in
ParquetInputSplit.end() using:
`MessageTypeParser.parseMessageType(requestedSchema)`
running into the original parsing issue for Read, where parquet interpreted
the identifier inside the parentheses to be datatype.
Have changed it to compute the size & offset in `buildParquetInputSplit()`
using our `MessageType requestedSchema` object avoiding string schema parsing
and call the new constructor which is the same functionality of the old
constructor (It also computed the size/offset and then calls this new
constructor internally).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
