ConeyLiu commented on PR #3136:
URL: https://github.com/apache/parquet-java/pull/3136#issuecomment-2611997502
@dylanburati I have tried to support the footer with an unsigned int size in
Java implementation. However, due to some limitations, it can not be done. We
could write a footer(without encrypted) size larger than Int.MaxValue bytes,
while we can not read it due to some limitations in thrift.
```java
java.io.IOException: can not read class
org.apache.parquet.format.FileMetaData: MaxMessageSize reached
at org.apache.parquet.format.Util.read(Util.java:393)
at org.apache.parquet.format.Util.readFileMetaData(Util.java:156)
at
org.apache.parquet.format.converter.ParquetMetadataConverter$3.visit(ParquetMetadataConverter.java:1545)
at
org.apache.parquet.format.converter.ParquetMetadataConverter$3.visit(ParquetMetadataConverter.java:1542)
at
org.apache.parquet.format.converter.ParquetMetadataConverter$NoFilter.accept(ParquetMetadataConverter.java:1242)
at
org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:1542)
at
org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:637)
at
org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:571)
Caused by: org.apache.thrift.transport.TTransportException: MaxMessageSize
reached
at
org.apache.thrift.transport.TEndpointTransport.checkReadBytesAvailable(TEndpointTransport.java:94)
at
org.apache.thrift.protocol.TCompactProtocol.checkStringReadLength(TCompactProtocol.java:726)
at
org.apache.thrift.protocol.TCompactProtocol.readString(TCompactProtocol.java:675)
at
org.apache.parquet.format.InterningProtocol.readString(InterningProtocol.java:269)
at
org.apache.parquet.format.KeyValue$KeyValueStandardScheme.read(KeyValue.java:415)
at
org.apache.parquet.format.KeyValue$KeyValueStandardScheme.read(KeyValue.java:392)
at org.apache.parquet.format.KeyValue.read(KeyValue.java:327)
at
org.apache.parquet.format.FileMetaData$FileMetaDataStandardScheme.read(FileMetaData.java:1338)
```
Since it is a rare condition. Here we just fail fast with an exception.
cc @wgtmac
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
