huangxiaopingRD commented on PR #9154:
URL: https://github.com/apache/paimon/pull/9154#issuecomment-5274934234
This change will render some tables that didn't use the Avro manifest format
unreadable. Do we need to ensure compatibility when reading historical tables?
Below is an exception that occurred when reading a table with an ORC
manifest format.
```
Caused by:
org.apache.paimon.shade.org.apache.avro.InvalidAvroMagicException: Not an Avro
data file
at
org.apache.paimon.shade.org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:79)
at
org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.createReaderFromPath(AvroBulkFormat.java:87)
at
org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.<init>(AvroBulkFormat.java:71)
at
org.apache.paimon.format.avro.AvroBulkFormat.createReader(AvroBulkFormat.java:55)
at
org.apache.paimon.utils.FileUtils.createFormatReader(FileUtils.java:144)
at org.apache.paimon.utils.ObjectsFile.lambda$new$0(ObjectsFile.java:71)
at
org.apache.paimon.utils.ObjectsFile.createIterator(ObjectsFile.java:226)
at
org.apache.paimon.utils.SimpleObjectsCache.createSegments(SimpleObjectsCache.java:71)
at org.apache.paimon.utils.ObjectsCache.read(ObjectsCache.java:88)
at
org.apache.paimon.utils.ObjectsFile.readWithIOException(ObjectsFile.java:179)
at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:156)
at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:127)
at
org.apache.paimon.manifest.ManifestList.readDataManifests(ManifestList.java:89)
at
org.apache.paimon.operation.ManifestsReader.readManifests(ManifestsReader.java:161)
at
org.apache.paimon.operation.ManifestsReader.read(ManifestsReader.java:144)
at
org.apache.paimon.operation.AbstractFileStoreScan.readManifests(AbstractFileStoreScan.java:462)
at
org.apache.paimon.operation.AbstractFileStoreScan.plan(AbstractFileStoreScan.java:299)
at
org.apache.paimon.table.source.snapshot.SnapshotReaderImpl.read(SnapshotReaderImpl.java:394)
at
org.apache.paimon.table.source.snapshot.ReadPlanStartingScanner.scan(ReadPlanStartingScanner.java:45)
at
org.apache.paimon.table.source.AbstractBatchTableScan.lambda$planWithoutAuth$1(AbstractBatchTableScan.java:134)
at java.base/java.util.Optional.orElseGet(Optional.java:364)
at
org.apache.paimon.table.source.AbstractBatchTableScan.planWithoutAuth(AbstractBatchTableScan.java:134)
at
org.apache.paimon.table.source.AbstractDataTableScan.plan(AbstractDataTableScan.java:113)
at
org.apache.paimon.spark.PaimonBaseScan.getInputSplits(PaimonBaseScan.scala:58)
at
org.apache.paimon.spark.read.PaimonSupportsRuntimeFiltering.inputSplits(PaimonSupportsRuntimeFiltering.scala:43)
at
org.apache.paimon.spark.read.PaimonSupportsRuntimeFiltering.inputSplits$(PaimonSupportsRuntimeFiltering.scala:41)
at
org.apache.paimon.spark.PaimonBaseScan.inputSplits(PaimonBaseScan.scala:42)
at
org.apache.paimon.spark.read.BaseScan.estimateStatistics(BaseScan.scala:152)
at
org.apache.paimon.spark.read.BaseScan.estimateStatistics$(BaseScan.scala:150)
at
org.apache.paimon.spark.PaimonBaseScan.estimateStatistics(PaimonBaseScan.scala:167)
at
org.apache.spark.sql.execution.datasources.v2.DescribeTableExec.addTableStats(DescribeTableExec.scala:146)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
