0dunay0 commented on PR #10673: URL: https://github.com/apache/paimon/pull/10673#issuecomment-6582381179
> There is one additional data-completeness blocker in the newly enabled ORC/Parquet copy path: partition enumeration is silently capped at 32,767 before applying the partition predicate. > Please retain efficient flat prefix listing for object stores, while pruning hidden directories before descending on hierarchical filesystems Thanks, both are fixed: - All partitions are listed before the predicate is applied. - Object stores keep the flat prefix listing. Other file systems skip hidden directories before entering them. With a resolving or caching FileIO, the choice follows the file system of the partition path. I also allowed extra nullable fields in structs inside arrays and maps of an existing target. Could you take another look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
