0dunay0 commented on PR #10673:
URL: https://github.com/apache/paimon/pull/10673#issuecomment-6582381179

   > There is one additional data-completeness blocker in the newly enabled 
ORC/Parquet copy path: partition enumeration is silently capped at 32,767 
before applying the partition predicate.
   
   > Please retain efficient flat prefix listing for object stores, while 
pruning hidden directories before descending on hierarchical filesystems
   
   Thanks, both are fixed:
   - All partitions are listed before the predicate is applied.
   - Object stores keep the flat prefix listing. Other file systems skip hidden 
directories before entering them. With a resolving or caching FileIO, the 
choice follows the file system of the partition path.
   
   I also allowed extra nullable fields in structs inside arrays and maps of an 
existing target. Could you take another look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to