GrigorievNick opened a new issue #2903: URL: https://github.com/apache/iceberg/issues/2903
Iceberg manages file location in metadata, so there is no reason to keep hive table file structure. But iceberg still writes data in partition per folder. In my case partitions are organized as ranges and my storage is s3. One of the main issues, that sometimes I need to split ranges into two or coalesce them. So because it's ranged, I actually need only split one-two files on the partition border. But because S3 does not support rename, if the partition is part of the prefix, I will need to copy all data in the partition. Iceberg is a great tool to manage files and looks like its architecture does not require a strict file folder hierarchy. So I wonder do there is a way to say iceberg always writes all files to the same folder? ``` /tmp/iceberg_cdc_test/iceberg_catalog/hdl/enrichment_table/data/idRange=0-50/ts_day=2021-07-30/00000-8-8b701a28-8a19-4b57-a84e-f2ff5b12bbb6-00001.orc ``` Example of files written by an iceberg in the partition. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
