GitHub user westonpace added a comment to the discussion: [C++] Footer dumps for IPC Feather/ Parquet
I agree some kind of atomic swap would be required. This would require the footer to be in a separate location as @jayspomodoro mentioned. I suppose one way to think of it is that there is no reason an IPC file couldn't have multiple footers (all stored in various locations) to support reading the file at various points in its creation. > Edit: come to think of it, this seems a lot like the problem Iceberg solves 🤔 Yes, this does sound like creating a mini-table format, although one based on appends instead of multiple files. Note that S3 express and Azure blob storage now both support appends so there could even be some cloud storage solutions where this is useful. Databases that do a lot of ingestion often use some kind of append based WAL. I think they are often built with something like Avro today however. I agree with @amoeba that having a specific use case in mind might be helpful. GitHub link: https://github.com/apache/arrow/discussions/46505#discussioncomment-13206738 ---- This is an automatically sent email for user@arrow.apache.org. To unsubscribe, please send an email to: user-unsubscr...@arrow.apache.org
