GitHub user jayspomodoro added a comment to the discussion: [C++] Footer dumps for IPC Feather/ Parquet
The end to end workflow that I am using currently: Think of three files - update_file - always readable main_file - file in which recordbatches get appended footer_file - contains the footer data At any moment I have to dump new data (recordbatch), I write it in the main_file as usual. I dump the new footer in the footer_file, overwriting the entire contents. I maintain the offset of the main_file before this update. I seek to this offset in the update_file, overwrite the original footer metadata with the new recordbatch just by copying it, and then copy the new footer on top of it. This maintains the structure of the feather/parquet at the end of these steps. Of course I had to maintain flocks for synchronous read and writes. GitHub link: https://github.com/apache/arrow/discussions/46505#discussioncomment-13209070 ---- This is an automatically sent email for user@arrow.apache.org. To unsubscribe, please send an email to: user-unsubscr...@arrow.apache.org
