mboehm7 commented on PR #1697: URL: https://github.com/apache/systemds/pull/1697#issuecomment-1250322402
Well, regarding the overall design I would recommend to follow the existing binary format. We write sequence files of key-value (index-block) pairs from both local and distributed writers such that the files can be read in any execution mode. Right now it seems you directly serialize the entire block, similar to what the buffer pool eviction did. A version ID at the binning of the file/blocks is fine but we should strive to keep the file layout static, except for new encoding schemes this should be possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
