marsupialtail commented on PR #13662: URL: https://github.com/apache/arrow/pull/13662#issuecomment-1215633181
Actually I misunderstood the requirements. I think that synchronous writing is not required in this use case, and it is okay to use O_DIRECT without O_SYNC, to use the SSD cache to speed up the write. In this case the write will not be persisted but it will be out of the page cache, achieving the objective first listed in the JIRA to reduce memory usage. In this case, O_DIRECT without O_SYNC is nearly 20x faster than fadvise + O_SYNC on my system. fadvise without O_SYNC fails to reduce page cache memory usage. I recommend we revert back to this PR: https://github.com/apache/arrow/pull/13640. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
