bvaradar commented on issue #2201: URL: https://github.com/apache/hudi/issues/2201#issuecomment-721211608
@getniz : Let me clarify what I mentioned by physical memory not getting affected. Normal Queries will not be reading data from older versions of the file. So, from data perspective, there is no overhead. Keeping tons of versions though could potentially slow down the query as the file listing might take a long time. In terms of metadata memory, the default implementation retains file metadata (of all versions) in memory. So, you will see a corresponding increase although I expect it to scale with number of files * versions. But, Hudi also supports other file metadata stores (using rocks-db : hoodie.filesystem.view.type=EMBEDDED_KV_STORE and hoodie.embed.timeline.server=true) if the default setup does not scale for you. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
