ThinkerLei commented on PR #6384: URL: https://github.com/apache/hudi/pull/6384#issuecomment-1501394656
Now, I did a simple memory overhead test for this change .Some related information, please see the attachment of [HUDI-4613](https://issues.apache.org/jira/browse/HUDI-4613). This change will add about 300～400Byte memory overhead for each logFile. And the FileGroupsCreationTime will decrease from tens of seconds to less than one second ( about 60000 logfiles). FileGroupsCreationTime with this change: view.AbstractTableFileSystemView: addFilesToView: NumFiles=59590, NumFileGroups=400, FileGroupsCreationTime=853, StoreTimeTaken=2 FileGroupsCreationTime without this change: addFilesToView: NumFiles=59961, NumFileGroups=400, FileGroupsCreationTime=11709, StoreTimeTaken=2 @danny0405 @yihua -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
