yuanlihan commented on issue #9918: URL: https://github.com/apache/druid/issues/9918#issuecomment-646401944
> @yuanlihan thanks for your answer, minimise intermediatePersistPeriod will create much more small file @cxlRay that's true. But the temporary persist files will be cleaned up when the hourly tasks finished. And the persisted incremental files/indexes(with extra indexes to speed up query processing) will be more efficient than the in-memory `incrementalIndex`. As far as I know, when a query scan the latest in-memory `incrementalIndex`(like latest 10 minutes's data), Druid processes the in-memory fact table held by `incrementalIndex` row by row. Actually, I had tried to minimise the value of `maxRowsInMemory` to reduce in-memory rows but this will also introduce some overheads caused by frequently persisting. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
