ly923976094 commented on issue #3454: what is the difference between mmap and heap? URL: https://github.com/apache/incubator-pinot/issues/3454#issuecomment-467289319 > Our documentation is still in progress. We expect to finish it by the time we make the first apache release (which should be real soon, once we verify the current candidate in production). Meanwhile, a few comments: > > 1. You seem to be running out of direct memory while loading a completed segment. Changing HEAP to MMAP will help there (assuming you Operating system over-commits mmap memory). Your time column unit is in MILLISECONDS. This can cause dictionary to explode unless you are rounding off the time values. Please check if you are doing so. So, if your millisecond value is 1551113826143 then round it off to (for example) the nearest 5 minute value 1551113710000. You will need to do this before you send your Kafka events, if you have control over it. We are thinking of providing an automatic switch in pinot to do this (i.e. round off milliseonds to nearest 5 mins or 15 mins or hour, whatever) > 2. You may want to try using auto-segment siizing. See https://cwiki.apache.org/confluence/display/PINOT/Consuming+and+Indexing+rows+in+Realtime yes, If I change to MMAP, there is no memory overflow. I want to know the basic difference between MMAP and HEAP. For example: query efficiency, memory usage, etc @mcvsubbu, The unit of time you are talking about is milliseconds causing the dictionary to explode,Is this the cause of memory overflow?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
