findingrish commented on issue #13547: URL: https://github.com/apache/druid/issues/13547#issuecomment-1352803429
> Use 512MB heap per task (or even 256MB) rather than 1GB when CPU-to-memory ratio is higher, so we can get more tasks and therefore use more processors. This would work for both MM and Indexer. Task memory is determined using the total memory allocated for tasks. It could be 512m/1024m/2048m as of now. Ingestion of `trips_60m` requires min 1g task memory. So we allocate 512m for tasks only when tasks memory is very less (< 2g) > Allow TASKS to use some memory that we had set aside for segment page cache. Currently, that's half the memory of the server, as long as it's above a minimum threshold. The beauty of TASKS is that they only use memory if they are actually launched. So, it's OK to eat into that 50% buffer; we'll only actually use it when all task slots are actually full. Then, when they're not, it all becomes available for page cache again. (This doesn't apply to INDEXER, where the memory is always used. Can't pull the same trick here.) We are using 80% of total memory for the services (up from 50% https://github.com/apache/druid/pull/13563). > Cap druid.worker.capacity at the number of processors; doesn't help much to have it be higher. This is already done https://github.com/apache/druid/blob/master/examples/bin/start-druid-main.py#L467 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
