jihoonson commented on issue #7900: Develop a new Indexer process for running ingestion tasks URL: https://github.com/apache/incubator-druid/issues/7900#issuecomment-506847670 @jon-wei nice proposal! Thank you for writing it up. The proposal looks good to me overall. I just have a couple of questions. > * When a merge notice is received, the Indexer will attempt to record two memory allocations in its memory usage tracking state: > * a fixed amount of heap. This can scale based on some proportion of globalIngestionHeapLimitBytes. > * a fixed amount of direct memory, needed for decompression buffers and dictionary conversions. This can scale based on the size in bytes of the fuzzy +1 factor in the familiar (druid.processing.numThreads + druid.processing.numMergeBuffers + 1) * druid.processing.buffer.sizeBytes formula. This sounds like each task needs to reserve some memory up front. How does the indexer schedule the memory reservation request?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
