quenlang commented on issue #7824: Kafka index service use a lot of direct memory during segment publish URL: https://github.com/apache/incubator-druid/issues/7824#issuecomment-502035506 I found a lot of ```Allocating new littleEndByteBuf``` operations in the log. Finally, allocated 333023 littleEndByteBuf and the usage of direct memory arrived 18GB, and OS killed this task. The ingested data was very small, just 2GB in six hours, and all the dimensions have very low cardinality. Why the task need so many direct memory? @himanshug Can you give me some advices? Thank you so much! ``` 2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,020] 2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,021] 2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,022] 2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,023] 2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,024] 2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,025] 2019-06-14T06:24:34,446 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,026] 2019-06-14T06:24:34,446 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,027] 2019-06-14T06:24:34,446 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,028] 2019-06-14T06:24:34,446 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,029] ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
