kaka11chen opened a new issue #7508: Druid historical node running caused os kernel creating too many kernel's dentry cache slab objects. URL: https://github.com/apache/incubator-druid/issues/7508 Druid historical node running caused os kernel creating too many kernel's dentry cache slab objects. ### Affected Version all versions ### Description Druid historical node running caused os kernel created too many kernel's dentry cache slab objects, run out of memory, and caused performance issue(Use 'sar -B' and found majfit/s high). #### OS Kernel version: 2.6.32-754.9.1.el6.x86_64 #### meminfo: Slab objects occupy 65G, most is dentry. $ cat /proc/meminfo MemTotal: 198340116 kB MemFree: 3411764 kB Buffers: 329324 kB Cached: 50916944 kB SwapCached: 0 kB Active: 81630316 kB Inactive: 44296244 kB Active(anon): 74680324 kB Inactive(anon): 148 kB Active(file): 6949992 kB Inactive(file): 44296096 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 136 kB Writeback: 0 kB AnonPages: 74679564 kB Mapped: 4465992 kB Shmem: 176 kB Slab: 65894636 kB SReclaimable: 40123808 kB SUnreclaim: 25770828 kB KernelStack: 23616 kB PageTables: 216640 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 99170056 kB Committed_AS: 75530612 kB VmallocTotal: 34359738367 kB VmallocUsed: 715408 kB VmallocChunk: 34359015920 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 10508 kB DirectMap2M: 1939456 kB DirectMap1G: 199229440 kB #### slabtop OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 193075920 193075920 100% 0.19K 9653796 20 38615184K dentry 193031130 193031130 99% 0.12K 6434371 30 25737484K size-128 12781095 12779631 99% 0.10K 345435 37 1381740K buffer_head 214746 214542 99% 0.55K 30678 7 122712K radix_tree_node 46860 46802 99% 0.98K 11715 4 46860K ext4_inode_cache #### It will create dentry every second. $ sudo strace -fp 29211 -e trace=open,stat,close,unlink Process 29211 attached with 370 threads [pid 33862] stat("/data/druid/var/tmp/druid-groupBy-2c2e19ef-f883-496f-b88d-10dab759eab0_e0677686-1751-43a5-b244-817accd836f8", 0x7efaa8549a10) = -1 ENOENT (No such file or directory) [pid 33856] stat("/data/druid/var/tmp/druid-groupBy-4033e0b6-2878-4fbc-b941-ab11f075e468_8889a78d-7afb-4290-af6c-5047fd499330", 0x7efaa8b4fb10) = -1 ENOENT (No such file or directory) [pid 33881] stat("/data/druid/var/tmp/druid-groupBy-e944fe64-64b6-42db-96b3-27857fef7dc1_0490c768-3b39-4b37-b413-d84a03de7025", 0x7efa6b1efb80) = -1 ENOENT (No such file or directory) [pid 33936] stat("/data/druid/var/tmp/druid-groupBy-b7452444-8665-482b-bec6-42de3abd756e_800e1931-f294-44b1-b844-e4bbec77aff5", 0x7ef9679f7af0) = -1 ENOENT (No such file or directory) [pid 33890] stat("/data/druid/var/tmp/druid-groupBy-a72b752a-d047-4438-8d3a-8d4f30b98699_7d6382f6-62eb-4e47-85e7-7060014742c2", 0x7efa6a8e6800) = -1 ENOENT (No such file or directory) [pid 33885] stat("/data/druid/var/tmp/druid-groupBy-96f1429b-53e5-4893-a084-19890e2c1ce7_c0a62ded-af6f-4ed2-a486-2489f639795f", 0x7efa6adeb970) = -1 ENOENT (No such file or directory) [pid 33869] stat("/data/druid/var/tmp/druid-groupBy-ad823d55-a654-424c-9f96-6c7cd3fb8cef_03746689-7ea4-4d0a-aaf8-a99d3e17288d", 0x7efa6bdfb980) = -1 ENOENT (No such file or directory) [pid 33878] stat("/data/druid/var/tmp/druid-groupBy-867609dd-82a8-43c3-863d-2fe8672f7937_97d3ebb4-cb0a-4e16-ab51-4a830094334e", 0x7efa6b4f29f0) = -1 ENOENT (No such file or directory) [pid 33880] stat("/data/druid/var/tmp/druid-groupBy-ef4d32c8-a5b9-4557-b712-bfc7f1589cf7_9e82521f-32d5-496a-9c2d-2a9e187e67dd", 0x7efa6b2f0b10) = -1 ENOENT (No such file or directory) [pid 33875] stat("/data/druid/var/tmp/druid-groupBy-f6398511-ed75-4547-8d02-c7004f38c1f1_6078cf5e-ef53-4c60-8f57-6413937a0fed", 0x7efa6b7f5880) = -1 ENOENT (No such file or directory) [pid 33927] stat("/data/druid/var/tmp/druid-groupBy-f3647873-65bc-4d5f-8e3b-ae74781b4601_9154c59f-456a-4d69-a5d9-9ef90d60d1b5", 0x7efa683c1a80) = -1 ENOENT (No such file or directory) [pid 33942] stat("/data/druid/var/tmp/druid-groupBy-c46029f8-1a26-4782-9413-6af301907289_14b7d398-b600-45c3-9861-46306f20e7ee", 0x7ef9673f1a10) = -1 ENOENT (No such file or directory) [pid 33861] stat("/data/druid/var/tmp/druid-groupBy-ce632ef2-25a6-4ddb-b09e-67acd1708497_7d188a95-1fb6-4a50-99a3-73ee49d08f53", 0x7efaa864a990) = -1 ENOENT (No such file or directory) [pid 33897] stat("/data/druid/var/tmp/druid-groupBy-963f0eb8-cdae-4fd4-a119-1152db5effd3_2f33c650-7f91-4e7f-a7b2-ba992a328461", 0x7efa6a1dfb90) = -1 ENOENT (No such file or directory) [pid 33867] stat("/data/druid/var/tmp/druid-groupBy-d410a69b-a283-475d-a977-f4a5e04ad47a_431f471e-e02a-4fc1-a7ed-ab658215575a", 0x7efa6bffd880) = -1 ENOENT (No such file or directory) [pid 33903] stat("/data/druid/var/tmp/druid-groupBy-821bd461-493a-4b8d-a948-96733175c553_fdadd570-0dd5-4e97-b7e9-cefc8e06cc15", 0x7efa69bd9a90) = -1 ENOENT (No such file or directory) [pid 33899] stat("/data/druid/var/tmp/druid-groupBy-e4133c0c-e2ef-46ad-9533-ead801d83491_9e1ababa-4baf-4858-a332-a080bd70a97d", 0x7efa69fdd880) = -1 ENOENT (No such file or directory) ### Root Cause In the LimitedTemporaryStorage::close() method, if will check directory existence whatever the directory if created @Override public void close() { synchronized (files) { if (closed) { return; } closed = true; for (File file : ImmutableSet.copyOf(files)) { delete(file); } files.clear(); if (storageDirectory.exists() && !storageDirectory.delete()) { log.warn("Cannot delete storageDirectory: %s", storageDirectory); } } } ### Solution I will add a flag to check if the directory created.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
