[ https://issues.apache.org/jira/browse/KAFKA-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Cai updated KAFKA-3904: ----------------------------- Description: I noticed when my application was running long (> 1 day), I will get 'Too many open files' error. I used 'lsof' to list all the file descriptors used by the process, it's over 32K, but most of them belongs to the .lock file, e.g. this same lock file shows 2700 times. I looked at the code, I think the problem is in: File lockFile = new File(stateDir, ProcessorStateManager.LOCK_FILE_NAME); FileChannel channel = new RandomAccessFile(lockFile, "rw").getChannel(); Each time new RandomAccessFile is called, a new fd will be created, we probably should either close or reuse this RandomAccessFile object. lsof result: java 14799 hcai *740u REG 9,0 0 2415928585 /mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock java 14799 hcai *743u REG 9,0 0 2415928585 /mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock java 14799 hcai *746u REG 9,0 0 2415928585 /mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock java 14799 hcai *755u REG 9,0 0 2415928585 /mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock hcai@teststream02001:~$ lsof -p 14799 | grep lock | grep 0_16 | wc 2709 24381 319662 was: Today most of the rocksDB configs are hard written inside {{RocksDBStore}}, or the default values are directly used. We need to make them configurable for advanced users. For example, some default values may not work perfectly for some scenarios: https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576 One way of doing that is to introduce a "RocksDBStoreConfigs" objects similar to "StreamsConfig", which defines all related rocksDB options configs, that can be passed as key-value pairs to "StreamsConfig". > File descriptor leaking (Too many open files) for long running stream process > ----------------------------------------------------------------------------- > > Key: KAFKA-3904 > URL: https://issues.apache.org/jira/browse/KAFKA-3904 > Project: Kafka > Issue Type: Bug > Components: streams > Reporter: Henry Cai > Assignee: Henry Cai > Labels: api, newbie > > I noticed when my application was running long (> 1 day), I will get 'Too > many open files' error. > I used 'lsof' to list all the file descriptors used by the process, it's over > 32K, but most of them belongs to the .lock file, e.g. this same lock file > shows 2700 times. > I looked at the code, I think the problem is in: > File lockFile = new File(stateDir, ProcessorStateManager.LOCK_FILE_NAME); > FileChannel channel = new RandomAccessFile(lockFile, "rw").getChannel(); > Each time new RandomAccessFile is called, a new fd will be created, we > probably should either close or reuse this RandomAccessFile object. > lsof result: > java 14799 hcai *740u REG 9,0 0 2415928585 > /mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock > java 14799 hcai *743u REG 9,0 0 2415928585 > /mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock > java 14799 hcai *746u REG 9,0 0 2415928585 > /mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock > java 14799 hcai *755u REG 9,0 0 2415928585 > /mnt/stream/join/rocksdb/ads-demo-30/0_16/.lock > hcai@teststream02001:~$ lsof -p 14799 | grep lock | grep 0_16 | wc > 2709 24381 319662 -- This message was sent by Atlassian JIRA (v6.3.4#6332)