Greg Fodor created KAFKA-4317: --------------------------------- Summary: RocksDB checkpoint files lost on kill -9 Key: KAFKA-4317 URL: https://issues.apache.org/jira/browse/KAFKA-4317 Project: Kafka Issue Type: Improvement Components: streams Affects Versions: 0.10.0.1 Reporter: Greg Fodor Assignee: Guozhang Wang
Right now, the checkpoint files for logged RocksDB stores are written during a graceful shutdown, and removed upon restoration. Unfortunately this means that in a scenario where the process is forcibly killed, the checkpoint files are not there, so all RocksDB stores are rematerialized from scratch on the next launch. In a way, this is good, because it simulates bootstrapping a new node (for example, its a good way to see how much I/O is used to rematerialize the stores) however it leads to longer recovery times when a non-graceful shutdown occurs and we want to get the job up and running again. It seems that two possible things to consider: - Simply do not remove checkpoint files on restoring. This way a kill -9 will result in only repeating the restoration of all the data generated in the source topics since the last graceful shutdown. - Continually update the checkpoint files (perhaps on commit) -- this would result in the least amount of overhead/latency in restarting, but the additional complexity may not be worth it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)