Hi Kudu devs! There've been a few discussions recently regarding changes to Kudu's metadata storage. There are a number of areas that could benefit from improving this layer, and I've been coalescing some of these ideas to lock down what changes make sense in the near future. Here's a list of a few considerations:
https://docs.google.com/document/d/1jXFqIZvLwkkmSjLC0wy-mDA0l0GmVuZJUMUB_7q5QUE/edit?usp=sharing To summarize some recent discussions, there are a few general changes that would be beneficial, in order of implementation difficulty: 1. Moving the metadata to the WAL directory (KUDU-1489) 2. Batch commit of metadata (KUDU-2204) 3. Mirroring of metadata (KUDU-2117) 1. and 3. mainly serve to reduce the points of failure (although 1. also has performance benefits if the WAL is SSD-mounted), while 2. could solve potential scalability issues with elections, update-heavy workloads, etc. Empirically, scalability is not as big of an adoption-bottleneck as it was before. Additionally, it's not clear that the listed scalability issues are the biggest bottlenecks to larger data volumes. Moving forward, we should keep track of user stories that would benefit from such improvements. There hasn't been a huge amount of noise regarding the two remaining points of failure (the WALs directory and metadata directory); it's unclear whether this is due to an acceptance of the fact, or due to the recency of the release (Kudu 1.6). In any case, the simplicity of 1. makes it the more likely candidate in the nearer future, provided it doesn't preclude future work on 2. or 3. As always, backwards compatibility and downgrades are concerns, and of the three options, 1. is by far the easiest, as (unlike the others) it will likely not change any on-disk formatting. That said, we'll have to put in some thought moving forward, e.g. if we do end up wanting to implement 2. and 3., the upgrade paths would be a bit trickier with the added configuration. With these points in mind, it seems the reasonable path forward is go with 1. and introduce a flag for users to colocate WALs and metadata. If you have thoughts or comments on the matter, feel free to respond! Best, Andrew
