kbendick commented on pull request #2680: URL: https://github.com/apache/iceberg/pull/2680#issuecomment-987528140
> We have used RocksDB in the past with Flink, and it does require some tuning to avoid OOM. For example, we have to change the memory allocator to `jemalloc`, because the default `glibc` allocator has major memory fragmentation issues with RocksDB. Furthermore, there are also various settings to control the amount of memory allocated to each tree level, index, cache, etc... It's true that RocksDB introduces more complexity. But it's really a trade off. User's don't necessarily have to opt-in either. 1. It can save a large amount of money for jobs that need a very large amount of memory to keep their entire state in memory. 2. Mostly I've found, at least with Flink, that once the core set of parameters were tuned for our common use cases, we could apply those everywhere and get away with that being "good enough". 3. Tuning very large JVMs, with the amount of memory needed to store equivalent amount of data that can be stored for much less on disk with RocksDB is its own challenge. There's simply no truly escaping a certain level of complexity at a certain scale. Additionally, RocksDB is arguably the most common state backend for Flink. Now, as of Spark 3.2, there's a RocksDB state store for structured streaming contributed by databricks. Given these two things, I don't think that the additional settings available is a huge issue. I think the Flink project has done a lot of work in minimizing the necessary tuning for RocksDB. With time, similar things could be introduced here. But even making this a possibility will be a really large win for very large jobs which might have a large amount of state or need to otherwise spill to disk. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
