Any reason they can't share a single RocksDB state backend instance?
On Fri, Apr 28, 2017 at 8:44 AM, Aljoscha Krettek <aljos...@apache.org> wrote: > The problem here is that this will try to open 300 RocksDB instances on > each of the TMs (depending on how the parallelism is spread between the > machines this could be more or less). As the exception says, this will open > too many files because each RocksDB instance has a directory with several > files in it. > > One possible solution would be to increase the limit on open files but I > don’t think that opening 300 RocksDB instances on one machine is a good > idea for any size of machine. I think with this many patterns you could > start thinking about writing the pattern matching yourself and multiplexing > the several patterns in one stateful function or operator. > > @Stefan, what do you think about having this many Rocks instances? >
