I wrote a quick script today to see how kudu behaves if I create many tables. After creating 334 tables, I started getting timeouts. I see this in the master log file:
W0216 11:37:48.961221 49810 catalog_manager.cc:2490] CreateTablet RPC for tablet 9b259d5c5ff74f04820240f2159bc1a0 on TS faaf4e9b6e5945d7a14953c4cc34f164 (telx-sb-dev2:7050) failed: IO error: Couldn't create tablet metadata: Failed to write tablet metadata 9b259d5c5ff74f04820240f2159bc1a0: Call to mkstemp() failed on name template /var/lib/kudu/tserver/tablet-meta/9b259d5c5ff74f04820240f2159bc1a0.tmp.XXXXXX: Too many open files (error 24) I decreased block_manager_max_open_files, but still got the same result. Lsof shows that the open files are for the WAL: kudu-tser 49648 kudu 1021u REG 8,5 67108864 16385457 /var/lib/kudu/tserver/wals/62b73d1b7f7a4e61a0a30a551e66230b/wal-000000001 kudu-tser 49648 kudu 1022r REG 8,5 67108864 16385457 /var/lib/kudu/tserver/wals/62b73d1b7f7a4e61a0a30a551e66230b/wal-000000001 kudu-tser 49648 kudu 1023u REG 8,5 24000000 16385458 /var/lib/kudu/tserver/wals/62b73d1b7f7a4e61a0a30a551e66230b/index.000000000 The files do not get closed until the tables are deleted, even though no running process has any of those tables open. Is there a setting that will reduce the number of WAL files that get created or held open at any given point in time?
