Hello again! Thanks for the tips. I tried checking for fate operations, but there didn't seem to be any hanging around very long. I tried increasing the fate threads, but it didn't help. After that, I tried profiling the manager process, and it seems like the bulk of CPU time is spent talking to zookeeper. Initially, that was in the TableLoadBalancer, but I changed to use the SimpleLoadBalancer and the cpu time shifted to MetadataTableUtils.addTablet/TableZooHelper/ZooCache, where it is populating table ids. Looking in zookeeper, it seems like all the tables end up under /accumulo/<uuid>/tables, which keeps growing in size. CPU doesn't seem particularly high, so I'm not entirely sure this is the culprit. But it seems to me that it's taking an increasingly long time to populate the table cache as the number of zk table nodes increases. Does that seem feasible? Is there anything I can do to mitigate the issue?
Thanks, Emilio Lahr-Vivaz General Atomics, CCRi ________________________________ From: Dave Marion <dlmar...@comcast.net> Sent: Thursday, June 13, 2024 7:56 PM To: user@accumulo.apache.org <user@accumulo.apache.org> Subject: -EXT-Re: debugging slow table creation WARNING: This message is from an external source. Evaluate the message carefully BEFORE clicking on links or opening attachments. Emilio, The create table operation is a Fate operation that runs in the Manager. My immediate thought is that maybe the number of Fate operations that you are creating for your other tables is making the create table operation wait for an available thread. I don't have the code in front of me, but I believe there are Fate commands in the shell and via the admin utility that will let you see the status of the Fate operations. If your create operation is sitting there in a submitted state, then it's waiting for a thread. There is a property that you can modify to increase the number of Fate threads. If it's in the running state for a long time, then stacking the Manager to determine where it's spending its time would help us. Dave On Jun 13, 2024 7:23 PM, "Lahr-Vivaz, Emilio" <emilio.lahr-vi...@ga-ccri.com> wrote: Hello, We've noticed that creating a table in Accumulo 2.1 tends to get slower and slower as the number of tables in the system increases, and once we get have several thousand tables creating more really bogs down (on the order of minutes). Does anyone have any tips on debugging this issue, or known configurations that might help? Or is this not a use case that Accumulo was designed for? I can provide more details on the cluster setup, if it would be helpful. Thanks, Emilio Lahr-Vivaz General Atomics, CCRi
