milleruntime opened a new issue #1809:
URL: https://github.com/apache/accumulo/issues/1809
**Describe the bug**
When a table is created, Accumulo creates around 20 Zookeeper watchers
associated with that table. This is with the default configuration for that
table. The more properties and iterators that are configured on the table, the
more watchers that will exist. When the table is dropped, anywhere from 8 to
15 watchers will persist indefinitely. The only way for ZK to drop these
watchers is for a restart of the server persisting the connections (ZK, tserver
or master). This becomes a problem on a large cluster with a lot of tables
being created and deleted as ZK will eventually become inoperable. Restarting
ZK or the master is not always advisable since this can lead to more problems
on an active cluster.
**Versions (OS, Maven, Java, and others, as appropriate):**
- Affected version(s) of this project: 1.10, 2.0, 2.1
**To Reproduce**
1. Start up a cluster using [Uno](https://github.com/apache/fluo-uno) and
have netcat installed for running ZK four letter commands. You will probably
have modify the ZK whitelist in zoo.cfg. For example, `vi
<uno_home>/install/apache-zookeeper-3.6.1-bin/conf/zoo.cfg` and modify the
property: `4lw.commands.whitelist=*`.
2. Create a table. For example, `accumulo shell -e "createtable test"`
3. Get the table ID for that table: `accumulo shell -e "tables -l"`
4. Get a count of the number of watchers associated with that table ID. For
table ID=4:
`echo wchp | nc localhost 2181 | grep "tables/4"`. This returned a count =
23
5. Drop the table: `accumulo shell -e "droptable test -f"`
6. Get the number of watchers again for that table ID. Command returned a
count = 15
**Expected behavior**
ZK Watchers associated with a table should be dropped when the table is
deleted.
**Additional context**
There is a very good chance this will be fixed with the 2.1 change #1454.
But until that change is made, this is a critical bug in 1.10 and 2.0.
The watchers that persist seem to be associated with table configuration:
<pre>
/accumulo/d12e80e5-3008-43af-b050-195094437b44/tables/p/conf/table.split.threshold
/accumulo/d12e80e5-3008-43af-b050-195094437b44/tables/p/conf/table.replication
/accumulo/d12e80e5-3008-43af-b050-195094437b44/tables/p/conf/table.balancer
/accumulo/d12e80e5-3008-43af-b050-195094437b44/tables/p/conf/table.groups.enabled
/accumulo/d12e80e5-3008-43af-b050-195094437b44/tables/p/conf/table.compaction.minor.logs.threshold
/accumulo/d12e80e5-3008-43af-b050-195094437b44/tables/p/namespace
/accumulo/d12e80e5-3008-43af-b050-195094437b44/tables/p/conf/table.classpath.context
/accumulo/d12e80e5-3008-43af-b050-195094437b44/tables/p/conf/tserver.dir.memdump
/accumulo/d12e80e5-3008-43af-b050-195094437b44/tables/p/conf/table.compaction.selector
/accumulo/d12e80e5-3008-43af-b050-195094437b44/tables/p/conf/table.majc.compaction.strategy
/accumulo/d12e80e5-3008-43af-b050-195094437b44/tables/p/conf/tserver.walog.max.referenced
/accumulo/d12e80e5-3008-43af-b050-195094437b44/tables/p/conf/table.compaction.dispatcher
/accumulo/d12e80e5-3008-43af-b050-195094437b44/tables/p/conf/table.split.endrow.size.max
/accumulo/d12e80e5-3008-43af-b050-195094437b44/tables/p/name
/accumulo/d12e80e5-3008-43af-b050-195094437b44/tables/p/conf/tserver.memory.maps.native.enabled
</pre>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
