Eric Newton created ACCUMULO-4120:
-------------------------------------
Summary: large root tablet causes system failure
Key: ACCUMULO-4120
URL: https://issues.apache.org/jira/browse/ACCUMULO-4120
Project: Accumulo
Issue Type: Bug
Components: tserver
Affects Versions: 1.7.0, 1.6.4
Environment: 300 node test cluster
Reporter: Eric Newton
Fix For: 1.8.0
On a large test cluster, a table was storing very large row keys that were
similar for 1 - 10 M per row id (which, is a bad schema... but not the problem,
yet).
Large row keys made the tablet large, so it split. And the first 1-10M of the
row keys were stored in the metadata table.
The metadata table has a small split size, so it split.
This ended up recording several keys in the root tablet that were very large.
For example, a single metadata table file was 700M (compressed) and contained
34 keys.
The problem is that *everyone* wants to read the root tablet to find the
metadata tablets. And that was causing the tablet server hosting the tablet to
run out of heap.
Possible solution: bring down the cluster, put it in "safe mode" where only the
metadata table is brought online. Raise the split size of the metadata table to
something large (1G?). Merge the metadata table which should remove the large
records from the root tablet.
There's a utility (SplitLarge) than can be used to remove large keys from the
RFiles of the offending table. Once the ridiculous keys are stripped out the
table can be brought online and merged, which will remove the large keys from
the metadata table.
As long is this is done on a small number of nodes, the servers should have
enough memory to satisfy the requests to perform the metadata table queries and
updates.
We may want to consider adding key size to the metadata table constraint to
prevent these things in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
