Hrm, looking at the code, it appears that the !METADATA table has some
inconsistencies. A (at least?) one row lacks of the "prev-row" column.
Something like:
some_row ~tab:~pr
Sadly, I'm not positive about the cause given the information. Can you
scan all of the !METADATA table without error?
That said, I don't think that the above would prevent you from being
able to change the iterators on a table because that's stored purely in
ZooKeeper. Maybe it tries to read some data from !METADATA before doing
the ZooKeeper update?
On 2/26/14, 11:49 PM, Russ Weeks wrote:
Hi, Josh,
Thanks for responding so quickly... sorry, I should've mentioned that
I'm running Accumulo 1.5.0.
Nothing really helpful that I can see in the monitor or debug monitor
logs. It just interleaves the 2 errors above from the
state.MetaDataTableScanner and util.TabletIterator loggers.
-Russ
On Wed, Feb 26, 2014 at 8:37 PM, Josh Elser <[email protected]
<mailto:[email protected]>> wrote:
Hi Russ,
What version of Accumulo are you running?
Any other exceptions in the Monitor?
- Josh
On 2/26/14, 11:10 PM, Russ Weeks wrote:
Hi, All,
I can't delete any tables through the accumulo shell, I keep
getting the
message: "Internal error processing waitForTableOperation".
Recently my accumulo cluster has started showing this error
repeatedly
in the master log:
[state.MetaDataTableScanner] WARN : No prev-row for key extent:
{z< : []
9223372036854775807
false= loc243f8e8ad6f5875>. 10.128.1.17:9997
<http://10.128.1.17:9997> ~tab ~pr-� �}
And this error in the monitor log:
[util.TabletIterator] WARN : gc:agile01 Metadata problem :
tablet !0;!0<
has no prev end row
And this in the GC log:
[util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no
prev end row
[util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : []
9223372036854775807 false,~ : [] 9223372036854775807 false)
Maybe coincidence, but it seems like things started getting
weird when I
added a WholeRowIterator through the accumulo shell (which is
nonsense,
I know). The iterators on one of my tables (36M entries) look
like this:
- Iterator WholeRowIterator, scan scope options:
- iteratorPriority = 50
- iteratorClassName =
org.apache.accumulo.core.__iterators.user.__WholeRowIterator
- Iterator rowdel, majc scope options:
- iteratorPriority = 30
- iteratorClassName =
org.apache.accumulo.core.__iterators.user.__RowDeletingIterator
- Iterator transformer, majc scope options:
- iteratorPriority = 15
- iteratorClassName =
com.phemi.agile.datapolicy.__DataColumnTransformer
- maxBufferSize = 256M
- Iterator vers, majc scope options:
- iteratorPriority = 20
- iteratorClassName =
org.apache.accumulo.core.__iterators.user.__VersioningIterator
- maxVersions = 1
- Iterator vers, minc scope options:
- iteratorPriority = 20
- iteratorClassName =
org.apache.accumulo.core.__iterators.user.__VersioningIterator
- maxVersions = 1
- Iterator vers, scan scope options:
- iteratorPriority = 20
- iteratorClassName =
org.apache.accumulo.core.__iterators.user.__VersioningIterator
- maxVersions = 1
I tried deleting DataColumnTransformer, RowDeletingIterator and
WholeRowIterator. No errors were returned from the deleteiter
command in
the shell but after running listiter, the WholeRowIterator and
RowDeletingIterator are still there.
I *am* still able to create tables, but tables that I create
can't then
be deleted.
Any advice appreciated,
-Russ
