Hi, Keith, Thanks for your response. I think that's exactly what happened. I screwed up the config command and wound up specifying a couple of iterators on the !METADATA table (and indeed all tables in the system).
I've removed those iterators using "config -d" and now I can scan the !METADATA table without errors in the shell or the master log. But I'm still not able to delete tables. When I run the command "deletetable test_table", I see this message in the shell: ERROR: org.apache.accumulo.core.client.AccumuloException: Internal error processing waitForTableOperation And this exception in the master log: 2014-02-27 13:32:12,311 [zookeeper.DistributedReadWriteLock] INFO : Added lock entry 0 userData 1a33c6891b997e71 lockType WRITE 2014-02-27 13:32:12,313 [tableOps.Utils] INFO : table 3n (1a33c6891b997e71) locked for write operation: DELETE 2014-02-27 13:32:12,314 [tables.TableManager] FATAL: Failed to transition table to state DELETING 2014-02-27 13:32:12,320 [fate.Fate] WARN : Failed to execute Repo, tid=1a33c6891b997e71 java.lang.RuntimeException: org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException at org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:148) at org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:224) at org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:207) at org.apache.accumulo.server.master.tableOps.TraceRepo.call(TraceRepo.java:65) at org.apache.accumulo.fate.Fate$TransactionRunner.run(Fate.java:64) at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:34) at java.lang.Thread.run(Thread.java:724) Caused by: org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException at org.apache.accumulo.server.master.state.tables.TableManager$1.mutate(TableManager.java:141) at org.apache.accumulo.fate.zookeeper.ZooReaderWriter.mutate(ZooReaderWriter.java:134) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.accumulo.server.zookeeper.ZooReaderWriter$1.invoke(ZooReaderWriter.java:67) at com.sun.proxy.$Proxy10.mutate(Unknown Source) at org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:116) ... 6 more 2014-02-27 13:32:12,327 [tableOps.Utils] INFO : table 3n (1a33c6891b997e71) unlocked for write -Russ On Thu, Feb 27, 2014 at 10:47 AM, Keith Turner <[email protected]> wrote: > > > > On Wed, Feb 26, 2014 at 11:10 PM, Russ Weeks <[email protected]>wrote: > >> Hi, All, >> >> I can't delete any tables through the accumulo shell, I keep getting the >> message: "Internal error processing waitForTableOperation". >> >> Recently my accumulo cluster has started showing this error repeatedly in >> the master log: >> [state.MetaDataTableScanner] WARN : No prev-row for key extent: {z< : [] >> 9223372036854775807 false= loc243f8e8ad6f5875>. 10.128.1.17:9997 ~tab >> ~pr-� �} >> >> And this error in the monitor log: >> [util.TabletIterator] WARN : gc:agile01 Metadata problem : tablet !0;!0< >> has no prev end row >> >> And this in the GC log: >> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no prev >> end row >> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : [] >> 9223372036854775807 false,~ : [] 9223372036854775807 false) >> >> Maybe coincidence, but it seems like things started getting weird when I >> added a WholeRowIterator through the accumulo shell (which is nonsense, I >> know). The iterators on one of my tables (36M entries) look like this: >> > > Was that iterator added at the table level or system level? Adding at the > system level could impact the metadata table, see ACCUMULO-324. > > >> >> - Iterator WholeRowIterator, scan scope options: >> - iteratorPriority = 50 >> - iteratorClassName = >> org.apache.accumulo.core.iterators.user.WholeRowIterator >> - Iterator rowdel, majc scope options: >> - iteratorPriority = 30 >> - iteratorClassName = >> org.apache.accumulo.core.iterators.user.RowDeletingIterator >> - Iterator transformer, majc scope options: >> - iteratorPriority = 15 >> - iteratorClassName = >> com.phemi.agile.datapolicy.DataColumnTransformer >> - maxBufferSize = 256M >> - Iterator vers, majc scope options: >> - iteratorPriority = 20 >> - iteratorClassName = >> org.apache.accumulo.core.iterators.user.VersioningIterator >> - maxVersions = 1 >> - Iterator vers, minc scope options: >> - iteratorPriority = 20 >> - iteratorClassName = >> org.apache.accumulo.core.iterators.user.VersioningIterator >> - maxVersions = 1 >> - Iterator vers, scan scope options: >> - iteratorPriority = 20 >> - iteratorClassName = >> org.apache.accumulo.core.iterators.user.VersioningIterator >> - maxVersions = 1 >> >> I tried deleting DataColumnTransformer, RowDeletingIterator and >> WholeRowIterator. No errors were returned from the deleteiter command in >> the shell but after running listiter, the WholeRowIterator and >> RowDeletingIterator are still there. >> >> I *am* still able to create tables, but tables that I create can't then >> be deleted. >> >> Any advice appreciated, >> -Russ >> >> >
