If you run 'accumulo org.apache.accumulo.server.fate.Admin -<list option, not sure what it is>', what does that look like?
On Thu, Feb 27, 2014 at 4:50 PM, Russ Weeks <[email protected]>wrote: > Looking at Zookeeper, it seems like some of my tables are stuck in the > DELETING state. I can't delete these tables. I *am* able to create and > delete new tables, or delete other tables that are not in the DELETING > state. > > Is there any way to force the table out of the DELETING state? > > Thanks, > -Russ > > [zk: localhost:2181(CONNECTED) 21] get > /accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/state > DELETING > cZxid = 0x2001138c0 > ctime = Wed Feb 26 20:08:17 PST 2014 > mZxid = 0x2001138ef > mtime = Wed Feb 26 20:08:24 PST 2014 > pZxid = 0x2001138c0 > cversion = 0 > dataVersion = 2 > aclVersion = 0 > ephemeralOwner = 0x0 > dataLength = 8 > numChildren = 0 > [zk: localhost:2181(CONNECTED) 22] get > /accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/name > test_table > cZxid = 0x2001138bf > ctime = Wed Feb 26 20:08:17 PST 2014 > mZxid = 0x2001138bf > mtime = Wed Feb 26 20:08:17 PST 2014 > pZxid = 0x2001138bf > cversion = 0 > dataVersion = 0 > aclVersion = 0 > ephemeralOwner = 0x0 > dataLength = 10 > numChildren = 0 > > > > > On Thu, Feb 27, 2014 at 1:34 PM, Russ Weeks <[email protected]>wrote: > >> Hi, Keith, >> >> Thanks for your response. I think that's exactly what happened. I >> screwed up the config command and wound up specifying a couple of iterators >> on the !METADATA table (and indeed all tables in the system). >> >> I've removed those iterators using "config -d" and now I can scan the >> !METADATA table without errors in the shell or the master log. But I'm >> still not able to delete tables. When I run the command "deletetable >> test_table", I see this message in the shell: >> >> ERROR: org.apache.accumulo.core.client.AccumuloException: Internal error >> processing waitForTableOperation >> >> And this exception in the master log: >> >> 2014-02-27 13:32:12,311 [zookeeper.DistributedReadWriteLock] INFO : Added >> lock entry 0 userData 1a33c6891b997e71 lockType WRITE >> 2014-02-27 13:32:12,313 [tableOps.Utils] INFO : table 3n >> (1a33c6891b997e71) locked for write operation: DELETE >> 2014-02-27 13:32:12,314 [tables.TableManager] FATAL: Failed to transition >> table to state DELETING >> 2014-02-27 13:32:12,320 [fate.Fate] WARN : Failed to execute Repo, >> tid=1a33c6891b997e71 >> java.lang.RuntimeException: >> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException >> at >> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:148) >> at >> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:224) >> at >> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:207) >> at >> org.apache.accumulo.server.master.tableOps.TraceRepo.call(TraceRepo.java:65) >> at org.apache.accumulo.fate.Fate$TransactionRunner.run(Fate.java:64) >> at >> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:34) >> at java.lang.Thread.run(Thread.java:724) >> Caused by: >> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException >> at >> org.apache.accumulo.server.master.state.tables.TableManager$1.mutate(TableManager.java:141) >> at >> org.apache.accumulo.fate.zookeeper.ZooReaderWriter.mutate(ZooReaderWriter.java:134) >> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.apache.accumulo.server.zookeeper.ZooReaderWriter$1.invoke(ZooReaderWriter.java:67) >> at com.sun.proxy.$Proxy10.mutate(Unknown Source) >> at >> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:116) >> ... 6 more >> 2014-02-27 13:32:12,327 [tableOps.Utils] INFO : table 3n >> (1a33c6891b997e71) unlocked for write >> >> >> -Russ >> >> >> >> >> On Thu, Feb 27, 2014 at 10:47 AM, Keith Turner <[email protected]> wrote: >> >>> >>> >>> >>> On Wed, Feb 26, 2014 at 11:10 PM, Russ Weeks >>> <[email protected]>wrote: >>> >>>> Hi, All, >>>> >>>> I can't delete any tables through the accumulo shell, I keep getting >>>> the message: "Internal error processing waitForTableOperation". >>>> >>>> Recently my accumulo cluster has started showing this error repeatedly >>>> in the master log: >>>> [state.MetaDataTableScanner] WARN : No prev-row for key extent: {z< : >>>> [] 9223372036854775807 false= loc243f8e8ad6f5875>. 10.128.1.17:9997~tab >>>> ~pr-� �} >>>> >>>> And this error in the monitor log: >>>> [util.TabletIterator] WARN : gc:agile01 Metadata problem : tablet >>>> !0;!0< has no prev end row >>>> >>>> And this in the GC log: >>>> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no >>>> prev end row >>>> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : [] >>>> 9223372036854775807 false,~ : [] 9223372036854775807 false) >>>> >>>> Maybe coincidence, but it seems like things started getting weird when >>>> I added a WholeRowIterator through the accumulo shell (which is nonsense, I >>>> know). The iterators on one of my tables (36M entries) look like this: >>>> >>> >>> Was that iterator added at the table level or system level? Adding at >>> the system level could impact the metadata table, see ACCUMULO-324. >>> >>> >>>> >>>> - Iterator WholeRowIterator, scan scope options: >>>> - iteratorPriority = 50 >>>> - iteratorClassName = >>>> org.apache.accumulo.core.iterators.user.WholeRowIterator >>>> - Iterator rowdel, majc scope options: >>>> - iteratorPriority = 30 >>>> - iteratorClassName = >>>> org.apache.accumulo.core.iterators.user.RowDeletingIterator >>>> - Iterator transformer, majc scope options: >>>> - iteratorPriority = 15 >>>> - iteratorClassName = >>>> com.phemi.agile.datapolicy.DataColumnTransformer >>>> - maxBufferSize = 256M >>>> - Iterator vers, majc scope options: >>>> - iteratorPriority = 20 >>>> - iteratorClassName = >>>> org.apache.accumulo.core.iterators.user.VersioningIterator >>>> - maxVersions = 1 >>>> - Iterator vers, minc scope options: >>>> - iteratorPriority = 20 >>>> - iteratorClassName = >>>> org.apache.accumulo.core.iterators.user.VersioningIterator >>>> - maxVersions = 1 >>>> - Iterator vers, scan scope options: >>>> - iteratorPriority = 20 >>>> - iteratorClassName = >>>> org.apache.accumulo.core.iterators.user.VersioningIterator >>>> - maxVersions = 1 >>>> >>>> I tried deleting DataColumnTransformer, RowDeletingIterator and >>>> WholeRowIterator. No errors were returned from the deleteiter command in >>>> the shell but after running listiter, the WholeRowIterator and >>>> RowDeletingIterator are still there. >>>> >>>> I *am* still able to create tables, but tables that I create can't then >>>> be deleted. >>>> >>>> Any advice appreciated, >>>> -Russ >>>> >>>> >>> >> >
