Hi, Keith, hduser@agile01:~/accumulo/bin$ accumulo org.apache.accumulo.server.fate.Admin print txid: 254bf92a31b14e3b status: SUCCESSFUL op: TableRangeOp locked: [] locking: [] top: null txid: 32cedb4ad5b9b3e3 status: SUCCESSFUL op: DeleteTable locked: [] locking: [] top: null
I might have jumped the gun a bit, but I found how to use the zookeeper CLI to force the table state to ONLINE... I can delete my tables now, I hope I haven't messed up the FSM at a higher level. -Russ On Thu, Feb 27, 2014 at 2:00 PM, Keith Turner <[email protected]> wrote: > If you run 'accumulo org.apache.accumulo.server.fate.Admin -<list option, > not sure what it is>', what does that look like? > > > > > On Thu, Feb 27, 2014 at 4:50 PM, Russ Weeks <[email protected]>wrote: > >> Looking at Zookeeper, it seems like some of my tables are stuck in the >> DELETING state. I can't delete these tables. I *am* able to create and >> delete new tables, or delete other tables that are not in the DELETING >> state. >> >> Is there any way to force the table out of the DELETING state? >> >> Thanks, >> -Russ >> >> [zk: localhost:2181(CONNECTED) 21] get >> /accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/state >> DELETING >> cZxid = 0x2001138c0 >> ctime = Wed Feb 26 20:08:17 PST 2014 >> mZxid = 0x2001138ef >> mtime = Wed Feb 26 20:08:24 PST 2014 >> pZxid = 0x2001138c0 >> cversion = 0 >> dataVersion = 2 >> aclVersion = 0 >> ephemeralOwner = 0x0 >> dataLength = 8 >> numChildren = 0 >> [zk: localhost:2181(CONNECTED) 22] get >> /accumulo/af046a2e-14d4-4a3b-a16f-cc0e88d35f80/tables/3n/name >> test_table >> cZxid = 0x2001138bf >> ctime = Wed Feb 26 20:08:17 PST 2014 >> mZxid = 0x2001138bf >> mtime = Wed Feb 26 20:08:17 PST 2014 >> pZxid = 0x2001138bf >> cversion = 0 >> dataVersion = 0 >> aclVersion = 0 >> ephemeralOwner = 0x0 >> dataLength = 10 >> numChildren = 0 >> >> >> >> >> On Thu, Feb 27, 2014 at 1:34 PM, Russ Weeks <[email protected]>wrote: >> >>> Hi, Keith, >>> >>> Thanks for your response. I think that's exactly what happened. I >>> screwed up the config command and wound up specifying a couple of iterators >>> on the !METADATA table (and indeed all tables in the system). >>> >>> I've removed those iterators using "config -d" and now I can scan the >>> !METADATA table without errors in the shell or the master log. But I'm >>> still not able to delete tables. When I run the command "deletetable >>> test_table", I see this message in the shell: >>> >>> ERROR: org.apache.accumulo.core.client.AccumuloException: Internal error >>> processing waitForTableOperation >>> >>> And this exception in the master log: >>> >>> 2014-02-27 13:32:12,311 [zookeeper.DistributedReadWriteLock] INFO : >>> Added lock entry 0 userData 1a33c6891b997e71 lockType WRITE >>> 2014-02-27 13:32:12,313 [tableOps.Utils] INFO : table 3n >>> (1a33c6891b997e71) locked for write operation: DELETE >>> 2014-02-27 13:32:12,314 [tables.TableManager] FATAL: Failed to >>> transition table to state DELETING >>> 2014-02-27 13:32:12,320 [fate.Fate] WARN : Failed to execute Repo, >>> tid=1a33c6891b997e71 >>> java.lang.RuntimeException: >>> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException >>> at >>> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:148) >>> at >>> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:224) >>> at >>> org.apache.accumulo.server.master.tableOps.DeleteTable.call(DeleteTable.java:207) >>> at >>> org.apache.accumulo.server.master.tableOps.TraceRepo.call(TraceRepo.java:65) >>> at org.apache.accumulo.fate.Fate$TransactionRunner.run(Fate.java:64) >>> at >>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:34) >>> at java.lang.Thread.run(Thread.java:724) >>> Caused by: >>> org.apache.accumulo.server.master.state.tables.TableManager$IllegalTableTransitionException >>> at >>> org.apache.accumulo.server.master.state.tables.TableManager$1.mutate(TableManager.java:141) >>> at >>> org.apache.accumulo.fate.zookeeper.ZooReaderWriter.mutate(ZooReaderWriter.java:134) >>> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >>> org.apache.accumulo.server.zookeeper.ZooReaderWriter$1.invoke(ZooReaderWriter.java:67) >>> at com.sun.proxy.$Proxy10.mutate(Unknown Source) >>> at >>> org.apache.accumulo.server.master.state.tables.TableManager.transitionTableState(TableManager.java:116) >>> ... 6 more >>> 2014-02-27 13:32:12,327 [tableOps.Utils] INFO : table 3n >>> (1a33c6891b997e71) unlocked for write >>> >>> >>> -Russ >>> >>> >>> >>> >>> On Thu, Feb 27, 2014 at 10:47 AM, Keith Turner <[email protected]> wrote: >>> >>>> >>>> >>>> >>>> On Wed, Feb 26, 2014 at 11:10 PM, Russ Weeks >>>> <[email protected]>wrote: >>>> >>>>> Hi, All, >>>>> >>>>> I can't delete any tables through the accumulo shell, I keep getting >>>>> the message: "Internal error processing waitForTableOperation". >>>>> >>>>> Recently my accumulo cluster has started showing this error repeatedly >>>>> in the master log: >>>>> [state.MetaDataTableScanner] WARN : No prev-row for key extent: {z< : >>>>> [] 9223372036854775807 false= loc243f8e8ad6f5875>. 10.128.1.17:9997~tab >>>>> ~pr-� �} >>>>> >>>>> And this error in the monitor log: >>>>> [util.TabletIterator] WARN : gc:agile01 Metadata problem : tablet >>>>> !0;!0< has no prev end row >>>>> >>>>> And this in the GC log: >>>>> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has no >>>>> prev end row >>>>> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : [] >>>>> 9223372036854775807 false,~ : [] 9223372036854775807 false) >>>>> >>>>> Maybe coincidence, but it seems like things started getting weird when >>>>> I added a WholeRowIterator through the accumulo shell (which is nonsense, >>>>> I >>>>> know). The iterators on one of my tables (36M entries) look like this: >>>>> >>>> >>>> Was that iterator added at the table level or system level? Adding at >>>> the system level could impact the metadata table, see ACCUMULO-324. >>>> >>>> >>>>> >>>>> - Iterator WholeRowIterator, scan scope options: >>>>> - iteratorPriority = 50 >>>>> - iteratorClassName = >>>>> org.apache.accumulo.core.iterators.user.WholeRowIterator >>>>> - Iterator rowdel, majc scope options: >>>>> - iteratorPriority = 30 >>>>> - iteratorClassName = >>>>> org.apache.accumulo.core.iterators.user.RowDeletingIterator >>>>> - Iterator transformer, majc scope options: >>>>> - iteratorPriority = 15 >>>>> - iteratorClassName = >>>>> com.phemi.agile.datapolicy.DataColumnTransformer >>>>> - maxBufferSize = 256M >>>>> - Iterator vers, majc scope options: >>>>> - iteratorPriority = 20 >>>>> - iteratorClassName = >>>>> org.apache.accumulo.core.iterators.user.VersioningIterator >>>>> - maxVersions = 1 >>>>> - Iterator vers, minc scope options: >>>>> - iteratorPriority = 20 >>>>> - iteratorClassName = >>>>> org.apache.accumulo.core.iterators.user.VersioningIterator >>>>> - maxVersions = 1 >>>>> - Iterator vers, scan scope options: >>>>> - iteratorPriority = 20 >>>>> - iteratorClassName = >>>>> org.apache.accumulo.core.iterators.user.VersioningIterator >>>>> - maxVersions = 1 >>>>> >>>>> I tried deleting DataColumnTransformer, RowDeletingIterator and >>>>> WholeRowIterator. No errors were returned from the deleteiter command in >>>>> the shell but after running listiter, the WholeRowIterator and >>>>> RowDeletingIterator are still there. >>>>> >>>>> I *am* still able to create tables, but tables that I create can't >>>>> then be deleted. >>>>> >>>>> Any advice appreciated, >>>>> -Russ >>>>> >>>>> >>>> >>> >> >
