Hi, Josh, No, I can't scan the !METADATA table. I get about 2k of output (here<https://gist.github.com/rweeks/055f0f19c5b959af8afc>, if useful) and then the shell blocks. I'm seeing a lot of these exceptions in the monitor log:
Caused by: org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:212) at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:82) at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:164) ... 6 more -Russ On Wed, Feb 26, 2014 at 9:33 PM, Josh Elser <[email protected]> wrote: > Hrm, looking at the code, it appears that the !METADATA table has some > inconsistencies. A (at least?) one row lacks of the "prev-row" column. > Something like: > > some_row ~tab:~pr > > Sadly, I'm not positive about the cause given the information. Can you > scan all of the !METADATA table without error? > > That said, I don't think that the above would prevent you from being able > to change the iterators on a table because that's stored purely in > ZooKeeper. Maybe it tries to read some data from !METADATA before doing the > ZooKeeper update? > > > On 2/26/14, 11:49 PM, Russ Weeks wrote: > >> Hi, Josh, >> >> Thanks for responding so quickly... sorry, I should've mentioned that >> I'm running Accumulo 1.5.0. >> >> Nothing really helpful that I can see in the monitor or debug monitor >> logs. It just interleaves the 2 errors above from the >> state.MetaDataTableScanner and util.TabletIterator loggers. >> >> -Russ >> >> >> On Wed, Feb 26, 2014 at 8:37 PM, Josh Elser <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Russ, >> >> What version of Accumulo are you running? >> >> Any other exceptions in the Monitor? >> >> - Josh >> >> >> On 2/26/14, 11:10 PM, Russ Weeks wrote: >> >> Hi, All, >> >> I can't delete any tables through the accumulo shell, I keep >> getting the >> message: "Internal error processing waitForTableOperation". >> >> Recently my accumulo cluster has started showing this error >> repeatedly >> in the master log: >> [state.MetaDataTableScanner] WARN : No prev-row for key extent: >> {z< : [] >> 9223372036854775807 >> false= loc243f8e8ad6f5875>. 10.128.1.17:9997 >> <http://10.128.1.17:9997> ~tab ~pr-� �} >> >> >> And this error in the monitor log: >> [util.TabletIterator] WARN : gc:agile01 Metadata problem : >> tablet !0;!0< >> has no prev end row >> >> And this in the GC log: >> [util.TabletIterator] WARN : Metadata problem : tablet !0;!0< has >> no >> prev end row >> [util.TabletIterator] INFO : Resetting !METADATA scanner to [!0 : >> [] >> 9223372036854775807 false,~ : [] 9223372036854775807 false) >> >> Maybe coincidence, but it seems like things started getting >> weird when I >> added a WholeRowIterator through the accumulo shell (which is >> nonsense, >> I know). The iterators on one of my tables (36M entries) look >> like this: >> >> - Iterator WholeRowIterator, scan scope options: >> - iteratorPriority = 50 >> - iteratorClassName = >> org.apache.accumulo.core.__iterators.user.__WholeRowIterator >> >> - Iterator rowdel, majc scope options: >> - iteratorPriority = 30 >> - iteratorClassName = >> org.apache.accumulo.core.__iterators.user.__RowDeletingIterator >> >> - Iterator transformer, majc scope options: >> - iteratorPriority = 15 >> - iteratorClassName = >> com.phemi.agile.datapolicy.__DataColumnTransformer >> >> - maxBufferSize = 256M >> - Iterator vers, majc scope options: >> - iteratorPriority = 20 >> - iteratorClassName = >> org.apache.accumulo.core.__iterators.user.__VersioningIterator >> >> - maxVersions = 1 >> - Iterator vers, minc scope options: >> - iteratorPriority = 20 >> - iteratorClassName = >> org.apache.accumulo.core.__iterators.user.__VersioningIterator >> >> - maxVersions = 1 >> - Iterator vers, scan scope options: >> - iteratorPriority = 20 >> - iteratorClassName = >> org.apache.accumulo.core.__iterators.user.__VersioningIterator >> >> - maxVersions = 1 >> >> I tried deleting DataColumnTransformer, RowDeletingIterator and >> WholeRowIterator. No errors were returned from the deleteiter >> command in >> the shell but after running listiter, the WholeRowIterator and >> RowDeletingIterator are still there. >> >> I *am* still able to create tables, but tables that I create >> can't then >> be deleted. >> >> Any advice appreciated, >> -Russ >> >> >>
