dlmarion opened a new issue, #2979:
URL: https://github.com/apache/accumulo/issues/2979
**Describe the bug**
Ran cingest on a 2.1.0-SNAPSHOT cluster with per-table crypto configured.
Specifically, the following properties are set in `accumulo.properties`:
```
instance.crypto.opts.factory=org.apache.accumulo.core.spi.crypto.PerTableCryptoServiceFactory
general.custom.crypto.recovery.service=org.apache.accumulo.core.spi.crypto.AESCryptoService
general.custom.crypto.wal.service=org.apache.accumulo.core.spi.crypto.AESCryptoService
general.custom.crypto.key.uri=file:///opt/accumulo-testing/accumulo/accumulo-2.1.0-SNAPSHOT/conf/per-table-enc-key
```
and the `ci` table had the property
`table.crypto.opts.service=org.apache.accumulo.core.spi.crypto.AESCryptoService`.
I loaded the table with a bunch of data, then I deleted it via the shell. The
`deletetable` command never returned and it appears that one of the metadata
tablets was re-assigned during the delete operation. When trying to load the
metadata tablet, the tserver logged the following error:
```
2022-09-28T16:50:44,468 [log.RecoveryLogsIterator] DEBUG: Opening recovery
log dir 11946b4a-99b6-4b7e-b0d4-ccc1f981783a
2022-09-28T16:50:44,469 [crypto.CryptoFactoryLoader] DEBUG: Creating new
crypto factory class
org.apache.accumulo.core.spi.crypto.GenericCryptoServiceFactory
2022-09-28T16:50:44,472 [tserver.AssignmentHandler] WARN : exception trying
to assign tablet !0<;~ null
java.lang.RuntimeException: Error recovering tablet !0<;~ from log files
at org.apache.accumulo.tserver.tablet.Tablet.<init>(Tablet.java:379)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.AssignmentHandler.run(AssignmentHandler.java:165)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.ActiveAssignmentRunnable.run(ActiveAssignmentRunnable.java:63)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.io.IOException: java.io.UncheckedIOException:
java.io.IOException: incorrect header check
at
org.apache.accumulo.tserver.log.TabletServerLogger.recover(TabletServerLogger.java:544)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.TabletServer.recover(TabletServer.java:1204)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.tablet.Tablet.<init>(Tablet.java:334)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
... 7 more
Caused by: java.io.UncheckedIOException: java.io.IOException: incorrect
header check
at
org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader$BCFileLoader.load(CachableBlockFile.java:223)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.client.rfile.RFileScanner$NoopCache.getBlock(RFileScanner.java:165)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.client.rfile.RFileScanner$NoopCache.getBlock(RFileScanner.java:158)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getMetaBlock(CachableBlockFile.java:376)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.file.rfile.RFile$Reader.<init>(RFile.java:1164)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.file.rfile.RFile$Reader.<init>(RFile.java:1256)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.core.client.rfile.RFileScanner.iterator(RFileScanner.java:354)
~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.log.RecoveryLogsIterator.validateFirstKey(RecoveryLogsIterator.java:156)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.log.RecoveryLogsIterator.<init>(RecoveryLogsIterator.java:77)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.log.SortedLogRecovery.findMaxTabletId(SortedLogRecovery.java:107)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.log.SortedLogRecovery.findLogsThatDefineTablet(SortedLogRecovery.java:147)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.log.SortedLogRecovery.recover(SortedLogRecovery.java:291)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.log.TabletServerLogger.recover(TabletServerLogger.java:542)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.accumulo.tserver.TabletServer.recover(TabletServer.java:1204)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.tablet.Tablet.<init>(Tablet.java:334)
~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
... 7 more
Caused by: java.io.IOException: incorrect header check
at
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native
Method) ~[hadoop-client-api-3.3.4.jar:?]
at
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:225)
~[hadoop-client-api-3.3.4.jar:?]
...
```
One tablet for the metadata table is in an offline state and the `ci` table
is in a deleting state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
