dlmarion opened a new issue, #3674:
URL: https://github.com/apache/accumulo/issues/3674
**Describe the bug**
Call chain:
AddSplitsCommand ->
TableOperationsImpl.addSplits (in a loop) ->
TabletClientHandler.splitTablet (throws NotServingTabletException) ->
TabletServer.splitTablet ->
Tablet.split ->
Tablet.initiateClose
On the first iteration of the loop in `TableOperationsImpl.addSplits`,
`Tablet.initiateClose` sets the `closingState` to `CLOSING`, closes the
Compactable, and kicks off a minc. The MinorCompactionTask fails to load the
VolumeChooser due to a invalid classloader context being set on the table.
```
2023-07-31T20:56:08,007 [tablet.MinorCompactionTask] ERROR: Unknown error
during minor compaction for extent: 1<<
java.lang.RuntimeException: Failed to create instance for 1 configured to
use org.apache.accumulo.core.spi.fs.PreferredVolumeChooser via volume.chooser
at
org.apache.accumulo.core.spi.fs.DelegatingChooser.lambda$createVolumeChooser$0(DelegatingChooser.java:161)
~[accumulo-core-2.1.1.jar:2.1.1]
at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1908)
~[?:?]
at
org.apache.accumulo.core.spi.fs.DelegatingChooser.createVolumeChooser(DelegatingChooser.java:142)
~[accumulo-core-2.1.1.jar:2.1.1]
at
org.apache.accumulo.core.spi.fs.DelegatingChooser.getVolumeChooserForTable(DelegatingChooser.java:95)
~[accumulo-core-2.1.1.jar:2.1.1]
at
org.apache.accumulo.core.spi.fs.DelegatingChooser.getDelegateChooser(DelegatingChooser.java:70)
~[accumulo-core-2.1.1.jar:2.1.1]
at
org.apache.accumulo.core.spi.fs.DelegatingChooser.choose(DelegatingChooser.java:59)
~[accumulo-core-2.1.1.jar:2.1.1]
at
org.apache.accumulo.server.fs.VolumeManagerImpl.choose(VolumeManagerImpl.java:479)
~[accumulo-server-base-2.1.1.jar:2.1.1]
at
org.apache.accumulo.tserver.tablet.Tablet.chooseTabletDir(Tablet.java:250)
~[accumulo-tserver-2.1.1.jar:2.1.1]
at
org.apache.accumulo.tserver.tablet.Tablet.getNextMapFilename(Tablet.java:258)
~[accumulo-tserver-2.1.1.jar:2.1.1]
at
org.apache.accumulo.tserver.tablet.MinorCompactionTask.run(MinorCompactionTask.java:79)
~[accumulo-tserver-2.1.1.jar:2.1.1]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-core-2.1.1.jar:2.1.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
~[accumulo-core-2.1.1.jar:2.1.1]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.IllegalArgumentException: Unknown context invalid
at
org.apache.accumulo.start.classloader.vfs.ContextManager.getClassLoader(ContextManager.java:166)
~[accumulo-start-2.1.1.jar:2.1.1]
at
org.apache.accumulo.start.classloader.vfs.AccumuloVFSClassLoader.getContextClassLoader(AccumuloVFSClassLoader.java:420)
~[accumulo-start-2.1.1.jar:2.1.1]
at
org.apache.accumulo.core.classloader.DefaultContextClassLoaderFactory.getClassLoader(DefaultContextClassLoaderFactory.java:99)
~[accumulo-core-2.1.1.jar:2.1.1]
at
org.apache.accumulo.core.classloader.ClassLoaderUtil.getClassLoader(ClassLoaderUtil.java:79)
~[accumulo-core-2.1.1.jar:2.1.1]
at
org.apache.accumulo.core.classloader.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:87)
~[accumulo-core-2.1.1.jar:2.1.1]
at
org.apache.accumulo.core.conf.ConfigurationTypeHelper.getClassInstance(ConfigurationTypeHelper.java:199)
~[accumulo-core-2.1.1.jar:2.1.1]
at
org.apache.accumulo.server.ServiceEnvironmentImpl.instantiate(ServiceEnvironmentImpl.java:69)
~[accumulo-server-base-2.1.1.jar:2.1.1]
at
org.apache.accumulo.core.spi.fs.DelegatingChooser.lambda$createVolumeChooser$0(DelegatingChooser.java:154)
~[accumulo-core-2.1.1.jar:2.1.1]
```
`Tablet.split` then calls `Tablet.completeClose`which sets the `closeState`
to `CLOSED`. I don't have enough information to know where this first iteration
fails after this point, but I'm assuming that it does fail and the Tablet state
is set to CLOSED. The first iteration ends in TableOperationsImpl.addSplits
with an exception, which may not be logged on the client side, and then the
operation is tried again. On the subsequent call to `Tablet.initiateClose` in
`Tablet.split` an IllegalStateException is thrown because the Tablet is already
`CLOSED`. `Tablet.split` logs `File 1<< not splitting : Tablet 1<< already
CLOSING` and returns null. Returning null causes
`TabletClientHandler.splitTablet` to throw a `NotServingTabletException` which
is caught by `TableOperationsImpl.addSplits` and retried. It appears that this
will retry *forever*.
**Versions (OS, Maven, Java, and others, as appropriate):**
- Affected version(s) of this project: 2.1.2-SNAPSHOT
**To Reproduce**
User was able to replicate the issue with the following:
```
root@uno> insert -t test1 a c1 q1 v1
root@uno> insert -t test1 b c1 q1 v1
root@uno> insert -t test1 c c1 q1 v1
root@uno> config -t test1 -s table.class.loader.context=invalid
root@uno> flush -t test1
2023-07-31T22:21:28,451 [shell.Shell] INFO : Flush of table test1
initiated...
root@uno> config -t test1 -d table.class.loader.context
root@uno> scan -t test1
a c1:q1 [] v1
b c1:q1 [] v1
c c1:q1 [] v1
root@uno> addsplits -t test1 b
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
