Hi,
Short answer, if you are using debian 12 (bookworm) with linstor, you
may want to avoid upgrading to these versions of these two packages:
ii openjdk-17-jdk-headless:amd64 17.0.13+11-2~deb12u1
amd64 OpenJDK Development Kit (JDK) (headless)
ii openjdk-17-jre-headless:amd64 17.0.13+11-2~deb12u1
amd64 OpenJDK Java runtime, using Hotspot JIT (headless)
I'm not entirely sure why, but as soon as these updated packages are
installed, it breaks linstor, with the following error reports.
Solution, downgrade to:
ii openjdk-17-jdk-headless:amd64 17.0.12+7-2~deb12u1
amd64 OpenJDK Development Kit (JDK) (headless)
ii openjdk-17-jre-headless:amd64 17.0.12+7-2~deb12u1
amd64 OpenJDK Java runtime, using Hotspot JIT (headless)
Again, as soon as I downgrade, the errors go away, however, I do still
have one broken resource which I needed to fix by taking a snapshot of
it, then creating a new resource from the snapshot. Still kind of
getting that one worked out (only synced to two nodes thid node stalled
at 62% and then went to 0%, but currently sync is in progress to a
fourth so hopefully will complete).
For reference, I'm using linstor version:
linstor 1.14.0; GIT-hash: 4885d705ccb38ce359ed58eab024bbc94413aaff
So this may have been fixed already in a later version.
Copy of one error report, let me know if you would like more, or want
any other information:
ERROR REPORT 672452F6-F4BC4-000640
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 1.29.0
Build ID: b2be7208a777f0743d4c7187062678cd5416fccf
Build time: 2024-07-31T11:02:51+00:00
Error time: 2024-11-02 22:11:08
Node: san5.xxxx
Thread: MainWorkerPool-2
============================================================
Reported error:
===============
Category: LinStorException
Class name: StorageException
Class canonical name: com.linbit.linstor.storage.StorageException
Generated at: Method 'genericExecutor', Source
file 'Commands.java', Line #120
Error message: Failed to get physical devices for
volume group: vg_hdd
ErrorContext:
Cause: External command threw an IOException
Details: External command: pvdisplay --config 'devices {
filter=["r|^/dev/drbd.*|"] }' --columns -o pv_name -S vg_name=vg_hdd
--noheadings --nosuffix
Call backtrace:
Method Native Class:Line number
genericExecutor N
com.linbit.linstor.storage.utils.Commands:120
genericExecutor N
com.linbit.linstor.storage.utils.Commands:63
genericExecutor N
com.linbit.linstor.storage.utils.Commands:51
listPhysicalVolumes N
com.linbit.linstor.layer.storage.lvm.utils.LvmCommands:595
getPhysicalVolumes N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:641
getLvmConfig N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:167
recacheLvmConfig N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:223
execWithRetry N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:669
getLvsInfoImpl N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:317
getLvsInfo N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:282
fetchAllocatedSizes N
com.linbit.linstor.layer.storage.lvm.LvmThinProvider:710
getVlmAllocatedCapacities N
com.linbit.linstor.core.apicallhandler.StltApiCallHandlerUtils:185
executeInScope N
com.linbit.linstor.api.protobuf.ReqVlmAllocated:84
lambda$executeReactive$0 N
com.linbit.linstor.api.protobuf.ReqVlmAllocated:69
doInScope N
com.linbit.linstor.core.apicallhandler.ScopeRunner:175
lambda$fluxInScope$0 N
com.linbit.linstor.core.apicallhandler.ScopeRunner:100
call N
reactor.core.publisher.MonoCallable:72
trySubscribeScalarMap N
reactor.core.publisher.FluxFlatMap:127
subscribeOrReturn N
reactor.core.publisher.MonoFlatMapMany:49
subscribe N
reactor.core.publisher.Flux:8759
onNext N
reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:195
request N
reactor.core.publisher.Operators$ScalarSubscription:2545
onSubscribe N
reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
subscribe N
reactor.core.publisher.MonoJust:55
subscribe N
reactor.core.publisher.MonoDeferContextual:55
subscribe N
reactor.core.publisher.Flux:8773
trySubscribeScalarMap N
reactor.core.publisher.FluxFlatMap:200
subscribeOrReturn N
reactor.core.publisher.MonoFlatMapMany:49
subscribe N
reactor.core.publisher.Flux:8759
onNext N
reactor.core.publisher.FluxFlatMap$FlatMapMain:427
slowPath N
reactor.core.publisher.FluxArray$ArraySubscription:127
request N
reactor.core.publisher.FluxArray$ArraySubscription:100
onSubscribe N
reactor.core.publisher.FluxFlatMap$FlatMapMain:371
subscribe N
reactor.core.publisher.FluxMerge:70
subscribe N
reactor.core.publisher.Flux:8773
onComplete N
reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber:258
subscribe N
reactor.core.publisher.FluxConcatArray:78
subscribe N
reactor.core.publisher.InternalFluxOperator:62
subscribe N
reactor.core.publisher.FluxDefer:54
subscribe N
reactor.core.publisher.Flux:8773
onNext N
reactor.core.publisher.FluxFlatMap$FlatMapMain:427
drainAsync N
reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453
drain N
reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724
onNext N
reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256
drainFused N
reactor.core.publisher.SinkManyUnicast:319
drain N
reactor.core.publisher.SinkManyUnicast:362
tryEmitNext N
reactor.core.publisher.SinkManyUnicast:237
tryEmitNext N
reactor.core.publisher.SinkManySerialized:100
processInOrder N
com.linbit.linstor.netcom.TcpConnectorPeer:416
doProcessMessage N
com.linbit.linstor.proto.CommonMessageProcessor:228
lambda$processMessage$2 N
com.linbit.linstor.proto.CommonMessageProcessor:165
onNext N
reactor.core.publisher.FluxPeek$PeekSubscriber:185
runAsync N
reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:440
run N
reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:527
call N
reactor.core.scheduler.WorkerTask:84
call N
reactor.core.scheduler.WorkerTask:37
run N
java.util.concurrent.FutureTask:264
run N
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
runWorker N
java.util.concurrent.ThreadPoolExecutor:1136
run N
java.util.concurrent.ThreadPoolExecutor$Worker:635
run N java.lang.Thread:840
Caused by:
==========
Category: Exception
Class name: IOException
Class canonical name: java.io.IOException
Generated at: Method 'start', Source file
'ProcessBuilder.java', Line #1143
Error message: Cannot run program "pvdisplay":
error=0, Failed to exec spawn helper: pid: 97865, exit value: 1
Call backtrace:
Method Native Class:Line number
start N
java.lang.ProcessBuilder:1143
start N
java.lang.ProcessBuilder:1073
exec N
com.linbit.extproc.ExtCmd:130
exec N com.linbit.extproc.ExtCmd:91
genericExecutor N
com.linbit.linstor.storage.utils.Commands:79
genericExecutor N
com.linbit.linstor.storage.utils.Commands:63
genericExecutor N
com.linbit.linstor.storage.utils.Commands:51
listPhysicalVolumes N
com.linbit.linstor.layer.storage.lvm.utils.LvmCommands:595
getPhysicalVolumes N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:641
getLvmConfig N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:167
recacheLvmConfig N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:223
execWithRetry N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:669
getLvsInfoImpl N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:317
getLvsInfo N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:282
fetchAllocatedSizes N
com.linbit.linstor.layer.storage.lvm.LvmThinProvider:710
getVlmAllocatedCapacities N
com.linbit.linstor.core.apicallhandler.StltApiCallHandlerUtils:185
executeInScope N
com.linbit.linstor.api.protobuf.ReqVlmAllocated:84
lambda$executeReactive$0 N
com.linbit.linstor.api.protobuf.ReqVlmAllocated:69
doInScope N
com.linbit.linstor.core.apicallhandler.ScopeRunner:175
lambda$fluxInScope$0 N
com.linbit.linstor.core.apicallhandler.ScopeRunner:100
call N
reactor.core.publisher.MonoCallable:72
trySubscribeScalarMap N
reactor.core.publisher.FluxFlatMap:127
subscribeOrReturn N
reactor.core.publisher.MonoFlatMapMany:49
subscribe N
reactor.core.publisher.Flux:8759
onNext N
reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:195
request N
reactor.core.publisher.Operators$ScalarSubscription:2545
onSubscribe N
reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
subscribe N
reactor.core.publisher.MonoJust:55
subscribe N
reactor.core.publisher.MonoDeferContextual:55
subscribe N
reactor.core.publisher.Flux:8773
trySubscribeScalarMap N
reactor.core.publisher.FluxFlatMap:200
subscribeOrReturn N
reactor.core.publisher.MonoFlatMapMany:49
subscribe N
reactor.core.publisher.Flux:8759
onNext N
reactor.core.publisher.FluxFlatMap$FlatMapMain:427
slowPath N
reactor.core.publisher.FluxArray$ArraySubscription:127
request N
reactor.core.publisher.FluxArray$ArraySubscription:100
onSubscribe N
reactor.core.publisher.FluxFlatMap$FlatMapMain:371
subscribe N
reactor.core.publisher.FluxMerge:70
subscribe N
reactor.core.publisher.Flux:8773
onComplete N
reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber:258
subscribe N
reactor.core.publisher.FluxConcatArray:78
subscribe N
reactor.core.publisher.InternalFluxOperator:62
subscribe N
reactor.core.publisher.FluxDefer:54
subscribe N
reactor.core.publisher.Flux:8773
onNext N
reactor.core.publisher.FluxFlatMap$FlatMapMain:427
drainAsync N
reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453
drain N
reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724
onNext N
reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256
drainFused N
reactor.core.publisher.SinkManyUnicast:319
drain N
reactor.core.publisher.SinkManyUnicast:362
tryEmitNext N
reactor.core.publisher.SinkManyUnicast:237
tryEmitNext N
reactor.core.publisher.SinkManySerialized:100
processInOrder N
com.linbit.linstor.netcom.TcpConnectorPeer:416
doProcessMessage N
com.linbit.linstor.proto.CommonMessageProcessor:228
lambda$processMessage$2 N
com.linbit.linstor.proto.CommonMessageProcessor:165
onNext N
reactor.core.publisher.FluxPeek$PeekSubscriber:185
runAsync N
reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:440
run N
reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:527
call N
reactor.core.scheduler.WorkerTask:84
call N
reactor.core.scheduler.WorkerTask:37
run N
java.util.concurrent.FutureTask:264
run N
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
runWorker N
java.util.concurrent.ThreadPoolExecutor:1136
run N
java.util.concurrent.ThreadPoolExecutor$Worker:635
run N java.lang.Thread:840
Caused by:
==========
Category: Exception
Class name: IOException
Class canonical name: java.io.IOException
Generated at: Method 'forkAndExec', Source file
'ProcessImpl.java, Unknown line number
Error message: error=0, Failed to exec spawn
helper: pid: 97865, exit value: 1
Call backtrace:
Method Native Class:Line number
forkAndExec Y
java.lang.ProcessImpl:unknown
<init> N java.lang.ProcessImpl:314
start N java.lang.ProcessImpl:244
start N
java.lang.ProcessBuilder:1110
start N
java.lang.ProcessBuilder:1073
exec N
com.linbit.extproc.ExtCmd:130
exec N com.linbit.extproc.ExtCmd:91
genericExecutor N
com.linbit.linstor.storage.utils.Commands:79
genericExecutor N
com.linbit.linstor.storage.utils.Commands:63
genericExecutor N
com.linbit.linstor.storage.utils.Commands:51
listPhysicalVolumes N
com.linbit.linstor.layer.storage.lvm.utils.LvmCommands:595
getPhysicalVolumes N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:641
getLvmConfig N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:167
recacheLvmConfig N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:223
execWithRetry N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:669
getLvsInfoImpl N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:317
getLvsInfo N
com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:282
fetchAllocatedSizes N
com.linbit.linstor.layer.storage.lvm.LvmThinProvider:710
getVlmAllocatedCapacities N
com.linbit.linstor.core.apicallhandler.StltApiCallHandlerUtils:185
executeInScope N
com.linbit.linstor.api.protobuf.ReqVlmAllocated:84
lambda$executeReactive$0 N
com.linbit.linstor.api.protobuf.ReqVlmAllocated:69
doInScope N
com.linbit.linstor.core.apicallhandler.ScopeRunner:175
lambda$fluxInScope$0 N
com.linbit.linstor.core.apicallhandler.ScopeRunner:100
call N
reactor.core.publisher.MonoCallable:72
trySubscribeScalarMap N
reactor.core.publisher.FluxFlatMap:127
subscribeOrReturn N
reactor.core.publisher.MonoFlatMapMany:49
subscribe N
reactor.core.publisher.Flux:8759
onNext N
reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:195
request N
reactor.core.publisher.Operators$ScalarSubscription:2545
onSubscribe N
reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
subscribe N
reactor.core.publisher.MonoJust:55
subscribe N
reactor.core.publisher.MonoDeferContextual:55
subscribe N
reactor.core.publisher.Flux:8773
trySubscribeScalarMap N
reactor.core.publisher.FluxFlatMap:200
subscribeOrReturn N
reactor.core.publisher.MonoFlatMapMany:49
subscribe N
reactor.core.publisher.Flux:8759
onNext N
reactor.core.publisher.FluxFlatMap$FlatMapMain:427
slowPath N
reactor.core.publisher.FluxArray$ArraySubscription:127
request N
reactor.core.publisher.FluxArray$ArraySubscription:100
onSubscribe N
reactor.core.publisher.FluxFlatMap$FlatMapMain:371
subscribe N
reactor.core.publisher.FluxMerge:70
subscribe N
reactor.core.publisher.Flux:8773
onComplete N
reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber:258
subscribe N
reactor.core.publisher.FluxConcatArray:78
subscribe N
reactor.core.publisher.InternalFluxOperator:62
subscribe N
reactor.core.publisher.FluxDefer:54
subscribe N
reactor.core.publisher.Flux:8773
onNext N
reactor.core.publisher.FluxFlatMap$FlatMapMain:427
drainAsync N
reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453
drain N
reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724
onNext N
reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256
drainFused N
reactor.core.publisher.SinkManyUnicast:319
drain N
reactor.core.publisher.SinkManyUnicast:362
tryEmitNext N
reactor.core.publisher.SinkManyUnicast:237
tryEmitNext N
reactor.core.publisher.SinkManySerialized:100
processInOrder N
com.linbit.linstor.netcom.TcpConnectorPeer:416
doProcessMessage N
com.linbit.linstor.proto.CommonMessageProcessor:228
lambda$processMessage$2 N
com.linbit.linstor.proto.CommonMessageProcessor:165
onNext N
reactor.core.publisher.FluxPeek$PeekSubscriber:185
runAsync N
reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:440
run N
reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:527
call N
reactor.core.scheduler.WorkerTask:84
call N
reactor.core.scheduler.WorkerTask:37
run N
java.util.concurrent.FutureTask:264
run N
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
runWorker N
java.util.concurrent.ThreadPoolExecutor:1136
run N
java.util.concurrent.ThreadPoolExecutor$Worker:635
run N java.lang.Thread:840
END OF ERROR REPORT.
