swhitman0 opened a new issue, #7316:
URL: https://github.com/apache/cloudstack/issues/7316
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete
the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
Storage volumes, Kubernetes Service
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main
branch.
-->
~~~
4.17.2.0
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network,
advanced networking, etc. N/A otherwise
-->
* Advanced networking with vxlan isolation of guest networks
* Kubernetes service enabled
* Cloudstack CSI driver deployed
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
Rocky Linnux 8.7
##### SUMMARY
<!-- Explain the problem/feature briefly -->
When a Kubernetes persistent volume claim is deleted, the Cloudstack volume
is not deleted, and an error message is written to the management log.
##### STEPS TO REPRODUCE
<!--
For bugs, show how to reproduce the problem using a minimal test case. Use
Screenshots if accurate.
For new features, show how the quality would be used.
-->
* Install the CSI driver in the K8s cluster using the command :
```
kubectl apply -f
https://github.com/apalia/cloudstack-csi-driver/releases/latest/download/manifest.yaml
```
* Deploy this storage class:
```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cloudstack-custom
provisioner: csi.cloudstack.apache.org
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: false
parameters:
csi.cloudstack.apache.org/disk-offering-id: <id of custom disk offering>
```
* Deploy this pvs and pod:
<!-- Paste example playbooks or commands between quotes below -->
~~~
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: example-pvc
spec:
storageClassName: cloudstack-custom
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
name: example-pod
spec:
containers:
- name: example
image: busybox
volumeMounts:
- mountPath: "/data"
name: example-volume
stdin: true
stdinOnce: true
tty: true
volumes:
- name: example-volume
persistentVolumeClaim:
claimName: example-pvc
~~~
* Delete the pod and pvc
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
I expected the Cloudstack storage volume backing the PVC would be deleted.
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
The storage volume was not deleted, and the following error was printed to
the management log:
<!-- Paste verbatim command output between quotes below -->
~~~
2023-03-05 14:18:39,144 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-92:ctx-3f7cf487 job-1657/job-1658 ctx-d6957fbc)
(logid:fd682bcf) Invocation exception, caused by:
com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume
pvc-0e82d1a8-9af2-40b3-b037-ebcf151febd3 from VM testk8s-node-186b0939378;
com.cloud.exception.InternalErrorException: Could not detach volume. Probably
the VM is in boot state at the moment
2023-03-05 14:18:39,144 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-92:ctx-3f7cf487 job-1657/job-1658 ctx-d6957fbc)
(logid:fd682bcf) Rethrow exception
com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume
pvc-0e82d1a8-9af2-40b3-b037-ebcf151febd3 from VM testk8s-node-186b0939378;
com.cloud.exception.InternalErrorException: Could not detach volume. Probably
the VM is in boot state at the moment
2023-03-05 14:18:39,144 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-92:ctx-3f7cf487 job-1657/job-1658) (logid:fd682bcf) Done
with run of VM work job: com.cloud.vm.VmWorkDetachVolume for VM 188, job
origin: 1657
2023-03-05 14:18:39,144 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-92:ctx-3f7cf487 job-1657/job-1658) (logid:fd682bcf) Unable
to complete AsyncJobVO: {id:1658, userId: 4, accountId: 2, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkDetachVolume, cmdInfo:
rO0ABXNyAB9jb20uY2xvdWQudm0uVm1Xb3JrRGV0YWNoVm9sdW1lhvReA-s81AMCAAFMAAh2b2x1bWVJZHQAEExqYXZhL2xhbmcvTG9uZzt4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1ldAASTGphdmEvbGFuZy9TdHJpbmc7eHAAAAAAAAAAAgAAAAAAAAAEAAAAAAAAALx0ABRWb2x1bWVBcGlTZXJ2aWNlSW1wbHNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAvA,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 128437426857854, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Sun Mar 05 14:18:36 EST 2023, removed: null}, job
origin:1657
com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume
pvc-0e82d1a8-9af2-40b3-b037-ebcf151febd3 from VM testk8s-node-186b0939378;
com.cloud.exception.InternalErrorException: Could not detach volume. Probably
the VM is in boot state at the moment
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateDetachVolumeFromVM(VolumeApiServiceImpl.java:2793)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateDetachVolumeFromVM(VolumeApiServiceImpl.java:4404)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4435)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy231.handleVmWorkJob(Unknown Source)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
