Hello Bharat, You’ve some well-known options as below: https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner https://github.com/rook/rook - CephFS & NFS can fit
Thanks, Jayanth Reddy From: Bharat Bhushan Saini <bharat.sa...@kloudspot.com.INVALID> Date: Tuesday, 27 February 2024 at 11:01 To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: CKS Storage Provisioner Info Hi All, Thanks for helping out for the storage provisioner. In CKS service I am able to bound PVC as ReadWriteOnce with Leaseweb CloudStack CSI Driver. Is there any opportunity in the cloudstack with other driver that I can bound storage as ReadWriteMany. I tried with Leaseweb and got below warning Warning ProvisioningFailed 6s (x5 over 21s) csi.cloudstack.apache.org_cloudstack-csi-controller-7f89c8cd47-ztllw_c6112933-3587-4445-8702-74b255b1e56f failed to provision volume with StorageClass "cloudstack-custom": rpc error: code = InvalidArgument desc = Volume capabilities not supported. Only SINGLE_NODE_WRITER supported. Thanks and Regards, Bharat Saini [signature_636195584] From: Kiran Chavala <kiran.chav...@shapeblue.com> Date: Monday, 26 February 2024 at 5:54 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: CKS Storage Provisioner Info EXTERNAL EMAIL: Please verify the sender email address before taking any action, replying, clicking any link or opening any attachment. Hi Bharath Note the CKS provisioner works on KVM based cloudstack environment Regards Kiran From: Kiran Chavala <kiran.chav...@shapeblue.com> Date: Monday, 26 February 2024 at 5:45 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: CKS Storage Provisioner Info Hi Bharat Bhusan Please follow these steps 1. Deploy a Kubernetes cluster on cluster NAME STATUS ROLES AGE VERSION ty-control-18de52c04f7 Ready control-plane 5m11s v1.28.4 ty-node-18de52c4185 Ready <none> 4m55s v1.28.4 kubectl get secrets -A NAMESPACE NAME TYPE DATA AGE kube-system cloudstack-secret Opaque 1 10m 2. Check the deployment, it should be in pending ~ kubectl get deployments -A NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE kube-system cloudstack-csi-controller 0/1 1 0 46s 3. Edit the deployment and remove the nodeSelector part ~kubectl edit deployment/cloudstack-csi-controller -n kube-system nodeSelector: kubernetes.io/os: linux node-role.kubernetes.io/master: "" 4. Check the deployment again and it should be running state ~kubectl get deployments -A NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE kube-system cloudstack-csi-controller 1/1 1 1 2m39s 5. Replace the disk offering in the storage class yaml Provide the custom disk offering UUID (service offerings > disk offering > custom disk offering 4c518474-5d7b-4285-a07c-c57e214abb3b vi 0-storageclass.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: cloudstack-custom provisioner: csi.cloudstack.apache.org reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: false parameters: csi.cloudstack.apache.org/disk-offering-id: 4c518474-5d7b-4285-a07c-c57e214abb3b kubectl apply -f 0-storageclass.yaml k8s git:(master) ✗ kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE cloudstack-custom csi.cloudstack.apache.org Delete WaitForFirstConsumer false 72s 6. Apply the pvc yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: example-pvc-block spec: storageClassName: cloudstack-custom volumeMode: Block accessModes: - ReadWriteOnce resources: requests: storage: 1Gi kubectl apply -f pvc-block.yaml k8s git:(master) ✗ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE example-pvc-block Pending cloudstack-custom 31s 7. Apply the pod block yaml apiVersion: v1 kind: Pod metadata: name: example-pod-block spec: containers: - name: example image: ubuntu volumeDevices: - devicePath: "/dev/example-block" name: example-volume stdin: true stdinOnce: true tty: true volumes: - name: example-volume persistentVolumeClaim: claimName: example-pvc-block kubectl apply -f pod-block.yaml ➜ k8s git:(master) ✗ k get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE default example-pod-block 1/1 Running 0 107s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 9s default-scheduler Successfully assigned default/example-pod-block to ty-node-18de52c4185 Normal SuccessfulAttachVolume 6s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-02999f04-dd0a-407c-8805-125c7c56d51b" Normal SuccessfulMountVolume 4s kubelet MapVolume.MapPodDevice succeeded for volume "pvc-02999f04-dd0a-407c-8805-125c7c56d51b" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-02999f04-dd0a-407c-8805-125c7c56d51b/dev" Normal SuccessfulMountVolume 4s kubelet MapVolume.MapPodDevice succeeded for volume "pvc-02999f04-dd0a-407c-8805-125c7c56d51b" volumeMapPath "/var/lib/kubelet/pods/076bd828-7130-4f72-a0ee-29f93043bbb1/volumeDevices/kubernetes.io~csi" Normal Pulling 3s kubelet Pulling image "ubuntu" Regards Kiran From: Jayanth Reddy <jayanthreddy5...@gmail.com> Date: Monday, 26 February 2024 at 4:35 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: CKS Storage Provisioner Info Hello Bharat, If that is the case, please update your "api-url" variable for the cloud-config. FWIW, please also visit "endpoint.url" in global configuration to include the HTTPS endpoint. Thanks, Jayanth Sent from Outlook for Android<https://aka.ms/AAb9ysg<https://checkpoint.url-protection.com/v1/url?o=https%3A//aka.ms/AAb9ysg&g=ODlmZjM0NzJlYWQ3Y2ZkNw==&h=ZjE0OTI0MDUzNzEwYWNlMzhmYTEwM2E0MDU3OWRiYjc2YzNjMTdkNGIxYTEzZTkyMGQyN2ZhZmZlMzU4OGNhYQ==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==>> ________________________________ From: Bharat Bhushan Saini <bharat.sa...@kloudspot.com.INVALID> Sent: Monday, February 26, 2024 4:29:22 pm To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: CKS Storage Provisioner Info Hi Jayanth, For to secure communication, I already enabled the https over the management server and turned off the http. The API URL only listen over http? Thanks and Regards, Bharat Saini [signature_3317674547] From: Jayanth Reddy <jayanthreddy5...@gmail.com> Date: Monday, 26 February 2024 at 4:19 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: CKS Storage Provisioner Info EXTERNAL EMAIL: Please verify the sender email address before taking any action, replying, clicking any link or opening any attachment. Are you able to access the CloudStack WebUI in the URL http://10.1.10.2:8080/client ? As long as k8s nodes have connectivity on the path /client/api on your management, this should work fine. Perhaps a host-level firewall in your management server? Thanks On Mon, Feb 26, 2024 at 4:08 PM Bharat Bhushan Saini <bharat.sa...@kloudspot.com.invalid> wrote: > Hi Vivek, > > > > Please check the findings > > > > ping 10.1.x.2 > > PING 10.1.x.2 (10.1.x.2): 56 data bytes > > 64 bytes from 10.1.x.2: icmp_seq=0 ttl=64 time=0.616 ms > > 64 bytes from 10.1.x.2: icmp_seq=1 ttl=64 time=0.716 ms > > ^C--- 10.1.10.2 ping statistics --- > > 2 packets transmitted, 2 packets received, 0% packet loss > > round-trip min/avg/max/stddev = 0.616/0.666/0.716/0.050 ms > > > > ping cloudstack.internal.com > > PING cloudstack.internal.com (10.1.x.2): 56 data bytes > > 64 bytes from 10.1.x.2: icmp_seq=0 ttl=64 time=0.555 ms > > 64 bytes from 10.1.x.2: icmp_seq=1 ttl=64 time=0.620 ms > > 64 bytes from 10.1.x.2: icmp_seq=2 ttl=64 time=0.664 ms > > ^C--- cloudstack.internal.kloudspot.com ping statistics --- > > 3 packets transmitted, 3 packets received, 0% packet loss > > round-trip min/avg/max/stddev = 0.555/0.613/0.664/0.045 ms > > > > telnet 10.1.x.2 8080 > > Trying 10.1.x.2... > > telnet: Unable to connect to remote host: Connection refused > > > > > > I am able to ping the management IP and URL but not able to get access > from the port as it is not open in the cluster. > NOTE: I use the management IP in the API URL. > > > > Thanks and Regards, > > Bharat Saini > > > > [image: signature_4099962424] > > > > *From: *Vivek Kumar <vivek.ku...@indiqus.com.INVALID> > *Date: *Monday, 26 February 2024 at 3:49 PM > *To: *users@cloudstack.apache.org <users@cloudstack.apache.org> > *Subject: *Re: CKS Storage Provisioner Info > > EXTERNAL EMAIL: Please verify the sender email address before taking any > action, replying, clicking any link or opening any attachment. > > > Hello Bharat, > > Is the cloudstack URL is reachable from your cluster, can you manually > check i.e ping, telnet on that port ? > > > > > > On 26-Feb-2024, at 3:43 PM, Bharat Bhushan Saini > <bharat.sa...@kloudspot.com.INVALID> wrote: > > > > Hi Wei/Jayanth, > > > > Thanks for sharing the details. I am able to fetch out the api and keys > and deployed the driver as suggested by @vivek and GH page. > > > > Now I encountered with one more issue that the cloudstack csi node goes > in CrashLoopBackOff Error. I am trying to get some more info regarding this > which is as below > > > > > {"level":"error","ts":1708932622.5365772,"caller":"zap/options.go:212","msg":"finished > unary call with code > Internal","grpc.start_time":"2024-02-26T07:30:22Z","grpc.request.deadline":"2024-02-26T07:32:22Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeGetInfo","error":"rpc > error: code = Internal desc = Get \" > http://10.1.10.2:8080/client/api?apiKey=k83H56KFdhFqpv7cXPU11nkwxPt8f2rXnm1WWVIRdeErqZr72Pzp7ySmricPWs7FQQuMmClznDhMz7uqnRD2wA&command=listVirtualMachines&id=cf4940eb-52a4-4205-b056-1575926cb488&response=json&signature=t4jdPVL7jqhGt5pWC0kjx%2Bxzr3o%3D\ > <http://10.1.10.2:8080/client/api?apiKey=k83H56KFdhFqpv7cXPU11nkwxPt8f2rXnm1WWVIRdeErqZr72Pzp7ySmricPWs7FQQuMmClznDhMz7uqnRD2wA&command=listVirtualMachines&id=cf4940eb-52a4-4205-b056-1575926cb488&response=json&signature=t4jdPVL7jqhGt5pWC0kjx%2Bxzr3o%3D%5C>": > < > http://10.1.10.2:8080/client/api?apiKey=k83H56KFdhFqpv7cXPU11nkwxPt8f2rXnm1WWVIRdeErqZr72Pzp7ySmricPWs7FQQuMmClznDhMz7uqnRD2wA&command=listVirtualMachines&id=cf4940eb-52a4-4205-b056-1575926cb488&response=json&signature=t4jdPVL7jqhGt5pWC0kjx%2Bxzr3o%3D\%22: > <http://10.1.10.2:8080/client/api?apiKey=k83H56KFdhFqpv7cXPU11nkwxPt8f2rXnm1WWVIRdeErqZr72Pzp7ySmricPWs7FQQuMmClznDhMz7uqnRD2wA&command=listVirtualMachines&id=cf4940eb-52a4-4205-b056-1575926cb488&response=json&signature=t4jdPVL7jqhGt5pWC0kjx%2Bxzr3o%3D%5C%22:>> > dial tcp 10.1.10.2:8080: connect: connection > refused","grpc.code":"Internal","grpc.time_ms":1.138,"stacktrace":" > github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.DefaultMessageProducer\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/logging/zap/options.go:212\ngithub.com/grpc-ecosystem > <http://github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.DefaultMessageProducer%5Cn%5Ct/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/logging/zap/options.go:212%5Cngithub.com/grpc-ecosystem<https://checkpoint.url-protection.com/v1/url?o=http%3A//github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.DefaultMessageProducer%255Cn%255Ct/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware%40v1.4.0/logging/zap/options.go%3A212%255Cngithub.com/grpc-ecosystem&g=NGY3ZjJlOTMwNWIwY2Y2ZQ==&h=Y2YxZWQ3NzQzODg1MDFmZjU2ZGVhZDg4MGQ0NmExNjIxN2JkNDFhOTZjZWVhZWQ2YjVmOGM4N2M5MWQ3ZmUyOQ==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==>> > < > http://github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.DefaultMessageProducer/n/t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/logging/zap/options.go:212/ngithub.com/grpc-ecosystem<https://checkpoint.url-protection.com/v1/url?o=http%3A//github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.DefaultMessageProducer/n/t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware%40v1.4.0/logging/zap/options.go%3A212/ngithub.com/grpc-ecosystem&g=NTZjYzE4ZGQ0NDAwNDYyZg==&h=YTA2ODFkZWY5M2NmMmMwYjIwY2NjMTQ0N2RmMzJmMTdmOGViMzI5ZjQ3ZDFmNDEyNDM5YWQzZDMyM2IxN2ZiNA==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==> > >/go-grpc-middleware/logging/zap.UnaryServerInterceptor.func1\n\t/home/runner/go/pkg/mod/ > github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/logging/zap/server_interceptors.go:39\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.60.1/server.go:1183\ngithub.com/container-storage-interface/spec/lib/go/csi > <http://github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/logging/zap/server_interceptors.go:39%5Cngoogle.golang.org/grpc.chainUnaryInterceptors.func1%5Cn%5Ct/home/runner/go/pkg/mod/google.golang.org/grpc@v1.60.1/server.go:1183%5Cngithub.com/container-storage-interface/spec/lib/go/csi<https://checkpoint.url-protection.com/v1/url?o=http%3A//github.com/grpc-ecosystem/go-grpc-middleware%40v1.4.0/logging/zap/server_interceptors.go%3A39%255Cngoogle.golang.org/grpc.chainUnaryInterceptors.func1%255Cn%255Ct/home/runner/go/pkg/mod/google.golang.org/grpc%40v1.60.1/server.go%3A1183%255Cngithub.com/container-storage-interface/spec/lib/go/csi&g=ZGZjNWZhZjYxM2FhOTJjYQ==&h=YTQ3MzYwZjU0ZWE1MjBlZWI3N2MxMmQ0NGMzYjYzYWY5YWI3ODJlOTgxZTUyMGQ1MDY5MGRhMzdiNThmODZjYw==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==>> > < > http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1/n/t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.60.1/server.go:1183/ngithub.com/container-storage-interface/spec/lib/go/csi<https://checkpoint.url-protection.com/v1/url?o=http%3A//ngoogle.golang.org/grpc.chainUnaryInterceptors.func1/n/t/home/runner/go/pkg/mod/google.golang.org/grpc%40v1.60.1/server.go%3A1183/ngithub.com/container-storage-interface/spec/lib/go/csi&g=MThhNDViZWY0NTI0MjhiMQ==&h=Mjk3NGI1MGNjNGU3NGFkMDIwNTJkNjdhZjFjNWNkOGY4OWE5OTQwZmI3MmM3MDk4NDMwZDk2NGRjMDIyMjUwZg==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==> > >._Node_NodeGetInfo_Handler\n\t/home/runner/go/pkg/mod/ > github.com/container-storage-interface/spec@v1.9.0/lib/go/csi/csi.pb.go:7351\ngoogle.golang.org/grpc > <http://github.com/container-storage-interface/spec@v1.9.0/lib/go/csi/csi.pb.go:7351%5Cngoogle.golang.org/grpc<https://checkpoint.url-protection.com/v1/url?o=http%3A//github.com/container-storage-interface/spec%40v1.9.0/lib/go/csi/csi.pb.go%3A7351%255Cngoogle.golang.org/grpc&g=YmNlY2M2YzhkNmI2NGI2Yg==&h=YzQ4OWZiZWY0OGVlODYzMWZmZDdlN2MwNTM1MjkxMGRhZTAwZDMxMGFjYjkyODU1YjJjYjE2MmMzODBjOTVhYg==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==>> > < > http://github.com/container-storage-interface/spec@v1.9.0/lib/go/csi/csi.pb.go:7351/ngoogle.golang.org/grpc>.(*Server<http://github.com/container-storage-interface/spec@v1.9.0/lib/go/csi/csi.pb.go:7351/ngoogle.golang.org/grpc%3e.(*Server><http://github.com/container-storage-interface/spec@v1.9.0/lib/go/csi/csi.pb.go:7351/ngoogle.golang.org/grpc%3e.(*Server%3chttp:/github.com/container-storage-interface/spec@v1.9.0/lib/go/csi/csi.pb.go:7351/ngoogle.golang.org/grpc%3e.(*Server%3e><https://checkpoint.url-protection.com/v1/url?o=http%3A//github.com/container-storage-interface/spec%40v1.9.0/lib/go/csi/csi.pb.go%3A7351/ngoogle.golang.org/grpc%253e.%28%2AServer%253chttp%3A/github.com/container-storage-interface/spec%40v1.9.0/lib/go/csi/csi.pb.go%3A7351/ngoogle.golang.org/grpc%253e.%28%2AServer%253e%253chttp%3A/github.com/container-storage-interface/spec%40v1.9.0/lib/go/csi/csi.pb.go%3A7351/ngoogle.golang.org/grpc%253e.%28%2AServer%253chttp%3A/github.com/container-storage-interface/spec%40v1.9.0/lib/go/csi/csi.pb.go%3A7351/ngoogle.golang.org/grpc%253e.%28%2AServer%253e%253e&g=NzdhNzBlZTUyMTgwZmJkYQ==&h=NTk4YmFlOWE5N2Q2ZGI1YmU1NzczNjJkNWE4MmFjZDExNzRjODU3YWE4MGE4MWEzMDUzMmIzY2UxMGNjMTJkNg==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==> > ).processUnaryRPC\n\t/home/runner/go/pkg/mod/ > google.golang.org/grpc@v1.60.1/server.go:1372\ngoogle.golang.org/grpc > <http://google.golang.org/grpc@v1.60.1/server.go:1372%5Cngoogle.golang.org/grpc<https://checkpoint.url-protection.com/v1/url?o=http%3A//google.golang.org/grpc%40v1.60.1/server.go%3A1372%255Cngoogle.golang.org/grpc&g=OWZlM2JhMjViOWM2NDZhYg==&h=YzE5MGMzYTllYTllZGZiZmFiMDgyMjhmZGRmNGE1MDZhMmQ4M2U4YTdlYTU5OTgzNmUxZGZiOWM2MWJmOTJlYQ==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==>> > < > http://google.golang.org/grpc@v1.60.1/server.go:1372/ngoogle.golang.org/grpc>.(*Server<http://google.golang.org/grpc@v1.60.1/server.go:1372/ngoogle.golang.org/grpc%3e.(*Server><http://google.golang.org/grpc@v1.60.1/server.go:1372/ngoogle.golang.org/grpc%3e.(*Server%3chttp:/google.golang.org/grpc@v1.60.1/server.go:1372/ngoogle.golang.org/grpc%3e.(*Server%3e><https://checkpoint.url-protection.com/v1/url?o=http%3A//google.golang.org/grpc%40v1.60.1/server.go%3A1372/ngoogle.golang.org/grpc%253e.%28%2AServer%253chttp%3A/google.golang.org/grpc%40v1.60.1/server.go%3A1372/ngoogle.golang.org/grpc%253e.%28%2AServer%253e%253chttp%3A/google.golang.org/grpc%40v1.60.1/server.go%3A1372/ngoogle.golang.org/grpc%253e.%28%2AServer%253chttp%3A/google.golang.org/grpc%40v1.60.1/server.go%3A1372/ngoogle.golang.org/grpc%253e.%28%2AServer%253e%253e&g=YTI5MTljNzVmMTBkZTlmMQ==&h=YzNjN2ZmYTU4MjgyZTQ3NDI1ZTc4ODcxZDYyMThlYjRjMjI3Y2VhMjU0ZTU4MDg1NmUxYzgwN2VmNmFmYTgxYw==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==> > ).handleStream\n\t/home/runner/go/pkg/mod/ > google.golang.org/grpc@v1.60.1/server.go:1783\ngoogle.golang.org/grpc > <http://google.golang.org/grpc@v1.60.1/server.go:1783%5Cngoogle.golang.org/grpc<https://checkpoint.url-protection.com/v1/url?o=http%3A//google.golang.org/grpc%40v1.60.1/server.go%3A1783%255Cngoogle.golang.org/grpc&g=MGVhZTIxOGUxY2U0YzQ4MQ==&h=MmJmZmVmY2M4ODg2NmRkZjkyNmY2NDdmMmFlNWVhNTIyYjY2YmMwOGQ1NDhmNDM2NDgwMmJhNTAxMjBhMWRmZg==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==>> > < > http://google.golang.org/grpc@v1.60.1/server.go:1783/ngoogle.golang.org/grpc>.(*Server<http://google.golang.org/grpc@v1.60.1/server.go:1783/ngoogle.golang.org/grpc%3e.(*Server><http://google.golang.org/grpc@v1.60.1/server.go:1783/ngoogle.golang.org/grpc%3e.(*Server%3chttp:/google.golang.org/grpc@v1.60.1/server.go:1783/ngoogle.golang.org/grpc%3e.(*Server%3e><https://checkpoint.url-protection.com/v1/url?o=http%3A//google.golang.org/grpc%40v1.60.1/server.go%3A1783/ngoogle.golang.org/grpc%253e.%28%2AServer%253chttp%3A/google.golang.org/grpc%40v1.60.1/server.go%3A1783/ngoogle.golang.org/grpc%253e.%28%2AServer%253e%253chttp%3A/google.golang.org/grpc%40v1.60.1/server.go%3A1783/ngoogle.golang.org/grpc%253e.%28%2AServer%253chttp%3A/google.golang.org/grpc%40v1.60.1/server.go%3A1783/ngoogle.golang.org/grpc%253e.%28%2AServer%253e%253e&g=N2Y1ZjNkZWI0ZTA1N2EwYw==&h=OGFkOTU3YjAxOThhZjBkNTRhMTI5MDNlNDk3Y2NiZTc5Y2VmNzY3ZTVlZjI5OTE5NzdmNjc3NGQ3MWIzYTZhYg==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==> > ).serveStreams.func2.1\n\t/home/runner/go/pkg/mod/ > google.golang.org/grpc@v1.60.1/server.go:1016 < > http://google.golang.org/grpc@v1.60.1/server.go:1016>"}<http://google.golang.org/grpc@v1.60.1/server.go:1016%3e%22%7d><http://google.golang.org/grpc@v1.60.1/server.go:1016%3e%22%7d%3chttp:/google.golang.org/grpc@v1.60.1/server.go:1016%3e%22%7d%3e><https://checkpoint.url-protection.com/v1/url?o=http%3A//google.golang.org/grpc%40v1.60.1/server.go%3A1016%253e%2522%257d%253chttp%3A/google.golang.org/grpc%40v1.60.1/server.go%3A1016%253e%2522%257d%253e%253chttp%3A/google.golang.org/grpc%40v1.60.1/server.go%3A1016%253e%2522%257d%253chttp%3A/google.golang.org/grpc%40v1.60.1/server.go%3A1016%253e%2522%257d%253e%253e&g=ZWQxZjM2NTUwZDY4YzU2OA==&h=ZDM1MGFlZmRmNzY1NGNiMzZjNzVhYmYxMDk0Y2Q2MjI5ZmY0MWY1MTcyZGNiNWMwYjJkNGNiYjkzZjg0MjVjYw==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==> > > > > kubectl get pods -A > > NAMESPACE > NAME READY > STATUS RESTARTS AGE > > default > example-pod 0/1 > Pending 0 87m > > kube-system > cloud-controller-manager-574bcb86c-vzp4m 1/1 > Running 0 155m > > kube-system > cloudstack-csi-controller-7f89c8cd47-ftgnf 5/5 > Running 0 150m > > kube-system > cloudstack-csi-controller-7f89c8cd47-j4s4z 5/5 > Running 0 150m > > kube-system > cloudstack-csi-controller-7f89c8cd47-ptvss 5/5 > Running 0 150m > > kube-system > cloudstack-csi-node-56hxg 2/3 > CrashLoopBackOff 34 (99s ago) 150m > > kube-system > cloudstack-csi-node-98cf2 2/3 > CrashLoopBackOff 34 (39s ago) 150m > > kube-system > coredns-5dd5756b68-5wwxk 1/1 > Running 0 4h17m > > kube-system > coredns-5dd5756b68-mbpwt 1/1 > Running 0 4h17m > > kube-system > etcd-kspot-app-control-18de3ee6b6f 1/1 > Running 0 4h17m > > kube-system > kube-apiserver-kspot-app-control-18de3ee6b6f 1/1 > Running 0 4h17m > > kube-system > kube-controller-manager-kspot-app-control-18de3ee6b6f 1/1 > Running 0 4h17m > > kube-system > kube-proxy-56r4l 1/1 > Running 0 4h17m > > kube-system > kube-proxy-mf6cc 1/1 > Running 0 4h17m > > kube-system > kube-scheduler-kspot-app-control-18de3ee6b6f 1/1 > Running 0 4h17m > > kube-system > weave-net-59t9z 2/2 > Running 1 (4h17m ago) 4h17m > > kube-system > weave-net-7xvpp 2/2 > Running 0 4h17m > > kubernetes-dashboard > dashboard-metrics-scraper-5657497c4c-g89lq 1/1 > Running 0 4h17m > > kubernetes-dashboard > kubernetes-dashboard-5b749d9495-fqplb 1/1 > Running 0 4h17m > > > > kubectl get csinode > > NAME DRIVERS AGE > > kspot-app-control-18de3ee6b6f 0 4h23m > > kspot-app-node-18de3eeb7b7 0 4h23m > > > > kubectl describe csinode > > Name: kspot-app-control-18de3ee6b6f > > Labels: <none> > > Annotations: storage.alpha.kubernetes.io/migrated-plugins: < > http://storage.alpha.kubernetes.io/migrated-plugins:><http://storage.alpha.kubernetes.io/migrated-plugins:%3e><http://storage.alpha.kubernetes.io/migrated-plugins:%3e%3chttp:/storage.alpha.kubernetes.io/migrated-plugins:%3e%3e><https://checkpoint.url-protection.com/v1/url?o=http%3A//storage.alpha.kubernetes.io/migrated-plugins%3A%253e%253chttp%3A/storage.alpha.kubernetes.io/migrated-plugins%3A%253e%253e%253chttp%3A/storage.alpha.kubernetes.io/migrated-plugins%3A%253e%253chttp%3A/storage.alpha.kubernetes.io/migrated-plugins%3A%253e%253e%253e&g=MzEyZjAyODUyMWVkNWU3Ng==&h=ZDY3YWNkOTBkNWNmY2Q4N2RkMWE5ZDdiZDM4ODcyMDM4NzUzYTA4MTczYTZhMzE2OWIzMGU1MmE5NzBiYTkwYg==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==> > > > kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/vsphere-vo > < > http://kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/vsphere-vo<https://checkpoint.url-protection.com/v1/url?o=http%3A//kubernetes.io/aws-ebs%2Ckubernetes.io/azure-disk%2Ckubernetes.io/azure-file%2Ckubernetes.io/cinder%2Ckubernetes.io/gce-pd%2Ckubernetes.io/vsphere-vo&g=MTQ4NjJkYTcxOTZlMjkwNg==&h=ZDYwMTY4M2MzNGFhZDlkMDFjZjBjNDc5MTA4N2ExMDk4Y2ViM2Y3ZjFiODRlOTczYTdkOTcxMTg3NjQ1YjQ1OA==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==> > >... > > CreationTimestamp: Mon, 26 Feb 2024 05:42:57 +0000 > > Spec: > > Events: <none> > > Name: kspot-app-node-18de3eeb7b7 > > Labels: <none> > > Annotations: storage.alpha.kubernetes.io/migrated-plugins: < > http://storage.alpha.kubernetes.io/migrated-plugins:><http://storage.alpha.kubernetes.io/migrated-plugins:%3e><http://storage.alpha.kubernetes.io/migrated-plugins:%3e%3chttp:/storage.alpha.kubernetes.io/migrated-plugins:%3e%3e><https://checkpoint.url-protection.com/v1/url?o=http%3A//storage.alpha.kubernetes.io/migrated-plugins%3A%253e%253chttp%3A/storage.alpha.kubernetes.io/migrated-plugins%3A%253e%253e%253chttp%3A/storage.alpha.kubernetes.io/migrated-plugins%3A%253e%253chttp%3A/storage.alpha.kubernetes.io/migrated-plugins%3A%253e%253e%253e&g=NDM2ZTZjMmUxNTQ5NGZhNw==&h=ZTE1ZWQ2NWRlOWI5NzYwODMzMmNjZWQ0MDA1NDUxZTRmMTQxZjM5ZDBlODBkOTc5MDZjNGI4MjMxNmNjZmY5ZA==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==> > > > kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/vsphere-vo > < > http://kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/vsphere-vo<https://checkpoint.url-protection.com/v1/url?o=http%3A//kubernetes.io/aws-ebs%2Ckubernetes.io/azure-disk%2Ckubernetes.io/azure-file%2Ckubernetes.io/cinder%2Ckubernetes.io/gce-pd%2Ckubernetes.io/vsphere-vo&g=MGU4NDRjMjFlYTc1ZWViYw==&h=NmFhZjQ4NmNjNWI0OGJkNzNkZmU2ZTY3MTExMDYwN2FhNGQyYjFhYTQ5Nzc5NjQ3MThhZmRjMWI4YjJmMzA2Ng==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==> > >... > > CreationTimestamp: Mon, 26 Feb 2024 05:43:12 +0000 > > Spec: > > Events: <none> > > > > > > Thanks and Regards, > > Bharat Saini > > > > > > > > From: Wei ZHOU <ustcweiz...@gmail.com <mailto:ustcweiz...@gmail.com > <ustcweiz...@gmail.com>>> > > Date: Monday, 26 February 2024 at 1:52 AM > > To: users@cloudstack.apache.org <mailto:users@cloudstack.apache.org > <users@cloudstack.apache.org>> <users@cloudstack.apache.org < > mailto:users@cloudstack.apache.org <users@cloudstack.apache.org>>> > > Subject: Re: CKS Storage Provisioner Info > > > > EXTERNAL EMAIL: Please verify the sender email address before taking any > action, replying, clicking any link or opening any attachment. > > > > > > +1 > > > > Or use the api key of "admin" user. > > > > -Wei > > > > On Sun, Feb 25, 2024 at 7:57 PM Jayanth Reddy < > jayanthreddy5...@gmail.com <mailto:jayanthreddy5...@gmail.com > <jayanthreddy5...@gmail.com>>> > > wrote: > > > > > Hello Bharat, > > > With your login as "admin" user, you should be able to generate keys > for > > > any user. Please do the below > > > > > > 1. Go to "Accounts" > > > 2. Select the account named "admin" > > > 3. Scroll down and click "users" > > > 4. Select the "admin-kubeadmin" user > > > 5. Then select the button for generation of keys. > > > > > > Please let me know if that helps. > > > > > > Thanks, > > > Jayanth > > > > > > Sent from Outlook for > > > Android<https://aka.ms/AAb9ysg<https://checkpoint.url-protection.com/v1/url?o=https%3A//aka.ms/AAb9ysg&g=ODI0YTE5YWNlMTUzOTAzNg==&h=YjYxNzQ1MzY3YTAxYjc3YTNiYzRkOTM5NTRhN2YyNzdmOTczYjdkNjVhNTU5NjgzNmI3YWUxM2Y0NzkyZjYxYg==&p=Y3AxZTpueHRnZW5pbmZpbml0ZWRhdGFjZW50ZXI6YzpvOmJiOWM1NjZhMzkyOWRmODY4MWZkMWFkYzRkMmU1MDQ4OnYxOmg6VA==>> > > > > > > > > > > > --------------------------- Disclaimer: ------------------------------ > > This message and its contents are intended solely for the designated > addressee and are proprietary to Kloudspot. The information in this email > is meant exclusively for Kloudspot business use. Any use by individuals > other than the addressee constitutes misuse and an infringement of > Kloudspot's proprietary rights. If you are not the intended recipient, > please return this email to the sender. Kloudspot cannot guarantee the > security or error-free transmission of e-mail communications. Information > could be intercepted, corrupted, lost, destroyed, arrive late or > incomplete, or contain viruses. Therefore, Kloudspot shall not be liable > for any issues arising from the transmission of this email. > > > > > -- > This message is intended only for the use of the individual or entity to > which it is addressed and may contain confidential and/or privileged > information. If you are not the intended recipient, please delete the > original message and any copy of it from your computer system. You are > hereby notified that any dissemination, distribution or copying of this > communication is strictly prohibited unless proper authorization has been > obtained for such action. If you have received this communication in error, > please notify the sender immediately. Although IndiQus attempts to sweep > e-mail and attachments for viruses, it does not guarantee that both are > virus-free and accepts no liability for any damage sustained as a result of > viruses. > > --------------------------- Disclaimer: ------------------------------ > This message and its contents are intended solely for the designated > addressee and are proprietary to Kloudspot. The information in this email > is meant exclusively for Kloudspot business use. Any use by individuals > other than the addressee constitutes misuse and an infringement of > Kloudspot's proprietary rights. If you are not the intended recipient, > please return this email to the sender. Kloudspot cannot guarantee the > security or error-free transmission of e-mail communications. Information > could be intercepted, corrupted, lost, destroyed, arrive late or > incomplete, or contain viruses. Therefore, Kloudspot shall not be liable > for any issues arising from the transmission of this email. > --------------------------- Disclaimer: ------------------------------ This message and its contents are intended solely for the designated addressee and are proprietary to Kloudspot. The information in this email is meant exclusively for Kloudspot business use. Any use by individuals other than the addressee constitutes misuse and an infringement of Kloudspot's proprietary rights. If you are not the intended recipient, please return this email to the sender. Kloudspot cannot guarantee the security or error-free transmission of e-mail communications. Information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Therefore, Kloudspot shall not be liable for any issues arising from the transmission of this email. --------------------------- Disclaimer: ------------------------------ This message and its contents are intended solely for the designated addressee and are proprietary to Kloudspot. The information in this email is meant exclusively for Kloudspot business use. Any use by individuals other than the addressee constitutes misuse and an infringement of Kloudspot's proprietary rights. If you are not the intended recipient, please return this email to the sender. Kloudspot cannot guarantee the security or error-free transmission of e-mail communications. Information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Therefore, Kloudspot shall not be liable for any issues arising from the transmission of this email. Disclaimer *** This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not authorised to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful and strictly prohibited. This e-mail may contain viruses. NxtGen Datacenter & Cloud Technologies Private Ltd (“NxtGen”) has taken every reasonable precaution to minimize this risk but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. NxtGen reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the NxtGen e-mail system. *** End of Disclaimer ***NXTGEN***