Hi Bharath
Note the CKS provisioner works on KVM based cloudstack environment
Regards
Kiran
From: Kiran Chavala <kiran.chav...@shapeblue.com>
Date: Monday, 26 February 2024 at 5:45 PM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: CKS Storage Provisioner Info
Hi Bharat Bhusan
Please follow these steps
1. Deploy a Kubernetes cluster on cluster
NAME STATUS ROLES AGE VERSION
ty-control-18de52c04f7 Ready control-plane 5m11s v1.28.4
ty-node-18de52c4185 Ready <none> 4m55s v1.28.4
kubectl get secrets -A
NAMESPACE NAME TYPE
DATA AGE
kube-system cloudstack-secret Opaque
1 10m
2. Check the deployment, it should be in pending
~ kubectl get deployments -A
NAMESPACE NAME READY UP-TO-DATE
AVAILABLE AGE
kube-system cloudstack-csi-controller 0/1 1 0
46s
3. Edit the deployment and remove the nodeSelector part
~kubectl edit deployment/cloudstack-csi-controller -n kube-system
nodeSelector:
kubernetes.io/os: linux
node-role.kubernetes.io/master: ""
4. Check the deployment again and it should be running state
~kubectl get deployments -A
NAMESPACE NAME READY UP-TO-DATE
AVAILABLE AGE
kube-system cloudstack-csi-controller 1/1 1 1
2m39s
5. Replace the disk offering in the storage class yaml
Provide the custom disk offering UUID (service offerings > disk offering >
custom disk offering
4c518474-5d7b-4285-a07c-c57e214abb3b
vi 0-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cloudstack-custom
provisioner: csi.cloudstack.apache.org
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: false
parameters:
csi.cloudstack.apache.org/disk-offering-id:
4c518474-5d7b-4285-a07c-c57e214abb3b
kubectl apply -f 0-storageclass.yaml
k8s git:(master) ✗ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY
VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
cloudstack-custom csi.cloudstack.apache.org Delete
WaitForFirstConsumer false 72s
6. Apply the pvc yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: example-pvc-block
spec:
storageClassName: cloudstack-custom
volumeMode: Block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
kubectl apply -f pvc-block.yaml
k8s git:(master) ✗ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
AGE
example-pvc-block Pending
cloudstack-custom 31s
7. Apply the pod block yaml
apiVersion: v1
kind: Pod
metadata:
name: example-pod-block
spec:
containers:
- name: example
image: ubuntu
volumeDevices:
- devicePath: "/dev/example-block"
name: example-volume
stdin: true
stdinOnce: true
tty: true
volumes:
- name: example-volume
persistentVolumeClaim:
claimName: example-pvc-block
kubectl apply -f pod-block.yaml
➜ k8s git:(master) ✗ k get pods -A
NAMESPACE NAME READY
STATUS RESTARTS AGE
default example-pod-block 1/1
Running 0 107s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9s default-scheduler Successfully
assigned default/example-pod-block to ty-node-18de52c4185
Normal SuccessfulAttachVolume 6s attachdetach-controller
AttachVolume.Attach succeeded for volume
"pvc-02999f04-dd0a-407c-8805-125c7c56d51b"
Normal SuccessfulMountVolume 4s kubelet
MapVolume.MapPodDevice succeeded for volume
"pvc-02999f04-dd0a-407c-8805-125c7c56d51b" globalMapPath
"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-02999f04-dd0a-407c-8805-125c7c56d51b/dev"
Normal SuccessfulMountVolume 4s kubelet
MapVolume.MapPodDevice succeeded for volume
"pvc-02999f04-dd0a-407c-8805-125c7c56d51b" volumeMapPath
"/var/lib/kubelet/pods/076bd828-7130-4f72-a0ee-29f93043bbb1/volumeDevices/kubernetes.io~csi"
Normal Pulling 3s kubelet Pulling image
"ubuntu"
Regards
Kiran
From: Jayanth Reddy <jayanthreddy5...@gmail.com>
Date: Monday, 26 February 2024 at 4:35 PM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: CKS Storage Provisioner Info
Hello Bharat,
If that is the case, please update your "api-url" variable for the cloud-config.
FWIW, please also visit "endpoint.url" in global configuration to include the
HTTPS endpoint.
Thanks,
Jayanth
Sent from Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Bharat Bhushan Saini <bharat.sa...@kloudspot.com.INVALID>
Sent: Monday, February 26, 2024 4:29:22 pm
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: CKS Storage Provisioner Info
Hi Jayanth,
For to secure communication, I already enabled the https over the management
server and turned off the http.
The API URL only listen over http?
Thanks and Regards,
Bharat Saini
[signature_3317674547]
From: Jayanth Reddy <jayanthreddy5...@gmail.com>
Date: Monday, 26 February 2024 at 4:19 PM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: CKS Storage Provisioner Info
EXTERNAL EMAIL: Please verify the sender email address before taking any
action, replying, clicking any link or opening any attachment.
Are you able to access the CloudStack WebUI in the URL
http://10.1.10.2:8080/client ? As long as k8s nodes have connectivity on
the path /client/api on your management, this should work fine. Perhaps a
host-level firewall in your management server?
Thanks
On Mon, Feb 26, 2024 at 4:08 PM Bharat Bhushan Saini
<bharat.sa...@kloudspot.com.invalid> wrote:
> Hi Vivek,
>
>
>
> Please check the findings
>
>
>
> ping 10.1.x.2
>
> PING 10.1.x.2 (10.1.x.2): 56 data bytes
>
> 64 bytes from 10.1.x.2: icmp_seq=0 ttl=64 time=0.616 ms
>
> 64 bytes from 10.1.x.2: icmp_seq=1 ttl=64 time=0.716 ms
>
> ^C--- 10.1.10.2 ping statistics ---
>
> 2 packets transmitted, 2 packets received, 0% packet loss
>
> round-trip min/avg/max/stddev = 0.616/0.666/0.716/0.050 ms
>
>
>
> ping cloudstack.internal.com
>
> PING cloudstack.internal.com (10.1.x.2): 56 data bytes
>
> 64 bytes from 10.1.x.2: icmp_seq=0 ttl=64 time=0.555 ms
>
> 64 bytes from 10.1.x.2: icmp_seq=1 ttl=64 time=0.620 ms
>
> 64 bytes from 10.1.x.2: icmp_seq=2 ttl=64 time=0.664 ms
>
> ^C--- cloudstack.internal.kloudspot.com ping statistics ---
>
> 3 packets transmitted, 3 packets received, 0% packet loss
>
> round-trip min/avg/max/stddev = 0.555/0.613/0.664/0.045 ms
>
>
>
> telnet 10.1.x.2 8080
>
> Trying 10.1.x.2...
>
> telnet: Unable to connect to remote host: Connection refused
>
>
>
>
>
> I am able to ping the management IP and URL but not able to get access
> from the port as it is not open in the cluster.
> NOTE: I use the management IP in the API URL.
>
>
>
> Thanks and Regards,
>
> Bharat Saini
>
>
>
> [image: signature_4099962424]
>
>
>
> *From: *Vivek Kumar <vivek.ku...@indiqus.com.INVALID>
> *Date: *Monday, 26 February 2024 at 3:49 PM
> *To: *users@cloudstack.apache.org <users@cloudstack.apache.org>
> *Subject: *Re: CKS Storage Provisioner Info
>
> EXTERNAL EMAIL: Please verify the sender email address before taking any
> action, replying, clicking any link or opening any attachment.
>
>
> Hello Bharat,
>
> Is the cloudstack URL is reachable from your cluster, can you manually
> check i.e ping, telnet on that port ?
>
>
>
>
> > On 26-Feb-2024, at 3:43 PM, Bharat Bhushan Saini
> <bharat.sa...@kloudspot.com.INVALID> wrote:
> >
> > Hi Wei/Jayanth,
> >
> > Thanks for sharing the details. I am able to fetch out the api and keys
> and deployed the driver as suggested by @vivek and GH page.
> >
> > Now I encountered with one more issue that the cloudstack csi node goes
> in CrashLoopBackOff Error. I am trying to get some more info regarding this
> which is as below
> >
> >
> {"level":"error","ts":1708932622.5365772,"caller":"zap/options.go:212","msg":"finished
> unary call with code
> Internal","grpc.start_time":"2024-02-26T07:30:22Z","grpc.request.deadline":"2024-02-26T07:32:22Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeGetInfo","error":"rpc
> error: code = Internal desc = Get \"
> http://10.1.10.2:8080/client/api?apiKey=k83H56KFdhFqpv7cXPU11nkwxPt8f2rXnm1WWVIRdeErqZr72Pzp7ySmricPWs7FQQuMmClznDhMz7uqnRD2wA&command=listVirtualMachines&id=cf4940eb-52a4-4205-b056-1575926cb488&response=json&signature=t4jdPVL7jqhGt5pWC0kjx%2Bxzr3o%3D\
> <http://10.1.10.2:8080/client/api?apiKey=k83H56KFdhFqpv7cXPU11nkwxPt8f2rXnm1WWVIRdeErqZr72Pzp7ySmricPWs7FQQuMmClznDhMz7uqnRD2wA&command=listVirtualMachines&id=cf4940eb-52a4-4205-b056-1575926cb488&response=json&signature=t4jdPVL7jqhGt5pWC0kjx%2Bxzr3o%3D%5C>":
> <
> http://10.1.10.2:8080/client/api?apiKey=k83H56KFdhFqpv7cXPU11nkwxPt8f2rXnm1WWVIRdeErqZr72Pzp7ySmricPWs7FQQuMmClznDhMz7uqnRD2wA&command=listVirtualMachines&id=cf4940eb-52a4-4205-b056-1575926cb488&response=json&signature=t4jdPVL7jqhGt5pWC0kjx%2Bxzr3o%3D\%22:
> <http://10.1.10.2:8080/client/api?apiKey=k83H56KFdhFqpv7cXPU11nkwxPt8f2rXnm1WWVIRdeErqZr72Pzp7ySmricPWs7FQQuMmClznDhMz7uqnRD2wA&command=listVirtualMachines&id=cf4940eb-52a4-4205-b056-1575926cb488&response=json&signature=t4jdPVL7jqhGt5pWC0kjx%2Bxzr3o%3D%5C%22:>>
> dial tcp 10.1.10.2:8080: connect: connection
> refused","grpc.code":"Internal","grpc.time_ms":1.138,"stacktrace":"
> github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.DefaultMessageProducer\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/logging/zap/options.go:212\ngithub.com/grpc-ecosystem
> <http://github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.DefaultMessageProducer%5Cn%5Ct/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/logging/zap/options.go:212%5Cngithub.com/grpc-ecosystem>
> <
> http://github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.DefaultMessageProducer/n/t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/logging/zap/options.go:212/ngithub.com/grpc-ecosystem
> >/go-grpc-middleware/logging/zap.UnaryServerInterceptor.func1\n\t/home/runner/go/pkg/mod/
> github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/logging/zap/server_interceptors.go:39\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.60.1/server.go:1183\ngithub.com/container-storage-interface/spec/lib/go/csi
> <http://github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/logging/zap/server_interceptors.go:39%5Cngoogle.golang.org/grpc.chainUnaryInterceptors.func1%5Cn%5Ct/home/runner/go/pkg/mod/google.golang.org/grpc@v1.60.1/server.go:1183%5Cngithub.com/container-storage-interface/spec/lib/go/csi>
> <
> http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1/n/t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.60.1/server.go:1183/ngithub.com/container-storage-interface/spec/lib/go/csi
> >._Node_NodeGetInfo_Handler\n\t/home/runner/go/pkg/mod/
> github.com/container-storage-interface/spec@v1.9.0/lib/go/csi/csi.pb.go:7351\ngoogle.golang.org/grpc
> <http://github.com/container-storage-interface/spec@v1.9.0/lib/go/csi/csi.pb.go:7351%5Cngoogle.golang.org/grpc>
> <
> http://github.com/container-storage-interface/spec@v1.9.0/lib/go/csi/csi.pb.go:7351/ngoogle.golang.org/grpc>.(*Server<http://github.com/container-storage-interface/spec@v1.9.0/lib/go/csi/csi.pb.go:7351/ngoogle.golang.org/grpc%3e.(*Server><http://github.com/container-storage-interface/spec@v1.9.0/lib/go/csi/csi.pb.go:7351/ngoogle.golang.org/grpc%3e.(*Server%3chttp:/github.com/container-storage-interface/spec@v1.9.0/lib/go/csi/csi.pb.go:7351/ngoogle.golang.org/grpc%3e.(*Server%3e>
> ).processUnaryRPC\n\t/home/runner/go/pkg/mod/
> google.golang.org/grpc@v1.60.1/server.go:1372\ngoogle.golang.org/grpc
> <http://google.golang.org/grpc@v1.60.1/server.go:1372%5Cngoogle.golang.org/grpc>
> <
> http://google.golang.org/grpc@v1.60.1/server.go:1372/ngoogle.golang.org/grpc>.(*Server<http://google.golang.org/grpc@v1.60.1/server.go:1372/ngoogle.golang.org/grpc%3e.(*Server><http://google.golang.org/grpc@v1.60.1/server.go:1372/ngoogle.golang.org/grpc%3e.(*Server%3chttp:/google.golang.org/grpc@v1.60.1/server.go:1372/ngoogle.golang.org/grpc%3e.(*Server%3e>
> ).handleStream\n\t/home/runner/go/pkg/mod/
> google.golang.org/grpc@v1.60.1/server.go:1783\ngoogle.golang.org/grpc
> <http://google.golang.org/grpc@v1.60.1/server.go:1783%5Cngoogle.golang.org/grpc>
> <
> http://google.golang.org/grpc@v1.60.1/server.go:1783/ngoogle.golang.org/grpc>.(*Server<http://google.golang.org/grpc@v1.60.1/server.go:1783/ngoogle.golang.org/grpc%3e.(*Server><http://google.golang.org/grpc@v1.60.1/server.go:1783/ngoogle.golang.org/grpc%3e.(*Server%3chttp:/google.golang.org/grpc@v1.60.1/server.go:1783/ngoogle.golang.org/grpc%3e.(*Server%3e>
> ).serveStreams.func2.1\n\t/home/runner/go/pkg/mod/
> google.golang.org/grpc@v1.60.1/server.go:1016 <
> http://google.golang.org/grpc@v1.60.1/server.go:1016>"}<http://google.golang.org/grpc@v1.60.1/server.go:1016%3e%22%7d><http://google.golang.org/grpc@v1.60.1/server.go:1016%3e%22%7d%3chttp:/google.golang.org/grpc@v1.60.1/server.go:1016%3e%22%7d%3e>
> >
> > kubectl get pods -A
> > NAMESPACE
> NAME READY
> STATUS RESTARTS AGE
> > default
> example-pod 0/1
> Pending 0 87m
> > kube-system
> cloud-controller-manager-574bcb86c-vzp4m 1/1
> Running 0 155m
> > kube-system
> cloudstack-csi-controller-7f89c8cd47-ftgnf 5/5
> Running 0 150m
> > kube-system
> cloudstack-csi-controller-7f89c8cd47-j4s4z 5/5
> Running 0 150m
> > kube-system
> cloudstack-csi-controller-7f89c8cd47-ptvss 5/5
> Running 0 150m
> > kube-system
> cloudstack-csi-node-56hxg 2/3
> CrashLoopBackOff 34 (99s ago) 150m
> > kube-system
> cloudstack-csi-node-98cf2 2/3
> CrashLoopBackOff 34 (39s ago) 150m
> > kube-system
> coredns-5dd5756b68-5wwxk 1/1
> Running 0 4h17m
> > kube-system
> coredns-5dd5756b68-mbpwt 1/1
> Running 0 4h17m
> > kube-system
> etcd-kspot-app-control-18de3ee6b6f 1/1
> Running 0 4h17m
> > kube-system
> kube-apiserver-kspot-app-control-18de3ee6b6f 1/1
> Running 0 4h17m
> > kube-system
> kube-controller-manager-kspot-app-control-18de3ee6b6f 1/1
> Running 0 4h17m
> > kube-system
> kube-proxy-56r4l 1/1
> Running 0 4h17m
> > kube-system
> kube-proxy-mf6cc 1/1
> Running 0 4h17m
> > kube-system
> kube-scheduler-kspot-app-control-18de3ee6b6f 1/1
> Running 0 4h17m
> > kube-system
> weave-net-59t9z 2/2
> Running 1 (4h17m ago) 4h17m
> > kube-system
> weave-net-7xvpp 2/2
> Running 0 4h17m
> > kubernetes-dashboard
> dashboard-metrics-scraper-5657497c4c-g89lq 1/1
> Running 0 4h17m
> > kubernetes-dashboard
> kubernetes-dashboard-5b749d9495-fqplb 1/1
> Running 0 4h17m
> >
> > kubectl get csinode
> > NAME DRIVERS AGE
> > kspot-app-control-18de3ee6b6f 0 4h23m
> > kspot-app-node-18de3eeb7b7 0 4h23m
> >
> > kubectl describe csinode
> > Name: kspot-app-control-18de3ee6b6f
> > Labels: <none>
> > Annotations: storage.alpha.kubernetes.io/migrated-plugins: <
> http://storage.alpha.kubernetes.io/migrated-plugins:><http://storage.alpha.kubernetes.io/migrated-plugins:%3e><http://storage.alpha.kubernetes.io/migrated-plugins:%3e%3chttp:/storage.alpha.kubernetes.io/migrated-plugins:%3e%3e>
> >
> kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/vsphere-vo
> <
> http://kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/vsphere-vo
> >...
> > CreationTimestamp: Mon, 26 Feb 2024 05:42:57 +0000
> > Spec:
> > Events: <none>
> > Name: kspot-app-node-18de3eeb7b7
> > Labels: <none>
> > Annotations: storage.alpha.kubernetes.io/migrated-plugins: <
> http://storage.alpha.kubernetes.io/migrated-plugins:><http://storage.alpha.kubernetes.io/migrated-plugins:%3e><http://storage.alpha.kubernetes.io/migrated-plugins:%3e%3chttp:/storage.alpha.kubernetes.io/migrated-plugins:%3e%3e>
> >
> kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/vsphere-vo
> <
> http://kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/vsphere-vo
> >...
> > CreationTimestamp: Mon, 26 Feb 2024 05:43:12 +0000
> > Spec:
> > Events: <none>
> >
> >
> > Thanks and Regards,
> > Bharat Saini
> >
> >
> >
> > From: Wei ZHOU <ustcweiz...@gmail.com <mailto:ustcweiz...@gmail.com
> <ustcweiz...@gmail.com>>>
> > Date: Monday, 26 February 2024 at 1:52 AM
> > To: users@cloudstack.apache.org <mailto:users@cloudstack.apache.org
> <users@cloudstack.apache.org>> <users@cloudstack.apache.org <
> mailto:users@cloudstack.apache.org <users@cloudstack.apache.org>>>
> > Subject: Re: CKS Storage Provisioner Info
> >
> > EXTERNAL EMAIL: Please verify the sender email address before taking any
> action, replying, clicking any link or opening any attachment.
> >
> >
> > +1
> >
> > Or use the api key of "admin" user.
> >
> > -Wei
> >
> > On Sun, Feb 25, 2024 at 7:57 PM Jayanth Reddy <
> jayanthreddy5...@gmail.com <mailto:jayanthreddy5...@gmail.com
> <jayanthreddy5...@gmail.com>>>
> > wrote:
> >
> > > Hello Bharat,
> > > With your login as "admin" user, you should be able to generate keys
> for
> > > any user. Please do the below
> > >
> > > 1. Go to "Accounts"
> > > 2. Select the account named "admin"
> > > 3. Scroll down and click "users"
> > > 4. Select the "admin-kubeadmin" user
> > > 5. Then select the button for generation of keys.
> > >
> > > Please let me know if that helps.
> > >
> > > Thanks,
> > > Jayanth
> > >
> > > Sent from Outlook for Android<https://aka.ms/AAb9ysg>
> > >
> > >
> > >
> > --------------------------- Disclaimer: ------------------------------
> > This message and its contents are intended solely for the designated
> addressee and are proprietary to Kloudspot. The information in this email
> is meant exclusively for Kloudspot business use. Any use by individuals
> other than the addressee constitutes misuse and an infringement of
> Kloudspot's proprietary rights. If you are not the intended recipient,
> please return this email to the sender. Kloudspot cannot guarantee the
> security or error-free transmission of e-mail communications. Information
> could be intercepted, corrupted, lost, destroyed, arrive late or
> incomplete, or contain viruses. Therefore, Kloudspot shall not be liable
> for any issues arising from the transmission of this email.
> >
>
>
> --
> This message is intended only for the use of the individual or entity to
> which it is addressed and may contain confidential and/or privileged
> information. If you are not the intended recipient, please delete the
> original message and any copy of it from your computer system. You are
> hereby notified that any dissemination, distribution or copying of this
> communication is strictly prohibited unless proper authorization has been
> obtained for such action. If you have received this communication in error,
> please notify the sender immediately. Although IndiQus attempts to sweep
> e-mail and attachments for viruses, it does not guarantee that both are
> virus-free and accepts no liability for any damage sustained as a result of
> viruses.
>
> --------------------------- Disclaimer: ------------------------------
> This message and its contents are intended solely for the designated
> addressee and are proprietary to Kloudspot. The information in this email
> is meant exclusively for Kloudspot business use. Any use by individuals
> other than the addressee constitutes misuse and an infringement of
> Kloudspot's proprietary rights. If you are not the intended recipient,
> please return this email to the sender. Kloudspot cannot guarantee the
> security or error-free transmission of e-mail communications. Information
> could be intercepted, corrupted, lost, destroyed, arrive late or
> incomplete, or contain viruses. Therefore, Kloudspot shall not be liable
> for any issues arising from the transmission of this email.
>
--------------------------- Disclaimer: ------------------------------
This message and its contents are intended solely for the designated addressee
and are proprietary to Kloudspot. The information in this email is meant
exclusively for Kloudspot business use. Any use by individuals other than the
addressee constitutes misuse and an infringement of Kloudspot's proprietary
rights. If you are not the intended recipient, please return this email to the
sender. Kloudspot cannot guarantee the security or error-free transmission of
e-mail communications. Information could be intercepted, corrupted, lost,
destroyed, arrive late or incomplete, or contain viruses. Therefore, Kloudspot
shall not be liable for any issues arising from the transmission of this email.
