Re: SNMP Traps Not Working
Here's the cloudstack code for the SNMP appender: https://github.com/apache/cloudstack/blob/4.18/plugins/alert-handlers/snmp-alerts/src/main/java/org/apache/cloudstack/alert/snmp/SnmpTrapAppender.java Here's the appender as configured in my log4j-cloud.xml file: On Thu, Aug 17, 2023 at 4:36 PM João Jandre Paraquetti wrote: > Hello, Willard > > Could you share your whole log4j config file? > > I'm asking because the default one will have the following configuration: > > > > > > > > > > > This is the only "logger" configuration that is using the SNMP appender > by default. And the package that it is using > (`org.apache.cloudstack.alerts`) does not exist. Therefore, it will > never log anything in the SNMP appender. > > After looking at the code, I think that the packages that are meant to > be appended with the SNMP appender are `com.cloud.alert` and > `com.cloud.usage`. The interface of these implementations is in > "org.apache.cloudstack.alert", but still, the configuration there would > be invalid anyways as "org.apache.cloudstack.alerts" (ending with an > "s") does not exist. The implementations of that interface are all in > `com.cloud.alert` and `com.cloud.usage`. Therefore, you could use those > packages instead. Here is an example of how to configure that: > > > > > > > > > > > > On 08/08/2023 08:37, Will Conrad wrote: > > I followed the cloudstack documentation here: > > > > > https://docs.cloudstack.apache.org/en/4.18.0.0/adminguide/management.html?highlight=snmp#configuring-snmp-and-syslog-managers > > > > to get SNMP traps working, however, no traps are being sent. I don't see > > any related errors logged. I see no instances of failed library loads, > but > > I do not see. Do I need to manually install the libsnmp4j library myself? > > > > I know it's reading the log4j config as I get this notification in the > > management server log: > > > > 2023-08-07 18:20:42,842 INFO [c.c.u.LogUtils] (main:null) (logid:) log4j > > configuration found at /etc/cloudstack/management/log4j-cloud.xml > > > > Here is the log4j config for the SNMP appender (note that I have > configured > > syslog as well, and that is working). > > > >> class="org.apache.cloudstack.alert.snmp.SnmpTrapAppender"> > > > > > > > > > > > > > > > > > > > > > class="org.apache.cloudstack.alert.snmp.SnmpEnhancedPatternLayout"> > > > > > > > > > > > > > > > > > > > > Any insight will be greatly appreciated. > > > > Regards, > > > > Willard > > >
Re: SNMP Traps Not Working
It's not using localhost. It's using a valid IPv4 address. Only one NIC exists on the host (other than lo), and tcpdump is listening on that adapter. Nothing is being sent from the server with a destination UDP (or TCP) port 162. On Thu, Aug 17, 2023 at 3:43 PM Simon Weller wrote: > Will, > > Is your mgmt server using an RFC 1918 address, or is it using localhost? > > -Si > > On Thu, Aug 17, 2023 at 2:13 PM Will Conrad .invalid> > wrote: > > > H K, > > > > Thanks for responding. > > > > Not a communication issue. The traps are not being sent. I've turned up > > tcpdump on the mgmt host and monitored for traffic to our trap receiver. > No > > trap ever leaves the server. > > > > Regards, > > > > Will > > > > > > > > On Thu, Aug 17, 2023 at 3:06 PM K B Shiv Kumar > > > wrote: > > > > > Sorry didn't go through it fully. Most likely an SNMP communication > > issue. > > > I hope your SNMP server is enabled to support version 2c and it is not > > > disabled. Nowadays v3 is the default I believe. > > > > > > Regards, > > > Shiv > > > (Sent from mobile device. Please excuse brevity and typos.) > > > > > > On Fri, 18 Aug 2023, 00:32 K B Shiv Kumar, wrote: > > > > > > > Hi Will > > > > > > > > Is your community "public"? Any info in the logs? Did you try Syslog? > > Why > > > > I'm asking is to zero in on SNMP traps vs Syslog or the alerting > > > mechanism > > > > itself. > > > > > > > > Regards, > > > > Shiv > > > > (Sent from mobile device. Please excuse brevity and typos.) > > > > > > > > On Thu, 17 Aug 2023, 23:17 Will Conrad, > .invalid> > > > > wrote: > > > > > > > >> Hi All, > > > >> > > > >> Bump > > > >> > > > >> Any word on this? Does anyone know how to get SNMP working in > > > cloudstack? > > > >> > > > >> On Tue, Aug 8, 2023 at 7:37 AM Will Conrad > > > >> wrote: > > > >> > > > >> > I followed the cloudstack documentation here: > > > >> > > > > >> > > > > >> > > > > >> > > > > > > https://docs.cloudstack.apache.org/en/4.18.0.0/adminguide/management.html?highlight=snmp#configuring-snmp-and-syslog-managers > > > >> > > > > >> > to get SNMP traps working, however, no traps are being sent. I > don't > > > see > > > >> > any related errors logged. I see no instances of failed library > > loads, > > > >> but > > > >> > I do not see. Do I need to manually install the libsnmp4j library > > > >> myself? > > > >> > > > > >> > I know it's reading the log4j config as I get this notification in > > the > > > >> > management server log: > > > >> > > > > >> > 2023-08-07 18:20:42,842 INFO [c.c.u.LogUtils] (main:null) > (logid:) > > > >> log4j > > > >> > configuration found at /etc/cloudstack/management/log4j-cloud.xml > > > >> > > > > >> > Here is the log4j config for the SNMP appender (note that I have > > > >> > configured syslog as well, and that is working). > > > >> > > > > >> > > > >> > class="org.apache.cloudstack.alert.snmp.SnmpTrapAppender"> > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> >> > >> > > class="org.apache.cloudstack.alert.snmp.SnmpEnhancedPatternLayout"> > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > Any insight will be greatly appreciated. > > > >> > > > > >> > Regards, > > > >> > > > > >> > Willard > > > >> > > > > >> > > > > > > > > > > -- > > > This message is intended only for the use of the individual or entity > to > > > which it is addressed and may contain confidential and/or privileged > > > information. If you are not the intended recipient, please delete the > > > original message and any copy of it from your computer system. You are > > > hereby notified that any dissemination, distribution or copying of this > > > communication is strictly prohibited unless proper authorization has > been > > > obtained for such action. If you have received this communication in > > > error, > > > please notify the sender immediately. Although IndiQus attempts to > sweep > > > e-mail and attachments for viruses, it does not guarantee that both are > > > virus-free and accepts no liability for any damage sustained as a > result > > > of > > > viruses. > > > > > >
Re: SNMP Traps Not Working
H K, Thanks for responding. Not a communication issue. The traps are not being sent. I've turned up tcpdump on the mgmt host and monitored for traffic to our trap receiver. No trap ever leaves the server. Regards, Will On Thu, Aug 17, 2023 at 3:06 PM K B Shiv Kumar wrote: > Sorry didn't go through it fully. Most likely an SNMP communication issue. > I hope your SNMP server is enabled to support version 2c and it is not > disabled. Nowadays v3 is the default I believe. > > Regards, > Shiv > (Sent from mobile device. Please excuse brevity and typos.) > > On Fri, 18 Aug 2023, 00:32 K B Shiv Kumar, wrote: > > > Hi Will > > > > Is your community "public"? Any info in the logs? Did you try Syslog? Why > > I'm asking is to zero in on SNMP traps vs Syslog or the alerting > mechanism > > itself. > > > > Regards, > > Shiv > > (Sent from mobile device. Please excuse brevity and typos.) > > > > On Thu, 17 Aug 2023, 23:17 Will Conrad, > > wrote: > > > >> Hi All, > >> > >> Bump > >> > >> Any word on this? Does anyone know how to get SNMP working in > cloudstack? > >> > >> On Tue, Aug 8, 2023 at 7:37 AM Will Conrad > >> wrote: > >> > >> > I followed the cloudstack documentation here: > >> > > >> > > >> > > >> > https://docs.cloudstack.apache.org/en/4.18.0.0/adminguide/management.html?highlight=snmp#configuring-snmp-and-syslog-managers > >> > > >> > to get SNMP traps working, however, no traps are being sent. I don't > see > >> > any related errors logged. I see no instances of failed library loads, > >> but > >> > I do not see. Do I need to manually install the libsnmp4j library > >> myself? > >> > > >> > I know it's reading the log4j config as I get this notification in the > >> > management server log: > >> > > >> > 2023-08-07 18:20:42,842 INFO [c.c.u.LogUtils] (main:null) (logid:) > >> log4j > >> > configuration found at /etc/cloudstack/management/log4j-cloud.xml > >> > > >> > Here is the log4j config for the SNMP appender (note that I have > >> > configured syslog as well, and that is working). > >> > > >> > >> > class="org.apache.cloudstack.alert.snmp.SnmpTrapAppender"> > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> >>> > class="org.apache.cloudstack.alert.snmp.SnmpEnhancedPatternLayout"> > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > Any insight will be greatly appreciated. > >> > > >> > Regards, > >> > > >> > Willard > >> > > >> > > > > -- > This message is intended only for the use of the individual or entity to > which it is addressed and may contain confidential and/or privileged > information. If you are not the intended recipient, please delete the > original message and any copy of it from your computer system. You are > hereby notified that any dissemination, distribution or copying of this > communication is strictly prohibited unless proper authorization has been > obtained for such action. If you have received this communication in > error, > please notify the sender immediately. Although IndiQus attempts to sweep > e-mail and attachments for viruses, it does not guarantee that both are > virus-free and accepts no liability for any damage sustained as a result > of > viruses. >
Re: SNMP Traps Not Working
Hi All, Bump Any word on this? Does anyone know how to get SNMP working in cloudstack? On Tue, Aug 8, 2023 at 7:37 AM Will Conrad wrote: > I followed the cloudstack documentation here: > > > https://docs.cloudstack.apache.org/en/4.18.0.0/adminguide/management.html?highlight=snmp#configuring-snmp-and-syslog-managers > > to get SNMP traps working, however, no traps are being sent. I don't see > any related errors logged. I see no instances of failed library loads, but > I do not see. Do I need to manually install the libsnmp4j library myself? > > I know it's reading the log4j config as I get this notification in the > management server log: > > 2023-08-07 18:20:42,842 INFO [c.c.u.LogUtils] (main:null) (logid:) log4j > configuration found at /etc/cloudstack/management/log4j-cloud.xml > > Here is the log4j config for the SNMP appender (note that I have > configured syslog as well, and that is working). > > class="org.apache.cloudstack.alert.snmp.SnmpTrapAppender"> > > > > > > > > > >class="org.apache.cloudstack.alert.snmp.SnmpEnhancedPatternLayout"> > > > > > > > > > > Any insight will be greatly appreciated. > > Regards, > > Willard >
SNMP Traps Not Working
I followed the cloudstack documentation here: https://docs.cloudstack.apache.org/en/4.18.0.0/adminguide/management.html?highlight=snmp#configuring-snmp-and-syslog-managers to get SNMP traps working, however, no traps are being sent. I don't see any related errors logged. I see no instances of failed library loads, but I do not see. Do I need to manually install the libsnmp4j library myself? I know it's reading the log4j config as I get this notification in the management server log: 2023-08-07 18:20:42,842 INFO [c.c.u.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/management/log4j-cloud.xml Here is the log4j config for the SNMP appender (note that I have configured syslog as well, and that is working). Any insight will be greatly appreciated. Regards, Willard
Re: Password Reset Broken for Redhat and derivatives?
Hello Community, In need of feedback on this thread. What's the status of the documentation I've referred to? Why is it 404? Is cloud-init no longer a supported method for enabling password management? Is a fix being worked on? Really need to know how cloudstack is handling this so I can determine what action I need (or need not) take. Thank you! Regards, Willard On Tue, Jul 18, 2023 at 8:31 AM Will Conrad wrote: > Additional info RE this issue: > > The documentation I referred to here: > https://docs.cloudstack.apache.org/en/latest/adminguide/_cloud_init.html#linux-with-cloud-init > > Was linked to (and still is linked to) from the documentation here: > https://docs.cloudstack.apache.org/en/latest/adminguide/templates.html > (though it is 404 not found) in section "4.Password Management". The > "cloud-init integration" link. > > In the cloud-init-output.log of a machine created from the template I get > this warning: > > 2023-07-18 12:00:22,372 - util.py[WARNING]: Failed to fetch password from > virtual router > (note I have redacted the virtual router IP, but it is what it should be). > > Password reset works fine for our Ubuntu template. > > Regards, > > Willard > > On Mon, Jul 17, 2023 at 2:29 PM Will Conrad > wrote: > >> >> Set up our template images according to documentation located here >> >> >> https://docs.cloudstack.apache.org/en/latest/adminguide/_cloud_init.html#linux-with-cloud-init >> >> However, password reset is no longer working for Redhat and derivatives >> and the above linked documentation is now "404". What gives? >> >> Is cloudstack killing support for redhat and derivatives? Have I stumbled >> on a bug? >> >> Regards, >> >> Willard Conrad >> Devops Engineer >> Hivelocity, LLC >> >> >>
Re: Password Reset Broken for Redhat and derivatives?
Additional info RE this issue: The documentation I referred to here: https://docs.cloudstack.apache.org/en/latest/adminguide/_cloud_init.html#linux-with-cloud-init Was linked to (and still is linked to) from the documentation here: https://docs.cloudstack.apache.org/en/latest/adminguide/templates.html (though it is 404 not found) in section "4.Password Management". The "cloud-init integration" link. In the cloud-init-output.log of a machine created from the template I get this warning: 2023-07-18 12:00:22,372 - util.py[WARNING]: Failed to fetch password from virtual router (note I have redacted the virtual router IP, but it is what it should be). Password reset works fine for our Ubuntu template. Regards, Willard On Mon, Jul 17, 2023 at 2:29 PM Will Conrad wrote: > > Set up our template images according to documentation located here > > > https://docs.cloudstack.apache.org/en/latest/adminguide/_cloud_init.html#linux-with-cloud-init > > However, password reset is no longer working for Redhat and derivatives > and the above linked documentation is now "404". What gives? > > Is cloudstack killing support for redhat and derivatives? Have I stumbled > on a bug? > > Regards, > > Willard Conrad > Devops Engineer > Hivelocity, LLC > > >
Password Reset Broken for Redhat and derivatives?
Set up our template images according to documentation located here https://docs.cloudstack.apache.org/en/latest/adminguide/_cloud_init.html#linux-with-cloud-init However, password reset is no longer working for Redhat and derivatives and the above linked documentation is now "404". What gives? Is cloudstack killing support for redhat and derivatives? Have I stumbled on a bug? Regards, Willard Conrad Devops Engineer Hivelocity, LLC
Re: Direct Download/Bypass Secondary Storage option for templates
Can you elaborate on this statement? "but only if there aren’t VMs using the template as the backing file on that primary storage pool." The documentation here states that deleting a template does not affect any VMs using the template. https://cloudstack.apache.org/api/apidocs-4.18/apis/deleteTemplate.html Does cloudstack have a recommended best practice for managing templates/versions? Say, for instance, If we want to ensure we have the latest cloud image every two weeks for an ubuntu2204 template? Would manually deleting the cached file from primary storage and running the prepareTemplate API call on the template do what I seek?Would it be safe? prepareTemplate API documented here: https://cloudstack.apache.org/api/apidocs-4.18/apis/prepareTemplate.html On Tue, Jun 20, 2023 at 10:51 AM Nicolas Vazquez < nicolas.vazq...@shapeblue.com> wrote: > Hi Will, > > You can refer to the Github repository > https://github.com/apache/cloudstack and submit a PR for this, or can > also raise an issue and describe this functionality as a new > feature/improvement: https://github.com/apache/cloudstack/issues/new. > > The only problem I see with your approach is that only updating secondary > storage will not ensure that new dpeloyments will get the latest version, > since CloudStack keeps copies of the template on primary storage also. New > deployments use the cached templates on primary storage when it is > available to avoid copying from secondary storage. I think that the feature > should remove all the existing copies in primary storage also and replace > them with the latest version, but only if there aren’t VMs using the > template as the backing file on that primary storage pool. > > Regards, > Nicolas Vazquez > > > From: Will Conrad > Date: Tuesday, 20 June 2023 at 08:46 > To: users@cloudstack.apache.org > Subject: Re: Direct Download/Bypass Secondary Storage option for templates > Hi Nicolas, > > Where do we stand on this? Is this something that could be easily updated > or feature added? I'm not familiar with the code base. If I were to explore > the idea of making an update and submitting a PR, where would I start? > > Regards, > > Willard > > On Fri, Jun 16, 2023 at 6:41 AM Will Conrad > wrote: > > > Nicolas, > > > > "In your requirement do you mean to change the template URL and > > re-download the template from a different location or simply download > again > > the template from the same URL after updating the served file?" > > > > For our use case the URL will stay the same, but the file will be updated > > (though I could imagine other use cases where an update of the URL would > be > > useful too). Consider a URL like > > http://some.distro.com/cloud/latest/imgname.img > > where /latest/ automatically takes you to the latest updated version of > > that file over time. We would not be looking to trigger an update on all > of > > the primary storage pools where instances have been deployed, just the > > secondary storage such that new deploys use the updated image. > > > > Over time, distro cloud images get updated with the latest updates (that > > one would receive just by running apt update && apt upgrade, for > instance). > > If we continue to deploy with the same image, updates will cause vm > > deployment to take longer and longer over time, as well as consume more > and > > more network bandwidth, due to the updates each vm will have to perform > at > > time of deployment. > > > > "Afaik there is no supported way to safely do this except manually > > updating the files on the downloaded storage pools (secondary and > primary) > > which should be updated carefully. The tables template_store_ref and > > template_spool_ref indicate the downloaded copies on secondary and > primary > > pools of each template." > > > > I assume you're referring to the question about copying over/updating the > > secondary storage image manually. I consider this idea a hack that could > > cause issues or at the very least be at risk of breaking when a new > > cloudstack update gets deployed. I don't like the idea myself. I'm just > > trying to explore all the options available to us and see where your > > guidance takes me. > > > > The reality is that we can make direct download work, but to do so would > > require additional infrastructure (for instance, a webserver in each > zone > > where we point the URL to, that we now have to maintain and monitor). > > Ideally we would just like to let the secondary storage server(s) fulfill > > their role. > > > > On Thu, Jun 15, 2023 at 10:30 P
Re: Direct Download/Bypass Secondary Storage option for templates
Hi Nicolas, Where do we stand on this? Is this something that could be easily updated or feature added? I'm not familiar with the code base. If I were to explore the idea of making an update and submitting a PR, where would I start? Regards, Willard On Fri, Jun 16, 2023 at 6:41 AM Will Conrad wrote: > Nicolas, > > "In your requirement do you mean to change the template URL and > re-download the template from a different location or simply download again > the template from the same URL after updating the served file?" > > For our use case the URL will stay the same, but the file will be updated > (though I could imagine other use cases where an update of the URL would be > useful too). Consider a URL like > http://some.distro.com/cloud/latest/imgname.img > where /latest/ automatically takes you to the latest updated version of > that file over time. We would not be looking to trigger an update on all of > the primary storage pools where instances have been deployed, just the > secondary storage such that new deploys use the updated image. > > Over time, distro cloud images get updated with the latest updates (that > one would receive just by running apt update && apt upgrade, for instance). > If we continue to deploy with the same image, updates will cause vm > deployment to take longer and longer over time, as well as consume more and > more network bandwidth, due to the updates each vm will have to perform at > time of deployment. > > "Afaik there is no supported way to safely do this except manually > updating the files on the downloaded storage pools (secondary and primary) > which should be updated carefully. The tables template_store_ref and > template_spool_ref indicate the downloaded copies on secondary and primary > pools of each template." > > I assume you're referring to the question about copying over/updating the > secondary storage image manually. I consider this idea a hack that could > cause issues or at the very least be at risk of breaking when a new > cloudstack update gets deployed. I don't like the idea myself. I'm just > trying to explore all the options available to us and see where your > guidance takes me. > > The reality is that we can make direct download work, but to do so would > require additional infrastructure (for instance, a webserver in each zone > where we point the URL to, that we now have to maintain and monitor). > Ideally we would just like to let the secondary storage server(s) fulfill > their role. > > On Thu, Jun 15, 2023 at 10:30 PM Nicolas Vazquez < > nicolas.vazq...@shapeblue.com> wrote: > >> Hi Will, >> >> When registering a template CloudStack usually downloads it first to >> secondary storage and then copies it through the different primary storage >> pools when a VM deployment requires it. The aim of the direct download >> feature is to skip the first step and directly downloading the templates >> into primary storage pools without any secondary storage intervention. >> >> In your requirement do you mean to change the template URL and >> re-download the template from a different location or simply download again >> the template from the same URL after updating the served file? Afaik there >> is no supported way to safely do this except manually updating the files on >> the downloaded storage pools (secondary and primary) which should be >> updated carefully. The tables template_store_ref and template_spool_ref >> indicate the downloaded copies on secondary and primary pools of each >> template. >> >> Regards, >> Nicolas Vazquez >> >> >> From: Will Conrad >> Date: Thursday, 15 June 2023 at 14:47 >> To: users@cloudstack.apache.org >> Subject: Re: Direct Download/Bypass Secondary Storage option for templates >> Nicolas, >> >> The reason we're considering using the directdownload feature is to >> simplify template maintenance/updates. I presume that's what it was >> designed for. We want to be able to, preferably through cloudstack >> functionality, update the template image file associated with a template. >> We planned on achieving this utilizing directdownload to decouple the >> image >> file from the registered template itself when it occurred to us that a >> "regrab" button in the template properties webui or an API call to tell >> secondary storage to redownload the source would very much simplify this >> process. >> >> This brings my questions to: >> >> How difficult would it be to implement something like that? >> >> Is there another way to update the imagefile associated with a template? I >> mean, could we ma
Re: Direct Download/Bypass Secondary Storage option for templates
Nicolas, "In your requirement do you mean to change the template URL and re-download the template from a different location or simply download again the template from the same URL after updating the served file?" For our use case the URL will stay the same, but the file will be updated (though I could imagine other use cases where an update of the URL would be useful too). Consider a URL like http://some.distro.com/cloud/latest/imgname.img where /latest/ automatically takes you to the latest updated version of that file over time. We would not be looking to trigger an update on all of the primary storage pools where instances have been deployed, just the secondary storage such that new deploys use the updated image. Over time, distro cloud images get updated with the latest updates (that one would receive just by running apt update && apt upgrade, for instance). If we continue to deploy with the same image, updates will cause vm deployment to take longer and longer over time, as well as consume more and more network bandwidth, due to the updates each vm will have to perform at time of deployment. "Afaik there is no supported way to safely do this except manually updating the files on the downloaded storage pools (secondary and primary) which should be updated carefully. The tables template_store_ref and template_spool_ref indicate the downloaded copies on secondary and primary pools of each template." I assume you're referring to the question about copying over/updating the secondary storage image manually. I consider this idea a hack that could cause issues or at the very least be at risk of breaking when a new cloudstack update gets deployed. I don't like the idea myself. I'm just trying to explore all the options available to us and see where your guidance takes me. The reality is that we can make direct download work, but to do so would require additional infrastructure (for instance, a webserver in each zone where we point the URL to, that we now have to maintain and monitor). Ideally we would just like to let the secondary storage server(s) fulfill their role. On Thu, Jun 15, 2023 at 10:30 PM Nicolas Vazquez < nicolas.vazq...@shapeblue.com> wrote: > Hi Will, > > When registering a template CloudStack usually downloads it first to > secondary storage and then copies it through the different primary storage > pools when a VM deployment requires it. The aim of the direct download > feature is to skip the first step and directly downloading the templates > into primary storage pools without any secondary storage intervention. > > In your requirement do you mean to change the template URL and re-download > the template from a different location or simply download again the > template from the same URL after updating the served file? Afaik there is > no supported way to safely do this except manually updating the files on > the downloaded storage pools (secondary and primary) which should be > updated carefully. The tables template_store_ref and template_spool_ref > indicate the downloaded copies on secondary and primary pools of each > template. > > Regards, > Nicolas Vazquez > > > From: Will Conrad > Date: Thursday, 15 June 2023 at 14:47 > To: users@cloudstack.apache.org > Subject: Re: Direct Download/Bypass Secondary Storage option for templates > Nicolas, > > The reason we're considering using the directdownload feature is to > simplify template maintenance/updates. I presume that's what it was > designed for. We want to be able to, preferably through cloudstack > functionality, update the template image file associated with a template. > We planned on achieving this utilizing directdownload to decouple the image > file from the registered template itself when it occurred to us that a > "regrab" button in the template properties webui or an API call to tell > secondary storage to redownload the source would very much simplify this > process. > > This brings my questions to: > > How difficult would it be to implement something like that? > > Is there another way to update the imagefile associated with a template? I > mean, could we manually overwrite the file on secondary storage? Would that > break anything? > > What is Cloudstack's recommended best practice for managing template > images? > > > Regards, > > Willard (Will) > > > > On Wed, Jun 14, 2023 at 10:26 AM Nicolas Vazquez < > nicolas.vazq...@shapeblue.com> wrote: > > > No problem, I think these docs do not clearly state the supported storage > > providers, I will fix that. On this blog entry we have mentioned them: > > > https://www.shapeblue.com/cloudstack-feature-first-look-direct-download-agnostic-of-the-storage-provider/ > > > > Currently the direct download feature is supporte
Re: Direct Download/Bypass Secondary Storage option for templates
Nicolas, The reason we're considering using the directdownload feature is to simplify template maintenance/updates. I presume that's what it was designed for. We want to be able to, preferably through cloudstack functionality, update the template image file associated with a template. We planned on achieving this utilizing directdownload to decouple the image file from the registered template itself when it occurred to us that a "regrab" button in the template properties webui or an API call to tell secondary storage to redownload the source would very much simplify this process. This brings my questions to: How difficult would it be to implement something like that? Is there another way to update the imagefile associated with a template? I mean, could we manually overwrite the file on secondary storage? Would that break anything? What is Cloudstack's recommended best practice for managing template images? Regards, Willard (Will) On Wed, Jun 14, 2023 at 10:26 AM Nicolas Vazquez < nicolas.vazq...@shapeblue.com> wrote: > No problem, I think these docs do not clearly state the supported storage > providers, I will fix that. On this blog entry we have mentioned them: > https://www.shapeblue.com/cloudstack-feature-first-look-direct-download-agnostic-of-the-storage-provider/ > > Currently the direct download feature is supported on NFS, local storage > and shared mount point, but not for Ceph. > > Regards, > Nicolas Vazquez > > > From: Will Conrad > Date: Wednesday, 14 June 2023 at 10:58 > To: users@cloudstack.apache.org > Subject: Re: Direct Download/Bypass Secondary Storage option for templates > Nicolas, > > I feel silly for not having read that documentation all the way through. > Thank you for your assistance. > > I have another question, now. Since we've been working with this we > have been trying various methods of testing directdownload templates. Since > we were having problems with HTTPS, we tested HTTP. We have run into a > problem where the template fails to download if the guest is using ceph > storage. When we change to creating the VM on "local" storage, the template > download succeeds and the VM creates successfully. Are there any insights > you can provide here? Is there more documentation I may have missed? > > On Wed, Jun 14, 2023 at 9:39 AM Nicolas Vazquez < > nicolas.vazq...@shapeblue.com> wrote: > > > Thanks Will, > > > > Currently it is only possible to upload the certificate via API but not > > from the UI, please find it documented here: > > > https://docs.cloudstack.apache.org/en/latest/adminguide/templates.html#bypassing-secondary-storage-for-kvm-templates > > . > > > > In your case as the template is stored on Github you may want to upload a > > Github certificate to the hosts for the download to be trusted > > > > Regards, > > Nicolas Vazquez > > > > > > From: Will Conrad > > Date: Wednesday, 14 June 2023 at 10:06 > > To: users@cloudstack.apache.org > > Subject: Re: Direct Download/Bypass Secondary Storage option for > templates > > Hi Wei and Nicolas, > > > > Thank you for you responses. > > > > Wei, > > > > I checked the host, and confirmed that yes the ca-certificates package is > > installed and latest. > > "root@lax2-cs-hv01:~# apt list ca-certificates -a > > > > Listing... Done > > > > ca-certificates/jammy-updates,jammy-security,now 20230311ubuntu0.22.04.1 > > all [installed,automatic] > > > > ca-certificates/jammy 20211016 all > > > > > > > > Nicolas, > > > > "Have you tried uploading the required certificate for the https download > > via the uploadTemplateDirectDownloadCertificate API?" > > > > No I have not. I was unaware of the need to do this. Is there > documentation > > I may have missed? What certificate is supposed to be uploaded and how is > > it used? > > > > Regards, > > > > Willard > > > > On Tue, Jun 13, 2023 at 10:01 PM Nicolas Vazquez < > > nicolas.vazq...@shapeblue.com> wrote: > > > > > Hi Will, > > > > > > Have you tried uploading the required certificate for the https > download > > > via the uploadTemplateDirectDownloadCertificate API? > > > > > > Regards, > > > Nicolas Vazquez > > > > > > > > > From: Wei ZHOU > > > Date: Tuesday, 13 June 2023 at 20:01 > > > To: users@cloudstack.apache.org > > > Subject: Re: Direct Download/Bypass Secondary Storage option for > > templates > > > Hi Will, > > > > > > What hyperviso
Re: Direct Download/Bypass Secondary Storage option for templates
Nicolas, I feel silly for not having read that documentation all the way through. Thank you for your assistance. I have another question, now. Since we've been working with this we have been trying various methods of testing directdownload templates. Since we were having problems with HTTPS, we tested HTTP. We have run into a problem where the template fails to download if the guest is using ceph storage. When we change to creating the VM on "local" storage, the template download succeeds and the VM creates successfully. Are there any insights you can provide here? Is there more documentation I may have missed? On Wed, Jun 14, 2023 at 9:39 AM Nicolas Vazquez < nicolas.vazq...@shapeblue.com> wrote: > Thanks Will, > > Currently it is only possible to upload the certificate via API but not > from the UI, please find it documented here: > https://docs.cloudstack.apache.org/en/latest/adminguide/templates.html#bypassing-secondary-storage-for-kvm-templates > . > > In your case as the template is stored on Github you may want to upload a > Github certificate to the hosts for the download to be trusted > > Regards, > Nicolas Vazquez > > > From: Will Conrad > Date: Wednesday, 14 June 2023 at 10:06 > To: users@cloudstack.apache.org > Subject: Re: Direct Download/Bypass Secondary Storage option for templates > Hi Wei and Nicolas, > > Thank you for you responses. > > Wei, > > I checked the host, and confirmed that yes the ca-certificates package is > installed and latest. > "root@lax2-cs-hv01:~# apt list ca-certificates -a > > Listing... Done > > ca-certificates/jammy-updates,jammy-security,now 20230311ubuntu0.22.04.1 > all [installed,automatic] > > ca-certificates/jammy 20211016 all > > > > Nicolas, > > "Have you tried uploading the required certificate for the https download > via the uploadTemplateDirectDownloadCertificate API?" > > No I have not. I was unaware of the need to do this. Is there documentation > I may have missed? What certificate is supposed to be uploaded and how is > it used? > > Regards, > > Willard > > On Tue, Jun 13, 2023 at 10:01 PM Nicolas Vazquez < > nicolas.vazq...@shapeblue.com> wrote: > > > Hi Will, > > > > Have you tried uploading the required certificate for the https download > > via the uploadTemplateDirectDownloadCertificate API? > > > > Regards, > > Nicolas Vazquez > > > > > > From: Wei ZHOU > > Date: Tuesday, 13 June 2023 at 20:01 > > To: users@cloudstack.apache.org > > Subject: Re: Direct Download/Bypass Secondary Storage option for > templates > > Hi Will, > > > > What hypervisor do you use ? Have you installed ca-crrtificates package? > > > > -Wei > > > > On Tuesday, 13 June 2023, Will Conrad > > wrote: > > > > > Hello again, Community! > > > > > > We're trying to make use of DirectDownload templates which makes use of > > the > > > "Bypass Secondary Storage" feature, but we seem to be having issues > with > > > this functionality. > > > > > > After setting up a new template with "Direct Download" turned on and an > > > HTTPS URL our template file won't download. The download source is a > file > > > stored in github. This is what we see in the logs: > > > > > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: WARN > > > [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-5:) > > > (logid:7b08521c) Error downloading template 209 due to: Error on HTTPS > > > request: PKIX path building failed: > > > sun.security.provider.certpath.SunCertPathBuilderException: unable to > > > find valid certification path to requested target > > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: INFO > > > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:) > > > (logid:7b08521c) Trying to fetch storage pool > > > 3b59a095-9e71-3e97-92a8-56aa3f931a5e from libvirt > > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: WARN > > > [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-1:) > > > (logid:7b08521c) Error downloading template 209 due to: Error on HTTPS > > > request: PKIX path building failed: > > > sun.security.provider.certpath.SunCertPathBuilderException: unable to > > > find valid certification path to requested target > > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: INFO > > > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:) > > > (logid:78a6fa93) Trying to fetch storage pool > > > 3b59a095-9e71-3e97-92a8-56aa3f931a5e from libvirt > > > Jun
Re: Direct Download/Bypass Secondary Storage option for templates
Hi Wei and Nicolas, Thank you for you responses. Wei, I checked the host, and confirmed that yes the ca-certificates package is installed and latest. "root@lax2-cs-hv01:~# apt list ca-certificates -a Listing... Done ca-certificates/jammy-updates,jammy-security,now 20230311ubuntu0.22.04.1 all [installed,automatic] ca-certificates/jammy 20211016 all Nicolas, "Have you tried uploading the required certificate for the https download via the uploadTemplateDirectDownloadCertificate API?" No I have not. I was unaware of the need to do this. Is there documentation I may have missed? What certificate is supposed to be uploaded and how is it used? Regards, Willard On Tue, Jun 13, 2023 at 10:01 PM Nicolas Vazquez < nicolas.vazq...@shapeblue.com> wrote: > Hi Will, > > Have you tried uploading the required certificate for the https download > via the uploadTemplateDirectDownloadCertificate API? > > Regards, > Nicolas Vazquez > > > From: Wei ZHOU > Date: Tuesday, 13 June 2023 at 20:01 > To: users@cloudstack.apache.org > Subject: Re: Direct Download/Bypass Secondary Storage option for templates > Hi Will, > > What hypervisor do you use ? Have you installed ca-crrtificates package? > > -Wei > > On Tuesday, 13 June 2023, Will Conrad > wrote: > > > Hello again, Community! > > > > We're trying to make use of DirectDownload templates which makes use of > the > > "Bypass Secondary Storage" feature, but we seem to be having issues with > > this functionality. > > > > After setting up a new template with "Direct Download" turned on and an > > HTTPS URL our template file won't download. The download source is a file > > stored in github. This is what we see in the logs: > > > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: WARN > > [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-5:) > > (logid:7b08521c) Error downloading template 209 due to: Error on HTTPS > > request: PKIX path building failed: > > sun.security.provider.certpath.SunCertPathBuilderException: unable to > > find valid certification path to requested target > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: INFO > > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:) > > (logid:7b08521c) Trying to fetch storage pool > > 3b59a095-9e71-3e97-92a8-56aa3f931a5e from libvirt > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: WARN > > [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-1:) > > (logid:7b08521c) Error downloading template 209 due to: Error on HTTPS > > request: PKIX path building failed: > > sun.security.provider.certpath.SunCertPathBuilderException: unable to > > find valid certification path to requested target > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: INFO > > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:) > > (logid:78a6fa93) Trying to fetch storage pool > > 3b59a095-9e71-3e97-92a8-56aa3f931a5e from libvirt > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: INFO > > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:) > > (logid:78a6fa93) Asking libvirt to refresh storage pool > > 3b59a095-9e71-3e97-92a8-56aa3f931a5e > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: INFO > > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:) > > (logid:7b08521c) Trying to fetch storage pool > > 3b59a095-9e71-3e97-92a8-56aa3f931a5e from libvirt > > Jun 13 16:12:08 lax2-cs-hv01 java[26054]: INFO > > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:) > > (logid:78a6fa93) Trying to fetch storage pool > > eb9f16ef-3ba3-4c50-9e64-807b6f2c8994 from libvirt > > Jun 13 16:12:08 lax2-cs-hv01 java[26054]: INFO > > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:) > > (logid:78a6fa93) Asking libvirt to refresh storage pool > > eb9f16ef-3ba3-4c50-9e64-807b6f2c8994 > > Jun 13 16:12:08 lax2-cs-hv01 java[26054]: WARN > > [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-2:) > > (logid:7b08521c) Error downloading template 209 due to: Error on HTTPS > > request: PKIX path building failed: > > sun.security.provider.certpath.SunCertPathBuilderException: unable to > > find valid certification path to requested target > > > > We've been through this documentation: > > https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html#< > https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html> > > securing-process > > > > but everything seems to be in order, on our side. Any insights here? > > Happy to provide any logs or configuration information to assist. > > > > Regards, > > > > Willard Conrad > > > > DevOps Engineer > > > > Hivelocity, LLC > > > > > >
Direct Download/Bypass Secondary Storage option for templates
Hello again, Community! We're trying to make use of DirectDownload templates which makes use of the "Bypass Secondary Storage" feature, but we seem to be having issues with this functionality. After setting up a new template with "Direct Download" turned on and an HTTPS URL our template file won't download. The download source is a file stored in github. This is what we see in the logs: Jun 13 16:12:07 lax2-cs-hv01 java[26054]: WARN [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-5:) (logid:7b08521c) Error downloading template 209 due to: Error on HTTPS request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Jun 13 16:12:07 lax2-cs-hv01 java[26054]: INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:) (logid:7b08521c) Trying to fetch storage pool 3b59a095-9e71-3e97-92a8-56aa3f931a5e from libvirt Jun 13 16:12:07 lax2-cs-hv01 java[26054]: WARN [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-1:) (logid:7b08521c) Error downloading template 209 due to: Error on HTTPS request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Jun 13 16:12:07 lax2-cs-hv01 java[26054]: INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:) (logid:78a6fa93) Trying to fetch storage pool 3b59a095-9e71-3e97-92a8-56aa3f931a5e from libvirt Jun 13 16:12:07 lax2-cs-hv01 java[26054]: INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:) (logid:78a6fa93) Asking libvirt to refresh storage pool 3b59a095-9e71-3e97-92a8-56aa3f931a5e Jun 13 16:12:07 lax2-cs-hv01 java[26054]: INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:) (logid:7b08521c) Trying to fetch storage pool 3b59a095-9e71-3e97-92a8-56aa3f931a5e from libvirt Jun 13 16:12:08 lax2-cs-hv01 java[26054]: INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:) (logid:78a6fa93) Trying to fetch storage pool eb9f16ef-3ba3-4c50-9e64-807b6f2c8994 from libvirt Jun 13 16:12:08 lax2-cs-hv01 java[26054]: INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:) (logid:78a6fa93) Asking libvirt to refresh storage pool eb9f16ef-3ba3-4c50-9e64-807b6f2c8994 Jun 13 16:12:08 lax2-cs-hv01 java[26054]: WARN [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-2:) (logid:7b08521c) Error downloading template 209 due to: Error on HTTPS request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target We've been through this documentation: https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html#securing-process but everything seems to be in order, on our side. Any insights here? Happy to provide any logs or configuration information to assist. Regards, Willard Conrad DevOps Engineer Hivelocity, LLC
Re: Difference in functionality of Advanced Networking With and Without Security Groups
Thank you for your quick response, Wei. It was helpful. Regards, Willard On Tue, Jun 6, 2023 at 7:36 AM Wei ZHOU wrote: > Hi Will, > > In the advanced zone with security groups, you can only create Shared > networks. L2 and isolated/VPC are not supported. (In my opinion, we could > support L2 as well). > In the advanced zones, you can create Shared/L2/Isolated/VPC, but vms do > not have security groups. > > Advanced zone with SG is suitable for public cloud providers, and advanced > zone without SG is suitable for private clouds. > There is an idea from some years ago, to combine these two types into one, > but not implemented yet. It is very complicated. > > -Wei > > > On Tue, 6 Jun 2023 at 12:45, Will Conrad > wrote: > > > HI Community! > > > > My company is building a cloudstack implementation and have discovered > > that security-group enabled advanced zones seem to function unexpectedly > > differently than non-security-group enabled advanced zones. After > creating > > a security-group enabled advanced zone, when adding new networks to this > > zone, we seem to have lost the choices of "L2" and "isolated". Is this > > normal? Is this the way security groups were designed to function? I did > > read through the documentation for security groups, and noticed the > > "limitations" expressed as well as saw the documentation that VPC are not > > supported in security-group enabled zones. I'm looking for further > > clarification. > > > > As depicted in the below screenshot, "shared" is now the only option > where > > before "L2" and "isolated" were also options. > > > > Have I missed something? Have I misinterpreted something? Is there > further > > documentation that might describe the nuances of using security groups in > > advanced zones? > > > > Any assistance is appreciated. Thank you! > > > > Regards, > > > > Willard Conrad > > DevOps Engineer > > Hivelocity, LLC > > > > [image: image_720.png] > > >
Re: IP Spoofing and IP Theft
How might one go about achieving this functionality without using security groups? Is there another way *through cloudstack* to limit the users' ability to change their instance IP address or otherwise use an arbitrary IP address? For instance, if using a shared network for internet access with a publicly routable class C assigned, a new instance/vm assigned to that network will consume one of those IPs. What's to stop the user from manually changing their IP or manually adding another IP from that subnet, which is effectively "stealing" a second IP (aside from the obvious, that when cloudstack tries to assign that "stolen" IP to another instance there will be IP collisions on the network)? We really need to understand how this functionality works and what we can do to prevent bad actors from being bad actors. Regards, Willard Conrad DevOps Engineer Hivelocity, LLC On Mon, May 22, 2023 at 10:02 AM Will Conrad wrote: > Hi Wei, > > Thanks for your response. Advanced zone is being used with a guest network > type "shared". Disclaimer, I neither setup nor configured this > cloustack zone or instance. How can I tell if security groups were enabled > when the zone was created? At this point, I am leaning towards they > weren't, but need to confirm. > > Regards, > > Willard > > On Mon, May 22, 2023 at 8:40 AM Wei ZHOU wrote: > >> Hi Will, >> >> What type of zone and network do you use ? >> >> As said before, the functionality works in the Advanced zones with >> security >> groups (as well as the Basic zones). >> If you use the advanced zone and isolated networks (it seems so), there is >> no such functionality, as far as I know. >> >> -Wei >> >> >> On Mon, 22 May 2023 at 14:00, Will Conrad > .invalid> >> wrote: >> >> > Thank you everyone, for your responses. >> > >> > I feel the need to further clarify my question: >> > The spoofing and IP theft this thread is concerned with is related to >> bad >> > actors on cloudstack instances attempting to send out traffic as a >> > different IP or attempting to utilize network IPs that aren't/weren't >> > assigned to said VM by cloudstack. >> > >> > Based on some of the responses and a jira ticket from an old cloudstack >> > version: https://issues.apache.org/jira/browse/CLOUDSTACK-8559 >> > I thought I would confirm that the spoofing and IP theft I am >> immediately >> > concerned with would not be an issue. However, I find that I am able to >> > manually modify an instance IP (from within the instance) and maintain >> > connectivity using the modified IP after removing the original >> > cloudstack-assigned IP. >> > >> > Method of modification was using iproute2 tools from within the VM: ip >> addr >> > add ..., ip addr del ..., ip route add ... >> > >> > Example: created new instance, received cloudstack assigned public IP, >> > confirmed working. Logged into instance, manually added "stolen" IP, >> > manually removed cloudstack assigned IP, re-added default gateway, >> tested >> > connectivity. Instance was able to communicate on the internet by both >> > sending and receiving outbound pings, performing DNS resolution, and >> > accepting inbound ssh connects via the new manually added IP. >> > >> > This is contradictory to what I expected. Does something have to be >> done to >> > enable this anti-spoofing functionality? Are there details I am missing? >> > >> > Regards, >> > >> > Willard Conrad >> > DevOps Engineer >> > Hivelocity, LLC >> > >> > >> > >> > On Thu, May 18, 2023 at 11:07 AM Wei ZHOU >> wrote: >> > >> > > Yes, as Jithin said cloudstack uses iptables/ebtables/ipset to >> prevent IP >> > > spoofing in advanced zone with security groups. >> > > >> > > If the IP or mac address of vm instance is modified inside the vm by >> the >> > > user, the vm will not work. >> > > >> > > -Wei >> > > >> > > >> > > On Thursday, 18 May 2023, Jithin Raju >> wrote: >> > > >> > > > Hi Willard, >> > > > >> > > > I believe there is something implemented using iptables,ebtables to >> > > > prevent IP spoofing for security group enabled zones. You need to >> take >> > > this >> > > > into account if you are using security group enabled zones. >> > > > >> > > > -Jithin >> > > > >> > > > From: Will Conrad >> > > > Date: Thursday, 18 May 2023 at 1:08 PM >> > > > To: users@cloudstack.apache.org >> > > > Subject: IP Spoofing and IP Theft >> > > > Hello Community! >> > > > >> > > > It looks like cloudstack has built-iin protection to prevent IP >> > > spoofing, I >> > > > am wondering what kind (if any) of protections cloudstack has >> built-in >> > to >> > > > protect the environment from IP theft, or is this a consideration >> that >> > > > should be taken into account when designing the network layout and >> > > > offerings for tenants? >> > > > >> > > > Regards, >> > > > >> > > > Willard Conrad >> > > > DevOps Engineer >> > > > Hivelocity, LLC >> > > > >> > > > >> > > > >> > > > >> > > >> > >> >
Difference in functionality of Advanced Networking With and Without Security Groups
HI Community! My company is building a cloudstack implementation and have discovered that security-group enabled advanced zones seem to function unexpectedly differently than non-security-group enabled advanced zones. After creating a security-group enabled advanced zone, when adding new networks to this zone, we seem to have lost the choices of "L2" and "isolated". Is this normal? Is this the way security groups were designed to function? I did read through the documentation for security groups, and noticed the "limitations" expressed as well as saw the documentation that VPC are not supported in security-group enabled zones. I'm looking for further clarification. As depicted in the below screenshot, "shared" is now the only option where before "L2" and "isolated" were also options. Have I missed something? Have I misinterpreted something? Is there further documentation that might describe the nuances of using security groups in advanced zones? Any assistance is appreciated. Thank you! Regards, Willard Conrad DevOps Engineer Hivelocity, LLC [image: image_720.png]
Re: IP Spoofing and IP Theft
Hi Wei, Thanks for your response. Advanced zone is being used with a guest network type "shared". Disclaimer, I neither setup nor configured this cloustack zone or instance. How can I tell if security groups were enabled when the zone was created? At this point, I am leaning towards they weren't, but need to confirm. Regards, Willard On Mon, May 22, 2023 at 8:40 AM Wei ZHOU wrote: > Hi Will, > > What type of zone and network do you use ? > > As said before, the functionality works in the Advanced zones with security > groups (as well as the Basic zones). > If you use the advanced zone and isolated networks (it seems so), there is > no such functionality, as far as I know. > > -Wei > > > On Mon, 22 May 2023 at 14:00, Will Conrad > wrote: > > > Thank you everyone, for your responses. > > > > I feel the need to further clarify my question: > > The spoofing and IP theft this thread is concerned with is related to bad > > actors on cloudstack instances attempting to send out traffic as a > > different IP or attempting to utilize network IPs that aren't/weren't > > assigned to said VM by cloudstack. > > > > Based on some of the responses and a jira ticket from an old cloudstack > > version: https://issues.apache.org/jira/browse/CLOUDSTACK-8559 > > I thought I would confirm that the spoofing and IP theft I am immediately > > concerned with would not be an issue. However, I find that I am able to > > manually modify an instance IP (from within the instance) and maintain > > connectivity using the modified IP after removing the original > > cloudstack-assigned IP. > > > > Method of modification was using iproute2 tools from within the VM: ip > addr > > add ..., ip addr del ..., ip route add ... > > > > Example: created new instance, received cloudstack assigned public IP, > > confirmed working. Logged into instance, manually added "stolen" IP, > > manually removed cloudstack assigned IP, re-added default gateway, tested > > connectivity. Instance was able to communicate on the internet by both > > sending and receiving outbound pings, performing DNS resolution, and > > accepting inbound ssh connects via the new manually added IP. > > > > This is contradictory to what I expected. Does something have to be done > to > > enable this anti-spoofing functionality? Are there details I am missing? > > > > Regards, > > > > Willard Conrad > > DevOps Engineer > > Hivelocity, LLC > > > > > > > > On Thu, May 18, 2023 at 11:07 AM Wei ZHOU wrote: > > > > > Yes, as Jithin said cloudstack uses iptables/ebtables/ipset to prevent > IP > > > spoofing in advanced zone with security groups. > > > > > > If the IP or mac address of vm instance is modified inside the vm by > the > > > user, the vm will not work. > > > > > > -Wei > > > > > > > > > On Thursday, 18 May 2023, Jithin Raju > wrote: > > > > > > > Hi Willard, > > > > > > > > I believe there is something implemented using iptables,ebtables to > > > > prevent IP spoofing for security group enabled zones. You need to > take > > > this > > > > into account if you are using security group enabled zones. > > > > > > > > -Jithin > > > > > > > > From: Will Conrad > > > > Date: Thursday, 18 May 2023 at 1:08 PM > > > > To: users@cloudstack.apache.org > > > > Subject: IP Spoofing and IP Theft > > > > Hello Community! > > > > > > > > It looks like cloudstack has built-iin protection to prevent IP > > > spoofing, I > > > > am wondering what kind (if any) of protections cloudstack has > built-in > > to > > > > protect the environment from IP theft, or is this a consideration > that > > > > should be taken into account when designing the network layout and > > > > offerings for tenants? > > > > > > > > Regards, > > > > > > > > Willard Conrad > > > > DevOps Engineer > > > > Hivelocity, LLC > > > > > > > > > > > > > > > > > > > > > >
Re: IP Spoofing and IP Theft
Thank you everyone, for your responses. I feel the need to further clarify my question: The spoofing and IP theft this thread is concerned with is related to bad actors on cloudstack instances attempting to send out traffic as a different IP or attempting to utilize network IPs that aren't/weren't assigned to said VM by cloudstack. Based on some of the responses and a jira ticket from an old cloudstack version: https://issues.apache.org/jira/browse/CLOUDSTACK-8559 I thought I would confirm that the spoofing and IP theft I am immediately concerned with would not be an issue. However, I find that I am able to manually modify an instance IP (from within the instance) and maintain connectivity using the modified IP after removing the original cloudstack-assigned IP. Method of modification was using iproute2 tools from within the VM: ip addr add ..., ip addr del ..., ip route add ... Example: created new instance, received cloudstack assigned public IP, confirmed working. Logged into instance, manually added "stolen" IP, manually removed cloudstack assigned IP, re-added default gateway, tested connectivity. Instance was able to communicate on the internet by both sending and receiving outbound pings, performing DNS resolution, and accepting inbound ssh connects via the new manually added IP. This is contradictory to what I expected. Does something have to be done to enable this anti-spoofing functionality? Are there details I am missing? Regards, Willard Conrad DevOps Engineer Hivelocity, LLC On Thu, May 18, 2023 at 11:07 AM Wei ZHOU wrote: > Yes, as Jithin said cloudstack uses iptables/ebtables/ipset to prevent IP > spoofing in advanced zone with security groups. > > If the IP or mac address of vm instance is modified inside the vm by the > user, the vm will not work. > > -Wei > > > On Thursday, 18 May 2023, Jithin Raju wrote: > > > Hi Willard, > > > > I believe there is something implemented using iptables,ebtables to > > prevent IP spoofing for security group enabled zones. You need to take > this > > into account if you are using security group enabled zones. > > > > -Jithin > > > > From: Will Conrad > > Date: Thursday, 18 May 2023 at 1:08 PM > > To: users@cloudstack.apache.org > > Subject: IP Spoofing and IP Theft > > Hello Community! > > > > It looks like cloudstack has built-iin protection to prevent IP > spoofing, I > > am wondering what kind (if any) of protections cloudstack has built-in to > > protect the environment from IP theft, or is this a consideration that > > should be taken into account when designing the network layout and > > offerings for tenants? > > > > Regards, > > > > Willard Conrad > > DevOps Engineer > > Hivelocity, LLC > > > > > > > > >
IP Spoofing and IP Theft
Hello Community! It looks like cloudstack has built-iin protection to prevent IP spoofing, I am wondering what kind (if any) of protections cloudstack has built-in to protect the environment from IP theft, or is this a consideration that should be taken into account when designing the network layout and offerings for tenants? Regards, Willard Conrad DevOps Engineer Hivelocity, LLC
RE: urgent help needed: primary storage became unplugged after xenserver roboot
I seem to recal there were some clouds tack files overwritten with updates. I'd try redeploy it in cloudstack. I would also take a pool backup first. Also check the XS logs for errors. /var/log/messages and /var/log/SMlog Sent from my Verizon Wireless 4G LTE smartphone Original message From: Yiping Zhang yzh...@marketo.com Date: 01/13/2015 5:43 PM (GMT-05:00) To: users@cloudstack.apache.org Subject: urgent help needed: primary storage became unplugged after xenserver roboot Hi, All: I need some urgent help in restoring my CS instance (version 4.3.1). After patching xenserver 6.2, I rebooted the pool master (I have two xen hypervisors). After the pool master comes back, the SR for the primary storage became “unplugged”, and shown as broken in XenCenter. The volume is not mounted on pool master host any more. How do I get it back without losing any data ? Thanks for all helps. Yiping
RE: 4.2.1 anytime soon?
From what I understood, they hope to respin RC today for a second vote. -Original Message- From: Adrian Lewis [mailto:adr...@alsiconsulting.co.uk] Sent: Monday, December 09, 2013 10:23 AM To: users@cloudstack.apache.org Subject: RE: 4.2.1 anytime soon? Do we have any projected dates for the official 4.2.1 release yet? I understand that there's an ACL disclosure issue that is under discussion as a potential blocker but I can't quite work out whether there has been any progress on this. -Original Message- From: sebgoa [mailto:run...@gmail.com] Sent: 27 November 2013 18:11 To: users@cloudstack.apache.org Subject: Re: 4.2.1 anytime soon? On Nov 27, 2013, at 6:52 PM, Adrian Lewis adr...@alsiconsulting.co.uk wrote: Hi Sebastien, Thanks for the reply and the link - I never knew about that option. Are these RPMs likely to be upgrade-safe if I configure the mgmt. server with the relevant yum repo at a later date? These rpms are built from the 4.2 branch, so they represent the latest on that branch and may differ slightly (be ahead ~4.2.2) from what we voted on for 4.2.1 I have not tried that type of upgrade, which If I understand you right, may actually be a 'downgrade'. That said it should only be bug fix changes so the schemas should not change. My advice would be to wait couple days for the official 4.2.1 but if you are eager you can grab those rpms and start ironing out your upgrade procedure (from where you are at to ~4.2.1) on your dev systems. Cheers, Adrian -Original Message- From: sebgoa [mailto:run...@gmail.com] Sent: 27 November 2013 17:33 To: users@cloudstack.apache.org Subject: Re: 4.2.1 anytime soon? On Nov 27, 2013, at 12:41 PM, Adrian Lewis adr...@alsiconsulting.co.uk wrote: Hi All, Hoping that someone here might know what's holding up the release of 4.2.1? A vote was cast back on the 12th Nov which appeared to be all fine with everyone eventually. Citrix have released Cloudplatform 4.2.1. As far as I can tell, the only thing missing was finalising the release notes but these too seem to be finished. Has CCCEU13 killed off all of the momentum to release 4.2.1 or is there something else going on that I've missed? You have not missed anything. It's just lack of time in the day to check the final docs, make the release announcement etc.. I appreciate that beggars can't be choosers but I'm too scared to try building from source and am eagerly awaiting the 'easy' option of having RPMs made from an officially sanctioned release. You can always get the latest rpms from: http://jenkins.buildacloud.org/view/4.2/job/package-rhel63-4.2/ Even though they are not the official release ones (and won't be since we only release source tar ball) Any info on this welcomed. Many thanks, Adrian
Re: Traditional Windows workloads and Cloudstack
Are you really saturating you GigE link with only 5-10 users. It sounds like you may be running out of IOs, SQL is usually a very write intensive workload. Junaid Shahid shahid.jun...@gmail.com wrote: Thanks Todd! Well I think the service offering is at 200Mbps.. Also I we are not using any link aggregation at all. Let me float these ideas to my team. Thanks for your feedback! On Wed, Nov 13, 2013 at 5:36 PM, Todd Pigram t...@toddpigram.com wrote: Junaid, what did you set the the network rate to in the exchange service offering? Depending on your backend network setup for that offering you may get better results with setting it to a '0' for unlimited. On my internal CCP, our SQL servers service offering has network rate to '0' as I am using a 4 NIC LACP bond. just food for thought Todd On Wed, Nov 13, 2013 at 6:37 AM, Junaid Shahid shahid.jun...@gmail.com wrote: Hi all, We are running a mixture of Windows and Linux VMs under different accounts on our cloud, that is based on CloudPlatform 3 (I know that it's a mailing list for ACS, but I still need your feedback so read on please :)). The Primary storage is based on iSCSI with GigE link, and Xen hyperviser. Now the problem is that whenever we run Windows OSes with applications like Exchange, Sharepoint and particularly MS Lync (that includes AD and MSSQL as pre-requisites..), the GigE link to Primary Storage becomes so congested that it affects the whole cloud environment. Nothing remains usable anymore, the performance of Linux VMs also is affected in the process. So what does your experience say, what should we do: 1) Segregate the Windows VMs to their own cluster and their own separate Primary storage. 2) Use local storage for the pre-cloud era traditional Windows workloads such as MS Exchange etc. 3) Is cloud environment feasible at all for Hosted Exchange and the like, as Local storage that runs on the speed of the motherboard back-plane, of course cannot be matched by a GigE link alone. Awaiting your valuable feedback all :) -- Regards, Junaid Shahid, TODO:__ -- Regards, Junaid Shahid, TODO:__
Re: Traditional Windows workloads and Cloudstack
The additional 5-10 users shouldn't be such an extreme load. How many Mbps were you using with the 5-10 users? I am trying to clarify if the SAN or the storage network is the bottleneck. In either case,as previous stated it does all go back to capacity/workload planning. I know this is getting beyond cloudstack, but on the ZFS box you can run 'zpool iostat -v' to see your IO and throughput averages. Be careful on the dedicated ZIL, it can quickly become a bottleneck if you don't purchase an SSD capably of the load. Junaid Shahid shahid.jun...@gmail.com wrote: Yeah with 5-10 users only :) Also I think we don't have any write-cache (called ZILs in the ZFS lingo, I think) on the storage server too, so SQL would be even more problematic there.. On Wed, Nov 13, 2013 at 5:53 PM, Conrad Geiger cgei...@it1solutions.comwrote: Are you really saturating you GigE link with only 5-10 users. It sounds like you may be running out of IOs, SQL is usually a very write intensive workload. Junaid Shahid shahid.jun...@gmail.com wrote: Thanks Todd! Well I think the service offering is at 200Mbps.. Also I we are not using any link aggregation at all. Let me float these ideas to my team. Thanks for your feedback! On Wed, Nov 13, 2013 at 5:36 PM, Todd Pigram t...@toddpigram.com wrote: Junaid, what did you set the the network rate to in the exchange service offering? Depending on your backend network setup for that offering you may get better results with setting it to a '0' for unlimited. On my internal CCP, our SQL servers service offering has network rate to '0' as I am using a 4 NIC LACP bond. just food for thought Todd On Wed, Nov 13, 2013 at 6:37 AM, Junaid Shahid shahid.jun...@gmail.com wrote: Hi all, We are running a mixture of Windows and Linux VMs under different accounts on our cloud, that is based on CloudPlatform 3 (I know that it's a mailing list for ACS, but I still need your feedback so read on please :)). The Primary storage is based on iSCSI with GigE link, and Xen hyperviser. Now the problem is that whenever we run Windows OSes with applications like Exchange, Sharepoint and particularly MS Lync (that includes AD and MSSQL as pre-requisites..), the GigE link to Primary Storage becomes so congested that it affects the whole cloud environment. Nothing remains usable anymore, the performance of Linux VMs also is affected in the process. So what does your experience say, what should we do: 1) Segregate the Windows VMs to their own cluster and their own separate Primary storage. 2) Use local storage for the pre-cloud era traditional Windows workloads such as MS Exchange etc. 3) Is cloud environment feasible at all for Hosted Exchange and the like, as Local storage that runs on the speed of the motherboard back-plane, of course cannot be matched by a GigE link alone. Awaiting your valuable feedback all :) -- Regards, Junaid Shahid, TODO:__ -- Regards, Junaid Shahid, TODO:__ -- Regards, Junaid Shahid, TODO:__
RE: Are we going to see CS 4.2 rc1 out anytime soon?
They are currently on the third round of voting. Checkout the dev list. -Original Message- From: Ron Wheeler [mailto:rwhee...@artifact-software.com] Sent: Wednesday, August 28, 2013 11:28 AM To: users@cloudstack.apache.org Subject: Re: Are we going to see CS 4.2 rc1 out anytime soon? On 28/08/2013 10:52 AM, Dean Kamali wrote: Hello everyone going over docs, CS 4.2 rc1 should be released today, just wondering if we are going to see it soon. Thanks Still 10 JIRA issues that I reported, open against installation docs. -- Ron Wheeler President Artifact Software Inc email: rwhee...@artifact-software.com skype: ronaldmwheeler phone: 866-970-2435, ext 102
RE: upgrading from CloudPlatform to CloudStack 4.1
What about upgrading to 4.0 first? I seem to recall a possible need for that on the dev list, but can't find it. Original message From: Brian Galura brian.gal...@citrix.com Date: 07/16/2013 7:24 PM (GMT-05:00) To: users@cloudstack.apache.org Subject: RE: upgrading from CloudPlatform to CloudStack 4.1 Looks like this doesn't work and is known. https://issues.apache.org/jira/browse/CLOUDSTACK-2929 ERROR [cloud.upgrade.DatabaseUpgradeChecker] (Timer-1:) There is no upgrade path from 3.0.6.20121222035904 to 4.1.0 -Original Message- From: Brian Galura [mailto:brian.gal...@citrix.com] Sent: Tuesday, July 16, 2013 2:53 PM To: users@cloudstack.apache.org Subject: RE: upgrading from CloudPlatform to CloudStack 4.1 I plan on doing this later today on a test installation. Ill report my results. Ive built the nonoss rpms from source using the latest code from git so it will be interesting. -Original Message- From: Kristoffer Sheather @ CloudCentral [mailto:kristoffer.sheat...@cloudcentral.com.au] Sent: Tuesday, July 16, 2013 2:41 PM To: users@cloudstack.apache.org Subject: re: upgrading from CloudPlatform to CloudStack 4.1 Take a backup of your database and test the upgrade process. Regards, Kristoffer Sheather Cloud Central Scale Your Data Center In The Cloud Phone: 1300 144 007 | Mobile: +61 414 573 130 | Email: k...@cloudcentral.com.au LinkedIn: | Skype: kristoffer.sheather | Twitter: http://twitter.com/kristofferjon From: Brian Galura brian.gal...@citrix.com Sent: Wednesday, July 17, 2013 7:39 AM To: users@cloudstack.apache.org users@cloudstack.apache.org Subject: upgrading from CloudPlatform to CloudStack 4.1 Has anyone tried upgrading from CloudPlatform to a newer version of CloudStack? I know the deployment methods are different but will the db migrations run cleanly?
Re: How many vms per primary storage can offer best performance?
I would also say that 8 spindles for 15-20 VMs is low. You are going to run out of iops. Sent from my Verizon Wireless 4G LTE Smartphone Original message From: Ahmad Emneina aemne...@gmail.com Date: 07/04/2013 9:10 AM (GMT-05:00) To: Cloudstack users mailing list users@cloudstack.apache.org Subject: Re: How many vms per primary storage can offer best performance? I would google NFS tuning and atomically test changes. Changes vary from the kernel level up through the switches (sizing frames) as well as introducing bonding. YMMV here NFS tuning is a huge part trial and error. On Thu, Jul 4, 2013 at 5:26 AM, WXR 1485739...@qq.com wrote: I use NFS share as primary storage,the NFS share is on a 8 SATA HDDs RAID10 volume. The network link is gigabit ethernet.The switch is dell powerconnect. When I just create 15-20 vm instances and start them(not run any software on them),I find the disk IO performance of the vm is very low. If a file copy job on a pc needs 10 minutes , the same job on the vm needs 20minutes. I don't know if it is normal,and I want to know the correct configuration of the primary storage,I need your suggests with enough experience.
RE: errors starting new instance with devcloud
The error you are getting is an HVM error (Hardware Virtualization). You do not have HVM in devcloud since it is a VM itself. You need a template without HVM. I though there was a work around already built into devcloud, but you can hack the db to mark the tempate without HVM by running this on the db: Update vm_templates set hvm=0. You might also shoot an email to the devlist to verify the expected behavior. Sent from my Verizon Wireless 4G LTE phone Original message From: Shane Witbeck sh...@digitalsanctum.com Date: 05/28/2013 7:20 PM (GMT-05:00) To: users@cloudstack.apache.org Subject: errors starting new instance with devcloud If have a fresh install of devcloud with management server running separately on a mac. I'm attempting to start a new instance and keep getting these errors: https://gist.github.com/digitalsanctum/5666858 I've tried on both master and 4.1 branches using a couple of different ISO's including the tiny Linux template that comes bundled. Any ideas? Thanks, Shane
RE: issue about windows server 2008 VM can NOT start
Have you check to make sure there are not any pending operations? Run xe task-list from the XenCenter console or via SSH of the host. What kind of primary storage are you using? I would also check the SMlog file(located /var/log/SMlog) on the host. When these issues occur, it's easier for me to take cloudstack out of the picture and try the start/stop on the host/XenCenter. After you resolved the issue, I would shut the machine back down and start it back up with CS. -Original Message- From: William Jiang [mailto:william.ji...@manwin.com] Sent: Thursday, May 16, 2013 3:50 PM To: Chip Childers; users@cloudstack.apache.org Subject: RE: issue about windows server 2008 VM can NOT start Thanks for reminding, I will provide error message in text. Hi, I need help for one of our windows server 2008 R2 instance which running on cloudstack 3.0.2+xenserver 6.0.2. As staring from yesterday, the status for this vm in cloudstack UI displayed as “stopping”, but we knew the VM is still up and running. Then the vm was down and status changed to “stopped” automatically. When I tried to start the VM, it failed and getting error as following in management-server log: 2013-05-16 12:04:12,105 DEBUG [xen.resource.XenServer56FP1Resource] (DirectAgent-383:null) Created VM 9d21618f-a75b-2a0f-e7bd-cd44e0554e90 for i-4-34-VM 2013-05-16 12:04:12,381 WARN [xen.resource.CitrixResourceBase] (DirectAgent-383:null) Unable to start i-4-34-VM due to 2013-05-16 12:04:12,463 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-383:null) The VM is in stopped state, detected problem during startup : i-4-34-VM And in cloudstack UI, I got the error as following: Unable to create a deployment for VM[user|i-4-34-VM] If you logged in xencenter, you will find the vm system disk “ROOT-34” doesn’t bind to any Virtual Machine. Any ideas or suggestions will be great appreciated. Thanks, William This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately. Ce courrier électronique est confidentiel et protégé. L'expéditeur ne renonce pas aux droits et obligations qui s'y rapportent. Toute diffusion, utilisation ou copie de ce message ou des renseignements qu'il contient par une personne autre que le (les) destinataire(s) désigné(s) est interdite. Si vous recevez ce courrier électronique par erreur, veuillez m'en aviser immédiatement, par retour de courrier électronique ou par un autre moyen.
RE: issue about windows server 2008 VM can NOT start
RE: 4. Sorry, I did not complete my thoughts. I assumed that you had a problem with the storage(I could be wrong). I take a blank template, attach the VDI and test. That is what I meant by taking CS out of the picture. -Original Message- From: William Jiang [mailto:william.ji...@manwin.com] Sent: Thursday, May 16, 2013 4:33 PM To: users@cloudstack.apache.org Subject: RE: issue about windows server 2008 VM can NOT start Hi Conrad, Thanks for your response. 1. most time I checked the pending operations by running xe task-list from xenserver console. I also run some command like xe vm-list, xe vm-disk-list, xe vdi-list to make sure the VHD configuration is ok. 2. we use iscsi device as our primary storage. 3. nothing found in /var/log/SMlog 4. what do you mean of it's easier for me to take cloudstack out of the picture ? can you give me more details about it? As I know, if the VM is stopped in cloudstack, you can't check the vm in xencenter as cloudstack removed corresponding info of that VM when you stop it. Thanks, William -Original Message- From: Conrad Geiger [mailto:cgei...@it1solutions.com] Sent: May-16-13 4:07 PM To: users@cloudstack.apache.org; Chip Childers Subject: RE: issue about windows server 2008 VM can NOT start Have you check to make sure there are not any pending operations? Run xe task-list from the XenCenter console or via SSH of the host. What kind of primary storage are you using? I would also check the SMlog file(located /var/log/SMlog) on the host. When these issues occur, it's easier for me to take cloudstack out of the picture and try the start/stop on the host/XenCenter. After you resolved the issue, I would shut the machine back down and start it back up with CS. -Original Message- From: William Jiang [mailto:william.ji...@manwin.com] Sent: Thursday, May 16, 2013 3:50 PM To: Chip Childers; users@cloudstack.apache.org Subject: RE: issue about windows server 2008 VM can NOT start Thanks for reminding, I will provide error message in text. Hi, I need help for one of our windows server 2008 R2 instance which running on cloudstack 3.0.2+xenserver 6.0.2. As staring from yesterday, the status for this vm in cloudstack UI displayed as “stopping”, but we knew the VM is still up and running. Then the vm was down and status changed to “stopped” automatically. When I tried to start the VM, it failed and getting error as following in management-server log: 2013-05-16 12:04:12,105 DEBUG [xen.resource.XenServer56FP1Resource] (DirectAgent-383:null) Created VM 9d21618f-a75b-2a0f-e7bd-cd44e0554e90 for i-4-34-VM 2013-05-16 12:04:12,381 WARN [xen.resource.CitrixResourceBase] (DirectAgent-383:null) Unable to start i-4-34-VM due to 2013-05-16 12:04:12,463 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-383:null) The VM is in stopped state, detected problem during startup : i-4-34-VM And in cloudstack UI, I got the error as following: Unable to create a deployment for VM[user|i-4-34-VM] If you logged in xencenter, you will find the vm system disk “ROOT-34” doesn’t bind to any Virtual Machine. Any ideas or suggestions will be great appreciated. Thanks, William This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately. Ce courrier électronique est confidentiel et protégé. L'expéditeur ne renonce pas aux droits et obligations qui s'y rapportent. Toute diffusion, utilisation ou copie de ce message ou des renseignements qu'il contient par une personne autre que le (les) destinataire(s) désigné(s) est interdite. Si vous recevez ce courrier électronique par erreur, veuillez m'en aviser immédiatement, par retour de courrier électronique ou par un autre moyen. This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately. Ce courrier électronique est confidentiel et protégé. L'expéditeur ne renonce pas aux droits et obligations qui s'y rapportent. Toute diffusion, utilisation ou copie de ce message ou des renseignements qu'il contient par une personne autre que le (les) destinataire(s) désigné(s) est interdite. Si vous recevez ce courrier électronique par erreur, veuillez m'en aviser immédiatement, par retour de courrier électronique ou par un autre moyen.
RE: CS 3.0.2 XenServer snapshots not cleaning up
That probably why the snapshots are not cleaning up. I wonder if CloudStack reports what 'should' be after the cleanup. When I've had these issues in the past, I've has to clear out a lot of space on the SR so It could clean up. You could also try manually running the leaf coalesce from the CLI. From my experience the XenServer 6.x does a better job. -Original Message- From: John Skinner [mailto:john.skin...@appcore.com] Sent: Thursday, May 09, 2013 11:50 AM To: users@cloudstack.apache.org Subject: Re: CS 3.0.2 XenServer snapshots not cleaning up SR is iSCSI. CloudStack GUI shows 283GB of 1TB allocated for that SR However, XenServer shows something different: [root@XH6-US1-PD1 ~]# xe sr-list uuid=79300477-8c0f-fc2b-b296-1f2007f7dc8d params=virtual-allocation,physical-utilisation,physical-size virtual-allocation ( RO) : 535314825216 physical-utilisation ( RO): 1078754017280 physical-size ( RO): 1099499044864 Physical utilization is actually higher than virtual allocation. From this it appears the SR is full, or really close. Something doesn't seem right here. -- The information in this message is intended for the named recipients only. It may contain information that is privileged, confidential or otherwise protected from disclosure. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or the taking of any action in reliance on the contents of this message is strictly prohibited. If you have received this e-mail in error, do not print it or disseminate it or its contents. In such event, please notify the sender by return e-mail and delete the e-mail file immediately thereafter. Thank you. On May 9, 2013, at 10:34 AM, Conrad Geiger cgei...@it1solutions.com wrote: That should be fine, I assume the SR has plenty of free space? What SR type? -Original Message- From: John Skinner [mailto:john.skin...@appcore.com] Sent: Thursday, May 09, 2013 11:16 AM To: users@cloudstack.apache.org Subject: Re: CS 3.0.2 XenServer snapshots not cleaning up 5.6 SP2 with hot fixes up to number 7 -- The information in this message is intended for the named recipients only. It may contain information that is privileged, confidential or otherwise protected from disclosure. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or the taking of any action in reliance on the contents of this message is strictly prohibited. If you have received this e-mail in error, do not print it or disseminate it or its contents. In such event, please notify the sender by return e-mail and delete the e-mail file immediately thereafter. Thank you. On May 9, 2013, at 10:07 AM, Conrad Geiger cgei...@it1solutions.com wrote: What version of XenServer and what hotfixes do you have installed? There are some know snapshot cleanup issues in certain versions. -Original Message- From: John Skinner [mailto:john.skin...@appcore.com] Sent: Thursday, May 09, 2013 11:03 AM To: users@cloudstack.apache.org Subject: CS 3.0.2 XenServer snapshots not cleaning up I have a volume that I discovered is no long taking snapshots. It appears that there are a lot of snapshots in the vdi tree for this vm that have not been cleaned up. The CS database says that there should only be 1 snapshot for the volume, but XenServer tells a much different story. How can I get rid of these snapshots without damaging the VDI tree? I apologize for the heavy text that follows. mysql select * from volumes where id = 4128; +--++---+-+-+---+---+--+-+--++++-+-+---+--+-++-+-+-+-+--++---+--+--+--+ | id | account_id | domain_id | pool_id | instance_id | device_id | name | size | folder | path | pod_id | data_center_id | iscsi_name | host_ip | volume_type | pool_type | disk_offering_id | template_id | first_snapshot_backup_uuid | recreatable | created | updated | removed | attached | chain_info | state | uuid | last_pool_id | update_count
RE: CS 3.0.2 XenServer snapshots not cleaning up
I meant to attach the CLI command: xe host-call-plugin host-uuid=host-UUID plugin=coalesce-leaf fn=leaf-coalesce args:vm_uuid=VM-UUID -Original Message- From: Conrad Geiger [mailto:cgei...@it1solutions.com] Sent: Thursday, May 09, 2013 12:42 PM To: users@cloudstack.apache.org Subject: RE: CS 3.0.2 XenServer snapshots not cleaning up That probably why the snapshots are not cleaning up. I wonder if CloudStack reports what 'should' be after the cleanup. When I've had these issues in the past, I've has to clear out a lot of space on the SR so It could clean up. You could also try manually running the leaf coalesce from the CLI. From my experience the XenServer 6.x does a better job. -Original Message- From: John Skinner [mailto:john.skin...@appcore.com] Sent: Thursday, May 09, 2013 11:50 AM To: users@cloudstack.apache.org Subject: Re: CS 3.0.2 XenServer snapshots not cleaning up SR is iSCSI. CloudStack GUI shows 283GB of 1TB allocated for that SR However, XenServer shows something different: [root@XH6-US1-PD1 ~]# xe sr-list uuid=79300477-8c0f-fc2b-b296-1f2007f7dc8d params=virtual-allocation,physical-utilisation,physical-size virtual-allocation ( RO) : 535314825216 physical-utilisation ( RO): 1078754017280 physical-size ( RO): 1099499044864 Physical utilization is actually higher than virtual allocation. From this it appears the SR is full, or really close. Something doesn't seem right here. -- The information in this message is intended for the named recipients only. It may contain information that is privileged, confidential or otherwise protected from disclosure. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or the taking of any action in reliance on the contents of this message is strictly prohibited. If you have received this e-mail in error, do not print it or disseminate it or its contents. In such event, please notify the sender by return e-mail and delete the e-mail file immediately thereafter. Thank you. On May 9, 2013, at 10:34 AM, Conrad Geiger cgei...@it1solutions.com wrote: That should be fine, I assume the SR has plenty of free space? What SR type? -Original Message- From: John Skinner [mailto:john.skin...@appcore.com] Sent: Thursday, May 09, 2013 11:16 AM To: users@cloudstack.apache.org Subject: Re: CS 3.0.2 XenServer snapshots not cleaning up 5.6 SP2 with hot fixes up to number 7 -- The information in this message is intended for the named recipients only. It may contain information that is privileged, confidential or otherwise protected from disclosure. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or the taking of any action in reliance on the contents of this message is strictly prohibited. If you have received this e-mail in error, do not print it or disseminate it or its contents. In such event, please notify the sender by return e-mail and delete the e-mail file immediately thereafter. Thank you. On May 9, 2013, at 10:07 AM, Conrad Geiger cgei...@it1solutions.com wrote: What version of XenServer and what hotfixes do you have installed? There are some know snapshot cleanup issues in certain versions. -Original Message- From: John Skinner [mailto:john.skin...@appcore.com] Sent: Thursday, May 09, 2013 11:03 AM To: users@cloudstack.apache.org Subject: CS 3.0.2 XenServer snapshots not cleaning up I have a volume that I discovered is no long taking snapshots. It appears that there are a lot of snapshots in the vdi tree for this vm that have not been cleaned up. The CS database says that there should only be 1 snapshot for the volume, but XenServer tells a much different story. How can I get rid of these snapshots without damaging the VDI tree? I apologize for the heavy text that follows. mysql select * from volumes where id = 4128; +--++---+-+-+---+---+--+-+--++++-+-+---+--+-++-+-+-+-+--++---+--+--+--+ | id | account_id | domain_id | pool_id | instance_id | device_id | name | size | folder | path | pod_id | data_center_id | iscsi_name | host_ip | volume_type | pool_type | disk_offering_id | template_id | first_snapshot_backup_uuid | recreatable | created | updated | removed
