Re: iso not downloading from local path

[Swastik Mittal]

Hey Dag,

On running ssvm-check I got :

root@s-1-VM:/usr/local/cloud/systemvm# ./ssvm-check.sh

First DNS server is  8.8.8.8
PING 8.8.8.8 (8.8.8.8): 48 data bytes
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
WARNING: cannot ping DNS server
route follows
Kernel IP routing table
Destination Gateway Genmask Flags Metric RefUse
Iface
0.0.0.0   10.1.0.20.0.0.0 UG0  00
eth2
8.8.8.8   10.1.0.2255.255.255.255 UGH   0  00
eth1
10.1.0.0  0.0.0.0 255.255.255.0   U 0  00
eth1
10.1.0.0  0.0.0.0 255.255.255.0   U 0  00
eth2
169.254.0.0 0.0.0.0 255.255.0.0 U 0  00 eth0
192.168.1.1 10.1.0.2255.255.255.255 UGH   0  00 eth1

Good: DNS resolves download.cloud.com

nfs is currently mounted
Mount point is /mnt/SecStorage/15b578c2-972f-3178-ac74-acc61e05ec25
Good: Can write to mount point

Management server is 10.1.0.77. Checking connectivity.
Good: Can connect to management server port 8250

Good: Java process is running

Tests Complete. Look for ERROR or WARNING above.
root@s-1-VM:/usr/local/cloud/systemvm#

It doesn't show any connection to secondary storage though. I has
connection to management server and my secondary storage currently is on
the same server.

On Thu, Feb 8, 2018 at 10:44 PM, Dag Sonstebo 
wrote:

> Swastik,
>
> Are you confident your SSVM can write to your secondary storage?
>
> Can you try to run /usr/local/cloud/systemvm/ssvm-check.sh and see what
> this comes back with?
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 08/02/2018, 11:06, "Swastik Mittal"  wrote:
>
> Hey Dag,
>
> As I start the downloading of iso I get a broken pipe error on my
> terminal
> logging http local file server details. But I checked the cloud.log
> and it
> showed processing and downloadStatus="NOT_DOWNLOADED".
> Downloadpath="/mnt/SecStorage" but NO ERROR untill after waiting for a
> while I myself turned off the http server. Till that time even the iso
> status din't show anything as no error string was generated and when I
> turned off the server it displayed partial http get request not
> supported.
> I don't think downloading from a local server would take time. I did
> allow
> it around 20 minutes. I think that broken pipe is connection to http
> server
> being suddenly stopped due to something. Not able to figure out how:).
>
> The download path shows /mnt/SecStorage which is in SSVM.
>
>
> Regards
> Swastik
>
> On Thu, Feb 8, 2018 at 2:59 PM, Dag Sonstebo <
> dag.sonst...@shapeblue.com>
> wrote:
>
> > Hi Swastik,
> >
> > Don’t worry about this – you were just doing a test where you were
> trying
> > to wget to the local SSVM filesystem as a test. The SSVM is just a
> small VM
> > with not a lot of disk space – so you are just checking connectivity
> here,
> > you should delete the partially downloaded file thereafter.
> >
> > When you do the template download from CloudStack the SSVM will write
> > straight to secondary storage, not to it’s own filesystem.
> >
> > Regards,
> > Dag Sonstebo
> > Cloud Architect
> > ShapeBlue
> >
> > On 08/02/2018, 08:41, "Swastik Mittal" 
> wrote:
> >
> > Hey
> >
> > A correction, I do see secondary storage and it shows only 1%
> utilized
> > so
> > my secondary storage does have enough memory.The issue seems to
> be
> > mostly
> > because of SSVM memory.
> >
> > Regards
> > Swastik
> >
> > On Thu, Feb 8, 2018 at 1:20 PM, Swastik Mittal <
> > mittal.swas...@gmail.com>
> > wrote:
> >
> > > Hey Dag and Glenn,
> > >
> > > On entering the local http server url, UI gives status that
> only
> > ports 80,
> > > 8080 and 443 are supported .
> > > So as my management server runs on 8080 and 443 would be for
> ssl so I
> > > hosted my iso on port 80.
> > > On logging into my ssvm and using wget to download it
> downloads upto
> > 35%
> > > and then stops it displaying not enough space on my terminal.
> So
> > definetly
> > > now there is no issue of credentials. And the broken pipe
> error on
> > > intiating download from local http from UI might be becuase I
> am
> > running
> > > short on space in my SSVM.
> > >
> > > But 

Re: Copy Volume Failed in CloudStack 4.5 (XenServer 6.5)

[Tutkowski, Mike]

If you go to the Global Settings tab in the GUI and search for “wait”, there 
are several possible timeouts that may apply.

The backup.snapshot.wait Global Setting seems like the one that probably 
applies here (per what Pierre-Luc was noting).

On 2/8/18, 4:15 PM, "Pierre-Luc Dion"  wrote:

I think there is a timout global settings you could change so the copy task
will take longer before it timeout and fail in cloudstack. This will not
improve your performance but might reduce failure.

On updating the database content, it could work, but only if the vhd
successfully copy, and mappings remain valid.

I hope this can help...



Le 6 févr. 2018 13 h 28, "anillakieni"  a
écrit :

Dear All,

Is somebody available here to assist me on fixing my issue.

Thanks,
Anil.

On Tue, Feb 6, 2018 at 9:00 PM, anillakieni  wrote:

> Hi All,
>
> I'm facing issue when copying  larger size volumes. i.e., Secondary
> Storage to Primary Storage (I mean attaching DATA volume to VM), after
> certain time around 37670 seconds.
>
> Version of:
> - CloudStack is 4.5.0
> - XenServer 6.5.0
> - MySQL 5.1.73
>
>
> The error and log is provided below, Could someone please assist me here
> which steps i have to take to fix this issue. Also, can we have a chance
to
> update the failed status to success through database tables because i have
> to upload the whole disk again to secondary storage and then later attach
> it to VM, which is consuming more time. My environment has very slow
> network transfers (I have only 1 Gig switch). Please let me know if we can
> tweak the DB to update the status of the disk or do we have any settings
to
> be changed to accept more time (wait time) for updating the status.
> "
>
> 2018-02-06 03:20:42,385 DEBUG [c.c.a.t.Request] (Work-Job-Executor-31:ctx-
c1c78a5a
> job-106186/job-106187 ctx-ea1ef3e6) (logid:c59b2359) Seq
> 38-367887794560851961: Received:  { Ans: , MgmtId: 47019105324719, via:
38,
> Ver: v1, Flags: 110, { CopyCmdAnswer } }
> 2018-02-06 03:20:42,389 DEBUG [o.a.c.s.v.VolumeObject]
> (Work-Job-Executor-31:ctx-c1c78a5a job-106186/job-106187 ctx-ea1ef3e6)
> (logid:c59b2359) *Failed to update state*
> *com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.JDBC4PreparedStatement@54bd3a25: SELECT volume_store_ref.id
> , volume_store_ref.store_id,
> volume_store_ref.volume_id, volume_store_ref.zone_id,
> volume_store_ref.created, volume_store_ref.last_updated,
> volume_store_ref.download_pct, volume_store_ref.size,
> volume_store_ref.physical_size, volume_store_ref.download_state,
> volume_store_ref.checksum, volume_store_ref.local_path,
> volume_store_ref.error_str, volume_store_ref.job_id,
> volume_store_ref.install_path, volume_store_ref.url,
> volume_store_ref.download_url, volume_store_ref.download_url_created,
> volume_store_ref.destroyed, volume_store_ref.update_count,
> volume_store_ref.updated, volume_store_ref.state, volume_store_ref.ref_cnt
> FROM volume_store_ref WHERE volume_store_ref.store_id = 1  AND
> volume_store_ref.volume_id = 1178  AND volume_store_ref.destroyed = 0
> ORDER BY RAND() LIMIT 1*
> at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(
> GenericDaoBase.java:425)
> at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(
> GenericDaoBase.java:361)
> at com.cloud.utils.db.GenericDaoBase.findOneIncludingRemovedBy(
> GenericDaoBase.java:889)
> at com.cloud.utils.db.GenericDaoBase.findOneBy(
> GenericDaoBase.java:900)
> at org.apache.cloudstack.storage.image.db.VolumeDataStoreDaoImpl.
> findByStoreVolume(VolumeDataStoreDaoImpl.java:209)
> at sun.reflect.GeneratedMethodAccessor306.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.springframework.aop.support.AopUtils.
> invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.
> invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.
> proceed(ReflectiveMethodInvocation.java:150)
> at com.cloud.utils.db.TransactionContextInterceptor.invoke(
> TransactionContextInterceptor.java:34)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.
> proceed(ReflectiveMethodInvocation.java:161)
> at 

Re: Copy Volume Failed in CloudStack 4.5 (XenServer 6.5)

[Pierre-Luc Dion]

I think there is a timout global settings you could change so the copy task
will take longer before it timeout and fail in cloudstack. This will not
improve your performance but might reduce failure.

On updating the database content, it could work, but only if the vhd
successfully copy, and mappings remain valid.

I hope this can help...



Le 6 févr. 2018 13 h 28, "anillakieni"  a
écrit :

Dear All,

Is somebody available here to assist me on fixing my issue.

Thanks,
Anil.

On Tue, Feb 6, 2018 at 9:00 PM, anillakieni  wrote:

> Hi All,
>
> I'm facing issue when copying  larger size volumes. i.e., Secondary
> Storage to Primary Storage (I mean attaching DATA volume to VM), after
> certain time around 37670 seconds.
>
> Version of:
> - CloudStack is 4.5.0
> - XenServer 6.5.0
> - MySQL 5.1.73
>
>
> The error and log is provided below, Could someone please assist me here
> which steps i have to take to fix this issue. Also, can we have a chance
to
> update the failed status to success through database tables because i have
> to upload the whole disk again to secondary storage and then later attach
> it to VM, which is consuming more time. My environment has very slow
> network transfers (I have only 1 Gig switch). Please let me know if we can
> tweak the DB to update the status of the disk or do we have any settings
to
> be changed to accept more time (wait time) for updating the status.
> "
>
> 2018-02-06 03:20:42,385 DEBUG [c.c.a.t.Request] (Work-Job-Executor-31:ctx-
c1c78a5a
> job-106186/job-106187 ctx-ea1ef3e6) (logid:c59b2359) Seq
> 38-367887794560851961: Received:  { Ans: , MgmtId: 47019105324719, via:
38,
> Ver: v1, Flags: 110, { CopyCmdAnswer } }
> 2018-02-06 03:20:42,389 DEBUG [o.a.c.s.v.VolumeObject]
> (Work-Job-Executor-31:ctx-c1c78a5a job-106186/job-106187 ctx-ea1ef3e6)
> (logid:c59b2359) *Failed to update state*
> *com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.JDBC4PreparedStatement@54bd3a25: SELECT volume_store_ref.id
> , volume_store_ref.store_id,
> volume_store_ref.volume_id, volume_store_ref.zone_id,
> volume_store_ref.created, volume_store_ref.last_updated,
> volume_store_ref.download_pct, volume_store_ref.size,
> volume_store_ref.physical_size, volume_store_ref.download_state,
> volume_store_ref.checksum, volume_store_ref.local_path,
> volume_store_ref.error_str, volume_store_ref.job_id,
> volume_store_ref.install_path, volume_store_ref.url,
> volume_store_ref.download_url, volume_store_ref.download_url_created,
> volume_store_ref.destroyed, volume_store_ref.update_count,
> volume_store_ref.updated, volume_store_ref.state, volume_store_ref.ref_cnt
> FROM volume_store_ref WHERE volume_store_ref.store_id = 1  AND
> volume_store_ref.volume_id = 1178  AND volume_store_ref.destroyed = 0
> ORDER BY RAND() LIMIT 1*
> at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(
> GenericDaoBase.java:425)
> at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(
> GenericDaoBase.java:361)
> at com.cloud.utils.db.GenericDaoBase.findOneIncludingRemovedBy(
> GenericDaoBase.java:889)
> at com.cloud.utils.db.GenericDaoBase.findOneBy(
> GenericDaoBase.java:900)
> at org.apache.cloudstack.storage.image.db.VolumeDataStoreDaoImpl.
> findByStoreVolume(VolumeDataStoreDaoImpl.java:209)
> at sun.reflect.GeneratedMethodAccessor306.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.springframework.aop.support.AopUtils.
> invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.
> invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.
> proceed(ReflectiveMethodInvocation.java:150)
> at com.cloud.utils.db.TransactionContextInterceptor.invoke(
> TransactionContextInterceptor.java:34)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.
> proceed(ReflectiveMethodInvocation.java:161)
> at org.springframework.aop.interceptor.
> ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.
> proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.
> invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy173.findByStoreVolume(Unknown Source)
> at org.apache.cloudstack.storage.datastore.
> ObjectInDataStoreManagerImpl.findObject(ObjectInDataStoreManagerImpl.
> java:353)
> at org.apache.cloudstack.storage.datastore.
> ObjectInDataStoreManagerImpl.findObject(ObjectInDataStoreManagerImpl.
> java:338)
> at 

RE: Customer Backup with CS 4.7 and VMWare

[Simon Weller]

Sounds exciting!

We've also been experimenting with Ceph volume/image mirroring (rbd mirror) 
lately, with the goal of potentially using it for multi-region DR and 
inter-region VM migration.

In an ideal world, we'd like migrate Ceph to a managed storage driver before we 
embark on adding mirroring support, but obviously implementation time frames 
will depend on commercial pressure.

Backup via a third party tool could potentially ease the prem to cloud 
migration challenges, so any features that can be leveraged there are only good 
thing.

- Si

Simon Weller/615-312-6068

-Original Message-
From: Paul Angus [paul.an...@shapeblue.com]
Received: Thursday, 08 Feb 2018, 12:49PM
To: users@cloudstack.apache.org [users@cloudstack.apache.org]
Subject: RE: Customer Backup with CS 4.7 and VMWare

Hi All, particularly Daniel, Sebastian and Simon,

We, as a community are very much aware of the shortfalls in 'backup and 
recovery' options available natively in CloudStack - which is basically 'do 
volume snapshots'.

Quite a while back, I published a high-level proposal of what I thought that a 
backup and recovery
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=70258527

I've continued to work on it, and am current speaking to a couple of backup 
vendors in order to have a framework concept that they will work with.
Hopefully within a few weeks I'll be able to publish a substantive proposal for 
everyone to see and comment on.

Stay tuned



paul.an...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




-Original Message-
From: daniel.herrm...@zv.fraunhofer.de [mailto:daniel.herrm...@zv.fraunhofer.de]
Sent: 07 February 2018 14:35
To: users@cloudstack.apache.org
Subject: Customer Backup with CS 4.7 and VMWare

Hi All,

We are using CS 4.7.1 with VMWare Hypervisor and advanced networking in a 
private cloud environment. Currently, most of our (internal) customers hosting 
internal services within this environment are using volume snapshots to 
facilitate backups of their virtual machines. Besides the obvious downsides of 
this approach (consistent snapshots of multiple volumes, …) we encounter 
serious problems using this features. In ~10% of the cases, snapshots get stuck 
in the BackingUp state, which sometimes causes the whole snapshot queue to 
stale. In some other cases, recurring snapshots are correctly configured, but 
CS does not try even try to create this snapshot, there is no entry in the 
database.

In summary, we are currently evaluating different options, hence my questions 
here:


  *   Are we the only ones encountering that massive problems with volume 
snapshots? Or is this a known problem? Anything we could look at or a hint 
where we could start troubleshooting?
  *   How are you actually providing backup services to the customer? Are there 
other solutions or products that integrate with CS?

When using another option than the volume snapshots in CS, the most important 
factor would be to keep the ability for the customer to configure everything in 
self-service.

Thanks and regards
Daniel

RE: Customer Backup with CS 4.7 and VMWare

[Paul Angus]

Hi All, particularly Daniel, Sebastian and Simon,

We, as a community are very much aware of the shortfalls in 'backup and 
recovery' options available natively in CloudStack - which is basically 'do 
volume snapshots'.

Quite a while back, I published a high-level proposal of what I thought that a 
backup and recovery 
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=70258527

I've continued to work on it, and am current speaking to a couple of backup 
vendors in order to have a framework concept that they will work with.   
Hopefully within a few weeks I'll be able to publish a substantive proposal for 
everyone to see and comment on.

Stay tuned



paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: daniel.herrm...@zv.fraunhofer.de 
[mailto:daniel.herrm...@zv.fraunhofer.de] 
Sent: 07 February 2018 14:35
To: users@cloudstack.apache.org
Subject: Customer Backup with CS 4.7 and VMWare

Hi All,

We are using CS 4.7.1 with VMWare Hypervisor and advanced networking in a 
private cloud environment. Currently, most of our (internal) customers hosting 
internal services within this environment are using volume snapshots to 
facilitate backups of their virtual machines. Besides the obvious downsides of 
this approach (consistent snapshots of multiple volumes, …) we encounter 
serious problems using this features. In ~10% of the cases, snapshots get stuck 
in the BackingUp state, which sometimes causes the whole snapshot queue to 
stale. In some other cases, recurring snapshots are correctly configured, but 
CS does not try even try to create this snapshot, there is no entry in the 
database.

In summary, we are currently evaluating different options, hence my questions 
here:


  *   Are we the only ones encountering that massive problems with volume 
snapshots? Or is this a known problem? Anything we could look at or a hint 
where we could start troubleshooting?
  *   How are you actually providing backup services to the customer? Are there 
other solutions or products that integrate with CS?

When using another option than the volume snapshots in CS, the most important 
factor would be to keep the ability for the customer to configure everything in 
self-service.

Thanks and regards
Daniel

Re: iso not downloading from local path

[Dag Sonstebo]

Swastik,

Are you confident your SSVM can write to your secondary storage?

Can you try to run /usr/local/cloud/systemvm/ssvm-check.sh and see what this 
comes back with?

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 08/02/2018, 11:06, "Swastik Mittal"  wrote:

Hey Dag,

As I start the downloading of iso I get a broken pipe error on my terminal
logging http local file server details. But I checked the cloud.log and it
showed processing and downloadStatus="NOT_DOWNLOADED".
Downloadpath="/mnt/SecStorage" but NO ERROR untill after waiting for a
while I myself turned off the http server. Till that time even the iso
status din't show anything as no error string was generated and when I
turned off the server it displayed partial http get request not supported.
I don't think downloading from a local server would take time. I did allow
it around 20 minutes. I think that broken pipe is connection to http server
being suddenly stopped due to something. Not able to figure out how:).

The download path shows /mnt/SecStorage which is in SSVM.


Regards
Swastik

On Thu, Feb 8, 2018 at 2:59 PM, Dag Sonstebo 
wrote:

> Hi Swastik,
>
> Don’t worry about this – you were just doing a test where you were trying
> to wget to the local SSVM filesystem as a test. The SSVM is just a small 
VM
> with not a lot of disk space – so you are just checking connectivity here,
> you should delete the partially downloaded file thereafter.
>
> When you do the template download from CloudStack the SSVM will write
> straight to secondary storage, not to it’s own filesystem.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 08/02/2018, 08:41, "Swastik Mittal"  wrote:
>
> Hey
>
> A correction, I do see secondary storage and it shows only 1% utilized
> so
> my secondary storage does have enough memory.The issue seems to be
> mostly
> because of SSVM memory.
>
> Regards
> Swastik
>
> On Thu, Feb 8, 2018 at 1:20 PM, Swastik Mittal <
> mittal.swas...@gmail.com>
> wrote:
>
> > Hey Dag and Glenn,
> >
> > On entering the local http server url, UI gives status that only
> ports 80,
> > 8080 and 443 are supported .
> > So as my management server runs on 8080 and 443 would be for ssl so 
I
> > hosted my iso on port 80.
> > On logging into my ssvm and using wget to download it downloads upto
> 35%
> > and then stops it displaying not enough space on my terminal. So
> definetly
> > now there is no issue of credentials. And the broken pipe error on
> > intiating download from local http from UI might be becuase I am
> running
> > short on space in my SSVM.
> >
> > But isn't the iso stored in secondary storage which is the 
management
> > server itself? My secondary storage is up as shown in infrastructure
> and
> > even my system vm template was downloaded in /mnt/secondary. But my
> > dashboard does not show secondary storage infact shows primary
> storage
> > having 1 TB of memory which is the complete memory of the system. Do
> I need
> > to increase the SSVM storage and if yes how do I do that?
> >
> > Regards
> > Swastik
> >
> > On Wed, Feb 7, 2018 at 8:43 PM, Swastik Mittal <
> mittal.swas...@gmail.com>
> > wrote:
> >
> >> Hey,
> >>
> >> Thanks Dag and Glen for reply.
> >>
> >> I'll try changing the port and check.
> >>
> >> Regards
> >> Swastik
> >>
> >>
> >>  Sent with Mailtrack
> >>  gmail-inbox/ndnaehgpjlnokgebbaldlmgkapkpjkkb?utm_source=gmail_
> medium=signature_campaign=signaturevirality>
> >>
> >> On Wed, Feb 7, 2018 at 5:57 PM, Glenn Wagner <
> glenn.wag...@shapeblue.com>
> >> wrote:
> >>
> >>> Hi
> >>>
> >>> As Dag pointed out (the errors) when using the simpleHTTP server
> its
> >>> better to use either port 8080 or 8000 so you don't have to setup
> SSL on
> >>> port 443
> >>>
> >>>  Example
> >>> python -m SimpleHTTPServer 8080
> >>>
> >>> Now try the wget or Curl from the SSVM to download the ISO use
> http not
> >>> https
> >>>
> >>> Regards
> >>> Glenn
> >>>
> >>>
> >>> glenn.wag...@shapeblue.com
> >>> www.shapeblue.com
> >>> Winter Suite, 1st Floor, The Avenues, Drama Street, Somerset 

Request for Participation

[Steve Roles]

Hi all - we are holding a CloudStack / Ceph day in London on Thursday, April 19.

It would be great to see some of you there, and we are also looking for 
speakers. If you have an interesting CloudStack talk and would like to come 
along to share, please let me know ASAP. Thanks!

https://www.eventbrite.co.uk/e/cloudstack-european-user-group-ceph-day-tickets-42670526694?ref=estw

Best regards,


steve.ro...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Re: iso not downloading from local path

[Swastik Mittal]

Hey Dag,

As I start the downloading of iso I get a broken pipe error on my terminal
logging http local file server details. But I checked the cloud.log and it
showed processing and downloadStatus="NOT_DOWNLOADED".
Downloadpath="/mnt/SecStorage" but NO ERROR untill after waiting for a
while I myself turned off the http server. Till that time even the iso
status din't show anything as no error string was generated and when I
turned off the server it displayed partial http get request not supported.
I don't think downloading from a local server would take time. I did allow
it around 20 minutes. I think that broken pipe is connection to http server
being suddenly stopped due to something. Not able to figure out how:).

The download path shows /mnt/SecStorage which is in SSVM.


Regards
Swastik

On Thu, Feb 8, 2018 at 2:59 PM, Dag Sonstebo 
wrote:

> Hi Swastik,
>
> Don’t worry about this – you were just doing a test where you were trying
> to wget to the local SSVM filesystem as a test. The SSVM is just a small VM
> with not a lot of disk space – so you are just checking connectivity here,
> you should delete the partially downloaded file thereafter.
>
> When you do the template download from CloudStack the SSVM will write
> straight to secondary storage, not to it’s own filesystem.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 08/02/2018, 08:41, "Swastik Mittal"  wrote:
>
> Hey
>
> A correction, I do see secondary storage and it shows only 1% utilized
> so
> my secondary storage does have enough memory.The issue seems to be
> mostly
> because of SSVM memory.
>
> Regards
> Swastik
>
> On Thu, Feb 8, 2018 at 1:20 PM, Swastik Mittal <
> mittal.swas...@gmail.com>
> wrote:
>
> > Hey Dag and Glenn,
> >
> > On entering the local http server url, UI gives status that only
> ports 80,
> > 8080 and 443 are supported .
> > So as my management server runs on 8080 and 443 would be for ssl so I
> > hosted my iso on port 80.
> > On logging into my ssvm and using wget to download it downloads upto
> 35%
> > and then stops it displaying not enough space on my terminal. So
> definetly
> > now there is no issue of credentials. And the broken pipe error on
> > intiating download from local http from UI might be becuase I am
> running
> > short on space in my SSVM.
> >
> > But isn't the iso stored in secondary storage which is the management
> > server itself? My secondary storage is up as shown in infrastructure
> and
> > even my system vm template was downloaded in /mnt/secondary. But my
> > dashboard does not show secondary storage infact shows primary
> storage
> > having 1 TB of memory which is the complete memory of the system. Do
> I need
> > to increase the SSVM storage and if yes how do I do that?
> >
> > Regards
> > Swastik
> >
> > On Wed, Feb 7, 2018 at 8:43 PM, Swastik Mittal <
> mittal.swas...@gmail.com>
> > wrote:
> >
> >> Hey,
> >>
> >> Thanks Dag and Glen for reply.
> >>
> >> I'll try changing the port and check.
> >>
> >> Regards
> >> Swastik
> >>
> >>
> >>  Sent with Mailtrack
> >>  gmail-inbox/ndnaehgpjlnokgebbaldlmgkapkpjkkb?utm_source=gmail_
> medium=signature_campaign=signaturevirality>
> >>
> >> On Wed, Feb 7, 2018 at 5:57 PM, Glenn Wagner <
> glenn.wag...@shapeblue.com>
> >> wrote:
> >>
> >>> Hi
> >>>
> >>> As Dag pointed out (the errors) when using the simpleHTTP server
> its
> >>> better to use either port 8080 or 8000 so you don't have to setup
> SSL on
> >>> port 443
> >>>
> >>>  Example
> >>> python -m SimpleHTTPServer 8080
> >>>
> >>> Now try the wget or Curl from the SSVM to download the ISO use
> http not
> >>> https
> >>>
> >>> Regards
> >>> Glenn
> >>>
> >>>
> >>> glenn.wag...@shapeblue.com
> >>> www.shapeblue.com
> >>> Winter Suite, 1st Floor, The Avenues, Drama Street, Somerset West,
> Cape
> >>> Town  7129South Africa
> >>> @shapeblue
> >>>
> >>>
> >>>
> >>>
> >>> -Original Message-
> >>> From: Dag Sonstebo [mailto:dag.sonst...@shapeblue.com]
> >>> Sent: Wednesday, 07 February 2018 2:13 PM
> >>> To: users@cloudstack.apache.org
> >>> Subject: Re: iso not downloading from local path
> >>>
> >>> Hi Swastik,
> >>>
> >>> Moving discussion to this thread.
> >>>
> >>> The simple test here is to log in to the SSVM from console or over
> ssh
> >>> and do a “wget” or “curl” against your download URL and see if
> this starts.
> >>>
> >>> I think your issue is with either your simpleHTTP server asking for
> >>> credentials, or your proxy configuration asking for credentials –
> “No
> >>> 

Re: Customer Backup with CS 4.7 and VMWare

[simon.voelker]

Hi,

colleague of Daniel’s here, who has been dealing a lot with volumesnapshot 
problems.

We are currently seeing two problems with the volumesnapshots.
The first is the long running snapshots, they go into the „BackingUp“ state and 
never finish and thus block all further snapshots. These can be fixed 
reasonably easy by hacking the database and setting the stuck snapshot to 
„Error“ Usually the next snapshot resumes normally. This can also happen due to 
restarts of the management service while OVF exports are running. In this case 
the ACPdoctor’s cleanup usually fixes the stale snapshots.

A second issue we are seeing since late last year is snapshots failing because 
they can not locate the volume they are supposed to snapshot. This seems to be 
due to inconsistencies between the database and vmware, specifically in the 
„Path“ and „ChainInfo“ variables. The accelerite support I am working with 
currently is supecting some race condition to cause this for VMs with multiple 
volumes.

The current suspicion is:
root volume snapshot triggers, this causes the creation of vmnamehere-0003.vmdk
data volume snapshot triggers, this causes the creation of vmnamehere-0004.vmdk
Once the snapshot is created, CP will parse the snapshot and then it is copied 
to secondary storage, during this process CP was unable to find the 0004.vmdk 
as part of the snapshot
When parsing the snapshot for the data disk, it does find the 0004.vmdk on 
datastore but not when parsing the snapshot info from vmsd as you can see above 
( notice the snapshot.numSnapshots = "4" in both ( when parsing ROOT and DATA 
disks) the cases)
Later, once the snapshot copying is done. CP will delete the snapshot of the VM 
and then consolidate the disks.
For the root disk cp tries to delete 0003.vmdk after copying, but this fails. 
It marks 0004 as top vmdk though.
CP keeps updating 0004.vmdk, but this vmdk is removed once the Vm runs a 
consolidation.

I’m currently looking to verify this is indeed the root cause of the described 
error.

Regards


Simon Völker
Fraunhofer-Gesellschaft e.V.
Abteilung C7 Kommunikationsmanagement
Schloss Birlinghoven IZB, 53754 Sankt Augustin
Telefon: (02241) 14-2311
E-mail: simon.voel...@zv.fraunhofer.de



Am 08.02.2018 um 09:25 schrieb 
daniel.herrm...@zv.fraunhofer.de:

Hi Sebastián,

Thank you for your answer. This is exactly the same problem we are facing. Some 
customers have >1TB volumes, and it just takes ages to complete them. Which by 
the way would not be the actual problem, but sometimes CS does not even create 
the snapshot from the recurring snapshot policy or, even worse, a snapshot is 
created but never (>7d) finishes and remains in the BackingUp state, causing 
new snapshots of this series not to be created.

I read about a solution with Veeam and Tags (e.g. let the customer tag the 
virtual machines, and Veeam automatically backups the tagged machines), but 
this adds problems such as:

- how to bill the usage of this method?
- We could restore the virtual machine to an earlier state. But if the customer 
accidently deleted the machine, we cannot create it back from the backup as CS 
would not recognize it again.

So... if anyone has further insight, we'd be happy to hear about it. (

Regards
Daniel

On 08.02.18, 08:21, "Sebastian Gomez" 
> wrote:

   Hello Daniel,

   We have the same environment, and the same problem.
   I agree, the volume snapshots are a pain in time needings. Volume snapshot
   does a full copy of the volume through the network from primary storage to
   the secondary. May be there is a storage configuration that could optimize
   this action, but we have iSCSI for primary storage and NFS por secondary...
   For some big volumes it takes up to 8 h to complete the snap.

   This is NOT a sustainable solution.

   Our customers uses backup agents of their backup solutions, and we (as
   providers) have a backup at VMware level (you can find many solutions like
   vRanger, veeam, ...), is the only way that we have found to have a
   disaster-recovery backup of the platform. We are working now on how to
   offer to customers using this backup as a service, facing to have a
   global-unique backup solution (and scalable) for all the platform and users.

   In this way, for example Veeam backup offers many options to allow users to
   recover their own data (configuring access via API), but the problem here
   is that in Cloudstack you can't recover a virtual machine on the
   virtualization layer without informing the cloud-manager...


   Perhaps someone else could light us.




   Regards.





   Atentamente,
   Sebastián Gómez

   On Wed, Feb 7, 2018 at 3:35 PM, 
> 
wrote:

Hi All,

We are using CS 4.7.1 with VMWare Hypervisor and advanced networking in a
private cloud environment. 

Re: iso not downloading from local path

[Dag Sonstebo]

Hi Swastik,

Don’t worry about this – you were just doing a test where you were trying to 
wget to the local SSVM filesystem as a test. The SSVM is just a small VM with 
not a lot of disk space – so you are just checking connectivity here, you 
should delete the partially downloaded file thereafter.

When you do the template download from CloudStack the SSVM will write straight 
to secondary storage, not to it’s own filesystem.

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 08/02/2018, 08:41, "Swastik Mittal"  wrote:

Hey

A correction, I do see secondary storage and it shows only 1% utilized so
my secondary storage does have enough memory.The issue seems to be mostly
because of SSVM memory.

Regards
Swastik

On Thu, Feb 8, 2018 at 1:20 PM, Swastik Mittal 
wrote:

> Hey Dag and Glenn,
>
> On entering the local http server url, UI gives status that only ports 80,
> 8080 and 443 are supported .
> So as my management server runs on 8080 and 443 would be for ssl so I
> hosted my iso on port 80.
> On logging into my ssvm and using wget to download it downloads upto 35%
> and then stops it displaying not enough space on my terminal. So definetly
> now there is no issue of credentials. And the broken pipe error on
> intiating download from local http from UI might be becuase I am running
> short on space in my SSVM.
>
> But isn't the iso stored in secondary storage which is the management
> server itself? My secondary storage is up as shown in infrastructure and
> even my system vm template was downloaded in /mnt/secondary. But my
> dashboard does not show secondary storage infact shows primary storage
> having 1 TB of memory which is the complete memory of the system. Do I 
need
> to increase the SSVM storage and if yes how do I do that?
>
> Regards
> Swastik
>
> On Wed, Feb 7, 2018 at 8:43 PM, Swastik Mittal 
> wrote:
>
>> Hey,
>>
>> Thanks Dag and Glen for reply.
>>
>> I'll try changing the port and check.
>>
>> Regards
>> Swastik
>>
>>
>>  Sent with Mailtrack
>> 

>>
>> On Wed, Feb 7, 2018 at 5:57 PM, Glenn Wagner 
>> wrote:
>>
>>> Hi
>>>
>>> As Dag pointed out (the errors) when using the simpleHTTP server its
>>> better to use either port 8080 or 8000 so you don't have to setup SSL on
>>> port 443
>>>
>>>  Example
>>> python -m SimpleHTTPServer 8080
>>>
>>> Now try the wget or Curl from the SSVM to download the ISO use http not
>>> https
>>>
>>> Regards
>>> Glenn
>>>
>>>
>>> glenn.wag...@shapeblue.com
>>> www.shapeblue.com
>>> Winter Suite, 1st Floor, The Avenues, Drama Street, Somerset West, Cape
>>> Town  7129South Africa
>>> @shapeblue
>>>
>>>
>>>
>>>
>>> -Original Message-
>>> From: Dag Sonstebo [mailto:dag.sonst...@shapeblue.com]
>>> Sent: Wednesday, 07 February 2018 2:13 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: iso not downloading from local path
>>>
>>> Hi Swastik,
>>>
>>> Moving discussion to this thread.
>>>
>>> The simple test here is to log in to the SSVM from console or over ssh
>>> and do a “wget” or “curl” against your download URL and see if this 
starts.
>>>
>>> I think your issue is with either your simpleHTTP server asking for
>>> credentials, or your proxy configuration asking for credentials – “No
>>> credentials configured for host ” seems fairly
>>> conclusive.
>>>
>>>
>>> Regards,
>>> Dag Sonstebo
>>> Cloud Architect
>>> ShapeBlue
>>>
>>> On 07/02/2018, 06:36, "Swastik Mittal"  wrote:
>>>
>>> I checked my cloud.log in my SSVM. After the URL request it mentions
>>>
>>> [storage.template.HttpTemplateDownloader]
>>> (agentRequest-Handler-3:null) No
>>> credentials configured for host 
>>>
>>> On Wed, Feb 7, 2018 at 11:32 AM, Swastik Mittal <
>>> mittal.swas...@gmail.com>
>>> wrote:
>>>
>>> > Hey,
>>> >
>>> > I do not have internet on management server and host so to upload
>>> an iso I
>>> > set secstorage.allowed.internal.sites to my CIDR. I used
>>> >
>>> > $ python -m SimpleHTTPServer 443
>>> >
>>> > to host my directory on http server where I had kept my downloaded
>>> iso. By
>>> > manually visiting the local http server I am able to download the
>>> file. But
>>> > on mentioning the 

Re: iso not downloading from local path

[Swastik Mittal]

Hey

A correction, I do see secondary storage and it shows only 1% utilized so
my secondary storage does have enough memory.The issue seems to be mostly
because of SSVM memory.

Regards
Swastik

On Thu, Feb 8, 2018 at 1:20 PM, Swastik Mittal 
wrote:

> Hey Dag and Glenn,
>
> On entering the local http server url, UI gives status that only ports 80,
> 8080 and 443 are supported .
> So as my management server runs on 8080 and 443 would be for ssl so I
> hosted my iso on port 80.
> On logging into my ssvm and using wget to download it downloads upto 35%
> and then stops it displaying not enough space on my terminal. So definetly
> now there is no issue of credentials. And the broken pipe error on
> intiating download from local http from UI might be becuase I am running
> short on space in my SSVM.
>
> But isn't the iso stored in secondary storage which is the management
> server itself? My secondary storage is up as shown in infrastructure and
> even my system vm template was downloaded in /mnt/secondary. But my
> dashboard does not show secondary storage infact shows primary storage
> having 1 TB of memory which is the complete memory of the system. Do I need
> to increase the SSVM storage and if yes how do I do that?
>
> Regards
> Swastik
>
> On Wed, Feb 7, 2018 at 8:43 PM, Swastik Mittal 
> wrote:
>
>> Hey,
>>
>> Thanks Dag and Glen for reply.
>>
>> I'll try changing the port and check.
>>
>> Regards
>> Swastik
>>
>>
>>  Sent with Mailtrack
>> 
>>
>> On Wed, Feb 7, 2018 at 5:57 PM, Glenn Wagner 
>> wrote:
>>
>>> Hi
>>>
>>> As Dag pointed out (the errors) when using the simpleHTTP server its
>>> better to use either port 8080 or 8000 so you don't have to setup SSL on
>>> port 443
>>>
>>>  Example
>>> python -m SimpleHTTPServer 8080
>>>
>>> Now try the wget or Curl from the SSVM to download the ISO use http not
>>> https
>>>
>>> Regards
>>> Glenn
>>>
>>>
>>> glenn.wag...@shapeblue.com
>>> www.shapeblue.com
>>> Winter Suite, 1st Floor, The Avenues, Drama Street, Somerset West, Cape
>>> Town  7129South Africa
>>> @shapeblue
>>>
>>>
>>>
>>>
>>> -Original Message-
>>> From: Dag Sonstebo [mailto:dag.sonst...@shapeblue.com]
>>> Sent: Wednesday, 07 February 2018 2:13 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: iso not downloading from local path
>>>
>>> Hi Swastik,
>>>
>>> Moving discussion to this thread.
>>>
>>> The simple test here is to log in to the SSVM from console or over ssh
>>> and do a “wget” or “curl” against your download URL and see if this starts.
>>>
>>> I think your issue is with either your simpleHTTP server asking for
>>> credentials, or your proxy configuration asking for credentials – “No
>>> credentials configured for host ” seems fairly
>>> conclusive.
>>>
>>>
>>> Regards,
>>> Dag Sonstebo
>>> Cloud Architect
>>> ShapeBlue
>>>
>>> On 07/02/2018, 06:36, "Swastik Mittal"  wrote:
>>>
>>> I checked my cloud.log in my SSVM. After the URL request it mentions
>>>
>>> [storage.template.HttpTemplateDownloader]
>>> (agentRequest-Handler-3:null) No
>>> credentials configured for host 
>>>
>>> On Wed, Feb 7, 2018 at 11:32 AM, Swastik Mittal <
>>> mittal.swas...@gmail.com>
>>> wrote:
>>>
>>> > Hey,
>>> >
>>> > I do not have internet on management server and host so to upload
>>> an iso I
>>> > set secstorage.allowed.internal.sites to my CIDR. I used
>>> >
>>> > $ python -m SimpleHTTPServer 443
>>> >
>>> > to host my directory on http server where I had kept my downloaded
>>> iso. By
>>> > manually visiting the local http server I am able to download the
>>> file. But
>>> > on mentioning the same url in registering the iso it shows
>>> registered
>>> > successfully but does not get downloaded.
>>> >
>>> > In the zone section in the iso it mentions not ready. I receive a
>>> broken
>>> > pipe error at the initial stage in my local file server log but
>>> then it
>>> > again shows processing request. Even the iso status in management
>>> shows
>>> > nothing and when I stop the local file server the status of iso
>>> shows 0%
>>> > downloaded and then partial get request cannot be served. I believe
>>> > management server keeps on pinging the local file server but the
>>> connection
>>> > is not getting established.
>>> >
>>> > I even refreshed my server again and again and also did wait for a
>>> long
>>> > enough time to see if the process is slow. logging into system VM
>>> and
>>> > running ./run.sh does not show any error and cloud services are
>>> running
>>> > fine. I am pretty sure about the CIDR I have mentioned in
>>> > secstorage.allowed.internal.sites. Any idea?
>>> >

Re: Customer Backup with CS 4.7 and VMWare

[daniel.herrmann]

Hi Sebastián,

Thank you for your answer. This is exactly the same problem we are facing. Some 
customers have >1TB volumes, and it just takes ages to complete them. Which by 
the way would not be the actual problem, but sometimes CS does not even create 
the snapshot from the recurring snapshot policy or, even worse, a snapshot is 
created but never (>7d) finishes and remains in the BackingUp state, causing 
new snapshots of this series not to be created.

I read about a solution with Veeam and Tags (e.g. let the customer tag the 
virtual machines, and Veeam automatically backups the tagged machines), but 
this adds problems such as:

- how to bill the usage of this method?
- We could restore the virtual machine to an earlier state. But if the customer 
accidently deleted the machine, we cannot create it back from the backup as CS 
would not recognize it again.

So... if anyone has further insight, we'd be happy to hear about it. (

Regards
Daniel
 
On 08.02.18, 08:21, "Sebastian Gomez"  wrote:

Hello Daniel,

We have the same environment, and the same problem.
I agree, the volume snapshots are a pain in time needings. Volume snapshot
does a full copy of the volume through the network from primary storage to
the secondary. May be there is a storage configuration that could optimize
this action, but we have iSCSI for primary storage and NFS por secondary...
For some big volumes it takes up to 8 h to complete the snap.

This is NOT a sustainable solution.

Our customers uses backup agents of their backup solutions, and we (as
providers) have a backup at VMware level (you can find many solutions like
vRanger, veeam, ...), is the only way that we have found to have a
disaster-recovery backup of the platform. We are working now on how to
offer to customers using this backup as a service, facing to have a
global-unique backup solution (and scalable) for all the platform and users.

In this way, for example Veeam backup offers many options to allow users to
recover their own data (configuring access via API), but the problem here
is that in Cloudstack you can't recover a virtual machine on the
virtualization layer without informing the cloud-manager...


Perhaps someone else could light us.




Regards.





Atentamente,
Sebastián Gómez

On Wed, Feb 7, 2018 at 3:35 PM,  wrote:

> Hi All,
>
> We are using CS 4.7.1 with VMWare Hypervisor and advanced networking in a
> private cloud environment. Currently, most of our (internal) customers
> hosting internal services within this environment are using volume
> snapshots to facilitate backups of their virtual machines. Besides the
> obvious downsides of this approach (consistent snapshots of multiple
> volumes, …) we encounter serious problems using this features. In ~10% of
> the cases, snapshots get stuck in the BackingUp state, which sometimes
> causes the whole snapshot queue to stale. In some other cases, recurring
> snapshots are correctly configured, but CS does not try even try to create
> this snapshot, there is no entry in the database.
>
> In summary, we are currently evaluating different options, hence my
> questions here:
>
>
>   *   Are we the only ones encountering that massive problems with volume
> snapshots? Or is this a known problem? Anything we could look at or a hint
> where we could start troubleshooting?
>   *   How are you actually providing backup services to the customer? Are
> there other solutions or products that integrate with CS?
>
> When using another option than the volume snapshots in CS, the most
> important factor would be to keep the ability for the customer to 
configure
> everything in self-service.
>
> Thanks and regards
> Daniel
>