Re: Rohit Vavaldas - GSoC - 2018

[Daan Hoogland]

Hey Rohit, nice to see you here. What you should be looking at is mostly
the register template API and service calls and then the update version
code for the last version that required an upgrade. Off the top of my head
it is in the class Upgrade410to411.java (don't kill me if I'm off by a few
chars) next to that there is ofcourse the upgrade process of the individual
instances of system VMs. A whole other story, part of which you can find in
the discuss thread [1] and a proposal [2] on the mailing list arcghives.

[1]
https://lists.apache.org/thread.html/6116f1d77cc2272f75f844a22597b296aa1f84f7acbd2a4f82127743@%3Cdev.cloudstack.apache.org%3E
[2]
https://lists.apache.org/thread.html/fe347ccf41073def042d35b51ce6007a0f3fe898690f5328e40143e1@%3Cdev.cloudstack.apache.org%3E

please feel free to inquire further ...



On Mon, Feb 26, 2018 at 5:02 PM, Rohit vavaldas 
wrote:

> Hi,
>
> I am Rohit Vavaldas, a graduate student pursuing my masters in computer
> science at the University of Oklahoma. I am very much interested to work on
> CloudStack and excited to participate in GSoC. I am interested to work on
> the issue to create an API driven SystemVM upgrade. The current flow of
> updating system VM template is to first register newer version of system VM
> template using 'registertemplate' API and then invoke 'preparetemplate' on
> the above-registered template. So, part of this, what should I be looking
> out for to develop this API and also can I please get some more information
> in this regard?
>
> Link to the issue:
>
> https://issues.apache.org/jira/browse/CLOUDSTACK-10262
>
>
>
> Thanks and Regards
> Rohit Vavaldas
>



-- 
Daan

Re: POM version in 4.11 branch

[Rafael Weingärtner]

So, this means that we have to fix the version to 4.11.1.0, right?

On Mon, Feb 26, 2018 at 1:06 PM, Khosrow Moossavi 
wrote:

> I see, thanks Rohit.
>
>
>
> On Mon, Feb 26, 2018 at 10:58 AM, Rohit Yadav 
> wrote:
>
> > Thanks for sharing Khosrow, yes that's an error. There was an issue
> around
> > db failure for 4.10.0.0 users upgrade to 4.11.0.0 -- which was sorted in
> > release notes instead of doing a new 4.11.0.1 release.
> >
> >
> > Let me fix that.
> >
> >
> > - Rohit
> >
> > 
> >
> >
> >
> > 
> > From: Khosrow Moossavi 
> > Sent: Monday, February 26, 2018 4:46:36 PM
> > To: dev@cloudstack.apache.org
> > Subject: POM version in 4.11 branch
> >
> > Hi
> >
> > While working on a PR, Rafael and myself noticed that the POM version of
> > 4.11 branch
> > is 4.11.0.1-SNAPSHOT and not 4.11.1.0-SNAPSHOT.
> > Was this planned? Shouldn't we be on micro version rather than patch
> > version?
> >
> > Khosrow Moossavi
> >
> > CloudOps
> >
> > rohit.ya...@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
>



-- 
Rafael Weingärtner

Re: [4.11] Management to VR connection issues

[Rene Moser]

On 02/26/2018 12:41 PM, Rohit Yadav wrote:

> - If waiting for ssh and apache2 as part of post-init solves the issue, this 
> would require a new systemvmtemplate as the systemd scripts cannot be changed 
> or make effect during first boot.

The waiting for ssh was not the issue, it was a result.

The hang of cloud-postinit caused by p.wait() when having a ton of
iptable rules was the issue. But this is addressed already. should be fine.

a systemctl list-jobs shows "no pending jobs" anymore, so the boot has
completed.

After that the VR should be accessable by SSH (3922) by managemement
right, but it is not.

Did you see  the changes after a reboot (please compare the screenshots
of the ip addr output I sent). After that reboot/network change, SSH
works...


> - I think the additional nics always used to show up for vmware, there is a 
> global setting to configure this (extra nics for vmware, probably because 
> older versions did not support dynamic nic addition on vmware vrs).

On 4.5.2, we only see 4 NICs. in 4.11 we see 5 of them. We were just
wondering if this could result in an issue. What global setting would
that be?


> - For VR timeouts, see logs and check if from management server host you're 
> able to SSH into the VR using the private IP and port 3922. See the 
> troubleshooting wiki: 
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM%2C+templates%2C+Secondary+storage+troubleshooting

Yes, after a manual reboot of the VR, we can SSH-in as I wrote. Without
a reboot of the VR, we get a "no route to host". So it seems not even an
arp ping is working.


> - Can you share/check which processes are consuming the RAM, 256MB ram is 
> usually enough for non-redundant VRs. (share output of top or check using 
> htop?). Make sure to use a latest Linux version (any Debian variant such as 
> Debian 8, 9 or Ubuntu 16.04+ may also work). The issue is vCenter/ESXi 6.5 
> for some reason, gives lower RAM compared to 6.0 and 5.5 and has poor support 
> for legacy os. I had faced/found this issue while testing redundant VRs which 
> take more RAM usually than normal VRs.

Using the shapeblue VR template (your template ;))

So the man docs says
https://manpages.debian.org/stretch/initscripts/tmpfs.5.en.html

unfortunately only a fstab entry worked for me, setting the
/etc/default/tmpfs didn't.

https://github.com/apache/cloudstack/pull/2468/commits/bd882a8f80763595a89a3b74330500e1965bfda3

Re: POM version in 4.11 branch

[Khosrow Moossavi]

I see, thanks Rohit.



On Mon, Feb 26, 2018 at 10:58 AM, Rohit Yadav 
wrote:

> Thanks for sharing Khosrow, yes that's an error. There was an issue around
> db failure for 4.10.0.0 users upgrade to 4.11.0.0 -- which was sorted in
> release notes instead of doing a new 4.11.0.1 release.
>
>
> Let me fix that.
>
>
> - Rohit
>
> 
>
>
>
> 
> From: Khosrow Moossavi 
> Sent: Monday, February 26, 2018 4:46:36 PM
> To: dev@cloudstack.apache.org
> Subject: POM version in 4.11 branch
>
> Hi
>
> While working on a PR, Rafael and myself noticed that the POM version of
> 4.11 branch
> is 4.11.0.1-SNAPSHOT and not 4.11.1.0-SNAPSHOT.
> Was this planned? Shouldn't we be on micro version rather than patch
> version?
>
> Khosrow Moossavi
>
> CloudOps
>
> rohit.ya...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>

Rohit Vavaldas - GSoC - 2018

[Rohit vavaldas]

Hi,

I am Rohit Vavaldas, a graduate student pursuing my masters in computer
science at the University of Oklahoma. I am very much interested to work on
CloudStack and excited to participate in GSoC. I am interested to work on
the issue to create an API driven SystemVM upgrade. The current flow of
updating system VM template is to first register newer version of system VM
template using 'registertemplate' API and then invoke 'preparetemplate' on
the above-registered template. So, part of this, what should I be looking
out for to develop this API and also can I please get some more information
in this regard?

Link to the issue:

https://issues.apache.org/jira/browse/CLOUDSTACK-10262



Thanks and Regards
Rohit Vavaldas

Re: POM version in 4.11 branch

[Rohit Yadav]

Thanks for sharing Khosrow, yes that's an error. There was an issue around db 
failure for 4.10.0.0 users upgrade to 4.11.0.0 -- which was sorted in release 
notes instead of doing a new 4.11.0.1 release.


Let me fix that.


- Rohit






From: Khosrow Moossavi 
Sent: Monday, February 26, 2018 4:46:36 PM
To: dev@cloudstack.apache.org
Subject: POM version in 4.11 branch

Hi

While working on a PR, Rafael and myself noticed that the POM version of
4.11 branch
is 4.11.0.1-SNAPSHOT and not 4.11.1.0-SNAPSHOT.
Was this planned? Shouldn't we be on micro version rather than patch
version?

Khosrow Moossavi

CloudOps

rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

POM version in 4.11 branch

[Khosrow Moossavi]

Hi

While working on a PR, Rafael and myself noticed that the POM version of
4.11 branch
is 4.11.0.1-SNAPSHOT and not 4.11.1.0-SNAPSHOT.
Was this planned? Shouldn't we be on micro version rather than patch
version?

Khosrow Moossavi

CloudOps

Fwd: Google Summer of Code 2018 Mentor Registration

[Daan Hoogland]

-- Forwarded message --
From: Ulrich Stärk 
Date: Sat, Feb 24, 2018 at 10:19 PM
Subject: Google Summer of Code 2018 Mentor Registration
To: ment...@community.apache.org
Cc: "d...@community.apache.org" 


Dear PMCs,

I'm happy to announce that the ASF has made it onto the list of accepted
organizations for
Google Summer of Code 2018! [1,2]

It is now time for mentors to sign up, so please pass this email on to your
community and
podlings. If you aren’t already subscribed to ment...@community.apache.org
you should do so now else
you might miss important information.

Mentor signup requires two steps: mentor signup in Google's system [3] and
PMC acknowledgement.

If you want to mentor a project in this year's SoC you will have to

1. Be an Apache committer.
2. Request an acknowledgement from the PMC for which you want to mentor
projects. Use the below
template and *do not forget to copy ment...@community.apache.org*. We will
use the email adress you
indicate to send the invite to be a mentor for Apache.

PMCs, read carefully please.

We request that each mentor is acknowledged by a PMC member. This is to
ensure the mentor is in good
standing with the community. When you receive a request for
acknowledgement, please ACK it and cc
ment...@community.apache.org

Lastly, it is not yet too late to record your ideas in Jira (see my
previous emails for details).
Students will now begin to explore ideas so if you haven’t already done so,
record your ideas
immediately!

Cheers,

Uli

mentor request email template:

to: private@.apache.org
cc: ment...@community.apache.org
subject: GSoC 2018 mentor request for 

 PMC,

please acknowledge my request to become a mentor for Google Summer of Code
2018 projects for Apache
.

I would like to receive the mentor invite to 





[1] https://summerofcode.withgoogle.com/organizations/
[2] https://summerofcode.withgoogle.com/organizations/5718432427802624/
[3] https://summerofcode.withgoogle.com/



-- 
Daan

Re: [cloudstack] annotated tag 4.9.1-RC1 updated (af26799 -> 2d788ee)

[Rafael Weingärtner]

No, I did not. I do not understand what happened. I tried to look for some
information online, but they were not clarifying (some were talking about a
tag being changed in the remote).

After re-reading the email for the fourth time, I noticed: " No new
revisions were added by this update ". Then, I ignored it.

On Mon, Feb 26, 2018 at 10:09 AM, Daan Hoogland 
wrote:

> did you resolve this yet, @rafael?
>
> it doesn't seem right that a commit other then a pom change has such a tag.
> On the other hand it says it is an RC-tag which could be deleted by now, or
> replaced by a GA-tag.
>
> On Thu, Feb 22, 2018 at 11:55 PM, Rafael Weingärtner <
> rafaelweingart...@gmail.com> wrote:
>
> > Does anybody understand what this one means? Did I make any mistake while
> > merging forward?
> > I pulled master branch to execute e forward merge from 4.11, and as soon
> as
> > I pushed the commit I noticed this email.
> >
> > -- Forwarded message --
> > From: 
> > Date: Thu, Feb 22, 2018 at 7:51 PM
> > Subject: [cloudstack] annotated tag 4.9.1-RC1 updated (af26799 ->
> 2d788ee)
> > To: "' comm...@cloudstack.apache.org" ,
> '@
> > gitbox.apache.org
> >
> >
> > This is an automated email from the ASF dual-hosted git repository.
> >
> > rafael pushed a change to annotated tag 4.9.1-RC1
> > in repository https://gitbox.apache.org/repos/asf/cloudstack.git.
> >
> >
> > *** WARNING: tag 4.9.1-RC1 was modified! ***
> >
> > from af26799  (commit)
> >   to 2d788ee  (tag)
> >  tagging af2679959b634d095b93b8265c6da294d360065d (commit)
> >  replaces portgroup_no_gc_tag
> >   by Boris
> >   on Mon Dec 12 15:44:56 2016 +0200
> >
> > - Log -
> > 4.9.1-RC1
> > ---
> >
> >
> > No new revisions were added by this update.
> >
> > Summary of changes:
> >
> > --
> > To stop receiving notification emails like this one, please contact
> > raf...@apache.org.
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
>
>
>
> --
> Daan
>



-- 
Rafael Weingärtner

Re: [cloudstack] annotated tag 4.9.1-RC1 updated (af26799 -> 2d788ee)

[Daan Hoogland]

did you resolve this yet, @rafael?

it doesn't seem right that a commit other then a pom change has such a tag.
On the other hand it says it is an RC-tag which could be deleted by now, or
replaced by a GA-tag.

On Thu, Feb 22, 2018 at 11:55 PM, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> Does anybody understand what this one means? Did I make any mistake while
> merging forward?
> I pulled master branch to execute e forward merge from 4.11, and as soon as
> I pushed the commit I noticed this email.
>
> -- Forwarded message --
> From: 
> Date: Thu, Feb 22, 2018 at 7:51 PM
> Subject: [cloudstack] annotated tag 4.9.1-RC1 updated (af26799 -> 2d788ee)
> To: "' comm...@cloudstack.apache.org" , '@
> gitbox.apache.org
>
>
> This is an automated email from the ASF dual-hosted git repository.
>
> rafael pushed a change to annotated tag 4.9.1-RC1
> in repository https://gitbox.apache.org/repos/asf/cloudstack.git.
>
>
> *** WARNING: tag 4.9.1-RC1 was modified! ***
>
> from af26799  (commit)
>   to 2d788ee  (tag)
>  tagging af2679959b634d095b93b8265c6da294d360065d (commit)
>  replaces portgroup_no_gc_tag
>   by Boris
>   on Mon Dec 12 15:44:56 2016 +0200
>
> - Log -
> 4.9.1-RC1
> ---
>
>
> No new revisions were added by this update.
>
> Summary of changes:
>
> --
> To stop receiving notification emails like this one, please contact
> raf...@apache.org.
>
>
>
> --
> Rafael Weingärtner
>



-- 
Daan

Re: Replacing download.cloud.com by download.cloudstack.org

[Wido den Hollander]

On 02/25/2018 12:11 AM, Chiradeep Vittal wrote:

Which releases still refer to them in the setup SQL?



I think it was 4.9? Does anybody know this exactly?

Wido


Sent from my iPhone


On Feb 24, 2018, at 12:31 PM, Wido den Hollander  wrote:




On 02/24/2018 12:14 AM, Chiradeep Vittal wrote:
Citrix is wondering if they can shut down download.cloud.com at this point.


Grepping through the source I see no references to download.cloud.com anymore.

I *think* it can be shut down. Any other opinions on this?

Wido


Thanks!

On Fri, Mar 10, 2017 at 2:39 AM, Wido den Hollander  wrote:


Op 10 maart 2017 om 11:36 schreef Wido den Hollander :


Hi,

Will Stevens and I have been working on a replacement for download.cloud.com.

This resulted in download.cloudstack.org which is also available over SSL!

download.cloudstack.org is a CNAME to cloudstack.apt-get.eu and is hosted by me 
in Amsterdam (on CloudStack and Ceph).

In the future I'd like to scale this out to more systems on different providers 
by using RR-DNS, but for now let's not make it very difficult.

There is a PR open [0] to start to do this, but we also have documentation 
which might need to be fixed.

RPM and DEB packages are also available on download.cloudstack.org:

- RPM: https://download.cloudstack.org/centos
- DEB: https://download.cloudstack.org/ubuntu

SystemVMs are also available: https://download.cloudstack.org/systemvm/

In total there is 126GB of data on the server:

0 centos7
4.0K  README
4.0K  release.asc
4.0K  RPM-GPG-KEY
12K   foss
316K  tools
629M  tmp
2.9G  archive
3.7G  rhel
5.3G  releases
8.3G  centos
19G   ubuntu
24G   systemvm
64G   templates

If you want to be able to add files to the system, send me your SSH key and we 
can fix this. A few PMC members already have access and can add files.

Any things which we need to fix in addition?



Forgot to add, the system already has rsync available, so you can sync it if 
you want:

$ rsync -avr --stats --progress download.cloudstack.org::cloudstack .

Might want to do that already so that we can quickly move to a different server 
should the one in Amsterdam be down for a long time.


Wido

[0]: https://github.com/apache/cloudstack/pull/1582

Re: [PROPOSAL] reducing VR downtime on upgrade

[Daan Hoogland]

Thanks Wido,
I suspect to start a PoC on fase 1 shortly.

On Wed, Feb 21, 2018 at 9:38 PM, Wido den Hollander  wrote:

>
>
> On 02/15/2018 04:36 PM, Daan Hoogland wrote:
>
>> The intention of this proposal is to have a way forward to reducing
>> maintenance downtime for virtual routers. There are two parts to this
>> proposal;
>>
>>1.  Dealing with legacy routers and replacing them before shutting
>> down.
>>2.  Unifying router embodiments and making use of redundancy
>> mechanisms to quickly failover from old to new.
>>
>> Ad .1 It will always be possible that a router is to old and will not be
>> able to talk to a new version that is to replace it. This might be due to a
>> keepalived update or replacement or just because it is very old. So though
>> Unifying the routers and making them redundant enabled will solve a lot of
>> use cases it will never deal with any conceivable situation, not even in
>> systems upgraded to a version in which all intended functionality has been
>> implemented. Dealing with any older router is to work as follows:
>>
>>1.  A check will be done to make sure the old VR is still up.
>>   *   If it is not there is no consideration it will be replaced as
>> quickly as possible. Possible improvements here are the iptables
>> configuration speedup and other generic optimisations unrelated to the
>> upgrade itself.
>>   *   If it is there we need to walk on eggs with provisioning the
>> new one
>>2.  A new VR will be instantiated
>>3.  Configuration data will be send but not applied.
>>4.  The interfaces will be added and if need be brought down.
>>5.  All configuration is applied
>>6.  The old VR is killed
>>7.  The interface on the new VR are brought up
>>
>>
> Looks good! We might want the VR to send out it's version as well over the
> local socket. Using that 'version' you could see if it supports various
> things.
>
> You could even have the VR send out 'features' so that you know what it's
> capable of.
>
> Ad .2 This is a long-term goal. At the moment we have five (or debatably
>> six) different incarnations of the virtual router:
>>
>>*   Basic zone dhcp server
>>*   Shared network ‘router’
>>*   VR
>>*   rVR
>>*   VPC
>>*   rVPC
>>
>
> Don't forget the metadata/password server it runs in almost all cases.
>
> Wido
>
>
> a first set of steps will be to reduce this to
>>
>>*   shared networks (where a basic zone is an automatic implementation
>> of a single shared network in a zone)
>>*   VR (which is always redundant enabled but may have only one
>> instance)
>>*   VPC (as above)
>> and then the next step is to unify VR and VPC as a VR is really only a
>> VPC with just one network
>> the final step is then to unify a shared network with a VPC and this one
>> is so far ahead that I don’t want to make too much statements about it now.
>> We will have to find the exact implementation hazards that we will face in
>> this step along the way. I think we are talking at least one year in when
>> we reach this point.
>>
>> As Shapeblue we will be starting a short PoC on the first part. We will
>> try to figure out if the process under .1 is feasible, or that we need to
>> wait configuring interfaces to the last mo
>> ment
>> and then do a ‘blind’ start.
>>
>> daan.hoogl...@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> @shapeblue
>>
>>
>


-- 
Daan

Re: [4.11] Management to VR connection issues

[Rohit Yadav]

Hi Rene,


- I think on the general issue of slow iptables rules application, we need to 
fix that. Does it help to increase aggregation timeouts?


- If waiting for ssh and apache2 as part of post-init solves the issue, this 
would require a new systemvmtemplate as the systemd scripts cannot be changed 
or make effect during first boot.


- I think the additional nics always used to show up for vmware, there is a 
global setting to configure this (extra nics for vmware, probably because older 
versions did not support dynamic nic addition on vmware vrs).


- For VR timeouts, see logs and check if from management server host you're 
able to SSH into the VR using the private IP and port 3922. See the 
troubleshooting wiki: 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM%2C+templates%2C+Secondary+storage+troubleshooting


- Can you share/check which processes are consuming the RAM, 256MB ram is 
usually enough for non-redundant VRs. (share output of top or check using 
htop?). Make sure to use a latest Linux version (any Debian variant such as 
Debian 8, 9 or Ubuntu 16.04+ may also work). The issue is vCenter/ESXi 6.5 for 
some reason, gives lower RAM compared to 6.0 and 5.5 and has poor support for 
legacy os. I had faced/found this issue while testing redundant VRs which take 
more RAM usually than normal VRs.


- Rohit






From: Rene Moser 
Sent: Monday, February 26, 2018 11:22:27 AM
To: us...@cloudstack.apache.org; dev@cloudstack.apache.org
Subject: Re: [4.11] Management to VR connection issues

Hi again

We found the main problem.

== cloud-postinit hang

When having many iptables rules resulting in cloud-postinit to hang for
10min unless it was killed by systemd. As a result the ssh daemon was
not started for 10 min because it is configured to be started after
cloud-postinit.

It seems the issue was already fixed by
https://github.com/apache/cloudstack/commit/ce67726c6d3db6e7db537e76da6217c5d5f4b10e

== VR still needs manual reboot

However, we still notice adapter changes after a reboot: see before
after screenshots of "ip addr" in
https://photos.app.goo.gl/9XsjOJjLqQ9SRjYV2. We still need to manually
reboot the VR to make the network actually working.

== VR has too many adapters?

Next thing we noticed there are many network adapters (NICs) for this
non-vpc router (see screenshot of the vcenter in
https://photos.app.goo.gl/9XsjOJjLqQ9SRjYV2). Adapter 4 and 5 seem
unnecessary. Any comments on that?

== VR with 256 MB RAM dows not work

Next issue we found is, that the VR must have more than 256MB RAM.
Otherwise systemd will complain the daemon can not be reloaded, because
the ram disk of /run has too less space.

Feb 23 16:24:36 r-413-VM postinit.sh[1089]: Failed to reload daemon:
Refusing to reload, not enough space available on /run/systemd.
Currently, 8.6M are free, but a safety buffer of 16.0M is enforced.
root@r-413-VM:~# df -h /run/
Filesystem  Size  Used Avail Use% Mounted on
tmpfs16M  7.2M  8.7M  46% /run

Increaing to 512MB RAM helped:

root@r-413-VM:~# df -h /run/
Filesystem  Size  Used Avail Use% Mounted on
tmpfs41M  7.8M   34M  19% /run

Unsure if this can be tuned on systemd level, didn't find a way yet.

== VR API Command timeouts

When executing command related to VR, e.g. restart network, start/stop
router the command won't reach the vcenter api, and times out. We are
unsure yet, why.

== VR minor fixes

Next we fixed 2 minor things along.

* rsyslogd config syntax issue
* IMHO we should start apache2 also after cloud-postinit

Also see https://github.com/apache/cloudstack/pull/2468

Regards
René

rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Re: [4.11] Management to VR connection issues

[Rene Moser]

Hi Paul

On 02/26/2018 11:35 AM, Paul Angus wrote:
> Rene,
> Have you checked the OS getting applied on vCenter?

It's "Other 3.x or later Linux (64-bit)" also see
https://photos.google.com/share/AF1QipPNnFnP8xMIHgYCQ0rZtDsyeVGoJIyHjPqWP8BP-hiNRnd0CxHuc8xn5GetIrpocQ/photo/AF1QipPIeKUnVU4hQ8_AuNT8galxvDyPsMqjkMkIqKPV?key=WmlJRUhMNnh3cXZheFp0WDZuMFZtTmpOQlo2Y2NR

RE: [4.11] Management to VR connection issues

[Paul Angus]

Rene,
Have you checked the OS getting applied on vCenter?
A lot of the issues went away once I changed the OS when testing over the 
weekend.

Kind regards,

Paul Angus

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Rene Moser [mailto:m...@renemoser.net] 
Sent: 26 February 2018 10:22
To: us...@cloudstack.apache.org; dev@cloudstack.apache.org
Subject: Re: [4.11] Management to VR connection issues

Hi again

We found the main problem.

== cloud-postinit hang

When having many iptables rules resulting in cloud-postinit to hang for 10min 
unless it was killed by systemd. As a result the ssh daemon was not started for 
10 min because it is configured to be started after cloud-postinit.

It seems the issue was already fixed by
https://github.com/apache/cloudstack/commit/ce67726c6d3db6e7db537e76da6217c5d5f4b10e

== VR still needs manual reboot

However, we still notice adapter changes after a reboot: see before after 
screenshots of "ip addr" in https://photos.app.goo.gl/9XsjOJjLqQ9SRjYV2. We 
still need to manually reboot the VR to make the network actually working.

== VR has too many adapters?

Next thing we noticed there are many network adapters (NICs) for this non-vpc 
router (see screenshot of the vcenter in 
https://photos.app.goo.gl/9XsjOJjLqQ9SRjYV2). Adapter 4 and 5 seem unnecessary. 
Any comments on that?

== VR with 256 MB RAM dows not work

Next issue we found is, that the VR must have more than 256MB RAM.
Otherwise systemd will complain the daemon can not be reloaded, because the ram 
disk of /run has too less space.

Feb 23 16:24:36 r-413-VM postinit.sh[1089]: Failed to reload daemon:
Refusing to reload, not enough space available on /run/systemd.
Currently, 8.6M are free, but a safety buffer of 16.0M is enforced.
root@r-413-VM:~# df -h /run/
Filesystem  Size  Used Avail Use% Mounted on
tmpfs16M  7.2M  8.7M  46% /run

Increaing to 512MB RAM helped:

root@r-413-VM:~# df -h /run/
Filesystem  Size  Used Avail Use% Mounted on
tmpfs41M  7.8M   34M  19% /run

Unsure if this can be tuned on systemd level, didn't find a way yet.

== VR API Command timeouts

When executing command related to VR, e.g. restart network, start/stop router 
the command won't reach the vcenter api, and times out. We are unsure yet, why.

== VR minor fixes

Next we fixed 2 minor things along.

* rsyslogd config syntax issue
* IMHO we should start apache2 also after cloud-postinit

Also see https://github.com/apache/cloudstack/pull/2468

Regards
René

Re: [4.11] Management to VR connection issues

[Rene Moser]

Hi again

We found the main problem.

== cloud-postinit hang

When having many iptables rules resulting in cloud-postinit to hang for
10min unless it was killed by systemd. As a result the ssh daemon was
not started for 10 min because it is configured to be started after
cloud-postinit.

It seems the issue was already fixed by
https://github.com/apache/cloudstack/commit/ce67726c6d3db6e7db537e76da6217c5d5f4b10e

== VR still needs manual reboot

However, we still notice adapter changes after a reboot: see before
after screenshots of "ip addr" in
https://photos.app.goo.gl/9XsjOJjLqQ9SRjYV2. We still need to manually
reboot the VR to make the network actually working.

== VR has too many adapters?

Next thing we noticed there are many network adapters (NICs) for this
non-vpc router (see screenshot of the vcenter in
https://photos.app.goo.gl/9XsjOJjLqQ9SRjYV2). Adapter 4 and 5 seem
unnecessary. Any comments on that?

== VR with 256 MB RAM dows not work

Next issue we found is, that the VR must have more than 256MB RAM.
Otherwise systemd will complain the daemon can not be reloaded, because
the ram disk of /run has too less space.

Feb 23 16:24:36 r-413-VM postinit.sh[1089]: Failed to reload daemon:
Refusing to reload, not enough space available on /run/systemd.
Currently, 8.6M are free, but a safety buffer of 16.0M is enforced.
root@r-413-VM:~# df -h /run/
Filesystem  Size  Used Avail Use% Mounted on
tmpfs16M  7.2M  8.7M  46% /run

Increaing to 512MB RAM helped:

root@r-413-VM:~# df -h /run/
Filesystem  Size  Used Avail Use% Mounted on
tmpfs41M  7.8M   34M  19% /run

Unsure if this can be tuned on systemd level, didn't find a way yet.

== VR API Command timeouts

When executing command related to VR, e.g. restart network, start/stop
router the command won't reach the vcenter api, and times out. We are
unsure yet, why.

== VR minor fixes

Next we fixed 2 minor things along.

* rsyslogd config syntax issue
* IMHO we should start apache2 also after cloud-postinit

Also see https://github.com/apache/cloudstack/pull/2468

Regards
René

Re: I'd like to introduce you to Khosrow

[Dag Sonstebo]

Welcome Khosrow!

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 26/02/2018, 08:35, "Nux!"  wrote:

Welcome aboard Khosrow :)

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro


dag.sonst...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

- Original Message -
> From: "Pierre-Luc Dion" 
> To: "dev" , "Khosrow Moossavi" 

> Sent: Thursday, 22 February, 2018 23:40:41
> Subject: I'd like to introduce you to Khosrow

> Hi fellow colleagues,
> 
> I might be a bit late with this email...
> 
> I'd like to introduce Khosrow Moossavi, who recently join our team and his
> focus is currently exclusively on dev for Cloudstack with cloud.ca.
> 
> Our 2 current priorities are:
> -fixing VRs,SVMs to run has HVM VMs in xenserver.
> - redesign, or rewrite, the remote management vpn for vpc, poc in progress
> for IKEv2...
> 
> 
> 
> Some of you might have interact with him already.
> 
> 
> Also, we are going to be more active for the upcomming 4.12 release.
> 
> 
> Cheers!

Re: I'd like to introduce you to Khosrow

[Nux!]

Welcome aboard Khosrow :)

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

- Original Message -
> From: "Pierre-Luc Dion" 
> To: "dev" , "Khosrow Moossavi" 
> Sent: Thursday, 22 February, 2018 23:40:41
> Subject: I'd like to introduce you to Khosrow

> Hi fellow colleagues,
> 
> I might be a bit late with this email...
> 
> I'd like to introduce Khosrow Moossavi, who recently join our team and his
> focus is currently exclusively on dev for Cloudstack with cloud.ca.
> 
> Our 2 current priorities are:
> -fixing VRs,SVMs to run has HVM VMs in xenserver.
> - redesign, or rewrite, the remote management vpn for vpc, poc in progress
> for IKEv2...
> 
> 
> 
> Some of you might have interact with him already.
> 
> 
> Also, we are going to be more active for the upcomming 4.12 release.
> 
> 
> Cheers!