RE: [DISCUSS] CloudStack 4.9.3.0 (LTS)

[Sean Lair]

Hi Rohit

I previous suggested these for 4.9.3.0

https://github.com/apache/cloudstack/pull/2041 (VR related jobs scheduled and 
run twice on mgmt servers)
https://github.com/apache/cloudstack/pull/2040 (Bug in monitoring of S2S VPNs - 
also exists in 4.10)
https://github.com/apache/cloudstack/pull/1966 (IPSEC VPNs do not work after 
vRouter reboot)

I'd also like to suggest these:
https://github.com/apache/cloudstack/pull/1246 (unable to use reserved IP range 
in a network)
https://github.com/apache/cloudstack/pull/2201 (VPC VR doesn't respond to DNS 
requests from remote access vpn clients)


Thanks
Sean

-Original Message-
From: Rohit Yadav [mailto:rohit.ya...@shapeblue.com] 
Sent: Monday, July 24, 2017 5:56 AM
To: dev@cloudstack.apache.org; us...@cloudstack.apache.org
Subject: Re: [DISCUSS] CloudStack 4.9.3.0 (LTS)

All,


We'll accept bugfixes on 4.9 branch till end of next week, following which I'll 
start release work towards 4.9.3.0 (LTS) release. Please help review 
outstanding PRs, share PRs that we should consider and advise/suggest issues 
that need to be reverted/backported, for example see: 
https://github.com/apache/cloudstack/pull/2052


Thank you for your support and co-operation.


- Rohit


From: Rohit Yadav 
Sent: Sunday, July 23, 2017 1:26:48 PM
To: dev@cloudstack.apache.org; us...@cloudstack.apache.org
Subject: Re: [DISCUSS] CloudStack 4.9.3.0 (LTS)

All,


I've started looking into reviewing/testing/merging of the PRs targeting 4.9+, 
I'll share some plans around 4.9.3.0 soon. Meanwhile, help in reporting any 
major/critical bugs and PRs we should consider reviewing/testing/merging. 
Thanks.


- Rohit

rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue




rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
  
 

Re: [DISCUSS] Move to Debian9 systemvmtemplate

[Rene Moser]

Hi Rohit


On 07/23/2017 06:08 PM, Rohit Yadav wrote:
> All,
> 
> 
> Just want to kick an initial discussion around migration to Debian9 based 
> systemvmtemplate, and get your feedback on the same.
> 
> Here's a work-in-progress PR: https://github.com/apache/cloudstack/pull/2198

Have you considered to replace veewee by packer?

Our friends from schubergphilis have already done some work here
https://github.com/MissionCriticalCloud/systemvm-packer.

However there would be also an official way to convert the definitions
https://www.packer.io/guides/veewee-to-packer.html

Regards René

Re: [DISCUSS] Move to Debian9 systemvmtemplate

[Rene Moser]

Hi

On 07/23/2017 06:21 PM, Paul Angus wrote:
> 
> I think that we should look at lighter Distros like Arch Linux in order to 
> get the boot times of the system VMs down.
> That in conjunction with improving the configuration will gives as a much 
> leaner, meaner System VM.

I tend to -1 for Arch.

Debian is a proven mature Distro. It is used as base for many critical
systems e.g. VyOS or also cumulus OS, not to mention Ubuntu.

Arch linux was not designed as a server OS. It has a rolling release
policy and it is great if you want to use the latest releases, you have
to update often.

Because we usually do not update VR OS that often, Arch is IMHO not a
good candidate. On the other hand Debian is a good one.

I would also doubt that Arch "boots" faster than Debian. IMHO the ways
_we_ configure the VR is still the performance killer.

Let's stick with Debian.

Re: Ubuntu 14.04 + qemu 2.5 vs. ACS 4.8 advice needed

[Dmytro Shevchenko]

Hello Wido,

we found source of this bug and also found that patch already implemented:

https://github.com/apache/cloudstack/commit/9dcfbceae71865b4de1a4744ceac8f48255733a2

sorry for inaccuracy, initially we tested on 4.5 release


On 24.07.17 14:56, Wido den Hollander wrote:

Op 21 juli 2017 om 17:57 schreef Dmytro Shevchenko :


Here is the error that we caught with Libvirt 1.3.1:

2017-07-21 11:46:27,501 WARN  [kvm.resource.LibvirtComputingResource] 
(agentRequest-Handler-1:null)  Plug Nic failed due to 
org.libvirt.LibvirtException: internal error: unable to execute QEMU command 
'device_add': Hot-plugged device without ROM bar can't have an option ROM
org.libvirt.LibvirtException: internal error: unable to execute QEMU command 
'device_add': Hot-plugged device without ROM bar can't have an option ROM
  at org.libvirt.ErrorHandler.processError(Unknown Source)
  at org.libvirt.Connect.processError(Unknown Source)


Can you show where this happens? Is that on VM start? If so, you see the XML in 
the agent.log and start then fails?

This seems like a libvirt 1.3.1 thing we might need to take a look at.

Wido


On 21.07.17 16:18, Andrija Panic wrote:

Hi all,

we are using ACS 4.8, on Ubuntu 14.04 (stock version qemu/libvirt -
2.0.0/1.2.2) and we have issues during live migration of "busy" VMs (high
RAM change rate), so we plan to introduce additional flag+global parameter
in ACS, which will auto converge flag libvirt to enable auto convergence
(thanks Mike. T :) )

qemu 1.6+ supports auto convergence, but before 2.5 it's mostly useless
(based on docs and my own testing)

Since we cant upgrade to Ubuntu 16.04 before ACS 4.9 or 4.10 release, we
are forced to play with qemu/libvirt verisons, so I used Ubuntu OpenStack
repo for "MITAKA" release, which provides QEMU 2.5 / LIBVIRT 1.3.1 (auto
convergence works like charm).


Now, we are facing different issues that ACS send command to libvirt, but
libvirt basically do some silly things and cant start some VMs at all, etc.

Anyone running ACS 4.8 with qemu 2.5+ and libvirt 1.3 , any advice how to
proceed (our developers are already debugging things, but this takes time) ?

Any info is really appreciated.

Thanks !

Re: Ubuntu 14.04 + qemu 2.5 vs. ACS 4.8 advice needed

[Wido den Hollander]

> Op 21 juli 2017 om 17:57 schreef Dmytro Shevchenko 
> :
> 
> 
> Here is the error that we caught with Libvirt 1.3.1:
> 
> 2017-07-21 11:46:27,501 WARN  [kvm.resource.LibvirtComputingResource] 
> (agentRequest-Handler-1:null)  Plug Nic failed due to 
> org.libvirt.LibvirtException: internal error: unable to execute QEMU command 
> 'device_add': Hot-plugged device without ROM bar can't have an option ROM
> org.libvirt.LibvirtException: internal error: unable to execute QEMU command 
> 'device_add': Hot-plugged device without ROM bar can't have an option ROM
>  at org.libvirt.ErrorHandler.processError(Unknown Source)
>  at org.libvirt.Connect.processError(Unknown Source)
> 

Can you show where this happens? Is that on VM start? If so, you see the XML in 
the agent.log and start then fails?

This seems like a libvirt 1.3.1 thing we might need to take a look at.

Wido

> 
> On 21.07.17 16:18, Andrija Panic wrote:
> > Hi all,
> >
> > we are using ACS 4.8, on Ubuntu 14.04 (stock version qemu/libvirt -
> > 2.0.0/1.2.2) and we have issues during live migration of "busy" VMs (high
> > RAM change rate), so we plan to introduce additional flag+global parameter
> > in ACS, which will auto converge flag libvirt to enable auto convergence
> > (thanks Mike. T :) )
> >
> > qemu 1.6+ supports auto convergence, but before 2.5 it's mostly useless
> > (based on docs and my own testing)
> >
> > Since we cant upgrade to Ubuntu 16.04 before ACS 4.9 or 4.10 release, we
> > are forced to play with qemu/libvirt verisons, so I used Ubuntu OpenStack
> > repo for "MITAKA" release, which provides QEMU 2.5 / LIBVIRT 1.3.1 (auto
> > convergence works like charm).
> >
> >
> > Now, we are facing different issues that ACS send command to libvirt, but
> > libvirt basically do some silly things and cant start some VMs at all, etc.
> >
> > Anyone running ACS 4.8 with qemu 2.5+ and libvirt 1.3 , any advice how to
> > proceed (our developers are already debugging things, but this takes time) ?
> >
> > Any info is really appreciated.
> >
> > Thanks !
> >
>

Re: [DISCUSS] Closing old Pull Requests on Github

[Wido den Hollander]

> Op 24 juli 2017 om 10:47 schreef Marc-Aurèle Brothier :
> 
> 
> Hi Wido,
> 
> I have one comment on this topic. Some of those PRs are lying there because
> no one took the time to merge them (I have a couple like that) since they
> were not very important (I think it's the reason), fixing only a small
> glitch or improving an output. If we start to close the PRs because there
> isn't activity on them, we should be sure to treat all PRs equally in term
> on timeline when they arrive. Using the labels to sort them and make
> filtering easier would also be something important IMO. Today there are
> 200+ PRs but we cannot filter them and have not much idea on their status,
> except by checking if they are "mergeable". This should not conflict with
> the Jira tickets & discussion that happened previously.

Understood! But that's a matter of resources the community has. Each PR needs 
to be looked at by a volunteer, a committer who all have limited resources.

It's not good that PR's didn't get the attention they needed, but it's a fact 
that it happened.

I don't think we have the resources to manually check and label 200 PRs and see 
which one can be merged.

If a author thinks the PR is still valid he/she can open it again. It's not a 
hard-close as I put in the message, but a way to filter what we need to put 
attention on.

They can be labeled and handled then.

Wido

> 
> Marco
> 
> On Mon, Jul 24, 2017 at 10:22 AM, Wido den Hollander  wrote:
> 
> > Hi,
> >
> > While writing this e-mail we have 191 Open Pull requests [0] on Github and
> > that number keeps hovering around ~200.
> >
> > We have a great number of PRs being merged, but a lot of code is old and
> > doesn't even merge anymore.
> >
> > My proposal would be that we close all PRs which didn't see any activity
> > in the last 3 months (Jun, July and May 2017) with the following message:
> >
> > "This Pull Request is being closed for not seeing any activity since May
> > 2017.
> >
> > The CloudStack project is in a transition from the Apache Foundation's Git
> > infrastructure to Github and due to that not all PRs we able to be tested
> > and/or merged.
> >
> > It's not our intention to say that we don't value the PR, but it's a way
> > to get a better overview of what needs to be merged.
> >
> > If you think closing this PR is a mistake, please add a comment and
> > re-open the PR! If you do that, could you please make sure that the PR
> > merges against the branch it was submitted against?
> >
> > Thank you very much for your understanding and cooperation!"
> >
> > How does that sound?
> >
> > Wido
> >
> >
> > [0]: https://github.com/apache/cloudstack/pulls
> >

Re: [DISCUSS] CloudStack 4.9.3.0 (LTS)

[Rohit Yadav]

All,


We'll accept bugfixes on 4.9 branch till end of next week, following which I'll 
start release work towards 4.9.3.0 (LTS) release. Please help review 
outstanding PRs, share PRs that we should consider and advise/suggest issues 
that need to be reverted/backported, for example see: 
https://github.com/apache/cloudstack/pull/2052


Thank you for your support and co-operation.


- Rohit


From: Rohit Yadav 
Sent: Sunday, July 23, 2017 1:26:48 PM
To: dev@cloudstack.apache.org; us...@cloudstack.apache.org
Subject: Re: [DISCUSS] CloudStack 4.9.3.0 (LTS)

All,


I've started looking into reviewing/testing/merging of the PRs targeting 4.9+, 
I'll share some plans around 4.9.3.0 soon. Meanwhile, help in reporting any 
major/critical bugs and PRs we should consider reviewing/testing/merging. 
Thanks.


- Rohit

rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Re: [DISCUSS] Closing old Pull Requests on Github

[Rohit Yadav]

My personal opinion is such activities are not useful and generally a waste of 
time, it might give us a feel good factor that the total open PRs (for that 
matter JIRA tickets) are low but it may do more harm than good. In general, a 
PR might take up to 6 months to merge depending on how engaging the author(s), 
reviewers and RMs were, so I would not close any PRs at least a year old, for 
example yesterday I reviewed/tested/merged few PRs originally submitted in 
2015/2016 but reworked/rebased in recent weeks/months. However, it might make 
sense to close duplicate PRs and in other circumstances but avoid a general 
rule.


Such an exercise may not be very useful, instead I would advise that we make an 
effort to engage with reviewers and author(s) to get them fixed/merged in our 
free time.


- Rohit


From: Wido den Hollander 
Sent: Monday, July 24, 2017 10:22:28 AM
To: dev@cloudstack.apache.org
Subject: [DISCUSS] Closing old Pull Requests on Github

Hi,

While writing this e-mail we have 191 Open Pull requests [0] on Github and that 
number keeps hovering around ~200.

We have a great number of PRs being merged, but a lot of code is old and 
doesn't even merge anymore.

My proposal would be that we close all PRs which didn't see any activity in the 
last 3 months (Jun, July and May 2017) with the following message:

"This Pull Request is being closed for not seeing any activity since May 2017.

The CloudStack project is in a transition from the Apache Foundation's Git 
infrastructure to Github and due to that not all PRs we able to be tested 
and/or merged.

It's not our intention to say that we don't value the PR, but it's a way to get 
a better overview of what needs to be merged.

If you think closing this PR is a mistake, please add a comment and re-open the 
PR! If you do that, could you please make sure that the PR merges against the 
branch it was submitted against?

Thank you very much for your understanding and cooperation!"

How does that sound?

Wido


[0]: https://github.com/apache/cloudstack/pulls

rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Re: [DISCUSS] Closing old Pull Requests on Github

[Marc-Aurèle Brothier]

Hi Wido,

I have one comment on this topic. Some of those PRs are lying there because
no one took the time to merge them (I have a couple like that) since they
were not very important (I think it's the reason), fixing only a small
glitch or improving an output. If we start to close the PRs because there
isn't activity on them, we should be sure to treat all PRs equally in term
on timeline when they arrive. Using the labels to sort them and make
filtering easier would also be something important IMO. Today there are
200+ PRs but we cannot filter them and have not much idea on their status,
except by checking if they are "mergeable". This should not conflict with
the Jira tickets & discussion that happened previously.

Marco

On Mon, Jul 24, 2017 at 10:22 AM, Wido den Hollander  wrote:

> Hi,
>
> While writing this e-mail we have 191 Open Pull requests [0] on Github and
> that number keeps hovering around ~200.
>
> We have a great number of PRs being merged, but a lot of code is old and
> doesn't even merge anymore.
>
> My proposal would be that we close all PRs which didn't see any activity
> in the last 3 months (Jun, July and May 2017) with the following message:
>
> "This Pull Request is being closed for not seeing any activity since May
> 2017.
>
> The CloudStack project is in a transition from the Apache Foundation's Git
> infrastructure to Github and due to that not all PRs we able to be tested
> and/or merged.
>
> It's not our intention to say that we don't value the PR, but it's a way
> to get a better overview of what needs to be merged.
>
> If you think closing this PR is a mistake, please add a comment and
> re-open the PR! If you do that, could you please make sure that the PR
> merges against the branch it was submitted against?
>
> Thank you very much for your understanding and cooperation!"
>
> How does that sound?
>
> Wido
>
>
> [0]: https://github.com/apache/cloudstack/pulls
>

[DISCUSS] Closing old Pull Requests on Github

[Wido den Hollander]

Hi,

While writing this e-mail we have 191 Open Pull requests [0] on Github and that 
number keeps hovering around ~200.

We have a great number of PRs being merged, but a lot of code is old and 
doesn't even merge anymore.

My proposal would be that we close all PRs which didn't see any activity in the 
last 3 months (Jun, July and May 2017) with the following message:

"This Pull Request is being closed for not seeing any activity since May 2017.

The CloudStack project is in a transition from the Apache Foundation's Git 
infrastructure to Github and due to that not all PRs we able to be tested 
and/or merged.

It's not our intention to say that we don't value the PR, but it's a way to get 
a better overview of what needs to be merged.

If you think closing this PR is a mistake, please add a comment and re-open the 
PR! If you do that, could you please make sure that the PR merges against the 
branch it was submitted against?

Thank you very much for your understanding and cooperation!"

How does that sound?

Wido


[0]: https://github.com/apache/cloudstack/pulls

Re: [DISCUSS] Move to Debian9 systemvmtemplate

[Rohit Yadav]

Only though testing we can know if there are any regressions, with Debian7 to 
be unsupported somewhere in 2018 we need to start some work around this.

Distros such as Arch Linux are not solid enough or work out of the box and have 
bleeding edge unstable software. Debian 9 "stretch" finally have 
openjdk-8-jre-headless (which surprising was not in debian8, though was in 
debian8's backports) so we can move away from using Azul's distro as well, 
along with having updated stable dependencies/versions.


Debian distributions are usually gold standard for security with other distros 
such as Tails and Kali Linux build on top of it, maybe a reason why systemvm 
templates were originally build using Debian as base. Moving to a new 
distribution will be a lot more work given highly assumed and dependent code 
(bash/python scripts) that targets Debian based systemvmtemplate. Perhaps a 
more modular and modern approach would be to split systemvm services and run 
them in containers, using a light base such as Alpine Linux, however that's a 
different discussion and a much bigger project to undertake.


- Rohit


From: Jayapal Uradi 
Sent: Monday, July 24, 2017 6:34:08 AM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Move to Debian9 systemvmtemplate


First one come to my mind is VPN feature.
But other general issues will depend on what are changed in debain9.

Thanks,
Jayapal




rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

> On Jul 24, 2017, at 9:54 AM, Satki Badal  wrote:
>
> Hi Jayapal,
>
> Do we know what all features re covered by Marvin and what all will require 
> manual testing ?
> I guess without this list we can never be sure is all the features got tested 
> or not.
>
> Thanks,
> Satki
>
>> On 24-Jul-2017, at 9:48 AM, Jayapal Uradi  
>> wrote:
>>
>> Hi,
>>
>> We should throughly check all the VR features to see feature breakage 
>> because of  Debian9.
>>
>> Thanks,
>> Jayapal
>>
>>
>>> On Jul 23, 2017, at 9:38 PM, Rohit Yadav  wrote:
>>>
>>> All,
>>>
>>>
>>> Just want to kick an initial discussion around migration to Debian9 based 
>>> systemvmtemplate, and get your feedback on the same.
>>>
>>> Here's a work-in-progress PR: https://github.com/apache/cloudstack/pull/2198
>>>
>>>
>>> Regards.
>>>
>>> rohit.ya...@shapeblue.com
>>> www.shapeblue.com
>>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> @shapeblue
>>>
>>>
>>>
>>
>> DISCLAIMER
>> ==
>> This e-mail may contain privileged and confidential information which is the 
>> property of Accelerite, a Persistent Systems business. It is intended only 
>> for the use of the individual or entity to which it is addressed. If you are 
>> not the intended recipient, you are not authorized to read, retain, copy, 
>> print, distribute or use this message. If you have received this 
>> communication in error, please notify the sender and delete all copies of 
>> this message. Accelerite, a Persistent Systems business does not accept any 
>> liability for virus infected mails.
>>
>

RE: [DISCUSS] Move to Debian9 systemvmtemplate

[Wido den Hollander]

> Op 23 juli 2017 om 18:21 schreef Paul Angus :
> 
> 
> 
> I think that we should look at lighter Distros like Arch Linux in order to 
> get the boot times of the system VMs down.

Debian 9 uses systemd and that makes it boot a lot faster.

> That in conjunction with improving the configuration will gives as a much 
> leaner, meaner System VM.

A lot of things can be done, but going to Debian 9 isn't that difficult.

Yes, we will need to fix a few things around systemd and such, but Debian is a 
proven and stable distribution.

Let's not try to do two things at the same time.

Our current Debian 7 is dangerously old and needs to be replaced. Debian 9 
seems like a very good candidate to me.

Wido

> 
> Kind regards,
> 
> Paul Angus
> 
> paul.an...@shapeblue.com 
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>   
>  
> 
> 
> -Original Message-
> From: Rohit Yadav [mailto:rohit.ya...@shapeblue.com] 
> Sent: 23 July 2017 17:08
> To: dev@cloudstack.apache.org
> Subject: [DISCUSS] Move to Debian9 systemvmtemplate
> 
> All,
> 
> 
> Just want to kick an initial discussion around migration to Debian9 based 
> systemvmtemplate, and get your feedback on the same.
> 
> Here's a work-in-progress PR: https://github.com/apache/cloudstack/pull/2198
> 
> 
> Regards.
> 
> rohit.ya...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>   
>  
>