Re: [DISCUSS][SECURITY] Feature: Secure CloudStack Communications

2017-08-23 Thread ilya 
Awesome work - thank you Rohit.

On 8/23/17 12:49 PM, Rohit Yadav wrote:
> All,
> 
> 
> No regression is seen in the smoke test run, however, I'll leave the PR open 
> for some time to gather further feedback and reviews.
> 
> 
> - Rohit
> 
> 
> From: Rohit Yadav 
> Sent: Friday, August 18, 2017 4:09:30 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [DISCUSS][SECURITY] Feature: Secure CloudStack Communications
> 
> All,
> 
> 
> The feature is ready for your review, please see:
> 
> https://github.com/apache/cloudstack/pull/2239
> 
> 
> Thanks and regards.
> 
> 
> From: Rohit Yadav 
> Sent: Thursday, July 13, 2017 12:59:02 PM
> To: dev@cloudstack.apache.org
> Subject: [DISCUSS][SECURITY] Feature: Secure CloudStack Communications
> 
> All,
> 
> 
> With upcoming features such as the application service (container service), 
> and existing features such as SAML, they all need some sort of certificate 
> management and the idea with the proposed feature is to build a pluggable 
> certificate authority manager (CA Manager). I would like to kick an initial 
> discussion around how we can secure components of CloudStacks. A CA 
> service/manager that can create/provision/deploy certificates providing both 
> automated and semi-automated ways for deploying/setup of certificates using 
> in-band (ssh, command-answer pattern) and out-of-band (ssh, ansible, chef 
> etc) to CloudStack services (such as systemvm agents, KVM agents, possible 
> webservices running in systemvms, VRs etc).
> 
> 
> While we do have some APIs and mechanisms to secure user/external facing 
> services where we can use custom or failsafe SSL/TLS certificates, it's far 
> from a complete solution. The present communications between CloudStack 
> management server, its peers and agents (served on port 8250) is one way SSL 
> handshaked connection, is not authenticated while may be secure by insecure 
> certificates.
> 
> 
> As a first step, it is proposed to create a general purpose pluggable CA 
> service with a default plugin implementation where CloudStack becomes a 
> Root-CA and can issue self-signed certificates. Such certificates may be 
> consumed by CloudStack agents (CPVM/SSVM/KVM) and other components/services 
> (such as SAML, container services etc). The pluggable CA framework should 
> allow developers to extend the functionality by implementing provider plugins 
> that may work with other CA providers such as LetsEncrypt, an 
> existing/internal CA infrastructure, or other certificate vendors.
> 
> 
> Please see an initial FS and ideas on implementation in the following FS. 
> Looking forward to your feedback.
> 
> 
> FS: 
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Secure+Agent+Communications
> 
> JIRA: https://issues.apache.org/jira/browse/CLOUDSTACK-9993
> 
> 
> Regards.
> 
> rohit.ya...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
> 
> 
> 
> 
> rohit.ya...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
> 
> 
> 
> 
> rohit.ya...@shapeblue.com 
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>   
>  
> 
>

Re: [DISCUSS] Moving to embedded Jetty and fatjar CloudStack

2017-08-23 Thread Rohit Yadav 
All,


Based on last Trilian based test runs, no regressions are seen. We'll aim for 
following changes and afterwards works towards merging the PR;


- An option to provide explicit webapp directory, for custom UI etc. The 
default UI would be bundled in the fat jar, with the UI assets/files also 
copied/packaged at /usr/share/cloudstack-management/webapp (or ui folder) to 
allow for customizations

- Fix packaging, refactor systemd/initd script

- Completely move to java/systemd, instead of using jsvc+systemd for 
systemd-enabled distros, and use init.d scripts for centos6/older distros

- Logging and other improvements to ServerDaemon (the main class that run jetty 
in the fatjar)

- A xml/yml file to configure SSL certificates, options etc.


- Rohit


From: Syed Ahmed 
Sent: Thursday, August 17, 2017 3:00:48 PM
To: dev@cloudstack.apache.org
Cc: Rohit Yadav; Cloudstack Users List
Subject: Re: [DISCUSS] Moving to embedded Jetty and fatjar CloudStack

I have always been a big fan of fat, dependency-free binaries. Good work guys!


rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

On Thu, Aug 10, 2017 at 11:23 AM, Wido den Hollander 
> wrote:

> Op 10 augustus 2017 om 14:09 schreef Rohit Yadav 
> >:
>
>
> All,
>
>
> Distro provided tomcat6/7/* has caused production issues for few users in the 
> past. Due to this, the ACS deployments are inconsistent with the version/jars 
> of tomcat in use. By allowing exploded war to be shipped, can allow admins to 
> sometimes overwrite cloudstack jars causing production issues. I think moving 
> to a CloudStack uber/fat jar will make it easier to deploy CloudStack in 
> environments and write custom init/systemd scripts and fix cloudstack setup 
> databases/management scripts without assuming the distro we're on.
>

Yes, I agree. A flat jar makes things a lot easier.

>
> With this discussion thread, I would like to engage with the community if 
> they've any reservations from moving away from tomcat to embedded jetty + 
> fat/uber jar based packaging. Please share your thoughts and comments.
>
>
> On very high level the packaging will provide the following:
>
> - A ServerDaemon class that can accept custom location of UI (webapp 
> directory), logging, and other environment options, part of the fatjar.
>
> - A config file (xml/yml or otherwise) where you can configure 
> keystore/SSL-certificates, paths (/client), ports, logging etc.
>
> - Default libraries/plugin path at /usr/share/cloudstack-management/lib, UI 
> path at /usr/share/cloudstack-management/webapp
>
> - A default file (available at /etc/default/cloudstack-management or symlink 
> at /etc/sysconfig etc) where you can specific custom variables, java options, 
> classpath etc.
>
> - Refactored init.d/systemd scripts to be commonly used b/w rpm/deb build 
> scripts
>
> - A new/improve logrotate file
>
> - Logging will be handled by log4j (the same xml/config file you normally use)
>
> - Currently we're using jsvc to handle mgmt server process, however we may 
> move to java+systemd completely
>

I suggest that we only use init.d on RHEL 6 and Ubuntu 14.04, but on RHEL 7 and 
Ubuntu 16.04 (any distro that runs with systemd) we should try to avoid using 
jsvc.

That way we don't have to daemonize the MGMT server and keep it attached to 
systemd. Also makes it easy to just print logs to stdout and have journalctl 
take care of them.

>
> Marc-Aurèle (ExoScale) and I have collaborated on this problem and we finally 
> have a PR (not complete) where we can show this actually works, please have a 
> look:

Awesome work!

>
> https://github.com/apache/cloudstack/pull/2226
>
>
> Once the PR is accepted, we can include a topic page in the 4.11/future 
> release notes docs about upgrading in-place and setting up ssl certs etc.
>
>
> Regards.
>
> rohit.ya...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>

Re: [DISCUSS] CloudStack 4.9.3.0 (LTS)

2017-08-23 Thread Rohit Yadav 
All,


Over the past couple of weeks, we've reviewed, tested and merge several PRs. In 
the recent Trillian test run against latest 4.9 branch, against KVM, VMware and 
XenServer all smoke tests passed except for rVR and rVPC related failures which 
are same as the failures in 4.9.2.0. The most common failure in the rVR setup 
is that the MASTER VR does not work properly, however a workaround is to reboot 
it, which causes BACKUP VR to takeover as MASTER and routes, private gateways 
and other issues resolve themselves.


During this effort, we've also tried to stabilize master branch and currently, 
most smoke tests are passing on master for KVM, XenServer, VMware except for 
rVR, rVPC tests, and some intermittent failures seen for some volume, vpn, 
snapshot related tests.


With this, we'll be running component tests and hopefully cut 4.9.3.0 RC1 for 
voting soon.


4.9 smoketest: https://github.com/apache/cloudstack/pull/2217

Master smoketests PR: https://github.com/apache/cloudstack/pull/2225


- Rohit


From: Rohit Yadav 
Sent: Thursday, August 10, 2017 2:34:29 PM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] CloudStack 4.9.3.0 (LTS)

Hi Alireza,


One of the issues mentioned may have been already fixed in 4.9 branch and will 
make its way in 4.9.3.0. For the other issue, a fix/PR does not exist already. 
Given, it's not a blocker/critical issue and freeze is tomorrow, if we don't 
have a PR soon it will likely get fixed in future releases. Thanks.


- Rohit


From: Alireza Eskandari 
Sent: Saturday, August 5, 2017 12:53:41 PM
To: dev@cloudstack.apache.org
Subject: RE: [DISCUSS] CloudStack 4.9.3.0 (LTS)

Hi Rohit
Please consider these bugs in next release:
https://issues.apache.org/jira/browse/CLOUDSTACK-10033
https://issues.apache.org/jira/browse/CLOUDSTACK-9994

rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue

Re: [DISCUSS][SECURITY] Feature: Secure CloudStack Communications

2017-08-23 Thread Rohit Yadav 
All,


No regression is seen in the smoke test run, however, I'll leave the PR open 
for some time to gather further feedback and reviews.


- Rohit


From: Rohit Yadav 
Sent: Friday, August 18, 2017 4:09:30 PM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS][SECURITY] Feature: Secure CloudStack Communications

All,


The feature is ready for your review, please see:

https://github.com/apache/cloudstack/pull/2239


Thanks and regards.


From: Rohit Yadav 
Sent: Thursday, July 13, 2017 12:59:02 PM
To: dev@cloudstack.apache.org
Subject: [DISCUSS][SECURITY] Feature: Secure CloudStack Communications

All,


With upcoming features such as the application service (container service), and 
existing features such as SAML, they all need some sort of certificate 
management and the idea with the proposed feature is to build a pluggable 
certificate authority manager (CA Manager). I would like to kick an initial 
discussion around how we can secure components of CloudStacks. A CA 
service/manager that can create/provision/deploy certificates providing both 
automated and semi-automated ways for deploying/setup of certificates using 
in-band (ssh, command-answer pattern) and out-of-band (ssh, ansible, chef etc) 
to CloudStack services (such as systemvm agents, KVM agents, possible 
webservices running in systemvms, VRs etc).


While we do have some APIs and mechanisms to secure user/external facing 
services where we can use custom or failsafe SSL/TLS certificates, it's far 
from a complete solution. The present communications between CloudStack 
management server, its peers and agents (served on port 8250) is one way SSL 
handshaked connection, is not authenticated while may be secure by insecure 
certificates.


As a first step, it is proposed to create a general purpose pluggable CA 
service with a default plugin implementation where CloudStack becomes a Root-CA 
and can issue self-signed certificates. Such certificates may be consumed by 
CloudStack agents (CPVM/SSVM/KVM) and other components/services (such as SAML, 
container services etc). The pluggable CA framework should allow developers to 
extend the functionality by implementing provider plugins that may work with 
other CA providers such as LetsEncrypt, an existing/internal CA infrastructure, 
or other certificate vendors.


Please see an initial FS and ideas on implementation in the following FS. 
Looking forward to your feedback.


FS: 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Secure+Agent+Communications

JIRA: https://issues.apache.org/jira/browse/CLOUDSTACK-9993


Regards.

rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue