Re: [VOTE] CentOS 7 KVM binaries

[Nathan Johnson]

> On Mar 4, 2022, at 9:04 AM, Simon Weller  wrote:
> 
> We've added features that are only supported on newer versions of qemu-kvm 
> packages previously, such as the IOP and bandwidth limits.
> I'd have to go hunting here, but I think it was done by have a check on the 
> qemu-kvm version and if it was later than a certain release, exposing the 
> feature. 
> 
> @Nathan Johnson might be able to jump in here, as he may remember how that 
> was implemented.
> 
> -Si
> 

Yes, this is correct.  There are version checks all over the kvm plugin it 
looks like.  Here is an example of one I added:

https://github.com/apache/cloudstack/blob/main/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtVMDef.java#L1113

Re: Proposal for EFI firmware support on KVM

[Nathan Johnson]

> On Mar 13, 2019, at 10:56 AM, Nicolas Vazquez  
> wrote:
> 
> Hi Nathan,
> 
> I propose a similar approach than taken to enable DPDK on guest VMs for KVM: 
> https://github.com/apache/cloudstack/pull/2839. I think you can follow a 
> similar approach here:
> 
> - Add a property to enable EFI on the KVM hosts 'agent.properties' file
> - Pass VM additional configurations (pieces of XML to include on the VM 
> domain) on VM deployments or stopped VMs. These additional configurations are 
> passed to the hypervisors only if setting 
> 'enable.additional.vm.configuration' is enabled.
> 
> Regarding letting management server know which hosts are EFI enabled, I think 
> you can use host capabilities. When agents connect to management server, they 
> send the list of capabilities. A new capability for EFI can be sent to 
> management server only when the property on agent.properties is set to true. 
> Host tags is a good approach as well but capabilities will make it more 
> robust in case host tags are not created.

I will look into this, thanks for the suggestion.  So I guess the deployment 
planner would then need to be aware of that specific VM detail at that point - 
"efi" in this case?

Thanks!

> 
> 
> Regards,
> 
> Nicolas Vazquez
> 
> 
> From: Sven Vogel 
> Sent: Wednesday, March 13, 2019 11:55 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Proposal for EFI firmware support on KVM
> 
> Hi Nathan,
> 
> Sounds not bad. Yes we think it could be a good idea.
> 
> It point 2) is where can be found the efi=true?
> 
> And Second
> 
> How can we decide which machine will use efi or normal bios? efi=true flag?
> 
> The settings for the agent sound good.
> 
> After that it should be possible to use tianocore ovmf
> 
> https://github.com/tianocore/tianocore.github.io/wiki/OVMF
> 
> Greetings
> 
> Sven
> 
> 
> Von meinem iPhone gesendet
> 
> 
> __
> 
> Sven Vogel
> Teamlead Platform
> 
> EWERK RZ GmbH
> Br?hl 24, D-04109 Leipzig
> P +49 341 42649 - 11
> F +49 341 42649 - 18
> s.vo...@ewerk.com
> www.ewerk.com<http://www.ewerk.com>
> 
> Gesch?ftsf?hrer:
> Dr. Erik Wende, Hendrik Schubert, Frank Richter, Gerhard Hoyer
> Registergericht: Leipzig HRB 17023
> 
> Zertifiziert nach:
> ISO/IEC 27001:2013
> DIN EN ISO 9001:2015
> DIN ISO/IEC 2-1:2011
> 
> EWERK-Blog<https://blog.ewerk.com/> | 
> LinkedIn<https://www.linkedin.com/company/ewerk-group> | 
> Xing<https://www.xing.com/company/ewerk> | 
> Twitter<https://twitter.com/EWERK_Group> | 
> Facebook<https://de-de.facebook.com/EWERK.IT/>
> 
> Ausk?nfte und Angebote per Mail sind freibleibend und unverbindlich.
> 
> Disclaimer Privacy:
> Der Inhalt dieser E-Mail (einschlie?lich etwaiger beigef?gter Dateien) ist 
> vertraulich und nur f?r den Empf?nger bestimmt. Sollten Sie nicht der 
> bestimmungsgem??e Empf?nger sein, ist Ihnen jegliche Offenlegung, 
> Vervielf?ltigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte 
> informieren Sie in diesem Fall unverz?glich den Absender und l?schen Sie die 
> E-Mail (einschlie?lich etwaiger beigef?gter Dateien) von Ihrem System. Vielen 
> Dank.
> 
> The contents of this e-mail (including any attachments) are confidential and 
> may be legally privileged. If you are not the intended recipient of this 
> e-mail, any disclosure, copying, distribution or use of its contents is 
> strictly prohibited, and you should please notify the sender immediately and 
> then delete it (including any attachments) from your system. Thank you.
> 
> Am 13.03.2019 um 15:30 schrieb Nathan Johnson 
> mailto:njohn...@ena.com.invalid>>:
> 
> I've put together an approach for EFI support that we would like to get some 
> feedback on before I create a PR.  Constructive criticism would be 
> appreciated.
> 
> I've added the following properties to be configured in the agent.properties:
> 
> guest.loader.efi - boolean to switch efi on.  This must be true before it 
> will inject any  entries into the domain xml
> guest.loader.image - this would be the path to the bios/efi image
> guest.loader.nvram - this optionally points to an nvram image
> 
> 
> Even when a host is configured so that it can use EFI, it will only actually 
> create a virtual machine when both of the following conditions are met:
> 
> 1) the host has guest.locader.efi set to true in its agent.properties
> 2) the vm has the vm details parameter efi=true
> 
> At present there is no automatic way for the management server to know in 
> advance which hosts have EFI enabled.  I suppose this could be approximated 
> using tags.  It m

Re: Proposal for EFI firmware support on KVM

[Nathan Johnson]

Nathan Johnson
Senior R Engineer
Education Networks of America



> On Mar 14, 2019, at 9:14 AM, Wido den Hollander  wrote:
> 
> 
> 
> On 3/13/19 3:29 PM, Nathan Johnson wrote:
>> I've put together an approach for EFI support that we would like to get some 
>> feedback on before I create a PR.  Constructive criticism would be 
>> appreciated.
>> 
>> I've added the following properties to be configured in the agent.properties:
>> 
>> guest.loader.efi - boolean to switch efi on.  This must be true before it 
>> will inject any  entries into the domain xml
>> guest.loader.image - this would be the path to the bios/efi image
>> guest.loader.nvram - this optionally points to an nvram image
>> 
>> 
>> Even when a host is configured so that it can use EFI, it will only actually 
>> create a virtual machine when both of the following conditions are met:
>> 
>> 1) the host has guest.locader.efi set to true in its agent.properties
> 
> Can't we detect if the host is EFI capable without the need of adding a
> new flag to the agent.properties?

There is no way that I'm aware of.  libvirt expects its EFI configuration to 
come from the domain xml.

> 
> It advertises the capability of EFI to the Mgmt server and only then can
> efi=true Instances be started on that host.
> 
>> 2) the vm has the vm details parameter efi=true
>>> At present there is no automatic way for the management server to know
> in advance which hosts have EFI enabled.  I suppose this could be
> approximated using tags.  It might be nice to make this more automatic,
> and have the resource planner aware of the efi toggle on the VM, but I'm
> not sure how best to implement that or if it's even worth it.
>> 
> 
> As others already mentioned. The Agent/Host capabilities should be
> sufficient?

Technically any host with a sufficiently new version of qemu can support efi, 
but again the domain xml has to be modified so that it points to the EFI 
firmware image location on disk.  There really is no "host" level 
configuration, it's really per-domain.



smime.p7s
Description: S/MIME cryptographic signature

Re: New VP of CloudStack: Paul Angus

[Nathan Johnson]

Congratulations Paul!


Nathan Johnson
Senior R Engineer
Education Networks of America



> On Mar 11, 2019, at 10:16 AM, Tutkowski, Mike  
> wrote:
> 
> Hi everyone,
> 
> As you may know, the role of VP of CloudStack (Chair of the CloudStack PMC) 
> has a one-year term. My term has now come and gone.
> 
> I’m happy to announce that the CloudStack PMC has elected Paul Angus as our 
> new VP of CloudStack.
> 
> As many already know, Paul has been an active member of the CloudStack 
> Community for over six years now. I’ve worked with Paul on and off throughout 
> much of that time and I believe he’ll be a great fit for this role.
> 
> Please join me in welcoming Paul as the new VP of Apache CloudStack!
> 
> Thanks,
> Mike



smime.p7s
Description: S/MIME cryptographic signature

Proposal for EFI firmware support on KVM

[Nathan Johnson]

I've put together an approach for EFI support that we would like to get some 
feedback on before I create a PR.  Constructive criticism would be appreciated.

I've added the following properties to be configured in the agent.properties:

guest.loader.efi - boolean to switch efi on.  This must be true before it will 
inject any  entries into the domain xml
guest.loader.image - this would be the path to the bios/efi image
guest.loader.nvram - this optionally points to an nvram image


Even when a host is configured so that it can use EFI, it will only actually 
create a virtual machine when both of the following conditions are met:

1) the host has guest.locader.efi set to true in its agent.properties
2) the vm has the vm details parameter efi=true

At present there is no automatic way for the management server to know in 
advance which hosts have EFI enabled.  I suppose this could be approximated 
using tags.  It might be nice to make this more automatic, and have the 
resource planner aware of the efi toggle on the VM, but I'm not sure how best 
to implement that or if it's even worth it.

Thanks in advance!


Nathan Johnson
Senior R Engineer
Education Networks of America





smime.p7s
Description: S/MIME cryptographic signature

Welcome Paul Bencko!

[Nathan Johnson]

I would like to introduce a new team member Paul Bencko who will be helping us 
out with some ACS development related tasks.

Welcome Paul!

Nathan Johnson
Senior R Engineer
Education Networks of America





smime.p7s
Description: S/MIME cryptographic signature

Re: questions about 4.11 future

[Nathan Johnson]

> On Jan 4, 2019, at 10:18 AM, Wei ZHOU  wrote:
> 
> Hi Nathan,
> 
> You can use ConfigKey instead of mysql change in cloudstack.

Ahh!  I had no idea these worked this way.  Thanks!  This makes matters much 
easier.

Nathan

> 
> -Wei
> 
> Nathan Johnson  于2019年1月3日周四 下午5:10写道：
> 
>> 
>> 
>>> On Jan 3, 2019, at 9:50 AM, Rafael Weingärtner <
>> rafaelweingart...@gmail.com> wrote:
>>> 
>>> I have a question. Is it a bug or a feature?
>> 
>> That is a very good question, and up for interpretation.  Please feel free
>> to weigh in here:
>> 
>> https://github.com/apache/cloudstack/issues/3096
>> 
>> 
>> 
>>> 
>>> I have seen new features being introduced in this 4.11 (LTS) version,
>> and I
>>> keep asking myself, why do we do that?
>>> According to the LTS document we have, LTS versions should only receive
>>> fixes, and not new features.
>>> 
>>>> The following are the types of changes that are permitted and guarantees
>>>> provided to users:
>>>> 
>>>>  - Defect fixes only.  Enhancements for that expand support for
>>>>  existing plugins (e.g. XenServer 7) and additional JDK/JRE or MySQL
>>>>  versions may be considered if the change is sufficiently isolated and
>> do
>>>>  not introduce significant quality release to the branch.
>>>>  - Database changes will be limited to those required to address defect
>>>>  fixes
>>>>  - Supported JDK/JRE, MySQL, and Linux distribution versions will not
>>>>  be removed throughout the cycle
>>>> 
>>>> Reference: https://cwiki.apache.org/confluence/display/CLOUDSTACK/LTS
>>> 
>>> On Thu, Jan 3, 2019 at 1:44 PM Paul Angus 
>> wrote:
>>> 
>>>> There definitely will be a 4.11.3 release, I would expect it to be
>> around
>>>> end of Q1.
>>>> We shouldn't have a problem handling a new unique entry in both with an
>>>> INSERT IGNORE in the 4.12 upgrade script
>>>> 
>>>> 
>>>> Kind regards,
>>>> 
>>>> Paul Angus
>>>> 
>>>> paul.an...@shapeblue.com
>>>> www.shapeblue.com
>>>> Amadeus House, Floral Street, London  WC2E 9DPUK
>>>> @shapeblue
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -Original Message-
>>>> From: Simon Weller 
>>>> Sent: 03 January 2019 15:23
>>>> To: Rohit Yadav ; dev@cloudstack.apache.org
>>>> Subject: Re: questions about 4.11 future
>>>> 
>>>> Rohit,
>>>> 
>>>> 
>>>> Thoughts on this? We can base it on 4.11 or the pending 4.12 master.
>>>> 
>>>> 
>>>> - Si
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> From: Nathan Johnson 
>>>> Sent: Wednesday, January 2, 2019 12:23 PM
>>>> To: Rohit Yadav
>>>> Cc: dev@cloudstack.apache.org
>>>> Subject: questions about 4.11 future
>>>> 
>>>> First off, is there going to be a 4.11.3 release?
>>>> 
>>>> Assuming so, at what point would it be appropriate to add database
>>>> migrations?  I have a bug fix I'd like to open a PR on, but it will
>> require
>>>> a small database change - namely inserting a record into the
>> configuration
>>>> table.
>>>> 
>>>> Is it appropriate for me to start a new migration path for 4.11.3 to
>>>> facilitate this fix?  Or would it be more appropriate for the release
>>>> manager to do this?
>>>> 
>>>> If you'd like, I have a 4.11.3 database migration started in my branch
>>>> (i.e., added a schema-41120to41130.sql / cleanup , added a
>>>> Upgrade41120to41130.java , and added the Upgrade41120to41130 class to
>> all
>>>> of the paths mentioned in DatabaseUpgradeChecker).  if you'd like me to
>>>> open a PR on effectively an empty 4.11.3 migrations path, and then I
>> could
>>>> make a second PR that just adds the appropriate sql statement(s) for the
>>>> actual bug fix PR.
>>>> 
>>>> Thanks in advance,
>>>> 
>>>> Nathan Johnson
>>>> 
>>> 
>>> 
>>> --
>>> Rafael Weingärtner
>> 
>> 



smime.p7s
Description: S/MIME cryptographic signature

Re: questions about 4.11 future

[Nathan Johnson]

> On Jan 3, 2019, at 9:50 AM, Rafael Weingärtner  
> wrote:
> 
> I have a question. Is it a bug or a feature?

That is a very good question, and up for interpretation.  Please feel free to 
weigh in here:

https://github.com/apache/cloudstack/issues/3096



> 
> I have seen new features being introduced in this 4.11 (LTS) version, and I
> keep asking myself, why do we do that?
> According to the LTS document we have, LTS versions should only receive
> fixes, and not new features.
> 
>> The following are the types of changes that are permitted and guarantees
>> provided to users:
>> 
>>   - Defect fixes only.  Enhancements for that expand support for
>>   existing plugins (e.g. XenServer 7) and additional JDK/JRE or MySQL
>>   versions may be considered if the change is sufficiently isolated and do
>>   not introduce significant quality release to the branch.
>>   - Database changes will be limited to those required to address defect
>>   fixes
>>   - Supported JDK/JRE, MySQL, and Linux distribution versions will not
>>   be removed throughout the cycle
>> 
>> Reference: https://cwiki.apache.org/confluence/display/CLOUDSTACK/LTS
> 
> On Thu, Jan 3, 2019 at 1:44 PM Paul Angus  wrote:
> 
>> There definitely will be a 4.11.3 release, I would expect it to be around
>> end of Q1.
>> We shouldn't have a problem handling a new unique entry in both with an
>> INSERT IGNORE in the 4.12 upgrade script
>> 
>> 
>> Kind regards,
>> 
>> Paul Angus
>> 
>> paul.an...@shapeblue.com
>> www.shapeblue.com
>> Amadeus House, Floral Street, London  WC2E 9DPUK
>> @shapeblue
>> 
>> 
>> 
>> 
>> -Original Message-
>> From: Simon Weller 
>> Sent: 03 January 2019 15:23
>> To: Rohit Yadav ; dev@cloudstack.apache.org
>> Subject: Re: questions about 4.11 future
>> 
>> Rohit,
>> 
>> 
>> Thoughts on this? We can base it on 4.11 or the pending 4.12 master.
>> 
>> 
>> - Si
>> 
>> 
>> 
>> 
>> 
>> From: Nathan Johnson 
>> Sent: Wednesday, January 2, 2019 12:23 PM
>> To: Rohit Yadav
>> Cc: dev@cloudstack.apache.org
>> Subject: questions about 4.11 future
>> 
>> First off, is there going to be a 4.11.3 release?
>> 
>> Assuming so, at what point would it be appropriate to add database
>> migrations?  I have a bug fix I'd like to open a PR on, but it will require
>> a small database change - namely inserting a record into the configuration
>> table.
>> 
>> Is it appropriate for me to start a new migration path for 4.11.3 to
>> facilitate this fix?  Or would it be more appropriate for the release
>> manager to do this?
>> 
>> If you'd like, I have a 4.11.3 database migration started in my branch
>> (i.e., added a schema-41120to41130.sql / cleanup , added a
>> Upgrade41120to41130.java , and added the Upgrade41120to41130 class to all
>> of the paths mentioned in DatabaseUpgradeChecker).  if you'd like me to
>> open a PR on effectively an empty 4.11.3 migrations path, and then I could
>> make a second PR that just adds the appropriate sql statement(s) for the
>> actual bug fix PR.
>> 
>> Thanks in advance,
>> 
>> Nathan Johnson
>> 
> 
> 
> -- 
> Rafael Weingärtner



smime.p7s
Description: S/MIME cryptographic signature

Re: Introduction

[Nathan Johnson]

Welcome!

> On Jan 2, 2019, at 4:53 AM, Abhishek Kumar  
> wrote:
> 
> Hello all!
> 
> 
> This is Abhishek Kumar. I've recently joined ShapeBlue as Software Engineer 
> to work on Cloudstack.
> Looking forward to learn and contribute in the project and community in a 
> meaningful manner.
> 
> 
> Regards,
> 
> 
> Abhishek Kumar
> 
> Software Engineer
> 
> ShapeBlue
> 
> abhishek.ku...@shapeblue.com
> 
> www.shapeblue.com
> 
> abhishek.ku...@shapeblue.com 
> www.shapeblue.com
> Amadeus House, Floral Street, London  WC2E 9DPUK
> @shapeblue
> 
> 
> 



smime.p7s
Description: S/MIME cryptographic signature

questions about 4.11 future

[Nathan Johnson]

First off, is there going to be a 4.11.3 release?

Assuming so, at what point would it be appropriate to add database migrations?  
I have a bug fix I’d like to open a PR on, but it will require a small database 
change - namely inserting a record into the configuration table.

Is it appropriate for me to start a new migration path for 4.11.3 to facilitate 
this fix?  Or would it be more appropriate for the release manager to do this?

If you’d like, I have a 4.11.3 database migration started in my branch (i.e., 
added a schema-41120to41130.sql / cleanup , added a Upgrade41120to41130.java , 
and added the Upgrade41120to41130 class to all of the paths mentioned in 
DatabaseUpgradeChecker).  if you’d like me to open a PR on effectively an empty 
4.11.3 migrations path, and then I could make a second PR that just adds the 
appropriate sql statement(s) for the actual bug fix PR.

Thanks in advance,

Nathan Johnson

smime.p7s
Description: S/MIME cryptographic signature

Re: Snapshots only on Primary Storage feature

[Nathan Johnson]

t;> automatically
>>>>> replicated to secondary storage, so this feature is useful to us.
>>>>> 
>>>>> I don't understand the rational for removing the feature just
>>>>> because
>>> it
>>>>> does not work on VMware.
>>>>> 
>>>>> On Fri, May 18, 2018, 6:27 AM Suresh Kumar Anaparti, <
>>>>> sureshkumar.anapa...@gmail.com> wrote:
>>>>> 
>>>>>> Si,
>>>>>> 
>>>>>> The PR# 1697 with the global setting
>>>>>> *snapshot.backup.rightafter** -
>>>>>> false* doesn't
>>>>>> work for VMware as create snapshot never takes a snapshot in
>>>>>> Primary
>>>>> pool,
>>>>>> it just returns the snapshot uuid. The backup snapshot does the
>>>> complete
>>>>>> job - creates a VM snapshot with the uuid, extracts and exports
>>>>>> the
>>>>> target
>>>>>> volume to secondary. On demand backup snapshot doesn't work as
>>>>>> there
>>> is
>>>>> no
>>>>>> snapshot in primary. Also, there'll be only one entry with
>>>>>> Primary
>>>> store
>>>>>> role in snapshot_store_ref, which is the latest snapshot taken
>>>>>> for
>>> that
>>>>>> volume.
>>>>>> 
>>>>>> -Suresh
>>>>>> 
>>>>>> On Fri, May 18, 2018 at 1:03 AM, Simon Weller
>>> >>>> 
>>>>>> wrote:
>>>>>> 
>>>>>>> The whole point of the original PR was to optionally disable
>>>>>>> this functionality.
>>>>>>> 
>>>>>>> We don't expose views of the backup state to our customers (we
>>>>>>> have
>>>> our
>>>>>>> own customer interfaces) and it's a large waste of space for
>>>>>>> us to
>>> be
>>>>>>> backing up tons of VM images when we have a solid primary
>>>>>>> storage infrastructure that already has lots of resiliency.
>>>>>>> 
>>>>>>> 
>>>>>>> I guess we're going to have to revisit this again before we
>>>>>>> can
>>>>> consider
>>>>>>> rebasing on 4.11.
>>>>>>> 
>>>>>>> 
>>>>>>> From: Suresh Kumar Anaparti 
>>>>>>> Sent: Thursday, May 17, 2018 2:21 PM
>>>>>>> To: dev
>>>>>>> Subject: Re: Snapshots only on Primary Storage feature
>>>>>>> 
>>>>>>> Hi Si,
>>>>>>> 
>>>>>>> No. not possible to disable the backup to secondary. It copies
>>>>>>> the
>>>>> volume
>>>>>>> snapshot to secondary in a background thread using asyncBackup
>>> param
>>>>> (set
>>>>>>> to true) and allows other operations during that time.
>>>>>>> 
>>>>>>> I understand that the backup was on demand when any operations
>>>>>>> are performed on the snapshot. But, backup during that time
>>>>>>> may take considerable time (depending on the snapshot size and
>>>>>>> the network bandwidth), which can result in the job timeout
>>>>>>> and the User may
>>>> assume
>>>>>>> that it is already Backed up based on its state, unless it is
>>>>> documented.
>>>>>>> 
>>>>>>> -Suresh
>>>>>>> 
>>>>>>> On Fri, May 18, 2018 at 12:23 AM, Simon Weller
>>>> >>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Suresh,
>>>>>>>> 
>>>>>>>> 
>>>>>>>> With this new merged  PR, is it possible to disable the
>>>>>>>> backup to secondary completely? I can't tell from the
>>>>>>>> reference spec and
>>>> we're
>>>>>> not
>>>>>>> on
>>>>>>>> a 4.10/4.11 base yet.
>>>>>>>> 
>>>>>>>> For the record, in the instances where a volume or template
>>>>>

Re: New committers: Nathan Johnson and Marc-Aurèle Brothier

[Nathan Johnson]

Thank you for all the kind words!

Nathan

Rafael Weingärtner <rafaelweingart...@gmail.com> wrote:

> The Project Management Committee (PMC) for Apache CloudStack has invited
> Nathan Johnson and Marc-Aurèle Brothier to become committers and we are
> pleased to announce that they have accepted.
>
> They have shown commitment to Apache CloudStack community, contributing
> with PRs in a constant fashion. Moreover, they have proved great abilities
> to interact with the community quite often in our mailing lists and Slack
> channel trying to help people.
>
> Let´s congratulate and welcome Apache CloudStack’s two newest committers.
>
> --
> Rafael Weingärtner

Re: [proposal] allow mac address to be specified for vm and nic creation

[Nathan Johnson]

Remi Bergsma <rberg...@schubergphilis.com> wrote:

> Hi Nathan,
>
> Great feature! I've also been in situations where we had to keep the mac  
> adress the same. Until now hacked the DB to make it happen, so this is  
> way better. Will see if I can test it in the coming days.

Thanks Remi!  Would very much appreciate independent testing!

Nathan

>
> Thanks, Remi
>
> _________
> From: Nathan Johnson  
> <njohn...@ena.com.invalid<mailto:njohn...@ena.com.invalid>>
> Sent: Tuesday, June 13, 2017 3:26 PM
> Subject: [proposal] allow mac address to be specified for vm and nic  
> creation
> To: <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
>
>
> The title pretty much says it all. Currently mac addresses are
> automagically generated based on the guru that is responsible for the
> network type. This would allow that behavior to be overridden by the API
> on deployVirtualMachine and addNicToVirtualMachine . One potential issue
> is that if the specified mac address was in the same range of potentially
> auto-generated mac addresses, there could be a collision, however this
> could be pretty easily mitigated by just testing for a mac already defined
> by that network and asking for the next
> getNextAvailableMacAddressInNetwork. The primary driver for this is to be
> able to import VMs from other hypervisors / environments where the mac
> address of the guest would need to stay the same, for instance if a piece
> of commercial software was tied to the MAC address. I have a working PR
> here:
>
> https://github.com/apache/cloudstack/pull/2143
>
> minus the logic around avoiding collisions where manually specified mac
> addresses for a network in the same range as those generated by the guru,
> and the guru generating a collision sometime later.
>
> Nathan Johnson
> R Engineer
> Education Networks of America

Re: [proposal] allow mac address to be specified for vm and nic creation

[Nathan Johnson]

Ron Wheeler <rwhee...@artifact-software.com> wrote:

> Does the PR include the documentation changes required to use it?

Currently this is API only, and the API annotations have been updated to  
show how it is used.  Automatic documentation generation that is currently  
used to generate API docs would pick this up.

>
> Networking is one of the areas where Cloudstack is hard to understand and  
> difficult to get setup properly.
> Anything that adds another option increases the complexity.
>
> Making Cloudstack more difficult to install in order to circumvent a  
> legal agreement seems like a bad idea.
>

This would do nothing to make cloudstack more difficult to install, and  
that is quite a leap to assume that this is circumventing any legal  
requirement.  Specifying mac addresses is available at the OS level for  
practically all operating systems, it’s allowed by most virtualization  
platforms, this is simply exposing an optional feature.   We happen to have  
a specific use case for it.  This does nothing to increase complexity for  
the operator, if you don’t use this feature then existing behavior is  
preserved.  It does add a marginal amount of complexity to the networking  
codebase, but the added bits do have unit and integration test coverage.

> Ron
>
> On 13/06/2017 9:49 AM, Nathan Johnson wrote:
>> Ron Wheeler <rwhee...@artifact-software.com> wrote:
>>
>>> This would seem to violate the license agreement.
>>> The license is supposed to be tied to specific hardware not a VM.
>>> This sounds like something to be addressed legally rather than  
>>> technically.
>> Possibly, but I don’t think that is so much a cloudstack concern as the
>> users of cloudstack.
>>
>>> It adds more complexity to a part of Cloudstack that already seems to be
>>> a barrier to entry.
>> This feature is actually to address a current barrier to entry for one of
>> our customers.
>>
>> Nathan
>
> -- 
> Ron Wheeler
> President
> Artifact Software Inc
> email: rwhee...@artifact-software.com
> skype: ronaldmwheeler
> phone: 866-970-2435, ext 102

Re: [proposal] allow mac address to be specified for vm and nic creation

[Nathan Johnson]

Ron Wheeler  wrote:

> This would seem to violate the license agreement.
> The license is supposed to be tied to specific hardware not a VM.
> This sounds like something to be addressed legally rather than technically.
>

Possibly, but I don’t think that is so much a cloudstack concern as the  
users of cloudstack.

> It adds more complexity to a part of Cloudstack that already seems to be  
> a barrier to entry.

This feature is actually to address a current barrier to entry for one of  
our customers.

Nathan

[proposal] allow mac address to be specified for vm and nic creation

[Nathan Johnson]

The title pretty much says it all.  Currently mac addresses are  
automagically generated based on the guru that is responsible for the  
network type.  This would allow that behavior to be overridden by the API  
on deployVirtualMachine and addNicToVirtualMachine .  One potential issue  
is that if the specified mac address was in the same range of potentially  
auto-generated mac addresses, there could be a collision, however this  
could be pretty easily mitigated by just testing for a mac already defined  
by that network and asking for the next  
getNextAvailableMacAddressInNetwork.  The primary driver for this is to be  
able to import VMs from other hypervisors / environments where the mac  
address of the guest would need to stay the same, for instance if a piece  
of commercial software was tied to the MAC address.  I have a working PR  
here:

https://github.com/apache/cloudstack/pull/2143

minus the logic around avoiding collisions where manually specified mac  
addresses for a network in the same range as those generated by the guru,  
and the guru generating a collision sometime later.

Nathan Johnson
R Engineer
Education Networks of America

Re: [PROPOSAL] Separate creation and backup operations for a volume snapshot

[Nathan Johnson]

Harika Punna <harika.pu...@accelerite.com> wrote:
>
> Currently, Volume Snapshots in Cloudstack take considerable amount of  
> time to complete as snapshot involves creation on primary and backup on  
> secondary. I would like to introduce an optional parameter in  
> CreateSnapshotCmd API to separate these operations.
>
>
> More details in the FS:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Separate+creation+and+backup+operations+for+a+volume+snapshot
>
>
> Thanks,
> Harika.

Hello Harika.  There was a related discussion around a global configuration  
parameter snapshot.backup.rightafter

See the thread here:
https://issues.apache.org/jira/browse/CLOUDSTACK-4858

And here is the PR where this was merged into 4.9:

https://github.com/apache/cloudstack/pull/1697

This is distinct from what you propose, and I like the idea of being able  
to specify whether or not to back up at snapshot creation time, versus only  
having a global configuration parameter.

Also one remaining issue with the current implementation of snapshots and  
backups is that if the snapshot.backup.rightafter parameter is set to  
false, on doing certain operations with snapshots (create template from  
snapshot iirc, perhaps some others) it will then need to take the backup to  
secondary at that time, and it will be very slow.  I think at some point  
you have to take that hit, so maybe there is no way around this.

But that brings me to a followup point: since the PR was merged above, ACS  
will backup on-demand any snapshots that need to be on secondary that only  
exist on primary. it would be nice to also have an optional cleanup thread  
and an expiration of backups to secondary.  Or maybe API endpoints that  
would allow some external management of backups.

What do you think?


Nathan Johnson
R Engineer
Education Networks of America

Re: proper place to put migrations

[Nathan Johnson]

Wido den Hollander <w...@widodh.nl> wrote:
>
> I think 4.9.3 will be released. You could put it into master I think and  
> right now in 4920to41000.sql
>
> Wido

Thank you.  I have created a PR here:

https://github.com/apache/cloudstack/pull/2078

So another question:  In the past, whenever I would reference a JIRA bug in  
a github PR / commit message, they would get copied back into the JIRA  
ticket.  This no longer seems to be happening.  Is there some new  
requirement to get these mirrored back?

Thanks,

Nathan Johnson
R Engineer
Education Networks of America

proper place to put migrations

[Nathan Johnson]

I have created a JIRA bug here that will require the use of a migration to  
fix.

https://issues.apache.org/jira/browse/CLOUDSTACK-9902

What is the most appropriate branch to fork from to submit a PR in this  
case, and what is the most appropriate migration script to edit (or create  
a new one)?  Is there going to be a new 4.9.x release?  If so, what is the  
next one called?  4.9.2.1 or 4.9.3.0 ?  Or should this go against master  
and just get added to schema-4920to41000.sql?

Thanks in advance

Nathan Johnson
R Engineer
Education Networks of America

Re: Alternative Cloudstack UI for KVM and Basic Zones (with SG)

[Nathan Johnson]

Very nice!

Ivan Kudryavtsev  wrote:

> Hello, Cloudstack community.
>
> We are proud to present our last development effort to you. During the last
> 5 months we spend some time to develop alternative Cloudstack UI for basic
> zones with KVM hypervisor and security groups. This is basically the thing
> we are using in our clouds. During the design of the software we tried to
> fulfill the expectations of our average cloud users and simplify operations
> as much as possible.
>
> The project is OSS and can be found at GitHub with bunch of screenshots and
> deployment guide. It's under active development so, we will ge glad if you
> join and provide us with additional feedback, UX considerations and other
> interesting information.
>
> Project page at GitHub: https://bwsw.github.io/cloudstack-ui/
> Source code: https://github.com/bwsw/cloudstack-ui
>
> Have a good day. Looking forward hearing your feedback.
>
> -- 
> With best regards, Ivan Kudryavtsev
> Bitworks Software, Ltd.
> Cell: +7-923-414-1515
> WWW: http://bw-sw.com/

Re: Modern template hosting

[Nathan Johnson]

Paul Angus  wrote:

> Hi Nathan,
>
> Ideally, if you put the template location in (or use a template defined  
> in)  test_data.py then the actual location can be overridden by anyone  
> testing.
>
> For Trillian, we've copied all of the templates that people have define  
> to a local repo and then replace the URLs in test_data.py to reduce  
> bandwidth use and download times.
>
> Ie:
>
> "bootableIso":
> {
> "displaytext": "Test Bootable ISO",
> "name": "testISO",
> "bootable": True,
> "ispublic": False,
> "url": "{{ marvin_images_location 
> }}/TinyCore-current.iso",
> "ostype": 'Other Linux (64-bit)',
> "mode": 'HTTP_DOWNLOAD'
> },
>


Thanks for the tip, this could work pretty well for us I think.

Re: Introducing Vishwas

[Nathan Johnson]

Vishwas Pathak <vishwas_pat...@accelerite.com> wrote:

> Hello CloudStack team,
>
> My name is Vishwas Pathak and I am working with Accelerite CloudPlatform  
> testing team. Earlier I have worked in storage systems, cloud computing,  
> virtualization and telecom messaging domains. Excited to be part of this  
> community and happy to contribute.

Welcome Vishwas!


Nathan Johnson
R Engineer
Education Networks of America

Modern template hosting

[Nathan Johnson]

So not to re-open a can of worms, but I’m in a situation where I need to  
come up with a Marvin component test that depends on a template based on a  
kernel that’s relatively new, i.e., newer than Centos 5.3 / Ubuntu 10.04 .   
I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but from  
looking at the thread "Migrating CloudStack content from  
download.cloud.com” it looks like there is resistance to using this at  
least for hosting system vm templates over concerns of neutrality.  Would  
this be suitable for a component test?  If not, what is a “blessed”  
template location?

Thanks in advance!

Nathan Johnson
R Engineer



618 Grassmere Park Drive, Suite 12
Nashville, TN 37211
General Office: 615-312-6000

website | blog | support

 

Re: Adding VirtIO SCSI to KVM hypervisors

[Nathan Johnson]

Sergey Levitskiy <sergey.levits...@autodesk.com> wrote:

> Actually, minor correction. When adding to VM/templates the name of the  
> detail is rootDiskController for Root controller and dataDiskController  
> for additional disks.
> Also, if you want to make changes on a global scale the changes need to  
> go to vm_template_details and user_vm_details tables respectively.

Thanks!  Very helpful

>
> On 2/21/17, 8:03 PM, "Sergey Levitskiy" <sergey.levits...@autodesk.com>  
> wrote:
>
> Here it is the logic.
> 1. Default value is taken from a global configuration 
> vmware.root.disk.controller 
> 2. To override add the same config to template or VM (starting from 4.10 
> UI allows adding advanced settings to templates and/or VMs). If added to a 
> template all VMs deployed from it will inherit this value. If added to VM and 
> then template is created it will also inherits all advanced settings.
>
>
>
>
> On 2/21/17, 7:06 PM, "Nathan Johnson" <njohn...@ena.com> wrote:
>
> Sergey Levitskiy <sergey.levits...@autodesk.com> wrote:
>
>> On vmware rootdiskcontroller is passed over to the hypervisor in VM start
>> command. I know for the fact that the following rootdiskcontroller option
>> specified in template/vm details work fine:
>> ide
>> scsi
>> lsilogic
>> lsilogic1068
>>
>> In general, any scsi controller option that vmware recognizes should work.
>>
>> Thanks,
>> Sergey
>
> Thanks Sergey!  So do you happen to know where on the management 
> server
> side the determination is made as to which rootDiskController 
> parameter to
> pass?
>
>
>
>
>> On 2/21/17, 6:13 PM, "Nathan Johnson" <njohn...@ena.com> wrote:
>>
>> Wido den Hollander <w...@widodh.nl> wrote:
>>
>>>> Op 25 januari 2017 om 4:44 schreef Simon Weller <swel...@ena.com>:
>>>>
>>>>
>>>> Maybe this is a good opportunity to discuss modernizing the OS
>>>> selections so that drivers (and other features) could be selectable per
>>>> OS.
>>>
>>> That seems like a good idea. If you select Ubuntu 16.04 or CentOS 7.3
>>> then for example it will give you a VirtIO SCSI disk on KVM, anything
>>> previous to that will get VirtIO-blk.
>>
>> So one thing I noticed, there is a possibility of a rootDiskController
>> parameter passed to the Start Command.  So this means that the Management
>> server could control whether to use scsi or virtio, assuming I’m reading
>> this correctly, and we shouldn’t necessarily have to rely on the os type
>> name inside the agent code.  From a quick glance at the vmware code, it
>> looks like maybe they already use this parameter?  It would be great if
>> someone familiar with the vmware code could chime in here.
>>
>> Thanks,
>>
>> Nathan
>>
>>
>>
>>> Wido
>>>
>>>> Thoughts?
>>>>
>>>>
>>>> 
>>>> From: Syed Ahmed <sah...@cloudops.com>
>>>> Sent: Tuesday, January 24, 2017 10:46 AM
>>>> To: dev@cloudstack.apache.org
>>>> Cc: Simon Weller
>>>> Subject: Re: Adding VirtIO SCSI to KVM hypervisors
>>>>
>>>> To maintain backward compatibility we would have to add a config option
>>>> here unfortunately. I do like the idea however. We can make the default
>>>> VirtIO ISCSI and keep the VirtIO-blk as an alternative for existing
>>>> installations.
>>>>
>>>> On Mon, Jan 23, 2017 at 8:05 AM, Wido den Hollander
>>>> <w...@widodh.nl<mailto:w...@widodh.nl>> wrote:
>>>>
>>>>> Op 21 januari 2017 om 23:50 schreef Wido den Hollander
>>>>> <w...@widodh.nl<mailto:w...@widodh.nl>>:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Op 21 jan. 2017 om 22:59 heeft Syed Ahmed
>>>>>> <sah...@cloudops.com<mailto:sah...@cloudops.com>> het volgende
>>>>>> geschreven:
>>>>>>
>>>>>> Exposing this via an API would be tricky but it can definitely be
>>>>>> added as
>>>>>> a cluster-wide or a global setting in my opinion. By enabling that,
>>>>>> all the
>>>>>> instances would be using VirtIO SCSI. Is there a reason you'd want  
>>>>>> some
>>>>>&

Re: Adding VirtIO SCSI to KVM hypervisors

[Nathan Johnson]

Sergey Levitskiy <sergey.levits...@autodesk.com> wrote:

> Here it is the logic.
> 1. Default value is taken from a global configuration  
> vmware.root.disk.controller   
> 2. To override add the same config to template or VM (starting from 4.10  
> UI allows adding advanced settings to templates and/or VMs). If added to  
> a template all VMs deployed from it will inherit this value. If added to  
> VM and then template is created it will also inherits all advanced  
> settings.
>

Excellent, thanks.  Do you happen to know where this is stored in the  
database?

Thanks again!

>
>
>
> On 2/21/17, 7:06 PM, "Nathan Johnson" <njohn...@ena.com> wrote:
>
> Sergey Levitskiy <sergey.levits...@autodesk.com> wrote:
>
>> On vmware rootdiskcontroller is passed over to the hypervisor in VM start
>> command. I know for the fact that the following rootdiskcontroller option
>> specified in template/vm details work fine:
>> ide
>> scsi
>> lsilogic
>> lsilogic1068
>>
>> In general, any scsi controller option that vmware recognizes should work.
>>
>> Thanks,
>> Sergey
>
> Thanks Sergey!  So do you happen to know where on the management server
> side the determination is made as to which rootDiskController parameter to
> pass?
>
>
>
>
>> On 2/21/17, 6:13 PM, "Nathan Johnson" <njohn...@ena.com> wrote:
>>
>> Wido den Hollander <w...@widodh.nl> wrote:
>>
>>>> Op 25 januari 2017 om 4:44 schreef Simon Weller <swel...@ena.com>:
>>>>
>>>>
>>>> Maybe this is a good opportunity to discuss modernizing the OS
>>>> selections so that drivers (and other features) could be selectable per
>>>> OS.
>>>
>>> That seems like a good idea. If you select Ubuntu 16.04 or CentOS 7.3
>>> then for example it will give you a VirtIO SCSI disk on KVM, anything
>>> previous to that will get VirtIO-blk.
>>
>> So one thing I noticed, there is a possibility of a rootDiskController
>> parameter passed to the Start Command.  So this means that the Management
>> server could control whether to use scsi or virtio, assuming I’m reading
>> this correctly, and we shouldn’t necessarily have to rely on the os type
>> name inside the agent code.  From a quick glance at the vmware code, it
>> looks like maybe they already use this parameter?  It would be great if
>> someone familiar with the vmware code could chime in here.
>>
>> Thanks,
>>
>> Nathan
>>
>>
>>
>>> Wido
>>>
>>>> Thoughts?
>>>>
>>>>
>>>> 
>>>> From: Syed Ahmed <sah...@cloudops.com>
>>>> Sent: Tuesday, January 24, 2017 10:46 AM
>>>> To: dev@cloudstack.apache.org
>>>> Cc: Simon Weller
>>>> Subject: Re: Adding VirtIO SCSI to KVM hypervisors
>>>>
>>>> To maintain backward compatibility we would have to add a config option
>>>> here unfortunately. I do like the idea however. We can make the default
>>>> VirtIO ISCSI and keep the VirtIO-blk as an alternative for existing
>>>> installations.
>>>>
>>>> On Mon, Jan 23, 2017 at 8:05 AM, Wido den Hollander
>>>> <w...@widodh.nl<mailto:w...@widodh.nl>> wrote:
>>>>
>>>>> Op 21 januari 2017 om 23:50 schreef Wido den Hollander
>>>>> <w...@widodh.nl<mailto:w...@widodh.nl>>:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Op 21 jan. 2017 om 22:59 heeft Syed Ahmed
>>>>>> <sah...@cloudops.com<mailto:sah...@cloudops.com>> het volgende
>>>>>> geschreven:
>>>>>>
>>>>>> Exposing this via an API would be tricky but it can definitely be
>>>>>> added as
>>>>>> a cluster-wide or a global setting in my opinion. By enabling that,
>>>>>> all the
>>>>>> instances would be using VirtIO SCSI. Is there a reason you'd want  
>>>>>> some
>>>>>> instances to use VirtIIO and others to use VirtIO SCSI?
>>>>>
>>>>> Even a global setting would be a bit of work and hacky as well.
>>>>>
>>>>> I do not see any reason to keep VirtIO, it os just that devices will be
>>>>> named sdX instead of vdX in the guest.
>>>>
>>>> To add, t

Re: Adding VirtIO SCSI to KVM hypervisors

[Nathan Johnson]

Sergey Levitskiy <sergey.levits...@autodesk.com> wrote:

> On vmware rootdiskcontroller is passed over to the hypervisor in VM start  
> command. I know for the fact that the following rootdiskcontroller option  
> specified in template/vm details work fine:
> ide
> scsi
> lsilogic
> lsilogic1068
>
> In general, any scsi controller option that vmware recognizes should work.
>
> Thanks,
> Sergey

Thanks Sergey!  So do you happen to know where on the management server  
side the determination is made as to which rootDiskController parameter to  
pass?




>
>
> On 2/21/17, 6:13 PM, "Nathan Johnson" <njohn...@ena.com> wrote:
>
> Wido den Hollander <w...@widodh.nl> wrote:
>
>>> Op 25 januari 2017 om 4:44 schreef Simon Weller <swel...@ena.com>:
>>>
>>>
>>> Maybe this is a good opportunity to discuss modernizing the OS
>>> selections so that drivers (and other features) could be selectable per
>>> OS.
>>
>> That seems like a good idea. If you select Ubuntu 16.04 or CentOS 7.3
>> then for example it will give you a VirtIO SCSI disk on KVM, anything
>> previous to that will get VirtIO-blk.
>
> So one thing I noticed, there is a possibility of a rootDiskController
> parameter passed to the Start Command.  So this means that the Management
> server could control whether to use scsi or virtio, assuming I’m reading
> this correctly, and we shouldn’t necessarily have to rely on the os type
> name inside the agent code.  From a quick glance at the vmware code, it
> looks like maybe they already use this parameter?  It would be great if
> someone familiar with the vmware code could chime in here.
>
> Thanks,
>
> Nathan
>
>
>
>> Wido
>>
>>> Thoughts?
>>>
>>>
>>> 
>>> From: Syed Ahmed <sah...@cloudops.com>
>>> Sent: Tuesday, January 24, 2017 10:46 AM
>>> To: dev@cloudstack.apache.org
>>> Cc: Simon Weller
>>> Subject: Re: Adding VirtIO SCSI to KVM hypervisors
>>>
>>> To maintain backward compatibility we would have to add a config option
>>> here unfortunately. I do like the idea however. We can make the default
>>> VirtIO ISCSI and keep the VirtIO-blk as an alternative for existing
>>> installations.
>>>
>>> On Mon, Jan 23, 2017 at 8:05 AM, Wido den Hollander
>>> <w...@widodh.nl<mailto:w...@widodh.nl>> wrote:
>>>
>>>> Op 21 januari 2017 om 23:50 schreef Wido den Hollander
>>>> <w...@widodh.nl<mailto:w...@widodh.nl>>:
>>>>
>>>>
>>>>
>>>>
>>>>> Op 21 jan. 2017 om 22:59 heeft Syed Ahmed
>>>>> <sah...@cloudops.com<mailto:sah...@cloudops.com>> het volgende
>>>>> geschreven:
>>>>>
>>>>> Exposing this via an API would be tricky but it can definitely be
>>>>> added as
>>>>> a cluster-wide or a global setting in my opinion. By enabling that,
>>>>> all the
>>>>> instances would be using VirtIO SCSI. Is there a reason you'd want some
>>>>> instances to use VirtIIO and others to use VirtIO SCSI?
>>>>
>>>> Even a global setting would be a bit of work and hacky as well.
>>>>
>>>> I do not see any reason to keep VirtIO, it os just that devices will be
>>>> named sdX instead of vdX in the guest.
>>>
>>> To add, the Qemu wiki [0] says:
>>>
>>> "A virtio storage interface for efficient I/O that overcomes virtio-blk
>>> limitations and supports advanced SCSI hardware."
>>>
>>> At OpenStack [1] they also say:
>>>
>>> "It has been designed to replace virtio-blk, increase it's performance
>>> and improve scalability."
>>>
>>> So it seems that VirtIO is there to be removed. I'd say switch to VirtIO
>>> SCSI at version 5.X? :)
>>>
>>> Wido
>>>
>>> [0]: http://wiki.qemu.org/Features/VirtioSCSI
>>> [1]: https://wiki.openstack.org/wiki/LibvirtVirtioScsi
>>>
>>>> That might break existing Instances when not using labels or UUIDs in
>>>> the Instance when mounting.
>>>>
>>>> Wido
>>>>
>>>>>> On Sat, Jan 21, 2017 at 4:22 PM, Simon Weller
>>>>>> <swel...@ena.com<mailto:swel...@ena.com>> wrote:
>>>>>>
>>>>>> For the record, we've been looking into this as we

Re: Adding VirtIO SCSI to KVM hypervisors

[Nathan Johnson]

hmed  
>>>>>> <sah...@cloudops.com<mailto:sah...@cloudops.com>>:
>>>>>>
>>>>>>
>>>>>> Wido,
>>>>>>
>>>>>> Were you thinking of adding this as a global setting? I can see why it
>>>>> will
>>>>>> be useful. I'm happy to review any ideas you might have around this.
>>>>>
>>>>> Well, not really. We don't have any structure for this in place right  
>>>>> now
>>>>> to define what type of driver/disk we present to a guest.
>>>>>
>>>>> See my answer below.
>>>>>
>>>>>> Thanks,
>>>>>> -Syed
>>>>>> On Sat, Jan 21, 2017 at 04:46 Laszlo Hornyak  
>>>>>> <laszlo.horn...@gmail.com<mailto:laszlo.horn...@gmail.com>>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Wido,
>>>>>>>
>>>>>>> If I understand correctly from the documentation and your examples,
>>>>> virtio
>>>>>>> provides virtio interface to the guest while virtio-scsi provides  
>>>>>>> scsi
>>>>>>> interface, therefore an IaaS service should not replace it without  
>>>>>>> user
>>>>>>> request / approval. It would be probably better to let the user set
>>>>> what
>>>>>>> kind of IO interface the VM needs.
>>>>>
>>>>> You'd say, but we already do those. Some Operating Systems get a IDE  
>>>>> disk,
>>>>> others a SCSI disk and when Linux guest support it according to our
>>>>> database we use VirtIO.
>>>>>
>>>>> CloudStack has no way of telling how to present a volume to a guest. I
>>>>> think it would be a bit to much to just make that configurable. That  
>>>>> would
>>>>> mean extra database entries, API calls. A bit overkill imho in this  
>>>>> case.
>>>>>
>>>>> VirtIO SCSI is supported by all Linux distributions for a very long  
>>>>> time.
>>>>>
>>>>> Wido
>>>>>
>>>>>>> Best regards,
>>>>>>> Laszlo
>>>>>>>
>>>>>>> On Fri, Jan 20, 2017 at 10:21 PM, Wido den Hollander  
>>>>>>> <w...@widodh.nl<mailto:w...@widodh.nl>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> VirtIO SCSI [0] has been supported a while now by Linux and all
>>>>> kernels,
>>>>>>>> but inside CloudStack we are not using it. There is a issue for this
>>>>> [1].
>>>>>>>> It would bring more (theoretical) performance to VMs, but one of the
>>>>>>>> motivators (for me) is that we can support TRIM/DISCARD [2].
>>>>>>>>
>>>>>>>> This would allow for RBD images on Ceph to shrink, but it can also
>>>>> give
>>>>>>>> back free space on QCOW2 images if quests run fstrim. Something all
>>>>>>> modern
>>>>>>>> distributions all do weekly in a CRON.
>>>>>>>>
>>>>>>>> Now, it is simple to swap VirtIO for VirtIO SCSI. This would however
>>>>> mean
>>>>>>>> that disks inside VMs are then called /dev/sdX instead of /dev/vdX.
>>>>>>>>
>>>>>>>> For GRUB and such this is no problems. This usually work on UUIDs
>>>>> and/or
>>>>>>>> labels, but for static mounts on /dev/vdb1 for example things break.
>>>>>>>>
>>>>>>>> We currently don't have any configuration method on how we want to
>>>>>>> present
>>>>>>>> a disk to a guest, so when attaching a volume we can't say that we
>>>>> want
>>>>>>> to
>>>>>>>> use a different driver. If we think that a Operating System supports
>>>>>>> VirtIO
>>>>>>>> we use that driver in KVM.
>>>>>>>>
>>>>>>>> Any suggestion on how to add VirtIO SCSI support?
>>>>>>>>
>>>>>>>> Wido
>>>>>>>>
>>>>>>>>
>>>>>>>> [0]: http://wiki.qemu.org/Features/VirtioSCSI
>>>>>>>> [1]: https://issues.apache.org/jira/browse/CLOUDSTACK-8239
>>>>>>>> [2]: https://issues.apache.org/jira/browse/CLOUDSTACK-8104
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> EOF


Nathan Johnson
R Engineer



618 Grassmere Park Drive, Suite 12
Nashville, TN 37211
General Office: 615-312-6000

website | blog | support

 

Jira permissions

[Nathan Johnson]

Is there any way I could be added to the list of people who can be assigned  
bugs in Jira?  My username is nathanejohnson .

Thanks in advance,
Nathan

Re: stopVirtualMachine command ignores the forced parameter

[Nathan Johnson]

Jeff Hair  wrote:

> Hi,
>
> I'm looking at the documentation and then the code for stopVirtualMachine.
> The forced parameter is passed down into the VM manager, where it seems to
> be ignored. This means that cleanupEvenIfFailed during VM stop will always
> be false, despite what the API command says. Going back to commit a4f4c986
> in 2013, we can see that the forced parameter was used. During subsequent
> refactoring, this seems to have vanished.



Jeff, after digging in I agree with you, this does look to be ignored.  So
I guess the question is: does cleanupEvenIfFailed mean the same thing as
the “forced" parameter?  If so, we could create a new overloaded
VMEntityManager / VMEntityManagerImpl stopvirtualmachine method with a bool
parameter, and then have that pass off to a new UserVMManagerImpl stop
method with a bool tacked on, and then it could pass this bool into
advanceStop.  Kind of messy, but overall wouldn’t be that much code to
change.

Thoughts?

Re: hidden configuration items revisited

[Nathan Johnson]

Erik Weber  wrote:

> I believe Kishan suggested that you could change those Hidden config items
> to Secure (an existing category), as Secure items are returned with the
> listConfiguration API.

This is chicken and egg.  I need the unencrypted values so I can decrypt  
other payloads.  I need the keys plaintext, one way or another.

Re: hidden configuration items

[Nathan Johnson]

Anshul Gangwar  wrote:
>
> ms ——authenticate—> CPVM ——for VNC console—>Hypervisor
>   ^
>   | gets images from CPVM
>   web  browser
>
> Which of the above components you want to keep and which to remove?

Currently when you hit the management server and pass it a vmid, you get a  
response that is a html payload with a link in an iframe to the console  
proxy VM, along with a parameter passed on the get string that constitutes  
an encrypted JSON payload.  This encrypted JSON payload includes all of the  
information that would be needed to connect via VNC.  We want to be able to  
make the request from our middleware, intercept and decrypt this JSON  
payload and be be able to use an alternative web based VNC client.

>
> Also you can look into other implementations of Console proxy which are  
> rarely used to get more info.
>

Could you point me to one?  I would be very interested to look at an  
alternative.

Thank you,
Nathan

Re: hidden configuration items revisited

[Nathan Johnson]

Kishan Kavala <kishan.kav...@accelerite.com> wrote:

> Nathan,
>   You can use "Secure" category instead of "Hidden". Config items with 
> "Secure" category are encrypted and also included in listConfigurations API 
> response.

The data that I need (specifically security.encryption.iv and  
security.encryption.key) are already marked “Hidden".  These are the keys  
that are used to encrypt the response that I need to decrypt in the  
middleware.  The configuration item I’m proposing to add is a boolean, so  
that doesn’t make sense to be “Secure” either.  Unless I’m misunderstanding?

Thanks for the reply

Nathan


>
> ~kishan
>
> -Original Message-
> From: Nathan Johnson [mailto:njohn...@ena.com]
> Sent: 08 May 2016 22:01
> To: dev@cloudstack.apache.org
> Subject: hidden configuration items revisited
>
> I would like to get some feedback for a proposed addition of a feature  
> that would allow “Hidden” configuration items to be returned from the  
> listConfigurations endpoint.
>
> 1) There will be a new optional parameter for listConfigurations called  
> showhidden .  Defaults to false.  Existing behavior is preserved unless  
> showhidden is set to true.
>
> 2) There is a now configuration item, com.cloud.allowshowhidden , which  
> defaults to false.  This must be set to true in order for showhidden to  
> be allowed.  If showhidden=true is passed and  
> com.cloud.allowshowhidden=false, an InvalidParameterValueException is  
> thrown.
>
> So the web UI would still hide hidden configuration items regardless of  
> the state of com.cloud.allowshowhidden since it will not be passing  
> showhidden=true.  The main value of this would be from API  
> implementations / middleware, which is what our front-end talks to  
> instead of directly to cloudstack management server.
>
> Obviously there is an explicit reason hidden configuration items are not  
> displayed via the API at present.  The Hidden configuration items contain  
> some very sensitive data, such as private keys etc.  I would like to  
> submit a pull request that would make sense to everyone and still be  
> secure by default and not open up pandora’s box so to speak.  I have this  
> working in our lab, but I wanted to get a bit of feedback before  
> submitting a PR.
>
> So several questions:
>
> 1) Would it make sense for com.cloud.allowshowhidden to be a “Hidden”
> configuration item?  The up side of this is that you could not toggle  
> this value from the API.  Marking it hidden means that a rogue root admin  
> api key holder could not grant themselves more access.  The down side is  
> that I’m not sure how to easily change this value outside of manually  
> going into the database and changing it, and one should hope that root  
> admin api key holders are well trusted.  Currently I have this  
> implemented as an “Advanced” configuration item.
>
> 2) I picked com.cloud.allowshowhidden out of my hat.  Is there a more  
> appropriate name that I should use?
>
>
>
>
> DISCLAIMER
> ==
> This e-mail may contain privileged and confidential information which is  
> the property of Accelerite, a Persistent Systems business. It is intended  
> only for the use of the individual or entity to which it is addressed. If  
> you are not the intended recipient, you are not authorized to read,  
> retain, copy, print, distribute or use this message. If you have received  
> this communication in error, please notify the sender and delete all  
> copies of this message. Accelerite, a Persistent Systems business does  
> not accept any liability for virus infected mails.

hidden configuration items revisited

[Nathan Johnson]

I would like to get some feedback for a proposed addition of a feature that  
would allow “Hidden” configuration items to be returned from the  
listConfigurations endpoint.

1) There will be a new optional parameter for listConfigurations called  
showhidden .  Defaults to false.  Existing behavior is preserved unless  
showhidden is set to true.

2) There is a now configuration item, com.cloud.allowshowhidden , which  
defaults to false.  This must be set to true in order for showhidden to be  
allowed.  If showhidden=true is passed and com.cloud.allowshowhidden=false,  
an InvalidParameterValueException is thrown.

So the web UI would still hide hidden configuration items regardless of the  
state of com.cloud.allowshowhidden since it will not be passing  
showhidden=true.  The main value of this would be from API implementations  
/ middleware, which is what our front-end talks to instead of directly to  
cloudstack management server.

Obviously there is an explicit reason hidden configuration items are not  
displayed via the API at present.  The Hidden configuration items contain  
some very sensitive data, such as private keys etc.  I would like to submit  
a pull request that would make sense to everyone and still be secure by  
default and not open up pandora’s box so to speak.  I have this working in  
our lab, but I wanted to get a bit of feedback before submitting a PR.

So several questions:

1) Would it make sense for com.cloud.allowshowhidden to be a “Hidden”  
configuration item?  The up side of this is that you could not toggle this  
value from the API.  Marking it hidden means that a rogue root admin api  
key holder could not grant themselves more access.  The down side is that  
I’m not sure how to easily change this value outside of manually going into  
the database and changing it, and one should hope that root admin api key  
holders are well trusted.  Currently I have this implemented as an  
“Advanced” configuration item.

2) I picked com.cloud.allowshowhidden out of my hat.  Is there a more  
appropriate name that I should use?

hidden configuration items

[Nathan Johnson]

If you hit the /client/console endpoint with a vmid, it sends you back some  
data that contains a link to a the console proxy VM and passes an encrypted  
json payload that has the user, password and port for a vnc connection.   
Normally this is meant to load in an iframe.  We want to decrypt this  
response to direct a VNC client to the appropriate host / port / user /  
pass and bypass use of the console proxy VM.  The key and iv appear to be  
stored in the configuration table under the names security.encryption.key  
and security.encryption.iv, but as they are hidden we cannot get these  
credentials via the listConfigurations endpoint as-is.  So my question is:

What would be the most appropriate way to open up the possibility of  
showing “hidden” configuration items via this API to our middleware?  Some  
sort of entry in a config file somewhere?  An entry in the configuration  
table itself?  Or is there some other way to get this information I’m  
looking for?

Thanks in advance.

Nathan

Jenkins issue

[Nathan Johnson]

I am aware that there have been some transient issues with Jenkins, but I
am repeatedly seeing this error and I suspect this might be something
different:

Unpacking 
https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.3.3/apache-maven-3.3.3-bin.zip
 to 
/home/jenkins/jenkins-slave/tools/hudson.tasks.Maven_MavenInstallation/maven-3.3.3
 on jenkins-test-971

[cloudstack-pr-analysis] $ 
/home/jenkins/jenkins-slave/tools/hudson.tasks.Maven_MavenInstallation/maven-3.3.3/bin/mvn
 clean --projects=org.apache.cloudstack:cloudstack 
org.apache.rat:apache-rat-plugin:0.10:check

[ERROR] Error executing Maven.

[ERROR] 1 problem was encountered while building the effective settings

[FATAL] Non-parseable settings /home/jenkins/.m2/settings.xml: unexpected 
character in markup % (position: START_TAG seen ...\n 
<%... @14:18)  @ /home/jenkins/.m2/settings.xml, line 14, column 18

I keep getting the same error.  I noticed a comment from rafaelweingartner on
this PR:

https://github.com/apache/cloudstack/pull/1466#issuecomment-214861439

Does anyone have any thoughts on this?

Thank you,
Nathan Johnson