Proposed design doc largely LGTM.
I've some additional suggestions and feedback to make requirements & the
first phase implementation more clearer and simpler:
*
+1 on implementing it in a hypervisor and storage agnostic manner.
*
Let's have the FS VMs owned by the caller (account or project), not like a
system-owned appliance. It would then be just like CKS in that sense. This is
because there is nothing special about the feature as in users can't do it,
it's really for (a) users who want the benefit of shared storage but don't want
to setup themselves and (b) orchestrate such a feature via API/SDKs/automation.
Advanced users may not prefer to use it who want too many customisation and
complexities.
*
To keep the first phase simple, let's drop adding support for metrics/usage of
FS VM and any other lifecycle that would need an agent or need for the
management servers to SSH/manage the FS VM at all. Then, the scope can be
limited to:
*
Orchestrate the initial FS VM setup that can be easily done via user-data
(config drive or VR depending on the network, cloud-init can orchestrate NFS
exports), the FS VM's nfs service can also listen on all nics/IPs. This would
make adding the FS capability to work out of the box if somebody want to attach
the FS to other networks later (than the one it was initially created on).
*
Keep it simple: as there is no agent or mgmt server access needed or required;
any change to the FS properties or lifecycle could be done by a FS VM reboot or
recreation, as the FS VM is stateless and a separate data disk holds the file
share storage. For such operations, the UI can clearly mention a warning or
note that such an operation would cause downtime due to reboot/recreate
lifecycle operation of the FS VM.
*
Suggestions for the Lifecycle operations:
*
(*list & update API are given, should support pagination, listing by
name/keyword, by network, account/domain/project etc)
*
Create FS: initial user-data base FS VM setup (during initial setup, disk can
be check/formatted + mounted with fstab rules)
*
Recreate/restart FS: destroy & recreate FS VM, attach data disk before starting
the VM (setup can check and initialise disk if needed; and grow/expand
filesystem if underlying volume was resized).
*
Attach/detach FS (to/from network): simply CloudStack nic/network attach/detach
(worth checking if cloud-init or something in the systemvm template
automatically takes care of nic setup in the FS VM)
*
Expand FS size: this could be simply UI-based proxy to resizing data disk, but
resizing would cause recreating (or rebooting) or the FS VM, for it to grow the
FS (due to lack of agent or SSH access, this may be acceptable in the first
phase)
*
Delete FS: deleting FS with/without expunging the data disk; and for users to
recover a non-expunged FS (similar to VMs)
*
FSM states: FS should have states that correspond to the FS VM running and
state of the underlying data disk
*
Misc: Ensure FS VM is HA enabled, worth also either assuming some default
compute offering or allow caller to specify compute offering for the FS VM.
*
Network support: support all networks except L2 or networks which don't have
userdata & dhcp capabilities
*
Hypervisor & Storage support: agnostic
*FS = File Shares (suggested name)
Regards.
________________________________
From: Alex Mattioli <alex.matti...@shapeblue.com>
Sent: Wednesday, June 19, 2024 15:13
To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
Cc: us...@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: [Proposal] Storage Filesystem as a First Class Feature
+1 on that, keeping it hypervisor agnostic is key.
-----Original Message-----
From: Nux <n...@li.nux.ro>
Sent: Wednesday, June 19, 2024 10:14 AM
To: dev@cloudstack.apache.org
Cc: us...@cloudstack.apache.org
Subject: Re: [Proposal] Storage Filesystem as a First Class Feature
Thanks Piotr,
This is the second time virtio-fs has been mentioned and just researched it a
bit, it looks like something really nice to have in Cloudstack, definitely
something to look at in the future.
Nice as it is though, it has a big drawback, it's KVM-only, so for now we'll
stick to "old school" tech that can be used in an agnostic matter.
You are more than welcome to share thoughts on the other details presented,
perhaps pros/cons on filesystems and other gotchas you may have encountered
yourself.
On 2024-06-19 07:04, Piotr Pisz wrote:
> Hi,
> We considered a similar problem in our company.
> Shared storage is needed between VMs running on different networks.
> NFS/CephFS is ok as long as the VM can see the source.
> The best solution would be to use https://virtio-fs.gitlab.io/ Any FS
> would be used on the host side (e.g. NFS or CephFS) and exported to
> the VM natively (the network problem disappears).
> But you should start by introducing an appropriate mechanism on the CS
> side (similar in operation to Manila Share from Openstack).
> So, the initiative itself is very good.
>
> Overall, CloudStack has been heading in the right direction lately :-)
>
> Best regards,
> Piotr
>
>
> -----Original Message-----
> From: Nux <n...@li.nux.ro>
> Sent: Wednesday, June 19, 2024 12:59 AM
> To: dev@cloudstack.apache.org; Users <us...@cloudstack.apache.org>
> Subject: Re: [Proposal] Storage Filesystem as a First Class Feature
>
> Hi, I'd like to draw the attention to some of the more operational
> aspects of this feature, mainly the storage appliance internals and
> UI.
>
> So long story short, I've discussed with Abhisar and others and we'll
> be deploying a VM based on the Cloudstack Debian systemvm template
> which will export NFS v3/4 for user VMs to consume.
>
> Below are some of the more finer details, please have a read if you
> are interested in this feature and feel free to comment and make
> suggestions.
>
> 1 - The appliance will only have a single export, that export will be
> a single disk (data volume). Keep it simple.
> 2 - GPT partition table and a single partition, filesystem probably
> XFS and/or customisable - something stock Debian supports, simple and
> boring stuff.
> 3 - NFS export should be simple, we can standardise on a path name eg
> /nfs or /fileshare and it will be identical on all appliances.
> 4 - Starting specs: 2 cores, 4 GB RAM - should be OK for a small NFS
> server, the appliance can be upgraded to bigger offerings.
> 5 - Disk offering should be flagged accordingly, the disk offering
> will have a flag/checkbox for "storage appliance" use.
> 6 - This appliance will not be a system VM, it will be a "blackbox",
> but the approach will be similar here to CKS.
> 7 - Security model: by default we export to * (all hosts) into a
> single network - for isolated networks - in SG zones we need to play
> with security groups & a global setting for dumb shared networks
> (without SG) because of security implications - requires further
> discussion.
> 8 - We export with default, best practices NFS options - anything
> against no_root_squash?
> 9 - Explore exporting the file share via multiple protocols - sftp,
> tftp, smb, nfs, http(s)? - The issue here is authentication becomes a
> problem, also user permissions will get messy and possibly conflict
> with no_root_squash, in fact might require an all_squash and
> everything mapped to a single user that will be then also used for all
> those other services.
> Also
> logging will become necessary. Thoughts?
> 10 - UI details, but this will probably show up in the Storage section
> somehow.
> 11 - Display free/used space, create alerts for full disk etc for this
> appliance.
> 12 - Formatting and setting up to be done by an internal agent,
> specifics are sent via the kernel cmd line of the VM, similar to how
> we configure system VMs.
>
> What do you folks think of these points and have I missed anything
> crucial?
>
>
>
> On 2024-06-04 05:04, Abhisar Sinha wrote:
>> Hi,
>>
>> I would like to propose supporting storage filesystem as a
>> first-class feature in Cloudstack.
>> The File Share can be associated with one or more guest networks or
>> vpc tiers and can be used by any VM on the network in a shared
>> manner. It is designed to be resizable and highly available. This
>> feature can later be used as integration endpoints with the CSI
>> driver, go-sdk, Terraform, Ansible and others.
>>
>> The draft functional spec is here :
>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Filesys
> tem+as
> +a+First+Class+Feature
>>
>> Looking forward to your comments and suggestions.
>>
>> Thanks,
>> Abhisar
