RE: [Lustre-discuss] Logical volumes over Lustre storage
If you want, we can do some preliminary testing for you. We're running 1.6beta. lin From: Peter Braam [mailto:[EMAIL PROTECTED] Sent: Tuesday, January 09, 2007 5:45 PM To: Lin Shen (lshen) Cc: Sridharan Ramaswamy (srramasw); lustre-discuss@clusterfs.com; cliff white Subject: Re: [Lustre-discuss] Logical volumes over Lustre storage Nathan - This looks like a feature we should get asap, and a very small one to prepare probably. Sridaran - With this OST file systems (and MDT file systems) can be grown (online) using LVM growth followed by ext3 dynamic resize. You can also grow a Lustre file system by adding OST's. In due course there will be tools to empty an OST with object migration, and there might in due course be an ext4 online shrinker (there is an offline shrinker for ext2 - but it doesn't support extents as far as I know). - Peter - Lin Shen (lshen) wrote: On a related question, this is what Andreas said: On Dec 21, 2006 16:41 -0800, Lin Shen (lshen) wrote: Can a Luster file system be expanded on fly if it's created on a logical volume by adding space to the logical volume? You can add OSTs on the fly in 1.6, but not 1.4. You could also grow the LV dynamically, but there is no way to pass the ioctl to ext3 to do the online resizing. That might at least be possible to implement in 1.6 because there is at least a mountpoint for every device, but it doesn't work quite yet. Does this mean whatever resizing that has been done on the underline LVM, it won't been reflected to the OST? BTW, is it a big effort to implement the missing part, ie., passing ioctl to ext3 for online resizing? lin From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Peter Braam Sent: Tuesday, January 09, 2007 5:18 PM To: Sridharan Ramaswamy (srramasw) Cc: lustre-discuss@clusterfs.com; cliff white Subject: Re: [Lustre-discuss] Logical volumes over Lustre storage Hi You can also put your OST's on LVM's and get some flexibility wit resizing, but probably you had already figured that out! - Peter - Sridharan Ramaswamy (srramasw) wrote: Nathan, Cliff - Thanks for the clarification! Think I understand this more now. Lustre can grow at the granularity of LUN size by adding it as another OST. Not in between. I presume the scenario is same for reduce/decreasing(take an OST offline and add to another MDT). - Sridhar -Original Message- From: cliff white [mailto:[EMAIL PROTECTED] Sent: Tuesday, January 09, 2007 11:21 AM To: Nathaniel Rutman Cc: Sridharan Ramaswamy (srramasw); lustre-discuss@clusterfs.com Subject: Re: [Lustre-discuss] Logical volumes over Lustre storage Nathaniel Rutman wrote: Sridharan Ramaswamy (srramasw) wrote: Is it possible to get traditional LVM like functionality over Lustre based storage? My understanding is Lustre can aggregate storage OSTs to provide a single mount point. But the aggregate storage it provides can't be carved out into different logical volumes, is this correct? For eg, assume each OST is 8GB node1: OST1, OST2, MDT
Re: [Lustre-discuss] Logical volumes over Lustre storage
Sridharan Ramaswamy (srramasw) wrote: Is it possible to get traditional LVM like functionality over Lustre based storage? My understanding is Lustre can aggregate storage OSTs to provide a single mount point. But the aggregate storage it provides can't be carved out into different logical volumes, is this correct? For eg, assume each OST is 8GB node1: OST1, OST2, MDT node2: OST3, OST4 Using *single* Lustre instance is it possible to provide two mount points with 16GB each? And ability to administratively change the storage available to each of those mount points, say instead of 16GB each to 20GB and 12GB. Something similar to lvextend, lvreduce. I presume you can achieve something like the above by creating two Lustre instances(running in different ports) and mount them separately as 16GB filesystem. But that seems too much overhead. They don't have to run in different ports, but yes, you can achieve what you want by making two different Lustre filesystems. That does mean you need two different MDTs. (Similarly, you would have to have two different filesystems on your LVM.) In Lustre 1.6 you can dynamically add OSTs to a filesystem to increase the space available. Appreciate any insights on this area. thanks, Sridhar ___ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Logical volumes over Lustre storage
Nathaniel Rutman wrote: Sridharan Ramaswamy (srramasw) wrote: Is it possible to get traditional LVM like functionality over Lustre based storage? My understanding is Lustre can aggregate storage OSTs to provide a single mount point. But the aggregate storage it provides can't be carved out into different logical volumes, is this correct? For eg, assume each OST is 8GB node1: OST1, OST2, MDT node2: OST3, OST4 Using *single* Lustre instance is it possible to provide two mount points with 16GB each? And ability to administratively change the storage available to each of those mount points, say instead of 16GB each to 20GB and 12GB. Something similar to lvextend, lvreduce. As Nathan mentions below you will be able to dynamically add OSTS in the future. However, you can _not split a single partition between two filesystems (which is possible with LVM) so you would not be able to split the 8GB LUN. You could start with 16GB, add an 8GB LUN and grow to 24GB, (You can do this today, but you have to stop the filesystem to change the config) but you could not add 1/2 the LUN. So a 12GB/20GB split with 8 GB LUNS is not possible, and will not be possible in the future. cliffw I presume you can achieve something like the above by creating two Lustre instances(running in different ports) and mount them separately as 16GB filesystem. But that seems too much overhead. They don't have to run in different ports, but yes, you can achieve what you want by making two different Lustre filesystems. That does mean you need two different MDTs. (Similarly, you would have to have two different filesystems on your LVM.) In Lustre 1.6 you can dynamically add OSTs to a filesystem to increase the space available. Appreciate any insights on this area. thanks, Sridhar ___ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
RE: [Lustre-discuss] Logical volumes over Lustre storage
Nathan, Cliff - Thanks for the clarification! Think I understand this more now. Lustre can grow at the granularity of LUN size by adding it as another OST. Not in between. I presume the scenario is same for reduce/decreasing(take an OST offline and add to another MDT). - Sridhar -Original Message- From: cliff white [mailto:[EMAIL PROTECTED] Sent: Tuesday, January 09, 2007 11:21 AM To: Nathaniel Rutman Cc: Sridharan Ramaswamy (srramasw); lustre-discuss@clusterfs.com Subject: Re: [Lustre-discuss] Logical volumes over Lustre storage Nathaniel Rutman wrote: Sridharan Ramaswamy (srramasw) wrote: Is it possible to get traditional LVM like functionality over Lustre based storage? My understanding is Lustre can aggregate storage OSTs to provide a single mount point. But the aggregate storage it provides can't be carved out into different logical volumes, is this correct? For eg, assume each OST is 8GB node1: OST1, OST2, MDT node2: OST3, OST4 Using *single* Lustre instance is it possible to provide two mount points with 16GB each? And ability to administratively change the storage available to each of those mount points, say instead of 16GB each to 20GB and 12GB. Something similar to lvextend, lvreduce. As Nathan mentions below you will be able to dynamically add OSTS in the future. However, you can _not split a single partition between two filesystems (which is possible with LVM) so you would not be able to split the 8GB LUN. You could start with 16GB, add an 8GB LUN and grow to 24GB, (You can do this today, but you have to stop the filesystem to change the config) but you could not add 1/2 the LUN. So a 12GB/20GB split with 8 GB LUNS is not possible, and will not be possible in the future. cliffw I presume you can achieve something like the above by creating two Lustre instances(running in different ports) and mount them separately as 16GB filesystem. But that seems too much overhead. They don't have to run in different ports, but yes, you can achieve what you want by making two different Lustre filesystems. That does mean you need two different MDTs. (Similarly, you would have to have two different filesystems on your LVM.) In Lustre 1.6 you can dynamically add OSTs to a filesystem to increase the space available. Appreciate any insights on this area. thanks, Sridhar -- -- ___ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Logical volumes over Lustre storage
Hi You can also put your OST's on LVM's and get some flexibility wit resizing, but probably you had already figured that out! - Peter - Sridharan Ramaswamy (srramasw) wrote: Nathan, Cliff - Thanks for the clarification! Think I understand this more now. Lustre can grow at the granularity of LUN size by adding it as another OST. Not in between. I presume the scenario is same for reduce/decreasing(take an OST offline and add to another MDT). - Sridhar -Original Message- From: cliff white [mailto:[EMAIL PROTECTED] Sent: Tuesday, January 09, 2007 11:21 AM To: Nathaniel Rutman Cc: Sridharan Ramaswamy (srramasw); lustre-discuss@clusterfs.com Subject: Re: [Lustre-discuss] Logical volumes over Lustre storage Nathaniel Rutman wrote: Sridharan Ramaswamy (srramasw) wrote: Is it possible to get traditional LVM like functionality over Lustre based storage? My understanding is Lustre can aggregate storage OSTs to provide a single mount point. But the aggregate storage it provides can't be carved out into different logical volumes, is this correct? For eg, assume each OST is 8GB node1: OST1, OST2, MDT node2: OST3, OST4 Using *single* Lustre instance is it possible to provide two mount points with 16GB each? And ability to administratively change the storage available to each of those mount points, say instead of 16GB each to 20GB and 12GB. Something similar to lvextend, lvreduce. As Nathan mentions below you will be able to dynamically add OSTS in the future. However, you can _not split a single partition between two filesystems (which is possible with LVM) so you would not be able to split the 8GB LUN. You could start with 16GB, add an 8GB LUN and grow to 24GB, (You can do this today, but you have to stop the filesystem to change the config) but you could not add 1/2 the LUN. So a 12GB/20GB split with 8 GB LUNS is not possible, and will not be possible in the future. cliffw I presume you can achieve something like the above by creating two Lustre instances(running in different ports) and mount them separately as 16GB filesystem. But that seems too much overhead. They don't have to run in different ports, but yes, you can achieve what you want by making two different Lustre filesystems. That does mean you need two different MDTs. (Similarly, you would have to have two different filesystems on your LVM.) In Lustre 1.6 you can dynamically add OSTs to a filesystem to increase the space available. Appreciate any insights on this area. thanks, Sridhar -- -- ___ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
RE: [Lustre-discuss] Logical volumes over Lustre storage
Hi Peter, Can you please elaborate on how resizing an LVM will cause Lustre to resize? Yeah, I know you can have OST on top of an LVM-based logical-volumn. But my understanding (from a previous mail in this alias) is OST can't reflect the size change of the underlying logical volumn it is based on. In plain words, if I do a lvextend or lvreduce on that logical-volumn, Lustre won't pick the OST size change and hence it won't reflect in the size change in the filesystem that OST is participating in. Is this understanding correct? BTW, you might have already guessed, our interest in Lustre is to use is more like a shared general purpose filesystem. In that domain I believe LVM is quite handy for an administrator to move around storage as need under different volumes. Wondering whether Lustre could still provide, if not exactly the same, but similar capability. thanks, Sridhar From: Peter Braam [mailto:[EMAIL PROTECTED] Sent: Tuesday, January 09, 2007 5:18 PM To: Sridharan Ramaswamy (srramasw) Cc: cliff white; Nathaniel Rutman; lustre-discuss@clusterfs.com Subject: Re: [Lustre-discuss] Logical volumes over Lustre storage Hi You can also put your OST's on LVM's and get some flexibility wit resizing, but probably you had already figured that out! - Peter - Sridharan Ramaswamy (srramasw) wrote: Nathan, Cliff - Thanks for the clarification! Think I understand this more now. Lustre can grow at the granularity of LUN size by adding it as another OST. Not in between. I presume the scenario is same for reduce/decreasing(take an OST offline and add to another MDT). - Sridhar -Original Message- From: cliff white [mailto:[EMAIL PROTECTED] Sent: Tuesday, January 09, 2007 11:21 AM To: Nathaniel Rutman Cc: Sridharan Ramaswamy (srramasw); lustre-discuss@clusterfs.com Subject: Re: [Lustre-discuss] Logical volumes over Lustre storage Nathaniel Rutman wrote: Sridharan Ramaswamy (srramasw) wrote: Is it possible to get traditional LVM like functionality over Lustre based storage? My understanding is Lustre can aggregate storage OSTs to provide a single mount point. But the aggregate storage it provides can't be carved out into different logical volumes, is this correct? For eg, assume each OST is 8GB node1: OST1, OST2, MDT node2: OST3, OST4 Using *single* Lustre instance is it possible to provide two mount points with 16GB each? And ability to administratively change the storage available to each of those mount points, say instead of 16GB each to 20GB and 12GB. Something similar to lvextend, lvreduce. As Nathan mentions below you will be able to dynamically add OSTS in the future. However, you can _not split a single partition between two filesystems (which is possible with LVM) so you would not be able to split the 8GB LUN. You could start with 16GB, add an 8GB LUN and grow to 24GB, (You can do this today, but you have to stop the filesystem to change the config) but you could not add 1/2 the LUN. So a 12GB/20GB split with 8 GB LUNS is not possible, and will not be possible in the future. cliffw I presume you can achieve something like the above by creating two Lustre instances(running in different ports) and mount
Re: [Lustre-discuss] Logical volumes over Lustre storage
Nathan - This looks like a feature we should get asap, and a very small one to prepare probably. Sridaran - With this OST file systems (and MDT file systems) can be grown (online) using LVM growth followed by ext3 dynamic resize. You can also grow a Lustre file system by adding OST's. In due course there will be tools to empty an OST with object migration, and there might in due course be an ext4 online shrinker (there is an offline shrinker for ext2 - but it doesn't support extents as far as I know). - Peter - Lin Shen (lshen) wrote: On a related question, this is what Andreas said: On Dec 21, 2006 16:41 -0800, Lin Shen (lshen) wrote: Can a Luster file system be expanded on fly if it's created on a logical volume by adding space to the logical volume? You can add OSTs on the fly in 1.6, but not 1.4. You could also grow the LV dynamically, but there is no way to pass the ioctl to ext3 to do the online resizing. That might at least be possible to implement in 1.6 because there is at least a mountpoint for every device, but it doesn't work quite yet. Does this mean whatever resizing that has been done on the underline LVM, it won't been reflected to the OST? BTW, is it a big effort to implement the missing part, ie., passing ioctl to ext3 for online resizing? lin *From:* [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] *On Behalf Of *Peter Braam *Sent:* Tuesday, January 09, 2007 5:18 PM *To:* Sridharan Ramaswamy (srramasw) *Cc:* lustre-discuss@clusterfs.com; cliff white *Subject:* Re: [Lustre-discuss] Logical volumes over Lustre storage Hi You can also put your OST's on LVM's and get some flexibility wit resizing, but probably you had already figured that out! - Peter - Sridharan Ramaswamy (srramasw) wrote: Nathan, Cliff - Thanks for the clarification! Think I understand this more now. Lustre can grow at the granularity of LUN size by adding it as another OST. Not in between. I presume the scenario is same for reduce/decreasing(take an OST offline and add to another MDT). - Sridhar -Original Message- From: cliff white [mailto:[EMAIL PROTECTED] Sent: Tuesday, January 09, 2007 11:21 AM To: Nathaniel Rutman Cc: Sridharan Ramaswamy (srramasw); lustre-discuss@clusterfs.com Subject: Re: [Lustre-discuss] Logical volumes over Lustre storage Nathaniel Rutman wrote: Sridharan Ramaswamy (srramasw) wrote: Is it possible to get traditional LVM like functionality over Lustre based storage? My understanding is Lustre can aggregate storage OSTs to provide a single mount point. But the aggregate storage it provides can't be carved out into different logical volumes, is this correct? For eg, assume each OST is 8GB node1: OST1, OST2, MDT node2: OST3, OST4 Using *single* Lustre instance is it possible to provide two mount points with 16GB each? And ability to administratively change the storage available to each of those mount points, say instead of 16GB each to 20GB and 12GB. Something similar to lvextend, lvreduce. As Nathan mentions below you will be able to dynamically add OSTS in the future. However, you can _not split a single partition between two filesystems (which is possible with LVM) so you would not be able to split the 8GB LUN. You could start with 16GB, add an 8GB LUN and grow to 24GB, (You can do this today, but you have to stop the filesystem to change the config) but you could not add 1/2 the LUN. So a 12GB/20GB split with 8 GB LUNS is not possible, and will not be possible in the future. cliffw I presume you can achieve something like the above by creating two Lustre instances(running in different ports) and mount them separately as 16GB filesystem. But that seems too much overhead. They don't have to run in different ports, but yes, you can achieve what you want by making two different Lustre filesystems. That does mean you need two different MDTs. (Similarly, you would have to have two different filesystems on your LVM.) In Lustre 1.6 you can dynamically add OSTs to a filesystem to increase the space available. Appreciate any insights on this area. thanks, Sridhar -- -- ___ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre
RE: [Lustre-discuss] Logical volumes over Lustre storage
On Tue, 2007-01-09 at 17:31 -0800, Sridharan Ramaswamy (srramasw) wrote: Sridharan, Yeah, I know you can have OST on top of an LVM-based logical-volumn. But my understanding (from a previous mail in this alias) is OST can't reflect the size change of the underlying logical volumn it is based on. Right. In plain words, if I do a lvextend or lvreduce on that logical-volumn, Lustre won't pick the OST size change and hence it won't reflect in the size change in the filesystem that OST is participating in. Is this understanding correct? This is correct, because on top of the logical volume is an ldiskfs (ext3 with a few patches) filesystem and it's in that filesystem that the lustre OST does it's work -- stores objects and so forth. So as I am sure you know with a regular ext3 filesystem, simply expanding the LV it's on won't reflect a change in the filesystem's size without first growing the filesystem. This is why an OST doesn't see an LV-only growth as making more space available. That ldiskfs (ext3) filesystem sitting between lustre and the LV needs to grow too. That is what Peter was referring to in his follow up message in this thread about dynamic resizing. Cheers, b. signature.asc Description: This is a digitally signed message part ___ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
RE: [Lustre-discuss] Logical volumes over Lustre storage
Thanks Peter! I think this will be a great compliment to Lustre's existing feature set. - Sridhar From: Peter Braam [mailto:[EMAIL PROTECTED] Sent: Tuesday, January 09, 2007 5:45 PM To: Lin Shen (lshen) Cc: Sridharan Ramaswamy (srramasw); lustre-discuss@clusterfs.com; cliff white Subject: Re: [Lustre-discuss] Logical volumes over Lustre storage Nathan - This looks like a feature we should get asap, and a very small one to prepare probably. Sridaran - With this OST file systems (and MDT file systems) can be grown (online) using LVM growth followed by ext3 dynamic resize. You can also grow a Lustre file system by adding OST's. In due course there will be tools to empty an OST with object migration, and there might in due course be an ext4 online shrinker (there is an offline shrinker for ext2 - but it doesn't support extents as far as I know). - Peter - Lin Shen (lshen) wrote: On a related question, this is what Andreas said: On Dec 21, 2006 16:41 -0800, Lin Shen (lshen) wrote: Can a Luster file system be expanded on fly if it's created on a logical volume by adding space to the logical volume? You can add OSTs on the fly in 1.6, but not 1.4. You could also grow the LV dynamically, but there is no way to pass the ioctl to ext3 to do the online resizing. That might at least be possible to implement in 1.6 because there is at least a mountpoint for every device, but it doesn't work quite yet. Does this mean whatever resizing that has been done on the underline LVM, it won't been reflected to the OST? BTW, is it a big effort to implement the missing part, ie., passing ioctl to ext3 for online resizing? lin From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Peter Braam Sent: Tuesday, January 09, 2007 5:18 PM To: Sridharan Ramaswamy (srramasw) Cc: lustre-discuss@clusterfs.com; cliff white Subject: Re: [Lustre-discuss] Logical volumes over Lustre storage Hi You can also put your OST's on LVM's and get some flexibility wit resizing, but probably you had already figured that out! - Peter - Sridharan Ramaswamy (srramasw) wrote: Nathan, Cliff - Thanks for the clarification! Think I understand this more now. Lustre can grow at the granularity of LUN size by adding it as another OST. Not in between. I presume the scenario is same for reduce/decreasing(take an OST offline and add to another MDT). - Sridhar -Original Message- From: cliff white [mailto:[EMAIL PROTECTED] Sent: Tuesday, January 09, 2007 11:21 AM To: Nathaniel Rutman Cc: Sridharan Ramaswamy (srramasw); lustre-discuss@clusterfs.com Subject: Re: [Lustre-discuss] Logical volumes over Lustre storage Nathaniel Rutman wrote: Sridharan Ramaswamy (srramasw) wrote: Is it possible to get traditional LVM like functionality over Lustre based storage? My understanding is Lustre can aggregate storage OSTs to provide a single mount point. But the aggregate storage it provides can't be carved out into different logical volumes, is this correct? For eg, assume each OST is 8GB node1: OST1
