Re: [Lustre-discuss] Question about quotas.
Perhaps quota wasn't able to turned on for some OSTs, did you see any error
message (in syslog) when starting MDT OSTs.
From: ?
theodoros.stylianos.kondy...@gmail.commailto:theodoros.stylianos.kondy...@gmail.com
Date: Wednesday, January 16, 2013 2:26 AM
To: lustre-discuss@lists.lustre.orgmailto:lustre-discuss@lists.lustre.org
lustre-discuss@lists.lustre.orgmailto:lustre-discuss@lists.lustre.org
Subject: [Lustre-discuss] Question about quotas.
Hello to everyone.
We have a lustre (v1.8.4) test cluster with 2 MDSs, 2OSSs and 4 Clients.
I am experimenting with the quotas but something seems to not work.
In first I did a
cli1# lfs quotacheck /lustre
cli1# lfs quotaon -ug /lustre
cli1# lfs quota /lustre
To get ::
Disk quotas for user root (uid 0):
Filesystem kbytes quota limit grace files quota limit grace
/lustre/jtest1/ [0] 0 0 - [0] 0 0 -
Some errors happened when getting quota info. Some devices may be not working
or deactivated. The data in [] is inaccurate.
Disk quotas for group root (gid 0):
Filesystem kbytes quota limit grace files quota limit grace
/lustre/jtest1/ [0] 0 0 - [0] 0 0 -
Some errors happened when getting quota info. Some devices may be not working
or deactivated. The data in [] is inaccurate.
So I tried ::
mds2# cat /proc/fs/lustre/lquota/jtest1-MDT/quota_type
ug3
oss1# cat /proc/fs/lustre/lquota/jtest1-OST000{1,3}/quota_type
ug3
ug3
oss2# cat /proc/fs/lustre/lquota/jtest1-OST000{0,2}/quota_type
ug3
ug3
Then I tried ::
cli1# lfs quotaon -ugf /lustre
cli1# lfs quota /lustre
user quotas are not enabled.
group quotas are not enabled.
mds2# cat /proc/fs/lustre/lquota/jtest1-MDT/quota_type
3
oss1# cat /proc/fs/lustre/lquota/jtest1-OST000{1,3}/quota_type
3
3
oss2# cat /proc/fs/lustre/lquota/jtest1-OST000{0,2}/quota_type
3
3
And then again ::
cli1# lfs quotaon -ug /lustre
cli1# lfs quota /lustre
Disk quotas for user root (uid 0):
Filesystem kbytes quota limit grace files quota limit grace
/lustre/jtest1/ [0] 0 0 - [0] 0 0
-
Some errors happened when getting quota info. Some devices may be not
working or deactivated. The data in [] is inaccurate.
Disk quotas for group root (gid 0):
Filesystem kbytes quota limit grace files quota limit grace
/lustre/jtest1/ [0] 0 0 - [0] 0 0
-
Some errors happened when getting quota info. Some devices may be not
working or deactivated. The data in [] is inaccurate.
mds2# cat /proc/fs/lustre/lquota/jtest1-MDT/quota_type
ug3
oss1# cat /proc/fs/lustre/lquota/jtest1-OST000{1,3}/quota_type
ug3
ug3
oss2# cat /proc/fs/lustre/lquota/jtest1-OST000{0,2}/quota_type
ug3
ug3
So I would like to make a few questions in case anyone knows.
First of all is this a known issue, or am I doing something wrong here? I am
forcing the quotaon and that reacts as if I did a quotaoff.
Furthermore I would like to ask about this number 3 in the quota_type files in
case anyone knows what does it mean and why is this necessary.
Finally if I am not doing something wrong here, is there a way to fix this ?
Thank you in advance for your time and any replies/guidance/directions.
Stelios.
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Question about quotas.
Hello, 3 is mean that quota disabled. ICan you show us Lustre log (and/or dmesg) just after cli1# lfs quotaon -ugf /lustre Best regards, Artem Blagodarenko On 15.01.2013, at 22:26, Θεόδωρος Στυλιανός Κονδύλης wrote: mds2# cat /proc/fs/lustre/lquota/jtest1-MDT/quota_type 3 ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] is Luster ready for prime time?
Hello, just signed up today, please forgive me if this question has been covered recently. - in a bit of a rush to get an answer on this as we need to make a decision soon, the idea of using luster was thrown into the mix very late in the decision making process. We are looking to procure a new storage solution which will predominately be used for HPC output but will also be used as our main business centric storage for day to day use. Meaning the file system needs to be available 24/7/365.The last time I was involved in considering Luster was about 6 years ago and it was at that time being considered for scratch space for HPC usage only. Our VMs and databases would remain on non-luster storage as we already have that in place and it works well.The luster file system potentially would have everything else. Projects we work on typically take up to 2 years to complete and during that time we would want all assets to remain on the file system. Some of the vendors on our short list include HDS(Blue Arc), Isilon and NetApp.Last week we started bouncing the idea of using Luster around. I'd love to use it if it is considered stable enough to do so. your thoughts and/or comments would be greatly appreciated. thanks for your time. greg ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] is Luster ready for prime time?
Somewhat surprised that no one has responded yet; although it’s likely that the responses would be rather subjective…including mine, of course! Generally I would say that it would be interesting to know more about your datasets and intended workload; however, you mention this is to be used as your day-to-day main business storage…so I imagine those characteristics would greatly vary… mine certainly do; that much is for sure! I don’t really think uptime would be as much an issue here; there are lots of redundancies, recovery mechanisms, and plenty of stable branches to choose from…the question becomes what are the feature-set needs, performance usability for different file types and workloads, and general comfort level with greater complexity and somewhat less resources. That said, I’d personally be a bit wary of using it as a general filesystem for all your needs. I do find it interesting that your short list is a wide range mix of storage and filesystem types; traditional NAS, scale-out NAS, and then some block storage with a parallel filesytem in Lustre. Why no GPFS on the list for comparison? I currently manage, or have used in the past [bluearc], all the storage / filesystems and more from your list. The reason being is that different storage and filesystems components have some things they are good at… while other things they might not be as good at doing. So I diversify by putting different storage/filesystem component pieces in the areas where they excel at best… Regards, Charles From: lustre-discuss-boun...@lists.lustre.org [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of greg whynott Sent: Thursday, January 17, 2013 12:18 PM To: lustre-discuss@lists.lustre.org Subject: [Lustre-discuss] is Luster ready for prime time? Hello, just signed up today, please forgive me if this question has been covered recently. - in a bit of a rush to get an answer on this as we need to make a decision soon, the idea of using luster was thrown into the mix very late in the decision making process. We are looking to procure a new storage solution which will predominately be used for HPC output but will also be used as our main business centric storage for day to day use. Meaning the file system needs to be available 24/7/365. The last time I was involved in considering Luster was about 6 years ago and it was at that time being considered for scratch space for HPC usage only. Our VMs and databases would remain on non-luster storage as we already have that in place and it works well.The luster file system potentially would have everything else. Projects we work on typically take up to 2 years to complete and during that time we would want all assets to remain on the file system. Some of the vendors on our short list include HDS(Blue Arc), Isilon and NetApp. Last week we started bouncing the idea of using Luster around. I'd love to use it if it is considered stable enough to do so. your thoughts and/or comments would be greatly appreciated. thanks for your time. greg ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] is Luster ready for prime time?
Hi Greg, In general like all file systems you're limited to how stable and reliable your hardware platform is. If you're building something yourself then Lustre becomes much more work. This is due to the need to keep up with stability patches as well as addressing issues directly related to your use case and hardware profile. In my opinion lustre is no less stable than any other file system technology, especially when you're talking about 1.8 revisions (which are very stable) however you have many more things which can go wrong, as you're usually talking about many more components which can fail. A correctly architect cluster with proper fail over environment should leave the file system trouble free, unless of course you hit a bug. There are many people on this list (including my self) that run Lustre as a /home file system without issues, again in most cases issues are introduced when you're over taxing your hardware, or you have hardware failure and a poor fail over environment. There are many vendors which can setup a very robust file system for you, however again, remember if you're looking for the cheapest option, you get what you pay for. -cf On 01/17/2013 10:17 AM, greg whynott wrote: Hello, just signed up today, please forgive me if this question has been covered recently. - in a bit of a rush to get an answer on this as we need to make a decision soon, the idea of using luster was thrown into the mix very late in the decision making process. We are looking to procure a new storage solution which will predominately be used for HPC output but will also be used as our main business centric storage for day to day use. Meaning the file system needs to be available 24/7/365. The last time I was involved in considering Luster was about 6 years ago and it was at that time being considered for scratch space for HPC usage only. Our VMs and databases would remain on non-luster storage as we already have that in place and it works well.The luster file system potentially would have everything else. Projects we work on typically take up to 2 years to complete and during that time we would want all assets to remain on the file system. Some of the vendors on our short list include HDS(Blue Arc), Isilon and NetApp.Last week we started bouncing the idea of using Luster around. I'd love to use it if it is considered stable enough to do so. your thoughts and/or comments would be greatly appreciated. thanks for your time. greg ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] is Luster ready for prime time?
Greg, I'm echoing Charles' comments a bit. Specific filesystems are not good at everything. While it is my opinion that Lustre can be very stable, and like Colin stated the underlying hardware and configuration is crucial to that end, the filesystem may not be the best performing at every data access model. Like every other filesystem Lustre has use cases where it excels and others where overhead may be less than optimal. Other filesystems and storage devices also suffer from one size fits most. Many here would likely be biased toward Lustre but many of those people have also used many other options on the market and ended up here. --Jeff -- -- Jeff Johnson Co-Founder Aeon Computing jeff.john...@aeoncomputing.com www.aeoncomputing.com t: 858-412-3810 x101 f: 858-412-3845 m: 619-204-9061 4170 Morena Boulevard, Suite D - San Diego, CA 92117 On 1/17/13 9:17 AM, greg whynott wrote: Hello, just signed up today, please forgive me if this question has been covered recently. - in a bit of a rush to get an answer on this as we need to make a decision soon, the idea of using luster was thrown into the mix very late in the decision making process. We are looking to procure a new storage solution which will predominately be used for HPC output but will also be used as our main business centric storage for day to day use. Meaning the file system needs to be available 24/7/365. The last time I was involved in considering Luster was about 6 years ago and it was at that time being considered for scratch space for HPC usage only. Our VMs and databases would remain on non-luster storage as we already have that in place and it works well.The luster file system potentially would have everything else. Projects we work on typically take up to 2 years to complete and during that time we would want all assets to remain on the file system. Some of the vendors on our short list include HDS(Blue Arc), Isilon and NetApp.Last week we started bouncing the idea of using Luster around. I'd love to use it if it is considered stable enough to do so. your thoughts and/or comments would be greatly appreciated. thanks for your time. greg ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] is Luster ready for prime time?
Hi Charles, I received a few off list challenging email messages along with a few fishing ones, but its all good. its interesting how a post asking a question can make someone appear angry. 8) Our IO profiles from the different segments of our business do vary greatly. The HPC is more or less the typical load you would expect to see, depending on which software is in use for the for the job being ran. We have hundreds of artists and administrative staff who use the file system in a variety of ways. Some examples would include but not limited to: saving out multiple revisions of photoshop documents (typically in the hundreds of megs to +1gig range), video editing (stereoscopic 2k and 4k images(again from 10's 100's to gigs in size) including uncompressed video, excel, word and similar files, thousands of project files (from software such as Maya, Nuke and similar) these also vary largely in size, from 1 to thousands of megs in size. The intention is keep our data bases and VM requirements on the existing file system which is comprised of about 100 10k SAS drives, it works well. We did consider GPFS but that consideration went out the door once I started talking to them and hammering in some numbers into their online calculator. Things got a bit crazy quickly. They have different pricing for the different types and speeds of Intel CPUs. I got the feeling they were trying to squeeze every penny out of customers they could. felt very Brocade-ish and left a bad taste with us. wouldn't of been much of a problem as some other shops I've worked at, but here we do have a finite budget to work within. The NAS vendors could all be considered scale out I suspect. All 3 can scale out the storage and front end. NA C-mode can have up to 24 heads, Blue Arc goes up to 4 or 8 depending on the class, Isilon can go up to 24 nodes or more as well if memory serves me correctly, and they all have a single name space solution in place. They each have their limits, but for our use case they are really subjective. We will not hit the limits of their scalability before we are considering a fork lift refresh. In our view, for what they offer it is perty much a wash for them - any would meet our needs. NetApp still has a silly agg/vol size limit, at least it is up to 90TB now (from 9 in the past(formatted fs use)).. in April it is suppose to go much higher. The block storage idea in the mix - since all our HPC is linux, they all would become luster clients. To provide a gateway into the luster storage for none linux/luster hosts the thinking was a clustered pair of linux boxes running SAMBA/NFS which were also Luster clients.Its just an idea being bounced around at this point. The data serving requirements of the non HPC parts of the business are much less. The video editors most likely would stay on our existing storage solution as that is working out very well for them, but even if we did put them onto the Luster FS, I think they would be fine. based on that, it didn't seem so crazy to consider block access in this method. that said, I think we would be one of the first in ME to do so, pioneers if you will... diversify - we will end up in the same boat for the same reasons. thanks Charles, greg On Thu, Jan 17, 2013 at 2:20 PM, Hammitt, Charles Allen chamm...@email.unc.edu wrote: ** ** Somewhat surprised that no one has responded yet; although it’s likely that the responses would be rather subjective…including mine, of course!** ** ** ** Generally I would say that it would be interesting to know more about your datasets and intended workload; however, you mention this is to be used as your day-to-day main business storage…so I imagine those characteristics would greatly vary… mine certainly do; that much is for sure! ** ** I don’t really think uptime would be as much an issue here; there are lots of redundancies, recovery mechanisms, and plenty of stable branches to choose from…the question becomes what are the feature-set needs, performance usability for different file types and workloads, and general comfort level with greater complexity and somewhat less resources. That said, I’d personally be a bit wary of using it as a general filesystem for *all* your needs. ** ** ** ** I do find it interesting that your short list is a wide range mix of storage and filesystem types; traditional NAS, scale-out NAS, and then some block storage with a parallel filesytem in Lustre. Why no GPFS on the list for comparison? ** ** I currently manage, or have used in the past *[bluearc]*, all the storage / filesystems and more from your list. The reason being is that different storage and filesystems components have some things they are good at… while other things they might not be as good at doing. So I diversify by putting different storage/filesystem component pieces in the areas where they excel at
