Re: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size
Placement policy only applies to writes and I had thought that gpfs did enough writing to memory "pagepool" to figure out the size before committing the write to pool. I also admit I don't know all of the innards of gpfs. Pehaps being a copy on write type filesystem prevents this for occurring. On Mon, Oct 31, 2016 at 1:29 PM, Chris Scottwrote: > Hi Brian > > This is exactly what I do with a SSD tier on top of 10K and 7.2K tiers. > > HAWC is another recent option that might address Eric's requirement but > needs further consideration of the read requirements you want from the > small files. > > Cheers > Chris > > On 31 October 2016 at 17:23, Brian Marshall wrote: > >> When creating a "fast tier" storage pool in a filesystem is the normal >> case to create a placement policy that places all files in the fast tier >> and migrates out old and large files? >> >> >> Brian Marshall >> >> On Mon, Oct 31, 2016 at 1:20 PM, Jez Tucker >> wrote: >> >>> Hey Bryan >>> >>> There was a previous RFE for path placement from the UG, but Yuri told >>> me this was not techically possible as an inode has no knowledge about the >>> parent dentry. (IIRC).You can see this in effect in the C API. It is >>> possible to work this out at kernel level, but it's so costly that it >>> becomes non-viable at scale / performance. >>> >>> IBMers please chip in and expand if you will. >>> >>> Jez >>> >>> >>> On 31/10/16 17:09, Bryan Banister wrote: >>> >>> The File Placement Policy that you are trying to set cannot use the size >>> of the file to determine the placement of the file in a GPFS Storage Pool. >>> This is because GPFS has no idea what the file size will be when the file >>> is open()’d for writing. >>> >>> >>> >>> Hope that helps! >>> >>> -Bryan >>> >>> >>> >>> PS. I really wish that we could use a path for specifying data placement >>> in a GPFS Pool, and not just the file name, owner, etc. I’ll submit a RFE >>> for this. >>> >>> >>> >>> *From:* gpfsug-discuss-boun...@spectrumscale.org [ >>> mailto:gpfsug-discuss-boun...@spectrumscale.org >>> ] *On Behalf Of *J. Eric >>> Wonderley >>> *Sent:* Monday, October 31, 2016 11:53 AM >>> *To:* gpfsug main discussion list >>> *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger >>> files onto a pool based on size >>> >>> >>> >>> I wanted to do something like this... >>> >>> >>> [root@cl001 ~]# cat /opt/gpfs/home.ply >>> /*Failsafe migration of old small files back to spinning media >>> pool(fc_8T) */ >>> RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) >>> WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' >>> /*Write files larger than 16MB to pool called "fc_8T" */ >>> RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 >>> /*Move anything else to system pool */ >>> RULE 'default' SET POOL 'system' >>> >>> Apparently there is no happiness using FILE_SIZE in a placement policy: >>> [root@cl001 ~]# mmchpolicy home /opt/gpfs/home.ply >>> Error while validating policy `home.ply': rc=22: >>> PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable >>> name in this context. >>> PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE >>> {{{FILE_SIZE}}}>16777216 >>> runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home >>> /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. >>> mmchpolicy: Command failed. Examine previous error messages to determine >>> cause. >>> >>> Can anyone suggest a way to accomplish this using policy? >>> >>> -- >>> >>> Note: This email is for the confidential use of the named addressee(s) >>> only and may contain proprietary, confidential or privileged information. >>> If you are not the intended recipient, you are hereby notified that any >>> review, dissemination or copying of this email is strictly prohibited, and >>> to please notify the sender immediately and destroy this email and any >>> attachments. Email transmission cannot be guaranteed to be secure or >>> error-free. The Company, therefore, does not make any guarantees as to the >>> completeness or accuracy of this email or any attachments. This email is >>> for informational purposes only and does not constitute a recommendation, >>> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >>> or perform any type of transaction of a financial product. >>> >>> >>> ___ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at >>> spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> ___ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >> >> ___ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >>
Re: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size
When creating a "fast tier" storage pool in a filesystem is the normal case to create a placement policy that places all files in the fast tier and migrates out old and large files? Brian Marshall On Mon, Oct 31, 2016 at 1:20 PM, Jez Tuckerwrote: > Hey Bryan > > There was a previous RFE for path placement from the UG, but Yuri told > me this was not techically possible as an inode has no knowledge about the > parent dentry. (IIRC).You can see this in effect in the C API. It is > possible to work this out at kernel level, but it's so costly that it > becomes non-viable at scale / performance. > > IBMers please chip in and expand if you will. > > Jez > > > On 31/10/16 17:09, Bryan Banister wrote: > > The File Placement Policy that you are trying to set cannot use the size > of the file to determine the placement of the file in a GPFS Storage Pool. > This is because GPFS has no idea what the file size will be when the file > is open()’d for writing. > > > > Hope that helps! > > -Bryan > > > > PS. I really wish that we could use a path for specifying data placement > in a GPFS Pool, and not just the file name, owner, etc. I’ll submit a RFE > for this. > > > > *From:* gpfsug-discuss-boun...@spectrumscale.org [mailto:gpfsug-discuss- > boun...@spectrumscale.org ] *On > Behalf Of *J. Eric Wonderley > *Sent:* Monday, October 31, 2016 11:53 AM > *To:* gpfsug main discussion list > *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger files > onto a pool based on size > > > > I wanted to do something like this... > > > [root@cl001 ~]# cat /opt/gpfs/home.ply > /*Failsafe migration of old small files back to spinning media pool(fc_8T) > */ > RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) > WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' > /*Write files larger than 16MB to pool called "fc_8T" */ > RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 > /*Move anything else to system pool */ > RULE 'default' SET POOL 'system' > > Apparently there is no happiness using FILE_SIZE in a placement policy: > [root@cl001 ~]# mmchpolicy home /opt/gpfs/home.ply > Error while validating policy `home.ply': rc=22: > PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable > name in this context. > PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE > {{{FILE_SIZE}}}>16777216 > runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home > /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. > mmchpolicy: Command failed. Examine previous error messages to determine > cause. > > Can anyone suggest a way to accomplish this using policy? > > -- > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at > spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size
Hi Brian This is exactly what I do with a SSD tier on top of 10K and 7.2K tiers. HAWC is another recent option that might address Eric's requirement but needs further consideration of the read requirements you want from the small files. Cheers Chris On 31 October 2016 at 17:23, Brian Marshallwrote: > When creating a "fast tier" storage pool in a filesystem is the normal > case to create a placement policy that places all files in the fast tier > and migrates out old and large files? > > > Brian Marshall > > On Mon, Oct 31, 2016 at 1:20 PM, Jez Tucker wrote: > >> Hey Bryan >> >> There was a previous RFE for path placement from the UG, but Yuri told >> me this was not techically possible as an inode has no knowledge about the >> parent dentry. (IIRC).You can see this in effect in the C API. It is >> possible to work this out at kernel level, but it's so costly that it >> becomes non-viable at scale / performance. >> >> IBMers please chip in and expand if you will. >> >> Jez >> >> >> On 31/10/16 17:09, Bryan Banister wrote: >> >> The File Placement Policy that you are trying to set cannot use the size >> of the file to determine the placement of the file in a GPFS Storage Pool. >> This is because GPFS has no idea what the file size will be when the file >> is open()’d for writing. >> >> >> >> Hope that helps! >> >> -Bryan >> >> >> >> PS. I really wish that we could use a path for specifying data placement >> in a GPFS Pool, and not just the file name, owner, etc. I’ll submit a RFE >> for this. >> >> >> >> *From:* gpfsug-discuss-boun...@spectrumscale.org [ >> mailto:gpfsug-discuss-boun...@spectrumscale.org >> ] *On Behalf Of *J. Eric >> Wonderley >> *Sent:* Monday, October 31, 2016 11:53 AM >> *To:* gpfsug main discussion list >> *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger >> files onto a pool based on size >> >> >> >> I wanted to do something like this... >> >> >> [root@cl001 ~]# cat /opt/gpfs/home.ply >> /*Failsafe migration of old small files back to spinning media >> pool(fc_8T) */ >> RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) >> WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' >> /*Write files larger than 16MB to pool called "fc_8T" */ >> RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 >> /*Move anything else to system pool */ >> RULE 'default' SET POOL 'system' >> >> Apparently there is no happiness using FILE_SIZE in a placement policy: >> [root@cl001 ~]# mmchpolicy home /opt/gpfs/home.ply >> Error while validating policy `home.ply': rc=22: >> PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable >> name in this context. >> PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE >> {{{FILE_SIZE}}}>16777216 >> runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home >> /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. >> mmchpolicy: Command failed. Examine previous error messages to determine >> cause. >> >> Can anyone suggest a way to accomplish this using policy? >> >> -- >> >> Note: This email is for the confidential use of the named addressee(s) >> only and may contain proprietary, confidential or privileged information. >> If you are not the intended recipient, you are hereby notified that any >> review, dissemination or copying of this email is strictly prohibited, and >> to please notify the sender immediately and destroy this email and any >> attachments. Email transmission cannot be guaranteed to be secure or >> error-free. The Company, therefore, does not make any guarantees as to the >> completeness or accuracy of this email or any attachments. This email is >> for informational purposes only and does not constitute a recommendation, >> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >> or perform any type of transaction of a financial product. >> >> >> ___ >> gpfsug-discuss mailing list >> gpfsug-discuss at >> spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ___ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size
Thanks for that suggestion, which is something we already do, but as you say is not as flexible. I think Jez is correct about the overhead being too high to support directory path for data placement, -B -Original Message- From: Edward Wahl [mailto:ew...@osc.edu] Sent: Monday, October 31, 2016 12:26 PM To: Bryan Banister Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size On Mon, 31 Oct 2016 17:09:55 + Bryan Banisterwrote: > -Bryan > > PS. I really wish that we could use a path for specifying data > placement in a GPFS Pool, and not just the file name, owner, etc. > I’ll submit a RFE for this. So... use a fileset with a linked directory and a fileset placement policy to a pool? Might be a bit more rigid for what you really want, and it would be messy, but it would work just fine. -- Ed Wahl Ohio Supercomputer Center 614-292-9302 Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size
On Mon, 31 Oct 2016 17:09:55 + Bryan Banisterwrote: > -Bryan > > PS. I really wish that we could use a path for specifying data > placement in a GPFS Pool, and not just the file name, owner, etc. > I’ll submit a RFE for this. So... use a fileset with a linked directory and a fileset placement policy to a pool? Might be a bit more rigid for what you really want, and it would be messy, but it would work just fine. -- Ed Wahl Ohio Supercomputer Center 614-292-9302 ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size
Hey Bryan There was a previous RFE for path placement from the UG, but Yuri told me this was not techically possible as an inode has no knowledge about the parent dentry. (IIRC).You can see this in effect in the C API. It is possible to work this out at kernel level, but it's so costly that it becomes non-viable at scale / performance. IBMers please chip in and expand if you will. Jez On 31/10/16 17:09, Bryan Banister wrote: The File Placement Policy that you are trying to set cannot use the size of the file to determine the placement of the file in a GPFS Storage Pool. This is because GPFS has no idea what the file size will be when the file is open()’d for writing. Hope that helps! -Bryan PS. I really wish that we could use a path for specifying data placement in a GPFS Pool, and not just the file name, owner, etc. I’ll submit a RFE for this. *From:*gpfsug-discuss-boun...@spectrumscale.org [mailto:gpfsug-discuss-boun...@spectrumscale.org] *On Behalf Of *J. Eric Wonderley *Sent:* Monday, October 31, 2016 11:53 AM *To:* gpfsug main discussion list *Subject:* [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size I wanted to do something like this... [root@cl001 ~]# cat /opt/gpfs/home.ply /*Failsafe migration of old small files back to spinning media pool(fc_8T) */ RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' /*Write files larger than 16MB to pool called "fc_8T" */ RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 /*Move anything else to system pool */ RULE 'default' SET POOL 'system' Apparently there is no happiness using FILE_SIZE in a placement policy: [root@cl001 ~]# mmchpolicy home /opt/gpfs/home.ply Error while validating policy `home.ply': rc=22: PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable name in this context. PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE {{{FILE_SIZE}}}>16777216 runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. mmchpolicy: Command failed. Examine previous error messages to determine cause. Can anyone suggest a way to accomplish this using policy? Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size
The File Placement Policy that you are trying to set cannot use the size of the file to determine the placement of the file in a GPFS Storage Pool. This is because GPFS has no idea what the file size will be when the file is open()’d for writing. Hope that helps! -Bryan PS. I really wish that we could use a path for specifying data placement in a GPFS Pool, and not just the file name, owner, etc. I’ll submit a RFE for this. From: gpfsug-discuss-boun...@spectrumscale.org [mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of J. Eric Wonderley Sent: Monday, October 31, 2016 11:53 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size I wanted to do something like this... [root@cl001 ~]# cat /opt/gpfs/home.ply /*Failsafe migration of old small files back to spinning media pool(fc_8T) */ RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' /*Write files larger than 16MB to pool called "fc_8T" */ RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 /*Move anything else to system pool */ RULE 'default' SET POOL 'system' Apparently there is no happiness using FILE_SIZE in a placement policy: [root@cl001 ~]# mmchpolicy home /opt/gpfs/home.ply Error while validating policy `home.ply': rc=22: PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable name in this context. PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE {{{FILE_SIZE}}}>16777216 runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. mmchpolicy: Command failed. Examine previous error messages to determine cause. Can anyone suggest a way to accomplish this using policy? Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
[gpfsug-discuss] wanted...gpfs policy that places larger files onto a pool based on size
I wanted to do something like this... [root@cl001 ~]# cat /opt/gpfs/home.ply /*Failsafe migration of old small files back to spinning media pool(fc_8T) */ RULE 'theshold' MIGRATE FROM POOL 'system' THRESHOLD(90,70) WEIGHT(ACCESS_TIME) TO POOL 'fc_8T' /*Write files larger than 16MB to pool called "fc_8T" */ RULE 'bigfiles' SET POOL 'fc_8T' WHERE FILE_SIZE>16777216 /*Move anything else to system pool */ RULE 'default' SET POOL 'system' Apparently there is no happiness using FILE_SIZE in a placement policy: [root@cl001 ~]# mmchpolicy home /opt/gpfs/home.ply Error while validating policy `home.ply': rc=22: PCSQLERR: 'FILE_SIZE' is an unsupported or unknown attribute or variable name in this context. PCSQLCTX: at line 4 of 6: RULE 'bigfiles' SET POOL 'fc_8T' WHERE {{{FILE_SIZE}}}>16777216 runRemoteCommand_v2: cl002.cl.arc.internal: tschpolicy /dev/home /var/mmfs/tmp/tspolicyFile.mmchpolicy.113372 -t home.ply failed. mmchpolicy: Command failed. Examine previous error messages to determine cause. Can anyone suggest a way to accomplish this using policy? ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss