Re: [gpfsug-discuss] Handling bad file names in policies?
We have both: WILDCARDSARELITERAL yes QUOTESARELITERAL yes Set. And use --noquote for mmbackup, the backup runs, but creates a file: /filesystem/mmbackup.unsupported.CLIENTNAME Which contains a list of files that are not backed up due to \n in the filename. So it doesn't break backup, but they don't get backed up either. I believe this is because the TSM client can't back the file up rather than mmbackup no longer allowing them. I had an RFE at some point to get dsmc changed ... but it got closed WONTFIX. Simon On 09/10/2021, 10:09, "gpfsug-discuss-boun...@spectrumscale.org on behalf of Jonathan Buzzard" wrote: On 08/10/2021 19:14, Wahl, Edward wrote: > This goes back as far as I can recall to <=GPFS 3.5 days. And no, I > cannot recall what version of TSM-EE that was. But newline has been > the only stopping point, for what seems like forever. Having filed > many an mmbackup bug, I don't recall ever crashing on filenames. > (tons of OTHER reasons, but not character set) We even generate an > error report from this and email users to fix it. We accept basically > almost everything else, and I have to say, we see some really crazy > things sometimes. I think my current favorite is the full windows > paths as a filename. (eg: > "Y:\Temp\temp\290\work\0\Material_ERTi-5.in" ) > I will have to do a test but I am sure newlines have worked just fine in the past. At the very least they have not stopped an entire backup from working when using dsmc incr. Now mmbackup that's a different kettle of fish. If you have not seen mmbackup fail entirely because of a random "special" character you simply have not been using it long enough :-) For the longest of times I would simply not go anywhere near it because it was not fit for purpose. > > Current IBM documentation doesn't go backwards past 4.2 but it says: > > "For IBM Spectrum Scale™ file systems with special characters > frequently used in the names of files or directories, backup failures > might occur. Known special characters that require special handling > include: *, ?, ", ’, carriage return, and the new line character. > > In such cases, enable the Tivoli Storage Manager client options > WILDCARDSARELITERAL and QUOTESARELITERAL on all nodes that are used > in backup activities and make sure that the mmbackup option --noquote > is used when invoking mmbackup." > > So maybe we could handle newlines somehow. But my lazy searches > didn't show what TSM doesn't accept. > We strongly advise our users (our GPFS file system is for an HPC system) in training not to use "special" characters. That is followed with a warning that if they do then we don't make any promises to backup their files :-) From time to time I run a dsmc incr in a screen and capture the output to a log file and then look at the list of failed files and prompt users to "fix" them. Though sometimes I just "fix" them myself if the correction is going to be obvious and then email them to tell them what has happened. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Handling bad file names in policies?
On 08/10/2021 19:14, Wahl, Edward wrote: This goes back as far as I can recall to <=GPFS 3.5 days. And no, I cannot recall what version of TSM-EE that was. But newline has been the only stopping point, for what seems like forever. Having filed many an mmbackup bug, I don't recall ever crashing on filenames. (tons of OTHER reasons, but not character set) We even generate an error report from this and email users to fix it. We accept basically almost everything else, and I have to say, we see some really crazy things sometimes. I think my current favorite is the full windows paths as a filename. (eg: "Y:\Temp\temp\290\work\0\Material_ERTi-5.in" ) I will have to do a test but I am sure newlines have worked just fine in the past. At the very least they have not stopped an entire backup from working when using dsmc incr. Now mmbackup that's a different kettle of fish. If you have not seen mmbackup fail entirely because of a random "special" character you simply have not been using it long enough :-) For the longest of times I would simply not go anywhere near it because it was not fit for purpose. Current IBM documentation doesn't go backwards past 4.2 but it says: "For IBM Spectrum Scale™ file systems with special characters frequently used in the names of files or directories, backup failures might occur. Known special characters that require special handling include: *, ?, ", ’, carriage return, and the new line character. In such cases, enable the Tivoli Storage Manager client options WILDCARDSARELITERAL and QUOTESARELITERAL on all nodes that are used in backup activities and make sure that the mmbackup option --noquote is used when invoking mmbackup." So maybe we could handle newlines somehow. But my lazy searches didn't show what TSM doesn't accept. We strongly advise our users (our GPFS file system is for an HPC system) in training not to use "special" characters. That is followed with a warning that if they do then we don't make any promises to backup their files :-) From time to time I run a dsmc incr in a screen and capture the output to a log file and then look at the list of failed files and prompt users to "fix" them. Though sometimes I just "fix" them myself if the correction is going to be obvious and then email them to tell them what has happened. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Handling bad file names in policies?
Well How about: define(DISPLAY_NEWLINE,[CASE WHEN ($1) *HAS NEWLINE* THEN *REPLACE NEWLINE WITH ALTERNATE CHARACTER* ELSE varchar(1) END]) Define your show to have the DISPLAY_NEWLINE in place of the file name? Sorry I don't know offhand how to do the find newline and replace newline sql string code, I don't have gpfs at home sadly. On Fri, Oct 8, 2021, 1:42 PM Wahl, Edward wrote: > Sadly the ESCAPE only works for EXTERNAL LISTs, correct? Not sure that > I can easily modify an EXERNAL LIST to do what I want, which is a LIST > policy using MISC_ATTRIBUTES and find all files without X, etc. > > And using mmlsattr on hundreds of millions of files will take until the > next millennium, so I really would like to stick with the policy engine. > Perhaps > I can do some RULE 1 feeds RULE 2 type thing? > > > > Sort of thing I’m looking at: > > > > define( immut, MISC_ATTRIBUTES LIKE '%X%') > > RULE 'listimmut' LIST 'not-immut' WHERE NOT (exclude_list) and NOT (immut) > > > > > > Ed Wahl > > OSC > > > > > > *From:* gpfsug-discuss-boun...@spectrumscale.org < > gpfsug-discuss-boun...@spectrumscale.org> *On Behalf Of *Olaf Weiser > *Sent:* Tuesday, October 5, 2021 2:10 AM > *To:* gpfsug-discuss@spectrumscale.org > *Cc:* gpfsug-discuss@spectrumscale.org > *Subject:* Re: [gpfsug-discuss] Handling bad file names in policies? > > > > Hi Ed, > > > > not a ready to run for "everything".. but just to remind, there is an > ESCAPE statement > > by this you can > > > > cat policy2 > RULE EXTERNAL LIST 'allfiles' EXEC '/var/mmfs/etc/list.exe' ESCAPE '%/#' > > > > and turn a file name into smth , what a policy can use > > > > I haven't used it for a while , but here is an example from a while ago .. > ;-) > > > > [root@c25m4n03 stupid_files]# ll > total 0 > -rw-r--r-- 1 root root 21 Mar 22 03:44 dämlicher filename > -rw-r--r-- 1 root root 2 Mar 22 03:59 üöä???ßß spacefilen > [root@c25m4n03 stupid_files]# > > > > > > policy: > > 101378 247907919 0 -- > /gpfs/fpofs/files/stupid_files/d%C3%A4mlicher%20filename > 101381 1945364096 0 -- > /gpfs/fpofs/files/stupid_files/%C3%BC%C3%BC%C3%BC%C3%B6%C3%B6%C3%A4%C3%A4%3F%3F%3F%C3%9F%C3%9F%20spacefilename > [I]2013-03-22@13:12:58.687 Policy execution. 2 files dispatched. > > > > > verify with policy (ESCAPE '%/ä ') > > 101378 247907919 0 -- /gpfs/fpofs/files/stupid_files/dämlicher filename > [...] > > > > > > hope this helps.. > > cheers > > > > > > > > > > - Ursprüngliche Nachricht - > Von: "Jonathan Buzzard" > Gesendet von: gpfsug-discuss-boun...@spectrumscale.org > An: gpfsug-discuss@spectrumscale.org > CC: > Betreff: [EXTERNAL] Re: [gpfsug-discuss] Handling bad file names in > policies? > Datum: Di, 5. Okt 2021 01:29 > > > On 04/10/2021 23:23, Wahl, Edward wrote: > > > I know I've run into this before way back, but my notes on how I solved > > this aren't getting the job done in Scale 5.0.5.8 and my notes are from > > 3.5. > > Anyone know a way to get a LIST policy to properly feed bad filenames > > into the output or an external script? > > > > When I say bad I mean things like control characters, spaces, etc. Not > > concerned about the dreaded 'newline' as we force users to fix those or > > the files do not get backed up in Tivoli. > > > > Since when? Last time I checked which was admittedly circa 2008, TSM > would backup files with newlines in them no problem. mmbackup on the > other hand in that time frame would simply die and backup nothing if > there was a single file on the file system with a newline in it. > > I would take a look at the mmbackup scripts which can handle such stuff > (least ways in >4.2) which would also suggest dsmc can handle it. > > As an aside I now think I know how you end up with newlines in file > names. Basically you cut and paste the file name complete with newlines > (most likely at the end) into a text field when saving the file. > Personally I think any program should baulk at that point but what do I > know. > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > <https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!KGKeukY!gbBLWYl7S7BX4mw1st0Uqn0jAON438v_xU5im5y1VOf3admLYLebW4C0k2nP$> > > > > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Handling bad file names in policies?
This is an interesting idea, but not at all what I was working towards, and is getting me off track. (and I'm known to get distracted and explore interesting Rabbit Holes, red herrings, et al) I've next to no issues with the filenames in day to day operations. On the positive side, this is a one off. What I need is a LIST policy, and the return leaves off the entire filename. Ed Wahl From: gpfsug-discuss-boun...@spectrumscale.org on behalf of Alec Sent: Friday, October 8, 2021 3:36 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Handling bad file names in policies? Why not just configure a file placement policy using a non existent pool or a bad encryption key to prevent files with non-printables characters from even being created in the first place. Alec On Fri, Oct 8, 2021, 11:49 AM Wahl, Edward mailto:ew...@osc.edu>> wrote: This goes back as far as I can recall to <=GPFS 3.5 days. And no, I cannot recall what version of TSM-EE that was. But newline has been the only stopping point, for what seems like forever. Having filed many an mmbackup bug, I don't recall ever crashing on filenames. (tons of OTHER reasons, but not character set) We even generate an error report from this and email users to fix it. We accept basically almost everything else, and I have to say, we see some really crazy things sometimes. I think my current favorite is the full windows paths as a filename. (eg: "Y:\Temp\temp\290\work\0\Material_ERTi-5.in" ) Current IBM documentation doesn't go backwards past 4.2 but it says: "For IBM Spectrum Scale™ file systems with special characters frequently used in the names of files or directories, backup failures might occur. Known special characters that require special handling include: *, ?, ", ’, carriage return, and the new line character. In such cases, enable the Tivoli Storage Manager client options WILDCARDSARELITERAL and QUOTESARELITERAL on all nodes that are used in backup activities and make sure that the mmbackup option --noquote is used when invoking mmbackup." So maybe we could handle newlines somehow. But my lazy searches didn't show what TSM doesn't accept. Ed Wahl OSC -Original Message- From: gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org> mailto:gpfsug-discuss-boun...@spectrumscale.org>> On Behalf Of Jonathan Buzzard Sent: Monday, October 4, 2021 7:29 PM To: gpfsug-discuss@spectrumscale.org<mailto:gpfsug-discuss@spectrumscale.org> Subject: Re: [gpfsug-discuss] Handling bad file names in policies? On 04/10/2021 23:23, Wahl, Edward wrote: > I know I've run into this before way back, but my notes on how I > solved this aren't getting the job done in Scale 5.0.5.8 and my notes > are from 3.5. > Anyone know a way to get a LIST policy to properly feed bad filenames > into the output or an external script? > > When I say bad I mean things like control characters, spaces, etc. > Not concerned about the dreaded 'newline' as we force users to fix > those or the files do not get backed up in Tivoli. > Since when? Last time I checked which was admittedly circa 2008, TSM would backup files with newlines in them no problem. mmbackup on the other hand in that time frame would simply die and backup nothing if there was a single file on the file system with a newline in it. I would take a look at the mmbackup scripts which can handle such stuff (least ways in >4.2) which would also suggest dsmc can handle it. As an aside I now think I know how you end up with newlines in file names. Basically you cut and paste the file name complete with newlines (most likely at the end) into a text field when saving the file. Personally I think any program should baulk at that point but what do I know. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<https://urldefense.com/v3/__http://spectrumscale.org__;!!KGKeukY!jPEGEXlh4N27v0ev7VeN2w8CsqZiWAWqEtQpQ7eHaetmvPuD0-JVJrZx0hAA$> https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!KGKeukY!nVH69Xr88S0X5DmO8QbaI7eozd9pDvmtMN40tZU8vWuduEF4J01ZTfnypvOy$ ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<https://urldefense.com/v3/__http://spectrumscale.org__;!!KGKeukY!jPEGEXlh4N27v0ev7VeN2w8CsqZiWAWqEtQpQ7eHaetmvPuD0-JVJrZx0hAA$> http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!KGKeukY!jPEGEXlh4N27v0ev7VeN2w8CsqZiWAWqEtQpQ7eHaetmvPuD0-JVJuNfee8K$> __
Re: [gpfsug-discuss] Handling bad file names in policies?
Sadly the ESCAPE only works for EXTERNAL LISTs, correct? Not sure that I can easily modify an EXERNAL LIST to do what I want, which is a LIST policy using MISC_ATTRIBUTES and find all files without X, etc. And using mmlsattr on hundreds of millions of files will take until the next millennium, so I really would like to stick with the policy engine. Perhaps I can do some RULE 1 feeds RULE 2 type thing? Sort of thing I’m looking at: define( immut, MISC_ATTRIBUTES LIKE '%X%') RULE 'listimmut' LIST 'not-immut' WHERE NOT (exclude_list) and NOT (immut) Ed Wahl OSC From: gpfsug-discuss-boun...@spectrumscale.org On Behalf Of Olaf Weiser Sent: Tuesday, October 5, 2021 2:10 AM To: gpfsug-discuss@spectrumscale.org Cc: gpfsug-discuss@spectrumscale.org Subject: Re: [gpfsug-discuss] Handling bad file names in policies? Hi Ed, not a ready to run for "everything".. but just to remind, there is an ESCAPE statement by this you can cat policy2 RULE EXTERNAL LIST 'allfiles' EXEC '/var/mmfs/etc/list.exe' ESCAPE '%/#' and turn a file name into smth , what a policy can use I haven't used it for a while , but here is an example from a while ago .. ;-) [root@c25m4n03 stupid_files]# ll total 0 -rw-r--r-- 1 root root 21 Mar 22 03:44 dämlicher filename -rw-r--r-- 1 root root 2 Mar 22 03:59 üöä???ßß spacefilen [root@c25m4n03 stupid_files]# policy: 101378 247907919 0 -- /gpfs/fpofs/files/stupid_files/d%C3%A4mlicher%20filename 101381 1945364096 0 -- /gpfs/fpofs/files/stupid_files/%C3%BC%C3%BC%C3%BC%C3%B6%C3%B6%C3%A4%C3%A4%3F%3F%3F%C3%9F%C3%9F%20spacefilename [I]2013-03-22@13:12:58.687<mailto:2013-03-22@13:12:58.687> Policy execution. 2 files dispatched. verify with policy (ESCAPE '%/ä ') 101378 247907919 0 -- /gpfs/fpofs/files/stupid_files/dämlicher filename [...] hope this helps.. cheers - Ursprüngliche Nachricht - Von: "Jonathan Buzzard" mailto:jonathan.buzz...@strath.ac.uk>> Gesendet von: gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org> An: gpfsug-discuss@spectrumscale.org<mailto:gpfsug-discuss@spectrumscale.org> CC: Betreff: [EXTERNAL] Re: [gpfsug-discuss] Handling bad file names in policies? Datum: Di, 5. Okt 2021 01:29 On 04/10/2021 23:23, Wahl, Edward wrote: > I know I've run into this before way back, but my notes on how I solved > this aren't getting the job done in Scale 5.0.5.8 and my notes are from > 3.5. > Anyone know a way to get a LIST policy to properly feed bad filenames > into the output or an external script? > > When I say bad I mean things like control characters, spaces, etc. Not > concerned about the dreaded 'newline' as we force users to fix those or > the files do not get backed up in Tivoli. > Since when? Last time I checked which was admittedly circa 2008, TSM would backup files with newlines in them no problem. mmbackup on the other hand in that time frame would simply die and backup nothing if there was a single file on the file system with a newline in it. I would take a look at the mmbackup scripts which can handle such stuff (least ways in >4.2) which would also suggest dsmc can handle it. As an aside I now think I know how you end up with newlines in file names. Basically you cut and paste the file name complete with newlines (most likely at the end) into a text field when saving the file. Personally I think any program should baulk at that point but what do I know. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!KGKeukY!gbBLWYl7S7BX4mw1st0Uqn0jAON438v_xU5im5y1VOf3admLYLebW4C0k2nP$> ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Handling bad file names in policies?
Why not just configure a file placement policy using a non existent pool or a bad encryption key to prevent files with non-printables characters from even being created in the first place. Alec On Fri, Oct 8, 2021, 11:49 AM Wahl, Edward wrote: > This goes back as far as I can recall to <=GPFS 3.5 days. And no, I cannot > recall what version of TSM-EE that was. But newline has been the only > stopping point, for what seems like forever. > Having filed many an mmbackup bug, I don't recall ever crashing on > filenames. (tons of OTHER reasons, but not character set) We even > generate an error report from this and email users to fix it. > We accept basically almost everything else, and I have to say, we see some > really crazy things sometimes. I think my current favorite is the full > windows paths as a filename. > (eg: "Y:\Temp\temp\290\work\0\Material_ERTi-5.in" ) > > > Current IBM documentation doesn't go backwards past 4.2 but it says: > > "For IBM Spectrum Scale™ file systems with special characters frequently > used in the names of files or directories, backup failures might occur. > Known special characters that require special handling include: *, ?, ", ’, > carriage return, and the new line character. > > In such cases, enable the Tivoli Storage Manager client options > WILDCARDSARELITERAL and QUOTESARELITERAL on all nodes that are used in > backup activities and make sure that the mmbackup option --noquote is used > when invoking mmbackup." > > So maybe we could handle newlines somehow. But my lazy searches didn't > show what TSM doesn't accept. > > Ed Wahl > OSC > > -Original Message- > From: gpfsug-discuss-boun...@spectrumscale.org < > gpfsug-discuss-boun...@spectrumscale.org> On Behalf Of Jonathan Buzzard > Sent: Monday, October 4, 2021 7:29 PM > To: gpfsug-discuss@spectrumscale.org > Subject: Re: [gpfsug-discuss] Handling bad file names in policies? > > On 04/10/2021 23:23, Wahl, Edward wrote: > > > I know I've run into this before way back, but my notes on how I > > solved this aren't getting the job done in Scale 5.0.5.8 and my notes > > are from 3.5. > > Anyone know a way to get a LIST policy to properly feed bad filenames > > into the output or an external script? > > > > When I say bad I mean things like control characters, spaces, etc. > > Not concerned about the dreaded 'newline' as we force users to fix > > those or the files do not get backed up in Tivoli. > > > > Since when? Last time I checked which was admittedly circa 2008, TSM would > backup files with newlines in them no problem. mmbackup on the other hand > in that time frame would simply die and backup nothing if there was a > single file on the file system with a newline in it. > > I would take a look at the mmbackup scripts which can handle such stuff > (least ways in >4.2) which would also suggest dsmc can handle it. > > As an aside I now think I know how you end up with newlines in file names. > Basically you cut and paste the file name complete with newlines (most > likely at the end) into a text field when saving the file. > Personally I think any program should baulk at that point but what do I > know. > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > > https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!KGKeukY!nVH69Xr88S0X5DmO8QbaI7eozd9pDvmtMN40tZU8vWuduEF4J01ZTfnypvOy$ > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Handling bad file names in policies?
This goes back as far as I can recall to <=GPFS 3.5 days. And no, I cannot recall what version of TSM-EE that was. But newline has been the only stopping point, for what seems like forever. Having filed many an mmbackup bug, I don't recall ever crashing on filenames. (tons of OTHER reasons, but not character set) We even generate an error report from this and email users to fix it. We accept basically almost everything else, and I have to say, we see some really crazy things sometimes. I think my current favorite is the full windows paths as a filename. (eg: "Y:\Temp\temp\290\work\0\Material_ERTi-5.in" ) Current IBM documentation doesn't go backwards past 4.2 but it says: "For IBM Spectrum Scale™ file systems with special characters frequently used in the names of files or directories, backup failures might occur. Known special characters that require special handling include: *, ?, ", ’, carriage return, and the new line character. In such cases, enable the Tivoli Storage Manager client options WILDCARDSARELITERAL and QUOTESARELITERAL on all nodes that are used in backup activities and make sure that the mmbackup option --noquote is used when invoking mmbackup." So maybe we could handle newlines somehow. But my lazy searches didn't show what TSM doesn't accept. Ed Wahl OSC -Original Message- From: gpfsug-discuss-boun...@spectrumscale.org On Behalf Of Jonathan Buzzard Sent: Monday, October 4, 2021 7:29 PM To: gpfsug-discuss@spectrumscale.org Subject: Re: [gpfsug-discuss] Handling bad file names in policies? On 04/10/2021 23:23, Wahl, Edward wrote: > I know I've run into this before way back, but my notes on how I > solved this aren't getting the job done in Scale 5.0.5.8 and my notes > are from 3.5. > Anyone know a way to get a LIST policy to properly feed bad filenames > into the output or an external script? > > When I say bad I mean things like control characters, spaces, etc. > Not concerned about the dreaded 'newline' as we force users to fix > those or the files do not get backed up in Tivoli. > Since when? Last time I checked which was admittedly circa 2008, TSM would backup files with newlines in them no problem. mmbackup on the other hand in that time frame would simply die and backup nothing if there was a single file on the file system with a newline in it. I would take a look at the mmbackup scripts which can handle such stuff (least ways in >4.2) which would also suggest dsmc can handle it. As an aside I now think I know how you end up with newlines in file names. Basically you cut and paste the file name complete with newlines (most likely at the end) into a text field when saving the file. Personally I think any program should baulk at that point but what do I know. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!KGKeukY!nVH69Xr88S0X5DmO8QbaI7eozd9pDvmtMN40tZU8vWuduEF4J01ZTfnypvOy$ ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Handling bad file names in policies?
Hi Ed, not a ready to run for "everything".. but just to remind, there is an ESCAPE statement by this you can cat policy2 RULE EXTERNAL LIST 'allfiles' EXEC '/var/mmfs/etc/list.exe' ESCAPE '%/#' and turn a file name into smth , what a policy can use I haven't used it for a while , but here is an example from a while ago .. ;-) [root@c25m4n03 stupid_files]# lltotal 0-rw-r--r-- 1 root root 21 Mar 22 03:44 dämlicher filename-rw-r--r-- 1 root root 2 Mar 22 03:59 üöä???ßß spacefilen[root@c25m4n03 stupid_files]# policy: 101378 247907919 0 -- /gpfs/fpofs/files/stupid_files/d%C3%A4mlicher%20filename101381 1945364096 0 -- /gpfs/fpofs/files/stupid_files/%C3%BC%C3%BC%C3%BC%C3%B6%C3%B6%C3%A4%C3%A4%3F%3F%3F%C3%9F%C3%9F%20spacefilename[I]2013-03-22@13:12:58.687 Policy execution. 2 files dispatched. verify with policy (ESCAPE '%/ä ') 101378 247907919 0 -- /gpfs/fpofs/files/stupid_files/dämlicher filename[...] hope this helps.. cheers - Ursprüngliche Nachricht -Von: "Jonathan Buzzard" Gesendet von: gpfsug-discuss-boun...@spectrumscale.orgAn: gpfsug-discuss@spectrumscale.orgCC:Betreff: [EXTERNAL] Re: [gpfsug-discuss] Handling bad file names in policies?Datum: Di, 5. Okt 2021 01:29 On 04/10/2021 23:23, Wahl, Edward wrote:> I know I've run into this before way back, but my notes on how I solved> this aren't getting the job done in Scale 5.0.5.8 and my notes are from> 3.5. > Anyone know a way to get a LIST policy to properly feed bad filenames> into the output or an external script?>> When I say bad I mean things like control characters, spaces, etc. Not> concerned about the dreaded 'newline' as we force users to fix those or> the files do not get backed up in Tivoli.>Since when? Last time I checked which was admittedly circa 2008, TSMwould backup files with newlines in them no problem. mmbackup on theother hand in that time frame would simply die and backup nothing ifthere was a single file on the file system with a newline in it.I would take a look at the mmbackup scripts which can handle such stuff(least ways in >4.2) which would also suggest dsmc can handle it.As an aside I now think I know how you end up with newlines in filenames. Basically you cut and paste the file name complete with newlines(most likely at the end) into a text field when saving the file.Personally I think any program should baulk at that point but what do Iknow.JAB.--Jonathan A. Buzzard Tel: +44141-5483420HPC System Administrator, ARCHIE-WeSt.University of Strathclyde, John Anderson Building, Glasgow. G4 0NG___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
