Re: [gpfsug-discuss] shutdown

2017-03-03 Thread Jan-Frode Myklebust 
Unmount filesystems cleanly "mmumount all -a", stop gpfs "mmshutdown -N
gss_ppc64" and poweroff "xdsh gss_ppc64 poweroff".


-jf
fre. 3. mar. 2017 kl. 20.55 skrev Joseph Grace :

> Please excuse the newb question but we have a planned power outage coming
> and I can't locate a procedure to shutdown an ESSp GL2 system?
> Thanks,
>
>  Joe Grace
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] shutdown

2017-03-03 Thread Joseph Grace 
Please excuse the newb question but we
have a planned power outage coming and I can't locate a procedure to shutdown
an ESSp GL2 system?
Thanks,

 Joe Grace


___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] mmbackup logging issue

2017-03-03 Thread Edward Wahl 
Comments inline

On Fri, 3 Mar 2017 12:59:48 +
Jez Tucker  wrote:

> Hi Ash,
> 
>This is the primary reason for using snapshots for mmbackup ( -S 
> snapname ) and making sure that only mmbackup sends data to backup 
> rather than an oob dsmc:
> 
> Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* 
> contain 0 failed paths but there were 10012 failures.
> Cannot reconcile shadow database.
> Unable to compensate for all TSM errors in new shadow database.
>Preserving previous shadow database.
>Run next mmbackup with -q to synchronize shadow database.  exit 12 
> < ick!
> 
> Other obvious causes are very, very, odd filenames / paths.

Yes, GPFS allows much more lenient filenames than TSM does.  You can attempt to
cut this back a bit with 'WILDCARDSARELITERAL yes' and 'QUOTESARELITERAL yes'
in your dsm.opt  (and I recommend SKIPACLUPDATECHECK yes and UPDATECTIME
yes )

But this still won't catch everything.  We still tend to find foreign character
sets, control sequences that look like people trying to exit emacs, etc. in
file names.  Valid for the Filesystem, not so much for TSM.

Ed

> 
> Jez
> 
> 
> On 28/02/17 16:10, Ashish Thandavan wrote:
> > Dear all,
> >
> > We have a small GPFS cluster and a separate server running TSM and one 
> > of the three NSD servers backs up our GPFS filesystem to the TSM 
> > server using mmbackup. After a recent upgrade from v3.5 to 4.1.1, 
> > we've noticed that mmbackup no longer logs stuff like it used to :
> >
> > ...
> > Thu Jan 19 05:45:41 2017 mmbackup:Backing up files: 0 backed up, 
> > 870532 expired, 2 failed.
> > Thu Jan 19 06:15:41 2017 mmbackup:Backing up files: 0 backed up, 
> > 870532 expired, 3 failed.
> > Thu Jan 19 06:45:41 2017 mmbackup:Backing up files: 0 backed up, 
> > 870532 expired, 3 failed.
> > ...
> >
> >
> > instead of
> >
> > ...
> > Sat Dec  3 12:01:00 2016 mmbackup:Backing up files: 105030 backed up, 
> > 635456 expired, 30 failed.
> > Sat Dec  3 12:31:00 2016 mmbackup:Backing up files: 205934 backed up, 
> > 635456 expired, 57 failed.
> > Sat Dec  3 13:01:00 2016 mmbackup:Backing up files: 321702 backed up, 
> > 635456 expired, 169 failed.
> > ...
> >
> > like it used to pre-upgrade.
> >
> > I am therefore unable to see how far long it has got, and indeed if it 
> > completed successfully, as this is what it logs at the end of a job :
> >
> > ...
> > Tue Jan 17 18:07:31 2017 mmbackup:Completed policy backup run with 0 
> > policy errors, 10012 files failed, 0 severe errors, returning rc=9.
> > Tue Jan 17 18:07:31 2017 mmbackup:Policy for backup returned 9 Highest 
> > TSM error 12
> > mmbackup: TSM Summary Information:
> > Total number of objects inspected: 20617273
> > Total number of objects backed up: 0
> > Total number of objects updated: 0
> > Total number of objects rebound: 0
> > Total number of objects deleted: 0
> > Total number of objects expired: 1
> > Total number of objects failed: 10012
> > Total number of objects encrypted: 0
> > Total number of bytes inspected: 3821624716861
> > Total number of bytes transferred: 3712040943672
> > Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* 
> > contain 0 failed paths but there were 10012 failures.
> > Cannot reconcile shadow database.
> > Unable to compensate for all TSM errors in new shadow database.
> >   Preserving previous shadow database.
> >   Run next mmbackup with -q to synchronize shadow database.  exit 12
> >
> > If it helps, the mmbackup job is kicked off with the following options :
> >  /usr/lpp/mmfs/bin/mmbackup gpfs -n 8 -t full -B 2 -L 1 
> > --tsm-servers gpfs_weekly_stanza -N glossop1a | /usr/bin/tee 
> > /var/log/mmbackup/gpfs_weekly/backup_log.`date +%Y%m%d_%H_%M`
> >
> > (The excerpts above are from the backup_log. file.)
> >
> > Our NSD servers are running GPFS 4.1.1-11, TSM is at 7.1.1.100 and the 
> > File system version is 12.06 (3.4.0.3). Has anyone else seen this 
> > behaviour with mmbackup and if so, found a fix?
> >
> > Thanks,
> >
> > Regards,
> > Ash
> >
> 
> 



-- 

Ed Wahl
Ohio Supercomputer Center
614-292-9302
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] Save the Date 9th/10th May - UK/European/Global (?) Spectrum Scale user group meeting

2017-03-03 Thread GPFS UG Chair (Simon Thompson) 
Hi All,

Save the date for the next UK Spectrum Scale user group meeting on
9th/10th May 2017. As last year, this will be a 2 day meeting and having
out-grown IBM South Bank client centre who have hosted us over the past
few years, we're heading off to Manchester to a bigger venue.

Further details on registration and agenda will be available in the next
few weeks.

Thanks to IBM for supporting the event funding the venue, we are working
up our sponsorship package for other companies for an evening function.
This year we are able to take up to 200 delegates and we're expecting that
it will be expanded out to other European customers looking for an English
language event, and we're flattered that Doug listed us a the Global event
:-D

Thanks

Simon
(Group Chair)


___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] mmbackup logging issue

2017-03-03 Thread Jez Tucker 

Comments inline...

On 03/03/17 13:05, Ashish Thandavan wrote:

Hi Jez,



  This is the primary reason for using snapshots for mmbackup ( -S  
snapname ) and making sure that only mmbackup sends data to backup 
rather than an oob dsmc:


Tue Jan 17 18:07:31 2017 mmbackup:Audit files 
/cs/mmbackup.audit.gpfs* contain 0 failed paths but there were 10012 
failures.

Cannot reconcile shadow database.
Unable to compensate for all TSM errors in new shadow database.
  Preserving previous shadow database.
  Run next mmbackup with -q to synchronize shadow database. exit 12   
< ick!




I was going to send a separate email about this, but as you mention it 
-- I was wondering if the TSM server requires the shadow files to also 
be backed up? And that reconciliation of the shadow database will fail 
if they are not?

No, that's not a requirement.

(Unless of course, you mean above that the shadow database failure is 
purely on account of the 10012 failures...)
More than likely.  Check the dsmerror.log and the mmbackup logs. There 
are also other possibilities which could exhibit the similar cases.




Re the use of snapshots for mmbackup, are you saying dsmc does not get 
involved in sending the files to the TSM server? I haven't used 
snapshots for mmbackup, but will certainly try!
Snapshots provide a consistent dataset to backup.  I.E. no errors from 
'trying to backup a file which has been deleted since I first knew about 
it in the scan phase'.


But as per previous related thread 'Tracking deleted files', YMMV 
depending on your workload and environment.




Regards,
Ash



___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] mmbackup logging issue

2017-03-03 Thread Jez Tucker 

Hi Ash,

  This is the primary reason for using snapshots for mmbackup ( -S 
snapname ) and making sure that only mmbackup sends data to backup 
rather than an oob dsmc:


Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* 
contain 0 failed paths but there were 10012 failures.

Cannot reconcile shadow database.
Unable to compensate for all TSM errors in new shadow database.
  Preserving previous shadow database.
  Run next mmbackup with -q to synchronize shadow database.  exit 12 
< ick!


Other obvious causes are very, very, odd filenames / paths.

Jez


On 28/02/17 16:10, Ashish Thandavan wrote:

Dear all,

We have a small GPFS cluster and a separate server running TSM and one 
of the three NSD servers backs up our GPFS filesystem to the TSM 
server using mmbackup. After a recent upgrade from v3.5 to 4.1.1, 
we've noticed that mmbackup no longer logs stuff like it used to :


...
Thu Jan 19 05:45:41 2017 mmbackup:Backing up files: 0 backed up, 
870532 expired, 2 failed.
Thu Jan 19 06:15:41 2017 mmbackup:Backing up files: 0 backed up, 
870532 expired, 3 failed.
Thu Jan 19 06:45:41 2017 mmbackup:Backing up files: 0 backed up, 
870532 expired, 3 failed.

...


instead of

...
Sat Dec  3 12:01:00 2016 mmbackup:Backing up files: 105030 backed up, 
635456 expired, 30 failed.
Sat Dec  3 12:31:00 2016 mmbackup:Backing up files: 205934 backed up, 
635456 expired, 57 failed.
Sat Dec  3 13:01:00 2016 mmbackup:Backing up files: 321702 backed up, 
635456 expired, 169 failed.

...

like it used to pre-upgrade.

I am therefore unable to see how far long it has got, and indeed if it 
completed successfully, as this is what it logs at the end of a job :


...
Tue Jan 17 18:07:31 2017 mmbackup:Completed policy backup run with 0 
policy errors, 10012 files failed, 0 severe errors, returning rc=9.
Tue Jan 17 18:07:31 2017 mmbackup:Policy for backup returned 9 Highest 
TSM error 12

mmbackup: TSM Summary Information:
Total number of objects inspected: 20617273
Total number of objects backed up: 0
Total number of objects updated: 0
Total number of objects rebound: 0
Total number of objects deleted: 0
Total number of objects expired: 1
Total number of objects failed: 10012
Total number of objects encrypted: 0
Total number of bytes inspected: 3821624716861
Total number of bytes transferred: 3712040943672
Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* 
contain 0 failed paths but there were 10012 failures.

Cannot reconcile shadow database.
Unable to compensate for all TSM errors in new shadow database.
  Preserving previous shadow database.
  Run next mmbackup with -q to synchronize shadow database.  exit 12

If it helps, the mmbackup job is kicked off with the following options :
 /usr/lpp/mmfs/bin/mmbackup gpfs -n 8 -t full -B 2 -L 1 
--tsm-servers gpfs_weekly_stanza -N glossop1a | /usr/bin/tee 
/var/log/mmbackup/gpfs_weekly/backup_log.`date +%Y%m%d_%H_%M`


(The excerpts above are from the backup_log. file.)

Our NSD servers are running GPFS 4.1.1-11, TSM is at 7.1.1.100 and the 
File system version is 12.06 (3.4.0.3). Has anyone else seen this 
behaviour with mmbackup and if so, found a fix?


Thanks,

Regards,
Ash




--
*Jez Tucker*
Head of Research and Development, Pixit Media
07764193820 | jtuc...@pixitmedia.com 
www.pixitmedia.com  | Tw:@pixitmedia.com 



--

This email is confidential in that it is intended for the exclusive 
attention of the addressee(s) indicated. If you are not the intended 
recipient, this email should not be read or disclosed to any other person. 
Please notify the sender immediately and delete this email from your 
computer system. Any opinions expressed are not necessarily those of the 
company from which this email was sent and, whilst to the best of our 
knowledge no viruses or defects exist, no responsibility can be accepted 
for any loss or damage arising from its receipt or subsequent use of this 
email.
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] mmbackup logging issue

2017-03-03 Thread Peter Childs 
That's basically what we did, They are only environment variables, so if you 
not using bash to call mmbackup you will need to change the lines accordingly.

What they do is in the manual the issue is the default changed between 
versions.

Peter Childs
ITS Research Infrastructure
Queen Mary, University of London


From: gpfsug-discuss-boun...@spectrumscale.org 
 on behalf of Sobey, Richard A 

Sent: Friday, March 3, 2017 9:20:24 AM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] mmbackup logging issue

HI all

We have the same problem (less of a problem, more lack of visibilitiy).

Can I just add those lines to the top of our mmbackup.sh script?

-Original Message-
From: gpfsug-discuss-boun...@spectrumscale.org 
[mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of Ashish Thandavan
Sent: 02 March 2017 16:50
To: gpfsug-discuss@spectrumscale.org
Subject: Re: [gpfsug-discuss] mmbackup logging issue

Dear Peter,


On 02/03/17 16:34, Peter Childs wrote:
> We had that issue.
>
> we had to
>
> export MMBACKUP_PROGRESS_CONTENT=5
> export MMBACKUP_PROGRESS_INTERVAL=300
>
> before we run it to get it back.
>
> Lets just say IBM changed the behaviour, We ended up opening a PRM to
> get that answer ;) We also set -L 1
>
> you can change how often the messages are displayed by changing
> MMBACKUP_PROGRESS_INTERVAL flexable but the default is different;)
>

I'll set those variables before kicking off the next mmbackup and hope that 
fixes it.

Thank you!!

Regards,
Ash

--
-
Ashish Thandavan

UNIX Support Computing Officer
Department of Computer Science
University of Oxford
Wolfson Building
Parks Road
Oxford OX1 3QD

Phone: 01865 610733
Email: ashish.thanda...@cs.ox.ac.uk

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] mmbackup logging issue

2017-03-03 Thread Sobey, Richard A 
HI all

We have the same problem (less of a problem, more lack of visibilitiy).

Can I just add those lines to the top of our mmbackup.sh script?

-Original Message-
From: gpfsug-discuss-boun...@spectrumscale.org 
[mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of Ashish Thandavan
Sent: 02 March 2017 16:50
To: gpfsug-discuss@spectrumscale.org
Subject: Re: [gpfsug-discuss] mmbackup logging issue

Dear Peter,


On 02/03/17 16:34, Peter Childs wrote:
> We had that issue.
>
> we had to
>
> export MMBACKUP_PROGRESS_CONTENT=5
> export MMBACKUP_PROGRESS_INTERVAL=300
>
> before we run it to get it back.
>
> Lets just say IBM changed the behaviour, We ended up opening a PRM to 
> get that answer ;) We also set -L 1
>
> you can change how often the messages are displayed by changing 
> MMBACKUP_PROGRESS_INTERVAL flexable but the default is different;)
>

I'll set those variables before kicking off the next mmbackup and hope that 
fixes it.

Thank you!!

Regards,
Ash

--
-
Ashish Thandavan

UNIX Support Computing Officer
Department of Computer Science
University of Oxford
Wolfson Building
Parks Road
Oxford OX1 3QD

Phone: 01865 610733
Email: ashish.thanda...@cs.ox.ac.uk

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss