Re: BMC IAM and DFHSM DATALOSS
Hi Dave, Not sure if this is you issue or not but we do have an APAR OA63738 that affects FSM when migrating to WORM tapes where we have seen what you describe. Take a look. If you still need help let me know the case # and I will look into it. Max Smith DFSMS Development -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: BMC IAM and DFHSM DATALOSS
On Mon, 6 Mar 2023 22:20:59 -0500, Rob Schramm wrote: >Is CICS involved? > >Rob > >On Mon, Mar 6, 2023, 13:41 Joel C. Ewing wrote: > >> I see two possibilities: >> Hey there Rob.That's where it was detected first, however we've been able to recreate outside of CICS. I now believe the problem lies within HSM at this point. The good folks over at BMC, provided me with FDREPORT job to dump the DSCB on our test case, and indeed the DS1IND08 is turned on indicating the file has been updated since the last recall. As I initially mentioned, we see the problem on z/OS V2.4 and V2.5.I do not know when this started happening, but our developers mentioned "months" or longer, so that leads me to think about previous maintenance levels. I think I still have a sysres laying around from prior to our last maint cycle in the fall, that I may IPL in the sandbox, and see if we can recreate there. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: BMC IAM and DFHSM DATALOSS
Is CICS involved? Rob On Mon, Mar 6, 2023, 13:41 Joel C. Ewing wrote: > I see two possibilities: > > Either IAM is somehow managing to update the file in a way that doesn't > end up with the changed bit set for the file -- might possibly happen if > the file is never properly closed by the task that updates it-- maybe if > the file is opened for update the changed bit only gets set at close if > some write to the file actually took place.. > > or > > some process is resetting the changed-bit after the file is closed but > before HSM migration kicks in for the dataset. > > I believe there is a DUMPCLASS option to RESET the changed bits on files > on a full volume DUMP -- on the assumption that you want the changed bit > to only reflect changes that occurred after that backup dump to avoid > unneeded dumps for later incremental backups of the same volume. This > sounds like something you would definitely NOT want to specify on > volumes also subject to migration with FSM enabled. If FSM is enabled, > the changed bit being off tells HSM if it needs to migrate the dataset > and still has a previously-migrated version of the dataset that was > recalled but not yet deleted from its migration volume, that all it > needs to do to migrate is to change the inactive old migrated dataset to > active status, delete the live dataset, and point it do the old migrated > copy. So a sequence of recall dataset, update dataset, > somehow-not-set-or-reset-change-bit, migrate-with-FSM-enabled is > guaranteed to lose the updates after the recall. > > Recalled, inactive migration versions of datasets on ML2 volumes can > persist for many weeks depending on the RECYCLE threshold for the ML2 > volumes. > > Joel C. Ewing > > On 3/6/23 11:07, Dave Jousma wrote: > > All, > > > > We just learned that we have a problem with IAM (Innovation Access > Method) the high perf replacement for VSAM. No idea how long it has been > going on, and I have tickets open with both BMC and IBM on this. Seems to > only affect IAM files, and exists in both z/OS V2.4, and V2.5, and multiple > versions of IAM. We are currently at 10.x, but the problem occurs in V9.x > as well. We see the problem mostly in our Development space, not in PROD > (or nothing reported yet, but problem exists there too). We dont migrate > much in PROD, and is why we havent had the problem reported there. > > > > The scenario here is > > - existing IAM file gets updates, records added or updated, doesnt > matter. > > - file goes unreferenced for 7 days, so HSM migrates > > - file gets recalled, the updates that were made are gone. > > > > In HSM we do have FSM enabled (Fast Subsequent Migration), where if HSM > is called to migrate the dataset, and it hasnt changed since last > migration, it just deletes the dataset and reconnects it to the already > migrated version in HSM. > > > > We've learned that turning FSM off circumvents the problem. IBM tells > us that "HSM checks the DS1RECAL and DS1IND08 flags in the Format-1 DSCB of > a data set to determine if a data set is eligible for fast subsequent > migration. The DS1RECAL flag is used to indicate if a data set has been > recalled and the DS1IND08 flag is used to indicate if a data set been > modified since it was last recalled. " > > > > IBM doc seems to indicate that OPEN handles setting these bits, BMC > support says they arent messing with these bits. I dont know *who* is to > blame. > > > > I'd be curious to hear from other installations that use IAM, has HSM > with FSM turned on. > > > > The recreate scenario is (assumes new test file) with FSM turned on > > 1 - Allocate test IAM file and add a record easily identifiable > > 2 - HSM Migrate the file > > 3 - HSM Recall the file > > 4 - RECORD ADDED IN #1 is there. > > 5 - Add another record with new identifiable info > > 6 - HSM migrate the file. > > 7 - HSM Restore the file. > > 8 - Record from #1 is there, record from #4 is not there. > > > > With FSM turned off, the dataset gets fresh migration copy every time, > so the issue is masked. > > > > -- > > For IBM-MAIN subscribe / signoff / archive access instructions, > > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > > -- > Joel C. Ewing > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: BMC IAM and DFHSM DATALOSS
I see two possibilities: Either IAM is somehow managing to update the file in a way that doesn't end up with the changed bit set for the file -- might possibly happen if the file is never properly closed by the task that updates it-- maybe if the file is opened for update the changed bit only gets set at close if some write to the file actually took place.. or some process is resetting the changed-bit after the file is closed but before HSM migration kicks in for the dataset. I believe there is a DUMPCLASS option to RESET the changed bits on files on a full volume DUMP -- on the assumption that you want the changed bit to only reflect changes that occurred after that backup dump to avoid unneeded dumps for later incremental backups of the same volume. This sounds like something you would definitely NOT want to specify on volumes also subject to migration with FSM enabled. If FSM is enabled, the changed bit being off tells HSM if it needs to migrate the dataset and still has a previously-migrated version of the dataset that was recalled but not yet deleted from its migration volume, that all it needs to do to migrate is to change the inactive old migrated dataset to active status, delete the live dataset, and point it do the old migrated copy. So a sequence of recall dataset, update dataset, somehow-not-set-or-reset-change-bit, migrate-with-FSM-enabled is guaranteed to lose the updates after the recall. Recalled, inactive migration versions of datasets on ML2 volumes can persist for many weeks depending on the RECYCLE threshold for the ML2 volumes. Joel C. Ewing On 3/6/23 11:07, Dave Jousma wrote: All, We just learned that we have a problem with IAM (Innovation Access Method) the high perf replacement for VSAM. No idea how long it has been going on, and I have tickets open with both BMC and IBM on this. Seems to only affect IAM files, and exists in both z/OS V2.4, and V2.5, and multiple versions of IAM. We are currently at 10.x, but the problem occurs in V9.x as well. We see the problem mostly in our Development space, not in PROD (or nothing reported yet, but problem exists there too). We dont migrate much in PROD, and is why we havent had the problem reported there. The scenario here is - existing IAM file gets updates, records added or updated, doesnt matter. - file goes unreferenced for 7 days, so HSM migrates - file gets recalled, the updates that were made are gone. In HSM we do have FSM enabled (Fast Subsequent Migration), where if HSM is called to migrate the dataset, and it hasnt changed since last migration, it just deletes the dataset and reconnects it to the already migrated version in HSM. We've learned that turning FSM off circumvents the problem. IBM tells us that "HSM checks the DS1RECAL and DS1IND08 flags in the Format-1 DSCB of a data set to determine if a data set is eligible for fast subsequent migration. The DS1RECAL flag is used to indicate if a data set has been recalled and the DS1IND08 flag is used to indicate if a data set been modified since it was last recalled. " IBM doc seems to indicate that OPEN handles setting these bits, BMC support says they arent messing with these bits. I dont know *who* is to blame. I'd be curious to hear from other installations that use IAM, has HSM with FSM turned on. The recreate scenario is (assumes new test file) with FSM turned on 1 - Allocate test IAM file and add a record easily identifiable 2 - HSM Migrate the file 3 - HSM Recall the file 4 - RECORD ADDED IN #1 is there. 5 - Add another record with new identifiable info 6 - HSM migrate the file. 7 - HSM Restore the file. 8 - Record from #1 is there, record from #4 is not there. With FSM turned off, the dataset gets fresh migration copy every time, so the issue is masked. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- Joel C. Ewing -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: BMC IAM and DFHSM DATALOSS
Try VTOC backups after Recall 3 and update 5 and restore 7 and compare. On Mon, Mar 6, 2023 at 11:08 AM Dave Jousma <01a0403c5dc1-dmarc-requ...@listserv.ua.edu> wrote: > > All, > > We just learned that we have a problem with IAM (Innovation Access Method) > the high perf replacement for VSAM. No idea how long it has been going on, > and I have tickets open with both BMC and IBM on this. Seems to only affect > IAM files, and exists in both z/OS V2.4, and V2.5, and multiple versions of > IAM. We are currently at 10.x, but the problem occurs in V9.x as well. We > see the problem mostly in our Development space, not in PROD (or nothing > reported yet, but problem exists there too). We dont migrate much in PROD, > and is why we havent had the problem reported there. > > The scenario here is > - existing IAM file gets updates, records added or updated, doesnt matter. > - file goes unreferenced for 7 days, so HSM migrates > - file gets recalled, the updates that were made are gone. > > In HSM we do have FSM enabled (Fast Subsequent Migration), where if HSM is > called to migrate the dataset, and it hasnt changed since last migration, it > just deletes the dataset and reconnects it to the already migrated version in > HSM. > > We've learned that turning FSM off circumvents the problem. IBM tells us > that "HSM checks the DS1RECAL and DS1IND08 flags in the Format-1 DSCB of a > data set to determine if a data set is eligible for fast subsequent > migration. The DS1RECAL flag is used to indicate if a data set has been > recalled and the DS1IND08 flag is used to indicate if a data set been > modified since it was last recalled. " > > IBM doc seems to indicate that OPEN handles setting these bits, BMC support > says they arent messing with these bits. I dont know *who* is to blame. > > I'd be curious to hear from other installations that use IAM, has HSM with > FSM turned on. > > The recreate scenario is (assumes new test file) with FSM turned on > 1 - Allocate test IAM file and add a record easily identifiable > 2 - HSM Migrate the file > 3 - HSM Recall the file > 4 - RECORD ADDED IN #1 is there. > 5 - Add another record with new identifiable info > 6 - HSM migrate the file. > 7 - HSM Restore the file. > 8 - Record from #1 is there, record from #4 is not there. > > With FSM turned off, the dataset gets fresh migration copy every time, so the > issue is masked. > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- Mike A Schwab, Springfield IL USA Where do Forest Rangers go to get away from it all? -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
BMC IAM and DFHSM DATALOSS
All, We just learned that we have a problem with IAM (Innovation Access Method) the high perf replacement for VSAM. No idea how long it has been going on, and I have tickets open with both BMC and IBM on this. Seems to only affect IAM files, and exists in both z/OS V2.4, and V2.5, and multiple versions of IAM. We are currently at 10.x, but the problem occurs in V9.x as well. We see the problem mostly in our Development space, not in PROD (or nothing reported yet, but problem exists there too). We dont migrate much in PROD, and is why we havent had the problem reported there. The scenario here is - existing IAM file gets updates, records added or updated, doesnt matter. - file goes unreferenced for 7 days, so HSM migrates - file gets recalled, the updates that were made are gone. In HSM we do have FSM enabled (Fast Subsequent Migration), where if HSM is called to migrate the dataset, and it hasnt changed since last migration, it just deletes the dataset and reconnects it to the already migrated version in HSM. We've learned that turning FSM off circumvents the problem. IBM tells us that "HSM checks the DS1RECAL and DS1IND08 flags in the Format-1 DSCB of a data set to determine if a data set is eligible for fast subsequent migration. The DS1RECAL flag is used to indicate if a data set has been recalled and the DS1IND08 flag is used to indicate if a data set been modified since it was last recalled. " IBM doc seems to indicate that OPEN handles setting these bits, BMC support says they arent messing with these bits. I dont know *who* is to blame. I'd be curious to hear from other installations that use IAM, has HSM with FSM turned on. The recreate scenario is (assumes new test file) with FSM turned on 1 - Allocate test IAM file and add a record easily identifiable 2 - HSM Migrate the file 3 - HSM Recall the file 4 - RECORD ADDED IN #1 is there. 5 - Add another record with new identifiable info 6 - HSM migrate the file. 7 - HSM Restore the file. 8 - Record from #1 is there, record from #4 is not there. With FSM turned off, the dataset gets fresh migration copy every time, so the issue is masked. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
