ISP 7.1.7.400 on Linux - Back a while ago, we were working to eliminate an ISP server. So, we removed replication from all of the nodes (or so we thought) and removed the target replication server.
Well, somewhere along the way, it didn't completely/successfully remove replication from one of the nodes. This wasn't discovered until after we removed the target server. Now this node is in a limbo state. Can't backup to it (we ended up renaming it and recreating the node) - can't delete it - can't expire it - can't remove replication from it (this is where we hope there is a hidden "force" command) - can't re-add the target replication server. *Expire Inventory produces this error:* 7/2/2019 12:04:05 PM ANR2017I Administrator ZFORRAY issued command: EXPIRE INVENTORY NODE=RO-ARIAPROD-BAD 7/2/2019 12:04:05 PM ANR0984I Process 54 for EXPIRE INVENTORY started in the BACKGROUND at 12:04:06 PM. 7/2/2019 12:04:05 PM ANR0811I Inventory client file expiration started as process 54. 7/2/2019 12:04:05 PM ANR0405I Session 8351 ended for administrator ZFORRAY (WinNT). 7/2/2019 12:04:05 PM ANR9999D_0415665470 admIsNodeConfiguredForRepl(admrepl.c:4966) Thread<202543>: Replication Mode not found for node RO-ARIAPROD-BAD 7/2/2019 12:04:05 PM ANR9999D Thread<202543> issued message 9999 from: 7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x0000000106b995 OutDiagToCons 7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x0000000106ebcc outDiagfExt 7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x000000006284e4 admIsNodeConfiguredForRepl 7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x0000000062868b admIsNodeReplica 7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x0000000062876c admIsNodeIdReplica 7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x000000009e345f ExpirationProcessThread 7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x00000001122f02 StartThread 7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x00003978807aa1 *UNKNOWN* 7/2/2019 12:04:05 PM ANR9999D Thread<202543> 0x000039784e8c4d *UNKNOWN* 7/2/2019 12:04:06 PM ANR2369I Database backup volume and recovery plan file expiration starting under process 54. 7/2/2019 12:04:06 PM ANR0167I Inventory file expiration process 54 processed for 0 minutes. 7/2/2019 12:04:06 PM ANR0812I Inventory file expiration process 54 completed: processed 0 nodes, examined 0 objects, deleting 0 backup objects, 0 archive objects, 0 DB backup volumes, and 0 recovery plan files. 0 objects were retried 0 errors were encountered and 0 objects were skipped. 7/2/2019 12:04:06 PM ANR0985I Process 54 for EXPIRE INVENTORY running in the BACKGROUND completed with completion state FAILURE at 12:04:07 PM. *Trying to delete its filespace generates a similar error:* 7/2/2019 12:05:38 PM ANR2017I Administrator ZFORRAY issued command: DELETE FILESPACE RO-ARIAPROD-BAD 7 t=ba namet=fsid 7/2/2019 12:05:38 PM ANR9999D_0415665470 admIsNodeConfiguredForRepl(admrepl.c:4966) Thread<202566>: Replication Mode not found for node RO-ARIAPROD-BAD 7/2/2019 12:05:38 PM ANR9999D Thread<202566> issued message 9999 from: 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x0000000106b995 OutDiagToCons 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x0000000106ebcc outDiagfExt 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x000000006284e4 admIsNodeConfiguredForRepl 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x000000005a0ccf AdmDeleteFilespace 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00000000555fe0 AdmCommandLocal 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x0000000055772f admCommand 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00000000d90ee2 SmAdminCommandThread 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00000001122f02 StartThread 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00003978807aa1 *UNKNOWN* 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x000039784e8c4d *UNKNOWN* 7/2/2019 12:05:38 PM ANR9999D_3725110895 AdmDeleteFilespace(admfs.c:1186) Thread<202566>: Error 9990 obtaining replication information on node RO-ARIAPROD-BAD 7/2/2019 12:05:38 PM ANR9999D Thread<202566> issued message 9999 from: 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x0000000106b995 OutDiagToCons 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x0000000106ebcc outDiagfExt 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x000000005a0cfe AdmDeleteFilespace 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00000000555fe0 AdmCommandLocal 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x0000000055772f admCommand 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00000000d90ee2 SmAdminCommandThread 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00000001122f02 StartThread 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x00003978807aa1 *UNKNOWN* 7/2/2019 12:05:38 PM ANR9999D Thread<202566> 0x000039784e8c4d *UNKNOWN* 7/2/2019 12:05:38 PM ANR2017I Administrator ZFORRAY issued command: ROLLBACK 7/2/2019 12:05:38 PM ANR0405I Session 8359 ended for administrator ZFORRAY (WinNT). -- *Zoltan Forray* Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator VMware Administrator Xymon Monitor Administrator Virginia Commonwealth University UCC/Office of Technology Services www.ucc.vcu.edu zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details visit http://phishing.vcu.edu/
