[Freeipa-users] Re: possible issue with ipa-backup on RHEL 9.3

2023-12-26 Thread Alexander Bokovoy via FreeIPA-users

On Пят, 22 сне 2023, Charles Hedrick via FreeIPA-users wrote:

A bit more info. Looking at errors, a normal backup terminates with

[20/Dec/2023:23:01:32.943228301 -0500] - INFO - archive_copyfile - Copying 
/etc/dirsrv/slapd-CS-RUTGERS-EDU/pwdfile.txt to /var/lib/dirsrv/slapd-\
CS-RUTGERS-EDU/bak/CS-RUTGERS-EDU/config_files/pwdfile.txt
[20/Dec/2023:23:01:32.957342035 -0500] - INFO - archive_copyfile - Copying 
/etc/dirsrv/slapd-CS-RUTGERS-EDU/certmap.conf to /var/lib/dirsrv/slapd\
-CS-RUTGERS-EDU/bak/CS-RUTGERS-EDU/config_files/certmap.conf
[20/Dec/2023:23:01:32.969828971 -0500] - INFO - archive_copyfile - Copying 
/etc/dirsrv/slapd-CS-RUTGERS-EDU/slapd-collations.conf to /var/lib/dir\
srv/slapd-CS-RUTGERS-EDU/bak/CS-RUTGERS-EDU/config_files/slapd-collations.conf
[20/Dec/2023:23:01:32.983763256 -0500] - INFO - task_backup_thread - Backup 
finished.
[2

The backup that hung is missing the last line, "Backup finished." ldap
stopped giving normal responses about a minute later, according to the
access log.


This looks like a thing internal to 389-ds. If you'd see it reproduced,
make sure to have debuginfo packages for 389-ds and freeipa installed
and then attempt to get a backtrace from 389-ds processes before you'd
kill them.




From: Charles Hedrick
Sent: Friday, December 22, 2023 9:56 AM
To: freeipa-users@lists.fedorahosted.org 
Subject: possible issue with ipa-backup on RHEL 9.3

I just upgraded one of three servers from RHEL 9.2. to 9.3. I have a clone of 
our three servers, on which all three have been upgraded to 9.3.

All of the servers run a cron job

/sbin/ipa-backup --online --data > /usr/local/scripts/ipa-backup.log 2>&1

The LDAP server hung (needed kill -9) at about the time that job ran, on the 
production server but not the testing copy. Obviously I can't prove that the 
backup caused the hang, but it's suspicious. I've commented out the cron job, 
since the backup isn't actually all the useful. If we have to restore we'd use 
a snapshot of the VM.

The backup completed successfully on the clone. On the production server it 
failed. Here is the log:

Preparing backup on krb4.cs.rutgers.edu
Local roles match globally used roles, proceeding.
Backing up userRoot in CS-RUTGERS-EDU to LDIF
Waiting for LDIF to finish
Backing up CS-RUTGERS-EDU
Waiting for BAK to finish
cannot connect to 'ldapi://%2Frun%2Fslapd-CS-RUTGERS-EDU.socket':
The ipa-backup command failed. See /var/log/ipabackup.log for more information

I'm wondering whether there's a bug that only happens under load.

We're been doing this in production for years with no trouble up to RHEL 9.2.






--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
--
___
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue


[Freeipa-users] Re: possible issue with ipa-backup on RHEL 9.3

2023-12-22 Thread Charles Hedrick via FreeIPA-users
A bit more info. Looking at errors, a normal backup terminates with

[20/Dec/2023:23:01:32.943228301 -0500] - INFO - archive_copyfile - Copying 
/etc/dirsrv/slapd-CS-RUTGERS-EDU/pwdfile.txt to /var/lib/dirsrv/slapd-\
CS-RUTGERS-EDU/bak/CS-RUTGERS-EDU/config_files/pwdfile.txt
[20/Dec/2023:23:01:32.957342035 -0500] - INFO - archive_copyfile - Copying 
/etc/dirsrv/slapd-CS-RUTGERS-EDU/certmap.conf to /var/lib/dirsrv/slapd\
-CS-RUTGERS-EDU/bak/CS-RUTGERS-EDU/config_files/certmap.conf
[20/Dec/2023:23:01:32.969828971 -0500] - INFO - archive_copyfile - Copying 
/etc/dirsrv/slapd-CS-RUTGERS-EDU/slapd-collations.conf to /var/lib/dir\
srv/slapd-CS-RUTGERS-EDU/bak/CS-RUTGERS-EDU/config_files/slapd-collations.conf
[20/Dec/2023:23:01:32.983763256 -0500] - INFO - task_backup_thread - Backup 
finished.
[2

The backup that hung is missing the last line, "Backup finished." ldap stopped 
giving normal responses about a minute later, according to the access log.

From: Charles Hedrick
Sent: Friday, December 22, 2023 9:56 AM
To: freeipa-users@lists.fedorahosted.org 
Subject: possible issue with ipa-backup on RHEL 9.3

I just upgraded one of three servers from RHEL 9.2. to 9.3. I have a clone of 
our three servers, on which all three have been upgraded to 9.3.

All of the servers run a cron job

/sbin/ipa-backup --online --data > /usr/local/scripts/ipa-backup.log 2>&1

The LDAP server hung (needed kill -9) at about the time that job ran, on the 
production server but not the testing copy. Obviously I can't prove that the 
backup caused the hang, but it's suspicious. I've commented out the cron job, 
since the backup isn't actually all the useful. If we have to restore we'd use 
a snapshot of the VM.

The backup completed successfully on the clone. On the production server it 
failed. Here is the log:

Preparing backup on krb4.cs.rutgers.edu
Local roles match globally used roles, proceeding.
Backing up userRoot in CS-RUTGERS-EDU to LDIF
Waiting for LDIF to finish
Backing up CS-RUTGERS-EDU
Waiting for BAK to finish
cannot connect to 'ldapi://%2Frun%2Fslapd-CS-RUTGERS-EDU.socket':
The ipa-backup command failed. See /var/log/ipabackup.log for more information

I'm wondering whether there's a bug that only happens under load.

We're been doing this in production for years with no trouble up to RHEL 9.2.

--
___
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue