Re: [Rpm-maint] [rpm-software-management/rpm] Fix resource leaks on zstd open error paths (#945)
Merged #945 into master. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/945#event-2822298006___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Generate file class dictionary after file classification for stability (#944)
Merged #944 into master. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/944#event-2822297248___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Optimize sqlite database rebuild (#947)
Postpone sql index creation until closing time on database rebuilds, this is more than 50% improvement on rebuilddb time on both SSD and traditional disks. You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/947 -- Commit Summary -- * Optimize sqlite database rebuild -- File Changes -- M lib/backend/sqlite.c (8) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/947.patch https://github.com/rpm-software-management/rpm/pull/947.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/947 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)
In the process of testing dcrpm, when I execute the command line lsof - F p /var/lib/rpm/.dbenv.lock, lsof displays the PIDs of all processes, including 1, instead of only displaying the PIDs of the .dbenv.lock file; It may have been generated after I installed typing (Python module) independently. Of course, it is not necessarily repeated. Therefore, I am afraid that dcrpm will execute lsof-f P /var/lib/rpm/.dbenv.lock to get the wrong PIDs and kill them all. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/232#issuecomment-557400253___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)
Yes, it should work just fine. We actually have packaging for dcrpm at https://github.com/facebookincubator/rpm-backports/tree/master/rpms/dcrpm that we use on CentOS 7 (and macOS) with no issues. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/232#issuecomment-557398238___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)
@jaymzh I see that dcrpm used typing(begin python-3.5 as a new feature added) , but my project is running on centos 7.5 1804 and using the python 2.7.5 version, Is typing(dcrpm) compatible with Python 2.7.5? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/232#issuecomment-557397334___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)
You could instead run it as nonroot: `su -c "rpm -q " nobody` (or possibly `su -c "rpm -q " -s /bin/bash nobody` if nobody's shell is not a real shell). Or you can make sure [dcrpm](https://github.com/facebookincubator/dcrpm) runs in a regular basis to detect and correct your RPM db... but certainly preventing the issues is better. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/232#issuecomment-557379740___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)
Can not avoid it? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/232#issuecomment-557374397___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)
@jaymzh Yes, the processes running as root account. `the memmap() the DB read-write, and write it back out when they are done... hence if then something changes at the same time, one of those process can write back old pages.` `# ls -lthra /var/lib/rpm total 70M -rw-r--r--. 1 root root0 Oct 23 18:04 .dbenv.lock -rw-r--r--. 1 root root0 Oct 23 18:04 .rpm.lock -rw-r--r--. 1 root root 8.0K Nov 21 11:03 Triggername drwxr-xr-x. 52 root root 4.0K Nov 21 11:04 .. -rw-r--r--. 1 root root 8.0K Nov 21 11:04 Conflictname -rw-r--r--. 1 root root 60M Nov 21 16:46 Packages -rw-r--r--. 1 root root 48K Nov 21 16:46 Name -rw-r--r--. 1 root root 4.2M Nov 21 16:46 Basenames -rw-r--r--. 1 root root 28K Nov 21 16:46 Group -rw-r--r--. 1 root root 256K Nov 21 16:46 Requirename -rw-r--r--. 1 root root 1.9M Nov 21 16:46 Providename -rw-r--r--. 1 root root 24K Nov 21 16:46 Obsoletename -rw-r--r--. 1 root root 2.1M Nov 21 16:46 Dirnames -rw-r--r--. 1 root root 20K Nov 21 16:46 Installtid -rw-r--r--. 1 root root 44K Nov 21 16:46 Sigmd5 -rw-r--r--. 1 root root 76K Nov 21 16:46 Sha1header drwxr-xr-x. 2 root root 4.0K Nov 21 17:21 . -rw-r--r--. 1 root root 344K Nov 22 11:30 __db.001 -rw-r--r--. 1 root root 188K Nov 22 11:30 __db.002 -rw-r--r--. 1 root root 1.3M Nov 22 11:30 __db.003` the /var/lib/rpm/.dbenv.lock and /var/lib/rpm/.rpm.lock can not protect the multi processes serial execution?? I don't know much about database. `>>> the memmap() the DB read-write, and write it back out when they are done... hence if then something changes at the same time, one of those process can write back old pages.` It indicate that the BDB use memmap(), and the multi proccesses rpm -q software.xxx, then changed the memory page, lead to old page write back. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/232#issuecomment-557374263___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)
@jianwei1216 Are the processes running as root? If so, the memmap() the DB read-write, and write it back out when they are done... hence if then something changes at the same time, one of those process can write back old pages. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/232#issuecomment-557176087___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Generate file class dictionary after file classification for stability (#944)
Ack, thanks for verifying. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/944#issuecomment-557076155___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Fix Arm detection improvements (#946)
Theres some issues with the detection of particular Arm hardware features. We never want to use the crypto feature as part of rpm functionality, its deployment is widely variable, and the HW optimisation should be run time detected with fall back through various optimisation paths. The 8c3a7b8fa commit where this functionality also has other issues it didnt include the VFP HWCAP for armv6 which rpm included previously, but just the VFP3, and NEON is a requirement, and hence always present in armv8 so its assumed that its part of armv8. Signed-off-by: Peter Robinson pbrobin...@gmail.com You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/946 -- Commit Summary -- * Fix Arm detection improvements -- File Changes -- M lib/rpmrc.c (18) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/946.patch https://github.com/rpm-software-management/rpm/pull/946.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/946 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Generate file class dictionary after file classification for stability (#944)
Tested it with our khmeros-fonts build. Results are still reproducible. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/944#issuecomment-557048211___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Fix resource leaks on zstd open error paths (#945)
If zstd stream initialization fails, the opened fd and the stream itself are leaked. Handle error exit in a central label. You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/945 -- Commit Summary -- * Fix resource leaks on zstd open error paths -- File Changes -- M rpmio/rpmio.c (12) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/945.patch https://github.com/rpm-software-management/rpm/pull/945.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/945 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)
Recently, I have encountered similar problems in my CentOS 7 environment. `[root@controller-3 ~]# uname -a Linux controller-3 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux [root@controller-3 ~]# cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core) [root@controller-3 ~]# rpm -q rpm rpm-4.11.3-32.el7.x86_64 [root@controller-3 ~]# rpm -q libdb libdb-5.3.21-24.el7.x86_64` But I can't reproduce it! This is the context of the problem at the sceneļ¼ 1. There are 8 processes running concurrently exec rpm -q software.xxx; 2. Each process generates 8 rpm -q software.xxx in 1 minute; 3. It's been running for half a year; 4. Millions of rpm -q software.xxx processes have been generated on CentOS 7; 5. memory exhaustion; 6. after recovery rpmdb (/var/lib/rpm), OS recovery; I really want to know the root cause of this problem!!! Can you tell me? Thank you very much! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/232#issuecomment-557007863___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Generate file class dictionary after file classification for stability (#944)
@bmwiedemann if you can verify this works for you, I'd appreciate. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/944#issuecomment-556981828___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Generate file class dictionary after file classification for stability (#944)
Store the file type strings in the classifier, and generate the dictionary and its ids serially after the parallel section completes to ensure stable order. Besides making the classifying really run in parallel again, this also moves the pool- and file-counting related constraints out of the parallel section for theoretically better parallelization. This is an alternative fix to #934, using omp order clause simply made the classification run serially. You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/944 -- Commit Summary -- * Revert Preserve processing order in file classification * Generate file class dictionary after file classification for stability -- File Changes -- M build/rpmfc.c (34) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/944.patch https://github.com/rpm-software-management/rpm/pull/944.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/944 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
