Hello, Sorry i have been working a high priority case. I have removed the confusing test plan and replaced with just running openscap with htop to see the memory consumption rates.
I have also updated the "Where problems can occur" section. Thank You, Heather Lemon ** Description changed: [ Impact ] What is Openscap? openscap is a program that scans user directories and looks for security vulnerabilities, security compliance, and generates reports. More specifically a scan can happen on bare-metal machines, virtual machines, virtual machine images (qcow2 and others), containers (including Docker), and container images. During a scan - openscap stores objects in memory and it can be a very memory intensive program. Security compliance comes from here, The scap security guide. https://github.com/ComplianceAsCode/content/releases/tag/<latest release>. What is currently happening is a program from opscap specifically probe_file is consuming 100% cpu during a scan. It is also scanning /proc/ which is fixed in a later version. The probe_file executable consumes all the CPU core of the system during a CIS scan with openscap tool. excessive resource usage running a specific rule which is related to this bug [1]. This has been fixed in OpenSCAP 1.3, while Jammy runs 1.2.17. A fix for this patch has been made [2]. [ Test Plan ] Steps to Reproduce: In a Jammy VM: Have SSSD installed. This is crucial to files being opened and scanned causing error. [3] apt-cache policy python3-openscap jammy version - 1.2.17-0.1ubuntu7.22.04.2 sudo apt-get install python3-openscap # Download the latest Scap Security Guide wget https://github.com/ComplianceAsCode/content/releases/tag/v0.1.77/scap-security-guide-0.1.77.tar.gz gunzip -d scap-security-guide-0.1.77.tar.gz tar -xf scap-security-guide-0.1.77.tar cd scap-security-guide-0.1.77/ sudo mkdir -p mkdir /usr/share/xml/scap/ssg/content cp ssg-ubuntu2204-ds.xml /usr/share/xml/scap/ssg/content/ # show guide for ubuntu oscap info /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml + # This part is only necessary if you have a vm with a lot of RAM. Consume a portion of RAM with python script. My testing was 1/2 capacity. ``` import time # Set a target memory usage in gigabytes target_gigabytes = 4 # Calculate the number of bytes target_bytes = target_gigabytes * (1024 ** 3) # Create a list to hold the allocated memory memory_hog = [] print(f"Allocating {target_gigabytes}GB of memory. This might freeze your system.") try: while True: # Append a large block of bytes to the list # Adjust the size of the block (e.g., 1024*1024) to control allocation speed memory_hog.append(b" " * (1024 * 1024 * 100)) # Allocate 100MB at a time if sum(len(x) for x in memory_hog) >= target_bytes: break print("Allocation complete. Memory held.") # The script will now hold the allocated memory until you close it. while True: time.sleep(1) except MemoryError: print("Memory allocation failed. The system ran out of resources.") except Exception as e: print(f"An error occurred: {e}") ``` CTRL+C to quit script once testing is completed. - # create 100 users - for i in $(seq 1 100); do sudo useradd -N -g users user$i; echo "user-ubu" | sudo passwd user$i; done - # create 1000 text files - for i in $(seq 1 100); do echo "This is test file number $i." > file$i.txt; 1000 $(id -u user$i); done - # each user opens 100 files and reads it - for i in $(seq 1 1000); do -u user1 file_1.txt 1000 100 & done - --> this will start 100 processes having 100 threads each, which are opening 1000 files each (shared between threads) + # Run htop and sort by cpu. + This will show probe_file at the top when sorted **specifically** when running the rule called: **Verify that Shared Library Files Have Root Ownership** - # Run oscap in a new terminal at the same time as executing the third step. - oscap xccdf eval --rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned --results-arf /tmp/oscap_results.xml /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml + # Run oscap in a new terminal at the same time as htop + oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_stig --results-arf /tmp/oscap_results.xml /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml - # While oscap runs, strace probe_file for some time in a new terminal + + # Optional: While oscap runs, strace probe_file for some time in a new terminal timeout 10s strace -fttTvyy -o oscap_10s.strace -s 64 -p <pid of probe_file> - # find probe_file executable - ps -aux | grep probe_file - ``` - root 6165 0.0 0.0 288064 11264 pts/3 Sl+ 09:12 0:00 /usr/lib/x86_64-linux-gnu/openscap/probe_file - root 6197 52.4 0.1 288064 22248 pts/3 Sl+ 09:12 0:09 /usr/lib/x86_64-linux-gnu/openscap/probe_file - ubuntu 6408 0.0 0.0 9212 2560 pts/5 R+ 09:12 0:00 grep --color=auto probe_file - ``` - # attach strace - sudo timeout 10s strace -fttTvyy -o oscap_10s20250823.strace -s 64 -p 6197 - strace: Process 6197 attached with 5 threads - You can also pull up system monitor and watch memory be consumed. - Or just run htop and sort by memory. [see screenshot] Once this happens, it becomes laggy and program is slow. look at logs for errors specifically lstat in strace output. A crash occurs, but the program still succeeds. Title Ensure All Files Are Owned by a Group Rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned FAIL: 304:pthread_timedjoin_np: 0, Success W: oscap: Can't receive message: 103, Software caused connection abort. E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 Result error or something like Title Ensure All Files Are Owned by a Group Rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned W: oscap: Obtrusive data from probe! Result fail [ Where Problems Could Occur ] Calculating max ratio could be incorrect. Return memory check code could fail and as a result not run the scan or exit early. On systems with a small amount of memory (<2GB) openscap stops sooner than needed and on systems with a very large amount of memory it stops later than needed. [ Other Info ] We will not be fixing noble as there was not a substantial difference when the patch was applied or not. It was difficult to tell visually if the patch fixed the memory consumption. Backport from upstream. Commands # show guide for ubuntu oscap info /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml Openscap test packages https://launchpad.net/~hypothetical-lemon/+archive/ubuntu/lp2116751-openscap-bugfix/+packages [1] https://bugzilla.redhat.com/show_bug.cgi?id=1932833 [2] https://github.com/OpenSCAP/openscap/pull/1803 [3] https://documentation.ubuntu.com/server/how-to/sssd/with-active-directory/index.html ** Description changed: [ Impact ] What is Openscap? openscap is a program that scans user directories and looks for security vulnerabilities, security compliance, and generates reports. More specifically a scan can happen on bare-metal machines, virtual machines, virtual machine images (qcow2 and others), containers (including Docker), and container images. During a scan - openscap stores objects in memory and it can be a very memory intensive program. Security compliance comes from here, The scap security guide. https://github.com/ComplianceAsCode/content/releases/tag/<latest release>. What is currently happening is a program from opscap specifically probe_file is consuming 100% cpu during a scan. It is also scanning /proc/ which is fixed in a later version. The probe_file executable consumes all the CPU core of the system during a CIS scan with openscap tool. excessive resource usage running a specific rule which is related to this bug [1]. This has been fixed in OpenSCAP 1.3, while Jammy runs 1.2.17. A fix for this patch has been made [2]. [ Test Plan ] Steps to Reproduce: In a Jammy VM: Have SSSD installed. This is crucial to files being opened and scanned causing error. [3] apt-cache policy python3-openscap jammy version - 1.2.17-0.1ubuntu7.22.04.2 sudo apt-get install python3-openscap # Download the latest Scap Security Guide wget https://github.com/ComplianceAsCode/content/releases/tag/v0.1.77/scap-security-guide-0.1.77.tar.gz gunzip -d scap-security-guide-0.1.77.tar.gz tar -xf scap-security-guide-0.1.77.tar cd scap-security-guide-0.1.77/ sudo mkdir -p mkdir /usr/share/xml/scap/ssg/content cp ssg-ubuntu2204-ds.xml /usr/share/xml/scap/ssg/content/ # show guide for ubuntu oscap info /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml # This part is only necessary if you have a vm with a lot of RAM. Consume a portion of RAM with python script. My testing was 1/2 capacity. ``` import time # Set a target memory usage in gigabytes target_gigabytes = 4 # Calculate the number of bytes target_bytes = target_gigabytes * (1024 ** 3) # Create a list to hold the allocated memory memory_hog = [] print(f"Allocating {target_gigabytes}GB of memory. This might freeze your system.") try: while True: # Append a large block of bytes to the list # Adjust the size of the block (e.g., 1024*1024) to control allocation speed memory_hog.append(b" " * (1024 * 1024 * 100)) # Allocate 100MB at a time if sum(len(x) for x in memory_hog) >= target_bytes: break print("Allocation complete. Memory held.") # The script will now hold the allocated memory until you close it. while True: time.sleep(1) except MemoryError: print("Memory allocation failed. The system ran out of resources.") except Exception as e: print(f"An error occurred: {e}") ``` CTRL+C to quit script once testing is completed. - # Run htop and sort by cpu. - This will show probe_file at the top when sorted **specifically** when running the rule called: **Verify that Shared Library Files Have Root Ownership** + Continuing... + # Run htop and sort by cpu. + This will show probe_file at the top when sorted **specifically** when running the rule called: **Verify that Shared Library Files Have Root Ownership** # Run oscap in a new terminal at the same time as htop oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_stig --results-arf /tmp/oscap_results.xml /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml - # Optional: While oscap runs, strace probe_file for some time in a new terminal timeout 10s strace -fttTvyy -o oscap_10s.strace -s 64 -p <pid of probe_file> You can also pull up system monitor and watch memory be consumed. Once this happens, it becomes laggy and program is slow. look at logs for errors specifically lstat in strace output. A crash occurs, but the program still succeeds. Title Ensure All Files Are Owned by a Group Rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned FAIL: 304:pthread_timedjoin_np: 0, Success W: oscap: Can't receive message: 103, Software caused connection abort. E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 Result error or something like Title Ensure All Files Are Owned by a Group Rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned W: oscap: Obtrusive data from probe! Result fail [ Where Problems Could Occur ] Calculating max ratio could be incorrect. Return memory check code could fail and as a result not run the scan or exit early. On systems with a small amount of memory (<2GB) openscap stops sooner than needed and on systems with a very large amount of memory it stops later than needed. [ Other Info ] We will not be fixing noble as there was not a substantial difference when the patch was applied or not. It was difficult to tell visually if the patch fixed the memory consumption. Backport from upstream. Commands # show guide for ubuntu oscap info /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml Openscap test packages https://launchpad.net/~hypothetical-lemon/+archive/ubuntu/lp2116751-openscap-bugfix/+packages [1] https://bugzilla.redhat.com/show_bug.cgi?id=1932833 [2] https://github.com/OpenSCAP/openscap/pull/1803 [3] https://documentation.ubuntu.com/server/how-to/sssd/with-active-directory/index.html ** Description changed: [ Impact ] What is Openscap? openscap is a program that scans user directories and looks for security vulnerabilities, security compliance, and generates reports. More specifically a scan can happen on bare-metal machines, virtual machines, virtual machine images (qcow2 and others), containers (including Docker), and container images. During a scan - openscap stores objects in memory and it can be a very memory intensive program. Security compliance comes from here, The scap security guide. https://github.com/ComplianceAsCode/content/releases/tag/<latest release>. What is currently happening is a program from opscap specifically probe_file is consuming 100% cpu during a scan. It is also scanning /proc/ which is fixed in a later version. The probe_file executable consumes all the CPU core of the system during a CIS scan with openscap tool. excessive resource usage running a specific rule which is related to this bug [1]. This has been fixed in OpenSCAP 1.3, while Jammy runs 1.2.17. A fix for this patch has been made [2]. [ Test Plan ] Steps to Reproduce: In a Jammy VM: Have SSSD installed. This is crucial to files being opened and scanned causing error. [3] apt-cache policy python3-openscap jammy version - 1.2.17-0.1ubuntu7.22.04.2 sudo apt-get install python3-openscap # Download the latest Scap Security Guide wget https://github.com/ComplianceAsCode/content/releases/tag/v0.1.77/scap-security-guide-0.1.77.tar.gz gunzip -d scap-security-guide-0.1.77.tar.gz tar -xf scap-security-guide-0.1.77.tar cd scap-security-guide-0.1.77/ sudo mkdir -p mkdir /usr/share/xml/scap/ssg/content cp ssg-ubuntu2204-ds.xml /usr/share/xml/scap/ssg/content/ # show guide for ubuntu oscap info /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml # This part is only necessary if you have a vm with a lot of RAM. Consume a portion of RAM with python script. My testing was 1/2 capacity. ``` import time # Set a target memory usage in gigabytes target_gigabytes = 4 # Calculate the number of bytes target_bytes = target_gigabytes * (1024 ** 3) # Create a list to hold the allocated memory memory_hog = [] print(f"Allocating {target_gigabytes}GB of memory. This might freeze your system.") try: while True: # Append a large block of bytes to the list # Adjust the size of the block (e.g., 1024*1024) to control allocation speed memory_hog.append(b" " * (1024 * 1024 * 100)) # Allocate 100MB at a time if sum(len(x) for x in memory_hog) >= target_bytes: break print("Allocation complete. Memory held.") # The script will now hold the allocated memory until you close it. while True: time.sleep(1) except MemoryError: print("Memory allocation failed. The system ran out of resources.") except Exception as e: print(f"An error occurred: {e}") ``` CTRL+C to quit script once testing is completed. Continuing... # Run htop and sort by cpu. This will show probe_file at the top when sorted **specifically** when running the rule called: **Verify that Shared Library Files Have Root Ownership** # Run oscap in a new terminal at the same time as htop oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_stig --results-arf /tmp/oscap_results.xml /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml # Optional: While oscap runs, strace probe_file for some time in a new terminal timeout 10s strace -fttTvyy -o oscap_10s.strace -s 64 -p <pid of probe_file> You can also pull up system monitor and watch memory be consumed. Once this happens, it becomes laggy and program is slow. look at logs for errors specifically lstat in strace output. A crash occurs, but the program still succeeds. Title Ensure All Files Are Owned by a Group Rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned FAIL: 304:pthread_timedjoin_np: 0, Success W: oscap: Can't receive message: 103, Software caused connection abort. E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 E: probe_file: Invalid value of the `recurse_direction' attribute: -1 Result error or something like Title Ensure All Files Are Owned by a Group Rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned W: oscap: Obtrusive data from probe! Result fail [ Where Problems Could Occur ] Calculating max ratio could be incorrect. Return memory check code could fail and as a result not run the scan or exit early. On systems with a small amount of memory (<2GB) openscap stops sooner than needed and on systems with a very large amount of memory it stops later than needed. + - Openscap will throw a warning and fail that specific scanned directory and then it normally continues and finishes successfully. Example: + W: probe_file: Memory usage ratio limit reached! limit=0.100000, current=0.312033, used=573 MB, free=131 MB, total=1837 MB, count of items=1001 + Result fail + ... continue scanning.. [ Other Info ] We will not be fixing noble as there was not a substantial difference when the patch was applied or not. It was difficult to tell visually if the patch fixed the memory consumption. Backport from upstream. Commands # show guide for ubuntu oscap info /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml Openscap test packages https://launchpad.net/~hypothetical-lemon/+archive/ubuntu/lp2116751-openscap-bugfix/+packages [1] https://bugzilla.redhat.com/show_bug.cgi?id=1932833 [2] https://github.com/OpenSCAP/openscap/pull/1803 [3] https://documentation.ubuntu.com/server/how-to/sssd/with-active-directory/index.html -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2116751 Title: openscap probe_file process consumes excessive resources during CIS scan To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openscap/+bug/2116751/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
