** Description changed: [ Impact ] * Since openscap 1.2.17-0.1ubuntu7.22.04.3 (LP: #2116751), Jammy USG compliance scans on machines with large directory trees (~100k+ files - under scanned paths) print thousands of spurious `E: probe_file: Failed - to check available memory` errors despite ample free memory. + under scanned paths) can print thousands of spurious `E: probe_file: + Failed to check available memory` errors despite ample free memory. * Additionally, rules that often collect more than 1000 items, like `xccdf_org.ssgproject.content_rule_no_files_unowned_by_user` and `xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned` are incorrectly reported as failures when the spurious errors begin, leading to inaccurate test results. * Each spurious error leaks the collected item, so `probe_file`'s memory usage grows until it hits the memory usage limit introduced in LP: #2116751, printing `W: probe_file: Memory usage ratio limit - reached!` or getting OOMed. + reached!` and aborting early or getting OOMed. * The fix corrects this with two upstream commits: - 12f9c02a612bb "Reset errno before call to strtoll": correctly initialize errno to 0 in the success case so that a single error doesn't propagate to future collected files - 83592087efc58 "Fix leak of item in probe_item_collect": Correctly free the collected items in the error case, following up on LP: #2116751. [ Test Plan ] * The bug is probabilistic, scaling with the number of files scanned, so a deterministic reproducer is not feasible. A script that performs the steps below is attached to the bug. 0. Set up a clean jammy system 1. Install the current jammy package and content: apt-cache policy libopenscap8 # 1.2.17-0.1ubuntu7.22.04.3 sudo apt install libopenscap8 scap-security-guide (alternatively install usg + usg-benchmarks for the usg path) 2. Create a large directory tree: sudo bash -c 'mkdir -p /srv/manyfiles && cd /srv/manyfiles && \ for i in $(seq 1 200000); do : > f$i; done' 3. Evaluate the two known-affected rules directly: sudo oscap xccdf eval --profile '(all)' \ --rule xccdf_org.ssgproject.content_rule_no_files_unowned_by_user \ --rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned \ --results /tmp/oscap-results.xml \ /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml \ 2>&1 | tee /tmp/scan.log Or run the full USG script: sudo usg audit cis_level1_workstation 4a. Failure output (unpatched): - /tmp/scan.log contains hundreds of "E: probe_file: Failed to check available memory" lines - the two rules show "Result fail" (or error) even on systems where they should pass per the rule definition - observe probe_file memory usage growing drastically or a "W: probe_file: Memory usage ratio limit reached!" message 4b. Success output (patched): - at most a handful "Failed to check available memory" lines, genuine errors are what LP: #2116751 was trying to catch - the two rules report their genuine result; i.e. passes on a compliant system per the rule definition - probe_file memory usage stays reasonable (less than 20+ GiB, no OOMs). [ Where problems could occur ] * 12f9c02 (initializing the errno to 0) is strictly a small correctness improvement. If the change were wrong, misbehavior would manifest either as false "memory limit reached" errors (as is the current case) or genuine failures becoming hidden as successes. The change itself has been shipped upstream since 1.3.7; Noble and later already contain it. * 8359208 (freeing the items on the error paths) could result in a double free / use-after-free in a probe_* process, printing errors like "W: oscap: Can't receive message", "Result error", or a segfault. This change is upstream as of 1.3.9, so it is also already present in Noble and later. * Both changes touch a code path that is only active when a single rule collects more than 1000 items (the memcheck threshold from LP: #2116751), so the majority of rules are unaffected. [ Other Info ] * These patches affect /usr/bin/oscap and the probe_* executables, which are in the libopenscap8 binary package in Jammy. * Only Jammy is affected: Noble and later ship openscap 1.3.9+ which already contain both commits * Follow-ups not included in this SRU: - 3ce4482466 "Improve counting collected items": replaces SEXP_list_length() with an explicit counter in probe_ctx. Not necessary, as the exact number of items doesn't really matter when triggering the memory safegaurd. This change is too large to have the same functional impact as just initializing the errno to 0 as is done in 12f9c02. - b20fca6e0d and f2461c7498: hard cap of 10000 collected items per object plus the OSCAP_PROBE_MAX_COLLECTED_ITEMS environment variable. This is a new feature, so not backporting. - ded3d58cd6: OSCAP_PROBE_MEMORY_USAGE_RATIO environment variable to tune the memory guard (discussed in LP: #2116751 comment #14). * References: - LP: #2116751 (SRU that introduced the memory guard and message spam: d/p/lp2116751-lower-memory-limits.patch) - LP: #2056775 (--debug masks the symptoms) - Errno fix: https://github.com/OpenSCAP/openscap/commit/12f9c02a612bb - Item leak fix: https://github.com/OpenSCAP/openscap/commit/83592087efc58 - Upstream bug report: https://github.com/OpenSCAP/openscap/issues/1858 - Upstream bug report: https://github.com/OpenSCAP/openscap/issues/1867 - Upstream bug report: https://github.com/OpenSCAP/openscap/issues/2345
** Description changed: [ Impact ] * Since openscap 1.2.17-0.1ubuntu7.22.04.3 (LP: #2116751), Jammy USG compliance scans on machines with large directory trees (~100k+ files under scanned paths) can print thousands of spurious `E: probe_file: Failed to check available memory` errors despite ample free memory. * Additionally, rules that often collect more than 1000 items, like `xccdf_org.ssgproject.content_rule_no_files_unowned_by_user` and `xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned` are incorrectly reported as failures when the spurious errors begin, leading to inaccurate test results. * Each spurious error leaks the collected item, so `probe_file`'s memory usage grows until it hits the memory usage limit introduced in LP: #2116751, printing `W: probe_file: Memory usage ratio limit reached!` and aborting early or getting OOMed. * The fix corrects this with two upstream commits: - 12f9c02a612bb "Reset errno before call to strtoll": correctly initialize errno to 0 in the success case so that a single error doesn't propagate to future collected files - 83592087efc58 "Fix leak of item in probe_item_collect": Correctly free the collected items in the error case, following up on LP: #2116751. [ Test Plan ] * The bug is probabilistic, scaling with the number of files scanned, - so a deterministic reproducer is not feasible. A script that performs - the steps below is attached to the bug. + so a deterministic reproducer is not feasible. A genuine error is likely + to trigger the spurious errors, but introducing a genuine error is also + difficult in a test plan. A script that performs the steps below is + attached to the bug. 0. Set up a clean jammy system 1. Install the current jammy package and content: apt-cache policy libopenscap8 # 1.2.17-0.1ubuntu7.22.04.3 sudo apt install libopenscap8 scap-security-guide (alternatively install usg + usg-benchmarks for the usg path) 2. Create a large directory tree: sudo bash -c 'mkdir -p /srv/manyfiles && cd /srv/manyfiles && \ for i in $(seq 1 200000); do : > f$i; done' 3. Evaluate the two known-affected rules directly: sudo oscap xccdf eval --profile '(all)' \ --rule xccdf_org.ssgproject.content_rule_no_files_unowned_by_user \ --rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned \ --results /tmp/oscap-results.xml \ /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml \ 2>&1 | tee /tmp/scan.log Or run the full USG script: sudo usg audit cis_level1_workstation 4a. Failure output (unpatched): - /tmp/scan.log contains hundreds of "E: probe_file: Failed to check available memory" lines - the two rules show "Result fail" (or error) even on systems where they should pass per the rule definition - observe probe_file memory usage growing drastically or a "W: probe_file: Memory usage ratio limit reached!" message 4b. Success output (patched): - - at most a handful "Failed to check available memory" lines, genuine errors are what LP: #2116751 was trying to catch + - at most a handful of "Failed to check available memory" lines--genuine errors are what LP: #2116751 was trying to catch - the two rules report their genuine result; i.e. passes on a compliant system per the rule definition - probe_file memory usage stays reasonable (less than 20+ GiB, no OOMs). [ Where problems could occur ] * 12f9c02 (initializing the errno to 0) is strictly a small correctness improvement. If the change were wrong, misbehavior would manifest either as false "memory limit reached" errors (as is the current case) or genuine failures becoming hidden as successes. The change itself has been shipped upstream since 1.3.7; Noble and later already contain it. * 8359208 (freeing the items on the error paths) could result in a double free / use-after-free in a probe_* process, printing errors like "W: oscap: Can't receive message", "Result error", or a segfault. This change is upstream as of 1.3.9, so it is also already present in Noble and later. * Both changes touch a code path that is only active when a single rule collects more than 1000 items (the memcheck threshold from LP: #2116751), so the majority of rules are unaffected. [ Other Info ] * These patches affect /usr/bin/oscap and the probe_* executables, which are in the libopenscap8 binary package in Jammy. * Only Jammy is affected: Noble and later ship openscap 1.3.9+ which already contain both commits * Follow-ups not included in this SRU: - - 3ce4482466 "Improve counting collected items": replaces SEXP_list_length() with an explicit counter in probe_ctx. Not necessary, as the exact number of items doesn't really matter when triggering the memory safegaurd. This change is too large to have the same functional impact as just initializing the errno to 0 as is done in 12f9c02. + - 3ce4482466 "Improve counting collected items": replaces SEXP_list_length() with an explicit counter in probe_ctx. Not necessary, as the exact number of items doesn't really matter when triggering the memory safegaurd. This change is too large to backport when it has the same functional impact as just initializing the errno to 0 as is done in 12f9c02. - b20fca6e0d and f2461c7498: hard cap of 10000 collected items per object plus the OSCAP_PROBE_MAX_COLLECTED_ITEMS environment variable. This is a new feature, so not backporting. - ded3d58cd6: OSCAP_PROBE_MEMORY_USAGE_RATIO environment variable to tune the memory guard (discussed in LP: #2116751 comment #14). * References: - LP: #2116751 (SRU that introduced the memory guard and message spam: d/p/lp2116751-lower-memory-limits.patch) - LP: #2056775 (--debug masks the symptoms) - Errno fix: https://github.com/OpenSCAP/openscap/commit/12f9c02a612bb - Item leak fix: https://github.com/OpenSCAP/openscap/commit/83592087efc58 - Upstream bug report: https://github.com/OpenSCAP/openscap/issues/1858 - Upstream bug report: https://github.com/OpenSCAP/openscap/issues/1867 - Upstream bug report: https://github.com/OpenSCAP/openscap/issues/2345 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2175424 Title: openscap 1.2.17-0.1ubuntu7.22.04.3: spurious rule failures on large filesystems To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openscap/+bug/2175424/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
