[Bug 1828074] Re: Memory usage issues with large Apache httpd configuration files
This bug was fixed in the package augeas - 1.10.1-2ubuntu1 --- augeas (1.10.1-2ubuntu1) bionic; urgency=medium * d/p/0005-Replace-pure-funcs-calls-in-path-expr-w-result.patch: Replace pure function invocations in path expressions with their result. (LP: #1828074) Thanks to David Lutterkort . * d/rules: Disable gnulib tests due to that tests are not testing augeas itself. (Closes: #919662) (LP: #1813566) Thanks to Matthias Klose. -- Miriam España Acebal Mon, 20 Sep 2021 11:18:53 +0200 ** Changed in: augeas (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828074 Title: Memory usage issues with large Apache httpd configuration files To manage notifications about this bug go to: https://bugs.launchpad.net/augeas/+bug/1828074/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828074] Re: Memory usage issues with large Apache httpd configuration files
Verification for Bionic
--
Checking the package is upgradable:
root@test-bug1828074-bionic:/etc/apache2/sites-available# apt list --upgradable
-a
Listing... Done
augeas-lenses/bionic 1.10.1-2ubuntu1 all [upgradable from: 1.10.1-2]
augeas-lenses/bionic,now 1.10.1-2 all [installed,upgradable to: 1.10.1-2ubuntu1]
augeas-tools/bionic 1.10.1-2ubuntu1 amd64 [upgradable from: 1.10.1-2]
augeas-tools/bionic,now 1.10.1-2 amd64 [installed,upgradable to:
1.10.1-2ubuntu1]
libaugeas0/bionic 1.10.1-2ubuntu1 amd64 [upgradable from: 1.10.1-2]
libaugeas0/bionic,now 1.10.1-2 amd64 [installed,upgradable to: 1.10.1-2ubuntu1]
If we executes(with memorybomb.conf in sites-available) the following:
augtool> match
/files/etc/apache2/sites-available/*[label()=~regexp('.*\\.conf')]//*[self::directive=~regexp('([Ii][Nn][Cc][Ll][Uu][Dd][Ee])|([Ii][Nn][Cc][Ll][Uu][Dd][Ee])|([Ii][Nn][Cc][Ll][Uu][Dd][Ee][Oo][Pp][Tt][Ii][Oo][Nn][Aa][Ll])')]
(no matches)
We can see in the attached image the expenditure in memory before and
after the fixing.
** Attachment added: "Image for bionic verification"
https://bugs.launchpad.net/ubuntu/+source/augeas/+bug/1828074/+attachment/5533838/+files/Before-and-after-fix-augeas-lp1828074.png
** Tags removed: verification-needed verification-needed-bionic
** Tags added: verification-done verification-done-bionic
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1828074
Title:
Memory usage issues with large Apache httpd configuration files
To manage notifications about this bug go to:
https://bugs.launchpad.net/augeas/+bug/1828074/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828074] Re: Memory usage issues with large Apache httpd configuration files
Hello Larry, or anyone else affected, Accepted augeas into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/augeas/1.10.1-2ubuntu1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-bionic. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: augeas (Ubuntu Bionic) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828074 Title: Memory usage issues with large Apache httpd configuration files To manage notifications about this bug go to: https://bugs.launchpad.net/augeas/+bug/1828074/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828074] Re: Memory usage issues with large Apache httpd configuration files
** Description changed:
[SRU]
[Impact]
The use of memory grows as a site conf file enlarges when using augeas.
[Test Plan]
- Copy the attached conf file here to /etc/apache2/sites-available and make
the site available to apache (a2ensite memorybomb).
- Mime module has to be enabled as well (check it with a2query -m | grep
mime). If not, enabling it with a2enmod mime.
- Reload apache2 configuration ( systemctl reload apache2 ). If something
wrong here, maybe a non-graphical web browser like lynx is need to be installed.
- Open htop in a different terminal
- Run augtool -I httpd.aug
- In augtool prompt use: match
/files/etc/apache2/sites-available/*[label()=~regexp('.*\\.conf')]//*[self::directive=~regexp('([Ii][Nn][Cc][Ll][Uu][Dd][Ee])|([Ii][Nn][Cc][Ll][Uu][Dd][Ee])|([Ii][Nn][Cc][Ll][Uu][Dd][Ee][Oo][Pp][Tt][Ii][Oo][Nn][Aa][Ll])')]
- Check in htop that augtool is consuming:
- Before fixing the issue: near 2238MB
- After fixing the issue (updated packages needed are augeas-lenses,
augeas-tools and libaugeas0): near 179 MB
PPA with built package with fix for testing: ppa:mirespace/sru-
lp-1828074-bionic-augeas-large-http-files
[Where problems could occur]
This patch is cherry picked from upstream, so the bug is corrected in
- following augeas package versions.
+ following augeas package versions and no other conflicts have arisen
+ with this change.
+
+ Also, the change itself acts as a cache of evaluated expressions
+ (substituting the abstract syntax tree in memory for the value it
+ produces), so the way expressions are evaluated is the same before and
+ after the change and no other part of augeas has to be affected by this.
[Other Info]
Fixed upstream in version 1.11.0, commit id :
bbf31f719db54916993be9042254f6d77b61cb13
Commit description: Replace pure function invocations in path
expressions with their result.
In path expressions, we generally need to evaluate functions against
every node that we consider for the result set. For example, in the path
expression /files/etc/hosts/*[ipaddr =~ regexp('127\\.')], the regexp
function was evaluated against every entry in /etc/hosts. Evaluating that
function requires the construction and compilation of a new regexp.
Because of how memory is managed during evaluation of path expressions,
the memory used by all these copies of the same regexp is only freed
after we are done evaluating the path expression. This causes
unacceptable memory usage in large files (see hercules-team#569).
To avoid these issues, we now distinguish between pure and impure
functions in the path expression interpreter. When we encounter a pure
function, we change the AST for the path expression so that the function
invocation is replaced with the result of invoking the function. With
the example above, that means we only construct and compile the regexp
'127\\.' once, regardless of how many nodes it gets checked against.
That leads to a dramatic reduction in the memory required to evaluate
path expressions with such constructs against large files.
Fixes hercules-team#569
Bug on upstream: https://github.com/hercules-team/augeas/issues/569
[Original Report]
This bug in libaugeas was fixed in version 1.11.0, but is not in the 18.04.2
repo.
See: https://github.com/hercules-team/augeas/issues/569
I found it using certbot and started a bug report there.
See: https://github.com/certbot/certbot/issues/7035
Which led me here to see if the newer version could be back-ported.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1828074
Title:
Memory usage issues with large Apache httpd configuration files
To manage notifications about this bug go to:
https://bugs.launchpad.net/augeas/+bug/1828074/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828074] Re: Memory usage issues with large Apache httpd configuration files
** Merge proposal linked: https://code.launchpad.net/~mirespace/ubuntu/+source/augeas/+git/augeas/+merge/408941 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828074 Title: Memory usage issues with large Apache httpd configuration files To manage notifications about this bug go to: https://bugs.launchpad.net/augeas/+bug/1828074/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828074] Re: Memory usage issues with large Apache httpd configuration files
** Description changed:
[SRU]
[Impact]
- The use of memory grows as a site conf file enlarges when using augeas.
+ The use of memory grows as a site conf file enlarges when using augeas.
[Test Plan]
- - Copy the attached conf file here to /etc/apache2/sites-available and make
the site available to apache (a2ensite memorybomb).
- - Mime module has to be enabled as well (check it with a2query -m | grep
mime). If not, enabling it with a2enmod mime.
- - Reload apache2 configuration ( systemctl reload apache2 ). If something
wrong here, maybe a non-graphical web browser like lynx is need to be
installed.
- - Open htop in a different terminal
- - Run augtool -I httpd.aug
- - In augtool prompt use: match
/files/etc/apache2/sites-available/*[label()=~regexp('.*\\.conf')]//*[self::directive=~regexp('([Ii][Nn][Cc][Ll][Uu][Dd][Ee])|([Ii][Nn][Cc][Ll][Uu][Dd][Ee])|([Ii][Nn][Cc][Ll][Uu][Dd][Ee][Oo][Pp][Tt][Ii][Oo][Nn][Aa][Ll])')]
- - Check in htop that augtool is consuming:
- - Before fixing the issue: near 2238MB
- - After fixing the issue (updated packages needed are augeas-lenses,
augeas-tools ans libaugeas0): near 179 MB
-
- PPA with built package with fix for testing:
ppa:mirespace/sru-lp-1828074-bionic-augeas-large-http-files
+ - Copy the attached conf file here to /etc/apache2/sites-available and make
the site available to apache (a2ensite memorybomb).
+ - Mime module has to be enabled as well (check it with a2query -m | grep
mime). If not, enabling it with a2enmod mime.
+ - Reload apache2 configuration ( systemctl reload apache2 ). If something
wrong here, maybe a non-graphical web browser like lynx is need to be installed.
+ - Open htop in a different terminal
+ - Run augtool -I httpd.aug
+ - In augtool prompt use: match
/files/etc/apache2/sites-available/*[label()=~regexp('.*\\.conf')]//*[self::directive=~regexp('([Ii][Nn][Cc][Ll][Uu][Dd][Ee])|([Ii][Nn][Cc][Ll][Uu][Dd][Ee])|([Ii][Nn][Cc][Ll][Uu][Dd][Ee][Oo][Pp][Tt][Ii][Oo][Nn][Aa][Ll])')]
+ - Check in htop that augtool is consuming:
+ - Before fixing the issue: near 2238MB
+ - After fixing the issue (updated packages needed are augeas-lenses,
augeas-tools and libaugeas0): near 179 MB
+
+ PPA with built package with fix for testing: ppa:mirespace/sru-
+ lp-1828074-bionic-augeas-large-http-files
[Where problems could occur]
- This patch is cherry picked from upstream, so the bug is corrected in
+ This patch is cherry picked from upstream, so the bug is corrected in
following augeas package versions.
[Other Info]
- Fixed upstream in version 1.11.0, commit id :
+ Fixed upstream in version 1.11.0, commit id :
bbf31f719db54916993be9042254f6d77b61cb13
- Commit description: Replace pure function invocations in path
+ Commit description: Replace pure function invocations in path
expressions with their result.
- In path expressions, we generally need to evaluate functions against
- every node that we consider for the result set. For example, in the path
- expression /files/etc/hosts/*[ipaddr =~ regexp('127\\.')], the regexp
- function was evaluated against every entry in /etc/hosts. Evaluating that
- function requires the construction and compilation of a new regexp.
+ In path expressions, we generally need to evaluate functions against
+ every node that we consider for the result set. For example, in the path
+ expression /files/etc/hosts/*[ipaddr =~ regexp('127\\.')], the regexp
+ function was evaluated against every entry in /etc/hosts. Evaluating that
+ function requires the construction and compilation of a new regexp.
- Because of how memory is managed during evaluation of path expressions, the
memory used by all these copies of the same regexp is only freed after we are
done evaluating the path expression. This causes unacceptable memory usage in
large files (see hercules-team#569).
-
- To avoid these issues, we now distinguish between pure and impure functions
in the path expression interpreter. When we encounter a pure function, we
change the AST for the path expression so that the function invocation is
replaced with the result of invoking the function. With the example above, that
means we only construct and compile the regexp '127\\.' once, regardless of how
many nodes it gets checked against. That leads to a dramatic reduction in the
memory required to evaluate path expressions with such constructs against large
files.
-
- Fixes hercules-team#569
+ Because of how memory is managed during evaluation of path expressions,
+ the memory used by all these copies of the same regexp is only freed
+ after we are done evaluating the path expression. This causes
+ unacceptable memory usage in large files (see hercules-team#569).
- Bug on upstream: https://github.com/hercules-team/augeas/issues/569
+ To avoid these issues, we now distinguish between pure and impure
+ functions in the path expression
[Bug 1828074] Re: Memory usage issues with large Apache httpd configuration files
Applied patch for fixing the increase of memory (attached is the conf site file for testing purposes) as suggested by the reporter. In the upstream's bug description is reported also a segmentation fault caused by very very large files (https://github.com/certbot/certbot/issues/7035) but is unplanned when this is going to be fixed upstream. This behaviour also occurs in the package tested here. ** Bug watch added: github.com/certbot/certbot/issues #7035 https://github.com/certbot/certbot/issues/7035 ** Attachment added: "Conf site file for test the fixing" https://bugs.launchpad.net/ubuntu/+source/augeas/+bug/1828074/+attachment/5526639/+files/memorybomb.conf -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828074 Title: Memory usage issues with large Apache httpd configuration files To manage notifications about this bug go to: https://bugs.launchpad.net/augeas/+bug/1828074/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828074] Re: Memory usage issues with large Apache httpd configuration files
** Description changed:
+ [SRU]
+
+ [Impact]
+
+ The use of memory grows as a site conf file enlarges when using augeas.
+
+ [Test Plan]
+
+ - Copy the attached conf file here to /etc/apache2/sites-available and make
the site available to apache (a2ensite memorybomb).
+ - Mime module has to be enabled as well (check it with a2query -m | grep
mime). If not, enabling it with a2enmod mime.
+ - Reload apache2 configuration ( systemctl reload apache2 ). If something
wrong here, maybe a non-graphical web browser like lynx is need to be
installed.
+ - Open htop in a different terminal
+ - Run augtool -I httpd.aug
+ - In augtool prompt use: match
/files/etc/apache2/sites-available/*[label()=~regexp('.*\\.conf')]//*[self::directive=~regexp('([Ii][Nn][Cc][Ll][Uu][Dd][Ee])|([Ii][Nn][Cc][Ll][Uu][Dd][Ee])|([Ii][Nn][Cc][Ll][Uu][Dd][Ee][Oo][Pp][Tt][Ii][Oo][Nn][Aa][Ll])')]
+ - Check in htop that augtool is consuming:
+ - Before fixing the issue: near 2238MB
+ - After fixing the issue (updated packages needed are augeas-lenses,
augeas-tools ans libaugeas0): near 179 MB
+
+ PPA with built package with fix for testing:
ppa:mirespace/sru-lp-1828074-bionic-augeas-large-http-files
+
+ [Where problems could occur]
+
+ This patch is cherry picked from upstream, so the bug is corrected in
+ following augeas package versions.
+
+ [Other Info]
+
+ Fixed upstream in version 1.11.0, commit id :
+ bbf31f719db54916993be9042254f6d77b61cb13
+
+ Commit description: Replace pure function invocations in path
+ expressions with their result.
+
+ In path expressions, we generally need to evaluate functions against
+ every node that we consider for the result set. For example, in the path
+ expression /files/etc/hosts/*[ipaddr =~ regexp('127\\.')], the regexp
+ function was evaluated against every entry in /etc/hosts. Evaluating that
+ function requires the construction and compilation of a new regexp.
+
+ Because of how memory is managed during evaluation of path expressions, the
memory used by all these copies of the same regexp is only freed after we are
done evaluating the path expression. This causes unacceptable memory usage in
large files (see hercules-team#569).
+
+ To avoid these issues, we now distinguish between pure and impure functions
in the path expression interpreter. When we encounter a pure function, we
change the AST for the path expression so that the function invocation is
replaced with the result of invoking the function. With the example above, that
means we only construct and compile the regexp '127\\.' once, regardless of how
many nodes it gets checked against. That leads to a dramatic reduction in the
memory required to evaluate path expressions with such constructs against large
files.
+
+ Fixes hercules-team#569
+
+ Bug on upstream: https://github.com/hercules-team/augeas/issues/569
+
+ [Original Report]
+
This bug in libaugeas was fixed in version 1.11.0, but is not in the 18.04.2
repo.
See: https://github.com/hercules-team/augeas/issues/569
I found it using certbot and started a bug report there.
See: https://github.com/certbot/certbot/issues/7035
Which led me here to see if the newer version could be back-ported.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1828074
Title:
Memory usage issues with large Apache httpd configuration files
To manage notifications about this bug go to:
https://bugs.launchpad.net/augeas/+bug/1828074/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828074] Re: Memory usage issues with large Apache httpd configuration files
** Changed in: augeas (Ubuntu Bionic) Status: Triaged => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828074 Title: Memory usage issues with large Apache httpd configuration files To manage notifications about this bug go to: https://bugs.launchpad.net/augeas/+bug/1828074/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828074] Re: Memory usage issues with large Apache httpd configuration files
** Changed in: augeas (Ubuntu Bionic) Assignee: (unassigned) => Miriam España Acebal (mirespace) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828074 Title: Memory usage issues with large Apache httpd configuration files To manage notifications about this bug go to: https://bugs.launchpad.net/augeas/+bug/1828074/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828074] Re: Memory usage issues with large Apache httpd configuration files
Standard support for cosmic has ended as of July 2019 ** Changed in: augeas (Ubuntu Cosmic) Status: Triaged => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828074 Title: Memory usage issues with large Apache httpd configuration files To manage notifications about this bug go to: https://bugs.launchpad.net/augeas/+bug/1828074/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828074] Re: Memory usage issues with large Apache httpd configuration files
** Tags added: bitesize -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828074 Title: Memory usage issues with large Apache httpd configuration files To manage notifications about this bug go to: https://bugs.launchpad.net/augeas/+bug/1828074/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828074] Re: Memory usage issues with large Apache httpd configuration files
** Changed in: augeas Status: Unknown => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828074 Title: Memory usage issues with large Apache httpd configuration files To manage notifications about this bug go to: https://bugs.launchpad.net/augeas/+bug/1828074/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828074] Re: Memory usage issues with large Apache httpd configuration files
Thank you for taking the time to report this bug and helping to make Ubuntu better. We can cherry-pick a fix to Ubuntu 18.04. Does the upstream commit apply to the 18.04 Ubuntu packaging cleanly? It would help if someone could prepare the upload please. ** Bug watch added: github.com/hercules-team/augeas/issues #569 https://github.com/hercules-team/augeas/issues/569 ** Also affects: augeas via https://github.com/hercules-team/augeas/issues/569 Importance: Unknown Status: Unknown ** Changed in: augeas (Ubuntu) Status: New => Triaged ** Changed in: augeas (Ubuntu) Importance: Undecided => High ** Also affects: augeas (Ubuntu Cosmic) Importance: Undecided Status: New ** Also affects: augeas (Ubuntu Bionic) Importance: Undecided Status: New ** Changed in: augeas (Ubuntu) Status: Triaged => Fix Released ** Changed in: augeas (Ubuntu Bionic) Status: New => Triaged ** Changed in: augeas (Ubuntu Cosmic) Status: New => Triaged ** Changed in: augeas (Ubuntu Bionic) Importance: Undecided => High ** Changed in: augeas (Ubuntu Cosmic) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828074 Title: Memory usage issues with large Apache httpd configuration files To manage notifications about this bug go to: https://bugs.launchpad.net/augeas/+bug/1828074/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
