Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)
On Fri, Mar 02, 2018 at 05:15:46PM +0300, Evgeny Kotkov wrote: > Unless I am missing something, this might be worth considering before the > 1.10 GA release. Evgeny, you were entirely right about calling this out as a release blocker. I am sorry for having suggested otherwise.
Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)
On 02.03.2018 19:21, Stefan Sperling wrote: > I am just questioning the usefulness of halting the presses and restarting > the soak for another month for something that isn't a security / data > corruption issue. It's a potential DOS that only needs read access. Falls under security by my definition. -- Brane
Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)
On Fri, Mar 02, 2018 at 07:21:15PM +0100, Stefan Sperling wrote: > I am just questioning the usefulness of halting the presses and restarting > the soak for another month for something that isn't a security / data > corruption issue. I anticipate that problems of similar severity to this > one will still be discovered after we release 1.10.0, regardless of > whether we release 1.10.0 at end of March or later. > > Though maybe my idea of the impact of this bug is wrong? > If this really makes some repositories entirely unusable with authz enabled > then of course it should be considered a blocker. Is it this severe? I have misread our flowchart at: http://subversion.apache.org/docs/community-guide/svn-soak-management.png It seems for this kind of issue we'd extend soak by just one week instead of four? I wouldn't mind a one-week extension for this kind of bug fix. One week seems reasonable.
Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)
On Fri, Mar 02, 2018 at 09:02:02PM +0300, Evgeny Kotkov wrote: > Stefan Sperling writes: > > > I'd rather ship 1.10.0 at the prospected release date followed closely > > by 1.10.1 to fix bugs such as these, than delay general access to 1.10.0 > > even further. > > While I do not have significant objections against such plan, I find the > idea of shipping a performance feature that causes a massive slowdown > instead of an improvement somewhat controversial. (In other words, > I am -0 for that.) > > > You may not have realized this, but I have been waiting for 1.10.0 to > > happen for *over a year* https://svn.haxx.se/dev/archive-2017-01/0043.shtml > > For all this time, I have wanted the conflict resolver to get real world > > exposure because I need feedback from users out there to improve it. > > I kept mostly quiet because I didn't want to push too hard for this > > release all by myself because of the relatively high share of burden > > this would imply. So I waited for activity from the community to make > > it happen as a true collective effort. > > Not too sure about how this is connected to the soak period and to the > release process — speaking of which, I would say that your e-mail may > discourage people from reporting issues during the soak period. I am not trying to discourage people from reporting and fixing problems. I am sorry if what I wrote could be interpreted in this way. I am just questioning the usefulness of halting the presses and restarting the soak for another month for something that isn't a security / data corruption issue. I anticipate that problems of similar severity to this one will still be discovered after we release 1.10.0, regardless of whether we release 1.10.0 at end of March or later. Though maybe my idea of the impact of this bug is wrong? If this really makes some repositories entirely unusable with authz enabled then of course it should be considered a blocker. Is it this severe?
Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)
Stefan Sperling writes: > I'd rather ship 1.10.0 at the prospected release date followed closely > by 1.10.1 to fix bugs such as these, than delay general access to 1.10.0 > even further. While I do not have significant objections against such plan, I find the idea of shipping a performance feature that causes a massive slowdown instead of an improvement somewhat controversial. (In other words, I am -0 for that.) > You may not have realized this, but I have been waiting for 1.10.0 to > happen for *over a year* https://svn.haxx.se/dev/archive-2017-01/0043.shtml > For all this time, I have wanted the conflict resolver to get real world > exposure because I need feedback from users out there to improve it. > I kept mostly quiet because I didn't want to push too hard for this > release all by myself because of the relatively high share of burden > this would imply. So I waited for activity from the community to make > it happen as a true collective effort. Not too sure about how this is connected to the soak period and to the release process — speaking of which, I would say that your e-mail may discourage people from reporting issues during the soak period. > If this one bug really bothers you enough to hold the planned release back > it makes me wonder why you didn't push for a fix much earlier. We have had > plenty of time. I haven't been and am not pushing for a fix. Rather than that, I have just included the additional information about the problem with a comment that it might be viable to look into before the GA release. Moreover, I reported the issue at the very moment I found it with an edge-case reproduction. Once I was asked to bisect for a specific revision, I should have probably stated that I won't have time to do that. But I have been thinking that I would be able to find some. When I stumbled across it again, I found the revision and the simple reproduction — but as it seems, this hasn't been the most appropriate time for including these new details. Putting all that aside, I wouldn't say that it is productive to discuss issues in such way. In my opinion, we should be doing it the other way around by actually encouraging reports of various problems during the soak period. Regards, Evgeny Kotkov
Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)
On Fri, Mar 02, 2018 at 05:15:46PM +0300, Evgeny Kotkov wrote: > Unless I am missing something, this might be worth considering before the > 1.10 GA release. Not about the actual bug, just a meta comment on the process: I'd rather ship 1.10.0 at the prospected release date followed closely by 1.10.1 to fix bugs such as these, than delay general access to 1.10.0 even further. You may not have realized this, but I have been waiting for 1.10.0 to happen for *over a year* https://svn.haxx.se/dev/archive-2017-01/0043.shtml For all this time, I have wanted the conflict resolver to get real world exposure because I need feedback from users out there to improve it. I kept mostly quiet because I didn't want to push too hard for this release all by myself because of the relatively high share of burden this would imply. So I waited for activity from the community to make it happen as a true collective effort. I was glad when Julian volunteered to drive the process. If this one bug really bothers you enough to hold the planned release back it makes me wonder why you didn't push for a fix much earlier. We have had plenty of time. I don't mean to disrespect the fact that you may not have had time recently -- we are all constraint for time these days. But I also believe we shouldn't panic over every bug that slips into this .0 release. It is OK to ship 1.10.0 with known bugs that aren't very serious security / data corruption issues. There's a section in the release notes which lists known issues. I would prefer to document this memory consumption problem there and patch it like any other regular bug.
Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)
Stefan Fuhrmann writes: > Would it be possible for you to bisect this to find the offending revision? > My random guess would be that in the context of mod_dav_svn, we might use > an unsuitable pool for authz caching. While looking through the various 1.10-related topics, I remembered about this issue. I have been able to narrow it down in my environment to https://svn.apache.org/r1778923 (Ensure that even long-lived DAV connections use up-to-date authz.) Perhaps, a simpler reproduction script would be to issue an 'svn log' for a medium-sized repository. In my environment, doing so for the trunk of TortoiseSVN's repository with 25,000 revisions causes the httpd process to consume up to a 1 GB of RAM while processing the request. Overall, the log takes around 11 seconds instead of 2, compared to 1.9.7. Unless I am missing something, this might be worth considering before the 1.10 GA release. Thanks, Evgeny Kotkov
Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)
On 05.12.2017 22:05, Evgeny Kotkov wrote: Julian Foad writes: After any issues raised in this discussion are resolved, we feel we should go ahead and produce RC1 as soon as possible. I think that I am seeing a 1.10 regression in terms of httpd's memory usage during large imports. In my environment, when I `svn import` an unpacked version of Boost (https://www.boost.org/) that consists of around 60,000 files, I see that the memory consumption by the server process rapidly grows up to 1.5 GB. The environment is a Windows 8.1 x64 machine with httpd 2.4.29 configured to use short-circuit authz and HTTPS. Apparently, this behavior is specific to 1.10, as I cannot reproduce it with 1.9 binaries. (I haven't investigated the issue any further at this time, and it might as well be reproducible in other configurations.) Would it be possible for you to bisect this to find the offending revision? My random guess would be that in the context of mod_dav_svn, we might use an unsuitable pool for authz caching. -- Stefan^2.
Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)
Julian Foad writes: > After any issues raised in this discussion are resolved, we feel we should > go ahead and produce RC1 as soon as possible. I think that I am seeing a 1.10 regression in terms of httpd's memory usage during large imports. In my environment, when I `svn import` an unpacked version of Boost (https://www.boost.org/) that consists of around 60,000 files, I see that the memory consumption by the server process rapidly grows up to 1.5 GB. The environment is a Windows 8.1 x64 machine with httpd 2.4.29 configured to use short-circuit authz and HTTPS. Apparently, this behavior is specific to 1.10, as I cannot reproduce it with 1.9 binaries. (I haven't investigated the issue any further at this time, and it might as well be reproducible in other configurations.) Thanks, Evgeny Kotkov