Re: [serf-dev] Re: Reduce the 1.7 release feature set a bit?
On Fri, Aug 20, 2010 at 10:23:28PM -0700, Justin Erenkrantz wrote: BTW, those who complain that no one's working on ra_serf hasn't looked at the diffs between 1.6.x and trunk. New authn framework, digest auth, Kerberos auth, error handling fixes, better proxy support, HTTPv2 support, etc. Many many kudos to everyone driving ra_serf forward! *group hug* -- justin I didn't mean to imply that nobody was working on serf / ra_serf. There was a lot of serf hacking going on during the June hackathon. It's just that specific issues were appearing to be lingering. Without much digging I cannot tell whether a problem is rooted in ra_serf or serf proper, and I don't follow serf development. So I rely on you guys to point out when a fix has been made in serf proper. I'll try to reproduce the problem with the new serf release once it's available. Thanks! Stefan
Re: Reduce the 1.7 release feature set a bit?
On Fri, Aug 20, 2010 at 11:55:10AM -0700, Bert Huijben wrote: From: Stefan Sperling - Conflict storage Do we really need to have this ready for 1.7? We might as well keep using the 1.6-style of recording conflict information in temporary files, that act as conflict markers. At the very least, doing so would not be a regression over 1.6, and it can be improved upon during the 1.8 cycle. I think we need some parts of this to compensate for the new obstructed/missing handling. You can have a different unversioned directory in place where a versioned directory should be. And update can't record that as a conflict using just the old information. Right. So that's one special case in the wc-ng world we simply cannot represent with the 1.6.x conflict storage framework. But do we really need an entirely new framework, or just a bit of meta-data in wc.db to get this working? And drop this meta-data again when the proper framework is in place? I guess now that we're dealing with DB schemas rather than entries files, we can make random changes to our meta-data easily as we go from one release to the next. The temorary hack would just need to be supported in read-only form later. One of the reasons that this isn't handled, is that this upgrade can't be handled atomicly before we move to single-db. (Tree conflicts would move from the parent to the child DBs, while we only upgrade them at once. So any error while upgrading would break your conflict storage) I don't think I understand what problem you're describing here. Can you elaborate? Will there still be a problem after moving the single db? - svnrdump It's marked as complete, but I don't really know yet if we can consider it complete yet. I plan to review it thoroughly again in its current state before deciding whether or not I'd like to see it in 1.7. In particular, the current test coverage seems quite poor, and I'm afraid of that hiding potential problems. However, if anyone already has reviewed its current state and has found it to be of releasable quality, please let us know. But if noone has the cycles available to review it thoroughly, and in particular to improve test coverage, I'd say we should postpone it to 1.8. I think the dump part is stable. That's good to hear! The load part needs more testing, but if it could load the ASF repository I'm happy to ship it as is and fix possible bugs later. (But I don't expect that this will complete now) I'm hesitant to ship features we haven't tested. While the impact of potential problems would be low in this case, having users run into problems with new features always leaves a bad impression about Subversion's quality. I'd like people to have a warm and fuzzy feeling about storing their data in Subversion, knowing that they can rely on Subversion and trust it with their data. Any bugs they see, even in non-critical features, damages that trust. - file externals improvements Are file externals usable at all right now or are they even more broken than they were in 1.6? If they aren't more broken, could we postpone fixing them to 1.8? At least, we should ship the feature in a state that matches the 1.6 state. It's not a killer feature that many people rely on, so I don't think 1.7 should definitely wait for improvements in this area. The problem here is that we never documented how it worked in 1.6. It worked as a hack in 1.6 and as a slightly different hack now. Issues are continuously noted and more hacks are added to fix specific use cases. But is the user experience for file externals any different after upgrading to the current trunk code? If not, then we can just ship it as it is. Else, we can re-design and re-implement the feature, or fix the use cases that are broken in current trunk code, depending on which is less work. I'm -0 on fixing any more specific use cases until we have a proper design to support these file externals/internals/hacks. (I don't want to see 2 additional if statements added to every function in libsvn_wc and libsvn_client, just because we never designed this feature) If adding a couple of if-statements restores 1.6 behaviour and allows us to ship 1.7 faster, then I'd prefer that route. - svn patch I'd like to really almost-feature-freeze this now (the last feature freeze was before danna's Gsoc). The only feature I'd still like to add is conflict recording (an exception dannas and I had in mind during the pre-gsoc freeze, too). Doesn't this require the new conflict store mentioned before? Not necessarily. We could simply treat the presence of a .svnpatch.rej file as a conflict marker. Stefan
RE: Reduce the 1.7 release feature set a bit?
-Original Message- From: Stefan Sperling [mailto:s...@elego.de] Sent: zaterdag 21 augustus 2010 3:32 To: Bert Huijben Cc: dev@subversion.apache.org Subject: Re: Reduce the 1.7 release feature set a bit? On Fri, Aug 20, 2010 at 11:55:10AM -0700, Bert Huijben wrote: From: Stefan Sperling - Conflict storage Do we really need to have this ready for 1.7? We might as well keep using the 1.6-style of recording conflict information in temporary files, that act as conflict markers. At the very least, doing so would not be a regression over 1.6, and it can be improved upon during the 1.8 cycle. I think we need some parts of this to compensate for the new obstructed/missing handling. You can have a different unversioned directory in place where a versioned directory should be. And update can't record that as a conflict using just the old information. Right. So that's one special case in the wc-ng world we simply cannot represent with the 1.6.x conflict storage framework. But do we really need an entirely new framework, or just a bit of meta-data in wc.db to get this working? And drop this meta-data again when the proper framework is in place? I guess now that we're dealing with DB schemas rather than entries files, we can make random changes to our meta-data easily as we go from one release to the next. The temorary hack would just need to be supported in read-only form later. The entire framework is the adding of the data in the db. If you start doing some details it is not necessary harder to do the rest at the same time One of the reasons that this isn't handled, is that this upgrade can't be handled atomicly before we move to single-db. (Tree conflicts would move from the parent to the child DBs, while we only upgrade them at once. So any error while upgrading would break your conflict storage) I don't think I understand what problem you're describing here. Can you elaborate? Will there still be a problem after moving the single db? Currently we take one entries file.. and turn that in one wc.db. If something fails we delete the wc.db and leave the entries file in place. The problem with tree conflicts on directories is that the conflict is stored in the parents wc.db, while in the new schema it belongs in the child. If you say the upgrade succeeded on the parent, the data is no longer in the parent wc.db. But the child is not updated yet, so the data is not in the child. It's just in memory at that time. With single-db we can have a transaction over the data move (or we move all data at once) and in both cases you can roll back. - file externals improvements Are file externals usable at all right now or are they even more broken than they were in 1.6? If they aren't more broken, could we postpone fixing them to 1.8? At least, we should ship the feature in a state that matches the 1.6 state. It's not a killer feature that many people rely on, so I don't think 1.7 should definitely wait for improvements in this area. The problem here is that we never documented how it worked in 1.6. It worked as a hack in 1.6 and as a slightly different hack now. Issues are continuously noted and more hacks are added to fix specific use cases. But is the user experience for file externals any different after upgrading to the current trunk code? If not, then we can just ship it as it is. Else, we can re-design and re-implement the feature, or fix the use cases that are broken in current trunk code, depending on which is less work. Well... since 1.6.0 some developers have tried to fix 'some' user issues.. and added if's here and there. And then another issue was found.. adding more ifs And before we get to 1.7 we add hundreds of more ifs to support an undersigned feature. And we can't say that it was similar to 1.6, because it was never designed. It probabably is: It's still an ugly hack without documentation. But the hack gets bigger and bigger, and if it continuous this way we can talk about wc-ng-ng in two years, instead of in twenty years. I'm -0 on fixing any more specific use cases until we have a proper design to support these file externals/internals/hacks. (I don't want to see 2 additional if statements added to every function in libsvn_wc and libsvn_client, just because we never designed this feature) If adding a couple of if-statements restores 1.6 behaviour and allows us to ship 1.7 faster, then I'd prefer that route. What is 1.6 behavior? And what about adding 2000 if statements? Without a design you don't know how many if's you need and there are at least more than a few in the current code and file external interaction with other features is virtually untested by our test suite. So this would probably require many more. Summary: NEEDS-DESIGN before adding more ifs. - svn patch I'd like to really almost-feature-freeze this now (the last feature
RE: Reduce the 1.7 release feature set a bit?
-Original Message- From: Bert Huijben [mailto:b...@qqmail.nl] On Behalf Of Bert Huijben Sent: zaterdag 21 augustus 2010 9:18 To: 'Stefan Sperling' Cc: dev@subversion.apache.org Subject: RE: Reduce the 1.7 release feature set a bit? Doesn't this require the new conflict store mentioned before? Not necessarily. We could simply treat the presence of a .svnpatch.rej file as a conflict marker. -1 This would slow down every status walk by statting that file suffix for every file, because we didn't store it in the DB and getting the information directly. (And we would have to do that in all future versions to keep compatibility with users removing that file to resolve the conflict) I would rather drop the entire patch support for 1.7 then going that way. (wc-ng-ng tomorrow?) And what about if that file already exists? (Normal conflicts store the location of the conflict marker to allow that situation). And Atomic (wq) behavior on creating the conflict marker? We had similar issues when creating the property reject files and this actually uses some parts of the new code to avoid these problems now. (It uses a wq operation to write out the reject marker file). The new conflict store would always allow recreating these markers from the info in the db (and pristine store). Including the first time. We should do things properly for wc-ng instead of adding new hacks that we have to support (and possibly slows us down) indefinitely. Even if this costs a few additional weeks before release. Bert
Reduce the 1.7 release feature set a bit?
I think that the recent NODE_DATA developments in wc-ng show that wc-ng is still in flux, or at least a bit further from stabilisation than we'd like it to be. That's fine! I'd much rather see us release a great working copy library than get the release out 3 months earlier. Our new focus on enterprise users doesn't demand the 6-month release cycle we once decided upon (and subsequently never carried out). Extra cycles spent on wc-ng won't hurt our users even if it means delaying 1.7 for a couple of months. A few enterprise users I've met are still on 1.4 (yes, really... send praises to Debian etch). Nevertheless, we'll need to get to a point where we can start focussing on stabilisation, to avoid a 1.5-like release debacle (if it's not already\ happening :) I think it would benefit the project a lot if anyone not involved in wc-ng focused solely on fixing release-critical issues from now on. We'll need to draw that line in the sand at some point, and we might as well draw it now. Looking over s.a.o/roadmap.html, I see a couple of things scheduled which don't seem to be strictly necessary in order to give our users a stable and functional Subversion 1.7 release that contains an awful lot of improvements over 1.6. I think that dropping all or any of these items from the roadmap would allow us to release 1.7 sooner, without much harm. - Conflict storage Do we really need to have this ready for 1.7? We might as well keep using the 1.6-style of recording conflict information in temporary files, that act as conflict markers. At the very least, doing so would not be a regression over 1.6, and it can be improved upon during the 1.8 cycle. As an aside, I'd like to make svn patch record conflict markers for rejected hunks for 1.7 if possible, and the implementation of that depends on this roadmap item. If we decide to postpone a better conflict store to 1.8, then I can make svn patch and other subcommands treat the presence of .svnpatch.rej files as a conflict marker. The sooner we can make a decision on this, the better. - Performance improvements branch I think these changes are very valuable and need to be reviewed and eventually go into trunk. But we could postpone this to 1.8 as well. Apparently we don't seem to have the cycles available right now to review them thoroughly, since not many people have done so. So why add the extra pressure during the current release cycle? - svnrdump It's marked as complete, but I don't really know yet if we can consider it complete yet. I plan to review it thoroughly again in its current state before deciding whether or not I'd like to see it in 1.7. In particular, the current test coverage seems quite poor, and I'm afraid of that hiding potential problems. However, if anyone already has reviewed its current state and has found it to be of releasable quality, please let us know. But if noone has the cycles available to review it thoroughly, and in particular to improve test coverage, I'd say we should postpone it to 1.8. - file externals improvements Are file externals usable at all right now or are they even more broken than they were in 1.6? If they aren't more broken, could we postpone fixing them to 1.8? At least, we should ship the feature in a state that matches the 1.6 state. It's not a killer feature that many people rely on, so I don't think 1.7 should definitely wait for improvements in this area. - atomic revprops This is still work-in-progress, and I'm not sure how far we are from completing it. The real problem it's trying to solve (race in svnsync) can be worked around reliably using file locks to synchronise svnsync processes. I think we can leave judgement to danielsh over whether this feature is easy enough to get ready for 1.7, or whether any additional time spent on it should rather be spent on more release-critical items. Should we not fix this for 1.7, the svnbook should be temporarily amended to document the problem and the workaround (maybe we should already have done so...) I think all the above are great features, and should eventually be released. All I'm asking is whether we should try to narrow our focus a bit more in order to get 1.7 shipped as early as possible (hopefully still this year). The less features we have to test and review, the faster we can release. Obviously, I myself cannot make binding decisions over any of these items, so please share your thoughts. Some random notes on other roadmap items while here: - svn patch I'd like to really almost-feature-freeze this now (the last feature freeze was before danna's Gsoc). The only feature I'd still like to add is conflict recording (an exception dannas and I had in mind during the pre-gsoc freeze, too). - libsvn_ra_serf stabilization I consider this one very release critical, because HTTPv2 depends on it. It would be a shame to not ship HTTPv2, which is basically done, because of
Re: Reduce the 1.7 release feature set a bit?
On Fri, Aug 20, 2010 at 3:29 AM, Stefan Sperling s...@elego.de wrote: just me): http://subversion.tigris.org/issues/show_bug.cgi?id=3684 Lieven fixed this in serf trunk a few months back. IOW, I can reproduce with serf 0.6.1, but not serf trunk. Greg/Lieven: shall we cut a new release? Greg did the last release, so I would prefer he do it. I'll also go figure out what needs to be backported in ra_serf from trunk to 1.6.x to work against current serf releases. -- justin
Re: [serf-dev] Re: Reduce the 1.7 release feature set a bit?
On Fri, Aug 20, 2010 at 11:19 AM, Justin Erenkrantz jus...@erenkrantz.com wrote: On Fri, Aug 20, 2010 at 11:15 AM, Lieven Govaerts lieven.govae...@gmail.com wrote: The memory usage problem is probably still there. Last time I looked at it with valgrind (at the elego days), most of the memory was allocated in the sqlite libraries. This has probably improved a lot with the single-db move, but I haven't tested yet. FWIW, I didn't see the memory go above ~48MB on my Mac when checking out SVN trunk with subversion serf trunk. So, I'm not tempted to pursue it further at this time. Getting ra_serf in 1.6.x happy with current versions of serf is more important (to me). =) -- justin 1.6.x + r879757 backport group now works with serf trunk again. All 'make check' tests (via davautocheck.sh) pass except: FAIL: merge_tests.py 133: replace that causes a tree-conflict Considering everything else passes, I'll go out on a limb and say that's likely not due to the serf version. (I'm guessing it's a known failure??) Greg will likely cut a Serf release this weekend that will incorporate r1401 from serf that will let the old-style authentication handlers in 1.6.x-era ra_serf work. I guess it'll either be 0.7.0 or 0.6.2 - not sure what his plans are - I just validated against serf trunk. (If we didn't incorporate r1401 in the new serf release, we'd have to backport all of the authn changes in ra_serf from trunk and I'd rather not do that. ra_serf in SVN trunk already uses the new serf authn framework.) BTW, those who complain that no one's working on ra_serf hasn't looked at the diffs between 1.6.x and trunk. New authn framework, digest auth, Kerberos auth, error handling fixes, better proxy support, HTTPv2 support, etc. Many many kudos to everyone driving ra_serf forward! *group hug* -- justin
