Re: Plumbing work to make shelving complete and robust

2018-08-24 Thread Nathan Hartman
On Fri, Aug 24, 2018 at 12:26 PM Julian Foad  wrote:

> Only by implementing more than one edit driver for each editor (and vice
> versa) can we prove that the components are cleanly separated by an API and
> thus re-usable.
>
> Then there is the need for a test framework that can generate all
> different combinations of changes and test each of the subsystems with all
> of those changes.
>
> These improvements are not just to benefit shelving. In the longer term,
> if we are to make a major improvement like supporting moves/renames
> properly, in my opinion we need to do it starting from a baseline where we
> consistently and cleanly use APIs that encode the current semantics of
> versioned changes, and then migrate those.
>
>
I'm not expert enough on Subversion internals to give in-depth
technical feedback here, but overall this refactoring sounds like a
good game plan, as it will improve consistency and testability, and
it sounds like it will make it possible (and/or easier) to
implement all sorts of nifty new features in the long run. Not only
should this benefit shelving, but checkpointing (which in my view
seems like an extended version of shelving) should benefit
significantly as well.


Re: Plumbing work to make shelving complete and robust

2018-08-24 Thread Julian Foad
Julian Foad wrote:
> [...] There are 
> currently two implementations of a "dump" editor and neither is 
> immediately usable. The one in "svnadmin dump" is not written as a clean 
> editor, and instead calls directly into the FS layer to fetch its data. 
> The one in "svnrdump" is clean but does not support all the features of 
> the other such as non-delta mode and verify mode.

For connecting to the WC and to shelving, it is not a problem that the 
"svnrdump" dump editor lacks support for those features. It's a problem that 
the implementation is located in the "svnrdump" executable rather than in a 
library, but that is easy to overcome.

>   => We need to unify these two implementations, to have a clean re-
> usable dump editor.

So moving the "svnrdump" implementation into the library can be the first step 
(done: r1838835), and unifying can be the second step.

The attached patch 'unifying-dump-editors-1.patch' goes some way towards that.

> The delta edit driver in "svnadmin load" also does not seem to be shared 
> with the one in "svnrdump load".
> 
>   => We need to unify these two implementations, to have a re-usable 
> load edit-driver.
> 
> To be clear, re-using the dump/load functionality is not a direct 
> requirement for shelving; rather, it is an existing functionality that 
> supports input and output of shelvable changes. By providing an input 
> and output mechanism that can be attached to various APIs (repository, 
> WC, and shelves) it is useful for testing that the APIs all work 
> consistently.

Only by implementing more than one edit driver for each editor (and vice versa) 
can we prove that the components are cleanly separated by an API and thus 
re-usable.

Then there is the need for a test framework that can generate all different 
combinations of changes and test each of the subsystems with all of those 
changes.

These improvements are not just to benefit shelving. In the longer term, if we 
are to make a major improvement like supporting moves/renames properly, in my 
opinion we need to do it starting from a baseline where we consistently and 
cleanly use APIs that encode the current semantics of versioned changes, and 
then migrate those.

-- 
- Julian
Towards unifying the two repository dump editor implementations.

('Dump editor' here means a dump-stream writer driven through the
svn_delta_editor_t API.)

The older dump editor in 'dump.c' supports 'svnadmin dump' and 'svnadmin
verify'. The newer dump editor in 'dump_editor.c' supports 'svnrdump dump'.
The latter can only produce delta-mode dumps.

This patch extracts some of the ancillary functionality (some kinds of
verification) out of the original dump editor into a separate wrapper, and
applies this verification wrapper to either the original dump editor or the
newer one. This brings us closer to being able to use the newer dump editor
as a drop-in replacement for the old one.

  ### So far, only 2 of the 3 checks performed in dump mode are moved to
  the new checking wrapper: fspath and copyfrom, but not mergeinfo.

* subversion/libsvn_repos/dump.c
  (check_mergeinfo_revisions,
   verify_fspath,
   check_for_old_copy_reference): New, extracted from dump_node().
  (edit_baton,
   dump_node,
   get_dump_editor): Remove the three checks that are moved to the wrapper,
and their related parameters.
  (rc_*,
   get_ref_checking_editor): New: a reference-checking wrapper editor
implementing those three checks.
  (get_dump_only_editor): New function. Wraps the basic editor with the new
reference-checking wrapper editor.
### For now, calls the newer dump editor in 'deltas' mode, else the
  older one; thus in deltas mode the mergeinfo revs check is missing.
  (svn_repos_dump_fs4): Call get_dump_only_editor() instead of
get_gump_editor().
  (get_verify_editor): New function, wrapping get_dump_editor().
  (verify_one_revision): Call get_verify_editor() instead of
get_gump_editor().

--This line, and those below, will be ignored--

Index: subversion/libsvn_repos/dump.c
===
--- subversion/libsvn_repos/dump.c	(revision 1838836)
+++ subversion/libsvn_repos/dump.c	(working copy)
@@ -651,16 +651,12 @@ struct edit_baton
   /* True if checking UCS normalization during a verify. */
   svn_boolean_t check_normalization;
 
   /* The first revision dumped in this dumpstream. */
   svn_revnum_t oldest_dumped_rev;
 
-  /* If not NULL, set to true if any references to revisions older than
- OLDEST_DUMPED_REV were found in the dumpstream. */
-  svn_boolean_t *found_old_reference;
-
   /* If not NULL, set to true if any mergeinfo was dumped which contains
  revisions older than OLDEST_DUMPED_REV. */
   svn_boolean_t *found_old_mergeinfo;
 
   /* Structure allows us to verify the paths currently being dumped.
  If NULL, validity checks are being skipped. */
@@ -874,12 +870,43 @@ verify_mergeinfo_revisions(svn_boolean_t
 

Buildbots: require Python-SQLite v3.8.2 [was: FSFS recovery should prune rep-cache even if its use is disabled]

2018-08-24 Thread Julian Foad

Brane, this one's yours:

  https://ci.apache.org/buildslaves/svn-x64-macosx-dgvrs

The new tests for #4077 are failing on at least this buildbot-slave.

  "Can't read rep-cache schema 2 using old Python-SQLite version (3, 7, 
13) < (3,8,2)"


(danielsh says: the magic number 3.8.2 is the minimum version of SQLite
that trunk can be built with.)

Can you upgrade it please?

--
- Julian


Re: FSFS recovery should prune rep-cache even if its use is disabled

2018-08-24 Thread Daniel Shahaf
Julian Foad wrote on Fri, 24 Aug 2018 12:17 +0100:
> Julian Foad:
> > Committed in http://svn.apache.org/r1838813
> 
> And nominated for backport to 1.10.x and 1.9.x.
> 
> The tests are failing on some buildbots:
> 
>  "Can't read rep-cache schema 2 using old Python-SQLite version (3, 7, 
> 13) < (3,8,2)"
> 

Clarification: the magic number 3.8.2 is the minimum version of SQLite
that trunk can be built with.

> If we're using SQLite and want to have tests written in Python, would it 
> not be reasonable to require a version of Python-SQLite that can read 
> it?
> 
> Can someone upgrade the buildbots? Brane, this one's yours: 
> https://ci.apache.org/buildslaves/svn-x64-macosx-dgvrs


Re: FSFS recovery should prune rep-cache even if its use is disabled

2018-08-24 Thread Julian Foad
Julian Foad:
> Committed in http://svn.apache.org/r1838813

And nominated for backport to 1.10.x and 1.9.x.

The tests are failing on some buildbots:

 "Can't read rep-cache schema 2 using old Python-SQLite version (3, 7, 13) < 
(3,8,2)"

If we're using SQLite and want to have tests written in Python, would it not be 
reasonable to require a version of Python-SQLite that can read it?

Can someone upgrade the buildbots? Brane, this one's yours: 
https://ci.apache.org/buildslaves/svn-x64-macosx-dgvrs

-- 
- Julian


Re: FSFS recovery should prune rep-cache even if its use is disabled

2018-08-24 Thread Julian Foad
Daniel Shahaf wrote:
> Julian Foad wrote on Thu, Aug 23, 2018 at 10:21:17 +0100:
> > +++ subversion/libsvn_fs_fs/recovery.c  (working copy)
> > @@ -468,15 +468,15 @@ recover_body(void *baton, apr_pool_t *po
> >/* Prune younger-than-(newfound-youngest) revisions from the rep
> > - cache if sharing is enabled taking care not to create the cache
> > - if it does not exist. */
> > -  if (ffd->rep_sharing_allowed)
> > + cache, no matter whether sharing is currently enabled, taking care
> > + not to create the cache if it does not exist. */
> > +  if (ffd->format >= SVN_FS_FS__MIN_REP_SHARING_FORMAT)
> 
> Looks good to me: that should fix both #4077 and #4214.

Committed in http://svn.apache.org/r1838813

I added tests there for both cases (enabled, disabled).

> I would only suggest expanding the comment [...]

Looks good to me, so I included your text.

Thanks.
- Julian