Re: Cannot add files due to hierarchical RDC svn:auto-props when matching rules merge properties between text and binary file types

[Johan Corveleyn]

On Sat, Oct 15, 2016 at 10:10 PM, Johan Corveleyn  wrote:
> On Fri, Oct 14, 2016 at 4:21 PM, Branko Čibej  wrote:
>> On 14.10.2016 09:17, Johan Corveleyn wrote:
>>> FWIW, I think you have a valid usecase, though it's a bit unusual. Of
>>> your proposed solutions, I think 1 and 2 are not possible (1 breaks
>>> backwards compatibility
>>
>> On the other hand, I'd argue that the behaviour as described is a bug
>> and that 1 is actually correct: an identical rule (identical glob
>> pattern) in svn:auto-props property should completely override the rule
>> from an inherited svn:auto-props property. Anything else leads to
>> madness, as this thread shows.
>
> Hm, yes, maybe you're right. In case of identical glob pattern in
> svn:auto-props perhaps, yes.
>
> Rereading the relevant section of the release notes [1], perhaps it
> was even explicitly designed to behave that way, but maybe there is a
> bug in the implementation. The release notes say:
>
> [[[
> Patterns defined in svn:auto-props property override any identical
> patterns in the auto-props runtime config. When multiple
> svn:auto-props properties apply to a file, the pattern from the
> nearest inheritable property takes precedence. See this section of
> design wiki for a full explanation.
> ]]]
>
> FTR: I haven't yet tried to reproduce the problem Rob was seeing.
> Maybe I'll try later.
>
> [1] 
> http://subversion.apache.org/docs/release-notes/1.8.html#repos-dictated-config
>

Well, it reproduces allright, both with 1.9.4 and with a trunk client:

[[[
C:\autoprops>svn --version -q
1.9.4-SlikSvn

C:\autoprops>svnadmin create repos

C:\autoprops>svn mkdir -mm --parents file:///c:/autoprops/repos/trunk/dir
Committing transaction...
Committed revision 1.

C:\autoprops>svn co file:///c:/autoprops/repos/ wc
Awc\trunk
Awc\trunk\dir
Checked out revision 1.

C:\autoprops>cd wc

C:\autoprops\wc>svn ps svn:auto-props "*.txt = svn:eol-style=native" .
property 'svn:auto-props' set on '.'

C:\autoprops\wc>svn ps svn:auto-props "*.txt =
svn:mime-type=application/octet-stream" trunk
property 'svn:auto-props' set on 'trunk'

C:\autoprops\wc>svn st
 M  .
 M  trunk

C:\autoprops\wc>svn ci -mm
Sending.
Sendingtrunk
Committing transaction...
Committed revision 2.

C:\autoprops\wc>svn up
Updating '.':
At revision 2.

C:\autoprops\wc>cd trunk/dir

C:\autoprops\wc\trunk\dir>svn pg svn:auto-props --show-inherited-props
C:\autoprops\wc - *.txt = svn:eol-style=native

C:\autoprops\wc\trunk - *.txt = svn:mime-type=application/octet-stream


C:\autoprops\wc\trunk\dir>echo test>test.txt

C:\autoprops\wc\trunk\dir>svn add test.txt
svn: E29: Can't set 'svn:eol-style': file
'C:\autoprops\wc\trunk\dir\test.txt' has binary mime type property
]]]

(same error when I execute the last command with a trunk client)

-- 
Johan

Re: Cannot add files due to hierarchical RDC svn:auto-props when matching rules merge properties between text and binary file types

[Johan Corveleyn]

On Fri, Oct 14, 2016 at 4:21 PM, Branko Čibej  wrote:
> On 14.10.2016 09:17, Johan Corveleyn wrote:
>> FWIW, I think you have a valid usecase, though it's a bit unusual. Of
>> your proposed solutions, I think 1 and 2 are not possible (1 breaks
>> backwards compatibility
>
> On the other hand, I'd argue that the behaviour as described is a bug
> and that 1 is actually correct: an identical rule (identical glob
> pattern) in svn:auto-props property should completely override the rule
> from an inherited svn:auto-props property. Anything else leads to
> madness, as this thread shows.

Hm, yes, maybe you're right. In case of identical glob pattern in
svn:auto-props perhaps, yes.

Rereading the relevant section of the release notes [1], perhaps it
was even explicitly designed to behave that way, but maybe there is a
bug in the implementation. The release notes say:

[[[
Patterns defined in svn:auto-props property override any identical
patterns in the auto-props runtime config. When multiple
svn:auto-props properties apply to a file, the pattern from the
nearest inheritable property takes precedence. See this section of
design wiki for a full explanation.
]]]

FTR: I haven't yet tried to reproduce the problem Rob was seeing.
Maybe I'll try later.

[1] 
http://subversion.apache.org/docs/release-notes/1.8.html#repos-dictated-config

-- 
Johan

Re: Cannot add files due to hierarchical RDC svn:auto-props when matching rules merge properties between text and binary file types

[Branko Čibej]

On 14.10.2016 09:17, Johan Corveleyn wrote:
> FWIW, I think you have a valid usecase, though it's a bit unusual. Of
> your proposed solutions, I think 1 and 2 are not possible (1 breaks
> backwards compatibility

On the other hand, I'd argue that the behaviour as described is a bug
and that 1 is actually correct: an identical rule (identical glob
pattern) in svn:auto-props property should completely override the rule
from an inherited svn:auto-props property. Anything else leads to
madness, as this thread shows.

I wouldn't worry about backwards-compatibility in this case, for the
simple reason that cases where things brake should be quite rare.
Witness that this is the first bug report on this topic (that I know
of). Furthermore, it's relatively easy to fix any cases that happened to
rely on this buggy behaviour.

I'd even say that the whole property value should override any inherited
ones ... but that would be harder to justify, given current behaviour.

-- Brane

Re: Cannot add files due to hierarchical RDC svn:auto-props when matching rules merge properties between text and binary file types

[Rob Hofer]

Hello Johan,

Separating the binary *.v files from the text based *.v files into separate
directories is exactly what we are doing. The hope was that, with the
introduction of inherited properties (svn:auto-props), we could setup a
different set of auto-props rules for just that one low-level directory
where *.v would mean something "non-standard" from a global/default
auto-props rule defined on the root of the repository.

There are some other file extensions we run into also, such as *.doc (MS
Word document/binary vs MS-DOS "document" (just a text file)), and also
*.lib, which is a common ASIC standard-cell library format (text file), vs
a statically linked library file (binary file).

Interestingly enough, the use case for *.doc works today as is, because the
global rule (default case) is binary, and the exception is a text file.
This works because on the lower level directory where we want *.doc to be a
text file, the mime-type override works, and the other properties such as
svn:eol-style and svn:keywords are being *added*, so the desired effect is
achieved. In the opposite case, where the higher level rule is a text file
(with properties like eol-style and keywords), the lower level rule
(binary) does not work because while the mime-type can be overridden, those
other properties cannot be blocked or removed from the upstream rule. And
then 'svn add' fails because combining either property (eol-style or
keywords) with a binary file results in a client error (and I agree that
behavior should probably remain the same as it is today).

Since the auto-props mechanism does not have a subtractive method of
combining multiple auto-props rule matches, it seems to me that the easiest
path to add such a feature is to define a new value to those existing
properties, such as 'none' or 'ignore' so that it is still additive in
nature to existing auto-props rules (thereby a svn:eol-style=none would
override svn:eol-style=Native). This also has the benefit outside this
particular text vs binary use case - where someone may want to disable
keyword substitution on a lower level directory but currently there is no
way to do that other than disabling auto-props entirely. And today that is
our current workaround for this issue - we simply have to add these files
with --no-auto-props and then add the correct properties manually. With the
introduction of inherited properties though, it seems this can be fully
automated.

I will submit the suggestion / feature request to your issue tracker. I
understand that doing so is no guarantee it will ever be implemented.

Sincerely,
Rob


*--Rob HoferSenior Electrical EngineerGS Communication Products
fpga319-295-4541rob.ho...@rockwellcollins.com
rockwellcollins.com
*

Any Export License Required Information and License Restricted
Third Party Intellectual Property (TPIP) content must be encrypted
and sent to rob.ho...@corp.rockwellcollins.com.

On Fri, Oct 14, 2016 at 2:17 AM, Johan Corveleyn  wrote:

> On Thu, Oct 13, 2016 at 2:15 AM, Eric Johnson  wrote:
> > Your constraints, as currently specified, seem to require actual
> logic
> >
> > Thoughts follow your email.
> >
> >
> > On 10/12/16 1:44 PM, Rob Hofer wrote:
> >>
> >> We have a rather common use case where we have an svn:auto-props rule
> set
> >> globally (set on root of repository) to define source code files as text
> >> based, but also have some files provided by 3rd parties which compress
> or
> >> encrypt similar files with the same file extension (which we have no
> control
> >> over), and hence we want to set an svn:auto-props rule locally on those
> >> directories to make those files binary type. But the hierarchical
> >> svn:auto-props rules add properties from multiple definitions of the
> same
> >> match filter (*.v), and result is a conflicting set of properties that
> block
> >> the add at the client (eol-style with mime-type=octet-stream).
> >>
> >> For example, a binary and text based Verilog file (*.v):
> >> %> file binary.v
> >> binary.v: gzip compressed data, was "binary.v", from Unix, last
> modified:
> >> Mon Feb 18 19:44:25 2013, max compression
> >> %> file text.v
> >> text.v: ASCII text
> >>
> >> The RDC auto-props for this directory shows these Verilog and VHDL types
> >> (local directory expects binary types, global are source-code text
> files).
> >> %> svn propget svn:auto-props --show-inherited-props .
> >> http://crsvn/gsadc - *.v  = svn:eol-style=LF;svn:keywords=Id
> Revision
> >> Date Author URL;svn:mime-type=text/x-verilog
> >> . - *.v   = svn:needs-lock;svn:mime-type=application/octet-stream
> >>
> >> Adding the files to SVN control fails, unless --no-auto-props is used
> >> (undesirable work around):
> >> %> svn add binary.v
> >> svn: E29: Can't set 'svn:eol-style': file '/module/lay/binary.v' has
> >> binary mime type property
> >> %> svn add --no-auto-props binary.v
> >> A  (bin)  

Re: Cannot add files due to hierarchical RDC svn:auto-props when matching rules merge properties between text and binary file types

[Johan Corveleyn]

On Thu, Oct 13, 2016 at 2:15 AM, Eric Johnson  wrote:
> Your constraints, as currently specified, seem to require actual logic
>
> Thoughts follow your email.
>
>
> On 10/12/16 1:44 PM, Rob Hofer wrote:
>>
>> We have a rather common use case where we have an svn:auto-props rule set
>> globally (set on root of repository) to define source code files as text
>> based, but also have some files provided by 3rd parties which compress or
>> encrypt similar files with the same file extension (which we have no control
>> over), and hence we want to set an svn:auto-props rule locally on those
>> directories to make those files binary type. But the hierarchical
>> svn:auto-props rules add properties from multiple definitions of the same
>> match filter (*.v), and result is a conflicting set of properties that block
>> the add at the client (eol-style with mime-type=octet-stream).
>>
>> For example, a binary and text based Verilog file (*.v):
>> %> file binary.v
>> binary.v: gzip compressed data, was "binary.v", from Unix, last modified:
>> Mon Feb 18 19:44:25 2013, max compression
>> %> file text.v
>> text.v: ASCII text
>>
>> The RDC auto-props for this directory shows these Verilog and VHDL types
>> (local directory expects binary types, global are source-code text files).
>> %> svn propget svn:auto-props --show-inherited-props .
>> http://crsvn/gsadc - *.v  = svn:eol-style=LF;svn:keywords=Id Revision
>> Date Author URL;svn:mime-type=text/x-verilog
>> . - *.v   = svn:needs-lock;svn:mime-type=application/octet-stream
>>
>> Adding the files to SVN control fails, unless --no-auto-props is used
>> (undesirable work around):
>> %> svn add binary.v
>> svn: E29: Can't set 'svn:eol-style': file '/module/lay/binary.v' has
>> binary mime type property
>> %> svn add --no-auto-props binary.v
>> A  (bin)  binary.v
>> %> svn add text.v
>> svn: E29: Can't set 'svn:eol-style': file '/module/lay/text.v' has
>> binary mime type property
>> %> svn add --no-auto-props text.v
>> A text.v
>>
>> Subversion auto-detects the binary file and at least sets the mime-type,
>> but other properties are missing because --no-auto-props is the only way to
>> add the files without error.
>> %> svn proplist -v binary.v
>> Properties on 'binary.v':
>>   svn:mime-type
>> application/octet-stream
>> %> svn proplist -v text.v
>>
>> %> svn --version
>> svn, version 1.9.4 (r1740329)
>>compiled May 18 2016, 12:05:49 on x86_64-unknown-linux-gnu
>>
>> (Incidentally, the commit will fail because our hook is checking these
>> svn:auto-props rules and the file must match the binary rules or the text
>> rule, based on the mime-type). So the only way today to add these files to
>> SVN is to use --no-auto-props on add, and go into the server and disable the
>> pre-commit hook during the commit, then put the pre-commit hook back.  Since
>> a pre-commit hook is the only way to enforce the use of auto-props
>> correctly, disabling the hook is not an optimal solution. Once added to SVN,
>> the issue never comes up again because the properties do not change, and the
>> pre-commit hook checks the modifications on future commits based upon the
>> mime-type (binary or text rules). The issue is only during the initial add
>> to SVN due to conflicting properties being applied to the file based on how
>> the svn:auto-props definitions are being interpreted.
>>
>> Proposed solution:
>> 1. Make lower level svn:auto-props rules completely override upstream
>> ones, rather than additively merging properties, for rules with same exact
>> match filter (local *.v redefines upstream *.v completely).
>> 2. Make SVN ignore properties such as eol-style and keywords when the
>> mime-type is a binary type rather than issue a fatal error to the
>> user/client (warning instead that some properties are being ignored).
>> 3. Provide a subtractive property rule to undo an upstream property. E.g.,
>> svn:eol-style=none, or svn:keyword=none, such that a lower level rule can
>> unset a property defined upstream (and make svn:eol-style=none behave same
>> as if svn:eol-style was not applied at all).
>>
>> Note: Before RDC svn:auto-props (pre 1.8), this use case required having
>> two entries in the ~/.subversion/config, with one always commented out. When
>> encountering one type or the other (text or binary), user would have to
>> uncomment/comment the proper auto-props rule in their config file before the
>> add, and then change their config back for the normal case. This was very
>> unwieldly and required careful synchronization of all user runtime config
>> files and hook scripts and manual intervention by the user during adds.
>> Hierarchical RDC properties should eliminate this problem, but it falls a
>> little short because of how hierarchical RDC svn:auto-props rules merge
>> mutually exclusive properties together. I believe this is very similar to
>> multiple matches in the ~/.subversion/config runtime file, for example a *.v
>> rule and a 

Re: Cannot add files due to hierarchical RDC svn:auto-props when matching rules merge properties between text and binary file types

[Eric Johnson]

Your constraints, as currently specified, seem to require actual logic

Thoughts follow your email.

On 10/12/16 1:44 PM, Rob Hofer wrote:
We have a rather common use case where we have an svn:auto-props rule 
set globally (set on root of repository) to define source code files 
as text based, but also have some files provided by 3rd parties which 
compress or encrypt similar files with the same file extension (which 
we have no control over), and hence we want to set an svn:auto-props 
rule locally on those directories to make those files binary type. But 
the hierarchical svn:auto-props rules add properties from multiple 
definitions of the same match filter (*.v), and result is a 
conflicting set of properties that block the add at the client 
(eol-style with mime-type=octet-stream).


For example, a binary and text based Verilog file (*.v):
%> file binary.v
binary.v: gzip compressed data, was "binary.v", from Unix, last 
modified: Mon Feb 18 19:44:25 2013, max compression

%> file text.v
text.v: ASCII text

The RDC auto-props for this directory shows these Verilog and VHDL 
types (local directory expects binary types, global are source-code 
text files).

%> svn propget svn:auto-props --show-inherited-props .
http://crsvn/gsadc - *.v  = svn:eol-style=LF;svn:keywords=Id 
Revision Date Author URL;svn:mime-type=text/x-verilog

. - *.v   = svn:needs-lock;svn:mime-type=application/octet-stream

Adding the files to SVN control fails, unless --no-auto-props is used 
(undesirable work around):

%> svn add binary.v
svn: E29: Can't set 'svn:eol-style': file '/module/lay/binary.v' 
has binary mime type property

%> svn add --no-auto-props binary.v
A  (bin)  binary.v
%> svn add text.v
svn: E29: Can't set 'svn:eol-style': file '/module/lay/text.v' has 
binary mime type property

%> svn add --no-auto-props text.v
A text.v

Subversion auto-detects the binary file and at least sets the 
mime-type, but other properties are missing because --no-auto-props is 
the only way to add the files without error.

%> svn proplist -v binary.v
Properties on 'binary.v':
  svn:mime-type
application/octet-stream
%> svn proplist -v text.v

%> svn --version
svn, version 1.9.4 (r1740329)
   compiled May 18 2016, 12:05:49 on x86_64-unknown-linux-gnu

(Incidentally, the commit will fail because our hook is checking these 
svn:auto-props rules and the file must match the binary rules or the 
text rule, based on the mime-type). So the only way today to add these 
files to SVN is to use --no-auto-props on add, and go into the server 
and disable the pre-commit hook during the commit, then put the 
pre-commit hook back.  Since a pre-commit hook is the only way to 
enforce the use of auto-props correctly, disabling the hook is not an 
optimal solution. Once added to SVN, the issue never comes up again 
because the properties do not change, and the pre-commit hook checks 
the modifications on future commits based upon the mime-type (binary 
or text rules). The issue is only during the initial add to SVN due to 
conflicting properties being applied to the file based on how the 
svn:auto-props definitions are being interpreted.


Proposed solution:
1. Make lower level svn:auto-props rules completely override upstream 
ones, rather than additively merging properties, for rules with same 
exact match filter (local *.v redefines upstream *.v completely).
2. Make SVN ignore properties such as eol-style and keywords when the 
mime-type is a binary type rather than issue a fatal error to the 
user/client (warning instead that some properties are being ignored).
3. Provide a subtractive property rule to undo an upstream property. 
E.g., svn:eol-style=none, or svn:keyword=none, such that a lower level 
rule can unset a property defined upstream (and make 
svn:eol-style=none behave same as if svn:eol-style was not applied at 
all).


Note: Before RDC svn:auto-props (pre 1.8), this use case required 
having two entries in the ~/.subversion/config, with one always 
commented out. When encountering one type or the other (text or 
binary), user would have to uncomment/comment the proper auto-props 
rule in their config file before the add, and then change their config 
back for the normal case. This was very unwieldly and required careful 
synchronization of all user runtime config files and hook scripts and 
manual intervention by the user during adds.  Hierarchical RDC 
properties should eliminate this problem, but it falls a little short 
because of how hierarchical RDC svn:auto-props rules merge mutually 
exclusive properties together. I believe this is very similar to 
multiple matches in the ~/.subversion/config runtime file, for example 
a *.v rule and a *-rtl.v rule could collide, except now it is possible 
to have the very same rule filter (*.v) defined in more than one 
location in the subversion hierarchy. See proposed solution #1 as my 
desired behavior from the SVN client.



Only a few options I see:

 - different 

Cannot add files due to hierarchical RDC svn:auto-props when matching rules merge properties between text and binary file types

[Rob Hofer]

We have a rather common use case where we have an svn:auto-props rule set
globally (set on root of repository) to define source code files as text
based, but also have some files provided by 3rd parties which compress or
encrypt similar files with the same file extension (which we have no
control over), and hence we want to set an svn:auto-props rule locally on
those directories to make those files binary type. But the hierarchical
svn:auto-props rules add properties from multiple definitions of the same
match filter (*.v), and result is a conflicting set of properties that
block the add at the client (eol-style with mime-type=octet-stream).

For example, a binary and text based Verilog file (*.v):
%> file binary.v
binary.v: gzip compressed data, was "binary.v", from Unix, last modified:
Mon Feb 18 19:44:25 2013, max compression
%> file text.v
text.v: ASCII text

The RDC auto-props for this directory shows these Verilog and VHDL types
(local directory expects binary types, global are source-code text files).
%> svn propget svn:auto-props --show-inherited-props .
http://crsvn/gsadc - *.v  = svn:eol-style=LF;svn:keywords=Id Revision
Date Author URL;svn:mime-type=text/x-verilog
. - *.v   = svn:needs-lock;svn:mime-type=application/octet-stream

Adding the files to SVN control fails, unless --no-auto-props is used
(undesirable work around):
%> svn add binary.v
svn: E29: Can't set 'svn:eol-style': file '/module/lay/binary.v' has
binary mime type property
%> svn add --no-auto-props binary.v
A  (bin)  binary.v
%> svn add text.v
svn: E29: Can't set 'svn:eol-style': file '/module/lay/text.v' has
binary mime type property
%> svn add --no-auto-props text.v
A text.v

Subversion auto-detects the binary file and at least sets the mime-type,
but other properties are missing because --no-auto-props is the only way to
add the files without error.
%> svn proplist -v binary.v
Properties on 'binary.v':
  svn:mime-type
application/octet-stream
%> svn proplist -v text.v

%> svn --version
svn, version 1.9.4 (r1740329)
   compiled May 18 2016, 12:05:49 on x86_64-unknown-linux-gnu

(Incidentally, the commit will fail because our hook is checking these
svn:auto-props rules and the file must match the binary rules or the text
rule, based on the mime-type). So the only way today to add these files to
SVN is to use --no-auto-props on add, and go into the server and disable
the pre-commit hook during the commit, then put the pre-commit hook back.
Since a pre-commit hook is the only way to enforce the use of auto-props
correctly, disabling the hook is not an optimal solution. Once added to
SVN, the issue never comes up again because the properties do not change,
and the pre-commit hook checks the modifications on future commits based
upon the mime-type (binary or text rules). The issue is only during the
initial add to SVN due to conflicting properties being applied to the file
based on how the svn:auto-props definitions are being interpreted.

Proposed solution:
1. Make lower level svn:auto-props rules completely override upstream ones,
rather than additively merging properties, for rules with same exact match
filter (local *.v redefines upstream *.v completely).
2. Make SVN ignore properties such as eol-style and keywords when the
mime-type is a binary type rather than issue a fatal error to the
user/client (warning instead that some properties are being ignored).
3. Provide a subtractive property rule to undo an upstream property. E.g.,
svn:eol-style=none, or svn:keyword=none, such that a lower level rule can
unset a property defined upstream (and make svn:eol-style=none behave same
as if svn:eol-style was not applied at all).

Note: Before RDC svn:auto-props (pre 1.8), this use case required having
two entries in the ~/.subversion/config, with one always commented out.
When encountering one type or the other (text or binary), user would have
to uncomment/comment the proper auto-props rule in their config file before
the add, and then change their config back for the normal case. This was
very unwieldly and required careful synchronization of all user runtime
config files and hook scripts and manual intervention by the user during
adds.  Hierarchical RDC properties should eliminate this problem, but it
falls a little short because of how hierarchical RDC svn:auto-props rules
merge mutually exclusive properties together. I believe this is very
similar to multiple matches in the ~/.subversion/config runtime file, for
example a *.v rule and a *-rtl.v rule could collide, except now it is
possible to have the very same rule filter (*.v) defined in more than one
location in the subversion hierarchy. See proposed solution #1 as my
desired behavior from the SVN client.

Sincerely,
Rob


*--Rob HoferSenior Electrical EngineerGS Communication Products
fpga319-295-4541rob.ho...@rockwellcollins.com
rockwellcollins.com
*

Any Export License Required