Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-06 Thread Edward d'Auvergne 
On 4 October 2015 at 13:32, Stefan Sperling  wrote:
> On Sun, Oct 04, 2015 at 12:52:33PM +0200, Edward d'Auvergne wrote:
>> I would maybe suggest introducing an option for disabling the entire
>> automatic property subsystem, i.e. it combines both of these, and
>> overrides them.  This is an interesting thought experiment - devise
>> any name for such a top level automatic property override - and this
>> should convincingly demonstrate the confusion that the current setting
>> names induces.
>
> I think that would just make the situation worse by adding to
> the current mess.
>
> I agree the documentation needs to be improved. Would you have time
> to update the documentation and send patches for it?

Ok, I can look into it.  Hpwever the instructions for setting the
MAGIC environmental variable needs some discussion (see below).


>> The problem with the svn:mime-type block is that for svn clients <
>> 1.9, 'svn import' can never work.  Specifically because the libmagic
>> layer overwrites any user decision to disable automatic properties.
>
> There's another trick to disable libmagic which works with all
> client versions. I just tested it with 1.8.
>
> Set the MAGIC environment variable to a file which does not exist.
>
>   env MAGIC=/nonexistent svn import ...
>
> This effectively disabled libmagic support.

I've tried all combinations and nothing works (on my Linux system with
svn 1.8.14).  For example:

$ env MAGIC=/nonexistent svn import ...
$ MAGIC=/nonexistent svn import ...
[block]

$ env MAGIC=False svn import ...
$ MAGIC=False svn import ...
[block]

$ export MAGIC=/nonexistent
$ svn import ...
[block]

$ export MAGIC=False
$ svn import ...
[block]

$ touch empty_magic
$ MAGIC=$BASE_DIR/empty_magic svn import ...
[block]

$ env MAGIC=$BASE_DIR/empty_magic svn import ...
[block]

$ head -n 1 /usr/share/mime/magic > empty_magic
$ MAGIC=$BASE_DIR/empty_magic svn import ...
[block]

Not a single one of these combinations work - svn:mime-type is always
being set by 'svn import'.  Therefore blocking svn:mime-type in a
pre-commit hook completely kills the option of using 'svn import', and
there is no way to avoid this.  Looking more carefully at what is
happening:

$ cd $BASE_DIR
$ cp dash/Images/DashBoot_logo_Mod.png test4
$ cp dash/dash-set.xml test4
$ cd test4
$ svn add DashBoot_logo_Mod.png
$ svn add dash-set.xml
$ svn plist -R
Properties on 'DashBoot_logo_Mod.png':
  svn:mime-type
$ svn pget svn:mime-type DashBoot_logo_Mod.png
application/octet-stream

So setting MAGIC is having an effect, but Subversion is falling back,
probably via a non-magic internal code path, to the default
svn:mime-type of "application/octet-stream" for PNG and some other
files.  The MAGIC env variable prevents the XML files from not having
svn:mime-type set, but other files are still being set.  Hence 'svn
import' is blocked, and there is zero way to avoid the setting of
svn:mime-type.  Because of this behaviour, I'm willing to bet that svn
1.9 will also suffer from the inability to disable the automatic
property setting of svn:mime-type (for some reason I can't do this on
a self compiled but not installed svn trunk version, as it is using
the wrong svnlook version and setting SVNLOOK doesn't seem to help).

Regards,

Edward


P. S.  Again, as always, here are some detailed instructions to allow
for a 100% reproducibility of all these tests by anyone interested:


Set up:

$ export BASE_DIR="/flightgear/repo_testing/mimetype/"
$ export BACKUP_DIR="/flightgear/backup/"

Initialise a new svn repository:

$ cd $BASE_DIR
$ svnadmin create test4.svn
$ svn co file://$BASE_DIR/test4.svn/ test4
Checked out revision 0.

Copy scripts from FGAddon backup:

$ rsync -av svn.code.sf.net::p/flightgear/fgaddon/hooks $BACKUP_DIR
$ cd $BACKUP_DIR
$ cd fgaddon/hooks/
$ cp -vp pre-commit fgaddon-svn-* $BASE_DIR/test4.svn/hooks/

Download the DaSH aircraft from
https://drive.google.com/file/d/0B-4Q5DonNb8yTi1Cd1J5Um1xMTg/view to
$BASE_DIR.

Testing 'svn import':

$ cd $BASE_DIR
$ tar xvf dash_v1.tar
dash/
dash/avlModel/
dash/dash-fdm.xml
dash/dash-set.xml
dash/Engines/
[snip]
$ head -n 1 /usr/share/mime/magic > empty_magic
$ export MAGIC=$BASE_DIR/empty_magic
$ MAGIC=$BASE_DIR/empty_magic svn import dash
file://$BASE_DIR/test4.svn/DaSH_svn_import/ -m "Initial import of the
DaSH human powered aircraft.\n\nFor details see the forum thread at
http://forum.flightgear.org/viewtopic.php?f=4=24495 ."
Adding  (bin)  dash/._thumbnail.jpg
Adding dash/Engines
Adding dash/Engines/dash-engine.xml
Adding dash/Engines/dash-prop.xml
Adding dash/Images
Adding  (bin)  dash/Images/._DashBoot_logo_Mod.png
Adding  (bin)  dash/Images/._dash-3view.png
Adding  (bin)  dash/Images/DashBoot_logo_Mod.png
Adding  (bin)  dash/Images/dash-3view.png
Adding dash/Models
Adding  (bin)  dash/Models/._SLE_21b.ac
Adding  (bin)  dash/Models/._dash-model.xml
Adding dash/Models/SLE_21b.ac
Adding

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-06 Thread Stefan Sperling 
On Tue, Oct 06, 2015 at 12:20:47PM +0200, Edward d'Auvergne wrote:
> So setting MAGIC is having an effect, but Subversion is falling back,
> probably via a non-magic internal code path, to the default
> svn:mime-type of "application/octet-stream" for PNG and some other
> files.

If you don't allow svn:mime-type for binary files, you're gonna have
a hard time. Subversion has always been setting this property on binary
files. Please read our documentation about this (at svnbook.org).

If you don't set this property on binary files, your users will
complain that output of 'svn diff' contains binary garbage,
and 'svn merge' will suddenly try to merge PNG files as text.

Make your pre-commit hook smarter. Limit it to *.xml files.

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-05 Thread Stefan Sperling 
On Mon, Oct 05, 2015 at 12:06:59AM +0200, Bert Huijben wrote:
> I'm not sure if I would call it a security problem when a user adds a file of
> their choosing to Subversion though :-)

Yes, typical SVN use cases are of no concern.

One case I could imagine where this might matter is some automated
service which records user-provided files in SVN.
Another case is where people use SVN to version home directories
which include a Downloads folder with files such as images from
the internet.

That's all a bit far fetched, though I know of one person using SVN
to keep /home under version control.
 
> This whole discussion -in its many iterations- is one of the reasons why I
> never looked at enabling this feature on Windows.

And nobody has been asking for it to be enabled?

RE: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-05 Thread Bert Huijben 
> -Original Message-
> From: Stefan Sperling [mailto:s...@elego.de]
> Subject: Re: Bug report: The auto-props setting of svn:mime-type is
> impossible to avoid.
> 

> > This whole discussion -in its many iterations- is one of the reasons why
I
> > never looked at enabling this feature on Windows.
> 
> And nobody has been asking for it to be enabled?

I don't remember anybody asking...

I doubt many users explicitly enable it on other platforms either, but we do
an autodetect there, which usually just works.


Bert

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-04 Thread Ryan Schmidt 

On Oct 4, 2015, at 5:52 AM, Edward d'Auvergne wrote:

> So the following setting disables [auto-props]:
> 
>enable-auto-props = no
> 
> If a svn:mime-type was defined in the config file, this is now
> disabled.  However in this case, [magic-auto-props] then decides to
> tag everything with svn:mime-type.  So the desired user action of
> disabling automatic properties is replaced by the second layer of
> magic that automatically adds a property, contrary to the direct
> English translation of "enable-auto-props = no" -> "do not enable
> automatic properties".  Note how this design is incredibly confusing
> from the user's perspective, when the desired action is to avoid
> having svn:mime-type set.
> 
> This is clearly overcome with the new option:
> 
>enable-magic-file = no
> 
> However the svn documentation really needs to clarify this
> double-layered automatic property setting and, to override it, that
> you need to set both:
> 
>enable-auto-props = no
>enable-magic-file = no

My introduction to Subversion as many years ago so I didn't know it worked that 
way. Thanks for pointing it out.

You didn't mention whether you tried using the Repository Dictated 
Configuration feature of Subversion 1.8 and up to push this configuration to 
your clients. Does that work, and satisfy your needs?


> The problem with the svn:mime-type block is that for svn clients <
> 1.9, 'svn import' can never work.  Specifically because the libmagic
> layer overwrites any user decision to disable automatic properties.
> The only solution I can see is to give these instructions in the
> FlightGear documentation:
> 
>- To use the 'svn import' command, you must have version 1.9 or
> higher of Subversion installed, and in your ~/.subversion/config file
> set both the "enable-auto-props = no" and "enable-magic-file = no"
> options.
> 
>- To avoid the automatic setting of svn properties with 'svn add'
> and the blocking of commits due to the svn:mime-type property being
> set, you must use version 1.9 or higher of Subversion, and in your
> ~/.subversion/config file set both the "enable-auto-props = no" and
> "enable-magic-file = no" options.

Sorry, I must've missed why Subversion 1.9 is required?

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-04 Thread Ivan Zhakov 
On 2 October 2015 at 11:06, Edward d'Auvergne  wrote:
> Hi all,
>
> I was wondering if this should be considered a bug.  At the FlightGear
> project we have a 6 GB data svn repository for aircraft (
> https://sourceforge.net/projects/flightgear/ ,
> https://sourceforge.net/p/flightgear/fgaddon/HEAD/tree/ ).  A large
> quantity of the files, almost 30,000 in number, are XML text files.
>
> The problem is that modern Subversion clients appear to all be using
> libmagic to determine the file type, to then set the svn:mime-type
> property.  However this is rather problematic:
>
> $ file -ib Aircraft/747-400/747-400-set.xml
> application/xml; charset=us-ascii
>
> Libmagic detects all these XML files as "application/xml".  Hence they
> are treated as binary files!  There are many other text files with
> extensions such as *.ac, *.nas, etc. present in the repository that in
> the future might be detected by libmagic as "application/xyz".
>
Easy fix would be change svn_mime_type_is_binary() to consider
'application/xml' as text file. We can also extend for other specific
application/* mimetypes that are text files. We already have special
handling for 'image/x-xbitmap' to consider it as text file.

Does it make sense?

-- 
Ivan Zhakov

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-04 Thread Stefan Sperling 
On Sun, Oct 04, 2015 at 12:52:33PM +0200, Edward d'Auvergne wrote:
> I would maybe suggest introducing an option for disabling the entire
> automatic property subsystem, i.e. it combines both of these, and
> overrides them.  This is an interesting thought experiment - devise
> any name for such a top level automatic property override - and this
> should convincingly demonstrate the confusion that the current setting
> names induces.

I think that would just make the situation worse by adding to
the current mess.

I agree the documentation needs to be improved. Would you have time
to update the documentation and send patches for it?

> The problem with the svn:mime-type block is that for svn clients <
> 1.9, 'svn import' can never work.  Specifically because the libmagic
> layer overwrites any user decision to disable automatic properties.

There's another trick to disable libmagic which works with all
client versions. I just tested it with 1.8.

Set the MAGIC environment variable to a file which does not exist.

  env MAGIC=/nonexistent svn import ...

This effectively disabled libmagic support.

RE: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-04 Thread Bert Huijben 


> -Original Message-
> From: Stefan Sperling [mailto:s...@elego.de]
> Sent: zondag 4 oktober 2015 13:32
> To: Edward d'Auvergne 
> Cc: Greg Stein ; users@subversion.apache.org
> Subject: Re: Bug report: The auto-props setting of svn:mime-type is
> impossible to avoid.
> 
> On Sun, Oct 04, 2015 at 12:52:33PM +0200, Edward d'Auvergne wrote:
> > I would maybe suggest introducing an option for disabling the entire
> > automatic property subsystem, i.e. it combines both of these, and
> > overrides them.  This is an interesting thought experiment - devise
> > any name for such a top level automatic property override - and this
> > should convincingly demonstrate the confusion that the current setting
> > names induces.
> 
> I think that would just make the situation worse by adding to
> the current mess.
> 
> I agree the documentation needs to be improved. Would you have time
> to update the documentation and send patches for it?
> 
> > The problem with the svn:mime-type block is that for svn clients <
> > 1.9, 'svn import' can never work.  Specifically because the libmagic
> > layer overwrites any user decision to disable automatic properties.
> 
> There's another trick to disable libmagic which works with all
> client versions. I just tested it with 1.8.
> 
> Set the MAGIC environment variable to a file which does not exist.
> 
>   env MAGIC=/nonexistent svn import ...
> 
> This effectively disabled libmagic support.

For Windows I don't think you need any disabling, as our build doesn't even
support building with libmagic there. (Perhaps somebody patched something in
their own build... but I don't think that is really an interesting
scenario). 

Note that the Cygwin version is an exception to most of this... It is a
Windows build but tries to do everything the unix way.
 
Bert

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-04 Thread Branko Čibej 
On 04.10.2015 11:35, Ivan Zhakov wrote:
> On 2 October 2015 at 11:06, Edward d'Auvergne  wrote:
>> Hi all,
>>
>> I was wondering if this should be considered a bug.  At the FlightGear
>> project we have a 6 GB data svn repository for aircraft (
>> https://sourceforge.net/projects/flightgear/ ,
>> https://sourceforge.net/p/flightgear/fgaddon/HEAD/tree/ ).  A large
>> quantity of the files, almost 30,000 in number, are XML text files.
>>
>> The problem is that modern Subversion clients appear to all be using
>> libmagic to determine the file type, to then set the svn:mime-type
>> property.  However this is rather problematic:
>>
>> $ file -ib Aircraft/747-400/747-400-set.xml
>> application/xml; charset=us-ascii
>>
>> Libmagic detects all these XML files as "application/xml".  Hence they
>> are treated as binary files!  There are many other text files with
>> extensions such as *.ac, *.nas, etc. present in the repository that in
>> the future might be detected by libmagic as "application/xyz".
>>
> Easy fix would be change svn_mime_type_is_binary() to consider
> 'application/xml' as text file. We can also extend for other specific
> application/* mimetypes that are text files. We already have special
> handling for 'image/x-xbitmap' to consider it as text file.
>
> Does it make sense?

Please let's not have that discussion again. A quick search of the
archives will show that we've talked this idea to death already.

To summarize: To Subversion, "text" implies that you can do textual
diff/merge with the contents. With XML, that may sometimes work, but
usually won't. And the problem doesn't stop with 'application/xml'
because there are literally hundreds of media types for documents that
use XML.

On the other hand, I am surprised that the logic that uses libmagic
isn't turned off with 'enable-auto-props=no'. After all, using libmagic
is just a convenient extension to the definitions in the [auto-props]
section.

-- Brane

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-04 Thread Stefan Sperling 
On Sun, Oct 04, 2015 at 09:16:04PM +0200, Branko Čibej wrote:
> On the other hand, I am surprised that the logic that uses libmagic
> isn't turned off with 'enable-auto-props=no'. After all, using libmagic
> is just a convenient extension to the definitions in the [auto-props]
> section.

Recall that the idea was to make setting svn:mime-type convenient.

Before we added this feature, people had to fiddle with their client
config, know what MIME-types are about, what useful values to set
them to, and keep their config consistent across all systems they
used since libmagic support pre-dates the svn:autoprops feature.
In a sane universe, almost nobody would ever bother setting it up that way.

I am not opposed to the idea, though. But not because of the fairly
minor configurability issues raised in this thread. Rather, because
the more I learn about how libmagic actually works, the more potentially
dangerous, from the security point of view, it seems to be.
The libmagic parser could potentially cause security problems whenever
poeple are adding untrusted files to SVN. Which is a good reason to make
this feature opt-in.

RE: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-04 Thread Bert Huijben 
> -Original Message-
> From: Stefan Sperling [mailto:s...@elego.de]
> Sent: zondag 4 oktober 2015 22:01
> To: Branko Čibej 
> Cc: users@subversion.apache.org
> Subject: Re: Bug report: The auto-props setting of svn:mime-type is
> impossible to avoid.
> 
> On Sun, Oct 04, 2015 at 09:16:04PM +0200, Branko Čibej wrote:
> > On the other hand, I am surprised that the logic that uses libmagic
> > isn't turned off with 'enable-auto-props=no'. After all, using libmagic
> > is just a convenient extension to the definitions in the [auto-props]
> > section.
> 
> Recall that the idea was to make setting svn:mime-type convenient.
> 
> Before we added this feature, people had to fiddle with their client
> config, know what MIME-types are about, what useful values to set
> them to, and keep their config consistent across all systems they
> used since libmagic support pre-dates the svn:autoprops feature.
> In a sane universe, almost nobody would ever bother setting it up that way.
> 
> I am not opposed to the idea, though. But not because of the fairly
> minor configurability issues raised in this thread. Rather, because
> the more I learn about how libmagic actually works, the more potentially
> dangerous, from the security point of view, it seems to be.
> The libmagic parser could potentially cause security problems whenever
> people are adding untrusted files to SVN. Which is a good reason to make
> this feature opt-in.

I would support changing Subversion to -by default- disable this feature, 
unless it is enabled in the configuration (or some flag passed to 'svn add' in 
1.10+).

I'm not sure if I would call it a security problem when a user adds a file of 
their choosing to Subversion though :-)


This whole discussion -in its many iterations- is one of the reasons why I 
never looked at enabling this feature on Windows.

Bert

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-04 Thread Stefan Sperling 
On Sat, Oct 03, 2015 at 11:13:08AM +0200, Edward d'Auvergne wrote:
> is it really not a bug when:
> 
> enable-auto-props = yes
> 
> and:
> 
> enable-auto-props = no
> 
> both enable auto-props?
> 
> Cheers,
> 
> Edward

I think your best way forward is what Ryan suggested: Ensure svn:mime-type
for *.xml is set to 'text/plain' by setting svn:autoprops at the root of
your repository. Then, over time, as your users upgrade to 1.8 clients
and beyond, this problem will fix itself.

Note that setting mime-type to 'text/plain' will have the same effect
as if a file which is a "text file" (as guessed by SVN based on
implementation-defined heuristics) does not have a mime-type set.

When I added the libmagic support (yes, I'm responsible) I had to ensure
the existing auto-props mechanism kept working. So mime-type detection
with libmagic happens *after* autoprops and only if autoprops don't set
a mime-type. That's where this design decision came from.

I'm sorry this implementation doesn't work out of the box for you,
and if I had known about your situation I might have made different
choices while implementing this feature years ago. Alas, it's water
under the bridge now, and you're the first person to complain about
this ever, and there are mulitple ways you can work around the problem.

So please stop calling behaviour you don't agree with a "bug".
Call it what it is, a particular behaviour, which might not fit
every single use case out there.

I hope this works out fine for you in the end, one way or another.
I'm very happy to see SVN supporting an awesome project like yours!

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-03 Thread Ryan Schmidt 

On Oct 3, 2015, at 3:25 AM, Edward d'Auvergne wrote:

> On 2 October 2015 at 11:33, Stefan Sperling wrote:
> 
>> - configure autoprops for *.xml in ~/.subversion/config to set
>>   the svn:mime-type property to 'text/plain'.
>>   Autoprops always override automatic detection with libmagic.
> 
> From the perspective of Subversion users, libmagic is a 3rd party.  We
> cannot trust them to not change the magic file again to something that
> treats text files in svn as binary files.  This FG aircraft repository
> is one of the biggest version controlled repositories in the open
> source world, by far.  It has an incredible diversity of text files
> with different extensions, and we would like these to forever be
> treated as text.  Due to the bad behaviour of libmagic, banning
> svn:mime-type from the FG repository is the only practical option.

So configure autoprops correctly for those various extensions.

You can even use repository-dictated configuration to make those autoprops 
definitions automatic for all your diverse users.

http://blogs.collab.net/subversion/the-road-to-repository-dictated-configuration-day-2-autoprops#.Vg-ZjbRobfA





On Oct 3, 2015, at 3:48 AM, Edward d'Auvergne wrote:

> The damage is to the repository backend and to the development process
> itself.  Treating text files as binary has a number of significant
> consequences for development.  For example:
> 
> - 'svn up' can no longer perform merges (G).
> - 'svn merge' or the svnmerge.py script can no longer function to
> merge changes to the same file.
> - A one character change to a massive text file results in a diff and
> db/revs backend file the size of the whole file, and not just a few
> lines for a proper diff.
> - Changes can no longer be reviewed (
> https://sourceforge.net/p/flightgear/mailman/flightgear-fgaddon-commitlogs/
> ).
> 
> This is a severe bug.

Not a bug: intended behavior for binary files. The "bug" is that Subversion, 
via libmagic, has decided a file is binary, when you consider it to be text.

The XML specification defines a number of correct MIME types for XML files, 
including some beginning with "text/xml" (intended for human-readable XML 
files) and "application/xml" (intended for XML files not meant for humans to 
read). libmagic can identify whether a file is XML, but it cannot know whether 
the file is meant for human consumption, so it picks on of these. The person 
committing the file can correct this prior to committing if it is inaccurate. 
Or if you don't want the property set at all, use autoprops.


> Here is an interesting sign as well:
> 
> $ svn propset svn:mime-type "application/xml" dash-set.xml
> svn: warning: 'application/xml' is a binary mime-type but file
> '/flightgear/repo_testing/mimetype/test1/DaSH_svn_import/dash-set.xml'
> looks like text; diff, merge, blame, and other operations will stop
> working on this file
> property 'svn:mime-type' set on 'dash-set.xml'
> $
> 
> Unfortunately 'svn add' and 'svn import' set this binary mime-type
> without warning, and when explicitly told not to.

Subversion doesn't do things without being told to do so. There are defaults 
for how MIME types are set on files, which you can override.


> Manually monitoring this huge beast is out of the question.

Of course you wouldn't monitor it manually; you would monitor it via a 
pre-commit hook script that prevents commits that don't conform to your 
requirements.

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-03 Thread Edward d'Auvergne 
On 3 October 2015 at 11:01, Greg Stein  wrote:
> I do understand this, but this is not a "bug". It is how Subversion is
> designed to work. For strange edge cases, the xml people want content to be
> labeled application/xml rather than a text subtype. ... I *do* understand
> that this negatively impacts your development workflow.
>
> Note that you can always remove the property after the "svn add", but before
> you "svn commit". Others have commented on ways to avoid the automatic
> placement of the property, though that will be increasingly difficult in
> heterogeneous environments with different svn clients. Subversion just
> "helpfully" provides a property, but you *don't* have to commit it that way.
> And committing it that way, and removing it later, has no effect on your
> actual content.
>
> Your hook is a good way to prevent *new* files with that svn:mime-type on
> them. Your repository has tens of thousands now, sure, but honestly... the
> concern is about new files, and their rate of addition. It seems once
> somebody sees a "cannot merge a binary file" error, then they can just go
> delete the property and continue their work.
>
> If one of the knobs that Stefan suggest is not working properly, then we can
> track that down and fix it.

Hi,

The "svn add", "svn propdel svn:mime-type -R", "svn ci" workflow is
what we currently must use.  Though it would be good if we could cut
out the middle "svn propdel" step, as the propdel command can be
difficult to find or missing in certain svn GUI clients (I've heard
this about TortoiseSVN).  The big problem is with the "svn import"
command, which essentially combines "svn add *" and "svn ci" into one
without having a local checked out copy of the repository.  In this
case, inserting "svn propdel" in the middle is not possible.  Anyway,
is it really not a bug when:

enable-auto-props = yes

and:

enable-auto-props = no

both enable auto-props?

Cheers,

Edward

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-03 Thread Edward d'Auvergne 
On 3 October 2015 at 01:05, Greg Stein  wrote:
> On Fri, Oct 02, 2015 at 10:06:26AM +0200, Edward d'Auvergne wrote:
>>...
>> As this bad behaviour can be so incredibly damaging for this
>> repository,
>
> Note: the files themselves are not "damaged" -- Subversion will never
> alter the contents of a file when it is first imported/added. It may
> make a file look different in the working copy, according to
> svn:eol-style. But the svn:mime-type is just a property associated
> with the file. Its only real use is when the file is served via HTTP
> to a web browser.

Hi Greg,

The damage is to the repository backend and to the development process
itself.  Treating text files as binary has a number of significant
consequences for development.  For example:

- 'svn up' can no longer perform merges (G).
- 'svn merge' or the svnmerge.py script can no longer function to
merge changes to the same file.
- A one character change to a massive text file results in a diff and
db/revs backend file the size of the whole file, and not just a few
lines for a proper diff.
- Changes can no longer be reviewed (
https://sourceforge.net/p/flightgear/mailman/flightgear-fgaddon-commitlogs/
).

This is a severe bug.  Here is an interesting sign as well:

$ svn propset svn:mime-type "application/xml" dash-set.xml
svn: warning: 'application/xml' is a binary mime-type but file
'/flightgear/repo_testing/mimetype/test1/DaSH_svn_import/dash-set.xml'
looks like text; diff, merge, blame, and other operations will stop
working on this file
property 'svn:mime-type' set on 'dash-set.xml'
$

Unfortunately 'svn add' and 'svn import' set this binary mime-type
without warning, and when explicitly told not to.


> If a file is committed to the repository with an svn:mime-type, why
> don't you just delete/correct the property in a following commit?

Does this answer your question ;)

[edward@localhost flightgear-fgaddon]$ svn info
Path: .
Working Copy Root Path: /flightgear/flightgear-fgaddon
URL: svn+ssh://edauver...@svn.code.sf.net/p/flightgear/fgaddon/trunk
Relative URL: ^/trunk
Repository Root: svn+ssh://edauver...@svn.code.sf.net/p/flightgear/fgaddon
Repository UUID: 9eb9bc84-db9d-4e05-893b-26dd41ac94a6
[snip]
[edward@localhost flightgear-fgaddon]$ find -type f | grep -v .svn | wc -l
69458
[edward@localhost flightgear-fgaddon]$ find -name "*.xml" | wc -l
27262
[edward@localhost flightgear-fgaddon]$ find -name "*.ac" | wc -l
12502
[edward@localhost flightgear-fgaddon]$ find -name "*.nas" | wc -l
2840

Manually monitoring this huge beast is out of the question.

Cheers,

Edward

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-03 Thread Ryan Schmidt 

On Oct 3, 2015, at 4:13 AM, Edward d'Auvergne wrote:

> is it really not a bug when:
> 
>enable-auto-props = yes
> 
> and:
> 
>enable-auto-props = no
> 
> both enable auto-props?

Obviously, "enable-auto-props = yes" should enable auto-props and 
"enable-auto-props = no" should disable them. "no" is the default according to 
the documentation:

http://svnbook.red-bean.com/en/1.7/svn.advanced.confarea.html#svn.advanced.confarea.opts.config

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-03 Thread Greg Stein 
I do understand this, but this is not a "bug". It is how Subversion is
designed to work. For strange edge cases, the xml people want content to be
labeled application/xml rather than a text subtype. ... I *do* understand
that this negatively impacts your development workflow.

Note that you can always remove the property after the "svn add", but
before you "svn commit". Others have commented on ways to avoid the
automatic placement of the property, though that will be increasingly
difficult in heterogeneous environments with different svn clients.
Subversion just "helpfully" provides a property, but you *don't* have to
commit it that way. And committing it that way, and removing it later, has
no effect on your actual content.

Your hook is a good way to prevent *new* files with that svn:mime-type on
them. Your repository has tens of thousands now, sure, but honestly... the
concern is about new files, and their rate of addition. It seems once
somebody sees a "cannot merge a binary file" error, then they can just go
delete the property and continue their work.

If one of the knobs that Stefan suggest is not working properly, then we
can track that down and fix it.

Cheers,
-g


On Sat, Oct 3, 2015 at 3:48 AM, Edward d'Auvergne 
wrote:

> On 3 October 2015 at 01:05, Greg Stein  wrote:
> > On Fri, Oct 02, 2015 at 10:06:26AM +0200, Edward d'Auvergne wrote:
> >>...
> >> As this bad behaviour can be so incredibly damaging for this
> >> repository,
> >
> > Note: the files themselves are not "damaged" -- Subversion will never
> > alter the contents of a file when it is first imported/added. It may
> > make a file look different in the working copy, according to
> > svn:eol-style. But the svn:mime-type is just a property associated
> > with the file. Its only real use is when the file is served via HTTP
> > to a web browser.
>
> Hi Greg,
>
> The damage is to the repository backend and to the development process
> itself.  Treating text files as binary has a number of significant
> consequences for development.  For example:
>
> - 'svn up' can no longer perform merges (G).
> - 'svn merge' or the svnmerge.py script can no longer function to
> merge changes to the same file.
> - A one character change to a massive text file results in a diff and
> db/revs backend file the size of the whole file, and not just a few
> lines for a proper diff.
> - Changes can no longer be reviewed (
> https://sourceforge.net/p/flightgear/mailman/flightgear-fgaddon-commitlogs/
> ).
>
> This is a severe bug.  Here is an interesting sign as well:
>
> $ svn propset svn:mime-type "application/xml" dash-set.xml
> svn: warning: 'application/xml' is a binary mime-type but file
> '/flightgear/repo_testing/mimetype/test1/DaSH_svn_import/dash-set.xml'
> looks like text; diff, merge, blame, and other operations will stop
> working on this file
> property 'svn:mime-type' set on 'dash-set.xml'
> $
>
> Unfortunately 'svn add' and 'svn import' set this binary mime-type
> without warning, and when explicitly told not to.
>
>
> > If a file is committed to the repository with an svn:mime-type, why
> > don't you just delete/correct the property in a following commit?
>
> Does this answer your question ;)
>
> [edward@localhost flightgear-fgaddon]$ svn info
> Path: .
> Working Copy Root Path: /flightgear/flightgear-fgaddon
> URL: svn+ssh://edauver...@svn.code.sf.net/p/flightgear/fgaddon/trunk
> Relative URL: ^/trunk
> Repository Root: svn+ssh://edauver...@svn.code.sf.net/p/flightgear/fgaddon
> Repository UUID: 9eb9bc84-db9d-4e05-893b-26dd41ac94a6
> [snip]
> [edward@localhost flightgear-fgaddon]$ find -type f | grep -v .svn | wc -l
> 69458
> [edward@localhost flightgear-fgaddon]$ find -name "*.xml" | wc -l
> 27262
> [edward@localhost flightgear-fgaddon]$ find -name "*.ac" | wc -l
> 12502
> [edward@localhost flightgear-fgaddon]$ find -name "*.nas" | wc -l
> 2840
>
> Manually monitoring this huge beast is out of the question.
>
> Cheers,
>
> Edward
>

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-03 Thread Nico Kadel-Garcia 
On Sat, Oct 3, 2015 at 6:53 AM, Ryan Schmidt
 wrote:
>
> On Oct 3, 2015, at 4:13 AM, Edward d'Auvergne wrote:
>
>> is it really not a bug when:
>>
>>enable-auto-props = yes
>>
>> and:
>>
>>enable-auto-props = no
>>
>> both enable auto-props?
>
> Obviously, "enable-auto-props = yes" should enable auto-props and 
> "enable-auto-props = no" should disable them. "no" is the default according 
> to the documentation:
>
> http://svnbook.red-bean.com/en/1.7/svn.advanced.confarea.html#svn.advanced.confarea.opts.config

Whether or not it works on the client settings reliably, it's breaking
these developer's workflow for even accidentally enabled use of
auto-props to happen. I'm afraid their only real option is the
pre-commit hook to prevent poisoning the data, because they don't
necessarily have control of the client configurations. The risk of
that pre-commit hook is that it forbids binary xml's at all, which
does not seem such a big loss in this case.

Mind you, this is the sort of thing that a README at the top of the
"trunk" is useful for, to let users know of workflow problems in
their.

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-03 Thread Edward d'Auvergne 
On 2 October 2015 at 11:33, Stefan Sperling  wrote:
> On Fri, Oct 02, 2015 at 10:06:26AM +0200, Edward d'Auvergne wrote:
>> Hi all,
>>
>> I was wondering if this should be considered a bug.  At the FlightGear
>> project we have a 6 GB data svn repository for aircraft (
>> https://sourceforge.net/projects/flightgear/ ,
>> https://sourceforge.net/p/flightgear/fgaddon/HEAD/tree/ ).  A large
>> quantity of the files, almost 30,000 in number, are XML text files.
>>
>> The problem is that modern Subversion clients appear to all be using
>> libmagic to determine the file type, to then set the svn:mime-type
>> property.  However this is rather problematic:
>>
>> $ file -ib Aircraft/747-400/747-400-set.xml
>> application/xml; charset=us-ascii
>>
>> Libmagic detects all these XML files as "application/xml".  Hence they
>> are treated as binary files!  There are many other text files with
>> extensions such as *.ac, *.nas, etc. present in the repository that in
>> the future might be detected by libmagic as "application/xyz".
>>
>> We have looked at disabling [auto-props] both as the user in
>> ~/.subversion/config and as root in /etc/subversion/config by setting:
>>
>> enable-auto-props = no
>>
>> And by commenting out the entirety of the [auto-props] section.
>> However this does not prevent svn:mime-type from being set on all
>> files.
>>
>> As this bad behaviour can be so incredibly damaging for this
>> repository, I have written pre-commit repository hook script to
>> specifically to block any file that enters the data repository that
>> has svn:mime-type set (
>> http://thread.gmane.org/gmane.games.flightgear.devel/77606/focus=77796
>> ).  I also block svn:executable in another script.  Below I'll give a
>> worked example that anyone can use and replicate this problem with.
>> The problem is that Subversion clients setting svn:mime-type
>> automatically, even when told not to, and the hook script catching
>> this causes a lot of developer confusion and prevents the use of 'svn
>> import'.
>>
>> I was wondering if there was anything that has been missed here?  Is
>> this a real bug?  The svn:mime-type property is not needed and is not
>> desired for any file in this repository.  Any help would be
>> appreciated.
>>
>> Cheers,
>>
>> Edward
>
> Hi Edward,
>
> the pre-commit hook script you wrote blocks commits using an
> undesired client configuration, so that's a good first step.

Hi Stefan,

Thanks for the detailed response!


> For the second step, there are several ways of tweaking the client-side
> behaviour. I hope at least one of these alternatives will help you.
>
>  - compile SVN without libmagic support

Unfortunately this is difficult.  The FlightGear (FG) aircraft
collection contains about 500 aircraft, mostly by different and
multiple authors.  So we have GNU/Linux, Mac OS X, and MS Windows
users/developers.  And we have users of the command line svn clients
and GUI tools.  Standard svn, KDESvn, SlikSVN, TortoiseSVN, the list
goes on...  The huge diversity limits the practical options.


>  - write you own magic file and use the MAGIC environment variable
>to point svn at it, setting the desired mime-type for XML files

This might work.  I'll soon try out Philip's suggestion and see how
that works.  I'm not sure how this will work with the GUI clients.


>  - configure autoprops for *.xml in ~/.subversion/config to set
>the svn:mime-type property to 'text/plain'.
>Autoprops always override automatic detection with libmagic.

>From the perspective of Subversion users, libmagic is a 3rd party.  We
cannot trust them to not change the magic file again to something that
treats text files in svn as binary files.  This FG aircraft repository
is one of the biggest version controlled repositories in the open
source world, by far.  It has an incredible diversity of text files
with different extensions, and we would like these to forever be
treated as text.  Due to the bad behaviour of libmagic, banning
svn:mime-type from the FG repository is the only practical option.


>  - set 'enable-magic-file = no' in ~/.subversion/config
>This option exists only as of Subversion 1.9, I believe.
>Consider the following trick to view a default configuration in the
>file /tmp/svncfg/config, which lists all available options in your
>current version of svn, with comments explaining their purpose:
>  svn help --config-dir /tmp/svncfg
>  less /tmp/svncfg/config

This sounds like an option, but not many of the FG aircraft developers
will have access to this.  For example I'm using Mageia Linux 5, which
was only released a few months ago, that this ships with svn, version
1.8.14 (r1692801).  However the option:

enable-auto-props = no

Should be equivalent of:

disable-auto-props = yes

I do not know why when telling the svn client to not use auto-props,
the svn client ignores you and uses auto-props anyway.  To me this
looks like a bug (I hope the Subversion developers

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-03 Thread Edward d'Auvergne 
On 2 October 2015 at 12:30, Philip Martin  wrote:
> "Edward d'Auvergne"  writes:
>
>> I was wondering if there was anything that has been missed here?  Is
>> this a real bug?  The svn:mime-type property is not needed and is not
>> desired for any file in this repository.  Any help would be
>> appreciated.
>
> You can disable libmagic by setting the environment variable MAGIC
>
> $ svnadmin create repo
> $ svn co file://`pwd`/repo wc
> $ echo "" > wc/a.xml
> $ echo "" > wc/b.xml
> $ svn add wc/a.xml
> A  (bin)  wc/a.xml
> $ MAGIC= svn add wc/b.xml
> A wc/b.xml
> $ svn pl -vR wc
> Properties on 'wc/a.xml':
>   svn:mime-type
> application/xml

Hi Philip,

Thank you for the response.  I have tried out your suggestion with the
MAGIC env variable, however unfortunately on my Mageia Linux 5 system,
this had no effect.  The full details, to allow for 100%
reproducibility, are given below.

Cheers,

Edward


P. S.  Here are 3 different tests:


Set up:

$ export BASE_DIR="/flightgear/repo_testing/mimetype/"
$ export BACKUP_DIR="/flightgear/backup/"

Initialise a new svn repository:

$ cd $BASE_DIR
$ svnadmin create test2.svn
$ svn co file://$BASE_DIR/test2.svn/ test2
Checked out revision 0.

Copy scripts from FGAddon backup:

$ rsync -av svn.code.sf.net::p/flightgear/fgaddon/hooks $BACKUP_DIR
$ cd $BACKUP_DIR
$ cd hooks/
$ cp -vp pre-commit fgaddon-svn-* $BASE_DIR/test2.svn/hooks/

Download the DaSH aircraft from
https://drive.google.com/file/d/0B-4Q5DonNb8yTi1Cd1J5Um1xMTg/view to
$BASE_DIR.

Testing 'svn import':

$ cd $BASE_DIR
$ tar xvf dash_v1.tar
dash/
dash/avlModel/
dash/dash-fdm.xml
dash/dash-set.xml
dash/Engines/
[snip]
$ MAGIC= svn import dash file://$BASE_DIR/test2.svn/DaSH_svn_import/
-m "Initial import of the DaSH human powered aircraft.\n\nFor details
see the forum thread at
http://forum.flightgear.org/viewtopic.php?f=4=24495 ."
Adding  (bin)  dash/._thumbnail.jpg
Adding dash/Engines
Adding dash/Engines/dash-engine.xml
Adding dash/Engines/dash-prop.xml
Adding dash/Images
Adding  (bin)  dash/Images/._DashBoot_logo_Mod.png
Adding  (bin)  dash/Images/._dash-3view.png
Adding  (bin)  dash/Images/DashBoot_logo_Mod.png
Adding  (bin)  dash/Images/dash-3view.png
Adding dash/Models
Adding  (bin)  dash/Models/._SLE_21b.ac
Adding  (bin)  dash/Models/._dash-model.xml
Adding dash/Models/SLE_21b.ac
Adding dash/Models/dash-model.xml
Adding dash/avlModel
Adding  (bin)  dash/avlModel/._dae11.dat
Adding  (bin)  dash/avlModel/._dash.avl
Adding  (bin)  dash/avlModel/._plot.pdf
Adding  (bin)  dash/avlModel/._plot.ps
Adding dash/avlModel/dae11.dat
Adding dash/avlModel/dash.avl
Adding dash/avlModel/dash_stability.txt
Adding  (bin)  dash/avlModel/plot.pdf
Adding dash/avlModel/plot.ps
Adding dash/dash-fdm.xml
Adding dash/dash-set.xml
Adding  (bin)  dash/thumbnail.jpg
svn: E165001: Commit blocked by pre-commit hook (exit code 1) with output:

The svn:mime-type property is set on the files
['DaSH_svn_import/._thumbnail.jpg',
'DaSH_svn_import/Images/._DashBoot_logo_Mod.png', 'DaSH_svn_import/Images
/._dash-3view.png', 'DaSH_svn_import/Images/DashBoot_logo_Mod.png',
'DaSH_svn_import/Images/dash-3view.png', 'DaSH_svn_import/Models/._SLE_21b.ac',
'DaSH_svn_import/Models/._dash-model.xml',
'DaSH_svn_import/avlModel/._dae11.dat', 'DaSH_svn_import/avlModel/._dash.avl',
'DaSH_svn_import/avlModel/._plot.pdf', 'DaSH_svn_import/avlModel/._plot.ps',
'DaSH_svn_import/avlModel/plot.pdf', 'DaSH_svn_import/thumbnail.jpg'], aborting
the commit.

The current policy is that the svn:mime-type property is to be avoided in
FGAddon.  Before committing, please remove this property by typing 'svn propdel
svn:mime-type file_name' for all affected files.  Or to remove it recursively
from all files to be committed, in your aircraft directory type 'svn propdel
svn:mime-type -R'.

To avoid the svn:mime-type property being set by your subversion client, the
subversion configuration should be edited and all [auto-props] entries commented
out.  In addition the setting "enable-auto-props = yes" should be deleted from
all subversion config files (in certain svn clients, this will use libmagic to
determine the mime-type and set this for all new files).  This includes the user
configuration file at $HOME/.subversion/config or, if this is not present, the
global configuration file at /etc/subversion/config in GNU/Linux and Mac OS X or
%appdata%\subversion\config and %appdata%\roaming\subversion\config in MS
Windows.

Test 2:

$ export MAGIC=
$ svn import dash file://$BASE_DIR/test2.svn/DaSH_svn_import/ -m
"Initial import of the DaSH human powered aircraft.\n\nFor details see
the forum thread at
http://forum.flightgear.org/viewtopic.php?f=4=24495 ."
Failed!

Test 3:

$ unset MAGIC
$ svn import dash file://$BASE_DIR/test2.svn/DaSH_svn_import/ -m
"Initial import of the DaSH human powered

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-02 Thread Stefan Sperling 
On Fri, Oct 02, 2015 at 10:06:26AM +0200, Edward d'Auvergne wrote:
> Hi all,
> 
> I was wondering if this should be considered a bug.  At the FlightGear
> project we have a 6 GB data svn repository for aircraft (
> https://sourceforge.net/projects/flightgear/ ,
> https://sourceforge.net/p/flightgear/fgaddon/HEAD/tree/ ).  A large
> quantity of the files, almost 30,000 in number, are XML text files.
> 
> The problem is that modern Subversion clients appear to all be using
> libmagic to determine the file type, to then set the svn:mime-type
> property.  However this is rather problematic:
> 
> $ file -ib Aircraft/747-400/747-400-set.xml
> application/xml; charset=us-ascii
> 
> Libmagic detects all these XML files as "application/xml".  Hence they
> are treated as binary files!  There are many other text files with
> extensions such as *.ac, *.nas, etc. present in the repository that in
> the future might be detected by libmagic as "application/xyz".
> 
> We have looked at disabling [auto-props] both as the user in
> ~/.subversion/config and as root in /etc/subversion/config by setting:
> 
> enable-auto-props = no
> 
> And by commenting out the entirety of the [auto-props] section.
> However this does not prevent svn:mime-type from being set on all
> files.
> 
> As this bad behaviour can be so incredibly damaging for this
> repository, I have written pre-commit repository hook script to
> specifically to block any file that enters the data repository that
> has svn:mime-type set (
> http://thread.gmane.org/gmane.games.flightgear.devel/77606/focus=77796
> ).  I also block svn:executable in another script.  Below I'll give a
> worked example that anyone can use and replicate this problem with.
> The problem is that Subversion clients setting svn:mime-type
> automatically, even when told not to, and the hook script catching
> this causes a lot of developer confusion and prevents the use of 'svn
> import'.
> 
> I was wondering if there was anything that has been missed here?  Is
> this a real bug?  The svn:mime-type property is not needed and is not
> desired for any file in this repository.  Any help would be
> appreciated.
> 
> Cheers,
> 
> Edward

Hi Edward,

the pre-commit hook script you wrote blocks commits using an
undesired client configuration, so that's a good first step.

For the second step, there are several ways of tweaking the client-side
behaviour. I hope at least one of these alternatives will help you.

 - compile SVN without libmagic support

 - write you own magic file and use the MAGIC environment variable
   to point svn at it, setting the desired mime-type for XML files

 - configure autoprops for *.xml in ~/.subversion/config to set
   the svn:mime-type property to 'text/plain'.
   Autoprops always override automatic detection with libmagic.

 - set 'enable-magic-file = no' in ~/.subversion/config
   This option exists only as of Subversion 1.9, I believe.
   Consider the following trick to view a default configuration in the
   file /tmp/svncfg/config, which lists all available options in your
   current version of svn, with comments explaining their purpose:
 svn help --config-dir /tmp/svncfg
 less /tmp/svncfg/config

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-02 Thread Philip Martin 
"Edward d'Auvergne"  writes:

> I was wondering if there was anything that has been missed here?  Is
> this a real bug?  The svn:mime-type property is not needed and is not
> desired for any file in this repository.  Any help would be
> appreciated.

You can disable libmagic by setting the environment variable MAGIC

$ svnadmin create repo
$ svn co file://`pwd`/repo wc
$ echo "" > wc/a.xml
$ echo "" > wc/b.xml
$ svn add wc/a.xml
A  (bin)  wc/a.xml
$ MAGIC= svn add wc/b.xml 
A wc/b.xml
$ svn pl -vR wc
Properties on 'wc/a.xml':
  svn:mime-type
application/xml

-- 
Philip Martin
WANdisco

Re: Bug report: The auto-props setting of svn:mime-type is impossible to avoid.

2015-10-02 Thread Greg Stein 
On Fri, Oct 02, 2015 at 10:06:26AM +0200, Edward d'Auvergne wrote:
>...
> As this bad behaviour can be so incredibly damaging for this
> repository,

Note: the files themselves are not "damaged" -- Subversion will never
alter the contents of a file when it is first imported/added. It may
make a file look different in the working copy, according to
svn:eol-style. But the svn:mime-type is just a property associated
with the file. Its only real use is when the file is served via HTTP
to a web browser.

If a file is committed to the repository with an svn:mime-type, why
don't you just delete/correct the property in a following commit?

Cheers,
-g