Re: [fossil-users] filename contains illegal characters
On Sat, 01 Dec, Jan Nijtmans wrote: I agree with Richard on this. The most important part of the patch is already on trunk now: allowing '[', ']', '', '*' and '?' in filenames. Really? That's the part I am interested in, but I do not see that merged into trunk: sbellon@slim$ fossil status | egrep (checkout|tags) checkout: 99ab9fb47cfb20106564012b7678ca5d579060af 2012-12-01 04:49:38 UTC tags: trunk sbellon@slim$ grep not contain any of these characters src/file.c ** * Does not contain any of these characters in the path: \*[]? That's the part I am interested in and I think this should go into trunk. It makes the code even simpler and not more complex because it removes an exceptional case which is questionable anyway. Regarding the Unicode part of the patch, I agree that this may need some further attention. The added value of the remaining part is minimal: It would make the win32 build of fossil do the same as the cygwin build of fossil. So, you have the functionality already: Just build fossil.exe for cygwin, it does exactly the same. For some reasons, cygwin is no option here. That's not sufficient excuse to rush it into 1.25. And missing is still a test-case description of exactly what is the added value: What can you do with the changed version what cannot be done with the trunk version. Well ... having filenames with [ ] * and ? in the repository and in checkouts if the underlying OS supports it. Greetings, Stefan -- Stefan Bellon ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
On Sat, Dec 1, 2012 at 4:02 AM, Stefan Bellon sbel...@sbellon.de wrote: ** * Does not contain any of these characters in the path: \*[]? That's the part I am interested in and I think this should go into trunk. Added to trunk here: http://www.fossil-scm.org/fossil/info/647bb7b79f -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
2012/12/1 Stefan Bellon sbel...@sbellon.de: sbellon@slim$ fossil status | egrep (checkout|tags) checkout: 99ab9fb47cfb20106564012b7678ca5d579060af 2012-12-01 04:49:38 UTC tags: trunk sbellon@slim$ grep not contain any of these characters src/file.c ** * Does not contain any of these characters in the path: \*[]? Comment was not updated accordingly. Done now. Thanks! Regards, Jan Nijtmans ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
On Sat, 01 Dec, Richard Hipp wrote: On Sat, Dec 1, 2012 at 4:02 AM, Stefan Bellon sbel...@sbellon.de wrote: That's the part I am interested in and I think this should go into trunk. Added to trunk here: http://www.fossil-scm.org/fossil/info/647bb7b79f Silly me, I just looked at the comment and not the code. Now I'm a happy bunny. :-) Greetings, Stefan -- Stefan Bellon ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
On Tue, 27 Nov, Jan Nijtmans wrote: This version allows all characters in filenames that are permissible on UNIX, except '\'. And those files can be checked out using Cygwin and Windows as well, using Cygwin's solution: In the file system translate those characters to 'safe' characters in the range U+F000 to U+F0FF (which should never itself occur in valid fossil cards) I cannot find any security-risks with this. The only possible security risk I found is unrelated to this change, and it is already fixed in [897dfa48b4] (Thanks, Richard!) Anyone who thinks this is a bad idea? Please, speak up now, otherwise Richard might approve this! I would very much like to see this going into the upcoming 1.25 release. Is it scheduled to get merged back into trunk? Greetings, Stefan -- Stefan Bellon ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
2012/11/25 Jan Nijtmans jan.nijtm...@gmail.com: New test version in [d3bee356ba]. It changes the filename contains illegal characters fatal into a warning, which should make your situation more managable: Even though files containing '\' still cannot be handled, they don't stop the fossil addremove anymore from adding other files. New version, including the unicode-glob setting in [71cd228b0e]. This version allows all characters in filenames that are permissible on UNIX, except '\'. And those files can be checked out using Cygwin and Windows as well, using Cygwin's solution: In the file system translate those characters to 'safe' characters in the range U+F000 to U+F0FF (which should never itself occur in valid fossil cards) I cannot find any security-risks with this. The only possible security risk I found is unrelated to this change, and it is already fixed in [897dfa48b4] (Thanks, Richard!) Anyone who thinks this is a bad idea? Please, speak up now, otherwise Richard might approve this! Regards, Jan Nijtmans ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
2012/11/22 Jan Nijtmans jan.nijtm...@gmail.com: 2012/11/22 Stefan Bellon sbel...@sbellon.de: On Thu, 22 Nov, Jan Nijtmans wrote: Stefan, could you try out [e6a1910fa8]? This sounds very cool. :-) New test version in [d3bee356ba]. It changes the filename contains illegal characters fatal into a warning, which should make your situation more managable: Even though files containing '\' still cannot be handled, they don't stop the fossil addremove anymore from adding other files. Please allow some time to Richard considering this. The currently ticketing-rework has clearly much more priority than this. Regards, Jan Nijtmans ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
On Thu, 22 Nov, Jan Nijtmans wrote: Stefan, could you try out [e6a1910fa8]? This sounds very cool. :-) However I still get: $ ~/tmp/fossil addremove [...] /home/bellonsn/tmp/fossil: filename contains illegal characters: str\i\ng.h This is with: $ ~/tmp/fossil version This is fossil version 1.24 [e6a1910fa8] 2012-11-22 09:32:23 UTC I am on a x86_64 Debian GNU/Linux box. And I did the following to build this version of fossil: $ mkdir ~/tmp cd ~/tmp $ fossil clone http://fossil-scm.org/ fossil.fossil $ fossil open fossil.fossil e6a1910fa8 $ ./configure $ make Am I missing something? Greetings, Stefan -- Stefan Bellon ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
2012/11/22 Stefan Bellon sbel...@sbellon.de: On Thu, 22 Nov, Jan Nijtmans wrote: Stefan, could you try out [e6a1910fa8]? This sounds very cool. :-) However I still get: $ ~/tmp/fossil addremove [...] /home/bellonsn/tmp/fossil: filename contains illegal characters: str\i\ng.h ... Am I missing something? No, you are not missing anything. The backslash is still forbidden, but all other characters should be allowed with this change. It 'could' be possible to do something about backslash as well, but that will need some more clever thinking. I would like this change to be reviewed and accepted first, before trying to do more: Adapting '\' as well could impose security risks I cannot foresee, so it deserves a separate discussion/review. Regards, Jan Nijtmans ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
2012/11/19 Stefan Bellon sbel...@sbellon.de: I think this restriction * Does not contain any of these characters in the path: \*[]? is too arbitrary. If you try to create a file or directory in the Windows explorer and type e.g. a then you get a bubble help with the information that you must not use any of the following characters \ / : * ?| An EXPERIMENTAL fix is committed now in [82ce90f91c]. It handles the invalid characters in the same way as cygwin does: http://cygwin.com/cygwin-ug-net/using-specialnames.html In cygwin it is no problem at all to use such characters in filenames, except for the backslash. I found ':' problematic as well, as it is used in UNIX as path separator (path1:path2) and on Windows as drive separator (C:\foo). The advantage of this method is that using a cygwin-compiled fossil will have exactly the same result as using a win32 version of fossil. The cygwin shell will show all filenames as they are expected to be. From Windows explorer they might look strange, but nothing goes wrong. The only remaining problematic characters are '\' and ':', all others are - on Windows - translated to a safe Unicode range. At this moment I don't see a way to work around the two remaining problematic characters. I think this deserves a lot more testing. I will Regards, jan Nijtmans ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
On Tue, 20 Nov, Richard Hipp wrote: Some uses are less clear. For example, at http://www.fossil-scm.org/fossil/artifact/33d79f5b0a2?ln=850 we are converting an entire command-line, which does likely contain some filenames, but also might contain other text where *?| are legitimate characters. I think doing the conversion will work OK for every case where Fossil calls fossil_system(), but we need to be careful here. I think you are mixing two separate aspects here. One aspect is the encoding of the input string, the other aspect is the set of acceptable characters for a certain use case (e.g. allowed filenames on Windows). What about reflecting those two aspects as separate functionality, i.e. the encoding conversion should apply to *all* input data regardless of whether read from a file, supplied as command line argument or even read via an environment variable (however what encoding should be assumed is quite a different matter). Then the check regarding allowed character set can be applied on the already converted internal unicode representation. But perhaps I am missing some implementation details I am not aware of. Greetings, Stefan -- Stefan Bellon ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] filename contains illegal characters
Hi all, first of all, let me say a big thank you for the Fossil SCM. I'm already using it to store generated files during the build process in order to retrieve them again later on for the various past builds. Now I was trying to convert an existing SVN repository to fossil and stumbled across this problem: ADDED str'i'ng.h ADDED string.h ADDED string.h fossil: filename contains illegal characters: str\i\ng.h This is part of a test suite that tests for weirdly named include files: $ ls -1 str* str'i'ng.h string.h string.h str\i\ng.h Is there some way to add this file to fossil (via some configuration setting) or is this impossible? TIA. Greetings, Stefan -- Stefan Bellon ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
2012/11/19 Stefan Bellon sbel...@sbellon.de: Now I was trying to convert an existing SVN repository to fossil and stumbled across this problem: ADDED str'i'ng.h ADDED string.h ADDED string.h fossil: filename contains illegal characters: str\i\ng.h This sounds like a bug in the SVN -fossil converter, not in fossil itself. Quotes are not illegal in fossil, but the backslash character is. Proof: $ fossil add str'i'ng.h ADDED win/str'i'ng.h $ fossil status ... ADDED str'i'ng.h See, no problem to add it at all. Regards, Jan Nijtmans ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
Hi! On Mon, 19 Nov, Jan Nijtmans wrote: This sounds like a bug in the SVN -fossil converter, not in fossil itself. Just to avoid any misunderstanding: I am not using any converter. I just did $ cd svn_working_copy $ fossil init myproject.fossil $ fossil open --keep myproject.fossil $ fossil addremove And then got the message. Quotes are not illegal in fossil, but the backslash character is. Ok, so then this *IS* the problem. I am trying to add a filename called str\i\ng.h to the repository. So there's no way around than to omit this file from the fossil repository and generated it on the fly with some Makefile? Greetings, Stefan -- Stefan Bellon ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
On Mon, Nov 19, 2012 at 6:37 AM, Stefan Bellon sbel...@sbellon.de wrote: Hi! On Mon, 19 Nov, Jan Nijtmans wrote: This sounds like a bug in the SVN -fossil converter, not in fossil itself. Just to avoid any misunderstanding: I am not using any converter. I just did $ cd svn_working_copy $ fossil init myproject.fossil $ fossil open --keep myproject.fossil $ fossil addremove And then got the message. Quotes are not illegal in fossil, but the backslash character is. Ok, so then this *IS* the problem. I am trying to add a filename called str\i\ng.h to the repository. The \ character is problematic because Windows cannot contain that character in filenames (except as a separator between directory and file names). If you were allowed to check this file into Fossil, then the repository could not be checked out on Windows. So there's no way around than to omit this file from the fossil repository and generated it on the fly with some Makefile? Greetings, Stefan -- Stefan Bellon ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
On Mon, 19 Nov, Richard Hipp wrote: The \ character is problematic because Windows cannot contain that character in filenames (except as a separator between directory and file names). If you were allowed to check this file into Fossil, then the repository could not be checked out on Windows. But the same is true for string.h This file can be added on GNU/Linux to fossil but cannot be checked out on Windows. This seems to me an inconsistency. Either support all allowed filenames per OS and skip them on OSes that cannot support them, or go with the set of filenames that all OSes can support (at least the mainstream ones). But being able to add string.h but not str\i\ng.h seems strange to me. Greetings, Stefan -- Stefan Bellon ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
2012/11/19 Stefan Bellon sbel...@sbellon.de: This seems to me an inconsistency. Either support all allowed filenames per OS and skip them on OSes that cannot support them, or go with the set of filenames that all OSes can support (at least the mainstream ones). But being able to add string.h but not str\i\ng.h seems strange to me. Agreed. I tried it in cygwin, and there I can add files with special characters just fine. But cygwin uses a trick to accomplish that: http://cygwin.com/cygwin-ug-net/using-specialnames.html So, a solution could be, when checking out files on Windows which contain invalid characters there, translate those characters to characters in the range U+F000 - U+F0FF. This way, files with quotes can be inserted in fossil just fine. When using the cygwin shell, everything works fine. When using another shell, the filename might look strange (it looks like: string.h) but apart from that everything could be made to work fine. Regards, Jan Nijtmans ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
On Mon, Nov 19, 2012 at 7:39 AM, Stefan Bellon sbel...@sbellon.de wrote: On Mon, 19 Nov, Richard Hipp wrote: The \ character is problematic because Windows cannot contain that character in filenames (except as a separator between directory and file names). If you were allowed to check this file into Fossil, then the repository could not be checked out on Windows. But the same is true for string.h This file can be added on GNU/Linux to fossil but cannot be checked out on Windows. This seems to me an inconsistency. Either support all allowed filenames per OS and skip them on OSes that cannot support them, or go with the set of filenames that all OSes can support (at least the mainstream ones). But being able to add string.h but not str\i\ng.h seems strange to me. I made the decision to disallow problematic characters in Fossil filenames 6 years ago. In theory, Fossil could allow these characters in filenames. But doing so would enhance the potential for bugs and would greatly complicate testing of the code. And I didn't see any value in doing so. Simpler is better. The code that checks for valid filenames is here: http://www.fossil-scm.org/fossil/artifact/91e4570f9462?ln=479-513 You are welcomed to modify this in any way you see fit. If you want to submit a patch, we will consider adding it. Greetings, Stefan -- Stefan Bellon ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
On Mon, 19 Nov, Richard Hipp wrote: I made the decision to disallow problematic characters in Fossil filenames 6 years ago. In theory, Fossil could allow these characters in filenames. But doing so would enhance the potential for bugs and would greatly complicate testing of the code. And I didn't see any value in doing so. Simpler is better. Yes, agreed. The question is just whether restricting characters is simpler or whether those special Windows exclude rules are rather more complicated. See below for the reasoning. The code that checks for valid filenames is here: http://www.fossil-scm.org/fossil/artifact/91e4570f9462?ln=479-513 I think this restriction * Does not contain any of these characters in the path: \*[]? is too arbitrary. If you try to create a file or directory in the Windows explorer and type e.g. a then you get a bubble help with the information that you must not use any of the following characters \ / : * ?| So either all of those should get rejected by fossil's pathname filter or none at all. But if all of those get rejected, there may be other operating systems out there which have other restrictions (e.g. RISC OS does not allow for a full stop in a filename because . and / are swapped in RISC OS' file system). The above restricted characters are just the ones for Windows. In fact you get into the same kind of trouble with casing as well. On case-sensitive file systems you can add files called foobar.txt FooBar.txt FOOBAR.TXT and so on. But you cannot check them out on a case-insensitive file system (like on Windows) because they would end up in the same file. If you wanted to make sure this cannot happen, you'd have to forbid adding differently cased versions of a filename on e.g. UNIX just because it could cause trouble checking out on Windows. That leads me back to my original statement that I think that simple means allowing all filenames. I think the checkout code that creates files should be able of handling such failures (warn the user but continue) instead having the add code decide whether a filename is nice enough. You are welcomed to modify this in any way you see fit. If you want to submit a patch, we will consider adding it. I'm not yet familiar with fossil's code and what consequences a change at this code part could mean for other areas, i.e. which assumptions on filenames in fossil's internal data structures exist elsewhere. Therefore I'm a bit reluctant to perform such a change myself. Greetings, Stefan -- Stefan Bellon ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] filename contains illegal characters
This is perhaps a non-typical use case but I'm trying to take a daily snapshot of a network share here at work. Thousands of files in thousands of folders over which I don't have any control but for which I'd still like to see a daily change history. I'm using a command like this to get new files into a commit: fossil extras | xargs2 /addquotes fossil add For a few files I see the following error: fossil: filename contains illegal characters: prep_20100113[1] clean/ 135816_0001.ps I understand it's the square brackets that are causing this error but not why this is by design. Could someone explain the rationale? Also, is there a way to get these files into fossil without renaming them? Thanks, Kevin ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
On 28 June 2010 21:20, Kevin Greiner grein...@gmail.com wrote: For a few files I see the following error: fossil: filename contains illegal characters: prep_20100113[1] clean/ 135816_0001.ps I understand it's the square brackets that are causing this error but not why this is by design. Could someone explain the rationale? Also, is there a way to get these files into fossil without renaming them? Well, looking in file.c for the function file_is_simple_pathname (failure of which causes that error in add.c), I see: /* ** Return true if the filename given is a valid filename for ** a file in a repository. Valid filenames follow all of the ** following rules: ** ** * Does not begin with / ** * Does not contain any path element named . or .. ** * Does not contain any of these characters in the path: \*[]? ** * Does not end with /. ** * Does not contain two or more / characters in a row. ** * Contains at least one character */ So it's definitely being done by design. Now the * and ? ban I can see. Files with that in the name would cause confusion on almost every platform. Some quick tests, though, see that files like test[1].txt are permitted under both Linux and Windows so I can't quite understand the justification there. (Under Windows a file with [] in the name has to be put in quotes, mind, but so do files with spaces.) Tragically, fossil won't let me add test[1].txt even if I do put it in quotes, although it will allow me to add test 1.txt. So it looks to me like right now you just can't add those files until (if?) this gets changed. -- Perhaps people don't believe this, but throughout all of the discussions of entering China our focus has really been what's best for the Chinese people. It's not been about our revenue or profit or whatnot. --Sergey Brin, demonstrating the emptiness of the don't be evil mantra. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
On 28 June 2010 at 3:26 pm, Michael Richter wrote: On 28 June 2010 21:20, Kevin Greiner grein...@gmail.com wrote: For a few files I see the following error: fossil: filename contains illegal characters: prep_20100113[1] clean/ 135816_0001.ps I understand it's the square brackets that are causing this error but not why this is by design. Could someone explain the rationale? Also, is there a way to get these files into fossil without renaming them? Well, looking in file.c for the function file_is_simple_pathname (failure of which causes that error in add.c), I see: /* ** Return true if the filename given is a valid filename for ** a file in a repository. Valid filenames follow all of the ** following rules: ** ** * Does not begin with / ** * Does not contain any path element named . or .. ** * Does not contain any of these characters in the path: \*[]? ** * Does not end with /. ** * Does not contain two or more / characters in a row. ** * Contains at least one character */ So it's definitely being done by design. Now the * and ? ban I can see. Files with that in the name would cause confusion on almost every platform. Some quick tests, though, see that files like test[1].txt are permitted under both Linux and Windows so I can't quite understand the justification there. (Under Windows a file with [] in the name has to be put in quotes, mind, but so do files with spaces.) [] are there for the same reason as * and ?: ~ $ ls -d p[lu]* play public_html Quoting stops the shell from expanding that, but doesn't affect what fossil does with it. I think there's an argument for allowing wildcard characters in filenames, but it would inevitably cause other kinds of confusion for users and scripts, so I don't really know which way it should be. Regards, Eric ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] filename contains illegal characters
On 29 June 2010 02:18, Eric e...@deptj.eu wrote:
[] are there for the same reason as * and ?:
~ $ ls -d p[lu]*
play public_html
Ah. I was unaware of that expansion. I always used something like p{l,u}*
in those situations.
--
Perhaps people don't believe this, but throughout all of the discussions of
entering China our focus has really been what's best for the Chinese people.
It's not been about our revenue or profit or whatnot.
--Sergey Brin, demonstrating the emptiness of the don't be evil mantra.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
