RE: FW: MAX_PATH problems with mingw gcc

2013-10-27 Thread Vladimir Simonov 
 -Original Message-
 From: Joey Ye [mailto:joey.ye...@gmail.com] 
 Sent: Monday, October 21, 2013 6:31 AM
 To: Vladimir Simonov; Ian Lance Taylor; d...@redhat.com
 Vladimir,
 
 I found no more issue on patch itself. But ChangeLogs are missing.
 Please refer to format in /include/ChangeLog, and generate yours in your next 
 email for
 
 /include
 /libcpp
 /libiberty
 
 Maintainers of libiberty are in TO list.
 
Hi Jan, DJ,

Could you please clarify your positions about the problem and suggested patch?
Attaching the patch for references.

TBD. Here is not complete list of problems in gcc-based cross development when
build platform uses DOS-style FS and target platform uses Linux-style FS:
1. If you do crossbuild with gcov support, binaries refer gcda files like 
C:\Myproject\Mydir\Myobj.gcda.
After run on Linux files with names  like C:\Myproject\Mydir\Myobj.gcda are 
created.
No any directory tree;
2. Current filename_cmp fails to compare c:/a/b/.. and c:/a;
3. Include files names on DOS-style FS quite often become too long because of 
their 
denormalization after concatenation(i.e a/ + ../b becomes a/../b but may be 
used as b/);
4. The same (as 1.) problems appear in gdb during debug cross-built programs on 
Linux side.

Best regards
Vladimir



gcc-4.8.1-filename-normalize-2.patch
Description: gcc-4.8.1-filename-normalize-2.patch

Re: FW: MAX_PATH problems with mingw gcc

2013-10-20 Thread Joey Ye 
Vladimir,

I found no more issue on patch itself. But ChangeLogs are missing.
Please refer to format in /include/ChangeLog, and generate yours in
your next email for

/include
/libcpp
/libiberty

Maintainers of libiberty are in TO list.

- Joey

RE: FW: MAX_PATH problems with mingw gcc

2013-10-19 Thread Vladimir Simonov 
 -Original Message-
 From: Joey Ye [mailto:joey.ye...@gmail.com] 
 Sent: Friday, October 18, 2013 6:37 AM

 I do encourage you to upstream this patch, though I'm not the maintainer of 
 libiberty to approve it.

OK. Thank you.
Here it is.
Could you push it to appropriate people?

Attaching test for it also.
To do tests:
- copy patched filename_cmp.c and filenames.h to new dir
- touch  safe-ctype.h
- create  hashtab.h with
typedef int size_t;
typedef int hashval_t;
#define TOLOWER(c) (c)
This is enough for me to test it on cygwin32. It may be that other platforms 
will require another fake includes.
- gcc -O2 filename_cmp.c main.c -o test1

Best regards
Vladimir


gcc-4.8.1-filename-normalize-2.patch
Description: gcc-4.8.1-filename-normalize-2.patch
#include filenames.h

char *tests[] = {
test, /* 0 */
test,

./test, /* 1 */
./test,
/./test, /* 2 */
/test,
.\\test, /* 3 */
./test,
\\.\\test, /* 4 */
/test,
C, /* 5 */
C,
., /* 6 */
.,
, /* 7 */
,

/../test, /* 8 */
/test,
C:/test/dir1/dir2, /* 9 */
C:/test/dir1/dir2,
/test/./dir2, /* 10 */
/test/dir2,
/test/././dir2, /* 11 */
/test/dir2,
/test/././/dir2, /* 12 */
/test/dir2,
/test/../dir2, /* 13 */
/dir2,
C:/test/dir1/dir2/../dir3, /* 14 */
C:/test/dir1/dir3,
/test/dir1/dir2/../../dir3, /* 15 */
/test/dir3,
/test/dir1/dir2/../../../dir3, /* 16 */
/dir3,
/test/dir1/dir2/../../../../dir3, /* 17 */
/dir3,
C:/test/dir1/dir2/../../../../aaa/../../././././dir3, /* 18 */
C:/dir3,
/../..//././././dir3, /* 19 */
/dir3,

\\..\\test, /* 20 */
/test,
C:\\test\\dir1\\dir2, /* 21 */
C:/test/dir1/dir2,
\\test\\.\\dir2, /* 22 */
/test/dir2,
\\test\\.\\.\\dir2, /* 23 */
/test/dir2,
\\test\\.\\.dir2, /* 24 */
/test/dir2,
\\test\\..\\dir2, /* 25 */
/dir2,
C:\\test\\dir1\\dir2\\..\\dir3, /* 26 */
C:/test/dir1/dir3,
\\test\\dir1\\dir2\\..\\..\\dir3, /* 27 */
/test/dir3,
\\test\\dir1\\dir2\\..\\..\\..\\dir3, /* 28 */
/dir3,
\\test\\dir1\\dir2\\..\\..\\..\\..\\dir3, /* 29 */
/dir3,
\\test\\dir1\\dir2\\..\\..\\..\\..\\aaa\\..\\..\\..\\.\\.\\dir3, /* 30 */
/dir3,
C:\\..\\...\\.\\.\\.\\dir3, /* 31 */
C:/dir3,

../test, /* 32 */
../test,
C:test/dir1/dir2, /* 33 */
C:test/dir1/dir2,
test/./dir2, /* 34 */
test/dir2,
test/././dir2, /* 35 */
test/dir2,
test/././/dir2, /* 36 */
test/dir2,
test/../dir2, /* 37 */
dir2,
C:test/dir1/dir2/../dir3, /* 38 */
C:test/dir1/dir3,
test/dir1/dir2/../../dir3, /* 39 */
test/dir3,
C:test/dir1/dir2/../../../dir3, /* 40 */
C:dir3,
test/dir1/dir2/../../../../dir3, /* 41 */
../dir3,
test/dir1/dir2/../../../../aaa/../../././././dir3, /* 42 */
../../dir3,
../..//././././dir3, /* 43 */
../../dir3,
../../../../dir3/aaa, /* 44 */
../../../../dir3/aaa,

..\\test, /* 45 */
../test,
C:test\\dir1\\dir2, /* 46 */
C:test/dir1/dir2,
test\\.\\dir2, /* 47 */
test/dir2,
test\\.\\.\\dir2, /* 48 */
test/dir2,
test\\.\\.dir2, /* 49 */
test/dir2,
test\\..\\dir2, /* 50 */
dir2,
C:test\\dir1\\dir2\\..\\dir3, /* 51 */
C:test/dir1/dir3,
test\\dir1\\dir2\\..\\..\\dir3, /* 52 */
test/dir3,
C:test\\dir1\\dir2\\..\\..\\..\\dir3, /* 53 */
C:dir3,
test\\dir1\\dir2\\..\\..\\..\\..\\dir3, /* 54 */
../dir3,
test\\dir1\\dir2\\..\\..\\..\\..\\aaa\\..\\..\\.\\.\\.\\.\\dir3, /* 55 */
../../dir3,
..\\...\\.\\.\\.\\dir3, /* 56 */
../../dir3,
..\\..\\..\\..\\dir3\\aaa, /* 57 */
../../../../dir3/aaa,

/, /* 58 */
/,
\\, /* 59 */
/,
C:test\\dir1\\dir2\\..\\..\\..\\dir3\\, /* 60 */
C:dir3/,
C:test/dir1/dir2/..\\..\\..\\dir3/, /* 61 */
C:dir3/,
C:\\test\\dir1\\dir2\\..\\..\\..\\dir3\\, /* 62 */
C:/dir3/,
C:\\test/dir1/dir2/..\\..\\..\\dir3/, /* 63 */
C:/dir3/,

0
};

int main(int argc, char **argv)
{
  int i;

  i = 0;
  while (tests[i])
{
  char *p = strdup(tests[i]);
  filename_normalize (p);
  if (strcmp(p, tests[i+1]))
{
  printf(Error[%d]: %s != %s, orig %s\n, i/2, p, tests[i+1], 
tests[i]);
}
  free(p);
  i += 2;
}
}

RE: FW: MAX_PATH problems with mingw gcc

2013-10-17 Thread Vladimir Simonov 
 -Original Message-
 From: Joey Ye [mailto:joey.ye...@gmail.com] 
 
 There is an issue on file system with symbolic link, like Linux ext2/3. It 
 could vitally change behavior of GCC. The issue is described as following.
 
 Such a logic cannot be deduced simple from the name string, so your patch can 
 do nothing about it.
 
 For your patch IMHO the way out could be just define a real function for DOS 
 FS, and leave a blank function body otherwise. Like:
 filename_normalize (char *fn)
 {
 #if DOS
 ...
 #endif
 }

Thank you for pointing this problem. 

So, on file systems with symlinks support playing with filenames as strings 
is impossible.
This means that filename_normalize name is too pretentious - it will do nothing 
for most of gcc consumers.
From other side - it is useful for OS-es with small MAX_PATH.
Do you think it should be renamed as filename_shortcut/filename_simplify or 
something like it?
So readers won't expect too much from it even without looking at its body.

Is it allowed to write
# ifdef HAVE_DOS_BASED_FILE_SYSTEM
extern void filename_normalize (char *f); 
#else
#define filename_normalize (char *f)
#endif
directly in include/filenames.h?
This way we'll avoid unnecessary empty call on platforms with symlinks. 

And more common question.
I can imagine that different filenames produced after cross build on different 
OS-es for the same target
may confuse some upper-level tools, like code analyzers, code coverage, etc...
Does it make sense to push such solution to gcc mainsteam?
May be it is better to keep the patch for cross toolchains builders...

Regards
Vladimir

RE: FW: MAX_PATH problems with mingw gcc

2013-10-17 Thread Vladimir Simonov 
Thank you for review. Please see inline

 From: Joey Ye [mailto:joey.ye...@gmail.com] 
 Sent: Thursday, October 17, 2013 7:36 AM
 
 The macro FILENAME_NORMALIZE doesn't look necessary. I would prefer use 
 filename_normalize directly, just as filename_cmp is used in GCC rather than 
 FILENAME_CMP
The patch is quite old(from gcc 4.2, may be 4.3). As I remember those days 
filenames.h had only
extern int filename_cmp (const char *s1, const char *s2);
#define FILENAME_CMP(s1, s2)filename_cmp(s1, s2)

at the end. So I just followed its style. Nowadays the style is changed.
I'll use filename_normalize everywhere.



 Also it normalize filename like:
 c:\abc\ into c:/abc\
The ending \ isn't normalized. Need to refine logic here to handle this case.
Thank you.
I was concentrated on filenames shorting  correctness and missed this case.
In fact it is not too bad - the patch doesn't guarantee that all filenames 
inside gcc process will be
as short as possible and will have forward slashes. It just simplifies 
filenames in some key places.
Anyway, I'll add this case to my test and fix it.

Other comments are accepted without notes:)
I'll redo the patch in several days.

Thank you
Vladimir

Re: FW: MAX_PATH problems with mingw gcc

2013-10-17 Thread Joey Ye 
On Thu, Oct 17, 2013 at 4:18 PM, Vladimir Simonov
vladimir.simo...@acronis.com wrote:

 Thank you for pointing this problem.

 So, on file systems with symlinks support playing with filenames as strings 
 is impossible.
 This means that filename_normalize name is too pretentious - it will do 
 nothing for most of gcc consumers.
 From other side - it is useful for OS-es with small MAX_PATH.
 Do you think it should be renamed as filename_shortcut/filename_simplify or 
 something like it?
 So readers won't expect too much from it even without looking at its body.

 Is it allowed to write
 # ifdef HAVE_DOS_BASED_FILE_SYSTEM
 extern void filename_normalize (char *f);
 #else
 #define filename_normalize (char *f)
 #endif
 directly in include/filenames.h?
 This way we'll avoid unnecessary empty call on platforms with symlinks.
#define a lower case function name isn't a good practice. So please
resume your original change that define FILENAME_XXX here like:
#ifdef HAVE_DOS_BASED_FILE_SYSTEM
#define FILENAME_XXX(f) filename_xxx(f)
#else
#define FILENAME_XXX(f)
#endif

 And more common question.
 I can imagine that different filenames produced after cross build on 
 different OS-es for the same target
 may confuse some upper-level tools, like code analyzers, code coverage, etc...
 Does it make sense to push such solution to gcc mainsteam?
 May be it is better to keep the patch for cross toolchains builders...
IMO it is not a concern. One reason libiberty is there, is because
people know different OS-es have different filename system. A tool
should either use libiberty to process it, or smarter enough to handle
difference by themselves.

Another good thing about the idea of filename_normalize is that it
make possible to do real filename_cmp. Current filename_cmp fails to
compare c:/a/b/.. and c:/a. You patch is at least a start to solve it.

I do encourage you to upstream this patch, though I'm not the
maintainer of libiberty to approve it.

Thanks,
Joey

FW: MAX_PATH problems with mingw gcc

2013-10-16 Thread Vladimir Simonov 
Hi,

Resending filename-normalize patch to correct list gcc-patches@gcc.gnu.org.
All context, please, see below.

Regards
Vladimir Simonov

-Original Message-
From: Vladimir Simonov 
Sent: Wednesday, October 16, 2013 12:52 PM
To: 'Jiri Dobry'; 'gcc-h...@gcc.gnu.org'
Subject: RE: MAX_PATH problems with mingw gcc

Hi all,

Sending new version of filename-normalize patch because people asking to make 
it public.

This version also fixes problems with gcov support of Linux binaries 
crosscompiled on Windows.
Problem: If you do crossbuild with gcov support, binaries refer gcda files like 
C:\Myproject\Mydir\Myobj.gcda.
After run on Linux files with names  like C:\Myproject\Mydir\Myobj.gcda are 
created.
No any directory tree.
The last chunk of new patch fixes this.

Best regards
Vladimir Simonov

-Original Message-
From: Simonov, Vladimir 
Sent: Monday, May 27, 2013 8:15 PM
To: gcc-h...@gcc.gnu.org
Subject: MAX_PATH problems with mingw gcc

Hi all,

Constructions #include ../../some_module/some_header.h are quite popular in 
our project. They lead to funny for Linux user problem which becomes real 
headache when we build the project by mingw based cross gcc.

Several levels of such include may easy exceed Windows MAX_PATH limitation 
for file I/O functions (260 symbols).

For example
g:/w/project1/product1/bundle/console/mv_system/centralized/dialogs/selectors/../../../../machines/dialogs/selectors/../../../components/machines/browsers/tree_based_group_browser/../../../hierarchy_model_browser/hierarchy_model_browser.h:37:61:
 fatal error: ../../remote/async_invokers/client/async_action.h: No such file 
or directory  #include ../../remote/async_invokers/client/async_action.h

As you see gcc tries to open
g:/w/project1/product1/bundle/console/mv_system/centralized/dialogs/selectors/../../../../machines/dialogs/selectors/../../../components/machines/browsers/tree_based_group_browser/../../../hierarchy_model_browser/../../remote/async_invokers/client/async_action.h
which length is 263 symbols.

IMO the best solution would be to switch on Unicode version of file IO 
functions in libiberty but it is quite radical change.

Attaching patch which works for me many years and illustrates another approach.
Its main idea is to simplify paths in some key places working with filenames 
(libcpp, parse_include in directives.c and find_file_in_dir in files.c).
With this patch code above compiles OK.

There are many pro and contras, i.e. it adds additional, probably unnecessary 
work on Linux time but makes filenames shorter, it affects libiberty which is 
shared between gcc/binutils/gdb, but it may be useful in other packages, etc., 
etc.

Anyway I'll be glad to hear any comments and may be to see some fix for above 
problem in gcc code.

Best regards
Vladimir Simonov



gcc-4.8.1-filename-normalize.patch
Description: gcc-4.8.1-filename-normalize.patch

Re: FW: MAX_PATH problems with mingw gcc

2013-10-16 Thread Joey Ye 
Thanks for contribution. See review comments at following.

On Wed, Oct 16, 2013 at 6:45 PM, Vladimir Simonov
vladimir.simo...@acronis.com wrote:
 Hi,

 Resending filename-normalize patch to correct list gcc-patches@gcc.gnu.org.
 All context, please, see below.


 +extern void filename_normalize (char *f);
 +#define FILENAME_NORMALIZE(f)  filename_normalize(f)
The macro FILENAME_NORMALIZE doesn't look necessary. I would prefer
use filename_normalize directly, just as filename_cmp is used in GCC
rather than FILENAME_CMP

 +  FILENAME_NORMALIZE (path);
Likewise. Change to lower case file name.

 +#ifndef HAVE_DOS_BASED_FILE_SYSTEM
 +void
 +filename_normalize (char *fn)
 +#else
 +static void
 +filename_normalize_unix (char *fn)
 +#endif
Redefining function names is easy to be confusing. At least I went up
and down several times to understand difference of two functions. I
feel it much clear to define a single filename_normalize, and then use
HAVE_DOS_BASED_FILE_SYSTEM to inline additional stuff for DOS into
this function. Like
filename_normalize (char *fn)
{
...
#ifdef HAVE_DOS_BASED_FILE_SYSTEM
// Additional work that your current DOS version does, basically
moving fn if needed.
#endif
... // Rest of what you have now
}

 +  next = p + 1;
 +  if (*next == '\0')
 +break;
 +  if (!IS_DIR_SEPARATOR( *p))
 +  continue;
Fix wrong indent at continue.

Also it normalize filename like:
c:\abc\ into c:/abc\
The ending \ isn't normalized. Need to refine logic here to handle this case.

+#ifdef HAVE_DOS_BASED_FILE_SYSTEM
+void
+filename_normalize (char *fn)
+{
+  if (IS_DIR_SEPARATOR (fn[0]) || ! IS_ABSOLUTE_PATH (fn))
+/* Absolute path in Unix style or relative path */
+filename_normalize_unix (fn);
+  else if (fn[1] != '\0')
+filename_normalize_unix (fn + 2);
+}
+#endif
+
Inline into unified version of filename_normalize

 +filename_normalize (char *fn)
 +{
 +  char *p;
 +  int rest;
It will be more robust to check if fn is NULL at function entry. By
doing so, you can remove the if (pwd) condition in getpwd.c

if (!pwd)
 +{
 pwd = getcwd (XNEWVEC (char, MAXPATHLEN + 1), MAXPATHLEN + 1
 #ifdef VMS
  , 0
 #endif
  );
 +if (pwd)
 +  FILENAME_NORMALIZE (pwd);
 +}
All code inside {} should be two more spaces indent here. And remove if (pwd).

Thanks,
Joey

Re: FW: MAX_PATH problems with mingw gcc

2013-10-16 Thread Joey Ye 
On Wed, Oct 16, 2013 at 6:45 PM, Vladimir Simonov
vladimir.simo...@acronis.com wrote:
 There are many pro and contras, i.e. it adds additional, probably unnecessary 
 work on Linux
 time but makes filenames shorter, it affects libiberty which is shared 
 between gcc/binutils/gdb,
 but it may be useful in other packages, etc.,
 etc.
There is an issue on file system with symbolic link, like Linux
ext2/3. It could vitally change behavior of GCC. The issue is
described as following.

Given that on Linux you have following directory structure:

base/
   level1-a/
  level2-a/
   level2-b (- level1-a/level2-a)

level2-b is a symbolic link created by:
ln -s level1-a/level2-a level1-b

Now run ls base/level2-b/.. you probably would expect it equal to
ls base, as the logic in your patch implies. However, the actual
behavior is equal to ls base/level1-a instead, because
base/level2-b/.. == base/level1-a/level2-a/.. == base/level1-a

Such a logic cannot be deduced simple from the name string, so your
patch can do nothing about it.

For your patch IMHO the way out could be just define a real function
for DOS FS, and leave a blank function body otherwise. Like:
filename_normalize (char *fn)
{
#if DOS
...
#endif
}

Thanks,
Joey