Re: [PATCH] gettimeofday: do not use LoadLibrary when built for Windows Store apps

[Daiki Ueno]

Hello,

Bruno Haible  writes:

> diff --git a/lib/stat-w32.c b/lib/stat-w32.c
> index b9163f5..02ad9ab 100644
> --- a/lib/stat-w32.c
> +++ b/lib/stat-w32.c
> @@ -40,18 +40,20 @@
>  #include "pathmax.h"
>  #include "verify.h"
>  
> +#if !(_WIN32_WINNT >= _WIN32_WINNT_VISTA)
> +

I am totally unfamiliar with Windows code, but this change seems to
break MinGW cross build, because GetFinalPathNameByHandleFunc is defined
only if !(_WIN32_WINNT >= _WIN32_WINNT_VISTA), but referred to from
_gl_convert_FILETIME_to_timespec without the guard:

 ../../gl/stat-w32.c: In function '_gl_fstat_by_handle':
 ../../gl/stat-w32.c:259:23: error: 'GetFinalPathNameByHandleFunc' undeclared 
(first use in this function); did you mean 'GetFinalPathNameByHandleW'?
   259 |   || (GetFinalPathNameByHandleFunc != NULL
   |   ^~~~
   |   GetFinalPathNameByHandleW

Regards,
-- 
Daiki Ueno

Re: [PATCH] read-file: add variants that clear internal memory

[Daiki Ueno]

Bruno Haible  writes:

> Let me update the uses of the module 'read-file' in Gnulib.
> I think the next weekly CI run would have caught this.

Thank you; I completely missed those uses in Gnulib.

On a different note, it was suggested to disable stdio buffering if
RF_SENSITIVE is set.  I am attaching a patch for this.

>From 9165e495461db91b8abc42661fc543784d26d0d6 Mon Sep 17 00:00:00 2001
From: Daiki Ueno 
Date: Fri, 29 May 2020 05:45:40 +0200
Subject: [PATCH] read-file: disable buffering if RF_SENSITIVE is set

* lib/read-file.c (read_file): Call setvbuf if RF_SENSITIVE.
Suggested by Glenn Strauss.
---
 ChangeLog   | 6 ++
 lib/read-file.c | 3 +++
 2 files changed, 9 insertions(+)

diff --git a/ChangeLog b/ChangeLog
index 77c637414..0a0e2301a 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,9 @@
+2020-05-29  Daiki Ueno  
+
+	read-file: disable buffering if RF_SENSITIVE is set
+	* lib/read-file.c (read_file): Call setvbuf if RF_SENSITIVE.
+	Suggested by Glenn Strauss.
+
 2020-05-29  Daiki Ueno  
 
 	fopen-gnu-tests: fix "\x" escape usage
diff --git a/lib/read-file.c b/lib/read-file.c
index 36780cc15..3520cbb7b 100644
--- a/lib/read-file.c
+++ b/lib/read-file.c
@@ -195,6 +195,9 @@ read_file (const char *filename, int flags, size_t *length)
   if (!stream)
 return NULL;
 
+  if (flags & RF_SENSITIVE)
+setvbuf (stream, NULL, _IONBF, 0);
+
   out = fread_file (stream, flags, length);
 
   save_errno = errno;
-- 
2.26.2


Regards,
-- 
Daiki Ueno

Re: portability of fopen and 'e' (O_CLOEXEC) flag

[Daiki Ueno]

Bruno Haible  writes:

>> +  ASSERT (fwrite (DATA, 1, sizeof(DATA)-1, f) == sizeof(DATA)-1);
> ...
>> +  ASSERT (fread (buf, 1, sizeof(buf), f) == sizeof(DATA)-1);
>
> GNU coding style wants a space between 'sizeof' and the opening parenthesis.
> Other than that, your patch is perfect.

Thank you for the review; fixed and pushed.  Afterwards, I realized a
compilation error in the test code on FreeBSD, so I've added the
follow-up patch attached.

Regards,
-- 
Daiki Ueno
>From 9326739489050009e3b8834838abc02203c592e5 Mon Sep 17 00:00:00 2001
From: Daiki Ueno 
Date: Fri, 29 May 2020 04:54:31 +0200
Subject: [PATCH] fopen-gnu-tests: fix "\x" escape usage

* tests/test-fopen-gnu.c (DATA): Use safer escape sequence.
---
 ChangeLog  | 5 +
 tests/test-fopen-gnu.c | 2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/ChangeLog b/ChangeLog
index c39bfd372..77c637414 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,8 @@
+2020-05-29  Daiki Ueno  
+
+	fopen-gnu-tests: fix "\x" escape usage
+	* tests/test-fopen-gnu.c (DATA): Use safer escape sequence.
+
 2020-05-28  Bruno Haible  
 
 	Avoid dynamic loading of Windows API functions when possible.
diff --git a/tests/test-fopen-gnu.c b/tests/test-fopen-gnu.c
index 4d98dcd85..7de45ab8d 100644
--- a/tests/test-fopen-gnu.c
+++ b/tests/test-fopen-gnu.c
@@ -30,7 +30,7 @@
 #define BASE "test-fopen-gnu.t"
 
 /* 0x1a is an EOF on Windows.  */
-#define DATA "abc\x1adef"
+#define DATA "abc\x1axyz"
 
 int
 main (void)
-- 
2.26.2

Re: [PATCH] gettimeofday: do not use LoadLibrary when built for Windows Store apps

[Bruno Haible]

Hi,

Steve Lhomme wrote:
> LoadLibrary is forbidden in such apps (can only load DLLs from within the app
> package).
> The API entries are available to all apps linking with the Windows API as 
> found
> here:
> https://docs.microsoft.com/en-us/uwp/win32-and-com/win32-apis

Thanks for these infos.

> +#if !WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_DESKTOP) && _WIN32_WINNT >= 
> 0x0A00 /* _WIN32_WINNT_WIN10 */

The GetSystemTimePreciseAsFileTime function is available starting with Windows 
8,
therefore
  - the condition with WINAPI_FAMILY_PARTITION is not necessary,
  - the condition _WIN32_WINNT >= 0x0A00 is overly restrictive;
_WIN32_WINNT >= _WIN32_WINNT_WIN8 will work just as well.

I have added a page about the native Windows APIs at
https://gitlab.com/ghwiki/gnow-how/-/wikis/Platforms/Native_Windows

Then here is a patch to avoid LoadLibrary when possible.


2020-05-28  Bruno Haible  

Avoid dynamic loading of Windows API functions when possible.
Reported by Steve Lhomme  in
.
* lib/gettimeofday.c (GetProcAddress,
GetSystemTimePreciseAsFileTimeFuncType,
GetSystemTimePreciseAsFileTimeFunc, initialized, initialize): Don't
define in a build for Windows 8 or higher.
* lib/isatty.c (GetProcAddress, GetNamedPipeClientProcessIdFuncType,
GetNamedPipeClientProcessIdFunc, QueryFullProcessImageNameFuncType,
QueryFullProcessImageNameFunc, initialized, initialize): Don't define
in a build for Windows Vista or higher.
* lib/stat-w32.c (GetProcAddress, GetFileInformationByHandleExFuncType,
GetFileInformationByHandleExFunc, GetFinalPathNameByHandleFuncType,
GetFinalPathNameByHandleFunc, initialized, initialize): Likewise.

diff --git a/lib/gettimeofday.c b/lib/gettimeofday.c
index 1980479..3d53115 100644
--- a/lib/gettimeofday.c
+++ b/lib/gettimeofday.c
@@ -33,9 +33,11 @@
 
 #ifdef WINDOWS_NATIVE
 
+# if !(_WIN32_WINNT >= _WIN32_WINNT_WIN8)
+
 /* Avoid warnings from gcc -Wcast-function-type.  */
-# define GetProcAddress \
-   (void *) GetProcAddress
+#  define GetProcAddress \
+(void *) GetProcAddress
 
 /* GetSystemTimePreciseAsFileTime was introduced only in Windows 8.  */
 typedef void (WINAPI * GetSystemTimePreciseAsFileTimeFuncType) (FILETIME 
*lpTime);
@@ -54,6 +56,8 @@ initialize (void)
   initialized = TRUE;
 }
 
+# endif
+
 #endif
 
 /* This is a wrapper for gettimeofday.  It is used only on systems
@@ -84,8 +88,10 @@ gettimeofday (struct timeval *restrict tv, void *restrict tz)
  .  */
   FILETIME current_time;
 
+# if !(_WIN32_WINNT >= _WIN32_WINNT_WIN8)
   if (!initialized)
 initialize ();
+# endif
   if (GetSystemTimePreciseAsFileTimeFunc != NULL)
 GetSystemTimePreciseAsFileTimeFunc (_time);
   else
diff --git a/lib/isatty.c b/lib/isatty.c
index 6cdc0fb..fc771d1 100644
--- a/lib/isatty.c
+++ b/lib/isatty.c
@@ -39,9 +39,11 @@
 # include 
 #endif
 
+#if !(_WIN32_WINNT >= _WIN32_WINNT_VISTA)
+
 /* Avoid warnings from gcc -Wcast-function-type.  */
-#define GetProcAddress \
-  (void *) GetProcAddress
+# define GetProcAddress \
+   (void *) GetProcAddress
 
 /* GetNamedPipeClientProcessId was introduced only in Windows Vista.  */
 typedef BOOL (WINAPI * GetNamedPipeClientProcessIdFuncType) (HANDLE hPipe,
@@ -69,6 +71,8 @@ initialize (void)
   initialized = TRUE;
 }
 
+#endif
+
 static BOOL IsConsoleHandle (HANDLE h)
 {
   DWORD mode;
@@ -84,8 +88,10 @@ static BOOL IsCygwinConsoleHandle (HANDLE h)
   BOOL result = FALSE;
   ULONG processId;
 
+#if !(_WIN32_WINNT >= _WIN32_WINNT_VISTA)
   if (!initialized)
 initialize ();
+#endif
 
   /* GetNamedPipeClientProcessId
  

diff --git a/lib/stat-w32.c b/lib/stat-w32.c
index b9163f5..02ad9ab 100644
--- a/lib/stat-w32.c
+++ b/lib/stat-w32.c
@@ -40,18 +40,20 @@
 #include "pathmax.h"
 #include "verify.h"
 
+#if !(_WIN32_WINNT >= _WIN32_WINNT_VISTA)
+
 /* Avoid warnings from gcc -Wcast-function-type.  */
-#define GetProcAddress \
-  (void *) GetProcAddress
+# define GetProcAddress \
+   (void *) GetProcAddress
 
-#if _GL_WINDOWS_STAT_INODES == 2
+# if _GL_WINDOWS_STAT_INODES == 2
 /* GetFileInformationByHandleEx was introduced only in Windows Vista.  */
 typedef DWORD (WINAPI * GetFileInformationByHandleExFuncType) (HANDLE hFile,

FILE_INFO_BY_HANDLE_CLASS fiClass,
LPVOID lpBuffer,
DWORD 
dwBufferSize);
 static GetFileInformationByHandleExFuncType GetFileInformationByHandleExFunc = 
NULL;
-#endif
+# endif
 /* GetFinalPathNameByHandle was introduced only in Windows Vista.  */
 typedef DWORD (WINAPI * GetFinalPathNameByHandleFuncType) 

Re: [PATCH] stat: remove _GL_WINDOWS_STAT_INODES == 2 support

[Bruno Haible]

Steve Lhomme wrote:
> It may be outdated code, the value is never 2.

The value is currently never 2, correct. But you can see from the patch
that introduced this code [1] that the reason why it's not enabled is
that it requires -D_WIN32_WINNT=_WIN32_WINNT_WIN8 or higher. Obviously
at a moment when many people still ran Windows 7, I did not want to
enable it.

Bruno

[1] https://lists.gnu.org/archive/html/bug-gnulib/2017-05/msg00106.html

Re: explicit_bzero-tests: pacify -Wmissing-declarations

[Paul Eggert]

On 5/28/20 3:24 PM, Bruno Haible wrote:
> Agreed. Please go ahead. I'll deal with possible xlc/Sun cc/MSVC build 
> failures
> then.

OK, I installed the attached.
>From ef7cc081f5c0d1b71f60f71d5e0f4a9ddf76bb28 Mon Sep 17 00:00:00 2001
From: Paul Eggert 
Date: Thu, 28 May 2020 16:02:32 -0700
Subject: [PATCH] explicit_bzero-tests: improve -Wmissing-declarations
 pacification

* tests/test-explicit_bzero.c: Now noinline.
Suggested by Bruno Haible in:
https://lists.gnu.org/r/bug-gnulib/2020-05/msg00300.html
---
 ChangeLog   | 7 +++
 tests/test-explicit_bzero.c | 2 +-
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/ChangeLog b/ChangeLog
index 602923d3f..0a62f572d 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,10 @@
+2020-05-28  Paul Eggert  
+
+	explicit_bzero-tests: improve -Wmissing-declarations pacification
+	* tests/test-explicit_bzero.c: Now noinline.
+	Suggested by Bruno Haible in:
+	https://lists.gnu.org/r/bug-gnulib/2020-05/msg00300.html
+
 2020-05-28  Bruno Haible  
 
 	Fix build errors due to read-file changes (regression from 2020-05-27).
diff --git a/tests/test-explicit_bzero.c b/tests/test-explicit_bzero.c
index 385fd12a9..b5698f898 100644
--- a/tests/test-explicit_bzero.c
+++ b/tests/test-explicit_bzero.c
@@ -128,7 +128,7 @@ test_heap (void)
  2. Verify that the memory has been erased.
Implement them in the same function, so that they access the same memory
range on the stack.  */
-static int
+static int _GL_ATTRIBUTE_NOINLINE
 do_secret_stuff (volatile int pass)
 {
   char stackbuf[SECRET_SIZE];
-- 
2.26.1

Re: explicit_bzero-tests: pacify -Wmissing-declarations

[Bruno Haible]

Paul Eggert wrote:
> > Is it possible to portably mark 'do_secret_stuff' as not-inline?
> 
> We can use _GL_ATTRIBUTE_NOINLINE. That should be good enough for any
> GCC-compatible compiler. We don't need to worry much about the other 
> compilers;
> most of them will work anyway, and if any don't we could deal with them
> individually I suppose.

Agreed. Please go ahead. I'll deal with possible xlc/Sun cc/MSVC build failures
then.

Bruno

Re: explicit_bzero-tests: pacify -Wmissing-declarations

[Paul Eggert]

On 5/25/20 12:28 AM, Bruno Haible wrote:
> Is it possible to portably mark 'do_secret_stuff' as not-inline?

We can use _GL_ATTRIBUTE_NOINLINE. That should be good enough for any
GCC-compatible compiler. We don't need to worry much about the other compilers;
most of them will work anyway, and if any don't we could deal with them
individually I suppose.

Re: [PATCH] read-file: add variants that clear internal memory

[Bruno Haible]

> Both has been fixed and pushed.  Thank you for the review!

Let me update the uses of the module 'read-file' in Gnulib.
I think the next weekly CI run would have caught this.


2020-05-28  Bruno Haible  

Fix build errors due to read-file changes (regression from 2020-05-27).
* lib/git-merge-changelog.c (read_changelog_file): Update read_file
invocation.
* tests/test-sameacls.c (main): Likewise.
* tests/test-pipe-filter-gi1.c (main): Call read_file instead of
read_binary_file.
* tests/test-pipe-filter-ii1.c (main): Likewise.

diff --git a/lib/git-merge-changelog.c b/lib/git-merge-changelog.c
index 1e6dae1..7b74a49 100644
--- a/lib/git-merge-changelog.c
+++ b/lib/git-merge-changelog.c
@@ -1,5 +1,5 @@
 /* git-merge-changelog - git "merge" driver for GNU style ChangeLog files.
-   Copyright (C) 2008-2010 Bruno Haible 
+   Copyright (C) 2008-2020 Bruno Haible 
 
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@@ -300,7 +300,7 @@ read_changelog_file (const char *filename, struct 
changelog_file *result)
   /* Read the file in text mode, otherwise it's hard to recognize empty
  lines.  */
   size_t length;
-  char *contents = read_file (filename, );
+  char *contents = read_file (filename, 0, );
   if (contents == NULL)
 {
   fprintf (stderr, "could not read file '%s'\n", filename);
diff --git a/tests/test-pipe-filter-gi1.c b/tests/test-pipe-filter-gi1.c
index 4ee9375..0994610 100644
--- a/tests/test-pipe-filter-gi1.c
+++ b/tests/test-pipe-filter-gi1.c
@@ -80,7 +80,7 @@ main (int argc, char *argv[])
 
   /* Read some text from a file.  */
   input_filename = argv[2];
-  input = read_binary_file (input_filename, _size);
+  input = read_file (input_filename, RF_BINARY, _size);
   ASSERT (input != NULL);
 
   /* Convert it to uppercase, line by line.  */
diff --git a/tests/test-pipe-filter-ii1.c b/tests/test-pipe-filter-ii1.c
index 5f31d37..5a56c55 100644
--- a/tests/test-pipe-filter-ii1.c
+++ b/tests/test-pipe-filter-ii1.c
@@ -102,7 +102,7 @@ main (int argc, char *argv[])
 
   /* Read some text from a file.  */
   input_filename = argv[2];
-  input = read_binary_file (input_filename, _size);
+  input = read_file (input_filename, RF_BINARY, _size);
   ASSERT (input != NULL);
 
   /* Convert it to uppercase, line by line.  */
diff --git a/tests/test-sameacls.c b/tests/test-sameacls.c
index cdb10f4..6aad92f 100644
--- a/tests/test-sameacls.c
+++ b/tests/test-sameacls.c
@@ -55,14 +55,14 @@ main (int argc, char *argv[])
 size_t size2;
 char *contents2;
 
-contents1 = read_file (file1, );
+contents1 = read_file (file1, 0, );
 if (contents1 == NULL)
   {
 fprintf (stderr, "error reading file %s: errno = %d\n", file1, errno);
 fflush (stderr);
 abort ();
   }
-contents2 = read_file (file2, );
+contents2 = read_file (file2, 0, );
 if (contents2 == NULL)
   {
 fprintf (stderr, "error reading file %s: errno = %d\n", file2, errno);

Re: use O_CLOEXEC in more places

[Bruno Haible]

Paul Eggert wrote:
> > How about module 'fts'? Should the directory fds that it allocates also be
> > made O_CLOEXEC?
> 
> Yes, I'd say so; I see little reason for a child process to continue an fts 
> scan.

Done through this patch. In fact, you had already done most of the work
on 2017-08-12.

Bruno
>From 708c76165712d6cbad59379b7e039870f81ca63c Mon Sep 17 00:00:00 2001
From: Bruno Haible 
Date: Thu, 28 May 2020 21:48:13 +0200
Subject: [PATCH] fts: Make more robust in multithreaded applications.

* lib/fts.c (fts_open): Pass an O_CLOEXEC flag to open().
* modules/fts (Depends-on): Add 'open'.
---
 ChangeLog   | 6 ++
 lib/fts.c   | 2 +-
 modules/fts | 1 +
 3 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/ChangeLog b/ChangeLog
index e637b57..4145796 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,11 @@
 2020-05-28  Bruno Haible  
 
+	fts: Make more robust in multithreaded applications.
+	* lib/fts.c (fts_open): Pass an O_CLOEXEC flag to open().
+	* modules/fts (Depends-on): Add 'open'.
+
+2020-05-28  Bruno Haible  
+
 	relocatable-prog: Make more robust in multithreaded applications.
 	* lib/progreloc.c (O_CLOEXEC): Define fallback to 0 when use from module
 	relocatable-prog-wrapper.
diff --git a/lib/fts.c b/lib/fts.c
index 3b2f693..5e357be 100644
--- a/lib/fts.c
+++ b/lib/fts.c
@@ -402,7 +402,7 @@ fts_open (char * const *argv,
early, doing it here saves us the trouble of ensuring
later (where it'd be messier) that "." can in fact
be opened.  If not, revert to FTS_NOCHDIR mode.  */
-int fd = open (".", O_SEARCH);
+int fd = open (".", O_SEARCH | O_CLOEXEC);
 if (fd < 0)
   {
 /* Even if "." is unreadable, don't revert to FTS_NOCHDIR mode
diff --git a/modules/fts b/modules/fts
index 480700b..b06d5b8 100644
--- a/modules/fts
+++ b/modules/fts
@@ -22,6 +22,7 @@ hash
 i-ring
 lstat
 memmove
+open
 openat-h
 opendir
 opendirat
-- 
2.7.4

Re: portability of fopen and 'e' (O_CLOEXEC) flag

[Bruno Haible]

Hi Daiki,

Thank you for noticing this, and the rapid fix.

> Sorry, attached an old patch; this would be simpler (and also supports
> other platforms that need O_BINARY).

> +  ASSERT (fwrite (DATA, 1, sizeof(DATA)-1, f) == sizeof(DATA)-1);
...
> +  ASSERT (fread (buf, 1, sizeof(buf), f) == sizeof(DATA)-1);

GNU coding style wants a space between 'sizeof' and the opening parenthesis.
Other than that, your patch is perfect.

Thanks again!

Bruno

Re: portability of fopen and 'e' (O_CLOEXEC) flag

[Daiki Ueno]

Daiki Ueno  writes:

> Bruno Haible  writes:
>
>>> Here are proposed patches for other modules. Does this look right?
>>
>> There were no objections. I pushed the changes.
>
> Thank you for this.  I have rebased GnuTLS on top of it, but noticed a
> strange test failures on Windows CI, which involve reading binary files
> (OCSP response):
> https://gitlab.com/gnutls/gnutls/-/jobs/569815031
>
> It seems that the fopen module ignores a 'b' flag.  The attached patch
> fixes the failures.

Sorry, attached an old patch; this would be simpler (and also supports
other platforms that need O_BINARY).

>From 81695244eb467603009a2777c3a8438f1a707954 Mon Sep 17 00:00:00 2001
From: Daiki Ueno 
Date: Thu, 28 May 2020 11:40:49 +0200
Subject: [PATCH] fopen-gnu: make 'b' flag can be used with 'e' on Windows

* lib/fopen.c (rpl_fopen): Pass O_BINARY to open, if a 'b' flag is
specified on Windows.
* tests/test-fopen-gnu.c (DATA): New define.
(main): Add test for reading binary files with an 'e' flag.
---
 ChangeLog  |  8 
 lib/fopen.c|  4 
 tests/test-fopen-gnu.c | 17 +
 3 files changed, 29 insertions(+)

diff --git a/ChangeLog b/ChangeLog
index c17b76b72..ea2716b2f 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,11 @@
+2020-05-28  Daiki Ueno  
+
+	fopen-gnu: make 'b' flag can be used with 'e' on Windows
+	* lib/fopen.c (rpl_fopen): Pass O_BINARY to open, if a 'b' flag is
+	specified on Windows.
+	* tests/test-fopen-gnu.c (DATA): New define.
+	(main): Add test for reading binary files with an 'e' flag.
+
 2020-05-27  Bruno Haible  
 
 	Don't assume that UNICODE is not defined.
diff --git a/lib/fopen.c b/lib/fopen.c
index 20065e4c6..47d7f194d 100644
--- a/lib/fopen.c
+++ b/lib/fopen.c
@@ -101,6 +101,10 @@ rpl_fopen (const char *filename, const char *mode)
 #endif
 continue;
   case 'b':
+/* While it is non-standard, O_BINARY is guaranteed by
+   gnulib .  We can also assume that orig_fopen
+   supports the 'b' flag.  */
+open_flags_standard |= O_BINARY;
 #if GNULIB_FOPEN_GNU
 if (q < fdopen_mode_buf + BUF_SIZE)
   *q++ = *p;
diff --git a/tests/test-fopen-gnu.c b/tests/test-fopen-gnu.c
index cae40421a..eeb1712c7 100644
--- a/tests/test-fopen-gnu.c
+++ b/tests/test-fopen-gnu.c
@@ -29,15 +29,20 @@
 
 #define BASE "test-fopen-gnu.t"
 
+/* 0x1a is an EOF on Windows.  */
+#define DATA "abc\x1adef"
+
 int
 main (void)
 {
   FILE *f;
   int fd;
   int flags;
+  char buf[16];
 
   /* Remove anything from prior partial run.  */
   unlink (BASE "file");
+  unlink (BASE "binary");
 
   /* Create the file.  */
   f = fopen (BASE "file", "w");
@@ -64,8 +69,20 @@ main (void)
   ASSERT (f == NULL);
   ASSERT (errno == EEXIST);
 
+  /* Open a binary file and check that the 'e' mode doesn't interfere.  */
+  f = fopen (BASE "binary", "wbe");
+  ASSERT (f);
+  ASSERT (fwrite (DATA, 1, sizeof(DATA)-1, f) == sizeof(DATA)-1);
+  ASSERT (fclose (f) == 0);
+
+  f = fopen (BASE "binary", "rbe");
+  ASSERT (f);
+  ASSERT (fread (buf, 1, sizeof(buf), f) == sizeof(DATA)-1);
+  ASSERT (fclose (f) == 0);
+
   /* Cleanup.  */
   ASSERT (unlink (BASE "file") == 0);
+  ASSERT (unlink (BASE "binary") == 0);
 
   return 0;
 }
-- 
2.26.2



Regards,
-- 
Daiki Ueno

Re: portability of fopen and 'e' (O_CLOEXEC) flag

[Daiki Ueno]

Bruno Haible  writes:

>> Here are proposed patches for other modules. Does this look right?
>
> There were no objections. I pushed the changes.

Thank you for this.  I have rebased GnuTLS on top of it, but noticed a
strange test failures on Windows CI, which involve reading binary files
(OCSP response):
https://gitlab.com/gnutls/gnutls/-/jobs/569815031

It seems that the fopen module ignores a 'b' flag.  The attached patch
fixes the failures.

Regards,
-- 
Daiki Ueno
>From 17fbb2560a05e3006125f8793c8e814ef5baa847 Mon Sep 17 00:00:00 2001
From: Daiki Ueno 
Date: Thu, 28 May 2020 11:40:49 +0200
Subject: [PATCH] fopen-gnu: make 'b' flag can be used with 'e' on Windows

* lib/fopen.c (rpl_fopen): Pass O_BINARY to open, if a 'b' flag is
specified on Windows.
* tests/test-fopen-gnu.c (DATA): New define.
(main): Add test for reading binary files with an 'e' flag.
---
 ChangeLog  |  8 
 lib/fopen.c|  9 +++--
 tests/test-fopen-gnu.c | 17 +
 3 files changed, 32 insertions(+), 2 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index c17b76b72..ea2716b2f 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,11 @@
+2020-05-28  Daiki Ueno  
+
+	fopen-gnu: make 'b' flag can be used with 'e' on Windows
+	* lib/fopen.c (rpl_fopen): Pass O_BINARY to open, if a 'b' flag is
+	specified on Windows.
+	* tests/test-fopen-gnu.c (DATA): New define.
+	(main): Add test for reading binary files with an 'e' flag.
+
 2020-05-27  Bruno Haible  
 
 	Don't assume that UNICODE is not defined.
diff --git a/lib/fopen.c b/lib/fopen.c
index 20065e4c6..f60c51f95 100644
--- a/lib/fopen.c
+++ b/lib/fopen.c
@@ -51,6 +51,7 @@ rpl_fopen (const char *filename, const char *mode)
   int open_flags_standard;
 #if GNULIB_FOPEN_GNU
   int open_flags_gnu;
+  int open_flags_other;
 # define BUF_SIZE 80
   char fdopen_mode_buf[BUF_SIZE + 1];
 #endif
@@ -66,6 +67,7 @@ rpl_fopen (const char *filename, const char *mode)
   open_flags_standard = 0;
 #if GNULIB_FOPEN_GNU
   open_flags_gnu = 0;
+  open_flags_other = 0;
 #endif
   {
 const char *p = mode;
@@ -101,6 +103,9 @@ rpl_fopen (const char *filename, const char *mode)
 #endif
 continue;
   case 'b':
+#if defined _WIN32 && ! defined __CYGWIN__
+	open_flags_other |= O_BINARY;
+#endif
 #if GNULIB_FOPEN_GNU
 if (q < fdopen_mode_buf + BUF_SIZE)
   *q++ = *p;
@@ -142,9 +147,9 @@ rpl_fopen (const char *filename, const char *mode)
 #endif
   }
 #if GNULIB_FOPEN_GNU
-  open_flags = open_flags_standard | open_flags_gnu;
+  open_flags = open_flags_standard | open_flags_other | open_flags_gnu;
 #else
-  open_flags = open_flags_standard;
+  open_flags = open_flags_standard | open_flags_other;
 #endif
 
 #if FOPEN_TRAILING_SLASH_BUG
diff --git a/tests/test-fopen-gnu.c b/tests/test-fopen-gnu.c
index cae40421a..eeb1712c7 100644
--- a/tests/test-fopen-gnu.c
+++ b/tests/test-fopen-gnu.c
@@ -29,15 +29,20 @@
 
 #define BASE "test-fopen-gnu.t"
 
+/* 0x1a is an EOF on Windows.  */
+#define DATA "abc\x1adef"
+
 int
 main (void)
 {
   FILE *f;
   int fd;
   int flags;
+  char buf[16];
 
   /* Remove anything from prior partial run.  */
   unlink (BASE "file");
+  unlink (BASE "binary");
 
   /* Create the file.  */
   f = fopen (BASE "file", "w");
@@ -64,8 +69,20 @@ main (void)
   ASSERT (f == NULL);
   ASSERT (errno == EEXIST);
 
+  /* Open a binary file and check that the 'e' mode doesn't interfere.  */
+  f = fopen (BASE "binary", "wbe");
+  ASSERT (f);
+  ASSERT (fwrite (DATA, 1, sizeof(DATA)-1, f) == sizeof(DATA)-1);
+  ASSERT (fclose (f) == 0);
+
+  f = fopen (BASE "binary", "rbe");
+  ASSERT (f);
+  ASSERT (fread (buf, 1, sizeof(buf), f) == sizeof(DATA)-1);
+  ASSERT (fclose (f) == 0);
+
   /* Cleanup.  */
   ASSERT (unlink (BASE "file") == 0);
+  ASSERT (unlink (BASE "binary") == 0);
 
   return 0;
 }
-- 
2.26.2

Re: bug#41558: Regexp Bug

[Norihiro Tanaka]

On Tue, 26 May 2020 21:14:12 -0700
"anton.paras"  wrote:

> I posted to Stack Exchange, and they recommended that I file a bug. I'd 
> rather not copy+paste it all, so here's the link:
> 
> 
> 
> https://unix.stackexchange.com/questions/579889/why-doesnt-this-sed-command-replace-the-3rd-to-last-and
> 
> 
> 
> here's an example
> 
> 
> 
> > echo 'dog and foo and bar and baz land good' |??? sed -E 
> > 's/(.*)\band\b((.*\band\b){2})/\1XYZ\2/'
> 
> 
> 
> expected output:?dog XYZ foo and bar and baz land good
> 
> actual output:?dog and foo XYZ bar and baz land good
> 
> 
> here's my sed --version output:?sed (GNU sed) 4.2.2
> 
> 
> 
> I hope this is helpful, cheers!

$ echo 'foo and bar land' | env LC_ALL=en_US.utf8 sed -nE '/(.*\band){2}/p'
foo and bar land
$ echo 'foo and bar land' | env LC_ALL=en_US.utf8 sed -nE '/.*\band.*\band/p'
$

It seems that there is the bug in regex.

expected:
$ echo 'foo and bar land' | env LC_ALL=en_US.utf8 sed -nE '/(.*\band){2}/p'
$ echo 'foo and bar land' | env LC_ALL=en_US.utf8 sed -nE '/.*\band.*\band/p'
$

It also reproduces in grep.

$ echo 'foo and bar land' | env LC_ALL=en_US.utf8 grep -E '(.*\band){2}'
foo and bar land
$ echo 'foo and bar land' | env LC_ALL=en_US.utf8 grep -E '.*\band.*\band'
$

Re: [PATCH 2/2] stat: do not use LoadLibrary when built for Windows Store apps

[Steve Lhomme]

Any update on this ?

Just as in gettimeofday, one cannot use LoadLibrary to load system DLLs 
in UWP builds. But they are available by static linking with windowsapp 
and are guaranteed to be there.


On 2020-05-19 8:26, Steve Lhomme wrote:

LoadLibrary is forbidden in such apps (can only load DLLs from within the app
package).
The API entries are available to all apps linking with the Windows API as found
here:
https://docs.microsoft.com/en-us/uwp/win32-and-com/win32-apis

windowsapp.lib (and mincore.lib for Windows 8) are both available in MinGW as
well.

GetFinalPathNameByHandleA is only allowed in Win10 UWP apps
GetFileInformationByHandleEx is allowed in Win8 and Win10 UWP apps.
---
  lib/stat-w32.c | 10 ++
  1 file changed, 10 insertions(+)

diff --git a/lib/stat-w32.c b/lib/stat-w32.c
index 106d25120..6900dfcf5 100644
--- a/lib/stat-w32.c
+++ b/lib/stat-w32.c
@@ -61,6 +61,15 @@ static BOOL initialized = FALSE;
  static void
  initialize (void)
  {
+#if !WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_DESKTOP)
+  /* LoadLibrary not allowed but the functions are available with the windows 
runtime */
+#if _GL_WINDOWS_STAT_INODES == 2
+  GetFileInformationByHandleExFunc = GetFileInformationByHandleEx;
+#endif
+#if _WIN32_WINNT >= 0x0A00 /* _WIN32_WINNT_WIN10 */
+  GetFinalPathNameByHandleFunc = GetFinalPathNameByHandleA;
+#endif
+#else /* WINAPI_PARTITION_DESKTOP */
HMODULE kernel32 = LoadLibrary ("kernel32.dll");
if (kernel32 != NULL)
  {
@@ -71,6 +80,7 @@ initialize (void)
GetFinalPathNameByHandleFunc =
  (GetFinalPathNameByHandleFuncType) GetProcAddress (kernel32, 
"GetFinalPathNameByHandleA");
  }
+#endif /* WINAPI_PARTITION_DESKTOP */
initialized = TRUE;
  }
  
--

2.26.2

Re: [PATCH] gettimeofday: do not use LoadLibrary when built for Windows Store apps

[Steve Lhomme]

Any update on this patch ?

On Dekstop it's better to use kernel32.dll as it's loaded with every 
process, so the LoadLibrary is not loading any new DLL.


On Winstore/UWP apps you cannot use LoadLibrary, only LoadLibraryFromApp 
which cannot be used to load system DLLs. Static linking in necessary in 
this case. Any app targeting UWP is already using the windowsapp.lib 
(replacing the kernel32 lib they used to link with) so no need to fore 
linking with it via pkg-config, libtool, etc.


For example in CLang you either link with windowsapp or kernel32:
https://github.com/llvm-project/clang/blob/master/lib/Driver/ToolChains/MinGW.cpp#L269

On 2020-05-19 8:24, Steve Lhomme wrote:

LoadLibrary is forbidden in such apps (can only load DLLs from within the app
package).
The API entries are available to all apps linking with the Windows API as found
here:
https://docs.microsoft.com/en-us/uwp/win32-and-com/win32-apis

windowsapp.lib (and mincore.lib for Windows 8) are both available in MinGW as
well.

GetSystemTimePreciseAsFileTime is only allowed in Win10 UWP apps.
---
  lib/gettimeofday.c | 5 +
  1 file changed, 5 insertions(+)

diff --git a/lib/gettimeofday.c b/lib/gettimeofday.c
index 19804793a..087f7eada 100644
--- a/lib/gettimeofday.c
+++ b/lib/gettimeofday.c
@@ -45,12 +45,17 @@ static BOOL initialized = FALSE;
  static void
  initialize (void)
  {
+#if !WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_DESKTOP) && _WIN32_WINNT >= 
0x0A00 /* _WIN32_WINNT_WIN10 */
+  /* LoadLibrary not allowed but the functions are available with the windows 
runtime */
+  GetSystemTimePreciseAsFileTimeFunc = GetSystemTimePreciseAsFileTime;
+#else /* WINAPI_PARTITION_DESKTOP */
HMODULE kernel32 = LoadLibrary ("kernel32.dll");
if (kernel32 != NULL)
  {
GetSystemTimePreciseAsFileTimeFunc =
  (GetSystemTimePreciseAsFileTimeFuncType) GetProcAddress (kernel32, 
"GetSystemTimePreciseAsFileTime");
  }
+#endif /* WINAPI_PARTITION_DESKTOP */
initialized = TRUE;
  }
  
--

2.26.2