Re: random(3) in OpenBSD
Hi Bruno,
On Nov 14 19:33, Bruno Haible wrote:
> [CCing bug-gnulib]
>
> Hi Corinna,
>
> > Looking into all the BSD variants of random(), I found that OpenBSD
> > defaults the random() function to return non-deterministic output. It
> > works like this:
> >
> > - A global var random_deterministic defaults to 0.
> > - Calls to srandom() set random_deterministic to 0 and ignore their
> > input.
> > - Calls to srandom_deterministic() sets random_deterministic to 1 and
> > seeds the usual PRNG.
> > - initstate() and setstate() set random_deterministic to 1.
> > - random() returns either the next value from the PRNG if
> > random_deterministic is 1, or the next value generated by a call to
> > arc4random() if random_deterministic is 0. So by default, it
> > returns arc4random() values.
> >
> > With that behaviour, OpenBSD appears to stray from POSIX deliberately.
>
> Yup, I observed this too when I found that the gnulib MT test for random()
> yielded strange results on OpenBSD. [1]
>
> > I was wondering if you want to add this as incompatibility to your
> > document at
> > https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=doc/posix-functions/random.texi
> > perhaps.
>
> I documented it as an incompatibility for the srandom() function. But you are
> right, it's also an incompatibility for the random() function, since POSIX [2]
> says "random() shall produce by default a sequence of numbers that can be
> duplicated by calling srandom() with 1 as the seed".
Yes, that's the problem. However, I think your addition to random.texi
isn't quite right.
> +When @samp{srandom} was not called, this function returns a non-deterministic
> +sequence rather than a deterministic sequence of numbers on some platforms:
> +OpenBSD 7.4.
In fact, as I outlined above, srandom(3) does *not* initiate a seed for
the deterministic PRNG. It just sets the global var random_deterministic
to 0.
In turn, random(3) will call arc4random(), which typically generates
non-deterministic values from getentropy(3) or from /dev/random.
An application which requires deterministic, reproducible values from
a PRNG has to call initstate(), setstate(), or the OpenBSD-specific
function srandom_deterministic().
HTH,
Corinna
Re: [PATCH] Do not decorate symbols as dllexport on Cygwin
On Feb 17 09:30, Reuben Thomas wrote: > On Sat, 11 Feb 2023 at 12:40, Corinna Vinschen wrote: > > > On Feb 11 12:29, Reuben Thomas wrote: > > > On Sat, 11 Feb 2023 at 12:06, Corinna Vinschen > > wrote: > > > > > > > > > > > If you're sure that the native recode.dll has been built, is it > > possible > > > > that it's just not found, because it's not in Windows' DLL search path? > > > > > > > > > > > Yes, because it was working fine before, and I've not changed anything > > > about where it is built. > > > > Uhm, ok, but... given you didn't build with -no-undefined yet, I'm > > pretty sure libtool only built a static librecode.a, so there's a good > > chance cython was linked statrically against the recode lib, no? > > > > Thanks, that was the answer; adding the DLL's build directory to PATH on > Windows fixes the tests. I'm glad I could help. > I have some issues with building dependencies of Recode on Debian to > investigate, and then I'll make another release. Great, thanks! Corinna
Re: [PATCH] Do not decorate symbols as dllexport on Cygwin
On Feb 11 12:29, Reuben Thomas wrote: > On Sat, 11 Feb 2023 at 12:06, Corinna Vinschen wrote: > > > > > If you're sure that the native recode.dll has been built, is it possible > > that it's just not found, because it's not in Windows' DLL search path? > > > > > Yes, because it was working fine before, and I've not changed anything > about where it is built. Uhm, ok, but... given you didn't build with -no-undefined yet, I'm pretty sure libtool only built a static librecode.a, so there's a good chance cython was linked statrically against the recode lib, no? Corinna
Re: [PATCH] Do not decorate symbols as dllexport on Cygwin
On Feb 11 11:36, Reuben Thomas wrote: > On Fri, 10 Feb 2023 at 14:21, Bruno Haible wrote: > > > It complains about the symbols defined in libiconv. This means, you need > > to invoke the Gnulib module 'iconv' and add $(LIBICONV) or $(LTLIBICONV) > > to the LDFLAGS. > > > > Bruno to the rescue again! Many thanks. > > Having added the iconv gnulib module, the Windows build works, but the > tests, which use a Cython module, are now broken. The actual error is: > > Traceback (most recent call last): > File "D:\a\recode\recode\tests\pytest", line 135, in main > module = __import__(base[:-3]) > File "D:/a/recode/recode/tests/./t21_names.py", line 2, in > import common > File "D:/a/recode/recode/tests/./common.py", line 10, in > import Recode > ImportError: DLL load failed while importing Recode: The specified module > could not be found. > > I assume that the "DLL load failed" indicates a link error. If you're sure that the native recode.dll has been built, is it possible that it's just not found, because it's not in Windows' DLL search path? Keep in mind that Windows searches DLLs in the OS default paths, in $PATH, or in the same dir the executable has been started from. https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order So at a guess, you either just have to add the directory where recode.dll resides to $PATH, or you have to copy the DLL into a path already in $PATH. Corinna
Re: [PATCH] Do not decorate symbols as dllexport on Cygwin
Hi Reuben, On Feb 7 14:22, Corinna Vinschen wrote: > On Feb 6 20:50, Reuben Thomas wrote: > > On Mon, 6 Feb 2023 at 20:38, Bruno Haible wrote: > > > > > 1. 'recode' is updated to a current gnulib and publish a corresponding > > > tarball. (Hi Reuben :) ) > > > > > > > I've updated gnulib in recode git; I'd be grateful if I could have a report > > that it's doing what's needed here before I make a release, if possible! > > > > Git at: https://github.com/rrthomas/recode.git > > Working on it, but this may take a while. Stay tuned, please. The new recode gnulib build works as desired, thank you. However, I'm trying to build recode from the Cygwin-specific build script, and running it with default settings I get an error before recode even got built: help2man: can't get `--help' info from ./recode.exe Try `--no-discard-stderr' if option outputs to stderr WARNING: '/usr/bin/help2man' is missing on your system. [...] So even with help2man installed, it doesn't work because recode.exe didn't get build yet. The Cygwin build system prefers parallel make. And that's the problem. I think commit dcdd5d26c0c2 is wrong: -recode.1: main.c $(top_srcdir)/configure.ac recode$(EXEEXT) +recode.1: main.c $(top_srcdir)/configure.ac This means that recode.1 doesn't depend on the built binary anymore even though it *needs* said binary to create the man page from it. This breaks dependency tracking in make and so make happily runs the recode.1: rule before recode has been built, just depending on the existence of main.c and configure.ac. For cross-checking I tried a Linux build with `make -j8' and help2man gets called before recode even got built as well. So the above change breaks parallel make. Not sure how to fix this correctly, but I have another patch, which would be nice to have: In Cygwin projects using libtool, we always have to add -no-undefined iLDFLAGS. This is some old safe-guard in libtool to remind developers that when creating Windows DLLs, all external symbols must be resolved. Fortunately, libtool allows this flags also on other platforms which don't require its usage. Patch is attached. It would be nice if that's ok for inclusion. Thanks, Corinna >From 9ea601d4615d8420e7cf75659b647728774f65c9 Mon Sep 17 00:00:00 2001 From: Corinna Vinschen Date: Tue, 7 Feb 2023 16:09:29 +0100 Subject: [PATCH] src/Makefile.am: librecode_la_LDFLAGS: add -no-undefined -no-undefined is a libtool requirement to build shared libs on Windows-based platforms like Cygwin. --- src/Makefile.am | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/Makefile.am b/src/Makefile.am index 93d3ed3634b1..427cc8c97fe7 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -52,7 +52,7 @@ recode_LDADD = librecode.la librecode_la_SOURCES = charname.c combine.c fr-charname.c iconv.c \ names.c outer.c recode.c request.c strip-pool.c task.c $(ALL_STEPS) \ $(include_HEADERS) $(noinst_HEADERS) $(H_STEPS) -librecode_la_LDFLAGS = -version-info $(VERSION_INFO) $(LTLIBINTL) \ +librecode_la_LDFLAGS = -no-undefined -version-info $(VERSION_INFO) $(LTLIBINTL) \ $(LIB_CLOCK_GETTIME) $(LIB_GETRANDOM) $(LIB_HARD_LOCALE) $(LIB_MBRTOWC) $(LIB_SETLOCALE_NULL) librecode_la_LIBADD = ../lib/libgnu.la libmerged.la -- 2.39.1
Re: [PATCH] Do not decorate symbols as dllexport on Cygwin
On Feb 6 20:50, Reuben Thomas wrote: > On Mon, 6 Feb 2023 at 20:38, Bruno Haible wrote: > > > 1. 'recode' is updated to a current gnulib and publish a corresponding > > tarball. (Hi Reuben :) ) > > > > I've updated gnulib in recode git; I'd be grateful if I could have a report > that it's doing what's needed here before I make a release, if possible! > > Git at: https://github.com/rrthomas/recode.git Working on it, but this may take a while. Stay tuned, please. Thanks, Corinna
Re: [PATCH] Do not decorate symbols as dllexport on Cygwin
On Feb 6 21:38, Bruno Haible wrote: > Corinna Vinschen wrote: > > I just ran the setlocale_null-mt-* tests with gnulib from git master > > under Cygwin with my newlib patch, and the tests succeeded. I checked > > that configure is doing the right thing and config.h contains the right > > settings: > > > > $ ./configure > > [...] > > checking whether setlocale (LC_ALL, NULL) is multithread-safe... yes > > checking whether setlocale (category, NULL) is multithread-safe... yes > > [...] > > $ grep SETLOCALE_NULL config.h > > #define GNULIB_TEST_SETLOCALE_NULL 1 > > #define SETLOCALE_NULL_ALL_MTSAFE 1 > > #define SETLOCALE_NULL_ONE_MTSAFE 1 > > $ grep SETLOCALE_NULL gltests/config.h > > #define GNULIB_TEST_SETLOCALE_NULL 1 > > #define SETLOCALE_NULL_ALL_MTSAFE 1 > > #define SETLOCALE_NULL_ONE_MTSAFE 1 > > Excellent! Thank you. > > The lasts remaining steps, to resolve the issue, are (as far as I see it): > 1. 'recode' is updated to a current gnulib and publish a corresponding > tarball. (Hi Reuben :) ) > 2. Then the Cygwin program packagers build this tarball on a Cygwin of > version ≥ 3.4.6. Incidentally I'm the maintainer of the recode package in the Cygwin distro and I built 3.7.12 (as in Fedora 37) with -Wl,--export-all-symbols, which also does the trick for now. Thanks, Corinna
Re: [PATCH] Do not decorate symbols as dllexport on Cygwin
On Feb 6 18:37, Bruno Haible wrote: > Corinna Vinschen wrote: > > This patch will be in the next Cygwin release 3.4.6. > > Thanks! > > > I'm just a bit fuzzy what patches will be required for gnulib now... > > With this patch, the setlocale_null lock should be gone in Cygwin >= 3.4.6. > I can't really test it, but I hope the patch is correct. I just ran the setlocale_null-mt-* tests with gnulib from git master under Cygwin with my newlib patch, and the tests succeeded. I checked that configure is doing the right thing and config.h contains the right settings: $ ./configure [...] checking whether setlocale (LC_ALL, NULL) is multithread-safe... yes checking whether setlocale (category, NULL) is multithread-safe... yes [...] $ grep SETLOCALE_NULL config.h #define GNULIB_TEST_SETLOCALE_NULL 1 #define SETLOCALE_NULL_ALL_MTSAFE 1 #define SETLOCALE_NULL_ONE_MTSAFE 1 $ grep SETLOCALE_NULL gltests/config.h #define GNULIB_TEST_SETLOCALE_NULL 1 #define SETLOCALE_NULL_ALL_MTSAFE 1 #define SETLOCALE_NULL_ONE_MTSAFE 1 Thanks a lot, Corinna > > > 2023-02-06 Bruno Haible > > setlocale-null: Don't use a lock in Cygwin >= 3.4.6. > Road paved by Corinna Vinschen . > * m4/setlocale_null.m4 (gl_FUNC_SETLOCALE_NULL): Assume that > setlocale (LC_ALL, NULL) is multithread-safe in Cygwin >= 3.4.6. > * lib/setlocale_null.c: Update comments. > * tests/test-setlocale_null-mt-all.c: Likewise. > > diff --git a/lib/setlocale_null.c b/lib/setlocale_null.c > index 6ac563db14..89c8a06598 100644 > --- a/lib/setlocale_null.c > +++ b/lib/setlocale_null.c > @@ -173,7 +173,7 @@ setlocale_null_unlocked (int category, char *buf, size_t > bufsize) > #endif > } > > -#if !(SETLOCALE_NULL_ALL_MTSAFE && SETLOCALE_NULL_ONE_MTSAFE) /* musl libc, > macOS, FreeBSD, NetBSD, OpenBSD, AIX, Haiku, Cygwin */ > +#if !(SETLOCALE_NULL_ALL_MTSAFE && SETLOCALE_NULL_ONE_MTSAFE) /* musl libc, > macOS, FreeBSD, NetBSD, OpenBSD, AIX, Haiku, Cygwin < 3.4.6 */ > > /* Use a lock, so that no two threads can invoke setlocale_null_unlocked > at the same time. */ > @@ -198,7 +198,7 @@ setlocale_null_with_lock (int category, char *buf, size_t > bufsize) >return ret; > } > > -# elif HAVE_PTHREAD_API /* musl libc, macOS, FreeBSD, NetBSD, OpenBSD, AIX, > Haiku, Cygwin */ > +# elif HAVE_PTHREAD_API /* musl libc, macOS, FreeBSD, NetBSD, OpenBSD, AIX, > Haiku, Cygwin < 3.4.6 */ > > extern > # if defined _WIN32 || defined __CYGWIN__ > diff --git a/m4/setlocale_null.m4 b/m4/setlocale_null.m4 > index dd6a5ef538..b41df499a8 100644 > --- a/m4/setlocale_null.m4 > +++ b/m4/setlocale_null.m4 > @@ -1,4 +1,4 @@ > -# setlocale_null.m4 serial 6 > +# setlocale_null.m4 serial 7 > dnl Copyright (C) 2019-2023 Free Software Foundation, Inc. > dnl This file is free software; the Free Software Foundation > dnl gives unlimited permission to copy and/or distribute it, > @@ -13,9 +13,23 @@ AC_DEFUN([gl_FUNC_SETLOCALE_NULL], >AC_CACHE_CHECK([whether setlocale (LC_ALL, NULL) is multithread-safe], > [gl_cv_func_setlocale_null_all_mtsafe], > [case "$host_os" in > - # Guess no on musl libc, macOS, FreeBSD, NetBSD, OpenBSD, AIX, Haiku, > Cygwin. > - *-musl* | darwin* | freebsd* | midnightbsd* | netbsd* | openbsd* | > aix* | haiku* | cygwin*) > + # Guess no on musl libc, macOS, FreeBSD, NetBSD, OpenBSD, AIX, Haiku. > + *-musl* | darwin* | freebsd* | midnightbsd* | netbsd* | openbsd* | > aix* | haiku*) > gl_cv_func_setlocale_null_all_mtsafe=no ;; > + # Guess no on Cygwin < 3.4.6. > + cygwin*) > + AC_EGREP_CPP([Lucky user], > + [ > +#if defined __CYGWIN__ > + #include > + #if CYGWIN_VERSION_DLL_COMBINED >= CYGWIN_VERSION_DLL_MAKE_COMBINED (3004, > 6) > + Lucky user > + #endif > +#endif > + ], > + [gl_cv_func_setlocale_null_all_mtsafe=yes], > + [gl_cv_func_setlocale_null_all_mtsafe=no]) > +;; > # Guess yes on glibc, HP-UX, IRIX, Solaris, native Windows. > *-gnu* | gnu* | hpux* | irix* | solaris* | mingw*) > gl_cv_func_setlocale_null_all_mtsafe=yes ;; > diff --git a/tests/test-setlocale_null-mt-all.c > b/tests/test-setlocale_null-mt-all.c > index 6036c260cd..7480406639 100644 > --- a/tests/test-setlocale_null-mt-all.c > +++ b/tests/test-setlocale_null-mt-all.c > @@ -166,7 +166,7 @@ Solaris 11.0 OK > Solaris 11.4 OK > Solaris OpenIndiana OK > Haikucrash < 1 sec > -Cygwin crash < 1 sec > +Cygwin < 3.4.6 crash < 1 sec > mingwOK > MSVC OK (assuming compiler option /MD !) > */ > >
Re: [PATCH] Do not decorate symbols as dllexport on Cygwin
Hi Bruno, On Feb 5 22:22, Corinna Vinschen wrote: > On Feb 5 21:41, Bruno Haible wrote: > > Another option — since we are talking about a single symbol and a single > > platform — would be if the locking for setlocale_null were not necessary > > on Cygwin in the first place. I determined that it is necessary by running > > the unit test gnulib/tests/test-setlocale_null-mt-all.c [3] on Cygwin: > > without the lock, it crashed within less than 1 second. Could the > > implementation of setlocale() in Cygwin be changed in such a way that this > > test does not crash? Then the lock would be necessary. > > Well, we could do that by adding Cygwin-internal locking to setlocale > calls. But that would only be available in the next Cygwin version > of course. > > May I ask what's the idea to provide a thread-safe setlocale? It was > never defined as thread-safe and POSIX explicitely mentions that. Any > application expecting to call setlocale thread-safe is broken by design. > It should use the newlocale/duplocale/uselocale/freelocale API instead, > isn't it? Ahhh, I finally see what's going on. The problem is not thread-safety as such, but thread-safety when reading the value of the LC_ALL category. Glibc's setlocale isn't entirely thread-safe either, but there's a difference: - GLibc creates the global strings returned by setlocale(LC_xxx, NULL) at the time the locale data is changed. All setlocale(LC_xxx, NULL) calls only return pointer to strings created earlier. - Cygwin or, better, newlib, also return a pointer to global strings. However, while the global strings for the specific categories are created when the locale is changed, the string returned for LC_ALL gets created on the fly when setlocale(LC_ALL, NULL) is called. That's why test-setlocale_null-mt-all fails almost immediately. I created a patch to newlib's setlocale to tweak the LC_ALL string each time the locale is changed, while setlocale(LC_ALL, NULL) just returns the already prepared string. https://cygwin.com/git/?p=newlib-cygwin.git;a=commitdiff;h=23e49b18ce39 This patch will be in the next Cygwin release 3.4.6. I'm just a bit fuzzy what patches will be required for gnulib now... Thanks, Corinna
Re: [PATCH] Do not decorate symbols as dllexport on Cygwin
On Feb 5 21:41, Bruno Haible wrote: > Hi Corinna, > > Thanks for the heads-up and patch. > > > Note that dllimport/dllexport decorations are not at all required on > > Cygwin for quite some time. > > Is this true in all circumstances? I thought that when --disable-auto-import > is in use AND the shared library wants to export variables, dllimport > and dllexport decorations are necessary. > > > On Cygwin --export-all-symbols is default. However, if just a single > > symbol is decorated with dllexport, ld switches to exporting only these > > symbols. > > ... > > An example of that is current recode. Building this package with > > --enable-shared is broken, unless one adds -Wl,--export-all-symbols to > > the link stage command line explicitely, because of the decorated > > gl_get_setlocale_null_lock variable. > > So, there are three types of packages: > (A) Those that use --disable-auto-import (e.g. [1][2]) and provide an > explicit list of exports for their shared libraries. > (B) Those that don't use --disable-auto-import but want to limit > the exposed symbols for other reasons (e.g. namespace cleanliness). > Such as those that use the libtool options '-export-symbols' or > '-export-symbols-regex'. > (C) Those like 'recode', which want all symbols to be exported (e.g. > if namespace cleanliness is not an issue for them). > > For packages of type (A) or (B), the symbol gl_get_setlocale_null_lock > *MUST* be exported, otherwise there may be different get_setlocale_null_lock > functions accessed by different code, which will destroy the purpose of this > function, i.e. open the door to potential crashes due to use of setlocale(). > > For packages of type (C), the symbol gl_get_setlocale_null_lock *MUST NOT* > be exported, otherwise all other symbols will not be exported. > > AFAIU, your patch fixes packages of type (C), while at the same time breaking > packages of type (A) or (B). > > Is there an easy way to distinguish packages of type (C) from those of type > (A), (B)? > > Another option — since we are talking about a single symbol and a single > platform — would be if the locking for setlocale_null were not necessary > on Cygwin in the first place. I determined that it is necessary by running > the unit test gnulib/tests/test-setlocale_null-mt-all.c [3] on Cygwin: > without the lock, it crashed within less than 1 second. Could the > implementation of setlocale() in Cygwin be changed in such a way that this > test does not crash? Then the lock would be necessary. Well, we could do that by adding Cygwin-internal locking to setlocale calls. But that would only be available in the next Cygwin version of course. May I ask what's the idea to provide a thread-safe setlocale? It was never defined as thread-safe and POSIX explicitely mentions that. Any application expecting to call setlocale thread-safe is broken by design. It should use the newlocale/duplocale/uselocale/freelocale API instead, isn't it? Corinna > > Bruno > > [1] > https://git.savannah.gnu.org/gitweb/?p=gettext.git;a=blob;f=gettext-tools/woe32dll/export.h;hb=HEAD > [2] https://haible.de/bruno/woe32dll.html > [3] > https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=tests/test-setlocale_null-mt-all.c;hb=HEAD > >
[PATCH] Do not decorate symbols as dllexport on Cygwin
Note that dllimport/dllexport decorations are not at all required on
Cygwin for quite some time.
Worse, this breaks building DLLs and DLL import libs using libtool.
On Cygwin --export-all-symbols is default. However, if just a single
symbol is decorated with dllexport, ld switches to exporting only these
symbols.
So, when creating DLLs and DLL import libs which are also linked against
libgnu.a, the result is that the actual DLL (i. e. cygfoo-1.dll)
contains all symbols of all object files given on the command line.
However, the DLL import lib (i. e. libfoo.dll.a) will contain only the
default symbols and the single symbol gl_get_setlocale_null_lock.
This in turn breaks linking against the just created cygfoo-1.dll.
An example of that is current recode. Building this package with
--enable-shared is broken, unless one adds -Wl,--export-all-symbols to
the link stage command line explicitely, because of the decorated
gl_get_setlocale_null_lock variable.
Fix this by dropping the dllexport decoration when building for the
Cygwin target.
* lib/setlocale-lock.c: don't decorate gl_get_setlocale_null_lock
as dllexport on Cygwin.
* lib/setlocale_null.c: don't dllimport gl_get_setlocale_null_lock
on Cygwin.
---
lib/setlocale-lock.c | 2 +-
lib/setlocale_null.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/setlocale-lock.c b/lib/setlocale-lock.c
index b70ba09b0020..ab23cadd2939 100644
--- a/lib/setlocale-lock.c
+++ b/lib/setlocale-lock.c
@@ -41,7 +41,7 @@ typedef int dummy;
# if HAVE_VISIBILITY
/* Override the effect of the compiler option '-fvisibility=hidden'. */
# define DLL_EXPORTED __attribute__((__visibility__("default")))
-# elif defined _WIN32 || defined __CYGWIN__
+# elif defined _WIN32 && !defined __CYGWIN__
# define DLL_EXPORTED __declspec(dllexport)
# else
# define DLL_EXPORTED
diff --git a/lib/setlocale_null.c b/lib/setlocale_null.c
index 6ac563db14a9..5bf04a5ad84d 100644
--- a/lib/setlocale_null.c
+++ b/lib/setlocale_null.c
@@ -201,7 +201,7 @@ setlocale_null_with_lock (int category, char *buf, size_t
bufsize)
# elif HAVE_PTHREAD_API /* musl libc, macOS, FreeBSD, NetBSD, OpenBSD, AIX,
Haiku, Cygwin */
extern
-# if defined _WIN32 || defined __CYGWIN__
+# if defined _WIN32 && !defined __CYGWIN__
__declspec(dllimport)
# endif
pthread_mutex_t *gl_get_setlocale_null_lock (void);
--
2.39.1
Re: [Grep-devel] handling of non-BMP characters
On Dec 19 15:41, Corinna Vinschen wrote: > On Dec 19 08:51, Bruno Haible wrote: > > Corinna Vinschen wrote in > > <https://lists.gnu.org/archive/html/grep-devel/2018-12/msg00039.html>: > > > it would be > > > pretty nice if that code could get reverted back in to support > > > non-BMP charsets even on Cygwin. > > > > I agree that support for beyond-BMP characters should be added back to > > 'grep'. > > > > Your earlier fix from 2013-08-16 (and the fact that the test failure is > > occurring exactly on Windows and AIX platforms) shows that the problem is > > with wchar_t being only 16-bit wide on these platforms. > > > > The type 'char32_t' has been introduced in C11 to overcome this > > limitation.[1] > > > > I propose to > > > > 1) introduce in gnulib support for , char32_t, and mbrtoc32, so > > that we can use these instead of , wchar_t, and mbrtowc > > portably, > > > > 2) change those gnulib modules that don't behave well with beyond-BMP > > characters on Windows and AIX to use char32_t instead of wchar_t. > > > > Then the 'grep' code can be changed in a similar way, and this will > > fix the bug on Cygwin and AIX (though not on native Windows [2]). > > > > The advantage of this approach are minimal code changes in 'grep': just > > change some type and function names here and there, and add code for > > the additional (size_t)(-3) return value of mbrtoc32. > > IIUC this would also drop the requirement for #ifdef CYGWIN'ed code. ... in grep. > Sounds like a great idea to me! > > > Corinna > > > > > > > Bruno > > > > [1] > > https://stackoverflow.com/questions/21264035/why-did-c11-introduce-the-char16-t-and-char32-t-types > > [2] https://lists.gnu.org/archive/html/bug-gnulib/2011-02/msg00175.html
Re: [Grep-devel] handling of non-BMP characters
On Dec 19 08:51, Bruno Haible wrote: > Corinna Vinschen wrote in > <https://lists.gnu.org/archive/html/grep-devel/2018-12/msg00039.html>: > > it would be > > pretty nice if that code could get reverted back in to support > > non-BMP charsets even on Cygwin. > > I agree that support for beyond-BMP characters should be added back to 'grep'. > > Your earlier fix from 2013-08-16 (and the fact that the test failure is > occurring exactly on Windows and AIX platforms) shows that the problem is > with wchar_t being only 16-bit wide on these platforms. > > The type 'char32_t' has been introduced in C11 to overcome this limitation.[1] > > I propose to > > 1) introduce in gnulib support for , char32_t, and mbrtoc32, so > that we can use these instead of , wchar_t, and mbrtowc > portably, > > 2) change those gnulib modules that don't behave well with beyond-BMP > characters on Windows and AIX to use char32_t instead of wchar_t. > > Then the 'grep' code can be changed in a similar way, and this will > fix the bug on Cygwin and AIX (though not on native Windows [2]). > > The advantage of this approach are minimal code changes in 'grep': just > change some type and function names here and there, and add code for > the additional (size_t)(-3) return value of mbrtoc32. IIUC this would also drop the requirement for #ifdef CYGWIN'ed code. Sounds like a great idea to me! Corinna > > Bruno > > [1] > https://stackoverflow.com/questions/21264035/why-did-c11-introduce-the-char16-t-and-char32-t-types > [2] https://lists.gnu.org/archive/html/bug-gnulib/2011-02/msg00175.html
Re: [PATCH] root-uid: new module
On Jun 26 18:18, Eric Blake wrote:
[adding cygwin]
On 06/26/2012 06:12 PM, Paul Eggert wrote:
On 06/26/2012 05:01 PM, Eric Blake wrote:
Rather than forcing comparison to a single value, would it be better to
have a function is_privileged_uid(euid)?
Do you really need this functionality for an arbitrary uid as input? Or
do you just want to know if the current process is running with a certain
privilege?
Yes, that sounds better. I don't use Cygwin, though, so
I'm not qualified to write that.
Broadening the question, does the cygwin community have advice on the
best implementation of a function that returns true if the argument is a
uid with full privileges (for example, if the uid is a member of the
Administrator group, and can therefore read files in spite of chown()
disabling all the read permissions)? We're trying to improve gnulib
(and therefore coreutils, findutils, tar, ...) to use an
implementation-appropriate check rather than the more traditional
hard-coded check of uid==0, since that doesn't quite work everywhere.
Security in Windows is kind of complicated. Not even the membership in
the group Administrators is a sure indicator that the current process
has the approriate rights. In short, every process has a user token
attached which defines the permissions a process has, given as a list
of group memberships and a list of special privileges. As an additional
twist, a privilege in the token can be enabled or disabled.
Some of them, if disabled, are automatically enabled when calling
certain functions, others have to be enabled explicitely before calling
functions relying on the privilege. There are no rules to that, other
than what's defined by the documentation.
I guess, the most platform-independent approach would be to encapsulate
the whole problem into a function requesting a specific capability along
the lines of the POSIX 1.e definitions, like, say
if (has_capability (CAP_CHOWN))
...
The actual implementation of has_capability can then be defined in
a platform-specific file. The default implementation of has_capability
would be something along the lines of
int
has_capability (int cap)
{
return geteuid () == 0;
}
Systems supporting POSIX 1.e could use the appropriate functions,
Windows-based platforms would use an implementation based on the
Win32 functions to request the user token information.
As a sidenote, it would be nice if Cygwin would support the POSIX 1.e
capabilities, but nobody had the time or incentive to implement this
yet. This might be a nice project for somebody who wants to contribute
to Cygwin, since its implementation is mostly independent from the
already existing functionality.
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Project Co-Leader cygwin AT cygwin DOT com
Red Hat
Re: 16-bit wchar_t on Windows and Cygwin
On Feb 3 01:12, Bruno Haible wrote: Hi Eric, I was asking: should wwchar_t (or xwchar_t, but not xchar_t) be 2-bytes on cygwin, but unlike the POSIX definition of wchar_t being always 1 character per unit, the new type is explicitly documented as being multi-unit on some platforms but with sane semantics or should it always be 4-bytes, where conversion from wchar_t to wwchar_t requires some efforts, and where the new type must be used everywhere (which means wrapping a lot of APIs), but where you can once again assume POSIX semantics of 1 character per unit, simplifying life of callers at the expense of converting to the new type In the first case we wouldn't need a new type. The plan is the second alternative. The goal is *not* to have to extend each of quotearg.c, regcomp.c, mbchar.h, wc.c, etc. to handle UTF-16 explicitly with #ifdefs, more variables, and more logic. if it works out, should we also add wwchar_t natively into cygwin? More and more Unix platforms offer only UTF-8 locales. One can predict that in 10 years, all Unix platforms will offer only UTF-8 locales. At this point wchar_t will be UCS-4 on all these platforms (except AIX). The mbrtoc32 function from the C1X API that you pointed to will then be equivalent to mbrtowwc. So, you can view 'wwchar_t' as a temporary measure that will bridge the gap between the ANSI C Amd. 1 API and the C1X API. Maybe I'm just dense, but isn't wwchar_t equivalent to wint_t on all platforms? On UCS-4 platforms sizeof(wint_t) == sizeof(wchar_t) == 4 because there's no reason to make it bigger. On UCS-2 and UTF-16 platforms sizeof(wint_t) == 4 because it must be able to hold EOF as well. So, why not just use the wint_t type for the time being? Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat
Re: 16-bit wchar_t on Windows and Cygwin
On Feb 2 14:24, Eric Blake wrote: [dropping coreutils at this point] On 02/02/2011 04:29 AM, Bruno Haible wrote: Good point. I agree then that overriding wchar_t should better not be done. Here's a new proposal: - Define a type 'wwchar_t' on all platforms, equivalent to uint32_t on Windows platforms and to 'wchar_t' otherwise. - Define functions 'mbrtowwc', 'iswwalpha', 'wwcwidth', and similar. Their definition will be a trivial redirection to 'mbrtowc', 'iswalpha', 'wcwidth' on most platforms, and a use of libunistring modules on Windows platforms. I like the idea of making a new type wrapper. Are you thinking of making a sane wrapping around either 4-byte wchar_t or which maps to 2-byte wchar_t but sanely handles UTF-16 (which makes it a thin wrapper on both Linux and Cygwin, but needing more work on mingw), or are you thinking that it is always a 4-byte type (needing lots more memory manipulation on cygwin to convert between 2- and 4-byte representations when using cygwin's functions, or else reimplementing everything from scratch by completely bypassing cygwin)? As to the name: I agree the opinion of others that xchar_t is easier to type and easier to avoid typos of a missing 'w' than wwchar_t. On the other hand, I can see wwprintf that takes wide-wchar_t values, but gnulib already has xprintf as a counterpart to xmalloc (which calls exit() if the printf fails for memory allocation or other non-I/O related reasons), so we can't blindly use 'x' instead of 'ww' when replacing existing 'w' in POSIX APIs. May I suggest a compromise? What about xwchar_t? It avoids the potential typo by accidentally dropping the second w. It still contains wchar which implies that it's a *wide* char type. And the x could be read as extended. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat
Re: Bug in libiconv?
. The alternative is to have this table stored in a file charset.alias; but then every package that includes the module 'localcharset' from gnulib (that is, libiconv, gettext, coreutils, and many others) will want to modify this file during make install. And this causes a lot of headaches to packaging systems. Therefore, on platforms which have widely used packaging systems (Linux, MacOS X, Cygwin), it's better to avoid the need for this file. Now I'm puzzled. If that's the case, why does libiconv request the charset.alias file on *any* other system than DARWIN7, VMS, and Windows? Especially on Linux? I optimized only the MacOS X, VMS, and Windows OSes. It would have been more work to optimize all versions of Solaris, FreeBSD, AIX, etc. in the same way. charset.alias is requested on Linux, even though it normally does not exist, so that packagers and users have a chance to modify the behaviour. I beg to keep this choice to Cygwin users as well. It will be empty by default as well. The supported codesets are documented in http://cygwin.com/cygwin-ug-net/setup-locale.html#setup-locale-charsetlist If some weird alias is required, the user can add it to charset.alias. That's the optimal solution. Even if Cygwin/newlib handles Windows codepage aliases in all places where it matters for Cygwin, there are still places where it matters for gnulib, coreutils, gettext. Since gnulib, coreutils and gettext are ported to Cygwin anyway, the ported versions should live happily in the Cygwin world. They get what the system defines, and the system is Cygwin, not Windows. Everything else can be added to charset.alias, if required. Neither libiconv nor gettext defines or undefines _WIN32 or __WIN32__. But they are prepared to either setting. Isn't that just covering a PEBKAC? I mean, there's no good reason to define -mwin32 on the command line and the libiconv configure certainly doesn't add it. Whoever squeezed a -mwin32 onto the GCC command line, or even defined -D__WIN32__ manually, deserves the result. But such a user will then write a mail to a mailing list, and it will take time for me (or someone else) to investigate and answer it. By writing #if (defined _WIN32 || defined __WIN32__) !defined __CYGWIN__ I avoid this potential problem. Ok. However, the other variation #if defined _WIN32 || defined __WIN32__ || defined __CYGWIN__ should be only used in very rare circumstances. Usually it just means that some unnecessary Windowism is used on Cygwin, and that there's probably a POSIXy equivalent. If not, kick us here on the list and we can discuss it. Thanks again for your reply and for the hint to the bug in libiconv's code. You're welcome and thanks for this fruitful discussion. I'm glad if we can find a well-working compromise for some of the problems, especially in the unfortunate UTF-16 case. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat
