Re: gnulib-tool: Use the Python implementation by default

[Dmitry Selyutin]

Hi folks, sorry for the long silence!

I've been tracking your progress for a while, even though sporadically and
remaining silent. I'd like to say "thank you" to Bruno and Collin, who made
it this far and never surrendered. :-) Truth to be told, the code I
implemented leaves much to be desired, and you were extremely passionate
and professional about its quality (frankly, it deserved a worse reaction;
I apologize for the quality and overall style). :-) Many thanks to you for
making this come true!

Re: removing permissions for long unused accounts, take 2

[Dmitry Selyutin]

Hi Bruno,

On Wed, Jul 13, 2022 at 8:18 AM Bruno Haible  wrote:
>
>  Dmitry Selyutin
> OK to proceed?

I'm fine with revoking my write permissions. I still have plans to
check on Python gnulib, but, if I will do it, it's simple to restore
the permissions. Until then, let's follow the principle of least
privilege.

P.S. By the way, it's amazing to be in one list with such people, even
though we're talking of those who must have their permissions revoked.
:-)

-- 
Best regards,
Dmitry Selyutin

Re: gnulib-tool.py: Import mktemp

[Dmitry Selyutin]

Hi Tim,

thank you for noticing it! Pushed into master.

-- 
With best regards,
Dmitry Selyutin

signature.asc
Description: This is a digitally signed message part.

gnulib-tool: .XXXignore files generation

[Dmitry Selyutin]

Hi again,

that's yet another question on gnulib-tool. Now I'm implementing the code 
which takes care of .gitignore and .cvsignore files, and I must admit that the 
overall algorithm is a bit tricky (perhaps the only reason is that it was 
designed so that it is generic and can be applied both to git and CVS with 
small changes, which is a nice property). Could someone describe how it works? 
As far as I understand, we must iterate over added_files and removed_files so 
that on each iteration we yield the following information:
1. dirname
2. operation
3. basename
4. previous dirname

FWIW, this likely resembles the following in Python:
added_files = ["lib/bitrotate.c", "m4/cond.m4"]
removed_files = ["prefix/tests/zerosize-ptr.h", "dir/m4/ctype.m4"]
items = [tuple(["A"] + list(os.path.split(path))) for path in added_files]
items += [tuple(["D"] + list(os.path.split(path))) for path in 
removed_files]
for (head, tail) in zip(items, items[1:]):
print(head, tail)

This code will lead to the following output:
('A', 'lib', 'bitrotate.c') ('A', 'm4', 'cond.m4')
('A', 'm4', 'cond.m4') ('D', 'prefix/tests', 'zerosize-ptr.h')
('D', 'prefix/tests', 'zerosize-ptr.h') ('D', 'm4', 'ctype.m4')

However, the tricky part is what to do with that information. Could you help 
me on this part, please? Thank you for your help!

-- 
With best regards,
Dmitry Selyutin

Re: gnulib-tool: conditional dependencies calculation

[Dmitry Selyutin]

Hi all,

sorry for a long absence of news. I'm completely occupied by my work and lack 
time to continue working on gnulib-tool on regular basis. However, I'm 
remembering about the project, and would like to continue the work. I'd like 
to revive the old topic, already asked here.

I don't understand the way gnulib-tool calculates conditional dependencies. 
Let's consider that we import module "socket" (via `--conditional-dependencies 
--import socket` command line arguments). Here's the full dependency graph for 
_conditional_ modules:

[['stddef', 'unistd', 'fd-hook', 'sockets', 'socket']]
[['sockets', 'socket']]
[['msvc-nothrow', 'socket'], ['msvc-nothrow', 'sockets', 'socket']]
[['msvc-inval', 'msvc-nothrow', 'socket'], ['msvc-inval', 'msvc-nothrow', 
'sockets', 'socket']]
[['fd-hook', 'sockets', 'socket']]
[['unistd', 'fd-hook', 'sockets', 'socket']]

As you see, the graph comes to conclusion that stddef, sockets, msvc-nothrow, 
msvc-inval, fd-hook and unistd are conditional modules. Let's consider as 
example the "stddef" module; it is conditional since the "sockets" module is 
required by "socket" module under '[test "$ac_cv_header_winsock2_h" = yes]' 
condition, and it is not required to check other modules, since there is only 
one path which leads to "stddef" module inclusion. As a counter-part, let's 
consider stdaling, which is not considered conditional; here are its paths:

['stdalign', 'sys_socket', 'sockets', 'socket']], [['stdalign', 
'sys_socket', 'socket']

The first path is not treat as unconditional, since the "socket" module 
depends on the "sockets" module conditionally ('$ac_cv_header_winsock2_h'). 
However, the second path has no conditional dependencies in any its part, so, 
as result, the "stdalign" is considered to be imported unconditionally.

However, if I insert the simple loop which iterates over all modules after 
func_transitive_closure and prints ones which are conditional, I see that the 
following modules are marked as conditional: fd-hook, msvc-inval, msvc-
nothrow, sockets, unistd. FWIW, I've used the following code to print the 
conditional modules:

  for module in $final_modules; do
if func_cond_module_p $module; then
  echo ">>" "$module"
fi
  done
  exit 1

What am I missing? It looks like pygnulib and gnulib-tool have different 
understanding of conditional modules.

Thank you for your help!

-- 
With best regards,
Dmitry Selyutin

Re: Change the python interpretor to use /usr/bin/env python3

[Dmitry Selyutin]

Hello,

thank you for investigating this. I'm joining to questions asked by Paul;
moreover, as far as I know specifying python is usually the most correct
way to make python work regardless of version (i.e. it is supposed that one
day there will be no distinction between python and python3).
FWIW, the future reimplementation will drop python 2 entirely. And "python"
branch uses /usr/bin/python too.

пн, 2 апр. 2018 г., 6:52 Paul Eggert :

> Could you please clarify this? Which Fedora release and architecture, and
> what
> What is the output of 'ls -l /usr/bin/python*' and of 'dnf list --installed
> 'python*''? That sort of thing. I don't understand why '#!/usr/bin/python'
> would
> fail at the same time that '#!/usr/bin/env python' worked. Thanks.
>
>

Re: gnulib-tool: conditional dependencies calculation

[Dmitry Selyutin]

Hi Bruno,

thank you for answer! It seems that I was confused by the following entry in 
`diff -ur dir1 dir2` output:

@@ -123,8 +124,8 @@
 
 BUILT_SOURCES += configmake.h
 CLEANFILES += configmake.h configmake.h-t
-endif
 
+endif

I've totally missed the last line and noticed only that line with endif 
disappeared, but from what I see it is present, just with an additional new 
line before it. Sorry for the noise on it; this example was simply incorrect.

However, there are other discrepancies which I observe. Let's consider another 
example which were run on GNU hello:

./gnulib/pygnulib.py --with-tests --conditional-dependencies --no-changelog 
--aux-dir build-aux --doc-base doc --lib libgnu --m4-base m4/ --source-base 
lib/ --tests-base tests --local-dir gl --makefile-name=gnulib.mk --libtool --
import getaddrinfo

./gnulib/gnulib-tool --with-tests --conditional-dependencies --no-changelog 
--aux-dir build-aux --doc-base doc --lib libgnu --m4-base m4/ --source-base 
lib/ --tests-base tests --local-dir gl --makefile-name=gnulib.mk --libtool --
import getaddrinfo

Here is the supposed chain of dependencies for "float" module (conditions are 
below, each condition represents a condition between two modules):

('float', 'vasnprintf', 'snprintf', 'getaddrinfo')
[None, 'test $ac_cv_func_snprintf = no || test $REPLACE_SNPRINTF = 1', 
'test $HAVE_GETADDRINFO = 0']

>From my perspective it seems that "float" module should have been considered 
conditional; but this is not the case, running `diff -ur hello-py hello-sh` 
yields the following:

 ## begin gnulib module float

-if gl_GNULIB_ENABLED_float
 BUILT_SOURCES += $(FLOAT_H)
 
 # We need the following in order to create  when the system
@@ -180,8 +178,9 @@
rm -f $@
 endif
 MOSTLYCLEANFILES += float.h float.h-t
-endif
+
 EXTRA_DIST += float.c float.in.h itold.c
+
 EXTRA_libgnu_la_SOURCES += float.c itold.c

 ## end   gnulib module float


There are also other modules, which appear to be unconditional in gnulib-tool, 
but conditional in pygnulib; the other way incompatibilities also occur. I've 
listed the supposed dependencies below. The [C] means that module was supposed 
to be conditional by pygnulib, but not by gnulib-tool; [U] means that module 
was supposed to be unconditional by pygnulib, but gnulib-tool decided that it 
is conditional.

alloca-opt [C]
('alloca-opt', 'vasnprintf', 'snprintf', 'getaddrinfo')
[None, 'test $ac_cv_func_snprintf = no || test $REPLACE_SNPRINTF = 1', 'test 
$HAVE_GETADDRINFO = 0']

float [C]
('float', 'vasnprintf', 'snprintf', 'getaddrinfo')
[None, 'test $ac_cv_func_snprintf = no || test $REPLACE_SNPRINTF = 1', 'test 
$HAVE_GETADDRINFO = 0']

msvc-inval [U]
('msvc-inval', 'fwrite-tests', 'stdio-tests')
[None, None]
('msvc-inval', 'fread-tests', 'stdio-tests')
[None, None]
('msvc-inval', 'fgetc-tests', 'stdio-tests')
[None, None]
('msvc-inval', 'fputc-tests', 'stdio-tests')
[None, None]
('msvc-inval', 'msvc-nothrow', 'sockets', 'getaddrinfo')
[None, None, 'test $HAVE_GETADDRINFO = 0']
('msvc-inval', 'fdopen', 'fwrite-tests', 'stdio-tests')
['test $REPLACE_FDOPEN = 1', None, None]
('msvc-inval', 'fdopen', 'fread-tests', 'stdio-tests')
['test $REPLACE_FDOPEN = 1', None, None]
('msvc-inval', 'fdopen', 'fgetc-tests', 'stdio-tests')
['test $REPLACE_FDOPEN = 1', None, None]
('msvc-inval', 'fdopen', 'fputc-tests', 'stdio-tests')
['test $REPLACE_FDOPEN = 1', None, None]

stdio [U]
('stdio', 'snprintf', 'getaddrinfo')
[None, 'test $HAVE_GETADDRINFO = 0']
('stdio', 'fdopen', 'fputc-tests', 'stdio-tests')
[None, None, None]
('stdio', 'fdopen', 'fread-tests', 'stdio-tests')
[None, None, None]
('stdio', 'fdopen', 'fgetc-tests', 'stdio-tests')
[None, None, None]
('stdio', 'fdopen', 'fwrite-tests', 'stdio-tests')
[None, None, None]

unistd [U]
('unistd', 'fgetc-tests', 'stdio-tests')
[None, None]
('unistd', 'fread-tests', 'stdio-tests')
[None, None]
('unistd', 'fwrite-tests', 'stdio-tests')
[None, None]
('unistd', 'fputc-tests', 'stdio-tests')
[None, None]
('unistd', 'getpagesize', 'memchr-tests')
[None, None]
('unistd', 'fd-hook', 'sockets', 'getaddrinfo')
[None, None, 'test $HAVE_GETADDRINFO = 0']


I guess that the problem lies somewhere inside module.py, in __init__ method 
of TransitiveClosure class, but I cannot understand what's wrong. I suspect 
that the whole issue is somehow related to handling of the "-tests"-related 
modules.
Could you please take a look on it? I've been dealing with this issue for 
several evenings and yet no result. :-( Thank you!


-- 
With best regards,
Dmitry Selyutin


signature.asc
Description: This is a digitally signed message part.

gnulib-tool: conditional dependencies calculation

[Dmitry Selyutin]

During implementing support for --conditional-dependencies option I've found a 
strange behavior, which makes me thinking that I don't quite get how module's 
conditional state is being calculated. Let's consider the following gnulib-
tool invocation:

gnulib-tool --with-tests --conditional-dependencies --no-changelog --aux-dir 
build-aux --doc-base doc --lib libgnu --m4-base m4/ --source-base lib/ --
tests-base tests --local-dir gl --makefile-name=gnulib.mk --libtool --import 
fnmatch vasnprintf

Upon executing the same command with pygnulib.py from "python" branch, I 
clearly see the difference between way how modules are treat as conditional. 
For example, module configmake is considered to be conditional by pygnulib.py 
and unconditional by gnulib-tool. Here are all the possible paths which lead 
to configmake module, collected via TransitiveClosure.paths("configmake") 
invocation):

('configmake', 'localcharset', 'mbrtowc', 'mbsinit-tests')
('configmake', 'localcharset', 'mbrtowc', 'mbsrtowcs-tests')
('configmake', 'localcharset', 'mbrtowc', 'mbsinit', 'mbrtowc-tests')
('configmake', 'localcharset', 'mbrtowc', 'mbsinit', 'fnmatch')
('configmake', 'localcharset', 'mbrtowc', 'mbsinit', 'mbsrtowcs-tests')
('configmake', 'localcharset', 'mbrtowc', 'mbsinit', 'wcrtomb', 'wctomb', 
'wctob', 'mbrtowc-tests')
('configmake', 'localcharset', 'mbrtowc', 'mbsinit', 'wcrtomb', 'wctomb', 
'wctob', 'mbsrtowcs-tests')
('configmake', 'localcharset', 'mbrtowc', 'mbsrtowcs', 'fnmatch')
('configmake', 'localcharset', 'mbrtowc', 'mbtowc', 'btowc', 'wcrtomb-tests')

The relationship between configmake and localcharset is an unconditional 
dependency; however, localcharset is listed as a conditional dependency in 
mbrtowc "Depends-on" section, and thus it seems reasonable that not only 
localcharset is a conditional dependency, but all modules required by it.

No other module seem to require localcharset except of mbrtowc.
1) mbrtowc is the only one module which requires localcharset;
2) mbrtowc requires localcharset conditionally;
3) all the dependencies of localcharset shall be considered conditional.

Is the following assumption correct and I've found a bug in gnulib-tool, or am 
I misunderstanding the whole concept of conditional dependencies? I'd be very 
grateful if someone could help me with this issue, currently it is the last 
part of the support for --conditional-dependencies in Python.

Thank you!

-- 
With best regards,
Dmitry Selyutin

signature.asc
Description: This is a digitally signed message part.

Re: gnulib-tool.py does not write the gitignore file correctly

[Dmitry Selyutin]

Hi Darshit,

unfortunately I'm currently overloaded with my work, so I only have rare 
occasions to ever get to gnulib's code. From your description it looks the 
feature is omitted entirely, perhaps due to the lack of time to implement it 
in 2012, but I think it is certainly doable. I'll try to look at it in a week 
or two, hopefully this part is not so complex. Pull requests are welcome 
anyway! ;-)

On Monday, January 22, 2018 3:08:57 PM MSK Darshit Shah wrote:
> Hi,
> 
> While using gnulib-tool.py for GNU Wget, I noticed a large number of new
> files in the git repository. It turns out that the original gnulib-tool
> used to write the .gitignore file with a list of all the files it copied
> into m4/. But, gnulib-tool.py does *NOT* write the gitignore file.
> 
> Could this please be fixed? It's not a major issue, just a minor
> inconvenience.


-- 
With best regards,
Dmitry Selyutin

signature.asc
Description: This is a digitally signed message part.

Re: pygnulib: progress and some questions (func_emit_autoconf_snippets)

[Dmitry Selyutin]

Hi Bruno,

thank you for the detailed answer. I think I still have to do some
debugging to understand why the Python version of the algorithm for
autoconf snippets generation works incorrectly, but I hope I understood how
it should work.

29 дек. 2017 г. 2:32 ДП пользователь "Bruno Haible" 
написал:

> Hi Dmitry,
>
> > > 1. At line 4260, what's the place where $modules variable gets
> assigned?
> >
> > Indeed the comments for function func_emit_autoconf_snippets are
> misleading:
> > This functions receives two module lists:
> >   $modules (implicit),
> >   $1 (explicit: first argument).
> >
> > Where does $modules come from?
> >   - For the invocation in func_import, line 5640 and 5659
> > it is set at line 5298. But I'm not sure this was intended this way,
> maybe
> > another list should be used instead of "$main_modules"?
> >   - For the invocation in func_create_testdir, line 6288 and 6290 and
> 6402 and 6404 and 6417
> > it does not appear to be set in a consistent and correct way.
> > (It is set differently if $single_configure and ! $single_configure.)
> > This is probably a bug.
> > The set of modules that _ought_ to be used here is the set of modules
> > being used for the current configure file. That's my current
> understanding.
> > But I'd really need to look at a couple of test cases in order to
> > firmly answer the question.
>
> I've understood it now: the code in func_create_testdir was not really a
> bug,
> because --conditional-dependencies was not allowed in this case anyway.
>
> Here's the patch that implements --conditional-dependencies also with
> --with-tests.
>
> Bruno
>

pygnulib: progress and some questions (func_emit_autoconf_snippets)

[Dmitry Selyutin]

Hello,

this is a kind reminder that python branch is still alive, though it's quite 
difficult to find a time 
for coding. :-) Anyway, I think I'll finish the "import" section in about a 
month: right now I'm 
slowly processing gnulib-comp.m4 generation part of the code. I have a question 
regarding a 
function named func_emit_autoconf_snippets.

TL;DR: feel free to skip to ===QUESTIONS=== section if this is a way too long 
introduction to 
the topic. :-) The questions mainly target Bruno as the original author, but 
I'd be very grateful 
for any help.


Could you please explain in details how func_emit_autoconf_snippets works? The 
most 
important for me is to understand how to map it into the API I've recently 
implemented: 
currently pygnulib can create a special object, "module database", which allows 
to iterate over 
modules and dependencies very quickly. Basically it is a hash table where each 
entry contains 
yet another hash table.


Given a dependency DEP, it is really easy to iterate over each demander DEM and 
condition 
COND like this:

for (dem, cond) in db.demanders(dep):
print(dem, cond)

One can also iterate over all dependencies of the demander DEM, acquiring each 
dependency 
DEP and the condition COND:

for (dep, cond) in db.dependencies(dem):
print(dep, cond)

So given a set of modules MODULES, I can hopefully obtain all the required 
information to 
implement func_emit_autoconf_snippets analogue. However, I can't quite 
understand how the 
modules are processed. The questions are followed; they all are relevant only 
for --
conditional-dependencies switch since otherwise the algorithm is trivial (just 
treat each 
module as unconditional).


===QUESTIONS===
1. At line 4260, what's the place where $modules variable gets assigned? The 
nearest explicit 
assignment I've found is at line 5298; is it correct that this is the one whose 
result is used at 
4260?
2. What's the logic for dependencies processing, starting at line 4298?
3. Do you have an idea how could the algorithm mapped to the described API, 
given that I 
have module database, main modules, tests modules and final modules (and 
hopefully every 
other bit of information)? There's no need to check whether module exists, 
since all the 
mentioned collections (database and modules lists) contain not strings, but a 
real module 
instances.


Thank you for your help!


-- 
With best regards,
Dmitry Selyutin



signature.asc
Description: This is a digitally signed message part.

Re: [PATCH] gnulib-tool.py: Append, don't replace existing VCS ignore files

[Dmitry Selyutin]

Hi Darshit,

I've just pushed your patch. I took the liberty to 
format it as if it was sent via git format-patch. 
Thank you again!


signature.asc
Description: This is a digitally signed message part.

Re: [PATCH] gnulib-tool.py: Append, don't replace existing VCS ignore files

[Dmitry Selyutin]

Hi Darshit,

the reason is actually very simple: I just had no time to apply it due to
activities on a regular work. Hopefully I'll continue developing
gnulib-tool in the end of this week. The patch seems to be OK though; I'll
try to push it this evening. Thank you again and sorry for the delay.

20 нояб. 2017 г. 2:53 ДП пользователь "Darshit Shah" <darnirml...@gmail.com>
написал:

It's been a while. Atleast some response on why it isn't being accepted
would
be appreciated

* Darshit Shah <darnirml...@gmail.com> [171113 09:07]:
> Hi,
>
> Is there something blocking this patch? It doesn't seem like it has been
merged
> yet.
>
> * Darshit Shah <darnirml...@gmail.com> [170913 00:19]:
> > * Dmitry Selyutin <ghostman...@gmail.com> [170911 21:32]:
> > > Hi all,
> > >
> > > > As explained above, this patch helps converge gnulib-tool.py and
> > > > gnulib-tool. There is a behavioural difference which I also
consider to
> > > > be a regression, and hence a string need to fix it.
> > > Thanks for the patch! Looks perfectly fine from my side. Could you
push it
> > > into the repository, please?
> > >
> > > > But we are not there yet. Maybe in one or two weeks, but not now.
> > > I think it is a way too optimistic forecast. Anyway, I do my best,
though my
> > > Python skills are a little bit rusty. :-)
> > >
> > > > Dmitry's work with the Python script has been great
> > > Well, I wouldn't say after looking into the code, but at least it is
working
> > > somehow and can be a basic for a much better API. Remember that we're
staying
> > > on giants' shoulders, I've just rewritten what was created before in
another
> > > language. :-) Anyway, thank you for your kind words!
> > >
> > >
> > > --
> > > With best regards,
> > > Dmitry Selyutin
> >
> > Bruno,
> >
> > Could you please push this patch to master?
> >
> >
> > --
> > Thanking You,
> > Darshit Shah
> > PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6
>
>
>
> --
> Thanking You,
> Darshit Shah
> PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6



--
Thanking You,
Darshit Shah
PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6

Re: gnulib-tool.py: --symbolic and --hardlink support

[Dmitry Selyutin]

Hi Bruno, Paul,

thank you for tips! Now I think I've come up with the decision how to handle 
both options (and this decision basically resembles the original handling). 
I've updated both configuration and parser logic on the python branch, and now 
I move forward to the file processing. Thank you!

-- 
With best regards,
Dmitry Selyutin

signature.asc
Description: This is a digitally signed message part.

[pygnulib] gnulib-tool: license transformers

[Dmitry Selyutin]

Hi all,

I continue developing a new pygnulib version. One of my goals is to replace 
command-line 
invocations with native Python constructs; one of the details that caught my 
attention is that 
I've carefully ported some of the sed invocations, especially the ones related 
to copyright text 
replacement (cf. gnulib-tool:4866-4962).

The most part of these sed invocations seems clear and can be greatly 
simplified in Python 
(e.g. there is no need to construct regex just to replace "GNU General" with 
"GNU Lesser 
General" and so on). However, I still cannot understand how 3orGPLv2 part works 
(see line 
4889); as far as I remember, the --lgpl=3orGPLv2 appeared somewhere after the 
Python 
implementation was done and is not still supported by the current 
gnulib-tool.py.

Could you help me, please? I don't quite get what happens here. I'd also be 
very grateful if you 
could also shed some light on lines 4915 and 4929 (related to numeric version 
parsing).

P.S. Could suggest some files to train my regular expressions on?

-- 
With best regards,
Dmitry Selyutin


signature.asc
Description: This is a digitally signed message part.

Re: alternatives to 'strlcpy'

[Dmitry Selyutin]

>> Though I would change its ssize_t 
>> to ptrdiff_t, so that it depends only on stddef.h.
>   - The mixed use of ssize_t vs. size_t.
What's wrong with ssize_t? As for me, ptrdiff_t is not a good choice here, 
because it represents not length, but difference between pointers. String 
length IS a difference between pointer in some way, but well, I think that's 
not the first allegory that comes to mind when one talks about strings. As for 
me, the only reason to avoid ssize_t is a dependency to stddef.h, but well, I 
think it is going to be used in most projects anyway, especially if these 
projects are somewhat complex. I thought size_t also comes from stddef, but I 
may be wrong. :-)

>   - The argument order.
That's also I think largely a historical issue. BTW, on x86_64 it makes some 
sense due to use of $rdi/$rsi. The only register that must be copied is $rcx. 
After then, one can implement memcpy/strcpy in just a few lines of assembly. 
Anyway, I agree that the argument order should better be changed, to avoid 
confusing strgcpy with its relatives (strncpy, strlcpy, strscpy).

>   If DSTSIZE is 0, it returns -1 and sets errno to EINVAL.
>   If DSTSIZE is nonzero and the result does not fit, it produces a NUL-
>   terminated truncated result, and returns -1 and sets errno to E2BIG.
I really dislike the idea of setting errno, since it involves thread-local 
storage. I know it's a "cold" path, but nevertheless, I really favor returning 
error codes if it is possible. And it is possible with ssize_t. Moreover, more 
modern POSIX functions do actually favor returning error code directly.


Anyway, I really think it is a good idea to provide yet another function, for 
me it seems the most logical choice; I cannot guess what "g" in strgcpy means 
though. Guarded? GNU? Generic? Godlike? :-)
The only possible reason to avoid doing such change is that some people may 
think that these Open Source guys went crazy and invented at least three 
functions to copy strings (four, including strscpy from the Linux kernel); 
moreover, since some people do not actually know the difference between strcpy 
and strcat, the situation is even worse. :-)

-- 
With best regards,
Dmitry Selyutin

signature.asc
Description: This is a digitally signed message part.

Re: new module 'strlcpy'

[Dmitry Selyutin]

How about strscpy from the Linux kernel?

https://www.kernel.org/doc/htmldocs/kernel-api/API-strscpy.html

28 сент. 2017 г. 4:23 ДП пользователь "Paul Eggert" 
написал:

> On 09/27/2017 05:35 PM, Bruno Haible wrote:
>
>> strlcpy with __warn_unused_result__ is the best solution for this task.
>>
>
> No it's not, because strlcpy always computes the string length of the
> source, and that is neither necessary nor desirable. Furthermore, in the
> bad style of programming that uses strlcpy, it's often acceptable to ignore
> the result since the programmer *wants* silent truncation. That is what
> strlcpy means in OpenBSD etc., and we shouldn't be trying to reuse their
> name for a function with different semantics.
>
> A better way to fix the test case is to remove its arbitrary limit on
> month name lengths. That way, it won't need snprintf or strlcpy or strncpy
> or anything like that. And it'll be following the GNU coding standards
> better. I can propose a patch along those lines, if you like. I do not want
> Gnulib to promote the use of strlcpy; that's a step in the wrong direction.
>
> If the same standards were set for test code
>> than for application code, it would become even more tedious and
>> boring to write unit tests than it already is.
>>
> In that case, snprintf suffices: maybe snprintf is not good enough for
> production code, but it's plenty good for this test case.
>
> If you really want a function whose semantics are like strlcpy's but has
> __warn_unused_result__ semantics (and while we're at it, also does not
> count overlong sources, because that's silly), then I suppose we could
> invent one and use it for Gnulib. But we should not call it strlcpy, and we
> shouldn't use it in production code.
>
>
>

Re: pygnulib: Portability fixes

[Dmitry Selyutin]

Hi Mathieu,

thank you for submitting it.

> Here is a patch that removes the use of 'enum.Flags' that didn't exist
> until python 3.6.  It seems that 'enum' classes are not well supported
> with older version too.  So it seems more reasonable to not use them at
> all.
Wow, that's a surprise. Sure, we must support all Python 3 versions. Thank 
you, I've applied it. Please check if everything is OK, I'm not a Git guru: 
since parser and config were a bit ahead, I did the same as you did, 
committed, then exported a patch via $(git format-patch HEAD~1), then put the 
original author and text (only lowercasing a one letter). Please let me know 
if you know a better way, I don't use Git intensively.

> I suppose that in the context of Gnulib it is important to be compatible 
with Python 2?
It is not. :-) Even in 2012 when I started the original work, Bruno wanted to 
stick to Python 3, but I still did everything in a crazy mix of both versions. 
Now the support for Python 2 seems to be redundant; hopefully most Unices 
either have Python 3 by default or at least have it in repositories, ports, 
etc. The cost of maintaining the script for both versions is really huge, I 
really think that Bruno was right from the very beginning.

-- 
With best regards,
Dmitry Selyutin

signature.asc
Description: This is a digitally signed message part.

Re: [PATCH] gnulib-tool.py: Clean the imports

[Dmitry Selyutin]

Hi Mathieu,

> Sorry, I was unaware of that branch.  What about adding some information
> about the gnulib-tool.py development process in a "README-hacking" file?
Frankly I don't know what to write here yet. :-) I mean that there must be 
something already to play with, but the whole thing is too half-baked to even 
mention it on the master branch.

> Since there is no "gnulib-tool.py" nor unit tests for the pygnulib
> module in that branch how are you working on it? with the REPL?  Am I
> overlooking something?
I really do have a simple script which is going to become a new gnulib-
tool.py. I've just committed it; it already can successfully parse the whole 
bunch of command-line options (except for some corner cases) and recently I 
could even teach it do do the same transitive closure the original script does 
upon modules import. :-) I've just committed it; please check the pygnulib.py 
from the python branch.

> I would be happy to get involved in this development which will a good
> excuse for improving my Python skills.
Well, I'm also not a Python guru; even though I know Python much better than I 
knew it when I started rewriting the gnulib-tool for the first time, I'd be 
really glad if we could learn from each other.
 
> Given that the "python" branch seems like a full rewrite of the pygnulib
> module, there is not much interest in applying my patch.
There's nothing wrong with the patch itself, it's correct and does the right 
thing, but I really think that it would be a lot better to invest your time 
into a new implementation. Especially if you want to train your Python skills, 
because the old code may really teach you some bad developer practices.


-- 
With best regards,
Dmitry Selyutin

signature.asc
Description: This is a digitally signed message part.

[pygnulib] simplify cache configure.ac parsing

[Dmitry Selyutin]

NOTE
This change does not affect the current gnulib-tool.py, just `python`
branch.
Still this change is going to be integrated later into the gnulib-tool.py.


I've been testing a new command-line parsing along with parsing cached
configuration (configure.ac, gnulib-cache.m4 and gnulib-comp.m4 processing).
I've noticed that we spend a lot of time whilst processing the contents of
AC_PREREQ and AC_CONFIG_AUX_DIR macros. These regular expressions have the
following form (I've removed some junk):

".*AC_PREREQ\\(\\[(.*?)\\]\\)"
".*AC_CONFIG_AUX_DIR\\(\\[(.*?)\\]\\)"

In Python, however, it seems to be enough to just use the following form:

"AC_PREREQ\\(\\[(.*?)\\]\\)"
"AC_CONFIG_AUX_DIR\\(\\[(.*?)\\]\\)"

Once I started using the latest form, the time required to process each of
these regular expressions decreased for about half a second. The regex works
even on the following cases:

"hello([AC_PREREQ([2.67])])"
"AC_PREREQ([2.67])"
"helloAC_PREREQ([2.67])world"

I suspect that the original form just was a copy-paste from the original
gnulib-tool, where it could have been used due to the usage of sed to parse
the contents of the configure.ac file. So the questions are:
1. Is the new behavior correct?
2. Shall I push this small optimization?


I'd like to do it, because right now everything else I've rewritten works
almost instantly, but I still have some doubts. What do you think?
BTW, the version from the pygnulib differs a bit already from the
gnulib-tool
shell script; I've attached the patch. I've also decided to use raw string
literals just to make regex less verbose.


-- 
With best regards,
Dmitry Selyutin
From 71a8d4a82caf17350cd3fad4ba6feb7b7fdb3e94 Mon Sep 17 00:00:00 2001
From: Dmitry Selyutin <ghostma...@gmail.com>
Date: Tue, 12 Sep 2017 18:47:55 +0300
Subject: [PATCH] config: simplify cache regular expressions

---
 pygnulib/config.py | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/pygnulib/config.py b/pygnulib/config.py
index d08181db6..b174b166a 100644
--- a/pygnulib/config.py
+++ b/pygnulib/config.py
@@ -430,9 +430,9 @@ class Base:
 class Cache(Base):
 """gnulib cached configuration"""
 _AUTOCONF_ = {
-"autoconf" : _re_.compile(".*AC_PREREQ\\(\\[(.*?)\\]\\)", _re_.S | _re_.M),
-"auxdir"   : _re_.compile("^AC_CONFIG_AUX_DIR\\(\\[(.*?)\\]\\)$", _re_.S | _re_.M),
-"libtool"  : _re_.compile("A[CM]_PROG_LIBTOOL", _re_.S | _re_.M)
+"autoconf" : _re_.compile(r"AC_PREREQ\(\[(.*?)\]\)", _re_.S | _re_.M),
+"auxdir"   : _re_.compile(r"AC_CONFIG_AUX_DIR\(\[(.*?)\]\)$", _re_.S | _re_.M),
+"libtool"  : _re_.compile(r"A[CM]_PROG_LIBTOOL", _re_.S | _re_.M)
 }
 _GNULIB_CACHE_ = {
 "local" : (str, "gl_LOCAL_DIR"),
@@ -470,7 +470,7 @@ class Cache(Base):
 _GNULIB_CACHE_STR_ += [_key_]
 else:
 _GNULIB_CACHE_LIST_ += [_key_]
-_GNULIB_CACHE_PATTERN_ = _re_.compile("^(gl_.*?)\\(\\[(.*?)\\]\\)$", _re_.S | _re_.M)
+_GNULIB_CACHE_PATTERN_ = _re_.compile(r"^(gl_.*?)\(\[(.*?)\]\)$", _re_.S | _re_.M)
 
 
 def __init__(self, root, m4_base, autoconf=None, **kwargs):
-- 
2.13.4

Re: gnulib-tool.py

[Dmitry Selyutin]

Hi Tim,

this is a kind reminder that it is still unstable and buggy, so I hope that 
you still maintain runners with the original gnilub-tool script. Still this 
news sounds great!

-- 
With best regards,
Dmitry Selyutin

signature.asc
Description: This is a digitally signed message part.

Re: [PATCH] gnulib-tool.py: Append, don't replace existing VCS ignore files

[Dmitry Selyutin]

Hi all,

> As explained above, this patch helps converge gnulib-tool.py and
> gnulib-tool. There is a behavioural difference which I also consider to
> be a regression, and hence a string need to fix it.
Thanks for the patch! Looks perfectly fine from my side. Could you push it 
into the repository, please?

> But we are not there yet. Maybe in one or two weeks, but not now.
I think it is a way too optimistic forecast. Anyway, I do my best, though my 
Python skills are a little bit rusty. :-)

> Dmitry's work with the Python script has been great
Well, I wouldn't say after looking into the code, but at least it is working 
somehow and can be a basic for a much better API. Remember that we're staying 
on giants' shoulders, I've just rewritten what was created before in another 
language. :-) Anyway, thank you for your kind words!


-- 
With best regards,
Dmitry Selyutin

signature.asc
Description: This is a digitally signed message part.

Re: gnulib-tool.py: follow gnulib-tool changes

[Dmitry Selyutin]

> Well, the logic that binary files (*.mo, *.class) should be copied as-is,
> not transformed, should be kept, no? You'll replace the implementation
> of the transform?
The binary files shall not be touched, but text files shall not be processed
with sed. That's what I mean; the patch is OK, but the code around is
flawed.

> If you replace 'sed' here, you save a subprocess
> invocation, though.
Exactly; I think one of the strongest motivations to perform the whole
rewrite
in Python is the fact that the original gnulib-tool spawns processes too
often.
Moreover, process invocation is quite a dumb technique when you have
built-in
language features instead.

2017-09-09 18:41 GMT+03:00 Bruno Haible <br...@clisp.org>:

> Hi Dmitry,
>
> > > [PATCH 6/6] gnulib-tool.py: follow gnulib-tool changes, part 14
> > > gnulib-tool: don't transform binary files with sed
> > All these sed transformers shall be IMHO entirely deprecated. I don't
> quite
> > remember why I used sed
>
> Using 'sed' is acceptable here because the input comes from a file and the
> output goes to a file anyway. If you replace 'sed' here, you save a
> subprocess
> invocation, though. This will be interesting when you/we are going to start
> optimizing the thing.
>
> Another possible optimization here is that first, we do a
>   cp lookedup tmpfile
> and then
>   sed -e transformer < lookedup > tmpfile
> We could eliminate the cp command when there is a transformer.
>
> > be aware though that this part of code is going to be removed.
>
> Well, the logic that binary files (*.mo, *.class) should be copied as-is,
> not transformed, should be kept, no? You'll replace the implementation
> of the transform?
>
> > > [PATCH 5/6] gnulib-tool.py: follow gnulib-tool changes, part 13
> > > gnulib-tool: concatenate lib_SOURCES to a single line
> > A bit tricky one, but OK from my side. The only thing I noted is that
> > `startpos,pos = match.span()` can be a bit better formatted into
> > `(startpos, pos) = match.span()`.
>
> Done. I had verified that both syntaxes work, but did not know which one is
> the preferred one.
>
> Bruno
>
>


-- 
With best regards,
Dmitry Selyutin

Re: gnulib-tool.py: follow gnulib-tool changes

[Dmitry Selyutin]

Hi Bruno,


thanks for these patches. I've left some comments below.


> [PATCH 6/6] gnulib-tool.py: follow gnulib-tool changes, part 14
> gnulib-tool: don't transform binary files with sed
All these sed transformers shall be IMHO entirely deprecated. I don't quite
remember why I used sed; may be I could not find a way to express the same
idea
using Python "re" module.
Anyway, feel free to push it, because right now it will make the behavior
more
stable; be aware though that this part of code is going to be removed.


> [PATCH 5/6] gnulib-tool.py: follow gnulib-tool changes, part 13
> gnulib-tool: concatenate lib_SOURCES to a single line
A bit tricky one, but OK from my side. The only thing I noted is that
`startpos,pos = match.span()` can be a bit better formatted into
`(startpos, pos) = match.span()`.



Everything is OK though, feel free to push.

2017-09-09 17:52 GMT+03:00 Bruno Haible <br...@clisp.org>:

> Hi Dmitry,
>
> Here's the next round of patches, for your review. I'm slowly working
> through
> the past gnulib-tool changes.
>
> Bruno
>
>


-- 
With best regards,
Dmitry Selyutin

Re: gnulib-tool.py argument parsing

[Dmitry Selyutin]

Hi Bruno,

to be honest, the command-line parsing in the gnulib-tool.py certainly
sucks.
I'm creating a new command-line parser from the scratch, and it works like
charm.
For example, it is much easier to avoid the error you described.
I'll take a look on how to fix the current parser and try to come up with
patch.
I guess there will be a lot of such errors until we switch to a new parser
though.

P.S. Frankly some of this errors are caused by the original solution.
I mean that some of the options (e.g. "modes", like --list, --import, etc.)
are not really options.
It was quite difficult to teach arparse to understand which options can be
combined and which not.

2017-09-09 0:44 GMT+03:00 Bruno Haible <br...@clisp.org>:

> Hi Dmitry,
>
> I'm banging my head against this:
>
>   $ ./gnulib-tool.py --test dirent
>   <works, creates a temporary directory>
>
>   $ ./gnulib-tool.py --test --destdir=../testdir-dirent dirent
>   usage: gnulib-tool.py --help
>   ./gnulib-tool.py: error: unrecognized arguments: dirent
>
> The argument parsing conventions for POSIX/GNU programs distinguish
>   * option with 0 or 1 arguments,
>   * non-option arguments.
> In strict POSIX and in gnulib-tool, the non-option arguments come last,
> that is, the user is not allowed to write
>
> $ ./gnulib-tool.py dirent --test --destdir=../testdir-dirent
> or
> $ ./gnulib-tool.py --test dirent --destdir=../testdir-dirent
>
> I'm fine with argument reordering, that is, to *allow* different ways
> of specifying the arguments. But gnulib-tool.py is currently *forcing*
> a specific argument order which is
>   1. invalid for gnulib-tool
>   2. not according to strict POSIX.
> Namely, it *forces* the syntax
> $ ./gnulib-tool.py --test dirent --destdir=../testdir-dirent
> or
> $ ./gnulib-tool.py --destdir=../testdir-dirent --test dirent
>
>
> This "unrecognized arguments" error is explained in
> https://stackoverflow.com/questions/12818146/
>
> So, I think the fix will be to
>
> 1) replace the line
> cmdargs = parser.parse_args()
> with
> cmdargs, nonoption_args = parser.parse_known_args()
>
> 2) Revisit all uses of nargs='+' and nargs='*'.
>
> Bruno
>
>


-- 
With best regards,
Dmitry Selyutin

Re: gnulib-tool.py: follow gnulib-tool changes

[Dmitry Selyutin]

Hi Bruno,

I have only one question regarding --no-changelog option.
Are you sure you want to entirely deprecate it?
Because right now it is a no-op, but parser still doesn't bark when it
founds it.
The current behavior of gnulib-tool.py matches the same of gnulib-tool.
Once you will apply the change, parser shall complain on unknown
--no-changelog,
And this change will for sure break some setups (e.g.wget2 still uses it).
CC'ing Tim.

As for the rest, feel free to push the changes.
I'll do the same in the "python" branch.

2017-09-09 12:44 GMT+03:00 Bruno Haible <br...@clisp.org>:

> Hi Dmitry,
>
> I've started to copy all modifications done to gnulib-tool since 2012
> over to gnulib-tool.py. Here is the first round of such updates.
> OK to push?
>
> Most of these are not really tested, and I'm a beginner regarding Python.
> Therefore it's very possible that you find mistakes in these patches.
>
> Bruno
>



-- 
With best regards,
Dmitry Selyutin

Re: gnulib-tool.py: trivial fixes

[Dmitry Selyutin]

Sure, perfectly OK.

2017-09-09 12:40 GMT+03:00 Bruno Haible <br...@clisp.org>:

> Hi Dmitry,
>
> Here are some proposed fixes for trivial issues. May I push these?
>
> Bruno
>
>


-- 
With best regards,
Dmitry Selyutin

Re: gnulib-tool.py

[Dmitry Selyutin]

Yes, whitespaces and EOF's were the real cause. I've run the autopep8 tool
and it solved the issue.
Tim, I've managed to build wget2 against the latest master (though I had to
change $gnulib_tool and checkout master inside submodule).

2017-09-08 17:51 GMT+03:00 Darshit Shah <darnirml...@gmail.com>:

> * Dmitry Selyutin <ghostman...@gmail.com> [170908 16:39]:
> > Hi Bruno,
> >
> > below is the sequence of commands.
> >
> > # creating a new branch on commit before pushing gnulib-tool
> > $ git branch pygnulib-stable ec2d72558
> >
> > # checking out branch and rebasing on the latest master changes
> > $ git checkout pygnulib-stable
> > $ git rebase master
> >
> > # cherry-picking only commits which touch original implementation
> > $ git cherry-pick eb41868f9 6edef0c50 f9364747f d1a39ef33
> >
> > # checking out master and merging, trying to push
> > $ git checkout master
> > $ git merge pygnulib-stable
> > $ git push origin master:master
> >
> > I've attached the log. All I can see is a lot of complaints on
> whitespaces
> > and EOF.
> > Does that just mean that I need to make the code a bit cleaner?
> > Or there may be another reason for hook to reject the commit?
> >
> > 2017-09-07 23:54 GMT+03:00 Bruno Haible <br...@clisp.org>:
> >
> > > Hi Dmitry,
> > >
> > > > As for merging into master, I tried to do it, but didn't succeed: git
> > > > complains a lot on whitespaces and similar stuff. I suspect it may be
> > > > caused by custom git hook.
> > >
> > > Yes, the custom git hook can emit errors and warnings. The messages
> about
> > > whitespace are only warnings, as far as I know. The error message(s)
> must
> > > be different.
> > >
> > > > Bruno, Paul, could you help me with merging?
> > >
> > > I might help you better if you show the error message(s) you got - and,
> > > obviously, the command sequence you tried.
> > >
> > > Bruno
> > >
> > >
> >
> >
> > --
> > With best regards,
> > Dmitry Selyutin
>
> Hi Dmitry,
>
> I think you forgot to attach the log files. I don't see anything
> attached to your mail. Could you please check it once?
>
> Since I don't know the exact warnings, I can't speak about those.
> However, from what I see, gnulib-tool.py has a lot of trailing
> whitespaces. That is lines that end with a whitespace character. This is
> usually discouraged, and my guess is exactly what the commit hooks are
> catching on to. You could try to set your editor to highlight such
> spaces or even to delete them on saving a file. Or you could use a regex
> like: `s/\s\+$//g` to remove all the trailing whitespaces
>
> --
> Thanking You,
> Darshit Shah
> PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6
>



-- 
With best regards,
Dmitry Selyutin

Re: gnulib-tool.py

[Dmitry Selyutin]

Hi Bruno,

below is the sequence of commands.

# creating a new branch on commit before pushing gnulib-tool
$ git branch pygnulib-stable ec2d72558

# checking out branch and rebasing on the latest master changes
$ git checkout pygnulib-stable
$ git rebase master

# cherry-picking only commits which touch original implementation
$ git cherry-pick eb41868f9 6edef0c50 f9364747f d1a39ef33

# checking out master and merging, trying to push
$ git checkout master
$ git merge pygnulib-stable
$ git push origin master:master

I've attached the log. All I can see is a lot of complaints on whitespaces
and EOF.
Does that just mean that I need to make the code a bit cleaner?
Or there may be another reason for hook to reject the commit?

2017-09-07 23:54 GMT+03:00 Bruno Haible <br...@clisp.org>:

> Hi Dmitry,
>
> > As for merging into master, I tried to do it, but didn't succeed: git
> > complains a lot on whitespaces and similar stuff. I suspect it may be
> > caused by custom git hook.
>
> Yes, the custom git hook can emit errors and warnings. The messages about
> whitespace are only warnings, as far as I know. The error message(s) must
> be different.
>
> > Bruno, Paul, could you help me with merging?
>
> I might help you better if you show the error message(s) you got - and,
> obviously, the command sequence you tried.
>
> Bruno
>
>


-- 
With best regards,
Dmitry Selyutin

Re: gnulib-tool.py

[Dmitry Selyutin]

Hi Tim,

thank you for your kind words!
As for merging into master, I tried to do it, but didn't succeed: git
complains a lot on whitespaces and similar stuff. I suspect it may be
caused by custom git hook. Bruno, Paul, could you help me with merging? I
suppose that right now almost everything shall be merged, except for the
files I've added recently:

pygnulib/error.py
pygnulib/generator.py
pygnulib/config.py
pygnulib/filesystem.py

These files represent the development branch; I'm trying to cleanup and
mostly rewrite the API. Could you help me with merge, please? If necessary,
I can try using some kind of tool which formats the code automatically.

6 сент. 2017 г. 7:42 ПП пользователь "Tim Rühsen" <tim.rueh...@gmx.de>
написал:

> On Dienstag, 5. September 2017 23:35:54 CEST Dmitry Selyutin wrote:
> > Hi Tim,
> >
> > that's exactly what I've done today. I've just pushed some changes into
> > repository; could you please try the latest version?
> > I didn't have time to integrate --no-changelog support yet (it seems it
> was
> > either unimplemented before or appeared after project was completed).
> > Thank you for trying the Python version; please let me know if it works
> for
> > you or if there are some other errors to be fixed.
> >
> > P.S. I've started cleaning the entire code base, also rewriting some slow
> > paths, but there is a lot of work to be done; stay tuned. :-)
>
> Awesome, Dmitry ! \o/
>
> Works like a charm on a quick test (will do some more testing the next
> days).
>
> That's really *much* faster than  before.
> Now:
> real0m34,265s
> user0m31,524s
> sys 0m2,279s
>
> Before:
> real1m20,295s
> user1m16,460s
> sys 0m16,050s
>
> The bottleneck now is autoreconf followed by libtoolize :-)
>
> If you remove the gnulib-tool from branch 'pygnulib' and instead add a link
> gnulib-tool -> gnulib-tool.py then one wouldn't have to edit ./bootstrap.
>
> I just can't say how much I appreciate your work !
>
> Regards, Tim
>

Re: [pygnulib] gnulib-python initial merge and some minor bug fixes

[Dmitry Selyutin]

Hi Bruno,

what do you think if I'll periodically merge stable versions into the
master? For example, I think currently the imported 'as is' version can be
merged. I'd like to work on API on a separate branch though, since I
roughly dislike the idea of developing on master.

For example, see my latest commit; even though it does not affect the API
yet, these classes (and similar stuff) are going to replace all this mess
with setters and getters, and when it will begin to happen, I don't want to
break the current gnulib-tool.py (even though it is considered to be
experimental yet).

P.S. I've just started doing the very same thing around Config class that I
had done five years ago, but previously it had taken approximately 1000
lines more than now. Wow, just wow.

21 авг. 2017 г. 11:36 ПП пользователь "Bruno Haible" 
написал:

Hi Dmitry,

> The development is going to happen inside a standalone branch.
> Once stuff becomes mature enough, it'll be pushed into the master.

I would like to make it easy for users to invoke gnulib-tool.py
for users, without changing their autogen.sh / bootstrap / Makefile.devel
/...
scripts or recipes. I'll do this by introducing an environment variable,
say, GNULIB_TOOL_PYTHON so that people can do

  $ GNULIB_TOOL_PYTHON=yes ./autogen.sh

instead of

  $ ./autogen.sh

This will be a couple of lines of code in gnulib-tool, to invoke
gnulib-tool.py with the same arguments.

The prerequisite for this is only that gnulib-tool.py is properly
invokable through python or python3, and that it sits in the 'master'
branch. It is *not* a requirement that it is "mature enough".

Bruno

Re: [pygnulib] gnulib-python initial merge and some minor bug fixes

[Dmitry Selyutin]

P.S. Please ignore binary .pyc files; they won't be present in the commit,
just a leftover from the testing.

2017-08-20 11:10 GMT+03:00 Dmitry Selyutin <ghostman...@gmail.com>:

> Sorry, I've forgotten to provide a link to the original repository.
>
> [0] https://github.com/ghostmansd/gnulib-python
>
> 2017-08-20 11:08 GMT+03:00 Dmitry Selyutin <ghostman...@gmail.com>:
>
>> Hello,
>>
>> this is the very first patch after five years of silence.
>> It all boils down to importing gnulib-python github repository[0] as is.
>> I'm going to import everything except of pygnulib/testing.py.
>> The second patch (hotfix.patch) just includes some quick fixes.
>>
>> The development is going to happen inside a standalone branch.
>> Once stuff becomes mature enough, it'll be pushed into the master.
>>
>> --
>> With best regards,
>> Dmitry Selyutin
>>
>
>
>
> --
> With best regards,
> Dmitry Selyutin
>



-- 
With best regards,
Dmitry Selyutin

Re: [pygnulib] gnulib-python initial merge and some minor bug fixes

[Dmitry Selyutin]

Sorry, I've forgotten to provide a link to the original repository.

[0] https://github.com/ghostmansd/gnulib-python

2017-08-20 11:08 GMT+03:00 Dmitry Selyutin <ghostman...@gmail.com>:

> Hello,
>
> this is the very first patch after five years of silence.
> It all boils down to importing gnulib-python github repository[0] as is.
> I'm going to import everything except of pygnulib/testing.py.
> The second patch (hotfix.patch) just includes some quick fixes.
>
> The development is going to happen inside a standalone branch.
> Once stuff becomes mature enough, it'll be pushed into the master.
>
> --
> With best regards,
> Dmitry Selyutin
>



-- 
With best regards,
Dmitry Selyutin

Re: --create-testdir eats CPU and does nothing

[Dmitry Selyutin]

Hi everyone!

I'm amazed that someone still remembers this project. Looking at its
sources, I see that there is room for improvement, but still the overall
project structure may still be integrated into gnulib. I would be glad to
continue working on the project in my free time, though my Python skills
are a bit rusty. As for parallelization, I think it may be a good idea to
use Python 3 since it now has some form of Go-like coroutines IIRC.

Do you think it's a good idea to revive the old project? It seems that it
may still be useful.

Re: symlink.c for Windows

[Dmitry Selyutin]

Hello, Eli!

First of all, thanks for your explanations! As I've said, I'm not so
skilled developer in C or C++.
As for compilers and toolchains, I'm pretty sure MinGW is not the only
supported platform, since we have a lot of checks for things like MSVC
inside gnulib files. gnulib is compatibility library which aims to support
set of functions and types for cross-platform develop, so yes, I don't
think MinGW is the only choice.
As for trailing slashes, the code was taken from gnulib's link.c. If it
needs a fix, it would be a good idea to fix it in all places. Probably it
worths adding support for DBCS locale, though it may be hard to add it
everywhere. Probably you could do it, since I'm not so good in Windows
locales? Thanks!
As for other library functions, I'm going to work deeper on this part when
I'll have time. It is possible to write wrappers around Windows functions.
I know that this may be a hard thing to do, but it is possible to do. I'll
try to add such functionality as soon as I can. Since system _supports_
symlinks, imho, we shall try to add support for it if we can do it.

Thanks for your kind explanations, remarks and advice. It's always a great
pleasure and honor for me to learn from experienced professionals, and I'm
really glad that here in gnulib I sometimes have such an opportunity. Thank
you again! If there are some things to be said, I'm always open for
discussion!


2013/9/9 Eli Zaretskii e...@gnu.org

  From: Dmitry Selyutin ghostman...@gmail.com
  Date: Mon, 9 Sep 2013 02:27:13 +0400
 
  Sorry for the long absence, there are lot of other things to be done.
 Still
  I'm trying to participate, though, as I can. Recently I've found that
 there
  is no symlink function for Windows, though Windows has CreateSymbolicLink
  function. I've tried to implement my own. I don't know C or C++ really
  well, but fix seemed to be trivial enough. MinGW doesn't support
 symlinks,
  though reports success.

 Caveat: I'm not a gnulib developer or maintainer, so what's below in
 no way reflects the opinions of the project about your contribution.

 That said, please allow me a few comments.

 First, I think having 'symlink' is just a small part of support for
 symbolic links.  Functions such as 'lstat' (a real one, not an alias
 of 'stat'), 'stat' that reports about the target of the symlink, and
 'readlink' are also needed.  Unlike with 'symlink', where the Windows
 implementation is simple, 'readlink' (and to some degree 'stat') are
 not, because there's no easy-to-use API on Windows that allows to find
 the target of a symlink.

 In addition, some other library functions, like 'access' and 'chmod',
 need to resolve symlinks internally, because the Windows APIs used by
 these functions report the attributes of the symlink, not of its
 target.

 IMO, having just 'symlink' without all the rest will provide a very
 incomplete support for symbolic links.

 A couple of comments to your code:

  +# if (defined _WIN32 || defined __WIN32__)  ! defined __CYGWIN__
  +
  +#  include windows.h
  +
  +#  if defined __MINGW32__
  +/* MinGW does not support symlinks. */

 I don't understand: if the code you wrote is not for MinGW, then for
 what Windows development environment is it, and how does that
 environment support symlinks, if it doesn't have the 'symlink'
 function?

  +  /* Reject trailing slashes on non-directories. */
  +  if ((len1  (file1[len1 - 1] == '/' || file1[len1 - 1] == '\\'))
  +  || (len2  (file2[len2 - 1] == '/' || file2[len2 - 1] == '\\')))

 The test of the trailing backslash will not do what you want in a
 Windows locale that uses DBCS character sets, because there the second
 byte of a two-byte character can be a backslash, even though the
 character is some CJK character, not a directory separator.  The
 result will be that the function will reject a perfectly valid file
 name.

  +  if (stat (file1, st) == 0  S_ISDIR (st.st_mode))
  +errno = EPERM;

 This doesn't allow creation of symbolic links to non-existent
 directories, which is a subtlety Posix programs might not expect
 (because Posix platforms don't distinguish between symlinks to
 directories and non-directories).

 Thanks again for working on this.




-- 
With best regards,
Dmitry Selyutin

E-mail: ghostma...@gmail.com
Phone: +7(985)334-07-70

symlink.c for Windows

[Dmitry Selyutin]

Hello everyone!

Sorry for the long absence, there are lot of other things to be done. Still
I'm trying to participate, though, as I can. Recently I've found that there
is no symlink function for Windows, though Windows has CreateSymbolicLink
function. I've tried to implement my own. I don't know C or C++ really
well, but fix seemed to be trivial enough. MinGW doesn't support symlinks,
though reports success.


diff --git a/lib/symlink.c b/lib/symlink.c
index d3c9f21..903ae6a 100644
--- a/lib/symlink.c
+++ b/lib/symlink.c
@@ -45,7 +45,12 @@ rpl_symlink (char const *contents, char const *name)

 #else /* !HAVE_SYMLINK */

-/* The system does not support symlinks.  */
+# if (defined _WIN32 || defined __WIN32__)  ! defined __CYGWIN__
+
+#  include windows.h
+
+#  if defined __MINGW32__
+/* MinGW does not support symlinks. */
 int
 symlink (char const *contents _GL_UNUSED,
  char const *name _GL_UNUSED)
@@ -54,4 +59,141 @@ symlink (char const *contents _GL_UNUSED,
   return -1;
 }

+#  else /* !defined __MINGW32__ */
+
+#   if !defined SYMBOLIC_LINK_FLAG_DIRECTORY
+#define SYMBOLIC_LINK_FLAG_DIRECTORY 0x1
+#   endif
+
+/* CreateSymbolicLink was introduced only in Windows Vista.
+   Also on Windows it is necessary to get special privileges. */
+typedef BOOL (WINAPI * CreateSymbolicLinkFuncType) (LPCTSTR
lpSymlinkFileName,
+LPCTSTR
lpTargetFileName,
+DWORD dwFlags);
+static CreateSymbolicLinkFuncType CreateSymbolicLinkFunc = NULL;
+static BOOL initialized = FALSE;
+
+static void
+initialize (void)
+{
+  HMODULE kernel32 = GetModuleHandle (kernel32.dll);
+  if (kernel32 != NULL)
+{
+  CreateSymbolicLinkFunc =
+(CreateSymbolicLinkFuncType) GetProcAddress (kernel32,
CreateSymbolicLinkA);
+}
+  initialized = TRUE;
+}
+
+int
+symlink (const char *file1, const char *file2)
+{
+  /* Enable privileges for symbolic link creation. */
+  HANDLE token;
+  struct stat st;
+  TOKEN_PRIVILEGES tp;
+  if (!OpenProcessToken (GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES,
token))
+goto error;
+  if (!LookupPrivilegeValueA (NULL, SeCreateSymbolicLinkPrivilege,
tp.Privileges[0].Luid))
+goto error;
+  tp.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;
+  tp.PrivilegeCount = 1;
+  if (!AdjustTokenPrivileges (token, false, tp, sizeof(TOKEN_PRIVILEGES),
NULL, NULL))
+goto error;
+  CloseHandle(token);
+
+  /* Initialize symbolic link function. */
+  size_t len1 = strlen (file1);
+  size_t len2 = strlen (file2);
+  if (!initialized)
+initialize ();
+
+  if (CreateSymbolicLinkFunc == NULL)
+goto error;
+
+  /* Reject trailing slashes on non-directories. */
+  if ((len1  (file1[len1 - 1] == '/' || file1[len1 - 1] == '\\'))
+  || (len2  (file2[len2 - 1] == '/' || file2[len2 - 1] == '\\')))
+{
+  if (stat (file1, st) == 0  S_ISDIR (st.st_mode))
+errno = EPERM;
+  else
+errno = ENOTDIR;
+  return -1;
+}
+
+  DWORD flags;
+  if (stat (file2, st) == 0  S_ISDIR (st.st_mode))
+isdir |= SYMBOLIC_LINK_FLAG_DIRECTORY;
+  /* Create symbolic link. */
+  if (CreateSymbolicLinkFunc (file2, file1, flags) == 0)
+{
+  /* Convert error codes from Windows to Unix. */
+  DWORD err = GetLastError ();
+  switch (err)
+{
+case ERROR_ACCESS_DENIED:
+  errno = EACCES;
+  break;
+
+case ERROR_INVALID_FUNCTION:
+  errno = EPERM;
+  break;
+
+case ERROR_NOT_SAME_DEVICE:
+  errno = EXDEV;
+  break;
+
+case ERROR_PATH_NOT_FOUND:
+case ERROR_FILE_NOT_FOUND:
+  errno = ENOENT;
+  break;
+
+case ERROR_INVALID_PARAMETER:
+  errno = ENAMETOOLONG;
+  break;
+
+case ERROR_TOO_MANY_LINKS:
+  errno = EMLINK;
+  break;
+
+case ERROR_ALREADY_EXISTS:
+  errno = EEXIST;
+  break;
+
+default:
+  errno = EIO;
+}
+  return -1;
+}
+
+  /* MinGW doesn't report errors, though really fails. */
+  if (stat (file1, st) != 0)
+  {
+errno = ENOSYS;
+return -1;
+  }
+  return 0;
+
+error:
+  CloseHandle(token);
+  errno = EPERM;
+  return -1;
+
+}
+
+#  endif /* !defined __MINGW32__ */
+
+# else
+/* The system does not support symlinks. */
+int
+symlink (char const *contents _GL_UNUSED,
+ char const *name _GL_UNUSED)
+{
+  errno = ENOSYS;
+  return -1;
+}
+
+# endif
+
 #endif /* !HAVE_SYMLINK */


-- 
With best regards,
Dmitry Selyutin

E-mail: ghostma...@gmail.com
Phone: +7(985)334-07-70

pygnulib: code separation

[Dmitry Selyutin]

Hello everyone!

Since it may be annoying to support files inside pygnulib to be the same
as in gnulib, I've thought that it may be a cool idea to separate entire
pygnulib code from gnulib. What is the proposal?

I propose to accept the following hierarchy for pygnulib:
  doc
README
LICENSE
INSTALL
  python
pygnulib
  __init__.py
  constants.py
  functions.py
  GLConfig.py
  GLEmiter.py
  GLError.py
  GLFileSystem.py
  GLImporter.py
  GLMakefileEditor.py
  GLModuleSystem.py
  GLPackage.py
  GLPackageSystem.py
  GLSystem.py
  GLTestDir.py
  GLTestFlags.py
  gnulib-tool
  install.py

Note that gnulib-tool now will be the name for the gnulib-tool.py.

What will do install.py script?

1. Copy pygnulib subdirectory to Python modules directory.
2. Clone git repository of the gnulib-tool (if directory with gnulib wasn't
set
with some option for install.py script) and move this repository to some
directory inside user's entire machine (e.g. to /usr/share/gnulib).
Also copy doc subdirectory to /usr/share/doc.
3. Adjust GNULIB_ROOT variable inside constants.py file to point to the
copied
(i.e. cloned and moved) gnulib git repository.
4. Copy gnulib-tool to /usr/local/bin directory.

gnulib-tool will accept a new --fetch function to update gnulib from git.

Such combination allows to separate gnulib files (at least before it will be
merged with gnulib original files). What do you think about it?


-- 
With best regards,
Dmitry Selyutin

E-mail: ghostma...@gmail.com
Phone: +7(985)334-07-70

Re: pygnulib: code separation

[Dmitry Selyutin]

Hi Karl,

that's why I've called it proposal. :-) Critics are really accepted,
since our project is mature enough to not admit such changes without
discussions.

I've thought about the future replacement since it was the goal of
Google Summer of Code (to reimplement gnulib-tool in Python), but I
don't think it will happen quickly (if it ever supposed to happen).

There is other way which we can follow, and after your advice and
critic I think it could be even better to follow this way.

We can reduce my repo exactly to contain pygnulib module and sometimes
update it if necessary inside gnulib repo (or we can even leave
decision to users: if they want to use pygnulib, they can get it from
pygnulib repo). The name of this gnulib-tool.py will be still the
same, however, I'll make an application with similar functions but
with other name (e.g. gnulib-manager) and other idea of updating. I'm
sure for some people ability to save gnulib like an eternal
directory on the system, for which they can easily get update, is
really a useful feature.

Thanks again for your notes and critics!

2013/5/11, Karl Berry k...@freefriends.org:
 Hi Dimitry (and all),

 I don't like to be so negative, but ... I was not aware of any general
 agreement on replacing the current gnulib world with a Python-based
 world.

 Note that gnulib-tool now will be the name for the gnulib-tool.py.

 It seems to me that gnulib will become completely unusable for me after
 things like this are done.

 directory inside user's entire machine (e.g. to /usr/share/gnulib).
 Also copy doc subdirectory to /usr/share/doc.

 I know it was just an example, but using /usr is certainly wrong.

 4. Copy gnulib-tool to /usr/local/bin directory.

 This seems to be suggesting, merely in passing, a fundamental change in
 the way gnulib is intended to be used.

 Maybe you should fork gnulib into pygnulib and maintain it as you like
 there, and people who like it can use it, and those of us who have a
 whole development structure built up around how gnulib is now can keep
 using it as it is?

 Best,
 k



-- 
With best regards,
Dmitry Selyutin

E-mail: ghostma...@gmail.com
Phone: +7(985)334-07-70

gnulib-tool.py: a new life

[Dmitry Selyutin]

Hello everyone!

First of all I'd like to apologize that I've been absent: I've been sent to
Italy to study Latin for this year to be able speak it. Since I've missed
more than month of study when I've come to Academy, I had to use all my
efforts to catch up with the rest. However, I've never forgotten about the
promise I've given to Bruno, though I wanted to make a small surprise for
Bruno and all great people who work to make gnulib even better. The other
factor explaining why the work was carried out in secret and result was
never publicated until this time is that I wanted to rewrite gnulib from
the beginning, so I don't wanted to use the same public git repo or create
public branch to let anyone see it before it will be ready.

Though even now I see some things which I can implement a bit other way or
change, I think that gnulib-tool.py and pygnulib, its basis, are now almost
ready for real work. There are some nuances, though they (except one) don't
seem to take any significant part. I feel that the largest part of work is
completed.

URL: https://github.com/ghostmansd/pygnulib
git: g...@github.com:ghostmansd/pygnulib.git

Addendum:
0. Since it's really painkill to support both Python and Python3K, only the
last one was left. I think the time has come, so I've removed support for
Python's versions less than 3.
1. Short documentation (which I'm planning to increase later) is located
here: *pygnulib/README*. It just briefly describes pygnulib and shows some
short examples. The most notable part of documentation can be acieved by
typing pydoc3 pygnulib. The other way to understand the way pygnulib
works is to see the sources, especially *gnulib-tool.py*, *
pygnulib/PyGNULib/GLImporter.py*, *pygnulib/PyGNULib/GLTestDir.py* files.
2. I've never writtten scripts to manually add python packages to their
location. If you need to use pygnulib in your own scripts, you must *
manually* copy it to corresponding directory. I advice to adjust
GNULIB_ROOT variable inside *pygnulib/PyGNULib/constants.py* file before
copying pygnulib to Python modules: it should help you not to get tired
from pointing to gnulib-root directory when creating GLSystem instances.
3. Don't forget to see *pygnulib/TODO* file. It contains some really
usefult information, including semi-bugs.
4. gnulib-tool --avoids now accepts modules as list too, not the way as it
was before (I've forgotten to tell about it in documentation). I think it's
a good idea to fix it, but when I've thought about it, I was to lazy to fix
it. If it is necessary, I'll fix it.

Any questions are appreciated. Testing is appreciated even more, though
I've tested only some basic actions with some basic packages like *hello*.

Since I'm studying in Italy yet, I won't have so much time until summer.
However, I hope that I can help even this time, though I can't promise it
will be often. I hope that situation will change when summer comes.

Good luck!

-- 
With best regards,
Dmitry Selyutin

E-mail: ghostma...@gmail.com
Phone: +7(985)334-07-70