Re: [fpc-devel] Unicode paths

[Marco van de Voort]

In our previous episode, Sven Barth said:
> > argv and argc is not really raw data, and on many platforms (like
> windows) processed.
> >
> > If you want to use the raw data, you'll need OS specific functions
> > (getcommandlinew for e.g.  Windows), or, in case of the Unix RTL, the
> > operatingsystem_parameter_* assembler symbols that convey the registers
> > loaded by the loader to the RTL.
> 
> The usage of which I strongly discourage as they conflict with dynamic
> packages and thus in the future won't be used on systems that support them.

Yeah, it is more logical to say "just do paramstr(unicodestring) and convert
it to the encoding yourself". On unix paramstr and argv are nearly the same,
and on windows they are emulated anyway.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Sven Barth]

Am 13.04.2016 09:04 schrieb "Marco van de Voort" :
>
> In our previous episode, Martin Schreiber said:
> > > Anyway, I never liked the fact that argv and argc are exported
symbols. If
> > > they weren't public, we could change them at will and/or have
differing
> > > implementations and encodings depending on target.  I would deprecate
them
> > > (being public)
> > >
> > Is there a possibility to still get the raw data in case one wants to
handle
> > encoding himself?
>
> argv and argc is not really raw data, and on many platforms (like
windows) processed.
>
> If you want to use the raw data, you'll need OS specific functions
> (getcommandlinew for e.g.  Windows), or, in case of the Unix RTL, the
> operatingsystem_parameter_* assembler symbols that convey the registers
> loaded by the loader to the RTL.

The usage of which I strongly discourage as they conflict with dynamic
packages and thus in the future won't be used on systems that support them.

Regards,
Sven
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Marco van de Voort]

In our previous episode, Martin Schreiber said:
> > Anyway, I never liked the fact that argv and argc are exported symbols. If
> > they weren't public, we could change them at will and/or have differing
> > implementations and encodings depending on target.  I would deprecate them
> > (being public)
> >
> Is there a possibility to still get the raw data in case one wants to handle 
> encoding himself?

argv and argc is not really raw data, and on many platforms (like windows) 
processed.

If you want to use the raw data, you'll need OS specific functions
(getcommandlinew for e.g.  Windows), or, in case of the Unix RTL, the
operatingsystem_parameter_* assembler symbols that convey the registers
loaded by the loader to the RTL.

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Martin Schreiber]

On Tuesday 12 April 2016 18:46:15 Marco van de Voort wrote:
>
> Anyway, I never liked the fact that argv and argc are exported symbols. If
> they weren't public, we could change them at will and/or have differing
> implementations and encodings depending on target.  I would deprecate them
> (being public)
>
Is there a possibility to still get the raw data in case one wants to handle 
encoding himself?

Martin
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Sven Barth]

Am 12.04.2016 18:46 schrieb "Marco van de Voort" :
>
> In our previous episode, Michael Van Canneyt said:
> > >> If argv/argc is known to be UTF8 encoded, then I see no problem with
> > >> keeping getopts ?
> > >
> > > Getopts is not utf8 but shortstring, so ACS on Windows. Or do you
plan to
> > > redo it using utf8string or unicodestring?
> >
> > I don't necessarily plan to do so, I was proposing it as a way out.
>
> If getopts is not leading, then the needs of the unicode
T(Custom)Application
> classes is probably the best indicator for which way to go.
>
> If they don't need argv and argc, just leave them as they are.
>
> Anyway, I never liked the fact that argv and argc are exported symbols. If
> they weren't public, we could change them at will and/or have differing
> implementations and encodings depending on target.  I would deprecate them
> (being public)

Agreed. They are only exported for ObjPas and that can be solved in the
same way as other symbols uses there are solved.

Regards,
Sven
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Marco van de Voort]

In our previous episode, Michael Van Canneyt said:
> >> If argv/argc is known to be UTF8 encoded, then I see no problem with
> >> keeping getopts ?
> >
> > Getopts is not utf8 but shortstring, so ACS on Windows. Or do you plan to
> > redo it using utf8string or unicodestring?
> 
> I don't necessarily plan to do so, I was proposing it as a way out.

If getopts is not leading, then the needs of the unicode T(Custom)Application
classes is probably the best indicator for which way to go.

If they don't need argv and argc, just leave them as they are.

Anyway, I never liked the fact that argv and argc are exported symbols. If
they weren't public, we could change them at will and/or have differing
implementations and encodings depending on target.  I would deprecate them
(being public)

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Michael Van Canneyt]

On Tue, 12 Apr 2016, Marco van de Voort wrote:


In our previous episode, Michael Van Canneyt said:

I guess the only way, as always, is to have two duplicate systems on
windows.  One wide that is for unicode, (unicodestring paramstr and
D2009 compatible tapplication), one for old legacy ansi apps. (getopts etc).



If argv/argc is known to be UTF8 encoded, then I see no problem with
keeping getopts ?


Getopts is not utf8 but shortstring, so ACS on Windows. Or do you plan to
redo it using utf8string or unicodestring?


I don't necessarily plan to do so, I was proposing it as a way out.

Michael.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Marco van de Voort]

In our previous episode, Michael Van Canneyt said:
> > I guess the only way, as always, is to have two duplicate systems on
> > windows.  One wide that is for unicode, (unicodestring paramstr and
> > D2009 compatible tapplication), one for old legacy ansi apps. (getopts etc).
 
> If argv/argc is known to be UTF8 encoded, then I see no problem with
> keeping getopts ?

Getopts is not utf8 but shortstring, so ACS on Windows. Or do you plan to
redo it using utf8string or unicodestring?


___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Michael Van Canneyt]

On Tue, 12 Apr 2016, Marco van de Voort wrote:


In our previous episode, Jonas Maebe said:

They are not supported, because we get the original command line data
using the ansi version of the API call (see setup_arguments() in
rtl/win/syswin.inc). If this is "fixed", then we also have to decide
what to do with the argv p(ansi)char (a good place would be to check
what Windows itself returns from the ansi API call when passing command
line arguments that contain characters that cannot be represented in the
ansi code page.


The central place of posix argv/argc in the command processing (including
getopts), also on nonposix systems is indeed a bit of a problem.

I guess the only way, as always, is to have two duplicate systems on
windows.  One wide that is for unicode, (unicodestring paramstr and
D2009 compatible tapplication), one for old legacy ansi apps. (getopts etc).



If argv/argc is known to be UTF8 encoded, then I see no problem with keeping 
getopts ?


Michael.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Mattias Gaertner]

On Mon, 11 Apr 2016 19:31:23 +0200
Mattias Gaertner  wrote:

> Hi,
> 
> Are there any plans to extend FPC to support Unicode file paths under
> Windows?

To clarify: I mean the compiler fpc.exe.

> Of course this also means the other tools, like fpcres, fpmake and
> fppkg.
> Would this require a new flag?

Mattias
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Mattias Gaertner]

On Mon, 11 Apr 2016 22:45:54 +0200 (CEST)
mar...@stack.nl (Marco van de Voort) wrote:

>[...]
> Well, afaik on Windows unicode commandlines are supported in $mode
> delphiuncode. Overloaded unicode(string) versions of paramstr handle it
> (unit uuchar)
> 
> 
> > And when executing the compiler directly e.g. from TProcess?
> 
> Specific binaries would be case by case basis, but I don't expect any to be
> upgraded all ready.  In trunk I updated executeprocess to unicodestring, but
> for TProcess I'm waiting on decisions on TSting(s)(list).  (system.classes)

Are we still talking about the compiler?

Mattias
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Jonas Maebe]

Marco van de Voort wrote:

Well, afaik on Windows unicode commandlines are supported in $mode
delphiuncode.


They are not supported, because we get the original command line data 
using the ansi version of the API call (see setup_arguments() in 
rtl/win/syswin.inc). If this is "fixed", then we also have to decide 
what to do with the argv p(ansi)char (a good place would be to check 
what Windows itself returns from the ansi API call when passing command 
line arguments that contain characters that cannot be represented in the 
ansi code page.


Another way to know it is not supported, is that it is not listed at 
http://wiki.freepascal.org/FPC_Unicode_support#RTL_changes (*)



Jonas

(*) I know you recently committed unicode support for executing external 
programs and it isn't listed there yet either, but that's because I 
think it is broken on *nix: encoding the command line parameters using 
DefaultFileSystmCodePage is wrong, because the receiving/executed 
applications will interpret them according to what they deduce as being 
DefaultSystemCodePage from the environment instead. We have a routine 
that can deduce this code page from the environment (GetSystemCodepage), 
so we should probably use that (although it may need some refactoring to 
be able to use it in case the environment for the executed process is 
specified separately from that of the invoking process)

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Marco van de Voort]

In our previous episode, Mattias Gaertner said:
> > > Of course this also means the other tools, like fpcres, fpmake and
> > > fppkg.
> > > Would this require a new flag?
> > 
> > Yes. cmd.exe with /u ? :-)
> 
> Is this already supported?

Well, afaik on Windows unicode commandlines are supported in $mode
delphiuncode. Overloaded unicode(string) versions of paramstr handle it
(unit uuchar)


> And when executing the compiler directly e.g. from TProcess?

Specific binaries would be case by case basis, but I don't expect any to be
upgraded all ready.  In trunk I updated executeprocess to unicodestring, but
for TProcess I'm waiting on decisions on TSting(s)(list).  (system.classes)

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Mattias Gaertner]

On Mon, 11 Apr 2016 21:05:55 +0200 (CEST)
mar...@stack.nl (Marco van de Voort) wrote:

> In our previous episode, Mattias Gaertner said:
> > Are there any plans to extend FPC to support Unicode file paths under
> > Windows?
> > Of course this also means the other tools, like fpcres, fpmake and
> > fppkg.
> > Would this require a new flag?
> 
> Yes. cmd.exe with /u ? :-)

Is this already supported?
And when executing the compiler directly e.g. from TProcess?

Mattias
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Marco van de Voort]

In our previous episode, Mattias Gaertner said:
> Are there any plans to extend FPC to support Unicode file paths under
> Windows?
> Of course this also means the other tools, like fpcres, fpmake and
> fppkg.
> Would this require a new flag?

Yes. cmd.exe with /u ? :-)
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Michael Van Canneyt]

On Mon, 11 Apr 2016, Mattias Gaertner wrote:


Hi,

Are there any plans to extend FPC to support Unicode file paths under
Windows?
Of course this also means the other tools, like fpcres, fpmake and
fppkg.
Would this require a new flag?


The rtl already support this. But none of the tools currently use unicode
names.

It would also require examining command-line parameters, I think, because I
am not sure these are currently unicodestring (or UTF8).

Michael.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Mattias Gaertner]

On Mon, 11 Apr 2016 20:02:32 +0200
Sven Barth  wrote:

> Am 11.04.2016 19:31 schrieb "Mattias Gaertner" :
>[..]
> In the end the Windows RTL should only use the *W functions and thus fully
> allow for Unicode paths. Or what do you mean? (especially with your last
> question?)

What about printing the paths in notes, hints, etc? What about ppu
files?

Mattias
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

Re: [fpc-devel] Unicode paths

[Sven Barth]

Am 11.04.2016 19:31 schrieb "Mattias Gaertner" :
>
> Hi,
>
> Are there any plans to extend FPC to support Unicode file paths under
> Windows?
> Of course this also means the other tools, like fpcres, fpmake and
> fppkg.
> Would this require a new flag?
>

In the end the Windows RTL should only use the *W functions and thus fully
allow for Unicode paths. Or what do you mean? (especially with your last
question?)

Regards,
Sven
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel

[fpc-devel] Unicode paths

[Mattias Gaertner]

Hi,

Are there any plans to extend FPC to support Unicode file paths under
Windows?
Of course this also means the other tools, like fpcres, fpmake and
fppkg.
Would this require a new flag?

Mattias
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel