[fpc-devel] Changing Windows API A routines in SysUtils to W in Windows NT
Hello, SysUtils for Windows utilizes only A Windows APIs, while it could use W APIs in Windows NT. The attached patch starts correcting improving the situation. Advantages of the patch: * Allows for unicode file names in TStringList.LoadFromFile if you install paswstrings as the unicode manager. Disadvantages of the patch: * None that I can imagine. Waiting for comments. -- Felipe Monteiro de Carvalho unicodesysutils.diff Description: Binary data ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
[fpc-devel] Re: Changing Windows API A routines in SysUtils to W in Windows NT
The related bug tracker item is: http://mantis.freepascal.org/view.php?id=21114 -- Felipe Monteiro de Carvalho ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
[fpc-devel] System 370: Episode 3. Addressing and it's limits Part One!
Episode 3. Addressing and it's limits Part One! First, let me apologise for this post as it's going to be a large one. Second, I don't talk about 64 bit modes here because I have never used them. But the basics will not have altered. IBM really does put a lot of effort into maintaining backwards compatibility. Secondly, I don't actually know anything about the internals of FreePascal or any other compiler come to that, some, or all, of the techniques discussed here, and in part 2, may be impractical or even impossible to implement. It should be noted however that it is not an exhaustive list. Thirdly, it should be noted here that if the intention is to provide support on Hercules based systems, that Hercules allows us to use the newer instructions introduced by the processor upgrades even though we are using processors that shouldn't, in theory, support them. This doesn't apply to providing 31 or 64 bit addressing however, as considerable operating system support is required to handle these modes. Finally, I may include bits of 370 assembler in this post. I don't see how I can avoid it. I will try and keep them as brief and non-technical as possible, but if you feel your eyes glazing over, ask and I will try and explain another way. So, does 370 architecture have a 4k limit on code and data? Well, yes... and no... Sort of... maybe... It depends... Prior to the upgrades of the 390 processor there was only 1 addressing mode, Base / Displacement or effective addressing. The newer processors introduced Program Counter (it's called the PSW on 390 systems) PC relative addressing but it only applies to code and, perhaps constants, and then only to some instructions; It doesn't apply to data, so the limits are still relevant. Base / Displacement consists of a 16 bit value, the first 4 bits enumerate a register, and the other 12 bits hold a displacement from 0 to 4095. The actual or Effective address for each storage operand is calculated as the unsigned addition of the value held in the base register to the displacement from the instruction itself. The effective address for each storage reference is real or virtual and 24 or 31 bit depending upon the mode the processor is in at the time. In our case it will, probably, always be a virtual address. It should be noted that the base register may not be register 0. Register 0 has an implied value of 0 when used for addressing purposes. It is plain that each instruction reference of the Base / Displacement form can only reference a range of 4k, hence the urban myth that that this is a limit on the size of a module. This is where USING enters the fray. USING is an instruction to the assembler. It tells it that a particular register holds the address (24 or 31 bit) of the label mentioned. It is still up to the programmer to load that address into the register, the assembler won't (actually can't) do this for us. Throughout, I am assuming that we will be using what IBM defines as standard linkage conventions between modules. Let's start with a basic bit of code that represents a function: PROG CSECT defines the name of our function £START set up standard linkage LR R12,R15 R15 has the address of PROG, copy it to R12 USING PROG,R12 Tell the assembler to use R12 as base code goes here £END return to caller SAVEAREA DS 18F save area for standard linkage working storage goes here LITPOOL LTORG constants get defined here END £START and £END are macros to set up the standard linkage stuff, SAVEAREA is a required area. The details don't really matter. If the total size of the code, working storage and constants grows beyond 4k, we will get assembly errors. However, we can use the USING instruction to help us out here. Part of the standard linkage is that R13 has to point to an area of 18 fullwords (32 bits each). By adding a USING SAVEAREA,R13 to our code; PROG CSECT defines the name of our function £START set up standard linkage LR R12,R15 R15 has the address of PROG, copy it to R12 USING PROG,R12 Tell the assembler to use R12 as base USING SAVEAREA,R13 use R13 as base register for working storage code goes here £END return to caller SAVEAREA DS 18F save area for standard linkage working storage goes here LITPOOL LTORG constants get defined here END Now we have defined 2 base registers. We are not allocating an extra register, we have to use R13 as a save area pointer anyway. We are using it to address storage after the save area. Now our code can be up to 4k, and our working storage plus constants can be 4k; 8k in total but still limited. One final example and we'll call it a day
Re: [fpc-devel] Changing Windows API A routines in SysUtils to W in Windows NT
On Mon, 6 Feb 2012, Felipe Monteiro de Carvalho wrote: Hello, SysUtils for Windows utilizes only A Windows APIs, while it could use W APIs in Windows NT. The attached patch starts correcting improving the situation. Advantages of the patch: * Allows for unicode file names in TStringList.LoadFromFile if you install paswstrings as the unicode manager. Disadvantages of the patch: * None that I can imagine. Waiting for comments. As you know, we are very keen on backwards compatibility. Are you sure that if a AnsiString filename which contains codepage-specific characters is used to open a file, it will still open the same file after this patch is applied ? In particular, does the opening of a DOS-style filename (Program~1\blah) still work as it used to ? Michael. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Re: Changing Windows API A routines in SysUtils to W in Windows NT
On 6-2-2012 17:54, Felipe Monteiro de Carvalho wrote: The related bug tracker item is: http://mantis.freepascal.org/view.php?id=21114 IMO This should not be done that way (at all): MS does it by respecting the PE flag for unicode and expects strings accordingly: If the PE says it's unicode, a default string should be unicode, (so apiW) if the header says ansi the default should be apiA. Just my three cents. But that still requires changes, I agree with that part. This might be an issue for xplatform, although I believe nixes have the same behaviour. smime.p7s Description: S/MIME Cryptographic Signature ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Changing Windows API A routines in SysUtils to W in Windows NT
As you know, we are very keen on backwards compatibility Ummm, I'll need to investigate that. But if it seams risky we could always add and (DefaultSystemCodePage = CP_UTF8) to the if, which would disable it for legacy code and enable it only for Unicode applications. If you use SetMultiByteConversionCodePage then you are clearly saying you desire Unicode support. On Mon, Feb 6, 2012 at 7:39 PM, Thaddy tha...@thaddy.com wrote: IMO This should not be done that way (at all): MS does it by respecting the PE flag for unicode and expects strings accordingly: If the PE says it's unicode, a default string should be unicode, (so apiW) if the header says ansi the default should be apiA. The Microsoft way is not the same as the Free Pascal way. We are not required to immitate them when implementing our routines. The Microsoft way has nasty side effects: 1 makes it impossible to support Unicode and support Windows 9x at the same time, but I doubt that Free Pascal wants to drop Windows 9x support (even if it is obsolete). 2 Breaks the interface because the string type changed -- Felipe Monteiro de Carvalho ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Changing Windows API A routines in SysUtils to W in Windows NT
On 6-2-2012 19:54, Felipe Monteiro de Carvalho wrote: The Microsoft way is not the same as the Free Pascal way. We are not required to immitate them when implementing our routines. The Microsoft way has nasty side effects: 1 makes it impossible to support Unicode and support Windows 9x at the same time, but I doubt that Free Pascal wants to drop Windows 9x support (even if it is obsolete). 2 Breaks the interface because the string type changed No, this is not true, not at all, to the contrary, that's what the PE flag is for: If you want to compile for any windows version, choose ansi mode and the OS respects and expects that. If you want a unicode default (i.e. win2000 and higher) choose unicode mode. The library should be totally opaque when the programmer decides what he wants. What you are doing is making changes in the generated code that shoukld be resolved at compile time. This can be resolved at link time, but I expect it should be resolved at compile time. That's why I agree with you that changes are necessary. (Although the PE flag is a linker instruction) What we are required to do is respect the OS we compile for. smime.p7s Description: S/MIME Cryptographic Signature ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Changing Windows API A routines in SysUtils to W in Windows NT
On 6-2-2012 19:54, Felipe Monteiro de Carvalho wrote: The Microsoft way is not the same as the Free Pascal way. We are not required to immitate them when implementing our routines. The Microsoft way has nasty side effects: 1 makes it impossible to support Unicode and support Windows 9x at the same time, but I doubt that Free Pascal wants to drop Windows 9x support (even if it is obsolete). 2 Breaks the interface because the string type changed No, this is not true, not at all, to the contrary, that's what the PE flag is for: If you want to compile for any windows version, choose ansi mode and the OS respects and expects that. If you want a unicode default (i.e. win2000 and higher) choose unicode mode. The library should be totally opaque when the programmer decides what he wants. What you are doing is making changes in the generated code that shoukld be resolved at compile time. This can be resolved at link time, but I expect it should be resolved at compile time. That's why I agree with you that changes are necessary. (Although the PE flag is a linker instruction) What we are required to do is respect the OS we compile for. smime.p7s Description: S/MIME Cryptographic Signature ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Re: Changing Windows API A routines in SysUtils to W in Windows NT
On 06.02.2012 19:39, Thaddy wrote: On 6-2-2012 17:54, Felipe Monteiro de Carvalho wrote: The related bug tracker item is: http://mantis.freepascal.org/view.php?id=21114 IMO This should not be done that way (at all): MS does it by respecting the PE flag for unicode and expects strings accordingly: If the PE says it's unicode, a default string should be unicode, (so apiW) if the header says ansi the default should be apiA. Just my three cents. But that still requires changes, I agree with that part. This might be an issue for xplatform, although I believe nixes have the same behaviour. Out of interest: Which flag are you talking about? Because I'm not aware of any such flag. Regards, Sven ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Changing Windows API A routines in SysUtils to W in Windows NT
On 06.02.2012 20:20, Thaddy wrote: On 6-2-2012 19:54, Felipe Monteiro de Carvalho wrote: The Microsoft way is not the same as the Free Pascal way. We are not required to immitate them when implementing our routines. The Microsoft way has nasty side effects: 1 makes it impossible to support Unicode and support Windows 9x at the same time, but I doubt that Free Pascal wants to drop Windows 9x support (even if it is obsolete). 2 Breaks the interface because the string type changed No, this is not true, not at all, to the contrary, that's what the PE flag is for: If you want to compile for any windows version, choose ansi mode and the OS respects and expects that. If you want a unicode default (i.e. win2000 and higher) choose unicode mode. The library should be totally opaque when the programmer decides what he wants. What you are doing is making changes in the generated code that shoukld be resolved at compile time. This can be resolved at link time, but I expect it should be resolved at compile time. That's why I agree with you that changes are necessary. (Although the PE flag is a linker instruction) What we are required to do is respect the OS we compile for. There is no option to set anything unicode related in the linker according to this page: http://msdn.microsoft.com/en-us/library/y0zzbyt4.aspx So what are you talking about? Regards, Sven ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Changing Windows API A routines in SysUtils to W in Windows NT
In our previous episode, Felipe Monteiro de Carvalho said: 1 makes it impossible to support Unicode and support Windows 9x at the same time, but I doubt that Free Pascal wants to drop Windows 9x support (even if it is obsolete). Afaik we already have. Recent versions require patches for win9x ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Re: Changing Windows API A routines in SysUtils to W in Windows NT
On 6-2-2012 20:21, Sven Barth wrote: Out of interest: Which flag are you talking about? Because I'm not aware of any such flag. Regards, Sven ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel It is known which versions are unicode. I am merely refering to the OS version flags. These are the ones which determine if the windows version is unicode or ansi. There's no point in running an ansi compiled exe on a unicode platform. Once a programmer decides his/hers minimum requirements, the compiler should obey. The OS does (up to a certain point) These decisions should be made by the compiler based on platform choice, not in rtl code. That should be optional (as of 2010/XE this is also D compatible) smime.p7s Description: S/MIME Cryptographic Signature ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Re: Changing Windows API A routines in SysUtils to W in Windows NT
On 2/6/2012 1:55 PM, Thaddy wrote:
It is known which versions are unicode. I am merely refering to the OS
version flags. These are the ones which determine if the windows version
is unicode or ansi.
There's no point in running an ansi compiled exe on a unicode platform.
Once a programmer decides his/hers minimum requirements, the compiler
should obey. The OS does (up to a certain point)
These decisions should be made by the compiler based on platform choice,
not in rtl code. That should be optional (as of 2010/XE this is also D
compatible)
In C on Windows the UNICODE symbol controls whether undecorated
functions should use the A or W versions, and what the TCHAR symbol
translates to. There's nothing stopping an application from calling the
A or W versions explicitly, and it's entirely possible for an otherwise
Unicode app to use the A versions for certain functions. That's handled
by the compiler's pre-processor, not the linker and not the loader.
Delphi does the same thing, except the change was built into the
language, rather than done using a conditional symbol, and the A or W
decision was hard-coded rather than wrapping them in {$IFDEF UNICODE}
blocks.
--
Craig Peterson
Scooter Software
___
fpc-devel maillist - fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Re: Changing Windows API A routines in SysUtils to W in Windows NT
On 6-2-2012 21:07, Sven Barth wrote: The check you are talking about in C++ is a compile time define that just switches the defines for e.g. CreateProcess to CreateProcessW or CreateProcessA (depending on whether UNICODE is defined or not when including the Windows headers). There is no other magic in Windows that changes the imports at runtime. Nope, but a compiler should choose the_optimum_ for the platform. Otherwise you can hand-code it. The A/W versions of the windows API have been in Delphi at least since D3, but I can check D2 in a minute. Of course it's a compile time switch! Point made by me in the first place. BTW you can even fool delphi and fpc to behave irratically (do what I want) by doing things like -uFPC -Mdelphi -dVER200 in fpc or defining -dVER200 when you have a D7 (VER150) . Sometimes handy to know the compilers (be it C or FP-C) at least understand the programmers meaning. (Just in case you wonder why I would fool the compiler to be over his age: parsing some badly behaving sourcecode) smime.p7s Description: S/MIME Cryptographic Signature ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Re: Changing Windows API A routines in SysUtils to W in Windows NT
On 6-2-2012 21:04, Craig Peterson wrote:
Delphi does the same thing, except the change was built into the
language, rather than done using a conditional symbol, and the A or W
decision was hard-coded rather than wrapping them in {$IFDEF UNICODE}
blocks.
I am not sure about this, (I do not know if the latest production
version has this) but I think you can (could) undefine unicode.
I propose FPC follows a similar strategy.
We already did it for Kol, which has a (almost) totally opaque
KOLstring/char type depending on switching on UNICODE_CTRLS or ANSI.
(And we had that switch some time before FPC)
smime.p7s
Description: S/MIME Cryptographic Signature
___
fpc-devel maillist - fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Changing Windows API A routines in SysUtils to W in Windows NT
06.02.2012 20:39, Felipe Monteiro de Carvalho пишет: Hello, SysUtils for Windows utilizes only A Windows APIs, while it could use W APIs in Windows NT. The attached patch starts correcting improving the situation. Advantages of the patch: * Allows for unicode file names in TStringList.LoadFromFile if you install paswstrings as the unicode manager. Disadvantages of the patch: * None that I can imagine. Waiting for comments. So, this is basically a first step of locking Windows RTL to use utf-8 by default and reducing chances it ever will call 'W' API without conversions? Sergei ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Re: Changing Windows API A routines in SysUtils to W in Windows NT
On 06.02.2012 21:07, Thaddy wrote: On 6-2-2012 21:04, Craig Peterson wrote: In C on Windows the UNICODE symbol controls whether undecorated functions should use the A or W versions, and what the TCHAR symbol translates to. This exactly the case and I do noy suggest this to be dropped. (This is not only C but also FP-C btw) But in FPC you'll need to recompile the Windows RTL if you want to have UNICODE defined (thus having e.g. CreateProcess default to CreateProcessW instead of CreateProcessA). In C you just include the header with the define either defined or not. Regards, Sven ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Changing Windows API A routines in SysUtils to W in Windows NT
On 6-2-2012 21:35, Sergei Gorelkin wrote: 06.02.2012 20:39, Felipe Monteiro de Carvalho пишет: So, this is basically a first step of locking Windows RTL to use utf-8 by default and reducing chances it ever will call 'W' API without conversions? That is another point that worries me too. Windows NT + are not native UTF8 but UTF16. That's why my suggestion to make the default string type fully opaque per platform requirements. That can be done on the compiler level with little effort in rtl/fcl. (For KOL it took weeks, not months) smime.p7s Description: S/MIME Cryptographic Signature ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Changing Windows API A routines in SysUtils to W in Windows NT
Of course, KOL is a framework, not a compiler. I merely want to state it can be done in a proper way. smime.p7s Description: S/MIME Cryptographic Signature ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Re: Changing Windows API A routines in SysUtils to W in Windows NT
On 6-2-2012 21:41, Sven Barth wrote: But in FPC you'll need to recompile the Windows RTL if you want to have UNICODE defined (thus having e.g. CreateProcess default to CreateProcessW instead of CreateProcessA). In C you just include the header with the define either defined or not. Nope, again no: this has to do with the RTL/FCL Any KOL program (try it) compiles in either unicode (-dUNICODE_CTRLS --Mdelphi -uFPC -dVER150) or ansi (skip the unicode_ctrls) and just plain works as expected. Not only with a Delphi compiler, also with fpc 2.6.0 and higher. This has nothing to do with the compiler itself. That behaves already like it should. smime.p7s Description: S/MIME Cryptographic Signature ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Changing Windows API A routines in SysUtils to W in Windows NT
06.02.2012 23:45, Marco van de Voort пишет: In our previous episode, Felipe Monteiro de Carvalho said: 1 makes it impossible to support Unicode and support Windows 9x at the same time, but I doubt that Free Pascal wants to drop Windows 9x support (even if it is obsolete). Afaik we already have. Recent versions require patches for win9x Just in case: I have written a small unicows.dll loader unit (under 10KBytes source code, in contrast with 300KByte Rob Kennedy's one) that can be used as a 'last goodbye' for win9x, allowing to switch most APIs to 'W'. The only issue is that it needs to be executed as early as possible, ideally before system.pp initialization. This probably can be solved by redefining the entry point passed to linker, or by placing its code into yet another sysinit unit, which is selected for win9x. If this is interesting I can provide more details. Sergei ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Changing Windows API A routines in SysUtils to W in Windows NT
On 6-2-2012 22:14, Sergei Gorelkin wrote: under 10KBytes Any decent code under 10k is interesting smile smime.p7s Description: S/MIME Cryptographic Signature ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Bounty for MIPS
Am 03.02.2012 01:37, schrieb Nikolai Zhubr: I can set up ssh for any FPC developer(s) (though I'll need some time to fix cables etc then) Let me know. It would be nice to get an account, currently I'am still busy with fixing compilation issue but I'd expect first compiled programs within a few days. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
