Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Sven Barth via Lazarus]

On 07.05.2017 12:17, Florian Klaempfl via Lazarus wrote:
> Am 07.05.2017 um 12:11 schrieb Sven Barth via Lazarus:
>> Am 07.05.2017 12:07 schrieb "Florian Klaempfl via Lazarus"
>> >:
>>>
>>> Am 07.05.2017 um 11:57 schrieb Graeme Geldenhuys via Lazarus:
 On 2017-05-07 09:10, Florian Klaempfl via Lazarus wrote:
>> Yeah, that would be the logical thing to do.
>
> Why? What makes a string literal UTF-8?
>

 As Mattias said, the fact that the source unit is UTF-8 encoded.
 Defined by a BOM marker, or -Fcutf8 or {$codepage utf8}. If the source
 unit is UTF-8 encoded, the literal string constant can't (and
 shouldn't) be in any other encoding.

 I would say the same if the source unit was stored in UTF-16
 encoding. Then string literals would be treated as UTF-16.
>>>
>>> And if a ISO/Ansi codepage is given? Things would probably fail.
>>>
>>> The point is: FPC is consistent in this regard: also sources with a
>>> given iso/ansi codepage are handled the same way. If there is a string
>>> literal with non-ascii chars, it is converted to UTF-16 using the
>>> codepage of the source. Very simple, very logical. It is a matter of
>>> preference if UTF-8, -16, -32 are chosen at this point, but FPC uses
>>> UTF-16. If it uses UTF-8, the problem would occur the other way around.
>>>
>>> If no codepage is given (by directive, command line, BOM), string
>>> literals are handled byte-wise as raw strings.
>>
>> Small correction: FPC only does this conversion if the codepage is
>> UTF-8, no other.
> 
> Then something is wrong/broken :)
> 

Well, the code in tscannerfile.readtoken() only does the conversion to
UTF-16 if the source codepage is UTF-8, otherwise it only converts to
UTF-16 if the string is already an UTF-16 string.
So probably not broken as it seems rather on purpose; if at all it's
wrong...

Regards,
Sven
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-07 11:17, Florian Klaempfl via Lazarus wrote:
> Then something is wrong/broken :)


I rest my case.  :-P


Regards,
  Graeme

-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Florian Klaempfl via Lazarus]

Am 07.05.2017 um 12:11 schrieb Sven Barth via Lazarus:
> Am 07.05.2017 12:07 schrieb "Florian Klaempfl via Lazarus"
> >:
>>
>> Am 07.05.2017 um 11:57 schrieb Graeme Geldenhuys via Lazarus:
>> > On 2017-05-07 09:10, Florian Klaempfl via Lazarus wrote:
>> >>> Yeah, that would be the logical thing to do.
>> >>
>> >> Why? What makes a string literal UTF-8?
>> >>
>> >
>> > As Mattias said, the fact that the source unit is UTF-8 encoded.
>> > Defined by a BOM marker, or -Fcutf8 or {$codepage utf8}. If the source
>> > unit is UTF-8 encoded, the literal string constant can't (and
>> > shouldn't) be in any other encoding.
>> >
>> > I would say the same if the source unit was stored in UTF-16
>> > encoding. Then string literals would be treated as UTF-16.
>>
>> And if a ISO/Ansi codepage is given? Things would probably fail.
>>
>> The point is: FPC is consistent in this regard: also sources with a
>> given iso/ansi codepage are handled the same way. If there is a string
>> literal with non-ascii chars, it is converted to UTF-16 using the
>> codepage of the source. Very simple, very logical. It is a matter of
>> preference if UTF-8, -16, -32 are chosen at this point, but FPC uses
>> UTF-16. If it uses UTF-8, the problem would occur the other way around.
>>
>> If no codepage is given (by directive, command line, BOM), string
>> literals are handled byte-wise as raw strings.
> 
> Small correction: FPC only does this conversion if the codepage is
> UTF-8, no other.

Then something is wrong/broken :)

-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Florian Klaempfl via Lazarus]

Am 07.05.2017 um 11:57 schrieb Graeme Geldenhuys via Lazarus:
> On 2017-05-07 09:10, Florian Klaempfl via Lazarus wrote:
>>> Yeah, that would be the logical thing to do. 
>>
>> Why? What makes a string literal UTF-8?
>>
> 
> As Mattias said, the fact that the source unit is UTF-8 encoded.
> Defined by a BOM marker, or -Fcutf8 or {$codepage utf8}. If the source
> unit is UTF-8 encoded, the literal string constant can't (and
> shouldn't) be in any other encoding.
> 
> I would say the same if the source unit was stored in UTF-16
> encoding. Then string literals would be treated as UTF-16.

And if a ISO/Ansi codepage is given? Things would probably fail.

The point is: FPC is consistent in this regard: also sources with a
given iso/ansi codepage are handled the same way. If there is a string
literal with non-ascii chars, it is converted to UTF-16 using the
codepage of the source. Very simple, very logical. It is a matter of
preference if UTF-8, -16, -32 are chosen at this point, but FPC uses
UTF-16. If it uses UTF-8, the problem would occur the other way around.

If no codepage is given (by directive, command line, BOM), string
literals are handled byte-wise as raw strings.

> 
> It's perfectly logical to me.

It is logical only in a limited view.

-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Sven Barth via Lazarus]

Am 07.05.2017 12:07 schrieb "Florian Klaempfl via Lazarus" <
lazarus@lists.lazarus-ide.org>:
>
> Am 07.05.2017 um 11:57 schrieb Graeme Geldenhuys via Lazarus:
> > On 2017-05-07 09:10, Florian Klaempfl via Lazarus wrote:
> >>> Yeah, that would be the logical thing to do.
> >>
> >> Why? What makes a string literal UTF-8?
> >>
> >
> > As Mattias said, the fact that the source unit is UTF-8 encoded.
> > Defined by a BOM marker, or -Fcutf8 or {$codepage utf8}. If the source
> > unit is UTF-8 encoded, the literal string constant can't (and
> > shouldn't) be in any other encoding.
> >
> > I would say the same if the source unit was stored in UTF-16
> > encoding. Then string literals would be treated as UTF-16.
>
> And if a ISO/Ansi codepage is given? Things would probably fail.
>
> The point is: FPC is consistent in this regard: also sources with a
> given iso/ansi codepage are handled the same way. If there is a string
> literal with non-ascii chars, it is converted to UTF-16 using the
> codepage of the source. Very simple, very logical. It is a matter of
> preference if UTF-8, -16, -32 are chosen at this point, but FPC uses
> UTF-16. If it uses UTF-8, the problem would occur the other way around.
>
> If no codepage is given (by directive, command line, BOM), string
> literals are handled byte-wise as raw strings.

Small correction: FPC only does this conversion if the codepage is UTF-8,
no other.

Regards,
Sven
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-07 09:10, Florian Klaempfl via Lazarus wrote:
>> Yeah, that would be the logical thing to do. 
>
> Why? What makes a string literal UTF-8?
> 

As Mattias said, the fact that the source unit is UTF-8 encoded.
Defined by a BOM marker, or -Fcutf8 or {$codepage utf8}. If the source
unit is UTF-8 encoded, the literal string constant can't (and
shouldn't) be in any other encoding.

I would say the same if the source unit was stored in UTF-16
encoding. Then string literals would be treated as UTF-16.

It's perfectly logical to me.

Regards,
  Graeme

-- 
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Florian Klaempfl via Lazarus]

Am 07.05.2017 um 10:30 schrieb Mattias Gaertner via Lazarus:
> On Sun, 7 May 2017 10:10:26 +0200
> Florian Klaempfl via Lazarus  wrote:
> 
>> Am 05.05.2017 um 13:35 schrieb Graeme Geldenhuys via Lazarus:
>>> On 2017-05-05 12:17, Mattias Gaertner via Lazarus wrote:  
 I wonder if it would help if FPC would store UTF-8 string literals as
 UTF-8   
>>>
>>> Yeah, that would be the logical thing to do.   
>>
>> Why? What makes a string literal UTF-8?
> 
> Here: $codepage utf-8 and non ASCII.
> 
> 
>>> FPC not doing that is what
>>> really confused me.  
>>
>> You have to distinuish between source encoding and string encoding.
> 
> Yes, but sometimes the string encoding is not obvious:
> 
> {$codepage utf8}
> const s = 'äöüالعَرَبِيَّة';
> begin
>   writeln(s); // needs widestringmanager
> end.

Yes. Which is good imo. The compiler should call the unicode writeln in
this case and this requires a widestringmanager (where I agree, its name
is chosen wrong, it should be named unicodestringmanager) anyways. If
the string were encoded UTF-8 and the ansistring writeln is called, your
example would work only on a system with an utf-8 console. But you can
achieve the same (broken output) by just leaving away the codepage
directive then. It will not work on non utf-8 consoles either.

-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Mattias Gaertner via Lazarus]

On Sun, 7 May 2017 10:10:26 +0200
Florian Klaempfl via Lazarus  wrote:

> Am 05.05.2017 um 13:35 schrieb Graeme Geldenhuys via Lazarus:
> > On 2017-05-05 12:17, Mattias Gaertner via Lazarus wrote:  
> >> I wonder if it would help if FPC would store UTF-8 string literals as
> >> UTF-8   
> > 
> > Yeah, that would be the logical thing to do.   
> 
> Why? What makes a string literal UTF-8?

Here: $codepage utf-8 and non ASCII.


> > FPC not doing that is what
> > really confused me.  
> 
> You have to distinuish between source encoding and string encoding.

Yes, but sometimes the string encoding is not obvious:

{$codepage utf8}
const s = 'äöüالعَرَبِيَّة';
begin
  writeln(s); // needs widestringmanager
end.

Mattias
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Florian Klaempfl via Lazarus]

Am 05.05.2017 um 13:35 schrieb Graeme Geldenhuys via Lazarus:
> On 2017-05-05 12:17, Mattias Gaertner via Lazarus wrote:
>> I wonder if it would help if FPC would store UTF-8 string literals as
>> UTF-8 
> 
> Yeah, that would be the logical thing to do. 

Why? What makes a string literal UTF-8?

> FPC not doing that is what
> really confused me.

You have to distinuish between source encoding and string encoding.

-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Mattias Gaertner via Lazarus]

On Fri, 5 May 2017 16:36:51 +0300
Juha Manninen via Lazarus  wrote:

> On Fri, May 5, 2017 at 4:21 PM, Mattias Gaertner via Lazarus
>  wrote:
> > Oops. Which one?  
> 
> The FAQ says:
> "Since FPC 3.0 you must add the flag -FcUTF8 or add {$codepage UTF8}
> at the beginning of the unit."

I improved it a bit.

Mattias
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Juha Manninen via Lazarus]

On Fri, May 5, 2017 at 4:21 PM, Mattias Gaertner via Lazarus
 wrote:
> Oops. Which one?

The FAQ says:
"Since FPC 3.0 you must add the flag -FcUTF8 or add {$codepage UTF8}
at the beginning of the unit."

The same page in "String Literals" section says:
 "In most cases {$codepage utf8} / -FcUTF8 is not needed."
which is the correct information.

Actually I don't know if that FAQ entry is yours. Many people have
added stuff there. The page is intimidating for a user who just wants
to support Unicode without a fuss.

Juha
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Juha Manninen via Lazarus]

On Fri, May 5, 2017 at 3:56 PM, Sven Barth via Lazarus
 wrote:
> That is mainly due to the compiler not supporting surrogate pairs for the
> UTF-8 -> UTF-16 conversion. If it would support them, then there wouldn't be
> a problem anymore...

That is a serious bug. Getting codepoints right is the absolute
minimum requirement for Unicode support. Surrogate pairs are the
UTF-16 equivalent of multi-byte codepoints in UTF-8.

Now I understand this was not caused by our UTF-8 run-time switch
"hack". It is a plain bug in FPC.
Is there a plan to fix it?

Juha
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Mattias Gaertner via Lazarus]

On Fri, 5 May 2017 14:12:05 +0300
Juha Manninen via Lazarus  wrote:

>[...]
> Then Mattias adds FAQs contradicting the earlier texts ...

Oops. Which one?

Mattias
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Sven Barth via Lazarus]

Am 05.05.2017 13:50 schrieb "Juha Manninen via Lazarus" <
lazarus@lists.lazarus-ide.org>:
>
> On Fri, May 5, 2017 at 2:29 PM, Michael Van Canneyt via Lazarus
>  wrote:
> > Then what is still the problem ?
>
> With BOM you get:
>  Error: UTF-8 code greater than 65535 found
> which is counter-intuitive when the file and the string literal are both
UTF-8.

That is mainly due to the compiler not supporting surrogate pairs for the
UTF-8 -> UTF-16 conversion. If it would support them, then there wouldn't
be a problem anymore... (though of course it would need to be ensured that
other parts of the RTL support surrogate pairs as well)

Regards,
Sven
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-05 12:49, Juha Manninen via Lazarus wrote:
> A wrong information easily propagates, thus it is important to get this right.

No worries, I agree. Thanks for correcting my terminology.

Regards,
  Graeme

-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Juha Manninen via Lazarus]

On Fri, May 5, 2017 at 2:02 PM, Graeme Geldenhuys via Lazarus
 wrote:
> If so, when why does LCL also call the above two functions?

Graeme, they are called by LazUtils package, LazUTF8 unit, not by LCL.
It is not limited to GUI programming.
A wrong information easily propagates, thus it is important to get this right.

Juha
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Juha Manninen via Lazarus]

On Fri, May 5, 2017 at 2:29 PM, Michael Van Canneyt via Lazarus
 wrote:
> Then what is still the problem ?

With BOM you get:
 Error: UTF-8 code greater than 65535 found
which is counter-intuitive when the file and the string literal are both UTF-8.
It is related to changing the default codepage at run-time which is a
hack from FPC's POV.
For the same reason we need this grid:
 
http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#String_Literals_Overview
So, it is not only a communication issue. It is truly messy. If only
it could be improved...

Juha
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Ondrej Pokorny via Lazarus]

On 05.05.2017 13:02, Graeme Geldenhuys via Lazarus wrote:

On 2017-05-05 10:41, Ondrej Pokorny via Lazarus wrote:

Just use "DefaultSystemCodePage := CP_UTF8" and every single-byte string
is unicode enabled.

So does that mean you don't have to also call the following two functions 
(which LCL does).

  SetMultiByteConversionCodePage(CP_UTF8);
  SetMultiByteRTLFileSystemCodePage(CP_UTF8);


So doing

DefaultSystemCodePage := CP_UTF8;

is all you need to switch the RTL, FCL and the String data type to UTF-8?

If so, when why does LCL also call the above two functions?


SetMultiByteConversionCodePage does only one thing: it sets 
DefaultSystemCodePage :) So yes, if you set DefaultSystemCodePage you 
don't have to call SetMultiByteConversionCodePage.


You are right - I forgot about 
SetMultiByteRTLFileSystemCodePage/DefaultRTLFileSystemCodePage.


BUT if I take a look into the RTL sources I see that it's used only in 
FindFirst/FindNext, FExpand and GetDir/do_GetDir. And only in the result 
strings. IMO it could be removed and replaced with DefaultSystemCodePage.


Ondrej
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-05 12:17, Mattias Gaertner via Lazarus wrote:
> I wonder if it would help if FPC would store UTF-8 string literals as
> UTF-8 

Yeah, that would be the logical thing to do. FPC not doing that is what
really confused me.

Regards,
  Graeme

-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Michael Van Canneyt via Lazarus]

On Fri, 5 May 2017, Mattias Gaertner via Lazarus wrote:


On Fri, 5 May 2017 12:52:48 +0200 (CEST)
Michael Van Canneyt via Lazarus  wrote:


[...]
I propose to let the compiler observe the BOM. 
But I don't think more is needed.


FPC observes the BOM. Same as Delphi.


Then what is still the problem ?



I wonder if it would help if FPC would store UTF-8 string literals as
UTF-8 and how much work that is.


Amount of work is probably not so much.

The question is whether it will cause problems for e.g. the JVM code
generator.

But I still fail to see the actual problem, aside from a lot of confusion by
users.

Confusion arises from a lack of clear information.

So in my opinion, we simply need to provide clear information, before we
start changing things.

Michael.
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Mattias Gaertner via Lazarus]

On Fri, 5 May 2017 12:52:48 +0200 (CEST)
Michael Van Canneyt via Lazarus  wrote:

>[...]
> I propose to let the compiler observe the BOM. 
> But I don't think more is needed.

FPC observes the BOM. Same as Delphi.

I wonder if it would help if FPC would store UTF-8 string literals as
UTF-8 and how much work that is.

Mattias
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Michael Schnell via Lazarus]

On 05.05.2017 12:16, Graeme Geldenhuys via Lazarus wrote:

In the end it’s about supporting Unicode. Does it really matter
what internal encoding it is to achieve the “Unicode support”
goal?


Yep it does.

There are ways around that issue (i.e. code aware strings) but in fact 
these trigger a new bunch of problems.


You might want to read -> 
http://wiki.freepascal.org/not_Delphi_compatible_enhancement_for_Unicode_Support


-Michael
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-05 11:55, Jürgen Hestermann via Lazarus wrote:
> I use UTF-8 internally and
> convert to/from UTF-16 for all Windows API functions and
> I never found any problem with it.
> The time that the API functions requires is so much longer than the
> time for string conversion that it does not matter at all.

This is what I've been doing for years, and I agree, it works great.
Windows is also the only platform (of any modern OS) that doesn't
use UTF-8 as standard - so I consider it the minority.


> A situation where it may be a problem is when reading
> (UTF-16 encoded) text files.

I'm yet to find a UTF-16 encoded text file in the wild. I'm not saying
they don't exist, I'm just saying they are extremely rare and more
like an anomaly. UTF-8 seems to rule the roost and the Internet.

This graph should say it all:

https://en.wikipedia.org/wiki/File:Utf8webgrowth.svg

  (source):  https://en.wikipedia.org/wiki/UTF-8

Even so, a simple conversion to UTF-8 at load time should resolve
all possible problems.



Regards,
  Graeme

-- 
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Juha Manninen via Lazarus]

On Fri, May 5, 2017 at 1:20 AM, Graeme Geldenhuys via Lazarus
 wrote:
> A case in point. Looking at the Wiki page you listed, I read the following:
> "
> Since FPC 3.0 you must add the flag -FcUTF8 or add {$codepage UTF8} at the 
> beginning of the unit.
> ...

Uhhh, the same page in "String Literals" section says:
 "In most cases {$codepage utf8} / -FcUTF8 is not needed."
which is the correct information.

Also this wiki page has become a mess when many people add stuff but
nobody removes any.
For example Michl added the grid about how constant assignment works
with and without {$codepage utf8}. It is nice but he didn't remove the
other paragraphs explaining the same thing. It looks like an extremely
complex topic for a new user, while in reality he should code like
with Delphi + remember only few simple rules.
Then Mattias adds FAQs contradicting the earlier texts ...

The comment from Martok was valid. This page is not good for users who
just want to get started quickly.
I will simplify the page. I will remove stuff and move the FAQ to a
new page. Sorry in advance for people who's text will be removed.

Juha
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-05 10:41, Ondrej Pokorny via Lazarus wrote:
> Just use "DefaultSystemCodePage := CP_UTF8" and every single-byte string 
> is unicode enabled.

So does that mean you don't have to also call the following two functions 
(which LCL does).

 SetMultiByteConversionCodePage(CP_UTF8);
 SetMultiByteRTLFileSystemCodePage(CP_UTF8);


So doing

   DefaultSystemCodePage := CP_UTF8;

is all you need to switch the RTL, FCL and the String data type to UTF-8?

If so, when why does LCL also call the above two functions?

Regards,
  Graeme

-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Ondrej Pokorny via Lazarus]

On 05.05.2017 12:01, Michael Van Canneyt via Lazarus wrote:

On Fri, 5 May 2017, Ondrej Pokorny via Lazarus wrote:
Believe me, I use it in production without any problems: I have 
unicode-aware TStrings, I can read files with unicode names, I can do 
everything with plain FPC trunk.


I am aware of this, I do it myself. But I work on Linux, where UTF8 is 
the norm.


So I cannot vouch for other platforms...


For now I am only on Windows and I have to say loadly: IT WORKS GREAT :)


I don't need a 100% UTF-16 Delphi-Compatible RTL for that at all.


This is the crux of the problem. Is this wanted/needed or do we stick 
to UTF8 ?


We claim Delphi compatibility. So IMHO we must provide a UTF-16 Delphi 
compatible RTL.


I write code that is compatible with FPC and Delphi 5 - 10.2 and it 
works fine. So you already have a Delphi-compatible RTL. The only (well 
documented) difference is that FPC uses single-byte string and Delphi 
uses 2-byte string.


The only place where you need to handle the difference is where you need 
the size of char (when you access string as buffer) - which is 
particularly low-level code:


MyStream.WriteBuffer(MyString[1], Length(MyString) * SizeOf(Char));

-> you need the extra SizeOf(Char) and not a constant (1 for fpc, 2 for 
unicode Delphi).


That's all. All high-level code is compatible already. Good job. I 
really do think it's not worth it to pollute FPC RTL with UnicodeString 
overloads of every function, class etc.


Better to keep 1 clean approach (UTF-8 RTL) and not confuse people with 
2 approaches (UTF-8 vs UTF-16). E.g. how do you want to call the new 
UnicodeString-TStrings class? You have 2 options:
1.) Break compatibility to legacy FPC. (New TStrings will use 
UnicodeString.)

2.) Break compatibility to Delphi. (TStrings will stay with 8-byte string.)

There is no obvious solution for the problem :/

And then if you will introduce a compiler switch to change String from 
1-byte to 2-bytes... Oh no, so much mess and so many variants to care 
about. Really, sometimes it's better to give people no options :) (Or 
have you already introduced the switch?)


Just stick with current utf8 approach that proved well :)

Ondrej
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Jürgen Hestermann via Lazarus]

Am 2017-05-05 um 12:16 schrieb Graeme Geldenhuys via Lazarus:
> In the end it’s about supporting Unicode. Does it really matter
> what internal encoding it is to achieve the “Unicode support”
> goal?

From a performance perspective it may be unwanted
to convert string encodings back and forth all the time.

Although, in my file manager I use UTF-8 internally and
convert to/from UTF-16 for all Windows API functions and
I never found any problem with it.
The time that the API functions requires is so much longer than the
time for string conversion that it does not matter at all.
Even fast API-functions like changing attributes only take
a second for thousands of files.

A situation where it may be a problem is when reading
(UTF-16 encoded) text files.
But I never stumbled over such a thing yet.

I would promote the use of UTF-8 whereever possible
while converting to target encodings only when unavoidable.
It makes life much easier if you only concentrate on one (the best)
Unicode encoding (UTF-8).

Therefore I see no use of a UTF-16 bases RTL.
I don't think that you would notice any performance difference
to the UTF-8 based RTL.
It would only waste valuable time that can be invested in other things.

--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Michael Van Canneyt via Lazarus]

On Fri, 5 May 2017, Juha Manninen via Lazarus wrote:


On Fri, May 5, 2017 at 9:43 AM, Michael Van Canneyt via Lazarus
 wrote:

What tricks do you still need in 3.0.x ?


The annoying tricky part with our UTF-8 solution is the assignment of
Unicode string literals.
With UTF-8 BOM it does not work at all, as discussed here.
Without BOM it depends on string type + compiler settings in an illogical way.
We would need a more robust solution for that. Do you have ideas?


I propose to let the compiler observe the BOM. 
But I don't think more is needed.


Michael.
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Mattias Gaertner via Lazarus]

On Fri, 5 May 2017 12:17:22 +0200
Ondrej Pokorny via Lazarus  wrote:

>[...]
> Embarcadero realized they made a mistake when they disabled (yes, only 
> disabled not removed) 8-byte strings from NEXTGEN compilers. UTF8String 
> and RawByteString are back for all NEXTGEN compilers since 10.1. You can 
> use them in Linux Delphi as well.
> 
> http://andy.jgknet.de/blog/2016/05/system-bytestrings-for-10-1-berlin/

Wow. I guess that means FPC lost the title of
"compiler with most confusing string types".


Mattias
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Mattias Gaertner via Lazarus]

On Fri, 5 May 2017 12:01:47 +0200 (CEST)
Michael Van Canneyt via Lazarus  wrote:

>[...]
> > Believe me, I use it in production without any problems: I have 
> > unicode-aware TStrings, I can read files with unicode names, I can do 
> > everything with plain FPC trunk.  
> 
> I am aware of this, I do it myself. 
> But I work on Linux, where UTF8 is the norm.
> 
> So I cannot vouch for other platforms...

It worked on Linux since years.
Since FPC 3.0 it works on Windows too.

Mattias
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-05 11:01, Michael Van Canneyt via Lazarus wrote:
> We claim Delphi compatibility. 
> So IMHO we must provide a UTF-16 Delphi compatible RTL.

In the end it’s about supporting Unicode. Does it really matter
what internal encoding it is to achieve the “Unicode support”
goal?


Regards,
  Graeme

-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Mattias Gaertner via Lazarus]

On Fri, 5 May 2017 10:56:41 +0100
Graeme Geldenhuys via Lazarus  wrote:

>[...]
> > or work with large amount of 8-bit strings.  
> 
> Why would you want to? Unicode supports all languages,

Maybe there is a misunderstanding. Let me rephrase my question:
What string do you use in Linux Delphi when working with UTF-8 strings?

Mattias
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-05 10:41, Mattias Gaertner via Lazarus wrote:
> I wonder what they do when you need to access the raw 8-bit file names,


OSX, iOS, Android and Linux all use UTF-8 as standard, so filename access
is not going to be any problem. Windows is moving more and more towards
UTF-16 everywhere, so that shouldn't be a problem either.


> or work with large amount of 8-bit strings.

Why would you want to? Unicode supports all languages, there simply is no
need for other non-Unicode encodings any more. If it is memory usage
you are worried about, convert your 8-bit strings as UTF-8 encoded text
(most Western countries text will all use low memory then - compared to
UTF-16 as an alternative).

Java has only supported Unicode since its inception in 1995, and Java runs
everywhere. It's never had a problem running on non-Unicode enabled
platforms.


Regards,
  Graeme

-- 
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Mattias Gaertner via Lazarus]

On Fri, 5 May 2017 10:01:24 +0100
Graeme Geldenhuys via Lazarus  wrote:

>[...]
> > AFAIK you are using UTF-8 in AnsiString in FPC 2.6.4. That works in
> > many cases, because of double fooling the compiler. This trick does not
> > work on Windows with RTL file functions though.  
> 
> Yes and true, but fpGUI supplies its own "wrapper" RTL file functions, thus
> it works 100% on all platforms for years. I believe LCL used to do the same.

Yes, and with FPC 3.0 many of them are no longer needed.

 
> RawByteString type (yet another string type in FPC & Delphi's arsenal) did
> not exist at the time, otherwise I would probably have defined...
> 
>   TfpgString = RawByteString;
> 
> and used that everywhere.

How would that help?

  
> > Of course it would be nicer, if we don't need tricks to get Unicode.  
> 
> Indeed, and that is why I love solutions implemented by Java and Qt
> Framework. They are simple, it works and not confusing.

IMO you are comparing apples and oranges.
The FP compiler provides a very easy Unicode solution - or even two
(UTF-8 and UTF-16). The problem are the old RTL and libs, which are
written for system encoding, not for Unicode.
You can design in FPC an Unicode RTL just like Java and QT. fpgui and
LazUtils are kind of a start of that.
Or you can help FPC finishing the Unicode RTL. So stop complaining and
help them.


> Even Embarcadero
> is doing some string type clean-up. Their new Linux compiler completely
> removed AnsiString support. After all, why do you need any other
> string types when you support the Unicode standard.

That's true for most cases.
I wonder what they do when you need to access the raw 8-bit file names,
or work with large amount of 8-bit strings.

Mattias
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Michael Van Canneyt via Lazarus]

On Fri, 5 May 2017, Ondrej Pokorny via Lazarus wrote:


On 05.05.2017 11:17, Michael Van Canneyt via Lazarus wrote:

On Fri, 5 May 2017, Graeme Geldenhuys via Lazarus wrote:


On 2017-05-05 07:43, Michael Van Canneyt via Lazarus wrote:

As far as I know, you don't need any tricks to work with unicode
filenames or output in 3.0.2. Maybe with exception of TStrings and
TFileStream.


Again, I didn't have time to follow FPC 3.x development much, and I 
was too

confused with all the Unicode changes.

With FPC 3.0.x, can you now load text files from disk using 
TStringList and

specify the encoding of the file at load time?

Something like:
 sl.LoadFromFile('some_utf8_file.txt', CP_UTF8);
 sl.LoadFromFile('some_utf16_file.txt', CP_UTF16);
 sl.LoadFromFile('some_latin1_file.txt', CP_Latin1);


Not yet. These are the exceptions I was talking about.


Current trunk 3.1.1 can do that since r34475 - you applied it :) I don't 
know if you ported it back to 3.0.x, though.


As far as I know, it is not backported, but Marco would need to confirm it.

Michael.
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-05 10:17, Michael Van Canneyt via Lazarus wrote:
>> Something like: 
>>
>>  sl.LoadFromFile('some_utf8_file.txt', CP_UTF8);
>>  sl.LoadFromFile('some_utf16_file.txt', CP_UTF16);
>>  sl.LoadFromFile('some_latin1_file.txt', CP_Latin1);
>
> Not yet. These are the exceptions I was talking about.


That also means FPC 3.0.x is then seriously flawed. It
supports Unicode, but it also doesn't support Unicode.

So what is the suggested work-around for FPC 3.0.2 to load
various text encoding files into a TStringList? Hopefully
the answer is not: "there is none"  :-/

Because that will seriously impair/break INI usage too. The
first example off the top of my head. XML and JSON probably
too.

Regards,
  Graeme

-- 
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-05 10:17, Ondrej Pokorny via Lazarus wrote:
> I don't know about 3.0.x but you can do it in trunk 3.1.1. I posted a 
> patch for it (r34475). 

Fantastic! Glad to see somebody was thinking in the same train of thought
as I did. :)

Is that scheduled to be back-ported to FPC 3.0.x?

Regards,
  Graeme

-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Ondrej Pokorny via Lazarus]

On 05.05.2017 11:17, Michael Van Canneyt via Lazarus wrote:

On Fri, 5 May 2017, Graeme Geldenhuys via Lazarus wrote:


On 2017-05-05 07:43, Michael Van Canneyt via Lazarus wrote:

As far as I know, you don't need any tricks to work with unicode
filenames or output in 3.0.2. Maybe with exception of TStrings and
TFileStream.


Again, I didn't have time to follow FPC 3.x development much, and I 
was too

confused with all the Unicode changes.

With FPC 3.0.x, can you now load text files from disk using 
TStringList and

specify the encoding of the file at load time?

Something like:
 sl.LoadFromFile('some_utf8_file.txt', CP_UTF8);
 sl.LoadFromFile('some_utf16_file.txt', CP_UTF16);
 sl.LoadFromFile('some_latin1_file.txt', CP_Latin1);


Not yet. These are the exceptions I was talking about.


Current trunk 3.1.1 can do that since r34475 - you applied it :) I don't 
know if you ported it back to 3.0.x, though.


TFileStream can also open files with unicode names - at least on Windows 
(since 3.0.0 if I am not mistaken). See


Function FileCreate (Const FileName : UnicodeString; ShareMode : 
Integer; Rights : Integer) : THandle;


in rtl/win/sysutils.pas

-->> There are absolutely no limitations whatsover, AFAIK. At least I 
don't know any and I don't experience any.


Ondrej
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Michael Van Canneyt via Lazarus]

On Fri, 5 May 2017, Ondrej Pokorny via Lazarus wrote:


On 05.05.2017 11:06, Graeme Geldenhuys via Lazarus wrote:

On 2017-05-05 07:43, Michael Van Canneyt via Lazarus wrote:

As far as I know, you don't need any tricks to work with unicode
filenames or output in 3.0.2. Maybe with exception of TStrings and
TFileStream.

Again, I didn't have time to follow FPC 3.x development much, and I was too
confused with all the Unicode changes.

With FPC 3.0.x, can you now load text files from disk using TStringList and
specify the encoding of the file at load time?

Something like:

   sl.LoadFromFile('some_utf8_file.txt', CP_UTF8);
   sl.LoadFromFile('some_utf16_file.txt', CP_UTF16);
   sl.LoadFromFile('some_latin1_file.txt', CP_Latin1);


I don't know about 3.0.x but you can do it in trunk 3.1.1. I posted a 
patch for it (r34475). I also extended TEncoding to support AnsiString, 
which was the requirement for TStrings encoding support.


Yes, this somewhat alleviates the problem; 
but this still is a single-byte TStrings, as opposed to the WideString

TStrings of Delphi. It's also still a single-byte filename argument.

Michael.
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-05 09:59, Michael Schnell via Lazarus wrote:
> (Most obvious drawback: not flexibly typed TStrings.)

I know not everybody likes Generics, but that is where I see
Generics could come in very handy. A single TStrings implementation
that supports multiple string types.

Or just implement a UTF-8 version. ;-)
On a side note:
  I have implemented a UTF-8 version of TStrings & TStringList somewhere
  on my hard drive.

Regards,
  Graeme

-- 
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Michael Van Canneyt via Lazarus]

On Fri, 5 May 2017, Graeme Geldenhuys via Lazarus wrote:


On 2017-05-05 07:43, Michael Van Canneyt via Lazarus wrote:

As far as I know, you don't need any tricks to work with unicode
filenames or output in 3.0.2. Maybe with exception of TStrings and
TFileStream.


Again, I didn't have time to follow FPC 3.x development much, and I was too
confused with all the Unicode changes.

With FPC 3.0.x, can you now load text files from disk using TStringList and
specify the encoding of the file at load time?

Something like: 


 sl.LoadFromFile('some_utf8_file.txt', CP_UTF8);
 sl.LoadFromFile('some_utf16_file.txt', CP_UTF16);
 sl.LoadFromFile('some_latin1_file.txt', CP_Latin1);


Not yet. These are the exceptions I was talking about.
But the FileOpen, Assign,Reset, Write of plain pascal 
do work with both Unicode and plain strings.


To fix the classes issues properly, we need a unicode RTL 
and a ANSI RTL if we wish to remain backwards compatible:

The Strings[] property can have only 1 type.

Michael.
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Ondrej Pokorny via Lazarus]

On 05.05.2017 11:06, Graeme Geldenhuys via Lazarus wrote:

On 2017-05-05 07:43, Michael Van Canneyt via Lazarus wrote:

As far as I know, you don't need any tricks to work with unicode
filenames or output in 3.0.2. Maybe with exception of TStrings and
TFileStream.

Again, I didn't have time to follow FPC 3.x development much, and I was too
confused with all the Unicode changes.

With FPC 3.0.x, can you now load text files from disk using TStringList and
specify the encoding of the file at load time?

Something like:

   sl.LoadFromFile('some_utf8_file.txt', CP_UTF8);
   sl.LoadFromFile('some_utf16_file.txt', CP_UTF16);
   sl.LoadFromFile('some_latin1_file.txt', CP_Latin1);


I don't know about 3.0.x but you can do it in trunk 3.1.1. I posted a 
patch for it (r34475). I also extended TEncoding to support AnsiString, 
which was the requirement for TStrings encoding support.


Ondrej
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-05 09:31, Kostas Michalopoulos via Lazarus wrote:
> After all, BMP does include practically all languages used today.

The bottom line:

   Unicode Standard <> BMP only!

If you think that, then rather promote your application as a UCS-2
compliant application, not a Unicode compliant application.

I can't remember my exact use case at the time, but the code-points
I needed to work with (using a data dump text file) were outside
the BMP range. I had to use a Java based text editor to correctly
edit the files.

Also, as Mattias said, the Emoji's, Musical notes, Scientific symbols,
Map symbols etc all fall outside the BMP.

Regards,
  Graeme

-- 
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-05 07:43, Michael Van Canneyt via Lazarus wrote:
> As far as I know, you don't need any tricks to work with unicode
> filenames or output in 3.0.2. Maybe with exception of TStrings and
> TFileStream.

Again, I didn't have time to follow FPC 3.x development much, and I was too
confused with all the Unicode changes.

With FPC 3.0.x, can you now load text files from disk using TStringList and
specify the encoding of the file at load time?

Something like:  

  sl.LoadFromFile('some_utf8_file.txt', CP_UTF8);
  sl.LoadFromFile('some_utf16_file.txt', CP_UTF16);
  sl.LoadFromFile('some_latin1_file.txt', CP_Latin1);

etc

Regards,
  Graeme

-- 
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-05 00:15, Mattias Gaertner via Lazarus wrote:

> I added a FAQ:
> http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#What_happens_when_I_use_.24codepage_utf8.3F

Ah, thanks for that explanation.

 
> AFAIK you are using UTF-8 in AnsiString in FPC 2.6.4. That works in
> many cases, because of double fooling the compiler. This trick does not
> work on Windows with RTL file functions though.

Yes and true, but fpGUI supplies its own "wrapper" RTL file functions, thus
it works 100% on all platforms for years. I believe LCL used to do the same.

RawByteString type (yet another string type in FPC & Delphi's arsenal) did
not exist at the time, otherwise I would probably have defined...

  TfpgString = RawByteString;

and used that everywhere.

 
> Of course it would be nicer, if we don't need tricks to get Unicode.

Indeed, and that is why I love solutions implemented by Java and Qt
Framework. They are simple, it works and not confusing. Even Embarcadero
is doing some string type clean-up. Their new Linux compiler completely
removed AnsiString support. After all, why do you need any other
string types when you support the Unicode standard.


Regards,
  Graeme

-- 
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Michael Schnell via Lazarus]

On 04.05.2017 16:56, Juha Manninen via Lazarus wrote:

I believe everybody is happy to get rid of the horrendous Windows
If if this is true, there is a decent need for backwards compatibility. 
That is why, theoretically, code aware strings is a good idea. 
Unfortunately the implementation of those, IMHO, is abysmal, as well in 
Delphi, as in fpc. (Most obvious drawback: not flexibly typed TStrings.)


-Michael
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Mattias Gaertner via Lazarus]

On Fri, 5 May 2017 11:31:00 +0300
Kostas Michalopoulos via Lazarus  wrote:

>[...]
> To play the devil's advocate, the fact that ALL reviews said that it has
> excellent support for Unicode means that characters outside the BMP *are*
> rare. After all, BMP does include practically all languages used today.
> 
> I mean, it isn't technically correct, it is just that in practice it is
> good enough for a very large number of tasks.

Devil's advocate: The new emojis are outside BMP.

Mattias
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-04 21:53, Juha Manninen via Lazarus wrote:
> It is briefly explained here:

I haven't been following FPC 3.x development much because I think the Unicode 
changes are terribly confusing.

A case in point. Looking at the Wiki page you listed, I read the following:

"
Since FPC 3.0 you must add the flag -FcUTF8 or add {$codepage UTF8} at the 
beginning of the unit.

  * Note: This changes all string literals to UTF-16, increasing the size of 
the binary and slowing it down. That's why Lazarus does not add it by default.
"

That makes NO sense to me. My units are always saved as UTF-8 encoded text. By 
helping the compiler out by explicitly telling it my files are UTF-8 encoded 
using -FcUTF8 or adding {$codepage UTF8} or saving the unit with a BOM marker 
breaks writeln() output under Linux/FreeBSD. Who knows what else it breaks. 
Apparently the breakage is because of the "NOTE" quoted above. Why the hell 
does FPC consider string literals UTF-16 when I explicitly told it the whole 
unit is UTF-8 encoded? FPC is doing the opposite of what I told it!

I simply can't see myself moving past FPC 2.6.4 at this point. FPC 3.x just 
doesn't make any sense.

Regards,
  Graeme

-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Juha Manninen via Lazarus]

On Thu, May 4, 2017 at 8:53 PM, Graeme Geldenhuys via Lazarus
 wrote:
> You made me curious, so I want to take a look. Hopefully it doesn’t
> depend too heavily on the rest of LCL, so I’ll be able to use it in
> other projects of mine.

It has no dependency for LCL, it is part of LazUtils package just like
the other Unicode stuff.
It can be used also with Delphi by copying 2 unit files.
It is briefly explained here:
 
http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#CodePoint_functions_for_encoding_agnostic_code

Juha
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-04 15:56, Juha Manninen via Lazarus wrote:
> I have seen comments saying that treating UTF-16 as fixed width 
> encoding is OK because the characters outside BMP are so rare. It is 
> like saying that a buggy spreadsheet app is OK because it calculates 
> the sums wrong only sometimes. IMO such people should not do
> programming.

+1
I purchased a commercial text editor renowned for having excellent
Unicode support - at least that is what ALL the reviews said. Umm
yeah, to my disappointment it internally uses UTF-16 (because it is
written in Delphi), and treats UTF-16 as 2-byte fixed width! WTF!

 
> I have not seen any feedback or comments about LazUnicode so far. I
> guess it means that nobody uses it. :(

You made me curious, so I want to take a look. Hopefully it doesn’t
depend too heavily on the rest of LCL, so I’ll be able to use it in
other projects of mine.

Regards,
  Graeme


-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[wkitty42--- via Lazarus]

On 05/04/2017 10:56 AM, Juha Manninen via Lazarus wrote:

On Thu, May 4, 2017 at 2:47 PM, wkitty42--- via Lazarus
 wrote:

On 05/03/2017 05:21 AM, Juha Manninen via Lazarus wrote:

Encoding does not matter any more, as long as it is Unicode.


reminds me of a saying that is attributed to Henry Ford...
Any customer can have a car painted any color that he wants so long as it is
black.


Ok, maybe my wording was not good.


i was making a joke... my apologies if it didn't make the great divide... you're 
good! :) :) :)




--
 NOTE: No off-list assistance is given without prior approval.
   *Please keep mailing list traffic on the list* unless
   private contact is specifically requested and granted.
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Juha Manninen via Lazarus]

On Thu, May 4, 2017 at 2:47 PM, wkitty42--- via Lazarus
 wrote:
> On 05/03/2017 05:21 AM, Juha Manninen via Lazarus wrote:
>> Encoding does not matter any more, as long as it is Unicode.
>
> reminds me of a saying that is attributed to Henry Ford...
> Any customer can have a car painted any color that he wants so long as it is
> black.

Ok, maybe my wording was not good.
I believe everybody is happy to get rid of the horrendous Windows
system codepages and the question marks in text.
Unicode is a good thing!
The LazUnicode unit provides a solution for encoding agnostic code.
Yes. It is no joke.

There has been many wars about UTF-8 <> UTF-16 encoding supremacy.
They look ridiculous for anybody who knows the complexity of Unicode.
The complexity is not related to encodings, it is at the abstract
Unicode level where encodings have no effect.

Currently Delphi and Lazarus use a different encoding. Some people see
it as a problem. LazUnicode proves it is not a problem. You can write
code that works with both. Codepoints in both encodings have variable
width, thus they must use the same fundamental concepts. It was easy
to write functions to support them both.
However LazUnicode does not solve the problems at abstract Unicode
level and nobody claims so.

Another fact is that lots of UTF-16 code out there is broken because
people treat UTF-16 as fixed width encoding for some reason. Using
LazUnicode unit improves the situation. The code will inevitably work
right with both encodings.
I have seen comments saying that treating UTF-16 as fixed width
encoding is OK because the characters outside BMP are so rare. It is
like saying that a buggy spreadsheet app is OK because it calculates
the sums wrong only sometimes.
IMO such people should not do programming.

I have not seen any feedback or comments about LazUnicode so far.
I guess it means that nobody uses it. :(

Juha
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[wkitty42--- via Lazarus]

On 05/03/2017 05:21 AM, Juha Manninen via Lazarus wrote:

Encoding does not matter any more, as long as it is Unicode.


reminds me of a saying that is attributed to Henry Ford...


Any customer can have a car painted any color that he wants so long as it is 
black.


:)


--
 NOTE: No off-list assistance is given without prior approval.
   *Please keep mailing list traffic on the list* unless
   private contact is specifically requested and granted.
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Mattias Gaertner via Lazarus]

On Thu, 4 May 2017 09:56:18 +0100
Tony Whyman via Lazarus  wrote:

>[...]
> I don't believe that string indexing even works for UTF8 strings at 
> present - at least not in a simple s[i] way.

It exists the same as for UTF-16 strings.

 
> Is it really that much overhead to have a simple codepage check before 
> calling the correct function to index a string? The obvious optimisation 
> would be to check for UTF8, then UTF16 then the Default codepage and 
> then the rest. Or perhaps UTF16 first for Windows. With register level 
> code you are talking about very few actual machine level operations.

The char type does not fit widechar. You would need widechar.

And in most cases the [] are used in loops. The compiler would have
to add checks on each access. It would be faster to convert the string
at the beginning to UnicodeString and back at the end.
A technique that many RTL functions do to support any string type.

 
> To me, a unified string type would have the advantage that:
> 
> - You would only have one managed string type "string" (and hence avoids 
> the confusion that exists today).

You can avoid the confusion by using only one string encoding,
either UTF-8 or UTF-16. The problem is that existing libraries often
support only one.

 
>[...]> - The only time that a programmer has to think about the character 
> encoding is when writing code that interacts directly with an external 
> interface.

That's already possible. With LazUTF8.
The problem is legacy code and sharing code with Delphi.

 
>[...]

Mattias
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Tony Whyman via Lazarus]

On 03/05/17 17:53, Sven Barth via Lazarus wrote:


Am 03.05.2017 14:37 schrieb "Tony Whyman via Lazarus" 
>:
> On the other hand, AnsiString and UnicodeString are still separate 
types. Why? Why should there not be a single unified string type with 
(e.g.) ASCII, UTF8 and UTF-16 (or MS Unicode) being just another code 
page?


Because indexed access to the string data would slow down quite a bit 
as the RTL would need to determine whether the string is a 1-Byte, 
2-Byte, 4-Byte or multi Byte String. Yes the compiler could do 
optimizations for this inside loops, but it would definitely slow down 
-O- code.


Regards,
Sven





I don't believe that string indexing even works for UTF8 strings at 
present - at least not in a simple s[i] way.


Is it really that much overhead to have a simple codepage check before 
calling the correct function to index a string? The obvious optimisation 
would be to check for UTF8, then UTF16 then the Default codepage and 
then the rest. Or perhaps UTF16 first for Windows. With register level 
code you are talking about very few actual machine level operations.


To me, a unified string type would have the advantage that:

- You would only have one managed string type "string" (and hence avoids 
the confusion that exists today).


- You would have standard string byte length and string character length 
functions (which yes, in the latter case, would have to have a codepage 
check as above).


- String indexing could be standardised as always returning the 
character at position 'i' (including UTF8 strings - albeit after having 
to "walk" the string).


- Automatic transliteration on string compare (with code page check of 
course) - and perhaps with the option to specific a non-standard collation.


- Readily portable code.

- The only time that a programmer has to think about the character 
encoding is when writing code that interacts directly with an external 
interface.


How often would that extra lookup be significant compared with the 
benefits that unified string handling would bring? And, there is no 
reason why you could not retain the UnicodeString type for cases where 
you really need to optimise UTF16 handling.


I see the unified string type as a further extension to AnsiString to 
include UTF16 and UCS2 code pages together with appropriate function 
support.


Tony


-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Marcos Douglas B. Santos via Lazarus]

On Wed, May 3, 2017 at 6:13 AM, Ondrej Pokorny via Lazarus
 wrote:
> Not if you need pre-unicode Delphi support :)
>
> (Well, Marcos didn't specify what Delphi version he wants to target but he
> stated "If Delphi sources don't use UTF8 [...]", which applies to
> pre-unicode Delphi versions.)

Yeah, sorry. I said that because I didn't know which encoding Delphi
is using nowadays.
But I would like to use only Delphi with Unicode support with
"dot.unit.support".

Marcos Douglas
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-03 20:47, Juha Manninen via Lazarus wrote:
> If you share and edit the sources between Delphi and Lazarus then you
> cannot use the full Unicode. 


Quite comical considering that the FPC team always makes such a big fuss
about "we want Delphi compatibility", and now it seems to be worse than
ever before.

Regards,
  Graeme

-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Sven Barth via Lazarus]

On 03.05.2017 21:47, Juha Manninen via Lazarus wrote:
> Why Delphi adds the BOM? Why can't it just read the file and
> understand it is UTF-8?

Probably for the same reason as FPC: the default code page if no BOM is
available and no command line option is set and no $codepage directive
is found is ISO-8859-1.
A BOM does the same as the command line option -FcUTF8 or the directive
{$codepage utf8}, namely switching the source codepage to UTF-8. Only
then string constants that contain UTF-8 characters are converted to
UnicodeString constants.

Regards,
Sven
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Ondrej Pokorny via Lazarus]

On 03.05.2017 21:47, Juha Manninen via Lazarus wrote:

How many people are editing their sources in both Delphi and Lazarus?


Me, but I keep the files ASCII-only because I need to target all Delphi 
versions down to D5 :/ My customers really demand it, unfortunately. I'd 
like to kill these dinosaurs, believe me - not the customers but the 
thousand different Delphi versions from ancient eras, during the best 
period even 2 of them in a year :)


I still feel that code should be in English, including comments - and I 
am a patriot, even if it is not popular nowadays :). The utf-8 bom seems 
to be a stupid issue, though.


Ondrej
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Sven Barth via Lazarus]

Am 03.05.2017 14:37 schrieb "Tony Whyman via Lazarus" <
lazarus@lists.lazarus-ide.org>:
> On the other hand, AnsiString and UnicodeString are still separate types.
Why? Why should there not be a single unified string type with (e.g.)
ASCII, UTF8 and UTF-16 (or MS Unicode) being just another code page?

Because indexed access to the string data would slow down quite a bit as
the RTL would need to determine whether the string is a 1-Byte, 2-Byte,
4-Byte or multi Byte String. Yes the compiler could do optimizations for
this inside loops, but it would definitely slow down -O- code.

Regards,
Sven
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Sven Barth via Lazarus]

Am 03.05.2017 11:34 schrieb "Graeme Geldenhuys via Lazarus" <
lazarus@lists.lazarus-ide.org>:
> For example, take a look at ConEmu for Windows.
>   * Tab support built-in
>   * Resizeable console windows

While not point and click you can resize console windows (and the window
buffer) without problems. And it's even remembered *per shortcut*!

>   * User defined encoding per console window

While not selectable upon start (I think) it can easily be changed with a
command.

>   * Font choice

Supported since at least XP

>   * better mouse & clipboard support

If you use the PowerShell window that experience is already vastly
improved. Could be that this is the case for the default console window in
Windows 10 as well.

>   * User defined "console engine" per window or tab.
> eg: I can have Bash run in one tab and the standard
>  windows console in another.

No one stops you from running a different shell in a command window (e.g. I
often switch to cmd inside PowerShell when some program give me trouble
with the latter).

>   * color customisation

Already supported since at least Windows 7 (can't currently check older
versions).

> I don't know why anybody would still want to run the standard Windows
> console - it is 20 years behind everybody else.

Because it does its job and its part of the OS without the need to install
some 3rd party application.

Regards,
Sven
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Martok via Lazarus]

Am 03.05.2017 um 11:03 schrieb Juha Manninen via Lazarus:
> How could this thing be communicated so that people understand?
It would probably help if there weren't three different pages about "Unicode
Support" on the wiki, all saying slightly different and conflicting things
(because they talk about different things, but that's really not obvious unless
you already know) and decidedly *not* saying what a user might want to know...

Maybe split the technical internals from a "simpler" user's guide?


Martok

-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Mattias Gaertner via Lazarus]

On Wed, 3 May 2017 13:37:24 +0100
Tony Whyman via Lazarus  wrote:

>[...]
> On the other hand, AnsiString and UnicodeString are still separate 
> types. Why? Why should there not be a single unified string type with 
> (e.g.) ASCII, UTF8 and UTF-16 (or MS Unicode) being just another code page?

Many 8bit string functions work for any 8bit encoding, and so do many
16bit string functions for any 16bit encoding. But almost no string
function works for both.

Mattias
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Tony Whyman via Lazarus]

On 03/05/17 09:52, Graeme Geldenhuys via Lazarus wrote:

[rant]
ps:
   Both FPC and Delphi is in such a messed up state when it comes to
   string and character types. It is the laughing stock of programming
   languages at the moment. At least EMBT is heading in the right
   direction with their Linux Delphi compiler - they completely removed
   AnsiString.

   FPC and Delphi can learn a huge lesson from Java and Qt in how to
   handle string and character types.
[/rant]

Regards,
   Graeme

Is Delphi/FPC string handling that much worse than 'C'?

To me, the great thing about AnsiString is that it provides unified 
handling of UTF8 and legacy codepages in a single managed type by 
including the code page id as a dynamic property of the string.


On the other hand, AnsiString and UnicodeString are still separate 
types. Why? Why should there not be a single unified string type with 
(e.g.) ASCII, UTF8 and UTF-16 (or MS Unicode) being just another code page?

--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Marcos Douglas B. Santos via Lazarus]

On Wed, May 3, 2017 at 5:59 AM, Graeme Geldenhuys via Lazarus
 wrote:
> On 2017-05-03 01:21, Marcos Douglas B. Santos via Lazarus wrote:
>> Sorry about that. I stopped using Delphi at version 7, that uses ANSI.
>> I thought that Delphi nowadays was using UTF16.
>
> They (Delphi) loves to follow Microsoft. Files are stored in UTF-8 (this
> is the norm), but they use UTF-16 internally.
>
> Lazarus stores files in UTF-8 and uses UTF-8 internally.
>
> Some background info
> 
> UTF-16 was the first encoding implementation for Unicode - at a time
> when they thought 2-bytes will be big enough for everything. They were
> wrong. So then they invented UTF-8 to solve the problem. But by that
> time Microsoft already standardised on UTF-16, so Delphi followed suite.
> Linux, FreeBSD etc saw the light and used UTF-8 instead.

Ok, thank you for these informations.

best regards,
Marcos Douglas
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Mattias Gaertner via Lazarus]

On Wed, 3 May 2017 12:03:41 +0300
Juha Manninen via Lazarus  wrote:

>[...]
> Please also look at program LazUnicodeTest in components/lazutils/test/.
> It does advanced Unicode stuff and works in both Delphi and Lazarus.

I tried it:

First it did not compile in Delphi, because of an unsupported inline. I
fix that with an IFDEF FPC.

Then it runs. The output is somewhat hard to interpret as the
Windows console shows many chars as '?' and the writelns do not
explain what it is supposed to show. 
The good news is that it works.

The bad news is, that it only works because Delphi silently altered
the source file and added the BOM.

Back in Lazarus compiling such a file gives the error:
LazUnicodeTest.lpr(28,10) Error: UTF-8 code greater than 65535 found


> It means any code dealing with Unicode can do it.
> 
> How could this thing be communicated so that people understand?
> Why other Lazarus developers don't want to mention it?
> I am puzzled. :(

You are right, that using Unicode with Lazarus only needs a couple of
rules to follow. Sharing code with Delphi adds a few more rules.

Mattias
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Jürgen Hestermann via Lazarus]

Am 2017-05-03 um 11:34 schrieb Graeme Geldenhuys via Lazarus:
> I don't know why anybody would still want to run the standard Windows
> console - it is 20 years behind everybody else.

The reason: It is available on every Windows machine.
The alternatives need to be installed first
so scripts designed for them don't work out of the box.

--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Juha Manninen via Lazarus]

On Wed, May 3, 2017 at 12:13 PM, Ondrej Pokorny via Lazarus
 wrote:
> Not if you need pre-unicode Delphi support :)

Ok, true. IMO such old Delphi versions should not be used any more for new code.
Maintenance tasks only I think.

Fortunately there is again a free Delphi Starter edition. It means
anybody can use the latest version. Things are surely getting better!

Juha
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Ondrej Pokorny via Lazarus]

On 03.05.2017 11:21, Juha Manninen via Lazarus wrote:
Windows already supports Unicode in everything ... except for console 
output! Why is that?


You can start the console with UTF-8 codepage: 
http://stackoverflow.com/questions/14109024/how-to-make-unicode-charset-in-cmd-exe-by-default


Then you have full unicode (utf-8) support.

Ondrej
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-03 10:25, Ondrej Pokorny via Lazarus wrote:
> You can start the console with UTF-8 codepage: 
> http://stackoverflow.com/questions/14109024/how-to-make-unicode-charset-in-cmd-exe-by-default
> 
> Then you have full unicode (utf-8) support.

Or use the much better console alternatives. The Windows platform is
finally catching on to what was been available in X11 environments for
years - multiple choice on consoles.

For example, take a look at ConEmu for Windows.
  * Tab support built-in
  * Resizeable console windows
  * User defined encoding per console window
  * Font choice
  * better mouse & clipboard support
  * User defined "console engine" per window or tab.
eg: I can have Bash run in one tab and the standard
 windows console in another.
  * color customisation
  * transparency support


  https://sourceforge.net/projects/conemu/


I don't know why anybody would still want to run the standard Windows
console - it is 20 years behind everybody else.

Regards,
  Graeme

-- 
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Juha Manninen via Lazarus]

On Wed, May 3, 2017 at 11:52 AM, Graeme Geldenhuys via Lazarus
 wrote:
> At least EMBT is heading in the right direction with their
> Linux Delphi compiler - they completely removed AnsiString.

I must agree with you. I hope it will be removed in (far) future when
nobody uses the old Windows system codepages any more.
Windows already supports Unicode in everything ... except for console
output! Why is that?

Anyway, please let's leave out encoding supremacy issues now.
My point has been that our Unicode solution makes the encoding issues
irrelevent. It is almost compatible at source level despite the
different encodings.
Think how improbable that is, yet it works.
See also the encoding agnostic support provided by LazUnicode.

Encoding does not matter any more, as long as it is Unicode.

Juha
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Ondrej Pokorny via Lazarus]

On 03.05.2017 11:03, Juha Manninen via Lazarus wrote:

I am puzzled why there were so many misleading and confusing replies,
also from knowledgeable Lazarus developers.
Remember, the question was about making sources compatible with Delphi.
The person (Marcos Douglas) did not know details of how strings work
in Delphi and Lazarus.
Now we finally have a system that allows (more or less) compatible
code when using Unicode. Why was it not even mentioned by you guys?

For example:

Ondrej Pokorny wrote:

Speaking from my experience, the only approach (not only the best one but
the only one) is not to use characters above #255.

Nonsense. Full Unicode is supported.


Not if you need pre-unicode Delphi support :)

(Well, Marcos didn't specify what Delphi version he wants to target but 
he stated "If Delphi sources don't use UTF8 [...]", which applies to 
pre-unicode Delphi versions.)


Ondrej
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Juha Manninen via Lazarus]

Marcos Douglas B. Santos wrote:
> I am develop on Windows. What problems do you mean?

Unicode is recommended also on Windows. No worries. You don't need to
use the old system codepages.
People who need them must convert them explicitly because the Unicode
system of Lazarus does not support them directly.

> Sorry about that. I stopped using Delphi at version 7, that uses ANSI.
> I thought that Delphi nowadays was using UTF16.
> I will install Delphi Tokyo Starter and discover these things.

You confuse separate things now.
The encoding of their "String" is now UTF-16.
Source files are saved as UTF-8.

> But if I put theses constants as resourcestrings, it's Ok as Mattias
> told me, right?

I don't think it makes any difference. You can use the full Unicode in
both cases.

---
I am puzzled why there were so many misleading and confusing replies,
also from knowledgeable Lazarus developers.
Remember, the question was about making sources compatible with Delphi.
The person (Marcos Douglas) did not know details of how strings work
in Delphi and Lazarus.
Now we finally have a system that allows (more or less) compatible
code when using Unicode. Why was it not even mentioned by you guys?

For example:

Ondrej Pokorny wrote:
> Speaking from my experience, the only approach (not only the best one but
> the only one) is not to use characters above #255.

Nonsense. Full Unicode is supported.

Mattias Gaertner wrote:
> Option a) You can use English in sources and load all non ASCII
> constants via resourcestrings or similar. Then the codepage is
> irrelevant.
> Option b) You can store all files as UTF-8 with BOM. Then FPC will
> store all non ASCII string constants as unicodestrings. Be careful when
> using PChar with them. This adds implicit conversions, so it might be
> slower.

That must be very confusing. Why didn't you just tell him to use the
default Unicode support in Lazarus which allows to write Delphi
compatible code, just by remembering couple of rules.

Also Tony's advice to use AnsiString explicitly is quite irresponsible
for a person who looks for Delphi compatibility.
AnsiString is not Delphi compatible any more in our system and it
brings you back to stone-age in Delphi, to the horrors of system
codepages.

Is it possible that people still don't know how Delphi compatible the
Lazarus Unicode system is (unless you need the old system codepages
obviously)?
For example Lazarus developer Werner (wp) didn't know the Ansi...()
string functions, like AnsiCompareStr(), are compatible with Delphi.
 
http://forum.lazarus.freepascal.org/index.php/topic,36664.msg244619.html#msg244619
Yes they are!

Please also look at program LazUnicodeTest in components/lazutils/test/.
It does advanced Unicode stuff and works in both Delphi and Lazarus.
It means any code dealing with Unicode can do it.

How could this thing be communicated so that people understand?
Why other Lazarus developers don't want to mention it?
I am puzzled. :(

Juha
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Graeme Geldenhuys via Lazarus]

On 2017-05-02 09:30, Juha Manninen via Lazarus wrote:
> From FPC's point of view our UTF-8 solution is a hack. 

FPC's point of view or Marco's point of view? Just curious - so what is
FPC’s “correct” solution then for using UTF-8 and the preferred
encoding? What's the alternative they offer?


[rant]
ps:
  Both FPC and Delphi is in such a messed up state when it comes to
  string and character types. It is the laughing stock of programming
  languages at the moment. At least EMBT is heading in the right
  direction with their Linux Delphi compiler - they completely removed
  AnsiString.

  FPC and Delphi can learn a huge lesson from Java and Qt in how to
  handle string and character types.
[/rant]

Regards,
  Graeme

-- 
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Marcos Douglas B. Santos via Lazarus]

On Tue, May 2, 2017 at 5:30 AM, Juha Manninen via Lazarus
 wrote:
>>>  1. Assign a constant always to a type String variable.
>>
>> What do you mean? Instead of create a constant, is it better create a
>> String variable and assign the string to it?
>
> From FPC's point of view our UTF-8 solution is a hack. In practice it
> means that success in assigning string literals depends on the string
> type.
> This:
>   S := 'Have  for FPC  Lazarus';
> always works if "S" is a "String". It may not work with other string types.
> It is all explained in the wiki page.

I understood, thanks.

> When all your string data is Unicode then you can code in a Delphi
> compatible way.
> Only the Windows system codepages impose a problem, but I got an
> impression you don't need them now.

I am develop on Windows. What problems do you mean?


Marcos Douglas
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Marcos Douglas B. Santos via Lazarus]

On Tue, May 2, 2017 at 5:58 AM, Juha Manninen via Lazarus
 wrote:
> On Sun, Apr 30, 2017 at 7:37 PM, Marcos Douglas B. Santos via Lazarus
>  wrote:
>> If Delphi sources don't use UTF8, how is the best way to mantain sources
>> that need to work in both compilers?
>
> I wonder if I have misunderstood something about your questions.
> What means "Delphi sources don't use UTF8"?
> AFAIK they do use UTF8.

Sorry about that. I stopped using Delphi at version 7, that uses ANSI.
I thought that Delphi nowadays was using UTF16.
I will install Delphi Tokyo Starter and discover these things.

> One more clarification about assigning string data:
> The potential problem is only with string literals, constants.
> Assignment between variables always goes right thanks to their dynamic
> encoding in FPC 3+.

But if I put theses constants as resourcestrings, it's Ok as Mattias
told me, right?

Thank you for these tips.

Best regards,
Marcos Douglas
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Juha Manninen via Lazarus]

On Tue, May 2, 2017 at 2:30 AM, Marcos Douglas B. Santos via Lazarus
 wrote:
> So, as Mattias said, we should code using ANSI chars and everything will be 
> Ok.

No, you can use all the Unicode freely.
The source files are saved as UTF-8 by default. Delphi does the same,
this detail is also compatible.


>> For Delphi compatible generics you can use FPC trunk and the Generics
>> Collection lib made by Maciej.
>
> Is it part of FPC? If not, could you can post the official URL?

It is part of FPC trunk.


>>  1. Assign a constant always to a type String variable.
>
> What do you mean? Instead of create a constant, is it better create a
> String variable and assign the string to it?

From FPC's point of view our UTF-8 solution is a hack. In practice it
means that success in assigning string literals depends on the string
type.
This:
  S := 'Have  for FPC  Lazarus';
always works if "S" is a "String". It may not work with other string types.
It is all explained in the wiki page.

When all your string data is Unicode then you can code in a Delphi
compatible way.
Only the Windows system codepages impose a problem, but I got an
impression you don't need them now.

Juha
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Marcos Douglas B. Santos via Lazarus]

On Mon, May 1, 2017 at 11:06 AM, Juha Manninen via Lazarus
 wrote:
> On Mon, May 1, 2017 at 12:30 PM, Tony Whyman via Lazarus
> ...
>
> No! The good idea is to use "String".

I agree.

>> 5. Take care when using string literals.
>> I added
>> {$IFDEF FPC}
>> {$codepage UTF8}
>> {$ENDIF}
>
> Yes, string literals are tricky but usually you should NOT use {$codepage 
> UTF8}.
> It is explained in the wiki page. I will not repeat it here.

So, as Mattias said, we should code using ANSI chars and everything will be Ok.

>> 7. Generics
>
> For Delphi compatible generics you can use FPC trunk and the Generics
> Collection lib made by Maciej.

Is it part of FPC? If not, could you can post the official URL?


>> I hope you find this a useful checklist.
>
> It contained so much false information that it only confuses people. :(
>
> I want to repeat that it is possible to write code dealing with
> Unicode that is fully compatible with Delphi at source level.
> It will be compatible with a future UTF-16 solution in Lazarus as well.
> Encoding agnostic (UTF-8 / UTF-16) code is possible even if you must
> iterate individual codepoints. See the wiki page for details.

That is I wanted to read. Thanks.

Some doubts:

> Remember these to keep your code compatible:
>  1. Normally use type "String".
>  1. Assign a constant always to a type String variable.

What do you mean? Instead of create a constant, is it better create a
String variable and assign the string to it?

>  2. Use type UnicodeString explicitly for API calls that need it.


Best regards,
Marcos Douglas
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Marcos Douglas B. Santos via Lazarus]

On Mon, May 1, 2017 at 8:47 AM, Mattias Gaertner via Lazarus
 wrote:
> Option a) You can use English in sources and load all non ASCII
> constants via resourcestrings or similar. Then the codepage is
> irrelevant.
> Option b) You can store all files as UTF-8 with BOM. Then FPC will
> store all non ASCII string constants as unicodestrings. Be careful when
> using PChar with them. This adds implicit conversions, so it might be
> slower.
>

Maybe option A could be the best.
I did not remember to use resourcestrings... it is a good tip, thanks.

Marcos Douglas
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Marcos Douglas B. Santos via Lazarus]

On Mon, May 1, 2017 at 6:30 AM, Tony Whyman via Lazarus
 wrote:
> Marcos,
>
> When I originally created the Firebird Pascal API package, I wrote it with
> FPC as the only intended target. However, I then got feedback asking for
> Delphi compatibility and I couldn't resist the challenge of seeing how easy
> it would be to convert the source code. In the end, it was not that
> difficult and the following summarises the changes needed (apart from the
> minor syntax changes):
>
> [...snip...]
>

Tony, thank you for all these tips.

Best regards,
Marcos Douglas
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Juha Manninen via Lazarus]

On Mon, May 1, 2017 at 7:40 PM, Tony Whyman via Lazarus
 wrote:
> I am not sure how much your second post rows back from this but I do think
> that false is a bit harsh.

Yes, sorry, it was correct when using the default types in FPC 3.
However making it compatible with Delphi requires some effort, as you noticed.
My experience is that the Unicode support provided by LazUtils leads
to more Delphi compatible code.
The Windows system codepages are a problem, yes. Then data must be
converted in some I/O layer.

Everybody please experiment with it. It can be used also for non-GUI code.

Juha
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Sven Barth via Lazarus]

Am 01.05.2017 18:50 schrieb "Tony Whyman via Lazarus" <
lazarus@lists.lazarus-ide.org>:
>
>
>
> On 01/05/17 16:33, Sven Barth via Lazarus wrote:
>>
>> Would you please elaborate on these and which FPC version you targeted?
>>
>> Regards,
>> Sven
>>
>>
>>
> I am working with FPC 3.0.x and an example of the solution I used
follows. Note that with FPC, there is no problem calling a constructor from
a placeholder, while with delphi I found it necessary to add an explicit
coercion. I also wanted to return the interface provided by the object.
Again with FPC, this is easy. With Delphi, I found it necessary to use
QueryInterface to do the same.
>
> The difference between the two seems to be that FPC waits until the
generic is instantiated before type checking while Delphi performs type
checking when compiling the generic itself - or thereabouts. IMHO, FPC is
much superior to Delphi in this area.

Your assumption about the way FPC and Delphi do things is indeed correct.

Comments of mine follow inside your example.

> {$IFDEF FPC}
>   TOutputBlockItemGroup<_TItem,_IItem> = class(TOutputBlockItem)
> {$ELSE}
>   TOutputBlockItemGroup<_TItem: TOutputBlockItem; _IItem: IUnknown> =
class(TOutputBlockItem)
> {$ENDIF}

You should be able to use the second variant with FPC as well. It would
allow the compiler to do more type checking when parsing the generic
instead of when specializing it.

>   public
> function GetItem(index: integer): _IItem;
> function Find(ItemType: byte): _IItem;
> property Items[index: integer]: _IItem read getItem; default;
>   end;
>
> 
>
> {$IFDEF FPC}
> { TOutputBlockItemGroup }
>
> function TOutputBlockItemGroup<_TItem,_IItem>.GetItem(index: integer):
_IItem;
> var P: POutputBlockItemData;
> begin
>   P := inherited getItem(index);
>   Result := _TItem.Create(self.Owner,P);
> end;
>
> function TOutputBlockItemGroup<_TItem,_IItem>.Find(ItemType: byte):
_IItem;
> var P: POutputBlockItemData;
> begin
>   P := inherited Find(ItemType);
>   Result := _TItem.Create(self.Owner,P);
> end;
>
>
> {$ELSE}
>
> { TOutputBlockItemGroup }
>
> function TOutputBlockItemGroup<_TItem,_IItem>.GetItem(index: integer):
_IItem;
> var P: POutputBlockItemData;
> Obj: TOutputBlockItem;
> begin
>   P := inherited getItem(index);
>   Obj := TOutputBlockItemClass(_TItem).Create(self.Owner,P);

Is this typecast really needed?

>   if Obj.QueryInterface(GetTypeData(TypeInfo(_IItem))^.Guid,Result) <> 0
then
>
IBError(ibxeInterfaceNotSupported,[GuidToString(GetTypeData(TypeInfo(_IItem))^.Guid)]);
> end;

"Result := Obj;" does not work? What about "Result := Obj as _IItem;"?

> function TOutputBlockItemGroup<_TItem,_IItem>.Find(ItemType: byte):
_IItem;
> var P: POutputBlockItemData;
> Obj: TOutputBlockItem;
> begin
>   P := inherited Find(ItemType);
>   Obj := TOutputBlockItemClass(_TItem).Create(self.Owner,P);
>   if Obj.QueryInterface(GetTypeData(TypeInfo(_IItem))^.Guid,Result) <> 0
then
>
IBError(ibxeInterfaceNotSupported,[GuidToString(GetTypeData(TypeInfo(_IItem))^.Guid)]);
> end;

Regards,
Sven
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Tony Whyman via Lazarus]

On 01/05/17 16:33, Sven Barth via Lazarus wrote:

Would you please elaborate on these and which FPC version you targeted?

Regards,
Sven



I am working with FPC 3.0.x and an example of the solution I used 
follows. Note that with FPC, there is no problem calling a constructor 
from a placeholder, while with delphi I found it necessary to add an 
explicit coercion. I also wanted to return the interface provided by the 
object. Again with FPC, this is easy. With Delphi, I found it necessary 
to use QueryInterface to do the same.


The difference between the two seems to be that FPC waits until the 
generic is instantiated before type checking while Delphi performs type 
checking when compiling the generic itself - or thereabouts. IMHO, FPC 
is much superior to Delphi in this area.


{$IFDEF FPC}
  TOutputBlockItemGroup<_TItem,_IItem> = class(TOutputBlockItem)
{$ELSE}
  TOutputBlockItemGroup<_TItem: TOutputBlockItem; _IItem: IUnknown> = 
class(TOutputBlockItem)

{$ENDIF}
  public
function GetItem(index: integer): _IItem;
function Find(ItemType: byte): _IItem;
property Items[index: integer]: _IItem read getItem; default;
  end;



{$IFDEF FPC}
{ TOutputBlockItemGroup }

function TOutputBlockItemGroup<_TItem,_IItem>.GetItem(index: integer): 
_IItem;

var P: POutputBlockItemData;
begin
  P := inherited getItem(index);
  Result := _TItem.Create(self.Owner,P);
end;

function TOutputBlockItemGroup<_TItem,_IItem>.Find(ItemType: byte): _IItem;
var P: POutputBlockItemData;
begin
  P := inherited Find(ItemType);
  Result := _TItem.Create(self.Owner,P);
end;


{$ELSE}

{ TOutputBlockItemGroup }

function TOutputBlockItemGroup<_TItem,_IItem>.GetItem(index: integer): 
_IItem;

var P: POutputBlockItemData;
Obj: TOutputBlockItem;
begin
  P := inherited getItem(index);
  Obj := TOutputBlockItemClass(_TItem).Create(self.Owner,P);
  if Obj.QueryInterface(GetTypeData(TypeInfo(_IItem))^.Guid,Result) <> 
0 then

IBError(ibxeInterfaceNotSupported,[GuidToString(GetTypeData(TypeInfo(_IItem))^.Guid)]);
end;

function TOutputBlockItemGroup<_TItem,_IItem>.Find(ItemType: byte): _IItem;
var P: POutputBlockItemData;
Obj: TOutputBlockItem;
begin
  P := inherited Find(ItemType);
  Obj := TOutputBlockItemClass(_TItem).Create(self.Owner,P);
  if Obj.QueryInterface(GetTypeData(TypeInfo(_IItem))^.Guid,Result) <> 
0 then

IBError(ibxeInterfaceNotSupported,[GuidToString(GetTypeData(TypeInfo(_IItem))^.Guid)]);
end;


--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Tony Whyman via Lazarus]

On 01/05/17 15:18, Juha Manninen via Lazarus wrote:

On Mon, May 1, 2017 at 12:30 PM, Tony Whyman via Lazarus
 wrote:

When I originally created the Firebird Pascal API package,

Now I realize your code may have been for FPC but not for Lazarus.
Even then the solution provided by LazUtils (2 files there) is good
because it allows compatible and portable code. Later when FPC's
UTF-16 support is ready, such code can be ported easily.

Juha
I assume that you mean that my code is non-visual which is indeed where 
I am coming from. If you want to write an application that is LCL/VCL 
compatible then that is another can of worms.


Your concluding remarks in your other post were:


>>I hope you find this a useful checklist.
It contained so much false information that it only confuses people.

I want to repeat that it is possible to write code dealing with
Unicode that is fully compatible with Delphi at source level.
It will be compatible with a future UTF-16 solution in Lazarus as well.
Encoding agnostic (UTF-8 / UTF-16) code is possible even if you must
iterate individual codepoints. See the wiki page for details.

Remember these to keep your code compatible:
  1. Normally use type "String".
  1. Assign a constant always to a type String variable.
  2. Use type UnicodeString explicitly for API calls that need it.
I am not sure how much your second post rows back from this but I do 
think that false is a bit harsh.


You seem to be coming from a view that strings are strings and the 
compiler should be allowed to work out what is the appropriate string 
encoding for the local environment. All the programmer has to do is 
declare the type as "string" and all will be good. I guess that is your 
definition of portable code: it is agnostic as regards the string encoding.


I am coming from a much messier perspective that says a portable program 
has to deal with whatever string encoding is thrown at it. It may be 
valid criticism to say that I was taking a particularly messy example 
and deriving generic rules from it - but few programs work in a vacuum 
and it is worth being aware of real world problems.


I my case, the real world problem is Firebird. Firebird will expect or 
give you  a string encoded not according to the local environment but 
that which was specified for the database connection and it is the API 
user that decides this and not the API. Ideally, the user specifies 
UTF8, but Firebird supports many other string encodings - but not UTF16 
or Unicode at present. In the original version of the library, the API 
was defined using the "string" type as were the internal structures. 
When I looked at moving to Delphi support, there was no way that this 
would work if "string" suddenly became "UnicodeString". All over the 
place I had assumed that "string" meant "AnsiString" including checking 
and setting the code page in order to match the connection character set 
with whatever code page was being used by the API user.


Could I have written the API without being aware of the character 
encoding? I doubt it. The connection character set is not something that 
the compiler can be aware of. Part of the role of the API library is to 
manage the character encoding on behalf of the user. On the other hand, 
by defining the API using the explicit AnsiString type, it should mean 
that if the API user uses the "string" type, then the compiler can 
automatically transliterate from the API to the API user's string types 
when string means "UnicodeString".


So is my messy example typical or atypical? Am I correct in offering it 
as a source of rules. Ideally, it is atypical. However, I would observe 
that few programs exist in isolation. They have to deal with external 
objects such as files, GUIs and TCP connections. The compiler cannot 
work out the character encoding for itself in these cases and either 
your program or some intermediate library has to be character coding 
aware in order to deal with these objects.


The bottom line is that it would be great if we never needed to be aware 
of the character encoding behind the string type. However, all too often 
you do and, because of that, when you are writing code that is portable 
between platforms and compilers, you either needed to be explicit in the 
string type throughout your program, or at least in the modules that 
deal with external interfaces.


Tony Whyman
--
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Juha Manninen via Lazarus]

On Mon, May 1, 2017 at 12:30 PM, Tony Whyman via Lazarus
 wrote:
> When I originally created the Firebird Pascal API package,

Now I realize your code may have been for FPC but not for Lazarus.
Even then the solution provided by LazUtils (2 files there) is good
because it allows compatible and portable code. Later when FPC's
UTF-16 support is ready, such code can be ported easily.

Juha
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

Re: [Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Tony Whyman via Lazarus]

Marcos,

When I originally created the Firebird Pascal API package, I wrote it 
with FPC as the only intended target. However, I then got feedback 
asking for Delphi compatibility and I couldn't resist the challenge of 
seeing how easy it would be to convert the source code. In the end, it 
was not that difficult and the following summarises the changes needed 
(apart from the minor syntax changes):


1. Standard set of defines for all modules:

{$IFDEF MSWINDOWS}
{$DEFINE WINDOWS}
{$ENDIF}

{$IFDEF FPC}
{$mode delphi}
{$ENDIF}

The first one is needed to ensure that the "WINDOWS" defined symbol 
works with Delphi as well as FPC, while "mode delphi" is needed for FPC.


2. All strings declared explicitly as AnsiString expect where they are 
used with system calls (e.g. FileNames).


This rule may vary between applications. However, as FPC defaults to 
AnsiString and Delphi to WideString, it is usually a good idea to be 
explicit in your string types. In my case, most strings were the ASCII 
subset of UTF8 and AnsiString worked for me.


Similarly AnsiChar instead of Char.

3. PByte instead of PChar.

Most uses of PChar in FPC/Lazarus aren't to character strings but are 
instead used to point to untyped buffers. However, in Delphi, PChar now 
means widestring and you will be caught out if you try and use PChar for 
an untyped buffer. PByte has the same semantic in both FPC and Delphi 
and is a pointer to an unsigned byte.


4. PAnsiChar instead of PChar.

When PChar is used to genuinely point to a character string then as in 
point 2, you should be explicit about which type of string. PWideString 
may also be appropriate for some cases.


5. Take care when using string literals.

I added

{$IFDEF FPC}
{$codepage UTF8}
{$ENDIF}

to sources with string literals just to make sure that FPC interprets 
all strings as UTF8. However, you need to be careful with Delphi. In 
some of the test scripts I ended up with source code such as:


{$IFDEF DCC}
ByName('title').AsString := UTF8Encode('Blob Test ©€');
ByName('Notes').AsString := UTF8Encode('Écoute moi');
{$ELSE}
ByName('title').AsString := 'Blob Test ©€';
ByName('Notes').AsString := 'Écoute moi';
{$ENDIF}

The first case (DCC defined) is Delphi and as my strings were all UTF8, 
I added an explicit UTF8Encode to force the literal to UTF8 before 
assigning to the string. Note the above is a test case with literals 
chosen to show up differences between character sets.


6. Varargs

Delphi is much fussier than FPC about varargs constructs. If you have to 
use them then the only way this seems to work is e.g.


function blah(...): sometype; cdecl varargs;

7. Generics

If you don't have to go here then don't. Unfortunately I had dug a hole 
for myself and made extensive use of generics. Some of the differences 
are minor syntax, but others... Delphi and FPC just don't work the same 
way. Examples of problems are:


- calling a constructor from a function defined using generics.

- extracting an interface from an object defined using generics.

Both are possible, but with very different syntax.

8.TSystemCodePage does not exist in Delphi

You may need to add

{$IF not declared(TSystemCodePage)}
  TSystemCodePage = word; {not defined in Delphi}
{$IFEND}

if you are using any Ansistring code page functions.

9. $IFEND or $ENDIF

As illustrated in point 8, older versions of Delphi demand $IFEND rather 
than $ENDIF to terminate conditional compilation.



I hope you find this a useful checklist.

Regards

Tony Whyman

MWA



On 30/04/17 17:37, Marcos Douglas B. Santos via Lazarus wrote:

It months ago I realized that Lazarus was saving the sources like that:

1- If there is only ANSI chars, save it as ANSI encode;
2. If there is more than ANSI chars, save it as UTF8 encode;

(correct me if I'm wrong about that)

I think this was changed when I updated my FPC to 3.* and Lazarus (trunk).
Everything continues works. This new behavior was completly 
transparent to me... but I have a doubt:


If Delphi sources don't use UTF8, how is the best way to mantain 
sources that need to work in both compilers?


I mean, I would like to work with FPC and Lazarus as I am used to 
doing, without take concerns about "remember to change the encode of 
these files", "remember to not put accented chars on code", or 
something like that.


Is there some wiki page that explains this?
Are you working on projects that have these problems? Could you tell 
me which is the best approach to deal with it?

Thank you.

Best regards,
Marcos Douglas





-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus

[Lazarus] Making sources compatible with Delphi (but Lazarus is priority)

[Marcos Douglas B. Santos via Lazarus]

It months ago I realized that Lazarus was saving the sources like that:

1- If there is only ANSI chars, save it as ANSI encode;
2. If there is more than ANSI chars, save it as UTF8 encode;

(correct me if I'm wrong about that)

I think this was changed when I updated my FPC to 3.* and Lazarus (trunk).
Everything continues works. This new behavior was completly transparent to
me... but I have a doubt:

If Delphi sources don't use UTF8, how is the best way to mantain sources
that need to work in both compilers?

I mean, I would like to work with FPC and Lazarus as I am used to doing,
without take concerns about "remember to change the encode of these files",
"remember to not put accented chars on code", or something like that.

Is there some wiki page that explains this?
Are you working on projects that have these problems? Could you tell me
which is the best approach to deal with it?
Thank you.

Best regards,
Marcos Douglas
-- 
___
Lazarus mailing list
Lazarus@lists.lazarus-ide.org
http://lists.lazarus-ide.org/listinfo/lazarus