[fpc-devel] XML and encoding problems
Hello, I am developing an application in Lazarus that is multi platform. It has very strong usage of xml that contains contains cyrillic text (Bulgarian language). All xml feeds are UTF-8 encoding and is converted internaly using iconv (UTF-8 - CP1251 and vice versa) when needed. On Linux this is working very well, but on Windows it is not. I have iconv.pas and link to iconv.dll but the problem is that when a xml document is read from file / stream after that the contents of the document are not UTF-8 encoded and iconv fails. Attached is a simple program that reads a file and prints xml values. On Linux the result is : [EMAIL PROTECTED]:~/temp/utf$ ./project1 user=Тихомир Руменов Тотев This is correct. Tested with gnome-terminal on native Linux system and with Bitvise Tunelier ssh client for Windows. On Windows the result is : E:\TMP\utfproject1.exe user=N???N? ?аN╡?? N??╡?? This is not correct. I tried debugging this but couldn't find the problem. Compiled with Lazarus 0.9.14 / FPC 2.0.2. Please give some help / hints / workaround. This is very cruical for me ATM. Thanks. utf.tar.bz2 Description: BZip2 compressed data ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [Fwd: Re: [fpc-devel] Magic numbers used in TryEncodeDate]
Wow! Thanks for that excellent explanation. As I mentioned to a collegue earlier this week. I never knew dates can become such a complex thing! I will use your email to document those functions, otherwise someone else might feel like I did before reading your explanation of the magic numbers. Now to add another problem. What happens with dates like 1752-09 which only has 20 days. Shouldn't this affect the DeltaDate? Should Encode/DecodeDate know that for that month, 1752-09, there never existed the days 3rd thru 13th. The correct days for September was 1, 2, 14, 15, ... or should that be the worries of the Calendar Component displaying that month and not Encode/DecodeDate's problem? Regards, - Graeme - On 22/04/06, Bram Kuijvenhoven [EMAIL PROTECTED] wrote: I posted this earlier today, but haven't seen it on the list yet. In fact, I have seen no messages at all at the list. Anyway, you have it now :) Original Message Subject: Re: [fpc-devel] Magic numbers used in TryEncodeDate Date: Fri, 21 Apr 2006 13:47:46 +0200 From: Bram Kuijvenhoven [EMAIL PROTECTED] To: FPC developers' list fpc-devel@lists.freepascal.org References: [EMAIL PROTECTED] Hi, Here is the explanation I promised: Graeme Geldenhuys wrote: What is the meaning of these magic numbers used in the TryEncodeDate method? I would like to document them for FPC once I know the answer. 146097 1461 153 693900 I saw the constant DateDelta, but that is never used in TryEncodeDate (whereas in Delphi it is). [...] Function TryEncodeDate(Year,Month,Day : Word; Var Date : TDateTime) : Boolean; var c, ya: cardinal; begin Result:=(Year0) and (Year1) and (Month in [1..12]) and (Day0) and (Day=MonthDays[IsleapYear(Year),Month]); If Result then begin if month 2 then Dec(Month,3) else begin Inc(Month,9); Dec(Year); end; c:= Year DIV 100; ya:= Year - 100*c; Date := (146097*c) SHR 2 + (1461*ya) SHR 2 + (153*cardinal(Month)+2) DIV 5 + cardinal(Day) - 693900; end end; Given are Year, Month and Day. We intend to convert this to the number of days since a certain data, such as 1/1/1, 1/1/1900 or 1/1/1970, or 30/12/1899 for TDateTime. Obviously, if we have some formula that enumerates the days, regardless of the starting date, we can always add or subtract a 'delta' to change the starting date. Intuitively we might think of a formula that looks like: 365 * Year + 30 * Month + Day Of course this formula is terribly wrong, and that is (mainly) because of two reasons: 1) not every year has 365 days in it 2) not every month has 30 days in it Let us first deal with 2). As noted, not every month has 30 days, but even worse, some have 30, some 31 and one has 28/29: February. We are looking for a more accurate formula than the 30 * Month above. It should represent the number of days in the months before Month in the year. A simple trick that will allow us to give a simple formula, and will help us dealing with leap Februaries, is: let a year run from March until February. The following map is taken: December 2005 - 'month' 9 of 'year' 2005 January 2006 - 'month' 10 of 'year' 2005 February 2006 - 'month' 11 of 'year' 2005 March2006 - 'month' 0 of 'year' 2006 April2006 - 'month' 1 of 'year' 2006 ... December 2006 - 'month' 9 of 'year' 2006 January 2007 - 'month' 10 of 'year' 2006 February 2007 - 'month' 11 of 'year' 2006 March2007 - 'month' 0 of 'year' 2007 (I will use 'year' and 'month' instead of year and month to differentiate them.) Now, the number of days in the current 'year' in the 'months' 0 .. Month-1 are given by: March: 0 - 0 April: 1 - 31 (the 31 days of March) May:2 - 61 (the 31 days of March + the 30 days of April) June: 3 - 92 (the 31 days of March + the 30 days of April + the 30 days of May) July: 4 - 122 (... + 30 days of June) August: 5 - 153 (... + 31 days of July) September: 6 - 184 (... + 31 days of August) October:7 - 214 (... + 30 days of September) November: 8 - 245 (... + 31 days of October) December: 9 - 275 (... + 30 days of November) January: 10 - 306 (... + 31 days of December) February: 11 - 337 (... + 31 days of January) This function is almost linear. The trick is to round a linear function, such that the increments are sometimes 30 and sometimes 31. For example take floor( 30.6 * Month + 0.4 ) = ( 153*Month + 2 ) DIV 5 This formula exactly gives the above function! To get the day of the 'year', we thus have the formula floor( 30.6 * Month + 0.4 ) + Day Let us now deal with 1): we need a formula for the number of days in the previous 'years'. Remember that a year, called a leap year, has 366 days instead of 365 when: it is a multiple of 4, but not a multiple of 100, unless it is a multiple
Re: [Fwd: Re: [fpc-devel] Magic numbers used in TryEncodeDate]
On 22/04/06, Bram Kuijvenhoven [EMAIL PROTECTED] wrote: IIRC this is commonly solved by differentiating between th Gregorian and Julian Calendar. I.e. the EncodeDate function etc. work with the Gregorian Calendar. Before that date in 1752 people used the Julian Calendar, but we map all TDateTimes to Gregorian dates. This makes sense. As far as I know, in most Calender components, there is an option to specify which calender system to use (Gregorian, Julian, etc..) and Gregorian being the default. Bottom Line: *Don't* invent a time travel machine! :-) Regards, - Graeme - -- There's no place like 127.0.0.1 ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
[fpc-devel] $Y raises internal error
Hello,
I have a D5 project that uses a {$Y+}. Whenever I include this unit (say
WithY.pas) into the uses clause of another unit (say Test.pas) *and* my
WithY.ppu is updated, fpc raises a:
Fatal: Internal Error 200310221
trying to:
Add Dependency of 'WithY' to 'AnFPCUnit'.
2.0.2 and currenct svn.
HTH
--
Joao Morais
___
fpc-devel maillist - fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel
[fpc-devel] Test, please ignore
___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
[fpc-devel] Web Service Toolkit For FPC Lazarus
Hi. My congratulations to the FPC team and Lazarus team. I'am working on this toolkit, to add Web Service support to FPC and Lazarus. My aim is to merge it with the FCL (LCL). The attached archive contains all the source code and a documentation file ( OpenDocument format, I use OpenOffice 2.0 to write it ). What do you think about it? Overview Web Service Toolkit for FPC Lazarus is a web services package for FPC and Lazarus; Web Service Toolkit is meant to ease web services consumption by FPC and Lazarus users. Web Service Toolkit is made of two parts, a command line tool ws_helper and a collection of support units. Given an interface definition file( describing à web service ), ws_helper will create a FPC unit containing a proxy implementing that interface. At runtime when a call targeting the web service is issued, the proxy's role is to : (a)marshall the call paramaters, (b)make the call to the target web service, (c)receive the call return and unmarshall output parameters to the caller. Behind the scene, the proxy will take care of the SOAP plumbing details. WebServiceToolKit.tar.gz Description: GNU Zip compressed data ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Magic numbers used in TryEncodeDate
Hi, Here is the explanation I promised: Graeme Geldenhuys wrote: What is the meaning of these magic numbers used in the TryEncodeDate method? I would like to document them for FPC once I know the answer. 146097 1461 153 693900 I saw the constant DateDelta, but that is never used in TryEncodeDate (whereas in Delphi it is). [...] Function TryEncodeDate(Year,Month,Day : Word; Var Date : TDateTime) : Boolean; var c, ya: cardinal; begin Result:=(Year0) and (Year1) and (Month in [1..12]) and (Day0) and (Day=MonthDays[IsleapYear(Year),Month]); If Result then begin if month 2 then Dec(Month,3) else begin Inc(Month,9); Dec(Year); end; c:= Year DIV 100; ya:= Year - 100*c; Date := (146097*c) SHR 2 + (1461*ya) SHR 2 + (153*cardinal(Month)+2) DIV 5 + cardinal(Day) - 693900; end end; Given are Year, Month and Day. We intend to convert this to the number of days since a certain data, such as 1/1/1, 1/1/1900 or 1/1/1970, or 30/12/1899 for TDateTime. Obviously, if we have some formula that enumerates the days, regardless of the starting date, we can always add or subtract a 'delta' to change the starting date. Intuitively we might think of a formula that looks like: 365 * Year + 30 * Month + Day Of course this formula is terribly wrong, and that is (mainly) because of two reasons: 1) not every year has 365 days in it 2) not every month has 30 days in it Let us first deal with 2). As noted, not every month has 30 days, but even worse, some have 30, some 31 and one has 28/29: February. We are looking for a more accurate formula than the 30 * Month above. It should represent the number of days in the months before Month in the year. A simple trick that will allow us to give a simple formula, and will help us dealing with leap Februaries, is: let a year run from March until February. The following map is taken: December 2005 - 'month' 9 of 'year' 2005 January 2006 - 'month' 10 of 'year' 2005 February 2006 - 'month' 11 of 'year' 2005 March2006 - 'month' 0 of 'year' 2006 April2006 - 'month' 1 of 'year' 2006 ... December 2006 - 'month' 9 of 'year' 2006 January 2007 - 'month' 10 of 'year' 2006 February 2007 - 'month' 11 of 'year' 2006 March2007 - 'month' 0 of 'year' 2007 (I will use 'year' and 'month' instead of year and month to differentiate them.) Now, the number of days in the current 'year' in the 'months' 0 .. Month-1 are given by: March: 0 - 0 April: 1 - 31 (the 31 days of March) May:2 - 61 (the 31 days of March + the 30 days of April) June: 3 - 92 (the 31 days of March + the 30 days of April + the 30 days of May) July: 4 - 122 (... + 30 days of June) August: 5 - 153 (... + 31 days of July) September: 6 - 184 (... + 31 days of August) October:7 - 214 (... + 30 days of September) November: 8 - 245 (... + 31 days of October) December: 9 - 275 (... + 30 days of November) January: 10 - 306 (... + 31 days of December) February: 11 - 337 (... + 31 days of January) This function is almost linear. The trick is to round a linear function, such that the increments are sometimes 30 and sometimes 31. For example take floor( 30.6 * Month + 0.4 ) = ( 153*Month + 2 ) DIV 5 This formula exactly gives the above function! To get the day of the 'year', we thus have the formula floor( 30.6 * Month + 0.4 ) + Day Let us now deal with 1): we need a formula for the number of days in the previous 'years'. Remember that a year, called a leap year, has 366 days instead of 365 when: it is a multiple of 4, but not a multiple of 100, unless it is a multiple of 400. So 2000 and 2004 are leap years, but 1900 is not. So every period of 400 years has exactly the same number of days, because the leap year rule is periodic with period 400. The number of days is 365*400 + 400/4 - 400/100 + 400/400 = 146097 So every century has about 146097/4 = 36524.25 days in it. Most centuries have 36524 days in it; only every fourth century has 36525 days in it. Similarly, every fourth year (within a century, e.g. 1901..1999) has 366 instead of 365 days in it. Recall that we used 'years' running from March until February the next year. This means that the leap day of a leap year is actually the last day before that year. For example, the 'year' 2000 runs from March 2000 until February 2001, and the leap day of 200 is 29 February 2000, the last day of the 'year' 1999. We now dissect a Year into two pieces: the century (c) and the year-within-the-century (ya). That is, we take Year = 100*c + ya, where 0 = ya 100. (Note: This does not correspond with the usual definition of century, which says e.g. that the 21th century runs from 2001 until 2100.) We now write the number of days in the years before Year = = the number of days in the centuries before c + the number of days in years before ya in the current century For us, a century starts at March c*100 and
Re: [fpc-devel] Magic numbers used in TryEncodeDate
Hi, Here is the explanation I promised: Graeme Geldenhuys wrote: What is the meaning of these magic numbers used in the TryEncodeDate method? I would like to document them for FPC once I know the answer. 146097 1461 153 693900 I saw the constant DateDelta, but that is never used in TryEncodeDate (whereas in Delphi it is). [...] Function TryEncodeDate(Year,Month,Day : Word; Var Date : TDateTime) : Boolean; var c, ya: cardinal; begin Result:=(Year0) and (Year1) and (Month in [1..12]) and (Day0) and (Day=MonthDays[IsleapYear(Year),Month]); If Result then begin if month 2 then Dec(Month,3) else begin Inc(Month,9); Dec(Year); end; c:= Year DIV 100; ya:= Year - 100*c; Date := (146097*c) SHR 2 + (1461*ya) SHR 2 + (153*cardinal(Month)+2) DIV 5 + cardinal(Day) - 693900; end end; Given are Year, Month and Day. We intend to convert this to the number of days since a certain data, such as 1/1/1, 1/1/1900 or 1/1/1970, or 30/12/1899 for TDateTime. Obviously, if we have some formula that enumerates the days, regardless of the starting date, we can always add or subtract a 'delta' to change the starting date. Intuitively we might think of a formula that looks like: 365 * Year + 30 * Month + Day Of course this formula is terribly wrong, and that is (mainly) because of two reasons: 1) not every year has 365 days in it 2) not every month has 30 days in it Let us first deal with 2). As noted, not every month has 30 days, but even worse, some have 30, some 31 and one has 28/29: February. We are looking for a more accurate formula than the 30 * Month above. It should represent the number of days in the months before Month in the year. A simple trick that will allow us to give a simple formula, and will help us dealing with leap Februaries, is: let a year run from March until February. The following map is taken: December 2005 - 'month' 9 of 'year' 2005 January 2006 - 'month' 10 of 'year' 2005 February 2006 - 'month' 11 of 'year' 2005 March2006 - 'month' 0 of 'year' 2006 April2006 - 'month' 1 of 'year' 2006 ... December 2006 - 'month' 9 of 'year' 2006 January 2007 - 'month' 10 of 'year' 2006 February 2007 - 'month' 11 of 'year' 2006 March2007 - 'month' 0 of 'year' 2007 (I will use 'year' and 'month' instead of year and month to differentiate them.) Now, the number of days in the current 'year' in the 'months' 0 .. Month-1 are given by: March: 0 - 0 April: 1 - 31 (the 31 days of March) May:2 - 61 (the 31 days of March + the 30 days of April) June: 3 - 92 (the 31 days of March + the 30 days of April + the 30 days of May) July: 4 - 122 (... + 30 days of June) August: 5 - 153 (... + 31 days of July) September: 6 - 184 (... + 31 days of August) October:7 - 214 (... + 30 days of September) November: 8 - 245 (... + 31 days of October) December: 9 - 275 (... + 30 days of November) January: 10 - 306 (... + 31 days of December) February: 11 - 337 (... + 31 days of January) This function is almost linear. The trick is to round a linear function, such that the increments are sometimes 30 and sometimes 31. For example take floor( 30.6 * Month + 0.4 ) = ( 153*Month + 2 ) DIV 5 This formula exactly gives the above function! To get the day of the 'year', we thus have the formula floor( 30.6 * Month + 0.4 ) + Day Let us now deal with 1): we need a formula for the number of days in the previous 'years'. Remember that a year, called a leap year, has 366 days instead of 365 when: it is a multiple of 4, but not a multiple of 100, unless it is a multiple of 400. So 2000 and 2004 are leap years, but 1900 is not. So every period of 400 years has exactly the same number of days, because the leap year rule is periodic with period 400. The number of days is 365*400 + 400/4 - 400/100 + 400/400 = 146097 So every century has about 146097/4 = 36524.25 days in it. Most centuries have 36524 days in it; only every fourth century has 36525 days in it. Similarly, every fourth year (within a century, e.g. 1901..1999) has 366 instead of 365 days in it. Recall that we used 'years' running from March until February the next year. This means that the leap day of a leap year is actually the last day before that year. For example, the 'year' 2000 runs from March 2000 until February 2001, and the leap day of 200 is 29 February 2000, the last day of the 'year' 1999. We now dissect a Year into two pieces: the century (c) and the year-within-the-century (ya). That is, we take Year = 100*c + ya, where 0 = ya 100. (Note: This does not correspond with the usual definition of century, which says e.g. that the 21th century runs from 2001 until 2100.) We now write the number of days in the years before Year = = the number of days in the centuries before c + the number of days in years before ya in the current century For us, a
Re: [Fwd: Re: [fpc-devel] Magic numbers used in TryEncodeDate]
Oops, I see that in the posting mess of this thread I replied to the wrong email address -- I sent the below to Graeme's personal email instead of the list. (Note: a few minutes ago I suddenly saw my other messages appear in the list, along with some other messages posted in the past days; in the light of Florian's 'Test, please ignore' message I assume something was up with the mailing list.) Original Message Subject: Re: [Fwd: Re: [fpc-devel] Magic numbers used in TryEncodeDate] Date: Sat, 22 Apr 2006 12:09:46 +0200 From: Bram Kuijvenhoven [EMAIL PROTECTED] To: Graeme Geldenhuys [EMAIL PROTECTED] References: [EMAIL PROTECTED] [EMAIL PROTECTED] (I hope that this email arrives at the mailing list...) Graeme Geldenhuys wrote: Now to add another problem. What happens with dates like 1752-09 which only has 20 days. Shouldn't this affect the DeltaDate? Should Encode/DecodeDate know that for that month, 1752-09, there never existed the days 3rd thru 13th. The correct days for September was 1, 2, 14, 15, ... or should that be the worries of the Calendar Component displaying that month and not Encode/DecodeDate's problem? IIRC this is commonly solved by differentiating between th Gregorian and Julian Calendar. I.e. the EncodeDate function etc. work with the Gregorian Calendar. Before that date in 1752 people used the Julian Calendar, but we map all TDateTimes to Gregorian dates. As far as I know FPC does not have a system like Java, which was designed to work with any calendar. (also think of Chinese, Hebrew, ...) Having such a date system (perhaps in a separate unit) could allow us to have functions mapping things to Julian, Gregorian and mix-type dates (i.e. Julian before 1752/9 and Gregorian after). Note that TDateTime is giving a universal time scale, but that mapping between different calendars needs different routines. Since historical dates, mentioned in historical documents might be Julian-based, it might be useful for some people to be able to deal with more or mixed calendars. But then they might want to be able to deal with BC as well! But I recommend looking these things up somewhere before writing it down in the docs. (i.e. about the precise meaning of the Gregorian vs. the Julian calendar) This is just my IIRC :) Regards, Bram ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [Fwd: Re: [fpc-devel] Magic numbers used in TryEncodeDate]
But I recommend looking these things up somewhere before writing it down in the docs. (i.e. about the precise meaning of the Gregorian vs. the Julian calendar) This is just my IIRC :) There is a very exhaustive calendar faq on the internet. And if you read it, then you will understand that implementing full calendar support will result in an immensely overweight system that is not used 99% of the time. So the historic datetime routines and the normal, 1950+ routines should be kept separate. This is also because the conversion to the Gregorian calendar is dependant on where you live. Some Orthodox countries only converted in the 20th century. Most calendar systems only implement a simplified model of the date system (e.g. what can be easily caught with a few equations), and don't tackle the place-specific issues, and the myriads of exceptions. However for real historic date use, you'll need that. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Magic numbers used in TryEncodeDate]
Not sure if this came through the list before Graeme . -- Forwarded message -- From: Graeme Geldenhuys [EMAIL PROTECTED] Date: 22-Apr-2006 12:31 Subject: Re: [Fwd: Re: [fpc-devel] Magic numbers used in TryEncodeDate] To: FPC developers' list fpc-devel@lists.freepascal.org On 22/04/06, Bram Kuijvenhoven [EMAIL PROTECTED] wrote: IIRC this is commonly solved by differentiating between th Gregorian and Julian Calendar. I.e. the EncodeDate function etc. work with the Gregorian Calendar. Before that date in 1752 people used the Julian Calendar, but we map all TDateTimes to Gregorian dates. This makes sense. As far as I know, in most Calender components, there is an option to specify which calender system to use (Gregorian, Julian, etc..) and Gregorian being the default. Bottom Line: *Don't* invent a time travel machine! :-) Regards, - Graeme - -- There's no place like 127.0.0.1 ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] XML and encoding problems
Alexander Todorov wrote: Hello, I am developing an application in Lazarus that is multi platform. It has very strong usage of xml that contains contains cyrillic text (Bulgarian language). All xml feeds are UTF-8 encoding and is converted internaly using iconv (UTF-8 - CP1251 and vice versa) when needed. On Linux this is working very well, but on Windows it is not. I have iconv.pas and link to iconv.dll but the problem is that when a xml document is read from file / stream after that the contents of the document are not UTF-8 encoded and iconv fails. Attached is a simple program that reads a file and prints xml values. On Linux the result is : [EMAIL PROTECTED]:~/temp/utf$ ./project1 user=Тихомир Руменов Тотев This is correct. Tested with gnome-terminal on native Linux system and with Bitvise Tunelier ssh client for Windows. On Windows the result is : E:\TMP\utfproject1.exe user=N???N? ?аN╡?? N??╡?? This is not correct. I tried debugging this but couldn't find the problem. Compiled with Lazarus 0.9.14 / FPC 2.0.2. Please give some help / hints / workaround. This is very cruical for me ATM. Thanks. Result printed on console are not sufficient.Compare binary chunks, then later check for system locale support. Regards Boguslaw ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
[fpc-devel] Re: Internal error 20060409
The problem is not my change, but a bug in the ARM code generator. The proper workaround for now is to change the longbool parameter to a regular boolean for now in int64.inc (in your local copy). Ok, in this way it works, thanks. I have noticed some other issues too: in some cases seems that fpc is not able to correctly parse inline asm code, eg. inserting something like: mov r0, r1, lsl #2 that returns an internal error 200501051. A last question: what about thumb code? Some months ago I heared about a future implementation in fpc 2.1.1. Someone is still working on it? Francesco ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
