On Wed, 14 Sep 2005 16:50:26 +0530, Sastry <[EMAIL PROTECTED]> wrote
> Hi Sadahiro
>
> On 9/12/05, SADAHIRO Tomoyuki <[EMAIL PROTECTED]> wrote:
> >
> > I attribute the failure in tr/\x{12c}-\x{130}/\xc0-\xc4/; to
> > such an ambiguity of \xc0-\xc4. In this expression the left part
> > \x{12c}-\x{130} parsed before coerces \xc0-\xc4 into Unicode,
> > and results in the failure.
> So this is still a problem on EBCDIC! Is there a way to fix this?
> > #test case B # On ASCII platform, of course successful
> > $c = ($a = "\x89\x8a\x8b\x8c\x8d\x8f\x90\x91") =~ tr/\x{100}\x89-\x91/X/;
> > is($c, 8);
> > is($a, "XXXXXXXX");
> This test fails on EBCDIC. In S_scan_const(), there is a statement below.
> /* Insert oct or hex escaped character.
> * There will always enough room in sv since such
> * escapes will be longer than any UTF-8 sequence
> * they can end up as. */
>
> /* We need to map to chars to ASCII before doing the tests
> to cover EBCDIC
> */
> if (!UNI_IS_INVARIANT(NATIVE_TO_UNI(uv))) {
> if (!has_utf8 && uv > 255) {
>
> on an ASCII , the first if condition is true as uv is 137 and it
> falls in the variant range as uv >\x7F whereas on EBCDIC the if
> condition is false. Can you explain why this behaviour is?
see "else" for this "if." This condition tests whether uv needs
multiple octets in UTF-8/UTF-EBCDIC or only needs a single octet.
"\x89" in Latin-1 corresponds to a double-octet representation
in UTF-8, and true (that needs multiple octets) on ASCII platform.
"\x89" in EBCDIC corresponds to a single-octet representation
in UTF-EBCDIC, and false on EBCDIC platform.
Where "else" runs, there is no difference between ASCII and UTF-8;
or between single-octet EBCDIC and UTF-EBCDIC.
> Also I found that the characters are expanded during runtime in
> S_do_trans_simple_utf8()
If I understand it correctly, expansion of character ranges isn't
performed in do_trans_simple_utf8(). It is performed in scan_const()
for non-Unicode and pmtrans() for Unicode.
> Do you have any suggestion where the problem is?
(1) one way (I think worse)
Perl should treat the range in the native order (not in Unicode one)
through the parse time, the compile time, and the run time.
using uvchr_to_utf8() instead of uvuni_to_utf8(),
utf8n_to_uvchr() instead of utf8n_to_uvuni(),
in op.c#pmtrans and doop.c#do_trans_simple_utf8 etc.
But swash_fetch() also needs change (the current swash does not
know EBCDIC, only Unicode); changes of swash may lead to
corruption of lc(), uc(), regular expression \p{something} etc.
(2) another way (I think better)
No change of swash, pmtrans, do_trans_****.
Then all character ranges within 0..255 (not only for non-Unicode
but also for Unicode) to be expanded in scan_const().
(and pmtrans() will expand only uv >= 256).
I think this way requires only the change of toke.c#scan_const
and influences only tr///.
But the change will be quite big, since the current scan_const()
only expands non-Unicode and assumes a single octet encoding.
The range 0..255 in UTF-8/UTF-EBCDIC includes double-octet characters.
I'm not sure whether such a change should be enclosed
with #ifdef EBCDIC and #endif
Regards,
SADAHIRO Tomoyuki
