Re: hg: jdk/jdk11: 8202329: [AIX] Fix codepage mappings for IBM-943 and Big5

[Bhaktavatsal R Maram]

Hi,

Sorry for the trouble. It had got compiled in my build environment, might be 
due to encoding is different. Please check if following patch make sense.


diff -r 5cb2d4a8b8d4 test/jdk/sun/nio/cs/TestIBMBugs.java
--- a/test/jdk/sun/nio/cs/TestIBMBugs.java  Tue Jun 12 13:00:50 2018 +0530
+++ b/test/jdk/sun/nio/cs/TestIBMBugs.java  Fri Jul 06 10:24:56 2018 +0530
@@ -174,7 +174,7 @@
 }
 
 private static void bug8202329() throws Exception {
-String original = "\\\u007E\u00A5\u203E"; // "b"; \\ 
[backslash][tilde][yen][overscore]
+String original = "\\\u007E\u00A5\u203E"; // 
[backslash][tilde][yen][overscore]
 byte[] expectedBytes; // bytes after conversion
 String expectedStringfromBytes; // String constructed from bytes
 
@@ -187,7 +187,7 @@
 
 
 // Test IBM943, where \ and ~ are encoded to unmappable i.e., 0x3f
-// and ¥ andoverscore are encoded to 0x5c and 0x7e
+// [yen] and [overscore] are encoded to 0x5c and 0x7e
 charset = Charset.forName("IBM943");
 expectedBytes = new byte[] {0x3f, 0x3f, 0x5c, 0x7e};
 expectedStringfromBytes = "??\u00A5\u203E";
@@ -204,7 +204,7 @@
 
 
 // Test IBM943C, where \ and ~ are encoded to 0x5c and 0x7e
-// and ¥ an overscore are encoded to 0x5c and 0x7e
+// [yen] and [overscore] are encoded to 0x5c and 0x7e
 charset = Charset.forName("IBM943C");
 expectedBytes = new byte[] {0x5c, 0x7e, 0x5c, 0x7e};
 expectedStringfromBytes = "\\~\\~";


Thanks,
Bhaktavatsal Reddy
  

-"core-libs-dev"  wrote: -
To: volker.simo...@gmail.com
From: David Holmes 
Sent by: "core-libs-dev" 
Date: 07/06/2018 06:31AM
Cc: core-libs-dev Libs 
Subject: Re: hg: jdk/jdk11: 8202329: [AIX] Fix codepage mappings for IBM-943 
and Big5

Filed: https://bugs.openjdk.java.net/browse/JDK-8206436

David

On 6/07/2018 7:51 AM, David Holmes wrote:
> edit to TestIBMBugs.java contains non-ascii characters in the comments, 
> resulting in javac failure:
> 
> /scratch/opt/mach5/mesos/work_dir/jib-master/install/jdk11-jdk.1010/src.full/open/test/jdk/sun/nio/cs/TestIBMBugs.java:191:
>  
> error: unmappable character (0xC2) for encoding US-ASCII
>  // and \ufffd\ufffd andoverscore are encoded to 0x5c and 0x7e
> ^
> /scratch/opt/mach5/mesos/work_dir/jib-master/install/jdk11-jdk.1010/src.full/open/test/jdk/sun/nio/cs/TestIBMBugs.java:191:
>  
> error: unmappable character (0xA5) for encoding US-ASCII
>  // and \ufffd\ufffd andoverscore are encoded to 0x5c and 0x7e
>  ^
> /scratch/opt/mach5/mesos/work_dir/jib-master/install/jdk11-jdk.1010/src.full/open/test/jdk/sun/nio/cs/TestIBMBugs.java:208:
>  
> error: unmappable character (0xC2) for encoding US-ASCII
>  // and \ufffd\ufffd an overscore are encoded to 0x5c and 0x7e
> ^
> /scratch/opt/mach5/mesos/work_dir/jib-master/install/jdk11-jdk.1010/src.full/open/test/jdk/sun/nio/cs/TestIBMBugs.java:208:
>  
> error: unmappable character (0xA5) for encoding US-ASCII
>  // and \ufffd\ufffd an overscore are encoded to 0x5c and 0x7e
>  ^
> 4 errors
> result: Failed. Compilation failed: Compilation failed

Re: RFR(XS): 8202329 [AIX] Fix codepage mappings for IBM-943 and Big5

[Bhaktavatsal R Maram]

Hi Thomas,

I've added testcase along with fix. Please find webrev at 
http://cr.openjdk.java.net/~aleonard/8202329/webrev.01/

Thanks,
Bhaktavatsal Reddy

  

-"core-libs-dev"  wrote: -
To: "Thomas Stüfe" 
From: "Bhaktavatsal R Maram" 
Sent by: "core-libs-dev" 
Date: 06/01/2018 01:34PM
Cc: Java Core Libs 
Subject: Re: RFR(XS): 8202329 [AIX] Fix codepage mappings for IBM-943 and Big5

Hi Thomas,

Sorry, I was on vacation. I will submit webrev with jtreg testcase.

Thanks,
Bhaktavatsal
 

  

-"Thomas Stüfe"  wrote: -
To: Bhaktavatsal R Maram 
From: "Thomas Stüfe" 
Date: 05/23/2018 12:32AM
Cc: Ichiroh Takiguchi , Volker Simonis 
, Java Core Libs 
Subject: Re: RFR(XS): 8202329 [AIX] Fix codepage mappings for IBM-943 and Big5

Hi Bhaktavatsal,

the fix is fine. Reviewed. Thanks a lot for your work!

Could you please (its okay in a future patch) add a regression test
for IBM943 and IBM943C, in the form of a jtreg test? I would like to
test conversions for both code pages (at least for the crucial
tilde/backslash code points).

Additionally, that when started on AIX with Ja_JP.IBM-943, we
correctly default to IBM943C.

As an example, here is an old test I wrote years ago. You won't be
able to use it verbatim, so it is just a suggestion. Feel free to do
it differently.

http://cr.openjdk.java.net/~stuefe/other/IBM943MappingTest.java

If you have not written jtreg tests before:

http://openjdk.java.net/jtreg/

Also, under /test are many many examples.

Best Regards, Thomas

On Mon, May 21, 2018 at 9:47 AM, Bhaktavatsal R Maram
 wrote:
> Hi Thomas,
>
> Is this fix ready to merge?
>
> Thanks,
> Bhaktavatsal
>
>
>
>
> -"Thomas Stüfe"  wrote: -
> To: Ichiroh Takiguchi , Bhaktavatsal R Maram 
> 
> From: "Thomas Stüfe" 
> Date: 05/11/2018 11:49AM
> Cc: Volker Simonis , Java Core Libs 
> 
> Subject: Re: RFR(XS): 8202329 [AIX] Fix codepage mappings for IBM-943 and Big5
>
> Hi,
>
> I'll test and review next week. We also have some in-house tests which
> I'd like to run.
>
> You IBM folks should really apply for authorship so that this
> contribution process gets streamlined. After all, if something breaks
> in this code, you want to be able to fix it, yes? So even if you do
> not contribute much else, more patches may be forthcoming.
>
> Of course I hope these are not your last contributions :)
>
> Best, Thomas
>
>
>
> On Fri, May 11, 2018 at 7:57 AM, Ichiroh Takiguchi
>  wrote:
>> Hi.
>>
>> I tested this fix on AIX.
>>
>> I got following results.
>> $ LANG=Ja_JP ~/jdk/bin/java PrintDefaultCharset
>> Ja_JP   x-IBM943C   IBM-943CIBM-943C
>> $ LANG=Ja_JP.IBM-943 ~/jdk/bin/java PrintDefaultCharset
>> Ja_JP.IBM-943   x-IBM943C   IBM-943CIBM-943C
>> $ LANG=Zh_TW ~/jdk/bin/java PrintDefaultCharset
>> Zh_TW   x-IBM950IBM-950 IBM-950
>> $ LANG=Zh_TW.big5 ~/jdk/bin/java PrintDefaultCharset
>> Zh_TW.big5  x-IBM950IBM-950 IBM-950
>>
>> Also I reviewed source code, it's fine
>>
>> Since this testing requires locale installation for Ja_JP and Zh_TW,
>> so it's not easy to test it...
>> (At least, I think bos.loc.pc.Ja_JP and bos.loc.iso.Zh_TW filesets are
>> required)
>>
>>
>> On 2018-05-02 18:32, Volker Simonis wrote:
>>>
>>> Hi Bhaktavatsal Reddy,
>>>
>>> your change looks good. I can sponsor it.
>>>
>>> Just waiting for a second review...
>>>
>>> Thank you and best regards,
>>> Volker
>>>
>>>
>>> On Mon, Apr 30, 2018 at 11:29 AM, Bhaktavatsal R Maram
>>>  wrote:
>>>>
>>>> Hi All,
>>>>
>>>> Please review the fix.
>>>>
>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8202329
>>>> webrev: http://cr.openjdk.java.net/~aleonard/8202329/webrev.00/
>>>>
>>>> Thanks,
>>>> Bhaktavatsal Reddy
>>>>
>>>> -"core-libs-dev"  wrote:
>>>> -
>>>> To: Volker Simonis 
>>>> From: "Bhaktavatsal R Maram"
>>>> Sent by: "core-libs-dev"
>>>> Date: 04/26/2018 09:31PM
>>>> Cc: Java Core Libs 
>>>> Subject: Re: [AIX] Fix codepage mappings in Java for IBM-943 and Big5
>>>>
>>>> Hi Volker,
>>>>
>>>> Thank you. I will address your review comments and send webrev for
>>>> review.
>>>>
>>>> - Bhaktavatsal Reddy
>>>>
>>>>
>>>>
>>>> -Volker Simonis  wrote: -
>

Re: [AIX] Add charset IBM-964 (default charset for zh_TW.IBM-eucTW) to stdcs

[Bhaktavatsal R Maram]

Thank you for review.

This is not new charset. This codepage already exist in openJDK and current fix 
is to move it to standard charset list (only for AIX) as it is default charset 
for locale zh_TW.IBM-eucTW. This enabling jdk11 on locale zh_TW.IBM-eucTW.

May I request someone to create JIRA bug for this issue and sponsor the fix?

- Bhaktavatsal Reddy.
 
-"core-libs-dev"  wrote: -
To: core-libs-dev@openjdk.java.net
From: Xueming Shen 
Sent by: "core-libs-dev" 
Date: 05/11/2018 09:43PM
Subject: Re: [AIX] Add charset IBM-964 (default charset for zh_TW.IBM-eucTW) to 
stdcs

SimpleEUCEncoder is something leftover when I reimplemented the charsets 
to be
map-based-built-during-build-time. I would recommend to leave them as 
Bhaktavatsal
suggested for now. Ideally any new charsets added to the platform needs 
to be based on
the new model (open/make/data/charetmapping), instead of hard-coding the map
into the source code.

Thanks!
Sherman

On 5/10/18, 8:20 PM, Bhaktavatsal R Maram wrote:
> Hi Ichiroh,
>
> Yes, moving SimpleEUCEncoder.java to sun.nio.cs package would leave build 
> tools untouched. But, I think that for platforms other than AIX, 
> SimpleEUCEncoder.java in java.base module would not add any value.
>
> - Bhaktavatsal
>
> -Ichiroh Takiguchi  wrote: -
> To: Bhaktavatsal R Maram/India/IBM@IBMIN
> From: Ichiroh Takiguchi
> Date: 05/11/2018 06:31AM
> Cc: Java Core Libs
> Subject: Re: [AIX] Add charset IBM-964 (default charset for zh_TW.IBM-eucTW) 
> to stdcs
>
> Hello Bhaktavatsal.
>
> I think we should move
> from
> src/jdk.charsets/share/classes/sun/nio/cs/ext/SimpleEUCEncoder.java
> to
> src/java.base/share/classes/sun/nio/cs/SimpleEUCEncoder.java
> instead of moving to
>
> src/jdk.charsets/share/classes/sun/nio/cs/ext/SimpleEUCEncoder.java.template
>
> Then we don't need to touch following files
> make/jdk/src/classes/build/tools/charsetmapping/Main.java
> make/jdk/src/classes/build/tools/charsetmapping/SRC.java
>
> I think SimpleEUCEncoder.java is not large file,
> so it can be moved to java.base module.
>
> What do you think ?
>
> On 2018-05-10 22:35, Bhaktavatsal R Maram wrote:
>> Hi All,
>>
>> In bug 8201540 (Extend the set of supported charsets in java.base on
>> AIX)[1] we moved default charsets that are available in OpenJDK to
>> java.base module except to IBM964. Charset IBM964 is bit tricky. Its
>> Encoder is extended from sun.nio.cs.ext.SimpleEUCEncoder. So,
>> SimpleEUCEncoder also should be moved along with IBM964 to sun.nio.cs.
>> However, there is another charset IBM33722 which should always in
>> sun.nio.cs.ext whose encoder also extends SimpleEUCEncoder.
>>
>> Hence, the challenge is that sun.nio.cs.ext.IBM33722 should import
>> SimpleEUCEncoder from either sun.nio.cs or sun.nio.cs.ext based on the
>> location of IBM964.
>>
>> On AIX, following should be the package locations
>> sun.nio.cs.IBM964
>> sun.nio.cs.SimpleEUCEncoder
>> sun.nio.cs.ext.IBM33722 imports sun.nio.cs.SimpleEUCEncoder
>>
>> On Linux (and other platforms), following should be the package
>> locations
>> sun.nio.cs.ext.IBM964
>> sun.nio.cs.ext.SimpleEUCEncoder
>> sun.nio.cs.ext.IBM33722 imports sun.nio.cs.ext.SimpleEUCEncoder
>>
>> To achieve this, I've changed all source files involved to templates.
>> And, I also modified build tools to check if IBM964 is moved to stdcs
>> or not and handled import statement of SimpleEUCEncoder in
>> sun.nio.cs.ext.IBM33722 accordingly.
>>
>> Patch can be found here [2]. Please review. After jira bug is opened,
>> I'll host it again with bug ID.
>>
>> With this change, java -version is successful on locale
>> zh_TW.IBM-eucTW.
>>
>> -bash-4.4$ LANG=zh_TW.IBM-eucTW
>> ~/openjdk/jdk11/build/aix-ppc64-normal-server-release/jdk/bin/java
>> -version
>> openjdk version "11-internal" 2018-09-25
>> OpenJDK Runtime Environment (build 11-internal+0-adhoc.bhamaram.jdk11)
>> OpenJDK 64-Bit Server VM (build 11-internal+0-adhoc.bhamaram.jdk11,
>> mixed mode)
>>
>>
>> [1] https://bugs.openjdk.java.net/browse/JDK-8201540
>> [2] http://cr.openjdk.java.net/~aleonard/AIXIBM964/webrev.01/
>>
>> Thanks,
>> Bhaktavatsal Reddy

Re: RFR(XS): 8202329 [AIX] Fix codepage mappings for IBM-943 and Big5

[Bhaktavatsal R Maram]

Hi Thomas,

Sorry, I was on vacation. I will submit webrev with jtreg testcase.

Thanks,
Bhaktavatsal
 

  

-"Thomas Stüfe"  wrote: -
To: Bhaktavatsal R Maram 
From: "Thomas Stüfe" 
Date: 05/23/2018 12:32AM
Cc: Ichiroh Takiguchi , Volker Simonis 
, Java Core Libs 
Subject: Re: RFR(XS): 8202329 [AIX] Fix codepage mappings for IBM-943 and Big5

Hi Bhaktavatsal,

the fix is fine. Reviewed. Thanks a lot for your work!

Could you please (its okay in a future patch) add a regression test
for IBM943 and IBM943C, in the form of a jtreg test? I would like to
test conversions for both code pages (at least for the crucial
tilde/backslash code points).

Additionally, that when started on AIX with Ja_JP.IBM-943, we
correctly default to IBM943C.

As an example, here is an old test I wrote years ago. You won't be
able to use it verbatim, so it is just a suggestion. Feel free to do
it differently.

http://cr.openjdk.java.net/~stuefe/other/IBM943MappingTest.java

If you have not written jtreg tests before:

http://openjdk.java.net/jtreg/

Also, under /test are many many examples.

Best Regards, Thomas

On Mon, May 21, 2018 at 9:47 AM, Bhaktavatsal R Maram
 wrote:
> Hi Thomas,
>
> Is this fix ready to merge?
>
> Thanks,
> Bhaktavatsal
>
>
>
>
> -"Thomas Stüfe"  wrote: -
> To: Ichiroh Takiguchi , Bhaktavatsal R Maram 
> 
> From: "Thomas Stüfe" 
> Date: 05/11/2018 11:49AM
> Cc: Volker Simonis , Java Core Libs 
> 
> Subject: Re: RFR(XS): 8202329 [AIX] Fix codepage mappings for IBM-943 and Big5
>
> Hi,
>
> I'll test and review next week. We also have some in-house tests which
> I'd like to run.
>
> You IBM folks should really apply for authorship so that this
> contribution process gets streamlined. After all, if something breaks
> in this code, you want to be able to fix it, yes? So even if you do
> not contribute much else, more patches may be forthcoming.
>
> Of course I hope these are not your last contributions :)
>
> Best, Thomas
>
>
>
> On Fri, May 11, 2018 at 7:57 AM, Ichiroh Takiguchi
>  wrote:
>> Hi.
>>
>> I tested this fix on AIX.
>>
>> I got following results.
>> $ LANG=Ja_JP ~/jdk/bin/java PrintDefaultCharset
>> Ja_JP   x-IBM943C   IBM-943CIBM-943C
>> $ LANG=Ja_JP.IBM-943 ~/jdk/bin/java PrintDefaultCharset
>> Ja_JP.IBM-943   x-IBM943C   IBM-943CIBM-943C
>> $ LANG=Zh_TW ~/jdk/bin/java PrintDefaultCharset
>> Zh_TW   x-IBM950IBM-950 IBM-950
>> $ LANG=Zh_TW.big5 ~/jdk/bin/java PrintDefaultCharset
>> Zh_TW.big5  x-IBM950IBM-950 IBM-950
>>
>> Also I reviewed source code, it's fine
>>
>> Since this testing requires locale installation for Ja_JP and Zh_TW,
>> so it's not easy to test it...
>> (At least, I think bos.loc.pc.Ja_JP and bos.loc.iso.Zh_TW filesets are
>> required)
>>
>>
>> On 2018-05-02 18:32, Volker Simonis wrote:
>>>
>>> Hi Bhaktavatsal Reddy,
>>>
>>> your change looks good. I can sponsor it.
>>>
>>> Just waiting for a second review...
>>>
>>> Thank you and best regards,
>>> Volker
>>>
>>>
>>> On Mon, Apr 30, 2018 at 11:29 AM, Bhaktavatsal R Maram
>>>  wrote:
>>>>
>>>> Hi All,
>>>>
>>>> Please review the fix.
>>>>
>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8202329
>>>> webrev: http://cr.openjdk.java.net/~aleonard/8202329/webrev.00/
>>>>
>>>> Thanks,
>>>> Bhaktavatsal Reddy
>>>>
>>>> -"core-libs-dev"  wrote:
>>>> -
>>>> To: Volker Simonis 
>>>> From: "Bhaktavatsal R Maram"
>>>> Sent by: "core-libs-dev"
>>>> Date: 04/26/2018 09:31PM
>>>> Cc: Java Core Libs 
>>>> Subject: Re: [AIX] Fix codepage mappings in Java for IBM-943 and Big5
>>>>
>>>> Hi Volker,
>>>>
>>>> Thank you. I will address your review comments and send webrev for
>>>> review.
>>>>
>>>> - Bhaktavatsal Reddy
>>>>
>>>>
>>>>
>>>> -Volker Simonis  wrote: -
>>>> To: Bhaktavatsal R Maram 
>>>> From: Volker Simonis 
>>>> Date: 04/26/2018 09:12PM
>>>> Cc: Java Core Libs 
>>>> Subject: Re: [AIX] Fix codepage mappings in Java for IBM-943 and Big5
>>>>
>>>> Hi Bhaktavatsal Reddy,
>>>>
>>>> I've opened the following issue for this problem:
>>>>
>>>>

Re: RFR(XS): 8202329 [AIX] Fix codepage mappings for IBM-943 and Big5

[Bhaktavatsal R Maram]

Hi Thomas,

Is this fix ready to merge?

Thanks,
Bhaktavatsal
 

  

-"Thomas Stüfe" <thomas.stu...@gmail.com> wrote: -
To: Ichiroh Takiguchi <taki...@linux.vnet.ibm.com>, Bhaktavatsal R Maram 
<bhama...@in.ibm.com>
From: "Thomas Stüfe" <thomas.stu...@gmail.com>
Date: 05/11/2018 11:49AM
Cc: Volker Simonis <volker.simo...@gmail.com>, Java Core Libs 
<core-libs-dev@openjdk.java.net>
Subject: Re: RFR(XS): 8202329 [AIX] Fix codepage mappings for IBM-943 and Big5

Hi,

I'll test and review next week. We also have some in-house tests which
I'd like to run.

You IBM folks should really apply for authorship so that this
contribution process gets streamlined. After all, if something breaks
in this code, you want to be able to fix it, yes? So even if you do
not contribute much else, more patches may be forthcoming.

Of course I hope these are not your last contributions :)

Best, Thomas



On Fri, May 11, 2018 at 7:57 AM, Ichiroh Takiguchi
<taki...@linux.vnet.ibm.com> wrote:
> Hi.
>
> I tested this fix on AIX.
>
> I got following results.
> $ LANG=Ja_JP ~/jdk/bin/java PrintDefaultCharset
> Ja_JP   x-IBM943C   IBM-943CIBM-943C
> $ LANG=Ja_JP.IBM-943 ~/jdk/bin/java PrintDefaultCharset
> Ja_JP.IBM-943   x-IBM943C   IBM-943CIBM-943C
> $ LANG=Zh_TW ~/jdk/bin/java PrintDefaultCharset
> Zh_TW   x-IBM950IBM-950 IBM-950
> $ LANG=Zh_TW.big5 ~/jdk/bin/java PrintDefaultCharset
> Zh_TW.big5  x-IBM950IBM-950 IBM-950
>
> Also I reviewed source code, it's fine
>
> Since this testing requires locale installation for Ja_JP and Zh_TW,
> so it's not easy to test it...
> (At least, I think bos.loc.pc.Ja_JP and bos.loc.iso.Zh_TW filesets are
> required)
>
>
> On 2018-05-02 18:32, Volker Simonis wrote:
>>
>> Hi Bhaktavatsal Reddy,
>>
>> your change looks good. I can sponsor it.
>>
>> Just waiting for a second review...
>>
>> Thank you and best regards,
>> Volker
>>
>>
>> On Mon, Apr 30, 2018 at 11:29 AM, Bhaktavatsal R Maram
>> <bhama...@in.ibm.com> wrote:
>>>
>>> Hi All,
>>>
>>> Please review the fix.
>>>
>>> bug: https://bugs.openjdk.java.net/browse/JDK-8202329
>>> webrev: http://cr.openjdk.java.net/~aleonard/8202329/webrev.00/
>>>
>>> Thanks,
>>> Bhaktavatsal Reddy
>>>
>>> -"core-libs-dev" <core-libs-dev-boun...@openjdk.java.net> wrote:
>>> -
>>> To: Volker Simonis <volker.simo...@gmail.com>
>>> From: "Bhaktavatsal R Maram"
>>> Sent by: "core-libs-dev"
>>> Date: 04/26/2018 09:31PM
>>> Cc: Java Core Libs <core-libs-dev@openjdk.java.net>
>>> Subject: Re: [AIX] Fix codepage mappings in Java for IBM-943 and Big5
>>>
>>> Hi Volker,
>>>
>>> Thank you. I will address your review comments and send webrev for
>>> review.
>>>
>>> - Bhaktavatsal Reddy
>>>
>>>
>>>
>>> -Volker Simonis <volker.simo...@gmail.com> wrote: -
>>> To: Bhaktavatsal R Maram <bhama...@in.ibm.com>
>>> From: Volker Simonis <volker.simo...@gmail.com>
>>> Date: 04/26/2018 09:12PM
>>> Cc: Java Core Libs <core-libs-dev@openjdk.java.net>
>>> Subject: Re: [AIX] Fix codepage mappings in Java for IBM-943 and Big5
>>>
>>> Hi Bhaktavatsal Reddy,
>>>
>>> I've opened the following issue for this problem:
>>>
>>> 8202329: [AIX] Fix codepage mappings for IBM-943 and Big5
>>>
>>> https://bugs.openjdk.java.net/browse/JDK-8202329
>>>
>>> Looking at you fix, can you please replace the "#elif AIX" by "#ifdef
>>> AIX" and the original "#else" by "#ifdef __solaris__". The original
>>> else branch contains Solaris-only code anyway and it is an historical
>>> omission that there are still a lot of places in the code where "not
>>> Linux" implicitly means "Solaris", but that's often wrong.
>>>
>>> Regards,
>>> Volker
>>>
>>>
>>> On Thu, Apr 26, 2018 at 4:02 PM, Bhaktavatsal R Maram
>>> <bhama...@in.ibm.com> wrote:
>>>>
>>>> Oops! Looks like there is problem with attachment (might be because I
>>>> attached .class file as well). I'm pasting the fix and test program here in
>>>> mail.
>>>>
>>>> Test Program:
>>>>
>>>> import java.nio.charset.*;
>>>> class PrintDefaultCharset {
>>&g

Re: RFR(XS): 8202329 [AIX] Fix codepage mappings for IBM-943 and Big5

[Bhaktavatsal R Maram]

Hi Thomas,

Yes, there are many more to come :)

- Bhaktavatsal
  

-"Thomas Stüfe" <thomas.stu...@gmail.com> wrote: -
To: Ichiroh Takiguchi <taki...@linux.vnet.ibm.com>, Bhaktavatsal R Maram 
<bhama...@in.ibm.com>
From: "Thomas Stüfe" <thomas.stu...@gmail.com>
Date: 05/11/2018 11:49AM
Cc: Volker Simonis <volker.simo...@gmail.com>, Java Core Libs 
<core-libs-dev@openjdk.java.net>
Subject: Re: RFR(XS): 8202329 [AIX] Fix codepage mappings for IBM-943 and Big5

Hi,

I'll test and review next week. We also have some in-house tests which
I'd like to run.

You IBM folks should really apply for authorship so that this
contribution process gets streamlined. After all, if something breaks
in this code, you want to be able to fix it, yes? So even if you do
not contribute much else, more patches may be forthcoming.

Of course I hope these are not your last contributions :)

Best, Thomas



On Fri, May 11, 2018 at 7:57 AM, Ichiroh Takiguchi
<taki...@linux.vnet.ibm.com> wrote:
> Hi.
>
> I tested this fix on AIX.
>
> I got following results.
> $ LANG=Ja_JP ~/jdk/bin/java PrintDefaultCharset
> Ja_JP   x-IBM943C   IBM-943CIBM-943C
> $ LANG=Ja_JP.IBM-943 ~/jdk/bin/java PrintDefaultCharset
> Ja_JP.IBM-943   x-IBM943C   IBM-943CIBM-943C
> $ LANG=Zh_TW ~/jdk/bin/java PrintDefaultCharset
> Zh_TW   x-IBM950IBM-950 IBM-950
> $ LANG=Zh_TW.big5 ~/jdk/bin/java PrintDefaultCharset
> Zh_TW.big5  x-IBM950IBM-950 IBM-950
>
> Also I reviewed source code, it's fine
>
> Since this testing requires locale installation for Ja_JP and Zh_TW,
> so it's not easy to test it...
> (At least, I think bos.loc.pc.Ja_JP and bos.loc.iso.Zh_TW filesets are
> required)
>
>
> On 2018-05-02 18:32, Volker Simonis wrote:
>>
>> Hi Bhaktavatsal Reddy,
>>
>> your change looks good. I can sponsor it.
>>
>> Just waiting for a second review...
>>
>> Thank you and best regards,
>> Volker
>>
>>
>> On Mon, Apr 30, 2018 at 11:29 AM, Bhaktavatsal R Maram
>> <bhama...@in.ibm.com> wrote:
>>>
>>> Hi All,
>>>
>>> Please review the fix.
>>>
>>> bug: 
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.openjdk.java.net_browse_JDK-2D8202329=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWPdQMmqg=EObjWXMYLV3tP7TlWXJqeLzR7hJkCsURSqJDb0dSnTc=YRbmsm-yoh4kURT2ZktLR6ayWpHpmxdqdOrEFmmDqT4=
>>> webrev: 
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__cr.openjdk.java.net_-7Ealeonard_8202329_webrev.00_=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWPdQMmqg=EObjWXMYLV3tP7TlWXJqeLzR7hJkCsURSqJDb0dSnTc=-9QkhLYlAZtChOxa4KOBBrycxbhYS8BfxdMahwSwjOw=
>>>
>>> Thanks,
>>> Bhaktavatsal Reddy
>>>
>>> -"core-libs-dev" <core-libs-dev-boun...@openjdk.java.net> wrote:
>>> -
>>> To: Volker Simonis <volker.simo...@gmail.com>
>>> From: "Bhaktavatsal R Maram"
>>> Sent by: "core-libs-dev"
>>> Date: 04/26/2018 09:31PM
>>> Cc: Java Core Libs <core-libs-dev@openjdk.java.net>
>>> Subject: Re: [AIX] Fix codepage mappings in Java for IBM-943 and Big5
>>>
>>> Hi Volker,
>>>
>>> Thank you. I will address your review comments and send webrev for
>>> review.
>>>
>>> - Bhaktavatsal Reddy
>>>
>>>
>>>
>>> -Volker Simonis <volker.simo...@gmail.com> wrote: -
>>> To: Bhaktavatsal R Maram <bhama...@in.ibm.com>
>>> From: Volker Simonis <volker.simo...@gmail.com>
>>> Date: 04/26/2018 09:12PM
>>> Cc: Java Core Libs <core-libs-dev@openjdk.java.net>
>>> Subject: Re: [AIX] Fix codepage mappings in Java for IBM-943 and Big5
>>>
>>> Hi Bhaktavatsal Reddy,
>>>
>>> I've opened the following issue for this problem:
>>>
>>> 8202329: [AIX] Fix codepage mappings for IBM-943 and Big5
>>>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.openjdk.java.net_browse_JDK-2D8202329=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWPdQMmqg=iQCg2Acve4LeG-Zymt7gpXuSgJLFbCFHsSVHCETqGt8=3KL9rSzXZgjLGz-ayIEaq94QK5rTY0PlEgewOjarNPE=
>>>
>>> Looking at you fix, can you please replace the "#elif AIX" by "#ifdef
>>> AIX" and the original "#else" by "#ifdef __solaris__". The original
>>> else branch contains Solaris-only code anyway and it is an historical
>>> omission that there are still a lot of places in the code where "not
>>

Re: [AIX] Add charset IBM-964 (default charset for zh_TW.IBM-eucTW) to stdcs

[Bhaktavatsal R Maram]

Hi Ichiroh,

Yes, moving SimpleEUCEncoder.java to sun.nio.cs package would leave build tools 
untouched. But, I think that for platforms other than AIX, 
SimpleEUCEncoder.java in java.base module would not add any value.

- Bhaktavatsal

-Ichiroh Takiguchi <taki...@linux.vnet.ibm.com> wrote: -
To: Bhaktavatsal R Maram/India/IBM@IBMIN
From: Ichiroh Takiguchi <taki...@linux.vnet.ibm.com>
Date: 05/11/2018 06:31AM
Cc: Java Core Libs <core-libs-dev@openjdk.java.net>
Subject: Re: [AIX] Add charset IBM-964 (default charset for zh_TW.IBM-eucTW) to 
stdcs

Hello Bhaktavatsal.

I think we should move
from
   src/jdk.charsets/share/classes/sun/nio/cs/ext/SimpleEUCEncoder.java
to
   src/java.base/share/classes/sun/nio/cs/SimpleEUCEncoder.java
instead of moving to
   
src/jdk.charsets/share/classes/sun/nio/cs/ext/SimpleEUCEncoder.java.template

Then we don't need to touch following files
   make/jdk/src/classes/build/tools/charsetmapping/Main.java
   make/jdk/src/classes/build/tools/charsetmapping/SRC.java

I think SimpleEUCEncoder.java is not large file,
so it can be moved to java.base module.

What do you think ?

On 2018-05-10 22:35, Bhaktavatsal R Maram wrote:
> Hi All,
> 
> In bug 8201540 (Extend the set of supported charsets in java.base on
> AIX)[1] we moved default charsets that are available in OpenJDK to
> java.base module except to IBM964. Charset IBM964 is bit tricky. Its
> Encoder is extended from sun.nio.cs.ext.SimpleEUCEncoder. So,
> SimpleEUCEncoder also should be moved along with IBM964 to sun.nio.cs.
> However, there is another charset IBM33722 which should always in
> sun.nio.cs.ext whose encoder also extends SimpleEUCEncoder.
> 
> Hence, the challenge is that sun.nio.cs.ext.IBM33722 should import
> SimpleEUCEncoder from either sun.nio.cs or sun.nio.cs.ext based on the
> location of IBM964.
> 
> On AIX, following should be the package locations
> sun.nio.cs.IBM964
> sun.nio.cs.SimpleEUCEncoder
> sun.nio.cs.ext.IBM33722 imports sun.nio.cs.SimpleEUCEncoder
> 
> On Linux (and other platforms), following should be the package 
> locations
> sun.nio.cs.ext.IBM964
> sun.nio.cs.ext.SimpleEUCEncoder
> sun.nio.cs.ext.IBM33722 imports sun.nio.cs.ext.SimpleEUCEncoder
> 
> To achieve this, I've changed all source files involved to templates.
> And, I also modified build tools to check if IBM964 is moved to stdcs
> or not and handled import statement of SimpleEUCEncoder in
> sun.nio.cs.ext.IBM33722 accordingly.
> 
> Patch can be found here [2]. Please review. After jira bug is opened,
> I'll host it again with bug ID.
> 
> With this change, java -version is successful on locale 
> zh_TW.IBM-eucTW.
> 
> -bash-4.4$ LANG=zh_TW.IBM-eucTW
> ~/openjdk/jdk11/build/aix-ppc64-normal-server-release/jdk/bin/java
> -version
> openjdk version "11-internal" 2018-09-25
> OpenJDK Runtime Environment (build 11-internal+0-adhoc.bhamaram.jdk11)
> OpenJDK 64-Bit Server VM (build 11-internal+0-adhoc.bhamaram.jdk11, 
> mixed mode)
> 
> 
> [1] https://bugs.openjdk.java.net/browse/JDK-8201540
> [2] http://cr.openjdk.java.net/~aleonard/AIXIBM964/webrev.01/
> 
> Thanks,
> Bhaktavatsal Reddy

[AIX] Add charset IBM-964 (default charset for zh_TW.IBM-eucTW) to stdcs

[Bhaktavatsal R Maram]

Hi All,

In bug 8201540 (Extend the set of supported charsets in java.base on AIX)[1] we 
moved default charsets that are available in OpenJDK to java.base module except 
to IBM964. Charset IBM964 is bit tricky. Its Encoder is extended from 
sun.nio.cs.ext.SimpleEUCEncoder. So, SimpleEUCEncoder also should be moved 
along with IBM964 to sun.nio.cs. However, there is another charset IBM33722 
which should always in sun.nio.cs.ext whose encoder also extends 
SimpleEUCEncoder.

Hence, the challenge is that sun.nio.cs.ext.IBM33722 should import 
SimpleEUCEncoder from either sun.nio.cs or sun.nio.cs.ext based on the location 
of IBM964.

On AIX, following should be the package locations
sun.nio.cs.IBM964
sun.nio.cs.SimpleEUCEncoder
sun.nio.cs.ext.IBM33722 imports sun.nio.cs.SimpleEUCEncoder

On Linux (and other platforms), following should be the package locations
sun.nio.cs.ext.IBM964
sun.nio.cs.ext.SimpleEUCEncoder
sun.nio.cs.ext.IBM33722 imports sun.nio.cs.ext.SimpleEUCEncoder

To achieve this, I've changed all source files involved to templates. And, I 
also modified build tools to check if IBM964 is moved to stdcs or not and 
handled import statement of SimpleEUCEncoder in sun.nio.cs.ext.IBM33722 
accordingly.

Patch can be found here [2]. Please review. After jira bug is opened, I'll host 
it again with bug ID.

With this change, java -version is successful on locale zh_TW.IBM-eucTW.

-bash-4.4$ LANG=zh_TW.IBM-eucTW 
~/openjdk/jdk11/build/aix-ppc64-normal-server-release/jdk/bin/java -version
openjdk version "11-internal" 2018-09-25
OpenJDK Runtime Environment (build 11-internal+0-adhoc.bhamaram.jdk11)
OpenJDK 64-Bit Server VM (build 11-internal+0-adhoc.bhamaram.jdk11, mixed mode)


[1] https://bugs.openjdk.java.net/browse/JDK-8201540
[2] http://cr.openjdk.java.net/~aleonard/AIXIBM964/webrev.01/

Thanks,
Bhaktavatsal Reddy

RFR(XS): 8202329 [AIX] Fix codepage mappings for IBM-943 and Big5

[Bhaktavatsal R Maram]

Hi All,

Please review the fix.

bug: https://bugs.openjdk.java.net/browse/JDK-8202329
webrev: http://cr.openjdk.java.net/~aleonard/8202329/webrev.00/

Thanks,
Bhaktavatsal Reddy  

-"core-libs-dev" <core-libs-dev-boun...@openjdk.java.net> wrote: -
To: Volker Simonis <volker.simo...@gmail.com>
From: "Bhaktavatsal R Maram" 
Sent by: "core-libs-dev" 
Date: 04/26/2018 09:31PM
Cc: Java Core Libs <core-libs-dev@openjdk.java.net>
Subject: Re: [AIX] Fix codepage mappings in Java for IBM-943 and Big5

Hi Volker,

Thank you. I will address your review comments and send webrev for review.

- Bhaktavatsal Reddy 

  

-Volker Simonis <volker.simo...@gmail.com> wrote: -
To: Bhaktavatsal R Maram <bhama...@in.ibm.com>
From: Volker Simonis <volker.simo...@gmail.com>
Date: 04/26/2018 09:12PM
Cc: Java Core Libs <core-libs-dev@openjdk.java.net>
Subject: Re: [AIX] Fix codepage mappings in Java for IBM-943 and Big5

Hi Bhaktavatsal Reddy,

I've opened the following issue for this problem:

8202329: [AIX] Fix codepage mappings for IBM-943 and Big5
https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.openjdk.java.net_browse_JDK-2D8202329=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWPdQMmqg=iQCg2Acve4LeG-Zymt7gpXuSgJLFbCFHsSVHCETqGt8=3KL9rSzXZgjLGz-ayIEaq94QK5rTY0PlEgewOjarNPE=

Looking at you fix, can you please replace the "#elif AIX" by "#ifdef
AIX" and the original "#else" by "#ifdef __solaris__". The original
else branch contains Solaris-only code anyway and it is an historical
omission that there are still a lot of places in the code where "not
Linux" implicitly means "Solaris", but that's often wrong.

Regards,
Volker


On Thu, Apr 26, 2018 at 4:02 PM, Bhaktavatsal R Maram
<bhama...@in.ibm.com> wrote:
> Oops! Looks like there is problem with attachment (might be because I 
> attached .class file as well). I'm pasting the fix and test program here in 
> mail.
>
> Test Program:
>
> import java.nio.charset.*;
> class PrintDefaultCharset {
>  public static void main(String[] args) {
> System.out.println("LANG = "+System.getenv("LANG"));
> System.out.println("Default charset = 
> "+Charset.defaultCharset().name());
> System.out.println("file.encoding = 
> "+System.getProperty("file.encoding"));
> System.out.println("sun.jnu.encoding = 
> "+System.getProperty("sun.jnu.encoding"));
>  }
> }
>
>
> Fix:
>
> diff --git a/src/java.base/unix/native/libjava/java_props_md.c 
> b/src/java.base/unix/native/libjava/java_props_md.c
> --- a/src/java.base/unix/native/libjava/java_props_md.c
> +++ b/src/java.base/unix/native/libjava/java_props_md.c
> @@ -1,5 +1,5 @@
>  /*
> - * Copyright (c) 1998, 2016, Oracle and/or its affiliates. All rights 
> reserved.
> + * Copyright (c) 1998, 2018, Oracle and/or its affiliates. All rights 
> reserved.
>   * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
>   *
>   * This code is free software; you can redistribute it and/or modify it
> @@ -297,6 +297,18 @@
>  if (strcmp(p, "EUC-JP") == 0) {
>  *std_encoding = "EUC-JP-LINUX";
>  }
> +#elif defined _AIX
> +if (strcmp(p, "big5") == 0) {
> +/* On AIX Traditional Chinese Big5 codeset is mapped to IBM-950 
> */
> +*std_encoding = "IBM-950";
> +} else if (strcmp(p, "IBM-943") == 0) {
> +/*
> + * On AIX, IBM-943 is mapped to IBM-943C in which symbol 'yen' 
> and
> + * 'overline' are replaced with 'backslash' and 'tilde' from 
> ASCII
> + * making first 96 code points same as ASCII.
> + */
> +*std_encoding = "IBM-943C";
> +}
>  #else
>  if (strcmp(p,"eucJP") == 0) {
>  /* For Solaris use customized vendor defined character
>
>
> Thanks,
> Bhaktavatsal Reddy
>
>
> -"core-libs-dev" <core-libs-dev-boun...@openjdk.java.net> wrote: -
> To: "Java Core Libs" <core-libs-dev@openjdk.java.net>
> From: "Bhaktavatsal R Maram"
> Sent by: "core-libs-dev"
> Date: 04/26/2018 07:26PM
> Subject: [AIX] Fix codepage mappings in Java for IBM-943 and Big5
>
> Hi All,
>
> This issue is continuation to bug 8201540 (Extend the set of supported 
> charsets in java.base on AIX) in which we have moved default charsets of most 
> of the locales supported by Operating System to java.base module thus 
> enabling OpenJDK on those locales for AIX platform.
>
> As part of t

Re: [AIX] Fix codepage mappings in Java for IBM-943 and Big5

[Bhaktavatsal R Maram]

Hi Volker,

Thank you. I will address your review comments and send webrev for review.

- Bhaktavatsal Reddy 

  

-Volker Simonis <volker.simo...@gmail.com> wrote: -
To: Bhaktavatsal R Maram <bhama...@in.ibm.com>
From: Volker Simonis <volker.simo...@gmail.com>
Date: 04/26/2018 09:12PM
Cc: Java Core Libs <core-libs-dev@openjdk.java.net>
Subject: Re: [AIX] Fix codepage mappings in Java for IBM-943 and Big5

Hi Bhaktavatsal Reddy,

I've opened the following issue for this problem:

8202329: [AIX] Fix codepage mappings for IBM-943 and Big5
https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.openjdk.java.net_browse_JDK-2D8202329=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWPdQMmqg=iQCg2Acve4LeG-Zymt7gpXuSgJLFbCFHsSVHCETqGt8=3KL9rSzXZgjLGz-ayIEaq94QK5rTY0PlEgewOjarNPE=

Looking at you fix, can you please replace the "#elif AIX" by "#ifdef
AIX" and the original "#else" by "#ifdef __solaris__". The original
else branch contains Solaris-only code anyway and it is an historical
omission that there are still a lot of places in the code where "not
Linux" implicitly means "Solaris", but that's often wrong.

Regards,
Volker


On Thu, Apr 26, 2018 at 4:02 PM, Bhaktavatsal R Maram
<bhama...@in.ibm.com> wrote:
> Oops! Looks like there is problem with attachment (might be because I 
> attached .class file as well). I'm pasting the fix and test program here in 
> mail.
>
> Test Program:
>
> import java.nio.charset.*;
> class PrintDefaultCharset {
>  public static void main(String[] args) {
> System.out.println("LANG = "+System.getenv("LANG"));
> System.out.println("Default charset = 
> "+Charset.defaultCharset().name());
> System.out.println("file.encoding = 
> "+System.getProperty("file.encoding"));
> System.out.println("sun.jnu.encoding = 
> "+System.getProperty("sun.jnu.encoding"));
>  }
> }
>
>
> Fix:
>
> diff --git a/src/java.base/unix/native/libjava/java_props_md.c 
> b/src/java.base/unix/native/libjava/java_props_md.c
> --- a/src/java.base/unix/native/libjava/java_props_md.c
> +++ b/src/java.base/unix/native/libjava/java_props_md.c
> @@ -1,5 +1,5 @@
>  /*
> - * Copyright (c) 1998, 2016, Oracle and/or its affiliates. All rights 
> reserved.
> + * Copyright (c) 1998, 2018, Oracle and/or its affiliates. All rights 
> reserved.
>   * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
>   *
>   * This code is free software; you can redistribute it and/or modify it
> @@ -297,6 +297,18 @@
>  if (strcmp(p, "EUC-JP") == 0) {
>  *std_encoding = "EUC-JP-LINUX";
>  }
> +#elif defined _AIX
> +if (strcmp(p, "big5") == 0) {
> +/* On AIX Traditional Chinese Big5 codeset is mapped to IBM-950 
> */
> +*std_encoding = "IBM-950";
> +} else if (strcmp(p, "IBM-943") == 0) {
> +/*
> + * On AIX, IBM-943 is mapped to IBM-943C in which symbol 'yen' 
> and
> + * 'overline' are replaced with 'backslash' and 'tilde' from 
> ASCII
> + * making first 96 code points same as ASCII.
> +     */
> +    *std_encoding = "IBM-943C";
> +}
>  #else
>  if (strcmp(p,"eucJP") == 0) {
>  /* For Solaris use customized vendor defined character
>
>
> Thanks,
> Bhaktavatsal Reddy
>
>
> -"core-libs-dev" <core-libs-dev-boun...@openjdk.java.net> wrote: -
> To: "Java Core Libs" <core-libs-dev@openjdk.java.net>
> From: "Bhaktavatsal R Maram"
> Sent by: "core-libs-dev"
> Date: 04/26/2018 07:26PM
> Subject: [AIX] Fix codepage mappings in Java for IBM-943 and Big5
>
> Hi All,
>
> This issue is continuation to bug 8201540 (Extend the set of supported 
> charsets in java.base on AIX) in which we have moved default charsets of most 
> of the locales supported by Operating System to java.base module thus 
> enabling OpenJDK on those locales for AIX platform.
>
> As part of that, charsets for locales Ja_JP (IBM-943) and Zh_TW (big5) also 
> have been moved. However, corresponding charsets mapped in Java is not 
> correct for them on AIX. Following are the details:
>
> 1. IBM-943 [1] for locale Ja_JP should be mapped to IBM-943C [2]
>
> Fundamental difference between IBM-943 and IBM-943C is that IBM-943C is ASCII 
> compatible which means code points 'yen' and 'overline' of IBM-943 is 
> replaced with 'backslash' and 'tilde' from ASCII character set.
>
>
> 2. Big5 for locale Zh_TW should b

RE: RFR : 8202322: AIX: symbol visibility flags not support on xlc 12.1

[Bhaktavatsal R Maram]

Hi Matthias,

At this point, I think we can remove the flag as you found that it is not 
supported in XLC < 13. And with XLC 13, it require more work to use this flag.

Thanks,
Bhaktavatsal Reddy

  

-"Baesken, Matthias" <matthias.baes...@sap.com> wrote: -
To: "Langer, Christoph" <christoph.lan...@sap.com>, 
"'build-...@openjdk.java.net'" <build-...@openjdk.java.net>, 
"ppc-aix-port-...@openjdk.java.net" <ppc-aix-port-...@openjdk.java.net>, 
"core-libs-dev@openjdk.java.net" <core-libs-dev@openjdk.java.net>
From: "Baesken, Matthias" <matthias.baes...@sap.com>
Date: 04/26/2018 08:23PM
Cc: "Simonis, Volker" <volker.simo...@sap.com>, Bhaktavatsal R Maram
<bhama...@in.ibm.com>
Subject: RE: RFR : 8202322: AIX: symbol visibility flags not support on xlc 12.1

   
 Hello Christoph,   I think  all XLC versions  < 12.1   are unsupported  (and 
probably not working anyway)  for the OpenJDK  build .
 I am only aware   of  XLC  versions  12.1  and 13.1which  work / in case 
of 13.1  “might” work   for OpenJDK compilation .
 And for 12.1  I want to remove the flags  .
  
 ( waiting for the feedback  of   Bhaktavatsal Reddy ,  in case he  prefers it  
I remove them for all xlC versions including 13.1 )
  
 Best regards, Matthias
  
  
  
 
 
 
 From: Langer, Christoph 
 Sent: Donnerstag, 26. April 2018 16:38
 To: Baesken, Matthias <matthias.baes...@sap.com>; 'build-...@openjdk.java.net' 
<build-...@openjdk.java.net>; ppc-aix-port-...@openjdk.java.net; 
core-libs-dev@openjdk.java.net
 Cc: Simonis, Volker <volker.simo...@sap.com>
 Subject: RE: RFR : 8202322: AIX: symbol visibility flags not support on xlc 
12.1

 Hi Matthias,
  
 to me the change in principal looks good.
  
 I’m wondering if it is possible to do a comparison like xlc < 13 (e.g. extract 
major number before the first dot, then compare numerically) – but maybe it is 
too complicated and the current single version compare suits  our needs ?
  
 Best regards
 Christoph
  
  
 
 
 From: Baesken, Matthias 
 Sent: Donnerstag, 26. April 2018 16:14
 To: 'build-...@openjdk.java.net' <build-...@openjdk.java.net>; 
ppc-aix-port-...@openjdk.java.net; core-libs-dev@openjdk.java.net
 Cc: Langer, Christoph <christoph.lan...@sap.com>; Simonis, Volker 
<volker.simo...@sap.com>
 Subject: RFR : 8202322: AIX: symbol visibility flags not support on xlc 12.1

 Hello  ,  could you please review this small adjustment to  the symbol 
visibility compilation settings on AIX ?
 Currently  we use  XLC 12.1  to compile  JDK on AIX .
  
 However XLC 12.1   does not support  the “-qvisibility=hidden”  setting 
currently set on AIX.
 It was introduced with  XLC 13.1 . Christoph found some info about it here :
  
 
https://www.ibm.com/developerworks/aix/library/au-aix-symbol-visibility-part2/index.html
  
 Setting it  only generates  hundreds  of warnings  in the build log , warnings 
look like this :
 XlC12.1
  
 bash-4.4$ xlC -qversion
 IBM XL C/C++ for AIX, V12.1 (5765-J02, 5725-C72)
 Version: 12.01..0019
  
 bash-4.4$ xlC -qvisibility=hidden sizeof.c -o sizeof_aixxlc
 1506-173 (W) Option visibility=hidden is not valid. Enter xlC for list of 
valid options.
  
 Compare to XLC13.1
  
 bash-3.00$ xlC -qversion
 IBM XL C/C++ for AIX, V13.1 (5725-C72, 5765-J07)
 Version: 13.01..0008
 bash-3.00$ xlC -qvisibility=default sizeof.c -o sizeof_aixxlc
 bash-3.00$ xlC -qvisibility=hidden sizeof.c -o sizeof_aixxlc
  
  
 So it is better to avoid  setting these flags when using xlc12.1   .
 Please review :
  
 Bug :
  
 https://bugs.openjdk.java.net/browse/JDK-8202322
  
 Change :
  
 http://cr.openjdk.java.net/~mbaesken/webrevs/8202322/
  
  
 Best regards, Matthias
  
  
  

Re: RFR : 8202322: AIX: symbol visibility flags not support on xlc 12.1

[Bhaktavatsal R Maram]

Hi Matthias,

Its right in time. I am facing "Undefined Symbol" errors with XLC 13.1 due to 
"-qvisibility=hidden" which hides all symbols by default. I workaround that by 
removing this flag

-bash-4.4$ xlC -qversion
IBM XL C/C++ for AIX, V13.1.3 (5725-C72, 5765-J07)
Version: 13.01.0003.

Looks like there are many function declarations that require changes to make 
function visible.

Thanks,
Bhaktavatsal Reddy

  

-"core-libs-dev"  wrote: -
To: "'build-...@openjdk.java.net'" , 
"ppc-aix-port-...@openjdk.java.net" , 
"core-libs-dev@openjdk.java.net" 
From: "Baesken, Matthias" 
Sent by: "core-libs-dev" 
Date: 04/26/2018 07:45PM
Cc: "Simonis, Volker" 
Subject: RFR : 8202322: AIX: symbol visibility flags not support on xlc 12.1

Hello  ,  could you please review this small adjustment to  the symbol 
visibility compilation settings on AIX ?
Currently  we use  XLC 12.1  to compile  JDK on AIX .

However XLC 12.1   does not support  the "-qvisibility=hidden"  setting 
currently set on AIX.
It was introduced with  XLC 13.1 . Christoph found some info about it here :

https://www.ibm.com/developerworks/aix/library/au-aix-symbol-visibility-part2/index.html

Setting it  only generates  hundreds  of warnings  in the build log , warnings 
look like this :
XlC12.1

bash-4.4$ xlC -qversion
IBM XL C/C++ for AIX, V12.1 (5765-J02, 5725-C72)
Version: 12.01..0019

bash-4.4$ xlC -qvisibility=hidden sizeof.c -o sizeof_aixxlc
1506-173 (W) Option visibility=hidden is not valid. Enter xlC for list of valid 
options.

Compare to XLC13.1

bash-3.00$ xlC -qversion
IBM XL C/C++ for AIX, V13.1 (5725-C72, 5765-J07)
Version: 13.01..0008
bash-3.00$ xlC -qvisibility=default sizeof.c -o sizeof_aixxlc
bash-3.00$ xlC -qvisibility=hidden sizeof.c -o sizeof_aixxlc


So it is better to avoid  setting these flags when using xlc12.1   .
Please review :

Bug :

https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.openjdk.java.net_browse_JDK-2D8202322=DwIFAg=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWPdQMmqg=F6_aATE7bsRWZG5lPZhJUESHNGqST5MNbkULllIH8l4=M4IJ-YdB3dfN7lj0DEWozQbipDXmgKtSu3pEyMcJF_E=

Change :

https://urldefense.proofpoint.com/v2/url?u=http-3A__cr.openjdk.java.net_-7Embaesken_webrevs_8202322_=DwIFAg=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWPdQMmqg=F6_aATE7bsRWZG5lPZhJUESHNGqST5MNbkULllIH8l4=gb0OP9fQTAWWJbBZDK6L_0gTKf1pwna8aXaYr_uVv8Q=


Best regards, Matthias

Re: [AIX] Fix codepage mappings in Java for IBM-943 and Big5

[Bhaktavatsal R Maram]

Oops! Looks like there is problem with attachment (might be because I attached 
.class file as well). I'm pasting the fix and test program here in mail.

Test Program:

import java.nio.charset.*;
class PrintDefaultCharset {
 public static void main(String[] args) {
System.out.println("LANG = "+System.getenv("LANG"));
System.out.println("Default charset = 
"+Charset.defaultCharset().name());
System.out.println("file.encoding = 
"+System.getProperty("file.encoding"));
System.out.println("sun.jnu.encoding = 
"+System.getProperty("sun.jnu.encoding"));
 }
}


Fix:

diff --git a/src/java.base/unix/native/libjava/java_props_md.c 
b/src/java.base/unix/native/libjava/java_props_md.c
--- a/src/java.base/unix/native/libjava/java_props_md.c
+++ b/src/java.base/unix/native/libjava/java_props_md.c
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 1998, 2016, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2018, Oracle and/or its affiliates. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * This code is free software; you can redistribute it and/or modify it
@@ -297,6 +297,18 @@
 if (strcmp(p, "EUC-JP") == 0) {
 *std_encoding = "EUC-JP-LINUX";
 }
+#elif defined _AIX
+if (strcmp(p, "big5") == 0) {
+/* On AIX Traditional Chinese Big5 codeset is mapped to IBM-950 */
+*std_encoding = "IBM-950";
+} else if (strcmp(p, "IBM-943") == 0) {
+/*
+ * On AIX, IBM-943 is mapped to IBM-943C in which symbol 'yen' and
+ * 'overline' are replaced with 'backslash' and 'tilde' from ASCII
+ * making first 96 code points same as ASCII.
+ */
+*std_encoding = "IBM-943C";
+}
 #else
 if (strcmp(p,"eucJP") == 0) {
 /* For Solaris use customized vendor defined character

 
Thanks,
Bhaktavatsal Reddy
  

-"core-libs-dev" <core-libs-dev-boun...@openjdk.java.net> wrote: -
To: "Java Core Libs" <core-libs-dev@openjdk.java.net>
From: "Bhaktavatsal R Maram" 
Sent by: "core-libs-dev" 
Date: 04/26/2018 07:26PM
Subject: [AIX] Fix codepage mappings in Java for IBM-943 and Big5

Hi All,

This issue is continuation to bug 8201540 (Extend the set of supported charsets 
in java.base on AIX) in which we have moved default charsets of most of the 
locales supported by Operating System to java.base module thus enabling OpenJDK 
on those locales for AIX platform. 

As part of that, charsets for locales Ja_JP (IBM-943) and Zh_TW (big5) also 
have been moved. However, corresponding charsets mapped in Java is not correct 
for them on AIX. Following are the details:

1. IBM-943 [1] for locale Ja_JP should be mapped to IBM-943C [2]

Fundamental difference between IBM-943 and IBM-943C is that IBM-943C is ASCII 
compatible which means code points 'yen' and 'overline' of IBM-943 is replaced 
with 'backslash' and 'tilde' from ASCII character set.


2. Big5 for locale Zh_TW should be mapped to IBM-950 [3]

I've attached simple test program to print the default charset along with fix 
for this issue. When run test program (PrintDefaultCharset) with IBM JDK 8 (on 
AIX) for locales Ja_JP & Zh_TW, following is output.

-bash-4.4$ LANG=Ja_JP ~/JDKs/IBM/80/ON/sdk/jre/bin/java PrintDefaultCharset 
LANG = Ja_JP
Default charset = x-IBM943C
file.encoding = IBM-943C
sun.jnu.encoding = IBM-943C

-bash-4.4$ LANG=Zh_TW ~/JDKs/IBM/80/ON/sdk/jre/bin/java PrintDefaultCharset
LANG = Zh_TW
Default charset = x-IBM950
file.encoding = IBM-950
sun.jnu.encoding = IBM-950


Same test run with openJDK 11 gives following output

-bash-4.4$ LANG=Ja_JP ~/jdk/bin/java PrintDefaultCharset
LANG = Ja_JP
Default charset = x-IBM943
file.encoding = IBM-943
sun.jnu.encoding = IBM-943

-bash-4.4$ LANG=Zh_TW ~/jdk/bin/java PrintDefaultCharset
LANG = Zh_TW
Default charset = Big5
file.encoding = big5
sun.jnu.encoding = big5

I will get webrev hosted in 
https://urldefense.proofpoint.com/v2/url?u=http-3A__cr.openjdk.java.net=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWPdQMmqg=Prdd2GWj8c4aCa1qPr65xklNyDyu64w_6X7qkNaI-D8=8C1bILMg6JLJtbL0KLRPfU0MkIPkAmq_IlJgdTfpjdI=
 for this change and send it for review once JIRA bug is created.

[1] 
https://urldefense.proofpoint.com/v2/url?u=http-3A__demo.icu-2Dproject.org_icu-2Dbin_convexp-3Fconv-3Dibm-2D943-5FP130-2D1999-26s-3DJAVA=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWPdQMmqg=Prdd2GWj8c4aCa1qPr65xklNyDyu64w_6X7qkNaI-D8=RJOiyJTR1jkgxxnRZu5JL97irAnHo1M4wMp7x21dgvs=
[2] 
https://urldefense.proofpoint.com/v2/url?u=http-3A__demo.icu-2Dproject.org_icu-2Dbin_convexp-3Fconv-3Dibm-2D943-5FP15A-2D2003-26s-3DALL=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWP

[AIX] Fix codepage mappings in Java for IBM-943 and Big5

[Bhaktavatsal R Maram]

Hi All,

This issue is continuation to bug 8201540 (Extend the set of supported charsets 
in java.base on AIX) in which we have moved default charsets of most of the 
locales supported by Operating System to java.base module thus enabling OpenJDK 
on those locales for AIX platform. 

As part of that, charsets for locales Ja_JP (IBM-943) and Zh_TW (big5) also 
have been moved. However, corresponding charsets mapped in Java is not correct 
for them on AIX. Following are the details:

1. IBM-943 [1] for locale Ja_JP should be mapped to IBM-943C [2]

Fundamental difference between IBM-943 and IBM-943C is that IBM-943C is ASCII 
compatible which means code points 'yen' and 'overline' of IBM-943 is replaced 
with 'backslash' and 'tilde' from ASCII character set.


2. Big5 for locale Zh_TW should be mapped to IBM-950 [3]

I've attached simple test program to print the default charset along with fix 
for this issue. When run test program (PrintDefaultCharset) with IBM JDK 8 (on 
AIX) for locales Ja_JP & Zh_TW, following is output.

-bash-4.4$ LANG=Ja_JP ~/JDKs/IBM/80/ON/sdk/jre/bin/java PrintDefaultCharset 
LANG = Ja_JP
Default charset = x-IBM943C
file.encoding = IBM-943C
sun.jnu.encoding = IBM-943C

-bash-4.4$ LANG=Zh_TW ~/JDKs/IBM/80/ON/sdk/jre/bin/java PrintDefaultCharset
LANG = Zh_TW
Default charset = x-IBM950
file.encoding = IBM-950
sun.jnu.encoding = IBM-950


Same test run with openJDK 11 gives following output

-bash-4.4$ LANG=Ja_JP ~/jdk/bin/java PrintDefaultCharset
LANG = Ja_JP
Default charset = x-IBM943
file.encoding = IBM-943
sun.jnu.encoding = IBM-943

-bash-4.4$ LANG=Zh_TW ~/jdk/bin/java PrintDefaultCharset
LANG = Zh_TW
Default charset = Big5
file.encoding = big5
sun.jnu.encoding = big5

I will get webrev hosted in http://cr.openjdk.java.net for this change and send 
it for review once JIRA bug is created.

[1] http://demo.icu-project.org/icu-bin/convexp?conv=ibm-943_P130-1999=JAVA
[2] http://demo.icu-project.org/icu-bin/convexp?conv=ibm-943_P15A-2003=ALL
[3] 
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_71/com.ibm.aix.nlsgdrf/big5.htm


Thanks,
Bhaktavatsal Reddy

Re: RFR(S): 8201540: [AIX] Extend the set of supported charsets in java.base

[Bhaktavatsal R Maram]

Hi Volker,

Thank you for reviewing the patch. 

> you change looks good, although I can't really verify all the charset
> aliases. For example Wikipedia mentions that "ibm-932" is equivalent
> to "ibm-942" [1] but you made it an alias for "ibm-942C". What's
> actually the difference between "ibm-942C" and "ibm-942"?

IBM-942C is a customized version of IBM-942, in which following characters are 
replaced with ASCII thus making first 96 character mappings same as ASCII.

0x1A is mapped to 0x1C (in IBM-942) and to 0x1A (in IBM-942C)
0x1C is mapped to 0x7F (in IBM-942) and to 0x1C (in IBM-942C)
0x5C is mapped to 0xA5 (in IBM-942) and to 0x5C (in IBM-942C)
0x7E is mapped to 0x203E (in IBM-942) and to 0x7E (in IBM-942C)
0x7F is mapped to 0x1A (in IBM-942) and to 0x7F (in IBM-942C)

Similarly, IBM-943C is a customization for IBM-943 in which character mappings 
for Yen(¥) and overline(‾) are replaced by their ASCII equivalents backslash 
(\) and tilde (~). So, we should be mapping OS code-page IBM-943 to code-page 
IBM-943C in Java. I am working on fixing these inconsistencies in another 
defect in-order not to confuse things (I hope it is alright). Current patch 
mainly address moving default codepage from extended codepage list to standard 
codepage list. Also, There are few codepages which are missing in OpenJDK.


> I can sponsor your change although I would appreciate if somebody else
> from IBM could have another look at your change. I tried to compare
> with "IBM Java 9" but it doesn't seem to exist. They only refer to
> AdoptOpenJDK and AdoptOpenJDK just uses a vanilla version of OpenJDK.

Right! OpenJ9 version of JDK9 in AdoptOpenJDK is vanilla version of OpenJDK 
with OpenJ9. I've picked aliases for this patch from IBM JDK 8.

> Finally, I hope you won't mind if I update the copyright years on the
> files you changed before pushing (this is a convention in the OpenJDK
> project).

Sorry, I forgot to take care of copyright. Please change it this time before 
pushing. I will take care of it henceforth.


Thanks,
Bhaktavatsal Reddy

-Volker Simonis <volker.simo...@gmail.com> wrote: -
To: Bhaktavatsal R Maram <bhama...@in.ibm.com>
From: Volker Simonis <volker.simo...@gmail.com>
Date: 04/17/2018 08:30PM
Cc: Alan Bateman <alan.bate...@oracle.com>, Tim Ellison 
<tim_elli...@uk.ibm.com>, ppc-aix-port-...@openjdk.java.net, Java Core Libs 
<core-libs-dev@openjdk.java.net>
Subject: Re: RFR(S): 8201540: [AIX] Extend the set of supported charsets in 
java.base

Hi Bhaktavatsal Reddy,

you change looks good, although I can't really verify all the charset
aliases. For example Wikipedia mentions that "ibm-932" is equivalent
to "ibm-942" [1] but you made it an alias for "ibm-942C". What's
actually the difference between "ibm-942C" and "ibm-942"?

I can sponsor your change although I would appreciate if somebody else
from IBM could have another look at your change. I tried to compare
with "IBM Java 9" but it doesn't seem to exist. They only refer to
AdoptOpenJDK and AdoptOpenJDK just uses a vanilla version of OpenJDK.

Finally, I hope you won't mind if I update the copyright years on the
files you changed before pushing (this is a convention in the OpenJDK
project).

Best regards,
Volker

[1] 
https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_Code-5Fpage-5F932-5F-28IBM-29=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWPdQMmqg=DencrOI40Trgt_TxNW4dYVWqYtpT7dPnHzaSOEsw_ZQ=xYfspcI7N7ZAbVMqyjM7YIb_kd-RsFPn6pINIFz_Oa4=

On Mon, Apr 16, 2018 at 1:10 PM, Bhaktavatsal R Maram
<bhama...@in.ibm.com> wrote:
> Hi All,
>
> I've regenerated webrev using "hg rename" to create template files. webrev 
> looks much neat now.. Thanks Alan for suggestion.
>
> webrev - 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__cr.openjdk.java.net_-7Egromero_8201540_v2_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWPdQMmqg=DencrOI40Trgt_TxNW4dYVWqYtpT7dPnHzaSOEsw_ZQ=mDikak1wXAwU-a0yd6dJml9X5N1DJg-GkQmgPl4v_5g=
>
> Thanks,
> Bhaktavatsal Reddy
>
>
> -"core-libs-dev" <core-libs-dev-boun...@openjdk.java.net> wrote: -
> To: Alan Bateman <alan.bate...@oracle.com>
> From: "Bhaktavatsal R Maram"
> Sent by: "core-libs-dev"
> Date: 04/16/2018 02:38PM
> Cc: Tim Ellison <tim_elli...@uk.ibm.com>, ppc-aix-port-...@openjdk.java.net, 
> Java Core Libs <core-libs-dev@openjdk.java.net>
> Subject: Re: RFR(S): 8201540: [AIX] Extend the set of supported charsets in 
> java.base
>
> Hi Alan,
>
> I deleted IBM943C.java (using hg remove) and added new file 
> IBM943C.java.template (using hg add). I now understand that using "hg rename" 
> i

Re: RFR(S): 8201540: [AIX] Extend the set of supported charsets in java.base

[Bhaktavatsal R Maram]

Hi All,

I've regenerated webrev using "hg rename" to create template files. webrev 
looks much neat now.. Thanks Alan for suggestion.

webrev - http://cr.openjdk.java.net/~gromero/8201540/v2/

Thanks,
Bhaktavatsal Reddy


-"core-libs-dev" <core-libs-dev-boun...@openjdk.java.net> wrote: -
To: Alan Bateman <alan.bate...@oracle.com>
From: "Bhaktavatsal R Maram" 
Sent by: "core-libs-dev" 
Date: 04/16/2018 02:38PM
Cc: Tim Ellison <tim_elli...@uk.ibm.com>, ppc-aix-port-...@openjdk.java.net, 
Java Core Libs <core-libs-dev@openjdk.java.net>
Subject: Re: RFR(S): 8201540: [AIX] Extend the set of supported charsets in 
java.base

Hi Alan,

I deleted IBM943C.java (using hg remove) and added new file 
IBM943C.java.template (using hg add). I now understand that using "hg rename" 
is giving more meaningful representation in webrev/index.html.

I will re-generate webrev by renaming source files to templates using "hg 
rename"

Thanks,
Bhaktavatsal Reddy

  

-Alan Bateman <alan.bate...@oracle.com> wrote: -
To: Bhaktavatsal R Maram <bhama...@in.ibm.com>, Volker Simonis 
<volker.simo...@gmail.com>
From: Alan Bateman <alan.bate...@oracle.com>
Date: 04/16/2018 02:16PM
Cc: Java Core Libs <core-libs-dev@openjdk.java.net>, Tim Ellison 
<tim_elli...@uk.ibm.com>, ppc-aix-port-...@openjdk.java.net
Subject: Re: RFR(S): 8201540: [AIX] Extend the set of supported charsets in 
java.base


On 16/04/2018 09:22, Bhaktavatsal R Maram wrote:
>
> 3. Source files for IBM-942C and IBM-943C are changed to template to support 
> #1
>
You might want to double check the webrev as it looks like you've added 
templates where as I assume you mean to use "hg rename" to rename 
IBM942C.java and IBM943C.java.

-Alan

Re: RFR(S): 8201540: [AIX] Extend the set of supported charsets in java.base

[Bhaktavatsal R Maram]

Hi Alan,

I deleted IBM943C.java (using hg remove) and added new file 
IBM943C.java.template (using hg add). I now understand that using "hg rename" 
is giving more meaningful representation in webrev/index.html.

I will re-generate webrev by renaming source files to templates using "hg 
rename"

Thanks,
Bhaktavatsal Reddy

  

-Alan Bateman <alan.bate...@oracle.com> wrote: -
To: Bhaktavatsal R Maram <bhama...@in.ibm.com>, Volker Simonis 
<volker.simo...@gmail.com>
From: Alan Bateman <alan.bate...@oracle.com>
Date: 04/16/2018 02:16PM
Cc: Java Core Libs <core-libs-dev@openjdk.java.net>, Tim Ellison 
<tim_elli...@uk.ibm.com>, ppc-aix-port-...@openjdk.java.net
Subject: Re: RFR(S): 8201540: [AIX] Extend the set of supported charsets in 
java.base


On 16/04/2018 09:22, Bhaktavatsal R Maram wrote:
>
> 3. Source files for IBM-942C and IBM-943C are changed to template to support 
> #1
>
You might want to double check the webrev as it looks like you've added 
templates where as I assume you mean to use "hg rename" to rename 
IBM942C.java and IBM943C.java.

-Alan

RFR(S): 8201540: [AIX] Extend the set of supported charsets in java.base

[Bhaktavatsal R Maram]

Hi All,

Please review.

Bug: https://bugs.openjdk.java.net/browse/JDK-8201540
webrev: http://cr.openjdk.java.net/~gromero/8201540/v1/webrev/

In this patch,

1. Default charsets Big5, Big5_Solaris, Big5_HKSCS, GB18030, IBM856, IBM921, 
IBM922, IBM942, IBM942C, IBM943, IBM943C, IBM950, IBM970, IBM1046, IBM1124, 
IBM1383, ISO_8859_6, ISO_8859_8, MS1252, TIS_620 for different locales 
supported on AIX are added to standard charsets in java.base module

2. More aliases are added to existing charsets.

3. Source files for IBM-942C and IBM-943C are changed to template to support #1

4. Modified file make/jdk/src/classes/build/tools/charsetmapping/SPI.java to 
increase the Hashtable capacity that holds aliases of standard charsets. As the 
no.of charsets we include in standard charsets are more in this patch, existing 
capacity of Hashtable used to hold aliases is not efficient. Without change to 
this file, JDK won't get built and following error is seen

Exception in thread "main" java.lang.RuntimeException: Cannot find a suitable 
size within given constraints
at build.tools.charsetmapping.Hasher.build(Hasher.java:122)
at build.tools.charsetmapping.Hasher.genClass(Hasher.java:261)
at build.tools.charsetmapping.SPI.genClass(SPI.java:130)
at build.tools.charsetmapping.Main.main(Main.java:99)

Please note that webrev/index.html is not showing below 2 files I deleted. 
However, webrev/jdk11.changeset file show them deleted properly.
  src/jdk.charsets/share/classes/sun/nio/cs/ext/IBM942C.java
  src/jdk.charsets/share/classes/sun/nio/cs/ext/IBM943C.java

I've built the JDK with this patch on both Linux and Aix.


Thanks,
Bhaktavatsal Reddy


  

-Volker Simonis <volker.simo...@gmail.com> wrote: -
To: Bhaktavatsal R Maram <bhama...@in.ibm.com>
From: Volker Simonis <volker.simo...@gmail.com>
Date: 04/13/2018 08:51PM
Cc: Alan Bateman <alan.bate...@oracle.com>, Java Core Libs 
<core-libs-dev@openjdk.java.net>, Tim Ellison <tim_elli...@uk.ibm.com>, 
ppc-aix-port-...@openjdk.java.net
Subject: Re: Missing many locales support on AIX platform

Hi Bhaktavatsal Reddy,

thanks for addressing this long standing issue. I've opened "8201540:
[AIX] Extend the set of supported charsets in java.base" [1] to track
this issue.

As I wrote in the bug report, this problem is the consequence of an
emergency fix (JDK-8081332) I did back in 2015 to fix the build on AIX
after the integration of the modularity support change (see
discussion: [2]). At that time I only added the minimal set of
charsets which were required to fix the build.

It would be great if you can get in touch with your IBM colleagues to
find out which are the default extended charsets supported by IBM J9
on AIX. I think we should try to use a similar set here in our OpenJDK
port which is also used by the OpenJ9 for building OpenJ9 on AIX.

Also, your IBM colleagues can help you to host a webrev which will
make the further review of these changes much easier. I've but Tim
from IBM on CC which should have an overview of all the IBM people
involved in the OpenJDK project. Besides that I know at least the
following people who might help you:

Michihiro Horie: ho...@jp.ibm.com
Hiroshi H Horii: ho...@jp.ibm.com
Gustavo Romero: grom...@linux.vnet.ibm.com
Matthew Brandyberyy: mbra...@linux.vnet.ibm.com

Thank you and best regards,
Volker

[1] 
https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.openjdk.java.net_browse_JDK-2D8201540=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWPdQMmqg=cieXGANE8bD3liEC2gJzl5hfZorR2qIfggL6U9t-Et8=hC4gcA6uomYgY26uR74VelobSIK1ReDsjRZGmI46UBY=
[2] 
https://urldefense.proofpoint.com/v2/url?u=http-3A__mail.openjdk.java.net_pipermail_2d-2Ddev_2015-2DMay_005431.html=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWPdQMmqg=cieXGANE8bD3liEC2gJzl5hfZorR2qIfggL6U9t-Et8=qfYLaVVn7tapNDhv5bY6yE72nhngaWqte4wtRhQ3wB8=

On Fri, Apr 13, 2018 at 2:42 PM, Bhaktavatsal R Maram
<bhama...@in.ibm.com> wrote:
> Hi Alan,
>
> Thank you for your response. I'm happy that my patch was attached. But, I 
> don't see attachment. So, I inlined patch which contain diffs from 2 
> changesets in mail text. If a Jira bug is opened for this issue, probably I 
> can attach complete and consolidated patch there.
>
> At high level, I'm adding following charsets to standard charset in 
> java.base. For this, I have to change IBM943C and IBM942C from source to 
> template to handle java package and aliases. It is also required to add 
> codepage 932 as alias for IBM942C because both are one and same.
>
> Big5, Big5_HKSCS, GB18030, IBM942, IBM942C, IBM943, IBM943C, IBM950, IBM970, 
> IBM1124, TIS_620
>
>
> These are default charsets for some of locales supported by Operating System 
> (AIX). Since these are not available in standard charset, JDK can't be used 
> in those locale even

Re: Missing many locales support on AIX platform

[Bhaktavatsal R Maram]

Hi Volker,

Thank you. I will take help of my colleagues and host webrev with all the 
default charsets that are missing for AIX. 
 
- Bhaktavatsal Reddy

-Volker Simonis <volker.simo...@gmail.com> wrote: -
To: Bhaktavatsal R Maram <bhama...@in.ibm.com>
From: Volker Simonis <volker.simo...@gmail.com>
Date: 04/13/2018 08:51PM
Cc: Alan Bateman <alan.bate...@oracle.com>, Java Core Libs 
<core-libs-dev@openjdk.java.net>, Tim Ellison <tim_elli...@uk.ibm.com>, 
ppc-aix-port-...@openjdk.java.net
Subject: Re: Missing many locales support on AIX platform

Hi Bhaktavatsal Reddy,

thanks for addressing this long standing issue. I've opened "8201540:
[AIX] Extend the set of supported charsets in java.base" [1] to track
this issue.

As I wrote in the bug report, this problem is the consequence of an
emergency fix (JDK-8081332) I did back in 2015 to fix the build on AIX
after the integration of the modularity support change (see
discussion: [2]). At that time I only added the minimal set of
charsets which were required to fix the build.

It would be great if you can get in touch with your IBM colleagues to
find out which are the default extended charsets supported by IBM J9
on AIX. I think we should try to use a similar set here in our OpenJDK
port which is also used by the OpenJ9 for building OpenJ9 on AIX.

Also, your IBM colleagues can help you to host a webrev which will
make the further review of these changes much easier. I've but Tim
from IBM on CC which should have an overview of all the IBM people
involved in the OpenJDK project. Besides that I know at least the
following people who might help you:

Michihiro Horie: ho...@jp.ibm.com
Hiroshi H Horii: ho...@jp.ibm.com
Gustavo Romero: grom...@linux.vnet.ibm.com
Matthew Brandyberyy: mbra...@linux.vnet.ibm.com

Thank you and best regards,
Volker

[1] 
https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.openjdk.java.net_browse_JDK-2D8201540=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWPdQMmqg=cieXGANE8bD3liEC2gJzl5hfZorR2qIfggL6U9t-Et8=hC4gcA6uomYgY26uR74VelobSIK1ReDsjRZGmI46UBY=
[2] 
https://urldefense.proofpoint.com/v2/url?u=http-3A__mail.openjdk.java.net_pipermail_2d-2Ddev_2015-2DMay_005431.html=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=KUVGEwJiRVpNtQ9wUhGP6BKqzSTV1OWX31WWPdQMmqg=cieXGANE8bD3liEC2gJzl5hfZorR2qIfggL6U9t-Et8=qfYLaVVn7tapNDhv5bY6yE72nhngaWqte4wtRhQ3wB8=

On Fri, Apr 13, 2018 at 2:42 PM, Bhaktavatsal R Maram
<bhama...@in.ibm.com> wrote:
> Hi Alan,
>
> Thank you for your response. I'm happy that my patch was attached. But, I 
> don't see attachment. So, I inlined patch which contain diffs from 2 
> changesets in mail text. If a Jira bug is opened for this issue, probably I 
> can attach complete and consolidated patch there.
>
> At high level, I'm adding following charsets to standard charset in 
> java.base. For this, I have to change IBM943C and IBM942C from source to 
> template to handle java package and aliases. It is also required to add 
> codepage 932 as alias for IBM942C because both are one and same.
>
> Big5, Big5_HKSCS, GB18030, IBM942, IBM942C, IBM943, IBM943C, IBM950, IBM970, 
> IBM1124, TIS_620
>
>
> These are default charsets for some of locales supported by Operating System 
> (AIX). Since these are not available in standard charset, JDK can't be used 
> in those locale even if they are available in jdk.charset module (java 
> -version fails).
>
> I've followed some of the discussions around this in community and understand 
> that default charset of a locale should be made available in java.charset 
> module by using stdcs-* mechanism. On Linux, they were added to java.base in 
> similar way. As it is missing for AIX, I've added them to enable JDK support 
> for more locales.
>
>
> Thanks,
> Bhaktavatsal Reddy
>
> -Alan Bateman <alan.bate...@oracle.com> wrote: -
> To: Bhaktavatsal R Maram <bhama...@in.ibm.com>, core-libs-dev@openjdk.java.net
> From: Alan Bateman <alan.bate...@oracle.com>
> Date: 04/13/2018 03:52PM
> Subject: Re: Missing many locales support on AIX platform
>
> On 13/04/2018 10:35, Bhaktavatsal R Maram wrote:
>> Given that patch is big, I am sending patch as attachment again after 
>> changing some mail settings. Hopefully, it will make to community this time.
> Your patch was attached.
>
> If I read it correctly, you've switched IBM943C to a template but there
> aren't additional aliases so this part is effectively a no-op, is that
> right? For IBM932C, you've moved it to be template and added several
> aliases.
>
> The rest is AIX specific and I hope the SAP engineers that maintain the
> AIX port can help you. It may be that you are testing with locales that
> aren't supported configurations for the AIX port in OpenJDK. As a
> general point, we try to keep as many of the exotic and multibyte
> charsets out of java.base. They are of course still available to
> applications via the API and the jdk.charsets service provider module.
>
> -Alan
>
>

Re: Missing many locales support on AIX platform

[Bhaktavatsal R Maram]

Hi Alan,

Thank you for your response. I'm happy that my patch was attached. But, I don't 
see attachment. So, I inlined patch which contain diffs from 2 changesets in 
mail text. If a Jira bug is opened for this issue, probably I can attach 
complete and consolidated patch there.

At high level, I'm adding following charsets to standard charset in java.base. 
For this, I have to change IBM943C and IBM942C from source to template to 
handle java package and aliases. It is also required to add codepage 932 as 
alias for IBM942C because both are one and same.

Big5, Big5_HKSCS, GB18030, IBM942, IBM942C, IBM943, IBM943C, IBM950, IBM970, 
IBM1124, TIS_620


These are default charsets for some of locales supported by Operating System 
(AIX). Since these are not available in standard charset, JDK can't be used in 
those locale even if they are available in jdk.charset module (java -version 
fails).

I've followed some of the discussions around this in community and understand 
that default charset of a locale should be made available in java.charset 
module by using stdcs-* mechanism. On Linux, they were added to java.base in 
similar way. As it is missing for AIX, I've added them to enable JDK support 
for more locales.
 

Thanks,
Bhaktavatsal Reddy

-Alan Bateman <alan.bate...@oracle.com> wrote: -
To: Bhaktavatsal R Maram <bhama...@in.ibm.com>, core-libs-dev@openjdk.java.net
From: Alan Bateman <alan.bate...@oracle.com>
Date: 04/13/2018 03:52PM
Subject: Re: Missing many locales support on AIX platform

On 13/04/2018 10:35, Bhaktavatsal R Maram wrote:
> Given that patch is big, I am sending patch as attachment again after 
> changing some mail settings. Hopefully, it will make to community this time.
Your patch was attached.

If I read it correctly, you've switched IBM943C to a template but there 
aren't additional aliases so this part is effectively a no-op, is that 
right? For IBM932C, you've moved it to be template and added several 
aliases.

The rest is AIX specific and I hope the SAP engineers that maintain the 
AIX port can help you. It may be that you are testing with locales that 
aren't supported configurations for the AIX port in OpenJDK. As a 
general point, we try to keep as many of the exotic and multibyte 
charsets out of java.base. They are of course still available to 
applications via the API and the jdk.charsets service provider module.

-Alan

Re: Missing many locales support on AIX platform

[Bhaktavatsal R Maram]

No luck! :(

How can I attach patch? Any suggestions?

Thanks,
Bhaktavatsal Reddy

  

-Bhaktavatsal R Maram/India/IBM wrote: -
To: core-libs-dev@openjdk.java.net
From: Bhaktavatsal R Maram/India/IBM
Date: 04/13/2018 03:05PM
Subject: Re: Missing many locales support on AIX platform

Given that patch is big, I am sending patch as attachment again after changing 
some mail settings. Hopefully, it will make to community this time.

Thanks,
Bhaktavatsal Reddy
 

  

-Bhaktavatsal R Maram/India/IBM wrote: -
To: core-libs-dev@openjdk.java.net
From: Bhaktavatsal R Maram/India/IBM
Date: 04/12/2018 11:47PM
Subject: Re: Missing many locales support on AIX platform

Please review this patch for this issue. I have added IBM-943C as well to 
standard charset along with IBM-943.

# HG changeset patch
# User bhamaram
# Date 1523510846 -19800
#  Thu Apr 12 10:57:26 2018 +0530
# Node ID a76dd37b1c6b47e7ac99f279c4ba76832dd9736f
# Parent  129d60b5dac71872f3317d94c724304100bc2048
java.lang.IllegalArgumentException from java -version on AIX with different 
locales

diff -r 129d60b5dac7 -r a76dd37b1c6b make/data/charsetmapping/charsets
--- a/make/data/charsetmapping/charsets Thu Apr 12 09:03:46 2018 -0400
+++ b/make/data/charsetmapping/charsets Thu Apr 12 10:57:26 2018 +0530
@@ -933,11 +933,16 @@
 
 charset x-IBM942C IBM942C
 package sun.nio.cs.ext
-typesource
+typetemplate
 alias   cp942C   # JDK historical
 alias   ibm942C
 alias   ibm-942C
 alias   942C
+alias   cp932
+alias   ibm932
+alias   ibm-932
+alias   932
+alias   x-ibm932
 
 charset x-IBM943 IBM943
 package sun.nio.cs.ext
diff -r 129d60b5dac7 -r a76dd37b1c6b make/data/charsetmapping/stdcs-aix
--- a/make/data/charsetmapping/stdcs-aixThu Apr 12 09:03:46 2018 -0400
+++ b/make/data/charsetmapping/stdcs-aixThu Apr 12 10:57:26 2018 +0530
@@ -1,6 +1,16 @@
 #
 #   generate these charsets into sun.nio.cs
 #
+Big5
+Big5_HKSCS
 EUC_CN
 EUC_KR
 GBK
+GB18030
+IBM942
+IBM942C
+IBM943
+IBM950
+IBM970
+IBM1124
+TIS_620
diff -r 129d60b5dac7 -r a76dd37b1c6b 
src/jdk.charsets/share/classes/sun/nio/cs/ext/IBM942C.java
--- a/src/jdk.charsets/share/classes/sun/nio/cs/ext/IBM942C.javaThu Apr 
12 09:03:46 2018 -0400
+++ /dev/null   Thu Jan 01 00:00:00 1970 +
@@ -1,93 +0,0 @@
-/*
- * Copyright (c) 2003, 2004, Oracle and/or its affiliates. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.  Oracle designates this
- * particular file as subject to the "Classpath" exception as provided
- * by Oracle in the LICENSE file that accompanied this code.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- */
-
-package sun.nio.cs.ext;
-
-import java.nio.charset.Charset;
-import java.nio.charset.CharsetDecoder;
-import java.nio.charset.CharsetEncoder;
-import java.util.Arrays;
-import sun.nio.cs.DoubleByte;
-import sun.nio.cs.HistoricallyNamedCharset;
-import static sun.nio.cs.CharsetMapping.*;
-
-public class IBM942C extends Charset implements HistoricallyNamedCharset
-{
-public IBM942C() {
-super("x-IBM942C", ExtendedCharsets.aliasesFor("x-IBM942C"));
-}
-
-public String historicalName() {
-return "Cp942C";
-}
-
-public boolean contains(Charset cs) {
-return ((cs.name().equals("US-ASCII"))
-|| (cs instanceof IBM942C));
-}
-
-public CharsetDecoder newDecoder() {
-return new DoubleByte.Decoder(this,
-  IBM942.b2c,
-  b2cSB,
-  0x40,
-  0xfc);
-}
-
-public CharsetEncoder newEncoder() {
-return new DoubleByte.Encoder(this, c2b, c2bIndex);
-}
-
-final static char[] b2cSB;
-final static char[] c2b;
-final static char[] c2bIndex;
-
-static {
-IBM942.initb2c();
-
-// the mappings need udpate are
-//u+001a  <-> 0x1a
-//u+001c  <-> 0x1c
-

Re: Missing many locales support on AIX platform

[Bhaktavatsal R Maram]

Given that patch is big, I am sending patch as attachment again after changing 
some mail settings. Hopefully, it will make to community this time.

Thanks,
Bhaktavatsal Reddy
 

  

-Bhaktavatsal R Maram/India/IBM wrote: -
To: core-libs-dev@openjdk.java.net
From: Bhaktavatsal R Maram/India/IBM
Date: 04/12/2018 11:47PM
Subject: Re: Missing many locales support on AIX platform

Please review this patch for this issue. I have added IBM-943C as well to 
standard charset along with IBM-943.

# HG changeset patch
# User bhamaram
# Date 1523510846 -19800
#  Thu Apr 12 10:57:26 2018 +0530
# Node ID a76dd37b1c6b47e7ac99f279c4ba76832dd9736f
# Parent  129d60b5dac71872f3317d94c724304100bc2048
java.lang.IllegalArgumentException from java -version on AIX with different 
locales

diff -r 129d60b5dac7 -r a76dd37b1c6b make/data/charsetmapping/charsets
--- a/make/data/charsetmapping/charsets Thu Apr 12 09:03:46 2018 -0400
+++ b/make/data/charsetmapping/charsets Thu Apr 12 10:57:26 2018 +0530
@@ -933,11 +933,16 @@
 
 charset x-IBM942C IBM942C
 package sun.nio.cs.ext
-typesource
+typetemplate
 alias   cp942C   # JDK historical
 alias   ibm942C
 alias   ibm-942C
 alias   942C
+alias   cp932
+alias   ibm932
+alias   ibm-932
+alias   932
+alias   x-ibm932
 
 charset x-IBM943 IBM943
 package sun.nio.cs.ext
diff -r 129d60b5dac7 -r a76dd37b1c6b make/data/charsetmapping/stdcs-aix
--- a/make/data/charsetmapping/stdcs-aixThu Apr 12 09:03:46 2018 -0400
+++ b/make/data/charsetmapping/stdcs-aixThu Apr 12 10:57:26 2018 +0530
@@ -1,6 +1,16 @@
 #
 #   generate these charsets into sun.nio.cs
 #
+Big5
+Big5_HKSCS
 EUC_CN
 EUC_KR
 GBK
+GB18030
+IBM942
+IBM942C
+IBM943
+IBM950
+IBM970
+IBM1124
+TIS_620
diff -r 129d60b5dac7 -r a76dd37b1c6b 
src/jdk.charsets/share/classes/sun/nio/cs/ext/IBM942C.java
--- a/src/jdk.charsets/share/classes/sun/nio/cs/ext/IBM942C.javaThu Apr 
12 09:03:46 2018 -0400
+++ /dev/null   Thu Jan 01 00:00:00 1970 +
@@ -1,93 +0,0 @@
-/*
- * Copyright (c) 2003, 2004, Oracle and/or its affiliates. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.  Oracle designates this
- * particular file as subject to the "Classpath" exception as provided
- * by Oracle in the LICENSE file that accompanied this code.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- */
-
-package sun.nio.cs.ext;
-
-import java.nio.charset.Charset;
-import java.nio.charset.CharsetDecoder;
-import java.nio.charset.CharsetEncoder;
-import java.util.Arrays;
-import sun.nio.cs.DoubleByte;
-import sun.nio.cs.HistoricallyNamedCharset;
-import static sun.nio.cs.CharsetMapping.*;
-
-public class IBM942C extends Charset implements HistoricallyNamedCharset
-{
-public IBM942C() {
-super("x-IBM942C", ExtendedCharsets.aliasesFor("x-IBM942C"));
-}
-
-public String historicalName() {
-return "Cp942C";
-}
-
-public boolean contains(Charset cs) {
-return ((cs.name().equals("US-ASCII"))
-|| (cs instanceof IBM942C));
-}
-
-public CharsetDecoder newDecoder() {
-return new DoubleByte.Decoder(this,
-  IBM942.b2c,
-  b2cSB,
-  0x40,
-  0xfc);
-}
-
-public CharsetEncoder newEncoder() {
-return new DoubleByte.Encoder(this, c2b, c2bIndex);
-}
-
-final static char[] b2cSB;
-final static char[] c2b;
-final static char[] c2bIndex;
-
-static {
-IBM942.initb2c();
-
-// the mappings need udpate are
-//u+001a  <-> 0x1a
-//u+001c  <-> 0x1c
-//u+005c  <-> 0x5c
-//u+007e  <-> 0x7e
-//u+007f  <-> 0x7f
-
-b2cSB = Arrays.copyOf(IBM942.b2cSB, IBM942.b2cSB.length);
-b2cSB[0x1a] = 0x1a;
-b2cSB[0x1c] = 0x1c;
-b2cSB[0x5c] = 0x5c;
-b2cSB[0x7e

Re: Missing many locales support on AIX platform

[Bhaktavatsal R Maram]

Cp943C";
+}
+
+public boolean contains(Charset cs) {
+return ((cs.name().equals("US-ASCII"))
+|| (cs instanceof IBM943C));
+}
+
+public CharsetDecoder newDecoder() {
+return new DoubleByte.Decoder(this,
+  IBM943.b2c,
+  b2cSB,
+  0x40,
+  0xfc);
+}
+
+public CharsetEncoder newEncoder() {
+return new DoubleByte.Encoder(this, c2b, c2bIndex);
+}
+
+final static char[] b2cSB;
+final static char[] c2b;
+final static char[] c2bIndex;
+
+static {
+IBM943.initb2c();
+b2cSB = new char[0x100];
+for (int i = 0; i < 0x80; i++) {
+b2cSB[i] = (char)i;
+}
+for (int i = 0x80; i < 0x100; i++) {
+b2cSB[i] = IBM943.b2cSB[i];
+}
+
+IBM943.initc2b();
+c2b = Arrays.copyOf(IBM943.c2b, IBM943.c2b.length);
+c2bIndex = Arrays.copyOf(IBM943.c2bIndex, IBM943.c2bIndex.length);
+for (char c = '\0'; c < '\u0080'; ++c) {
+    int index = c2bIndex[c >> 8];
+c2b[index + (c & 0xff)] = c;
+}
+}
+}


Thanks,
Bhaktavatsal Reddy

  

-"core-libs-dev" <core-libs-dev-boun...@openjdk.java.net> wrote: -
To: core-libs-dev@openjdk.java.net
From: "Bhaktavatsal R Maram" 
Sent by: "core-libs-dev" 
Date: 04/12/2018 03:16PM
Subject: Re: Missing many locales support on AIX platform

Patch I attached to previous mail is not seen. So, pasting patch here. Hope, it 
is fine.


# HG changeset patch
# User bhamaram
# Date 1523510846 -19800
# Node ID 6134bbe986f8cc42b055081f2c48e91039a8
# Parent  0c3e252cea44f06aef570ef464950ab97c669970
java.lang.IllegalArgumentException from java -version on AIX with different 
locales

diff -r 0c3e252cea44 -r 6134bbe98611 make/data/charsetmapping/charsets
--- a/make/data/charsetmapping/charsets Thu Apr 12 10:19:31 2018 +0800
+++ b/make/data/charsetmapping/charsets Thu Apr 12 10:57:26 2018 +0530
@@ -933,11 +933,16 @@
 
 charset x-IBM942C IBM942C
 package sun.nio.cs.ext
-typesource
+typetemplate
 alias   cp942C   # JDK historical
 alias   ibm942C
 alias   ibm-942C
 alias   942C
+alias   cp932
+alias   ibm932
+alias   ibm-932
+alias   932
+alias   x-ibm932
 
 charset x-IBM943 IBM943
 package sun.nio.cs.ext
diff -r 0c3e252cea44 -r 6134bbe98611 make/data/charsetmapping/stdcs-aix
--- a/make/data/charsetmapping/stdcs-aixThu Apr 12 10:19:31 2018 +0800
+++ b/make/data/charsetmapping/stdcs-aixThu Apr 12 10:57:26 2018 +0530
@@ -1,6 +1,16 @@
 #
 #   generate these charsets into sun.nio.cs
 #
+Big5
+Big5_HKSCS
 EUC_CN
 EUC_KR
 GBK
+GB18030
+IBM942
+IBM942C
+IBM943
+IBM950
+IBM970
+IBM1124
+TIS_620
diff -r 0c3e252cea44 -r 6134bbe98611 
src/jdk.charsets/share/classes/sun/nio/cs/ext/IBM942C.java
--- a/src/jdk.charsets/share/classes/sun/nio/cs/ext/IBM942C.javaThu Apr 
12 10:19:31 2018 +0800
+++ /dev/null   Thu Jan 01 00:00:00 1970 +
@@ -1,93 +0,0 @@
-/*
- * Copyright (c) 2003, 2004, Oracle and/or its affiliates. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.  Oracle designates this
- * particular file as subject to the "Classpath" exception as provided
- * by Oracle in the LICENSE file that accompanied this code.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- */
-
-package sun.nio.cs.ext;
-
-import java.nio.charset.Charset;
-import java.nio.charset.CharsetDecoder;
-import java.nio.charset.CharsetEncoder;
-import java.util.Arrays;
-import sun.nio.cs.DoubleByte;
-import sun.nio.cs.HistoricallyNamedCharset;
-import static sun.nio.cs.CharsetMapping.*;
-
-public class IBM942C extends Charset implements HistoricallyNamedCharset
-{
-public IBM942C() {
-super("x-IBM942C", ExtendedCharsets.aliasesFor("x-IBM942C"));
-}
-
-public String historicalName() {
-return &qu

Re: Missing many locales support on AIX platform

[Bhaktavatsal R Maram]

rc/jdk.charsets/share/classes/sun/nio/cs/ext/IBM942C.java.template
--- /dev/null   Thu Jan 01 00:00:00 1970 +
+++ b/src/jdk.charsets/share/classes/sun/nio/cs/ext/IBM942C.java.template   
Thu Apr 12 10:57:26 2018 +0530
@@ -0,0 +1,93 @@
+/*
+ * Copyright (c) 2003, 2004, Oracle and/or its affiliates. All rights reserved.
+ * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+ *
+ * This code is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 only, as
+ * published by the Free Software Foundation.  Oracle designates this
+ * particular file as subject to the "Classpath" exception as provided
+ * by Oracle in the LICENSE file that accompanied this code.
+ *
+ * This code is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+ * version 2 for more details (a copy is included in the LICENSE file that
+ * accompanied this code).
+ *
+ * You should have received a copy of the GNU General Public License version
+ * 2 along with this work; if not, write to the Free Software Foundation,
+ * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
+ *
+ * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
+ * or visit www.oracle.com if you need additional information or have any
+ * questions.
+ */
+
+package $PACKAGE$;
+
+import java.nio.charset.Charset;
+import java.nio.charset.CharsetDecoder;
+import java.nio.charset.CharsetEncoder;
+import java.util.Arrays;
+import sun.nio.cs.DoubleByte;
+import sun.nio.cs.HistoricallyNamedCharset;
+import static sun.nio.cs.CharsetMapping.*;
+
+public class IBM942C extends Charset implements HistoricallyNamedCharset
+{
+public IBM942C() {
+super("x-IBM942C", $ALIASES$);
+}
+
+public String historicalName() {
+return "Cp942C";
+}
+
+public boolean contains(Charset cs) {
+return ((cs.name().equals("US-ASCII"))
+|| (cs instanceof IBM942C));
+}
+
+public CharsetDecoder newDecoder() {
+return new DoubleByte.Decoder(this,
+  IBM942.b2c,
+  b2cSB,
+  0x40,
+  0xfc);
+}
+
+public CharsetEncoder newEncoder() {
+return new DoubleByte.Encoder(this, c2b, c2bIndex);
+}
+
+final static char[] b2cSB;
+final static char[] c2b;
+final static char[] c2bIndex;
+
+static {
+IBM942.initb2c();
+
+// the mappings need udpate are
+//u+001a  <-> 0x1a
+//u+001c  <-> 0x1c
+//u+005c  <-> 0x5c
+//u+007e  <-> 0x7e
+//u+007f  <-> 0x7f
+
+b2cSB = Arrays.copyOf(IBM942.b2cSB, IBM942.b2cSB.length);
+b2cSB[0x1a] = 0x1a;
+b2cSB[0x1c] = 0x1c;
+b2cSB[0x5c] = 0x5c;
+b2cSB[0x7e] = 0x7e;
+b2cSB[0x7f] = 0x7f;
+
+IBM942.initc2b();
+c2b = Arrays.copyOf(IBM942.c2b, IBM942.c2b.length);
+c2bIndex = Arrays.copyOf(IBM942.c2bIndex, IBM942.c2bIndex.length);
+c2b[c2bIndex[0] + 0x1a] = 0x1a;
+c2b[c2bIndex[0] + 0x1c] = 0x1c;
+c2b[c2bIndex[0] + 0x5c] = 0x5c;
+c2b[c2bIndex[0] + 0x7e] = 0x7e;
+c2b[c2bIndex[0] + 0x7f] = 0x7f;
+}
+}


Thanks,
Bhaktavatsal Reddy
 

  

-"core-libs-dev" <core-libs-dev-boun...@openjdk.java.net> wrote: -
To: core-libs-dev@openjdk.java.net
From: "Bhaktavatsal R Maram" 
Sent by: "core-libs-dev" 
Date: 04/12/2018 03:12PM
Subject: Missing many locales support on AIX platform

Hi,

On AIX platforms, Java is not starting on many locales. Java initialization 
fails with java.lang.IllegalArgumentException 

bash-4.4$ LANG=Ja_JP java -version
Error occurred during initialization of VM
java.lang.IllegalArgumentException: Null charset name
at java.nio.charset.Charset.lookup(java.base/Charset.java:455)
at java.nio.charset.Charset.defaultCharset(java.base/Charset.java:608)
at java.lang.StringCoding.decode(java.base/StringCoding.java:314)
at java.lang.String.(java.base/String.java:591)
at java.lang.String.(java.base/String.java:613)
at java.lang.System.initProperties(java.base/Native Method)
at java.lang.System.initPhase1(java.base/System.java:1908)

>From our analysis, following are locales that are impacted with this issue.

Ar_AA.IBM-1046
ar_AA.ISO8859-6 
ca_ES.IBM-1252@euro
Iw_IL.IBM-856
iw_IL.ISO8859-8 
Et_EE.IBM-922@euro 
Ja_JP.IBM-932
Ja_JP.IBM-943   
ja_JP.IBM-eucJP 
ko_KR.IBM-eucKR 
Lt_LT.IBM-921@euro 
th_TH.TIS-620
Uk_UA.IBM-1124
Vi_VN.IBM-1129
Zh_CN.GB18030
zh_CN.IBM-eucCN
Zh_HK.BIG5-HKSCS
Zh_TW.big5


The default charset for these loca

Missing many locales support on AIX platform

[Bhaktavatsal R Maram]

Hi,

On AIX platforms, Java is not starting on many locales. Java initialization 
fails with java.lang.IllegalArgumentException 

bash-4.4$ LANG=Ja_JP java -version
Error occurred during initialization of VM
java.lang.IllegalArgumentException: Null charset name
at java.nio.charset.Charset.lookup(java.base/Charset.java:455)
at java.nio.charset.Charset.defaultCharset(java.base/Charset.java:608)
at java.lang.StringCoding.decode(java.base/StringCoding.java:314)
at java.lang.String.(java.base/String.java:591)
at java.lang.String.(java.base/String.java:613)
at java.lang.System.initProperties(java.base/Native Method)
at java.lang.System.initPhase1(java.base/System.java:1908)

>From our analysis, following are locales that are impacted with this issue.

Ar_AA.IBM-1046
ar_AA.ISO8859-6 
ca_ES.IBM-1252@euro
Iw_IL.IBM-856
iw_IL.ISO8859-8 
Et_EE.IBM-922@euro 
Ja_JP.IBM-932
Ja_JP.IBM-943   
ja_JP.IBM-eucJP 
ko_KR.IBM-eucKR 
Lt_LT.IBM-921@euro 
th_TH.TIS-620
Uk_UA.IBM-1124
Vi_VN.IBM-1129
Zh_CN.GB18030
zh_CN.IBM-eucCN
Zh_HK.BIG5-HKSCS
Zh_TW.big5


The default charset for these locales are not available in the java.base module 
and hence java cannot be used with those locales. In the current fix, am fixing 
the issue for following locales.

Ja_JP.IBM-932
Ja_JP.IBM-943
ko_KR.IBM-eucKR
th_TH.TIS-620
Uk_UA.IBM-1124
Zh_CN.GB18030
Zh_HK.BIG5-HKSCS
Zh_TW.big5

I've attached fix with this mail. I will continue working on other locales and 
provide the fix. In the mean time, I kindly request you to open a bug and 
review the fix for the above locales.

Thanks,
Bhaktavatsal Reddy