Re: [9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Denis Kononenko]

Hi All,

> On 9/28/2016 2:02 AM, Artem Smotrakov wrote:
> > Currently the tests silently quit which looks like they pass. This
> makes
> > people think that everything went smoothly, but actually nothing
> was
> > tested.
> >
> I did not get the idea.  Looks like, if no NSS installed, the test
> would 
> be ignored; if NSS get installed, the test is actually get run.  If no
> 
> NSS get installed,  the test should quite silently and test nothing 
> because nothing should be tested.  That's the expected common behavior
> 
> in order to test specific configuration.

The problem is the tests report they passed but actually they were skipped. I 
have no objections against skipping tests but it would be better to give a hint 
somehow how many tests were skipped and why. 

> 
> If you don't want the test quit silently, I may prefer to check the 
> platform to install NSS libs rather than update this test cases. 
> Exposing the testing environment configuration problem is not the job
> of 
> this test.

There're 60+ tests related to PKCS11. Two years they have been "passed" on 3 
unsupported platforms on hosts where usually no NSS libraries were installed. 
How can we rely on these results?

> 
> > I would prefer to update PKCS11Test to report a failure in case of
> > unexpected platform.
> Then you need to know all expected platform.  The test is not only run
> 
> in a certain known environment (for example the platforms Mach5 or 
> JPRT), it can also be run by third party environment (OpenJDK 
> contributors).  If not possible, I think it is a hard job to know all
> 
> the expected platform exactly.

>From my experience the most common environment issue is a lack of GUI 
>libraries. Fortunately, in such case tests simply fail due to unsatisfied 
>dependencies. If they would pass due to uncertain environment we could end up 
>having completely untested UI functionality with green status.

> 
> Xuelei

Thank you,
Denis.

Re: [9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Xuelei Fan]

Looks fine to me.

Xuelei

On 9/30/2016 4:39 PM, Tim Du wrote:

Thanks Xuelei review it.
Latest webrev available here
:http://cr.openjdk.java.net/~tidu/8164322/webrev.02/ ,Please help to
review it. Thanks

Regards
Tim
On 2016/9/29 18:16, Xuelei Fan wrote:

On 9/29/2016 5:36 PM, Tim Du wrote:

Thank you for reviewing the codes.
Remove the codes which throw exception and add codes could show machine
not installed NSS library information.
The latest code is here:
http://cr.openjdk.java.net/~tidu/8164322/webrev.01/ ,please help to
review it.


Looks fine to me.  A very minor comment:
 303   + ",please initialize ...

Please add a white space between ",' and 'please".

Thanks,
Xuelei


For whether throw exception caused by OS/machine issue ,I file an
Enhancement https://bugs.openjdk.java.net/browse/JDK-8166893 , we can
keep discussing by it ,and then the test
sun/security/pkcs11/PKCS11Test.java could be supported to run on ARM
platform firstly. Thanks

Regards
Tim
On 2016/9/29 13:05, Xuelei Fan wrote:

On 9/29/2016 9:26 AM, Artem Smotrakov wrote:

Hi Xuelei,

This is not a problem with machine configuration. The actual test
cases
are not going to be run (even if there are NSS libs on a test machine)
until "osMap" contains an entry for specific platform.


OK.  Please fix for the purpose accordingly.

Tim, the current fix has two parts.  The 1st part, which throw
exception is a machine configuration problem, is not in the scope of
Artem's consideration, please remove it.  The 2nd part, which add new
items to the osMap, is fine to me.  I think we are on the same page
now.  Please update the webrev accordingly.  I will approve the 2nd
part fix.

Thanks,
Xuelei


Artem


On 09/28/2016 05:52 PM, Xuelei Fan wrote:

Hi Artem,

What do you think fix the testing infrastructure (JPRT/Mach5) with
proper configuration?  I think it is a easier than update a large
bunch of test codes.

I understand the concerns of yours, but I don't want add additional
effort for those who need to run the test on their own environment.
Some people run the test locally before submit JPRT jobs (or no
access
to JPRT systems for external people).

Update the testing infrastructure may solve both of your concerns and
my concerns.

I'm not sure the @requires tag would work or not.  It would be great
if you can find a solution with the tag.

Thanks,
Xuelei

On 9/29/2016 4:32 AM, Artem Smotrakov wrote:

Hi Xuelei,

I understand your concerns. But I'd prefer to be aware of situations
when a test reports that it passed when it actually did nothing.

How about using @requires key? We can try to specify all expected
platforms. If I understand correctly, jtreg won't run tests if
specified
requirements are not met. In this case, you need to look at the
report
to make sure all tests you are interested in were run.

Artem


On 09/28/2016 08:00 AM, Xuelei Fan wrote:

On 9/28/2016 9:20 PM, Denis Kononenko wrote:

There're 60+ tests related to PKCS11. Two years they have been
"passed" on 3 unsupported platforms on hosts where usually no NSS
libraries were installed. How can we rely on these results?

;-) The words of "unsupported platforms" are very confusing in this
mail thread.

Let's think more about what if a test failed.  What one will do
if a
test failed?
1. Test fail means source code problems for developers. One cannot
submit a change-set if a test failed.  He need to pay additional
effort and analysis the failure.  It take one developer a lot
effort
to know the root cause.  I would never like this unnecessary cost.
2. In order to get the test pass, he need to install the NSS libs
although NSS has nothing to do with his changeset.  It may be a
very
very hard step or even impossible (for example licenses issues)
step
for him.

TBH, I did not see much benefits to fail on unsupported
platforms.  I
agree that skip for pass is not a good idea, but fail to warn is
worse.

I think the root cause if "unsupported platforms" actually are
supported platforms, but by accident the NSS libraries are not
installed or not installed properly.

If one is not interested in NSS, the test get ignored (passed). If
one
is interested in NSS, he should install the NSS libs and the
test get
checked.

What do you think if fix the testing infrastructure with properly
installed NSS libs?

> The problem is the tests report they passed but actually they
were
> skipped. I have no objections against skipping tests but it would
> be better to give a hint somehow how many tests were skipped and
why.
Agreed.  Unfortunately, there are only two options, "pass" or
"fail",
at present.  It would be nice if there is a grey area. Any idea to
make summary of skipped tests and reasons?

Xuelei

Re: [9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Tim Du]

Thanks Xuelei review it.
Latest webrev available here 
:http://cr.openjdk.java.net/~tidu/8164322/webrev.02/ ,Please help to 
review it. Thanks


Regards
Tim
On 2016/9/29 18:16, Xuelei Fan wrote:

On 9/29/2016 5:36 PM, Tim Du wrote:

Thank you for reviewing the codes.
Remove the codes which throw exception and add codes could show machine
not installed NSS library information.
The latest code is here:
http://cr.openjdk.java.net/~tidu/8164322/webrev.01/ ,please help to
review it.


Looks fine to me.  A very minor comment:
 303   + ",please initialize ...

Please add a white space between ",' and 'please".

Thanks,
Xuelei


For whether throw exception caused by OS/machine issue ,I file an
Enhancement https://bugs.openjdk.java.net/browse/JDK-8166893 , we can
keep discussing by it ,and then the test
sun/security/pkcs11/PKCS11Test.java could be supported to run on ARM
platform firstly. Thanks

Regards
Tim
On 2016/9/29 13:05, Xuelei Fan wrote:

On 9/29/2016 9:26 AM, Artem Smotrakov wrote:

Hi Xuelei,

This is not a problem with machine configuration. The actual test 
cases

are not going to be run (even if there are NSS libs on a test machine)
until "osMap" contains an entry for specific platform.


OK.  Please fix for the purpose accordingly.

Tim, the current fix has two parts.  The 1st part, which throw
exception is a machine configuration problem, is not in the scope of
Artem's consideration, please remove it.  The 2nd part, which add new
items to the osMap, is fine to me.  I think we are on the same page
now.  Please update the webrev accordingly.  I will approve the 2nd
part fix.

Thanks,
Xuelei


Artem


On 09/28/2016 05:52 PM, Xuelei Fan wrote:

Hi Artem,

What do you think fix the testing infrastructure (JPRT/Mach5) with
proper configuration?  I think it is a easier than update a large
bunch of test codes.

I understand the concerns of yours, but I don't want add additional
effort for those who need to run the test on their own environment.
Some people run the test locally before submit JPRT jobs (or no 
access

to JPRT systems for external people).

Update the testing infrastructure may solve both of your concerns and
my concerns.

I'm not sure the @requires tag would work or not.  It would be great
if you can find a solution with the tag.

Thanks,
Xuelei

On 9/29/2016 4:32 AM, Artem Smotrakov wrote:

Hi Xuelei,

I understand your concerns. But I'd prefer to be aware of situations
when a test reports that it passed when it actually did nothing.

How about using @requires key? We can try to specify all expected
platforms. If I understand correctly, jtreg won't run tests if
specified
requirements are not met. In this case, you need to look at the 
report

to make sure all tests you are interested in were run.

Artem


On 09/28/2016 08:00 AM, Xuelei Fan wrote:

On 9/28/2016 9:20 PM, Denis Kononenko wrote:

There're 60+ tests related to PKCS11. Two years they have been
"passed" on 3 unsupported platforms on hosts where usually no NSS
libraries were installed. How can we rely on these results?

;-) The words of "unsupported platforms" are very confusing in this
mail thread.

Let's think more about what if a test failed.  What one will do 
if a

test failed?
1. Test fail means source code problems for developers. One cannot
submit a change-set if a test failed.  He need to pay additional
effort and analysis the failure.  It take one developer a lot 
effort

to know the root cause.  I would never like this unnecessary cost.
2. In order to get the test pass, he need to install the NSS libs
although NSS has nothing to do with his changeset.  It may be a 
very
very hard step or even impossible (for example licenses issues) 
step

for him.

TBH, I did not see much benefits to fail on unsupported 
platforms.  I

agree that skip for pass is not a good idea, but fail to warn is
worse.

I think the root cause if "unsupported platforms" actually are
supported platforms, but by accident the NSS libraries are not
installed or not installed properly.

If one is not interested in NSS, the test get ignored (passed). If
one
is interested in NSS, he should install the NSS libs and the 
test get

checked.

What do you think if fix the testing infrastructure with properly
installed NSS libs?

> The problem is the tests report they passed but actually they 
were

> skipped. I have no objections against skipping tests but it would
> be better to give a hint somehow how many tests were skipped and
why.
Agreed.  Unfortunately, there are only two options, "pass" or 
"fail",

at present.  It would be nice if there is a grey area. Any idea to
make summary of skipped tests and reasons?

Xuelei

Re: [9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Xuelei Fan]

On 9/29/2016 5:36 PM, Tim Du wrote:

Thank you for reviewing the codes.
Remove the codes which throw exception and add codes could show machine
not installed NSS library information.
The latest code is here:
http://cr.openjdk.java.net/~tidu/8164322/webrev.01/ ,please help to
review it.


Looks fine to me.  A very minor comment:
 303   + ",please initialize ...

Please add a white space between ",' and 'please".

Thanks,
Xuelei


For whether throw exception caused by OS/machine issue ,I file an
Enhancement https://bugs.openjdk.java.net/browse/JDK-8166893 , we can
keep discussing by it ,and then the test
sun/security/pkcs11/PKCS11Test.java could be supported to run on ARM
platform firstly. Thanks

Regards
Tim
On 2016/9/29 13:05, Xuelei Fan wrote:

On 9/29/2016 9:26 AM, Artem Smotrakov wrote:

Hi Xuelei,

This is not a problem with machine configuration. The actual test cases
are not going to be run (even if there are NSS libs on a test machine)
until "osMap" contains an entry for specific platform.


OK.  Please fix for the purpose accordingly.

Tim, the current fix has two parts.  The 1st part, which throw
exception is a machine configuration problem, is not in the scope of
Artem's consideration, please remove it.  The 2nd part, which add new
items to the osMap, is fine to me.  I think we are on the same page
now.  Please update the webrev accordingly.  I will approve the 2nd
part fix.

Thanks,
Xuelei


Artem


On 09/28/2016 05:52 PM, Xuelei Fan wrote:

Hi Artem,

What do you think fix the testing infrastructure (JPRT/Mach5) with
proper configuration?  I think it is a easier than update a large
bunch of test codes.

I understand the concerns of yours, but I don't want add additional
effort for those who need to run the test on their own environment.
Some people run the test locally before submit JPRT jobs (or no access
to JPRT systems for external people).

Update the testing infrastructure may solve both of your concerns and
my concerns.

I'm not sure the @requires tag would work or not.  It would be great
if you can find a solution with the tag.

Thanks,
Xuelei

On 9/29/2016 4:32 AM, Artem Smotrakov wrote:

Hi Xuelei,

I understand your concerns. But I'd prefer to be aware of situations
when a test reports that it passed when it actually did nothing.

How about using @requires key? We can try to specify all expected
platforms. If I understand correctly, jtreg won't run tests if
specified
requirements are not met. In this case, you need to look at the report
to make sure all tests you are interested in were run.

Artem


On 09/28/2016 08:00 AM, Xuelei Fan wrote:

On 9/28/2016 9:20 PM, Denis Kononenko wrote:

There're 60+ tests related to PKCS11. Two years they have been
"passed" on 3 unsupported platforms on hosts where usually no NSS
libraries were installed. How can we rely on these results?

;-) The words of "unsupported platforms" are very confusing in this
mail thread.

Let's think more about what if a test failed.  What one will do if a
test failed?
1. Test fail means source code problems for developers. One cannot
submit a change-set if a test failed.  He need to pay additional
effort and analysis the failure.  It take one developer a lot effort
to know the root cause.  I would never like this unnecessary cost.
2. In order to get the test pass, he need to install the NSS libs
although NSS has nothing to do with his changeset.  It may be a very
very hard step or even impossible (for example licenses issues) step
for him.

TBH, I did not see much benefits to fail on unsupported platforms.  I
agree that skip for pass is not a good idea, but fail to warn is
worse.

I think the root cause if "unsupported platforms" actually are
supported platforms, but by accident the NSS libraries are not
installed or not installed properly.

If one is not interested in NSS, the test get ignored (passed). If
one
is interested in NSS, he should install the NSS libs and the test get
checked.

What do you think if fix the testing infrastructure with properly
installed NSS libs?

> The problem is the tests report they passed but actually they were
> skipped. I have no objections against skipping tests but it would
> be better to give a hint somehow how many tests were skipped and
why.
Agreed.  Unfortunately, there are only two options, "pass" or "fail",
at present.  It would be nice if there is a grey area. Any idea to
make summary of skipped tests and reasons?

Xuelei

Re: [9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Xuelei Fan]

On 9/29/2016 9:26 AM, Artem Smotrakov wrote:

Hi Xuelei,

This is not a problem with machine configuration. The actual test cases
are not going to be run (even if there are NSS libs on a test machine)
until "osMap" contains an entry for specific platform.


OK.  Please fix for the purpose accordingly.

Tim, the current fix has two parts.  The 1st part, which throw exception 
is a machine configuration problem, is not in the scope of Artem's 
consideration, please remove it.  The 2nd part, which add new items to 
the osMap, is fine to me.  I think we are on the same page now.  Please 
update the webrev accordingly.  I will approve the 2nd part fix.


Thanks,
Xuelei


Artem


On 09/28/2016 05:52 PM, Xuelei Fan wrote:

Hi Artem,

What do you think fix the testing infrastructure (JPRT/Mach5) with
proper configuration?  I think it is a easier than update a large
bunch of test codes.

I understand the concerns of yours, but I don't want add additional
effort for those who need to run the test on their own environment.
Some people run the test locally before submit JPRT jobs (or no access
to JPRT systems for external people).

Update the testing infrastructure may solve both of your concerns and
my concerns.

I'm not sure the @requires tag would work or not.  It would be great
if you can find a solution with the tag.

Thanks,
Xuelei

On 9/29/2016 4:32 AM, Artem Smotrakov wrote:

Hi Xuelei,

I understand your concerns. But I'd prefer to be aware of situations
when a test reports that it passed when it actually did nothing.

How about using @requires key? We can try to specify all expected
platforms. If I understand correctly, jtreg won't run tests if specified
requirements are not met. In this case, you need to look at the report
to make sure all tests you are interested in were run.

Artem


On 09/28/2016 08:00 AM, Xuelei Fan wrote:

On 9/28/2016 9:20 PM, Denis Kononenko wrote:

There're 60+ tests related to PKCS11. Two years they have been
"passed" on 3 unsupported platforms on hosts where usually no NSS
libraries were installed. How can we rely on these results?

;-) The words of "unsupported platforms" are very confusing in this
mail thread.

Let's think more about what if a test failed.  What one will do if a
test failed?
1. Test fail means source code problems for developers.  One cannot
submit a change-set if a test failed.  He need to pay additional
effort and analysis the failure.  It take one developer a lot effort
to know the root cause.  I would never like this unnecessary cost.
2. In order to get the test pass, he need to install the NSS libs
although NSS has nothing to do with his changeset.  It may be a very
very hard step or even impossible (for example licenses issues) step
for him.

TBH, I did not see much benefits to fail on unsupported platforms.  I
agree that skip for pass is not a good idea, but fail to warn is worse.

I think the root cause if "unsupported platforms" actually are
supported platforms, but by accident the NSS libraries are not
installed or not installed properly.

If one is not interested in NSS, the test get ignored (passed). If one
is interested in NSS, he should install the NSS libs and the test get
checked.

What do you think if fix the testing infrastructure with properly
installed NSS libs?

> The problem is the tests report they passed but actually they were
> skipped. I have no objections against skipping tests but it would
> be better to give a hint somehow how many tests were skipped and why.
Agreed.  Unfortunately, there are only two options, "pass" or "fail",
at present.  It would be nice if there is a grey area. Any idea to
make summary of skipped tests and reasons?

Xuelei

Re: [9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Xuelei Fan]

Hi Artem,

What do you think fix the testing infrastructure (JPRT/Mach5) with 
proper configuration?  I think it is a easier than update a large bunch 
of test codes.


I understand the concerns of yours, but I don't want add additional 
effort for those who need to run the test on their own environment. 
Some people run the test locally before submit JPRT jobs (or no access 
to JPRT systems for external people).


Update the testing infrastructure may solve both of your concerns and my 
concerns.


I'm not sure the @requires tag would work or not.  It would be great if 
you can find a solution with the tag.


Thanks,
Xuelei

On 9/29/2016 4:32 AM, Artem Smotrakov wrote:

Hi Xuelei,

I understand your concerns. But I'd prefer to be aware of situations
when a test reports that it passed when it actually did nothing.

How about using @requires key? We can try to specify all expected
platforms. If I understand correctly, jtreg won't run tests if specified
requirements are not met. In this case, you need to look at the report
to make sure all tests you are interested in were run.

Artem


On 09/28/2016 08:00 AM, Xuelei Fan wrote:

On 9/28/2016 9:20 PM, Denis Kononenko wrote:

There're 60+ tests related to PKCS11. Two years they have been
"passed" on 3 unsupported platforms on hosts where usually no NSS
libraries were installed. How can we rely on these results?

;-) The words of "unsupported platforms" are very confusing in this
mail thread.

Let's think more about what if a test failed.  What one will do if a
test failed?
1. Test fail means source code problems for developers.  One cannot
submit a change-set if a test failed.  He need to pay additional
effort and analysis the failure.  It take one developer a lot effort
to know the root cause.  I would never like this unnecessary cost.
2. In order to get the test pass, he need to install the NSS libs
although NSS has nothing to do with his changeset.  It may be a very
very hard step or even impossible (for example licenses issues) step
for him.

TBH, I did not see much benefits to fail on unsupported platforms.  I
agree that skip for pass is not a good idea, but fail to warn is worse.

I think the root cause if "unsupported platforms" actually are
supported platforms, but by accident the NSS libraries are not
installed or not installed properly.

If one is not interested in NSS, the test get ignored (passed). If one
is interested in NSS, he should install the NSS libs and the test get
checked.

What do you think if fix the testing infrastructure with properly
installed NSS libs?

> The problem is the tests report they passed but actually they were
> skipped. I have no objections against skipping tests but it would
> be better to give a hint somehow how many tests were skipped and why.
Agreed.  Unfortunately, there are only two options, "pass" or "fail",
at present.  It would be nice if there is a grey area. Any idea to
make summary of skipped tests and reasons?

Xuelei

Re: [9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Artem Smotrakov]

Hi Xuelei,

I understand your concerns. But I'd prefer to be aware of situations 
when a test reports that it passed when it actually did nothing.


How about using @requires key? We can try to specify all expected 
platforms. If I understand correctly, jtreg won't run tests if specified 
requirements are not met. In this case, you need to look at the report 
to make sure all tests you are interested in were run.


Artem


On 09/28/2016 08:00 AM, Xuelei Fan wrote:

On 9/28/2016 9:20 PM, Denis Kononenko wrote:
There're 60+ tests related to PKCS11. Two years they have been 
"passed" on 3 unsupported platforms on hosts where usually no NSS 
libraries were installed. How can we rely on these results?
;-) The words of "unsupported platforms" are very confusing in this 
mail thread.


Let's think more about what if a test failed.  What one will do if a 
test failed?
1. Test fail means source code problems for developers.  One cannot 
submit a change-set if a test failed.  He need to pay additional 
effort and analysis the failure.  It take one developer a lot effort 
to know the root cause.  I would never like this unnecessary cost.
2. In order to get the test pass, he need to install the NSS libs 
although NSS has nothing to do with his changeset.  It may be a very 
very hard step or even impossible (for example licenses issues) step 
for him.


TBH, I did not see much benefits to fail on unsupported platforms.  I 
agree that skip for pass is not a good idea, but fail to warn is worse.


I think the root cause if "unsupported platforms" actually are 
supported platforms, but by accident the NSS libraries are not 
installed or not installed properly.


If one is not interested in NSS, the test get ignored (passed). If one 
is interested in NSS, he should install the NSS libs and the test get 
checked.


What do you think if fix the testing infrastructure with properly 
installed NSS libs?


> The problem is the tests report they passed but actually they were
> skipped. I have no objections against skipping tests but it would
> be better to give a hint somehow how many tests were skipped and why.
Agreed.  Unfortunately, there are only two options, "pass" or "fail", 
at present.  It would be nice if there is a grey area. Any idea to 
make summary of skipped tests and reasons?


Xuelei

Re: [9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Xuelei Fan]

I agree with the update for that part to add new items.  But please 
don't throw exception if NSS libs was not found (that's another part of 
the fix).


Xuelei

On 9/28/2016 8:22 AM, Artem Smotrakov wrote:

Right, thank you Max.

Currently PKCS11 tests are seen as passed on unsupported (by the tests)
platforms, but actually nothing was tested. We'd better know about it.

Artem


On 09/27/2016 05:04 PM, Wang Weijun wrote:

On Sep 28, 2016, at 7:59 AM, Xuelei Fan  wrote:


Currently the tests silently quit which looks like they pass. This
makes
people think that everything went smoothly, but actually nothing was
tested.


I did not get the idea.

I think what Artem meant is that without the platform name in osMap,
the platform is not tested at all. If the platform has NSS libs, we
won't be able to know whether it works.

--Max

Re: [9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Wang Weijun]

> On Sep 28, 2016, at 7:59 AM, Xuelei Fan  wrote:
> 
>> Currently the tests silently quit which looks like they pass. This makes
>> people think that everything went smoothly, but actually nothing was
>> tested.
>> 
> I did not get the idea.

I think what Artem meant is that without the platform name in osMap, the 
platform is not tested at all. If the platform has NSS libs, we won't be able 
to know whether it works.

--Max

Re: [9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Valerie Peng]

I think it makes sense to error out on unknown platforms.
This doesn't conflict with skipping tests (or silently passing) for 
unsupported platforms.
We just need a way to distinguish them. I'd probably go for the same 
approach that Max suggested, i.e. add the list of (known) unsupported 
platforms to osMap and errors out if the requested platform cannot be 
found in osMap.


Valerie

On 9/27/2016 7:22 AM, Wang Weijun wrote:

Looking at the webrev, it looks we've never tested on "Linux-arm-32" and 
"Linux-aarch64-64" before and we only realized it now. This is a true problem.

On the other hand, I also agree with Xuelei's concern. If a new platform is 
added and it does not have NSS libs tests will fail.

How about this? If there is such a new platform called "mhos-arch-32", we can 
add

osMap("mhos-arch-32", new String[]{});

and, make nssLibDirs == null a failure, but nssLibDirs.length == 0 can return 
null.

Is this good?

Thanks
Max



On Sep 27, 2016, at 7:25 PM, Xuelei Fan  wrote:

I think, it is the expected behavior to ignore the test if a platform does not 
support it.  If showing failures, every testing on unsupported platform will 
fail, and additional effort MUST be paid to evaluate the root cause of the 
failure.  We should try to avoid that.

Xuelei

On 9/27/2016 6:32 PM, Tim Du wrote:

Hi All:

Would you help to review the patch for sun/security/pkcs11/PKCS11Test.java?
The test keep pass on not supported platforms, it will make nobody
notice the test was skipped,which is not our expected. Update case to
show failure, when platform not supported. And add the support for
Linux-arm-32 and Linux-arm-64 platforms. Thanks.

JBS: https://bugs.openjdk.java.net/browse/JDK-8164322
Webrev: http://cr.openjdk.java.net/~tidu/8164322/webrev.00/


Regards
Tim

Re: [9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Artem Smotrakov]

(cc'ing Denis who reported the bug)

I think making PKCS11 tests fail on unexpected platform would be helpful 
for people who port JDK on new platforms and run tests on them. 
Currently the tests silently quit which looks like they pass. This makes 
people think that everything went smoothly, but actually nothing was tested.


I would prefer to update PKCS11Test to report a failure in case of 
unexpected platform.


But before that it would be good to make sure that PKCS11 tests work 
fine on those generic platforms which are listed in "osMap".


Tim,

Did you run PKCS11 tests on those platforms? (it might be good to submit 
multiple runs).


One minor comment, could you please check that indentation is correct in 
lines 625-629?


Artem

On 09/27/2016 07:22 AM, Wang Weijun wrote:

Looking at the webrev, it looks we've never tested on "Linux-arm-32" and 
"Linux-aarch64-64" before and we only realized it now. This is a true problem.

On the other hand, I also agree with Xuelei's concern. If a new platform is 
added and it does not have NSS libs tests will fail.

How about this? If there is such a new platform called "mhos-arch-32", we can 
add

osMap("mhos-arch-32", new String[]{});

and, make nssLibDirs == null a failure, but nssLibDirs.length == 0 can return 
null.

Is this good?

Thanks
Max



On Sep 27, 2016, at 7:25 PM, Xuelei Fan  wrote:

I think, it is the expected behavior to ignore the test if a platform does not 
support it.  If showing failures, every testing on unsupported platform will 
fail, and additional effort MUST be paid to evaluate the root cause of the 
failure.  We should try to avoid that.

Xuelei

On 9/27/2016 6:32 PM, Tim Du wrote:

Hi All:

Would you help to review the patch for sun/security/pkcs11/PKCS11Test.java?
The test keep pass on not supported platforms, it will make nobody
notice the test was skipped,which is not our expected. Update case to
show failure, when platform not supported. And add the support for
Linux-arm-32 and Linux-arm-64 platforms. Thanks.

JBS: https://bugs.openjdk.java.net/browse/JDK-8164322
Webrev: http://cr.openjdk.java.net/~tidu/8164322/webrev.00/


Regards
Tim

Re: [9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Wang Weijun]

Looking at the webrev, it looks we've never tested on "Linux-arm-32" and 
"Linux-aarch64-64" before and we only realized it now. This is a true problem.

On the other hand, I also agree with Xuelei's concern. If a new platform is 
added and it does not have NSS libs tests will fail.

How about this? If there is such a new platform called "mhos-arch-32", we can 
add

   osMap("mhos-arch-32", new String[]{});

and, make nssLibDirs == null a failure, but nssLibDirs.length == 0 can return 
null.

Is this good?

Thanks
Max


> On Sep 27, 2016, at 7:25 PM, Xuelei Fan  wrote:
> 
> I think, it is the expected behavior to ignore the test if a platform does 
> not support it.  If showing failures, every testing on unsupported platform 
> will fail, and additional effort MUST be paid to evaluate the root cause of 
> the failure.  We should try to avoid that.
> 
> Xuelei
> 
> On 9/27/2016 6:32 PM, Tim Du wrote:
>> Hi All:
>> 
>> Would you help to review the patch for sun/security/pkcs11/PKCS11Test.java?
>> The test keep pass on not supported platforms, it will make nobody
>> notice the test was skipped,which is not our expected. Update case to
>> show failure, when platform not supported. And add the support for
>> Linux-arm-32 and Linux-arm-64 platforms. Thanks.
>> 
>> JBS: https://bugs.openjdk.java.net/browse/JDK-8164322
>> Webrev: http://cr.openjdk.java.net/~tidu/8164322/webrev.00/
>> 
>> 
>> Regards
>> Tim

Re: [9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Xuelei Fan]

I think, it is the expected behavior to ignore the test if a platform 
does not support it.  If showing failures, every testing on unsupported 
platform will fail, and additional effort MUST be paid to evaluate the 
root cause of the failure.  We should try to avoid that.


Xuelei

On 9/27/2016 6:32 PM, Tim Du wrote:

Hi All:

Would you help to review the patch for sun/security/pkcs11/PKCS11Test.java?
The test keep pass on not supported platforms, it will make nobody
notice the test was skipped,which is not our expected. Update case to
show failure, when platform not supported. And add the support for
Linux-arm-32 and Linux-arm-64 platforms. Thanks.

JBS: https://bugs.openjdk.java.net/browse/JDK-8164322
Webrev: http://cr.openjdk.java.net/~tidu/8164322/webrev.00/


Regards
Tim

[9] RFR: JDK-8164322: sun/security/pkcs11/PKCS11Test.java shall be updated to run on ARM platforms

[Tim Du]

Hi All:

Would you help to review the patch for sun/security/pkcs11/PKCS11Test.java?
The test keep pass on not supported platforms, it will make nobody 
notice the test was skipped,which is not our expected. Update case to 
show failure, when platform not supported. And add the support for  
Linux-arm-32 and Linux-arm-64 platforms. Thanks.


JBS: https://bugs.openjdk.java.net/browse/JDK-8164322
Webrev: http://cr.openjdk.java.net/~tidu/8164322/webrev.00/


Regards
Tim