Re: javac server: No port file values materialized. on AIX

[Thomas Stüfe]

Hi Andreas,

thanks! I will use this patch on my AIX builds. I will come back to you if
the error resurfaces.

Kind Regards, Thoams

On Sun, Jan 24, 2016 at 12:33 PM, Andreas Lundblad <
andreas.lundb...@oracle.com> wrote:

> On Mon, Jan 18, 2016 at 03:39:39PM +0100, Volker Simonis wrote:
> > On Mon, Jan 18, 2016 at 3:34 PM, Andreas Lundblad
> >  wrote:
> > >> >> Interesting observation. The code for waiting for valid port file
> values
> > >> >> basically looks like
> > >> >>
> > >> >>  for (int i = 0; i < 10; i++) {
> > >> >>  checkPortFile();
> > >> >>  if (successful)
> > >> >>  break;
> > >> >>  sleep(500);
> > >> >>  }
> > >> >>
> > >> >> so the fact that it even reaches 6676 ms looks suspicious when it
> comes to
> > >> >> load.
> > >> >
> > >> >
> > >> > Why? Under load those sleep(500)'s might not return for much
> longer; and the
> > >> > whole things might be time preempted at any point for an extended
> period of
> > >> > time.
> > >
> > > What I meant was that the fact that the code takes 6676 ms to complete
> increases my suspicion about it being due to a load issue.
> > >
> > >
> > >> What about putting another loop around this loop which prints a
> > >> warning to stdout (e.g. "..trying to connect to sjavac server since X
> > >> seconds") for another five or so times. We could also print the system
> > >> load [1] although I'm not sure it's worth it. On our AIX machines it
> > >> is often a network/NFS problem which causes long startup times of new
> > >> executables and this won't be observable by looking at the system load
> > >> (but it may at least give a hint').
> > >>
> > >> [1]
> http://docs.oracle.com/javase/6/docs/api/java/lang/management/OperatingSystemMXBean.html#getSystemLoadAverage%28%29
> > >
> > > I don't think a loop around the loop is necessary. The actual code
> (which is slightly different from the snippet I posted above) already
> prints a message between each attempt.
> > >
> > > I was thinking of just bumping the timeout from 5 seconds to, say, 60
> seconds. If it's a load issue, we should se something like "Port file
> values found after 9000 ms", in which case we know for sure that it was a
> premature timeout issue. If no port files materialize after >60 seconds, we
> can probably safely assume that the issue is due to something else.
> > >
> >
> > Sounds good. Let's try it.
>
> Patch has now been pushed [1].
>
> Please let me know if any of you still experience the "No portfile values
> materialized" problem.
>
> -- Andreas
>
> [1] http://hg.openjdk.java.net/jdk9/dev/langtools/rev/3e4edb085bf0
>

Re: javac server: No port file values materialized. on AIX

[Andreas Lundblad]

On Mon, Jan 18, 2016 at 03:39:39PM +0100, Volker Simonis wrote:
> On Mon, Jan 18, 2016 at 3:34 PM, Andreas Lundblad
>  wrote:
> >> >> Interesting observation. The code for waiting for valid port file values
> >> >> basically looks like
> >> >>
> >> >>  for (int i = 0; i < 10; i++) {
> >> >>  checkPortFile();
> >> >>  if (successful)
> >> >>  break;
> >> >>  sleep(500);
> >> >>  }
> >> >>
> >> >> so the fact that it even reaches 6676 ms looks suspicious when it comes 
> >> >> to
> >> >> load.
> >> >
> >> >
> >> > Why? Under load those sleep(500)'s might not return for much longer; and 
> >> > the
> >> > whole things might be time preempted at any point for an extended period 
> >> > of
> >> > time.
> >
> > What I meant was that the fact that the code takes 6676 ms to complete 
> > increases my suspicion about it being due to a load issue.
> >
> >
> >> What about putting another loop around this loop which prints a
> >> warning to stdout (e.g. "..trying to connect to sjavac server since X
> >> seconds") for another five or so times. We could also print the system
> >> load [1] although I'm not sure it's worth it. On our AIX machines it
> >> is often a network/NFS problem which causes long startup times of new
> >> executables and this won't be observable by looking at the system load
> >> (but it may at least give a hint').
> >>
> >> [1] 
> >> http://docs.oracle.com/javase/6/docs/api/java/lang/management/OperatingSystemMXBean.html#getSystemLoadAverage%28%29
> >
> > I don't think a loop around the loop is necessary. The actual code (which 
> > is slightly different from the snippet I posted above) already prints a 
> > message between each attempt.
> >
> > I was thinking of just bumping the timeout from 5 seconds to, say, 60 
> > seconds. If it's a load issue, we should se something like "Port file 
> > values found after 9000 ms", in which case we know for sure that it was a 
> > premature timeout issue. If no port files materialize after >60 seconds, we 
> > can probably safely assume that the issue is due to something else.
> >
> 
> Sounds good. Let's try it.

Patch has now been pushed [1].

Please let me know if any of you still experience the "No portfile values 
materialized" problem.

-- Andreas

[1] http://hg.openjdk.java.net/jdk9/dev/langtools/rev/3e4edb085bf0

Re: javac server: No port file values materialized. on AIX

[Severin Gehwolf]

Hi,

I'm doubtful this is an issue specific to AIX since I'm seing this now
on a Linux box. See below.

On Mon, 2016-01-18 at 15:39 +0100, Volker Simonis wrote:
> On Mon, Jan 18, 2016 at 3:34 PM, Andreas Lundblad
>  wrote:
> > > > > Interesting observation. The code for waiting for valid port file 
> > > > > values
> > > > > basically looks like
> > > > > 
> > > > >  for (int i = 0; i < 10; i++) {
> > > > >  checkPortFile();
> > > > >  if (successful)
> > > > >  break;
> > > > >  sleep(500);
> > > > >  }
> > > > > 
> > > > > so the fact that it even reaches 6676 ms looks suspicious when it 
> > > > > comes to
> > > > > load.
> > > > 
> > > > 
> > > > Why? Under load those sleep(500)'s might not return for much longer; 
> > > > and the
> > > > whole things might be time preempted at any point for an extended 
> > > > period of
> > > > time.
> > 
> > What I meant was that the fact that the code takes 6676 ms to complete 
> > increases my suspicion about it being due to a load issue.

FWIW, I'm seeing this issue now on the Zero builder:
http://builder.classpath.org/jenkins/job/OpenJDK9_hs_rt_Zero/203/console

The builder might be under load which could cause this, but
nevertheless it's the first time I'm seeing this. Has the above code
been introduced recently?

Is there a bug I can CC myself to?

Thanks,
Severin

Re: javac server: No port file values materialized. on AIX

[Andreas Lundblad]

On Wed, Jan 20, 2016 at 10:43:47AM +0100, Severin Gehwolf wrote:
> Hi,
> 
> I'm doubtful this is an issue specific to AIX since I'm seing this now
> on a Linux box. See below.
> 
> On Mon, 2016-01-18 at 15:39 +0100, Volker Simonis wrote:
> > On Mon, Jan 18, 2016 at 3:34 PM, Andreas Lundblad
> >  wrote:
> > > > > > Interesting observation. The code for waiting for valid port file 
> > > > > > values
> > > > > > basically looks like
> > > > > > 
> > > > > >  for (int i = 0; i < 10; i++) {
> > > > > >  checkPortFile();
> > > > > >  if (successful)
> > > > > >  break;
> > > > > >  sleep(500);
> > > > > >  }
> > > > > > 
> > > > > > so the fact that it even reaches 6676 ms looks suspicious when it 
> > > > > > comes to
> > > > > > load.
> > > > > 
> > > > > 
> > > > > Why? Under load those sleep(500)'s might not return for much longer; 
> > > > > and the
> > > > > whole things might be time preempted at any point for an extended 
> > > > > period of
> > > > > time.
> > > 
> > > What I meant was that the fact that the code takes 6676 ms to complete 
> > > increases my suspicion about it being due to a load issue.
> 
> FWIW, I'm seeing this issue now on the Zero builder:
> http://builder.classpath.org/jenkins/job/OpenJDK9_hs_rt_Zero/203/console
> 
> The builder might be under load which could cause this, but
> nevertheless it's the first time I'm seeing this. Has the above code
> been introduced recently?
 
The code has been there for a while, but it has only been activated by default 
recently.

> Is there a bug I can CC myself to?

Yes: JDK-8145392

A patch which bumps the default from 5 seconds to 60 seconds is currently under 
review.

-- Andreas

Re: javac server: No port file values materialized. on AIX

[David Holmes]

On 20/01/2016 7:43 PM, Severin Gehwolf wrote:

Hi,

I'm doubtful this is an issue specific to AIX since I'm seing this now
on a Linux box. See below.


It isn't specific to AIX. As Erik said earlier in the thread we see this 
occasionally on Solaris too.


David



On Mon, 2016-01-18 at 15:39 +0100, Volker Simonis wrote:

On Mon, Jan 18, 2016 at 3:34 PM, Andreas Lundblad
 wrote:

Interesting observation. The code for waiting for valid port file values
basically looks like

  for (int i = 0; i < 10; i++) {
  checkPortFile();
  if (successful)
  break;
  sleep(500);
  }

so the fact that it even reaches 6676 ms looks suspicious when it comes to
load.



Why? Under load those sleep(500)'s might not return for much longer; and the
whole things might be time preempted at any point for an extended period of
time.


What I meant was that the fact that the code takes 6676 ms to complete 
increases my suspicion about it being due to a load issue.


FWIW, I'm seeing this issue now on the Zero builder:
http://builder.classpath.org/jenkins/job/OpenJDK9_hs_rt_Zero/203/console

The builder might be under load which could cause this, but
nevertheless it's the first time I'm seeing this. Has the above code
been introduced recently?

Is there a bug I can CC myself to?

Thanks,
Severin

Re: javac server: No port file values materialized. on AIX

[Severin Gehwolf]

On Wed, 2016-01-20 at 14:04 +0100, Andreas Lundblad wrote:
> > 
> > FWIW, I'm seeing this issue now on the Zero builder:
> > http://builder.classpath.org/jenkins/job/OpenJDK9_hs_rt_Zero/203/console
> > 
> > The builder might be under load which could cause this, but
> > nevertheless it's the first time I'm seeing this. Has the above code
> > been introduced recently?
>  
> The code has been there for a while, but it has only been activated by 
> default recently.
> 
> > Is there a bug I can CC myself to?
> 
> Yes: JDK-8145392
> 
> A patch which bumps the default from 5 seconds to 60 seconds is currently 
> under review.

OK, thanks.

Cheers,
Severin

Re: javac server: No port file values materialized. on AIX

[Andreas Lundblad]

> >> Interesting observation. The code for waiting for valid port file values
> >> basically looks like
> >>
> >>  for (int i = 0; i < 10; i++) {
> >>  checkPortFile();
> >>  if (successful)
> >>  break;
> >>  sleep(500);
> >>  }
> >>
> >> so the fact that it even reaches 6676 ms looks suspicious when it comes to
> >> load.
> >
> >
> > Why? Under load those sleep(500)'s might not return for much longer; and the
> > whole things might be time preempted at any point for an extended period of
> > time.

What I meant was that the fact that the code takes 6676 ms to complete 
increases my suspicion about it being due to a load issue.


> What about putting another loop around this loop which prints a
> warning to stdout (e.g. "..trying to connect to sjavac server since X
> seconds") for another five or so times. We could also print the system
> load [1] although I'm not sure it's worth it. On our AIX machines it
> is often a network/NFS problem which causes long startup times of new
> executables and this won't be observable by looking at the system load
> (but it may at least give a hint').
> 
> [1] 
> http://docs.oracle.com/javase/6/docs/api/java/lang/management/OperatingSystemMXBean.html#getSystemLoadAverage%28%29

I don't think a loop around the loop is necessary. The actual code (which is 
slightly different from the snippet I posted above) already prints a message 
between each attempt.

I was thinking of just bumping the timeout from 5 seconds to, say, 60 seconds. 
If it's a load issue, we should se something like "Port file values found after 
9000 ms", in which case we know for sure that it was a premature timeout issue. 
If no port files materialize after >60 seconds, we can probably safely assume 
that the issue is due to something else.

-- Andreas

Re: javac server: No port file values materialized. on AIX

[Volker Simonis]

On Mon, Jan 18, 2016 at 3:34 PM, Andreas Lundblad
 wrote:
>> >> Interesting observation. The code for waiting for valid port file values
>> >> basically looks like
>> >>
>> >>  for (int i = 0; i < 10; i++) {
>> >>  checkPortFile();
>> >>  if (successful)
>> >>  break;
>> >>  sleep(500);
>> >>  }
>> >>
>> >> so the fact that it even reaches 6676 ms looks suspicious when it comes to
>> >> load.
>> >
>> >
>> > Why? Under load those sleep(500)'s might not return for much longer; and 
>> > the
>> > whole things might be time preempted at any point for an extended period of
>> > time.
>
> What I meant was that the fact that the code takes 6676 ms to complete 
> increases my suspicion about it being due to a load issue.
>
>
>> What about putting another loop around this loop which prints a
>> warning to stdout (e.g. "..trying to connect to sjavac server since X
>> seconds") for another five or so times. We could also print the system
>> load [1] although I'm not sure it's worth it. On our AIX machines it
>> is often a network/NFS problem which causes long startup times of new
>> executables and this won't be observable by looking at the system load
>> (but it may at least give a hint').
>>
>> [1] 
>> http://docs.oracle.com/javase/6/docs/api/java/lang/management/OperatingSystemMXBean.html#getSystemLoadAverage%28%29
>
> I don't think a loop around the loop is necessary. The actual code (which is 
> slightly different from the snippet I posted above) already prints a message 
> between each attempt.
>
> I was thinking of just bumping the timeout from 5 seconds to, say, 60 
> seconds. If it's a load issue, we should se something like "Port file values 
> found after 9000 ms", in which case we know for sure that it was a premature 
> timeout issue. If no port files materialize after >60 seconds, we can 
> probably safely assume that the issue is due to something else.
>

Sounds good. Let's try it.

> -- Andreas

Re: javac server: No port file values materialized. on AIX

[Volker Simonis]

On Mon, Jan 18, 2016 at 8:35 AM, David Holmes  wrote:
> On 18/01/2016 3:08 AM, Andreas Lundblad wrote:
>>
>> On Fri, Jan 15, 2016 at 10:59:08AM +0100, Volker Simonis wrote:
>>>
>>> Maybe the timeout of five seconds is too small?
>>>
>>> Our AIX boxes are not the fastest and we also have a lot of stuff on NFS
>>> shares.
>>>
>>> I recently saw this one:
>>>
>>> [CLIENT] Exception caught: java.io.IOException: No port file values
>>> materialized. Giving up after 6676 ms
>>>
>>> Normally the timeout should be just about a little more than 5 seconds
>>> but here the exception reports more than 6 seconds which might be a
>>> hint that the machine was severely overloaded.
>>>
>>> Regards,
>>> Volker
>>
>>
>> Interesting observation. The code for waiting for valid port file values
>> basically looks like
>>
>>  for (int i = 0; i < 10; i++) {
>>  checkPortFile();
>>  if (successful)
>>  break;
>>  sleep(500);
>>  }
>>
>> so the fact that it even reaches 6676 ms looks suspicious when it comes to
>> load.
>
>
> Why? Under load those sleep(500)'s might not return for much longer; and the
> whole things might be time preempted at any point for an extended period of
> time.

Yes, exactly.

What about putting another loop around this loop which prints a
warning to stdout (e.g. "..trying to connect to sjavac server since X
seconds") for another five or so times. We could also print the system
load [1] although I'm not sure it's worth it. On our AIX machines it
is often a network/NFS problem which causes long startup times of new
executables and this won't be observable by looking at the system load
(but it may at least give a hint').

[1] 
http://docs.oracle.com/javase/6/docs/api/java/lang/management/OperatingSystemMXBean.html#getSystemLoadAverage%28%29

>
> David
> -
>
>
>> -- Andreas
>>
>

Re: javac server: No port file values materialized. on AIX

[David Holmes]

On 18/01/2016 3:08 AM, Andreas Lundblad wrote:

On Fri, Jan 15, 2016 at 10:59:08AM +0100, Volker Simonis wrote:

Maybe the timeout of five seconds is too small?

Our AIX boxes are not the fastest and we also have a lot of stuff on NFS shares.

I recently saw this one:

[CLIENT] Exception caught: java.io.IOException: No port file values
materialized. Giving up after 6676 ms

Normally the timeout should be just about a little more than 5 seconds
but here the exception reports more than 6 seconds which might be a
hint that the machine was severely overloaded.

Regards,
Volker


Interesting observation. The code for waiting for valid port file values 
basically looks like

 for (int i = 0; i < 10; i++) {
 checkPortFile();
 if (successful)
 break;
 sleep(500);
 }

so the fact that it even reaches 6676 ms looks suspicious when it comes to load.


Why? Under load those sleep(500)'s might not return for much longer; and 
the whole things might be time preempted at any point for an extended 
period of time.


David
-



-- Andreas

Re: javac server: No port file values materialized. on AIX

[Andreas Lundblad]

On Fri, Jan 15, 2016 at 10:59:08AM +0100, Volker Simonis wrote:
> Maybe the timeout of five seconds is too small?
> 
> Our AIX boxes are not the fastest and we also have a lot of stuff on NFS 
> shares.
> 
> I recently saw this one:
> 
> [CLIENT] Exception caught: java.io.IOException: No port file values
> materialized. Giving up after 6676 ms
> 
> Normally the timeout should be just about a little more than 5 seconds
> but here the exception reports more than 6 seconds which might be a
> hint that the machine was severely overloaded.
> 
> Regards,
> Volker

Interesting observation. The code for waiting for valid port file values 
basically looks like

for (int i = 0; i < 10; i++) {
checkPortFile();
if (successful)
break;
sleep(500);
}

so the fact that it even reaches 6676 ms looks suspicious when it comes to load.

-- Andreas

Re: javac server: No port file values materialized. on AIX

[Volker Simonis]

Maybe the timeout of five seconds is too small?

Our AIX boxes are not the fastest and we also have a lot of stuff on NFS shares.

I recently saw this one:

[CLIENT] Exception caught: java.io.IOException: No port file values
materialized. Giving up after 6676 ms

Normally the timeout should be just about a little more than 5 seconds
but here the exception reports more than 6 seconds which might be a
hint that the machine was severely overloaded.

Regards,
Volker


On Thu, Jan 14, 2016 at 1:23 PM, Andreas Lundblad
 wrote:
> On Thu, Jan 14, 2016 at 09:06:56AM +0100, Erik Joelsson wrote:
>> There is a bug for that as we see the same thing on Solaris from
>> time to time, as I see you know. Andreas, who is the main engineer
>> working on sjavac, is currently on vacation. We don't currently know
>> the cause of this issue. Any help would be appreciated.
>>
>> /Erik
>
> As Erik said, I'm on vacation until Monday.
>
> This issue is however currently my top priority. I'm thankful that you let me 
> know that the fix for JDK-8145944 did not solve JDK-8145392.
>
> best regards,
> Andreas

Re: javac server: No port file values materialized. on AIX

[Erik Joelsson]

There is a bug for that as we see the same thing on Solaris from time to 
time, as I see you know. Andreas, who is the main engineer working on 
sjavac, is currently on vacation. We don't currently know the cause of 
this issue. Any help would be appreciated.


/Erik

On 2016-01-14 08:29, Thomas Stüfe wrote:

Hi all,

on AIX, I keep getting

Server log:
--- Server log start ---
--- Server log end -
[CLIENT] Exception caught: java.io.IOException: No port file values
materialized. Giving up after 5375 ms
java.io.IOException: No port file values materialized. Giving up after 5375
ms
 at
com.sun.tools.sjavac.server.PortFile.waitForValidValues(PortFile.java:242)
 at
com.sun.tools.sjavac.client.SjavacClient.fork(SjavacClient.java:262)
 at
com.sun.tools.sjavac.client.SjavacClient.makeSureServerIsRunning(SjavacClient.java:216)
 at
com.sun.tools.sjavac.client.SjavacClient.tryConnect(SjavacClient.java:173)
 at
com.sun.tools.sjavac.client.SjavacClient.compile(SjavacClient.java:126)
 at com.sun.tools.sjavac.client.ClientMain.run(ClientMain.java:84)
 at com.sun.tools.sjavac.client.ClientMain.run(ClientMain.java:47)
 at com.sun.tools.sjavac.Main.go(Main.java:56)
 at com.sun.tools.sjavac.Main.main(Main.java:46)

javac server issues when building.

I build hs-rt, and I manually applied the patch from

https://bugs.openjdk.java.net/browse/JDK-8145944

which fixes the windows-delayed-delete-issue, but that (obviously) does not
help on AIX.

My current workaround is --disable-javac-server. But I would prefer running
with javac server, my AIX build machine is slow enough as it is :)

Any suggestions?

Thanks! Kind Regards, Thomas

Re: javac server: No port file values materialized. on AIX

[Andreas Lundblad]

On Thu, Jan 14, 2016 at 09:06:56AM +0100, Erik Joelsson wrote:
> There is a bug for that as we see the same thing on Solaris from
> time to time, as I see you know. Andreas, who is the main engineer
> working on sjavac, is currently on vacation. We don't currently know
> the cause of this issue. Any help would be appreciated.
> 
> /Erik

As Erik said, I'm on vacation until Monday.

This issue is however currently my top priority. I'm thankful that you let me 
know that the fix for JDK-8145944 did not solve JDK-8145392.

best regards,
Andreas

Re: javac server: No port file values materialized. on AIX

[Thomas Stüfe]

Thanks, Andreas,

well, we do have a workaround. I'll wait until you find out more. Have a
nice rest-of-vacation!

Kind Regards, Thomas

On Thu, Jan 14, 2016 at 1:23 PM, Andreas Lundblad <
andreas.lundb...@oracle.com> wrote:

> On Thu, Jan 14, 2016 at 09:06:56AM +0100, Erik Joelsson wrote:
> > There is a bug for that as we see the same thing on Solaris from
> > time to time, as I see you know. Andreas, who is the main engineer
> > working on sjavac, is currently on vacation. We don't currently know
> > the cause of this issue. Any help would be appreciated.
> >
> > /Erik
>
> As Erik said, I'm on vacation until Monday.
>
> This issue is however currently my top priority. I'm thankful that you let
> me know that the fix for JDK-8145944 did not solve JDK-8145392.
>
> best regards,
> Andreas
>