Re: [cross-project-issues-dev] hipp9 overloaded

2017-02-05 Thread Ed Willink 

Hi

On the 'ocl' hipp, I have had to increase the timeout on a 12 minute job 
from 60 minutes to 180 to get it to run. This at 6:00 GMT on a Sunday!


While observing the progress interactively, it is my impression that the 
JUnit tests are not dramatically slower, but anything involving repos or 
GIT, presumably the network, crawls.


Regards

Ed Willink

On 05/02/2017 14:24, Stephan Herrmann wrote:

Could this be a general issue?
The factor of 4 looks similar to what I see in hipp10 (Object Teams).

Some figures:
Two individual test suites A and B as part of the Object Teams build 
over time

(from some builds which I happened to keep):

up-to 2016-12-08:
suite A: around 30 min.
suite B: around 30 min.

2017-01-18:
suite A at 47 min.
suite B presumably timed out after 120 min (details no longer 
available)


2017-01-22:
suite A at 44 min.
suite B completed at 110 min

2017-01-27:
suite A at 60 min.
suite B presumably timed out after 120 min (details no longer 
available)


2017-02-04:
suite A timed out at 120 min
suite B died at 50 min (not sure if this could possibly be a time 
out)


Several intermediate builds timed out, providing no useful test results.

To check if this could be a regression in the software being tested, I 
inspected logs from
local test runs over the given period, which, however, show no 
performance regression.


When I looked up the various hipps on this hipp10, I couldn't see much 
info about
build history, in fact only the platform hipp has data in the build 
history view.


The same suite A is actually part of JDT/Core so we can do some 
comparison of the
Object Teams HIPP with eclipse.jdt.core-Gerrit on Platform HIPP (same 
machine hipp10!).
On the latter HIPP we observe alternating time outs but also test runs 
completing

normally at the typical 30 min.


In this situation I have to run each build several times until I 
happen to get
complete test results. Of course these unnecessary test runs aggravate 
the situation...


Short term work around: stop using hipp for testing, test only locally 
(saves much time)
use hipp only for building / publishing - which will also take some 
load from that machine.


Maybe we should all do this to retain good response times at least for 
the actual

production builds?

best,
Stephan


On 02/02/2017 08:08 AM, Markus Knauer wrote:
I see similar problems on hipp8. Its reported load is between 1 and 2 
at the moment, sometimes less, i.e. not very high, but the EPP
package builds are extremely slow, much slower than usual. A few 
months ago in October these builds used to take 1:20h [1], now we
are close to 6 hours [2] which is 4 times slower for the same build. 
In both cases, builds were running on a day during Simultaneous
Release week which rules out this as a reason for the increased 
duration.


Maybe its worth to look at this host, too, as the increased build 
time doesn't seem to be caused by a high load of the CPU.


And I highly appreciate your efforts to distribute CPU resources in a 
better way!


Much thanks,
Markus

[1] https://hudson.eclipse.org/packaging/job/neon.epp-tycho-build/448/
[2] https://hudson.eclipse.org/packaging/job/oxygen.epp-tycho-build/183/



On 2 February 2017 at 01:22, Frederic Gurr > wrote:


Hi,

Unfortunately hipp9 has been experiencing a near-constant high load
during work hours. Due to a few projects hogging resources other
projects are severely constrained and have problems running their 
builds

in a timely manner.

To fix this, I have taken the following near term steps:

- Papyrus HIPP
  - disabled execution of concurrent builds for 
"Papyrus-Gerrit-Oxygen"

  - limited the number of executors to 2

- EMF Compare HIPP
  - limited the number of executors to 2

I will also move the EMF Compare HIPP to a different hipp machine
tomorrow at 9am EST. Downtime will be less than one hour.

We will also investigate how CPU usage can be limited to ensure a 
fair

distribution of resources.

Regards,

Fred
___
cross-project-issues-dev mailing list
cross-project-issues-dev@eclipse.org 

To change your delivery options, retrieve your password, or 
unsubscribe from this list, visit

https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev







___
cross-project-issues-dev mailing list
cross-project-issues-dev@eclipse.org
To change your delivery options, retrieve your password, or 
unsubscribe from this list, visit

https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev



___
cross-project-issues-dev mailing list
cross-project-issues-dev@eclipse.org
To change your delivery options, re

Re: [cross-project-issues-dev] hipp9 overloaded

2017-02-05 Thread Stephan Herrmann 

Could this be a general issue?
The factor of 4 looks similar to what I see in hipp10 (Object Teams).

Some figures:
Two individual test suites A and B as part of the Object Teams build over time
(from some builds which I happened to keep):

up-to 2016-12-08:
suite A: around 30 min.
suite B: around 30 min.

2017-01-18:
suite A at 47 min.
suite B presumably timed out after 120 min (details no longer available)

2017-01-22:
suite A at 44 min.
suite B completed at 110 min

2017-01-27:
suite A at 60 min.
suite B presumably timed out after 120 min (details no longer available)

2017-02-04:
suite A timed out at 120 min
suite B died at 50 min (not sure if this could possibly be a time out)

Several intermediate builds timed out, providing no useful test results.

To check if this could be a regression in the software being tested, I 
inspected logs from
local test runs over the given period, which, however, show no performance 
regression.

When I looked up the various hipps on this hipp10, I couldn't see much info 
about
build history, in fact only the platform hipp has data in the build history 
view.

The same suite A is actually part of JDT/Core so we can do some comparison of 
the
Object Teams HIPP with eclipse.jdt.core-Gerrit on Platform HIPP (same machine 
hipp10!).
On the latter HIPP we observe alternating time outs but also test runs 
completing
normally at the typical 30 min.


In this situation I have to run each build several times until I happen to get
complete test results. Of course these unnecessary test runs aggravate the 
situation...

Short term work around: stop using hipp for testing, test only locally (saves 
much time)
use hipp only for building / publishing - which will also take some load from 
that machine.

Maybe we should all do this to retain good response times at least for the 
actual
production builds?

best,
Stephan


On 02/02/2017 08:08 AM, Markus Knauer wrote:

I see similar problems on hipp8. Its reported load is between 1 and 2 at the 
moment, sometimes less, i.e. not very high, but the EPP
package builds are extremely slow, much slower than usual. A few months ago in 
October these builds used to take 1:20h [1], now we
are close to 6 hours [2] which is 4 times slower for the same build. In both 
cases, builds were running on a day during Simultaneous
Release week which rules out this as a reason for the increased duration.

Maybe its worth to look at this host, too, as the increased build time doesn't 
seem to be caused by a high load of the CPU.

And I highly appreciate your efforts to distribute CPU resources in a better 
way!

Much thanks,
Markus

[1] https://hudson.eclipse.org/packaging/job/neon.epp-tycho-build/448/
[2] https://hudson.eclipse.org/packaging/job/oxygen.epp-tycho-build/183/



On 2 February 2017 at 01:22, Frederic Gurr mailto:frederic.g...@eclipse.org>> wrote:

Hi,

Unfortunately hipp9 has been experiencing a near-constant high load
during work hours. Due to a few projects hogging resources other
projects are severely constrained and have problems running their builds
in a timely manner.

To fix this, I have taken the following near term steps:

- Papyrus HIPP
  - disabled execution of concurrent builds for "Papyrus-Gerrit-Oxygen"
  - limited the number of executors to 2

- EMF Compare HIPP
  - limited the number of executors to 2

I will also move the EMF Compare HIPP to a different hipp machine
tomorrow at 9am EST. Downtime will be less than one hour.

We will also investigate how CPU usage can be limited to ensure a fair
distribution of resources.

Regards,

Fred
___
cross-project-issues-dev mailing list
cross-project-issues-dev@eclipse.org 

To change your delivery options, retrieve your password, or unsubscribe 
from this list, visit
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev







___
cross-project-issues-dev mailing list
cross-project-issues-dev@eclipse.org
To change your delivery options, retrieve your password, or unsubscribe from 
this list, visit
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev



___
cross-project-issues-dev mailing list
cross-project-issues-dev@eclipse.org
To change your delivery options, retrieve your password, or unsubscribe from 
this list, visit
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev

Re: [cross-project-issues-dev] hipp9 overloaded

2017-02-02 Thread Denis Roy 
Probably shared disk access... We'll have to look at that as well.

Denis


On 02/02/2017 02:08 AM, Markus Knauer wrote:
> I see similar problems on hipp8. Its reported load is between 1 and 2
> at the moment, sometimes less, i.e. not very high, but the EPP package
> builds are extremely slow, much slower than usual. A few months ago in
> October these builds used to take 1:20h [1], now we are close to 6
> hours [2] which is 4 times slower for the same build. In both cases,
> builds were running on a day during Simultaneous Release week which
> rules out this as a reason for the increased duration.
>
> Maybe its worth to look at this host, too, as the increased build time
> doesn't seem to be caused by a high load of the CPU.
>
> And I highly appreciate your efforts to distribute CPU resources in a
> better way!
>
> Much thanks,
> Markus
>
> [1] https://hudson.eclipse.org/packaging/job/neon.epp-tycho-build/448/
> [2] https://hudson.eclipse.org/packaging/job/oxygen.epp-tycho-build/183/
>
>
>
> On 2 February 2017 at 01:22, Frederic Gurr  > wrote:
>
> Hi,
>
> Unfortunately hipp9 has been experiencing a near-constant high load
> during work hours. Due to a few projects hogging resources other
> projects are severely constrained and have problems running their
> builds
> in a timely manner.
>
> To fix this, I have taken the following near term steps:
>
> - Papyrus HIPP
>   - disabled execution of concurrent builds for
> "Papyrus-Gerrit-Oxygen"
>   - limited the number of executors to 2
>
> - EMF Compare HIPP
>   - limited the number of executors to 2
>
> I will also move the EMF Compare HIPP to a different hipp machine
> tomorrow at 9am EST. Downtime will be less than one hour.
>
> We will also investigate how CPU usage can be limited to ensure a fair
> distribution of resources.
>
> Regards,
>
> Fred
> ___
> cross-project-issues-dev mailing list
> cross-project-issues-dev@eclipse.org
> 
> To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
> 
>
>
>
>
>
>
> ___
> cross-project-issues-dev mailing list
> cross-project-issues-dev@eclipse.org
> To change your delivery options, retrieve your password, or unsubscribe from 
> this list, visit
> https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev

___
cross-project-issues-dev mailing list
cross-project-issues-dev@eclipse.org
To change your delivery options, retrieve your password, or unsubscribe from 
this list, visit
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev

Re: [cross-project-issues-dev] hipp9 overloaded

2017-02-01 Thread Markus Knauer 
I see similar problems on hipp8. Its reported load is between 1 and 2 at
the moment, sometimes less, i.e. not very high, but the EPP package builds
are extremely slow, much slower than usual. A few months ago in October
these builds used to take 1:20h [1], now we are close to 6 hours [2] which
is 4 times slower for the same build. In both cases, builds were running on
a day during Simultaneous Release week which rules out this as a reason for
the increased duration.

Maybe its worth to look at this host, too, as the increased build time
doesn't seem to be caused by a high load of the CPU.

And I highly appreciate your efforts to distribute CPU resources in a
better way!

Much thanks,
Markus

[1] https://hudson.eclipse.org/packaging/job/neon.epp-tycho-build/448/
[2] https://hudson.eclipse.org/packaging/job/oxygen.epp-tycho-build/183/



On 2 February 2017 at 01:22, Frederic Gurr 
wrote:

> Hi,
>
> Unfortunately hipp9 has been experiencing a near-constant high load
> during work hours. Due to a few projects hogging resources other
> projects are severely constrained and have problems running their builds
> in a timely manner.
>
> To fix this, I have taken the following near term steps:
>
> - Papyrus HIPP
>   - disabled execution of concurrent builds for "Papyrus-Gerrit-Oxygen"
>   - limited the number of executors to 2
>
> - EMF Compare HIPP
>   - limited the number of executors to 2
>
> I will also move the EMF Compare HIPP to a different hipp machine
> tomorrow at 9am EST. Downtime will be less than one hour.
>
> We will also investigate how CPU usage can be limited to ensure a fair
> distribution of resources.
>
> Regards,
>
> Fred
> ___
> cross-project-issues-dev mailing list
> cross-project-issues-dev@eclipse.org
> To change your delivery options, retrieve your password, or unsubscribe
> from this list, visit
> https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
>
___
cross-project-issues-dev mailing list
cross-project-issues-dev@eclipse.org
To change your delivery options, retrieve your password, or unsubscribe from 
this list, visit
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev

[cross-project-issues-dev] hipp9 overloaded

2017-02-01 Thread Frederic Gurr 
Hi,

Unfortunately hipp9 has been experiencing a near-constant high load
during work hours. Due to a few projects hogging resources other
projects are severely constrained and have problems running their builds
in a timely manner.

To fix this, I have taken the following near term steps:

- Papyrus HIPP
  - disabled execution of concurrent builds for "Papyrus-Gerrit-Oxygen"
  - limited the number of executors to 2

- EMF Compare HIPP
  - limited the number of executors to 2

I will also move the EMF Compare HIPP to a different hipp machine
tomorrow at 9am EST. Downtime will be less than one hour.

We will also investigate how CPU usage can be limited to ensure a fair
distribution of resources.

Regards,

Fred
___
cross-project-issues-dev mailing list
cross-project-issues-dev@eclipse.org
To change your delivery options, retrieve your password, or unsubscribe from 
this list, visit
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev