Re: Koji builders' specs

2017-01-04 Thread Dan Horák 
On Wed, 04 Jan 2017 13:58:38 +0100
Pavel Raiskup  wrote:

> On Wednesday, January 4, 2017 12:06:40 PM CET Pavel Raiskup wrote:
> > On Monday, January 2, 2017 3:47:52 PM CET Pavel Raiskup wrote:
> > > On Monday, January 2, 2017 2:40:34 PM CET Pavel Raiskup wrote:
> > > > Also, the question is whether it isn't really glibc bug, because
> > > > schedulers non-ppc64le architectures look to be more "fair"
> > > > regardless the default.
> > > 
> > > Sorry, not really -- at least i686 (build cross-compiled from
> > > x86_64 probably) is also affected:
> > > https://koji.fedoraproject.org/koji/taskinfo?taskID=17147577
> 
> It is nice to see that the build finished in the end (even though it
> took ~2 days).  So not really infinite ...
> 
> ... but what worries me is that both builds, even this one,
> https://koji.fedoraproject.org/koji/taskinfo?taskID=17147897
> finished almost at the same time;  both tests took about 48 hours?
> 
> Is there some timeout, signal, or scheduler attempt to change
> something after 48 hours? :)

there is a timeout set in koji builder config that tells mock to kill
builds after 48 after hours


Dan
 
> Pavel
> 
> > FTR, I just filed:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1410052
> > Who knows, maybe that could move this forward.
> > 
> > That's a result of gnulib discussion:
> > https://lists.gnu.org/archive/html/bug-gnulib/2017-01/msg9.html
> > 
> > Pavel
> > ___
> > devel mailing list -- devel@lists.fedoraproject.org
> > To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> > 
> 
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Koji builders' specs

2017-01-04 Thread Pavel Raiskup 
On Wednesday, January 4, 2017 12:06:40 PM CET Pavel Raiskup wrote:
> On Monday, January 2, 2017 3:47:52 PM CET Pavel Raiskup wrote:
> > On Monday, January 2, 2017 2:40:34 PM CET Pavel Raiskup wrote:
> > > Also, the question is whether it isn't really glibc bug, because
> > > schedulers non-ppc64le architectures look to be more "fair" regardless
> > > the default.
> > 
> > Sorry, not really -- at least i686 (build cross-compiled from x86_64
> > probably) is also affected:
> > https://koji.fedoraproject.org/koji/taskinfo?taskID=17147577

It is nice to see that the build finished in the end (even though it took ~2
days).  So not really infinite ...

... but what worries me is that both builds, even this one,
https://koji.fedoraproject.org/koji/taskinfo?taskID=17147897
finished almost at the same time;  both tests took about 48 hours?

Is there some timeout, signal, or scheduler attempt to change something after 48
hours? :)

Pavel

> FTR, I just filed:
> https://bugzilla.redhat.com/show_bug.cgi?id=1410052
> Who knows, maybe that could move this forward.
> 
> That's a result of gnulib discussion:
> https://lists.gnu.org/archive/html/bug-gnulib/2017-01/msg9.html
> 
> Pavel
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> 

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Koji builders' specs

2017-01-04 Thread Pavel Raiskup 
On Monday, January 2, 2017 3:47:52 PM CET Pavel Raiskup wrote:
> On Monday, January 2, 2017 2:40:34 PM CET Pavel Raiskup wrote:
> > Also, the question is whether it isn't really glibc bug, because
> > schedulers non-ppc64le architectures look to be more "fair" regardless
> > the default.
> 
> Sorry, not really -- at least i686 (build cross-compiled from x86_64
> probably) is also affected:
> https://koji.fedoraproject.org/koji/taskinfo?taskID=17147577

FTR, I just filed:
https://bugzilla.redhat.com/show_bug.cgi?id=1410052
Who knows, maybe that could move this forward.

That's a result of gnulib discussion:
https://lists.gnu.org/archive/html/bug-gnulib/2017-01/msg9.html

Pavel
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Koji builders' specs

2017-01-02 Thread Miroslav Suchý 
Dne 2.1.2017 v 11:24 Miroslav Suchý napsal(a):
> Hmm, that is nice idea. Plugin for Mock which will print:
>  * number of CPU
>  * RAM + swap
>  * storage size available
> 
> I will try to work on that.

Done.  You can expect it in next version of Mock:
https://github.com/rpm-software-management/mock/wiki/Plugin-HwInfo

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Koji builders' specs

2017-01-02 Thread Pavel Raiskup 
On Monday, January 2, 2017 2:40:34 PM CET Pavel Raiskup wrote:
> Also, the question is whether it isn't really glibc bug, because
> schedulers non-ppc64le architectures look to be more "fair" regardless
> the default.

Sorry, not really -- at least i686 (build cross-compiled from x86_64
probably) is also affected:
https://koji.fedoraproject.org/koji/taskinfo?taskID=17147577

Pavel
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Koji builders' specs

2017-01-02 Thread Pavel Raiskup 
On Monday, January 2, 2017 11:24:35 AM CET Miroslav Suchý wrote:
> Dne 20.12.2016 v 16:03 Pavel Raiskup napsal(a):
> > The only thing I was able to find is version of mock in the log output.
> 
> Hmm, that is nice idea. Plugin for Mock which will print:
>  * number of CPU
>  * RAM + swap
>  * storage size available
> 
> I will try to work on that.

Perfect, thanks!

Also it would be nice to see:

 1. a note that multiple builds might be run concurrently on single Koji builder
 2. what is the HW host running the builder VM

But I see those are not requests for mock..  Thus some wiki showing
builder's hostname to specs would be useful.

Also #2 is useless, because such hardware is usually not available for
maintainers ... (without ticket against fedora koji maintainers).  But
some link for ticket creating would be useful on such wiki page.

Pavel
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Koji builders' specs

2017-01-02 Thread Pavel Raiskup 
On Monday, January 2, 2017 11:00:44 AM CET Kamil Dudka wrote:
> > Note this test was just changed upstream to use locks instead of volatiles,
> > which significantly improved performance on a 40 core NUMA system at least:
> > http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=commitdiff;h=480d374
> 
> Thanks for letting me know!  I have applied the above patch on the coreutils 
> package in Fedora rawhide:
> 
> http://pkgs.fedoraproject.org/cgit/rpms/coreutils.git/commit/?id=b3b3da0c
> 
> ... and it seems to have resolved the issue with hanging Koji builds.

As we discussed in person with Kamil, unfortunately the upstream fix
doesn't help.  And also most probably, I was wrong before when I told that
this is _not_ arch specific;  it looks like it actually is ppc64le only
and, indeed!, the test_rwlock() test took incredibly long before the
upstream fix (not only on ppc64le).  So a good performance speedup done
upstream, thanks!

But yes, there's still problem with test_rwlock() on ppc64le.  Usually I
see this in build.log [1]:

...
+ set +x
   1 Starting test_lock ... OK
   9 Starting test_rwlock ... OK
  10 Starting test_recursive_lock ... OK
  13 Starting test_once ... OK
  13 
...

but sometimes there's infinite hang [2]:

...
+ set +x
1 Starting test_lock ... OK
[HANG HERE]

That integer number on each line means "how many seconds left when the test
(being written to stdout) finished" [3].

Kamil pointed out that the test-lock.c might be wrong.  The point is that
there are multiple independent read-only threads (up to 10?) in "infinite
loop until all read-write threads finish".  Which causes probably writer's
starvation.  POSIX only says that "Implementations may favor writers over
readers to avoid writer starvation.", but it is not guaranteed in general.

More precisely, we seem to have PTHREAD_RWLOCK_PREFER_READER_NP in glibc
by default, so I'm not sure whether we shouldn't set
PTHREAD_RWLOCK_PREFER_WRITER_NP in test-lock.c somewhere... at least if
possible.  Also, the question is whether it isn't really glibc bug,
because schedulers non-ppc64le architectures look to be more "fair"
regardless the default.

Also, I was unable to reproduce anywhere else than in Koji.

[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=17147902
[2] https://koji.fedoraproject.org/koji/taskinfo?taskID=17147995
[3] 
http://pkgs.fedoraproject.org/cgit/rpms/gettext.git/commit/?h=private-praiskup-test-lock-upstream-fix=937f421ab3276c7fc363c1af9e

Pavel
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Koji builders' specs

2017-01-02 Thread Miroslav Suchý 
Dne 20.12.2016 v 16:03 Pavel Raiskup napsal(a):
> The only thing I was able to find is version of mock in the log output.

Hmm, that is nice idea. Plugin for Mock which will print:
 * number of CPU
 * RAM + swap
 * storage size available

I will try to work on that.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Koji builders' specs

2017-01-02 Thread Kamil Dudka 
On Sunday, December 25, 2016 13:13:31 Pádraig Brady wrote:
> On 22/12/16 13:13, Pádraig Brady wrote:
> > On 21/12/16 16:04, Dan Horák wrote:
> >> On Tue, 20 Dec 2016 16:03:49 +0100
> >> 
> >> Pavel Raiskup  wrote:
> >>> Hi all,
> >>> 
> >>> where is documented what system/hw is used on (primary) Koji builders?
> >>> I'm interested in memory, storage, filesystem, host operating system,
> >>> guest operating system (if those are VMs), etc.
> >>> 
> >>> The only thing I was able to find is version of mock in the log
> >>> output.
> >>> 
> >>> FWIW, I'd like to reproduce hang in gnulib's test-lock [1] test case.
> >>> Unless I'm able to reproduce that somehow, I'll disable the test for
> >>> the time being (done in 'coreutils' and I guess elsewhere, too).
> >> 
> >> oh no, test-lock again :-( A year (or two) ago it turned to be a kernel
> >> bug - https://bugzilla.redhat.com/show_bug.cgi?id=1155291
> > 
> > I saw Kamil recently disabled that test for the coreutils update.
> > Note the test itself may have issues as discussed at:
> > https://lists.gnu.org/archive/html/bug-gnulib/2015-07/msg00032.html
> 
> Note this test was just changed upstream to use locks instead of volatiles,
> which significantly improved performance on a 40 core NUMA system at least:
> http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=commitdiff;h=480d374

Thanks for letting me know!  I have applied the above patch on the coreutils 
package in Fedora rawhide:

http://pkgs.fedoraproject.org/cgit/rpms/coreutils.git/commit/?id=b3b3da0c

... and it seems to have resolved the issue with hanging Koji builds.

Kamil

> cheers,
> Pádraig
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Koji builders' specs

2016-12-25 Thread Pádraig Brady 
On 22/12/16 13:13, Pádraig Brady wrote:
> On 21/12/16 16:04, Dan Horák wrote:
>> On Tue, 20 Dec 2016 16:03:49 +0100
>> Pavel Raiskup  wrote:
>>
>>> Hi all,
>>>
>>> where is documented what system/hw is used on (primary) Koji builders?
>>> I'm interested in memory, storage, filesystem, host operating system,
>>> guest operating system (if those are VMs), etc.
>>>
>>> The only thing I was able to find is version of mock in the log
>>> output.
>>>
>>> FWIW, I'd like to reproduce hang in gnulib's test-lock [1] test case.
>>> Unless I'm able to reproduce that somehow, I'll disable the test for
>>> the time being (done in 'coreutils' and I guess elsewhere, too).
>>
>> oh no, test-lock again :-( A year (or two) ago it turned to be a kernel
>> bug - https://bugzilla.redhat.com/show_bug.cgi?id=1155291
> 
> I saw Kamil recently disabled that test for the coreutils update.
> Note the test itself may have issues as discussed at:
> https://lists.gnu.org/archive/html/bug-gnulib/2015-07/msg00032.html

Note this test was just changed upstream to use locks instead of volatiles,
which significantly improved performance on a 40 core NUMA system at least:
http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=commitdiff;h=480d374

cheers,
Pádraig
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Koji builders' specs

2016-12-22 Thread Pádraig Brady 
On 21/12/16 16:04, Dan Horák wrote:
> On Tue, 20 Dec 2016 16:03:49 +0100
> Pavel Raiskup  wrote:
> 
>> Hi all,
>>
>> where is documented what system/hw is used on (primary) Koji builders?
>> I'm interested in memory, storage, filesystem, host operating system,
>> guest operating system (if those are VMs), etc.
>>
>> The only thing I was able to find is version of mock in the log
>> output.
>>
>> FWIW, I'd like to reproduce hang in gnulib's test-lock [1] test case.
>> Unless I'm able to reproduce that somehow, I'll disable the test for
>> the time being (done in 'coreutils' and I guess elsewhere, too).
> 
> oh no, test-lock again :-( A year (or two) ago it turned to be a kernel
> bug - https://bugzilla.redhat.com/show_bug.cgi?id=1155291

I saw Kamil recently disabled that test for the coreutils update.
Note the test itself may have issues as discussed at:
https://lists.gnu.org/archive/html/bug-gnulib/2015-07/msg00032.html
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Koji builders' specs

2016-12-21 Thread Dan Horák 
On Tue, 20 Dec 2016 16:03:49 +0100
Pavel Raiskup  wrote:

> Hi all,
> 
> where is documented what system/hw is used on (primary) Koji builders?
> I'm interested in memory, storage, filesystem, host operating system,
> guest operating system (if those are VMs), etc.
> 
> The only thing I was able to find is version of mock in the log
> output.
> 
> FWIW, I'd like to reproduce hang in gnulib's test-lock [1] test case.
> Unless I'm able to reproduce that somehow, I'll disable the test for
> the time being (done in 'coreutils' and I guess elsewhere, too).

oh no, test-lock again :-( A year (or two) ago it turned to be a kernel
bug - https://bugzilla.redhat.com/show_bug.cgi?id=1155291


Dan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Koji builders' specs

2016-12-21 Thread Kevin Fenzi 
On Tue, 20 Dec 2016 16:03:49 +0100
Pavel Raiskup  wrote:

> Hi all,
> 
> where is documented what system/hw is used on (primary) Koji builders?
> I'm interested in memory, storage, filesystem, host operating system,
> guest operating system (if those are VMs), etc.
> 
> The only thing I was able to find is version of mock in the log
> output.
> 
> FWIW, I'd like to reproduce hang in gnulib's test-lock [1] test case.
> Unless I'm able to reproduce that somehow, I'll disable the test for
> the time being (done in 'coreutils' and I guess elsewhere, too).
> 
> [1] https://koji.fedoraproject.org/koji/taskinfo?taskID=16970779

I don't think we have any document like that, but:

buildvm-01 to buildvm-32:

These are Fedora 25 Vm's with 15GB mem, 6 cpus, 256GB disk of which
128 is swap (in prep for testing tmpfs mock plugin, which is currently
NOT enabled). 

arm*: 

calexida armv7 SOC's. Fedora 25. Each has ~300GB disk, 4GB ram, 4 cpus. 

buildhw*:

dell poweredge fx blades with 128GB ram, Fedora 25, 48 cpus, 2 400GB
drives in raid1.

buildppc: 

VM on power8, Fedora 25, 10GB mem, 5 cpus, 150GB disk. 

buildvm-aarch64: 

VM's on HP moonshot cartridges. Fedora 24 (soon 25), 25GB memory,
4CPUs, 150GB disk. 

In the cases where there are VM's, they are running on RHEL7. 

It looks like your task hung on x86_64, i686 and ppc64.

The first two shoudl be easy to come by. Let us know (via an
infrastructure ticket) if you need a ppc64 vm. 

kevin




pgp4lizkj3HNG.pgp
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Koji builders' specs

2016-12-20 Thread Pavel Raiskup 
Hi all,

where is documented what system/hw is used on (primary) Koji builders?
I'm interested in memory, storage, filesystem, host operating system, guest
operating system (if those are VMs), etc.

The only thing I was able to find is version of mock in the log output.

FWIW, I'd like to reproduce hang in gnulib's test-lock [1] test case.
Unless I'm able to reproduce that somehow, I'll disable the test for the time
being (done in 'coreutils' and I guess elsewhere, too).

[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=16970779

Thanks,
Pavel
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org