Re: [Development] CI is currently down! Hardware issues.

[Tony Sarajärvi via Development]

Hi again.

The backend seems to be working after a standard fix method was applied: power 
off - power on.

All systems are back to normal.

-Tony
-- 
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] CI is currently down! Hardware issues.

[Tony Sarajärvi via Development]

Hi

We had a backend hardware failure last night, and it stopped our CI entirely. 
Investigations on-going and as of yet, not estimates on the downtime.

-Tony
-- 
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] CI down for a while

[Tony Sarajärvi]

Hi

We hit a small SNAFU with the CI and we’re fixing it. Downtime depends all on 
the speed of the system sadly :/
It’s been fixing for 2 hours now and closing to its end. My estimate is an hour 
more of downtime.

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CI just blew up

[Tony Sarajärvi]

Trying to restart the CI now. We had to reinstall OpenNebula and redeploy all 
hosts etc. Seemed like the quickest way to get things online today.

We don’t know what corrupted the SQL database, nor do we have anything to debug 
it with left after nuking it. Let’s just see if it happens again.

Tier2 images should be left, we did not wipe the NFS partition, so no 
provisioning storm should be needed.

Good luck! I’ll continue on this tomorrow

-Tony

From: Tony Sarajärvi
Sent: Thursday, January 28, 2021 3:59 PM
To: development@qt-project.org
Subject: RE: CI just blew up

Seems this isn’t repairable. The poo entangled itself so well into the fan, 
that I have to replace the whole fan now.

This could take a few hours...

-Tony

From: Tony Sarajärvi
Sent: Thursday, January 28, 2021 3:31 PM
To: development@qt-project.org<mailto:development@qt-project.org>
Subject: CI just blew up

Hi

Something blew up in the CI. Currently my finger is pointing towards 
OpenNebula’s SQL database blowing up. But that’s after 5 minutes of looking at 
logs... so let’s see.

In other words: working on it.

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CI just blew up

[Tony Sarajärvi]

Seems this isn’t repairable. The poo entangled itself so well into the fan, 
that I have to replace the whole fan now.

This could take a few hours...

-Tony

From: Tony Sarajärvi
Sent: Thursday, January 28, 2021 3:31 PM
To: development@qt-project.org
Subject: CI just blew up

Hi

Something blew up in the CI. Currently my finger is pointing towards 
OpenNebula’s SQL database blowing up. But that’s after 5 minutes of looking at 
logs... so let’s see.

In other words: working on it.

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] CI just blew up

[Tony Sarajärvi]

Hi

Something blew up in the CI. Currently my finger is pointing towards 
OpenNebula’s SQL database blowing up. But that’s after 5 minutes of looking at 
logs... so let’s see.

In other words: working on it.

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Disk quota exceeded in the CI

[Tony Sarajärvi]

Hi

Disk quota got exceeded and I’m trying to fix it somehow. CI is at a stand 
still until I fix it.

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Monthly maintenance break - December (Mon 14th Dec 2020)

[Tony Sarajärvi]

Hi all!

Seems the break went smoothly and we’ve been back online for a while now.

Biggest change was the update of OpenNebula and the underlying hosts we build 
on. We now run everything on Ubuntu 20.04 (compared to 18.04 previously), 
meaning new KVM and QEMU might affect the VMs in some way.

Other than that, security patches were applied here and there.

Good luck and merry x-mas!

-Tony

From: Development  On Behalf Of Ville-Pekka 
Karhu
Sent: Thursday, December 10, 2020 4:15 PM
To: Qt development mailing list 
Subject: [Development] Monthly maintenance break - December (Mon 14th Dec 2020)


Hi all!



We’ll have our scheduled maintenance break on next Monday (14th of Dec). We’ll 
begin our work at 8:00 EET and you can prepare for 3 hours of CI not working.



Br,

VP

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Short break in OpenNebula

[Tony Sarajärvi]

Hi

I have to clean up the database a bit, and for that I have to shut down 
OpenNebula. We’ll be back in a jiffy

-T
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] QThread::create mandatory in Qt 6?

[Tony Sarajärvi]

>The big problem I see on Windows is unfortunately not the C++ standard 
>library, but the C standard library. MinGW defaults to crtdll, the one from 
>MSVC 6, which is not >C99-compliant (MSVC 6 is from 1998 after all). A better 
>alternative exists in the form of the Universal CRT, but that only comes by 
>default with Windows 10, so we'd need to >drop support for 8.1 -- have we done 
>that? -- and recompile GCC and/or LLVM so they ask the linker to link to ucrt.

I have no idea what I'm talking about but I've currently set myself to a 
receive mode for everything regarding these compilers. So I tried to educate 
myself and stumbled upon this: 
https://www.cs.colorado.edu/~main/cs1300/doc/mingwfaq.html


msvcrt vs crtdll?
Mingw requires the Microsoft runtime libraries. You'll need either crtdll.dll 
or msvcrt.dll on a system in order to run a program. There are two 
distributions of mingw, one for each runtime library option. Use the version of 
mingw for the type of runtime library you wish to work with. Which is better? 
That depends on what you need. Crtdll is on all Win32 systems, but is no longer 
updated. It is available on platforms such as Windows 3.11/Windows NT 3.5 and 
platforms that use Win32s. Msvcrt is not guaranteed to be on every Windows 
operating system, but it is more up-to-date and may offer more features. It 
handles threading better than crtdll and supplies functions such as _i64toa and 
_atoi64 that crtdll doesn't. There is a possibility that crtdll may not be 
supported in future releases of mingw.


And do we have to mention or say mingw-w64 every time we say mingw? Or can we 
take that for granted?

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Do we want to support Windows LLVM builds from winlibs?

[Tony Sarajärvi]

> > One problem with it is that it doesn’t support std::future or std::invoke.
> > See https://bugreports.qt.io/browse/QTBUG-88392
> > 
> > So how about it, do we want to support that?

> See the other thread.

> We wait for them to support those.

Oh man how did I miss that one. Sorry :/

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Do we want to support Windows LLVM builds from winlibs?

[Tony Sarajärvi]

Hi,

I’ve been experimenting with getting builds on Windows done with LLVM. I’m 
trying to use the package from winlibs.com and to be more precies, this one: 
https://github.com/brechtsanders/winlibs_mingw/releases/download/10.2.0-10.0.1-9.0.0-r1/winlibs-x86_64-posix-seh-gcc-10.2.0-llvm-10.0.1-mingw-w64-8.0.0-r1.7z

One problem with it is that it doesn’t support std::future or std::invoke.
See https://bugreports.qt.io/browse/QTBUG-88392

So how about it, do we want to support that?
-Tony

Tony Sarajärvi
CI Tech Lead
[The Qt Company logo]<http://qt.io/>
[Facebook]<http://www.facebook.com/Qt>
[Twitter]<http://www.twitter.com/qtproject>
[LinkedIn]<https://www.linkedin.com/company/the-qt-company/>
[Google+]<https://plus.google.com/104580575722059274792>
[YouTube]<https://www.youtube.com/QtStudios>


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Crash in the CI infra

[Tony Sarajärvi]

Things might work again.

-T

From: Tony Sarajärvi
Sent: Thursday, November 5, 2020 8:22 AM
To: Qt development mailing list 
Subject: Crash in the CI infra

Good morning

Last evening we had a host crash where our NFS server went down. Thus all CI 
builds came to a halt. We’re trying to recover from it as I type.

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Crash in the CI infra

[Tony Sarajärvi]

Good morning

Last evening we had a host crash where our NFS server went down. Thus all CI 
builds came to a halt. We’re trying to recover from it as I type.

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Monthly maintenance break - November (Mon 2nd Nov 2020)

[Tony Sarajärvi]

Hi all!

The maintenance break is over. Things that got updated:

  *   Gerrit
  *   One build host got a firmware upgrade
  *   A couple of security fixes for distros on various servers

Happy building!
-Tony

From: Tony Sarajärvi
Sent: Tuesday, October 27, 2020 8:21 AM
To: Qt development mailing list 
Subject: Monthly maintenance break - November (Mon 2nd Nov 2020)

Hi all!

We’ll have our scheduled maintenance break next Monday (2nd of Nov). We’ll 
begin our work at 8:00 EET and you can prepare for 3 hours of CI not working. 
Which parts of the infra will be updated is yet unknown, since we’ll see what 
updates are available and plan the update on Friday.

-Tony
[The Qt Company logo]<http://qt.io/>
[Facebook]<http://www.facebook.com/Qt>
[Twitter]<http://www.twitter.com/qtproject>
[LinkedIn]<https://www.linkedin.com/company/the-qt-company/>
[Google+]<https://plus.google.com/104580575722059274792>
[YouTube]<https://www.youtube.com/QtStudios>


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Monthly maintenance break - November (Mon 2nd Nov 2020)

[Tony Sarajärvi]

Hi all!

We’ll have our scheduled maintenance break next Monday (2nd of Nov). We’ll 
begin our work at 8:00 EET and you can prepare for 3 hours of CI not working. 
Which parts of the infra will be updated is yet unknown, since we’ll see what 
updates are available and plan the update on Friday.

-Tony
[The Qt Company logo]
[Facebook]
[Twitter]
[LinkedIn]
[Google+]
[YouTube]


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Maintenance mishap with a coincidence

[Tony Sarajärvi]

Hi

We had a maintenance break yesterday and we left snapshots on just in case we 
have to revert something.
Well…Coin in our CI created so much data, that due to the snapshot it filled 
its LUN, therefore bringing the CI to a halt.

I should have caught the event before it reached full storage capacity before 
it actually happened due to our warning systems. However, IT changed some email 
policy yesterday, so I never received any e-mails from work since yesterday 
afternoon. Bad luck 

I’m working on untangling the system. A moment please…

-Tony

Tony Sarajärvi
CI Tech Lead

The Qt Company
Tutkijantie 4
90590 Oulu, Finland
tony.saraja...@qt.io
+358 50 4821416
http://qt.io
[The Qt Company logo]<http://qt.io/>
[Facebook]<http://www.facebook.com/Qt>
[Twitter]<http://www.twitter.com/qtproject>
[LinkedIn]<https://www.linkedin.com/company/the-qt-company/>
[Google+]<https://plus.google.com/104580575722059274792>
[YouTube]<https://www.youtube.com/QtStudios>


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Our CI is suffering from malfunctions currently

[Tony Sarajärvi]

Hi all

The system _might_ be up. All tier2 images were lost, and we’ve had some 
problems with tier1 images as well. But the only way to be sure things work are 
to start the systems and see what happens.  So cross your fingers, here we go!

Btrfs, when it breaks down, it surely does it properly.

-Tony

From: Tony Sarajärvi
Sent: Wednesday, June 24, 2020 10:31 AM
To: Qt development mailing list 
Subject: RE: Our CI is suffering from malfunctions currently

Seems the BTRFS file system is permanently busted. We have to recreate the 
thing and I estimate it taking the rest of the day. Sorry!

-T

From: Tony Sarajärvi
Sent: Wednesday, June 24, 2020 8:13 AM
To: Qt development mailing list 
mailto:development@qt-project.org>>
Subject: Our CI is suffering from malfunctions currently

Hi

Yesterday our NFS server’s root partition got filled up by logs apparently. As 
we got everything fixed, it did not take long for it to start filling up again. 
Within ours it got filled up again. So, the CI is currently down until we fix 
things…

Bear with us
-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Our CI is suffering from malfunctions currently

[Tony Sarajärvi]

We don't have tier1 or tier2 images available for the hosts, so they can't do 
much currently. It was a close call that would have lost them permanently as 
well, but luckily it seems that btrfs restore can still read them.

We really need to have tier1 images stored somewhere else as well. The NFS on 
the compellent running on btrfs is not safe enough. We need more copies 
distributed and that's something we'll P0 prioritize after we get things 
running.

-T

-Original Message-
From: Volker Hilsheimer  
Sent: Wednesday, June 24, 2020 10:57 AM
To: Tony Sarajärvi 
Cc: development@qt-project.org
Subject: Re: Our CI is suffering from malfunctions currently

Hi Tony,

Thanks for the update, and good luck with the fixing!

What are the implications? Should we not stage anything until you give the 
all-clear?

Cheers,
Volker


> On 24 Jun 2020, at 09:30, Tony Sarajärvi  wrote:
> 
> Seems the BTRFS file system is permanently busted. We have to recreate the 
> thing and I estimate it taking the rest of the day. Sorry!
>  
> -T
>  
> From: Tony Sarajärvi 
> Sent: Wednesday, June 24, 2020 8:13 AM
> To: Qt development mailing list 
> Subject: Our CI is suffering from malfunctions currently
>  
> Hi
>  
> Yesterday our NFS server’s root partition got filled up by logs apparently. 
> As we got everything fixed, it did not take long for it to start filling up 
> again. Within ours it got filled up again. So, the CI is currently down until 
> we fix things…
>  
> Bear with us
> -Tony

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Our CI is suffering from malfunctions currently

[Tony Sarajärvi]

Seems the BTRFS file system is permanently busted. We have to recreate the 
thing and I estimate it taking the rest of the day. Sorry!

-T

From: Tony Sarajärvi
Sent: Wednesday, June 24, 2020 8:13 AM
To: Qt development mailing list 
Subject: Our CI is suffering from malfunctions currently

Hi

Yesterday our NFS server’s root partition got filled up by logs apparently. As 
we got everything fixed, it did not take long for it to start filling up again. 
Within ours it got filled up again. So, the CI is currently down until we fix 
things…

Bear with us
-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Our CI is suffering from malfunctions currently

[Tony Sarajärvi]

Hi

Yesterday our NFS server’s root partition got filled up by logs apparently. As 
we got everything fixed, it did not take long for it to start filling up again. 
Within ours it got filled up again. So, the CI is currently down until we fix 
things…

Bear with us
-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Drop MSVC 2015 in Qt 5.15?

[Tony Sarajärvi]

How much maintenance will there be, if 5.15.0 is compatible with it already? 
Are you going to remove the code from 5.15.1 that makes it compatible with it? 
Or are you having new features coming in 5.15.1+ that would require additional 
coding to make it MSVC 2015 compatible?

-T

-Original Message-
From: Development  On Behalf Of Jani 
Heikkinen
Sent: Monday, May 25, 2020 9:19 AM
To: Ville Voutilainen ; Thiago Macieira 

Cc: Qt development mailing list 
Subject: Re: [Development] Drop MSVC 2015 in Qt 5.15?

Hi,

MSVC2015 prebuild binaries will be in Qt 5.15.0 (and has been there since Alpha 
release). So from that point of view if think we shouldn't drop it anymore from 
5.15 series. 

br,
Jani 

> -Original Message-
> From: Development  On Behalf Of 
> Ville Voutilainen
> Sent: lauantai 23. toukokuuta 2020 22.00
> To: Thiago Macieira 
> Cc: Qt development mailing list 
> Subject: Re: [Development] Drop MSVC 2015 in Qt 5.15?
> 
> On Sat, 23 May 2020 at 04:01, Thiago Macieira 
> 
> wrote:
> >
> > On Friday, 22 May 2020 02:21:29 PDT Tony Sarajärvi wrote:
> > > Hi
> > >
> > > Now open for discussion: 
> > > https://bugreports.qt.io/browse/QTQAINFRA-
> 3745
> > >
> > > Based on Thiago’s comment that we don’t want to keep supporting it 
> > > for
> > > years:
> > > https://codereview.qt-project.org/c/qt/qtdoc/+/269546
> >
> > My opinion is clear. +1
> 
> Kill it with fire. +1
> ___
> Development mailing list
> Development@qt-project.org
> https://lists.qt-project.org/listinfo/development
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Drop MSVC 2015 in Qt 5.15?

[Tony Sarajärvi]

Hi

Now open for discussion: https://bugreports.qt.io/browse/QTQAINFRA-3745

Based on Thiago’s comment that we don’t want to keep supporting it for years:
https://codereview.qt-project.org/c/qt/qtdoc/+/269546

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Utf8 as the 8bit encoding on Windows

[Tony Sarajärvi]

>>Requiring that Windows version is, unfortunately, not yet acceptable. Many 
>>corporations, Intel included, have a slow Windows upgrade cycle. According to 
>>the WIkipedia page[1], MSFT is offering a 2½-year support cycle for each of 
>>those Windows 10 releases, so we need to support pre-1903 releases until at 
>>least May 11, 2021. That's past the Qt 6 release date.

The CI tends to always update to the latest Windows available, just as we 
upgrade to the latest Ubuntu or macOS when available. But what Qt supports is 
another thing. In 90% of conversations I have, they are mixed together as the 
same thing.

Looking at the Qt documentation, it says that Qt supports any generic Linux 
with GCC 5 or later: 
https://code.qt.io/cgit/qt/qtdoc.git/tree/doc/src/platforms/supported-platforms.qdocinc?h=5.15#n54

The same document also says that Qt supports Windows 10 without mentioning the 
build numbers.

Unless the wording in that document is changed, it means that Qt supports all 
older versions of Windows 10 as well and any Linux distro, as long as it has 
GCC 5, but we just don't have them in the CI.

-T

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Maintenance break tomorrow Apr 1st

[Tony Sarajärvi]

Update.

We are now running on the new setup. I did forget that I had to do some manual 
BIOS / UEFI tweaking on every single host, so it took us this time to get all 
up and running, but we were live at aprox 11:00 EEST.

But that didn't last long... 2 hours later our new NFS backend crashed for what 
it seems like overwhelming load. So we 10-20 folded the CPU and RAM it had and 
heavily reduced the pre-fetching of data as we set up a new host. Now it's been 
up and kicking since.

I hope the VMs themselves are more steady now 

-Tony


-Original Message-
From: Tony Sarajärvi 
Sent: Wednesday, April 1, 2020 7:58 AM
To: development@qt-project.org
Subject: RE: Maintenance break tomorrow Apr 1st

Good morning!

We'll be bringing the infra down now.

A piece of good news I didn't even think of yesterday. The new setup has been 
tested with 5 hosts and they already exist there. So the shutdown will be very 
brief, but at launch you will only see the 5 hosts to build with in the 
beginning. The schedule I presented remains for the rest of the hosts.

It starts now...

-Tony

-Original Message-
From: Tony Sarajärvi 
Sent: Tuesday, March 31, 2020 3:03 PM
To: development@qt-project.org
Subject: Maintenance break tomorrow Apr 1st

Hi

We’ll be bringing the OpenNebula down tomorrow morning at 8:00 EEST for an 
upgrade.

We’ll update OpenNebula itself to version 5.10.1 as well as an implementation 
of using hugepages when running virtual machines.
We’ve seen it possibly eliminating the slowness they’ve been suffering. It 
seems like the slowness is due to vast amounts of memory allocations in a 
fragmented memory space.

Also, the latest firmware’s will be installed on the hosts. The firmware 
install will take aprox 1h per host, but that can be done for all of them in 
parallel.
After that we’ll have to deploy all the hosts, and it will take some time, 
since we have to upload all tier1 images to them. Because of this upload we’ll 
do it in batches of 5-6 hosts at a time.

First ones should be coming online at around 10 o’clock, if everything goes as 
planned. So give it an extra 2 hours as usual 

For our internal employees: the parked VMs have to be nuked. Save your work!!!

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Maintenance break tomorrow Apr 1st

[Tony Sarajärvi]

Good morning!

We'll be bringing the infra down now.

A piece of good news I didn't even think of yesterday. The new setup has been 
tested with 5 hosts and they already exist there. So the shutdown will be very 
brief, but at launch you will only see the 5 hosts to build with in the 
beginning. The schedule I presented remains for the rest of the hosts.

It starts now...

-Tony

-Original Message-
From: Tony Sarajärvi 
Sent: Tuesday, March 31, 2020 3:03 PM
To: development@qt-project.org
Subject: Maintenance break tomorrow Apr 1st

Hi

We’ll be bringing the OpenNebula down tomorrow morning at 8:00 EEST for an 
upgrade.

We’ll update OpenNebula itself to version 5.10.1 as well as an implementation 
of using hugepages when running virtual machines.
We’ve seen it possibly eliminating the slowness they’ve been suffering. It 
seems like the slowness is due to vast amounts of memory allocations in a 
fragmented memory space.

Also, the latest firmware’s will be installed on the hosts. The firmware 
install will take aprox 1h per host, but that can be done for all of them in 
parallel.
After that we’ll have to deploy all the hosts, and it will take some time, 
since we have to upload all tier1 images to them. Because of this upload we’ll 
do it in batches of 5-6 hosts at a time.

First ones should be coming online at around 10 o’clock, if everything goes as 
planned. So give it an extra 2 hours as usual 

For our internal employees: the parked VMs have to be nuked. Save your work!!!

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Maintenance break tomorrow Apr 1st

[Tony Sarajärvi]

Hi

We’ll be bringing the OpenNebula down tomorrow morning at 8:00 EEST for an 
upgrade.

We’ll update OpenNebula itself to version 5.10.1 as well as an implementation 
of using hugepages when running virtual machines.
We’ve seen it possibly eliminating the slowness they’ve been suffering. It 
seems like the slowness is due to vast amounts of memory allocations in a 
fragmented memory space.

Also, the latest firmware’s will be installed on the hosts. The firmware 
install will take aprox 1h per host, but that can be done for all of them in 
parallel.
After that we’ll have to deploy all the hosts, and it will take some time, 
since we have to upload all tier1 images to them. Because of this upload we’ll 
do it in batches of 5-6 hosts at a time.

First ones should be coming online at around 10 o’clock, if everything goes as 
planned. So give it an extra 2 hours as usual 

For our internal employees: the parked VMs have to be nuked. Save your work!!!

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Qt infra services down!

[Tony Sarajärvi]

Hi

We had a small hiccup and the Compellent got filled up. We got people on it 
figuring out how to fix the situation.

Because of this, a large part of our infra services are in frozen state and 
nothing really works atm.

-T

Sent from my Huawei phone
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Plans of platforms in the CI and released binaries

[Tony Sarajärvi]

I did not know about ticket https://bugreports.qt.io/browse/QTBUG-74687 where 
discussions about Windows 7 in Qt6 have been discussed. Nor did apparently no 
one else in our meetings, since we were not told about those discussions.

Thus we got comments in https://codereview.qt-project.org/c/qt/qt5/+/285317 
where we’re about to drop Windows 7 already in Qt5.15 ( according to these 
plans I’ve posted ), and to me it seems like the community doesn’t want to drop 
the support for Windows 7 in Qt 5.15 yet. But as usual, we could still have 
support for it, but drop it from the CI. But I can understand why that isn’t 
wanted.

So without anyone really responding to this thread, am I still to understand 
that you wish to have Windows 7 in Qt 5.15 CI? (and even if I get 2 replies 
that say “hell yeah”, those 2 might be the only 2 people in the world using 
that )

-Tony

From: Development  On Behalf Of Tony 
Sarajärvi
Sent: maanantai 30. joulukuuta 2019 11.20
To: development@qt-project.org
Subject: [Development] Plans of platforms in the CI and released binaries

Hi all!

We’ve had a few meetings here trying to figure out what we will have in the CI 
and ultimately what binaries we will serve in our releases.

Qt could support something that isn’t tested in the CI. We simply have to limit 
the amount of combinations we test due to technical limits of the CI infra. We 
try to cover to most common use cases however.

Also, for us to be able to produce binaries we release, it has to be in the CI, 
and also tested. So those facts are separated as own columns in this sheet I 
attached.

There’s the current 5.13 and 5.14 configurations visible and planned 5.15 and 
very much so planned Qt6.

Please go through it and respond with comments.

-Tony


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Plans of platforms in the CI and released binaries

[Tony Sarajärvi]

Hi all!

We've had a few meetings here trying to figure out what we will have in the CI 
and ultimately what binaries we will serve in our releases.

Qt could support something that isn't tested in the CI. We simply have to limit 
the amount of combinations we test due to technical limits of the CI infra. We 
try to cover to most common use cases however.

Also, for us to be able to produce binaries we release, it has to be in the CI, 
and also tested. So those facts are separated as own columns in this sheet I 
attached.

There's the current 5.13 and 5.14 configurations visible and planned 5.15 and 
very much so planned Qt6.

Please go through it and respond with comments.

-Tony




CI_Platforms.xlsx
Description: CI_Platforms.xlsx
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Datacenter migration on the go - maintenance break 6-8.12.2019

[Tony Sarajärvi]

Hi

The biggest chunk of servers is now moved. One of the final items was 
testresults.qt.io which was briefly down moments ago. If you encounter pages 
that are down or services that don't respond normally, then please contact us 
immediately at qt...@qt.io

-T

From: Development  On Behalf Of EXT Petri 
Puurunen
Sent: torstai 5. joulukuuta 2019 11.33
To: development@qt-project.org
Subject: [Development] Datacenter migration on the go - maintenance break 
6-8.12.2019

Hello,

Part of a infrastructure renewal-process on datacenter there will be a planned 
maintenance break during the weekend 6th - 8th December.
This will possibly affect to some services and might cause inoperability. We 
will try to minimize downtime and do the changes late on evening / early in the 
morning (UTC +2).

On behalf of The QT Company,


Petri Puurunen
System Specialist | Service Operations
Telia Inmics-Nebula






___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Security patches and Firmware update on the Coin host

[Tony Sarajärvi]

Good morning.

We are running security patches and firmware upgrades on the Coin host, just in 
case you are wondering why Coin isn’t responding.

The work should be finished about now, but nothing goes fast when you work with 
upgrading firmware on Dell hardware or update ESXi, so hold onto your horses 
just a bit longer 

-Tony



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Network problems in Finland - CI is down

[Tony Sarajärvi]

The CI is coming back up. I’m surprised if it works though. I most likely 
forgot to fix or start some service 
Not all infra services are back up and running, but the critical ones should be 
if we remembered them correctly.
Work continues in the morning.

Good luck!
-Tony

From: Tony Sarajärvi
Sent: tiistai 27. elokuuta 2019 19.02
To: development@qt-project.org
Subject: RE: Network problems in Finland - CI is down

Hi

IT found out the root cause for our problems. Unfortunately our entire infra 
practically went down along with locking the hard drives our infra machines 
use. The problem that caused all is fixed, but getting everything up and 
running again takes time. We have dozens of servers that are locked that need 
to be fixed. We have people working on it all evening now, and hopefully we get 
things working by tomorrow morning. Local time is now 19:00.

Cheers,
-Tony

From: Development 
mailto:development-boun...@qt-project.org>> 
On Behalf Of Heikki Halmet
Sent: tiistai 27. elokuuta 2019 15.12
To: development@qt-project.org<mailto:development@qt-project.org>
Subject: [Development] Network problems in Finland - CI is down

Hi,

We are having network problems in Finland, which affecting to our CI. People 
are working on this with high priority.



Heikki Halmet
Software Engineer | CI Lead
Release and Test Automation | CI Team
The Qt Company | Finland | Oulu


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Network problems in Finland - CI is down

[Tony Sarajärvi]

Hi

IT found out the root cause for our problems. Unfortunately our entire infra 
practically went down along with locking the hard drives our infra machines 
use. The problem that caused all is fixed, but getting everything up and 
running again takes time. We have dozens of servers that are locked that need 
to be fixed. We have people working on it all evening now, and hopefully we get 
things working by tomorrow morning. Local time is now 19:00.

Cheers,
-Tony

From: Development  On Behalf Of Heikki 
Halmet
Sent: tiistai 27. elokuuta 2019 15.12
To: development@qt-project.org
Subject: [Development] Network problems in Finland - CI is down

Hi,

We are having network problems in Finland, which affecting to our CI. People 
are working on this with high priority.



Heikki Halmet
Software Engineer | CI Lead
Release and Test Automation | CI Team
The Qt Company | Finland | Oulu


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] How to document distros and compilers that Qt supports

[Tony Sarajärvi]

Hi all!

We've been going back to defining what "supported platform" means many times. 
This time however I want to loosen the reins a bit:
https://codereview.qt-project.org/c/qt/qtdoc/+/269546

We stumble upon a problem where the distro itself isn't supported and / or its 
repositories are in worst cases removed, but we claim that our LTS release 
still supports it, and nothing newer. Thus I suggest we change the terminology 
to say that Qt supports anything beyond that was existing when we released Qt. 
Note, that I don't say "we support" as The Qt Company, but what Qt supports, 
and that's what the documentation is about really.

MacOS is another good example where we claim to support the latest 3 available 
at the time of release. But in 4 years of an LTS release we actually don't have 
a single one of those 3 left that were out when we released 5.x.0. Are we now 
supporting the ones documented when the initial minor release came out, or the 
latest 3 available at the time of the latest patch release? Especially when 
talking about macOS people tend to update to the latest one, so why support 
older ones.

Or should we go ahead and rewrite that documentation even more? We say that Qt 
5.12 supports Ubuntu 18.04 with GCC 7.3, but then we say that Qt supports 
Generic Linux with just about any compiler. Doesn't Ubuntu sit in this category 
already? Wouldn't it be enough that we say Linux with GCC 4.8+?

And different features have different library dependencies. Nothing prevents 
you from using Qt 5.12 on SLES 12, but you won't get QtWayland there. Or bluez 
this and that version is needed for specific Bluetooth features to enable which 
aren't necessarily available in a stock RHEL 6.x series. So let's add library 
dependencies to the list?

I think that in general if there is a bug in Qt for any major distro, the bug 
will get fixed. How often is the response that Qt does't support that? The 
documentation part should definitely define the minimum compiler version that 
is needed, just as the talk currently about Qt6 that's going on in a different 
thread. But beyond that to what comes to the distros we support, I don't think 
we should document it so tightly what we support, because of the reasons above.

And just as a reminder here, the stuff that's in the CI have "nothing" to do 
with what Qt supports, or even what The Qt Company supports. The CI tries to 
cover the most common use cases, the latest compilers to cover where we are 
going, and the minimum set so that we don't start regressing from the rear. And 
Qt packages (Qt binaries) can even be done with a distro that's been modified 
with libraries built from sources and thus making it a custom distro in a way, 
only so that The Qt Company's Linux binaries work in most Linuxes. Not an issue 
in macOS or Windows worlds naturally.

Regards,
-Tony


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Supported compilers for Qt 6

[Tony Sarajärvi]

How about ICC 19?
https://software.intel.com/en-us/articles/intel-c-compiler-190-for-linux-release-notes-for-intel-parallel-studio-xe-2019

-T

-Original Message-
From: Development  On Behalf Of Ville 
Voutilainen
Sent: maanantai 12. elokuuta 2019 12.24
To: Lars Knoll 
Cc: Qt development mailing list 
Subject: Re: [Development] Supported compilers for Qt 6

On Mon, 12 Aug 2019 at 12:16, Lars Knoll  wrote:

> > We gain a slightly better baseline wrt. bugs; GCC for instance 
> > doesn't backport to closed branches, and GCC 7 is already closed, 
> > and 8 will follow relatively quickly. While we can't keep up with 
> > that during our (especially LTS) release cycle, at least we're not 
> > targetting already-obsolete versions too badly. :)
>
> So I assume that’s a vote for the alternative proposal? :)

Yes. I prefer the alternative proposal to the original one. :) 
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Maintenance break coming up at 12:00 EEST, i.e. 20 minutes

[Tony Sarajärvi]

Vmware isn't doing the inflating of the virtual disk as fast as we suspected. 
Looks like we have to wait another 1.5h. Sorry for the delay.

-Tony

From: Tony Sarajärvi
Sent: keskiviikko 31. heinäkuuta 2019 11.43
To: development@qt-project.org
Subject: Maintenance break coming up at 12:00 EEST, i.e. 20 minutes

Hi

We have a quieter period going on right now and we have a small infra related 
thing we want to get done. So we'll perform a small maintenance break in 20 
minutes which will bring the CI down. I'll make a backup clone before I 
proceed, so I'll give it an estimate of 1h. Perhaps 2h.

-Tony


Tony Sarajärvi
CI Tech Lead

The Qt Company
Elektroniikkatie 13
90590, Oulu, Finland
tony.saraja...@qt.io<mailto:tony.saraja...@qt.io>
+358 50 482 1416
http://qt.io
[The Qt Company logo]<http://qt.io/>
[Facebook]<http://www.facebook.com/Qt>
[Twitter]<http://www.twitter.com/qtproject>
[LinkedIn]<https://www.linkedin.com/company/the-qt-company/>
[Google+]<https://plus.google.com/104580575722059274792>
[YouTube]<https://www.youtube.com/QtStudios>


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Maintenance break coming up at 12:00 EEST, i.e. 20 minutes

[Tony Sarajärvi]

Hi

We have a quieter period going on right now and we have a small infra related 
thing we want to get done. So we'll perform a small maintenance break in 20 
minutes which will bring the CI down. I'll make a backup clone before I 
proceed, so I'll give it an estimate of 1h. Perhaps 2h.

-Tony


Tony Sarajärvi
CI Tech Lead

The Qt Company
Elektroniikkatie 13
90590, Oulu, Finland
tony.saraja...@qt.io<mailto:tony.saraja...@qt.io>
+358 50 482 1416
http://qt.io
[The Qt Company logo]<http://qt.io/>
[Facebook]<http://www.facebook.com/Qt>
[Twitter]<http://www.twitter.com/qtproject>
[LinkedIn]<https://www.linkedin.com/company/the-qt-company/>
[Google+]<https://plus.google.com/104580575722059274792>
[YouTube]<https://www.youtube.com/QtStudios>


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] A monologue about platforms in the Qt world

[Tony Sarajärvi]

by RedHat themselves? 
And there's the big question #1!

Because of ignorance we didn't think about this when we documented things. We 
just blindly say that Qt 5.12 supports RHEL 7.4, because that's what we have in 
the CI! Yeah, but are we really supporting it in 2021, when RedHat themselves 
have pulled the plug on it over 2 years before that? Could we just update to 
the latest RHEL after all? Especially since we didn't use RHEL 7.4 to begin 
with, but our own distro as we modified the original RHEL?
Tony Sarajärvi
CI Tech Lead

The Qt Company
Elektroniikkatie 13
90590, Oulu, Finland
tony.saraja...@qt.io
+358 50 482 1416
http://qt.io 









___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Qt infra "weekly" report

[Tony Sarajärvi]

Hi

State of the CI

  *   Hardware and host wise all seem pretty ok. We had our first host crash in 
a month sadly. Reported it to Canonical just in case: 
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1814628
  *   Running network test servers as containers seem to cause such problems 
that our agent on the VM gets restarted and our state machine doesn't currently 
like that. We have a fix to handle these situations, but it requires us to 
recreate our tier1 images with a new bootstrap agent.
  *   Coin on the other hand seems to be suffering from random archive creation 
problems and what not. Sadly these affect the throughput of the system quite 
heavily currently.

Firewall changes

  *   The changes are almost all done. The next changes that will cause a small 
maintenance break in the CI will be on week 11. And those changes will be the 
last ones, and we can then remove the old firewall from the rack. The changes 
on that week will cause the external IP address of testresults.qt.io to change. 
Domain name remains the same though.

Changes to Coin servers

  *   Coin is currently running on a dedicated hardware using internal SSDs. We 
now have the capacity to change that so that we use Compellent hard drives. 
With this move we can start cloning the Coin VM and play around with multiple 
instances of it if necessary. This is something we will do in the near future, 
but we haven't decided a specific date for the big move yet.

Cheers!
-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Qt infra "weekly" report

[Tony Sarajärvi]

Hi

State of the CI

  *   Currently all OK again. We had issues with Coin update breaking the code 
that re-runs flaky tests, but that was caught and fixed. Then we also had 
Google changing the way Android’s SDK manager is used, so we had to create a 
patch for all branches. 5.12 is at the time of writing fixed, and dev is being 
built currently.
  *   The SSDs were installed in the Compellent, so we can proceed with a few 
minor changes we had planned. Now we shouldn’t have to use the excuse that we 
run out of disk space as long as we grow the disks on the servers 
  *   Last evening we moved the third big set of infra over to the new 
firewall. After a few hours of down time the new firewall picked up the load. 
This is visible to the users in such a way that our public IP address changed 
for those under the new firewall.

Cheers!
-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Maintenance break right now

[Tony Sarajärvi]

Hi

If I forgot to tell you that we have a maintenance break, so did I also forget 
to tell you that it is over. Aprox 8-9 hours ago I launched the CI and builds 
began passing again.

-Tony

From: Tony Sarajärvi
Sent: tiistai 22. tammikuuta 2019 18.24
To: development@qt-project.org
Subject: Maintenance break right now

Sorry all, but we forgot to publicly announce that our firewall change is 
proceeding to the next phase right now. This means that for the next hours some 
of our web pages are going offline and the CI is brought down.

I'll inform you when all is up and running again.

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Maintenance break right now

[Tony Sarajärvi]

Sorry all, but we forgot to publicly announce that our firewall change is 
proceeding to the next phase right now. This means that for the next hours some 
of our web pages are going offline and the CI is brought down.

I'll inform you when all is up and running again.

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Good idea to update Windows 10 to 1809 - Redstone 5?

[Tony Sarajärvi]

Hi

We have this thing that we like to try to update things to the latest. But 
regarding Windows 10 I was told we shouldn't perhaps do that, because Qt's code 
will select and different path depending on the API levels available from 
Windows.

Does this community want to have their say and give thumbs up or down?

https://bugreports.qt.io/browse/QTQAINFRA-2255

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Qt infra "weekly" report

[Tony Sarajärvi]

Hi

Just as we say everything is working, we had 2 crashes yesterday in openNebula. 
As always when you fix one bottle neck (the hosts in this case) the bottle neck 
is somewhere else. But hey…it worked for over 2 weeks 

-Tony

From: Tony Sarajärvi
Sent: maanantai 14. tammikuuta 2019 12.09
To: 'development@qt-project.org' 
Subject: Qt infra "weekly" report

Hi

State of the CI

  *   18.04 hosts seem to be working! The virtual machines don’t slow down now 
and we’re hoping this will solve a few of the mysteries we’ve had.
 *   The feedback we’ve received have all been positive. The infra is 
working better than ever now.
  *   The vmx instructions were enabled on the move to 18.04 and nested VMs are 
already being used.
  *   If we look at the ping latencies from network to network, it seems like 
our new firewall is doing its job. Most spikes we get tend to be related to the 
internet now (we ping AWS).
  *   Our Compellent got a SW upgrade, which should increase its performance. 
However, since we didn’t pull 100% out of it, it’s hard to say if it really 
give us anything. The improvement might be seen as faster tier1 images if we 
are under heavy load.
  *   This week Dell should come and install more SSDs into our Compellent. 
This enables us to do a few changes we have been postponing. Again, the goal is 
to make the CI more robust.

Cheers!
-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Qt infra "weekly" report

[Tony Sarajärvi]

Hi

State of the CI

  *   18.04 hosts seem to be working! The virtual machines don't slow down now 
and we're hoping this will solve a few of the mysteries we've had.
 *   The feedback we've received have all been positive. The infra is 
working better than ever now.
  *   The vmx instructions were enabled on the move to 18.04 and nested VMs are 
already being used.
  *   If we look at the ping latencies from network to network, it seems like 
our new firewall is doing its job. Most spikes we get tend to be related to the 
internet now (we ping AWS).
  *   Our Compellent got a SW upgrade, which should increase its performance. 
However, since we didn't pull 100% out of it, it's hard to say if it really 
give us anything. The improvement might be seen as faster tier1 images if we 
are under heavy load.
  *   This week Dell should come and install more SSDs into our Compellent. 
This enables us to do a few changes we have been postponing. Again, the goal is 
to make the CI more robust.

Cheers!
-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CI upgrade

[Tony Sarajärvi]

>
>On Thursday, 3 January 2019 08:10:14 -02 Tony Sarajärvi wrote:
>> openNebula uses QEMU that uses libvirt to create a VM using KVM:

>Despite the name, in this mode QEMU is not emulating anything. When assisted 
>by KVM this is real hardware virtualisation >and qemu only serves to provide 
>the virtual HW. This is called "qemu system" because it emulates the entire 
>system and runs >a full OS.

>That's very different from "qemu user" which we use to test the ARM builds on 
>a regular x86 Linux host. That one does not >emulate the whole system, just 
>the CPU and it translates system calls from one ABI to the other. This mode is 
>far buggier since >the system call translation layer is very incomplete.

>So my answer stands: I don't see how qemu could be involved.

Alright, thanks!  Then we have put our thinking hats and on try to figure out 
why this upgrade began causing different problems, like crashing tests.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CI upgrade

[Tony Sarajärvi]

>On Wednesday, 2 January 2019 04:34:43 -02 Tony Sarajärvi wrote:
>> agent:2019/01/01 20:15:26 build.go:193: Config: Using QtTest library 
>> 5.12.0, Qt 5.12.0 (x86_64-little_endian-lp64 shared (dynamic) release 
>> build; by GCC
>> 5.3.1 20160406 (Red Hat 5.3.1-6))

>This is a native x86-64 build and run. I don't see how it could be related to 
>QEMU.

>The other one you said is Windows 10, so I don't see how it would be QEMU 
>either.

openNebula uses QEMU that uses libvirt to create a VM using KVM:

root@calm-squid:/var/log/libvirt/qemu# cat one-2474429.log
2018-12-31 06:28:04.959+: starting up libvirt version: 3.6.0, package: 
1ubuntu6.8 (Marc Deslauriers  Wed, 23 May 2018 
13:23:59 -0400), qemu version: 2.10.1(Debian 1:2.10+dfsg-0ubuntu3.8), hostname: 
calm-squid
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 
QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name 
guest=one-2474429,debug-threads=on -S -object 
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-8102-one-2474429/master-key.aes
 -machine pc-i440fx-artful,accel=kvm,usb=off,dump-guest-core=off -cpu 
qemu64,ssse3=on,svm=off,vmx=off -m 6144 -realtime mlock=off -smp 
2,sockets=1,cores=2,threads=1 -uuid edb33d53-187d-4448-9835-54c303297131 
-no-user-config -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-8102-one-2474429/monitor.sock,server,nowait
 -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown 
-boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device 
ahci,id=sata0,bus=pci.0,addr=0x4 -drive 
file=/var/lib/one//datastores/0/2474429/disk.0,format=qcow2,if=none,id=drive-sata0-0-0,cache=unsafe
 -device ide-hd,bus=sata0.0,drive=drive-sata0-0-0,id=sata0-0-0,bootindex=1 
-drive 
file=/var/lib/one//datastores/0/2474429/disk.1,format=raw,if=none,media=cdrom,id=drive-sata0-0-1,readonly=on
 -device ide-cd,bus=sata0.1,drive=drive-sata0-0-1,id=sata0-0-1 -netdev 
tap,fd=31,id=hostnet0,vhost=on,vhostfd=33 -device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=02:06:d3:d3:d0:ee,bus=pci.0,addr=0x3 
-vnc 0.0.0.0:29394 -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
2018-12-31T06:29:41.783344Z qemu-system-x86_64: terminating on signal 15 from 
pid 6092 (/usr/sbin/libvirtd)
2018-12-31 06:29:43.984+: shutting down, reason=destroyed


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CI upgrade

[Tony Sarajärvi]

Hi

I'd like to know your opinion about these failures we've been seeing during the 
last 24 hours. Are they related to QEMU being updated perhaps?

on RHEL 7.4:
agent:2019/01/01 20:15:25 build.go:193: /home/qt/work/install/bin/qmltestrunner
agent:2019/01/01 20:15:26 build.go:193: * Start testing of 
qmltestrunner *
agent:2019/01/01 20:15:26 build.go:193: Config: Using QtTest library 5.12.0, Qt 
5.12.0 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 5.3.1 
20160406 (Red Hat 5.3.1-6))
agent:2019/01/01 20:15:26 build.go:193: PASS   : 
qmltestrunner::shadersource-dynamic-sourceobject::initTestCase()
agent:2019/01/01 20:15:26 build.go:193: BPASS  : 
qmltestrunner::shadersource-dynamic-sourceobject::test_endresult()
agent:2019/01/01 20:15:26 build.go:193: PASS   : 
qmltestrunner::shadersource-dynamic-sourceobject::cleanupTestCase()
agent:2019/01/01 20:15:26 build.go:193: PASS   : 
qmltestrunner::shadersource-dynamic-shadersource::initTestCase()
agent:2019/01/01 20:15:26 build.go:193: make: *** [check] Segmentation fault 
(core dumped)

on Windows 10:
agent:2019/01/01 03:56:28 build.go:193:   call 
C:\Users\qt\work\qt\qt3d\tests\auto\render\qrendertargetoutput\target_wrapper.bat
  release\tst_qrendertargetoutput.exe -o 
C:\Users\qt\work\testresults\tests\auto\render\qrendertargetoutput782111623,xml 
-o -,txt
agent:2019/01/01 03:56:28 build.go:193: jom: 
C:\Users\qt\work\qt\qt3d\tests\auto\render\qrendertargetoutput\Makefile.Release 
[check] Error 2
agent:2019/01/01 03:56:28 build.go:193: jom: 
C:\Users\qt\work\qt\qt3d\tests\auto\render\qrendertargetoutput\Makefile 
[release-check] Error 2
agent:2019/01/01 03:56:28 build.go:237: Process finished with error: exit 
status 2
agent:2019/01/01 03:56:28 build.go:196: Error reading from stdout/err: exit 
status 2
agent:2019/01/01 03:56:28 build.go:435: Executing scheduled instruction 1 of 2 
- Upload all core dumps if there are some

I also reported these 3 autotest crashes before we switched over to the new 
software stack:
https://bugreports.qt.io/browse/QTBUG-72781
https://bugreports.qt.io/browse/QTBUG-72782
https://bugreports.qt.io/browse/QTBUG-72790

-Tony
From: Tony Sarajärvi 
Sent: maanantai 31. joulukuuta 2018 8.28
To: development@qt-project.org
Subject: CI upgrade

Good morning.

Since our last verifications of 3 main branches show green light to make the 
upgrade of the CI, I'll go ahead and proceed right away. The switch over will 
only take seconds, and I'll start monitoring the system today to see that it 
actually works.

What changes:

  *   Our hosts running the virtual machines get updated from Ubuntu 17.10.x -> 
18.04.x (Maintained by MAAS and its PXE boot)
 *   libvirt and QEMU API versions get updated from 3.6.0 -> 4.0.0
 *   Hypervisor from QEMU 2.10.1 -> 2.11.1
  *   OpenNebula 5.4.1 -> 5.4.13
  *   MAAS 2.2.2 -> 2.4.2
 *   And the host MAAS runs on from Ubuntu 16.04.3 -> 18.04.1

Most of all we have high hopes that these new versions will bring a lot of 
stability to the CI.

-Tony


___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Maintenance break tomorrow morning on the 19th Dec

[Tony Sarajärvi]

Hi

We’ll perform some firewall switch overs tomorrow morning. While at it, we’ll 
move some infra services from one address space to an other enabling us to 
create better rules for traffic.

We’ll probably perform a shutdown of the CI at 9 o’clock EET, and we’ll 
probably be done with it in an hour…give or take as usual 

-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Qt infra "weekly" report

[Tony Sarajärvi]

Hi

State of the CI

  *   We're still running Ubuntu 17.10.
  *   The setup of the secondary CI network is complete (as far as we know 
currently)
  *   There was a small bug in MAAS which took some time to find which delayed 
my work for about a day. I'm back on track and new hosts in new network seem to 
be up. They didn't call openNebula yet, so I'm pretty sure we have a DNS route 
& firewall port issue here. Expecting to have trial builds running by the end 
of the day.

Our Compellent will receive software update tomorrow. This is an enabler to add 
more disk space later on. The reboot shouldn't affect any running systems.


Cheers!
-Tony
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Scheduled maintenance

[Tony Sarajärvi]

Good morning

This morning we had a scheduled maintenance "break" where we started using a 
new firewall. This shouldn't have caused interruptions if we had preconfigured 
everything correctly. That however wasn't the case. We have a few ports which 
we need to open in the wall so that traffic goes through. For that reason the 
CI is currently now working. We hope to get it back online in a moment.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] Qt infra "weekly" report

[Tony Sarajärvi]

Hi

With me being extremely busy lately, I've forgotten all about this report I 
promised to send out. Sorry about that!

State of the CI

  *   The kernels we updated back in august were a minor success. They don't 
crash anymore, and that leaves us with more time to do something else than just 
rebooting hosts. Unfortunately even the current Ubuntu 17.10 as host doesn't 
run Windows 10 VMs very well, and they tend to cause problems, eventually 
slowing down the also the other VMs on the same hosts. It's troublesome and we 
are working on getting Ubuntu 18.04 in place.

Update of openNebula

  *   Version 5.6.1 of ONE seems to handle images a bit differently. I'm 
currently working on figuring out how to get those working. I did manage to get 
the basic setup working so that it actually successfully runs provisioning. But 
after that ONE stops talking to the hosts and I'm lost to why that is.

Update of MAAS and the Hosts

  *   The network setup of our CI infra didn't work as we had planned and it 
took quite a bit of time to figure out why things actually work. Using this 
information and reflecting that on the new dev-network didn't help much, since 
the dev-network with updated distros and SW worked differently again. Although 
being a copy to begin with. Now we know how everything works and what to 
configure and where. So this is close to the point where we can start updating 
production environments.

A new firewall has been installed and we are currently moving networks over to 
the new FW one by one. The old FW had problems coping with our load and it 
dropped packages left and right. The new one is a much more powerful one, so at 
least those problems should go away.

Also we are doubling the amount of disk space we have in the Compellent, 
enabling us to use the space we have more freely. We have plans already to move 
some of the CI storages over to the Compellent from local storage, so that we 
don't run out of disk space every now and then. This also enables us to store 
artifacts that are a bit older. This then makes those builds faster that can 
still reuse the artifacts.

That's it for today, have a nice weekend!
-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] CI machines out of disk space

[Tony Sarajärvi]

Yeah, we were already wondering about that. Before the builds starts, it has 
160 GB of free disk space. Could the build really eat up all that?

We've had kernel crashing on the host side of the VM, and that has caused the 
VM to put its disks on read-only mode. That has then created errors that can't 
write on disk. That's different error message though. Just wondering if the 
error message this time is wrong, but from the same reason... 

-Tony

-Original Message-
From: Development  On 
Behalf Of Thiago Macieira
Sent: keskiviikko 31. lokakuuta 2018 6.17
To: development@qt-project.org
Subject: [Development] CI machines out of disk space

https://testresults.qt.io/logs/qt/qtbase/
4d1c701336289c6c70c86b1bdea0324214a5687a/
LinuxUbuntu_18_04x86_64LinuxQEMUarmv7GCCqtci-linux-Ubuntu-18.04-x86_64-
ea77a2DeveloperBuild_DisableTests/2a48a3b7dcb9652b1532a3ad14de7580b5b9b1ee/
build_1542285435/log.txt.gz

agent:2018/10/31 03:51:26 build.go:193: {standard input}: Assembler messages:
agent:2018/10/31 03:51:26 build.go:193: {standard input}: Fatal error: can't 
write 76 bytes to section .text of .obj/qgraphicssceneindex.o because: 'No 
space left on device'

--
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center



___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Coin go version has been updated

[Tony Sarajärvi]

I’m also interested in what you thought had fixed the issues, as they clearly 
state they remove the compatibility with the old distros?

And what features do we need from the new go-lang versions?

-T

From: Development  On 
Behalf Of Aapo Keskimölö
Sent: lauantai 20. lokakuuta 2018 15.37
To: development@qt-project.org
Cc: Coin Team 
Subject: Re: [Development] Coin go version has been updated

It seems that the mac machines are still having the same issue with this go 
version:/

I have reverted the go version coin master back to 1.8.3.

Kind regards/Ystävällisin terveisin,
Aapo Keskimölö

On 20 Oct 2018, at 13.51, Aapo Keskimölö 
mailto:aapo.keskim...@qt.io>> wrote:
Hi all,


I have bumped the go compiler on the master machine 1.8.3->1.11.1 because the 
current go version is not supporting all features that are required to continue 
development of Coin.


In the past this caused issues with mac builds: 
https://bugreports.qt.io/browse/QTQAINFRA-2172


If you experience any problems with your integrations that might be related to 
the new go version, you can reply to this email or in internal IRC.
Kind regards/Ystävällisin terveisin,
Aapo Keskimölö
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Shutting down the CI right now!

[Tony Sarajärvi]

Liang: Ok, one thing is my bad in all of this. I told you yesterday after noon 
that I've reverted the host-passthrough thing. Well...yeah, the configuration. 
I just forgot to host the setting to the hosts. Just realized it. So _now_ the 
old settings of qemu64 instead of host-passthrough is in effect. See if that 
helps.

-Tony

-Original Message-
From: Liang Qi 
Sent: perjantai 12. lokakuuta 2018 8.21
To: Tony Sarajärvi 
Cc: development@qt-project.org
Subject: Re: [Development] Shutting down the CI right now!

Thanks for the work, and sometimes in spare time.

But there is no success for qt5 integrations and qtbase in any branch since the 
work from Wednesday night. See https://bugreports.qt.io/browse/QTQAINFRA-2273 , 
Angle building timeout (on Windows)

So I suggest not to restage in above areas at least before the issue got fixed.

Best Regards,
Liang

> On 12 Oct 2018, at 07:16, Tony Sarajärvi  wrote:
> 
> Wow… ok it took longer than expected (as usual). Not only did we have to 
> remove snapshots, we also had to run unmap on the Compellent, so this whole 
> thing took a while. But nothing got lost, thanks to a last minute 
> intervention.
>  
> So, the CI is up again, and good luck stage storming it
>  
> -Tony
>  
> From: Tony Sarajärvi 
> Sent: torstai 11. lokakuuta 2018 14.49
> To: development@qt-project.org
> Subject: Shutting down the CI right now!
>  
> Hi
>  
> Due to a lot of changes while developing and snapshots being huge, our 
> Compellent backend is nearly 100% full. I’ll shut down the CI for a few hours 
> to avoid everything breaking due to lack of disk space. We’ll be back after 
> snapshots have been deleted.
>  
> -Tony
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Shutting down the CI right now!

[Tony Sarajärvi]

Wow... ok it took longer than expected (as usual). Not only did we have to 
remove snapshots, we also had to run unmap on the Compellent, so this whole 
thing took a while. But nothing got lost, thanks to a last minute intervention.

So, the CI is up again, and good luck stage storming it

-Tony

From: Tony Sarajärvi
Sent: torstai 11. lokakuuta 2018 14.49
To: development@qt-project.org
Subject: Shutting down the CI right now!

Hi

Due to a lot of changes while developing and snapshots being huge, our 
Compellent backend is nearly 100% full. I'll shut down the CI for a few hours 
to avoid everything breaking due to lack of disk space. We'll be back after 
snapshots have been deleted.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] Shutting down the CI right now!

[Tony Sarajärvi]

Hi

Due to a lot of changes while developing and snapshots being huge, our 
Compellent backend is nearly 100% full. I'll shut down the CI for a few hours 
to avoid everything breaking due to lack of disk space. We'll be back after 
snapshots have been deleted.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] Maintenance break for the CI

[Tony Sarajärvi]

Hi

Let's do a small maintenance break for the CI right away. It's quite unstable 
currently, and a reset of all the hosts usually helps. While I do that, I'll 
also enable nested virtual machines. This will enable a few things we are 
working on.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] Bug in Ubuntu 18.04 provisioning preventing builds in dev branch currently

[Tony Sarajärvi]

Hi

A small bug in the provisioning scripts of Ubuntu 18.04 prevents the builds 
from going through in dev branch currently. We run apt update only after we try 
to install the packages. So it's trying to install a package so old it doesn't 
exist in the repos anymore.

A fix has landed in 5.12 already (a side effect of another change), so a merge 
from 5.12 -> dev will fix the situation.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] MSVC 2017 32bit pre-built binaries for Qt 5.12 (replacing MSVC2015 32bit)

[Tony Sarajärvi]

We currently work like this in 5.12

For all submodule builds we do:
MSVC 2015 x64 - a DeveloperBuild that isn't a DebugAndRelease build that's 
_not_ exported to release
MSVC 2015 x86 - a DebugAndRelease build that's exported to release
MSVC 2015 x64 - a DebugAndRelease build that's exported to release
MSVC 2017 x64 - a DebugAndRelease build that's exported to release

on top of that for every qt5.git build we do an additional
MSVC 2017 x86 - a DebugAndRelease build that's _not_ exported to release

So we actually build MSVC with both 2015 and 2017 for both x86 and x64. So it's 
just a matter of selecting which we want to export to release.

-Tony

-Original Message-
From: Development  On 
Behalf Of Allan Sandfeld Jensen
Sent: tiistai 2. lokakuuta 2018 21.05
To: development@qt-project.org
Subject: Re: [Development] MSVC 2017 32bit pre-built binaries for Qt 5.12 
(replacing MSVC2015 32bit)

On Dienstag, 2. Oktober 2018 14:47:42 CEST Alex Blasche wrote:
> Hi,
> 
> We had this discussion for Qt 5.11 already but it is time to decide 
> for Qt 5.12. The discussion details can be found under
> 
> https://bugreports.qt.io/browse/QTBUG-63708
> 
> Based on download figures and dropped MSVC2015 support for webengine 
> we plan to change MSVC2015 32bit binaries over to MSVC2017 32bit. I 
> realize this will not be happy news for everybody but considering the 
> LTS implications of Qt 5.12 it makes sense. Nevertheless if you 
> believe you have strong counter arguments please comment on the issue in Jira.
> 
Would it be possible to just replace 32bit MSVC2015 with 32bit MSVC2017 on the 
CIs that test the individual modules, but keep both for qt5.git if we have the 
resources for it?

`Allan


___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] QT infra weekly report 2018-09-27

[Tony Sarajärvi]

Hi

State of the CI
- Somewhere along the axis of Ubuntu 17.10 - KVM we're hitting a SNAFU in a 
cycle of about 2 weeks. After that the virtual machines done on our hosts start 
slowing down enormously. We're working on replacing 17.10 with 18.04, but the 
work has been slowed down by us giving CPR to the current setup. We now have 
good redeploy scripts so that us redeploying these hosts weekly doesn't slow us 
down much.

Update of openNebula, MAAS & the build hosts
- Scripts have been updated and we have a working MAAS and openNebula. The host 
deployment to 18.04 is our current issue and we're working on figuring out how 
to deploy those.

- Coin is constantly running out of disk space, and with the current setup we 
are lacking redundancy of the hardware. We're planning to expand and / or move 
the storage to the Compellent storage. This enables us to start using standard 
rack servers for running Coin, thus giving us the redundancy we want, as any of 
the current build servers works as backup.

Rest of the infra:
- A new firewall has arrived. We've begun the swap over to the new one. Current 
cable connections documented and cluster preconfigured off site. When 
completed, it will be sent on site and connected to racks.
- We have one more infra server (the backup one I mentioned in the previous 
mail) taking off load of the 3 running ones. This shouldn't affect you at all. 
Mostly we hope to keep the loads more balanced so that in peak infra usages we 
don't run out of capacity. Shouldn't affect the CI at all.
- All public IP addresses in use. Andrew L. configuring new IRC server  which 
is going to be using the old one. We need maintenance break for  it when taking 
it into a production.

-Tony

Click _here_ if you wish to unsubscribe.
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] Retry problems in the CI

[Tony Sarajärvi]

Hi

You all might have noticed that it's almost impossible to get anything through 
the CI, especially into QtBase. The reason seems to be our 'retry' mechanism 
that has vanished. We don't retry failing cases currently. Thus every flaky 
case that fails, now immediately fails the entire integration, instead of being 
retried to see if it passes on the consequent builds.

We're working on finding the culprit. Hang on there!

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] Coin production update

[Tony Sarajärvi]

Coin production was updated today at Fri Sep 21 7:00 EEST 2018 with new 
baseline.

For CI related issues, you may create bug report as instructed in 
https://wiki.qt.io/Reporting_Bugs#Reporting_bugs_for_Coin and/or consult the 
internal #qtci IRC-channel.

See change-list attached for related patches.

Br,
-Tony


product_baseline_20180921.log
Description: product_baseline_20180921.log
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] CI still down a while

[Tony Sarajärvi]

Ok, systems seem to be up and running. There still might be services here and 
there that we haven't caught yet being offline. Keeping a third eye on the CI 
today for you.

-Tony

From: Tony Sarajärvi
Sent: keskiviikko 12. syyskuuta 2018 7.32
To: 'development@qt-project.org' 
Subject: CI still down a while

Hi

We encountered a problem last evening Finnish time. We overallocated our hard 
disk space on our infra backend and it led to us running out of disk space. All 
services were brought to a halt. We're restoring systems right now. A while 
longer and we'll be back online.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] CI still down a while

[Tony Sarajärvi]

Hi

We encountered a problem last evening Finnish time. We overallocated our hard 
disk space on our infra backend and it led to us running out of disk space. All 
services were brought to a halt. We're restoring systems right now. A while 
longer and we'll be back online.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] Recall: QT infra weekly report

[Tony Sarajärvi]

Tony Sarajärvi would like to recall the message, "QT infra weekly report".
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] QT infra weekly report

[Tony Sarajärvi]

Hi

We’ve decided to start sharing a bit more information about the state of our 
infra and what we’re up to. This will be a brief e-mail that will cover the 
main topics. 

State of the CI
- We've been running 2 weeks now with a stable kernel. The hosts have actually 
been up without reboots now for 2 weeks, which must be a record for us. This 
means, we won't be touching them now as they work. 

Update of OpenNebula
- 5.6 has been installed. Scripting it will require still some work, as the 
same deployment procedure doesn’t work right of the bat. Will be tested in the 
dev environment.

Update of CI hosts to Ubuntu 18.04 and upgrade MAAS
- Current MAAS version doesn't support Ubuntu 18.04, so we need to update MAAS.
- A newer version of MAAS requires it to run on Ubuntu 18.04. So we need to 
update the host of MAAS as well (no chicken and the egg problem here).
- This all lead us to need a dev/sandbox environment with all new VLAN tags. 
That's done now and we can now test everything before breaking production 
(again).
- One host has been allocated into the dev environment (So CI is down one from 
33 we have in total)

Mac hardware in the CI
- Currently OpenNebula doesn’t have Mac hardware attached to it. A Mac Pro 
could be attached to it for testing, but we don’t have a PDU (power 
distribution unit) for it yet. This needs some further planning what we really 
want to do with these.
 
Our packet loss situation
- We’ve had problems with odd packet loss. We suspected that our firewall was 
to blame, and surely it hit high peaks in load.
- QoS was disabled, as manufacturer told us it will increase load on the 
firewall by 20-50%
- Packet loss was reduced from 5-8% down to 0.5%.
- If we can pin point the culprit of the remaining 0.5% to be the firewall, we 
might get a budget to replace those firewalls ASAP (work ongoing).

Our Blade infrastructure is going to have a backup Blade allocated to the pool 
currently running our infra. It needed more RAM however, and the order has been 
placed.

Coin’s server will be allocated terabytes of disk space from the Compellent to 
cope with vast amount of builds produced by product repo development.
This became an issue as the Compellent is getting quite full, so we’re 
currently dumping old pre Qt 5.6 images into USB drives for disposal (read: 
storage).

You might have seen issues with Licenses in the INTEGRITY builds. These have 
now been resolved, but if you see more of these or have any license related 
questions, contact mailto:it.helpd...@qt.io and CC andrey.le...@qt.io who takes 
care of them and mailto:qt...@qt.io (that’s us) as well.  Naturally we keep an 
eye on builds as well, but quite often is the case such that people staging the 
commits see the problem right after the failure and not the next time we glance 
Coin's front page.


This first e-mail had some longer back stories attached to the points above. In 
the future I'll try to be more brief, as you all will know the back stories 
from this e-mail 

-Tony

Click _here_ if you wish to unsubscribe.
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] Coin momentarily down

[Tony Sarajärvi]

Hi

We have a small glitch in the CI currently. We keep spawning VMs like crazy. 
The CI is down for a while as we figure out why and clean up the mess.
Shouldn’t take more than say 30 minutes…. shouldn’t 

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Upgrade the XCode running in the CI's macOS 10.11

[Tony Sarajärvi]

The timing  https://codereview.qt-project.org/#/c/237412/

-Tony

-Original Message-
From: Development  On 
Behalf Of Thiago Macieira
Sent: keskiviikko 22. elokuuta 2018 3.15
To: development@qt-project.org
Subject: Re: [Development] Upgrade the XCode running in the CI's macOS 10.11
Importance: High

On Sunday, 8 July 2018 17:15:36 PDT Thiago Macieira wrote:
> This is the second time I've run into a compiler bug with that XCode's 
> ancient Clang.

Third strike.

Can we PLEASE REMOVE ancient Clang that is running on macOS 10.11 in the CI for 
5.12? Yes, I've read the email that it would leave us without 10.11 testing. If 
that's the result, then do it.

And BEFORE the final downmerge?

I can't integrate changes that compile on regular Clang (Linux and FreeBSD), 
modern Apple Clang, GCC, MSVC 2015 and MSVC 2017.

If that isn't done, I can't guarantee CBOR will work. I *do not* have time to 
investigate this problem or find work arounds, so this will impact my ability 
to provide bug fixes. 

Unless the consensus is to drop the work I was doing that ran into issues. See
https://codereview.qt-project.org/233481

--
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center



___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] Redeploying hosts with original kernel

[Tony Sarajärvi]

Hi

It seems like a culprit for our extremely odd behaving windows VMs might be the 
custom kernel we are applying (that tried to fix an NFS issue). After using the 
original kernel again, it seems at least for now we can run full speed on a 
host again. So I'll redeploy all hosts (some done already). This will slow down 
CI for a while, but as things are quite horrible currently, you won't probably 
notice anything but a better performance in a few hours.

This could bring back the NFS bug, but as fix for it was proposed in kernel -38 
and MAAS is giving us the kernel -46 now by default, it could either be fixed 
in the newest hosts we deploy, or the fix never worked to begin with. Anyway, 
I'll pick the least harmful of the 2 problems we have.

Good luck to us!

-T
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] Performing infra upgrade in the CI

[Tony Sarajärvi]

Hi

I'm performing a small infra upgrade. I'll be upgrading MAAS to the latest 
version. This shouldn't cause any CI downtime. If something goes badly wrong, 
I'll just revert my change and let's get back to business as usual.

Fingers crossed,
-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Maintenance in network aprox between the 25th and 27th of April

[Tony Sarajärvi]

Hi

We are right in the middle of redeploying all the hosts currently. So ATM 
capacity is down by 50%, but I won't bring it down from that before we get 
updated hardware online.

However, due to problems left and right (as usual), the deployment back takes 
time. With so much going on, I really don't even know why hosts don't come back 
online.

The new capacity we have has been set up, but the hardware has been wired 
differently than the previous batch we got. So we are using a different RAID 
controller, and you can guess twice how MAAS deployment works with it.

-Tony

From: Tony Sarajärvi
Sent: tiistai 24. huhtikuuta 2018 14.24
To: 'development@qt-project.org' <development@qt-project.org>
Subject: RE: Maintenance in network aprox between the 25th and 27th of April

Hi

This is a reminder of the service break we are going to be having the next few 
days.

A brief summary of what's going to change and what they affect:

Wednesday:

  *   New switches are being installed
 *   Side by side with the current ones, no interruptions as such
  *   New hosts are being connected to the CI
 *   This increases our capacity in the CI after they get deployed
  *   Our blade chassis hosting our infra is going to be moved
 *   This cuts of everyone from our infra for a minute

(This includes our VPN, and some external services like testresults.qt.io)


Thursday:

  *   All CI hosts are going down for upgrade
 *   Affects CI in such that our 20 hosts have to be taken out of service 
one by one
 *   Bringing them back takes ~2h per host
  *   MacMinis are taken down and re-routed
 *   This prevents macOS builds in our Jenkins farm

Friday:

  *   MacMinis are brought back online
  *   Troubleshooting


In case of questions, contact qt...@qt.io<mailto:qt...@qt.io>



With regards,
-Tony


From: Tony Sarajärvi
Sent: perjantai 6. huhtikuuta 2018 9.36
To: 'development@qt-project.org' 
<development@qt-project.org<mailto:development@qt-project.org>>
Subject: Maintenance in network aprox between the 25th and 27th of April

Hi

We are going to perform maintenance on our switches which will probably cause 
interruptions in all Qt services not located in AWS. New switches are estimated 
to arrive so that the work can begin approximately on the 25th. We'll inform 
you with more precise data as we get it.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Maintenance in network aprox between the 25th and 27th of April

[Tony Sarajärvi]

Hi

This is a reminder of the service break we are going to be having the next few 
days.

A brief summary of what's going to change and what they affect:

Wednesday:

  *   New switches are being installed
 *   Side by side with the current ones, no interruptions as such
  *   New hosts are being connected to the CI
 *   This increases our capacity in the CI after they get deployed
  *   Our blade chassis hosting our infra is going to be moved
 *   This cuts of everyone from our infra for a minute

(This includes our VPN, and some external services like testresults.qt.io)


Thursday:

  *   All CI hosts are going down for upgrade
 *   Affects CI in such that our 20 hosts have to be taken out of service 
one by one
 *   Bringing them back takes ~2h per host
  *   MacMinis are taken down and re-routed
 *   This prevents macOS builds in our Jenkins farm

Friday:

  *   MacMinis are brought back online
  *   Troubleshooting


In case of questions, contact qt...@qt.io



With regards,
-Tony


From: Tony Sarajärvi
Sent: perjantai 6. huhtikuuta 2018 9.36
To: 'development@qt-project.org' <development@qt-project.org>
Subject: Maintenance in network aprox between the 25th and 27th of April

Hi

We are going to perform maintenance on our switches which will probably cause 
interruptions in all Qt services not located in AWS. New switches are estimated 
to arrive so that the work can begin approximately on the 25th. We'll inform 
you with more precise data as we get it.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] Maintenance in network aprox between the 25th and 27th of April

[Tony Sarajärvi]

Hi

We are going to perform maintenance on our switches which will probably cause 
interruptions in all Qt services not located in AWS. New switches are estimated 
to arrive so that the work can begin approximately on the 25th. We'll inform 
you with more precise data as we get it.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] CI maintenance break on March 21 @ 8:00 - 9:00 EET

[Tony Sarajärvi]

All back up again!

If you see unexpected problems, due contact us again at 
qt...@qt.io<mailto:qt...@qt.io>

-Tony

From: Tony Sarajärvi
Sent: keskiviikko 21. maaliskuuta 2018 8.53
To: 'development@qt-project.org' <development@qt-project.org>
Cc: Qt Development <qt.developm...@qt.io>
Subject: RE: CI maintenance break on March 21 @ 8:00 - 9:00 EET

Hi

No problems so far, but so many firmware updates one after the other that this 
one hour window is not going to be enough. We are going to run it a few minutes 
past that.

-Tony

From: Tony Sarajärvi
Sent: keskiviikko 21. maaliskuuta 2018 8.00
To: development@qt-project.org<mailto:development@qt-project.org>
Cc: Qt Development <qt.developm...@qt.io<mailto:qt.developm...@qt.io>>
Subject: RE: CI maintenance break on March 21 @ 8:00 - 9:00 EET

Hi

Reminding you all that this maintenance break is about to happen now.

Good luck to us all.

-T

From: Tony Sarajärvi
Sent: perjantai 9. maaliskuuta 2018 14.08
To: development@qt-project.org<mailto:development@qt-project.org>
Cc: Qt Development <qt.developm...@qt.io<mailto:qt.developm...@qt.io>>
Subject: CI maintenance break on March 21 @ 8:00 - 9:00 EET

Hi

We'd like to have a small maintenance break. If the suggested time doesn't fit 
you at all, please contact us and we'll reschedule.

Purpose: We'd like to update the firmware on the hardware running Coin.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] CI maintenance break on March 21 @ 8:00 - 9:00 EET

[Tony Sarajärvi]

Hi

Reminding you all that this maintenance break is about to happen now.

Good luck to us all.

-T

From: Tony Sarajärvi
Sent: perjantai 9. maaliskuuta 2018 14.08
To: development@qt-project.org
Cc: Qt Development <qt.developm...@qt.io>
Subject: CI maintenance break on March 21 @ 8:00 - 9:00 EET

Hi

We'd like to have a small maintenance break. If the suggested time doesn't fit 
you at all, please contact us and we'll reschedule.

Purpose: We'd like to update the firmware on the hardware running Coin.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] Problems in provisioning

[Tony Sarajärvi]

Hi

You probably have seen that we have problems in provisioning. grub-update 
command isn't found and hdiutil can't unmount while installing Java on 
macOS'es. I'm suspecting the culprit for these problems is an update in Coin 
where it began enforcing errors in the scripts. Yes, we had the problems in the 
scripts before, but we just ignored the errors. So, now as we enforce, we hit a 
stand still until they are fixed (or we revert the Coin change).

Sadly it's a Saturday and people working on this is close to 0.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] The CI is down atm

[Tony Sarajärvi]

(top posting, thank you outlook)
The NFS is the backbone of how our CI works. I've been drawing a describing 
picture of how our infra is set up so that everyone can understand the bits and 
pieces of it, but I've seen so super busy lately that it hasn't progressed 
lately. Began working on it during the slower period at Christmas.

Basically we have 20 hosts that need access to the same data. For this we have 
a Dell Compellent storage system in the background and OpenNebula having the 
qcow2 images stored there. Now these qcow2 images are shared over NFS and we 
have 20 hosts reading that data. If they constantly read that data over the 
local network, the NFS server would not cope with it. So that's why all hosts 
have NFS caches so that the data they need is mostly available every time it's 
needed.

It would be a dramatic change in the infra setup if we distributed all these 
qcow2 images to the hosts beforehand, and we don't have the storage space on 
the hosts to store the images either ☹

-Tony
-Original Message-
From: Allan Jensen 
Sent: lauantai 17. maaliskuuta 2018 11.25
To: Tony Sarajärvi <tony.saraja...@qt.io>
Cc: development@qt-project.org
Subject: Re: The CI is down atm

On Samstag, 17. März 2018 07:42:30 CET Tony Sarajärvi wrote:
> Ok I brought it up again. Hosts seem to be dying left and right. The 
> current MAAS provided Ubuntu 16.04 LTS seem to have at least 3 
> different symptoms of how it crashes.
> 
>  
> 
> kernel BUG at /build/linux-Fk60NP/linux-4.10.0/include/linux/swapops.h:129!
> 
> kernel BUG at /build/linux-Fk60NP/linux-4.10.0/fs/fscache/operation.c:494!
> 
> or
> 
> kernel BUG at /build/linux-Fk60NP/linux-4.10.0/fs/fscache/operation.c:68!
> 
>  
> 
> + possible memory corruptions on multiple host causing crashes. These 
> + are
> being checked by memtest, but it hasn’t found anything wrong.
> 
Googling around a bit, these errors seems to be pretty common and happen across 
many different distros and for years. The general conclusion seems to be that 
fscache is just not very stable.

Can't you somehow avoid NFS instead?

'Allan


___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] The CI is down atm

[Tony Sarajärvi]

Ok I brought it up again. Hosts seem to be dying left and right. The current 
MAAS provided Ubuntu 16.04 LTS seem to have at least 3 different symptoms of 
how it crashes.

kernel BUG at /build/linux-Fk60NP/linux-4.10.0/include/linux/swapops.h:129!
kernel BUG at /build/linux-Fk60NP/linux-4.10.0/fs/fscache/operation.c:494!
or
kernel BUG at /build/linux-Fk60NP/linux-4.10.0/fs/fscache/operation.c:68!

+ possible memory corruptions on multiple host causing crashes. These are being 
checked by memtest, but it hasn’t found anything wrong.

So, coin is building again, but more crashes are due. I should swap to 17.04 or 
17.10 even though they have their own share of problems. They _might_ have 
gotten fixed during this period when we went back to 16.04. And as 16.04 went 
from bad to worse recently, the 17.xx are surely more stable right now.

Fingers crossed!

-Tony

Oh, and capacity is heavily reduced before the crashed servers are deployed 
back. And a long queue is in already naturally. ☹

We aren’t monitoring 24/7, so please inform at qt...@qt.io<mailto:qt...@qt.io> 
if you suspect the CI has crashed. Thank you!

From: Tony Sarajärvi
Sent: lauantai 17. maaliskuuta 2018 8.03
To: development@qt-project.org
Subject: The CI is down atm

Hi

The CI is down atm and I can’t even log in to the server right now. I’m trying 
to fix this somehow right away.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] The CI is down atm

[Tony Sarajärvi]

Hi

The CI is down atm and I can't even log in to the server right now. I'm trying 
to fix this somehow right away.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] About our CI update notifications

[Tony Sarajärvi]

Hi
We're trying to improve our communication toward you regarding our CI and 
what's going on with it. So our suggestion comes as follows:
· - In case of a scheduled update, we send an e-mail to the public 
developer mailing list the day before
· - In case of emergency update or pure restart, no notifications are 
needed
· - If Coin is unavailable for more than 30 min, we send an e-mail to 
the public developer mail list and inform about it followed by an e-mail when 
it's back up again.
· - Keep community updated in case of prolonged downtime
· - We mention a contact address where e-mails should be sent if 
abnormalities are seen after the update/restart, usually 
qt.ci@qt.io
How does this sound to you recipients of these e-mails? Enough or would this be 
excess spam to some degree?

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] CI maintenance break on March 21 @ 8:00 - 9:00 EET

[Tony Sarajärvi]

Hi

We'd like to have a small maintenance break. If the suggested time doesn't fit 
you at all, please contact us and we'll reschedule.

Purpose: We'd like to update the firmware on the hardware running Coin.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] CI going down for update

[Tony Sarajärvi]

Hi

Let me recap what we did last weekend.


  *   The Coin storage now has more storage space available. This should mean 
less rebooting the system for cleanup.
  *   We also noticed a VMware teak that increases the I/O from the storage. 
This could potentially make Coin more stable as the bandwidth doesn’t run out.
  *   We now have LVM installed, so further expansion of the storage is easier 
now.

Because we installed LVM, we had to redo the partitions. This required us to 
move data about a lot. We could have just deleted the logs and artifacts and 
let Coin rebuild them (we would have gotten the system up already on Friday 
evening), but by copying over all the data from a backup we enabled Coin to 
resume like nothing had happened (didn’t need to rebuild all the existing 
artifacts). It also enabled all the current logs to be visible for our company 
internal users that read the data straight from Coin instead of 
testresults.qt.io where they are published. With all the flaky autotests we 
had, we might have ended up trying to recover from lost artifacts until 
Saturday morning anyway. So the delay wasn’t bad if we think about it that way 

On an unrelated note. We’ve reverted some of the build hosts to Ubuntu 16.04 as 
the underlying platform. It looks like it’s just as stable if not more stable 
than the 17.xx ones. At least as of yet, not a single one has crashed! If this 
continues, this is _very_ good news for the CI. It will knock down a very 
annoying problem where the hosts that’s running all the virtual machines kept 
crashing and failing all the builds. Let’s hope this trend continues.

Happy building!
-Tony

From: Simon Hausmann
Sent: perjantai 16. helmikuuta 2018 17.10
To: Tony Sarajärvi <tony.saraja...@qt.io>; development@qt-project.org
Subject: Re: CI going down for update




Thanks for the update! Good luck and thanks for working on this :)





Simon


From: Tony Sarajärvi
Sent: Friday, February 16, 2018 3:44:49 PM
To: development@qt-project.org<mailto:development@qt-project.org>
Subject: RE: CI going down for update


Hi



We have to do things the hard way, meaning copying over a lot of stuff instead 
of just converting stuff. No unexpected things have poped up during the hard 
way however, but the copy phase takes a lot of time. ETA (or nothing really 
arrives now… so EToC): in a few hours.



-T



From: Tony Sarajärvi
Sent: perjantai 16. helmikuuta 2018 9.20
To: 'development@qt-project.org' 
<development@qt-project.org<mailto:development@qt-project.org>>
Subject: CI going down for update



Hi



We’ll take the CI down for update for a short while. The web page won’t work at 
all during that time, since we’ll close the whole server.



Thanks for your patience.



-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] CI going down for update

[Tony Sarajärvi]

Hi

We have to do things the hard way, meaning copying over a lot of stuff instead 
of just converting stuff. No unexpected things have poped up during the hard 
way however, but the copy phase takes a lot of time. ETA (or nothing really 
arrives now... so EToC): in a few hours.

-T

From: Tony Sarajärvi
Sent: perjantai 16. helmikuuta 2018 9.20
To: 'development@qt-project.org' <development@qt-project.org>
Subject: CI going down for update

Hi

We'll take the CI down for update for a short while. The web page won't work at 
all during that time, since we'll close the whole server.

Thanks for your patience.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] CI going down for update

[Tony Sarajärvi]

Hi

We'll take the CI down for update for a short while. The web page won't work at 
all during that time, since we'll close the whole server.

Thanks for your patience.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] CI infra broken

[Tony Sarajärvi]

Finally figured out the problem with the network test server. Sorry about the 
delay. QtBase's autotests have a greater chance of passing again (note: didn't 
say they will pass )

-Tony

-Original Message-
From: Tony Sarajärvi 
Sent: keskiviikko 31. tammikuuta 2018 13.42
To: 'Michal Klocek' <michal.klo...@qt.io>; development@qt-project.org
Subject: RE: [Development] CI infra broken

Hi

Yes you may use it, but it's currently in the state where stores are in the 
morning of Black Friday. We have everyone running in now and wondering why they 
don't get service. Talking about the commits you push in. Enormous peaks here. 
On top of that, I deleted all tier2 images everywhere. Basically meaning that 
the stores has nothing placed in shelves but everything has to be brought out 
from the back. I should have done that before I opened everything.

I also found that qtbase's autotest fail because the network test server has 
some problems after the restart. Looking into that. It often doesn't like 
reboots and fails when launching services. Shouldn't take long to fix.

-T

-Original Message-
From: Development 
[mailto:development-bounces+tony.sarajarvi=qt...@qt-project.org] On Behalf Of 
Michal Klocek
Sent: keskiviikko 31. tammikuuta 2018 12.44
To: development@qt-project.org
Subject: Re: [Development] CI infra broken

Hi

Please let us know if we can use ci already or not.

I also get:

"Timeout while creating source archives. Probably Coin issue, you can re-stage 
and ping Coin admin."

Br

Michal

On 01/31/2018 05:42 AM, Tony Sarajärvi wrote:
> Hi
> 
> Now that seems to be a bug in Coin. It doesn't even begin to retry it. The 
> log is still from last night the first time I launched it. Sigh... I'll try 
> to force it somehow.
> 
> -Tony
> 
> -Original Message-
> From: Allan Jensen
> Sent: keskiviikko 31. tammikuuta 2018 0.17
> To: Tony Sarajärvi <tony.saraja...@qt.io>
> Cc: development@qt-project.org
> Subject: Re: CI infra broken
> 
> On Dienstag, 30. Januar 2018 23:13:08 CET Tony Sarajärvi wrote:
>> Hi
>>  
>> We worked around the bug, but it took hours to move all data involved here.
>> The system is back up and running now. 
> 
> It appears to still be broken just like it was before everything went 
> offline, half the CIs can't provision and fail with messages like:
> 
> E: Failed to fetch 
> http://security.ubuntu.com/ubuntu/pool/main/p/python3.5/
> libpython3.5_3.5.2-2ubuntu0~16.04.4_amd64.deb  Could not connect to
> proxy.intra.qt.io:3128 (10.215.0.6). - connect (113: No route to host)
> 
> 
> 
> 'Allan
> 
> 
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development
> 
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] CI infra broken

[Tony Sarajärvi]

Hi

Yes you may use it, but it's currently in the state where stores are in the 
morning of Black Friday. We have everyone running in now and wondering why they 
don't get service. Talking about the commits you push in. Enormous peaks here. 
On top of that, I deleted all tier2 images everywhere. Basically meaning that 
the stores has nothing placed in shelves but everything has to be brought out 
from the back. I should have done that before I opened everything.

I also found that qtbase's autotest fail because the network test server has 
some problems after the restart. Looking into that. It often doesn't like 
reboots and fails when launching services. Shouldn't take long to fix.

-T

-Original Message-
From: Development 
[mailto:development-bounces+tony.sarajarvi=qt...@qt-project.org] On Behalf Of 
Michal Klocek
Sent: keskiviikko 31. tammikuuta 2018 12.44
To: development@qt-project.org
Subject: Re: [Development] CI infra broken

Hi

Please let us know if we can use ci already or not.

I also get:

"Timeout while creating source archives. Probably Coin issue, you can re-stage 
and ping Coin admin."

Br

Michal

On 01/31/2018 05:42 AM, Tony Sarajärvi wrote:
> Hi
> 
> Now that seems to be a bug in Coin. It doesn't even begin to retry it. The 
> log is still from last night the first time I launched it. Sigh... I'll try 
> to force it somehow.
> 
> -Tony
> 
> -Original Message-
> From: Allan Jensen
> Sent: keskiviikko 31. tammikuuta 2018 0.17
> To: Tony Sarajärvi <tony.saraja...@qt.io>
> Cc: development@qt-project.org
> Subject: Re: CI infra broken
> 
> On Dienstag, 30. Januar 2018 23:13:08 CET Tony Sarajärvi wrote:
>> Hi
>>  
>> We worked around the bug, but it took hours to move all data involved here.
>> The system is back up and running now. 
> 
> It appears to still be broken just like it was before everything went 
> offline, half the CIs can't provision and fail with messages like:
> 
> E: Failed to fetch 
> http://security.ubuntu.com/ubuntu/pool/main/p/python3.5/
> libpython3.5_3.5.2-2ubuntu0~16.04.4_amd64.deb  Could not connect to
> proxy.intra.qt.io:3128 (10.215.0.6). - connect (113: No route to host)
> 
> 
> 
> 'Allan
> 
> 
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development
> 
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] CI infra broken

[Tony Sarajärvi]

Hi

We worked around the bug, but it took hours to move all data involved here. The 
system is back up and running now.

You might have noticed quite a few provisioning failures (details show a proxy 
problem) failing your builds right after I got things up and running. Two 
things here. The proxy was never launched after we migrated it. An oversight on 
our part. Secondly the provisioning scripts have a check whether or not a proxy 
is available. I had created a bug there. Sorry about that. I have a fix created 
now for that. Anyway, just restage the commits and things should roll again. I 
did that for a few of the tasks for you :)

The earlier e-mail about CI updates ongoing where I mentioned that the capacity 
is lower than usual for the time being. That’s still the situation. This day 
just caused delays there. We’re now at 50%, but we should be at 75% or so in 
the morning. Hopefully during tomorrow we’ll have all hosts available 
again…with the new version of openNebula running newer KVM hosts (hopefully 
more stable). A lot of hopes there 

With regards,
-Tony

From: Tony Sarajärvi
Sent: tiistai 30. tammikuuta 2018 14.33
To: 'development@qt-project.org' <development@qt-project.org>
Subject: RE: CI infra broken

Hi

Seems we ran out of luck again. The openNebula VM seems to be broken. We 
reached a bug in vSphere that’s still unresolved by VMware. We have to recreate 
the VM, and that will take some time. Estimate for getting the CI up is 
tomorrow.

WR,
-Tony

From: Tony Sarajärvi
Sent: tiistai 30. tammikuuta 2018 12.20
To: 'development@qt-project.org' 
<development@qt-project.org<mailto:development@qt-project.org>>
Subject: CI infra broken

Hi

We have problems with our infra currently. We’re working on it. For the time 
being, the CI is down.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] CI infra broken

[Tony Sarajärvi]

Hi

Seems we ran out of luck again. The openNebula VM seems to be broken. We 
reached a bug in vSphere that's still unresolved by VMware. We have to recreate 
the VM, and that will take some time. Estimate for getting the CI up is 
tomorrow.

WR,
-Tony

From: Tony Sarajärvi
Sent: tiistai 30. tammikuuta 2018 12.20
To: 'development@qt-project.org' <development@qt-project.org>
Subject: CI infra broken

Hi

We have problems with our infra currently. We're working on it. For the time 
being, the CI is down.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] CI infra broken

[Tony Sarajärvi]

Hi

We have problems with our infra currently. We're working on it. For the time 
being, the CI is down.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] CI updates ongoing

[Tony Sarajärvi]

Ah, sorry for the confusion. I'm not touching the distros we test on. That is 
the virtual machines, the configurations in qt5.git.

I'm updating the host OS's that run the virtual machines. The thing that 
previously used to be ESXi from VMware before we switched to using Ubuntu/KVM 
with openNebula.

The exception being the virtual machine running openNebula itself. That will 
also be updated.

But as said. The updates will occur now as ubuntu moved their distros to their 
archives. Our deployment scripts don't work right of the bat anymore. We could 
modify them, but as we want to upgrade anyway at some point, we might as well 
do it now.

-Tony
Sent from my BlackBerry - the most secure mobile device
From: dange...@gmail.com
Sent: January 27, 2018 01:13
To: tony.saraja...@qt.io
Cc: development@qt-project.org
Subject: Re: [Development] CI updates ongoing


On 26 January 2018 at 21:34, Tony Sarajärvi <tony.saraja...@qt.io> wrote:
> Hi
>
>
>
> Our CI hosts run on Ubuntu, but sadly not on the latest version. We have
> 16.04 and 16.10 releases in use which need to be updated quickly.
>
> As we transition over, we will move hosts gradually to a new version of the
> CI running on 17.10. So during the next few days the CI will be a bit slower
> with reduced capacity.

Do you mean there won't be any CI node testing Ubuntu LTS? (Ok, 18.04
is due in a few months, but 18.04.1 is anyhow the recommended one for
LTS users, so we're talking ~6 months from now).

Thanks,
--
Giuseppe D'Angelo
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] CI updates ongoing

[Tony Sarajärvi]

Hi

Our CI hosts run on Ubuntu, but sadly not on the latest version. We have 16.04 
and 16.10 releases in use which need to be updated quickly.
As we transition over, we will move hosts gradually to a new version of the CI 
running on 17.10. So during the next few days the CI will be a bit slower with 
reduced capacity.

With regards,
-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] CI down. OpenNebula VM has crashed.

[Tony Sarajärvi]

We're back online, but don't know the reason for the crash yet. We'll monitor.

-T

From: Tony Sarajärvi
Sent: perjantai 12. tammikuuta 2018 8.26
To: development@qt-project.org
Subject: CI down. OpenNebula VM has crashed.

Hi

We've had a breakdown of the hypervisor we are using in the CI. We began 
investigations.

Regards,
-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] CI down. OpenNebula VM has crashed.

[Tony Sarajärvi]

Hi

We've had a breakdown of the hypervisor we are using in the CI. We began 
investigations.

Regards,
-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] CI down for a moment

[Tony Sarajärvi]

Hi all!

We had situation where it seems like cachefilesd bugged and didn't do cache 
culling. Our hosts filled up and caches got messed up. After cleaning the 
corrupted images, we still had hosts having incorrect data in caches. So as a 
quick fix I'm redeploying all hosts now. It takes some time, but we get a clean 
slate to work on again.

This was a last measure, but now I have an estimate of when the CI is back 
again. Before this I could only guess how to fix things and how long it would 
have taken.

Hosts come back gradually, and the we'll be at 20% capacity in aprox 2-3 hours. 
Then 2-3 hours more to get another 20%.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Reporting errors for powershell scripts executed by Coin

[Tony Sarajärvi]

Hi

We ignore the results of PS1 scripts because A) We have bad scripts that try to 
remove folders that don't exists anymore (should be cleaned) and B) as your 
example said, if we enabled enforcing of them now, things would break apart 
(fix needed as well )

-T

-Original Message-
From: Development 
[mailto:development-bounces+tony.sarajarvi=qt...@qt-project.org] On Behalf Of 
Konstantin Tokarev
Sent: torstai 21. joulukuuta 2017 20.37
To: development@qt-project.org
Subject: [Development] Reporting errors for powershell scripts executed by Coin

Hi all,

AFAIU, some time ago Coin considered ps1 script to be failed its exit code was 
non-null (e.g, Exit(1) certainly aborted further provisioning process).

Now I see that Exit(1) is ignored (see [1]). What is the correct way now, throw 
exception?

[1] 
https://testresults.qt.io/coin/api/results/provisioning/qtci-windows-10-x86_64-10-f8e8db/provision_1513869138/log.txt.gz

Words "conan exited with code 1" are printed before script does Exit(1)



-- 
Regards,
Konstantin
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] Gerrit problems fixed (?)

[Tony Sarajärvi]

Hi again!

The gerrit problems we've seen (example below (pardon for top posting with 
Outlook)) are potentially fixed. The problem was found to be in the AWS VM. 
Following actions were made to remedy the situation:

  *   Port range increased to allow more connections
  *   Mount options adjusted optimizing file access
  *   I/O scheduler is now 'noop'

With these changes the symptoms are for now gone. If they start appearing 
again, just hit reply to this thread and let's see if we can reproduce and fix 
it.

Thank you,
-Tony


From: Tony Sarajärvi
Sent: keskiviikko 13. joulukuuta 2017 21.39
To: 'development@qt-project.org' <development@qt-project.org>
Subject: RE: Builds missing qmake?

Hi again.

Should have told you this as well. When you see this error message coming from 
the CI:
"Could not access requested repository (: RAN: /usr/bin/git 
--git-dir=/home/vmbuilder/ci-working-dir/git-repos/qt-project/qt/qtwebview 
--no-pager fetch --tags origin 
+refs/builds/qtci/5.9/1513182815:refs/builds/qtci/5.9/1513182815 STDOUT: 
packet_write_wait: Connection to 54.229.21.112 port 29418: Broken pipe fatal: 
Could not read from remote repository. Please make sure you have the correct 
access rights and the repository exists. STDERR:"

It means that Coin is unable to retrieve needed sources from codereview. Again, 
as of now we don't know the root cause for this. We can produce this even 
outside Coin, so it's a problem outside our CI. We are investigating where and 
why the connections are lost.

When this happens you don't have to contact us however. Just restage your 
commit and the next time around the sources might be retrieved.

I apologize for the inconvenience
-Tony

From: Tony Sarajärvi
Sent: keskiviikko 13. joulukuuta 2017 11.26
To: 'development@qt-project.org' 
<development@qt-project.org<mailto:development@qt-project.org>>
Subject: Builds missing qmake?

Hi

If you stumble upon a problem in your build where the error message says that 
it can't find qmake, contact us maintaining the CI immediately. It's not a 
random failure in such that it goes away by restaging a few times. It means 
that for some reason (still unclear) we have a pointer indicating where the 
binaries should be, but they aren't. So they are never even uncompressed. But 
we can't really fail the build on missing artifacts, since we have no way of 
knowing if those binaries are optional or not. The error above clearly says 
that qtbase binaries are now uncompressed, hence the missing qmake, but we 
don't want to hard code all the really dependent packages for every module. 
Instead, we're trying to figure out why those binaries are missing in the first 
place.

So, qmake missing error -> contact us. We are needed to fix it currently.

-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] CI is suffering from problems

[Tony Sarajärvi]

Hi

This problem was fixed for now. We have the CI running again since yesterday 
afternoon. The system needed a cold boot of the server to update all disk 
related info after our cleanup. Free space etc were reported incorrectly before 
that.

To avoid this, we should never ever let the disk get 100% full, and we have to 
be more careful with that. Monitoring didn’t help here as we ignored the 
warnings we received ☹

-T
From: Tony Sarajärvi
Sent: torstai 21. joulukuuta 2017 14.19
To: development@qt-project.org
Subject: CI is suffering from problems

Hi

You have probably already noticed this, but here’s an email notifying you that 
we’re on it. Our OpenNebula isn’t able to clone any new virtual machines, 
complaining that the disk is full even though it isn’t. As said…we continue 
investigating.

Regards,
-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] CI is suffering from problems

[Tony Sarajärvi]

Hi

You have probably already noticed this, but here's an email notifying you that 
we're on it. Our OpenNebula isn't able to clone any new virtual machines, 
complaining that the disk is full even though it isn't. As said...we continue 
investigating.

Regards,
-Tony
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development