Re: [Development] QProcess fork() failure and overcommit

2017-03-08 Thread Thiago Macieira 
Em quarta-feira, 8 de março de 2017, às 14:44:51 CET, Ulf Hermann escreveu:
> > That said, look at the numbers: it increased from ~100 µs to 16.9 ms in
> > the
> > worst case. You can't perceive that difference.
> 
> This limits the number of processes you can spawn per second to about 60
> (and it goes lower if you eat more RAM).

Why would Qt Creator want to spawn more than 60 processes per second? 

> I certainly can notice this
> when using the version of qbs compiled into QtCreator to compile a
> project. In order to keep an 8-core CPU loaded and send extra jobs to
> icecream you need a higher rate.
> 
> There is a workaround for that by now, which just uses a separate
> process to spawn the children, but it used to be a problem.

Oh, that explains.Separate process is the correct solution. I

've just tried gmake in a very simple project and it called vfork() 26 times 
in a total runtime of 0.5 seconds -- that is, 52 times per second. That's a 
total count in the same order than the overhead inside Qt Creator.

Note: gmake uses vfork(), whose stated goal is to avoid doing a page table 
duplication; QProcess and forkfd cannot do that.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QProcess fork() failure and overcommit

2017-03-08 Thread Ulf Hermann 

That said, look at the numbers: it increased from ~100 µs to 16.9 ms in the
worst case. You can't perceive that difference.


This limits the number of processes you can spawn per second to about 60 
(and it goes lower if you eat more RAM). I certainly can notice this 
when using the version of qbs compiled into QtCreator to compile a 
project. In order to keep an 8-core CPU loaded and send extra jobs to 
icecream you need a higher rate.


There is a workaround for that by now, which just uses a separate 
process to spawn the children, but it used to be a problem.


Ulf
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QProcess fork() failure and overcommit

2017-03-08 Thread Thiago Macieira 
Em quarta-feira, 8 de março de 2017, às 13:51:49 CET, Christian Kandeler 
escreveu:
> I use the default settings for overcommit (0/heuristic). My application
> starts QProcesses in a loop. If the application has not allocated
> additional memory, strace -T for clone() gives:
> 
> <0.79>
> <0.91>
> <0.000205>
> <0.90>
> <0.000119>
> <0.000112>
> <0.000231>
> <0.96>
> <0.000114>
> <0.000112>
> 
> Here's the same application using 1GB of additional memory:
> 
> <0.012753>
> <0.016715>
> <0.015152>
> <0.014449>
> <0.012960>
> <0.016914>
> <0.013560>
> <0.013337>
> <0.012911>
> <0.013019>
> 
> So, factors around 100?

Not unexpected, if you use 100x more memory. clone() has a fixed cost for 
creating the necessary book-keeping and it may have some non-O(1) dependent on 
the number of running processes running, but it's definitely O(n) on the memory 
-- the Linux kernel does not apply the book-keeping on the 2MB page table 
directory level, only on the actual 4 kB page table entries.

That said, look at the numbers: it increased from ~100 µs to 16.9 ms in the 
worst case. You can't perceive that difference.

> (Also, I get lots of ERESTARTNOINTR in the latter case).

Ignore that, it's not visible.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QProcess fork() failure and overcommit

2017-03-08 Thread Christian Kandeler 
On 03/07/2017 10:05 PM, Thiago Macieira wrote:
> Em terça-feira, 7 de março de 2017, às 17:53:41 CET, René J.V. Bertin 
> escreveu:
>> One tends to forget (I do at least) that spawning a little helper process
>> can be quite expensive, sometimes prohibitively so. Makes you wonder what
>> kind of cross-platform alternatives there are!
> 
> It shouldn't be.
> 
> fork() does need to copy the page tables and mark the pages as shared. It 
> also 
> means COW of certain pages. But the overhead shouldn't be that big.
> 
> If anyone is seeing this, can you run your application with strace to get the 
> timings? 
> 
>   strace -T -e clone

I use the default settings for overcommit (0/heuristic). My application
starts QProcesses in a loop. If the application has not allocated
additional memory, strace -T for clone() gives:

<0.79>
<0.91>
<0.000205>
<0.90>
<0.000119>
<0.000112>
<0.000231>
<0.96>
<0.000114>
<0.000112>

Here's the same application using 1GB of additional memory:

<0.012753>
<0.016715>
<0.015152>
<0.014449>
<0.012960>
<0.016914>
<0.013560>
<0.013337>
<0.012911>
<0.013019>

So, factors around 100?
(Also, I get lots of ERESTARTNOINTR in the latter case).


Christian
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QProcess fork() failure and overcommit

2017-03-07 Thread Thiago Macieira 
Em terça-feira, 7 de março de 2017, às 22:28:11 CET, Simon Hausmann escreveu:
> I understand that this is in the context of Linux without overcommit.

Which is exactly the situation in which forking should be fast.

If it wasn't overcommitting, then the kernel would try and reserve the pages, 
possibly by freeing least-recently-used pages in other processes.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QProcess fork() failure and overcommit

2017-03-07 Thread Simon Hausmann 
I understand that this is in the context of Linux without overcommit.

Simon

> On 7 Mar 2017, at 22:05, Thiago Macieira  wrote:
> 
> Em terça-feira, 7 de março de 2017, às 17:53:41 CET, René J.V. Bertin 
> escreveu:
>> One tends to forget (I do at least) that spawning a little helper process
>> can be quite expensive, sometimes prohibitively so. Makes you wonder what
>> kind of cross-platform alternatives there are!
> 
> It shouldn't be.
> 
> fork() does need to copy the page tables and mark the pages as shared. It 
> also 
> means COW of certain pages. But the overhead shouldn't be that big.
> 
> If anyone is seeing this, can you run your application with strace to get the 
> timings? 
> 
>strace -T -e clone
> 
> Add -f if the forking is happening outside the main thread.
> 
> -- 
> Thiago Macieira - thiago.macieira (AT) intel.com
>  Software Architect - Intel Open Source Technology Center
> 
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QProcess fork() failure and overcommit

2017-03-07 Thread Thiago Macieira 
Em terça-feira, 7 de março de 2017, às 17:53:41 CET, René J.V. Bertin 
escreveu:
> One tends to forget (I do at least) that spawning a little helper process
> can be quite expensive, sometimes prohibitively so. Makes you wonder what
> kind of cross-platform alternatives there are!

It shouldn't be.

fork() does need to copy the page tables and mark the pages as shared. It also 
means COW of certain pages. But the overhead shouldn't be that big.

If anyone is seeing this, can you run your application with strace to get the 
timings? 

strace -T -e clone

Add -f if the forking is happening outside the main thread.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QProcess fork() failure and overcommit

2017-03-07 Thread Thiago Macieira 
Em terça-feira, 7 de março de 2017, às 14:32:13 CET, Christian Kandeler 
escreveu:
> This kind of stuff seems to happen when the parent process has allocated
> a lot of memory. I haven't debugged into it, but one idea might be that
> the page tables themselves get to a non-trivial size and thus copying
> them causes allocation problems.
> Note also that starting a QProcess becomes enormously slow in such
> applications; for instance, on my machine here, I can only start about
> 20 processes per second from an application that has 1 GB of memory
> allocated.

What OS is that? René was talking about macOS.

Also, what's 1 GB for you? Virtual size or resident set size?

The page table overhead in the OS shouldn't be more than a couple of 
megabytes.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QProcess fork() failure and overcommit

2017-03-07 Thread René J . V . Bertin 
On Tuesday March 07 2017 17:29:54 Kevin Funk wrote:

> It's pulled in by the Grantlee library. KF5::TextEditor depends on 
> Qt5::Script 
> as well.

There you have it, thanks Kevin.

One tends to forget (I do at least) that spawning a little helper process can 
be quite expensive, sometimes prohibitively so. Makes you wonder what kind of 
cross-platform alternatives there are!

R
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QProcess fork() failure and overcommit

2017-03-07 Thread Kevin Funk 
On Tuesday, 7 March 2017 16:42:31 CET René J.V. Bertin wrote:
> On Tuesday March 7 2017 14:20:18 Simon Hausmann wrote:
> Hi
> 
> > But for example if KDevelop ends up using QtScript and you're on a 64-bit
> > system, then we end up allocating 2GB of address space, which the kernel
> > has to copy (in terms of page tables) when forking. It could be that
> > you're short of that.
> That could be, but if my understanding of overcommit_memory=2 is correct I
> should still have enough margin for even that. Unless there's an additional
> per-process limit that comes into play?
> >We should probably mark these address spaces with MADV_DONTFORK.
> 
> I suppose I could test such a change, if that helps.
> 
> >But can you check first if KDevelop ends up using QtScript by chance?
> 
> The CMake project manager appears to use QScript for evaluating math
> expressions but even a session that doesn't load the corresponding plugin
> ends up loading the QtScript library according to its /proc/.../maps file.
> If Kevin picks up on this thread he may be able to say more about the paths
> through which KDevelop ends up using QtScript.

Here it is:

% lddtree .../libKDevPlatformShell.so
  (snip)
libKDevPlatformLanguage.so.10 => .../libKDevPlatformLanguage.so.10
libGrantlee_Templates.so.5 => .../libGrantlee_Templates.so.5
libQt5Script.so.5 => .../libQt5Script.so.5

It's pulled in by the Grantlee library. KF5::TextEditor depends on Qt5::Script 
as well.

Regards,
Kevin

> %> fgrep -i 5script /proc/24536/maps
> 7fd00cc9e000-7fd00cef5000 r-xp  00:27 37325 
> /opt/local/libexec/qt5/lib/libQt5Script.so.5.8.0 7fd00cef5000-7fd00d0f5000
> ---p 00257000 00:27 37325 
> /opt/local/libexec/qt5/lib/libQt5Script.so.5.8.0 7fd00d0f5000-7fd00d107000
> r--p 00257000 00:27 37325 
> /opt/local/libexec/qt5/lib/libQt5Script.so.5.8.0 7fd00d107000-7fd00d109000
> rw-p 00269000 00:27 37325 
> /opt/local/libexec/qt5/lib/libQt5Script.so.5.8.0
> 
> R.
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development


-- 
Kevin Funk | kf...@kde.org | http://kfunk.org

signature.asc
Description: This is a digitally signed message part.
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QProcess fork() failure and overcommit

2017-03-07 Thread René J . V . Bertin 
On Tuesday March 7 2017 14:20:18 Simon Hausmann wrote:
Hi

> But for example if KDevelop ends up using QtScript and you're on a 64-bit 
> system, then we end up allocating 2GB of address space, which the kernel has 
> to copy (in terms of page tables) when forking. It could be that you're short 
> of that.

That could be, but if my understanding of overcommit_memory=2 is correct I 
should still have enough margin for even that. Unless there's an additional 
per-process limit that comes into play?

>We should probably mark these address spaces with MADV_DONTFORK.

I suppose I could test such a change, if that helps.

>But can you check first if KDevelop ends up using QtScript by chance?

The CMake project manager appears to use QScript for evaluating math 
expressions but even a session that doesn't load the corresponding plugin ends 
up loading the QtScript library according to its /proc/.../maps file. If Kevin 
picks up on this thread he may be able to say more about the paths through 
which KDevelop ends up using QtScript.

%> fgrep -i 5script /proc/24536/maps
7fd00cc9e000-7fd00cef5000 r-xp  00:27 37325  
/opt/local/libexec/qt5/lib/libQt5Script.so.5.8.0
7fd00cef5000-7fd00d0f5000 ---p 00257000 00:27 37325  
/opt/local/libexec/qt5/lib/libQt5Script.so.5.8.0
7fd00d0f5000-7fd00d107000 r--p 00257000 00:27 37325  
/opt/local/libexec/qt5/lib/libQt5Script.so.5.8.0
7fd00d107000-7fd00d109000 rw-p 00269000 00:27 37325  
/opt/local/libexec/qt5/lib/libQt5Script.so.5.8.0

R.
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QProcess fork() failure and overcommit

2017-03-07 Thread Simon Hausmann 
Hi,


It would be interesting to see /proc//maps of the KDevelop process when 
fork fails. I would imagine - like Christian - that there is a shortage of 
memory mappings. It's odd that this is an issue, as there shouldn't be big 
memory mappings _usually_. But for example if KDevelop ends up using QtScript 
and you're on a 64-bit system, then we end up allocating 2GB of address space, 
which the kernel has to copy (in terms of page tables) when forking. It could 
be that you're short of that.


We should probably mark these address spaces with MADV_DONTFORK.


But can you check first if KDevelop ends up using QtScript by chance?


Simon


From: Development <development-bounces+simon.hausmann=qt...@qt-project.org> on 
behalf of René J.V. Bertin <rjvber...@gmail.com>
Sent: Tuesday, March 7, 2017 2:04:58 PM
To: development@qt-project.org
Subject: [Development] QProcess fork() failure and overcommit

Hi,

I have a bit of an intriguing issue I hope someone here could help me 
understand. If not, sorry for the noise.

I'm seeing occasional QProcess failures where QProcess:waitForStarted() fails 
and gives rise to errors like

kdevplatform.vcs: "DVCSJob::start: git ls-files -- kclock.cpp failed to start: 
Resource error (fork failure): Cannot allocate memory"

Those come and go in episodes; restarting the affected application always 
helped too (for a while).

Last time this happened htop reported I had 6955Mb used of 7899Mb, and not even 
10% swap used (1151Mb of 16Gb) but despite that the issue went away when I 
turned overcommit back on (I usually run with overcommit_memory=2 and 
overcommit_ratio=80).

This only happened to me with KDevelop5 until now, i.e. a Qt5/KF5 based 
application, running on a KDE4 (= Qt4-based) desktop.
If memory contention were really the cause here I would expect to see traces of 
similar failures elsewhere too because KDevelop isn't exactly the only KDE 
application that makes generous use of QProcess.

Is there anything in QProces (Qt5) vs. the Qt4 version that could explain 
fork() failing, or else can it be the way QProcess is being used which causes 
this in KDevelop but not other applications?

FWIW, not only do I see this in KDevelop exclusively, I've also seen it only 
when executing git commands through QProcess.

I've been seeing this for several months now, meaning with Qt 5.6.2, 5.7.1 and 
now 5.8.0 and kernels 4.7, 4.9 and possibly 4.5.7 (all with Con Kolivas's 
patches).

Thanks,
René
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QProcess fork() failure and overcommit

2017-03-07 Thread Christian Kandeler 
On 03/07/2017 02:54 PM, René J. V. Bertin wrote:
>> This kind of stuff seems to happen when the parent process has allocated
>> a lot of memory. I haven't debugged into it, but one idea might be that
> 
> What is a lot here? Typical usage for one of the KDevelop sessions that tends 
> to 
> be affected is around 70Mb total according to KSysGuard. Hardly a value that 
> shocks me...

More than that. Let's say a Gig, as in my example.

>> Note also that starting a QProcess becomes enormously slow in such
>> applications; for instance, on my machine here, I can only start about
>> 20 processes per second from an application that has 1 GB of memory
>> allocated.
> 
> As opposed to how many from a leaner application?

Hundreds.


Christian

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QProcess fork() failure and overcommit

2017-03-07 Thread René J . V . Bertin 
Christian Kandeler wrote:


> This kind of stuff seems to happen when the parent process has allocated
> a lot of memory. I haven't debugged into it, but one idea might be that

What is a lot here? Typical usage for one of the KDevelop sessions that tends 
to 
be affected is around 70Mb total according to KSysGuard. Hardly a value that 
shocks me...

> the page tables themselves get to a non-trivial size and thus copying
> them causes allocation problems.

Something application-specific thus? That would fit with the observation that 
not 
all KDevelop sessions are always affected at the same time.

> Note also that starting a QProcess becomes enormously slow in such
> applications; for instance, on my machine here, I can only start about
> 20 processes per second from an application that has 1 GB of memory
> allocated.

As opposed to how many from a leaner application?

R.

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] QProcess fork() failure and overcommit

2017-03-07 Thread Christian Kandeler 
On 03/07/2017 02:04 PM, René J.V. Bertin wrote:
> I have a bit of an intriguing issue I hope someone here could help me 
> understand. If not, sorry for the noise.
> 
> I'm seeing occasional QProcess failures where QProcess:waitForStarted() fails 
> and gives rise to errors like
> 
> kdevplatform.vcs: "DVCSJob::start: git ls-files -- kclock.cpp failed to 
> start: Resource error (fork failure): Cannot allocate memory"
> 
> Those come and go in episodes; restarting the affected application always 
> helped too (for a while).
> 
> Last time this happened htop reported I had 6955Mb used of 7899Mb, and not 
> even 10% swap used (1151Mb of 16Gb) but despite that the issue went away when 
> I turned overcommit back on (I usually run with overcommit_memory=2 and 
> overcommit_ratio=80).
> 
> This only happened to me with KDevelop5 until now, i.e. a Qt5/KF5 based 
> application, running on a KDE4 (= Qt4-based) desktop.
> If memory contention were really the cause here I would expect to see traces 
> of similar failures elsewhere too because KDevelop isn't exactly the only KDE 
> application that makes generous use of QProcess.
> 
> Is there anything in QProces (Qt5) vs. the Qt4 version that could explain 
> fork() failing, or else can it be the way QProcess is being used which causes 
> this in KDevelop but not other applications?

This kind of stuff seems to happen when the parent process has allocated
a lot of memory. I haven't debugged into it, but one idea might be that
the page tables themselves get to a non-trivial size and thus copying
them causes allocation problems.
Note also that starting a QProcess becomes enormously slow in such
applications; for instance, on my machine here, I can only start about
20 processes per second from an application that has 1 GB of memory
allocated.


Christian
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] QProcess fork() failure and overcommit

2017-03-07 Thread René J . V . Bertin 
Hi,

I have a bit of an intriguing issue I hope someone here could help me 
understand. If not, sorry for the noise.

I'm seeing occasional QProcess failures where QProcess:waitForStarted() fails 
and gives rise to errors like

kdevplatform.vcs: "DVCSJob::start: git ls-files -- kclock.cpp failed to start: 
Resource error (fork failure): Cannot allocate memory"

Those come and go in episodes; restarting the affected application always 
helped too (for a while).

Last time this happened htop reported I had 6955Mb used of 7899Mb, and not even 
10% swap used (1151Mb of 16Gb) but despite that the issue went away when I 
turned overcommit back on (I usually run with overcommit_memory=2 and 
overcommit_ratio=80).

This only happened to me with KDevelop5 until now, i.e. a Qt5/KF5 based 
application, running on a KDE4 (= Qt4-based) desktop.
If memory contention were really the cause here I would expect to see traces of 
similar failures elsewhere too because KDevelop isn't exactly the only KDE 
application that makes generous use of QProcess.

Is there anything in QProces (Qt5) vs. the Qt4 version that could explain 
fork() failing, or else can it be the way QProcess is being used which causes 
this in KDevelop but not other applications?

FWIW, not only do I see this in KDevelop exclusively, I've also seen it only 
when executing git commands through QProcess.

I've been seeing this for several months now, meaning with Qt 5.6.2, 5.7.1 and 
now 5.8.0 and kernels 4.7, 4.9 and possibly 4.5.7 (all with Con Kolivas's 
patches).

Thanks,
René
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development