Re: [Development] Evolving Qt's multithreading API
On Monday 04 March 2013 23:56:35 Thiago Macieira wrote: On terça-feira, 5 de março de 2013 08.54.35, André Somers wrote: Op 5-3-2013 8:41, Thiago Macieira schreef: On terça-feira, 5 de março de 2013 08.38.17, André Somers wrote: I'd appreciate a reply on the point that when using a QFutureWatcher, you don't know if the future is already done or not at the moment you connect. It's not done, by construction. So, if I understand you correctly, the following is completely safe right? QFuturevoid future = QtConcurrent::run(someFunction, 42, blah); //... perhaps some code, but not returning to the event loop QFutureWatcher* watcher = new QFutureWatcher(this); watcher.setFuture(future); connect(watcher, SIGNAL(finished(), this, SLOT(resultOfSomeFunctionReady())); Yes. No. If you do that, there is even a qWarning telling you there will be a race: http://code.woboq.org/qt5/qtbase/src/corelib/thread/qfuturewatcher.cpp.html#_ZN18QFutureWatcherBase13connectNotifyERK11QMetaMethod You should setup the connection before the future. -- Olivier Woboq - Qt services and support - http://woboq.com - http://code.woboq.org ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On terça-feira, 5 de março de 2013 10.11.21, Olivier Goffart wrote: Yes. No. If you do that, there is even a qWarning telling you there will be a race: http://code.woboq.org/qt5/qtbase/src/corelib/thread/qfuturewatcher.cpp.html# _ZN18QFutureWatcherBase13connectNotifyERK11QMetaMethod You should setup the connection before the future. Fair enough. You can create the QFutureWatcher before the future. The code is even more clear that way. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On 2 March 2013 18:17, Andre Somers an...@familiesomers.nl wrote: Actually, how do you feel about adding an optional _then_ argument to the list of arguments of the functions in QtConcurrent and whatever is decided to replace the QtConcurrent::run feature? Such a _then_ argument could be a slot signature, a function pointer or a lambda function (basically, whatever you can use in the Qt 5 QObject::connect). That would make it quite a bit easier to work with, I think. It would eliminate the need to create (and delete) a QFutureWatcher for a lot of cases. The method you passed into the _then_ argument would be called when the future is ready. It would be very nice if the _then_ argument could optionally have an argument of the return type of QFuture::result. For this to work, QFuture would not need to be a QObject itself, and because you pass in the _then_ argument with the call itself, you don't have the race issues that you get if you need to connect in separate calls after you have already fired off the thread: no need for trampoline objects or the like. Do you mean like the last argument to QTimer::singleShot()? (this needs to be updated to support Qt 5 slots, by the way) Yes, that could work. To make it truly optional though, the main parallel function would have to be pre-bound by the programmer (e.g. using https://codereview.qt-project.org/#change,45294), since the argument list is variable-size. QFutureT runFunction(QFunctionT main, QFunctionS then = NullFunction); Would the extra binding step nullify the convenience of not requiring a signalling object? Or, what if the run-function returns a QFutureWatcher which already has a QFuture attached -- would that ease the burden from the programmer a bit? It's such a shame that template classes can't be meta-objects... otherwise we could emit the return value directly. That would truly be event-driven C++! Regards, Sze-Howe ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
Op 4-3-2013 16:17, Sze Howe Koh schreef: On 2 March 2013 18:17, Andre Somers an...@familiesomers.nl wrote: Actually, how do you feel about adding an optional _then_ argument to the list of arguments of the functions in QtConcurrent and whatever is decided to replace the QtConcurrent::run feature? Such a _then_ argument could be a slot signature, a function pointer or a lambda function (basically, whatever you can use in the Qt 5 QObject::connect). That would make it quite a bit easier to work with, I think. It would eliminate the need to create (and delete) a QFutureWatcher for a lot of cases. The method you passed into the _then_ argument would be called when the future is ready. It would be very nice if the _then_ argument could optionally have an argument of the return type of QFuture::result. For this to work, QFuture would not need to be a QObject itself, and because you pass in the _then_ argument with the call itself, you don't have the race issues that you get if you need to connect in separate calls after you have already fired off the thread: no need for trampoline objects or the like. Do you mean like the last argument to QTimer::singleShot()? (this needs to be updated to support Qt 5 slots, by the way) Yes, very much like that, but updated to also support the Qt5 like way of connecting. Yes, that could work. To make it truly optional though, the main parallel function would have to be pre-bound by the programmer (e.g. using https://codereview.qt-project.org/#change,45294), since the argument list is variable-size. QFutureT runFunction(QFunctionT main, QFunctionS then = NullFunction); Would the extra binding step nullify the convenience of not requiring a signalling object? It would not quite, but it would be a bit of a let-down. There still is the benefit of not having to worry about race conditions. However, if we modify the syntax a little bit, I think we can avoid the additional binding step: QFutureT runFunction(QFunctionT then, QFunctionT main); QFutureT runFunction(QFuntionvoid then, QFuntionT main); QFutureT runFunction(QFunctionT main); (Copying your use of QFunction, a quick look at the WIP you mentioned doesn't quite reveal to me how it is supposed to work, and I probably use it wrong. The then argument could be lambda function, a QObject* and a slot signature, a pointer to an object and a member function pointer, or perhaps even a plain function pointer. It would be nice if the argument for these could be either no argument at all, or of the type T so it can just receive the value from the method.) Sure, it would read easier to have the _then_ follow the actual method, if if that is not possible, then I would not mind having them be reversed. However - and I am not a template mage - I think it should be possible to keep the order as is? As long as the actual function doesn't take a final argument with the given signature, I don't think there would be confusion? Note that I'd also like to see the same possibity for the other QtConcurrent methods, and pershaps even the threadpool method as well. Or, what if the run-function returns a QFutureWatcher which already has a QFuture attached -- would that ease the burden from the programmer a bit? No, it would not! The returned QFutureWatcher then has unclear ownership. That sounds even worse than having to create one yourself. And: the future still can finish before you have a chance to make the connection between the QFutureWatcher::finished() signal and your own slot. It's such a shame that template classes can't be meta-objects... otherwise we could emit the return value directly. That would truly be event-driven C++! Yeah, that is a very unfortunate artifact of the way Qt implements signals and slots. However, it is what it is... André ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On segunda-feira, 4 de março de 2013 17.25.35, André Somers wrote:
QFutureT runFunction(QFunctionT then, QFunctionT main);
QFutureT runFunction(QFuntionvoid then, QFuntionT main);
QFutureT runFunction(QFunctionT main);
I think this is going too far. If you want to chain jobs, we need a more
complex job tracker. That's what ThreadWeaver is trying to do. But we need to
solve a simpler problem first.
And besides, you can always do:
QFutureWatcherT w = runFunction(...);
w.connect(QFutureWatcherT::finished, []() { other function goes here });
w.connect(QFutureWatcherT::finished, w, QObject::deleteLater);
--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center
signature.asc
Description: This is a digitally signed message part.
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
Op 4-3-2013 21:21, Thiago Macieira schreef:
On segunda-feira, 4 de março de 2013 17.25.35, André Somers wrote:
QFutureT runFunction(QFunctionT then, QFunctionT main);
QFutureT runFunction(QFuntionvoid then, QFuntionT main);
QFutureT runFunction(QFunctionT main);
I think this is going too far. If you want to chain jobs, we need a more
complex job tracker. That's what ThreadWeaver is trying to do. But we need to
solve a simpler problem first.
And besides, you can always do:
QFutureWatcherT w = runFunction(...);
w.connect(QFutureWatcherT::finished, []() { other function goes here });
w.connect(QFutureWatcherT::finished, w, QObject::deleteLater);
The point is not to chain jobs. Indeed, that is what threadweaver is
doing. I am sorry if I did not make the point clear. The point is not to
create complex chains (or really: networks) of jobs. ThreadWeaver
already supports that use case. Still, I suppose you could do something
like that. The lines I wrote however were not meant as a chain, just as
different versions of the same method with a then method taking a
parameter of the type of the function, a then method without a parameter
and a call without any then argument.
The point is to get notification when a job is done in the simples way
possible. Nothing more, nothing less, without having to check the status
of the future. The way I understand QtConcurrent, the work is started
immediately and may finish at any moment. So, by the time I create the
QFutureWatcher and added the connect, the work may already be done.
Right? That means that you have to check for that as well, or you risk
missing the finished signal. See the discussion on the trampoline object
elsewhere in this thread. You'd avoid that by passing what you want
happening directly to the invocation to QtConcurrent.
The resulting code using a then argument would be much simpler than what
you post above. I don't quite get that code at all, by the way. You seem
to be creating a QFutureWatcher on the stack, and still connecting to
its deleteLater method? Also, which method directly returns a
QFutureWatcher? The QtConcurrent I'm familiar with returns a QFutureT,
which you can assign to a QFutureWatcher.
I think the proposed API addition is actually quite natural. It is not
without precedent (QTimer::singleShot()), and would result in quite
simple, readable code. It also doesn't create big constraints on the
type of object returned from these functions: there is no need for them
to be QObjects themselves.
If you want more control or need more information, you'd still create a
QFutureWatcher of course, as it provides more information than a simple
done.
André
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
Yes, very much like that, but updated to also support the Qt5 like way of connecting. Yes, that could work. To make it truly optional though, the main parallel function would have to be pre-bound by the programmer (e.g. using https://codereview.qt-project.org/#change,45294), since the argument list is variable-size. QFutureT runFunction(QFunctionT main, QFunctionS then = NullFunction); Would the extra binding step nullify the convenience of not requiring a signalling object? It would not quite, but it would be a bit of a let-down. There still is the benefit of not having to worry about race conditions. However, if we modify the syntax a little bit, I think we can avoid the additional binding step: QFutureT runFunction(QFunctionT then, QFunctionT main); QFutureT runFunction(QFuntionvoid then, QFuntionT main); QFutureT runFunction(QFunctionT main); I'm gonna sound crazy but... An important part of the QtConcurrent code do that implicit biding, and if we were to duplicate those calls for different usages, it's a lot of boilerplate code. Also, as said above, it limit the api in the sens that all the require parameters must be put before the actual function and its parameters. Both issues make it harder to maintain and evolve the api. So, what about let the users make the binding part themselves ? The syntax could be like runFunction(QFunctionFoo::bind(myFunction, arg1, arg2)); assuming the biding part is done using a bunch of functions, it could be simplified to runFunction(qBind(myFunction, arg1, arg2)) instead of the actual QtConcurrent::run(myFunction, arg1, arg2) It's arguably less straight forward for the user, but on the other hand, as we will need extra parameters, putting the options/flags/callback/signals before the function and its parameter could be even more confusing. we could then have: runFunction(QFunctionT function, Options, callback) (Copying your use of QFunction, a quick look at the WIP you mentioned doesn't quite reveal to me how it is supposed to work, and I probably use it wrong. QFunction is somehow like std::function, but has support for c++98. It's actually something between std::function and std::bind ( but does not support placeholders ) The then argument could be lambda function, a QObject* and a slot signature, a pointer to an object and a member function pointer, or perhaps even a plain function pointer. It would be nice if the argument for these could be either no argument at all, or of the type T so it can just receive the value from the method.) Agree It's such a shame that template classes can't be meta-objects... otherwise we could emit the return value directly. That would truly be event-driven C++! Yeah, that is a very unfortunate artifact of the way Qt implements signals and slots. However, it is what it is... Would that even be possible ? It sure would be nice Corentin ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On segunda-feira, 4 de março de 2013 22.00.34, Andre Somers wrote: The point is to get notification when a job is done in the simples way possible. Nothing more, nothing less, without having to check the status of the future. I understand what you want, but I don't want that. QFutureWatcher serves that purpose already. I don't quite get that code at all, by the way. You seem to be creating a QFutureWatcher on the stack, and still connecting to its deleteLater method? It was pseudocode. It should have been: QFutureWatcherT *w = new QFutureWatcherT(); w-setFuture(QThread::start(...)); I think the proposed API addition is actually quite natural. It is not without precedent (QTimer::singleShot()), and would result in quite simple, readable code. I disagree on both accounts. It's not natural: there is no precedent for taking two runnables outside of QtConcurrent's own reducing functions (map- reduce and filter-reduce). In that case, it serves a very specific purpose, in a domain where the concept is understood. And besides, that's the very module we're trying to replace. There's also no good API that takes two slots -- the only example I can think of is the horrible QDBusConnection::callWithCallback. And that one is made simpler and easier by QDBusPendingReply and QDBusPendingCallWatcher, two classes inspired by QFuture and QFutureWatcher. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On segunda-feira, 4 de março de 2013 14.49.15, Thiago Macieira wrote:
I think the proposed API addition is actually quite natural. It is not
without precedent (QTimer::singleShot()), and would result in quite
simple, readable code.
I disagree on both accounts. It's not natural: there is no precedent for
taking two runnables outside of QtConcurrent's own reducing functions (map-
reduce and filter-reduce). In that case, it serves a very specific purpose,
in a domain where the concept is understood. And besides, that's the very
module we're trying to replace.
There's also no good API that takes two slots -- the only example I can
think of is the horrible QDBusConnection::callWithCallback. And that one
is made simpler and easier by QDBusPendingReply and
QDBusPendingCallWatcher, two classes inspired by QFuture and
QFutureWatcher.
Oh, and if you want simple, one more thing occurred to me:
runFunction([]() { main(); then(); });
If you need the return value:
runFunction([]() { then(main()); });
I will not accept C++11 isn't available for everyone as a reason to add a
more complicated API. The API should be designed for C++11 use and later made
to work with C++98, with as little modification as possible.
--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center
signature.asc
Description: This is a digitally signed message part.
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
Op 4-3-2013 23:56, Thiago Macieira schreef:
On segunda-feira, 4 de março de 2013 14.49.15, Thiago Macieira wrote:
I think the proposed API addition is actually quite natural. It is not
without precedent (QTimer::singleShot()), and would result in quite
simple, readable code.
I disagree on both accounts. It's not natural: there is no precedent for
taking two runnables outside of QtConcurrent's own reducing functions (map-
reduce and filter-reduce). In that case, it serves a very specific purpose,
in a domain where the concept is understood. And besides, that's the very
module we're trying to replace.
There's also no good API that takes two slots -- the only example I can
think of is the horrible QDBusConnection::callWithCallback. And that one
is made simpler and easier by QDBusPendingReply and
QDBusPendingCallWatcher, two classes inspired by QFuture and
QFutureWatcher.
Oh, and if you want simple, one more thing occurred to me:
runFunction([]() { main(); then(); });
If you need the return value:
runFunction([]() { then(main()); });
I will not accept C++11 isn't available for everyone as a reason to add a
more complicated API. The API should be designed for C++11 use and later made
to work with C++98, with as little modification as possible.
The above doesn't work at all. It will run the then() in the worker
thread. That is not what you want.
I'd appreciate a reply on the point that when using a QFutureWatcher,
you don't know if the future is already done or not at the moment you
connect.
André
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On Mar 4, 2013, at 11:56 PM, Thiago Macieira thiago.macie...@intel.com wrote:
Oh, and if you want simple, one more thing occurred to me:
runFunction([]() { main(); then(); });
If you need the return value:
runFunction([]() { then(main()); });
I will not accept C++11 isn't available for everyone as a reason to add a
more complicated API. The API should be designed for C++11 use and later made
to work with C++98, with as little modification as possible.
I think this is the direction we should go in: build the future API around
C++11 lambdas.
This reduces the need for QFuture and especially QFutureWatcher. Thiago pointed
out the waiting case above. Instead of connecting to
QFutureWatcher::finished(), create a signal and emit when the work is done:
MyObject::asyncWork()
{
runFunction([]() {
Result result = doWork();
emit done(result); // signal on MyObject
});
}
Morten
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On terça-feira, 5 de março de 2013 08.38.17, André Somers wrote: I'd appreciate a reply on the point that when using a QFutureWatcher, you don't know if the future is already done or not at the moment you connect. It's not done, by construction. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On Mar 5, 2013, at 8:38 AM, André Somers an...@familiesomers.nl wrote: I'd appreciate a reply on the point that when using a QFutureWatcher, you don't know if the future is already done or not at the moment you connect. The intended usage of QFutureWatcher is that you set it up with connections before starting the worker thread/task. Then QFutureWatcher::setFuture() will do the correct thing and emit finished() if the future is already done. Morten ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
Op 5-3-2013 8:41, Thiago Macieira schreef: On terça-feira, 5 de março de 2013 08.38.17, André Somers wrote: I'd appreciate a reply on the point that when using a QFutureWatcher, you don't know if the future is already done or not at the moment you connect. It's not done, by construction. So, if I understand you correctly, the following is completely safe right? QFuturevoid future = QtConcurrent::run(someFunction, 42, blah); //... perhaps some code, but not returning to the event loop QFutureWatcher* watcher = new QFutureWatcher(this); watcher.setFuture(future); connect(watcher, SIGNAL(finished(), this, SLOT(resultOfSomeFunctionReady())); So, when _does_ the work start for the future then? Because all the documentation for QtConcurrent suggests that the work is started immediately if there is a free thread in the pool. And if the work can indeed start immediately, how is is guaranteed that at the moment you create your watcher your future is not done yet? André ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On terça-feira, 5 de março de 2013 08.54.35, André Somers wrote: Op 5-3-2013 8:41, Thiago Macieira schreef: On terça-feira, 5 de março de 2013 08.38.17, André Somers wrote: I'd appreciate a reply on the point that when using a QFutureWatcher, you don't know if the future is already done or not at the moment you connect. It's not done, by construction. So, if I understand you correctly, the following is completely safe right? QFuturevoid future = QtConcurrent::run(someFunction, 42, blah); //... perhaps some code, but not returning to the event loop QFutureWatcher* watcher = new QFutureWatcher(this); watcher.setFuture(future); connect(watcher, SIGNAL(finished(), this, SLOT(resultOfSomeFunctionReady())); Yes. So, when _does_ the work start for the future then? Immediately. In fact, the future might be finished before run() returns. However, it's not notified. Because all the documentation for QtConcurrent suggests that the work is started immediately if there is a free thread in the pool. And if the work can indeed start immediately, how is is guaranteed that at the moment you create your watcher your future is not done yet? By construction. The code was designed so that if you do that, it will still emit finished(). I explained how to achieve this in another email in this thread. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On 27 February 2013 15:38, Laszlo Papp lp...@kde.org wrote: On Wed, Feb 27, 2013 at 1:13 AM, Joseph Crowell joseph.w.crow...@gmail.com wrote: I have threads that live for the full life of my application but telling them to stop on application shut down is painful. What a coincidence. I have just had the same scenario. :-) How did you guys solve this issue? Do you need different cleanup code for the cancelled halfway case and the finished naturally case? Sze-Howe ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On Sun, Mar 3, 2013 at 10:15 AM, Sze Howe Koh szehowe@gmail.com wrote: On 27 February 2013 15:38, Laszlo Papp lp...@kde.org wrote: On Wed, Feb 27, 2013 at 1:13 AM, Joseph Crowell joseph.w.crow...@gmail.com wrote: I have threads that live for the full life of my application but telling them to stop on application shut down is painful. What a coincidence. I have just had the same scenario. :-) How did you guys solve this issue? Do you need different cleanup code for the cancelled halfway case and the finished naturally case? That depends too much on your specific design to be able to be answered. If your slots can schedule more slots (for example), then they might receive a quit message in between two slots that depend on each other and the thread could be brought down halfway because the second slot is never entered... and maybe some relevant data corrupted (in that case, you would have to use flagging to cleanly shutdown, which isn't too difficult but turns your code into a spaghetti of paths). I wrote a clean threading example for my own reference a while ago. Took quite a bit of reading the documentation, but once you know what you're doing (a lot of which depends on proper design of your backend thread/object), it's not too hard to tell them to stop at application shutdown. Here's that example demonstrating clean shutdown: http://lists.qt-project.org/pipermail/interest/attachments/20121026/60482ba0/attachment.zip In that project I was also working on my own evolution of the threading API but eventually gave up after I deemed fighting qmake/moc to be a waste of time. Was pretty close to getting QThread to be able to target a Functor with some args etc like has been mentioned in this thread. That's what Qt's threading solution is really missing. d3fault ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
Op 22-2-2013 11:57, Sze Howe Koh schreef:
Some more common use case would be (pseudo-code)
auto watcher = new QFutureWatcher;
QObject::connect(watcher, SIGNAL(finished()), myObject, SLOT(doStuff()));
watcher-setFuture(QThrerad::run([]() { return computeStuff(); } ));
// who deletes the watcher?
If the caller creates the watcher, I think it's fine to let the caller
delete it too. It's an ordinary C++ practice.
Actually, how do you feel about adding an optional _then_ argument to
the list of arguments of the functions in QtConcurrent and whatever is
decided to replace the QtConcurrent::run feature? Such a _then_ argument
could be a slot signature, a function pointer or a lambda function
(basically, whatever you can use in the Qt 5 QObject::connect). That
would make it quite a bit easier to work with, I think. It would
eliminate the need to create (and delete) a QFutureWatcher for a lot of
cases. The method you passed into the _then_ argument would be called
when the future is ready. It would be very nice if the _then_ argument
could optionally have an argument of the return type of QFuture::result.
For this to work, QFuture would not need to be a QObject itself, and
because you pass in the _then_ argument with the call itself, you don't
have the race issues that you get if you need to connect in separate
calls after you have already fired off the thread: no need for
trampoline objects or the like.
André
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On Tuesday 26 February 2013 20:02:31 Иван Комиссаров wrote: My 2 cents. I would like to see something similar to Qt Creator's functions implemented in runextensions.h in QtConcurrent module itself - methods that allows to manipulate future using QFutureInterface (progress notification, partial results). Anyway, i wasn't able to find Qt Concurrent problems, can anyone point them? Is it really should be replaced with other solution? This is just my opinion: From an API point of view, I think Qt concurrent is fine. One problem is that you cannot select a thread pool for your operations, so you are limited to the default thread pool, which means you really should only use it for CPU intensive operation. There are still problem on the implementation: - In the templated code: I think the abstractions are on the wrong place. This result in a lot of code duplication for the different cases and make also the code difficult to follow. (and difficult to modify) - In the 'engine': Let's say there is room for improvements to limit the overhead. Using lockfree data structures where it make sens, or optimize better the code. (at some point one does a median computation that uses a lot of CPU) All of this is fixable. QtConcurrent was released in a good finished state for a first stable version, but it is was still young in a way. Then it could have deserved a bit more love over the years. But it stayed as it. I personally think QtConcurrent should not be replaced. Just improved. For example, as you mentioned, incorporating the changes from QtCreator, after proper API review. QtConcurrent::run() (which i feel is what most of this thread is about) could possibly be replaced by API in QThread or QThreadPool, to fix the problem that it can only be run in the main thread pool. -- Olivier Woboq - Qt services and support - http://woboq.com ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On Wednesday 27 February 2013 11:13:35 Joseph Crowell wrote:
I have threads that live for the full life of my application but telling
them to stop on application shut down is painful. For this reason I
would like to be able to cancel a thread and also have a virtual
function to handle what happens when tread cancel is requested.. i.e.
cleanly closing child objects and handling locked mutexes so I don't get
segfaults. This is possible with current QThread but could be much easier.
You could use the QThread's destructor for that.
MyThread::~MyThread() {
setShouldExit(true);
wait();
}
I don't know why we don't call {quit();wait();} in the default QThread
destructor instead of calling it undefined behaviour to destroy a running
thread. Possibly because it is already too late, and the derived class has
already been destroyed.
But maybe we still could do it now.
--
Olivier
Woboq - Qt services and support - http://woboq.com - http://code.woboq.org
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
I've been sitting silent on this, but I am quite in favor of having an easy to understand approach to using QThreads, which the proposal in this thread seems to be. From: Thiago Macieira thiago.macie...@intel.com To: development@qt-project.org Sent: Monday, February 25, 2013 11:06 AM Subject: Re: [Development] Evolving Qt's multithreading API If so, what's the cost of having two QObjects (trampoline object and returned object), and how does it compare to the cost of deferring the start until the next event loop iteration? In my mind, the costs of 3 different approaches to starting this new method are: - Trampoline object: 2 QObjects, delay of 0 loop iterations, 0 extra LOC for user - Queued auto-start: 1 QObject, delay of 1 loop iteration, 0 extra LOC for user - Manual start: 1 QObject, delay of 0 loop iterations, 1 extra LOC for user Which costs the least? How do the optimizations to the trampoline change things? Now you're mixing things. First we choose the API: does it start automatically or not? After we've done that, we choose how to implement it and that is an implementation detail. How about a parameter to the function(s) that specifies whether to auto-start or not; default could be either way. Personally, I can easily seem myself replacing my current QThread usages with this functionality; but I'd want to be able to receive both start/finished signals (for logging purposes) and be able to interact with the QThread object - so the Trampoline object would need to provide an interface by which to access the internal QThread, or be a pass thru for signals/slots of the QThread. Of course, I mostly deal with long living threads; but my point of using the new interface would be (i) the simplicity, and (ii) the clarity. $0,02 Ben ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On terça-feira, 26 de fevereiro de 2013 07.03.37, BRM wrote: Personally, I can easily seem myself replacing my current QThread usages with this functionality; but I'd want to be able to receive both start/finished signals (for logging purposes) and be able to interact with the QThread object - so the Trampoline object would need to provide an interface by which to access the internal QThread, or be a pass thru for signals/slots of the QThread. Why? Why do you need access to the QThread object? Please don't answer the started / finshed signals -- those will be provided. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
My 2 cents. I would like to see something similar to Qt Creator's functions implemented in runextensions.h in QtConcurrent module itself - methods that allows to manipulate future using QFutureInterface (progress notification, partial results). Anyway, i wasn't able to find Qt Concurrent problems, can anyone point them? Is it really should be replaced with other solution? Иван Комиссаров ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
From: Thiago Macieira thiago.macie...@intel.com To: development@qt-project.org Cc: Sent: Tuesday, February 26, 2013 10:46 AM Subject: Re: [Development] Evolving Qt's multithreading API On terça-feira, 26 de fevereiro de 2013 07.03.37, BRM wrote: Personally, I can easily seem myself replacing my current QThread usages with this functionality; but I'd want to be able to receive both start/finished signals (for logging purposes) and be able to interact with the QThread object - so the Trampoline object would need to provide an interface by which to access the internal QThread, or be a pass thru for signals/slots of the QThread. Why? Why do you need access to the QThread object? Please don't answer the started / finshed signals -- those will be provided. Querying the status of the thread, and waiting for termination - e.g. QThread::wait(); or connecting a signal to tell it to quit. Having a minimal API is great for creating it; but you may still need access to the other functions provided by the QThread API. Either direct access or a pass-thru API would suffice; but it's probably a lot easier to have: QThread* Trampoline::getThread() const; than to do all the signals/slots/etc in the Trampoline object. $0.02 Ben ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On 2/27/2013 2:12 AM, BRM wrote: From: Thiago Macieira thiago.macie...@intel.com To: development@qt-project.org Cc: Sent: Tuesday, February 26, 2013 10:46 AM Subject: Re: [Development] Evolving Qt's multithreading API On terça-feira, 26 de fevereiro de 2013 07.03.37, BRM wrote: Personally, I can easily seem myself replacing my current QThread usages with this functionality; but I'd want to be able to receive both start/finished signals (for logging purposes) and be able to interact with the QThread object - so the Trampoline object would need to provide an interface by which to access the internal QThread, or be a pass thru for signals/slots of the QThread. Why? Why do you need access to the QThread object? Please don't answer the started / finshed signals -- those will be provided. Querying the status of the thread, and waiting for termination - e.g. QThread::wait(); or connecting a signal to tell it to quit. Having a minimal API is great for creating it; but you may still need access to the other functions provided by the QThread API. Either direct access or a pass-thru API would suffice; but it's probably a lot easier to have: QThread* Trampoline::getThread() const; than to do all the signals/slots/etc in the Trampoline object. $0.02 Ben ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development I have threads that live for the full life of my application but telling them to stop on application shut down is painful. For this reason I would like to be able to cancel a thread and also have a virtual function to handle what happens when tread cancel is requested.. i.e. cleanly closing child objects and handling locked mutexes so I don't get segfaults. This is possible with current QThread but could be much easier. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
The question is what exactly do you mean with canceling. I was already in the same place needing a thread to stop in a controlled fashion, but issuing ie a real phtread_cancel is tricky as the kernel may not support thread canceling and stack unwinding in C++ combined with it leads to quite weird side effects ;-) Why not just using a stop and join mechanism? Worked for me quite well. Mike On Feb 27, 2013 2:13 AM, Joseph Crowell joseph.w.crow...@gmail.com wrote: On 2/27/2013 2:12 AM, BRM wrote: From: Thiago Macieira thiago.macie...@intel.com To: development@qt-project.org Cc: Sent: Tuesday, February 26, 2013 10:46 AM Subject: Re: [Development] Evolving Qt's multithreading API On terça-feira, 26 de fevereiro de 2013 07.03.37, BRM wrote: Personally, I can easily seem myself replacing my current QThread usages with this functionality; but I'd want to be able to receive both start/finished signals (for logging purposes) and be able to interact with the QThread object - so the Trampoline object would need to provide an interface by which to access the internal QThread, or be a pass thru for signals/slots of the QThread. Why? Why do you need access to the QThread object? Please don't answer the started / finshed signals -- those will be provided. Querying the status of the thread, and waiting for termination - e.g. QThread::wait(); or connecting a signal to tell it to quit. Having a minimal API is great for creating it; but you may still need access to the other functions provided by the QThread API. Either direct access or a pass-thru API would suffice; but it's probably a lot easier to have: QThread* Trampoline::getThread() const; than to do all the signals/slots/etc in the Trampoline object. $0.02 Ben ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development I have threads that live for the full life of my application but telling them to stop on application shut down is painful. For this reason I would like to be able to cancel a thread and also have a virtual function to handle what happens when tread cancel is requested.. i.e. cleanly closing child objects and handling locked mutexes so I don't get segfaults. This is possible with current QThread but could be much easier. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On segunda-feira, 25 de fevereiro de 2013 20.36.24, Sze Howe Koh wrote: Thank you for the comprehensive explanation. I know little about Qt's internal mechanisms, so I'm curious now. Could the guarantee also be provided by posting QThread::start() into the event loop during setup? QMetaObject::invokeMethod(thread, start, Qt::QueuedConnection) Yes, that guarantee could be done like that, but that also implies delaying the start. I'd rather start immediately, potentially allow it to finish immediately, but delay *just* the notification. If so, what's the cost of having two QObjects (trampoline object and returned object), and how does it compare to the cost of deferring the start until the next event loop iteration? It's comparing apples to oranges. One delays the start of the work, so the expense is measured in time, plus a bit of complexity of code to make it start if waitForFinished() is somehow called. The other expense is measured in memory. In my mind, the costs of 3 different approaches to starting this new method are: - Trampoline object: 2 QObjects, delay of 0 loop iterations, 0 extra LOC for user - Queued auto-start: 1 QObject, delay of 1 loop iteration, 0 extra LOC for user - Manual start: 1 QObject, delay of 0 loop iterations, 1 extra LOC for user Which costs the least? How do the optimizations to the trampoline change things? Now you're mixing things. First we choose the API: does it start automatically or not? After we've done that, we choose how to implement it and that is an implementation detail. QThread signals the status of the thread, but QFutureWatcher signals the status of the future, the task. The thread itself may not have finished. What I'm saying is that you should not return a QThread, but something else. And then you can connect QThread's finished() signal to that something else's finished(). As I described above, that alone will be enough to guarantee the right semantics. Then, should we put this new function outside the QThread class? The task-management interface feels like it abstracts away the underlying QThread, so it would be messy to mix the API. We could do that, but I think that putting it in QThread also makes sense because that's where people will go to look for it. If this method is restricted to a run-replacement -- a static method that takes a lambda -- then it should be in QThread and it should return a QThread*. It should not be more complex than that. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On 22 February 2013 21:58, André Somers an...@familiesomers.nl wrote: Op 22-2-2013 14:31, Corentin Jabot schreef: Think about QNetworkAccessManager : reply = nam-get(url); // start the request connect(reply, QNetworkReply::finished(), doSomething()); // you can connect later I think that work just fine because the connection is queued, But I'm not sure exactly how. It works for QNAM, because the request is guaranteed to be made async. The work is only started when control is returned to the eventloop. For this to work in the threading context, you'd have to make a similar guarantee. Otherwise, you might end up in the situation that you start the thread, and before the connection is made, the thread already finishes, resulting in the slot never being called. That would result in only starting the work in the other thread when either the result is requested, or control is returned to the eventloop so you're sure all connectes are made. Hmm... I haven't looked at the implementation of QNAM, but what if, during the construction of the new QThreads (specifically, subclasses that handle functions/QRunnable), the derived QThreads post their start() method to the event loop? Will that delay the starting until control returns to the event loop? Regards, Sze-Howe ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On 22 February 2013 21:31, Corentin Jabot corentin.ja...@gmail.com wrote: Here again, two different issues. 1/ can we use the c++11 functional features and variadic templates for the biding part. 2/ should we use the c++11 thread api somehow ( that looks like a huge - unnecessary ? - change ) I don't think there's a need to change Qt's underlying threading system (yet?). Pthreads and Win32 threads do the job fine. snip I was actually wondering if QtConcurrent couldn't be upgraded/recycled * adding QtConcurrent::runFunctionInNewThread(function, ...) * adding QtConcurrent::runFunctionInThreadPool(QThreadPool* p, function, ... ) * adding QtConcurrent::runFunctionInGlobalThreadPoool(function, ...); * deprecating QtConcurrent::run and make it an alias of QtConcurrent::runInGlobalThreadPoool We decided to halt development of QtConcurrent, and only keep it for compatibility purposes (see comments in https://codereview.qt-project.org/#change,39375) How do you propose we run QRunnable in a non-threadpool thread? This way, QThread keeps its current purpose of exclusively handling thread. (Or we could add another class or namespace, like QAsynchronous, reusing QtConcurrent would meant keeping c++03 compat and its suppose we will still return QFuture) Or a low-level QtGlobal function? qAsyncRun(func, args...) About returning QThread* : what about the function return value ? That should be accessible, easily. It's one of the reason I prefer QFuture over QThread* Yes, that's the weakness of QThread*. We have some mutually exclusive features at the moment; we'll have to decide which one we want more: - Easy signalling (QThread) - Ability to postpone start (QThread) - Easy return value retrieval (QFuture) Regards, Sze-Howe ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On 23 February 2013 00:11, Thiago Macieira thiago.macie...@intel.com wrote: The fact is that any QObject that is returned from those functions -- if any -- must belong to the calling thread. That implies the necessary guarantees in terms of signal emissions. For example, if the functions return a QObject pointer, a signal-signal connection from the actual target object's finished() signal to the returned object's finished() will apply the necessary queueing semantics. That also speaks against returning a QThread*. I haven't understood your point, sorry. Can you please clarify what necessary guarantees you were referring to, and how this speaks against returning a QThread*? I thought QThread and QFutureWatcher were designed to signal the status of the new thread while living in the original thread. I also couldn't find a need to link up 2 finished() signals -- QThread will emit finished() when QThread::run() returns, and QFutureWatcher emits finished() in response to a callout event, not another signal. Regards, Sze-Howe ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On sábado, 23 de fevereiro de 2013 20.26.11, Sze Howe Koh wrote: On 23 February 2013 00:11, Thiago Macieira thiago.macie...@intel.com wrote: The fact is that any QObject that is returned from those functions -- if any -- must belong to the calling thread. That implies the necessary guarantees in terms of signal emissions. For example, if the functions return a QObject pointer, a signal-signal connection from the actual target object's finished() signal to the returned object's finished() will apply the necessary queueing semantics. That also speaks against returning a QThread*. I haven't understood your point, sorry. Can you please clarify what necessary guarantees you were referring to, and how this speaks against returning a QThread*? Sure. This is an un-optimised approach. There are a few ways to make it more efficient, but that's for later. Suppose we have: Result *QThread::start(task functor or object) Internally, it will use an internal class derived from QThread that runs the task in its run() function. When the thread finishes, QThread finished(). Now, as you noticed, unless we do something, the thread could start and finish before the user has a chance to connect the finished() signal from the returned object. What I am suggesting is a trampoline object: in the setup phase, a this trampoline QObject is created and it contains just one signal: finished(). That signal is connected to the Result object's finished(). When the task finishes and run() is about to return, it emits that signal -- from the auxiliary thread. That means the actual thread might have emitted finished(), but the Result object didn't -- that can only happen at the next event loop iteration, due to the queue semantics. So you note that the returned object cannot be the QThread that we created. It must be something else. Interestingly, since it can't be the QThread, it means the trampoline object *can* be the QThread: we just need to connect QThread::finished() to QThreadTask::finished(). I thought QThread and QFutureWatcher were designed to signal the status of the new thread while living in the original thread. I also couldn't find a need to link up 2 finished() signals -- QThread will emit finished() when QThread::run() returns, and QFutureWatcher emits finished() in response to a callout event, not another signal. QThread signals the status of the thread, but QFutureWatcher signals the status of the future, the task. The thread itself may not have finished. What I'm saying is that you should not return a QThread, but something else. And then you can connect QThread's finished() signal to that something else's finished(). As I described above, that alone will be enough to guarantee the right semantics. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
I'm not sure what you mean. OpenMP, Pthreads and Boost threads are independent; there is no backend here. Anyway, as Thiago mentioned in another post, OpenMP is not supported on all the compilers that Qt supports, so we can't use it in Qt. (http://openmp.org/wp/openmp-compilers/) Regards, Sze-Howe On 22 February 2013 03:16, Ing. Reynier Pupo Gomez rgo...@uci.cu wrote: The only problem that I see in OpenMP is it backend, Pthread. I think actually Qt switch form PThread to boost threads. But on the other side the code injected for programing with OMP is very simple. It is possible to develop a secuential program and prepare for parallel use with non-intrusive directives depending on a compilation flag. Im just starting with this library, so I cant help so much. On Viernes, 22 de febrero de 2013 12:15:19 AM usted escribió: On 21 February 2013 02:31, Ing. Reynier Pupo Gómez rgo...@uci.cu wrote: What about using of OpenMP standard? It could be very usefull and well known by the C/C++ comunity. Thanks for the suggestion. I had a quick look, but it seems to be on the low-level side. I'm not sure if we want to use #pragmas for regular code... Have you had experience with it? Is it easy to use? Regards, Sze-Howe ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On 20 February 2013 22:45, Sze Howe Koh szehowe@gmail.com wrote: Hi all, Some time ago there was some talk about improving Qt's multithreading API. I'm summarizing them here to stop them from fading into obscurity, and to see if there's any interest in following them up. Here are the tasks mentioned: - Replace/Rewrite QtConcurrent [2] - Create/Find a good API to replace QtConcurrent::run() for one-shot tasks [1] - Find a third-party solution for high-level multithreading [2] - Find more uses for QFuture, outside of QtConcurrent [3] - Influence C++1y by creating a nice multithreading API [4] Some suggestions were raised: - Put a Qt-ish wrapper around TBB [1] - Integrate ThreadWeaver back into Qt? [2] Separately, someone was experimenting with ways to spawn a QObject in a secondary thread, without first constructing it in the current thread [5] Do you think any of these avenues are worth pursuing? I've had a quick look at TBB vs. ThreadWeaver. The latter specializes in task-oriented programming, while TBB is a more swiss-army-knife toolkit, which includes container-based operations similar to QtConcurrent. So, if we're to integrate 3rd-party option into Qt, TBB would be more worth it (although it'd involve more work too) Actually, I just realized that the open-source flavour of TBB is licensed under GPLv2 (http://threadingbuildingblocks.org/Licensing). Doesn't that mean that Qt TBB, if it were to become reality, can't be licensed under the LGPL? Regards, Sze-Howe [1] http://lists.qt-project.org/pipermail/development/2012-November/007901.html [2] http://lists.qt-project.org/pipermail/development/2012-November/007921.html [3] http://lists.qt-project.org/pipermail/development/2012-November/007944.html [4] http://lists.qt-project.org/pipermail/development/2012-November/007933.html [5] http://lists.qt-project.org/pipermail/interest/2013-January/005740.html ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
Op 22-2-2013 11:57, Sze Howe Koh schreef:
On Feb 22, 2013 12:33 AM, Olivier Goffart oliv...@woboq.com wrote:
Some more common use case would be (pseudo-code)
auto watcher = new QFutureWatcher;
QObject::connect(watcher, SIGNAL(finished()), myObject, SLOT(doStuff()));
watcher-setFuture(QThrerad::run([]() { return computeStuff(); } ));
// who deletes the watcher?
If the caller creates the watcher, I think it's fine to let the caller
delete it too. It's an ordinary C++ practice.
I don't really like the need to create a watcher in order to get a
signal at all, to be honest. Never did.
My own implementation of a task-based system that I recently did,
involved returning a QSharedPointerTask from the task manager. Task
derives from QObject, and has signals and slots that can be used
notification. It is similar to QThread, in that it lives in the thread
that requested the task. It is inspired on QNetworkAccessManager, that
immediately gives back an instance of a reply that you can either
directly connect to or ignore.
Using a shared pointer gave me these advantages:
* No need for explicit deletes on either side: the requester can hold on
to the pointer and use it directly, or use a generic signal from the
task manager object that also contains a shared pointer to the same
task. When nobody is interested in the task object any more, it is
automatically deleted.
* Easily connect to signals like finished() or error() (though it would
be nice if you could directly connect to a
QSharedPointerQObjectDerivedClass instead of having to use .data() );
no need for a separate watcher object
* Value semantics, just like QFuture: a shared pointer is quite cheap to
pass around
Just like QFuture, it is possible to wait for the task to finish when
needed.
I'd really like to see such a structure in Qt, as generic as possible.
Tasks are not only relevant in the context of threads, but also for
things like network operations and perhaps even I/O operations,
printing, etc.
André
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
2013/2/22 Sze Howe Koh szehowe@gmail.com:
Yes, that was my original plan. Someone complained that they couldn't
bind a function + arguments to QtConcurrent::run() first, then run it
at a later time. This approach gives them the opportunity to start it
whenever they want.
The problem is QtConcurrent does 2 separate things.
It binds function and run them asynchronously.
To me that's two different issues, so, if one want to bind a function
and run it later,
just use std::bind ( hence my QFunction patch for c++03 support )
On second thought, I'm not sure if this would be commonly needed. We
can make it start immediately instead, BUT this requires a guarantee
that the first thread can always connect the finished() signal, before
the new thread runs and finishes. Can that be guaranteed?
Think about QNetworkAccessManager :
reply = nam-get(url); // start the request
connect(reply, QNetworkReply::finished(), doSomething()); // you can
connect later
I think that work just fine because the connection is queued,
But I'm not sure exactly how.
If not,
we'll need:
QThread *thread = QThread::runFunction([]() { return
doSomethingComplicated(); });
connect(thread, QThread::finished, this, MyObject::doStuff);
thread-start(); // Manual start, after all connections have been made
OR
QThread *thread = QThread::runFunction([]() { return
doSomethingComplicated(); });
thread-wait(); // Wait for auto-started thread to end
doStuff();
The latter is like Corentin's approach using QFuture. Which is better?
(Personally I think the latter defeats the purpose of having a 2nd
thread)
Of course that was a silly example, you would connect the QFuture
finished signal
to a slot, or use QFutureSynchronizer to run run multiple treatments
concurrently.
Regarding deletion, we can follow the example of QAudioInput::start()
-- it returns a pointer to a QIODevice for reading, but retains
ownership of the device and auto-deletes it when stopped. Automatic
deletion is needed to prevent a memory leak in the nice simple pattern
you wrote above.
The implementation use the same sort of generator that QtConcurrent.
Thiago suggested making this feature only compatible C++11, which
would make it easier to maintain. I actually envisaged to send a mail
about that particular issue.
Can c++03 really be dropped for that particular feature ?
Also, I we were to make a c++11 only feature, what would be the
benefits over std::async ?
I would love to see an implementation using C++11. I think that
decision can't be made lightly though, and should be a separate
discussion -- it has implications for all other Qt modules.
What do you mean?
Do you mean an implementation of QThread that would use std::thread as an
internal rather than pthread/windows API? (qthread_cxx11.cpp)
Here again, two different issues.
1/ can we use the c++11 functional features and variadic templates
for the biding part.
2/ should we use the c++11 thread api somehow ( that looks like a huge
- unnecessary ? - change )
I think it would be good to have. But would just be another layer, and more
code to maintain.
I meant variadic templates and std::bind. QtConcurrent::run() uses a
code generator to produce 6 separate templates. In theory, we can
replace all that with 1 variadic template. No code generator, fewer
templates, less maintenance.
template typename TFunc, typename... TArgs
static QThread* setupSimpleThread(TFunc func, TArgs... args)
{
// Private class, derived from QThread
return new QFunctionThread(
std::bind(std::forwardTFunction(a_func),
std::forwardTArgs(a_args)...));
}
Exactly, there are real benefits to use c++11 only.
We do have many older compilers to support still. Is Qt ready to start
introducing features that require C++11? Are we in a position to drive
its adoption, as Thiago put it?
(http://www.macieira.org/blog/2011/09/cxx11-support-in-qt-5/)
QtConcurrent somehow does works so people with no c++11 support, could
still use that even if its not ideal.
I was actually wondering if QtConcurrent couldn't be upgraded/recycled
* adding QtConcurrent::runFunctionInNewThread(function, ...)
* adding QtConcurrent::runFunctionInThreadPool(QThreadPool* p,
function, ... )
* adding QtConcurrent::runFunctionInGlobalThreadPoool(function, ...);
* deprecating QtConcurrent::run and make it an alias of
QtConcurrent::runInGlobalThreadPoool
This way, QThread keeps its current purpose of exclusively handling thread.
(Or we could add another class or namespace, like QAsynchronous,
reusing QtConcurrent would meant keeping c++03 compat and its suppose
we will still return QFuture)
About returning QThread* : what about the function return value ? That
should be accessible, easily.
It's one of the reason I prefer QFuture over QThread*
Regards,
Corentin
Re: [Development] Evolving Qt's multithreading API
Op 22-2-2013 14:31, Corentin Jabot schreef:
On second thought, I'm not sure if this would be commonly needed. We
can make it start immediately instead, BUT this requires a guarantee
that the first thread can always connect the finished() signal, before
the new thread runs and finishes. Can that be guaranteed?
Think about QNetworkAccessManager :
reply = nam-get(url); // start the request
connect(reply, QNetworkReply::finished(), doSomething()); // you can
connect later
I think that work just fine because the connection is queued,
But I'm not sure exactly how.
It works for QNAM, because the request is guaranteed to be made async.
The work is only started when control is returned to the eventloop. For
this to work in the threading context, you'd have to make a similar
guarantee. Otherwise, you might end up in the situation that you start
the thread, and before the connection is made, the thread already
finishes, resulting in the slot never being called. That would result in
only starting the work in the other thread when either the result is
requested, or control is returned to the eventloop so you're sure all
connectes are made.
Alternatively, you'd have to start doing the work explicitly as written
here:
If not,
we'll need:
QThread *thread = QThread::runFunction([]() { return
doSomethingComplicated(); });
connect(thread, QThread::finished, this, MyObject::doStuff);
thread-start(); // Manual start, after all connections have been made
OR
QThread *thread = QThread::runFunction([]() { return
doSomethingComplicated(); });
thread-wait(); // Wait for auto-started thread to end
doStuff();
The latter is like Corentin's approach using QFuture. Which is better?
(Personally I think the latter defeats the purpose of having a 2nd
thread)
Of course that was a silly example, you would connect the QFuture
finished signal
to a slot, or use QFutureSynchronizer to run run multiple treatments
concurrently.
If only QFuture allowed you to connect... Unfortunately, it is not a
QObject.
André
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
2013/2/22 André Somers an...@familiesomers.nl If only QFuture allowed you to connect... Unfortunately, it is not a QObject Oh yeah, I almost forgot that bit. And somehow it looks like the core issue. I wonder why by the way: We could have something like QObject - QFutureBase (with all requiered signals/slots) - QFutureT or is there something I'm not seeing ? Of course now its too late, but we could introduce something new, like QFutureObject ? Corentin ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
However, std::function and std::bind were already in tr1, which AFAIK is already supported by all the compiler we support (Tier1 + Tier2) (MSVC 2008 and gcc 4.2 have it) For VC 2008 it is part of an add-on pack [1], but it is available. We could have something like QObject - QFutureBase (with all requiered signals/slots) - QFutureT or is there something I'm not seeing ? QObject is heavy - in terms of memory usage, construction/destruction cost etc. - if there are scenarios where you are creating thousands of such objects, this could be a problem. Regards, Rob. [1] http://www.microsoft.com/en-gb/download/details.aspx?id=6922 On 22 February 2013 14:28, Corentin Jabot corentin.ja...@gmail.com wrote: 2013/2/22 André Somers an...@familiesomers.nl If only QFuture allowed you to connect... Unfortunately, it is not a QObject Oh yeah, I almost forgot that bit. And somehow it looks like the core issue. I wonder why by the way: We could have something like QObject - QFutureBase (with all requiered signals/slots) - QFutureT or is there something I'm not seeing ? Of course now its too late, but we could introduce something new, like QFutureObject ? Corentin ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On sexta-feira, 22 de fevereiro de 2013 15.28.44, Corentin Jabot wrote: 2013/2/22 André Somers an...@familiesomers.nl If only QFuture allowed you to connect... Unfortunately, it is not a QObject Oh yeah, I almost forgot that bit. And somehow it looks like the core issue. I wonder why by the way: We could have something like QObject - QFutureBase (with all requiered signals/slots) - QFutureT or is there something I'm not seeing ? Of course now its too late, but we could introduce something new, like QFutureObject ? That's QFutureWatcher. The fact is that any QObject that is returned from those functions -- if any -- must belong to the calling thread. That implies the necessary guarantees in terms of signal emissions. For example, if the functions return a QObject pointer, a signal-signal connection from the actual target object's finished() signal to the returned object's finished() will apply the necessary queueing semantics. That also speaks against returning a QThread*. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On sexta-feira, 22 de fevereiro de 2013 15.21.50, Olivier Goffart wrote: Variadic template, we can clearly not rely on it. It only came in MSVC really recently (patch release in nov 2012) Yes, we can rely on it. That just means MSVC 2012 RTM doesn't get the feature. As Marc put it, C++98 costs more. My suggestion is: design the API for C++11, then after it's done, we look into how much more is needed to support C++98. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On sexta-feira, 22 de fevereiro de 2013 19.26.06, Sze Howe Koh wrote: Actually, I just realized that the open-source flavour of TBB is licensed under GPLv2 (http://threadingbuildingblocks.org/Licensing). Doesn't that mean that Qt TBB, if it were to become reality, can't be licensed under the LGPL? It's GPLv2+exceptions: The source code of Threading Building Blocks is distributed under version 2 of the GNU General Public License, with the so-called runtime exception, as follows (or see any header or implementation file): As a special exception, you may use this file as part of a free software library without restriction. Specifically, if other files instantiate templates or use macros or inline functions from this file, or you compile this file and link it with other files to produce an executable, this file does not by itself cause the resulting executable to be covered by the GNU General Public License. This exception does not however invalidate any other reasons why the executable file might be covered by the GNU General Public License. I believe it's the same exception as the one in GNU libstdc++, -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On 21 February 2013 02:16, Corentin Jabot corentin.ja...@gmail.com wrote:
Hi. I'm the one Olivier mentioned :p
I didn't have time to pursue further the work I started, but I intend
to, someday.
The plan, as suggested by thiago was to have a QThread::run(functor)
method acting exactly like QtConcurrent::run, but using a new QThead.
A similar QThreadPool::run function would call a functor in a thread
allocated in its pool.
I think the the plan is good; QtConcurrent::run() always felt
out-of-place with the filter-map-reduce API. We just need to be
mindful of how we integrate it into Qt -- the solution should cover
all 3 of functors, QRunnable, and parallel event loops.
I notice you also have QThread::run(QRunnable*). Let's take into
consideration the existing methods QThreadPool::start() and
QThreadPool::tryStart(). It's inconsistent to have:
QThread::run(QRunnable*)
QThreadPool::start(QRunnable*)
(Look for static polymorphism in
http://doc.qt.digia.com/qq/qq13-apis.html) I think we should either
make the new functions match the old names, or rename the old
functions to match the new names. (rename = create new names,
deprecate old names)
Those function return QFuture and do not require event loop so this
silly snippet work:
int main() {
auto future = QThread::run( []() { qDebug() Hello world ; } );
future.waitForFinished();
}
My goal was to make it as simple as I could.
Sounds good
Anyway, I don't think we should returns a QThread*. the underlying
implementation does not mater and shouldn't mater.
What would you do with a QThread* you can't do with a QFuture ?
Apologies, I didn't realize that you had a version that returns
QFuture; I only saw the one that returned void. (Line 115,
https://codereview.qt-project.org/#patch,sidebyside,45297,5,src/corelib/thread/qthread.h)
I think returning QThread* is better than void. I still can't decide
if QThread* or QFuture is better for these new methods, however.
Technically, returning a QThread does hide the implementation. The
programmer only sees the QThread, and doesn't know that you've
implemented a QRunnableThread underneath. The programmer already knows
about QThread (since s/he called QThread::run()), so getting a
QThread* won't be a big deal.
About QThread vs. QFuture... QThread:
- Lets the programmer postpone starting the thread
- Lets the programmer query and manipulate the thread using
event-driven programming (signals and slots)
On the other hand, QFuture makes it easy to retrieve the function's
return value.
I leaned towards returning QThread* because QtConcurrent::run()
couldn't use most of QFuture's features anyway, and QThread* can be
returned from all 3 static functions:
static QThread* QThread::setupSimpleThread(QRunnable *runnable);
static QThread* QThread::setupSimpleThread(Function func, Args arg, ...);
static QThread* QThread::setupEventLoop(QObject* worker);
...but I'm not sure if losing the functor's return value is worth it.
The Thread is started right-away, you can't really interrupt the
function until its done, etc but you could delete the thread while
it's running, so yeah, I'd rather not a return a pointer to the
thread.
I don't think we have to worry about people deleting the QThread while
it's running. If we did, we'd have similar problems the
QAudioInput::start() (returns a QIODevice*) and
QNetworkAccessManager::get() (returns a QNetworkReply*).
I find the signature QThread::run(function), quite straightforward,
there is no confusion about what it does.
I don't think the fact there is also the static void run() method is
confusing, but maybe that's just me.
I agree that, by itself, run() is a good name for this feature. The
issue, however, is that QThread already has something completely
different, also called run(). Thus, our choices become more limited.
I disagree that calling them all run() is non-confusing. It produces
an inconsistent API. Imagine that a new user finds a class with these
functions:
public static QFutureT run(QFunction);
public static void run(QRunnable*);
protected void run();
Will this new user be reasonably able to guess their differences?
That said, there is no real reason to put these functions in QThread,
except the name.
These functions could be put elsewhere, I actually started to work
with a bunch of free functions before integrated them to QThread
I think we should try to keep it inside QThread if possible; creating
another class/namespace feels excessive
The implementation use the same sort of generator that QtConcurrent.
Thiago suggested making this feature only compatible C++11, which
would make it easier to maintain. I actually envisaged to send a mail
about that particular issue.
Can c++03 really be dropped for that particular feature ?
Also, I we were to make a c++11 only feature, what would be the
benefits over std::async ?
I would love to see an implementation using C++11. I think that
Re: [Development] Evolving Qt's multithreading API
On 21 February 2013 02:31, Ing. Reynier Pupo Gómez rgo...@uci.cu wrote: What about using of OpenMP standard? It could be very usefull and well known by the C/C++ comunity. Thanks for the suggestion. I had a quick look, but it seems to be on the low-level side. I'm not sure if we want to use #pragmas for regular code... Have you had experience with it? Is it easy to use? Regards, Sze-Howe ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On Friday 22 February 2013 00:07:28 Sze Howe Koh wrote:
On 21 February 2013 02:16, Corentin Jabot corentin.ja...@gmail.com wrote:
Hi. I'm the one Olivier mentioned :p
I didn't have time to pursue further the work I started, but I intend
to, someday.
The plan, as suggested by thiago was to have a QThread::run(functor)
method acting exactly like QtConcurrent::run, but using a new QThead.
A similar QThreadPool::run function would call a functor in a thread
allocated in its pool.
I think the the plan is good; QtConcurrent::run() always felt
out-of-place with the filter-map-reduce API. We just need to be
mindful of how we integrate it into Qt -- the solution should cover
all 3 of functors, QRunnable, and parallel event loops.
I notice you also have QThread::run(QRunnable*). Let's take into
consideration the existing methods QThreadPool::start() and
QThreadPool::tryStart(). It's inconsistent to have:
QThread::run(QRunnable*)
QThreadPool::start(QRunnable*)
(Look for static polymorphism in
http://doc.qt.digia.com/qq/qq13-apis.html) I think we should either
make the new functions match the old names, or rename the old
functions to match the new names. (rename = create new names,
deprecate old names)
Yes. We need to find good APIs.
Those function return QFuture and do not require event loop so this
silly snippet work:
int main() {
auto future = QThread::run( []() { qDebug() Hello world ; } );
future.waitForFinished();
}
My goal was to make it as simple as I could.
Sounds good
Anyway, I don't think we should returns a QThread*. the underlying
implementation does not mater and shouldn't mater.
What would you do with a QThread* you can't do with a QFuture ?
Apologies, I didn't realize that you had a version that returns
QFuture; I only saw the one that returned void. (Line 115,
https://codereview.qt-project.org/#patch,sidebyside,45297,5,src/corelib/thre
ad/qthread.h) I think returning QThread* is better than void. I still can't
decide if QThread* or QFuture is better for these new methods, however.
This has uses case, but i'm not sure it is the main use case.
Some more common use case would be (pseudo-code)
auto watcher = new QFutureWatcher;
QObject::connect(watcher, SIGNAL(finished()), myObject, SLOT(doStuff()));
watcher-setFuture(QThrerad::run([]() { return computeStuff(); } ));
// who deletes the watcher?
I think this pattern should be made easier
Something like:
QObject::connect(
QThread::runFunction([]() { return doSomethingComplicated(); }),
QThread::finished,
this,
MyObject::doStuff //cannot use lambda here because we want a
// QueuedConnection
);
Technically, returning a QThread does hide the implementation. The
programmer only sees the QThread, and doesn't know that you've
implemented a QRunnableThread underneath. The programmer already knows
about QThread (since s/he called QThread::run()), so getting a
QThread* won't be a big deal.
About QThread vs. QFuture... QThread:
- Lets the programmer postpone starting the thread
- Lets the programmer query and manipulate the thread using
event-driven programming (signals and slots)
On the other hand, QFuture makes it easy to retrieve the function's
return value.
I leaned towards returning QThread* because QtConcurrent::run()
couldn't use most of QFuture's features anyway, and QThread* can be
returned from all 3 static functions:
static QThread* QThread::setupSimpleThread(QRunnable *runnable);
static QThread* QThread::setupSimpleThread(Function func, Args arg,
...); static QThread* QThread::setupEventLoop(QObject* worker);
So then the caller is responsible on calling start() and also deleting the
thread ?
...but I'm not sure if losing the functor's return value is worth it.
The Thread is started right-away, you can't really interrupt the
function until its done, etc but you could delete the thread while
it's running, so yeah, I'd rather not a return a pointer to the
thread.
I don't think we have to worry about people deleting the QThread while
it's running. If we did, we'd have similar problems the
QAudioInput::start() (returns a QIODevice*) and
QNetworkAccessManager::get() (returns a QNetworkReply*).
I find the signature QThread::run(function), quite straightforward,
there is no confusion about what it does.
I don't think the fact there is also the static void run() method is
confusing, but maybe that's just me.
I agree that, by itself, run() is a good name for this feature. The
issue, however, is that QThread already has something completely
different, also called run(). Thus, our choices become more limited.
I disagree that calling them all run() is non-confusing. It produces
an inconsistent API. Imagine that a new user finds a class with these
functions:
public static QFutureT
Re: [Development] Evolving Qt's multithreading API
On sexta-feira, 22 de fevereiro de 2013 00.15.19, Sze Howe Koh wrote: On 21 February 2013 02:31, Ing. Reynier Pupo Gómez rgo...@uci.cu wrote: What about using of OpenMP standard? It could be very usefull and well known by the C/C++ comunity. Thanks for the suggestion. I had a quick look, but it seems to be on the low-level side. I'm not sure if we want to use #pragmas for regular code... Have you had experience with it? Is it easy to use? OpenMP is completely out of scope. The most we can do is make Qt's own threading mechanisms work with OpenMP if the user wants to use it. But we can't dictate use of it because it requires compiler support. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
Op 20-2-2013 15:45, Sze Howe Koh schreef: Hi all, Some time ago there was some talk about improving Qt's multithreading API. I'm summarizing them here to stop them from fading into obscurity, and to see if there's any interest in following them up. There is also a proposal posted here: http://qt-project.org/forums/viewthread/2488 André ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On 20 February 2013 22:49, André Somers an...@familiesomers.nl wrote: Op 20-2-2013 15:45, Sze Howe Koh schreef: Hi all, Some time ago there was some talk about improving Qt's multithreading API. I'm summarizing them here to stop them from fading into obscurity, and to see if there's any interest in following them up. There is also a proposal posted here: http://qt-project.org/forums/viewthread/2488 André André meant http://qt-project.org/forums/viewthread/24884/ :-) I'd like to add this idea to the mix: Give Qt's low-level API more flexibility and consistency. Currently, Qt's suggestions for multithreading are: - Subclass a worker QObject and move it to a basic QThread (if an event loop is desired) - Subclass QThread (if an event loop is not desired) - Subclass QRunnable and run it via a QThreadPool (if recycling QThreads is desired) - Call QtConcurrent::run() (if a parallel function call is desired) - Use QtConcurrent's filter-map-reduce API (if high-level container processing is desired) Ignoring filter-map-reduce (the only true high-level option here), the low-level API looks quite disparate. Furthermore, some features are missing: - The ability to use QRunnable, or make a parallel function call, without being tied to a thread pool - The ability to emit a signal when a thread finishes with QRunnable - The ability to delay a parallel function call - The ability to elegantly separate code control and thread logic === PROPOSED NEW METHODS === 1) static QThread* QThread::setupSimpleThread(QRunnable *runnable); 2) static QThread* QThread::setupSimpleThread(Function func, Args arg, ...); 3) static QThread* QThread::setupEventLoop(QObject* worker); 4) void QThreadPool::start(Function func, Args arg, ...); 5) bool QThreadPool::tryStart(Function func, Args arg...); === BEHAVIOUR === (1) binds a QRunnable to a QThread, which will call the QRunnable::run() when start()'ed. (2) binds a function and zero or more arguments to a QThread, which will call the function when start()'ed. (3) moves the worker QObject to the new thread, ready to have its slots invoked when the QThread is start()'ed. (4) and (5) are similar to (2), but it uses a recyclable thread from a QThreadPool and (tries to) start immediately === ADVANTAGES === - The missing features mentioned earlier are provided - A symmetrical API for using both recycled and unrecycled threads(QThreadPool and QThread) - A unified, self-documenting API, which clearly distinguishes and enforces the 2 different ways to using QThread (both with and without an event loop) - A way to cleanly separate thread control (QThread) and threaded code (QRunnable), thus making it more idiot-proof. Also achieved without the huge overhead of the worker-object approach === IMPLEMENTATION === (1) The QThread implementation can be similar to QThreadPoolThread, except that it doesn't check the number of active threads. (2, 4, 5) Ideally, we'd use a variadic template with std::function + std::bind behind the scenes. But, since C++98 support is required, I guess we'll have to follow QtConcurrent::run()'s approach (multiple overloaded templates for different numbers of arguments). (3) setupEventLoop() is just a simple wrapper that instantiates a basic QThread and calls moveToThread() on the worker, before returning the QThread. Thoughts? Regards, Sze-Howe ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On Wednesday 20 February 2013 22:45:21 Sze Howe Koh wrote: Hi all, Some time ago there was some talk about improving Qt's multithreading API. I'm summarizing them here to stop them from fading into obscurity, and to see if there's any interest in following them up. Here are the tasks mentioned: - Replace/Rewrite QtConcurrent [2] - Create/Find a good API to replace QtConcurrent::run() for one-shot tasks [1] - Find a third-party solution for high-level multithreading [2] - Find more uses for QFuture, outside of QtConcurrent [3] - Influence C++1y by creating a nice multithreading API [4] Some suggestions were raised: - Put a Qt-ish wrapper around TBB [1] - Integrate ThreadWeaver back into Qt? [2] Separately, someone was experimenting with ways to spawn a QObject in a secondary thread, without first constructing it in the current thread [5] Do you think any of these avenues are worth pursuing? I've had a quick look at TBB vs. ThreadWeaver. The latter specializes in task-oriented programming, while TBB is a more swiss-army-knife toolkit, which includes container-based operations similar to QtConcurrent. So, if we're to integrate 3rd-party option into Qt, TBB would be more worth it (although it'd involve more work too) Someone has already been working of some feature such as: static QThread::run(QRunable*) static QThread::run(Function) static QThreadPool::run(Function) https://codereview.qt-project.org/#/t/65/ Remember also that C++11 contains already a some set of threading primitive such as std::thread, std::async, std::future -- Olivier Woboq - Qt services and support - http://woboq.com - http://code.woboq.org ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On 20 Feb 2013, at 4:57 PM, Olivier Goffart wrote: On Wednesday 20 February 2013 22:45:21 Sze Howe Koh wrote: Hi all, Some time ago there was some talk about improving Qt's multithreading API. I'm summarizing them here to stop them from fading into obscurity, and to see if there's any interest in following them up. Here are the tasks mentioned: - Replace/Rewrite QtConcurrent [2] - Create/Find a good API to replace QtConcurrent::run() for one-shot tasks [1] - Find a third-party solution for high-level multithreading [2] - Find more uses for QFuture, outside of QtConcurrent [3] - Influence C++1y by creating a nice multithreading API [4] Some suggestions were raised: - Put a Qt-ish wrapper around TBB [1] - Integrate ThreadWeaver back into Qt? [2] Separately, someone was experimenting with ways to spawn a QObject in a secondary thread, without first constructing it in the current thread [5] Do you think any of these avenues are worth pursuing? I've had a quick look at TBB vs. ThreadWeaver. The latter specializes in task-oriented programming, while TBB is a more swiss-army-knife toolkit, which includes container-based operations similar to QtConcurrent. So, if we're to integrate 3rd-party option into Qt, TBB would be more worth it (although it'd involve more work too) Someone has already been working of some feature such as: static QThread::run(QRunable*) static QThread::run(Function) static QThreadPool::run(Function) https://codereview.qt-project.org/#/t/65/ There is also this bug about fixing the examples to show the best practice instead of inheriting from QThread: https://bugreports.qt-project.org/browse/QTBUG-29059 It sounds like the preferred way will be something different after those patches. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
Hello, A few thoughts: - In general and especially for newcomers, encourage task-orientated concurrency and avoiding shared state where possible. This partly about documentation and the examples but also in the kind of approach that the APIs optimize for. - Having had the er, pleasure of debugging a number of concurrency issues in QThread, ThreadWeaver and other third party libraries over the years, I'd say there is a very strong case for going with a wrapper around a well-written and tested existing library where possible. A few years back I tended to use the 'create object, move to thread, send messages' approach to doing background work. The problem is that it usually isn't explicit enough when reading the code what data a particular thread can or cannot safety access. With cross-thread signals, you have the issue that when you emit a signal, it isn't immediately clear at that point that you may be sending data to another thread - depending on who happens to be listening. If you see that a value is being written into a channel on the other hand - as you would find in Go or Rust, it is much more explicit that the value or reference is being passed to another thread for use. Regards, Rob. On 20 February 2013 15:10, Sze Howe Koh szehowe@gmail.com wrote: On 20 February 2013 22:49, André Somers an...@familiesomers.nl wrote: Op 20-2-2013 15:45, Sze Howe Koh schreef: Hi all, Some time ago there was some talk about improving Qt's multithreading API. I'm summarizing them here to stop them from fading into obscurity, and to see if there's any interest in following them up. There is also a proposal posted here: http://qt-project.org/forums/viewthread/2488 André André meant http://qt-project.org/forums/viewthread/24884/ :-) I'd like to add this idea to the mix: Give Qt's low-level API more flexibility and consistency. Currently, Qt's suggestions for multithreading are: - Subclass a worker QObject and move it to a basic QThread (if an event loop is desired) - Subclass QThread (if an event loop is not desired) - Subclass QRunnable and run it via a QThreadPool (if recycling QThreads is desired) - Call QtConcurrent::run() (if a parallel function call is desired) - Use QtConcurrent's filter-map-reduce API (if high-level container processing is desired) Ignoring filter-map-reduce (the only true high-level option here), the low-level API looks quite disparate. Furthermore, some features are missing: - The ability to use QRunnable, or make a parallel function call, without being tied to a thread pool - The ability to emit a signal when a thread finishes with QRunnable - The ability to delay a parallel function call - The ability to elegantly separate code control and thread logic === PROPOSED NEW METHODS === 1) static QThread* QThread::setupSimpleThread(QRunnable *runnable); 2) static QThread* QThread::setupSimpleThread(Function func, Args arg, ...); 3) static QThread* QThread::setupEventLoop(QObject* worker); 4) void QThreadPool::start(Function func, Args arg, ...); 5) bool QThreadPool::tryStart(Function func, Args arg...); === BEHAVIOUR === (1) binds a QRunnable to a QThread, which will call the QRunnable::run() when start()'ed. (2) binds a function and zero or more arguments to a QThread, which will call the function when start()'ed. (3) moves the worker QObject to the new thread, ready to have its slots invoked when the QThread is start()'ed. (4) and (5) are similar to (2), but it uses a recyclable thread from a QThreadPool and (tries to) start immediately === ADVANTAGES === - The missing features mentioned earlier are provided - A symmetrical API for using both recycled and unrecycled threads(QThreadPool and QThread) - A unified, self-documenting API, which clearly distinguishes and enforces the 2 different ways to using QThread (both with and without an event loop) - A way to cleanly separate thread control (QThread) and threaded code (QRunnable), thus making it more idiot-proof. Also achieved without the huge overhead of the worker-object approach === IMPLEMENTATION === (1) The QThread implementation can be similar to QThreadPoolThread, except that it doesn't check the number of active threads. (2, 4, 5) Ideally, we'd use a variadic template with std::function + std::bind behind the scenes. But, since C++98 support is required, I guess we'll have to follow QtConcurrent::run()'s approach (multiple overloaded templates for different numbers of arguments). (3) setupEventLoop() is just a simple wrapper that instantiates a basic QThread and calls moveToThread() on the worker, before returning the QThread. Thoughts? Regards, Sze-Howe ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org
Re: [Development] Evolving Qt's multithreading API
One other thing - QtConcurrent::run(), QRunnable and invokeMethod() do not provide any standard mid-task cancellation mechanism, which I've found to be a very common need in the context of a client app which is offloading a chunk of heavy work to background threads to avoid UI lag - eg. file operations, parsing. QFuture provides a cancel() method and a QThread's event loop can be quit between processing events but these only work between task execution. Regards, Rob. On 20 February 2013 16:24, Robert Knight robertkni...@gmail.com wrote: Hello, A few thoughts: - In general and especially for newcomers, encourage task-orientated concurrency and avoiding shared state where possible. This partly about documentation and the examples but also in the kind of approach that the APIs optimize for. - Having had the er, pleasure of debugging a number of concurrency issues in QThread, ThreadWeaver and other third party libraries over the years, I'd say there is a very strong case for going with a wrapper around a well-written and tested existing library where possible. A few years back I tended to use the 'create object, move to thread, send messages' approach to doing background work. The problem is that it usually isn't explicit enough when reading the code what data a particular thread can or cannot safety access. With cross-thread signals, you have the issue that when you emit a signal, it isn't immediately clear at that point that you may be sending data to another thread - depending on who happens to be listening. If you see that a value is being written into a channel on the other hand - as you would find in Go or Rust, it is much more explicit that the value or reference is being passed to another thread for use. Regards, Rob. On 20 February 2013 15:10, Sze Howe Koh szehowe@gmail.com wrote: On 20 February 2013 22:49, André Somers an...@familiesomers.nl wrote: Op 20-2-2013 15:45, Sze Howe Koh schreef: Hi all, Some time ago there was some talk about improving Qt's multithreading API. I'm summarizing them here to stop them from fading into obscurity, and to see if there's any interest in following them up. There is also a proposal posted here: http://qt-project.org/forums/viewthread/2488 André André meant http://qt-project.org/forums/viewthread/24884/ :-) I'd like to add this idea to the mix: Give Qt's low-level API more flexibility and consistency. Currently, Qt's suggestions for multithreading are: - Subclass a worker QObject and move it to a basic QThread (if an event loop is desired) - Subclass QThread (if an event loop is not desired) - Subclass QRunnable and run it via a QThreadPool (if recycling QThreads is desired) - Call QtConcurrent::run() (if a parallel function call is desired) - Use QtConcurrent's filter-map-reduce API (if high-level container processing is desired) Ignoring filter-map-reduce (the only true high-level option here), the low-level API looks quite disparate. Furthermore, some features are missing: - The ability to use QRunnable, or make a parallel function call, without being tied to a thread pool - The ability to emit a signal when a thread finishes with QRunnable - The ability to delay a parallel function call - The ability to elegantly separate code control and thread logic === PROPOSED NEW METHODS === 1) static QThread* QThread::setupSimpleThread(QRunnable *runnable); 2) static QThread* QThread::setupSimpleThread(Function func, Args arg, ...); 3) static QThread* QThread::setupEventLoop(QObject* worker); 4) void QThreadPool::start(Function func, Args arg, ...); 5) bool QThreadPool::tryStart(Function func, Args arg...); === BEHAVIOUR === (1) binds a QRunnable to a QThread, which will call the QRunnable::run() when start()'ed. (2) binds a function and zero or more arguments to a QThread, which will call the function when start()'ed. (3) moves the worker QObject to the new thread, ready to have its slots invoked when the QThread is start()'ed. (4) and (5) are similar to (2), but it uses a recyclable thread from a QThreadPool and (tries to) start immediately === ADVANTAGES === - The missing features mentioned earlier are provided - A symmetrical API for using both recycled and unrecycled threads(QThreadPool and QThread) - A unified, self-documenting API, which clearly distinguishes and enforces the 2 different ways to using QThread (both with and without an event loop) - A way to cleanly separate thread control (QThread) and threaded code (QRunnable), thus making it more idiot-proof. Also achieved without the huge overhead of the worker-object approach === IMPLEMENTATION === (1) The QThread implementation can be similar to QThreadPoolThread, except that it doesn't check the number of active threads. (2, 4, 5) Ideally, we'd use a variadic template with std::function + std::bind behind the scenes. But, since C++98 support is required, I guess we'll have to
Re: [Development] Evolving Qt's multithreading API
On 20 February 2013 23:57, Olivier Goffart oliv...@woboq.com wrote: Someone has already been working of some feature such as: ... https://codereview.qt-project.org/#/t/65/ Ah, these are quite similar to my second post (http://lists.qt-project.org/pipermail/development/2013-February/009970.html). Some comments: static QThread::run(QRunable*) static QThread::run(Function) - We already have (protected) void QThread::run(), which is the core of all Qt threads. Thus, the new (public static) methods shouldn't be called 'run()'. - QThread is a thread control interface. I think it's preferable for the new (public static) methods to return a QThread pointer, so that the programmer can interact with the new thread. static QThreadPool::run(Function) - We already have (public) void QThreadPool::start(QRunnable*) and (public) bool QThreadPool::tryStart(QRunnable* runnable, int priority = 0). The new function-based method should mirror start() and tryStart() - We need to specify the thread pool to use (the global one, or a custom one), so this can't be static To recap, here are my proposed methods: 1) static QThread* QThread::setupSimpleThread(QRunnable *runnable); 2) static QThread* QThread::setupSimpleThread(Function func, Args arg, ...); 3) static QThread* QThread::setupEventLoop(QObject* worker); 4) void QThreadPool::start(Function func, Args arg, ...); 5) bool QThreadPool::tryStart(Function func, Args arg...); I suggested (1, 2, 3) because Qt currently recommends two ways of using QThread: i) (Without event loop) Subclass QThread, reimplement QThread::run(), instantiate the derived QThread ii) (With event loop) Subclass and instantiate a worker QObject, instantiate a basic QThread, and call moveToThread() These two approaches to using QThread look completely different, and the code isn't clear on whether an event loop is involved or not. Moving forward, - (1, 2, 3) provides the same feel for starting a new low-level thread, regardless of whether an event loop is involved or not - (1, 2, 3) makes it clear if an event loop is involved or not - (1) can replace (i) in most cases -- the effort required by the programmer is exactly the same (unless they want their new thread to emit signals) - (3) can replace (ii) in all cases -- it even results in slightly shorter code There are still some wrinkles to iron out though: - (4) is slightly inconsistent with its existing counterpart 'void QThreadPool::start(QRunnable* runnable, int priority = 0)'... any suggestions? - Should (3) accept an arbitrary number of QObjects? Remember also that C++11 contains already a some set of threading primitive such as std::thread, std::async, std::future Yes, although it's good to polish what we already have :) Regards, Sze-Howe ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On 21 February 2013 00:02, Rutledge Shawn shawn.rutle...@digia.com wrote: On 20 Feb 2013, at 4:57 PM, Olivier Goffart wrote: On Wednesday 20 February 2013 22:45:21 Sze Howe Koh wrote: Hi all, Some time ago there was some talk about improving Qt's multithreading API. I'm summarizing them here to stop them from fading into obscurity, and to see if there's any interest in following them up. Here are the tasks mentioned: - Replace/Rewrite QtConcurrent [2] - Create/Find a good API to replace QtConcurrent::run() for one-shot tasks [1] - Find a third-party solution for high-level multithreading [2] - Find more uses for QFuture, outside of QtConcurrent [3] - Influence C++1y by creating a nice multithreading API [4] Some suggestions were raised: - Put a Qt-ish wrapper around TBB [1] - Integrate ThreadWeaver back into Qt? [2] Separately, someone was experimenting with ways to spawn a QObject in a secondary thread, without first constructing it in the current thread [5] Do you think any of these avenues are worth pursuing? I've had a quick look at TBB vs. ThreadWeaver. The latter specializes in task-oriented programming, while TBB is a more swiss-army-knife toolkit, which includes container-based operations similar to QtConcurrent. So, if we're to integrate 3rd-party option into Qt, TBB would be more worth it (although it'd involve more work too) Someone has already been working of some feature such as: static QThread::run(QRunable*) static QThread::run(Function) static QThreadPool::run(Function) https://codereview.qt-project.org/#/t/65/ There is also this bug about fixing the examples to show the best practice instead of inheriting from QThread: https://bugreports.qt-project.org/browse/QTBUG-29059 It sounds like the preferred way will be something different after those patches. The Qt Project community is divided on what should be the preferred way. Some are strongly against subclassing QThread altogether, but others maintain that subclassing QThread has its valid use cases (see the actual changes, and the comments in https://codereview.qt-project.org/#change,45271) The first group doesn't like subclassing QThread, because it mixes thread control with threaded code, making it easy for newcomers to shoot themselves in the foot. The only way around that, currently, is to use a worker QObject all the time, with queued signal-slot communication. However, this produces overly-complex code and a big runtime overhead if you don't actually need an event loop. This, and the fact that subclassing is normal in C++, is the basis of the second group's stance. (IIRC, the very frequent foot-shooting led to this post: http://blog.qt.digia.com/blog/2010/06/17/youre-doing-it-wrong/, which was rebutted recently with http://woboq.com/blog/qthread-you-were-not-doing-so-wrong.html) I was hoping that the new, proposed API can address all concerns. Regards, Sze-Howe P.S. I realize I've made quite sweeping statements about people's stances; I sincerely hope I've represented all views accurately ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
Hi. I'm the one Olivier mentioned :p
I didn't have time to pursue further the work I started, but I intend
to, someday.
The plan, as suggested by thiago was to have a QThread::run(functor)
method acting exactly like QtConcurrent::run, but using a new QThead.
A similar QThreadPool::run function would call a functor in a thread
allocated in its pool.
Those function return QFuture and do not require event loop so this
silly snippet work:
int main() {
auto future = QThread::run( []() { qDebug() Hello world ; } );
future.waitForFinished();
}
My goal was to make it as simple as I could.
Anyway, I don't think we should returns a QThread*. the underlying
implementation does not mater and shouldn't mater.
What would you do with a QThread* you can't do with a QFuture ?
The Thread is started right-away, you can't really interrupt the
function until its done, etc but you could delete the thread while
it's running, so yeah, I'd rather not a return a pointer to the
thread.
I find the signature QThread::run(function), quite straightforward,
there is no confusion about what it does.
I don't think the fact there is also the static void run() method is
confusing, but maybe that's just me.
That said, there is no real reason to put these functions in QThread,
except the name.
These functions could be put elsewhere, I actually started to work
with a bunch of free functions before integrated them to QThread
The implementation use the same sort of generator that QtConcurrent.
Thiago suggested making this feature only compatible C++11, which
would make it easier to maintain. I actually envisaged to send a mail
about that particular issue.
Can c++03 really be dropped for that particular feature ?
Also, I we were to make a c++11 only feature, what would be the
benefits over std::async ?
---
About running QObject* methods on a separate thread, it can become
quite complex.
Which slot is called when the thread start ? Which signal makes the
thread quit ? Should the object be deleted afterwards ? Moved back to
its original thread ?
Considering the number of scenario, I'm not sure we would benefit from
a function - the only factorizable part is
QThread* t = new QThread();
obj-moveToThread(t);
connect(t, SIGNAL(finished()), t, SLOT(deleteLater()));
Maybe we should first agree on how QThread sould work before trying to
add yet-another-way.
Regards,
Corentin
2013/2/20 Sze Howe Koh szehowe@gmail.com:
On 21 February 2013 00:02, Rutledge Shawn shawn.rutle...@digia.com wrote:
On 20 Feb 2013, at 4:57 PM, Olivier Goffart wrote:
On Wednesday 20 February 2013 22:45:21 Sze Howe Koh wrote:
Hi all,
Some time ago there was some talk about improving Qt's multithreading
API. I'm summarizing them here to stop them from fading into
obscurity, and to see if there's any interest in following them up.
Here are the tasks mentioned:
- Replace/Rewrite QtConcurrent [2]
- Create/Find a good API to replace QtConcurrent::run() for one-shot
tasks
[1] - Find a third-party solution for high-level multithreading [2]
- Find more uses for QFuture, outside of QtConcurrent [3]
- Influence C++1y by creating a nice multithreading API [4]
Some suggestions were raised:
- Put a Qt-ish wrapper around TBB [1]
- Integrate ThreadWeaver back into Qt? [2]
Separately, someone was experimenting with ways to spawn a QObject in
a secondary thread, without first constructing it in the current
thread [5]
Do you think any of these avenues are worth pursuing?
I've had a quick look at TBB vs. ThreadWeaver. The latter specializes
in task-oriented programming, while TBB is a more swiss-army-knife
toolkit, which includes container-based operations similar to
QtConcurrent. So, if we're to integrate 3rd-party option into Qt, TBB
would be more worth it (although it'd involve more work too)
Someone has already been working of some feature such as:
static QThread::run(QRunable*)
static QThread::run(Function)
static QThreadPool::run(Function)
https://codereview.qt-project.org/#/t/65/
There is also this bug about fixing the examples to show the best practice
instead of inheriting from QThread:
https://bugreports.qt-project.org/browse/QTBUG-29059
It sounds like the preferred way will be something different after those
patches.
The Qt Project community is divided on what should be the preferred
way. Some are strongly against subclassing QThread altogether, but
others maintain that subclassing QThread has its valid use cases (see
the actual changes, and the comments in
https://codereview.qt-project.org/#change,45271)
The first group doesn't like subclassing QThread, because it mixes
thread control with threaded code, making it easy for newcomers to
shoot themselves in the foot. The only way around that, currently, is
to use a worker QObject all the time, with queued signal-slot
communication. However, this produces overly-complex code and a big
runtime overhead if you don't actually need an event loop.
Re: [Development] Evolving Qt's multithreading API
What about using of OpenMP standard? It could be very usefull and well known by the C/C++ comunity. -- Linux Registered User: #515619 Linux Registered Machine: #421715 ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Evolving Qt's multithreading API
On 21 February 2013 00:29, Robert Knight robertkni...@gmail.com wrote: One other thing - QtConcurrent::run(), QRunnable and invokeMethod() do not provide any standard mid-task cancellation mechanism, which I've found to be a very common need in the context of a client app which is offloading a chunk of heavy work to background threads to avoid UI lag - eg. file operations, parsing. QFuture provides a cancel() method and a QThread's event loop can be quit between processing events but these only work between task execution. Yes, C++ doesn't provide a way to interrupt a function block (there wouldn't be a safe way to do so, anyway). The onus is on the programmer to split their code into chunks/tasks, be it at a low level (threading primitives) or high level (task-oriented programming) On 20 February 2013 16:24, Robert Knight robertkni...@gmail.com wrote: Hello, A few thoughts: - In general and especially for newcomers, encourage task-orientated concurrency and avoiding shared state where possible. This partly about documentation and the examples but also in the kind of approach that the APIs optimize for. I believe that the move-worker-QObject-to-thread approach aimed for zero shared state -- all data is to be stored privately in the object, and transferred between threads only via queued signals and slots. Attempting to polish documentation/examples made me realize that we're still somewhat divided on what approach to optimize for, hence the start of this discussion :) Tangent: I recently realized that task-oriented programming is basically dataflow programming, which is inherently multithreaded. Very powerful for data processing. - Having had the er, pleasure of debugging a number of concurrency issues in QThread, ThreadWeaver and other third party libraries over the years, I'd say there is a very strong case for going with a wrapper around a well-written and tested existing library where possible. Agreed. TBB is looking quite attractive right now; do you have any other recommendations? A few years back I tended to use the 'create object, move to thread, send messages' approach to doing background work. The problem is that it usually isn't explicit enough when reading the code what data a particular thread can or cannot safety access. Would it have been possible to encapsulate data within objects that live in particular threads? I found that this approach worked quite well for me. I know it's safe for this object to work on this data, 'cos it's the only entity which can see it! (granted, my projects were small-ish) With cross-thread signals, you have the issue that when you emit a signal, it isn't immediately clear at that point that you may be sending data to another thread - depending on who happens to be listening. If you see that a value is being written into a channel on the other hand - as you would find in Go or Rust, it is much more explicit that the value or reference is being passed to another thread for use. Hmm... Data transferred through queued signals and slots are always copied, if I'm not mistaken (unless it's a pointer -- the pointed item isn't copied then). And the idea of signals/slots was to decouple the sender from the receiver(s), so that they don't need to know about each other. Can you give an example where the destination thread of my emitted data would affect my program? Regards, Sze-Howe ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
