Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Thursday 10 July 2014 10:22:14 Thiago Macieira wrote: On Thursday 10 July 2014 07:56:28 Koehne Kai wrote: Provide a define QT_LOG_TO_CONSOLE that let QCoreApplication.h record whether it should log to console (QT_LOG_TO_CONSOLE=1), to the system (QT_LOG_TO_CONSOLE=0), or both (QT_LOG_TO_CONSOLE=2). Qmake can wrap this into convenience CONFIG arguments: CONFIG+=log_to_console log_to_system It's easier to set an AA flag. http://qt-project.org/doc/qt-5/qcoreapplication.html#setAttribute http://qt-project.org/doc/qt-5/qt.html#ApplicationAttribute-enum PS: doc team, the second link above is unreadable. There was no other support for this suggestion, so I'm not taking it forward. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Friday 11 July 2014 14:47:04 Joerg Bornemann wrote: On 11-Jul-14 14:14, Joerg Bornemann wrote: Here's the code. Maybe I missed something. Ossi showed me what I missed. We cannot just change stdout unconditionally. So no reason to not do this, as long as the application call is prefixed with start /wait to run it in the foreground. Please comment on https://codereview.qt-project.org/89357 with the exact code that needs to be written for Windows. I won't have access to a Windows machine until after the feature freeze. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
-Original Message- From: development-bounces+kai.koehne=digia@qt-project.org [...] Some background: on Sailfish, we launch all major system services (and UI) through systemd services ( user session for the UI parts). Those processes may then choose to execute processes of their own (the most prominent example being the UI environment which will launch applications in response to user input). With a default setup, systemd will log everything sent to stderr of systems-launched processes, and child-launched processes will have their stderr output attributed to the systemd service which launched them. Out of curiosity: Would an opt-in solution (the application explicitly has to tell Qt to log to journal, e.g. through an API, maybe even through qtlogging.ini file) work for your use case? I'm wondering whether such a setup wouldn't be a good one for desktop distributions, too. I can see that e.g. warning and errors from kwin are best kept in journald. Not sure whether that means though that you necessarily want all spurious warnings and messages from every single KDE app in there. Regards Kai ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Thu, Jul 10, 2014 at 7:14 PM, Thiago Macieira thiago.macie...@intel.com wrote: I'm not asking that distributions turn journald on if they don't want to. Journald is optional. Good to have that fact stated clearly:-) snip section on remote debugging and having a way to force stderr It's there in two ways: 1) Option -t to ssh, which forces the allocation of a TTY and, thus, logging to stderr 2) The environment variable Note that it's also possible to query the system log remotely too. I believe that is done on Android, isn't it? From all I can find on the Internet, in order to enable capture of stderr, you need to have a rooted device. AFAIK android is not using systemd/journald, so what we do there does not directly apply to the situation here. I meant that each plugin for remote systems needs to decide how it's going to approach this problem. If it can access the remote log, all the better. If it can't, the best way is to use ssh -t -- with the added benefit that it causes stdio to become line-buffered. If ssh is not getting used, make sure the environment variable is set. Sure, we can come up with case-by-case ways to get to data. But I really want to keep the simple remote debugging use case working. That currently requires sshd and gdbserver to be installed on the device and I would personally prefer not to have additional dependencies on top of that. I found some documentation on how to configure journald to forward its logs to a pre-configured remote machine, which does not fit our use case. We need to access the logs from the machine of the developer and that will rarely be the central log server (I hope;-). At least journalctl can produce machine readable output, so we might get away by just retrieving that by running journalctl -o json. That is way more data than what we currently get via stderr, so this will need more bandwidth than the current approach. That might already be prohibitive on some hardware. But without code that is just speculation. And there's another advantage: if Creator is able to query the system log, it is also able to print the output of an already-running application that you attached the debugger to. I see why you would want journal data in your output, but I do see a problems making that work for remote debugging/running. Even sailfish is apparently forcing output to stderr when starting applications on-device via Qt Creator. Does somebody have experience with reading data from journald remotely? Best Regards, Tobias ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Fri, Jul 11, 2014 at 11:13 AM, Hunger Tobias tobias.hun...@digia.com wrote: I see why you would want journal data in your output, but I do see a problems making that work for remote debugging/running. Even sailfish is apparently forcing output to stderr when starting applications on-device via Qt Creator. Don't let our existing solution limit you. There may be better solutions, we were time-constrained, and now resource-constrained (i.e. we have to pick and choose what we put effort into..) so we haven't looked into changing this any more, as far as I know. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
Looks like there are many different uses cases and point of views, we may not reach an agreement. I agree that the default settings should be as unified as possible, but, on the other and, each platform having different capabilities, it make sense to set the default behavior on a platform-per-platform basis; there are only 2 or 3 possible choices after-all. II the documentation states explicitly what the behavior is on each platform and that the behavior is to be expected on that platform, I don't think people will get confused. I really think we need a way to override the log output in the application with an api such as Denis proposed, in addition to the existing environment variable. Having a global or per-application file may be to much, things should be kept simple. I still don't think qt creator should read the output from the journal when it can avoid it. Reading the system log + stdout, merging both in a sane way ( which means qt creator will have to time stamp the output from stdout) looks overly complicated to me. The extra information the journal provide will be lost anyway since Qt Creator's output is very basic (but, sufficient most of the time). And if the order in which each line is displayed is inaccurate, it could make the debugging much harder. The extra work of reading the journal may also create a delay in the output, which is annoying when debugging. The debugging with an other IDE should also work out-of-the-box. It's not that debugging should be handled specifically, more like debugging should not be handled differently, compared to any another application, and the burden should not be put on the IDE. That being said, if it comes to this, I have no problem with Qt creator setting QT_LOGGING_TO_CONSOLE by default. As for attaching windows applications to their parent's console automatically, that would be really neat. 2014-07-11 11:25 GMT+02:00 Robin Burchell robin...@viroteck.net: On Fri, Jul 11, 2014 at 11:13 AM, Hunger Tobias tobias.hun...@digia.com wrote: I see why you would want journal data in your output, but I do see a problems making that work for remote debugging/running. Even sailfish is apparently forcing output to stderr when starting applications on-device via Qt Creator. Don't let our existing solution limit you. There may be better solutions, we were time-constrained, and now resource-constrained (i.e. we have to pick and choose what we put effort into..) so we haven't looked into changing this any more, as far as I know. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On 10-Jul-14 20:03, Thiago Macieira wrote:
That means we can replace the GetConsoleWindow() call with
AttachConsole(ATTACH_PARENT_PROCESS). If it succeeds or if it returns
ERROR_ACCESS_DENIED, we have a console.
For the Windows folks around: should we do this?
I've just played around with this and unfortunately channel redirection
does not work with this approach.
myapp foo.txt
starts myapp in the background like myapp on *nix would do.
The log is written to the console window and doesn't appear in the file.
cmd /c myapp foo.txt
Blocks the shell as expected but redirection still doesn't work.
Here's the code. Maybe I missed something.
---snip---
#include QApplication
#include qt_windows.h
#include cstdio
#include fcntl.h
#include io.h
int main(int argc, char *argv[])
{
QApplication a(argc, argv);
if (!AttachConsole(ATTACH_PARENT_PROCESS))
return 1;
HANDLE h = GetStdHandle(STD_OUTPUT_HANDLE);
int fd = _open_osfhandle((intptr_t)h, _O_TEXT);
if (fd = 0)
return 2;
*stdout = *_fdopen(fd, w);
setvbuf(stdout, NULL, _IONBF, 0 );
printf(Hello World from GUI app!\n);
QMetaObject::invokeMethod(a, quit, Qt::QueuedConnection);
return a.exec();
}
---snap---
Joerg
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On 11-Jul-14 01:18, Thiago Macieira wrote: Logging to both is a problem if you have something that is able to read from both. On Windows, all debuggers and IDEs read from both. Therefore, the option is out. That's not true. Visual Studio displays output from OutputDebugString in its debug output pane. The stderr/stdout output is printed to the console window that gets opened when you run your console application. The output is separated and that makes perfect sense. Qt Creator could have a separate output pane for OutputDebugString and we're good. BTW QTestLib did log to both until 5.2.1. See 7fab8eb56b9309734d1b34d3d935d6a4cf14ce6e BR, Joerg ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On 11-Jul-14 14:14, Joerg Bornemann wrote:
Here's the code. Maybe I missed something.
Ossi showed me what I missed.
We cannot just change stdout unconditionally.
So no reason to not do this, as long as the application call is prefixed
with start /wait to run it in the foreground.
---snip---
#include QApplication
#include qt_windows.h
#include cstdio
#include fcntl.h
#include io.h
int main(int argc, char *argv[])
{
QApplication a(argc, argv);
HANDLE h = GetStdHandle(STD_OUTPUT_HANDLE);
if (h == INVALID_HANDLE_VALUE || h == NULL) {
if (!AttachConsole(ATTACH_PARENT_PROCESS))
return 1;
h = GetStdHandle(STD_OUTPUT_HANDLE);
int fd = _open_osfhandle((intptr_t)h, _O_TEXT);
if (fd = 0)
return 2;
*stdout = *_fdopen(fd, w);
setvbuf(stdout, NULL, _IONBF, 0 );
}
printf(Hello World from GUI app!\n);
QMetaObject::invokeMethod(a, quit, Qt::QueuedConnection);
return a.exec();
}
---snap---
Joerg
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Friday 11 July 2014 09:13:30 Hunger Tobias wrote: Note that it's also possible to query the system log remotely too. I believe that is done on Android, isn't it? From all I can find on the Internet, in order to enable capture of stderr, you need to have a rooted device. AFAIK android is not using systemd/journald, so what we do there does not directly apply to the situation here. It does because we use a logging system very similar to journald and the criteria to choose when to use that or when to use stderr is the same. Instead of sd_journald_send, we use __android_log_print. That's the only difference. I meant that each plugin for remote systems needs to decide how it's going to approach this problem. If it can access the remote log, all the better. If it can't, the best way is to use ssh -t -- with the added benefit that it causes stdio to become line-buffered. If ssh is not getting used, make sure the environment variable is set. Sure, we can come up with case-by-case ways to get to data. But I really want to keep the simple remote debugging use case working. That currently requires sshd and gdbserver to be installed on the device and I would personally prefer not to have additional dependencies on top of that. That's there. I've given you two possible solutions to get the log sent to stderr in those cases. I found some documentation on how to configure journald to forward its logs to a pre-configured remote machine, which does not fit our use case. We need to access the logs from the machine of the developer and that will rarely be the central log server (I hope;-). At least journalctl can produce machine readable output, so we might get away by just retrieving that by running journalctl -o json. That is way more data than what we currently get via stderr, so this will need more bandwidth than the current approach. That might already be prohibitive on some hardware. But without code that is just speculation. Right. That would be a third option, but it would also require figuring out what the target system is doing. Maybe for remote debugging we don't want to mess with journald or something else. Suppose you were ssh'ing into an OS X machine. Then we're not talking about journald, but about asl. And there's another advantage: if Creator is able to query the system log, it is also able to print the output of an already-running application that you attached the debugger to. I see why you would want journal data in your output, but I do see a problems making that work for remote debugging/running. Even sailfish is apparently forcing output to stderr when starting applications on-device via Qt Creator. When I wrote the paragraph above, I was thinking of local debugging and attaching to already-running processes, processes that Creator did not launch. For local debugging, it's more reasonable. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Friday 11 July 2014 08:59:35 Alejandro Exojo wrote: I agree. Which is why my proposal email does not talk about debug mode. Oh, I'm confused then. It does in the Patch Set 6, no? The commit does not match the proposal. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Friday 11 July 2014 14:32:51 Joerg Bornemann wrote: BTW QTestLib did log to both until 5.2.1. See 7fab8eb56b9309734d1b34d3d935d6a4cf14ce6e To be clear: since 5.2.0 until 5.2.1. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
-Original Message- From: development-bounces+kai.koehne=digia@qt-project.org [...] On the contrary, the end-user on Windows expects the application logs to be in a file somewhere near the executable if it's not a system service. So in my opinion the logs shouldn't write into system unless explicitly asked to. I think logging to a file is a useful option. But I disagree that it should be the default, at least for Windows. First, you've the problem to find a file 'somewhere near the executable' that you can write to. For end-users, you don't necessarily have write permissions to Program Files, and even if you have it, you don't want data to be put there, either. The only place that should always work would probably be %localappdata%. But even if you manage to do that: You need to take care of concurrent accesses (multiple instances of your application might run). Then someone comes along and complains about his user directory hitting the quota, so you've to implement some cleanup/rotation ... It gets quite a big task. Then there's the IDE side: You can adapt Qt Creator, but Visual Studio etc won't know a thing about your log file. Despite all its limits (one global buffer, only plain string logging, only one debugger/viewer can attach ...), the Windows system log is the recommended way to log debugging information for GUI applications. I'm not arguing against adding such a file based backend, but it should be the default. Moreover afaik not every enterprise user has access to System Logs on Windows. First time I hear this (and I don't think we have had a lot complains about it so far?). Anyway, this is one of the things where I'd rather tell people to fix their policies ( I know, I know, it's a hard fight ...). Also, you can just run your application in the debugger (as long as you've the rights to do this ;), or implement a simple log-to-file in 10 lines of code yourself. Regards Kai ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
-Original Message- From: development-bounces+kai.koehne=digia@qt-project.org [mailto:development-bounces+kai.koehne=digia@qt-project.org] On Behalf Of Thiago Macieira Sent: Wednesday, July 09, 2014 11:44 PM To: development@qt-project.org Subject: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357 == Problematic == Qt 5.3 and dev are currently able to send the output of qDebug, qWarning and the rest of the logging framework to somewhere other than stderr. That's done on Windows, BlackBerry, Android, and on Linux systems with journald. Additionally, there's a pending patch for OS X to use the Apple System Log [5] and I had a patch for MeeGo that made it go to the standard Unix syslog. We have a problem of wanting a cake and eating it too: 1) we want applications to send their logging to the system log where possible, which usually logs more information like the log level, time, application name and PID. Such a system is searchable and may allow us to add additional information, like the category (for category logging), file name, line number, function context, etc. 2) we also want to see the output when we run applications from the terminal or capture the output with QProcess. The absence of output causes bug reports like [2] and [4]. == Solutions == === Log to both === Aside from the extra overhead, this causes systems that capture both stderr and the system log to record and display the same message twice. That's the source of task [3]. This is not an option. === Log to stderr only === That results in loss of information, since the system logs are richer and more searchable. What's more, on Windows that would result in no output at all for GUI applications. I can't find a lot of information on whether GUI applications launched from an open command prompt would show anything. IIRC, they don't. This is not an option. === Log to system logs only === This causes reports like [4]. But note that the bug is actually in Creator, for failing to read the log store and display the data, like it does on Windows with the debug output. This is an option. We might consider also deferring the decision to the application: === Defer decision to application === Provide a define QT_LOG_TO_CONSOLE that let QCoreApplication.h record whether it should log to console (QT_LOG_TO_CONSOLE=1), to the system (QT_LOG_TO_CONSOLE=0), or both (QT_LOG_TO_CONSOLE=2). Qmake can wrap this into convenience CONFIG arguments: CONFIG+=log_to_console log_to_system And individual (system) mkspecs might set a default, so e.g. Windows might get QT_LOG_TO_CONSOLE=2 for gui apps. If nothing is set, let the heuristic kick in ... Regards Kai ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
Sorry for my vague statements :) Actually I wasn't talking about logging to file as a default for Windows. On the contrary I think this has to be done by each application individually -- someone needs as you rightly say 10 lines of code just to make sure everything went okay, others need a large logging facility with rotation, archiving, severity of messages and maybe even mirroring some messages to a database. There's no universal solution here, the former would complain about dependencies, the latter would extend the existing logger or invent their own. What I'm trying to say is that logging to file seems to be more common approach on Windows than logging to Event Log accessed from the Control Panel - Administrative unless it is a service or application produced by Microsoft :) Access to Event Log by end-user can really be restricted by Domain Policies which affects mostly enterprise users, though I totally agree that those who are affected should do something about it. Cheers 2014-07-10 12:46 GMT+06:00 Koehne Kai kai.koe...@digia.com: -Original Message- From: development-bounces+kai.koehne=digia@qt-project.org [...] On the contrary, the end-user on Windows expects the application logs to be in a file somewhere near the executable if it's not a system service. So in my opinion the logs shouldn't write into system unless explicitly asked to. I think logging to a file is a useful option. But I disagree that it should be the default, at least for Windows. First, you've the problem to find a file 'somewhere near the executable' that you can write to. For end-users, you don't necessarily have write permissions to Program Files, and even if you have it, you don't want data to be put there, either. The only place that should always work would probably be %localappdata%. But even if you manage to do that: You need to take care of concurrent accesses (multiple instances of your application might run). Then someone comes along and complains about his user directory hitting the quota, so you've to implement some cleanup/rotation ... It gets quite a big task. Then there's the IDE side: You can adapt Qt Creator, but Visual Studio etc won't know a thing about your log file. Despite all its limits (one global buffer, only plain string logging, only one debugger/viewer can attach ...), the Windows system log is the recommended way to log debugging information for GUI applications. I'm not arguing against adding such a file based backend, but it should be the default. Moreover afaik not every enterprise user has access to System Logs on Windows. First time I hear this (and I don't think we have had a lot complains about it so far?). Anyway, this is one of the things where I'd rather tell people to fix their policies ( I know, I know, it's a hard fight ...). Also, you can just run your application in the debugger (as long as you've the rights to do this ;), or implement a simple log-to-file in 10 lines of code yourself. Regards Kai ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
-Original Message- From: development-bounces+kai.koehne=digia@qt-project.org [...] Here's what I propose: Heuristically determine at runtime. On all platforms (including Windows), use only the fact that a terminal is present. If a terminal is present, write to it using stderr. Otherwise, write to the system log. * Implications for Qt Creator: 1) it must be able to read from the system log for desktop debugging (especially when running applications without Run in terminal). That's already done on Windows. It will be necessary to add support for journald (Linux) and asl (OS X). Well, actually your latest patch that checks for stderr!=/dev/null will just work fine with current Qt Creator. That's for journal/Linux at least, haven't had a look yet how asl works. Qt Creator needs to continue listening to stderr in any case: You still want to see stdout/stderr, after all. It's true that Qt Creator can in theory profit from additional information piped through systemd, but integrating both streams also has it's issues: E.g. the order of qDebug/stderr messages is no longer consistent, and interleaving lines might need special attention (though I guess we handle that already for stdout/stderr). 2) remote debugging: per platform, it needs to be decided whether we can access the log on the remote device. If so, that's the best situation. If it isn't, then we need to force Qt to log to stderr. 2a) set QT_LOGGING_TO_CONSOLE=1 in the environment 2b) force the allocation of a terminal (if using ssh, pass the -t option) See above. Unless stderr somehow ends up in the system log too we need to anyway have the terminal. I recommend advising Linux distributions to disable journald logging at least until Creator is updated. And the support for asl on OSX should be held until the same time. I agree (though I'm not sure whether Qt Creator is the blocker, or if it's enough to handle the issue in Qt. See above). * Implications for unit tests: Except for the unit tests testing logging to the system log in the first place, QT_LOGGING_TO_CONSOLE=1 should be set. I think QtTest itself should do that. * Support for journald in the pre-built Creator binary: 1) new plugin, link to libsystemd-journal.so.0. If the library isn't present in the target system, the plugin won't load. 2) dlopen libsystemd-journal.so.0. If it isn't present, then Qt can't very well be logging to it. 3) use QProcess to run journalctl -f * Support for journald in the pre-built Qt binaries: Disabled. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
Perhaps, it makes sense to select an output message sink in runtime?
E.g. before QCoreApplication, by means of specify some QLogging flags, to
override the installMessageHandler and so on?
Maybe make sense to introduce a some pre-defined qInstallMessageHandler's()
or something others functions, like:
qInstallMessageDestination(enum MessageDestinationFlag)
, where
enum MessageDestinationFlag
{
MessageToConsole,
MessageToDebugger,
MessageToSystemLog,
and etc.
}
For example, then user can select manually what output need:
int main()
{
qInstallMessageDestination(MessageToConsole | MessageToSystemLog);
QCoreApplication app;
}
Besides, in Windows there are four various directions of a messages output:
1) To the Console
2) To the Debug output, using the OutputDebugString function:
http://msdn.microsoft.com/ru-ru/library/windows/desktop/aa363362%28v=vs.85%29.aspx
3) To the SystemLog (Journal), using the
RegisterEventSource:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa363678%28v=vs.85%29.aspx
ReportEvent:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa363679%28v=vs.85%29.aspx
functions (as already was made in QtService from qt-solutions).
4) Using WPP Software Tracing feature:
http://msdn.microsoft.com/en-us/library/windows/hardware/ff556204%28v=vs.85%29.aspx
Thus, I prefer that the user himself chose where he wants to print out the
log messages.
Best regards,
Denis
2014-07-10 12:03 GMT+04:00 Dmitriy Purgin dpur...@gmail.com:
Sorry for my vague statements :)
Actually I wasn't talking about logging to file as a default for
Windows. On the contrary I think this has to be done by each
application individually -- someone needs as you rightly say 10 lines
of code just to make sure everything went okay, others need a large
logging facility with rotation, archiving, severity of messages and
maybe even mirroring some messages to a database. There's no universal
solution here, the former would complain about dependencies, the
latter would extend the existing logger or invent their own.
What I'm trying to say is that logging to file seems to be more common
approach on Windows than logging to Event Log accessed from the
Control Panel - Administrative unless it is a service or application
produced by Microsoft :) Access to Event Log by end-user can really be
restricted by Domain Policies which affects mostly enterprise users,
though I totally agree that those who are affected should do something
about it.
Cheers
2014-07-10 12:46 GMT+06:00 Koehne Kai kai.koe...@digia.com:
-Original Message-
From: development-bounces+kai.koehne=digia@qt-project.org
[...]
On the contrary, the end-user on Windows expects the application logs
to be
in a file somewhere near the executable if it's not a system service.
So in my
opinion the logs shouldn't write into system unless explicitly asked to.
I think logging to a file is a useful option. But I disagree that it
should be the default, at least for Windows.
First, you've the problem to find a file 'somewhere near the executable'
that you can write to. For end-users, you don't necessarily have write
permissions to Program Files, and even if you have it, you don't want data
to be put there, either. The only place that should always work would
probably be %localappdata%.
But even if you manage to do that: You need to take care of concurrent
accesses (multiple instances of your application might run). Then someone
comes along and complains about his user directory hitting the quota, so
you've to implement some cleanup/rotation ... It gets quite a big task.
Then there's the IDE side: You can adapt Qt Creator, but Visual Studio
etc won't know a thing about your log file.
Despite all its limits (one global buffer, only plain string logging,
only one debugger/viewer can attach ...), the Windows system log is the
recommended way to log debugging information for GUI applications. I'm not
arguing against adding such a file based backend, but it should be the
default.
Moreover afaik not every enterprise user has access to System Logs on
Windows.
First time I hear this (and I don't think we have had a lot complains
about it so far?). Anyway, this is one of the things where I'd rather tell
people to fix their policies ( I know, I know, it's a hard fight ...).
Also, you can just run your application in the debugger (as long as you've
the rights to do this ;), or implement a simple log-to-file in 10 lines of
code yourself.
Regards
Kai
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
Hi Thiago, Basically I agree with your statements, but I do not think we can rely on journald at this time. The first problem is of course systemd itself: Ubuntu is one of the biggest distros out there and we can not reasonably assume that to be running systemd before 14.04 is at the end of its life-cycle. Considering that this is a long term support release that will be a while. But even when systemd is in use journald might be disabled or misconfigured -- as you already said yourself. I hope gnome and KDE starting to depend on systemd user sessions will get that sorted out soon, but at this time those changes are not yet deployed in distributions. So I think being able to rely on logging to journald is still a long time off. So what is the fallback? Stderr again? Adding Journald into the picture will also make retrieving warnings from remote machines a lot more difficult. Currently we rely on SSH and/or gdbserver to retrieve all application output on Unix systems. I am not sure at this time how that would work if the application starts sending output to the journal. So having a way to force stderr in favor of system logs would be greatly appreciated. Best Regards, Tobias PS: Creator not supporting getting logs from system logs on Unix systems is not a bug, it is a feature nobody requested so far:-) ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
What if we use qtlogging.ini (or equivalent on other systems) to control whether the logs go to the system log or not? I agree that every ordinary KDE user doesn't need to have unread logs taking up disk space, but maybe developers would like to turn on output to the journal permanently for after-the-fact diagnosis when something unexpectedly crashes or misbehaves. There is already a discoverability problem about the available categories, so I was thinking we need a nice tool to manage qtlogging.ini (a widget-based app or a menu-driven console app). This tool could turn on and off the system logging too. Maybe it could be a feature of qtdiag instead of a separate tool (since logging is a kind of diagnostic device). On Windows I never liked CONFIG += console, because it's never in the .pro by default (usually not in manual tests or examples, and not in Creator-generated pro files either) and yet I would like to always have qDebug output in the console when the app is running in a console. So detecting whether the app is running in a console and automatically sending debug output there would mean that we no longer need that flag in the .pro, right? If it's technically possible, we should have done that years ago. I never want a separate pop-up console otherwise because it's an annoyance and has too few features. A log viewer which allows searching and filtering would be far more useful. So another idea is what about including with the Qt SDK a widget-based log viewer, with UI for turning the categories on and off, and also some searching and filtering features? If you are using it, the logging output would be redirected there instead of stderr or the console... but the question is how do you use it: by having it start the application? or have UI to select and attach to already-running applications? I also don't like Creator's default run in terminal checkbox because again I don't get to choose which terminal, and the one it chooses is always 80x25, white background, has no search feature, etc. So I have to remember to turn off the checkbox because the default is not what I want. IMO the Application Output pane should have enough features that it can be treated as the console in which the app is running. What is missing, that we don't already do this? Or if we write a log viewer tool, Creator could use that as the Application Output pane instead of needing to write a separate one. Or maybe it's just a matter of packaging Creator's output pane into a standalone log viewer tool, and add more filtering features to both. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Thursday 10 July 2014 08:13:09 Koehne Kai wrote: -Original Message- From: development-bounces+kai.koehne=digia@qt-project.org [...] Here's what I propose: Heuristically determine at runtime. On all platforms (including Windows), use only the fact that a terminal is present. If a terminal is present, write to it using stderr. Otherwise, write to the system log. * Implications for Qt Creator: 1) it must be able to read from the system log for desktop debugging (especially when running applications without Run in terminal). That's already done on Windows. It will be necessary to add support for journald (Linux) and asl (OS X). Well, actually your latest patch that checks for stderr!=/dev/null will just work fine with current Qt Creator. That's for journal/Linux at least, haven't had a look yet how asl works. The latest patch does not match the proposal. I want to remove the stderr check for Unix and for Windows, which means Creator should check the system logs. Qt Creator needs to continue listening to stderr in any case: You still want to see stdout/stderr, after all. It's true that Qt Creator can in theory profit from additional information piped through systemd, but integrating both streams also has it's issues: E.g. the order of qDebug/stderr messages is no longer consistent, and interleaving lines might need special attention (though I guess we handle that already for stdout/stderr). Agreed it must continue to read. As for the synchronisation, journald gives you the message's absolute timestamp, so you'll just need to do the same for each line read via QProcess. 2) remote debugging: per platform, it needs to be decided whether we can access the log on the remote device. If so, that's the best situation. If it isn't, then we need to force Qt to log to stderr. 2a) set QT_LOGGING_TO_CONSOLE=1 in the environment 2b) force the allocation of a terminal (if using ssh, pass the -t option) See above. Unless stderr somehow ends up in the system log too we need to anyway have the terminal. Addressed above. I recommend advising Linux distributions to disable journald logging at least until Creator is updated. And the support for asl on OSX should be held until the same time. I agree (though I'm not sure whether Qt Creator is the blocker, or if it's enough to handle the issue in Qt. See above). Creator must be able to handle reading from the system log before logging to the system is enabled for that system. On Linux, the code is already there, so we have to tell distributions *not* to enable journald yet. For OS X, it's putting a hold on https://codereview.qt-project.org/82922. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Thursday 10 July 2014 11:05:03 Tobias Hunger wrote: Hi Thiago, Basically I agree with your statements, but I do not think we can rely on journald at this time. Agreed. The first problem is of course systemd itself: Ubuntu is one of the biggest distros out there and we can not reasonably assume that to be running systemd before 14.04 is at the end of its life-cycle. Considering that this is a long term support release that will be a while. But even when systemd is in use journald might be disabled or misconfigured -- as you already said yourself. I hope gnome and KDE starting to depend on systemd user sessions will get that sorted out soon, but at this time those changes are not yet deployed in distributions. I'm not asking that distributions turn journald on if they don't want to. Journald is optional. However, they *can't* turn it on until we figure out how to read back what gets logged there. At this point, the showstopper is in journald itself. So I think being able to rely on logging to journald is still a long time off. So what is the fallback? Stderr again? Yes. If you don't turn journald on (it's optional), then we always fall back to stderr. Adding Journald into the picture will also make retrieving warnings from remote machines a lot more difficult. Currently we rely on SSH and/or gdbserver to retrieve all application output on Unix systems. I am not sure at this time how that would work if the application starts sending output to the journal. So having a way to force stderr in favor of system logs would be greatly appreciated. It's there in two ways: 1) Option -t to ssh, which forces the allocation of a TTY and, thus, logging to stderr 2) The environment variable Note that it's also possible to query the system log remotely too. I believe that is done on Android, isn't it? From all I can find on the Internet, in order to enable capture of stderr, you need to have a rooted device. I meant that each plugin for remote systems needs to decide how it's going to approach this problem. If it can access the remote log, all the better. If it can't, the best way is to use ssh -t -- with the added benefit that it causes stdio to become line-buffered. If ssh is not getting used, make sure the environment variable is set. And there's another advantage: if Creator is able to query the system log, it is also able to print the output of an already-running application that you attached the debugger to. PS: Creator not supporting getting logs from system logs on Unix systems is not a bug, it is a feature nobody requested so far:-) Yup. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Thursday 10 July 2014 16:50:22 Shawn Rutledge wrote: What if we use qtlogging.ini (or equivalent on other systems) to control whether the logs go to the system log or not? I agree that every ordinary KDE user doesn't need to have unread logs taking up disk space, but maybe developers would like to turn on output to the journal permanently for after-the-fact diagnosis when something unexpectedly crashes or misbehaves. That doesn't solve the problem of the default. What do we do when qtlogging.ini isn't present? And how do you debug an application that got installed on a system that already has qtlogging.ini? On Windows I never liked CONFIG += console, because it's never in the .pro by default (usually not in manual tests or examples, and not in Creator-generated pro files either) and yet I would like to always have qDebug output in the console when the app is running in a console. So detecting whether the app is running in a console and automatically sending debug output there would mean that we no longer need that flag in the .pro, right? If it's technically possible, we should have done that years ago. I don't think it's possible at all. If the application was compiled to the GUI subsystem, it *has* no stdout and stderr. We can open a new console window, but I don't think you can write to an existing one. I also don't like Creator's default run in terminal checkbox because again I don't get to choose which terminal, and the one it chooses is always 80x25, white background, has no search feature, etc. So I have to remember to turn off the checkbox because the default is not what I want. IMO the Application Output pane should have enough features that it can be treated as the console in which the app is running. What is missing, that we don't already do this? Colour support, TTY support, job control, etc. TTYs on Unix are much more complex than pipes. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Thursday 10 July 2014 07:56:28 Koehne Kai wrote: Provide a define QT_LOG_TO_CONSOLE that let QCoreApplication.h record whether it should log to console (QT_LOG_TO_CONSOLE=1), to the system (QT_LOG_TO_CONSOLE=0), or both (QT_LOG_TO_CONSOLE=2). Qmake can wrap this into convenience CONFIG arguments: CONFIG+=log_to_console log_to_system It's easier to set an AA flag. http://qt-project.org/doc/qt-5/qcoreapplication.html#setAttribute http://qt-project.org/doc/qt-5/qt.html#ApplicationAttribute-enum PS: doc team, the second link above is unreadable. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Thu, Jul 10, 2014 at 10:17:52AM -0700, Thiago Macieira wrote: On Thursday 10 July 2014 16:50:22 Shawn Rutledge wrote: On Windows I never liked CONFIG += console, because it's never in the .pro by default (usually not in manual tests or examples, and not in Creator-generated pro files either) and yet I would like to always have qDebug output in the console when the app is running in a console. So detecting whether the app is running in a console and automatically sending debug output there would mean that we no longer need that flag in the .pro, right? If it's technically possible, we should have done that years ago. I don't think it's possible at all. If the application was compiled to the GUI subsystem, it *has* no stdout and stderr. We can open a new console window, but I don't think you can write to an existing one. windows is stupid, but it can't be that stupid, can it? = google = AttachConsole(ATTACH_PARENT_PROCESS) GetStdHandle(STD_ERROR_HANDLE) (and _open_osfhandle(h) if one cares for stdio) ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Thursday 10 July 2014 19:34:59 Oswald Buddenhagen wrote: On Thu, Jul 10, 2014 at 10:17:52AM -0700, Thiago Macieira wrote: On Thursday 10 July 2014 16:50:22 Shawn Rutledge wrote: On Windows I never liked CONFIG += console, because it's never in the .pro by default (usually not in manual tests or examples, and not in Creator-generated pro files either) and yet I would like to always have qDebug output in the console when the app is running in a console. So detecting whether the app is running in a console and automatically sending debug output there would mean that we no longer need that flag in the .pro, right? If it's technically possible, we should have done that years ago. I don't think it's possible at all. If the application was compiled to the GUI subsystem, it *has* no stdout and stderr. We can open a new console window, but I don't think you can write to an existing one. windows is stupid, but it can't be that stupid, can it? = google = AttachConsole(ATTACH_PARENT_PROCESS) GetStdHandle(STD_ERROR_HANDLE) (and _open_osfhandle(h) if one cares for stdio) That means we can replace the GetConsoleWindow() call with AttachConsole(ATTACH_PARENT_PROCESS). If it succeeds or if it returns ERROR_ACCESS_DENIED, we have a console. For the Windows folks around: should we do this? -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On 10 July 2014 19:17, Thiago Macieira thiago.macie...@intel.com wrote: On Thursday 10 July 2014 16:50:22 Shawn Rutledge wrote: What if we use qtlogging.ini (or equivalent on other systems) to control whether the logs go to the system log or not? I agree that every ordinary KDE user doesn't need to have unread logs taking up disk space, but maybe developers would like to turn on output to the journal permanently for after-the-fact diagnosis when something unexpectedly crashes or misbehaves. That doesn't solve the problem of the default. What do we do when qtlogging.ini isn't present? I don't think it would need to go to the journal by default. If users don't know how to turn it on, they probably don't need the feature either. We just need to make it easy to turn on. And how do you debug an application that got installed on a system that already has qtlogging.ini? Edit it and run the application again and try to reproduce the bug again? Being able to debug after something has already happened is a rare privilege anyway. If you expect problems you can turn on the logging in advance though. Or maybe it could be configured on a per-application basis in addition to the defaults. Maybe we invent a standardized easy way that an app can allow its own QSettings to control the logging, like have a standard [qtlogging] section. We could make it opt-in by requiring that you call a method on your QSettings object to enable the standard logging section, and after that, users will be able to edit the settings file and turn on logging categories just as they can in qtlogging.ini. Or it could be always-available if that doesn't bother anybody... On Windows I never liked CONFIG += console, because it's never in the .pro by default (usually not in manual tests or examples, and not in Creator-generated pro files either) and yet I would like to always have qDebug output in the console when the app is running in a console. So detecting whether the app is running in a console and automatically sending debug output there would mean that we no longer need that flag in the .pro, right? If it's technically possible, we should have done that years ago. I don't think it's possible at all. If the application was compiled to the GUI subsystem, it *has* no stdout and stderr. We can open a new console window, but I don't think you can write to an existing one. I thought so, but I'm glad Ossi found a possible strategy. ;-) I also don't like Creator's default run in terminal checkbox because again I don't get to choose which terminal, and the one it chooses is always 80x25, white background, has no search feature, etc. So I have to remember to turn off the checkbox because the default is not what I want. IMO the Application Output pane should have enough features that it can be treated as the console in which the app is running. What is missing, that we don't already do this? Colour support, TTY support, job control, etc. TTYs on Unix are much more complex than pipes. What are the use cases? Somehow I don't usually need more than what the Creator output pane already provides... except that categorized logging opens up a possibility to control what you see there. But even if we go that far, it's not so hard to find some code for a terminal that we could use (there's even a certain QtQuick one I happen to know of ;-) And some people would like having a terminal window in Creator anyway; I've seen it in other IDEs. But of course it would add more bloat. Anyway just being able to see and manage the logging output in Creator does not preclude running the application in a terminal if that's what you want to do, right? But IMO it should not be the default, and it wouldn't hurt if it was more configurable. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
El Wednesday 09 July 2014, Thiago Macieira escribió: === Log to both === Aside from the extra overhead, this causes systems that capture both stderr and the system log to record and display the same message twice. That's the source of task [3]. This is not an option. Why not? Task [3] is an issue in Qt Creator, isn't it? Why it needs to read the system log on Windows? Logging to the console _and_ journald has proven very useful to us. We did it specifically for an application using Qt's API, and developers can code, press Run (in Creator), and see the output in the same window with all the QT_MESSAGE_PATTERN goodness. Afterwards, if it's too noisy, or if you disabled some category, you can check the logs on journald (we logged all categories, even the disabled ones, to the journal, since we had or own message handler). It's not hard at all to do it on your application if you want to fine tune it. Actually, I never expected to see the journald logging in Qt, but having it IMHO it's a boon, since you can use the feature for 3rd party applications. Only downside is that you might need to collect what your application outputs to stderr from a non-Qt library. Then yes, you might get duplicated messages. === Heuristically determine at runtime where to send the log === On Unix, I have implemented the same checks. We can also use the fact of whether the main application is in debug mode or not. I see this as two orthogonal things. Why would one prefer the console on a debug build? You might need to put the program in debug mode on the target hardware (only device where you have the peripherals), and need the enriched system logging and the debug symbols. This is the most inconvenient change, since you might need to change the systemd unit file to set the QT_LOGGING_TO_CONSOLE environment variable to have the built in journald logging. == Summary of systems == System services on Linux with journald: - default stderr: captured into system log - system logging: available - is stderr useful: yes, for launching from terminal Note that you can teach systemd to ignore stderr/stdout if wanted. If the app is written with Qt, that can be a sane default. And then some desktop Linux distributions decided to enable journald. The effect of that is task [6]: Qt Creator shows no output because it goes to journald. But this comment is/was in the code: We use isatty to catch the obvious case of someone running something interactively. We also support environment variables for Qt Creator use (...) I thought that it was agreed/known to Qt Creator developers that QT_LOGGING_TO_CONSOLE might be gently defaulted to true for applications run by Creator. That environment variable seems like API to me, and a useful one that Creator could use. Of course they couldn't predict some distributions would pass -journald to configure. From the other messages: * Support for journald in the pre-built Qt binaries: Disabled. Probably will be that way for a long time. If Qt binaries are built with and old Ubuntu, they just don't have libsystemd-journal. Note: if journald is writing to volatile storage, regular users can't read the log. Therefore, system logging is not available. Linux distributions should not enable journald logging unless regular users can read the output. I think users can read their user journal, independently of the storage. This is as user nobody: $ systemd-journalctl Showing user generated messages only. Users in the group 'adm' can see all messages. Pass -q to turn this message off. -- Alex (a.k.a. suy) | GPG ID 0x0B8B0BC2 http://barnacity.net/ | http://disperso.net ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
El Thursday 10 July 2014, Koehne Kai escribió: Provide a define QT_LOG_TO_CONSOLE that let QCoreApplication.h record whether it should log to console (QT_LOG_TO_CONSOLE=1), to the system (QT_LOG_TO_CONSOLE=0), or both (QT_LOG_TO_CONSOLE=2). Would it be too problematic to make the environment variable (QT_LOGGING_TO_CONSOLE, which already is in use) accept also a 2 for logging to both sources? To me, the really interesting feature of providing such code inside of Qt is to enable it without having to touch legacy/third party code. Having it for your application is also nice, but you'll likely need to override it somehow to integrate it with your configuration or UI. -- Alex (a.k.a. suy) | GPG ID 0x0B8B0BC2 http://barnacity.net/ | http://disperso.net ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Friday 11 July 2014 00:27:51 Alejandro Exojo wrote: Would it be too problematic to make the environment variable (QT_LOGGING_TO_CONSOLE, which already is in use) accept also a 2 for logging to both sources? Problematic, no. It can be easily implemented. But it's an option we discarded. It makes no sense to log to both destinations. If you're using a tool that can read the syslog, then it doesn't matter which one it was sent to, but it will be a problem if you send to both. If you're using a tool that can't, then you want to send to stderr only. To me, the really interesting feature of providing such code inside of Qt is to enable it without having to touch legacy/third party code. Having it for your application is also nice, but you'll likely need to override it somehow to integrate it with your configuration or UI. If you have legacy, set to 1. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Friday 11 July 2014 00:17:48 Alejandro Exojo wrote: El Wednesday 09 July 2014, Thiago Macieira escribió: === Log to both === Aside from the extra overhead, this causes systems that capture both stderr and the system log to record and display the same message twice. That's the source of task [3]. This is not an option. Why not? Task [3] is an issue in Qt Creator, isn't it? Why it needs to read the system log on Windows? Because that's the way to do it on Windows. It's been established for 15 or 20 years that applications log via OutputDebugString. GUI applications can't write to the console unless they AllocConsole or attach to an existing one -- and attaching is only permitted since Windows XP. The big problem I found when implementing this is that systemd and launchd will capture your stdout and stderr into the system log. So if we log to both, then the message appears duplicated in the syslog for all applications launched by those tools. Logging to the console _and_ journald has proven very useful to us. We did it specifically for an application using Qt's API, and developers can code, press Run (in Creator), and see the output in the same window with all the QT_MESSAGE_PATTERN goodness. Afterwards, if it's too noisy, or if you disabled some category, you can check the logs on journald (we logged all categories, even the disabled ones, to the journal, since we had or own message handler). You can still install your handler to do that. We're talking about the default and I'd like to have a consistent behaviour across different OS. It makes my implementation simpler. Logging to both is a problem if you have something that is able to read from both. On Windows, all debuggers and IDEs read from both. Therefore, the option is out. Only downside is that you might need to collect what your application outputs to stderr from a non-Qt library. Then yes, you might get duplicated messages. There you go. That's the default on OS X and will probably become the default on Linux if the desktops switch to user-mode systemd. === Heuristically determine at runtime where to send the log === On Unix, I have implemented the same checks. We can also use the fact of whether the main application is in debug mode or not. I see this as two orthogonal things. Why would one prefer the console on a debug build? You might need to put the program in debug mode on the target hardware (only device where you have the peripherals), and need the enriched system logging and the debug symbols. I agree. Which is why my proposal email does not talk about debug mode. This is the most inconvenient change, since you might need to change the systemd unit file to set the QT_LOGGING_TO_CONSOLE environment variable to have the built in journald logging. If we did it like that and if you ran debug-mode applications in your system. That's not a common situation. And besides, systemd will capture stderr anyway, just will less information. I thought that it was agreed/known to Qt Creator developers that QT_LOGGING_TO_CONSOLE might be gently defaulted to true for applications run by Creator. It isn't. Creator devs are skeptical about modifying the environment by default. It's bad enough that it modifies PATH during the build process -- that means if you build with a Qt whose tools are in /usr/bin, you lose any overrides you placed in PATH (such as icecc). Note: if journald is writing to volatile storage, regular users can't read the log. Therefore, system logging is not available. Linux distributions should not enable journald logging unless regular users can read the output. I think users can read their user journal, independently of the storage. This is as user nobody: $ systemd-journalctl Showing user generated messages only. Users in the group 'adm' can see all messages. Pass -q to turn this message off. $ systemd-journalctl No journal files were found. Like I said, it depends on the journald configuration. Distributions should not enable journald unless they also configure journald in a way that users can read back what they've logged. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Thursday 10 July 2014 10:07:56 Thiago Macieira wrote: [snip] Creator must be able to handle reading from the system log before logging to the system is enabled for that system. On Linux, the code is already there, so we have to tell distributions *not* to enable journald yet. AFAIU from this thread, journald support should not be enabled except regular users can read the output. Now what I'm missing here is: if Qt is built with journald support, can it be still be used if journald is *not* present? Yes, I happen to be maintaining Qt for a distro in which users can still choose the init system. -- UNIX is basically a simple operating system, but you have to be a genius to understand the simplicity. Dennis Ritchie Lisandro Damián Nicanor Pérez Meyer http://perezmeyer.com.ar/ http://perezmeyer.blogspot.com/ signature.asc Description: This is a digitally signed message part. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
El Friday 11 July 2014, Lisandro Damián Nicanor Pérez Meyer escribió: AFAIU from this thread, journald support should not be enabled except regular users can read the output. Now what I'm missing here is: if Qt is built with journald support, can it be still be used if journald is not present? No. Quote from: http://www.freedesktop.org/software/systemd/man/sd_journal_send.html If systemd-journald(8) is not running (the socket is not present), those functions do nothing, and also return 0. Yes, I happen to be maintaining Qt for a distro in which users can still choose the init system. And unfortunately, there is no clean way to test for it. There is the sd_booted() function, but: Internally, this function checks whether the directory /run/systemd/system/ exists. A simple check like this can also be implemented trivially in shell or any other language. Which wasn't enough in my tests. Since I used systemd to boot the (developer) system, but I ran the app on Creator, it would return true all the time, and I got no output on Creator. It might be even worse if systemd-shim provides that directory. Is it worth to try sd_journal_open() and see if it fails? Or check some other way if systemd actually launched the application? Also, the way is done now (more after the patch on review) is to check for the console first, and then the decision is done. If the heuristics to find a console are good enough for you, having journald support is irrelevant, since if you are in a console you'll get the stderr printing. -- Alex (a.k.a. suy) | GPG ID 0x0B8B0BC2 http://barnacity.net/ | http://disperso.net ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Thursday 10 July 2014 23:53:20 Lisandro Damián Nicanor Pérez Meyer wrote: On Thursday 10 July 2014 10:07:56 Thiago Macieira wrote: [snip] Creator must be able to handle reading from the system log before logging to the system is enabled for that system. On Linux, the code is already there, so we have to tell distributions *not* to enable journald yet. AFAIU from this thread, journald support should not be enabled except regular users can read the output. Now what I'm missing here is: if Qt is built with journald support, can it be still be used if journald is *not* present? Yes, I happen to be maintaining Qt for a distro in which users can still choose the init system. Don't enable journald if your system doesn't use journald. As for having journald optionally in the distro, if we can detect at runtime and fallback if it's not running, we should implement it. If we can't detect, then don't enable the feature in your distro unless you're sure that journald will be running (excepted misconfiguration, of course). -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Friday 11 July 2014 07:45:23 Alejandro Exojo wrote: It might be even worse if systemd-shim provides that directory. Is it worth to try sd_journal_open() and see if it fails? Or check some other way if systemd actually launched the application? No and no. sd_journal_open might fail because the application can't open the log file, but logging to the journal is the intention. We're asking that distros not enable logging there if the user can't open the journal, but some embedded devices may want exactly that. So this option is out. Checking if the application was launched by systemd is also not an option IMHO because it would mean subprocesses wouldn't use journald, but instead go to stderr, which in turn makes that output be attributed to the parent process. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
[Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
== Problematic == Qt 5.3 and dev are currently able to send the output of qDebug, qWarning and the rest of the logging framework to somewhere other than stderr. That's done on Windows, BlackBerry, Android, and on Linux systems with journald. Additionally, there's a pending patch for OS X to use the Apple System Log [5] and I had a patch for MeeGo that made it go to the standard Unix syslog. We have a problem of wanting a cake and eating it too: 1) we want applications to send their logging to the system log where possible, which usually logs more information like the log level, time, application name and PID. Such a system is searchable and may allow us to add additional information, like the category (for category logging), file name, line number, function context, etc. 2) we also want to see the output when we run applications from the terminal or capture the output with QProcess. The absence of output causes bug reports like [2] and [4]. == Solutions == === Log to both === Aside from the extra overhead, this causes systems that capture both stderr and the system log to record and display the same message twice. That's the source of task [3]. This is not an option. === Log to stderr only === That results in loss of information, since the system logs are richer and more searchable. What's more, on Windows that would result in no output at all for GUI applications. I can't find a lot of information on whether GUI applications launched from an open command prompt would show anything. IIRC, they don't. This is not an option. === Log to system logs only === This causes reports like [4]. But note that the bug is actually in Creator, for failing to read the log store and display the data, like it does on Windows with the debug output. This is an option. === Heuristically determine at runtime where to send the log === Which is what I'm trying to do above. On Windows, we check whether we have a console and whether stderr is not NUL. On Unix, I have implemented the same checks. We can also use the fact of whether the main application is in debug mode or not. == Summary of systems == - default stderr: what is stderr connected to when launched from the GUI menu - system logging: if there's an API and whether non-privileged users can read from the log - is stderr useful: whether there are conditions under which writing to stderr is useful Android: - default stderr: /dev/null - system logging: available - is stderr useful: barely, it requires a rooted device BlackBerry: - default stderr: /dev/null - system logging: available - is stderr useful: I don't know, it's possible d5a4732c applies to non-BB QNX only OS X and iOS: - default stderr: captured into system log - system logging: available - is stderr useful: yes on OS X, running applications from the terminal SailfishOS: - default stderr: captured into the wrong system log - system logging: available - is stderr useful: yes, for remote debugging applications Current Linux desktops with journald: - default stderr: captured into ~/.xsession-errors - system logging available: available - is stderr useful: yes, for launching from terminal [when Linux desktops start using user-mode systemd, it will be as below] System services on Linux with journald: - default stderr: captured into system log - system logging: available - is stderr useful: yes, for launching from terminal Unix without journald: - default stderr: irrelevant - system logging: not available (non-root users may not be able to read it) - is stderr useful: yes, mandatory Windows GUI apps: - default stderr: NUL - system logging: available - is stderr useful: yes Windows console apps: - default stderr: console - system logging: available - is stderr useful: yes == History == In the dawn of time, qDebug printed to stderr. But then came Windows with GUI applications that don't have a console and stderr is NUL. So Qt applications that did not use CONFIG += console would use OutputDebugString, which we'd detect by the use or not of Qt's WinMain function. That was the first system log API. During Qt 5.0 development, we added logging to BlackBerry's slog2. That was unconditional: all applications, all output. Similarly for the Android port: when it was introduced, it unconditionally logged to the the Android log. But in Qt 5.1, there was a commit (289120f8) that added an environment variable allowing logging to stderr. During 5.2 development, we stopped using WinMain and instead tried to detect whether the application has a console window or not. The first two patch sets of 7fb3906 [1] only checked for the console, but PS3 added a check for whether stderr was valid even if no console window was present. For 5.3, we added support for journald on Linux, mostly intended for SailfishOS. Support was like on Android at the time: an environment variable was needed to write to stderr; otherwise, it went to journald. For 5.3.1,
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Wednesday 09 July 2014 14:43:36 Thiago Macieira wrote: === Log to system logs only === This causes reports like [4]. But note that the bug is actually in Creator, for failing to read the log store and display the data, like it does on Windows with the debug output. This is an option. === Heuristically determine at runtime where to send the log === Which is what I'm trying to do above. On Windows, we check whether we have a console and whether stderr is not NUL. On Unix, I have implemented the same checks. We can also use the fact of whether the main application is in debug mode or not. Here's what I propose: Heuristically determine at runtime. On all platforms (including Windows), use only the fact that a terminal is present. If a terminal is present, write to it using stderr. Otherwise, write to the system log. * Implications for Qt Creator: 1) it must be able to read from the system log for desktop debugging (especially when running applications without Run in terminal). That's already done on Windows. It will be necessary to add support for journald (Linux) and asl (OS X). 2) remote debugging: per platform, it needs to be decided whether we can access the log on the remote device. If so, that's the best situation. If it isn't, then we need to force Qt to log to stderr. 2a) set QT_LOGGING_TO_CONSOLE=1 in the environment 2b) force the allocation of a terminal (if using ssh, pass the -t option) I recommend advising Linux distributions to disable journald logging at least until Creator is updated. And the support for asl on OSX should be held until the same time. * Implications for unit tests: Except for the unit tests testing logging to the system log in the first place, QT_LOGGING_TO_CONSOLE=1 should be set. I think QtTest itself should do that. * Support for journald in the pre-built Creator binary: 1) new plugin, link to libsystemd-journal.so.0. If the library isn't present in the target system, the plugin won't load. 2) dlopen libsystemd-journal.so.0. If it isn't present, then Qt can't very well be logging to it. 3) use QProcess to run journalctl -f * Support for journald in the pre-built Qt binaries: Disabled. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Wednesday 09 July 2014 14:43:36 Thiago Macieira wrote: Current Linux desktops with journald: - default stderr: captured into ~/.xsession-errors - system logging available: available - is stderr useful: yes, for launching from terminal [when Linux desktops start using user-mode systemd, it will be as below] Note: if journald is writing to volatile storage, regular users can't read the log. Therefore, system logging is *not* available. Linux distributions should not enable journald logging unless regular users can read the output. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
I feel the urge to give my two cents on that. At least on unix desktop platforms, I expect to get stderr output on the console. So, detecting its presence is certainly a good approach. When there is no console, I probably don't care about the logs at all. I *may* care about error/warning messages, granted that the application manages its outputs diligently, which is often not the case. One argument is that the system logging offers better rotating search facilities, which is true, but the logs would still be cluttered by probably useless messages. There are tons of signal/slot missing throughout KDE applications, as a end user what can I possibly do about it ? I do not have a huge ~/.xsession-error file, but for those who have, I think it's a problem for distributions to solve, not Qt. On the other end, I usually have a large /home partition and relatively small root partition, so, of to evil, I would prefer my home to be cluttered. I think that's true for most Linux installations; Depending on its features or importance to the system or to the user an application is either important, in which case some log should be conserved or unimportant logs should certainly be dropped. A think an application should be assumed to be unimportant ( not to log to the system's journal) by default. The choice should be given both to the developer ( with an api) and maintainers/end user (environment variable) to modify this behavior. As for Qt Creator, my opinion is that the logs of an in-development application become irrelevant the second you press the rebuild button and so its output should always go to stderr only. For platforms which offer no or poor console support, not-in-development applications logs should arguably go to the system journal to be forever forgotten : I don't expect end-users to know how access those logs, even less to provide the developers with them when something goes wrong. Corentin 2014-07-10 1:20 GMT+02:00 Thiago Macieira thiago.macie...@intel.com: On Wednesday 09 July 2014 14:43:36 Thiago Macieira wrote: Current Linux desktops with journald: - default stderr: captured into ~/.xsession-errors - system logging available: available - is stderr useful: yes, for launching from terminal [when Linux desktops start using user-mode systemd, it will be as below] Note: if journald is writing to volatile storage, regular users can't read the log. Therefore, system logging is *not* available. Linux distributions should not enable journald logging unless regular users can read the output. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Cake and eating it too for qDebug printing to system log - https://codereview.qt-project.org/89357
On Thursday 10 July 2014 02:54:12 Corentin Jabot wrote: At least on unix desktop platforms, I expect to get stderr output on the console. So, detecting its presence is certainly a good approach. When there is no console, I probably don't care about the logs at all. I *may* care about error/warning messages, granted that the application manages its outputs diligently, which is often not the case. As we discussed on IRC: *if* you care about the warnings, then you want to find them. And reading ~/.xsession-errors is not acceptable. Mine has close to 1.2 million lines in 7 days of uptime (roughly 7000 lines per hour), in an output that doesn't log application name, PID, TID or function name. We can turn those on with the logging framework, but they're not easily searchable. One argument is that the system logging offers better rotating search facilities, which is true, but the logs would still be cluttered by probably useless messages. There are tons of signal/slot missing throughout KDE applications, as a end user what can I possibly do about it ? Hence the searchability. $ journalctl -l /home/thiago/obj/qt/qt5/qtbase/bin/qdbusviewer -- Logs begin at Wed 2014-07-09 16:14:48 PDT, end at Wed 2014-07-09 19:23:29 PDT. -- Jul 09 19:23:29 tjmaciei-mobl4 qdbusviewer[36992]: Connecting to deprecated signal QDBusConnectionInterface::serviceOwnerChanged(QString,QString,QString) $ journalctl -l QT_CATEGORY=default -- Logs begin at Wed 2014-07-09 16:14:48 PDT, end at Wed 2014-07-09 19:23:29 PDT. -- Jul 09 19:23:29 tjmaciei-mobl4 qdbusviewer[36992]: Connecting to deprecated signal QDBusConnectionInterface::serviceOwnerChanged(QString,QString,QString) I can even search for which Qt categories are getting logged: $ journalctl -F QT_CATEGORY default I do not have a huge ~/.xsession-error file, but for those who have, I think it's a problem for distributions to solve, not Qt. Agreed. But we have to integrate with the mechanisms that the distros provide. A think an application should be assumed to be unimportant ( not to log to the system's journal) by default. The choice should be given both to the developer ( with an api) and maintainers/end user (environment variable) to modify this behavior. I disagree. I think the application should log and then the log system can decide whether to keep it or throw it away. The one thing I'm missing here is a user-configurable log system. If I'm a developer of a Qt application and I'm using the system Qt, I want to get my logs and not clutter the system. Which is where the recommendation comes from: Linux distributions should not enable a logging system that regular users can't access. As for Qt Creator, my opinion is that the logs of an in-development application become irrelevant the second you press the rebuild button and so its output should always go to stderr only. Yes and no. Creator must show it, but I don't think it matters to the user where it got the output from. If Creator can read from the log, it can get more information than just plain pipes. It could, for example, allow you to jump to the source line that produced the line in question. Granted, that could be done with a proper message formatting too, but it would be harder because stdout and stderr are just plain buffers (they're not even line buffered), whereas the log would be properly packetised. For platforms which offer no or poor console support, not-in-development applications logs should arguably go to the system journal to be forever forgotten :I don't expect end-users to know how access those logs, even less to provide the developers with them when something goes wrong. That's the case on Windows: GUI apps have stderr connected to the blackhole and the OutputDebugString buffer is only 32k in size. Use it or lose it. But again: it can be used. Hence the suggestion to use the system log. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
