Re: vanishing abrt logs
> I think abrt should figure out if the crashing binary is part of Would it be an option to create your own configuration for libreport? It is much simpler and robust to deliver a config tailored for your packages than trying to generalize the task and define a global policy Example: cat >/etc/libreport/events.d/my_system_service.conf < my_system_service.log EOF > Could we have a per-service switch that says Unfortunately, I am not sure we can say that a coredump file never contains any security sensitive data. For example, it contains a copy of /proc/[pid]/environ and if not cleared and an admin has a strange idea to export a crazy stuff, we are in trouble. Jakub -- Původní e-mail -- Od: Zbigniew Jędrzejewski-Szmek Komu: Development discussions related to Fedora Datum: 9. 4. 2019 9:04:48 Předmět: Re: vanishing abrt logs "On Mon, Apr 08, 2019 at 02:21:47PM +0200, jfi...@fedoraproject.org wrote: > I agree, but the file /proc/meminfo is not present, right? > And yes, the abrt thing just reads the data from journal and > > stores them on filesystem to be able to upload them to Bugzilla. > > > > > > Regarding the missing logs, the journal log lines should > > be extracted by this thing: > > https://github.com/abrt/abrt/blob/master/src/plugins/ccpp_event.conf#L25 > (https://github.com/abrt/abrt/blob/master/src/plugins/ccpp_event.conf#L25) > > which is a little bit complex, so no wonder if it is broken Thanks for the link. That code boils down to "journalctl -q -b --since=-3m --system -n 99 _COMM="$base_executable" and for systemd services this is very underwhelming, because it doesn't use the information that the process is part of service, and all logs from that service are relevant, e.g. when it launches a second executable or switches uid. I think abrt should figure out if the crashing binary is part of a system service, and switch to a log collection mode that is more like 'journalctl -u' in this case. Also, don't limit the messages to just 3 minutes or 99 messages. > (I am just not sure why the abrt test suite is not failing though). > > Regarding the core files, I was under the impression that > the core files can be attached only if the reporter wants to attach it. > IOW the core files were never attached automatically (due to security > issues). Could we have a per-service switch that says "this service never has private information, make the coredump file collection opt-out"? For example, we get a number of reports for systemd-logind, and by design it never has any private user data or keys. The user interface could be simplified. Similarly for hwrngd, and probably many others. > If you need some information that is relevant only to your packages, > we can work together to create a new abrt configuration which will > gather that information for your packages > (for example, dnf ships its own abrt configuration). Can you point me to it? rpm -ql $(rpm -qa|grep dnf)|grep abrt doesn't yield anything. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists. fedoraproject.org " ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: vanishing abrt logs
On Tue, 9 Apr 2019 at 08:58, Zbigniew Jędrzejewski-Szmek wrote: > > > If you need some information that is relevant only to your packages, > > we can work together to create a new abrt configuration which will > > gather that information for your packages > > (for example, dnf ships its own abrt configuration). > > Can you point me to it? rpm -ql $(rpm -qa|grep dnf)|grep abrt doesn't > yield anything. > You can find the DNF configuration for ABRT in /etc/libreport/events.d/collect_dnf.conf And more about event configurations here: https://abrt.readthedocs.io/en/latest/conf.html#event-configuration Martin ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: vanishing abrt logs
Dne 09. 04. 19 v 8:56 Zbigniew Jędrzejewski-Szmek napsal(a): > Can you point me to it? rpm -ql $(rpm -qa|grep dnf)|grep abrt doesn't > yield anything. What and how it is collected is determined by libreport with configs in /etc/libreport Some of those configs even have man page. You can start with `man report_event.conf`. Miroslav ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: vanishing abrt logs
On Mon, Apr 08, 2019 at 02:21:47PM +0200, jfi...@fedoraproject.org wrote: > I agree, but the file /proc/meminfo is not present, right? > And yes, the abrt thing just reads the data from journal and > > stores them on filesystem to be able to upload them to Bugzilla. > > > > > > Regarding the missing logs, the journal log lines should > > be extracted by this thing: > > https://github.com/abrt/abrt/blob/master/src/plugins/ccpp_event.conf#L25 > (https://github.com/abrt/abrt/blob/master/src/plugins/ccpp_event.conf#L25) > > which is a little bit complex, so no wonder if it is broken Thanks for the link. That code boils down to "journalctl -q -b --since=-3m --system -n 99 _COMM="$base_executable" and for systemd services this is very underwhelming, because it doesn't use the information that the process is part of service, and all logs from that service are relevant, e.g. when it launches a second executable or switches uid. I think abrt should figure out if the crashing binary is part of a system service, and switch to a log collection mode that is more like 'journalctl -u' in this case. Also, don't limit the messages to just 3 minutes or 99 messages. > (I am just not sure why the abrt test suite is not failing though). > > Regarding the core files, I was under the impression that > the core files can be attached only if the reporter wants to attach it. > IOW the core files were never attached automatically (due to security > issues). Could we have a per-service switch that says "this service never has private information, make the coredump file collection opt-out"? For example, we get a number of reports for systemd-logind, and by design it never has any private user data or keys. The user interface could be simplified. Similarly for hwrngd, and probably many others. > If you need some information that is relevant only to your packages, > we can work together to create a new abrt configuration which will > gather that information for your packages > (for example, dnf ships its own abrt configuration). Can you point me to it? rpm -ql $(rpm -qa|grep dnf)|grep abrt doesn't yield anything. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: vanishing abrt logs
Dne 08. 04. 19 v 13:18 jfi...@fedoraproject.org napsal(a): > Hi Kevin, > > I was under the impression that the problem with many emails from Bugzilla > has already been fixed: > https://bugzilla.redhat.com/show_bug.cgi?id=1660157 > https://github.com/abrt/libreport/commit/569bf0e3fed698e93b8e098bf6a0bb2f773aed6a > Yes, it was fixed in libreport-2.10.0. But if somebody report with older version, you will receive the Bugzilla spam. :( Miroslav ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: vanishing abrt logs
I agree, but the file /proc/meminfo is not present, right? And yes, the abrt thing just reads the data from journal and stores them on filesystem to be able to upload them to Bugzilla. Regarding the missing logs, the journal log lines should be extracted by this thing: https://github.com/abrt/abrt/blob/master/src/plugins/ccpp_event.conf#L25 (https://github.com/abrt/abrt/blob/master/src/plugins/ccpp_event.conf#L25) which is a little bit complex, so no wonder if it is broken (I am just not sure why the abrt test suite is not failing though). Regarding the core files, I was under the impression that the core files can be attached only if the reporter wants to attach it. IOW the core files were never attached automatically (due to security issues). If you need some information that is relevant only to your packages, we can work together to create a new abrt configuration which will gather that information for your packages (for example, dnf ships its own abrt configuration). Regards, Jakub -- Původní e-mail -- Od: Zbigniew Jędrzejewski-Szmek Komu: Development discussions related to Fedora Datum: 8. 4. 2019 13:49:41 Předmět: Re: vanishing abrt logs "On Mon, Apr 08, 2019 at 01:27:10PM +0200, jfi...@fedoraproject.org wrote: > Hi Zbyszek, > > > > If you want more files attached to ABRT Bugzilla reports, please add them to > systemd-coredump first :) > > https://github.com/systemd/systemd/blob/master/src/coredump/coredump.c#L 1074 > (https://github.com/systemd/systemd/blob/master/src/coredump/coredump.c#L 1074) In a sense, they already *are* attached to systemd-coredump. coredumpctl is basically just a journal query, and all the information I asked for is also present in the journal. abrt already extract *some* logs from the journal, so it's a question of adjusting the filters. The other thing which is missing from abrt reports is the corefile. In the past we used to have that attached. It would be great to have it again. (For example in case of memory structure errors, the backtrace itself is not particularly useful, but a lot could be diagnosed by examining the structures in memory. But I can't look at all reports quickly, and when I ask reporters if they have the core file, after a month or two the answer is usually "not anymore".). I *am* trying to solve the abrt reports that are files for my packages, but it's getting harder to do. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists. fedoraproject.org " ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: vanishing abrt logs
On Mon, Apr 08, 2019 at 01:27:10PM +0200, jfi...@fedoraproject.org wrote: > Hi Zbyszek, > > > > If you want more files attached to ABRT Bugzilla reports, please add them to > systemd-coredump first :) > > https://github.com/systemd/systemd/blob/master/src/coredump/coredump.c#L1074 > (https://github.com/systemd/systemd/blob/master/src/coredump/coredump.c#L1074) In a sense, they already *are* attached to systemd-coredump. coredumpctl is basically just a journal query, and all the information I asked for is also present in the journal. abrt already extract *some* logs from the journal, so it's a question of adjusting the filters. The other thing which is missing from abrt reports is the corefile. In the past we used to have that attached. It would be great to have it again. (For example in case of memory structure errors, the backtrace itself is not particularly useful, but a lot could be diagnosed by examining the structures in memory. But I can't look at all reports quickly, and when I ask reporters if they have the core file, after a month or two the answer is usually "not anymore".). I *am* trying to solve the abrt reports that are files for my packages, but it's getting harder to do. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: vanishing abrt logs
Hi Zbyszek, If you want more files attached to ABRT Bugzilla reports, please add them to systemd-coredump first :) https://github.com/systemd/systemd/blob/master/src/coredump/coredump.c#L1074 (https://github.com/systemd/systemd/blob/master/src/coredump/coredump.c#L1074) so ABRT folks can simply list them in their white-list: https://github.com/abrt/abrt/blob/master/src/plugins/abrt-dump-journal-core. c#L35 (https://github.com/abrt/abrt/blob/master/src/plugins/abrt-dump-journal-core.c#L35) Regards, Jakub PS: I forwarded your email to ABRT developers -- Původní e-mail -- Od: Zbigniew Jędrzejewski-Szmek Komu: Development discussions related to Fedora Datum: 7. 4. 2019 11:14:37 Předmět: vanishing abrt logs "Hi, I was looking at some abrt reports in the bugzilla [1,2,3], and I seems that the reports contain much less information than in the past. In particular, the "var_log_messages" attachment is always just a few useless lines that a) seems to be cut down just to messages *from* the service, and has no messages *about* the service b) the messages are only from a short time period before the crash When a service dies, systemd logs quite a bit of useful information [4] about the causes. We also have services like pcp, which log information about resource starvation [5]. This extra information would surely be helpful to diagnose abrt reports. Can we please make the reports contain useful logs again (at least full 'journalctl -u' for the service and all logs timed from around the crash) ? More information about the machine, for example /proc/meminfo would be great too. Zbyszek [1] https://bugzilla.redhat.com/show_bug.cgi?id=1660466 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1691889 [3] https://bugzilla.redhat.com/show_bug.cgi?id=1697032n [4] $ sudo systemd-run -p WatchdogSec=5 sleep 10 Running as run-re57f9e7da8234592bf74818522ad9644.service... $ journalctl --no-hostname -o short-precise -u run-re57f9e7da8234592bf 74818522ad9644.service -- Logs begin at Tue 2019-02-19 02:28:12 CET, end at Sun 2019-04-07 10:55:09 CEST. -- Apr 07 10:47:02.891163 systemd[1]: Started /usr/bin/sleep 100. Apr 07 10:47:08.071563 systemd[1]: run-re57f9e7da8234592bf74818522ad9644. service: Watchdog timeout (limit 5s)! Apr 07 10:47:08.072333 systemd[1]: run-re57f9e7da8234592bf74818522ad9644. service: Killing process 7704 (sleep) with signal SIGABRT. Apr 07 10:47:08.403477 systemd[1]: run-re57f9e7da8234592bf74818522ad9644. service: Main process exited, code=dumped, status=6/ABRT Apr 07 10:47:08.403608 systemd[1]: run-re57f9e7da8234592bf74818522ad9644. service: Failed with result 'watchdog'. Apr 07 10:47:08.403946 systemd[1]: run-re57f9e7da8234592bf74818522ad9644. service: Consumed 4ms CPU time. [5] pcp-pmie[3000]: Severe demand for real memory 11.2pgsout/s@krowka ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists. fedoraproject.org "___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: vanishing abrt logs
Hi Kevin, I was under the impression that the problem with many emails from Bugzilla has already been fixed: https://bugzilla.redhat.com/show_bug.cgi?id=1660157 (https://bugzilla.redhat.com/show_bug.cgi?id=1660157) https://github.com/abrt/libreport/commit/569bf0e3fed698e93b8e098bf6a0bb2f773 aed6a (https://github.com/abrt/libreport/commit/569bf0e3fed698e93b8e098bf6a0bb2f773aed6a) Regards, Jakub -- Původní e-mail -- Od: Kevin Fenzi Komu: devel@lists.fedoraproject.org Datum: 7. 4. 2019 20:19:40 Předmět: Re: vanishing abrt logs "On 4/7/19 1:59 AM, Zbigniew Jędrzejewski-Szmek wrote: > Hi, > > I was looking at some abrt reports in the bugzilla [1,2,3], and I seems that > the reports contain much less information than in the past. In particular, > the "var_log_messages" attachment is always just a few useless lines that > a) seems to be cut down just to messages *from* the service, and has no > messages *about* the service > b) the messages are only from a short time period before the crash > > When a service dies, systemd logs quite a bit of useful information [4] > about the causes. We also have services like pcp, which log information about > resource starvation [5]. > > This extra information would surely be helpful to diagnose abrt reports. > Can we please make the reports contain useful logs again (at least > full 'journalctl -u' for the service and all logs timed from around the crash) ? > > More information about the machine, for example /proc/meminfo would be > great too. I think: https://github.com/abrt/abrt/ is the correct place to file bugs/RFE's. I don't know if any of the abrt developers read the list or not. I really wish someone could fix libreport to only send 1 or 2 emails on new abrt bugzilla tickets again. It used to do this, but somehow broke with bugzilla5 upgrade, so now when someone files an abrt bug you get like 25 emails for it. I thought I filed a bug on this, but I can't seem to find it right now, will look more and refile if I can't find it. Also, someone noted the other day that abrt/libreport doesn't work with the new signin options on bugzilla5. You login with the redhat or fedora idps and it will ask you still for a username/password to file the bug. :( kevin ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists. fedoraproject.org " ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: vanishing abrt logs
On 4/7/19 1:59 AM, Zbigniew Jędrzejewski-Szmek wrote: > Hi, > > I was looking at some abrt reports in the bugzilla [1,2,3], and I seems that > the reports contain much less information than in the past. In particular, > the "var_log_messages" attachment is always just a few useless lines that > a) seems to be cut down just to messages *from* the service, and has no >messages *about* the service > b) the messages are only from a short time period before the crash > > When a service dies, systemd logs quite a bit of useful information [4] > about the causes. We also have services like pcp, which log information about > resource starvation [5]. > > This extra information would surely be helpful to diagnose abrt reports. > Can we please make the reports contain useful logs again (at least > full 'journalctl -u' for the service and all logs timed from around the > crash) ? > > More information about the machine, for example /proc/meminfo would be > great too. I think: https://github.com/abrt/abrt/ is the correct place to file bugs/RFE's. I don't know if any of the abrt developers read the list or not. I really wish someone could fix libreport to only send 1 or 2 emails on new abrt bugzilla tickets again. It used to do this, but somehow broke with bugzilla5 upgrade, so now when someone files an abrt bug you get like 25 emails for it. I thought I filed a bug on this, but I can't seem to find it right now, will look more and refile if I can't find it. Also, someone noted the other day that abrt/libreport doesn't work with the new signin options on bugzilla5. You login with the redhat or fedora idps and it will ask you still for a username/password to file the bug. :( kevin signature.asc Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
vanishing abrt logs
Hi, I was looking at some abrt reports in the bugzilla [1,2,3], and I seems that the reports contain much less information than in the past. In particular, the "var_log_messages" attachment is always just a few useless lines that a) seems to be cut down just to messages *from* the service, and has no messages *about* the service b) the messages are only from a short time period before the crash When a service dies, systemd logs quite a bit of useful information [4] about the causes. We also have services like pcp, which log information about resource starvation [5]. This extra information would surely be helpful to diagnose abrt reports. Can we please make the reports contain useful logs again (at least full 'journalctl -u' for the service and all logs timed from around the crash) ? More information about the machine, for example /proc/meminfo would be great too. Zbyszek [1] https://bugzilla.redhat.com/show_bug.cgi?id=1660466 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1691889 [3] https://bugzilla.redhat.com/show_bug.cgi?id=1697032n [4] $ sudo systemd-run -p WatchdogSec=5 sleep 10 Running as run-re57f9e7da8234592bf74818522ad9644.service... $ journalctl --no-hostname -o short-precise -u run-re57f9e7da8234592bf74818522ad9644.service -- Logs begin at Tue 2019-02-19 02:28:12 CET, end at Sun 2019-04-07 10:55:09 CEST. -- Apr 07 10:47:02.891163 systemd[1]: Started /usr/bin/sleep 100. Apr 07 10:47:08.071563 systemd[1]: run-re57f9e7da8234592bf74818522ad9644.service: Watchdog timeout (limit 5s)! Apr 07 10:47:08.072333 systemd[1]: run-re57f9e7da8234592bf74818522ad9644.service: Killing process 7704 (sleep) with signal SIGABRT. Apr 07 10:47:08.403477 systemd[1]: run-re57f9e7da8234592bf74818522ad9644.service: Main process exited, code=dumped, status=6/ABRT Apr 07 10:47:08.403608 systemd[1]: run-re57f9e7da8234592bf74818522ad9644.service: Failed with result 'watchdog'. Apr 07 10:47:08.403946 systemd[1]: run-re57f9e7da8234592bf74818522ad9644.service: Consumed 4ms CPU time. [5] pcp-pmie[3000]: Severe demand for real memory 11.2pgsout/s@krowka ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
