Re: [systemd-devel] Failed to get journal fields: Bad message
Lennart, On Mon, 2017-02-27 at 19:42 +0100, Lennart Poettering wrote: > On Mon, 27.02.17 08:29, Krzysztof Błaszkowski (k...@sysmikro.com.pl) > wrote: > > > > > > > > > The journal file format is primarily an append-based format > > > (though > > > some fields at the front are updated, to link the new additions > > > up). This makes it not too bad when it comes to disk corruptions, > > > and > > > data up to the point of corruption should be readable pretty > > > nicely > > > still. > > > > well, the most interesting information is located beyond that point > > very often. > > Well, note that journald will stop writing to journal files as soon > as > it notices a corruption of any kind, and will start new journal > files, > leaving the old ones the way they are (thus not making them worse). Is there a way to access the data collected beyond the point of corruption ? Notice that I lost logs for a month time because of system freeze however my ext4 recovered and is clean but *-journal. > We > also start new journal files in regular intervals, after a fixed time > interval or after a certain number of bytes wirtten, for whatever > comes first. splendid. this is like making poor candies more sweet. > > With that in place I am pretty sure our behaviour when it comes to > real-life corruptions these days isn't that bad: when something bad > happens, then we'll conserve in time what has been written already, > and if you destroy one file entirely what you loose will be bounded > both in time and in size. > > But anyway, I get the impression that you don't like the journal out > of principle, and I am not sure I want to try to convince you > otherwise. right, it will not convince me because we differ in very basic points of views. You agree for log losses "to some degree" (undefined) while I reckon this is not acceptable utterly. And because I think that I'm absolutely right that log losses are not acceptable then here is logical conclusion: a syslogd replacement must do exactly what syslogd used to do because if there are only milisecs to kernel failure left then there is no time for various funny "database doings" like hashing, indexing rows, compression - all that is rubbish. Logs must be appended only to current log file all the time as fast as syslogd or its replacement can do. Let filesystem do these things. and many don't need one "sweety tool for dumbs" because they are familiar with cat and grep. These various http crap admin idiots better educate themself. > My recommendation would be: if you don't like it don't use > it, set Storage=no in journald.conf and use whatever you like > better. You'll still benefit from journald and systemd collecting > all service stdout/stderr and early boot stuff for you, over plain > sysvinit/syslog, but it won't store anything on disk for you anymore. > ok. will give a try. Collected new battery yesterday and troubles gone. > > > > > > > > Anyway, if you have a corrupted journal file and current > > > journalctl > > > can't read it, then please file a bug, attach the journal file, > > > and > > > we'll look into improving journalctl. > > > > > > > journalctl --verify > > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user- > > 1105.journal > > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005 > > 495e > > 2bb26863-aa72c6a8a7437123.journal~ > > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005 > > 455b > > e5e6181f-8616123458bd2b50.journal~ > > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005 > > 4956 > > b0c61ef7-6f0ef8bfd59ad5e8.journal~ > > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1000@0 > > 0054 > > 95e2c558675-4af46c2d79464765.journal~ > > 549f50: Invalid entry item (7/9 offset: > > 00 > > 549f50: Invalid object contents: Bad > > message > > File corruption detected at > > /var/log/journal/52f04da61fa8108cd5f48bca58 > > 6164a2/system@0005495abfa1c9d8-cf2df0a23044.journal~:549f50 (of > > 8388608 bytes, 66%). > > FAIL: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005 > > 495a > > bfa1c9d8-cf2df0a23044.journal~ (Bad message) > > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1000@0 > > 0054 > > 55bfdbeb23d-1ff3046498e945ad.journal~ > > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005 > > 495e > > 3d56c587-44f1967167aac5a3.journal~ > > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@65be > > e6e3 > > 4f3c4b77b265431a1dd2a6df-0001-0005459b8fd2a11e.journal > > c84f48: Invalid entry item (8/24 offset: > > 00 > > c84f48: Invalid object contents: Bad > > message > > File corruption detected at > > /var/log/journal/52f04da61fa8108cd5f48bca58 > >
Re: [systemd-devel] Failed to get journal fields: Bad message
On Mon, 27.02.17 08:29, Krzysztof Błaszkowski (k...@sysmikro.com.pl) wrote: > > The journal file format is primarily an append-based format (though > > some fields at the front are updated, to link the new additions > > up). This makes it not too bad when it comes to disk corruptions, and > > data up to the point of corruption should be readable pretty nicely > > still. > > well, the most interesting information is located beyond that point > very often. Well, note that journald will stop writing to journal files as soon as it notices a corruption of any kind, and will start new journal files, leaving the old ones the way they are (thus not making them worse). We also start new journal files in regular intervals, after a fixed time interval or after a certain number of bytes wirtten, for whatever comes first. With that in place I am pretty sure our behaviour when it comes to real-life corruptions these days isn't that bad: when something bad happens, then we'll conserve in time what has been written already, and if you destroy one file entirely what you loose will be bounded both in time and in size. But anyway, I get the impression that you don't like the journal out of principle, and I am not sure I want to try to convince you otherwise. My recommendation would be: if you don't like it don't use it, set Storage=no in journald.conf and use whatever you like better. You'll still benefit from journald and systemd collecting all service stdout/stderr and early boot stuff for you, over plain sysvinit/syslog, but it won't store anything on disk for you anymore. > > Anyway, if you have a corrupted journal file and current journalctl > > can't read it, then please file a bug, attach the journal file, and > > we'll look into improving journalctl. > > > > journalctl --verify > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user- > 1105.journal > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005495e > 2bb26863-aa72c6a8a7437123.journal~ > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005455b > e5e6181f-8616123458bd2b50.journal~ > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@00054956 > b0c61ef7-6f0ef8bfd59ad5e8.journal~ > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1000@00054 > 95e2c558675-4af46c2d79464765.journal~ > 549f50: Invalid entry item (7/9 offset: > 00 > 549f50: Invalid object contents: Bad > message > File corruption detected at /var/log/journal/52f04da61fa8108cd5f48bca58 > 6164a2/system@0005495abfa1c9d8-cf2df0a23044.journal~:549f50 (of > 8388608 bytes, 66%). > FAIL: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005495a > bfa1c9d8-cf2df0a23044.journal~ (Bad message) > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1000@00054 > 55bfdbeb23d-1ff3046498e945ad.journal~ > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005495e > 3d56c587-44f1967167aac5a3.journal~ > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@65bee6e3 > 4f3c4b77b265431a1dd2a6df-0001-0005459b8fd2a11e.journal > c84f48: Invalid entry item (8/24 offset: > 00 > c84f48: Invalid object contents: Bad > message > File corruption detected at /var/log/journal/52f04da61fa8108cd5f48bca58 > 6164a2/system@0005459b8ff5e32b-620437abf7009d7d.journal~:c84f48 (of > 16777216 bytes, 78%). > FAIL: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005459b > 8ff5e32b-620437abf7009d7d.journal~ (Bad message) > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user- > 1000.journal > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1000@00054 > 95e3de63208-bbc6d86046b94014.journal~ > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1105@2a1f1 > b3f5241431483971743eded852c-d6d6-000547f91700222b.journal > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1000@00054 > 59b907b630d-a25c081908a032be.journal~ > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005495e > 8db292fb-cb1d4cbe87b5093f.journal~ > PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1001@1dd4c > 9dcd66446b483e91ea564f6f1b6-43c6-0005478f1ea62cb4.journal > PASS: > /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system.journal > > 3a86e0: Invalid > object > > File corruption detected at /var/log/journal/52f04da61fa8108cd5f48bca58 > 6164a2/user-1000@0005495ac03d6f02-bc5bf9a6b1d16fec.journal~:3a86e0 (of > 8388608 bytes, 45%). > FAIL: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1000@00054 > 95ac03d6f02-bc5bf9a6b1d16fec.journal~ (Bad message) > PASS:
Re: [systemd-devel] Failed to get journal fields: Bad message
On Sun, 2017-02-26 at 21:45 +0100, Lennart Poettering wrote: > On Sat, 25.02.17 11:10, Krzysztof Błaszkowski (k...@sysmikro.com.pl) > wrote: > > > > > > > Any thoughts ? wise only .. > > > > when there was /var/log/messages available there was no problem > > with > > accessing logs because the "database" was plain text but now. > > > > corrupted less or more it did not matter. it is responsibility of > > /var/log filesystem to perform right recovery .. > > > > I want to see what could happen before I had to hard reboot and I > > can't > > access "messages" even before a month ago. > > > > > > don't want to see reply like this: > > http://forums.fedoraforum.org/showthread.php?t=311314 > > > > because I will express how stupid *-journal idea is. > > and making double recovery scheme by the file system and poor one > > by *- > > journal is so brain fucked like I have never seen yet. > > The journal file format is primarily an append-based format (though > some fields at the front are updated, to link the new additions > up). This makes it not too bad when it comes to disk corruptions, and > data up to the point of corruption should be readable pretty nicely > still. > > well, the most interesting information is located beyond that point very often. > Anyway, if you have a corrupted journal file and current journalctl > can't read it, then please file a bug, attach the journal file, and > we'll look into improving journalctl. > journalctl --verify PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user- 1105.journal PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005495e 2bb26863-aa72c6a8a7437123.journal~ PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005455b e5e6181f-8616123458bd2b50.journal~ PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@00054956 b0c61ef7-6f0ef8bfd59ad5e8.journal~ PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1000@00054 95e2c558675-4af46c2d79464765.journal~ 549f50: Invalid entry item (7/9 offset: 00 549f50: Invalid object contents: Bad message File corruption detected at /var/log/journal/52f04da61fa8108cd5f48bca58 6164a2/system@0005495abfa1c9d8-cf2df0a23044.journal~:549f50 (of 8388608 bytes, 66%). FAIL: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005495a bfa1c9d8-cf2df0a23044.journal~ (Bad message) PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1000@00054 55bfdbeb23d-1ff3046498e945ad.journal~ PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005495e 3d56c587-44f1967167aac5a3.journal~ PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@65bee6e3 4f3c4b77b265431a1dd2a6df-0001-0005459b8fd2a11e.journal c84f48: Invalid entry item (8/24 offset: 00 c84f48: Invalid object contents: Bad message File corruption detected at /var/log/journal/52f04da61fa8108cd5f48bca58 6164a2/system@0005459b8ff5e32b-620437abf7009d7d.journal~:c84f48 (of 16777216 bytes, 78%). FAIL: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005459b 8ff5e32b-620437abf7009d7d.journal~ (Bad message) PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user- 1000.journal PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1000@00054 95e3de63208-bbc6d86046b94014.journal~ PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1105@2a1f1 b3f5241431483971743eded852c-d6d6-000547f91700222b.journal PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1000@00054 59b907b630d-a25c081908a032be.journal~ PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005495e 8db292fb-cb1d4cbe87b5093f.journal~ PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1001@1dd4c 9dcd66446b483e91ea564f6f1b6-43c6-0005478f1ea62cb4.journal PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system.journal 3a86e0: Invalid object File corruption detected at /var/log/journal/52f04da61fa8108cd5f48bca58 6164a2/user-1000@0005495ac03d6f02-bc5bf9a6b1d16fec.journal~:3a86e0 (of 8388608 bytes, 45%). FAIL: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1000@00054 95ac03d6f02-bc5bf9a6b1d16fec.journal~ (Bad message) PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/system@0005455b eea9f506-e4b5738d99c3f3ab.journal~ PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user- 1020.journal PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user- 1005.journal PASS: /var/log/journal/52f04da61fa8108cd5f48bca586164a2/user-1000@8d932 e8a67a74b8483761b6d509bd81f-05e0-0005459b907b791b.journal PASS:
Re: [systemd-devel] Failed to get journal fields: Bad message
On Sat, 25.02.17 11:10, Krzysztof Błaszkowski (k...@sysmikro.com.pl) wrote: > > Any thoughts ? wise only .. > > when there was /var/log/messages available there was no problem with > accessing logs because the "database" was plain text but now. > > corrupted less or more it did not matter. it is responsibility of > /var/log filesystem to perform right recovery .. > > I want to see what could happen before I had to hard reboot and I can't > access "messages" even before a month ago. > > > don't want to see reply like this: > http://forums.fedoraforum.org/showthread.php?t=311314 > > because I will express how stupid *-journal idea is. > and making double recovery scheme by the file system and poor one by *- > journal is so brain fucked like I have never seen yet. The journal file format is primarily an append-based format (though some fields at the front are updated, to link the new additions up). This makes it not too bad when it comes to disk corruptions, and data up to the point of corruption should be readable pretty nicely still. Anyway, if you have a corrupted journal file and current journalctl can't read it, then please file a bug, attach the journal file, and we'll look into improving journalctl. If you think that our ideas, and in particular journal files are stupid, then please use something else, there are plenty syslog implementations around, and they still work on systemd. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Failed to get journal fields: Bad message
Any thoughts ? wise only .. when there was /var/log/messages available there was no problem with accessing logs because the "database" was plain text but now. corrupted less or more it did not matter. it is responsibility of /var/log filesystem to perform right recovery .. I want to see what could happen before I had to hard reboot and I can't access "messages" even before a month ago. don't want to see reply like this: http://forums.fedoraforum.org/showthread.php?t=311314 because I will express how stupid *-journal idea is. and making double recovery scheme by the file system and poor one by *- journal is so brain fucked like I have never seen yet. -- Krzysztof Blaszkowski ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
