Re: [DNG] ..forensics on systemd or journald logs
On 22/11/17 17:35, Arnt Karlsen wrote: ..to reiterate: Is there a way to decode and read those binary systemd journal logs on classic POSIX/Unix etc forensic systems _not_ running systemd? Of course. Either install a tool that does it for you, i.e. journalctl, or write a tool to do it using the publicly available documentation. ..the "strings" approach suggested by John Hughes requires an intimate knowledge of systemd and might be relevant if the investigations were on "systemd sabotaging Devuan playing _new_ zero-day dirty tricks." Intimate knowledge? No, all it requires knowing is that most of the fields in a systemd journal are ascii keyword=value pairs. Tell you what, I'll see if I can write a little perl script to output a systemd journal in a format a little more pretty than strings(1) for you, give me a day, ok? ..so, the systemd crowd should have an interest in e.g. exposing "Devuan incompetence and paranoia" by coming up with an easy way to decode and read binary systemd journal logs without having to run systemd, to prove their case on "Devuan incompetence and paranoia on systemd", rather than confirm my current belief. incompetence is your word, not mine. Paranoia seems to fit some people. For example, what do you mean by "_new_ zero-day dirty tricks" above? ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ifconfig deprecated?
On Wed, Nov 22, 2017 at 09:25:18PM -0500, taii...@gmx.com wrote: > What is so much better about the ifconfig replacement ip? Why should I learn > how to use yet another tool that has no tangible benefit but is being > foisted on me? Because ifconfig is broken. It works only in simplest cases, and it can't even detect when it is wrong. From the technical side, net-tools use a different set of APIs than iproute (BSD ioctls vs netlink), the former are not extensible. Well, one could make a complete new set of ioctls, but that's quite pointless. Likewise, the user-facing format is different, as net-tools can't express the data: netstat: tcp6 0 0 2001:470:64f4::6:22 2001:470:64f4:0:edd:efff:fefc:2ad7:64167 ESTABLISHED 27287/sshd: kilobyt ss: tcp ESTAB 0 02001:470:64f4::6:22 2001:470:64f4:0:edd:efff:fefc:2ad7:64167 users:(("sshd",pid=27296,fd=3),("sshd",pid=27287,fd=3)) The former is incomplete: shows only one process that uses this socket, truncates fields, and fails to escape user-controlled strings it prints. Here's a car analogy: imagine a car that can go only over single-lane roads, and will crash if you try to drive over a multi-lane one. Such a car was adequate in 1920, or in a good part of today's Poland, but in a civilized country you won't make it father than a couple blocks. > Would I be correct in guessing it is made by red-hat? No. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢰⠒⠀⣿⡁ Imagine there are bandits in your house, your kid is bleeding out, ⢿⡄⠘⠷⠚⠋⠀ the house is on fire, and seven big-ass trumpets are playing in the ⠈⠳⣄ sky. Your cat demands food. The priority should be obvious... ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ifconfig deprecated?
Quoting taii...@gmx.com (taii...@gmx.com): > What is so much better about the ifconfig replacement ip? One, iproute2 is maintained. net-tools isn't. Unmaintained key system tools are a security and reliability risk that can IMO not be justified by merely not wanting to move on. IIRC, net-tools has been orphaned since 2003. Two, the iproute2 tools are lighter-weight than the net-tools ones, on account of using netlink instead of procfs and ioctl. Three, UI is more intuitive. (Yes, you say you are already used to what you're familiar with. I used to say this about nslookup.) The various iproute2 tools have a consistent syntax and are scripting-friendly. Four, iproute2's ip command can add/manage/display multiple IP addresses on a single interface. net-tools can only semi-approach this functionality via IP aliasing. The 'display' functionality of the net-tools toolkit is particularly broken in that department. 'ifconfig eth0' will show only one IP address. 'ip addr show dev eth0' will show all of them. Reason number 1 is sufficient, IMO. > Why should I learn how to use yet another tool that has no tangible > benefit but is being foisted on me? Why do you make this claim without bothering to do your homework? > Would I be correct in guessing it is made by red-hat? Current maintainer: Stephen Hemminger Original developer: Alexey Kuznetsov Hemminger was formerly at embedded networking firm Vyatta (previously InterTrust, Passedge, Informix, nCube, Sequent, and Tektronix), and is now Principal Software Engineer at Microsoft Corporation. Alexay is Chief Software Engineer at Parallels, Inc. Before that, he was at IceRock Development, Alawar Digital, Alawar Entertainment Inc., and my firm VA Linux System, Inc. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ifconfig deprecated?
What is so much better about the ifconfig replacement ip? Why should I learn how to use yet another tool that has no tangible benefit but is being foisted on me? Would I be correct in guessing it is made by red-hat? ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] rc.local removed from Debian 9, rly?
Quoting Steve Litt (sl...@troubleshooters.com): > Acronym for Apologist Troll. HANDY! (Acronym for 'Have A Nice Day, Y'all.') ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ..forensics on systemd or journald logs
On Wed, 22 Nov 2017 12:58:10 +, Arnt wrote in message <6ff3d9c1-e23c-4b0e-af51-5f8db1425...@gulbrandsen.priv.no>: > Arnt Karlsen writes: > > you appear to suggest that law enforcement wanting to read systemd > > journal logs, _should_ depend on the mercy of systemd developers > > not "filtering" away inconvenient evidence of e.g. systemd developer > > wrongdoing from said law enforcement. > > That's routine. Few readers read everything that can be read. For > example, look at postgres. Its binary file format reveals quite a bit > more than you can get using psql, and by design: The writer and > binary format are intended for storing things quickly and reliably, > and the reader for reading what was stored. Anything that's in the > file but wasn't stored by instruction of an SQL user is uninteresting > to psql, and the file format writer has no particular reason to avoid > storing other information. > > If you really want to look at the details in postgres, you can take a > good guess at whether two rows were inserted at the same time or one > later than the other. > > That's why forensics people use the files. Systemd is about the > millionth system to join the club. Flame postgres and vast numbers of > others before you flame systemd. Or better yet, limit your statements > about systemd to what's correct. > > Arnt ..it is very nice to learn I can read e.g. postgresql database files while boycotting e.g. postgresql, using strings and all sorts of fancy tricks to e.g. verify some postgresql developer's statement on systemd people playing nice or not. ..it would also be very nice to learn of a way to decode and read binary systemd journal logs without having to run systemd or without having to hire expensive expert witnesses to decode and read my own binary systemd journal logs from my final days on Debian Jessie. ..one very nice way of learning of a way to decode and read binary systemd journal logs without having to run systemd, is listening to wise answers from those who knows the correct truth about how to decode and read our own binary systemd journal log files. ;o) ..so, how about answering my question? Preferably correctly, if at all possible. If not, pointers to hearsay is useful to help try discover the (ugly?) truth. All I've seen this far, is confusion, deflection, trolling and diversion away from the context and my question. ..to reiterate: Is there a way to decode and read those binary systemd journal logs on classic POSIX/Unix etc forensic systems _not_ running systemd? ..e.g. using my namesake's example postgresql to translate the binary files into some human-readable format? ..the "strings" approach suggested by John Hughes requires an intimate knowledge of systemd and might be relevant if the investigations were on "systemd sabotaging Devuan playing _new_ zero-day dirty tricks." ..so, the systemd crowd should have an interest in e.g. exposing "Devuan incompetence and paranoia" by coming up with an easy way to decode and read binary systemd journal logs without having to run systemd, to prove their case on "Devuan incompetence and paranoia on systemd", rather than confirm my current belief. -- ..med vennlig hilsen = with Kind Regards from Arnt Karlsen ...with a number of polar bear hunters in his ancestry... Scenarios always come in sets of three: best case, worst case, and just in case. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] rc.local removed from Debian 9, rly?
On Wed, 22 Nov 2017 14:44:57 -0500, Steve wrote in message <20171122144457.02549...@mydesk.domain.cxm>: > On Wed, 22 Nov 2017 13:03:45 +0100 > Arnt Karlsenwrote: > > > On Wed, 22 Nov 2017 02:28:45 -0500, Steve wrote in message > > <20171122022845.1327c...@mydesk.domain.cxm>: > > > > > On Wed, 22 Nov 2017 02:59:11 +0100 > > > Arnt Karlsen wrote: > > > > > > > On Tue, 21 Nov 2017 18:21:14 +0100, John wrote in message > > > > : > > > > > > > > > (Damn but the systemd journal is great :-)) > > > > > > A T > > > > ..er, I _totally_ lost you here. > > Acronym for Apologist Troll. ..thanks, John and Rick stand corrected, their "A" suggestion was "a". ;o) > > SteveT > > Steve Litt > November 2017 featured book: Troubleshooting: Just the Facts > http://www.troubleshooters.com/tjust > ___ > Dng mailing list > Dng@lists.dyne.org > https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng -- ..med vennlig hilsen = with Kind Regards from Arnt Karlsen ...with a number of polar bear hunters in his ancestry... Scenarios always come in sets of three: best case, worst case, and just in case. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] rc.local removed from Debian 9, rly?
On Wed, 22 Nov 2017 13:02:37 +0100, John wrote in message <0788acc2-15f4-491f-61bf-d28664664...@atlantech.com>: > On 22/11/17 12:32, KatolaZ wrote: > > On Wed, Nov 22, 2017 at 12:24:28PM +0100, John Hughes wrote: > >> > >> I was amazed that KatolaZ couldn't imagine any way of reading text > >> from a file without a special application, doesn't he have > >> strings(1) on his "forensic system"? > > > > As for journalctl, you forget to mention that it is not available > > as a separate component from systemd. > > "Not available"? Attached to systemd with epoxy? Or an independent > executable that could easily be installed on a forensic system the > good old fashioned way. Or, if you prefer, just install the systemd > package and use some other init system: .._can_ we assume "systemd will never cover up crime"? ..e.g. Microsoft usually recommend against "3rd party software" "that may" (or not) "contain software virus", and on our side we have "root kits" written by very skillful people covering their tracks from their victims and law enforcement. > > I had never thhougt that I would have been suggested to look at logs > > by grepping the results of "strings" on a binary file. But I > > understand that this is considered "amazing technological progress" > > in some camps. > > Whatever gets the job done. Personally I'd just install the > application that knows how to read the file, but if I was unable to > do that for some reason or other I'd use one of the many useful tools > Unix like systems come with rather than claiming the job was > impossible. ..what if the job is "cover up crime in systemd journal logs"? -- ..med vennlig hilsen = with Kind Regards from Arnt Karlsen ...with a number of polar bear hunters in his ancestry... Scenarios always come in sets of three: best case, worst case, and just in case. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
[DNG] BookStack on Devuan 1.0.0 (Jessie)
For any that may be interested, I just submitted a pull request for an install script for BookStack on Devuan 1.0.0. Whether or not they accept it is another story, but you can clone my repo here: https://github.com/obeardly/devops.git or just grab the file here: https://github.com/obeardly/devops/tree/master/scripts If you're looking for a self-hosted Confluence alternative, I personally believe BookStack to be the best. At the moment, I believe BookStack is English only. I hope this is useful to someone besides myself. On a side note, I'm very close to completely replacing all Ubuntu, Debian, Fedora and CentOS systems at my new company. In another 6 weeks, with the exception of our virtual hosts, we will be all Devuan on both our virtual guests and bare metal. -- Linux O'Beardly @LinuxOBeardly http://o.beard.ly linux.obear...@gmail.com ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?
Le 22/11/2017 à 16:46, Arnt Gulbrandsen a écrit : Didier Kryn writes: Well, postgress is a database manager. You have a choice of several others; they must be able to deal with high fluxes of data. None of them is a critical system component. WTF? Postgres is a critical system component of every single server where I've ever installed that. The data in Postgres and the software that accesses it are the reason why the server exists at all. Good point, I tend to forget that there are special needs for heavy duty servers; but see below. System logs are a critical system component and they don't face high fluxes of data. You can, in principle, use syslog for applications with a high flux of logs, but it's at your own risk. Are you saying one should not use syslog for events caused by untrusted users? If the reason for having binary logs is performance, it means you are dealing with really massive logs. If untrusted users cause gigabytes of logs per day, you can either filter them online, which rsyslog can do pretty well, or try to use another method. I may have missed something, but I doubt anybody will ever read such massive logs. Also remember that syslog() uses one single socket and lock for all processes in the system, which means that emmiting a log message may imply waiting on a queue. If you write your own application, you can bypass the syslog() bottleneck in a number of ways. Didier ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] rc.local removed from Debian 9, rly?
On Wed, 22 Nov 2017 13:03:45 +0100 Arnt Karlsenwrote: > On Wed, 22 Nov 2017 02:28:45 -0500, Steve wrote in message > <20171122022845.1327c...@mydesk.domain.cxm>: > > > On Wed, 22 Nov 2017 02:59:11 +0100 > > Arnt Karlsen wrote: > > > > > On Tue, 21 Nov 2017 18:21:14 +0100, John wrote in message > > > : > > > > > > > (Damn but the systemd journal is great :-)) > > > > A T > > ..er, I _totally_ lost you here. Acronym for Apologist Troll. SteveT Steve Litt November 2017 featured book: Troubleshooting: Just the Facts http://www.troubleshooters.com/tjust ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] rc.local removed from Debian 9, rly?
Quoting Arnt Karlsen (a...@iaksess.no): > On Wed, 22 Nov 2017 02:28:45 -0500, Steve wrote in message > <20171122022845.1327c...@mydesk.domain.cxm>: > > A T > > ..er, I _totally_ lost you here. A vådeskudd? ("An > unintended discharge?", may happen if the wrong xterm > has keyboard focus when you type something...) ;o) I'll hazard a guess Steve was telegraphing 'a troll', though certainly 'a wet shot' på norsk would seem semi-appropriate, too, now that you mention it. -- Cheers, Luftputebåten min er full av ål. Rick Moen r...@linuxmafia.com McQ! (4x80) ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?
Didier Kryn writes: Well, postgress is a database manager. You have a choice of several others; they must be able to deal with high fluxes of data. None of them is a critical system component. WTF? Postgres is a critical system component of every single server where I've ever installed that. The data in Postgres and the software that accesses it are the reason why the server exists at all. System logs are a critical system component and they don't face high fluxes of data. You can, in principle, use syslog for applications with a high flux of logs, but it's at your own risk. Are you saying one should not use syslog for events caused by untrusted users? Arnt ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?
Le 22/11/2017 à 13:58, Arnt Gulbrandsen a écrit : If you really want to look at the details in postgres, you can take a good guess at whether two rows were inserted at the same time or one later than the other. Well, postgress is a database manager. You have a choice of several others; they must be able to deal with high fluxes of data. None of them is a critical system component. System logs are a critical system component and they don't face high fluxes of data. You can, in principle, use syslog for applications with a high flux of logs, but it's at your own risk. Didier ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?
On 2017-11-22 09:46 AM, Arnt Gulbrandsen wrote: Aldemir Akpinar writes: No, I've actually asked an honest question. In that case you'll get my honest answer. I've implemented several file/network formats vaguely like that journal format, one of them has likely been used by millions of people. In each case, the team decided to use a header/packet format, because that made both writing and reading simple. In the case of the network format we additionally included a magic number to catch version skew, and did it using binary because that made for the simplest reading code. Reading a 16-byte header using read(2) is simpler than reading a textual header. I don't remember anyone on either of the teams suggesting that using text had advantages for developers or users. Generally we just chose what was easier to ship reliably and parse/generate with simple code. In one case we used binary because even though the data were readable text, they weren't editable (the actual format had non-trivial restrictions). I don't remember the details, but for some reason we worried that people would hand-edit files and cause problems deep inside the reading program. I find it totally plausible that the systemd people would design the format for similar concerns and end up a format where a fixed-size header includes a tag type and length, then a variable-sized packets mostly containing log lines, and then another header. That kind of thing is so easy to read and write using https://linux.die.net/man/2/read and its companion functions. It is all a trade-off, making choices in implementation, and choice IS a good thing to have (Insert thumbs up to Devuan here), I don't think anybody is disputing that. This sub-thread mentions forensics, which to me means analysis of a system, a snapshot in time or worst case after complete failure. Due to the nature of things one is always going to be faced with binary files somewhere in the analysis that may or may not be easily decode-able or even recoverable. There is no question in my mind that a broken plain text file is far easier to traverse and sort out, which is really what the finger pointing game going on here is all about. Choice is good, having "no choice" forced upon you is bad, end of story. Clarke ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?
Aldemir Akpinar writes: No, I've actually asked an honest question. In that case you'll get my honest answer. I've implemented several file/network formats vaguely like that journal format, one of them has likely been used by millions of people. In each case, the team decided to use a header/packet format, because that made both writing and reading simple. In the case of the network format we additionally included a magic number to catch version skew, and did it using binary because that made for the simplest reading code. Reading a 16-byte header using read(2) is simpler than reading a textual header. I don't remember anyone on either of the teams suggesting that using text had advantages for developers or users. Generally we just chose what was easier to ship reliably and parse/generate with simple code. In one case we used binary because even though the data were readable text, they weren't editable (the actual format had non-trivial restrictions). I don't remember the details, but for some reason we worried that people would hand-edit files and cause problems deep inside the reading program. I find it totally plausible that the systemd people would design the format for similar concerns and end up a format where a fixed-size header includes a tag type and length, then a variable-sized packets mostly containing log lines, and then another header. That kind of thing is so easy to read and write using https://linux.die.net/man/2/read and its companion functions. Arnt ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?
On 22/11/17 15:08, Aldemir Akpinar wrote: On 22 November 2017 at 17:03, John Hughes> wrote: On 22/11/17 14:18, Aldemir Akpinar wrote: Could you elaborate why are you comparing a relational database system where its files must be binary with a logging system where its files doesn't need to binary? Need? Nothing "needs" to be in binary[*]. It's a design decision. Do the advantages of a structured format (mostly speed) override the disadvantages (higher costs for access if the reader software is unavailable? That's still not the answer to my question! There is no simple answer to your question because your question contains the logical fallacy of "begging the question", i.e. the question assumes its own answer. You said "why are you comparing a relational database system where its files must be binary with a logging system where its files doesn't need to binary?" That assumes that the files of a relational database system *must* be binary, which is simply untrue and that there is no advantage to making the files of a logging system binary which is debatable. The answer to your question is that there is no one single answer. The system designer will make his decisions about the different tradeoffs. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ifconfig deprecated?
Adam Borowski - 22.11.17, 11:06: > On Wed, Nov 22, 2017 at 10:45:12AM +0100, Martin Steigerwald wrote: > > For that I do not know a ip command out of the box. But > > > > merkaba:~> netstat -i > > Kernel-Schnittstellentabelle > > Iface MTURX-OK RX-ERR RX-DRP RX-OVRTX-OK TX-ERR TX-DRP TX-OVR > > Flg eth0 1500 6190114 0 0 0 2742172 0 0 > >0 BMRU > > lo 65536 334759 0 0 0334759 0 0 0 > > LRU > Except that netstat is another part of the sinking ship. Yeah, I just noticed that after sending this mail :) > The vast majority of netstat's functionality is in ss, usually even with > identical set of arguments, but as "netstat -i" shows information about > interfaces not sockets, it doesn't belong there. > > You want "ip -s a", I'm too lazy to see if you can get just the counters. That was it. I knew there was something like it. Thank you, -- Martin ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
[DNG] OT (..but relevant): reboot command on debian 8 no longer works
Just a heads up. Maybe this is old news to everyone but I run some Proxmox hosts which are based off Debian 8 (I believe) and we've had random problems with the reboot command hanging the system for a good 6 months now. Seems the simple 'reboot' command has been rendered useless. https://www.techrepublic.com/forums/discussions/debian-8-jessie-failing-to-reboot-from-the-command-line/ I still use 'reboot' on Devuan and things work fine. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?
On 22/11/17 14:22, Arnt Gulbrandsen wrote: Aldemir Akpinar writes: Could you elaborate why are you comparing a relational database system where its files must be binary with a logging system where its files doesn't need to binary? You make it sound is if binary files were some sort of horror that requires special justification. Please argue the point. Does a text format justify x% performance loss? y% increase in line count or code complexity? Pick x/y. Arnt ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng My understanding of why text files are better for important system logs is this:- When your server goes down big-style and you get all sorts of file corruption you stand a very good chance of working out what happened even if your text format log file is a bit mangled. If your binary format log file is mangled life is considerably more difficult - ask those that look after Microsoft Servers. I did, 'that's as bad as Windows!' he said. DaveT ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?
On 22 November 2017 at 17:22, Arnt Gulbrandsenwrote: > Aldemir Akpinar writes: > >> Could you elaborate why are you comparing a relational database system >> where its files must be binary with a logging system where its files >> doesn't need to binary? >> > > You make it sound is if binary files were some sort of horror that > requires special justification. Please argue the point. Does a text format > justify x% performance loss? y% increase in line count or code complexity? > Pick x/y. > > > Arnt > > > No, I've actually asked an honest question. I didn't imply anything at all. But all I get is trolling, not an answer to my Q :) Anyway, I'm done with these e-mails today. Doesn't help Devuan at all. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?
Aldemir Akpinar writes: Could you elaborate why are you comparing a relational database system where its files must be binary with a logging system where its files doesn't need to binary? You make it sound is if binary files were some sort of horror that requires special justification. Please argue the point. Does a text format justify x% performance loss? y% increase in line count or code complexity? Pick x/y. Arnt ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?
On 22 November 2017 at 17:03, John Hugheswrote: > On 22/11/17 14:18, Aldemir Akpinar wrote: > > > That's routine. Few readers read everything that can be read. For example, >> look at postgres. Its binary file format reveals quite a bit more than you >> can get using psql, and by design: The writer and binary format are >> intended for storing things quickly and reliably, and the reader for >> reading what was stored. Anything that's in the file but wasn't stored by >> instruction of an SQL user is uninteresting to psql, and the file format >> writer has no particular reason to avoid storing other information. >> >> >> >> > Could you elaborate why are you comparing a relational database system > where its files must be binary with a logging system where its files > doesn't need to binary? > > > Need? Nothing "needs" to be in binary[*]. It's a design decision. Do > the advantages of a structured format (mostly speed) override the > disadvantages (higher costs for access if the reader software is > unavailable? > > [*] or, to put it another way -- *everything on a computer is in binary*. > "Text" files are binary. The question is how easy is it to decode the file > format. It seems obvious that a "text" file is easy to decode, everyone > knows the format (but what character set is it in?), but don't forget that > the "text" file is stored on a filesystem, which is itself a complicated > "binary" structure. When you're talking about "forensics", i.e. looking at > something that may be broken in exciting ways, it's quite naïve to assume > that you can just mount the filesystem (which one?) and use cat, vi, grep > or whatever. > > That's still not the answer to my question! ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?
On 22/11/17 14:18, Aldemir Akpinar wrote: That's routine. Few readers read everything that can be read. For example, look at postgres. Its binary file format reveals quite a bit more than you can get using psql, and by design: The writer and binary format are intended for storing things quickly and reliably, and the reader for reading what was stored. Anything that's in the file but wasn't stored by instruction of an SQL user is uninteresting to psql, and the file format writer has no particular reason to avoid storing other information. Could you elaborate why are you comparing a relational database system where its files must be binary with a logging system where its files doesn't need to binary? Need? Nothing "needs" to be in binary[*]. It's a design decision. Do the advantages of a structured format (mostly speed) override the disadvantages (higher costs for access if the reader software is unavailable? [*] or, to put it another way -- *everything on a computer is in binary*. "Text" files are binary. The question is how easy is it to decode the file format. It seems obvious that a "text" file is easy to decode, everyone knows the format (but what character set is it in?), but don't forget that the "text" file is stored on a filesystem, which is itself a complicated "binary" structure. When you're talking about "forensics", i.e. looking at something that may be broken in exciting ways, it's quite naïve to assume that you can just mount the filesystem (which one?) and use cat, vi, grep or whatever. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?
> That's routine. Few readers read everything that can be read. For example, > look at postgres. Its binary file format reveals quite a bit more than you > can get using psql, and by design: The writer and binary format are > intended for storing things quickly and reliably, and the reader for > reading what was stored. Anything that's in the file but wasn't stored by > instruction of an SQL user is uninteresting to psql, and the file format > writer has no particular reason to avoid storing other information. > > If you really want to look at the details in postgres, you can take a good > guess at whether two rows were inserted at the same time or one later than > the other. > > That's why forensics people use the files. Systemd is about the millionth > system to join the club. Flame postgres and vast numbers of others before > you flame systemd. Or better yet, limit your statements about systemd to > what's correct. > > Arnt > Could you elaborate why are you comparing a relational database system where its files must be binary with a logging system where its files doesn't need to binary? -- aldemir ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?
Arnt Karlsen writes: you appear to suggest that law enforcement wanting to read systemd journal logs, _should_ depend on the mercy of systemd developers not "filtering" away inconvenient evidence of e.g. systemd developer wrongdoing from said law enforcement. That's routine. Few readers read everything that can be read. For example, look at postgres. Its binary file format reveals quite a bit more than you can get using psql, and by design: The writer and binary format are intended for storing things quickly and reliably, and the reader for reading what was stored. Anything that's in the file but wasn't stored by instruction of an SQL user is uninteresting to psql, and the file format writer has no particular reason to avoid storing other information. If you really want to look at the details in postgres, you can take a good guess at whether two rows were inserted at the same time or one later than the other. That's why forensics people use the files. Systemd is about the millionth system to join the club. Flame postgres and vast numbers of others before you flame systemd. Or better yet, limit your statements about systemd to what's correct. Arnt ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] rc.local removed from Debian 9, rly?
On Wed, Nov 22, 2017 at 07:50:44PM +0900, Olaf Meeuwissen wrote: > > Whether /etc/rc.local will be run (and on what run levels) is, IMHO, a > matter for *your* init system to decide. If your init system wants to > cater to a decades long tradition of running /etc/rc.local at system > startup, it should declare a dependency on initscripts or provide an > /etc/rc.local itself. > > If you want to use /etc/rc.local to tweak things, *you* should install > an init system that runs it (and Devuan's `init` package should list it > as a preferred alternative ;-) > > Any init systems that deviate from age old traditions, should, ideally, > clearly document that. If they don't, cluebat their maintainers ;-) > > And for the masses that don't know what /etc/rc.local is all about? > Well, they wouldn't know either way, so are pretty much unaffected by > all of this anyway. There's no need for an init package to impose an /etc/rc.local, except as a clue to the clueless. It's enough for it to execute it if it's there. -- hendrik ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] rc.local removed from Debian 9, rly?
On Wed, 22 Nov 2017 02:28:45 -0500, Steve wrote in message <20171122022845.1327c...@mydesk.domain.cxm>: > On Wed, 22 Nov 2017 02:59:11 +0100 > Arnt Karlsenwrote: > > > On Tue, 21 Nov 2017 18:21:14 +0100, John wrote in message > > : > > > > > (Damn but the systemd journal is great :-)) > > A T ..er, I _totally_ lost you here. A vådeskudd? ("An unintended discharge?", may happen if the wrong xterm has keyboard focus when you type something...) ;o) > SteveT > > Steve Litt > November 2017 featured book: Troubleshooting: Just the Facts > http://www.troubleshooters.com/tjust > ___ > Dng mailing list > Dng@lists.dyne.org > https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng -- ..med vennlig hilsen = with Kind Regards from Arnt Karlsen ...with a number of polar bear hunters in his ancestry... Scenarios always come in sets of three: best case, worst case, and just in case. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
[DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?
On Wed, 22 Nov 2017 07:19:20 +0100, John wrote in message: > On 22/11/17 02:59, Arnt Karlsen wrote: > > On Tue, 21 Nov 2017 18:21:14 +0100, John wrote in message > > : > > > >> (Damn but the systemd journal is great :-)) > > ..is there a way to decode and read those binary systemd journal > > logs on classic POSIX/Unix etc forensic systems _not_ running > > systemd? > > Is there any way to read a file in format X without a program that > reads format X? ..I'm asking you. Your other "answers" to this question suggests you may know the true answer to my question. > I suppose you could scatter iron filings on the disk the use a > scanning electron microscope to examine their positions and, using > paper, pencil and a copy of the systemd doc work out the contents by > hand. > > Or, being endowed with the minimum level of foresight necessary for > survival have a forensic system that includes tools for reading the > file formats you're likely to find on the system you want to > post-mortem. ..correct, that is precisely why I went for devuan and precisely why I ask you here now. ..you appear to suggest that law enforcement wanting to read systemd journal logs, _should_ depend on the mercy of systemd developers not "filtering" away inconvenient evidence of e.g. systemd developer wrongdoing from said law enforcement. ..depending on your jurisdiction, this feature of systemd is either a good thing or a bad thing, probably both and probably capable of facilitating the cover-up of organised crime, AFAICT. -- ..med vennlig hilsen = with Kind Regards from Arnt Karlsen ...with a number of polar bear hunters in his ancestry... Scenarios always come in sets of three: best case, worst case, and just in case. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] rc.local removed from Debian 9, rly?
On 22/11/17 12:32, KatolaZ wrote: On Wed, Nov 22, 2017 at 12:24:28PM +0100, John Hughes wrote: I was amazed that KatolaZ couldn't imagine any way of reading text from a file without a special application, doesn't he have strings(1) on his "forensic system"? As for journalctl, you forget to mention that it is not available as a separate component from systemd. "Not available"? Attached to systemd with epoxy? Or an independent executable that could easily be installed on a forensic system the good old fashioned way. Or, if you prefer, just install the systemd package and use some other init system: I had never thhougt that I would have been suggested to look at logs by grepping the results of "strings" on a binary file. But I understand that this is considered "amazing technological progress" in some camps. Whatever gets the job done. Personally I'd just install the application that knows how to read the file, but if I was unable to do that for some reason or other I'd use one of the many useful tools Unix like systems come with rather than claiming the job was impossible. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] rc.local removed from Debian 9, rly?
Hi all, I've read all the followup until 2017-11-22T10:21Z. I may follow up on selected posts, but I wanted to tackle this first. KatolaZ writes: > On Tue, Nov 21, 2017 at 04:05:47PM +0100, John Hughes wrote: >> On 21/11/17 15:53, KatolaZ wrote: >> >> >What matters is that we need to retain initscripts as "important". >> >> If you have sysvinit then it's a damn site more than "important", it's a >> dependency for sysvinit-core. > > I was not referring only to sysvinit. Since the expectation for any > "pluggable" init system is to not break anything that works (at least > in Devuan), this point must be taken into account by any candidate > alternative init system (at least in Devuan). Devuan has an `init` package that is Priority: required and Essential: yes (on Jessie) or Important:yes (on Ascii and Ceres). Trying to purge `init` will warn you sternly and require a magic incantation, something along the lines of WARNING: The following essential packages will be removed. This should NOT be done unless you know exactly what you are doing! init [...] You are about to do something potentially harmful. To continue type in the phrase 'Yes, do as I say!' The init package has a Pre-Depends: sysvinit-core | upstart. Both packages have a Depends: list that includes initscripts (without any alternatives for initscripts). Note that upstart is only available in Jessie (it's purely virtual on Ascii and Ceres). I have checked this on Jessie, Ascii and Ceres (using my Devuan Docker base images[1]). [1]: https://gitlab.com/paddy-hack/devuan/container_registry Given the above, I don't think there is not much need to make sure that the `initscripts` package is made Priority: important. On any of the Devuan versions it will be installed so /etc/rc.local will exist, be executable and run courtesy of sysvinit-core's or upstart's /sbin/init. Whether /etc/rc.local will be run (and on what run levels) is, IMHO, a matter for *your* init system to decide. If your init system wants to cater to a decades long tradition of running /etc/rc.local at system startup, it should declare a dependency on initscripts or provide an /etc/rc.local itself. If you want to use /etc/rc.local to tweak things, *you* should install an init system that runs it (and Devuan's `init` package should list it as a preferred alternative ;-) Any init systems that deviate from age old traditions, should, ideally, clearly document that. If they don't, cluebat their maintainers ;-) And for the masses that don't know what /etc/rc.local is all about? Well, they wouldn't know either way, so are pretty much unaffected by all of this anyway. Does that make sense? -- Olaf Meeuwissen, LPIC-2FSF Associate Member since 2004-01-27 GnuPG key: F84A2DD9/B3C0 2F47 EA19 64F4 9F13 F43E B8A4 A88A F84A 2DD9 Support Free Softwarehttps://my.fsf.org/donate Join the Free Software Foundation https://my.fsf.org/join ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] rc.local removed from Debian 9, rly?
On Wed, Nov 22, 2017 at 12:24:28PM +0100, John Hughes wrote: > On 22/11/17 11:42, Jaromil wrote: > >On Wed, 22 Nov 2017, John Hughes wrote: > > > >>No way to do that? Seriously? No way at all? > >jeez, is John a troll? > > My little joke about the usefulness of the systemd journal in diagnosing the > /etc/rc.local problem could conceivably be considered trolling. The > skirt-clutching replies to it could also be considered trolling. > > I was amazed that KatolaZ couldn't imagine any way of reading text from a > file without a special application, doesn't he have strings(1) on his > "forensic system"? John, you should possibly make an effort to quote people better, and to avoid mixing quotes from different sources, since this only causes confusion. As for journalctl, you forget to mention that it is not available as a separate component from systemd. But you seem used to show only the evidence that you like. I had never thhougt that I would have been suggested to look at logs by grepping the results of "strings" on a binary file. But I understand that this is considered "amazing technological progress" in some camps. Choices. HND KatolaZ P.S.: Go on with the flame. It's pointless to me. -- [ ~.,_ Enzo Nicosia aka KatolaZ - Devuan -- Freaknet Medialab ] [ "+. katolaz [at] freaknet.org --- katolaz [at] yahoo.it ] [ @) http://kalos.mine.nu --- Devuan GNU + Linux User ] [ @@) http://maths.qmul.ac.uk/~vnicosia -- GPG: 0B5F062F ] [ (@@@) Twitter: @KatolaZ - skype: katolaz -- github: KatolaZ ] signature.asc Description: Digital signature ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ifconfig deprecated?
Adam Borowskiwrote: >> merkaba:~> netstat -i > You want "ip -s a", I'm too lazy to see if you can get just the counters. Or use the contents of /proc/net/dev ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] rc.local removed from Debian 9, rly?
On 22/11/17 11:42, Jaromil wrote: On Wed, 22 Nov 2017, John Hughes wrote: No way to do that? Seriously? No way at all? jeez, is John a troll? My little joke about the usefulness of the systemd journal in diagnosing the /etc/rc.local problem could conceivably be considered trolling. The skirt-clutching replies to it could also be considered trolling. I was amazed that KatolaZ couldn't imagine any way of reading text from a file without a special application, doesn't he have strings(1) on his "forensic system"? it would explain his constant questions, keeping ignoring details that are already explicit in the thread and wasting our time. What explicit details have I ignored? As far as I can see you've never provided any link to the original complaints about /etc/rc.local not being run, just some ill-informed rubbish on stackoverflow(!) about things being "deprecated". You, personally, have repeatedly ignored information I provided, for example: You: 2017-11-21 14:20 +100 is it the case that one must run two systemctl commands in order for rc.local to be processed, or will rc.local just be found and executed? Me: 2017-11-21 14:48 +100, in reply: No, no systemctl commands are needed, systemd-rc-local-generator will enable rc-local.service if /etc/rc.local is executable, You, later in the same thread: 2017-11-21 15:19 +100 Then I believe we also agree that rc.local is a serious regressions? Me, in reply: What regression? You: 2017-11-21 16:18 +100 the fact that besides creating it and making it executable, one must also activate the service unit. But I'd already told you that "no systemctl commands are needed"! ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] rc.local removed from Debian 9, rly?
On Wed, 22 Nov 2017, John Hughes wrote: > > On 22/11/17 06:50, KatolaZ wrote: > > On Wed, Nov 22, 2017 at 02:59:11AM +0100, Arnt Karlsen wrote: > > > > > ..is there a way to decode and read those binary systemd journal logs > > > on classic POSIX/Unix etc forensic systems _not_ running systemd? > > > > > No, there is no way to do that, and we are probably spiralling down > > another flame... :( > > > > No way to do that? Seriously? No way at all? jeez, is John a troll? it would explain his constant questions, keeping ignoring details that are already explicit in the thread and wasting our time. also this reply is clearly aimed at triggering, it contains no information at all. ciao ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] rc.local removed from Debian 9, rly?
On 22/11/17 08:48, Didier Kryn wrote: Le 22/11/2017 à 07:19, John Hughes a écrit : Is there any way to read a file in format X without a program that reads format X? The question is why use yet another "proprietary format"? Just to force people to be use systemd for every task they need to do with their computer. Here we go again with the "assume bad faith". The systemd journal format is not a proprietary format. The systemd developers have said why they designed the format, but you think they have conspiratorial reasons for it. You are not even forced to use systemd to read journald log files, you can have journalctl on a system not running systemd. Hell, if you want you can just use strings(1). strings /run/log/journal/bea434ed778c45fca34c5986c88ac085/system.journal | grep MESSAGE= Personally I couldn't give a toss about the format, what's great about the journal is that it captures everything, especially things that would get written to the console and lost forever on a sysvinit based system. The original point was that the problem the bitcoin developers complained about should have been easy to diagnose if they were using systemd. Since they weren't then whatever went wrong with their /etc/rc.local just scrolled off into outer space like the beginning of a Star Wars film with nobody watching. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ifconfig deprecated?
On Wed, Nov 22, 2017 at 10:45:12AM +0100, Martin Steigerwald wrote: > For that I do not know a ip command out of the box. But > > merkaba:~> netstat -i > Kernel-Schnittstellentabelle > Iface MTURX-OK RX-ERR RX-DRP RX-OVRTX-OK TX-ERR TX-DRP TX-OVR Flg > eth0 1500 6190114 0 0 0 2742172 0 0 0 > BMRU > lo 65536 334759 0 0 0334759 0 0 0 LRU Except that netstat is another part of the sinking ship. The vast majority of netstat's functionality is in ss, usually even with identical set of arguments, but as "netstat -i" shows information about interfaces not sockets, it doesn't belong there. You want "ip -s a", I'm too lazy to see if you can get just the counters. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢰⠒⠀⣿⡁ Imagine there are bandits in your house, your kid is bleeding out, ⢿⡄⠘⠷⠚⠋⠀ the house is on fire, and seven big-ass trumpets are playing in the ⠈⠳⣄ sky. Your cat demands food. The priority should be obvious... ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ifconfig deprecated?
Hello Hendrik. Hendrik Boom - 21.11.17, 07:05: > On Tue, Nov 21, 2017 at 05:34:15PM -0500, Steve Litt wrote: > > On Mon, 20 Nov 2017 22:55:36 -0500 > > > > Hendrik Boomwrote: > > > On Tue, Nov 21, 2017 at 05:53:49PM +0100, Jaromil wrote: > > > > nono, as I wrote: that script doesn't works anymore, if ran on a > > > > freshly debootstrapped version of Debian 9. It seemed that rc.local > > > > wasn't executed anymore. But there is some confusion, since both > > > > brctl and ifconfig are legitimately deprecated. Assuming you have > > > > done better checking, then the failure may be caused by them > > > > bailing out. > > > > > > ifconfig is deprecated? What is to be used instead? > > > > A program called ip. It has commands such as ip route, ip addr, ip > > link, and several others. It's confusing and underdocumented, but so is > > ifconfig (which I never even began to master). > > I've never mastered ifconfig either. But I've noticed that the > simple command > >ifconfig > > tells me what interfaces are up at the moment and the packet counts > tell me whether they are actually functioning. > > It's a very simple way to diagnose the most common netweok > conectivity of all. For that I do not know a ip command out of the box. But merkaba:~> netstat -i Kernel-Schnittstellentabelle Iface MTURX-OK RX-ERR RX-DRP RX-OVRTX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 6190114 0 0 0 2742172 0 0 0 BMRU lo 65536 334759 0 0 0334759 0 0 0 LRU will also show it. I also recommend bmon for a more detailed analysis :) Thanks, -- Martin ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] ifconfig deprecated?
Hello Steve. Steve Litt - 21.11.17, 23:34: > On Mon, 20 Nov 2017 22:55:36 -0500 > > Hendrik Boomwrote: > > On Tue, Nov 21, 2017 at 05:53:49PM +0100, Jaromil wrote: > > > nono, as I wrote: that script doesn't works anymore, if ran on a > > > freshly debootstrapped version of Debian 9. It seemed that rc.local > > > wasn't executed anymore. But there is some confusion, since both > > > brctl and ifconfig are legitimately deprecated. Assuming you have > > > done better checking, then the failure may be caused by them > > > bailing out. > > > > ifconfig is deprecated? What is to be used instead? > > A program called ip. It has commands such as ip route, ip addr, ip > link, and several others. It's confusing and underdocumented, but so is > ifconfig (which I never even began to master). I do not find it underdocumented, considering: merkaba:~> apropos "^ip-" ip-address (8) - protocol address management ip-addrlabel (8) - protocol address label management ip-fou (8) - Foo-over-UDP receive port configuration ip-gue (8) - Generic UDP Encapsulation receive port configuration ip-l2tp (8) - L2TPv3 static unmanaged tunnel configuration ip-link (8) - network device configuration ip-macsec (8)- MACsec device configuration ip-maddress (8) - multicast addresses management ip-monitor (8) - state monitoring ip-mroute (8)- multicast routing cache management ip-neighbour (8) - neighbour/arp tables management. ip-netconf (8) - network configuration monitoring ip-netns (8) - process network namespace management ip-ntable (8)- neighbour table configuration ip-route (8) - routing table management ip-rule (8) - routing policy database management ip-tcp_metrics (8) - management for TCP Metrics ip-token (8) - tokenized interface identifier support ip-tunnel (8)- tunnel configuration ip-xfrm (8) - transform configuration Thanks, -- Martin ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng