Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?

[Didier Kryn]

Le 22/11/2017 à 16:46, Arnt Gulbrandsen a écrit :

Didier Kryn writes:
    Well, postgress is a database manager. You have a choice of 
several others; they must be able to deal with high fluxes of data. 
None of them is a critical system component.


WTF? Postgres is a critical system component of every single server 
where I've ever installed that. The data in Postgres and the software 
that accesses it are the reason why the server exists at all.


    Good point, I tend to forget that there are special needs for heavy 
duty servers; but see below.
    System logs are a critical system component and they don't face 
high fluxes of data. You can, in principle, use syslog for 
applications with a high flux of logs, but it's at your own risk.


Are you saying one should not use syslog for events caused by 
untrusted users?


    If the reason for having binary logs is performance, it means you 
are dealing with really massive logs. If untrusted users cause gigabytes 
of logs per day, you can either filter them online, which rsyslog can do 
pretty well, or try to use another method. I may have missed something, 
but I doubt anybody will ever read such massive logs. Also remember that 
syslog() uses one single socket and lock for all processes in the 
system, which means that emmiting a log message may imply waiting on a 
queue. If you write your own application, you can bypass the syslog() 
bottleneck in a number of ways.


    Didier

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?

[Arnt Gulbrandsen]

Didier Kryn writes:
Well, postgress is a database manager. You have a choice of 
several others; they must be able to deal with high fluxes of 
data. None of them is a critical system component.


WTF? Postgres is a critical system component of every single server where 
I've ever installed that. The data in Postgres and the software that 
accesses it are the reason why the server exists at all.


System logs are a critical system component and they don't 
face high fluxes of data. You can, in principle, use syslog for 
applications with a high flux of logs, but it's at your own 
risk.


Are you saying one should not use syslog for events caused by untrusted 
users?


Arnt

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?

[Didier Kryn]

Le 22/11/2017 à 13:58, Arnt Gulbrandsen a écrit :
If you really want to look at the details in postgres, you can take a 
good guess at whether two rows were inserted at the same time or one 
later than the other.


    Well, postgress is a database manager. You have a choice of several 
others; they must be able to deal with high fluxes of data. None of them 
is a critical system component.


    System logs are a critical system component and they don't face 
high fluxes of data. You can, in principle, use syslog for applications 
with a high flux of logs, but it's at your own risk.


    Didier


___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?

[Clarke Sideroad]

On 2017-11-22 09:46 AM, Arnt Gulbrandsen wrote:

Aldemir Akpinar writes:

No, I've actually asked an honest question.


In that case you'll get my honest answer. I've implemented several 
file/network formats vaguely like that journal format, one of them has 
likely been used by millions of people.


In each case, the team decided to use a header/packet format, because 
that made both writing and reading simple. In the case of the network 
format we additionally included a magic number to catch version skew, 
and did it using binary because that made for the simplest reading 
code. Reading a 16-byte header using read(2) is simpler than reading a 
textual header.


I don't remember anyone on either of the teams suggesting that using 
text had advantages for developers or users. Generally we just chose 
what was easier to ship reliably and parse/generate with simple code.


In one case we used binary because even though the data were readable 
text, they weren't editable (the actual format had non-trivial 
restrictions). I don't remember the details, but for some reason we 
worried that people would hand-edit files and cause problems deep 
inside the reading program.


I find it totally plausible that the systemd people would design the 
format for similar concerns and end up a format where a fixed-size 
header includes a tag type and length, then a variable-sized packets 
mostly containing log lines, and then another header. That kind of 
thing is so easy to read and write using 
https://linux.die.net/man/2/read and its companion functions.


It is all a trade-off, making choices in implementation, and choice IS a 
good thing to have (Insert thumbs up to Devuan here), I don't think 
anybody is disputing that.


This sub-thread mentions forensics, which to me means analysis of a 
system, a snapshot in time or worst case after complete failure.


Due to the nature of things one is always going to be faced with binary 
files somewhere in the analysis that may or may not be easily 
decode-able or even recoverable.


There is no question in my mind that a broken plain text file is far 
easier to traverse and sort out, which is really what the finger 
pointing game going on here is all about.


Choice is good, having "no choice" forced upon you is bad, end of story.

Clarke





___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?

[Arnt Gulbrandsen]

Aldemir Akpinar writes:

No, I've actually asked an honest question.


In that case you'll get my honest answer. I've implemented several 
file/network formats vaguely like that journal format, one of them has 
likely been used by millions of people.


In each case, the team decided to use a header/packet format, because that 
made both writing and reading simple. In the case of the network format we 
additionally included a magic number to catch version skew, and did it 
using binary because that made for the simplest reading code. Reading a 
16-byte header using read(2) is simpler than reading a textual header.


I don't remember anyone on either of the teams suggesting that using text 
had advantages for developers or users. Generally we just chose what was 
easier to ship reliably and parse/generate with simple code.


In one case we used binary because even though the data were readable text, 
they weren't editable (the actual format had non-trivial restrictions). I 
don't remember the details, but for some reason we worried that people 
would hand-edit files and cause problems deep inside the reading program.


I find it totally plausible that the systemd people would design the format 
for similar concerns and end up a format where a fixed-size header includes 
a tag type and length, then a variable-sized packets mostly containing log 
lines, and then another header. That kind of thing is so easy to read and 
write using https://linux.die.net/man/2/read and its companion functions.


Arnt

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?

[John Hughes]

On 22/11/17 15:08, Aldemir Akpinar wrote:


On 22 November 2017 at 17:03, John Hughes > wrote:


On 22/11/17 14:18, Aldemir Akpinar wrote:


Could you elaborate why are you comparing a relational database
system where its files must be binary with a logging system where
its files doesn't need to binary?



Need?  Nothing "needs" to be in binary[*].  It's a design
decision.  Do the advantages of a structured format (mostly speed)
override the disadvantages (higher costs for access if the reader
software is unavailable?





That's still not the answer to my question!


There is no simple answer to your question because your question 
contains the logical fallacy of "begging the question", i.e. the 
question assumes its own answer.


You said "why are you comparing a relational database system where its 
files must be binary with a logging system where its files doesn't need 
to binary?"


That assumes that the files of a relational database system *must* be 
binary, which is simply untrue and that there is no advantage to making 
the files of a logging system binary which is debatable.


The answer to your question is that there is no one single answer.  The 
system designer will make his decisions about the different tradeoffs.



___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?

[Dave Turner]

On 22/11/17 14:22, Arnt Gulbrandsen wrote:

Aldemir Akpinar writes:
Could you elaborate why are you comparing a relational database 
system where its files must be binary with a logging system where its 
files doesn't need to binary?


You make it sound is if binary files were some sort of horror that 
requires special justification. Please argue the point. Does a text 
format justify x% performance loss? y% increase in line count or code 
complexity? Pick x/y.


Arnt

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng


My understanding of why text files are better for important system logs 
is this:-


When your server goes down big-style and you get all sorts of file 
corruption you stand a very good chance of working out what happened 
even if your text format log file is a bit mangled.


If your binary format log file is mangled life is considerably more 
difficult - ask those that look after Microsoft Servers. I did, 'that's 
as bad as Windows!' he said.


DaveT

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?

[Aldemir Akpinar]

On 22 November 2017 at 17:22, Arnt Gulbrandsen 
wrote:

> Aldemir Akpinar writes:
>
>> Could you elaborate why are you comparing a relational database system
>> where its files must be binary with a logging system where its files
>> doesn't need to binary?
>>
>
> You make it sound is if binary files were some sort of horror that
> requires special justification. Please argue the point. Does a text format
> justify x% performance loss? y% increase in line count or code complexity?
> Pick x/y.
>
>
> Arnt
>
>
>
No, I've actually asked an honest question. I didn't imply anything at all.
But all I get is trolling, not an answer to my Q :)

Anyway, I'm done with these e-mails today. Doesn't help Devuan at all.
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?

[Arnt Gulbrandsen]

Aldemir Akpinar writes:
Could you elaborate why are you comparing a relational database 
system where its files must be binary with a logging system 
where its files doesn't need to binary?


You make it sound is if binary files were some sort of horror that requires 
special justification. Please argue the point. Does a text format justify 
x% performance loss? y% increase in line count or code complexity? Pick 
x/y.


Arnt

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?

[Aldemir Akpinar]

On 22 November 2017 at 17:03, John Hughes  wrote:

> On 22/11/17 14:18, Aldemir Akpinar wrote:
>
>
> That's routine. Few readers read everything that can be read. For example,
>> look at postgres. Its binary file format reveals quite a bit more than you
>> can get using psql, and by design: The writer and binary format are
>> intended for storing things quickly and reliably, and the reader for
>> reading what was stored. Anything that's in the file but wasn't stored by
>> instruction of an SQL user is uninteresting to psql, and the file format
>> writer has no particular reason to avoid storing other information.
>>
>>
>>
>>
> Could you elaborate why are you comparing a relational database system
> where its files must be binary with a logging system where its files
> doesn't need to binary?
>
>
> Need?  Nothing "needs" to be in binary[*].  It's a design decision.  Do
> the advantages of a structured format (mostly speed) override the
> disadvantages (higher costs for access if the reader software is
> unavailable?
>
> [*] or, to put it another way -- *everything on a computer is in binary*.
> "Text" files are binary.  The question is how easy is it to decode the file
> format.  It seems obvious that a "text" file is easy to decode, everyone
> knows the format (but what character set is it in?), but don't forget that
> the "text" file is stored on a filesystem, which is itself a complicated
> "binary" structure.  When you're talking about "forensics", i.e. looking at
> something that may be broken in exciting ways, it's quite naïve to assume
> that you can just mount the filesystem (which one?) and use cat, vi, grep
> or whatever.
>
>

That's still not the answer to my question!
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?

[John Hughes]

On 22/11/17 14:18, Aldemir Akpinar wrote:


That's routine. Few readers read everything that can be read. For
example, look at postgres. Its binary file format reveals quite a
bit more than you can get using psql, and by design: The writer
and binary format are intended for storing things quickly and
reliably, and the reader for reading what was stored. Anything
that's in the file but wasn't stored by instruction of an SQL user
is uninteresting to psql, and the file format writer has no
particular reason to avoid storing other information.




Could you elaborate why are you comparing a relational database system 
where its files must be binary with a logging system where its files 
doesn't need to binary?




Need?  Nothing "needs" to be in binary[*].  It's a design decision. Do 
the advantages of a structured format (mostly speed) override the 
disadvantages (higher costs for access if the reader software is 
unavailable?


[*] or, to put it another way -- *everything on a computer is in 
binary*.  "Text" files are binary.  The question is how easy is it to 
decode the file format.  It seems obvious that a "text" file is easy to 
decode, everyone knows the format (but what character set is it in?), 
but don't forget that the "text" file is stored on a filesystem, which 
is itself a complicated "binary" structure.  When you're talking about 
"forensics", i.e. looking at something that may be broken in exciting 
ways, it's quite naïve to assume that you can just mount the filesystem 
(which one?) and use cat, vi, grep or whatever.
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?

[Aldemir Akpinar]

> That's routine. Few readers read everything that can be read. For example,
> look at postgres. Its binary file format reveals quite a bit more than you
> can get using psql, and by design: The writer and binary format are
> intended for storing things quickly and reliably, and the reader for
> reading what was stored. Anything that's in the file but wasn't stored by
> instruction of an SQL user is uninteresting to psql, and the file format
> writer has no particular reason to avoid storing other information.
>
> If you really want to look at the details in postgres, you can take a good
> guess at whether two rows were inserted at the same time or one later than
> the other.
>
> That's why forensics people use the files. Systemd is about the millionth
> system to join the club. Flame postgres and vast numbers of others before
> you flame systemd. Or better yet, limit your statements about systemd to
> what's correct.
>
> Arnt
>

Could you elaborate why are you comparing a relational database system
where its files must be binary with a logging system where its files
doesn't need to binary?

--
aldemir
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?

[Arnt Gulbrandsen]

Arnt Karlsen writes:

you appear to suggest that law enforcement wanting to read systemd
journal logs, _should_ depend on the mercy of systemd developers not 
"filtering" away inconvenient evidence of e.g. systemd developer

wrongdoing from said law enforcement.


That's routine. Few readers read everything that can be read. For example, 
look at postgres. Its binary file format reveals quite a bit more than you 
can get using psql, and by design: The writer and binary format are 
intended for storing things quickly and reliably, and the reader for 
reading what was stored. Anything that's in the file but wasn't stored by 
instruction of an SQL user is uninteresting to psql, and the file format 
writer has no particular reason to avoid storing other information.


If you really want to look at the details in postgres, you can take a good 
guess at whether two rows were inserted at the same time or one later than 
the other.


That's why forensics people use the files. Systemd is about the millionth 
system to join the club. Flame postgres and vast numbers of others before 
you flame systemd. Or better yet, limit your statements about systemd to 
what's correct.


Arnt

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

[DNG] ..forensics on systemd or journald logs, was: rc.local removed from Debian 9, rly?

[Arnt Karlsen]

On Wed, 22 Nov 2017 07:19:20 +0100, John wrote in message 
:

> On 22/11/17 02:59, Arnt Karlsen wrote:
> > On Tue, 21 Nov 2017 18:21:14 +0100, John wrote in message
> > :
> >  
> >> (Damn but the systemd journal is great :-))  
> > ..is there a way to decode and read those binary systemd journal
> > logs on classic POSIX/Unix etc forensic systems _not_ running
> > systemd?  
> 
> Is there any way to read a file in format X without a program that
> reads format X?

..I'm asking you.  Your other "answers" to this question suggests 
you may know the true answer to my question.

> I suppose you could scatter iron filings on the disk the use a
> scanning electron microscope to examine their positions and, using
> paper, pencil and a copy of the systemd doc work out the contents by
> hand.
> 
> Or, being endowed with the minimum level of foresight necessary for 
> survival have a forensic system that includes tools for reading the
> file formats you're likely to find  on the system you want to
> post-mortem.

..correct, that is precisely why I went for devuan and precisely why 
I ask you here now.  

..you appear to suggest that law enforcement wanting to read systemd
journal logs, _should_ depend on the mercy of systemd developers not 
"filtering" away inconvenient evidence of e.g. systemd developer
wrongdoing from said law enforcement.

..depending on your jurisdiction, this feature of systemd is either 
a good thing or a bad thing, probably both and probably capable of
facilitating the cover-up of organised crime, AFAICT.

-- 
..med vennlig hilsen = with Kind Regards from Arnt Karlsen
...with a number of polar bear hunters in his ancestry...
  Scenarios always come in sets of three: 
  best case, worst case, and just in case.
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng