Re: Proposal: Disable compression of newsyslog by default

[Mike Karels]

On 25 Dec 2023, at 2:12, Xin Li wrote:

> On 2023-12-23 14:17, Mike Karels wrote:
>> On 23 Dec 2023, at 15:23, Craig Leres wrote:
>>
>>> On 12/23/23 06:52, Konstantin Belousov wrote:
 This is strange change at best.  I have no opinion about the disabling
 of compression of the rotated logs by default, but we already have knobs
 to do that.  Adding a knob that disables (or enables) other knobs to work
 is weird.
>>>
>>> I totally agree. This moves the compression knob from the config file to 
>>> the command line. And what if the user wants some but not all files to be 
>>> compressed? Or wants to use different compression with different log files?
>>
>> Another possibility would be to introduce some simple form of variables in
>> newsyslog.conf, replacing J by a variable reference, with the variable
>> being set near the beginning.  E.g.
>>
>>  V=zstd (or just V= for none?)
>>  ... $V
>>  ... $V
>>
>> Then there would be one global change, and much easier changing of the
>> default.
>>
>> It would also be possible to add  /etc/newsyslog.local.conf at the
>> beginning, and set variables there, making changes to the default file less
>> painful in the future.
>
> I've implemented support of  in https://reviews.freebsd.org/D43174 
> .  Let's don't add macro or variables to newsyslog.conf as it would be a 
> nightmare for compatibility with other newsyslog variants of other BSDs.

Looks like there is a fair amount of divergence from NetBSD, at least;
I don't think compatibility with other BSDs is a significant factor.
Compatibility with older FreeBSD is a bigger consideration.

An intermediate possibility would be to add an option letter for the
default compression to be used in each entry, and use  to set
that default.  It seems wrong to attempt to use  zstd with option
letters like J; it's contradictory, if not confusing.  It's still a knob
to disable other knobs, although it is in the right file.

Mike

Re: Proposal: Disable compression of newsyslog by default

[Xin Li]

On 2023-12-23 14:17, Mike Karels wrote:

On 23 Dec 2023, at 15:23, Craig Leres wrote:


On 12/23/23 06:52, Konstantin Belousov wrote:

This is strange change at best.  I have no opinion about the disabling
of compression of the rotated logs by default, but we already have knobs
to do that.  Adding a knob that disables (or enables) other knobs to work
is weird.


I totally agree. This moves the compression knob from the config file to the 
command line. And what if the user wants some but not all files to be 
compressed? Or wants to use different compression with different log files?


Another possibility would be to introduce some simple form of variables in
newsyslog.conf, replacing J by a variable reference, with the variable
being set near the beginning.  E.g.

V=zstd (or just V= for none?)
... $V
... $V

Then there would be one global change, and much easier changing of the
default.

It would also be possible to add  /etc/newsyslog.local.conf at the
beginning, and set variables there, making changes to the default file less
painful in the future.


I've implemented support of  in 
https://reviews.freebsd.org/D43174 .  Let's don't add macro or variables 
to newsyslog.conf as it would be a nightmare for compatibility with 
other newsyslog variants of other BSDs.


Cheers,


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Proposal: Disable compression of newsyslog by default

[Xin Li]

On 2023-12-24 10:03, Rodney W. Grimes wrote:

On 2023-12-23 06:52, Konstantin Belousov wrote:

On Fri, Dec 22, 2023 at 11:18:23PM -0800, Xin Li wrote:

Hi,

Inspired by D42961, I propose that we move forward with disabling the
compression by default in newsyslog, as implemented in
https://reviews.freebsd.org/D43169

Historically, newsyslog has compressed rotated log files to save disk space.
This approach was valuable in the early days where storage space was
limited.  However, the landscape has changed significantly.  Modern file
systems, such as ZFS, now offer native compression capabilities.
Additionally, the widespread availability of larger hard drives has
diminished the necessity for additional compression.  Notably, the need to
decompress log files for pattern searches poses a significant inconvenience,
further questioning the utility of this legacy feature.

In commit 906748d208d3, flags J, X, Y, Z can now indicate that a log file is
eligible for compression rather than directly enforcing it. It allows for a
more flexible approach, wherein the actual compression method can be set to
"none" or specified as one among bzip2, gzip, xz, or zstd.

Therefore I would propose that we change the default compression setting to
"none" in FreeBSD 15.0.  This change reflects our adaptation to the evolving
technological environment and user needs.  It also aligns with the broader
initiative to modernize our systems while maintaining flexibility and
efficiency.

I look forward to your thoughts and feedback on this proposal.


This is strange change at best.  I have no opinion about the disabling
of compression of the rotated logs by default, but we already have knobs
to do that.  Adding a knob that disables (or enables) other knobs to work
is weird.


And counter intuitive!


If you want to change the compression, update the default configuration file.


The primary issue with simply updating the default configuration file is
the increased workload it imposes during system upgrades. Since the
compression method flag is a part of newsyslog.conf, standard conflict
resolution tools like diff3 struggle to automatically resolve changes
involving these flags without manual intervention. This situation
necessitates that users manually reconcile their configuration with
every update to newsyslog.conf, even for minor alterations like
switching the default compression method.

Therefore, the proposal isn't about adding another knob within
newsyslog.conf. Rather, it's about introducing a command-line option for
newsyslog (to be used in /etc/crontab) that specifies the preferred
compression method, or the choice not to compress at all. This approach
is more self-contained compared to modifying each line in
newsyslog.conf. It offers a simpler, more straightforward solution for
administrators to manage their compression settings, reducing the
administrative burden during upgrades.


So now I edit /etc/crontab to change the behavior of newsyslog?
Thats VERY counter to how things "should be."  I am sorry, but
this change is just yet another "Oh my god, the defaults dont
work for me so I am gona force my idea of what the defaults should
be on everone else."

And you just pushed the merge conflict to another file, one that
is not even related to newsyslog, you do get your arguement doesnt
hold water?  A change in /etc/newsyslog.conf or /etc/crontab BOTH
lead to the fact people are going to have to merge things.

Now we are goning go crazy trying to figure out why my logs are
not compressed and finally gona end up finding this flag in
/etc/crontab that is causing newsyslog to ignore what I put
in its proper config file???  Again, very counter intuitive.


Thank you for your feedback.

In response to the concerns you've raised, I've initiated a new change, 
which you can review at https://reviews.freebsd.org/D43174 . This 
introduces a new configuration file option, "", in the 
newsyslog.conf filem, intended for use at the beginning of the file, and 
updated https://reviews.freebsd.org/D43169 with the default change done 
in the configuration file.


This change aims to provide a more intuitive way to manage compression 
settings directly within the newsyslog configuration, hopefully 
addressing the issues you highlighted.


Cheers,


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Proposal: Disable compression of newsyslog by default

[Rodney W. Grimes]

> On 2023-12-23 10:17, Ceri Davies wrote:
> > I really don?t like the idea of adding flags that make the program ignore a 
> > config file.  I also think this is premature until ZFS installs are default 
> > on all architectures.
> > 
> > However, if you must (and I see you have) change this, please don?t call 
> > the option ?legacy? as we wish to deprecate other behaviour in the future 
> > or add additional compression algorithms.
> > Perhaps it could be called ?useconfig? or something that describes what it 
> > actually does.
> > 
> > Also, could you please update the manage commit to note the change in 15.0 
> > because not everyone reads commit logs and we may end up violating the PoLA 
> > twice otherwise.
> 
> Thank you for sharing your thoughts on the commit. I want to clarify 
> that my recent change does not alter the existing behavior of the code. 
> In fact, the current legacy test cases still pass successfully. The 
> modification I introduced primarily adds the option to override the 
> default compression setting.
> 
> Regarding your suggestion to rename the ?legacy? option, the term was 
> chosen to reflect its purpose ? to preserve historical behavior while 
> acknowledging that it may be phased out in the future. Adding new 
> letters to the configuration format isn't scalable and lacks 
> flexibility.

The words "new, old, legacy, etc" are poor adjectives as they are
time variant.  Use of "legacy" isn't scalable and lacks flexibility too,
what do I call the next legacy move?  Legacy-legacy?  Please try
to use time invariant nouns when picking option names.

> Especially, consider an application that install their own 
> newsyslog.conf.d configuration, in which case the configuration is 
> overwritten during upgrades, or has to have some way to solve merge 
> conflicts, should they decide to add new log files, etc.  There's no 
> practical reason for users to adhere to a specific compression method 
> that the application writer told them, because the administrator have 
> better idea about what compressor is more suitable for their situation.
> 
> I'll add a note to the release notes later, but rest assured that there 
> is no PoLA violation yet.

Actually I think there is.  When admin A comes along and turns on
this global "do not compress" flag in /etc/crontab's invokation of
newsyslog then some time later admin B comes along and tries to fix
a full /var/log problem by working on /etc/newsyslogd.conf he pulls
his few grey remaining hairs out before he finds this "tweak" to
ignore what the config file is saying!  This POLA shall long outlive
your entry in the release notes for 15 and exist for ever.

> Cheers,

-- 
Rod Grimes rgri...@freebsd.org

Re: Proposal: Disable compression of newsyslog by default

[Rodney W. Grimes]

> On 2023-12-23 06:52, Konstantin Belousov wrote:
> > On Fri, Dec 22, 2023 at 11:18:23PM -0800, Xin Li wrote:
> >> Hi,
> >>
> >> Inspired by D42961, I propose that we move forward with disabling the
> >> compression by default in newsyslog, as implemented in
> >> https://reviews.freebsd.org/D43169
> >>
> >> Historically, newsyslog has compressed rotated log files to save disk 
> >> space.
> >> This approach was valuable in the early days where storage space was
> >> limited.  However, the landscape has changed significantly.  Modern file
> >> systems, such as ZFS, now offer native compression capabilities.
> >> Additionally, the widespread availability of larger hard drives has
> >> diminished the necessity for additional compression.  Notably, the need to
> >> decompress log files for pattern searches poses a significant 
> >> inconvenience,
> >> further questioning the utility of this legacy feature.
> >>
> >> In commit 906748d208d3, flags J, X, Y, Z can now indicate that a log file 
> >> is
> >> eligible for compression rather than directly enforcing it. It allows for a
> >> more flexible approach, wherein the actual compression method can be set to
> >> "none" or specified as one among bzip2, gzip, xz, or zstd.
> >>
> >> Therefore I would propose that we change the default compression setting to
> >> "none" in FreeBSD 15.0.  This change reflects our adaptation to the 
> >> evolving
> >> technological environment and user needs.  It also aligns with the broader
> >> initiative to modernize our systems while maintaining flexibility and
> >> efficiency.
> >>
> >> I look forward to your thoughts and feedback on this proposal.
> > 
> > This is strange change at best.  I have no opinion about the disabling
> > of compression of the rotated logs by default, but we already have knobs
> > to do that.  Adding a knob that disables (or enables) other knobs to work
> > is weird.

And counter intuitive!

> > If you want to change the compression, update the default configuration 
> > file.
> 
> The primary issue with simply updating the default configuration file is 
> the increased workload it imposes during system upgrades. Since the 
> compression method flag is a part of newsyslog.conf, standard conflict 
> resolution tools like diff3 struggle to automatically resolve changes 
> involving these flags without manual intervention. This situation 
> necessitates that users manually reconcile their configuration with 
> every update to newsyslog.conf, even for minor alterations like 
> switching the default compression method.
> 
> Therefore, the proposal isn't about adding another knob within 
> newsyslog.conf. Rather, it's about introducing a command-line option for 
> newsyslog (to be used in /etc/crontab) that specifies the preferred 
> compression method, or the choice not to compress at all. This approach 
> is more self-contained compared to modifying each line in 
> newsyslog.conf. It offers a simpler, more straightforward solution for 
> administrators to manage their compression settings, reducing the 
> administrative burden during upgrades.

So now I edit /etc/crontab to change the behavior of newsyslog?
Thats VERY counter to how things "should be."  I am sorry, but
this change is just yet another "Oh my god, the defaults dont
work for me so I am gona force my idea of what the defaults should
be on everone else."

And you just pushed the merge conflict to another file, one that
is not even related to newsyslog, you do get your arguement doesnt
hold water?  A change in /etc/newsyslog.conf or /etc/crontab BOTH
lead to the fact people are going to have to merge things.

Now we are goning go crazy trying to figure out why my logs are
not compressed and finally gona end up finding this flag in
/etc/crontab that is causing newsyslog to ignore what I put
in its proper config file???  Again, very counter intuitive.

> 
> I hope this clarifies the rationale behind the proposed changes and 
> their potential benefits.

I see only change for the sake of change and no real benefit, just
another hack cause someone is to weak to deal with locally modifing
configurations and dealing with the maintenance.  Do understand
every single change to any of this configuration stuff creates
burden for someone, I now have to add yet another file to my
maintanance to undo this change to /etc/crontab.  And I have to
special case that as "only apply if version >= 15".


-- 
Rod Grimes rgri...@freebsd.org

Re: Proposal: Disable compression of newsyslog by default

[Alex Dupre]

Xin Li wrote:
> Inspired by D42961, I propose that we move forward with disabling the
> compression by default in newsyslog, as implemented in
> https://reviews.freebsd.org/D43169

I like the possibility to entirely disable compression on log files, or
to select a specific compression algorithm, but I don't think we should
disable compression by default because there is a filesystem that
supports compression.

-- 
Alex Dupre

Re: Proposal: Disable compression of newsyslog by default

[Tomoaki AOKI]

On Sat, 23 Dec 2023 15:14:20 -0800
Xin Li  wrote:

> On 2023-12-23 07:09, Enji Cooper wrote:
> > This impacts embedded systems or jails which use UFS as the default
> > /var/log backed device. There are quite a few larger consumers of
> > FreeBSD out there that still use UFS instead of ZFS.
> 
> I appreciate your feedback!
> 
> Thank you for pointing out the implications of this change for embedded 
> systems and jails using UFS. I understand your concerns, especially 
> regarding larger FreeBSD consumers who might still rely on UFS instead 
> of ZFS.
> 
> Note that the committed change was designed to simplify code 
> maintenance, particularly for downstream software vendors.  By reducing 
> the number of configuration lines in newsyslog.conf to a single line in 
> /etc/crontab, it makes it easier for downstream maintainers to follow 
> the latest FreeBSD codebase, because they don't have to manually solve 
> merge conflicts when someone changes newsyslog.conf anymore.  This 
> should ease the integration and maintenance processes for these vendors.
> 
> > Adding this instead into bsdinstall and the documentation as a suggested
> >   knob seems like a good way to go.
> > 
> > Just something to keep in mind when making this change.
> 
> Now back to the proposed behavior change, regarding your suggestion to 
> change the default in the installer, I have reservations about this 
> approach. One of my primary motivations for this change is to move away 
> from using flags to specify which compression method should be used.  In 
> my view, the software package distributed configuration should not 
> dictate the compression method to be used by the user. Rather, its role 
> should be to inform newsyslog about the suitability of a file for 
> compression. This shift in approach aims to provide users with greater 
> flexibility and autonomy in managing their compression settings.
> 
> Cheers,

I think any config files which is updated by freebsd-update and/or
etcupdate and/or mergemaster should better be in /etc/defaults and any
configuration in them should be able to overridden by configuration
files directly under /etc, like rc.conf.

Is it possible to override EVERY configurations in /etc/newsyslog.conf
by any configuration files under /etc/newsyslog.conf.d
and/or /usr/local/etc/newsyslog.conf.d?

If so, just moving default one to /etc/defaults would help, I think,
like at commit d105b00084a533f41a1277d08cfacb15062d9b50 [1].

[1]
https://cgit.freebsd.org/src/commit/?id=d105b00084a533f41a1277d08cfacb15062d9b50

Regards.

-- 
Tomoaki AOKI

Re: Proposal: Disable compression of newsyslog by default

[Sulev-Madis Silber]

On 23 December 2023 09:18:23 EET, Xin Li  wrote:
>Hi,
>
>Inspired by D42961, I propose that we move forward with disabling the 
>compression by default in newsyslog, as implemented in 
>https://reviews.freebsd.org/D43169
>
>Historically, newsyslog has compressed rotated log files to save disk space. 
>This approach was valuable in the early days where storage space was limited.

it's still limited

>However, the landscape has changed significantly.  Modern file systems, such 
>as ZFS, now offer native compression capabilities.

not everyone uses them

>Additionally, the widespread availability of larger hard drives has diminished 
>the necessity for additional compression.

but data sizes also have increased massively

>Notably, the need to decompress log files for pattern searches poses a 
>significant inconvenience, further questioning the utility of this legacy 
>feature.

should be up to each admin to cba decompression vs. plain speed/size/etc

>In commit 906748d208d3, flags J, X, Y, Z can now indicate that a log file is 
>eligible for compression rather than directly enforcing it. It allows for a 
>more flexible approach, wherein the actual compression method can be set to 
>"none" or specified as one among bzip2, gzip, xz, or zstd.

that's good approach

>Therefore I would propose that we change the default compression setting to 
>"none" in FreeBSD 15.0.  This change reflects our adaptation to the evolving 
>technological environment and user needs.  It also aligns with the broader 
>initiative to modernize our systems while maintaining flexibility and 
>efficiency.

unsure about this. generic zroot install would be fine with this i guess, and 
usual log sizes? other custom installs need tuning anyway

>I look forward to your thoughts and feedback on this proposal.
>
>Cheers,

indeed. we have large disks now. but we fill them all. i started with 1.2g one. 
was too small. needed to compress for space. now i have 12t. it's still too 
small. i compress for space. they make them up to 22t nowadays. this is about 
2 times larger but still feels small. how did this happen? we just did this 
to ourselves. data sizes have kept up with storage and bandwidth. gamer might 
get 1gbit/s connection at home so (s)he only needs to wait for one hour to 
download new game. just as dialup user once did, wait for hours. or it could be 
photographer, graphics designer or architect who works. both cases still use 
compressed data as cpu and ram permits it and it saves a lot of time and space. 
now, it's also related to servers as those things don't disappear to or appear 
from just thin air. they come from machines, some of them hopefully running 
fbsd, where admins wonder how to deal with large log sizes. they need them for 
audit purposes. or statistics. hardware allows, so they compress it. 
"write-only-read-never" data benefits from, eg, xz a lot. as others already 
have told

so yeah, from (only!) 25+ years of experience, i can confirm that humankind has 
developed AND used everything at max. internet, first for military and 
educational uses, now for connecting washing machines. oh and, first hdd, state 
of art device then, can only store *part* of *compressed* photo now

now, this might not be related to default fbsd installs in common usage where 
default base syslog creates tiny amount of data per week

but one of reasons was given how everything fits uncompressed nowdays. to our 
disks and pipes. which it really doesn't

Re: Proposal: Disable compression of newsyslog by default

[Xin Li]

On 2023-12-23 07:09, Enji Cooper wrote:

This impacts embedded systems or jails which use UFS as the default
/var/log backed device. There are quite a few larger consumers of
FreeBSD out there that still use UFS instead of ZFS.


I appreciate your feedback!

Thank you for pointing out the implications of this change for embedded 
systems and jails using UFS. I understand your concerns, especially 
regarding larger FreeBSD consumers who might still rely on UFS instead 
of ZFS.


Note that the committed change was designed to simplify code 
maintenance, particularly for downstream software vendors.  By reducing 
the number of configuration lines in newsyslog.conf to a single line in 
/etc/crontab, it makes it easier for downstream maintainers to follow 
the latest FreeBSD codebase, because they don't have to manually solve 
merge conflicts when someone changes newsyslog.conf anymore.  This 
should ease the integration and maintenance processes for these vendors.



Adding this instead into bsdinstall and the documentation as a suggested
  knob seems like a good way to go.

Just something to keep in mind when making this change.


Now back to the proposed behavior change, regarding your suggestion to 
change the default in the installer, I have reservations about this 
approach. One of my primary motivations for this change is to move away 
from using flags to specify which compression method should be used.  In 
my view, the software package distributed configuration should not 
dictate the compression method to be used by the user. Rather, its role 
should be to inform newsyslog about the suitability of a file for 
compression. This shift in approach aims to provide users with greater 
flexibility and autonomy in managing their compression settings.


Cheers,


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Proposal: Disable compression of newsyslog by default

[Xin Li]

On 2023-12-23 10:17, Ceri Davies wrote:

I really don’t like the idea of adding flags that make the program ignore a 
config file.  I also think this is premature until ZFS installs are default on 
all architectures.

However, if you must (and I see you have) change this, please don’t call the 
option “legacy” as we wish to deprecate other behaviour in the future or add 
additional compression algorithms.
Perhaps it could be called “useconfig” or something that describes what it 
actually does.

Also, could you please update the manage commit to note the change in 15.0 
because not everyone reads commit logs and we may end up violating the PoLA 
twice otherwise.


Thank you for sharing your thoughts on the commit. I want to clarify 
that my recent change does not alter the existing behavior of the code. 
In fact, the current legacy test cases still pass successfully. The 
modification I introduced primarily adds the option to override the 
default compression setting.


Regarding your suggestion to rename the “legacy” option, the term was 
chosen to reflect its purpose – to preserve historical behavior while 
acknowledging that it may be phased out in the future. Adding new 
letters to the configuration format isn't scalable and lacks 
flexibility. Especially, consider an application that install their own 
newsyslog.conf.d configuration, in which case the configuration is 
overwritten during upgrades, or has to have some way to solve merge 
conflicts, should they decide to add new log files, etc.  There's no 
practical reason for users to adhere to a specific compression method 
that the application writer told them, because the administrator have 
better idea about what compressor is more suitable for their situation.


I'll add a note to the release notes later, but rest assured that there 
is no PoLA violation yet.


Cheers,




OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Proposal: Disable compression of newsyslog by default

[Xin Li]

On 2023-12-23 13:08, Steve Kargl wrote:

On Sat, Dec 23, 2023 at 11:13:23AM -0800, Xin Li wrote:


I appreciate your perspective on this issue. However, I believe there are
additional benefits to modifying the newsyslog code (which is already done
in commit 906748d208d3, by the way) beyond what can be achieved by simply
adjusting the defaults in newsyslog.conf.


Why ask for others opinions if you're simply going to commit
the change?  It seems the commit (23 Dec 2023 06:53:06 UTC)
occurred before the initial post in the discussion here
(23 Dec 2023 07:18:23 UTC).


This commit only added the option and does not change the existing code 
behavior.


Cheers,



OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Proposal: Disable compression of newsyslog by default

[Mike Karels]

On 23 Dec 2023, at 15:23, Craig Leres wrote:

> On 12/23/23 06:52, Konstantin Belousov wrote:
>> This is strange change at best.  I have no opinion about the disabling
>> of compression of the rotated logs by default, but we already have knobs
>> to do that.  Adding a knob that disables (or enables) other knobs to work
>> is weird.
>
> I totally agree. This moves the compression knob from the config file to the 
> command line. And what if the user wants some but not all files to be 
> compressed? Or wants to use different compression with different log files?

Another possibility would be to introduce some simple form of variables in
newsyslog.conf, replacing J by a variable reference, with the variable
being set near the beginning.  E.g.

V=zstd (or just V= for none?)
... $V
... $V

Then there would be one global change, and much easier changing of the
default.

It would also be possible to add  /etc/newsyslog.local.conf at the
beginning, and set variables there, making changes to the default file less
painful in the future.

>> If you want to change the compression, update the default configuration file.
>
> I also think this is the best approach.
>
> Given the current freebsd-update workflow, users will get to deal with 
> changes to the default newsyslog.conf via mergemaster.
>
> And having converted a number of systems from newsyslog compression to zfs 
> compression, just changing the config file is not the only change needed, 
> users will still need to compress/uncompress existing log files.

Good point.  Although newsyslog could be smart enough to recognize alternate
suffixes (or none), and rotate the files anyway.  Short of that, this suggests
that a new default config file should specify bzip2, but it would be easy to
localize.

Mike

Re: Proposal: Disable compression of newsyslog by default

[Craig Leres]

On 12/23/23 06:52, Konstantin Belousov wrote:

This is strange change at best.  I have no opinion about the disabling
of compression of the rotated logs by default, but we already have knobs
to do that.  Adding a knob that disables (or enables) other knobs to work
is weird.


I totally agree. This moves the compression knob from the config file to 
the command line. And what if the user wants some but not all files to 
be compressed? Or wants to use different compression with different log 
files?



If you want to change the compression, update the default configuration file.


I also think this is the best approach.

Given the current freebsd-update workflow, users will get to deal with 
changes to the default newsyslog.conf via mergemaster.


And having converted a number of systems from newsyslog compression to 
zfs compression, just changing the config file is not the only change 
needed, users will still need to compress/uncompress existing log files.


Craig

Re: Proposal: Disable compression of newsyslog by default

[Steve Kargl]

On Sat, Dec 23, 2023 at 11:13:23AM -0800, Xin Li wrote:
> 
> I appreciate your perspective on this issue. However, I believe there are
> additional benefits to modifying the newsyslog code (which is already done
> in commit 906748d208d3, by the way) beyond what can be achieved by simply
> adjusting the defaults in newsyslog.conf.

Why ask for others opinions if you're simply going to commit
the change?  It seems the commit (23 Dec 2023 06:53:06 UTC)
occurred before the initial post in the discussion here
(23 Dec 2023 07:18:23 UTC).

-- 
Steve

Re: Proposal: Disable compression of newsyslog by default

[Xin Li]

On 2023-12-23 06:52, Konstantin Belousov wrote:

On Fri, Dec 22, 2023 at 11:18:23PM -0800, Xin Li wrote:

Hi,

Inspired by D42961, I propose that we move forward with disabling the
compression by default in newsyslog, as implemented in
https://reviews.freebsd.org/D43169

Historically, newsyslog has compressed rotated log files to save disk space.
This approach was valuable in the early days where storage space was
limited.  However, the landscape has changed significantly.  Modern file
systems, such as ZFS, now offer native compression capabilities.
Additionally, the widespread availability of larger hard drives has
diminished the necessity for additional compression.  Notably, the need to
decompress log files for pattern searches poses a significant inconvenience,
further questioning the utility of this legacy feature.

In commit 906748d208d3, flags J, X, Y, Z can now indicate that a log file is
eligible for compression rather than directly enforcing it. It allows for a
more flexible approach, wherein the actual compression method can be set to
"none" or specified as one among bzip2, gzip, xz, or zstd.

Therefore I would propose that we change the default compression setting to
"none" in FreeBSD 15.0.  This change reflects our adaptation to the evolving
technological environment and user needs.  It also aligns with the broader
initiative to modernize our systems while maintaining flexibility and
efficiency.

I look forward to your thoughts and feedback on this proposal.


This is strange change at best.  I have no opinion about the disabling
of compression of the rotated logs by default, but we already have knobs
to do that.  Adding a knob that disables (or enables) other knobs to work
is weird.

If you want to change the compression, update the default configuration file.


The primary issue with simply updating the default configuration file is 
the increased workload it imposes during system upgrades. Since the 
compression method flag is a part of newsyslog.conf, standard conflict 
resolution tools like diff3 struggle to automatically resolve changes 
involving these flags without manual intervention. This situation 
necessitates that users manually reconcile their configuration with 
every update to newsyslog.conf, even for minor alterations like 
switching the default compression method.


Therefore, the proposal isn't about adding another knob within 
newsyslog.conf. Rather, it's about introducing a command-line option for 
newsyslog (to be used in /etc/crontab) that specifies the preferred 
compression method, or the choice not to compress at all. This approach 
is more self-contained compared to modifying each line in 
newsyslog.conf. It offers a simpler, more straightforward solution for 
administrators to manage their compression settings, reducing the 
administrative burden during upgrades.


I hope this clarifies the rationale behind the proposed changes and 
their potential benefits.


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Proposal: Disable compression of newsyslog by default

[Xin Li]

On 2023-12-23 00:51, Miroslav Lachman wrote:

On 23/12/2023 08:18, Xin Li wrote:

Hi,

Inspired by D42961, I propose that we move forward with disabling the 
compression by default in newsyslog, as implemented in 
https://reviews.freebsd.org/D43169


Historically, newsyslog has compressed rotated log files to save disk 
space. This approach was valuable in the early days where storage 
space was limited.  However, the landscape has changed significantly.  
Modern file systems, such as ZFS, now offer native compression 
capabilities. Additionally, the widespread availability of larger hard 
drives has diminished the necessity for additional compression.  
Notably, the need to decompress log files for pattern searches poses a 
significant inconvenience, further questioning the utility of this 
legacy feature.


In commit 906748d208d3, flags J, X, Y, Z can now indicate that a log 
file is eligible for compression rather than directly enforcing it. It 
allows for a more flexible approach, wherein the actual compression 
method can be set to "none" or specified as one among bzip2, gzip, xz, 
or zstd.


Therefore I would propose that we change the default compression 
setting to "none" in FreeBSD 15.0.  This change reflects our 
adaptation to the evolving technological environment and user needs.  
It also aligns with the broader initiative to modernize our systems 
while maintaining flexibility and efficiency.


I look forward to your thoughts and feedback on this proposal.


I don't think anything needs to be changed on newsyslog. Those who want 
to disable compression can do so in the "default" newsyslog.conf file. 
Why force this change in the newsyslog code?
I also don't think that the log file should not be compressed even on a 
compressed filesystem. Compressed log files can be handled by tools like 
zcat, zless, zgrep, etc. without first decompressing the log file. Text 
log files can still grow to large sizes, and if you have a daily backup 
job over a long distance network, if you use a protocol without own 
compression, it is still better to have compressed log files than 
uncompressed on a compressed filesystem. To me, it's the same as 
compressing large database dumps and not relying on filesystem compression.
YMMV, but I really don't see any benefit of changing the newsyslog code, 
just change defaults in newsyslog.conf.


I appreciate your perspective on this issue. However, I believe there 
are additional benefits to modifying the newsyslog code (which is 
already done in commit 906748d208d3, by the way) beyond what can be 
achieved by simply adjusting the defaults in newsyslog.conf.


Firstly, updating the newsyslog code provides us with greater 
flexibility when introducing new compression modes, such as 'xz -9e', or 
in supporting other compressors. This approach is more efficient than 
continually adding new letters to represent each compression method in 
the configuration file. It simplifies the process and avoids unnecessary 
complexity.


Furthermore, the previous method of representing compression methods in 
the configuration file can be cumbersome for users who have disabled or 
altered their compression configuration.  Each time there is a change in 
newsyslog.conf, these users face potential conflicts that require manual 
resolution.  Updating the code would make the configuration in one 
central place (/etc/crontab) therefore simplifying maintenance for users 
and administrators alike.


While I acknowledge your points regarding the utility of compressed log 
files, especially in scenarios involving daily backups over 
long-distance networks, I believe the flexibility and ease of 
maintenance offered by updating the code would be a significant 
improvement for many users.


Looking forward to further discussion on this.

Cheers,


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Proposal: Disable compression of newsyslog by default

[Michael Gmelin]

> On 23. Dec 2023, at 16:10, Enji Cooper  wrote:
> 
> 
>> On Dec 22, 2023, at 23:18, Xin Li  wrote:
>> 
>> Hi,
>> 
>> Inspired by D42961, I propose that we move forward with disabling the 
>> compression by default in newsyslog, as implemented in 
>> https://reviews.freebsd.org/D43169
>> 
>> Historically, newsyslog has compressed rotated log files to save disk space. 
>> This approach was valuable in the early days where storage space was 
>> limited.  However, the landscape has changed significantly.  Modern file 
>> systems, such as ZFS, now offer native compression capabilities. 
>> Additionally, the widespread availability of larger hard drives has 
>> diminished the necessity for additional compression.  Notably, the need to 
>> decompress log files for pattern searches poses a significant inconvenience, 
>> further questioning the utility of this legacy feature.
>> 
>> In commit 906748d208d3, flags J, X, Y, Z can now indicate that a log file is 
>> eligible for compression rather than directly enforcing it. It allows for a 
>> more flexible approach, wherein the actual compression method can be set to 
>> "none" or specified as one among bzip2, gzip, xz, or zstd.
>> 
>> Therefore I would propose that we change the default compression setting to 
>> "none" in FreeBSD 15.0.  This change reflects our adaptation to the evolving 
>> technological environment and user needs.  It also aligns with the broader 
>> initiative to modernize our systems while maintaining flexibility and 
>> efficiency.
>> 
>> I look forward to your thoughts and feedback on this proposal.
> 
> This impacts embedded systems or jails which use UFS as the default /var/log 
> backed device. There are quite a few larger consumers of FreeBSD out there 
> that still use UFS instead of ZFS.
> 
> Adding this instead into bsdinstall and the documentation as a suggested knob 
> seems like a good way to go.
> 
> Just something to keep in mind when making this change.

I would not change the default behavior (POLA violation), but adding an easy to 
change knob to disable compression sounds like a reasonable approach.

-m

Re: Proposal: Disable compression of newsyslog by default

[Enji Cooper]

> On Dec 22, 2023, at 23:18, Xin Li  wrote:
> 
> Hi,
> 
> Inspired by D42961, I propose that we move forward with disabling the 
> compression by default in newsyslog, as implemented in 
> https://reviews.freebsd.org/D43169
> 
> Historically, newsyslog has compressed rotated log files to save disk space. 
> This approach was valuable in the early days where storage space was limited. 
>  However, the landscape has changed significantly.  Modern file systems, such 
> as ZFS, now offer native compression capabilities. Additionally, the 
> widespread availability of larger hard drives has diminished the necessity 
> for additional compression.  Notably, the need to decompress log files for 
> pattern searches poses a significant inconvenience, further questioning the 
> utility of this legacy feature.
> 
> In commit 906748d208d3, flags J, X, Y, Z can now indicate that a log file is 
> eligible for compression rather than directly enforcing it. It allows for a 
> more flexible approach, wherein the actual compression method can be set to 
> "none" or specified as one among bzip2, gzip, xz, or zstd.
> 
> Therefore I would propose that we change the default compression setting to 
> "none" in FreeBSD 15.0.  This change reflects our adaptation to the evolving 
> technological environment and user needs.  It also aligns with the broader 
> initiative to modernize our systems while maintaining flexibility and 
> efficiency.
> 
> I look forward to your thoughts and feedback on this proposal.

This impacts embedded systems or jails which use UFS as the default /var/log 
backed device. There are quite a few larger consumers of FreeBSD out there that 
still use UFS instead of ZFS.

Adding this instead into bsdinstall and the documentation as a suggested knob 
seems like a good way to go.

Just something to keep in mind when making this change.

Cheers,
-Enji

Re: Proposal: Disable compression of newsyslog by default

[Konstantin Belousov]

On Fri, Dec 22, 2023 at 11:18:23PM -0800, Xin Li wrote:
> Hi,
> 
> Inspired by D42961, I propose that we move forward with disabling the
> compression by default in newsyslog, as implemented in
> https://reviews.freebsd.org/D43169
> 
> Historically, newsyslog has compressed rotated log files to save disk space.
> This approach was valuable in the early days where storage space was
> limited.  However, the landscape has changed significantly.  Modern file
> systems, such as ZFS, now offer native compression capabilities.
> Additionally, the widespread availability of larger hard drives has
> diminished the necessity for additional compression.  Notably, the need to
> decompress log files for pattern searches poses a significant inconvenience,
> further questioning the utility of this legacy feature.
> 
> In commit 906748d208d3, flags J, X, Y, Z can now indicate that a log file is
> eligible for compression rather than directly enforcing it. It allows for a
> more flexible approach, wherein the actual compression method can be set to
> "none" or specified as one among bzip2, gzip, xz, or zstd.
> 
> Therefore I would propose that we change the default compression setting to
> "none" in FreeBSD 15.0.  This change reflects our adaptation to the evolving
> technological environment and user needs.  It also aligns with the broader
> initiative to modernize our systems while maintaining flexibility and
> efficiency.
> 
> I look forward to your thoughts and feedback on this proposal.

This is strange change at best.  I have no opinion about the disabling
of compression of the rotated logs by default, but we already have knobs
to do that.  Adding a knob that disables (or enables) other knobs to work
is weird.

If you want to change the compression, update the default configuration file.

Re: Proposal: Disable compression of newsyslog by default

[Marek Zarychta]

W dniu 23.12.2023 o 09:51, Miroslav Lachman pisze:

On 23/12/2023 08:18, Xin Li wrote:

Hi,

Inspired by D42961, I propose that we move forward with disabling the 
compression by default in newsyslog, as implemented in 
https://reviews.freebsd.org/D43169


Historically, newsyslog has compressed rotated log files to save disk 
space. This approach was valuable in the early days where storage 
space was limited.  However, the landscape has changed 
significantly.  Modern file systems, such as ZFS, now offer native 
compression capabilities. Additionally, the widespread availability 
of larger hard drives has diminished the necessity for additional 
compression.  Notably, the need to decompress log files for pattern 
searches poses a significant inconvenience, further questioning the 
utility of this legacy feature.


In commit 906748d208d3, flags J, X, Y, Z can now indicate that a log 
file is eligible for compression rather than directly enforcing it. 
It allows for a more flexible approach, wherein the actual 
compression method can be set to "none" or specified as one among 
bzip2, gzip, xz, or zstd.


Therefore I would propose that we change the default compression 
setting to "none" in FreeBSD 15.0.  This change reflects our 
adaptation to the evolving technological environment and user needs.  
It also aligns with the broader initiative to modernize our systems 
while maintaining flexibility and efficiency.


I look forward to your thoughts and feedback on this proposal.


Thank you for this change and future MFCs. Log files and an urge to 
process them differ a lot between setups. I guess some people would back 
completely disabling compression, and some would be against this 
proposal. There are also small VMs with limited storage formatted with 
UFS where current defaults: "-c legacy" and using bzip2 (J from 
newsyslog.conf) to compress logs seems to be perfectly suited.




I don't think anything needs to be changed on newsyslog. Those who 
want to disable compression can do so in the "default" newsyslog.conf 
file. Why force this change in the newsyslog code?
I also don't think that the log file should not be compressed even on 
a compressed filesystem. Compressed log files can be handled by tools 
like zcat, zless, zgrep, etc. without first decompressing the log file. 


When newsyslog has to deal only with small/medium files and logs are 
just stored, if processed then very rarely, the compression method is 
not very relevant, but the situation changes if you have to deal with 
larger log files, which have to be processed after rotation.


While testing this change, I have done a very simple benchmark with the 
largest log file which I have to rotate daily. It looks to me that 
having a compressed log file sometimes makes sense even if you have to 
process these logs in place. Please take a look at the result of running 
zcat over the compressed file and the timings for the original, 
uncompressed one. All the files are stored on ZFS lz4 compressed dataset 
lying on mirrored, not so fast, SSD drives.


[r-b] /tmp# /usr/bin/time zcat flowd.0.zst > /dev/null
    3.22 real 3.03 user 0.18 sys
[r-b] /tmp# /usr/bin/time zcat flowd.0.zst.orig  > /dev/null
    3.40 real 3.24 user 0.15 sys
[r-b] /tmp# /usr/bin/time zcat flowd.0.gz > /dev/null
    6.58 real 6.46 user 0.12 sys
[r-b] /tmp# /usr/bin/time zcat flowd.0.xz > /dev/null
   18.15 real    16.97 user 1.18 sys
[r-b] /tmp# /usr/bin/time zcat flowd.0.bz2 > /dev/null
   45.47 real    45.33 user 0.12 sys


[r-b] /tmp# /usr/bin/time cat flowd.0 > /dev/null
    3.35 real 0.25 user 3.10 sys


[r-b] /tmp# ls -l flowd.0*
-rw---  1 root wheel 3823756280 Dec 23 09:56 flowd.0
-rw---  1 root wheel  195036235 Dec 23 09:56 flowd.0.bz2
-rw---  1 root wheel  305250854 Dec 23 09:56 flowd.0.gz
-rw---  1 root wheel  180642412 Dec 23 09:56 flowd.0.xz
-rw---  1 root wheel  252470186 Dec 23 09:56 flowd.0.zst (zstd with 
--long --adapt -T0)
-rw---  1 root wheel  297184160 Dec 23 09:56 flowd.0.zst.orig (pre 
906748d208d3 zstd options)



Text log files can still grow to large sizes, and if you have a daily 
backup job over a long distance network, if you use a protocol without 
own compression, it is still better to have compressed log files than 
uncompressed on a compressed filesystem. To me, it's the same as 
compressing large database dumps and not relying on filesystem 
compression.
YMMV, but I really don't see any benefit of changing the newsyslog 
code, just change defaults in newsyslog.conf.


There is a clear benefit for me, users who don't want their logs to be 
compressed, will have just to alter one line in /etc/crontab:


# Rotate log files every hour, if necessary.
0    *    *    *    *    root    newsyslog -c none

instead of changing /etc/newsyslog.conf line by line.

Thank you for the new newsyslog(8) -c option providing th

Re: Proposal: Disable compression of newsyslog by default

[Miroslav Lachman]

On 23/12/2023 08:18, Xin Li wrote:

Hi,

Inspired by D42961, I propose that we move forward with disabling the 
compression by default in newsyslog, as implemented in 
https://reviews.freebsd.org/D43169


Historically, newsyslog has compressed rotated log files to save disk 
space. This approach was valuable in the early days where storage space 
was limited.  However, the landscape has changed significantly.  Modern 
file systems, such as ZFS, now offer native compression capabilities. 
Additionally, the widespread availability of larger hard drives has 
diminished the necessity for additional compression.  Notably, the need 
to decompress log files for pattern searches poses a significant 
inconvenience, further questioning the utility of this legacy feature.


In commit 906748d208d3, flags J, X, Y, Z can now indicate that a log 
file is eligible for compression rather than directly enforcing it. It 
allows for a more flexible approach, wherein the actual compression 
method can be set to "none" or specified as one among bzip2, gzip, xz, 
or zstd.


Therefore I would propose that we change the default compression setting 
to "none" in FreeBSD 15.0.  This change reflects our adaptation to the 
evolving technological environment and user needs.  It also aligns with 
the broader initiative to modernize our systems while maintaining 
flexibility and efficiency.


I look forward to your thoughts and feedback on this proposal.


I don't think anything needs to be changed on newsyslog. Those who want 
to disable compression can do so in the "default" newsyslog.conf file. 
Why force this change in the newsyslog code?
I also don't think that the log file should not be compressed even on a 
compressed filesystem. Compressed log files can be handled by tools like 
zcat, zless, zgrep, etc. without first decompressing the log file. Text 
log files can still grow to large sizes, and if you have a daily backup 
job over a long distance network, if you use a protocol without own 
compression, it is still better to have compressed log files than 
uncompressed on a compressed filesystem. To me, it's the same as 
compressing large database dumps and not relying on filesystem compression.
YMMV, but I really don't see any benefit of changing the newsyslog code, 
just change defaults in newsyslog.conf.


Kind regards
Miroslav Lachman

Re: Proposal: Disable compression of newsyslog by default

[Fabian Keil]

Xin Li  wrote on 2023-12-22 at 23:18:23:

> Inspired by D42961, I propose that we move forward with disabling the 
> compression by default in newsyslog, as implemented in 
> https://reviews.freebsd.org/D43169
[...]
> Therefore I would propose that we change the default compression setting 
> to "none" in FreeBSD 15.0.  This change reflects our adaptation to the 
> evolving technological environment and user needs.  It also aligns with 
> the broader initiative to modernize our systems while maintaining 
> flexibility and efficiency.
> 
> I look forward to your thoughts and feedback on this proposal.

In ElectroBSD I simply removed the J flags in newsyslog.conf in 2021 [0].

I have no strong feelings about it, but it's unclear to me
why newsyslog[8] itself needs to be changed.

Fabian

[0] 

Proposal: Disable compression of newsyslog by default

[Xin Li]

Hi,

Inspired by D42961, I propose that we move forward with disabling the 
compression by default in newsyslog, as implemented in 
https://reviews.freebsd.org/D43169


Historically, newsyslog has compressed rotated log files to save disk 
space. This approach was valuable in the early days where storage space 
was limited.  However, the landscape has changed significantly.  Modern 
file systems, such as ZFS, now offer native compression capabilities. 
Additionally, the widespread availability of larger hard drives has 
diminished the necessity for additional compression.  Notably, the need 
to decompress log files for pattern searches poses a significant 
inconvenience, further questioning the utility of this legacy feature.


In commit 906748d208d3, flags J, X, Y, Z can now indicate that a log 
file is eligible for compression rather than directly enforcing it. It 
allows for a more flexible approach, wherein the actual compression 
method can be set to "none" or specified as one among bzip2, gzip, xz, 
or zstd.


Therefore I would propose that we change the default compression setting 
to "none" in FreeBSD 15.0.  This change reflects our adaptation to the 
evolving technological environment and user needs.  It also aligns with 
the broader initiative to modernize our systems while maintaining 
flexibility and efficiency.


I look forward to your thoughts and feedback on this proposal.

Cheers,


OpenPGP_signature.asc
Description: OpenPGP digital signature