Re: a proper-unix meta package (Re: Proposal: plocate as standard for bookworm)

[Peter Pentchev]

On Wed, Apr 14, 2021 at 10:48:29PM +, Holger Levsen wrote:
> On Thu, Mar 25, 2021 at 04:17:01PM +, Jonathan Dowland wrote:
> > I once considered packaging sed from v7 UNIX, which has all the
> > coreutils bundled in one source repo, and so literally considered
> > src:unix, although in both cases it was largely a joke and I didn't
> > pursue it.
>  
> *g*, nice!
> 
> I'd also be happy if this were src:unix (or maybe not) as long as there
> would be a src:proper-unix-system meta-package for those who care...

That would indeed be great, and thanks, jmtd, for your work! (even if
you ultimately decide not to complete it, it's still cool)

...and then there comes the "...but does Caldera really have the right
to those sources?" question, which may or may not have been already
settled in the US courts (there are people who say "the decision that
one specific action did not mean that SCO owns the copyright still
does not necessarily mean that there are no other reasons for
SCO-name-of-the-month to own the copyright")...

Sorry... some people do say that I sometimes worry too much and throw
wet blankets on others' enthusiasm... :/

G'luck,
Peter

-- 
Peter Pentchev  r...@ringlet.net r...@debian.org p...@storpool.com
PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13


signature.asc
Description: PGP signature

Re: a proper-unix meta package (Re: Proposal: plocate as standard for bookworm)

[Holger Levsen]

On Thu, Mar 25, 2021 at 04:17:01PM +, Jonathan Dowland wrote:
> I once considered packaging sed from v7 UNIX, which has all the
> coreutils bundled in one source repo, and so literally considered
> src:unix, although in both cases it was largely a joke and I didn't
> pursue it.
 
*g*, nice!

I'd also be happy if this were src:unix (or maybe not) as long as there
would be a src:proper-unix-system meta-package for those who care...


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁   holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀   OpenPGP: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
 ⠈⠳⣄

Fascism is not just an opinion, it usually becomes a capital crime.


signature.asc
Description: PGP signature

Re: a proper-unix meta package (Re: Proposal: plocate as standard for bookworm)

[Jonathan Dowland]

On Wed, Feb 10, 2021 at 12:21:36PM +, Holger Levsen wrote:
> As it's 2021 and 99% of the Linux user base has no idea what UNIX (or Linux)
> is, maybe it's time for a src:proper-unix-system package for those who care?

I once considered packaging sed from v7 UNIX, which has all the
coreutils bundled in one source repo, and so literally considered
src:unix, although in both cases it was largely a joke and I didn't
pursue it.

Oh, that's still lying around somewhere:
 although not with
src:unix. (And the credit getting sed to *build* as well as the initial
packaging, goes to David Jones)



-- 
👱🏻Jonathan Dowland
✎  j...@debian.org
🔗https://jmtd.net

Re: Proposal: plocate as standard for bookworm

[Steinar H. Gunderson]

On Mon, Feb 22, 2021 at 09:50:25AM +, Holger Levsen wrote:
> I'll still oppose installing a locate package on every system by default
> which burns energy every day on millions of computers for almost noone's
> benefit.
> 
> Those who want locate, can apt install it. The rest shouldn't default to
> having more toxic computers than need be. Energy wasted is toxic for this
> planet and thus all of us.

While I appreciate the idea, the amount of power burned every day by updatedb
is microscopic on any reasonable scale (on the order of “if I turn down the
temperature in my living room by one degree, it will probably offset 100k
Debian installations running updatedb every night”).

If we really want to make energy use an overriding concern in Debian,
there are _much_ more impactful things we should do, such as energy-tuning
our default install, or optimizing dpkg so it no longer uses several
CPU-seconds every time it wants to upgrade a package. Or throwing out a few
architectures so we don't waste power on their buildds.

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: Proposal: plocate as standard for bookworm

[Holger Levsen]

On Sun, Feb 21, 2021 at 08:52:44PM +0100, Michael Biebl wrote:
> plocate already ships a native .timer unit, so all it would take is to add
> RandomizedDelaySec= to plocate-updatedb.timer with a reasonably chosen
> value.

nice.

I'll still oppose installing a locate package on every system by default
which burns energy every day on millions of computers for almost noone's
benefit.

Those who want locate, can apt install it. The rest shouldn't default to
having more toxic computers than need be. Energy wasted is toxic for this
planet and thus all of us.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁   holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀ PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
 ⠈⠳⣄

There are no jobs on a dead planet.


signature.asc
Description: PGP signature

Re: Proposal: plocate as standard for bookworm

[Michael Biebl]

Am 19.02.21 um 15:50 schrieb Marc Haber:

On Fri, 19 Feb 2021 10:34:45 +0100, Florian Lohoff  wrote:

All locate variants are a PITA on servers, especially when virtualized.
Imagine a 100+ VM hypervisor at 6:30 starting an updatedb job on all VMs
in parallel. I debugged stuff like that for customers so yes - This
problem is real.


We really need such things to run from systemd-cron with a sufficient
random spread to defuse this kind of load spikes in virtualized
environments, yes.


plocate already ships a native .timer unit, so all it would take is to 
add RandomizedDelaySec= to plocate-updatedb.timer with a reasonably 
chosen value.

https://www.freedesktop.org/software/systemd/man/systemd.timer.html#RandomizedDelaySec=

Regards,
Michael



OpenPGP_signature
Description: OpenPGP digital signature

Re: Proposal: plocate as standard for bookworm

[Marc Haber]

On Fri, 19 Feb 2021 10:34:45 +0100, Florian Lohoff  wrote:
>All locate variants are a PITA on servers, especially when virtualized. 
>Imagine a 100+ VM hypervisor at 6:30 starting an updatedb job on all VMs
>in parallel. I debugged stuff like that for customers so yes - This
>problem is real.

We really need such things to run from systemd-cron with a sufficient
random spread to defuse this kind of load spikes in virtualized
environments, yes.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: Proposal: plocate as standard for bookworm

[Florian Lohoff]

On Sat, Feb 13, 2021 at 02:15:17PM -0800, Noah Meyerhans wrote:
> On Mon, Feb 08, 2021 at 07:28:56PM +0100, Richard Hartmann wrote:
> > I very dimly remember updatedb being a concern when cloud images were
> > first discussed. Back then and today, agreed, it does not make sense
> > there.
> 
> Agreed, but we don't install all Priority: standard packages on the
> cloud images anyway, and I don't see us going out of our way to add it
> to them even if plocate is promoted to standard.
> 
> > IMO, it makes sense on both servers and desktops, so rather than
> > through tasksel, I would think it's a useful default to have on all
> > non-virtual installations.
> 
> Personally I'd rather leave it out of the default install, and I really
> don't like the idea of running it on servers by default.  First, the
> additional IO may impact serving latencies.  Second, because servers
> (except maybe multi-user shell servers, but they're not the general
> case) are purpose-built systems, and the locate utility really doesn't
> contribute anything to the system's purpose.

All locate variants are a PITA on servers, especially when virtualized. 
Imagine a 100+ VM hypervisor at 6:30 starting an updatedb job on all VMs
in parallel. I debugged stuff like that for customers so yes - This
problem is real.

My ansible "essential" roles purge all variants of locate.

Flo
-- 
Florian Lohoff f...@zz.de


signature.asc
Description: PGP signature

Re: Proposal: plocate as standard for bookworm

[Noah Meyerhans]

On Mon, Feb 08, 2021 at 07:28:56PM +0100, Richard Hartmann wrote:
> I very dimly remember updatedb being a concern when cloud images were
> first discussed. Back then and today, agreed, it does not make sense
> there.

Agreed, but we don't install all Priority: standard packages on the
cloud images anyway, and I don't see us going out of our way to add it
to them even if plocate is promoted to standard.

> IMO, it makes sense on both servers and desktops, so rather than
> through tasksel, I would think it's a useful default to have on all
> non-virtual installations.

Personally I'd rather leave it out of the default install, and I really
don't like the idea of running it on servers by default.  First, the
additional IO may impact serving latencies.  Second, because servers
(except maybe multi-user shell servers, but they're not the general
case) are purpose-built systems, and the locate utility really doesn't
contribute anything to the system's purpose.

noah

Re: a proper-unix meta package (Re: Proposal: plocate as standard for bookworm)

[Gunnar Wolf]

Jonathan Carter dijo [Wed, Feb 10, 2021 at 07:29:14PM +0200]:
> >> Define "proper Unix"...
> > 
> > The definition depends on whether you are a longhair or shorthair.
> 
> If you're a proper blue-haired person, then the only proper Unix is Debian.

Please do note that your definition might be of sufficiency, but is
–to use our common parlance– a Suggests: or, at most, a
Recommends:. It has been observed there are many example cases where
hair style is orthogonal to Unix preferences.

Re: Proposal: plocate as standard for bookworm

[Russ Allbery]

Adrian Bunk  writes:
> On Tue, Feb 09, 2021 at 12:02:33PM -0800, Russ Allbery wrote:
>> Adrian Bunk  writes:
>>> On Mon, Feb 08, 2021 at 11:13:10PM +0100, Steinar H. Gunderson wrote:

 users on shared systems can expect it to be available without asking
 an administrator first. To me, locate has always been a standard
 tool on a UNIX system, so it makes sense to install it by default.

>>> "Shared systems" have become pretty rare.

>> They've become *rarer*, but they're still very common in the academic and
>> scientific research world.

> Let's not nitpick over words, I assume we can both agree that they still
> exist but are a much smaller share of Unix-like systems than 30 years ago.

I'm not nitpicking; I think shared systems are still common enough that
this is one of several reasons why we should install a locate
implementation by default.

> When I think of hardware running Debian in 2021,
> I am thinking of a Raspberry Pi running on an SD card.

Every person is different.

-- 
Russ Allbery (r...@debian.org)  

Re: Proposal: plocate as standard for bookworm

[Adrian Bunk]

On Tue, Feb 09, 2021 at 12:02:33PM -0800, Russ Allbery wrote:
> Adrian Bunk  writes:
> > On Mon, Feb 08, 2021 at 11:13:10PM +0100, Steinar H. Gunderson wrote:
> >>...
> >> users on
> >> shared systems can expect it to be available without asking an 
> >> administrator
> >> first. To me, locate has always been a standard tool on a UNIX system, so 
> >> it
> >> makes sense to install it by default.
> >>...
> 
> > "Shared systems" have become pretty rare.
> 
> They've become *rarer*, but they're still very common in the academic and
> scientific research world.

Let's not nitpick over words, I assume we can both agree that they still
exist but are a much smaller share of Unix-like systems than 30 years ago.

When I think of hardware running Debian in 2021,
I am thinking of a Raspberry Pi running on an SD card.

cu
Adrian

Re: a proper-unix meta package (Re: Proposal: plocate as standard for bookworm)

[Jonathan Carter]

On 2021/02/10 15:16, Bjørn Mork wrote:
> Adam Borowski  writes:
>>> As it's 2021 and 99% of the Linux user base has no idea what UNIX (or Linux)
>>> is, maybe it's time for a src:proper-unix-system package for those who care?
>>
>> Define "proper Unix"...
> 
> The definition depends on whether you are a longhair or shorthair.

If you're a proper blue-haired person, then the only proper Unix is Debian.

-Jonathan

Re: Proposal: plocate as standard for bookworm

[Vincent Lefevre]

On 2021-02-09 22:12:31 +0100, Ansgar wrote:
> "Steinar H. Gunderson" writes:
> > On Sat, Feb 06, 2021 at 10:16:29PM -0800, Josh Triplett wrote:
> >> Furthermore, any mechanism they use to configure one of them
> >> (e.g. for privacy or performance reasons) will not control the other,
> >> and again they may well be unaware of the existence of the other one.
> >
> > I'm not sure what privacy reasons you're referring to? I'm not aware that
> > neither mlocate/plocate nor e.g. tracker will leak data across users.
> 
> If you have an encrypted /home (or /home/), but unencrypted
> /var/lib/plocate, you leak information about the encrypted files.

This would be the user's fault. I can also see /home/ "leaks"
in log files. And mail "leaks" under /var/mail. So, in short, /var
should be encrypted.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: a proper-unix meta package (Re: Proposal: plocate as standard for bookworm)

[Holger Levsen]

On Wed, Feb 10, 2021 at 01:38:56PM +0100, Adam Borowski wrote:
> > As it's 2021 and 99% of the Linux user base has no idea what UNIX (or Linux)
> > is, maybe it's time for a src:proper-unix-system package for those who care?
> Define "proper Unix"...

well, I'd leave this to the people who care.

But surely there could be several flavors, maybe even coming from different
source packages.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁   holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀ PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
 ⠈⠳⣄

Everyone is entitled to their own opinion, but not their own facts.


signature.asc
Description: PGP signature

Re: a proper-unix meta package (Re: Proposal: plocate as standard for bookworm)

[Ansgar]

On Wed, 2021-02-10 at 13:38 +0100, Adam Borowski wrote:
> Define "proper Unix"...

A system including Emacs.  So we would need emacs at Priority: standard
or even important or required :]

Ansgar

Re: a proper-unix meta package (Re: Proposal: plocate as standard for bookworm)

[Bjørn Mork]

Adam Borowski  writes:
> On Wed, Feb 10, 2021 at 12:21:36PM +, Holger Levsen wrote:
>> On Wed, Feb 10, 2021 at 01:05:04PM +0100, Bjørn Mork wrote:
>> > I happen to disagree.  To me this is yet another step away from being a
>> > proper Unix system - to something else.  Which would be fine if it moved
>> > us forward.
>> 
>> As it's 2021 and 99% of the Linux user base has no idea what UNIX (or Linux)
>> is, maybe it's time for a src:proper-unix-system package for those who care?
>
> Define "proper Unix"...

The definition depends on whether you are a longhair or shorthair.


Bjørn

Re: a proper-unix meta package (Re: Proposal: plocate as standard for bookworm)

[Adam Borowski]

On Wed, Feb 10, 2021 at 12:21:36PM +, Holger Levsen wrote:
> On Wed, Feb 10, 2021 at 01:05:04PM +0100, Bjørn Mork wrote:
> > I happen to disagree.  To me this is yet another step away from being a
> > proper Unix system - to something else.  Which would be fine if it moved
> > us forward.
> 
> As it's 2021 and 99% of the Linux user base has no idea what UNIX (or Linux)
> is, maybe it's time for a src:proper-unix-system package for those who care?

Define "proper Unix"...


-- 
⢀⣴⠾⠻⢶⣦⠀ .globl _start↵.data↵rc: .ascii "/etc/init.d/rcS\0"↵.text↵_start
⣾⠁⢠⠒⠀⣿⡁ mov $57,%rax↵syscall↵cmp $0,%rax↵jne child↵parent:↵mov $61,%rax
⢿⡄⠘⠷⠚⠋⠀ mov $-1,%rdi↵xor %rsi,%rsi↵xor %rdx,%rdx↵syscall↵jmp parent↵child:
⠈⠳⣄ mov $59,%rax↵mov $rc,%rdi↵xor %rsi,%rsi↵xor %rdx,%rdx↵syscall

a proper-unix meta package (Re: Proposal: plocate as standard for bookworm)

[Holger Levsen]

On Wed, Feb 10, 2021 at 01:05:04PM +0100, Bjørn Mork wrote:
> I happen to disagree.  To me this is yet another step away from being a
> proper Unix system - to something else.  Which would be fine if it moved
> us forward.

As it's 2021 and 99% of the Linux user base has no idea what UNIX (or Linux)
is, maybe it's time for a src:proper-unix-system package for those who care?


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁   holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀ PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
 ⠈⠳⣄

Life is short but a sea of morons is forever.


signature.asc
Description: PGP signature

Re: Proposal: plocate as standard for bookworm

[Bjørn Mork]

Marvin Renich  writes:
> * Steinar H. Gunderson  [210209 14:27]:
>> On Tue, Feb 09, 2021 at 08:53:10PM +0200, Adrian Bunk wrote:
>> > And there are now also many non-technical Linux users who have never
>> > used a shell.
>> 
>> Well, why do we include netcat, telnet or hdparm? lsof? pciutils?
>> traceroute? host? All of these are irrelevant for a non-technical
>> non-shell user, yet a fairly common part of a Linux installation.
>
> These have come to be expected to be on a typical Linux system by almost
> every technically-knowledgeable Linux user.  Locate does not satisfy
> that criterion, and I think the dissension in this thread is evidence of
> that.

This is subjective.  I don't think arguing about the expectations of
"every technically-knowledgeable Linux user" is going to bring the
discussion anywhere.

FWIW, locate was "always" there.  Before Linux and before PCI.
See https://en.wikipedia.org/wiki/Locate_(Unix).

> However, I agree with others here that anyone who wants a locate
> implementation will know how to find all the packages with "locate" in
> their names and plocate looks to me, from the descriptions, to be the
> best choice.  I don't think it deserves "Priority: standard".

I happen to disagree.  To me this is yet another step away from being a
proper Unix system - to something else.  Which would be fine if it moved
us forward.

But removing functionality can hardly be argued to be a move in the
forward direction?

Sure, I can install mlocate or plocate.  But it's the kind of tool you
expect to be present, with an uptodate database, when you need it.  The
database makes locate different from all the other tools you mention.

I also use Linux systems I don't admininstrate.  I'd hate to have to ask
the admin for every basic Unix tool I need.  Some of the standard tools
you mention are only relevant for admins.  Those don't need to be
standard.  But the ones that are relevant for users should be.


Bjørn

Re: Proposal: plocate as standard for bookworm

[Andrei POPESCU]

On Lu, 08 feb 21, 23:13:10, Steinar H. Gunderson wrote:
> 
> The downside is, of course, the nightly updatedb job, which it is rare that
> anyone will notice, and the space used. 

That's assuming the system is left running over night. It might be 
noticeable if run on next system start.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser


signature.asc
Description: PGP signature

Re: Proposal: plocate as standard for bookworm

[Andrei POPESCU]

On Mi, 10 feb 21, 12:24:20, Andrey Rahmatullin wrote:
> On Tue, Feb 09, 2021 at 03:58:39PM -0500, Marvin Renich wrote:
> > These have come to be expected to be on a typical Linux system by almost
> > every technically-knowledgeable Linux user.  Locate does not satisfy
> > that criterion
> (I'm surprised by this, if this is true)

Possibly a consequence of the portability problem, advice on debian-user 
will almost always[1] use 'find' whenever the problem / solution 
involves searching for files.

For myself, I abandoned 'locate' as soon as I learned 'find', even of 
low power systems with storage on SD cards or connected via USB2.

[1] as in: can't remember *any* occurrence in recent years

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser


signature.asc
Description: PGP signature

Re: Proposal: plocate as standard for bookworm

[Andrey Rahmatullin]

On Tue, Feb 09, 2021 at 03:58:39PM -0500, Marvin Renich wrote:
> These have come to be expected to be on a typical Linux system by almost
> every technically-knowledgeable Linux user.  Locate does not satisfy
> that criterion
(I'm surprised by this, if this is true)

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Proposal: plocate as standard for bookworm

[Paul Wise]

On Tue, Feb 9, 2021 at 9:12 PM Ansgar wrote:

> Admittedly Debian's other defaults like making every file in $HOME
> world-readable by default are very unfriendly too on both multi-user
> systems (obviously) and single-user systems where suddenly even the
> "nobody" user has access to lots of interesting files...

Ubuntu are changing this, perhaps Debian should too for bookworm:

https://ubuntu.com/blog/private-home-directories-for-ubuntu-21-04

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Proposal: plocate as standard for bookworm

[Ansgar]

"Steinar H. Gunderson" writes:
> On Sat, Feb 06, 2021 at 10:16:29PM -0800, Josh Triplett wrote:
>> Furthermore, any mechanism they use to configure one of them
>> (e.g. for privacy or performance reasons) will not control the other,
>> and again they may well be unaware of the existence of the other one.
>
> I'm not sure what privacy reasons you're referring to? I'm not aware that
> neither mlocate/plocate nor e.g. tracker will leak data across users.

If you have an encrypted /home (or /home/), but unencrypted
/var/lib/plocate, you leak information about the encrypted files.

File indexing services run as a user would at least write only to
/home/ which would be encrypted.

Admittedly Debian's other defaults like making every file in $HOME
world-readable by default are very unfriendly too on both multi-user
systems (obviously) and single-user systems where suddenly even the
"nobody" user has access to lots of interesting files...

Ansgar

Re: Proposal: plocate as standard for bookworm

[Marvin Renich]

* Steinar H. Gunderson  [210209 14:27]:
> On Tue, Feb 09, 2021 at 08:53:10PM +0200, Adrian Bunk wrote:
> > And there are now also many non-technical Linux users who have never
> > used a shell.
> 
> Well, why do we include netcat, telnet or hdparm? lsof? pciutils?
> traceroute? host? All of these are irrelevant for a non-technical
> non-shell user, yet a fairly common part of a Linux installation.

These have come to be expected to be on a typical Linux system by almost
every technically-knowledgeable Linux user.  Locate does not satisfy
that criterion, and I think the dissension in this thread is evidence of
that.  Most of the ones you mention above might be necessary when trying
to fix a broken system.  Locate might save a little time, but find,
which is standard, will suffice for that purpose.

I just tried plocate, and it certainly looks faster to update than
mlocate.  Thanks for packaging this!  I've just removed mlocate.

However, I agree with others here that anyone who wants a locate
implementation will know how to find all the packages with "locate" in
their names and plocate looks to me, from the descriptions, to be the
best choice.  I don't think it deserves "Priority: standard".

...Marvin

Re: Proposal: plocate as standard for bookworm

[Russ Allbery]

Adrian Bunk  writes:
> On Mon, Feb 08, 2021 at 11:13:10PM +0100, Steinar H. Gunderson wrote:
>>...
>> users on
>> shared systems can expect it to be available without asking an administrator
>> first. To me, locate has always been a standard tool on a UNIX system, so it
>> makes sense to install it by default.
>>...

> "Shared systems" have become pretty rare.

They've become *rarer*, but they're still very common in the academic and
scientific research world.

-- 
Russ Allbery (r...@debian.org)  

Re: Proposal: plocate as standard for bookworm

[Steinar H. Gunderson]

On Tue, Feb 09, 2021 at 08:53:10PM +0200, Adrian Bunk wrote:
> And there are now also many non-technical Linux users who have never
> used a shell.

Well, why do we include netcat, telnet or hdparm? lsof? pciutils?
traceroute? host? All of these are irrelevant for a non-technical
non-shell user, yet a fairly common part of a Linux installation.

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: Proposal: plocate as standard for bookworm

[Adrian Bunk]

On Mon, Feb 08, 2021 at 11:13:10PM +0100, Steinar H. Gunderson wrote:
>...
> users on
> shared systems can expect it to be available without asking an administrator
> first. To me, locate has always been a standard tool on a UNIX system, so it
> makes sense to install it by default.
>...

"Shared systems" have become pretty rare.

And there are now also many non-technical Linux users who have never
used a shell.

cu
Adrian

Re: Proposal: plocate as standard for bookworm

[Steinar H. Gunderson]

On Mon, Feb 08, 2021 at 01:57:42PM -0800, Josh Triplett wrote:
> That doesn't make it usable for portable scripting. A script can't
> assume the presence of locate without declaring a dependency on it, and
> there's currently no virtual package for locate. And even if locate is
> present, a script can't assume that it'll return useful results, because
> it may not have an updated database, or even have a database initialized
> at all.

OK, but that's a very strict bar to set for to be useful for scripting.
locate is not a POSIX tool, and doesn't try to use be that either;
you wouldn't use it in an installer meant to run on foreign computers.
But in a controlled environment, you can build utility scripts around locate
-- and I know of several people that do.

> Statistics elsewhere in this thread suggest that it's not that cheap
> even on a fast system with an SSD. It's likely to be slower on systems
> with spinning disks. That's leaving aside the effect on caches.

Well, given that Paul didn't share the number of files on his system on-list,
I won't repeat it here; but let me suggest that it is way up on the
99-percentile of what I'd ever expect a desktop machine to have.
And yet, the cron job finishes in 40 seconds, during which the machine is
perfectly usable.

An interesting question is what _would_ be acceptable speed and resource use?

> What is the net benefit of having plocate installed by default (that
> isn't available by installing it)? What is the net drawback? Why does
> the former outweigh the latter?

Just like with any other package: It's more discoverable (how would any user
know that “apt install locate” is the locate you _least_ want?), and users on
shared systems can expect it to be available without asking an administrator
first. To me, locate has always been a standard tool on a UNIX system, so it
makes sense to install it by default.

The downside is, of course, the nightly updatedb job, which it is rare that
anyone will notice, and the space used. For my laptop, the latter is 19 MB;
for comparison, the systemd journal on the same machine (which is left on
buster defaults, and that I never use) is 2.8 GB :-)

> It's a database of files that previously existed on the system. If users
> are unaware of that database, it may leak information they did not
> expect. For instance, after deleting a file or unmounting a device, the
> locate database may still have a record of such files.

To address this specific issue: Once the file is gone, neither plocate nor
mlocate will show it anymore. If you are root, you can go into the database,
of course, but if so, you could also undelete files fairly reliably or just
stop it from being deleted in the first place.

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: Proposal: plocate as standard for bookworm

[Josh Triplett]

Steinar H. Gunderson wrote:
> On Sat, Feb 06, 2021 at 10:16:29PM -0800, Josh Triplett wrote:
> > locate is a purely user-facing tool,
> > not really usable for portable scripting, since neither its presence nor
> > its functioning can be assumed.
> 
> Really? Basic functionality is the same between locate.findutils, mlocate and
> plocate. Its presence can be resolved by... well, making it Priority:
> standard. :-)

That doesn't make it usable for portable scripting. A script can't
assume the presence of locate without declaring a dependency on it, and
there's currently no virtual package for locate. And even if locate is
present, a script can't assume that it'll return useful results, because
it may not have an updated database, or even have a database initialized
at all.

> > Many users won't even know it exists
> > (locate has far fewer users than find), and for all of those non-users,
> > the effort spent building the database will go entirely to waste.
> 
> Sure, but that waste is fairly small. On a typical system, you're using a few
> seconds every night.

Statistics elsewhere in this thread suggest that it's not that cheap
even on a fast system with an SSD. It's likely to be slower on systems
with spinning disks. That's leaving aside the effect on caches.

That may be a reasonable cost to pay for users who actively use locate.
It's not a cost we should impose on the majority of Debian installations
just so that someone can run locate without installing it first.

The defaults need to cater for 1) the broadest set of users, and 2)
users who are less likely to change the defaults. Most users don't run
locate, and those who do are more likely to be users who can and do
change the defaults. `apt install plocate` isn't hard for someone who
uses locate to do.

I think you've made a compelling case for plocate being the default
locate. And that already seems to be the case: plocate provides a
higher-priority alternative than mlocate or locate.

What is the net benefit of having plocate installed by default (that
isn't available by installing it)? What is the net drawback? Why does
the former outweigh the latter?

> > Furthermore, any mechanism they use to configure one of them
> > (e.g. for privacy or performance reasons) will not control the other,
> > and again they may well be unaware of the existence of the other one.
> 
> I'm not sure what privacy reasons you're referring to? I'm not aware that
> neither mlocate/plocate nor e.g. tracker will leak data across users.

It's a database of files that previously existed on the system. If users
are unaware of that database, it may leak information they did not
expect. For instance, after deleting a file or unmounting a device, the
locate database may still have a record of such files.

Re: Proposal: plocate as standard for bookworm

[Richard Hartmann]

On Sat, Feb 6, 2021 at 6:29 PM Steinar H. Gunderson  wrote:

> Now, there is plocate (written and maintained by myself). It is orders of
> magnitude faster than mlocate

I use plocate on most days. Directly installed on laptops and
desktops, and via backports on all my file servers.


> I believe a good, fast locate is something that we should have in our base
> install; it is fine to keep it out of the cloud image (as today), but it is
> genuinely useful for both desktops and servers, and with a low cost.

I very dimly remember updatedb being a concern when cloud images were
first discussed. Back then and today, agreed, it does not make sense
there.

IMO, it makes sense on both servers and desktops, so rather than
through tasksel, I would think it's a useful default to have on all
non-virtual installations.


Richard

Re: Proposal: plocate as standard for bookworm

[Steinar H. Gunderson]

On Sun, Feb 07, 2021 at 03:12:25PM +, Paul Wise wrote:
> On my desktop a no-change update takes 40s and the I/O usage is around
> 1800 K/s according to iotop-c, probably would be more painful on
> HDD-only systems.

That's interesting; how many files do you have on your machine, roughly?
(e.g. plocate '' | wc -l)

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: Proposal: plocate as standard for bookworm

[Paul Wise]

On Sun, Feb 7, 2021 at 9:58 AM Steinar H. Gunderson wrote:

> On my server, with ~12M files on rotating media, updatedb.plocate finishes in
> ~40 seconds, IIRC. (I'd re-check to be sure, but today is RAID Sunday... :-) )
> On my laptop with 875k files and a regular SSD, it's less than three. It does
> alter the filesystem cache somewhat, but the amount of RAM used can be
> bounded cgroups if desired; it runs from a systemd service.

On my desktop a no-change update takes 40s and the I/O usage is around
1800 K/s according to iotop-c, probably would be more painful on
HDD-only systems. If you multiply that by all the Debian systems on
Earth it seems like a substantial amount of resource use.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Proposal: plocate as standard for bookworm

[Steinar H. Gunderson]

On Sat, Feb 06, 2021 at 10:16:29PM -0800, Josh Triplett wrote:
> locate is a purely user-facing tool,
> not really usable for portable scripting, since neither its presence nor
> its functioning can be assumed.

Really? Basic functionality is the same between locate.findutils, mlocate and
plocate. Its presence can be resolved by... well, making it Priority:
standard. :-)

> Many users won't even know it exists
> (locate has far fewer users than find), and for all of those non-users,
> the effort spent building the database will go entirely to waste.

Sure, but that waste is fairly small. On a typical system, you're using a few
seconds every night.

> Furthermore, any mechanism they use to configure one of them
> (e.g. for privacy or performance reasons) will not control the other,
> and again they may well be unaware of the existence of the other one.

I'm not sure what privacy reasons you're referring to? I'm not aware that
neither mlocate/plocate nor e.g. tracker will leak data across users.

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: Proposal: plocate as standard for bookworm

[Holger Levsen]

On Sat, Feb 06, 2021 at 10:16:29PM -0800, Josh Triplett wrote:
> I absolutely think that plocate makes sense as the "default" locate; it
> seems like an improvement on mlocate in every way.
> 
> However, I don't think *any* locate should be in the base install, as
> long as that entails having any kind of regularly scheduled task that
> indexes the filesystem [...]
 
agreed on everything. It's easy to install if needed/wanted.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁   holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀ PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
 ⠈⠳⣄

Make racists afraid again.


signature.asc
Description: PGP signature

Re: Proposal: plocate as standard for bookworm

[Marco d'Itri]

On Feb 06, "Steinar H. Gunderson"  wrote:

> mlocate used to be Priority: standard; for some reason that I haven't been
> able to unearth (despite the efforts of several people), there is now an
Probably because long ago it replaced locate from findutils which used 
to be standard too.

> release and non-release architectures. The only non-Essential dependencies
> are liburing1 (33 kB) and libzstd1 (670 kB).
libzstd1 is already a dependency of apt, so no big deal.

> I believe a good, fast locate is something that we should have in our base
> install; it is fine to keep it out of the cloud image (as today), but it is
> genuinely useful for both desktops and servers, and with a low cost.
I agree.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Proposal: plocate as standard for bookworm

[Steinar H. Gunderson]

On Sun, Feb 07, 2021 at 12:40:55AM +, Paul Wise wrote:
> I support having locate in the base install, but I don't think that
> the cost of daily walking the entire filesystem is low; especially
> with HDDs and older storage or computers that can be a lot of I/O. I
> guess it also alters the Linux kernel block/filesystem caches?

updatedb.plocate, like updatedb.mlocate, doesn't actually watch the entire
filesystem. It keeps track of the mtime of each directory, and doesn't do the
readdir()/getdents() if it hasn't changed. So you get the cost of stat-ing
every directory, but that's actually fairly low.

On my server, with ~12M files on rotating media, updatedb.plocate finishes in
~40 seconds, IIRC. (I'd re-check to be sure, but today is RAID Sunday... :-) ) 
On my laptop with 875k files and a regular SSD, it's less than three. It does
alter the filesystem cache somewhat, but the amount of RAM used can be
bounded cgroups if desired; it runs from a systemd service.

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: Proposal: plocate as standard for bookworm

[Josh Triplett]

Steinar H. Gunderson wrote:
> Now, there is plocate (written and maintained by myself). It is orders of
> magnitude faster than mlocate (both on SSDs and HDDs alike), has the same
> security model, a smaller database (typically half the size), and fixes
> a few long-standing mlocate bugs. It is nearly fully command-line compatible
> with mlocate, so most users should feel right at home. It builds on all
> release and non-release architectures. The only non-Essential dependencies
> are liburing1 (33 kB) and libzstd1 (670 kB).
> 
> I believe a good, fast locate is something that we should have in our base
> install

I absolutely think that plocate makes sense as the "default" locate; it
seems like an improvement on mlocate in every way.

However, I don't think *any* locate should be in the base install, as
long as that entails having any kind of regularly scheduled task that
indexes the filesystem, even if that task has been made relatively
cheaper than other implementations. locate is a purely user-facing tool,
not really usable for portable scripting, since neither its presence nor
its functioning can be assumed. Many users won't even know it exists
(locate has far fewer users than find), and for all of those non-users,
the effort spent building the database will go entirely to waste.

On top of that, several desktop environments (including a default
"desktop" installation) have their own entirely separate mechanism for
indexing files, typically based on a change-watching API (e.g. inotify)
rather than a regularly scheduled update. For any users who have and use
such a mechanism, they'd then have *two* mechanisms indexing the
filesystem rather than just one, and they're likely to only use one of
those. Furthermore, any mechanism they use to configure one of them
(e.g. for privacy or performance reasons) will not control the other,
and again they may well be unaware of the existence of the other one.

We should absolutely have a locate implementation available for any who
want to install and use it, but we shouldn't make all users pay the cost
of locate's database update to satisfy the subset of users who ever
invoke locate.

- Josh Triplett

Re: Proposal: plocate as standard for bookworm

[Paul Wise]

On Sat, Feb 6, 2021 at 5:29 PM Steinar H. Gunderson wrote:

> I believe a good, fast locate is something that we should have in our base
> install; it is fine to keep it out of the cloud image (as today), but it is
> genuinely useful for both desktops and servers, and with a low cost.

I support having locate in the base install, but I don't think that
the cost of daily walking the entire filesystem is low; especially
with HDDs and older storage or computers that can be a lot of I/O. I
guess it also alters the Linux kernel block/filesystem caches? A
daemon monitoring filesystem changes using fanotify or similar and
then updating the database on a configured schedule would mitigate
this cost, but I guess would be harder to implement.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Proposal: plocate as standard for bookworm

[Paul Wise]

On Sun, Feb 7, 2021 at 12:05 AM Steinar H. Gunderson wrote:

> It's a pretty thin use-case, but someone could have scripts that call mlocate
> explicitly (not through the locate symlink). Or have something that is
> capable of reading mlocate's database. Or wish to have both to benchmark
> against each other :-) Unfortunately, we don't have an “Anti-Recommends”.

Or wish to transition from mlocate to plocate without doing a full
updatedb run by using the plocate-build script, which reads the
mlocate database and generates a plocate database.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Proposal: plocate as standard for bookworm

[Steinar H. Gunderson]

On Sat, Feb 06, 2021 at 10:18:45PM +0100, Bernd Zeimetz wrote:
>> Thoughts?
> I think plocate should have a Conflicts: mlocate. There is no need to
> install two locate implementations in parallel, it will just create
> useless IO.

It's a pretty thin use-case, but someone could have scripts that call mlocate
explicitly (not through the locate symlink). Or have something that is
capable of reading mlocate's database. Or wish to have both to benchmark
against each other :-) Unfortunately, we don't have an “Anti-Recommends”.

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: Proposal: plocate as standard for bookworm

[Bernd Zeimetz]

On 2/6/21 6:10 PM, Steinar H. Gunderson wrote:

> I believe a good, fast locate is something that we should have in our base
> install; it is fine to keep it out of the cloud image (as today), but it is
> genuinely useful for both desktops and servers, and with a low cost.

seconded, thanks for writing plocate!

> Thoughts?

I think plocate should have a Conflicts: mlocate. There is no need to
install two locate implementations in parallel, it will just create
useless IO.


-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F

Proposal: plocate as standard for bookworm

[Steinar H. Gunderson]

Hi,

mlocate used to be Priority: standard; for some reason that I haven't been
able to unearth (despite the efforts of several people), there is now an
override for buster, so that it's no longer installed by default (and mlocate
now has an override disparity).

I do wonder if this was intentional or not, but it's a bit beside the point:
Now, there is plocate (written and maintained by myself). It is orders of
magnitude faster than mlocate (both on SSDs and HDDs alike), has the same
security model, a smaller database (typically half the size), and fixes
a few long-standing mlocate bugs. It is nearly fully command-line compatible
with mlocate, so most users should feel right at home. It builds on all
release and non-release architectures. The only non-Essential dependencies
are liburing1 (33 kB) and libzstd1 (670 kB).

I believe a good, fast locate is something that we should have in our base
install; it is fine to keep it out of the cloud image (as today), but it is
genuinely useful for both desktops and servers, and with a low cost.

Thoughts?

/* Steinar */
-- 
Homepage: https://www.sesse.net/