Bug#1040947: perl: readline fails to detect updated file
Niko Tyni wrote: > On Tue, Jul 18, 2023 at 01:23:51AM +, Eric Wong wrote: > > > At this point (given Perl's maturity); it's less surprising if > > it kept it's lack of error detection in all cases. > > Thanks for caring, but please take your arguments upstream. First off, my apologies to Ian that I made him feel uncomfortable. I merely wanted to point out the severity of incompatible changes in case someone from upstream sees my comments. > It's them you need to convince if you want that change to be > reverted or amended. Unfortunately, upstream's use of a proprietary platform, said platform's terms-of-service and JavaScript CAPTCHA are all non-starters for somebody like me :< Thanks.
Bug#1040947: perl: readline fails to detect updated file
On Tue, Jul 18, 2023 at 01:23:51AM +, Eric Wong wrote: > At this point (given Perl's maturity); it's less surprising if > it kept it's lack of error detection in all cases. Thanks for caring, but please take your arguments upstream. It's them you need to convince if you want that change to be reverted or amended. -- Niko Tyni nt...@debian.org
Bug#1040947: perl: readline fails to detect updated file
Ian Jackson wrote:
> Hi. I'm sorry that something I had a hand in is causing you an
> inconvenience.
>
> I'm afraid it's not clear to me what "working" and "non-working"
> behaviour from your example program is. I don't feel I can reply
> comprehensively, so I will comment on some of the details in your
> message.
With each iteration of the loop in my example, the file size
increases as shown by `-s', yet readline isn't returning any
data after it sees a transient EOF.
> Eric Wong writes ("Re: Bug#1040947: perl: readline fails to detect updated
> file"):
> > Both can be data loss bugs, but checking a log file which is
> > being written to by another process is a much more common
> > occurence than FS failures[1] (or attempting readline on a
> > directory (as given in the #1016369 example))
>
> AFAICT you are saying that the fix to #1016369 broke a program which
> was tailing a logfile. I agree that one should be able to tail a
> logfile in perl. I don't think I have a complete opinion about
> precisely what set of calls ought to be used to do that, but I would
> expect them to mirror the calls needed in C with stdio.
Right, and C stdio.h is similarly tricky when it comes to
properly dealing with errors, too. I often ended up using
unistd.h read(2) or Perl sysread directly for stuff I really
care about; combined with checking stat(2) st_size to ensure
I've read everything.
Data I care about tends to have checksumming built into it's
data format (git objects/packs, gzipped texts/tarballs, FLAC audio)
Uncompressed log files are transient data that ceases to be
relevant after a short time.
> > Since this is Perl and TIMTOWTDI, I've never used IO::Handle->error;
> > instead I always check defined-ness on each critical return value and
> > also enable Perl warnings to catch undefined return values.
> > I've never used `eof' checks, either; checking `chomp' result
> > can ensure proper termination of lines to detect truncated reads.
>
> AFIACT you are saying that you have always treated an undef value
> from line-reading operations as EOF, and never checked for error.
> I think that is erroneous.
Maybe so, though most of the reads I do are less critical than
writes. A failed read means there's *already* lost data and
there's nothing one can do about it.
A failed write can be retried on a different FS or rolled back.
IME, write errors are far more common (but perhaps that's because
I have errors=remount-ro in all my fstabs)
In the case of an application's log file, the application is
already toast if there's any I/O error; thus any monitoring on
application-level log files would cease to be relevant.
> That IO errors are rare doesn't mean they oughtn't to be checked for.
> Reliable software must check for IO errors and not assume that undef
> means EOF.
>
> I believe perl's autodie gets this wrong, which is very unfortunate.
Right, autodie doesn't appear to handle readline at all.
> > [1] yes, my early (by my standards) upgrade to bookworm was triggered
> > by an SSD failure, but SSD failures aren't a common occurence
> > compared to tailing a log file.
>
> I don't think this is the right tradeoff calculus.
>
> *With* the fix to #1016369 it is *possible* to write a reliable
> program, but soee buggy programs lose data more often.
>
> *Without* the fix to #1016369 it is completely impossible to write a
> reliable program.
For reliable programs (e.g. file servers), it's required to
check expected vs actual bytes read; that pattern can be
applied regardless of #1016369.
> Having said all that, I don't see why the *eof* indicator ought to
> have to persist. It is only the *errors* that mustn't get lost. So I
> think it might be possible for perl to have behaviour that would
> make it possible to write reliable programs, which still helping buggy
> programs fail less often.
Right; EOF indicators should be transient for regular files.
It's wrong to consider EOF a permanent condition on regular files.
> But, even if that's possible, I'm not sure that it's a good idea.
> Buggy programs that lose data only in exceptional error conditions are
> a menace. Much better to make such buggy programs malfunction all the
> time - then they will be found and fixed.
This mentality of breaking imperfect-but-practically-working
code in favor of some perfect ideal is damaging to Perl (based
on my experience with changes to Ruby driving away users).
Fwiw, using `strace -P $PATH -e inject=syscall...' to inject
errors for certain paths, both gawk and mawk fail as expected
when it fails to read STDIN:
echo hello >x # create file named `x'
strace -P x -e inject=read:error=EIO gawk '{ print }'
Bug#1040947: perl: readline fails to detect updated file
Hi. I'm sorry that something I had a hand in is causing you an
inconvenience.
I'm afraid it's not clear to me what "working" and "non-working"
behaviour from your example program is. I don't feel I can reply
comprehensively, so I will comment on some of the details in your
message.
Eric Wong writes ("Re: Bug#1040947: perl: readline fails to detect updated
file"):
> Both can be data loss bugs, but checking a log file which is
> being written to by another process is a much more common
> occurence than FS failures[1] (or attempting readline on a
> directory (as given in the #1016369 example))
AFAICT you are saying that the fix to #1016369 broke a program which
was tailing a logfile. I agree that one should be able to tail a
logfile in perl. I don't think I have a complete opinion about
precisely what set of calls ought to be used to do that, but I would
expect them to mirror the calls needed in C with stdio.
> Since this is Perl and TIMTOWTDI, I've never used IO::Handle->error;
> instead I always check defined-ness on each critical return value and
> also enable Perl warnings to catch undefined return values.
> I've never used `eof' checks, either; checking `chomp' result
> can ensure proper termination of lines to detect truncated reads.
AFIACT you are saying that you have always treated an undef value
from line-reading operations as EOF, and never checked for error.
I think that is erroneous.
That IO errors are rare doesn't mean they oughtn't to be checked for.
Reliable software must check for IO errors and not assume that undef
means EOF.
I believe perl's autodie gets this wrong, which is very unfortunate.
> [1] yes, my early (by my standards) upgrade to bookworm was triggered
> by an SSD failure, but SSD failures aren't a common occurence
> compared to tailing a log file.
I don't think this is the right tradeoff calculus.
*With* the fix to #1016369 it is *possible* to write a reliable
program, but soee buggy programs lose data more often.
*Without* the fix to #1016369 it is completely impossible to write a
reliable program.
Having said all that, I don't see why the *eof* indicator ought to
have to persist. It is only the *errors* that mustn't get lost. So I
think it might be possible for perl to have behaviour that would
make it possible to write reliable programs, which still helping buggy
programs fail less often.
But, even if that's possible, I'm not sure that it's a good idea.
Buggy programs that lose data only in exceptional error conditions are
a menace. Much better to make such buggy programs malfunction all the
time - then they will be found and fixed.
Thanks for your attention.
Ian.
--
Ian JacksonThese opinions are my own.
Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.
Bug#1040947: perl: readline fails to detect updated file
Niko Tyni wrote: > On Sun, Jul 16, 2023 at 11:35:49AM +0300, Niko Tyni wrote: > > Control: forwarded -1 https://github.com/Perl/perl5/issues/21240 > > > > On Wed, Jul 12, 2023 at 09:17:54PM +, Eric Wong wrote: > > > Package: perl > > > Version: 5.36.0-7 > > > Severity: normal > > > Tags: upstream > > > > > > Dear Maintainer, > > > > > > The `readline' op (`') no longer detects updates to file > > > which is being written to in Perl 5.36.0 after updating to bookworm. > > > > > > The attached script worked fine with Perl 5.32.1 in oldstable, > > > both scripts show the size changing with the `-s FH' op. > > > Same behavior on both amd64 and i386. > > > > Thanks for the report. > > > > This changed with > > > > > > https://github.com/Perl/perl5/commit/80c1f1e45e8ef8c27d170fae7ade41971fe20218 > > > > which went upstream in 5.37.4. We backported it for 5.36 to fix #1016369 . > > > > I've forwarded it upstream as https://github.com/Perl/perl5/issues/21240 > > Upstream says it's a direct consequence of fixing #1016369 so > the earlier behaviour of reading after EOF was a bug. > > You need to clear the EOF flag by calling $rd->clearerr() or > seek($rd, 0, SEEK_CUR) to get the appended lines. +Cc Ian Jackson who reported #1016369 Bug #1016369 was reported for an issue which went unnoticed for years (or decades, even?). I hit this new bug within minutes of upgrading to bookworm. Both can be data loss bugs, but checking a log file which is being written to by another process is a much more common occurence than FS failures[1] (or attempting readline on a directory (as given in the #1016369 example)) Thus I expect there's far more people negatively affected by this bug than there were for #1016369. Since this is Perl and TIMTOWTDI, I've never used IO::Handle->error; instead I always check defined-ness on each critical return value and also enable Perl warnings to catch undefined return values. I've never used `eof' checks, either; checking `chomp' result can ensure proper termination of lines to detect truncated reads. When there's hardware (or network FS) failure; errors tend to get very loud anyways with the system becoming unusable, console spew, read-only remount, etc; so the error checking done via Perl is often redundant in that case. [1] yes, my early (by my standards) upgrade to bookworm was triggered by an SSD failure, but SSD failures aren't a common occurence compared to tailing a log file.
Bug#1040947: perl: readline fails to detect updated file
On Sun, Jul 16, 2023 at 11:35:49AM +0300, Niko Tyni wrote: > Control: forwarded -1 https://github.com/Perl/perl5/issues/21240 > > On Wed, Jul 12, 2023 at 09:17:54PM +, Eric Wong wrote: > > Package: perl > > Version: 5.36.0-7 > > Severity: normal > > Tags: upstream > > > > Dear Maintainer, > > > > The `readline' op (`') no longer detects updates to file > > which is being written to in Perl 5.36.0 after updating to bookworm. > > > > The attached script worked fine with Perl 5.32.1 in oldstable, > > both scripts show the size changing with the `-s FH' op. > > Same behavior on both amd64 and i386. > > Thanks for the report. > > This changed with > > > https://github.com/Perl/perl5/commit/80c1f1e45e8ef8c27d170fae7ade41971fe20218 > > which went upstream in 5.37.4. We backported it for 5.36 to fix #1016369 . > > I've forwarded it upstream as https://github.com/Perl/perl5/issues/21240 Upstream says it's a direct consequence of fixing #1016369 so the earlier behaviour of reading after EOF was a bug. You need to clear the EOF flag by calling $rd->clearerr() or seek($rd, 0, SEEK_CUR) to get the appended lines. -- Niko Tyni nt...@debian.org
Bug#1040947: perl: readline fails to detect updated file
Control: forwarded -1 https://github.com/Perl/perl5/issues/21240 On Wed, Jul 12, 2023 at 09:17:54PM +, Eric Wong wrote: > Package: perl > Version: 5.36.0-7 > Severity: normal > Tags: upstream > > Dear Maintainer, > > The `readline' op (`') no longer detects updates to file > which is being written to in Perl 5.36.0 after updating to bookworm. > > The attached script worked fine with Perl 5.32.1 in oldstable, > both scripts show the size changing with the `-s FH' op. > Same behavior on both amd64 and i386. Thanks for the report. This changed with https://github.com/Perl/perl5/commit/80c1f1e45e8ef8c27d170fae7ade41971fe20218 which went upstream in 5.37.4. We backported it for 5.36 to fix #1016369 . I've forwarded it upstream as https://github.com/Perl/perl5/issues/21240 -- Niko Tyni nt...@debian.org
Bug#1040947: perl: readline fails to detect updated file
Package: perl
Version: 5.36.0-7
Severity: normal
Tags: upstream
Dear Maintainer,
The `readline' op (`') no longer detects updates to file
which is being written to in Perl 5.36.0 after updating to bookworm.
The attached script worked fine with Perl 5.32.1 in oldstable,
both scripts show the size changing with the `-s FH' op.
Same behavior on both amd64 and i386.
-- System Information:
Debian Release: 12.0
APT prefers stable-security
APT policy: (500, 'stable-security'), (500, 'stable-debug'), (500, 'stable')
Architecture: i386 (i686)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL
set to en_US.UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages perl depends on:
ii dpkg 1.21.22
ii libperl5.365.36.0-7
ii perl-base 5.36.0-7
ii perl-modules-5.36 5.36.0-7
Versions of packages perl recommends:
ii netbase 6.4
Versions of packages perl suggests:
pn libtap-harness-archive-perl
pn libterm-readline-gnu-perl | libterm-readline-perl-perl
ii make4.3-4.1
ii perl-doc5.36.0-7
-- no debconf information
#!/usr/bin/perl -w
use v5.12;
use autodie;
use File::Temp;
use IO::Handle;
use Data::Dumper;
my $tmp = File::Temp->newdir('test-', TMPDIR => 1);
my $dst = "$tmp/log";
open my $wr, '>>', $dst;
open my $rd, '<', $dst;
say "# writing to $dst";
for my $n (1..5) {
sleep 1;
my @x = readline $rd;
say "# size ".(-s $rd), ' ', Dumper(\@x);
syswrite $wr, "$n\n";
}
