Hi Eric On Sat, 10 Sept 2022 at 03:19, Eric Wong <[email protected]> wrote: > > Ricardo Ribalda <[email protected]> wrote: > > The patch did not seem to have any effect :(, I never get a "IMAP > > LastError: " message > > Yeah, I guess IMAP servers will just shutdown the socket w/o > saying anything. At least I didn't get anything from dovecot... > > The below patch is a refinement of what I posted originally > and should stop the process instead of attempting to continue > and spew. > > > On the other hand, the -j worked! I can go up to -j ,15 without any error. > > Good to know. > > I wonder if making the default `-j ,4' for IMAP is reasonable if > unspecified. That's the default limit for HTTP(S) hosts, and I > seem to recall 4 being a reasonable limit for browsers. > > Thanks for the report and followup!
Similar output: ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me # https://lore.kernel.org/all/ limiting to 2022-09-08 19:33 +0000 and newer # /usr/local/google/home/ribalda/.local/share/lei/store 12/12 # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660159987..)+AND+dt%3A20220908193300.. 1486767 lei2mail 17 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486762 lei2mail 12 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486782 lei2mail 32 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486771 lei2mail 21 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486766 lei2mail 16 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486788 lei2mail 38 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486761 lei2mail 11 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486791 lei2mail 41 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486775 lei2mail 25 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486765 lei2mail 15 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486755 lei2mail 5 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486793 lei2mail 43 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486795 lei2mail 45 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486763 lei2mail 13 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486779 lei2mail 29 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486773 lei2mail 23 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486751 lei2mail 1 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486787 lei2mail 37 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486796 lei2mail 46 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486776 lei2mail 26 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486752 lei2mail 2 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486750 lei2mail 0 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486786 lei2mail 36 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486777 lei2mail 27 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486764 lei2mail 14 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486794 lei2mail 44 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486756 lei2mail 6 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486790 lei2mail 40 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486784 lei2mail 34 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486792 lei2mail 42 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486768 lei2mail 18 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486774 lei2mail 24 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486757 lei2mail 7 wq_worker: do_post_auth: Can't call method "uidvalidity" on an undefined value at /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. 1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm line 783. # https://lore.kernel.org/all/ 20/20 # 0 written to imaps://imap.gmail.com/lei/me (32 matches) ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me -j ,15 # https://lore.kernel.org/all/ limiting to 2022-09-08 19:33 +0000 and newer # /usr/local/google/home/ribalda/.local/share/lei/store 12/12 # /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660160023..)+AND+dt%3A20220908193307.. # https://lore.kernel.org/all/ 20/20 # 0 written to imaps://imap.gmail.com/lei/me (32 matches) Thanks! > > ------8<------ > From: Eric Wong <[email protected]> > Subject: [PATCH] lei: bail out earlier on IMAP writer failures > > Excessive IMAP connections can overload IMAP servers and cause > clients to be disconnected without diagnostic messages. > Use $lei->fail on these exceptions to propagate errors to the > CLI ASAP to avoid further errors down the line. > > This ought to make problems more apparent for users using IMAP > destinations. > > Reported-by: Ricardo Ribalda <[email protected]> > Link: > https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wnu...@mail.gmail.com/ > --- > lib/PublicInbox/LeiToMail.pm | 10 +++++++--- > lib/PublicInbox/NetReader.pm | 8 +++++++- > 2 files changed, 14 insertions(+), 4 deletions(-) > > diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm > index 2aa3977e..03cbde3b 100644 > --- a/lib/PublicInbox/LeiToMail.pm > +++ b/lib/PublicInbox/LeiToMail.pm > @@ -310,8 +310,11 @@ sub _imap_write_cb ($$) { > my $dedupe = $lei->{dedupe}; > $dedupe->prepare_dedupe if $dedupe; > my $append = $lei->{net}->can('imap_append'); > - my $uri = $self->{uri}; > - my $mic = $lei->{net}->mic_get($uri); > + my $uri = $self->{uri} // die 'BUG: no {uri}'; > + my $mic = $lei->{net}->mic_get($uri) // die <<EOM; > +E: $uri connection failed. > +E: Consider using `--jobs ,1' to limit IMAP connections > +EOM > my $folder = $uri->mailbox; > $uri->uidvalidity($mic->uidvalidity($folder)); > my $lse = $lei->{lse}; # may be undef > @@ -749,7 +752,8 @@ sub do_post_auth { > $au_peers->[1] = undef; > sysread($au_peers->[0], my $barrier1, 1); > } > - $self->{wcb} = $self->write_cb($lei); > + eval { $self->{wcb} = $self->write_cb($lei) }; > + $lei->fail($@) if $@; > if ($au_peers) { # wait for peer l2m to set write_cb > $au_peers->[3] = undef; > sysread($au_peers->[2], my $barrier2, 1); > diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm > index c1af03a3..4de2583e 100644 > --- a/lib/PublicInbox/NetReader.pm > +++ b/lib/PublicInbox/NetReader.pm > @@ -685,7 +685,13 @@ sub mic_get { > } > my $mic = mic_new($self, $mic_arg, $sec, $uri); > $cached //= {}; # invalid placeholder if no cache enabled > - $mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef; > + if ($mic && $mic->IsConnected) { > + $cached->{$sec} = $mic; > + } else { > + warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic; > + warn "IMAP errno: $!\n" if $!; > + undef; > + } > } > > sub imap_each { -- Ricardo Ribalda
