heads up: Email::MIME triggers Encode leak

[Eric Wong]

Hey all, apparently the use of Encode::FB_CROAK in Email::MIME
has been triggering leaks on Encode::decode failures.  It seems
like it's been a problem for several years, at least:

  https://rt.cpan.org/Ticket/Display.html?id=139622


I finally found it because I started porting a wonky malloc
debugger to Perl: https://80x24.org/mwrap-perl.git/

Re: Any advice for a searchable web archiver ?

[Eric Wong]

Marc Chantreux  wrote:
> hello,
> 
> > public-inbox is Perl, uses Email::MIME, and (optionally) uses
> > Xapian like notmuch.  The Perl bits around search indexing is
> > ported to Perl from what I understood of the C++ code in notmuch.
> 
> it seems you explored very interesting concepts. however i didn't
> understand how to install/test/hack on it.

Which parts of the INSTALL, HACKING, or README were unclear?

https://public-inbox.org/INSTALL
https://public-inbox.org/HACKING
https://public-inbox.org/README

I can try to clarify (sorry, not online much due to upcoming holidays)

It uses MakeMaker like most Perl modules, but haven't made a
tarball release, yet, so you can just:

git clone https://public-inbox.org/ public-inbox

to grab the source

Re: Any advice for a searchable web archiver ?

[Eric Wong]

Marc Chantreux  wrote:
> Hello,
> 
> As the sympa community (http://www.sympa.org) recently grown, we are
> thinking about revamping the whole UI and we would like to have
> a new web archiver based on:
> 
> * no default frontend but exposing the API through REST, websockets or
>   whatever.
> * maximizing the interactions between Sympa and CPAN
> * trying to avoid other dynamic langage or jvm dependency
>   (or considering them as temporary solutions)
> * being JMAP friendly (we bet on it to become a very healthy community)
> 
> My first idea was to use notmuch, PEP modules and Dancer on top of
> maildirs then i discover Dezi (inactive since 2015) and the use of
> Lucy (also used by the very active librecat project).
> 
> I know Dezi is a general search engine but i hope that taking care of
> a good email support for it than reinvent the wheel.

public-inbox is Perl, uses Email::MIME, and (optionally) uses
Xapian like notmuch.  The Perl bits around search indexing is
ported to Perl from what I understood of the C++ code in notmuch.

The web part is PSGI and I consider the URL format a stable API:

https://public-inbox.org/design_www.txt

I will probably add JSON support to it for external web services;
haven't looked into JMAP, yet...

There's also a standalone NNTP server based on Danga::Socket.

You can find an example of it for the git mailing list
:  https://public-inbox.org/git/

> Those are lot of things to look for if i want to have a clear opinion
> on a good strategy. Any advice would be warmly welcome.

It's probably not a perfect match for you guys, but it's all AGPL-3+.
The whole thing (code AND data) is designed to be completely
replicatable and forkable using git, so anybody can clone any instance
it's entirety.

Re: Email::MIME and headerless parts

[Eric Wong]

"Matthew Horsfall (alh)" <wolfs...@gmail.com> wrote:
> On Tue, Jan 10, 2017 at 5:33 AM, Eric Wong <e...@80x24.org> wrote:
> > [...]
> > The problem manifests when the part has no header.
> > For example, the first rendered part of the message at is
> > missing the first 3 lines of the body:
> 
> Thanks for bringing this up!
> 
> I tested the attached script and it appears that my PR here fixes it:
> 
>   https://github.com/rjbs/Email-MIME/pull/28/files
> 
> This hasn't been merged yet, there may still be some more work to do
> on it, but hopefully we'll get it sorted soon.

Thanks for the pointer!  I've introduced a monkey patch into
public-inbox with your changes to workaround the problem in currently
released versions of Email::MIME, for now:

   https://public-inbox.org/meta/20170110220718.5106-...@80x24.org/raw

So yeah, links to Richard's emails in my original message display
correctly, at least:

https://public-inbox.org/git/20170110004031.57985-1-hans...@google.com/
http://hjrcffqmbrq6wope.onion/git/20170110004031.57985-1-hans...@google.com/

Re: Where is the attached file opened with Email::MIME?

[Eric Wong]

Octavian Rasnita orasn...@gmail.com wrote:
Email::MIME-create(
body = $pdf,
attributes = {
content_type = application/pdf,
name = bogus-report.pdf,
filename = bogus-report.pdf,
encoding = quoted-printable,

Can you try base64 instead of quoted-printable for the
attachment?  For largely non-ASCII data (such as PDF),
base64 will be smaller.

Re: things need co-maint

[Eric Wong]

Ricardo Signes perl@rjbs.manxome.org wrote:
   Email-LocalDelivery

I use this for various projects since 2006 or so (from Debian).
Works great for me!

I think I looked at Email::Sender a few months ago for a newer project
but decided the version in Debian stable (0.110005) was too old for a
faster(?) moving target.