file type mapping for Office documents [was: Re: ANNOUNCE: amavisd-new-2.11.0-rc1 release candidate is available]

[Thomas Jarosch]

Hi Mark,

On Monday, 21. March 2016 09:08:24 Thomas Jarosch wrote:
> In the previous amavisd version, all Office files types
> were mapped to 'doc' since file(1) reported just "Office document".
> 
> Office 2007+ documents get properly distinguished by file(1).
> 
> We could either do this (I run that locally):
> 
>  [qr/^Microsoft Office Document\b/i  => 'doc'],  # OLE2: doc, ppt,
> xls, ... +[qr/^Microsoft (Word|Excel|PowerPoint)\b/i  => 'doc'],  #
> Office 2007+: docx, docm, pptx, xlsx, ...
> 
> 
> for the sake of "compatibility"
> 
> *or*
> 
> recognize each file type on it's own to give
> the users a better control about the content type.
> 
> What do you think?

could you revisit this one for 2.11.1? :)

I've opted for this one on my installation:

[qr/^Microsoft Office Document\b/i => 'doc'], # OLE2: doc, ppt, xls,...
[qr/^Microsoft Word\b/i=> 'doc'],
[qr/^Microsoft Excel\b/i   => 'xls'],
[qr/^Microsoft PowerPoint\b/i  => 'ppt'],


and also added this as fallback if file(1) is not 100% sure:

[qr/^Composite Document File\b/i   => 'doc'], # OLE2 container


Best regards,
Thomas

Re: ANNOUNCE: amavisd-new-2.11.0-rc1 release candidate is available

[Thomas Spuhler]

On Monday, March 21, 2016 09:08:24 AM Thomas Jarosch wrote:
> Hi Mark,
> 
> On Friday, 18. March 2016 01:38:59 Mark Martinec wrote:
> > - updated a default $map_full_type_to_short_type_re to recognize
> > 
> >a Microsoft Word document as type doc; thanks to Jörg Backschues;
> 
> thanks for the new release candidate!
> 
> In the previous amavisd version, all Office files types
> were mapped to 'doc' since file(1) reported just "Office document".
> 
> Office 2007+ documents get properly distinguished by file(1).
> 
> We could either do this (I run that locally):
> 
>  [qr/^Microsoft Office Document\b/i  => 'doc'],  # OLE2: doc, ppt, xls,
> ... +[qr/^Microsoft (Word|Excel|PowerPoint)\b/i  => 'doc'],  # Office
> 2007+: docx, docm, pptx, xlsx, ...
> 
> 
> for the sake of "compatibility"
> 
> *or*
> 
> recognize each file type on it's own to give
> the users a better control about the content type.
> 
> What do you think?
> 
> 
> btw: We've just contributed example Office documents
> to the file-tests test suite:
> https://github.com/christian-intra2net/file-tests/tree/office-types
> 
> Those files are good samples for toying with this stuff.
> 
> Cheers,
> Thomas

I packaged it for Mageia with the usual distro patches and upgraded the 
existing install. I don't see any errors.
This what  # systemctl -l status amavisd provides.

Mar 22 04:30:16 cauldron.btspuhler.com systemd[1]: Reloaded AMAVIS interface 
between MTA and content checkers.
Mar 22 04:30:16 cauldron.btspuhler.com amavis[28786]: (!)Net::Server: 
2016/03/22-04:30:16 Re-exec server during HUP
Mar 22 04:30:17 cauldron.btspuhler.com amavis[28786]: starting. (warm) 
/usr/sbin/amavisd at cauldron.btspuhler.com amavisd-new-2.11.0-rc1 (2
Mar 22 04:30:18 cauldron.btspuhler.com amavis[28786]: Net::Server: Group Not 
Defined.  Defaulting to EGID '957 957'
Mar 22 04:30:18 cauldron.btspuhler.com amavis[28786]: Net::Server: User Not 
Defined.  Defaulting to EUID '958'
Mar 22 04:30:18 cauldron.btspuhler.com amavis[28786]: No $altermime, 
not using it
Mar 22 04:30:18 cauldron.btspuhler.com amavis[28786]: No ext program for   
.lz4, tried: lz4c -d
Mar 22 04:30:18 cauldron.btspuhler.com amavis[28786]: No decoder for   .lz4
Mar 22 04:30:18 cauldron.btspuhler.com amavis[28786]: Using primary internal av 
scanner code for ClamAV-clamd
Mar 22 04:30:18 cauldron.btspuhler.com amavis[28786]: Found secondary av 
scanner ClamAV-clamscan at /usr/bin/clamscan


Seems like I need to provide a group and user ID for Net::Server

-- 
Best regards
Thomas Spuhler

All of my e-mails have a valid digital signature
ID 60114E63

signature.asc
Description: This is a digitally signed message part.

Re: ANNOUNCE: amavisd-new-2.11.0-rc1 release candidate is available

[Quanah Gibson-Mount]

--On Tuesday, March 22, 2016 9:17 AM +1000 Noel Butler 
 wrote:




amavis[1972]: (!)_DIE: Suicide in child_init_hook: BDB can't connect db
env. at /var/amavis/db: No such file or directory, No such file or
directory. at (eval 83) line 337.


sure would be nice if amavis would dump BDB for LMDB.. :P

--Quanah

--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.

Zimbra ::  the leader in open source messaging and collaboration
A division of Synacor, Inc

Re: ANNOUNCE: amavisd-new-2.11.0-rc1 release candidate is available

[Noel Butler]

Hi Mark,


"
- remove a stale database file __db.nanny.db on a reload or restart,
  as it can prevent a successful start when a previous start failed
  for some reason; a patch by Trent Lloyd;
"

Does this patch fix the constant logfile-filling logging of no db found, 
by just creating the damn thing and carrying on as normal ?


as in:
amavis[1972]: (!)_DIE: Suicide in child_init_hook: BDB can't connect db 
env. at /var/amavis/db: No such file or directory, No such file or

directory. at (eval 83) line 337.


Although we've not seen in in a few months, it does occasionally pop its 
ugly head up.


Thanks
Noel



--
If you have the urge to reply to all rather than reply to list, you best
first read  http://members.ausics.net/qwerty/

Re: ANNOUNCE: amavisd-new-2.11.0-rc1 release candidate is available

[Thomas Jarosch]

Hi Mark,

On Friday, 18. March 2016 01:38:59 Mark Martinec wrote:
> - updated a default $map_full_type_to_short_type_re to recognize
>a Microsoft Word document as type doc; thanks to Jörg Backschues;

thanks for the new release candidate!

In the previous amavisd version, all Office files types
were mapped to 'doc' since file(1) reported just "Office document".

Office 2007+ documents get properly distinguished by file(1).

We could either do this (I run that locally):

 [qr/^Microsoft Office Document\b/i  => 'doc'],  # OLE2: doc, ppt, xls, ...
+[qr/^Microsoft (Word|Excel|PowerPoint)\b/i  => 'doc'],  # Office 2007+: 
docx, docm, pptx, xlsx, ...


for the sake of "compatibility"

*or*

recognize each file type on it's own to give
the users a better control about the content type.

What do you think?


btw: We've just contributed example Office documents
to the file-tests test suite:
https://github.com/christian-intra2net/file-tests/tree/office-types

Those files are good samples for toying with this stuff.

Cheers,
Thomas

ANNOUNCE: amavisd-new-2.11.0-rc1 release candidate is available

[Mark Martinec]

A release candidade RC1 of the coming version 2.11.0
of amavisd-new is available at:

  https://www.ijs.si/software/amavisd/amavisd-new-2.11.0-rc1.tar.xz

Release notes are at:

  https://www.ijs.si/software/amavisd/release-notes.txt

Please try it out, feedback is welcome.



amavisd-new-2.11.0 release notes

Contents:
  DEPRECATION NOTICE
  COMPATIBILITY
  BUG FIXES
  NEW FEATURES
  OTHER
  SUPERVISED PROCESS NOTES


DEPRECATION NOTICE

- the old DomainKey signatures (a predecessor to DKIM) has been 
published

  as a historic document RFC 4870 and obsoleted by RFC 4871 in May 2007;
  Support for DomainKey signatures is likely to be removed with a next
  version of amavisd.


COMPATIBILITY

There are some minor incompatibilities between versions 2.10.1 and 
2.11.0:


- During startup more detailed testing is performed for taint bugs of
  a module Encode and the function utf8::is_utf8(), which may produce
  warnings on old versions of perl with its old core module Encode,
  or may exit on detecting more sinister bugs in these modules.
  Note that the module Encode may be upgraded independently of perl,
  if desired;

- with MySQL: changed character set 'utf8' to 'utf8mb4' for fields
  msgs.subject and msgs.from_addr, as previously some of the UTF-8
  characters could not be stored in a database;

- when logging to stderr a timestamp prefix to each message is only
  still inserted if $DEBUG is true.  When $DEBUG is false each message
  is prefixed with a syslog log level in angle brackets, and a timestamp
  is omitted (for compatibility with systemd);

- a perl module Digest::SHA is now a required module. It is a perl core
  module since perl 5.10, so it shouldn't introduce a new dependency,
  and it was a de-facto required module even previously, as it was 
needed

  for DKIM processing;


BUG FIXES

- delivery method was undefined when always_bcc was used;
  reported by Marieke Janssen;

- amavisd: avoid warnings issued by perl 5.21.7 and later:
Negative repeat count does nothing at ./amavisd line 16408
  and similarly in amavisd-status;

- remove a stale database file __db.nanny.db on a reload or restart,
  as it can prevent a successful start when a previous start failed
  for some reason; a patch by Trent Lloyd;


NEW FEATURES

- Polished rough corners to facilitate running amavisd as a 
non-daemonized

  supervised process, e.g. under systemd:

  * make it possible/easier to disable use of a pid_file;

  * send status notifications to systemd when a NOTIFY_SOCKET 
environment

variable is provided;

  * improved logging to stderr when $do_syslog and $logfile are 
undefined
(although logging through syslog might still be preferred, as 
writing
to a shared pipe from multiple child processes only guarantees 
atomicity
of writes shorter than PIPE_BUF, which is typically 512 bytes on 
*BSD,

and 4096 bytes on Linux);

  See below for a sample amavisd.service file.

- A log template macro 'report_json' can now take arguments, which can
  include or exclude fields (key/values) from the JSON report object.
  Arguments to a macro are either field names (keys) to be included
  in a report, or are field names to be excluded, each prefixed with
  an exclamation mark, to produce a report with all but excluded fields.

  Field names are case-sensitive. The order of fields in a serialized
  JSON object is unaffected by the order of field names in a filter.
  Unknown or non-present field names in a filter are silently ignored.

  Example:
[:report_json|mail_id|action|content_type|queued_as|mail_from|size]
  or:
[:report_json|!recipients|!elapsed|!os_fp|!subject|!subject_rot13]

  For better clarity, instead of listing field names as individual
  arguments to a macro, it is also possible to provide a single argument
  to a macro, in which field names are separated by whitespace:
[:report_json|mail_id action content_type queued_as mail_from size]
  or:
[:report_json|!message !recipients !to_addr   !elapsed !os_fp
   !subject !subject_rot13 !user_agent !tests !tests_ham 
!tests_spam]


  As an example, a setting in a config file may look like:
$log_templ = '[:report_json|mail_id action queued_as mail_from]';

  If at least one field name has an exclamation mark (i.e. is to be
  excluded), all but excluded fields are implied, so any field names
  without an exclamation mark are redundant.

  Currently this is a simple filter where subfields of a structured
  object cannot be selectively filtered (e.g. elapsed.SpamCheck).
  For finer control on JSON content use some external JSON-processing
  utility.  Based on a patch by Markus Benning.

- Two new configuration settings are added: %smtpd_tls_server_options
  and %smtp_tls_client_options. These two associative arrays are passed
  to IO::Socket::SSL->start_SSL when establishing a server-side or a
  client-side TLS session with an MTA, and provide more control over
  a TLS session - like providing certificates and