AW: Restore to O365 fails

2021-04-27 Thread Martin Nadvornik



Thank you for your input Jackson. We already have the Delivered-To header 
filtered so unfortunateley this is not a cause for our issue.

And thanks for your answer Janos, I also think it is worth to investigate 
further espacially because our users are used to restore mails and since we 
switched to EXO seemingly 90% of all restores fail. I will definitely pursue 
this further and if I can't find any clues the only remaining option is of 
course to open an issue with Microsoft on this.


Von: Jackson Craig 
Gesendet: Dienstag, 27. April 2021 15:59
An: 'Piler User'
Betreff: RE: Restore to O365 fails



Folks,

Most likely not the culprit, but thought I would share anyway - just ignore
if you already know it isn't due to this - this is something I encountered
with trying to restore mail to mailboxes. Just pasting in my notes that I
took at the time:


"Some mail will not restore to mailbox, and you get a
forwarding loop for the user (note that you always get
a forwarding loop for bcc the archive blind copy, this is
OK just ignore the message. It stops the restored mail getting
added into the archive again anyway).

This happens because of the Delivered-To header. Postfix adds
this header, and as there may already be one there with the
same recipient, it thinks the restored mail is in a mail
loop and you get the error (and no mail).

The quick workaround is to add a header check for the user, e.g.:

/^Delivered-To: m.gras...@asb-group.com/ IGNORE

add this to /etc/postfix/header_checks on the mail server

Remember to postmap the file after editing it."


  -
This email and any attachments are confidential to the intended recipient
and may also be privileged.  If you are not the intended recipient, please
delete it from your system and notify the sender.  You should not copy it
or use it for any purpose nor disclose or distribute its contents to any
other person.

-Original Message-----
From: Martin Nadvornik 
Sent: 27 April 2021 14:24
To: Piler User 
Subject: AW: Restore to O365 fails



Hi everyone,

we also have the same or at least a similar problem. However not with every
mail but with most. We were so far also unable to find the root cause for
this and can't see any distinctive differences between mails that fail to
restore and those that don't. In our specific setup piler is using a postfix
server as smart host which then redirects the messages to exchange online
via a transport rule. EXO accepts the restored messages our postfix sends
but never delivers them to the inbox.
Looking at the message trace in EXO shows that the message is beeing recived
but there are no detailed trace infos available.

Out first guess was also that EXO doesn't like recveiving messages with the
same message ID multiple times. We haven't tried REWRITE_MESSAGE_ID yet but
based on what Alex wrote this seems to not solve the problem.

Martin

Von: Karl Rossing 
Gesendet: Freitag, 16. April 2021 17:54
An: Piler User
Betreff: Re: Restore to O365 fails

We have had the same problem. I meant to open a ticket with Microsoft but so
far have not.

On Fri, Apr 16, 2021 at 8:53 AM Alexander Noack
mailto:a...@loetzimmer.de>> wrote:

I do realize that this is very likely not a Piler issue but rather O365
related… but maybe somebody already came across this issue and knows how to
remediate it.

When I restore a deleted email in my O365 account, the message transfer
always fails:

550 4.3.2 QUEUE.TransportAgent; message deleted by transport agent

The Piler server is setup as a connector in Exchange Online. The Smarthost
is
.mail.protection.outlook.com<http://mail.protection.outlook.c
om>.

Mails seem to be originating from
MicrosoftExchange329e71ec88ae4615bbc36ab6ce41109e@.onmicrosoft.com

The error states that there is an NDR sent to that address, but even after
assigning that address to myself, no NDR is ever received.

I have set $config['REWRITE_MESSAGE_ID'] = 1;

Any help is greatly appreciated!


Alex

Missiles & Space Batteries Limited is registered in Scotland under company
number 162626.  The company's registered office is at Hagmill Road, East
Shawhead, Coatbridge, ML5 4UZ.




AW: Restore to O365 fails

2021-04-27 Thread Martin Nadvornik



Hi everyone,

we also have the same or at least a similar problem. However not with every 
mail but with most. We were so far also unable to find the root cause for this 
and can't see any distinctive differences between mails that fail to restore 
and those that don't. In our specific setup piler is using a postfix server as 
smart host which then redirects the messages to exchange online via a transport 
rule. EXO accepts the restored messages our postfix sends but never delivers 
them to the inbox.
Looking at the message trace in EXO shows that the message is beeing recived 
but there are no detailed trace infos available.

Out first guess was also that EXO doesn't like recveiving messages with the 
same message ID multiple times. We haven't tried REWRITE_MESSAGE_ID yet but 
based on what Alex wrote this seems to not solve the problem.

Martin

Von: Karl Rossing 
Gesendet: Freitag, 16. April 2021 17:54
An: Piler User
Betreff: Re: Restore to O365 fails

We have had the same problem. I meant to open a ticket with Microsoft but so 
far have not.

On Fri, Apr 16, 2021 at 8:53 AM Alexander Noack 
mailto:a...@loetzimmer.de>> wrote:

I do realize that this is very likely not a Piler issue but rather O365 
related… but maybe somebody already came across this issue and knows how to 
remediate it.

When I restore a deleted email in my O365 account, the message transfer always 
fails:

550 4.3.2 QUEUE.TransportAgent; message deleted by transport agent

The Piler server is setup as a connector in Exchange Online. The Smarthost is 
.mail.protection.outlook.com.

Mails seem to be originating from 
MicrosoftExchange329e71ec88ae4615bbc36ab6ce41109e@.onmicrosoft.com

The error states that there is an NDR sent to that address, but even after 
assigning that address to myself, no NDR is ever received.

I have set $config['REWRITE_MESSAGE_ID'] = 1;

Any help is greatly appreciated!


Alex



Re: exact matching for archiving rules possible?

2020-09-10 Thread Martin Nadvornik
Hello Janos,

thank you as always for your quick help. You were absoluteley right,
using the two rules works perfectly.
In case someone stumbles upon this in future and thinks two rules are
too unwieldy, the regex "(^someth...@example.com)|(
someth...@example.com)" also works fine and keeps it in one rule.

Martin

Am 09.09.2020 um 20:32 schrieb s...@acts.hu:
> Hello Martin,
>
> On 2020-09-09 14:33, Martin Nadvornik wrote:
>>
>> I recently discovered that archiving rules like "From:
>> someth...@example.com" or "To: someth...@example.com" will also trigger
>> for e-mail addresses like "justsometh...@example.com". Is there any way
>> to perform an exaxct matching?
>> I already tried entering the following regexes when creating the rules
>> via the web gui:
>>
>> "someth...@example.com"
>> ^ someth...@example.com
>> ^someth...@example.com
>>
>> Any help is appreciated :)
>
> Take a message as an eml file, and run pilertest against it to see how
> piler sees the message. Only showing the To: results between asterisks
> (*).
>
> Let's take the following message (#1) (only the To: header):
>
> To: user.n...@example.com
>
> after parsing:
>
> to: *user.n...@example.com user name example com  (example.com )*
>
>
> Then another message (#2):
>
> To: "User, Name" user.n...@example.com
>
> after parsing:
>
> to: *user name user.n...@example.com user name example com 
> (example.com )*
>
>
> If you specified "^user.n...@example.com" (without quotes), then you
> could match
> message #1, but miss #2.
>
> If you specified "user.n...@example.com" (without quotes), then you
> could match
> both message #1 and #2, however also match otheruser.n...@example.com,
> which you
> don't want.
>
> So the solution takes 2 patterns:
>
> "^user.n...@example.com" to catch #1
>
> and
>
> " user.n...@example.com" to catch #2 (notice the prepended space!)
>
> Again, you shouldn't use the quotes when you type these patterns, the
> quotes
> here only serve as a clear distinction.
>
> Janos

-- 
Martin Nadvornik
IT Mitarbeiter

Diakonie Flüchtlingsdienst gem. GmbH
Steinergasse 3/12
1170 Wien, Austria
Tel: +43 (0) 1/ 402 67 54 - 1337
Fax: +43 (0) 1/ 402 67 54 - 16
Mobil: 0664/88350692
http://www.diakonie.at/fluechtlingsdienst

Verantwortung kann man nicht abschieben!



exact matching for archiving rules possible?

2020-09-09 Thread Martin Nadvornik



Hello everyone,

I recently discovered that archiving rules like "From:
someth...@example.com" or "To: someth...@example.com" will also trigger
for e-mail addresses like "justsometh...@example.com". Is there any way
to perform an exaxct matching?
I already tried entering the following regexes when creating the rules
via the web gui:

"someth...@example.com"
^ someth...@example.com
^someth...@example.com

Any help is appreciated :)

Thanks,
Martin

-- 
Martin Nadvornik
IT Mitarbeiter

Diakonie Flüchtlingsdienst gem. GmbH
Steinergasse 3/12
1170 Wien, Austria
Tel: +43 (0) 1/ 402 67 54 - 1337
Fax: +43 (0) 1/ 402 67 54 - 16
Mobil: 0664/88350692
http://www.diakonie.at/fluechtlingsdienst

Verantwortung kann man nicht abschieben!




Re: LDAPS

2020-09-03 Thread Martin Nadvornik




Hi Cameron,

it should work if you set the ldap host using the sheme "ldaps://". It 
might also be required to add "TLS_REQCERT never" to your ldap.conf as 
specified in http://www.mailpiler.org/wiki/current:ldap-authentication.
You can also have a look at 
https://bitbucket.org/jsuto/piler/issues/930/ldaps-auth-against-ad-not-working 
for an example


Cheers,
Martin

Am 2020-09-03 15:00, schrieb Robitaille, Cameron:

Hello:

Does anyone have some documentation regarding configuring LDAPS in the
config-site.php?

Thanks,

Cameron




Re: Piler 1.35 on Ubuntu 18.04 - Web UI doesn't show mail headers / contents

2020-07-31 Thread Martin Nadvornik



Hello Patrick,

of course check what Janos said first, but if it's just one single
e-mail account where mails are not shown in the webgui, it would be
interesting if the account has access in piler to an address that is
longer than 41 characters. I stumbled upon this for our piler instance
recently
(https://bitbucket.org/jsuto/piler/issues/1082/no-sphinx-results-with-long-email).
Let me know if you have the same issue.

Thanks,
Martin

Am 31.07.2020 um 17:34 schrieb s...@acts.hu:
>
>
> On 2020-07-31 17:16, Wagner, Patrick wrote:
>>
>> So that issue is fixed now, but has unearthed another - I can find and
>> open all mails just fine as auditor@local, and logging in with my
>> personal account works as well (I'm shown only my emails, as it should
>> be), but another email address / account I've tested doesn't show any
>> emails even though the usage report and my login as auditor definitely
>> contains emails for this email address? I've even used copy and paste
>> out of the usage report UI to change email address and username for
>> the piler account in question (basically a NOOP, as there was no typo
>> before either)
>
> check the sphinx query logged when this other user hits the search
> button.
> Verify that his email addresses are present in the query. If that's good,
> then download or export a single email he should see, and run pilertest
> on the email to see what piler thinks about it. Check that the From/To/Cc
> fields have his email address.
>
> Janos
>

-- 
Martin Nadvornik
IT Mitarbeiter

Diakonie Flüchtlingsdienst gem. GmbH
Steinergasse 3/12
1170 Wien, Austria
Tel: +43 (0) 1/ 402 67 54 - 1337
Fax: +43 (0) 1/ 402 67 54 - 16
Mobil: 0664/88350692
http://www.diakonie.at/fluechtlingsdienst

Verantwortung kann man nicht abschieben!




Re: Enterprise edition

2020-07-22 Thread Martin Nadvornik




Hi Marcelo,

can't speek for Janos, the developer of Piler, but I am not aware of any 
changes. What did you read that leads you into thinking piler is not 
open source anymore?
The source code is available like always at 
https://bitbucket.org/jsuto/piler/src/master/ and the master branch had 
the last commit 4 days ago...


There has always been an enterprise edition (at least as long as I know 
the project) but you are not required to pay anything to use piler when 
using the open source version.


Or maybe you got confused because the site currently seems a bit broken 
and clicking on "GoBD notes" leads to the enterprise documentation. The 
normal documentation ist still reachable under 
http://www.mailpiler.org/wiki/current:index however.


Janos: FYI the website seems broken. I guess "Please fill or disable 
this placeholder" is not something that is supposed to be there :)


Regards,
Martin

Am 2020-07-22 17:51, schrieb Marcelo Machado:

Hello everyone.

Is Mail Piler only commercial now?

Does the Open Source Edition still exist?

Marcelo Gomes




Re: Upgrade or reinstall on newer OS version ?

2020-01-10 Thread Martin Nadvornik
Hello Yann,

I too think Janos' suggestions are good. From my experience in migrating
a piler instance with ~15 million mails from one vm to another I would
suggest to only move to a completely fresh machine if you don't mind a
lengthy migration process. An OS and database upgrade is in my point of
view always easier than migrating. It was a very painful process when I
did a migration from our archive.

If you want to migrate to a new machine then I can suggest to follow
roughly the following steps I you want to migrate without downtime for
your users
1. set up new machine and test if piler works correctly
2. export a sample set from you old archive (like a couple 1000 mails
for example)
3. import the sample set and make sure import, indexing and searching
works as expected
4. point your email servers at your new machine but keep it sending to
your old archive additionally
5. export all mails from your old archive
6. import all mails in your new archive
7. test if everything works correctly in the new archive
8. switch your users to the new archive

Also keep a close eye on your syslog and mail.log while importing all
mails. When I did a migration of 15 million mails I filled up the log
partition in a short amount of time which led to a crash of pilerimport.
I also had various other issues when I imported the millions of mails I
exported from the old archive because of broken headers or malformed
content leading to segfault, detection as duplicate altough they weren't
and so on. So make sure you log everythin that happens when you import a
lot messages and analyse the log to make sure every mail was correctly
imported and nothing was lost.

I have never switched a piler database from mysql to mariadb but
generally there shouldn't be any issue. Just execute mysql_upgrade after
the update and you should be good to go. Don't forget to have working
backups of course before you do anything.

Best wishes
Martin

Am 10.01.2020 um 12:43 schrieb Jason Morrill:
> I think Janos offers good thoughts.
>
> Personally I would keep the old VM running and install everything
> fresh. Then point your email servers at this new Piler VM. Next,
> either migrate info from the old system or just keep it and don't
> migrate. Of course, not migrating would mean you need to have two
> separate systems when it comes time to look for old messages.
>
> I've never performed a migration and don't know what it entails so
> perhaps it's rather painful. But from what I've seen in this forum
> Janos is very responsive with questions.
>
> Oh, and one last thought ... you could just keep the old system
> running as-is forever. Just be certain you've got it reasonably locked
> down in terms of security and access. My personal motto is "if it
> ain't broke, don't fix it" and have been running Piler since 2014 on a
> Linux box that's never seen an upgrade in 5-years.
>
> Best of luck!
>
>   Jason Morrill
>
>   IT Director
>
>   Child and Family Agency
>
>   860-443-2896 x1422
>
>   http://www.childandfamilyagency.org/
>
>
> 
> *From:* Janos SUTO 
> *Sent:* Friday, January 10, 2020 4:12 AM
> *To:* Piler User 
> *Subject:* Re: Upgrade or reinstall on newer OS version ?
>  
> Hello Yann,
>
> I have no experience upgrading through several LTS versions. A mysql
> -> mariadb switch should not hurt. Anyway, be sure to backup the
> database, or even better snapshot the whole vm. When the os upgrade is
> ready, be sure to recompile Piler.
>
> Be sure to stop all piler related processes even the ctontabs while
> upgrading the os.
>
> The other option could be to start a new vm with the choice of a
> recent distro with the latest master of Piler. Then export all emails
> from old server, and import to new one. This approach has the benefit
> of having everything new and up to date, however you need a lengthy
> import process.
>
> Janos
> On 10 Jan 2020, at 09:18, Yann Lehmann  > wrote:
>
>
> Hello Everyone
>
> Our Piler server is doing good so far (only service on a virtual 
> machine), but it is running on Ubuntu 14.04, which is no more supported.
>
> I would like to either upgrade to Ubuntu 18.04 (actual LTS, or may be 
> wait some time and go to 20.04 the next LTS) or move to Debian 10. I 
> know that on Ubuntu, an in place upgrade is only possible from one LTS 
> version to the next, so in my case, it would be from 14.04 to 16.04, and 
> then to 18.04.
>
>
> I also know that if I upgrade from Ubuntu 14.04 to 16.04, I would have 
> to upgrade Mysql (or move to MariaDB) before the upgrade.
>
> What path would you advise: in place upgrade or move/reinstall on newer 
> os ? And what were your experiences with in place upgrades on Ubuntu ?
>
> Thank's in advance for any advice and best regards
> Yann
>
>
>


Re: groups based on LDAP attribute

2020-01-07 Thread Martin Nadvornik



Hello Janos,

although my answer is a bit late, I still want to thank you for your
explanation and example. It worked as expected for our use case. I
already knew about the custom authentication function but I never
thought of performing an ldap bind there to get the required addresses
and add them to the session data.

Thanks,
Martin

Am 22.11.2019 um 21:46 schrieb s...@acts.hu:
>
>
> Hello Martin,
>
> my first idea was the group feature until I saw you had ruled it out.
>
> Then the custom email query function is to the rescue, see
> http://www.mailpiler.org/wiki/current:custom-authentication for more.
>
> So let's say you have an ldap attribute called 'aaa' with value 'bbb'
> having the required email addresses, and the manager is
> mana...@yourdomain.com.
>
> Then put a similar function to config-site.php:
>
> $config['CUSTOM_EMAIL_QUERY_FUNCTION'] = 'my_custom_func';
>
> function my_custom_func($username = '') {
>    $session = Registry::get('session');
>    $data = $session->get("auth_data");
>
>    $extra_emails = [];
>
>    if($data['email'] == 'mana...@yourdomain.com') {
>   $ldap = new LDAP($ldap_host, $ldap_port, $ldap_helper_dn,
> $ldap_helper_password);
>
>   if($ldap->is_bind_ok()) {
>  $query = $ldap->query($ldap_base_dn, "aaa=bbb");
>  $extra_emails = $query->rows;
>   }
>    }
>
>    $data['emails'] = array_merge($data['emails'] , $extra_emails);
>
>    $session->set("auth_data", $data);
> }
>
> Note that you may have to fine tune the ldap query, need to do some
> polishing on the $query->rows
> part. Also be sure to check out checkLoginAgainstLDAP_real() function
> in model/user/auth.php
> to see how it works.
>
>
> Janos
>
> On 2019-11-22 20:30, Martin Nadvornik wrote:
>> Hello fellow piler users,
>>
>> I am currently trying to find a way to give managers the ability to
>> search e-mails of employees who are not anymore with the company. I know
>> that in general you can give the owner of one address the ability to
>> access other addresses by using an alias or distribution list. However
>> in our case the e-mail accounts of prior employees don't exist anymore
>> and their e-mail addresses should not be reachable. Therore using alias
>> or distribution lists are not an option.
>> It would be possible of course to create groups directly in piler but
>> maintaining such groups by hand is not really efficient. So I am looking
>> for a way to use the group feature using LDAP.
>> The easiest method for us would be if there was a config value that just
>> tells piler which LDAP attribute to use and just allows the logged in
>> user access to every addresses listed in the attribute. But this needs
>> some development effort as far as I can tell.
>>
>> Is there anyone who has done this before? Any recommendations to solve
>> this are welcome.
>>
>> Thanks,
>> Martin
>

-- 
Martin Nadvornik
IT Mitarbeiter

Diakonie Flüchtlingsdienst gem. GmbH
Steinergasse 3/12
1170 Wien, Austria
Tel: +43 (0) 1/ 402 67 54 - 1337
Fax: +43 (0) 1/ 402 67 54 - 16
Mobil: 0664/88350692
http://www.diakonie.at/fluechtlingsdienst

Verantwortung kann man nicht abschieben!




groups based on LDAP attribute

2019-11-22 Thread Martin Nadvornik



Hello fellow piler users,

I am currently trying to find a way to give managers the ability to
search e-mails of employees who are not anymore with the company. I know
that in general you can give the owner of one address the ability to
access other addresses by using an alias or distribution list. However
in our case the e-mail accounts of prior employees don't exist anymore
and their e-mail addresses should not be reachable. Therore using alias
or distribution lists are not an option.
It would be possible of course to create groups directly in piler but
maintaining such groups by hand is not really efficient. So I am looking
for a way to use the group feature using LDAP.
The easiest method for us would be if there was a config value that just
tells piler which LDAP attribute to use and just allows the logged in
user access to every addresses listed in the attribute. But this needs
some development effort as far as I can tell.

Is there anyone who has done this before? Any recommendations to solve
this are welcome.

Thanks,
Martin



Re: Forward messages from another server

2019-10-14 Thread Martin Nadvornik
Hi Marcelo,

if all messages are processed by postfix then /always_bcc/ should work
as described in http://www.mailpiler.org/wiki/current:installation
However be aware that depending on your setup, adding the /X-Envelope-To
Header/ to all mails could leak all bcc recipients if this header is
also included in outgoing mails.

Personally, I prefer using plus addressing. For this you just need to
add a file like /pcre:/etc/postfix/bcc_map /to the
parameter/sender_bcc_maps. /This file need to have the following content:
/^([^+]+).*@(.*)$/  archive+$1=$2@
You can read more about plus-addressing in piler here:
https://bitbucket.org/jsuto/piler/issues/335/postfix-related-x-envelope-to-reveals-bcc

Martin

Am 14.10.2019 um 17:44 schrieb Marcelo Machado:
> Thanks for your quick reply.
> I think I expressed myself poorly.
> All messages to and from Exchange go through Postfix. I need all these
> messages to be forwarded to MailPiler.
>
> Marcelo
>
> Em seg, 14 de out de 2019 às 11:11, Janos SUTO  <mailto:s...@acts.hu>> escreveu:
>
> Hello Marcelo,
>
> assuming you want to archive internal emails as well that provably
> never reach the postfix relay, I'd rather setup journaling on
> exchange.
>
> Janos
> 
> *From:* Marcelo Machado
> *Sent:* Mon Oct 14 15:32:45 GMT+02:00 2019
> *To:* Piler User
> *Subject:* Forward messages from another server
>
> I have a postfix that works as relay from an Exchange server. How
> do I get postfix to forward these messages to Mailpiler?
>
> Marcelo Gomes
>

-- 
Martin Nadvornik
IT Mitarbeiter

Diakonie Flüchtlingsdienst gem. GmbH
Steinergasse 3/12
1170 Wien, Austria
Tel: +43 (0) 1/ 402 67 54 - 1337
Fax: +43 (0) 1/ 402 67 54 - 16
Mobil: 0664/88350692
http://www.diakonie.at/fluechtlingsdienst

Verantwortung kann man nicht abschieben!



Re: Indexation of Excel files newer than 2007

2019-05-06 Thread Martin Nadvornik
Hello Christian,

catdoc is not capable of processing new office formats. As far as I know
there is no intention for catdoc to implement this in a foreseeable
future. The same problem exists for xls2csv. You could theoretically try
to call unoconv (https://github.com/unoconv/unoconv) before catdoc, but
it will probably have a big performance impact since it launches libre
office / open office for the conversion. But if you try this I would be
interested in your results since being limited to index only old office
formats is also something we would like to overcome. Alternatively if
you can find an open source software which is capable of efficiently
extracting plain text from current office formats it should be easily
implementable into piler (basically a few lines in extract.c as far as I
can tell). For excel there is https://github.com/xevo/xls2csv and
https://github.com/nagirrab/xls2csv which claim to be cabable of
proccessing xlsx files. But I haven't looked into them yet.

Kind Regards
Martin

Am 06.05.2019 um 06:45 schrieb Katterl Christian:
>
> Hello,
>
>  
>
> Indexation of Excel files newer than Excel 2007 fails in my installation.
>
> I am using catdoc 0.95 and it tells:
>
>  
>
> This file looks like ZIP archive or Office 2007 or later file.
>
> Not supported by catdoc
>
>  
>
> The Excel-File has been created using Excel 2010.
>
>  
>
> BR, Christian
>
>
>
> *ChristianKatterl*
> Teamleader Technical IT
>
>
>
> *Asamer Baustoffe AG*
> Unterthalham Straße 2
> 4694 Ohlsdorf
> Austria
> *tel * +43 50 799 - 2511
> *mobile * +43 664 811 54 99
> *email * c.katt...@asamer.at 
> *www.abag.at* 
>
>
> This message is confidential. It may not be disclosed to, or used by,
> anyone other than the addressee. If you receive this message by
> mistake, please advise the sender.
> Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334
>
>