Re: I want to know if piler can do the following

2022-06-05 Thread Janos SUTO
Unfortunately, piler is not an imap server, so it cannot do what you asked.

Janos

On Jun 6, 2022, 02:06, at 02:06, Asmodean Thor  wrote:
>I have decided to offload my mail from google, and look for another
>provider in the future.
>
>This is the solution I want:
>Google send and receive —> POP3 to piler on VPS —> delete from google
>server —> pull via IMAP from piler to various clients
>
>Thank you!


Re: Reindex Question / Clarification

2020-09-21 Thread Janos SUTO
Hello Ryan,

reindexing won't cause any waste in sphinx, so go ahead.

Btw. you have several main indices, eg. main1, main2, etc. It's possible to use 
main1 for 2017 data, main2 for 2018 data, etc. In that case you may move main1 
to a backup media to free up some space. The trick is to fix both 
indexer.delta.sh and indexer.main.sh to reference the given main index.

Janos

On 22 Sep 2020, 02:37, at 02:37, Ryan Blenis  wrote:
>Hi Janos,
>
>Quick question on reindexing and its effect on the size of the sphinx
>files.
>
>If I reindex something that is already in the index, will there be
>duplicates in sphinx wasting space, or is this detected and removed?
>
>Background: We weren't searching for anything old normally, so we
>cleared
>the sphinx data and only indexed the last years worth of data. Now we
>have
>requests for data from parts of 2018, 2017, etc. that we've reindexed
>as
>needed to make searchable, but the more requests come in it seems it
>would
>be better to reindex everything and just throw resources at it if
>needed
>and cut out the "manual" process of manually indexing anything prior to
>a
>year ago as the requests come in. If I reindex everything, will I be
>wasting space and should start fresh, or will the duplicates be
>filtered
>out at some point in the process (e.g. not wasting space)?
>
>Thank you as always.


Re: Piler 1.35 on Ubuntu 18.04 - Web UI doesn't show mail headers / contents

2020-07-31 Thread Janos SUTO
Hello Patrick,

fix the piler binary paths in config-site.php. Check out the current values in 
config.php.

Janos SUTO

On 31 Jul 2020, 14:07, at 14:07, "Wagner, Patrick"  
wrote:
>
>
>Hello everyone,
>
>
>just opened an older piler archive (version 1.35) and was surprised to
>see all the emails listed in the web GUI upper pane / list, including
>icons for attachments and correct messages sizes, but I cannot display
>the actual email headers or contents.
>
>The lower pane / detail pane per email is pretty much empty:
>
>
>Subject:
>From:
>To:
>Date:
>
>
>
>Attachment names are shown, however every attachment I try to open is 1
>
>bytes on download, so obviously unusable.
>
>Thankfully it seems to be a web GUI interface only, as a test with
>pilerexport yielded the email contents, including attachments.
>
>What's wrong? Is the Piler version too old for modern browsers? What do
>
>I need to fix?
>
>Thanks,
>
>Patrick
>
>
># piler -V
>piler 1.3.5, build 997, Janos SUTO 
>
>Build Date: Sun Apr 21 16:50:30 UTC 2019
>ldd version: ldd (Ubuntu GLIBC 2.27-3ubuntu1) 2.27
>gcc version: gcc version 7.3.0 (Ubuntu 7.3.0-27ubuntu1~18.04)
>OS: Linux 188ae4f9894f 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13
>10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>Configure command: ./configure --prefix=/usr --sysconfdir=/etc
>--localstatedir=/var --with-database=mariadb
>MySQL client library version: 10.2.9
>Extractors: /usr/bin/pdftotext /usr/bin/catdoc /usr/bin/catppt
>/usr/bin/xls2csv /usr/bin/unrtf /usr/bin/tnef
>
>Ubuntu 18.04.4 LTS
>
>PHP 7.2.24-0ubuntu0.18.04.6 (as supplied with the distro)
>
>MariaDB 10.1.44 (as supplied with the distro)
>
>
>*
>*


Re: Empty message header and body

2020-07-15 Thread Janos SUTO
Sorry, piler.key is a file, not an entry in piler.conf.

Janos

On 15 Jul 2020, 12:25, at 12:25, Marcelo Machado  wrote:
>Thanks for the answer.
>
>The parameter iv is cipher_list.There is no line with piler.key.
> I was unable to read the contents of piler.key.
>
>Marcelo
>
>Em qua., 15 de jul. de 2020 às 00:46, Janos SUTO  escreveu:
>
>> Hello Marcelo,
>>
>> check if piler.key and the iv parameter in piler.conf (if there's
>any) are
>> the same.
>>
>> Janos
>> On 14 Jul 2020, at 23:42, Marcelo Machado  wrote:
>>>
>>> Hello everyone.
>>>
>>> After I migrated MailPiler to a new hardware, the header and body of
>old
>>> messages no longer appear. The new ones are OK.
>>>
>>> I followed the migration documentation on the website.
>>>
>>> Marcelo Gomes
>>>
>>


Re: Empty message header and body

2020-07-14 Thread Janos SUTO
Hello Marcelo,

check if piler.key and the iv parameter in piler.conf (if there's any) are the 
same.

Janos

On 14 Jul 2020, 23:42, at 23:42, Marcelo Machado  wrote:
>Hello everyone.
>
>After I migrated MailPiler to a new hardware, the header and body of
>old
>messages no longer appear. The new ones are OK.
>
>I followed the migration documentation on the website.
>
>Marcelo Gomes


Re: Upgrade Piler 1.3.7 -> 1.3.8 / Upgrade Sphinx

2020-04-21 Thread Janos SUTO
Hello,

well, you need that python module, please install it.

And yes, I recommend sphinx 3.2.1 for a new installation. The 2.2.x series is 
not maintained by the sphinx team apparently. It's safe to upgrade from 3.1.1 
to 3.2.1.

Janos

On 22 Apr 2020, 00:58, at 00:58, "BKH-Netz.de - Postfach" 
 wrote:
>
>
>Hello,
>
>i upgraded my piler installation 1.3.7 to 1.3.8 on a Debian 9 (Buster).
>After deleting some mail i wanted to purge them using pilerpurge.
>
>Doing this i got an error:
>
>Traceback (most recent call last):
>  File "/usr/local/libexec/piler/pilerpurge.py", line 4, in 
>import MySQLdb as dbapi
>ModuleNotFoundError: No module named ‚MySQLdb'
>
>Using piler 1.3.7 i have never seen this error …
>
>—
>
>Do you recommend an upgrade to sphinx 3.2.1 on a new and fresh
>installation?
>Can i upgrade from sphinx 3.1.1 to 3.2.1 on an existing installation?
>
>Thanks :-)


Re: Wrong dates

2020-02-04 Thread Janos SUTO
Hello,

click on view headers, and paste the date header line here.

Janos

On 4 Feb 2020, 12:15, at 12:15, Marcelo Machado  wrote:
>Hello everyone.
>
>Can you help me why the dates are different?
>
>Marcelo
>
>
>[image: tempsnip.png]


Re: Upgrade or reinstall on newer OS version ?

2020-01-10 Thread Janos SUTO
Hello Yann,

I have no experience upgrading through several LTS versions. A mysql -> mariadb 
switch should not hurt. Anyway, be sure to backup the database, or even better 
snapshot the whole vm. When the os upgrade is ready, be sure to recompile Piler.

Be sure to stop all piler related processes even the ctontabs while upgrading 
the os.

The other option could be to start a new vm with the choice of a recent distro 
with the latest master of Piler. Then export all emails from old server, and 
import to new one. This approach has the benefit of having everything new and 
up to date, however you need a lengthy import process.

Janos


On 10 Jan 2020, 09:18, at 09:18, Yann Lehmann  wrote:
>
>
>Hello Everyone
>
>Our Piler server is doing good so far (only service on a virtual
>machine), but it is running on Ubuntu 14.04, which is no more
>supported.
>
>I would like to either upgrade to Ubuntu 18.04 (actual LTS, or may be 
>wait some time and go to 20.04 the next LTS) or move to Debian 10. I 
>know that on Ubuntu, an in place upgrade is only possible from one LTS
>version to the next, so in my case, it would be from 14.04 to 16.04,
>and
>then to 18.04.
>
>
>I also know that if I upgrade from Ubuntu 14.04 to 16.04, I would have
>to upgrade Mysql (or move to MariaDB) before the upgrade.
>
>What path would you advise: in place upgrade or move/reinstall on newer
>
>os ? And what were your experiences with in place upgrades on Ubuntu ?
>
>Thank's in advance for any advice and best regards
>Yann


Re: Authentication against multiple IMAP servers

2019-10-31 Thread Janos SUTO
Hello,

then there must be a syntax error on the introduced function. Check out the 
error logs of the webserver.

If it doesn't help, then show me the function in a private email.

Janos


 Original Message 
From: BKH - Post 
Sent: Thu Oct 31 11:25:37 GMT+01:00 2019
To: piler-user@list.acts.hu
Subject: Re: Authentication against multiple IMAP servers



Hello Janos,

i tried this and set CUSTOM_PRE_AUTH_FUNCTION as described in the docs.
For testing reasons i copied your "my_imap"-script into config-site.php.

After this tried to log into piler with different users but the login 
page is shown as acompletely blank page.
Disabling the script, which is mentioned in the docs, login page is 
shown correctly.

I changes domain values to my domains but this error persists.

Can you help me with this?

BKH

Am 29.10.19 um 20:20 schrieb s...@acts.hu:
>
>
> Hello,
>
> I've just updated the custom authentication docs at
> http://www.mailpiler.org/wiki/current:custom-authentication
>
> Please check out the CUSTOM_PRE_AUTH_FUNCTION feature at the end.
>
> Janos
>
> On 2019-10-29 10:12, BKH - Post wrote:
>> Hi @all,
>>
>> is there a possibility to establish an authentication against multiple
>> IMAP servers?
>> Having here a scenario with two to three internal IMAP servers in
>> different workgroups i don't want to setup three mailpiler instances.
>> In fact that setting setting up such system is not a problem i want to
>> reduce administrative tasks.
>>
>> Thanks for your advice.
>>
>> Regards
>>
>> BKH
>



Re: Upgrade from 1.3.5 -> 1.3.7

2019-10-25 Thread Janos SUTO
Hello,

I'll clarify this in the upgrade docs, however you may overwrite all files 
(except the configs), because there hasn't been any incompatible change.

As a precaution, always backup your config files and especially the encryption 
key.

Janos


 Original Message 
From: BKH - Post 
Sent: Fri Oct 25 12:01:35 GMT+02:00 2019
To: piler-user@list.acts.hu
Subject: Upgrade from 1.3.5 -> 1.3.7



Hi @all,

version 1.3.7 of mailpiler is out.
After reading upgrade instructions i found nothing regarding version 1.3.7.
As i want to do an upgraade of piler form the first time i need some 
hints to do this.

It is clear that making a backup of all configs is recommended.
Can i copy over all files?

Thanks for your advice

Regards

BKH




Re: Forward messages from another server

2019-10-14 Thread Janos SUTO
Hello Marcelo,

assuming you want to archive internal emails as well that provably never reach 
the postfix relay, I'd rather setup journaling on exchange.

Janos


 Original Message 
From: Marcelo Machado 
Sent: Mon Oct 14 15:32:45 GMT+02:00 2019
To: Piler User 
Subject: Forward messages from another server

I have a postfix that works as relay from an Exchange server. How do I get
postfix to forward these messages to Mailpiler?

Marcelo Gomes


Re: Cronjobs - change indexing interval

2019-08-24 Thread Janos SUTO
Hello,

try this:

*/10 * * * * 

Btw. check the mail log to see if it works as required. Whenever the indexer 
script starts it syslogs it.

Janos


 Original Message 
From: BKH 
Sent: Fri Aug 23 21:35:04 GMT+02:00 2019
To: piler-user@list.acts.hu
Subject: Cronjobs - change indexing interval



Hello,

for testing reasons i tried changing indexing interval.
I played around with this line:

5,35 * * * * /usr/local/libexec/piler/indexer.delta.sh

I changed minute an hour to differnte values, but could not figure out
to get it in a working state.
Indexing should take place e.g. every 10 minutes.
How can i achive this?

Thanks :-)

BKH





Re: Indexation of Excel files newer than 2007

2019-05-08 Thread Janos SUTO
Try the following. Get the latest master branch, and run the configure command. 
It prints a summary at the end including libzip status.

Anyway it's also possible to adopt the Tika project as adminshok suggested.

Janos


 Original Message 
From: Katterl Christian 
Sent: Wed May 08 10:52:33 GMT+02:00 2019
To: Piler User 
Subject: AW: Indexation of Excel files newer than 2007

In at least my case, this does not seem to work.

BR, Christian



Von: Janos SUTO 
Gesendet: Montag, 6. Mai 2019 11:33
An: Piler User 
Betreff: Re: Indexation of Excel files newer than 2007

Newer office files, eg. xlsx, etc should be handled internally by the parser, 
provided that you have libzip package installed as well as the header files, 
libzip-dev or similar.
Janos


 Christian Katterl
Teamleader Technical IT

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel +43 50 799 - 2511

mobile  +43 664 811 54 99
c.katt...@asamer.at
www.abag.at


This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this
message by mistake, please advise the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334


From: Katterl Christian
Sent: Mon May 06 10:19:07 GMT+02:00 2019
To: Piler User
Subject: AW: Indexation of Excel files newer than 2007


Hello again,


for docx, there would be: https://github.com/ankushshah89/python-docx2txt


Unfortunately, I am not a software-developer to make the adoptions by myself.


BR Christian




Von: Martin Nadvornik 
mailto:martin.nadvor...@diakonie.at>>
Gesendet: Montag, 6. Mai 2019 09:46
An: Piler User mailto:piler-user@list.acts.hu>>
Betreff: Re: Indexation of Excel files newer than 2007


Hello Christian,

catdoc is not capable of processing new office formats. As far as I know there 
is no intention for catdoc to implement this in a foreseeable future. The same 
problem exists for xls2csv. You could theoretically try to call unoconv 
(https://github.com/unoconv/unoconv) before catdoc, but it will probably have a 
big performance impact since it launches libre office / open office for the 
conversion. But if you try this I would be interested in your results since 
being limited to index only old office formats is also something we would like 
to overcome. Alternatively if you can find an open source software which is 
capable of efficiently extracting plain text from current office formats it 
should be easily implementable into piler (basically a few lines in extract.c 
as far as I can tell). For excel there is https://github.com/xevo/xls2csv and 
https://github.com/nagirrab/xls2csv which claim to be cabable of proccessing 
xlsx files. But I haven't looked into them yet.

Kind Regards
Martin
Am 06.05.2019 um 06:45 schrieb Katterl Christian:
Hello,

Indexation of Excel files newer than Excel 2007 fails in my installation.
I am using catdoc 0.95 and it tells:

This file looks like ZIP archive or Office 2007 or later file.
Not supported by catdoc

The Excel-File has been created using Excel 2010.

BR, Christian


Christian Katterl
Teamleader Technical IT

[cid:image001.png@01D503F5.230293E0]

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel  +43 50 799 - 2511
mobile +43 664 811 54 99
email c.katt...@asamer.at<mailto:c.katt...@asamer.at>
www.abag.at<https://www.abag.at>

This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this message by mistake, please advise 
the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334





Re: Indexation of Excel files newer than 2007

2019-05-06 Thread Janos SUTO
Newer office files, eg. xlsx, etc should be handled internally by the parser, 
provided that you have libzip package installed as well as the header files, 
libzip-dev or similar.

Janos


 Original Message 
From: Katterl Christian 
Sent: Mon May 06 10:19:07 GMT+02:00 2019
To: Piler User 
Subject: AW: Indexation of Excel files newer than 2007

Hello again,

for docx, there would be: https://github.com/ankushshah89/python-docx2txt

Unfortunately, I am not a software-developer to make the adoptions by myself.

BR Christian


Von: Martin Nadvornik 
Gesendet: Montag, 6. Mai 2019 09:46
An: Piler User 
Betreff: Re: Indexation of Excel files newer than 2007

Hello Christian,

catdoc is not capable of processing new office formats. As far as I know there 
is no intention for catdoc to implement this in a foreseeable future. The same 
problem exists for xls2csv. You could theoretically try to call unoconv 
(https://github.com/unoconv/unoconv) before catdoc, but it will probably have a 
big performance impact since it launches libre office / open office for the 
conversion. But if you try this I would be interested in your results since 
being limited to index only old office formats is also something we would like 
to overcome. Alternatively if you can find an open source software which is 
capable of efficiently extracting plain text from current office formats it 
should be easily implementable into piler (basically a few lines in extract.c 
as far as I can tell). For excel there is https://github.com/xevo/xls2csv and 
https://github.com/nagirrab/xls2csv which claim to be cabable of proccessing 
xlsx files. But I haven't looked into them yet.

Kind Regards
Martin
Am 06.05.2019 um 06:45 schrieb Katterl Christian:
Hello,

Indexation of Excel files newer than Excel 2007 fails in my installation.
I am using catdoc 0.95 and it tells:

This file looks like ZIP archive or Office 2007 or later file.
Not supported by catdoc

The Excel-File has been created using Excel 2010.

BR, Christian


Christian Katterl
Teamleader Technical IT

[cid:image001.png@01D503F5.230293E0]

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel  +43 50 799 - 2511
mobile +43 664 811 54 99
email c.katt...@asamer.at
www.abag.at


This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this message by mistake, please advise 
the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334



 Christian Katterl
Teamleader Technical IT

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel +43 50 799 - 2511

mobile  +43 664 811 54 99
c.katt...@asamer.at
www.abag.at


This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this
message by mistake, please advise the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334




Re: How to configure info@ adresses?

2019-05-03 Thread Janos SUTO
Hello,

I think it can be done using the group feature. Its purpose is to give access 
to another mailbox as well.

You may setup the group settings as admin in the GUI.

Janos


 Original Message 
From: Biderius - 
Sent: Fri May 03 12:34:41 GMT+02:00 2019
To: "piler-user@list.acts.hu" 
Subject: How to configure info@ adresses?

Hello,
I have configured piler using my LDAP as login-service. When I log in, I can 
see all my messages associated with my user-account.

But now, I'm facing the problem, that our company uses an info@ addresss which 
is splitted to severel agents via "virtual-transports" tables in postfix. How 
can I see these messages when I log in to piler?
I already tried to configure the $config[LDAP_MAIL_ATTR] param, but when I use 
this, I cant log in to piler.
How can I see both e-mail addresses?



Re: Downloaded files corrupt

2019-04-30 Thread Janos SUTO
Hello,

what language file do you use?
Can reproduce the issue using English?

Janos


 Original Message 
From: Katterl Christian 
Sent: Tue Apr 30 05:45:59 GMT+02:00 2019
To: "piler-user@list.acts.hu" 
Subject: Downloaded files corrupt

Hello,

when I try to download archived files (attachments), I noticed, that the files 
are corrupted after download.
They are 1 byte larger than the original.

I found this here:
https://bitbucket.org/jsuto/piler/issues/949/corrupted-files-are-downloaded-from-web-ui

I even tried the language-files of the current master-tree, where I assume, the 
issue is fixed?

BR, Christian


 Christian Katterl
Teamleader Technical IT

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel +43 50 799 - 2511

mobile  +43 664 811 54 99
c.katt...@asamer.at
www.abag.at


This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this
message by mistake, please advise the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334




Re: Rentention-rules and pilerpurge

2018-11-14 Thread Janos SUTO
Hello Ryan,

you are right about the verification hash problem. However, since you are about 
to purge those messages, it's probably not that big issue.

Btw. the GUI compares only the message hash values, and the rest of the 
metadata table entries (eg. the retained column) are not used currently.

Janos


 Original Message 
From: Ryan Blenis 
Sent: Tue Nov 13 23:08:59 GMT+01:00 2018
To: piler-user@list.acts.hu
Subject: Re: Rentention-rules and pilerpurge

> > Do you have any idea what I should
> > do  so that more messages are being purged? Especially the retention
> > period seems suspect I think...
> For already archived spam, you should identify them in the metadata
table, eg. check the subject
> field. Then set the retained column for such messages only to the past,
eg. a yesterday
> timestamp, and pilerpurge will take care of them.

Hello Janos,

Seeing your above reply to Marina's post regarding purging data brought me
back to an issue we were trying to clear up (we have a certain email "from"
address we'd like purged, as it is not required in the archive). However,
we lacked the foresight to set up retention rules prior to the email
account spewing out millions of emails.

I figured I'd just go in and manually set the retention date back in the
past as you suggest, however I found the following note on
http://www.mailpiler.org/wiki/current:administering-piler:

--
Note that the retention period is included in the per message verification
digest, so the retention period should not be changed after the message has
been archived.
--

I've run a test on the database by selecting one email and changing the
retention period to the past and running

pilerpurge -d

to see the effect, and it says it will remove it.

If this is true, what are the ramifications of this and the message
verification digest no longer aligning?

Is this a viable method to get rid of the cruft we've accumulated and would
like to purge?

Thank you.


Re: var/piler/tmp files

2018-10-18 Thread Janos SUTO
Hello Kim,

you should look at the mail log to find out what happened. There might be some 
parsing or encoding issues.

Janos


 Original Message 
From: "Kim Mølgaard" 
Sent: Thu Oct 18 09:18:52 GMT+02:00 2018
To: piler-user@list.acts.hu
Subject: var/piler/tmp files

Hi !
Piler v. 1.3.6 on Ubuntu 16.04.
My /var/piler/tmp is filled with files, lots of files.
I clean it out 3 times a day, by stopping piler and removing all files, but 
that can't be right.


Do anyone have any suggestions to what is going on, and how to deal with it ?
Regards

Kim Mølgaard
ITandMore
Tlf.: +45 93 89 83 79

CVR nr. 38147110



Re: Upgrading from old piler to new

2018-10-05 Thread Janos SUTO
Hello,

well, the official site has a section for the upgrade. It should work.

It's worth to backup the MySQL database, your configs and the piler.key.

Janos


 Original Message 
From: Michael Mansour 
Sent: Fri Oct 05 01:47:18 GMT+02:00 2018
To: Piler User 
Subject: Upgrading from old piler to new



Hi,

I have an older Mail Piler version and would like to update to the latest.

Are there any upgrade docs on-line or similar that I may look at so the 
upgrade is smooth?

Thanks.

Michael.


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus




Re: Moved piler VM and then rsync'ed piler store

2018-10-04 Thread Janos SUTO
Hello Michael,

check the ownership on all the directories down to the .m file.

Janos


 Original Message 
From: Michael Mansour 
Sent: Thu Oct 04 14:13:49 GMT+02:00 2018
To: piler-user@list.acts.hu
Subject: Moved piler VM and then rsync'ed piler store



Hi Piler users,

I had to shutdown a data centre and move to another data centre.

The original piler VM was an easy copy from one to the other, all worked.

The store here:

/var/piler/store

was on an NFS share in the old data centre. I create the same NFS in the 
new data centre, then rsync'ed using:

rsync -avz etc etc

from one to the other, for everything in /var/piler/store.

Everything moved across, then I change permissions on the new 
/var/piler/store using:

cd /var/piler/store/
chown -R piler:piler 00
find . -type d -exec chmod 750 {} \;
find . -type f -exec chmod 640 {} \;

This ran for a while but eventually completed.

When I got to the Piler interface with Auditor account, I can see the 
messages there but when clicking on any of the messages I can't view 
them and instead, get this in the logs:

Oct  4 21:52:40 server pilerget[19390]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()
Oct  4 21:52:40 server pilerget[19393]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()
Oct  4 21:52:49 server pilerget[19400]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()
Oct  4 21:52:49 server pilerget[19403]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()
Oct  4 21:52:51 server pilerget[19408]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()
Oct  4 21:52:51 server pilerget[19411]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()
Oct  4 21:52:51 server pilerget[19416]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()
Oct  4 21:52:51 server pilerget[19419]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()

and we do see:

# ll /var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m
-rw-r- 1 piler piler 1072 Sep 28 23:01 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m

so permissions are correct.

Any ideas what I did wrong ?

Thanks.

Michael.


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus




Re: CentOS 7 support only for free version

2018-07-07 Thread Janos SUTO
Hello Steve,

give me some time, and I will check if I can make it on centos 7.

Janos


 Original Message 
From: Stephen Ingram 
Sent: Sun Jul 08 01:55:10 GMT+02:00 2018
To: piler-user@list.acts.hu
Subject: CentOS 7 support only for free version

Is there any way to get the commercial version on CentOS 7 or do we have to
use the free version if we want CentOS 7 support?

Steve


Re: Limit on searchs?

2018-06-20 Thread Janos SUTO
Hello Thiago,

surely not correct. In fact it's pretty bad. For starters check if the Sphinx 
database is fine.

Janos


 Original Message 
From: Thiago Lima 
Sent: Wed Jun 20 00:38:54 GMT+02:00 2018
To: Piler User 
Subject: Limit on searchs?

After a crash and rebuild of the databases, my mail piler just shows 50
results per page and doesn't allow to change pages.

All the searchs returns "total: 0" and I can only see the first page of
results.

Is that correct or there's something wrong?


RE: only sent mails are stored or also received?

2018-04-12 Thread Janos SUTO
What were your issues? Perhaps I can help with them.

Anyway, using pilerimport is possible, though it's not efficient as you have an 
ever growing mailbox, and pilerimport needs more and more time to process.

Janos


 Original Message 
From: Jackson Craig 
Sent: Thu Apr 12 15:52:18 GMT+02:00 2018
To: 'Piler User' 
Subject: RE: only sent mails are stored or also received?

Hi,

 

For me, all I added to postfix  - in the main.cf configuration file, was :

 

#Keep a copy of every email by sending bcc

always_bcc = b...@yourdomain.com  

 

Of course, replace the domain part :)  At the end of each day, I have all the 
email that has passed through the server sitting ready to be collected. I did 
initially experiment with realtime forwarding to piler, but I had various 
issues with that so just went with this instead.

 

Then on the piler server, I have a cron entry (important to run this as the 
piler user and not for example root) to collect the mail over pop3:

 

01 20 * * * /usr/local/bin/pilerimport –K your.mailserver -u bcc -p bcc

 

Hope this helps,

 

Regards

Craig

 

  -

This email and any attachments are confidential to the intended recipient 

and may also be privileged.  If you are not the intended recipient, please 

delete it from your system and notify the sender.  You should not copy it 

or use it for any purpose nor disclose or distribute its contents to any 

other person.

 

From: Lucian Lazar [mailto:lucian.la...@soteha.it] 
Sent: 12 April 2018 13:44
To: Piler User 
Subject: Re: only sent mails are stored or also received?

 

Thank you, we use Postfix as well for our mailing servers. Is there any 
reference in the documentation regarding how to setup this BCC behavior? In 
this case, all original headers will be preserved? Like mailto:, mailfrom: and 
the people that were in the CC and BCC fields?

Thank you again

-- 

Lucian Lazar - Linux System Engineer
Soteha S.r.l. a Socio Unico
Via Giuseppe Zucchi, 39C 20095 Cusano Milanino (MI)
Ingresso principale da 39/G interno 3
T. +39 02 66043166 | F. +39 02 66048942
  lucian.la...@soteha.it

From: Jackson Craig  >
Reply-To: Piler User  >
Date: Thursday, 12 April 2018 at 15:40
To: 'Piler User'  >
Subject: RE: only sent mails are stored or also received?

 

Hi there,

 

One way is to have your mailserver send a bcc of every mail to a separate 
mailbox on the server,  which is then collected by piler. For example here, 
using postfix, we configure a bcc mail for every mail that passes through 
postfix. This then puts a copy of every mail into a mailbox on the server 
(incoming and outgoing). Mailpiler then collects this mail by logging into the 
mailserver using POP3. We do this at the end of each day. Of course the exact 
configuration will depend on your mailserver.

 

Regards

Craig Jackson

 

  -

This email and any attachments are confidential to the intended recipient 

and may also be privileged.  If you are not the intended recipient, please 

delete it from your system and notify the sender.  You should not copy it 

or use it for any purpose nor disclose or distribute its contents to any 

other person.

 

From: Lucian Lazar [  
mailto:lucian.la...@soteha.it] 
Sent: 12 April 2018 13:21
To:   piler-user@list.acts.hu
Subject: only sent mails are stored or also received?

 

Hello,

We are actually evaluating a viable solution for our mail legal hold needs. We 
currently need to archive all sent and received emails for all users in a given 
mailserver.

Please correct me if i’m wrong, but i have red that only sendemail from the 
users are being saved into Piler database? Is there any way we can als store 
emails received by the users?

Thank you

 

-- 

Lucian Lazar - Linux System Engineer
Soteha S.r.l. a Socio Unico
Via Giuseppe Zucchi, 39C 20095 Cusano Milanino (MI)
Ingresso principale da 39/G interno 3
T. +39 02 66043166 | F. +39 02 66048942
  lucian.la...@soteha.it


Missiles & Space Batteries Limited is registered in Scotland under company 
number 162626. The company's registered office is at Hagmill Road, East 
Shawhead, Coatbridge, ML5 4UZ. 


Missiles & Space Batteries Limited is registered in Scotland under company
number 162626.  The company's registered office is at Hagmill Road, East
Shawhead, Coatbridge, ML5 4UZ.



Re: only sent mails are stored or also received?

2018-04-12 Thread Janos SUTO
Hello Lucian,

piler archives every email you send to it. Just configure your mail server to 
copy both sent and received emails to piler, and you'll be fine.

Janos


 Original Message 
From: Lucian Lazar 
Sent: Thu Apr 12 14:20:36 GMT+02:00 2018
To: "piler-user@list.acts.hu" 
Subject: only sent mails are stored or also received?

Hello,
We are actually evaluating a viable solution for our mail legal hold needs. We 
currently need to archive all sent and received emails for all users in a given 
mailserver.
Please correct me if i’m wrong, but i have red that only sendemail from the 
users are being saved into Piler database? Is there any way we can als store 
emails received by the users?
Thank you

--
Lucian Lazar - Linux System Engineer
Soteha S.r.l. a Socio Unico
Via Giuseppe Zucchi, 39C 20095 Cusano Milanino (MI)
Ingresso principale da 39/G interno 3
T. +39 02 66043166 | F. +39 02 66048942
lucian.la...@soteha.it


Re: BCC not shown

2018-02-12 Thread Janos SUTO
Hello Christian,

check the Sphinx query in the mail log. I suspect that your bcc address is not 
in the query.

The fix depends on what authentication you are using.


Janos


 Original Message 
From: "Christian Röser - PELMA" 
Sent: Mon Feb 12 17:37:09 GMT+01:00 2018
To: Piler User 
Subject: BCC not shown



Hello,

I need some help. Today I made some tests with my piler installation and I 
found a problem that drives me crazy - but I assume there could be simple 
configuration problem. Messages I receive as bcc are not shown in the search 
results.

I checked if the exchange server behaves correctly, here seems everything fine. 
The system generates two e-mails, one for the normal recipient and a separate 
one for the bcc recipient. Piler receive both e-mails, stores the first one and 
recognizes for the second e-mail that it have the same message-id. In the 
metadata table exists an entry and in the rcpt table two entries for this id. 
Nevertheless only in the search results of the "normal" recipient the message 
is shown.

Is the some parameter missing?

Best regards,
Christian






Re: Feature Request: Disable Restore to Mailbox

2017-11-29 Thread Janos SUTO
Hello,

you can disable the feature: set the smarthost parameter to an empty string.

Janos


 Original Message 
From: Katterl Christian 
Sent: Wed Nov 29 10:43:42 GMT+01:00 2017
To: Piler User 
Subject: Feature Request: Disable Restore to Mailbox

Hello,

as the „restore-to-mailbox“ feature sends an item using smtp, the original date 
and time is screwed up and the item get’s restored with the date and time of 
restoring.

Exporting to  EML would be the better option, in my opinion.
Therefore, it would be nice to have a config-switch to disable this export to 
mailbox completey.

BR, Christian


Christian Katterl
Teamleader Technical IT

[cid:01a2-0001@01d368f6.8b9afdbc]

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel  +43 50 799 - 2511
mobile +43 664 811 54 99
email c.katt...@asamer.at
www.abag.at



This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this message by mistake, please advise 
the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334



Re: AW: LDAP Users can see other's mails

2017-11-28 Thread Janos SUTO
Thanks, it's even better. I'll add it to the FAQ.

Janos


 Original Message 
From: Katterl Christian <c.katt...@asamer.at>
Sent: Tue Nov 28 08:47:20 GMT+01:00 2017
To: Piler User <piler-user@list.acts.hu>
Subject: AW: AW: LDAP Users can see other's mails

Update:
By modifying the parameter from $config['LDAP_DISTRIBUTIONLIST_OBJECTCLASS'] = 
'group'; to $config['LDAP_DISTRIBUTIONLIST_OBJECTCLASS'] = 'user'; it seems to 
accomplish the same without changing code in /var/www/piler/model/user/auth.php



Christian Katterl
Teamleader Technical IT

[cid:0139-0001@01d3681d.1f8ac597]

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel  +43 50 799 - 2511
mobile +43 664 811 54 99
email c.katt...@asamer.at<mailto:c.katt...@asamer.at>
www.abag.at<https://www.abag.at>



This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this message by mistake, please advise 
the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334

Von: Janos SUTO [mailto:s...@acts.hu]
Gesendet: Dienstag, 28. November 2017 08:11
An: Piler User
Betreff: Re: AW: LDAP Users can see other's mails

OK, if it gives you a proper result, then case is solved. Be sure to save your 
fix in case of a future upgrade. Or I may introduce a configure option to apply 
your fix.
Janos

From: Katterl Christian
Sent: Tue Nov 28 06:35:25 GMT+01:00 2017
To: Piler User
Subject: AW: AW: LDAP Users can see other's mails



Hello,

maybe I have found a solution for this issue.

In /var/www/piler/model/user/auth.php

I changed the line 217, which originally looked like this:

$query = $ldap->query($ldap_base_dn, 
"(|(&(objectClass=$ldap_account_objectclass)($ldap_mail_attr=$username_prefix$username))(&(objectClass=$ldap_distributionlist_objectclass)($ldap_distributionlist_attr=$username_prefix$username)"
 . 
")(&(objectClass=$ldap_distributionlist_objectclass)($ldap_distributionlist_attr="
 . $a['dn'] . ")))", array());

To only:
  $query = $ldap->query($ldap_base_dn, 
"(|(&(objectClass=$ldap_account_objectclass)($ldap_mail_attr=$username_prefix$username)))",
 array());

I mean - I removed all the group- and distribution-list things.
I am not sure, what this else will/could cause (I am not a programmer)?

But from what I saw in a very quick test, now only my personal emails are shown.

BR, Christian


 Christian Katterl
Teamleader Technical IT

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel +43 50 799 - 2511

mobile  +43 664 811 54 99
c.katt...@asamer.at<mailto:c.katt...@asamer.at>
www.abag.at<http://www.abag.at>


This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this
message by mistake, please advise the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334


-Ursprüngliche Nachricht-
Von: s...@acts.hu<mailto:s...@acts.hu> [mailto:s...@acts.hu]
Gesendet: Sonntag, 26. November 2017 17:56
An: Piler User
Betreff: Re: AW: LDAP Users can see other's mails



Hmm, it's odd. Even if a user is member of a group with other users which is 
totally normal a user still shouldn't see others' emails.

Some of the addresses look like some distribution lists. Can you show me such a 
message you can see and meant for someone else?
I'm interested the headers only. (You may send it privately to my address). The 
selected messages should not belong to any distribution list you are on.

Janos

On 2017-11-23 12:09, Katterl Christian wrote:

 It seems that i can see all messages of members of the same ad-groups.


 In my case, piler would not need to take care of groups…..

 VON: Janos SUTO [mailto:s...@acts.hu]
 GESENDET: Donnerstag, 23. November 2017 09:45
 AN: Piler User
 BETREFF: Re: LDAP Users can see other's mails

 Show me the sphinx query from the mail log related to the given user.

 Janos

 -

 FROM: Katterl Christian
 SENT: Thu Nov 23 07:35:19 GMT+01:00 2017
 TO: "piler-user@list.acts.hu<mailto:piler-user@list.acts.hu>"
 SUBJECT: LDAP Users can see other's mails

 Dear all,

 i configured piler (1.3.1) on Debian (9) using LDAP authentication
 against Active Directory.

 Basically, authentication works.

 BUT:

 Successfully logged in users cannot only see their own mails, but also
 mails of other users?

 My ldap-config from config-site.php looks like this:

 $config['ENABLE_LDAP_AUTH'] = 1;

 $config['LDAP_HOST'] = 'mydomaincontroller.mydomain.myforest.tld';

 $config['LDAP_HELPER_DN'] =
 'CN=pilerldap,OU=ServicesAccounts,DC=mydomain,DC=myforest,DC=tld';

 $config['LDAP_HELPER_PASSWORD'] = 'highpressurecompressor';

 $config['LDAP_MAIL_ATTR'] = 'mail';

 $config['LDAP_ACCOUNT_OBJECTCLASS'] = 'user';

 $config['LDAP_DISTRIBUTIONLIST_OBJECTCLASS'] = 'group';


Re: AW: LDAP Users can see other's mails

2017-11-27 Thread Janos SUTO
OK, if it gives you a proper result, then case is solved. Be sure to save your 
fix in case of a future upgrade. Or I may introduce a configure option to apply 
your fix.

Janos


 Original Message 
From: Katterl Christian <c.katt...@asamer.at>
Sent: Tue Nov 28 06:35:25 GMT+01:00 2017
To: Piler User <piler-user@list.acts.hu>
Subject: AW: AW: LDAP Users can see other's mails

Hello,

maybe I have found a solution for this issue.

In /var/www/piler/model/user/auth.php

I changed the line 217, which originally looked like this:

$query = $ldap->query($ldap_base_dn, 
"(|(&(objectClass=$ldap_account_objectclass)($ldap_mail_attr=$username_prefix$username))(&(objectClass=$ldap_distributionlist_objectclass)($ldap_distributionlist_attr=$username_prefix$username)"
 . 
")(&(objectClass=$ldap_distributionlist_objectclass)($ldap_distributionlist_attr="
 . $a['dn'] . ")))", array());

To only:
  $query = $ldap->query($ldap_base_dn, 
"(|(&(objectClass=$ldap_account_objectclass)($ldap_mail_attr=$username_prefix$username)))",
 array());

I mean - I removed all the group- and distribution-list things.
I am not sure, what this else will/could cause (I am not a programmer)?

But from what I saw in a very quick test, now only my personal emails are shown.

BR, Christian


 Christian Katterl
Teamleader Technical IT

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel +43 50 799 - 2511

mobile  +43 664 811 54 99
c.katt...@asamer.at
www.abag.at


This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this
message by mistake, please advise the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334


-Ursprüngliche Nachricht-
Von: s...@acts.hu [mailto:s...@acts.hu]
Gesendet: Sonntag, 26. November 2017 17:56
An: Piler User
Betreff: Re: AW: LDAP Users can see other's mails



Hmm, it's odd. Even if a user is member of a group with other users which is 
totally normal a user still shouldn't see others' emails.

Some of the addresses look like some distribution lists. Can you show me such a 
message you can see and meant for someone else?
I'm interested the headers only. (You may send it privately to my address). The 
selected messages should not belong to any distribution list you are on.

Janos

On 2017-11-23 12:09, Katterl Christian wrote:
> It seems that i can see all messages of members of the same ad-groups.
>
>
> In my case, piler would not need to take care of groups…..
>
> VON: Janos SUTO [mailto:s...@acts.hu]
> GESENDET: Donnerstag, 23. November 2017 09:45
> AN: Piler User
> BETREFF: Re: LDAP Users can see other's mails
>
> Show me the sphinx query from the mail log related to the given user.
>
> Janos
>
> -
>
> FROM: Katterl Christian
> SENT: Thu Nov 23 07:35:19 GMT+01:00 2017
> TO: "piler-user@list.acts.hu"
> SUBJECT: LDAP Users can see other's mails
>
> Dear all,
>
> i configured piler (1.3.1) on Debian (9) using LDAP authentication
> against Active Directory.
>
> Basically, authentication works.
>
> BUT:
>
> Successfully logged in users cannot only see their own mails, but also
> mails of other users?
>
> My ldap-config from config-site.php looks like this:
>
> $config['ENABLE_LDAP_AUTH'] = 1;
>
> $config['LDAP_HOST'] = 'mydomaincontroller.mydomain.myforest.tld';
>
> $config['LDAP_HELPER_DN'] =
> 'CN=pilerldap,OU=ServicesAccounts,DC=mydomain,DC=myforest,DC=tld';
>
> $config['LDAP_HELPER_PASSWORD'] = 'highpressurecompressor';
>
> $config['LDAP_MAIL_ATTR'] = 'mail';
>
> $config['LDAP_ACCOUNT_OBJECTCLASS'] = 'user';
>
> $config['LDAP_DISTRIBUTIONLIST_OBJECTCLASS'] = 'group';
>
> $config['LDAP_DISTRIBUTIONLIST_ATTR'] = 'member';
>
> $config['LDAP_BASE_DN'] = DC=mydomain,DC=myforest,DC=tld;
>
> $config['LDAP_AUDITOR_MEMBER_DN'] = '';
>
> $config['LDAP_ADMIN_MEMBER_DN'] = '';
>
> Any ideas?
>
> BR, Christian
>
> CHRISTIAN KATTERL
> Teamleader Technical IT
>
> ASAMER BAUSTOFFE AG
> Unterthalham Straße 2
> 4694 Ohlsdorf
> Austria
> TEL  +43 50 799 - 2511
> MOBILE +43 664 811 54 99
> EMAIL c.katt...@asamer.at
> WWW.ABAG.AT [1]
>
> This message is confidential. It may not be disclosed to, or used by,
> anyone other than the addressee. If you receive this message by
> mistake, please advise the sender.
> Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334
>
>
>
> Links:
> --
> [1] https://www.abag.at




Re: LDAP Users can see other's mails

2017-11-23 Thread Janos SUTO
Show me the sphinx query from the mail log related to the given user.

Janos


 Original Message 
From: Katterl Christian 
Sent: Thu Nov 23 07:35:19 GMT+01:00 2017
To: "piler-user@list.acts.hu" 
Subject: LDAP Users can see other's mails

Dear all,

i configured piler (1.3.1) on Debian (9) using LDAP authentication against 
Active Directory.
Basically, authentication works.

BUT:
Successfully logged in users cannot only see their own mails, but also mails of 
other users?

My ldap-config from config-site.php looks like this:

$config['ENABLE_LDAP_AUTH'] = 1;
$config['LDAP_HOST'] = 'mydomaincontroller.mydomain.myforest.tld';
$config['LDAP_HELPER_DN'] = 
'CN=pilerldap,OU=ServicesAccounts,DC=mydomain,DC=myforest,DC=tld';
$config['LDAP_HELPER_PASSWORD'] = 'highpressurecompressor';
$config['LDAP_MAIL_ATTR'] = 'mail';
$config['LDAP_ACCOUNT_OBJECTCLASS'] = 'user';
$config['LDAP_DISTRIBUTIONLIST_OBJECTCLASS'] = 'group';
$config['LDAP_DISTRIBUTIONLIST_ATTR'] = 'member';
$config['LDAP_BASE_DN'] = DC=mydomain,DC=myforest,DC=tld;
$config['LDAP_AUDITOR_MEMBER_DN'] = '';
$config['LDAP_ADMIN_MEMBER_DN'] = '';

Any ideas?

BR, Christian


Christian Katterl
Teamleader Technical IT

[cid:00d9-0001@01d36425.3b18d2bf]

Asamer Baustoffe AG
Unterthalham Stra?e 2
4694 Ohlsdorf
Austria
tel  +43 50 799 - 2511
mobile +43 664 811 54 99
email c.katt...@asamer.at
www.abag.at



This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this message by mistake, please advise 
the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334



Re: View all emails from a domain

2017-07-05 Thread Janos SUTO
Hello Marcelo,

well, something like that is possible. Set ENABLE_SAAS to 1, and see if it 
suits your needs.

Note that I was working on adding multitenant capabilities to piler, however at 
some point I decided to stop. I won't remove what's already done, but I don't 
intend to improve it.

If you had many domains with many customers, then piler may lack a few 
features, like separating customers' data, supporting more nodes, etc.

Janos


 Original Message 
From: Marcelo Machado <mmgom...@gmail.com>
Sent: Wed Jul 05 12:56:49 GMT+02:00 2017
To: Piler User <piler-user@list.acts.hu>
Subject: Re: View all emails from a domain



Hi Janos.

I would like an audit user to view the messages only from one domain
and another audit user only from another domain. It is possible?

Marcelo Gomes

2017-07-05 1:22 GMT-03:00 Janos SUTO <s...@acts.hu>:
> Hello,
>
> a regular user can't see all emails, however an auditor user can.
>
> Janos
> 
> From: Marcelo Machado
> Sent: Tue Jul 04 21:54:58 GMT+02:00 2017
> To: piler-user@list.acts.hu
> Subject: View all emails from a domain
>
>
>
> Hello everyone.
>
> I have a piler server receiving messages from two different domains.
>
> Is possible one regular user or auditor user see messages from all
> users only of a one
> domain?
>
> Marcelo Gomes
>



Re: View all emails from a domain

2017-07-04 Thread Janos SUTO
Hello,

a regular user can't see all emails, however an auditor user can.

Janos


 Original Message 
From: Marcelo Machado 
Sent: Tue Jul 04 21:54:58 GMT+02:00 2017
To: piler-user@list.acts.hu
Subject: View all emails from a domain



Hello everyone.

I have a piler server receiving messages from two different domains.

Is possible one regular user or auditor user see messages from all
users only of a one
domain?

Marcelo Gomes



Re: Metadata Cleanup?

2017-07-03 Thread Janos SUTO
Hello Uwe,

besides deleting from metadata table, you may delete the corresponding entries 
from rcpt table as well.

However, my personal point of view is that the metadata table serves as a 
history of emails worth to keep. Anyway it's your archive.

And one more thing. Sphinx uses the deleted=1 rows as the killlist, see 
sphinx.conf.

Janos



 Original Message 
From: "Uwe Kiewel [ML]" 
Sent: Mon Jul 03 10:52:23 GMT+02:00 2017
To: "piler-user@list.acts.hu" 
Subject: Metadata Cleanup?



Hi

hou to cleanup old metadata of deleted mails?

My current status:

mysql> select count(deleted) from metadata where deleted = 0;
++
| count(deleted) |
++
|   2918 |
++
1 row in set (0.00 sec)

mysql> select count(deleted) from metadata where deleted = 1;
++
| count(deleted) |
++
|   8517 |
++
1 row in set (0.01 sec)


Will it be save just to execute "delete from metadata where deleted = 1" 
periodically?  


Thanks
Uwe




1.3.0 is soon to be released

2017-05-16 Thread Janos SUTO

Dear piler users,

I'm about to release 1.3.0. It has a few new features,
it's mostly a maintenance release with bugfixes, minor
improvements.

tl;dr #1: 1.3.0 will be released on this week

tl;dr #2: if you use piler on FreeBSD or other non-Linux platform,
please drop me a line. There will be changes affecting you, see
the end of the email.



A short list of changes:


* removed the pdf download icon from gui

tcpdf seems to have some issues if the email has images. Until
I find a better alternative this feature is hidden from the
users.


* added HTMLpurifier to sanitize the html mime parts

* rewritten pilerpurge in python

The purge utility uses lots of SQL as well, and it's much
easier to do the job using a script language. Also, it's
half the size in lines of code.


* improved search feature

the entered text is searched in all fields (subject, body, from, ...)
unless you specify an exact keyword (eg. subject:) to narrow the search.

Thanks for Epicenter to sponsor the feature!



* improving the smtp daemon

You may or may not noticed that I've created a new branch called 
'epoll'.


The current piler daemon uses the preforking model: it spawns 10 or so
child processes that actually wait for an incoming smtp connection, and
handle the received email.

The 'epoll' branch features a separate smtp daemon for piler called
piler-smtp. It's a single process daemon using the epoll() mechanism
to process multiple connections effectively. It stores the received
emails in its queue directory which is read by piler processes. The
number of piler processes should be your number of of cpu cores + 1,
ie. 2 for a single cpu system.

As you can see the current approach uses 10+1=11 processes, however,
with epoll model uses 1+2+1=4 processes, while it can serve up to
64 parallel connections (can be set in piler.conf).

One example when the latter might be handy is receiving emails from 
o365.

Usually 10 child processes are enough, however it seems that o365 has
no flow control, ie. in case of a spike in the email volume it tries
to overwhelm piler, and if all 10 children are busy, then return an
error to the sender, which is bad news.

In such case you have two options: either to increase the number of
the piler processes (that particular user had to increase the instance
size of the virtual machine in amazon aws to run 32 piler processes).

The other option is the epoll model which is able to handle such a spike
effectively with low memory footprint.

The bottomline is that 1.3.0 will be the last release to support the
preforking model. After that I'll switch to the epoll model. One 
drawback

however, it's Linux only. I'm aware of that some of you use piler on
FreeBSD. I'll contact you to figure out something for you as well.


Best regards,
Janos



Re: Starting in Piller

2017-03-04 Thread Janos SUTO

Look at the mail log. The piler daemon syslogs every
complaint or issue it encounters. You didn't expect
this, right? :-)

Anyway my best bet is that you have a mta running
(exim, postfix, whatever) thus blocking piler's
ability to bind to port 25.

Janos


On 2017-03-03 14:52, Marcelo Soares wrote:

Dears,

Is there a way to start piler in verbose mode?

I've installed on dependencies

Apache and php7 is running:

But piler service don't come up:

Att

Marcelo Soares
2017-02-23 18:11 GMT-03:00 admin-at-extremeshok-dot-com
<ad...@extremeshok.com>:


Usually mariadb is configured to use TCP 127.0.0.1:3206 [1]

__.https://eXtremeSHOK.com
.__

On 02/23/2017 11:09 PM, Janos SUTO wrote:


Hello Marcelo,

On 2017-02-23 21:05, Marcelo Soares wrote:

Dear,
Do I have to change  mysqlsocket=/var/run/mysqld/mysqld.sock to
mariadb? (Ubuntu 16.04 MariaDB10)


unlikely. Mariadb uses just the same socket file.
Anyway start mariadb, then check if that socket file
exists.

Janos





Links:
--
[1] http://127.0.0.1:3206




Re: Starting in Piller

2017-02-23 Thread Janos SUTO

Hello Marcelo,

On 2017-02-23 21:05, Marcelo Soares wrote:

Dear,
Do I have to change  mysqlsocket=/var/run/mysqld/mysqld.sock to
mariadb? (Ubuntu 16.04 MariaDB10)


unlikely. Mariadb uses just the same socket file.
Anyway start mariadb, then check if that socket file
exists.

Janos



Re: Starting in Piller

2017-02-21 Thread Janos SUTO

Hello Marcelo,

I recommend ubuntu xenial x64 with mariadb 10.1 (and not
the stock shipped mysql 5.7 which has some utf8 and other
problems with piler).

Janos

On 2017-02-21 18:58, Marcelo Soares wrote:

Hello Everybody,

My name is Marcelo Soares, from Brazil.

I am starting on Piler, i have a Zimbra Server with 600 mailboxes and
4tb.

Which is the best linux distro to use Piller? Starting from zero.

Att

Marcelo Soares




Re: Verification Failure

2017-02-20 Thread Janos SUTO
Hello Stephan,

you exported the message, and checked the current sha256 values.

You also queried the stored sha256 sums from the metadata table.

Comparing these values we can assume that either the metadata entries have 
changed or the stored message.

Janos


 Original Message 
From: "Stephan Schöffel" <s.schoef...@aktivoptik.de>
Sent: Mon Feb 20 15:52:01 GMT+01:00 2017
To: Piler User <piler-user@list.acts.hu>
Subject: Re: Verification Failure


Hello Janos,

still don't know exactly what to compare.

Form pilertest I get:

root@mailarchiv:/tmp/mailarchive# pilertest message.txt | grep 'body digest'
body digest: 
ae927d2d6228f75a9bf8be6d96b6264287f615bf0273a170aaa50c2f30b721da

 From sha256sum I get:

root@mailarchiv:/tmp/mailarchive# sha256sum message.txt
acaf159d68e628bbd5397dcc163d700a76e306168680b8beb1bc09761cb60c44 message.txt

 From the metadata table I get:

MariaDB [piler]> select digest, bodydigest  from metadata where piler_id 
= '4000586a3e193199bb2400c9daf66a7e' ;
+--+--+
| digest   | 
bodydigest   |
+--+--+
| 3758f9e4c13b590810786c0ced2f4bbb6901fee894120da436b09c25e3e21522 | 
bbfef40739b9e1447479ef066f1d0d98bda82eba8fcdff9328df266386072ab6 |
+--+--+
1 row in set (0.00 sec)


Cheers
Stephan

Am 09.02.2017 um 20:01 schrieb Janos SUTO:
> the sha256 values you can see in pilertest output (also run sha256 
> )
> with the stored results.
>
> Janos
>
> On 2017-02-02 11:08, Stephan Schöffel wrote:
>> Which values should I compare?
>>
>> Am 01.02.2017 um 10:27 schrieb Janos SUTO:
>>
>>> Hello Stephan,
>>>
>>> no, it shouldn't. I need some troubleshooting to figure it out. 3℅
>>> verification problem is way too much.
>>>
>>> So please do the following:
>>>
>>> Get a problematic message with pilerget, save to a file, run
>>> pilertest against it, and get the values from the metadata table,
>>> and compare the results.
>>>
>>> Janos
>>>
>>> Janos
>>>
>>> -
>>> FROM: "Stephan Schöffel"
>>> SENT: Wed Feb 01 09:16:22 GMT+01:00 2017
>>> TO: Piler User
>>>
>>> SUBJECT: Re: Verification Failure
>>>
>>> Janos, actually nothing of the both has occured. Can the parser bug
>>> occur without an update? Cheers Stephan Am 31.01.2017 um 19:26
>>> schrieb Janos SUTO: Hello, On 2017-01-30 12:10, Stephan Schöffel
>>> wrote: upon trying to export emails from the archive I get about 3%
>>> "verification FAILED" messages. Unfortunately, I cannot find any log
>>> file to check what's going on. Maybe you can give me a hint. the
>>> 'verification failed' message means that the computed checksums (two
>>> sha256 hash values) are not the same that were stored at archiving
>>> time. This might be the result of a parser bug (assuming there was
>>> an upgrade in the meantime), or the message itself is actually
>>> changed. I can't tell which. Janos




Re: Verification Failure

2017-02-09 Thread Janos SUTO
the sha256 values you can see in pilertest output (also run sha256 
)

with the stored results.

Janos

On 2017-02-02 11:08, Stephan Schöffel wrote:

Which values should I compare?

Am 01.02.2017 um 10:27 schrieb Janos SUTO:


Hello Stephan,

no, it shouldn't. I need some troubleshooting to figure it out. 3℅
verification problem is way too much.

So please do the following:

Get a problematic message with pilerget, save to a file, run
pilertest against it, and get the values from the metadata table,
and compare the results.

Janos

Janos

-
FROM: "Stephan Schöffel"
SENT: Wed Feb 01 09:16:22 GMT+01:00 2017
TO: Piler User

SUBJECT: Re: Verification Failure

Janos, actually nothing of the both has occured. Can the parser bug
occur without an update? Cheers Stephan Am 31.01.2017 um 19:26
schrieb Janos SUTO: Hello, On 2017-01-30 12:10, Stephan Schöffel
wrote: upon trying to export emails from the archive I get about 3%
"verification FAILED" messages. Unfortunately, I cannot find any log
file to check what's going on. Maybe you can give me a hint. the
'verification failed' message means that the computed checksums (two
sha256 hash values) are not the same that were stored at archiving
time. This might be the result of a parser bug (assuming there was
an upgrade in the meantime), or the message itself is actually
changed. I can't tell which. Janos


--

 [1]

 STEPHAN SCHÖFFEL

BITTE BEACHTEN SIE UNSERE NEUE ADRESSE:

AKTIVOPTIK SERVICE AG
John-F.-Kennedy-Straße 26
D-55543 Bad Kreuznach

Tel.: +49 671.796467 150
Mobil: +49 151.28448 602
Fax: +49 671.796467 101
E-Mail: s.schoef...@aktivoptik.de
Web: www.aktivoptik.de [1]

Sitz der Gesellschaft: Bad Kreuznach
HRB 20838 • Amtsgericht Bad Kreuznach
Vorstand: Rolf Schneider (Vorsitzender), Ruth Weißmann, Jan Schneider
Aufsichtsratsvorsitzender: Hans-Walter Liese
USt-IdNr.: DE 269 834 610

Bankverbindung:
Sparkasse Rhein-Nahe
IBAN: DE13 5605 0180 0010 0621 80
BIC: MALADE51KRE

 [2]

 www.aktivoptik.de [1]

 [3]

Links:
--
[1] http://www.aktivoptik.de
[2]
http://media.opti.de/onlinekatalog/2017/Ausstellerdetails/aktivoptik_Service_AG/?elb=208.1100.2593.1.
[3] https://www.facebook.com/aktivoptik.de




Re: Exclude mails with a specific header value

2017-02-08 Thread Janos SUTO
Hello,

this feature is not supported. Currently your best bet is to put an MTA, eg. 
postfix in front of piler, and let it do the filtering. Or you may do the 
equivalent of this on the sending side.

Janos


 Original Message 
From: Yann Lehmann 
Sent: Wed Feb 08 14:42:21 GMT+01:00 2017
To: piler-user@list.acts.hu
Subject: Exclude mails with a specific header value


Hi

Our groupware-system sends notification mails for tasks and appointments.

All this notification mails have a specific header like 'X--Type: 
notification-mail'. As the use of the groupware-system is growing, we 
are seeing more and more of this messages.

Is there a way to avoid them to be archived by Piler ?

Thanks and best regards
Yann Lehmann



Re: Verification Failure

2017-02-01 Thread Janos SUTO
Hello Stephan,

no, it shouldn't. I need some troubleshooting to figure it out. 3℅ verification 
problem is way too much.

So please do the following:

Get a problematic message with pilerget, save to a file, run pilertest against 
it, and get the values from the metadata table, and compare the results.

Janos



Janos⁣​


 Original Message 
From: "Stephan Schöffel" <s.schoef...@aktivoptik.de>
Sent: Wed Feb 01 09:16:22 GMT+01:00 2017
To: Piler User <piler-user@list.acts.hu>
Subject: Re: Verification Failure


Janos,

actually nothing of the both has occured. Can the parser bug occur 
without an update?

Cheers
Stephan


Am 31.01.2017 um 19:26 schrieb Janos SUTO:
> Hello,
>
> On 2017-01-30 12:10, Stephan Schöffel wrote:
>>
>> upon trying to export emails from the archive I get about 3%
>> "verification FAILED" messages. Unfortunately, I cannot find any log
>> file to check what's going on. Maybe you can give me a hint.
>
> the 'verification failed' message means that the computed checksums
> (two sha256 hash values) are not the same that were stored at
> archiving time.
>
> This might be the result of a parser bug (assuming there was an upgrade
> in the meantime), or the message itself is actually changed. I can't
> tell which.
>
> Janos
>




Re: Verification Failure

2017-01-31 Thread Janos SUTO

Hello,

On 2017-01-30 12:10, Stephan Schöffel wrote:


upon trying to export emails from the archive I get about 3%
"verification FAILED" messages. Unfortunately, I cannot find any log
file to check what's going on. Maybe you can give me a hint.


the 'verification failed' message means that the computed checksums
(two sha256 hash values) are not the same that were stored at
archiving time.

This might be the result of a parser bug (assuming there was an upgrade
in the meantime), or the message itself is actually changed. I can't
tell which.

Janos



Re: Sphinx index files rotation (and corrupted?)

2016-10-26 Thread Janos SUTO
Hello,

well, I'd definitely check the sphinx docs what they say about large 
installations.

Symlinks are just fine, and I even suggest a dedicated sphinx host, giving it 
enough CPU, memory and fast disk.

Janos

⁣​


 Original Message 
From: apfc...@gmail.com
Sent: Tue Oct 25 23:54:52 GMT+02:00 2016
To: Piler User <piler-user@list.acts.hu>
Subject: Re: Sphinx index files rotation (and corrupted?)

Hi Janos,

Ah, didn't know default only went up to 4 only. I did briefly go through
sphinx.conf and saw there was no main5, but didn't want to try manually
creating it.

Anyways, I stopped searchd, ran "indexer main5", started searchd and GUI is
returning results again. (indexer.main.sh and config-site.php were already
modified for main5)

BTW, what do you recommend for these index files as the archive grows into
terabytes? It'll be okay to have mainxx.* or even mainxxx.*? If we need to
move these files, using symbolic links won't be a problem?

Thank you very much for the help!

On Tue, Oct 25, 2016 at 2:59 PM, Janos SUTO <s...@acts.hu> wrote:

> Hello,
>
> On 2016-10-25 21:03, apfc...@gmail.com wrote:
>
>>
>> I've been rotating the main sphinx index (per
>> http://www.mailpiler.org/wiki/current:administering-piler) and am now
>> up to main5, but I don't see the main5.* created at
>> /var/piler/sphinx/. It's been almost 24 hours since I made the change
>> and restarted searchd and piler.
>>
>
> the default install creates main1,2,3 and 4 'only'. I assume you have
> created main5 index in sphinx.conf. Then you ran indexer main5 which
> created main5.* files in /var/piler/sphinx. You also fixed indexer.main.sh
> to use main5 instead of main4. And finally you edited config-site.php
> and added main5 to the index list in SPHINX_MAIN_INDEX.
>
> Please confirm you did exactly the steps above.
>
> I tried searching via the gui just now and no results are being
>> returned, not even from weeks ago when I exported/downloaded emails
>> successfully.
>>
>> Should I re-create the sphinx data?
>>
>
> not yet, see the procedure above.
>
> Janos
>
>


Re: Mysql 5.7

2016-10-24 Thread Janos SUTO

Hello Michael,

On 2016-10-23 22:48, Prof. Dr. Michael Schefczyk wrote:


Thank you very much for taking piler even further. While I only run a
small personal installation which would qualify for the free offer of
Mailarchiva, I still very much prefer truly open source software!


cool :-)


Currently, I run piler 1.1 on Debian Jessie with an internal standard
mysql database on the same virtual machine. In general, I am also
running a Percona XtraDB Cluster for my database needs. The cluster is
now upgraded to mysql 5.7. In the faq for piler, you do state an
incompatibility with mysql 5.7 (plus a possible fix, if I interpret it
correctly). The release notes on piler 1.2 refer to substantial sql
changes. Therefore I would like to ask if piler 1.2 is still
incompatible with mysql 5.7. If so, is the fix quoted to on the faq
page necessary and sufficient to solve this?


yes, the incompatibility is present with piler 1.2 and mysql 5.7, and 
the

fix is meant as a workaround for the utf8 issue.

Janos



Re: End Users are not seeing their email

2016-10-11 Thread Janos SUTO


You didn't provide any description what you are using,
your setup, piler version, anything.

Janos

On Tue, 11 Oct 2016, Karl Rossing wrote:



We have our email servers forwarding to mailpiler.

Using the auditor account, we were able to determine that the mail is 
arriving in mail piler however the sender is always postmaster and the 
recipient is always "undisclosed recipient"


How do we get the mail to be assigned to each user?

Thanks
Karl








CONFIDENTIALITY NOTICE:  This communication (including all attachments) is
confidential and is intended for the use of the named addressee(s) only and
may contain information that is private, confidential, privileged, and
exempt from disclosure under law.  All rights to privilege are expressly
claimed and reserved and are not waived.  Any use, dissemination,
distribution, copying or disclosure of this message and any attachments, in
whole or in part, by anyone other than the intended recipient(s) is strictly
prohibited.  If you have received this communication in error, please notify
the sender immediately, delete this communication from all data storage
devices and destroy all hard copies.






Re: AW: Access to mails with delimiter "+"

2016-08-21 Thread Janos SUTO

Hello,

On 2016-08-21 14:52, André Peters wrote:


A delimited address like me+t...@domain.tld is now shown as
"t...@domain.tld". I need to add "t...@domain.tld" to show delimited
mails in the search panel.
Anyways it shows the address correctly in the bottom panel.

Screenshots always help, I guess: https://i.imgur.com/Td8obp4.png

I gladly pay your hourly rate to implement this into the parser. I had
a look into it but my C isn't good at all. :-)


try the following: edit src/parser_utils.c, and locate the translateLine
function. It has the following nice condition around the 593th line:

if( (state->message_state == MSG_RECEIVED || state->message_state == 
MSG_FROM || state->message_state == MSG_TO
 || state->message_state == MSG_CC || state->message_state == 
MSG_RECIPIENT) && *p == '@'){ continue; }


and replace it with the following:

if( (state->message_state == MSG_RECEIVED || state->message_state == 
MSG_FROM || state->message_state == MSG_TO
 || state->message_state == MSG_CC || state->message_state == 
MSG_RECIPIENT) && (*p == '@' || *p == '+') ){ continue; }


(note that I wrapped the lines to fit my screen. You may keep it in one 
line).


Then recompile, and run pilertest against it, and check if the To 
address

is displayed correctly. Let me know if it works for you.

Janos



André

-Ursprüngliche Nachricht-
Von: Janos SUTO [mailto:s...@acts.hu]
Gesendet: Sonntag, 21. August 2016 14:32
An: Piler User <piler-user@list.acts.hu>
Betreff: Re: Access to mails with delimiter "+"

Hello,

try adding the following to config-site.php:

$config['CUSTOM_EMAIL_QUERY_FUNCTION'] = '';

function ($username = '') {
$session = Registry::get('session');
$data = $session->get("auth_data");

if($username == 'myn...@example.org') {
   array_push($data['emails'], "myname+appen...@example.org");
}

$session->set("auth_data", $data);
}

then check on the settings page if the appended email is listed.

Note that I didn't check if the plus sign  is handled correctly by the
parser and the indexer.

Janos


On 2016-08-21 12:18, André Peters wrote:

Hi,

Any idea how I can give access to mails with a delimiter? Like in
myname+appen...@example.org.

I wrote a function to include these aliases in the $data['emails']
array when logging in, but I cannot see them anyway. Any ideas?

André




Re: Archive size

2016-08-12 Thread Janos SUTO
Hello Konstantin,

it's possible to main merge once a week, but if it's the disk space then I'm 
not sure how it would help you.

Hopefully pilerpurge removes the attachments as well provided that they are not 
referenced anymore. Unless there's a bug somewhere. Anyway your best bet is to 
test it.

Janos




 Original Message 
From: Konstantin 
Sent: Fri Aug 12 13:35:33 GMT+02:00 2016
To: Piler User 
Subject: Archive size

Hello Janos,

We have an archive with following version: piler 0.1.25-master-branch,
build 869
A server has 2T hdd and 1.5T used currently.
Usually 420G is available but when indexer.main.sh running free disk space
reduced to 90G.
It is too low so i started looking for a reason what's taking up the space
on my drive.

Here are the questions i have.
1) I read a documentation for sphinx and understand why disk used so hard
during merge from dailydelta1 to main1.
Currently we have indexer.main.sh scheduled at 2am each day.
I would like to run it on Saturday only. For a week it will be ~180k emails.
Can it cause any issues during merge ?

2) We have pilerpurge sheduled. Old emails removed from search and i
thought that they removed from disk as well.
I looked for an attachments example and found this file
/var/piler/store/00/526/00/00/4000526faa5a1602877400bf63cb.a3

In piler.metadata message with
piler_id=4000526faa5a1602877400bf63cb has deleted=1
In piler.attachment i also see results for that piler_id:
*** 1. row ***
 id: 79578
 piler_id: 4000526faa5a1602877400bf63cb
 attachment_id: 3
 name: medshoppe
 type: application/octet-stream
 sig: 37cfd3b8d394d791780d2c478f98239ad0fd5b656d4e7d52c64035dea76e86de
 size: 3944
 ptr: 0
 deleted: 0

A wiki said 
what pilerpurge remove an old message.
I suppose that piler also remove attachments.
Is that correct?
Can you please confirm that piler build 869 remove old messages from disk ?

Thank you.
--
*This message was delivered using 100% recycled electrons*.


Re: Web UI Display encoded Mails

2016-07-25 Thread Janos SUTO
Hello Joe,

there shouldn't be such a flag. Can you send me an example email that looks 
wrong? And perhaps a screenshot how it looks for you.

Janos




 Original Message 
From: Joe Rady 
Sent: Mon Jul 25 11:33:15 GMT+02:00 2016
To: piler-user@list.acts.hu
Subject: Web UI Display encoded Mails

Hi Janos,

After importing a fair amount of mails, it appears that Mails being encoded by 
Lotus Notes (X-Mailer: Lotus Notes Release 8.5.3FP3 November 16, 2012) are not 
decoded (displayed) properly in the WebUI.
when i download the eml file, my mail.app decodes the Mail correctly.
are there specific flags that need to be activated for Lotus Notes?

thanks a million,
Joe




Re: Delete some special mails[Avira checked]

2016-07-14 Thread Janos SUTO

Hello,

set its retention value to the past (eg. yesterday), then run the 
pilerpurge.


Janos


On 2016-07-14 15:37, Jan Novak wrote:

Hi,

how can i delete some manually selected mails oft he archive?

Jan




Re: LDAP works via email address only and breaks SSO + allows disabled users access

2016-07-14 Thread Janos SUTO

Hello,

On 2016-07-14 10:25, S T wrote:

Hello,

RE: LDAP + Username

Currently it seems the only way to login using an LDAP connected
server is to use the/an email address for a user. Whilst this works it
breaks the unified username/password combo that users in our
organisation are used to using for multiple services.


why would it? You can use the accounts in the ldap server for 
authentication.



Because of this we find it stops SSO from working, because even if we
use the "strip domain" option, the user.name@ does not always match
the username of the user.


I don't think so. You may use ldap auth as well as sso.


Looking at the source code, there is only one entry in the config
(LDAP_MAIL_ATTR) for an ldap field for 'mail' in this instance, and
there is nothing similar for a username field, and from this I presume
that the username is not used to check the user?


you are right. The gui starts with the email address to determine other
user settings in case of ldap auth. With sso you are already 
authenticated,
and (assuming you have AD) it starts with the samaccountname to figure 
out

the email addresses, etc.



I think I could modify this code to split the user auth out to allow
to username and/or email address to be used for logon which would fix
both. But before I do that I wanted to check that I am not missing
something as I may have to backport these changes for newer versions
(unless this is something of interest in the main branch).


you may do that, but I think you might miss something.


RE: LDAP + Disabled accounts

Currently 1.1.1 allows an account that has been disabled to still
login, it has to be checked manually if an account is disabled. I've
seen lots of projects miss this out it. (Even ESXi didn't fix it until
5.5).


you haven't described your environment, but in general if an account is
disabled, then the underlying ldap server (or whatever you have) must 
reject
the authentication attempt from the gui, thus the gui you display a 
proper

auth failed error.

Janos



Re: existing users Not found

2016-07-13 Thread Janos SUTO

Hello,

On Wed, 13 Jul 2016, Carrie Knox wrote:


Here is my attempt to manually execute the mysql query

mysql> SELECT user.uid, isadmin, username, realname, domain, email
    -> FROM user,email
    -> WHERE user.uid='email.uid  group by user.uid'
    -> ORDER BY username
    -> ASC  LIMIT 0,20
    -> ;
+-+-+--+--+++
| uid | isadmin | username | realname | domain | email  
|
+-+-+--+--+++
|   0 |   1 | admin    | built-in piler admin | local  | admin@local    
|
|   0 |   1 | admin    | built-in piler admin | local  | auditor@local  
|
|   0 |   1 | admin    | built-in piler admin | local  | bb...@sgi.com  
|
|   0 |   1 | admin    | built-in piler admin | local  | jeff...@sgi.com    
|
|   0 |   1 | admin    | built-in piler admin | local  | k...@sgi.com   
|
|   0 |   1 | admin    | built-in piler admin | local  | lista...@sgi.com   
|
|   0 |   1 | admin    | built-in piler admin | local  | listarchive2@local 
|
|   0 |   1 | admin    | built-in piler admin | local  | listarchive@local  
|
|   0 |   1 | admin    | built-in piler admin | local  | localus...@sgi.com 
|
+-+-+--+--+++
9 rows in set, 1 warning (0.00 sec)


this doesn't look right. Drop both user and email tables, and recreate 
them, and add the default admin and auditor users (see util/db-mysql.sql 
script).


It should look like the this:

| uid | isadmin | username  | realname  
| domain| email |
+-+-+---+---+---+---+
|   0 |   1 | admin | built-in piler admin  
| local | admin@local   |
|   1 |   2 | auditor@local | builtin auditor   
| local | auditor@local |



Janos


Re: existing users Not found

2016-07-12 Thread Janos SUTO

Hello Carrie,

On 2016-07-12 23:10, Carrie Knox wrote:



The issue that I am seeing is that when I login as admin@local and
select administration -> users, no  existing users are listed.   I
would expect to see the admin@local, auditor@local and
listarchive@local accounts, but instead I am seeing "Not found" for
existing users.

If I select "Add new user" to re-create the listarchive@local account
it reports that the Username is an Existing User.

I'm not sure where to begin in resolving this issue.


hmm, it seems odd. I'd check the user table in the piler database, and
see if it looks correct. Also as admin@local try to create a new user, 
eg.

us...@sgi.com (I assume you have set this domain in the domain settings
page).

Also try to debug the sql queries to see what's going on: edit 
system/database/mysql.php

and add the following to the beginning of the query() function:

syslog(LOG_INFO, "sql: $sql " . implode(";", $arr));

then see the mail log for the sql queries, and try to run the same 
statements

using the mysql client and see the results. Let me know how it goes.

Janos



Re: no mails can be found after imap import

2016-06-29 Thread Janos SUTO

Hello,

this is the problem:

SELECT id FROM main1,dailydelta1,delta1 WHERE       folder IN () AND
^^^

To fix it you have to login as administrator and assign some folders to 
the given user. He should relogin, and verify on the settings page that he 
has his folders assigned.


Janos


On Wed, 29 Jun 2016, Joe Rady wrote:


Hi,

a Newbie here:

I downloaded the ovf, startet it up, made some settings, most importantly the 
support for folders.
(my users are VERY fond of their imap folders)
went into the web-gui and set up some users plus domains etc.
everything looked nice.

then i executed

pilerimport -i imap.fleuchaus.com -u xx -p XxXxXx -R -P 993

inside the folder /home/piler/

it was counting the mails, and threw some error messages that some mails could 
not be imported.

i did look at the web Gui (health monitor) and found the correct number of 
received mails and also a few hundred of duplicate mails (which does make 
sense)...
also i see space projection and all that

however, when searching for mails i get none.
my search in the mails list brought some ideas, but no solution.

it appears that there are mails occupying space but they cannot be found.

mail.log has some infos that i cannot make a good clue of:

a whole bunch of lines like:

          "Jun 27 15:10:23 piler pilerimport[3815]: error: helper: execl"

which with my very limited knowledge of C tells me that something could not be 
executed?

and a large number of

"Jun 27 15:13:51 piler pilerimport[3207]: 4000577126992ee13a74008e6e435bad: 
cannot open: 4000577126992ee13a74008e6e435bad.m
Jun 27 15:13:51 piler pilerimport[3207]: 4000577126992ee13a74008e6e435bad: error 
storing message: 4000577126992ee13a74008e6e435bad.m"


later on in the day i tried importing from a different imap account
and got a lot of

"Jun 27 17:17:15 piler reindex[6042]: 4000577124f62c8a32e400d171e161bc: 
mysql_stmt_execute error: *Duplicate entry '780' for key 'PRIMARY'* (errno: 1062)"



the indexer in the crontab seems to do its job, but the log reports the same 
over and over:

"Jun 29 11:05:15 piler piler-webui[1921]: sphinx query: 'SELECT id FROM 
main1,dailydelta1,delta1 WHERE       folder IN () AND  MATCH('') ORDER BY `sent` 
DESC LIMIT 0,20 OPTION max_matches=1000' in 0.00 s, 0 hits, 0 total found

So although the health monitor indicates existing mails, the web gui cannot 
find them.

my apologies if i´ve overlooked some clues on the mailing list or on the 
website, i´ve tried to find some?


Best regards

Joe Rady
Head of IT
Fleuchaus & Gallo Partnerschaft mbB
__




Re: Problem: SSO Debian 8 Piler 1.1.1 not working (smbd 2:4.2.10+dfsg-0+deb8u3)

2016-06-29 Thread Janos SUTO
Enable debug log level in apache, then how me all the apache entries related to 
the SSO login.

Janos




 Original Message 
From: Herman Harperink 
Sent: Wed Jun 29 08:26:34 GMT+02:00 2016
To: "piler-user@list.acts.hu" 
Subject: Problem: SSO Debian 8 Piler 1.1.1 not working (smbd 
2:4.2.10+dfsg-0+deb8u3)

Hi,

After all was working fine I implemented SSO, but got stuck.
Apache give a error 500 when I call sso.php, errorlog gives me:

[auth_ntlm_winbind:error] [pid 10416] (20)Not a directory: [client 
x.x.x.x:25962] early EOF from helper

I did all checks I could fine and detect no problems (wbinfo –t/u/g works fine)

I followed http://www.mailpiler.org/wiki/current:single-sign-on


All I can find related to this is a samba bug from may this year but I seems 
unrelated since I use a later version.

Any hints on this one?

Thanks in advance.




With kind regards / Met vriendelijke groeten,

Herman Harperink
ICT Administrator





[Electromach 
B.V.]




Mail:   harper...@electromach.nl
Web:www.electromach.nl
Phone:  +31-74-2472492
Fax:+31-74-2435925


[cid:image8fc88e.JPG@750f3b96.47b12bc4]


[cid:imageefe799.JPG@35ed1877.44bfbc0f]


[cid:image66d4ba.JPG@e51a336c.47bdeb9c]


[cid:image12674c.JPG@93033b57.4c81f435]



Address
Electromach B.V.
Jan Tinbergenstraat 193
7559 SP Hengelo










Re: Piler new manual installation does not show archived mail in the search console

2016-06-22 Thread Janos SUTO
See the download section on the official site. You should really read the site.

Janos 



On Jun 22, 2016, 14:25, at 14:25, Mahmoud Ramadan 
 wrote:
>Thanks Janos , yes the newer Sphinx version solved my issue , but i
>noticed
>that the URL of Piler source code on the web site is version 1.1.1
>which is
>not the latest version , i guess the last version is piler -v
>1.2.0-master
>build 935 which can be found in the pre-installed ISO , i did a search
>to
>get the source code but i did not find it so if you can update me with
>the
>link of the latest Piler source code version i will be thankful to you
>,
>thanks in advance.
>
>
>
>Best Regards,
>
>Mahmoud Ramadan Ali
>
>Network and VOIP Specialist.
>
>Mobil: (+2) 01276877112
>
>Blog | Website
>| LinkedIn
>
>
>On Mon, Jun 20, 2016 at 9:19 AM, James Macdonell 
>wrote:
>
>>
>>
>> On 06/20/2016 04:24 AM, Mahmoud Ramadan wrote:
>>
>> msql>  select * main1,daildelta1,delta1;
>> ERROR 1064 (42000): You have an error in your SQL syntax;
>>
>>
>> I think the spirit here is to sanity check to ensure the
>indexer.delta.sh
>> and indexer.main.sh crons are doing their thing... filling sphinx
>with
>> stuff.  If the counts are zero, the crons need help.
>>
>> MySQL [piler]> select count(*) from main1; select count(*) from
>> dailydelta1; select count(*) from delta1;
>> +--+
>> | count(*) |
>> +--+
>> | 32971970 |
>> +--+
>> 1 row in set (0.80 sec)
>>
>> +--+
>> | count(*) |
>> +--+
>> |24832 |
>> +--+
>> 1 row in set (0.00 sec)
>>
>> +--+
>> | count(*) |
>> +--+
>> | 3256 |
>> +--+
>> 1 row in set (0.00 sec)
>>
>> 
>>
>> --
>> James Macdonell
>> Information Security & Emerging Technologies
>> California State University, San Bernardino
>> 5500 University Parkway
>> San Bernardino, CA 92407-2393
>> Main Line: 909/537-7262http://its.csusb.edu/
>> PGP: 0xBE849E4B
>>
>>


Re: MTA / Mail resending[Avira checked]

2016-06-21 Thread Janos SUTO




On Tue, 21 Jun 2016, Jan Novak wrote:


Dear piler user,

i have installed the piler vm and it works great.
2 questions:

1.) how can a user resend a mail in archive back to him out oft he WebGui as 
normal user?


click on the restore link in the preview pane (make sure you have set the 
smarthost properly)



2.) the log tells me, that "no mta is installed" when as example cron wants to 
send a mail with the output of pilerimport (I like to get this info mail at this first 
start to see whats going on, maybe i will disable it later).


because no mta is installed in the VM. If you need one, then install one 
for you, just make sure that it won't interfere with piler.


Janos



Re: Piler new manual installation does not show archived mail in the search console

2016-06-21 Thread Janos SUTO
Sigh. Though Ozan provided a nice docs he overlooked that debian ships an 
outdated version of sphinx.

I suggest you to go to the official piler site read the installation docs and 
the FAQ at minimum, and stick to everything you can read there. Then everything 
will be fine and you will be fine.

Janos



On Jun 21, 2016, 09:55, at 09:55, Mahmoud Ramadan 
<mahmoud.ramadan@gmail.com> wrote:
>My Sphinx version is
>root@Piler-VM:~# search
>Sphinx 2.0.4-release (r3135)
>
> and Piler version is
>root@Piler-VM:~# piler -v
>piler 1.1.1, build 904, Janos SUTO <s...@acts.hu>
>
>Build Date: Thu Jun 16 08:54:16 EDT 2016
>ldd version: ldd (Debian EGLIBC 2.13-38+deb7u11) 2.13
>gcc version: gcc version 4.7.2 (Debian 4.7.2-5)
>Configure command: ./configure --localstatedir=/var
>--with-database=mysql
>--enable-starttls --enable-tcpwrappers
>
>I'm following this Wiki for the installation
>http://wiki.ozanh.com/doku.php?id=linux:debian:mailpiler
>Thanks
>
>
>Best Regards,
>
>Mahmoud Ramadan Ali
>
>Network and VOIP Specialist.
>
>Mobil: (+2) 01276877112
>
>Blog <http://hotciscolabs.blogspot.com/>| Website
><http://telecomandsecurity.com/>| LinkedIn
><http://ch.linkedin.com/pub/mahmoud-ali/99/923/421/en>
>
>On Tue, Jun 21, 2016 at 1:45 AM, Janos SUTO <s...@acts.hu> wrote:
>
>> You shouldn't get any error or warning when running delta indexer.
>What
>> sphinx version do you have?
>>
>> Janos
>> On Jun 21, 2016, at 09:07, Mahmoud Ramadan
><mahmoud.ramadan@gmail.com>
>> wrote:
>>>
>>> I used this command to give Piler all the permissions og the sphinx
>>> directory root@Piler-VM:~# chmod -R 777 /var/piler/sphinx
>>>
>>> and all the file in the directory have the permissions like that:
>>>
>>> root@Piler-VM:~# ls -l /var/piler/sphinx/
>>> total 168
>>> -rwxrwxrwx 1 piler piler  32 Jun 21 02:26 dailydelta1.spa
>>> -rwxrwxrwx 1 piler piler   1 Jun 21 02:26 dailydelta1.spd
>>> -rwxrwxrwx 1 piler piler 582 Jun 21 02:26 dailydelta1.sph
>>>
>>> Also i used the command indexer.delta.sh as below but sill no search
>>> results in the search console.
>>>
>>> root@Piler-VM:~# su piler
>>> $ /usr/local/libexec/piler/ indexer.delta.sh
>>> WARNING: IsEmpty != attribute size (32)
>>>
>>> Thanks in advance.
>>>
>>>
>>> Best Regards,
>>>
>>> Mahmoud Ramadan Ali
>>>
>>> Network and VOIP Specialist.
>>>
>>> Mobil: (+2) 01276877112
>>>
>>> Blog <http://hotciscolabs.blogspot.com/>| Website
>>> <http://telecomandsecurity.com/>| LinkedIn
>>> <http://ch.linkedin.com/pub/mahmoud-ali/99/923/421/en>
>>>
>>> On Mon, Jun 20, 2016 at 9:35 AM, Janos SUTO <s...@acts.hu> wrote:
>>>
>>>> Your cron jobs are either not running or you have a permission
>problem
>>>> on /var/piler/sphinx.
>>>>
>>>> So switch to user piler, run indexer.delta.sh, and verify that the
>>>> sphinx files are update and increased in size.
>>>>
>>>> Janos
>>>> On Jun 20, 2016, at 13:25, Mahmoud Ramadan <
>>>> mahmoud.ramadan@gmail.com> wrote:
>>>>>
>>>>> Any updates ? need help to solve this issue please , thanks in
>advance.
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Mahmoud Ramadan Ali
>>>>>
>>>>> Network and VOIP Specialist.
>>>>>
>>>>> Mobil: (+2) 01276877112
>>>>>
>>>>> Blog <http://hotciscolabs.blogspot.com/>| Website
>>>>> <http://telecomandsecurity.com/>| LinkedIn
>>>>> <http://ch.linkedin.com/pub/mahmoud-ali/99/923/421/en>
>>>>>
>>>>> On Sun, Jun 19, 2016 at 1:36 AM, Mahmoud Ramadan <
>>>>> mahmoud.ramadan@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>> I did the following troubleshooting steps;
>>>>>>
>>>>>> *Checking the crontab jobs:*
>>>>>>
>>>>>> root@Piler-VM:~# crontab -l -u piler
>>>>>> ### PILERSTART
>>>>>> 5,35 * * * * /usr/local/libexec/piler/ indexer.delta.sh
>>>>>> 30   2 * * * /usr/local/libexec/piler/ indexer.main.sh
>>>>>> */15 * * * * /usr/bin/indexer --quiet tag1 --rotate
>>>>>> */15 * * * * /usr/bin/indexer --quiet note1 --rotate

Re: Piler new manual installation does not show archived mail in the search console

2016-06-21 Thread Janos SUTO
You shouldn't get any error or warning when running delta indexer. What sphinx 
version do you have?

Janos



On Jun 21, 2016, 09:07, at 09:07, Mahmoud Ramadan 
<mahmoud.ramadan@gmail.com> wrote:
>I used this command to give Piler all the permissions og the sphinx
>directory root@Piler-VM:~# chmod -R 777 /var/piler/sphinx
>
>and all the file in the directory have the permissions like that:
>
>root@Piler-VM:~# ls -l /var/piler/sphinx/
>total 168
>-rwxrwxrwx 1 piler piler  32 Jun 21 02:26 dailydelta1.spa
>-rwxrwxrwx 1 piler piler   1 Jun 21 02:26 dailydelta1.spd
>-rwxrwxrwx 1 piler piler 582 Jun 21 02:26 dailydelta1.sph
>
>Also i used the command indexer.delta.sh as below but sill no search
>results in the search console.
>
>root@Piler-VM:~# su piler
>$ /usr/local/libexec/piler/indexer.delta.sh
>WARNING: IsEmpty != attribute size (32)
>
>Thanks in advance.
>
>
>Best Regards,
>
>Mahmoud Ramadan Ali
>
>Network and VOIP Specialist.
>
>Mobil: (+2) 01276877112
>
>Blog <http://hotciscolabs.blogspot.com/>| Website
><http://telecomandsecurity.com/>| LinkedIn
><http://ch.linkedin.com/pub/mahmoud-ali/99/923/421/en>
>
>On Mon, Jun 20, 2016 at 9:35 AM, Janos SUTO <s...@acts.hu> wrote:
>
>> Your cron jobs are either not running or you have a permission
>problem on
>> /var/piler/sphinx.
>>
>> So switch to user piler, run indexer.delta.sh, and verify that the
>sphinx
>> files are update and increased in size.
>>
>> Janos
>> On Jun 20, 2016, at 13:25, Mahmoud Ramadan
><mahmoud.ramadan@gmail.com>
>> wrote:
>>>
>>> Any updates ? need help to solve this issue please , thanks in
>advance.
>>>
>>> Best Regards,
>>>
>>> Mahmoud Ramadan Ali
>>>
>>> Network and VOIP Specialist.
>>>
>>> Mobil: (+2) 01276877112
>>>
>>> Blog <http://hotciscolabs.blogspot.com/>| Website
>>> <http://telecomandsecurity.com/>| LinkedIn
>>> <http://ch.linkedin.com/pub/mahmoud-ali/99/923/421/en>
>>>
>>> On Sun, Jun 19, 2016 at 1:36 AM, Mahmoud Ramadan <
>>> mahmoud.ramadan@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I did the following troubleshooting steps;
>>>>
>>>> *Checking the crontab jobs:*
>>>>
>>>> root@Piler-VM:~# crontab -l -u piler
>>>> ### PILERSTART
>>>> 5,35 * * * * /usr/local/libexec/piler/ indexer.delta.sh
>>>> 30   2 * * * /usr/local/libexec/piler/ indexer.main.sh
>>>> */15 * * * * /usr/bin/indexer --quiet tag1 --rotate
>>>> */15 * * * * /usr/bin/indexer --quiet note1 --rotate
>>>> 30   6 * * * /usr/bin/php
>/usr/local/libexec/piler/generate_stats.php
>>>> --webui /var/www/piler
>>>> */5 * * * * /usr/bin/find /var/www/piler/tmp -type f -name i.\*
>-exec rm
>>>> -f {} \;
>>>> ### PILEREND
>>>>
>>>> *Reading log files:*
>>>>
>>>> root@Piler-VM:~# tail -f /var/log/sphinxsearch/query.log
>>>> [Sat Jun 18 14:20:18.899 2016] 0.000 sec [ext2/0/ext 0 (0,1000)]
>>>> [main1,dailydelta1,delta1]
>>>> [Sat Jun 18 14:21:48.727 2016] 0.000 sec [ext2/0/ext 0 (0,1000)]
>>>> [main1,dailydelta1,delta1]
>>>>
>>>>
>>>>
>>>> *Checking permissions:*root@Piler-VM:~# ls -l /var/piler/sphinx
>>>> total 168
>>>> -rw-r--r-- 1 piler piler  32 Jun 19 03:05 dailydelta1.spa
>>>> -rw-r--r-- 1 piler piler   1 Jun 19 03:05 dailydelta1.spd
>>>> -rw-r--r-- 1 piler piler 582 Jun 19 03:05 dailydelta1.sph
>>>> -rw-r--r-- 1 piler piler   1 Jun 19 03:05 dailydelta1.spi
>>>> -rw-r--r-- 1 piler piler   0 Jun 19 03:05 dailydelta1.spk
>>>> -rw--- 1 piler piler   0 Jun 19 03:05 dailydelta1.spl
>>>>
>>>>
>>>>
>>>> *Reading log files:*root@Piler-VM:~# tail -f /var/log/mail.log
>>>> Jun 19 03:06:17 Piler-VM piler-webui[4484]: sphinx query: 'SELECT
>id
>>>> FROM main1,dailydelta1,delta1 WHEREMATCH('') ORDER BY
>`sent` DESC
>>>> LIMIT 0,1000 OPTION max_matches=1000' in 0.00 s, 0 hits, 0 total
>found
>>>>
>>>> Notice: i executed the "  select * main1,daildelta1,delta1; "
>command
>>>> from the web GUI because i tried to know how to execute it from CLI
>and
>>>> whenever i use the command from the Mysql CLI i get this error.
>>>>
>>>> root@Piler-VM:~# mysql -h localhost -u piler -p piler -P 

Re: Piler new manual installation does not show archived mail in the search console

2016-06-20 Thread Janos SUTO
Your cron jobs are either not running or you have a permission problem on 
/var/piler/sphinx.

So switch to user piler, run indexer.delta.sh, and verify that the sphinx files 
are update and increased in size.

Janos



On Jun 20, 2016, 13:25, at 13:25, Mahmoud Ramadan 
<mahmoud.ramadan@gmail.com> wrote:
>Any updates ? need help to solve this issue please , thanks in advance.
>
>Best Regards,
>
>Mahmoud Ramadan Ali
>
>Network and VOIP Specialist.
>
>Mobil: (+2) 01276877112
>
>Blog <http://hotciscolabs.blogspot.com/>| Website
><http://telecomandsecurity.com/>| LinkedIn
><http://ch.linkedin.com/pub/mahmoud-ali/99/923/421/en>
>
>On Sun, Jun 19, 2016 at 1:36 AM, Mahmoud Ramadan <
>mahmoud.ramadan@gmail.com> wrote:
>
>> Hi,
>> I did the following troubleshooting steps;
>>
>> *Checking the crontab jobs:*
>>
>> root@Piler-VM:~# crontab -l -u piler
>> ### PILERSTART
>> 5,35 * * * * /usr/local/libexec/piler/indexer.delta.sh
>> 30   2 * * * /usr/local/libexec/piler/indexer.main.sh
>> */15 * * * * /usr/bin/indexer --quiet tag1 --rotate
>> */15 * * * * /usr/bin/indexer --quiet note1 --rotate
>> 30   6 * * * /usr/bin/php /usr/local/libexec/piler/generate_stats.php
>> --webui /var/www/piler
>> */5 * * * * /usr/bin/find /var/www/piler/tmp -type f -name i.\* -exec
>rm
>> -f {} \;
>> ### PILEREND
>>
>> *Reading log files:*
>>
>> root@Piler-VM:~# tail -f /var/log/sphinxsearch/query.log
>> [Sat Jun 18 14:20:18.899 2016] 0.000 sec [ext2/0/ext 0 (0,1000)]
>> [main1,dailydelta1,delta1]
>> [Sat Jun 18 14:21:48.727 2016] 0.000 sec [ext2/0/ext 0 (0,1000)]
>> [main1,dailydelta1,delta1]
>>
>>
>>
>> *Checking permissions:*root@Piler-VM:~# ls -l /var/piler/sphinx
>> total 168
>> -rw-r--r-- 1 piler piler  32 Jun 19 03:05 dailydelta1.spa
>> -rw-r--r-- 1 piler piler   1 Jun 19 03:05 dailydelta1.spd
>> -rw-r--r-- 1 piler piler 582 Jun 19 03:05 dailydelta1.sph
>> -rw-r--r-- 1 piler piler   1 Jun 19 03:05 dailydelta1.spi
>> -rw-r--r-- 1 piler piler   0 Jun 19 03:05 dailydelta1.spk
>> -rw--- 1 piler piler   0 Jun 19 03:05 dailydelta1.spl
>>
>>
>>
>> *Reading log files:*root@Piler-VM:~# tail -f /var/log/mail.log
>> Jun 19 03:06:17 Piler-VM piler-webui[4484]: sphinx query: 'SELECT id
>FROM
>> main1,dailydelta1,delta1 WHEREMATCH('') ORDER BY `sent` DESC
>LIMIT
>> 0,1000 OPTION max_matches=1000' in 0.00 s, 0 hits, 0 total found
>>
>> Notice: i executed the "  select * main1,daildelta1,delta1; " command
>from
>> the web GUI because i tried to know how to execute it from CLI and
>whenever
>> i use the command from the Mysql CLI i get this error.
>>
>> root@Piler-VM:~# mysql -h localhost -u piler -p piler -P 9306
>> mysql> show databases;
>> ++
>> | Database   |
>> ++
>> | information_schema |
>> | piler  |
>> ++
>> 2 rows in set (0.00 sec)
>>
>> mysql> use piler;
>> Database changed
>> mysql>  select * main1,daildelta1,delta1;
>> ERROR 1064 (42000): You have an error in your SQL syntax; check the
>manual
>> that corresponds to your MySQL server version for the right syntax to
>use
>> near 'main1,daildelta1,delta1' at line 1
>>
>> Thanks in advance.
>>
>> Best Regards,
>>
>> Mahmoud Ramadan Ali
>>
>> Network and VOIP Specialist.
>>
>> Mobil: (+2) 01276877112
>>
>> Blog <http://hotciscolabs.blogspot.com/>| Website
>> <http://telecomandsecurity.com/>| LinkedIn
>> <http://ch.linkedin.com/pub/mahmoud-ali/99/923/421/en>
>>
>> On Sat, Jun 18, 2016 at 8:08 AM, Janos SUTO <s...@acts.hu> wrote:
>>
>>>
>>> as with any software if you encountered a problem, you should check
>the
>>> logs. Piler logs everything to the mail log, so be sure to read it.
>>>
>>> To address your problem, if you get no search results, then check
>the
>>> sphinx database. Don't try to select from v_messages, because it's
>in
>>> the mysql database (not in sphinx).
>>>
>>> Rather, try this query:
>>>
>>> select * main1,daildelta1,delta1;
>>>
>>> If it's empty, then check why the sphinx database is not updated.
>Missing
>>> cron entries? Bad permissions on /var/piler/sphinx?
>>>
>>> So I'd expect some troubleshooting on your behalf.
>>>
>>> Janos
>>>
>>> PS: don't call me/us your d

Re: AW: Error when using " Download the selected PDF " feature in the end user web GUI.

2016-06-14 Thread Janos SUTO

On 2016-06-14 19:59, André Peters wrote:

The directory is probably just not writeable by the user who runs
piler.


almost. The directory must be writable by the webserver, eg. www-data,
apache, etc. Btw. note that the installer takes care of it.

Janos




VON: Mahmoud Ramadan [mailto:mahmoud.ramadan@gmail.com]
GESENDET: Dienstag, 14. Juni 2016 19:56
AN: ad...@extremeshok.com
CC: piler-user@list.acts.hu
BETREFF: Re: Error when using " Download the selected PDF " feature in
the end user web GUI.

what do you mean by " check your permissions ? I'm a network and VoIP
specialist with Linux experience , I'm deploying open source solutions
like Piler for companies , Still waiting for someone to answer my
question , thanks in advance.

Best Regards,

Mahmoud Ramadan Ali

Network and VOIP Specialist.

Mobil: (+2) 01276877112

Blog [1]| Website [2]| LinkedIn [3]

On Tue, Jun 14, 2016 at 2:37 PM, admin-at-extremeshok-dot-com
 wrote:


Network and VOIP Specialist. ? really .. check your permissions

__.https://eXtremeSHOK.com
.__

On 14-Jun-16 2:02 PM, Mahmoud Ramadan wrote:


Network and VOIP Specialist.




Links:
--
[1] http://hotciscolabs.blogspot.com/
[2] http://telecomandsecurity.com/
[3] http://ch.linkedin.com/pub/mahmoud-ali/99/923/421/en




Re: Single mail stuck in queue, smtp times out

2016-06-03 Thread Janos SUTO
Hello Andre,

what piler version do you have? Also try increase the verbosity level to see at 
which stage the connection is stuck.

And finally it would be useful to see an ngrep output for only this specific 
email.

Anyway you may zip this email and send it to me.

Janos 



On Jun 3, 2016, 12:39, at 12:39, "André Peters"  wrote:
>
>Hi,
>
>I have a single 161kb message stuck in my postfix queue. Any time I try
>to flush it, the smtp connection to piler times out.
>It is just this one message. I have about 20 mails per minute and none
>failed in months. :-)
>
>Maybe the message contains some weird formatting which piler does not
>like?
>
>Janos, I may send you this message. 
>
>Syslog looks fine. Restarted all services, too. I have 10 workers
>online, so this is not a critical problem. The other mails are
>processed fine. 
>
>
>André
>
>Von meinem iPhone gesendet


Re: Diskspace of archive

2016-06-02 Thread Janos SUTO
Piler compresses and deduplicates emails. 1.2GB is the raw size, 165 MB is the 
size on the disk.

Janos



On Jun 2, 2016, 16:46, at 16:46, Yann Lehmann  wrote:
>
>Hi
>
>On the stats page of my test-installation of Piler, the 'Archive size' 
>is showed as '1.2 GB (165 MB)'.
>
>A 'su -sh /var/piler' returns 219M.
>
>Where does the difference come from ?
>
>Thanks and regards
>Yann Lehmann


Re: Some questions about piler (user rights, ignored messages, etc)

2016-05-24 Thread Janos SUTO

Hello,

On 2016-05-24 16:55, Yann Lehmann wrote:

Hi,

I am new to piler and after some trouble, I have a working system that
I will test in the next days.

I have some questions about the use of piler:

1) User rights
I would like one user being able to see all messages. Do I have to
grant this user rights on all existing adresses and add new users when
these are created ? Or is there a "magic" way to grant access to all
adresses ?


just grant auditor role to this user, and you are done.



2) Ignored messages
I have seen on the "health page", that a lot of messages are ignored.
For now, I have only one user. Could it be that messages not for this
user are ignored ? If yes, is there a setting to archive all messages,


no. Piler archives each email it receives. A message can be ignored if 
it
matches an archiving rule or if it's a duplicate (based on the 
message-id)

If you have lots of messages WITHOUT a valid message-id, then they would
be discarded as well.


even if users or addresses do not exist on the piler system ?

3) If a user wants a message sent back to his mailbox, I guess piler
will send the message to the 'smarthost' defined in the
'config-site.php' file. Our mailserver only accepts mails from


that's right


authenticated users. Is it possible to send credentials with the
message and how ?


not at the moment


4) Is there a list which explains the parameters of the 'piler.conf' 
file ?


see src/examples.conf

Janos



Re: piler on arm-based device

2016-04-19 Thread Janos SUTO

Hello Yann,

I've never tried to port piler to ARM. However, I'm curious whether
it can be compiled at all. Unfortunately I don't have access to such
a device. If you can't make it, and can afford to provide some access
to the device, then I may try to do the job. No promises, though.

Janos


On 2016-04-19 18:23, Yann Lehmann wrote:

Hi

I am setting up a small arm-based computer (Cubietruck) as a "test 
lab".


I would like to test it as a mail server and archiving solution
(though I have read it should not be run on the same machine than the
mail server).

Is it possible to compile piler for arm ?

Thanks and regards
Yann Lehmann

PS: sorry if the question has already been asked and answered




Re: Folders

2016-04-10 Thread Janos SUTO

Hello Raimer,

On 2016-04-09 14:12, Raimar Sandner wrote:

Hi all,

I'm reaching out one more time, maybe someone can help me with this 
folders-

problem described below.

Cheers
Raimar

On Montag, 1. Februar 2016 13:51:51 CEST Raimar Sandner wrote:

I'm having trouble making folders work in piler. I do realize this is 
not an
'official' feature, but it might be exactly the thing I need. This is 
my
use case: I want to archive the emails of two users, Alice and Bob, 
and
both should have access only to their own emails. I won't be using 
smtp,
just pilerimport from time to time. Ideally I would not like to bother 
with

the various email addresses or domains the users have.

To me this looks just like the feature folders could offer, using

pilerimport -F alice ...

or

pilerimport -F bob ...

However I cannot make this work the way I want, I'm confused how to 
manage
the folders. I deleted everything, databases and sphinx index to have 
a
fresh start as described in the FAQ, and I edited the configuration 
files
to enable folders. Then I imported a few mails with pilerimport -F 
alice,

but I cannot see them as auditor. I tried to create a regular user
alice@local and to give this user permission to the folder alice. As 
admin,
on the 'edit user' page I see 'Search folders' and 'Folders', but when 
I

enter 'alice' to either field this does not seem to have any visible
effect, when the user settings are loaded again both fields are empty. 
Also

I'm not sure if I have to create the folders somewhere before actually
using them. In the admin account there is no folder menu.

Sorry if I got this all wrong. Could you maybe explain how it is 
supposed to

work?


the folder idea may be used to override the usual permissions based on 
the sender
or recipient addresses. Thus it's no more any use if you are an auditor, 
you need

specific access rights for the given folder(s).

So login as admin, create a user, and assign him either alice or bob 
folder.
Then login as this user, and verify that you can actually see this 
folder on

the settings page.

Finally click on the search button, and show me the sphinx query (see 
the maillog).


Janos



Re: Pilerpurge and attachments

2016-04-05 Thread Janos SUTO

Hello Konstantin,

the 0.1.25 release was an interim build between two major
releases, and I don't plan to support fixes for that, sorry.

I suggest to upgrade to either 1.1.1, and I believe that
this issue would be gone.

Janos

On 2016-04-04 17:02, Konstantin wrote:

Hello Janos

I found pilerpurge remove attachment even if it has another attachment
depends on it (table `attachment`, id match value in ptr)

Is it possible:
- patch pilerpurge and recompile it to fix such behavior
- or do not remove attachments at all?

piler 0.1.25-master-branch, build 857

Thank you.




RE: 3 Message with wrong 2036 date

2016-03-02 Thread Janos SUTO

Hello Edwin,

can you show me the headers for these emails? In Tim's case it turned 
out

there were some bogus Date: headers, and we managed to fix it with a
manual sphinx update.

Janos

On 2016-02-18 17:17, Nichols, Edwin (BeneFACT) wrote:

Strangely, I have the same issue with 8 emails. The emails themselves
show a date of 2010, but the Piler interface shows them as being from
2036.  There may be a bug?

EDWIN NICHOLS  |  _Director – IT & Operations_



' 416-360-SRED (7733) ext.161

' Vancouver: 604-628-9870

' Calgary: 403-775-7565

' Toll Free: 1-855-TAX-BACK (829-2225)

, 647-689-3127

*  edwin.nich...@benefact.ca

8 WWW.BENEFACT.CA [1]

P PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL

-

Privacy Disclaimer - Français à suivre
This e-mail message (including attachments, if any) is intended for
the use of the individual or entity to which it is addressed and may
contain information that is privileged, proprietary, confidential. If
you are not the intended recipient, you are notified that any
dissemination, distribution, or copy of this communication is strictly
prohibited. If you have received this communication in error, please
notify the sender and erase this e-mail message immediately.

Déclaration de confidentialité
Le présent courrier électronique (y compris les pièces qui y sont
annexées, le cas échéant) s'adresse au destinataire indiqué et
peut contenir des renseignements de caractère privé ou confidentiel.
Si vous n'êtes pas le destinataire de ce document, nous vous
signalons qu'il est strictement interdit de le diffuser, de le
distribuer ou de le reproduire. Si ce message vous a été transmis
par erreur, veuillez en informer l'expéditeur et le supprimer
immédiatement.

FROM: Tim Stumbo [mailto:timstu...@gmail.com]
SENT: Thursday, February 18, 2016 9:45 AM
TO: piler-user@list.acts.hu
SUBJECT: 3 Message with wrong 2036 date

I have 3 messages that always appear at the top of the search field
due to them having an incorrect date. The year on the messages is
2036. I would like to either delete these 3 e-mails or fix the index
for them with the corrected date.

Could someone help me with this issue?

Thanks

Links:
--
[1] http://www.benefact.ca/




Re: Too Many Duplicate Messages With " message-id: null "

2016-03-02 Thread Janos SUTO

On 2016-02-29 12:06, Mahmoud Ramadan wrote:

Hi dears
I'm getting too many duplicated messages on Piler and by checking my
logs i found that the messages are received by Piler without message
ID (null) so Piler discards them.

Is there is any way to force Piler to accept duplicated messages and
skip message ID checking ?!  Thanks in advance.


set archive_emails_not_having_message_id=1 in piler.conf, then reload 
piler.


Janos



Re: Some Piler Messages are displayed with " Message Failed Verification ".

2016-03-02 Thread Janos SUTO

On 2016-02-29 13:37, Mahmoud Ramadan wrote:

Update: I noticed something weird ! When i changed the Piler contact
settings in Exchange to enable " Use MAPI rich text format" then all


please don't, as it forces exchange to transform the email, and this
practice may contradict with some regulations.


the messages received in rich text format are label as " Message is
Verified " BUT off course can NOT be displayed well as Piler does NOT
support archiving E-Mail in rich text format.


actually, piler can process rtf attachments by using unrtf.


And when i switch the setting again to disable " Use MAPI rich text
format" , the archived messages can be displayed fine BUT with "
Message Failed Verification ".


I'd like to get a complete non sensitive email. Select a message,
click on the download link, and let me see it. Also please confirm
that you have upgraded the gui as well.

Janos



SECURITY ALERT: the drown attack

2016-03-02 Thread Janos SUTO

Dear piler-users,

you probably have heard about the drown attack against
(see https://drownattack.com/ for more) tls/ssl.

In a nutshell it targets services (http, smtp, imap, pop3, etc)
allowing ssl v2 connections as well.

Piler 1.1.1 uses TLSv1_client_method() and TLSv1_server_method()
respectively. According to the openssl manual they force piler
to understand only the TLSv1 protocol, so piler 1.1.1 should be
safe.

However piler 1.1.0 (and older versions) uses SSLv3_client_method()
and SSLv23_server_method(). It means that pilerimport is safe from
the drown attack, since it doesn't understand SSL v2.
However the piler daemon (which understands SSL v2 as well) is
vulnerable to the drown attack.


Solution: upgrade to 1.1.1

Workaround: none. You may mitigate the issue if you have <= 1.1.0
if you allow connections only from trusted smtp clients.


Janos



Re: User that can read all mailboxes

2016-02-29 Thread Janos SUTO
I don't get your problem. What's the problem anyway?

Janos



On Feb 28, 2016, 22:29, at 22:29, "Matt ." <yamakasi@gmail.com> wrote:
>
>Do I miss something here ? I'm figuring out the right query method
>using the form.
>
>2016-02-20 22:50 GMT+01:00 Matt . <yamakasi@gmail.com>:
>> Hi sorry,
>>
>> I get than indeed 3 mails back from a domain where mail has been
>> generated for today.
>>
>> sphinx query: 'SELECT id FROM main1,dailydelta1,delta1 WHERE
>> MATCH('') ORDER BY `sent` DESC LIMIT 0,1000 OPTION max_matches=1000'
>> in 0.00 s, 3 hits, 3 total found
>>
>> But as there are more mails from the past, which I tried using the
>> search field with dates, etc before I'm unsure if that works well.
>>
>>
>>
>> 2016-02-20 22:00 GMT+01:00 Janos SUTO <s...@acts.hu>:
>>>
>>> I asked you to hit the search button, then show me the sphinx query,
>>> not "type *, then hit the search button, and show me the sphinx
>query".
>>>
>>> Note that there's indeed wildcard search, but not as you tried to
>use it.
>>>
>>> Janos
>>>
>>>
>>>
>>> On Sat, 20 Feb 2016, Matt . wrote:
>>>
>>>>
>>>> HI guys,
>>>>
>>>> I needed to get back on this, sorry.
>>>>
>>>> When I do a wildmark search I get this on my logs:
>>>>
>>>> sphinx query: 'SELECT id FROM main1,dailydelta1,delta1 WHERE
>>>> MATCH('@(subject,body)  *') ORDER BY `sent` DESC LIMIT 0,1000
>OPTION
>>>> max_matches=1000' in 0.06 s, 0 hits, 0 total found
>>>>
>>>> but noting back in piler.
>>>>
>>>> Thanks so far!
>>>>
>>>> 2016-02-17 15:25 GMT+01:00 Janos SUTO <s...@acts.hu>:
>>>>>
>>>>> hit the search button, then show me the matching sphinx query
>>>>> (look in the mail log).
>>>>>
>>>>> Janos
>>>>>
>>>>>
>>>>> On 2016-02-17 13:43, Matt . wrote:
>>>>>>
>>>>>>
>>>>>> Sure I did but whatever I search I get back no results.
>>>>>>
>>>>>> Crons are running.
>>>>>>
>>>>>> Do you have a wildcard example that should always work ?
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2016-02-17 13:41 GMT+01:00 admin-at-extremeshok-dot-com
>>>>>> <ad...@extremeshok.com>:
>>>>>>>
>>>>>>>
>>>>>>> login as the auditor@local user
>>>>>>>
>>>>>>> __.https://eXtremeSHOK.com
>>>>>>> .__
>>>>>>>
>>>>>>>
>>>>>>> On 17-Feb-16 2:18 PM, Matt . wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Not far as I can see, but maybe I'm searching the wrong way.
>Searches
>>>>>>>> don't give back anything at all atm.
>>>>>>>>
>>>>>>>> 2016-02-17 12:56 GMT+01:00 ad...@extremeshok.com
>>>>>>>> <ad...@extremeshok.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The default auditor account reads all email from all domains
>>>>>>>>>
>>>>>>>>> Sent from my iPhone
>>>>>>>>>
>>>>>>>>>> On 17 Feb 2016, at 10:47 AM, Matt . <yamakasi@gmail.com>
>wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> HI guys,
>>>>>>>>>>
>>>>>>>>>> Is there (already) a feature so my Auditor can read all
>mailboxes ?
>>>>>>>>>> This because Piler is not used on userbase here but on
>Auditor base.
>>>>>>>>>>
>>>>>>>>>> This for all domains/mailboxes.
>>>>>>>>>>
>>>>>>>>>> I hope I can accomplish this in some way
>>>>>>>>>>
>>>>>>>>>> Matt
>>>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>


Re: User that can read all mailboxes

2016-02-29 Thread Janos SUTO




On Sun, 28 Feb 2016, Matt . wrote:



Do I miss something here ? I'm figuring out the right query method
using the form.


not sure. You said you had running the cron jobs, I assume files under 
/var/piler/sphinx has recent timestamps, and their sizes grows.


Also check if youi have lots of status=stored lines in the mail log (ie. 
the piler daemon archives emails). Then check the metadata table in the

piler database and verify that it has lots of entries.

Janos



2016-02-20 22:50 GMT+01:00 Matt . <yamakasi@gmail.com>:

Hi sorry,

I get than indeed 3 mails back from a domain where mail has been
generated for today.

sphinx query: 'SELECT id FROM main1,dailydelta1,delta1 WHERE
MATCH('') ORDER BY `sent` DESC LIMIT 0,1000 OPTION max_matches=1000'
in 0.00 s, 3 hits, 3 total found

But as there are more mails from the past, which I tried using the
search field with dates, etc before I'm unsure if that works well.



2016-02-20 22:00 GMT+01:00 Janos SUTO <s...@acts.hu>:


I asked you to hit the search button, then show me the sphinx query,
not "type *, then hit the search button, and show me the sphinx query".

Note that there's indeed wildcard search, but not as you tried to use it.

Janos



On Sat, 20 Feb 2016, Matt . wrote:



HI guys,

I needed to get back on this, sorry.

When I do a wildmark search I get this on my logs:

sphinx query: 'SELECT id FROM main1,dailydelta1,delta1 WHERE
MATCH('@(subject,body)  *') ORDER BY `sent` DESC LIMIT 0,1000 OPTION
max_matches=1000' in 0.06 s, 0 hits, 0 total found

but noting back in piler.

Thanks so far!

2016-02-17 15:25 GMT+01:00 Janos SUTO <s...@acts.hu>:


hit the search button, then show me the matching sphinx query
(look in the mail log).

Janos


On 2016-02-17 13:43, Matt . wrote:



Sure I did but whatever I search I get back no results.

Crons are running.

Do you have a wildcard example that should always work ?



2016-02-17 13:41 GMT+01:00 admin-at-extremeshok-dot-com
<ad...@extremeshok.com>:



login as the auditor@local user

__.https://eXtremeSHOK.com
.__


On 17-Feb-16 2:18 PM, Matt . wrote:




Hi,

Not far as I can see, but maybe I'm searching the wrong way. Searches
don't give back anything at all atm.

2016-02-17 12:56 GMT+01:00 ad...@extremeshok.com
<ad...@extremeshok.com>:




The default auditor account reads all email from all domains

Sent from my iPhone


On 17 Feb 2016, at 10:47 AM, Matt . <yamakasi@gmail.com> wrote:


HI guys,

Is there (already) a feature so my Auditor can read all mailboxes ?
This because Piler is not used on userbase here but on Auditor base.

This for all domains/mailboxes.

I hope I can accomplish this in some way

Matt
















Re: User that can read all mailboxes

2016-02-20 Thread Janos SUTO


I asked you to hit the search button, then show me the sphinx query,
not "type *, then hit the search button, and show me the sphinx query".

Note that there's indeed wildcard search, but not as you tried to use it.

Janos


On Sat, 20 Feb 2016, Matt . wrote:



HI guys,

I needed to get back on this, sorry.

When I do a wildmark search I get this on my logs:

sphinx query: 'SELECT id FROM main1,dailydelta1,delta1 WHERE
MATCH('@(subject,body)  *') ORDER BY `sent` DESC LIMIT 0,1000 OPTION
max_matches=1000' in 0.06 s, 0 hits, 0 total found

but noting back in piler.

Thanks so far!

2016-02-17 15:25 GMT+01:00 Janos SUTO <s...@acts.hu>:

hit the search button, then show me the matching sphinx query
(look in the mail log).

Janos


On 2016-02-17 13:43, Matt . wrote:


Sure I did but whatever I search I get back no results.

Crons are running.

Do you have a wildcard example that should always work ?



2016-02-17 13:41 GMT+01:00 admin-at-extremeshok-dot-com
<ad...@extremeshok.com>:


login as the auditor@local user

__.https://eXtremeSHOK.com
.__


On 17-Feb-16 2:18 PM, Matt . wrote:



Hi,

Not far as I can see, but maybe I'm searching the wrong way. Searches
don't give back anything at all atm.

2016-02-17 12:56 GMT+01:00 ad...@extremeshok.com
<ad...@extremeshok.com>:



The default auditor account reads all email from all domains

Sent from my iPhone


On 17 Feb 2016, at 10:47 AM, Matt . <yamakasi@gmail.com> wrote:


HI guys,

Is there (already) a feature so my Auditor can read all mailboxes ?
This because Piler is not used on userbase here but on Auditor base.

This for all domains/mailboxes.

I hope I can accomplish this in some way

Matt











Re: 3 Message with wrong 2036 date

2016-02-18 Thread Janos SUTO

show me the Date: headers from these 3 emails.

Janos

On 2016-02-18 15:45, Tim Stumbo wrote:

I have 3 messages that always appear at the top of the search field
due to them having an incorrect date. The year on the messages is
2036. I would like to either delete these 3 e-mails or fix the index
for them with the corrected date.

Could someone help me with this issue?

Thanks




Re: Modify Indexer Cron Job

2016-02-17 Thread Janos SUTO



On Wed, 17 Feb 2016, Tim Stumbo wrote:


We would like the indexer job to run every 5 minutes instead of 30. Is that
an easy adjustment? 


yes. Edit piler's crontab, and you are done.

Janos

Re: User that can read all mailboxes

2016-02-17 Thread Janos SUTO

hit the search button, then show me the matching sphinx query
(look in the mail log).

Janos

On 2016-02-17 13:43, Matt . wrote:

Sure I did but whatever I search I get back no results.

Crons are running.

Do you have a wildcard example that should always work ?



2016-02-17 13:41 GMT+01:00 admin-at-extremeshok-dot-com 
:

login as the auditor@local user

__.https://eXtremeSHOK.com
.__


On 17-Feb-16 2:18 PM, Matt . wrote:


Hi,

Not far as I can see, but maybe I'm searching the wrong way. Searches
don't give back anything at all atm.

2016-02-17 12:56 GMT+01:00 ad...@extremeshok.com 
:


The default auditor account reads all email from all domains

Sent from my iPhone


On 17 Feb 2016, at 10:47 AM, Matt .  wrote:


HI guys,

Is there (already) a feature so my Auditor can read all mailboxes ?
This because Piler is not used on userbase here but on Auditor 
base.


This for all domains/mailboxes.

I hope I can accomplish this in some way

Matt







Re: Urgent: Piler " Message Failed Verification " Issue.

2016-02-15 Thread Janos SUTO

It seems that pilerget can return only the headers of the message
and not the body. If you indeed copied the whole output to the
message you sent, then it must be an error. However in that case
pilerget syslogs the problem. Please check for that problem.

Janos



On 2016-02-13 18:27, Mahmoud Ramadan wrote:

Hi Piler Users,
Here are the logs for one message marked as " Message Failed
Verification " and can NOT see it clearly in the Piler interface ,
Kindly advice .

root@piler:/var/log# pilerget 400056b6f35b19ba057c00e886907cb0
Received: from mailserver.telnsec.dom (192.168.0.121) by
 mailserver.telnsec.dom (192.168.221.5) with Microsoft SMTP Server
 (TLS) id 8.3.279.5; Sun, 7 Feb 2016 09:46:21 +0200
Content-Type: multipart/mixed;
boundary="_046ece66-61ca-4efb-87ef-93c20101f318_"
MIME-Version: 1.0
From: Network Performance Monitor 
Sender:

To: 
CC: , 
Subject: This is an alert from your SolarWinds server.
Message-ID:

Date: Sun, 7 Feb 2016 09:52:47 +0200
X-MS-Journal-Report:
Return-Path: <>

--_046ece66-61ca-4efb-87ef-93c20101f318_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

Sender: orion...@telnsec.com
Subject: This is an alert from your SolarWinds server.
Message-Id: <8B405D60B44E4F6DA361AADEB9808459@MonSRV2>
Recipient: hany.na...@telnsec.com
Recipient: ibraheem.ma...@telnsec.com

--_046ece66-61ca-4efb-87ef-93c20101f318_
Content-Type: message/rfc822;
name="This is an alert from your SolarWinds server."
Content-Description: This is an alert from your SolarWinds server.
Content-Disposition: attachment;
filename="This is an alert from your SolarWinds server."

Received: from MonSRV2 (19.16.2.11) by ma.mydomain.dom

This an alert indicating that one of the interfaces of CUCM wnet down

 (192.168.0.121) with Microsoft SMTP Server id 8.3.389.2; Sun, 7 Feb
2016
 09:52:44 +0200
thread-index: AdFhfCR7oDEup282R52+t6O6mQ0zdQ==
Thread-Topic: This is an alert from your SolarWinds server.
From: Network Performance Monitor 
To: 
CC: , 
Subject: This is an alert from your SolarWinds server.
Date: Sun, 7 Feb 2016 09:49:58 +0200
Message-ID: <8B405D60B44E4F6DA361AADEB9808459@MonSRV2>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft CDO for Windows 2000
Content-Class: urn:content-classes:message
Importance: normal
Priority: normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.4913
Return-Path: orion...@telnsec.com
X-MS-Exchange-Organization-OriginalArrivalTime: 07 Feb 2016
07:52:44.3026
 (UTC)
X-MS-Exchange-Organization-AuthSource: mailserver.telnsec.dom
X-MS-Exchange-Organization-AuthAs: Anonymous
X-MS-Exchange-Organization-PRD: telnsec.com [1]
X-MS-Exchange-Organization-SenderIdResult: None
Received-SPF: None (mailserver.telnsec.dom:
 orion...@telnsec.com does not designate permitted sender hosts)
X-MS-Exchange-Organization-SCL: 2
X-MS-Exchange-Organization-PCL: 2
X-MS-Exchange-Organization-Antispam-Report:
DV:3.3.14519.472;SID:SenderIDStatus None;OrigIP:192.168.0.151
X-MS-Exchange-Organization-OriginalSize: 784
X-MS-Exchange-Organization-AVStamp-Mailbox: MSFTFF;1;0;0 0 0
X-MS-Exchange-Organization-Recipient-Limit-Verified: True
X-MS-Exchange-Forest-RulesExecuted: mailserver

Best Regards,

Mahmoud Ramadan Ali

Network and VOIP Specialist.

Mobil: (+2) 01276877112

Blog [2]| Website [3]| LinkedIn [4]

Links:
--
[1] http://telnsec.com
[2] http://hotciscolabs.blogspot.com/
[3] http://telecomandsecurity.com/
[4] http://ch.linkedin.com/pub/mahmoud-ali/99/923/421/en




changes in the bitbucket issue tracker

2016-01-29 Thread Janos SUTO

Dear piler users,

I've decided to fix the bitbucket issue tracker settings, and disallow
creating issues as anonymous.

I expected the bitbucket jira issue tracker as a tool allowing piler
users to report bugs found in the software. And actually it has 
happened.


However other things also happened: several "issues", "major" "bugs"
making clear that the poster was pretty much clueless, having trouble
reading at least the FAQ.

So by excluding anonymous users I hope there will be a bit less 
annoyance

on bitbucket. I'd like to make it clear that all questions are welcome.
Here, on the mailing list, but bitbucket.

Janos



Re: error importing 'xxxx-imap-xxxxx.txt'

2016-01-24 Thread Janos SUTO

Hello Michael,

I've just reviewed the code, and found the matching part for each
open() and fopen() calls. Can you provide more clues, error messages,
etc. on the matter?

Janos

On 2016-01-14 16:57, Michael Volz wrote:

I kind of solved the problem. The problem was that the pilerimport
process had too many open file handles. I could finish the process by
increasing the limit. But I still guess it's not "right" of pilerimport
to keep all those files open, is it?

Am 04.01.2016 um 16:40 schrieb Michael Volz:

The files pilerimport complains about (e.g. 9964-imap-10014.txt) do
exist in the folder and contain email messages.

Am 04.01.2016 um 16:31 schrieb Michael Volz:

Hey everybody!

I have problems with pilerimport. I am running several imports from 
imap
folders and I am using the folder feature. After importing 
successfully
for some time (several hours), pilerimport begins many of the 
following

errors:

error importing '9964-imap-10014.txt'
failed to import: 9964-imap-10015.txt (id:
4000568a8e0a3a35d13c0068f88e0e35)

There are no errors in maillog or other logs. What could be the 
problem?

I invoke pilerimport as root, but in a directory that belongs to the
piler user. There are about 25 files in this folder. The command
line is like that:

pilerimport -i server -u user -p password -P port -f imap_folder_name 
-F

piler_folder_name

or:

pilerimport -i server -u user -p password -P port -R

Thanks for any help!

Michael








Re: MS Exhange Impersonation Issue With Piler .

2016-01-16 Thread Janos SUTO

Hello Mahmoud,

On Sat, 16 Jan 2016, Mahmoud Ramadan wrote:


I've successfully configured Piler integration with MS Exchange 2007 using Jornaling and everything 
is working fine. But i have an issue with Exhange Impersonation,I've configured user 
"mahmoud.aly" to get full access on user's "jack.gee" mailbox using the:

#add-mailboxpermission jack.gee -AccessRights FullAccess -user mahmoud.aly

And i've verified the new permissions But when i use the following command on Piler it 
takes sometime and then i get a connect() message as output and when checking the Piler i 
did NOT see the user "Jack.gee " mailbox messages imported into Piler !

#pilerimport -i mail.telnsec.com -u telnsec.com/mahmoud.aly/jack.gee -p 
mahmoudalypassword

connect()

So how to fix this issue ? any help will be appreciated,thanks in advance.


I think the imap server doesn't respond on port 143, try on 993 (imaps), 
ie. add the -P 993 to pilerimport, and try again.


Janos


Re: error importing 'xxxx-imap-xxxxx.txt'

2016-01-14 Thread Janos SUTO


I'd like you to view src/message.c, and locate process_message() function 
around the 265th line.


It has the following line in the ~283rd row:

fd = open(state->message_id_hash, O_CREAT|O_EXCL, S_IRUSR|S_IWUSR);

please check if you can see a few lines below that it has the following as 
well:


close(fd);


Janos


On Thu, 14 Jan 2016, Michael Volz wrote:



I kind of solved the problem. The problem was that the pilerimport
process had too many open file handles. I could finish the process by
increasing the limit. But I still guess it's not "right" of pilerimport
to keep all those files open, is it?

Am 04.01.2016 um 16:40 schrieb Michael Volz:

The files pilerimport complains about (e.g. 9964-imap-10014.txt) do
exist in the folder and contain email messages.

Am 04.01.2016 um 16:31 schrieb Michael Volz:

Hey everybody!

I have problems with pilerimport. I am running several imports from imap
folders and I am using the folder feature. After importing successfully
for some time (several hours), pilerimport begins many of the following
errors:

error importing '9964-imap-10014.txt'
failed to import: 9964-imap-10015.txt (id:
4000568a8e0a3a35d13c0068f88e0e35)

There are no errors in maillog or other logs. What could be the problem?
I invoke pilerimport as root, but in a directory that belongs to the
piler user. There are about 25 files in this folder. The command
line is like that:

pilerimport -i server -u user -p password -P port -f imap_folder_name -F
piler_folder_name

or:

pilerimport -i server -u user -p password -P port -R

Thanks for any help!

Michael












Re: Basic Pilar Questions Before Setup With MS Exchange 2007.

2016-01-14 Thread Janos SUTO


Hello Mahmoud,

On Thu, 14 Jan 2016, Mahmoud Ramadan wrote:


I'm planning to use Pilar and integrate it with MS Exchange 2007 for
archiving BUT i need clarification regarding some points that I'm confused
about.

1.How MS Exchange will send E-Mails for archiving to Pilar , i mean what are
the configuration steps required to integrate MS Exhcnage 2007 with Pilar ?


the preferred method is to setup smtp journaling, so exchange will send a 
copy of each email to piler via smtp. For the configuration steps, see

http://www.mailpiler.org/pdf/piler-1.1.0.pdf on page 16.

Perhaps it's time to improve the docs, and add these info as well.


2.I know that Pilar will archive every new messages BUT what about exsiting
mails in the Exchange DB.I have 4 Mailbox DBs and each of them 300 GB at
least so exporting the mailboxes as PST files and then converting them as
Mbox files to import them into Pilar is NOT practical as all ! so what is
the best way to import this big size of mailboxes into Pilar ?


the impersonation feature of exchange, see below


3.What is the role of the " Impersonation with Exchange " feature ? can
anyone explain it to me ?


in a nutshell you create (or grant the role) a use for this purpose. Then 
you login all the time with this user account AND you specify which 
mailbox to read.


So you can iterate over all mailboxes using a single user account only.
See http://www.mailpiler.org/wiki/current:importing-emails for more.


Janos



Re: New Install issues

2015-12-01 Thread Janos SUTO


On Mon, 30 Nov 2015, Warren H. Prince wrote:



Thanks for getting back to me, but I?m still stuck.

I got rid of postfix on the Piler server and can see email for 
arch...@princelaw.com mounting up on the mail server.  I have enabled IMAP_AUTH 
and SSL.  From the command line, if I issue:
pilerimport -i mail.princelaw.com -u archive.princelaw.com -p 


the idea behind the smtp copy is that you send emails TO THE ARCHIVE HOST,
not to the mail server. So you are using the wrong address, because it's
not on piler. You should use something like arch...@piler.princelaw.com or
arch...@archive.princelaw.com, not sure the hostname of your archive.



I get:
BAD [ALERT] Plaintext authentication not allowed without SSL/TLS, but 
your client did it anyway. If anyone was listening, the password was exposed. 
A1 NO [PRIVACYREQUIRED] Plaintext authentication disallowed on non-secure 
(SSL/TLS) connections.


but your client did it anyway. If anyone was listening, the password was
exposed. A1 NO [PRIVACYREQUIRED] Plaintext authentication disallowed on
non-secure (SSL/TLS) connections.

please check the documentation. I doubt that you have an imap account
named like 'archive.princelaw.com'. Also if plain text imap traffic is
disabled, then why do you try so? Add the following parameter: -P 993


although I can?t seem to see any errors in the mail server?s logs.  Bottom line 
is that Piler isn?t showing any email.  The mail.log on the Piler server shows 
nothing irregular?  just startup and connections.


yeah, piler is just sitting and waiting for the mail server to send some
mails finally.

Janos



Re: New Install issues

2015-12-01 Thread Janos SUTO

On Mon, 30 Nov 2015, Warren H. Prince wrote:


Thanks for getting back to me, I?ve made some progress, but still not quite 
home.  I was able to import some email with the command:
pilerimport -i mail.princelaw.com -u arch...@piler.princelaw.com -p  -P 993

And I can search that data, but I?ve waited an hour since that succeeded and 
nothing more has appeared.  I have 2 ?import? commands in crontab:

1    * * * * root       (cd /var/www/piler.princelaw.com; /usr/bin/php 
/usr/local/libexec/piler/gmail-imap-import.php .)
45   * * * * root       /usr/local/bin/pilerimport -d /var/piler/imap -r -q

But they don?t seem to be working.  Do I need to add the -i and -P parameters?  
If them, what about the -u and -p?  I?m not real anxious to put the password in 
a cron job.

what are you trying to do? Firstly, I'd suggest, let's fix the smtp 
delivery. When it's confirmed to work, then we may move on to the next 
issue.


Janos

Re: Empty message body

2015-11-30 Thread Janos SUTO

Hello Konstantin,


Can you please clarify what exactly delete pilerpurge ?
Will piler use old sig value from piler.attachment if
an attachment was removed for the new attachment with same name,type and 

size?

it shouldn't, however if it does, then it's definitely a bug.
The pilerpurge utility should delete the matching entries from the
attachment table. Anyway, I'll do some tests to make sure that
purging works properly.



I have two questions here:
1. Can someone suggest is it possible to extract message body?


try running pilerget 40005644f5d6083278fc00d9460b8312

It's likely to return some error, since pilerexport reported a 
verification problem.




2. We do not need search by content of doc,xls and pdf files.
Can i substitute catdoc and pdftotext with another binary to return 0 

for piler?

you can, however you may recompile piler after you remove catdoc and 
pdftotext. If the configure command can't find them, then piler won't

attempt to use them.

Janos



Re: Custom Skin

2015-11-10 Thread Janos SUTO

Hello Stephen,

On 2015-11-10 16:28, srisin...@meintl.com wrote:

Okay, I feel dumb in asking, but I have done everything possible to
try to add new themes to Piler. The only thing I could find to work is
to replace the metro-bootstrap.css file with a new one, but that
replaces everybody's default theme.

The instructions at
http://www.mailpiler.org/wiki/current:administering-piler are no
longer right. How about a little something for the llama?


hmmm, what would you change? Only a css file, or the tpl files as well?
I'll try to update the docs to reflect on how to customize it.

Janos



Re: retention time considerations

2015-11-02 Thread Janos SUTO

Hello Arthur,

On 2015-11-02 14:49, Arthur Emerson wrote:


Stupid question - is there any reason why this cannot be upgraded to a
64-bit time_t like most modern Unix-based operating systems are already
converting to, so they do not have 2038 problems?  This is not a new
issue...


You are right. I've just fixed all of the internal time related 
variables
(sent, arrived, retained) to time_t. If the platform is 64 bit, then 
time_t

is 64 bit as well. For 32 bit platforms time_t is 32 bit long.

The related mysql columns are 4 bytes long as well. Not sure if it's 
worth
to upgrade them to 8 byte long type. For dates < 2106 4 byte data type 
is

fine. If you plan to use piler in the next century, then you should use
bigint types for sent, arrived and retained columns.

Janos




On 11/1/15, 6:53 AM, "Janos SUTO" <s...@acts.hu> wrote:






Dear piler users,

piler uses a 4 byte value to store the retention timestamp. The 
trouble

is that
it has 23 more years of support, ie. up to 2038.

Usually it's not a problem, since most people keep emails for 3-10
years. However,
what if you have to store emails for a longer time, eg. to eternity?

You have two options:

#1: leave the default 7 years default retention, and never run the
purging cron
job. Thus no matter if an email is actually aged, it won't be removed
from the
archive.


#2: set a proper retention period, eg. 36500 days (~= 100 years). Note
that this
extreme value spans over 2038, and thus the retained column in the
metadata table
will be overflown, and a wrong value will be stored, and may cause
pilerpurge to
remove some emails that it shouldn't.

To mitigate the issue, I've tweaked the parser to use an unsigned 4 
byte

type
for the retained value. Thus I've extended the retained column support
to
2106. I'll modify some internal variables to support 2106 for the 
'sent'

and
'arrived' values soon, even though I have 20 years or more to fix it.

2106 is still not enough for 100 years (2106 < 2015 + 100), but I
believe that
for most users it's ok, just be aware of the limitation, and make sure
to
setup a reasonable retention time for your emails.

Email has been with us for ~30 years or so, and I believe it stays 
with

us
for another 30 years (give or take), though I'm not sure that people
will
still send emails in 2100. Who knows? The problem is that chances are 
I

won't be able to figure it out.

So to sum it up, if you are concerned about the 2038 issue, then 
upgrade
to the latest master branch which supports 2106 for the retained 
column.


The related bitbucket issue:
https://bitbucket.org/jsuto/piler/issues/626/year-2038-problem

Best regards,
Janos






retention time considerations

2015-11-01 Thread Janos SUTO

Dear piler users,

piler uses a 4 byte value to store the retention timestamp. The trouble 
is that

it has 23 more years of support, ie. up to 2038.

Usually it's not a problem, since most people keep emails for 3-10 
years. However,

what if you have to store emails for a longer time, eg. to eternity?

You have two options:

#1: leave the default 7 years default retention, and never run the 
purging cron
job. Thus no matter if an email is actually aged, it won't be removed 
from the

archive.


#2: set a proper retention period, eg. 36500 days (~= 100 years). Note 
that this
extreme value spans over 2038, and thus the retained column in the 
metadata table
will be overflown, and a wrong value will be stored, and may cause 
pilerpurge to

remove some emails that it shouldn't.

To mitigate the issue, I've tweaked the parser to use an unsigned 4 byte 
type
for the retained value. Thus I've extended the retained column support 
to
2106. I'll modify some internal variables to support 2106 for the 'sent' 
and

'arrived' values soon, even though I have 20 years or more to fix it.

2106 is still not enough for 100 years (2106 < 2015 + 100), but I 
believe that
for most users it's ok, just be aware of the limitation, and make sure 
to

setup a reasonable retention time for your emails.

Email has been with us for ~30 years or so, and I believe it stays with 
us
for another 30 years (give or take), though I'm not sure that people 
will

still send emails in 2100. Who knows? The problem is that chances are I
won't be able to figure it out.

So to sum it up, if you are concerned about the 2038 issue, then upgrade
to the latest master branch which supports 2106 for the retained column.

The related bitbucket issue: 
https://bitbucket.org/jsuto/piler/issues/626/year-2038-problem


Best regards,
Janos



Re: Pilerimport - IMAP help

2015-10-28 Thread Janos SUTO

thanks for the replies. Perhaps it's worth to share what you did
on the windows side, eg. what commands you ran, etc. before running
pilerimport.

Janos

On 2015-10-28 16:14, Cristiano Manzoni wrote:

pilerimport -i exchange.test.local -u
test.local/userwithrights/usermailbox -p Password -P 993 -x 'Junk
E-mail'

On Wed, Oct 28, 2015 at 4:04 PM Jason Caspersen
 wrote:


Hi everyone, I'm having some trouble trying to import the legacy
email from my Exchange 2010 server.

Long story short, I used the instructions from
http://www.mailpiler.org/wiki/current:importing-emails [1]

I need to impersonate each user, because I don't have access to the
AD password.

I have "full access" setup correctly and can view the mailbox email
using OWA http://mail.domain.com/owa/n...@domain.com [2], however
when I try use Pilerimport using the following format:

pilerimport -i mail.domain.com [3] -u domain.com/superuser/username
[4] -p superpassword

I get an error:

login failed, server reponse: A1 NO LOGIN failed.

On the flip side, if I know the users password it works correctly

pilerimport -i mail.domain.com [3] -u domain.com/user [5] -p
password

Has anyone successfully imported their Legacy email using IMAP
without having to reset the users AD account?

Thanks,

Jason Caspersen



Links:
--
[1] http://www.mailpiler.org/wiki/current:importing-emails
[2] http://mail.domain.com/owa/n...@domain.com
[3] http://mail.domain.com
[4] http://domain.com/superuser/username
[5] http://domain.com/user





RE: Exchange 2010 journal user messages appearing in archive

2015-10-19 Thread Janos SUTO

On 2015-10-19 21:18, Jason Morrill wrote:
Thank you for the suggested solution. It seems to have fixed my 
problem:


Edit /usr/local/etc/piler.conf
Revised, hostid= with my server name
Save changes
Restart rc.piler

Now records in the database don't have the extra recipient, 
jour...@here.com




Great! You can get rid of the bogus entries as well:

$ mysql -u piler -p piler
mysql> delete from rcpt where `to`='jour...@here.com';


Janos




Re: Exchange 2010 journal user messages appearing in archive

2015-10-19 Thread Janos SUTO

Hello Jason,

On 2015-10-19 18:40, Jason Morrill wrote:

I am journaling all messages to/from our Exchange 2010 server. I
followed the instructions creating a mail contact, enabling
journaling, and creating the Send Connector. And it's working great
... except

... all messages are being stored with their original recipient AND
the journal mail contact account. This means my archive includes
duplicate messages for every single message to/from our server. This
makes searching for messages problematic and messy. Less problematic
for storage space since Piler deduplicates nicely.

If my description is unclear here's an example:

1) Message arrives from some...@somwhere.com

2) It's delivered to the recipient myst...@here.com

3) The message is then journaled to jour...@here.com and delivered to
MailPiler

4) MailPiler stores the message as if it were sent to both
myst...@here.com and jour...@here.com

Is there any way to prevent MailPiler from recording the message being
sent to the "journal" address?


I'm not sure if I understand it correctly, but perhaps you have not set
the hostid parameter properly in piler.conf.

Let's say you have hostid=mailarchiver in piler.conf (this is the 
default

value).

Piler versions before the master branch (by default) adds the rcpt to
address (ie. the journaling address) to the recipient list provided
that the rcpt to address is not in the @hostid domain.

So using your example piler adds jour...@here.com to the recipient list.

How to fix it? Set hostid value to the domain part of the journaling
address, eg. here.com in your example.

For my domain (acts.hu) it would look like this:

recipient: s...@acts.hu
journal address: arch...@archive.acts.hu
hostid=archive.acts.hu

Note that the most recent builds (not released yet as a stable version)
use the 'process_rcpt_to_addresses' piler.conf variable to determine if
to process the email addresses in the smtp rcpt stage or not (by default 
no)


Let me know if it helps.

Janos




Re: beginner/install question, trying to use with docker

2015-10-18 Thread Janos SUTO

On 2015-10-18 00:54, Justin Mecham wrote:

A very accomplished developer friend of mine places all of his static
things in one docker image and all the acid/volatile data in another.


ok, then it's a vote for a single docker image

Janos



Re: beginner/install question, trying to use with docker

2015-10-16 Thread Janos SUTO

Hello Stefan,

On 2015-10-16 12:52, stefanz wrote:


 I wanted to try out piler, recently looked at docker (also a
beginner) I thought this was a good idea.
 So my first question: Is there a docker image available?


yes: https://bitbucket.org/jsuto/piler/downloads/docker-896.tar.gz

Note that it's a somewhat older build, however it should work.


 As I did not find one, but I found this thread
https://www.mail-archive.com/piler-user@list.acts.hu/msg00669.html
[1],
 I tried to use the Dockerfile there. It seems the file
http://www.mailpiler.org/download/docker-first-time-fix.sh [2] does
not seem to exist anymore.
 My second question:  is this file no longer required?


it is required as part of the docker image deployment. There has been
a site upgrade, and forgot to move it to the new docroot. I've just
fixed it, so try again, and thanks for noticing.

Note that the current docker image is a full stack, ie. containing
everything required for piler: the application itself, sphinx, mysql,
and nginx. Not sure if it's better to provide it as a whole and complete
package or would it be better to ship 4 docker containers each with a
single application, ie.

docker1: mysql
docker2: sphinx
docker3: nginx
docker4: piler

Let me know what you (and the list) think about it.

Janos



high availability options

2015-10-05 Thread Janos SUTO


Dear piler-users,

I've got a feature request for DAG-like high resiliency.

I've designed piler to run on a single host only. However it's a valid
business demand that let the archive be fault tolerant.

Though piler has no builtin high availability feature, something close
might be achieved.

Setup two piler boxes (they can be even in distant datacenters, not 
necessarily
on the same lan). These 2 nodes are independent, having no knowledge of 
each other.


The only difference is that the smtp copy must be done to two addresses:
archive@piler1 and archive@piler2. Thus you have an email archived at 
two
places. Users should access http(s)://archive which should point to 
piler1,

and in case of trouble you just point it to piler2.

I'm not sure if such a setup may be ok for you, but I'm interested in 
your

ideas and expectations.

Janos



RE: Search Issues

2015-10-01 Thread Janos SUTO


On 2015-09-30 22:47, srisin...@meintl.com wrote:


Janos, we can do multiple searches if MailPiler can't handle a complex
search like we were attempting. (I don't know if other archivers can
do it, either, so don't think I'm slamming MailPiler!) We were able to
find the data we needed; it just seemed more difficult than it should
have been.


well, try the single query, start with the date, and see if the result
looks ok.


RE: eml file names... That is a good point. If you try to save
multiple emails from Outlook, for example, it prompts saying that it
already has a file saved with that name (after the first). So, even
Microsoft doesn't automatically handle it. Would it be possible to
export the emails by subject, appending a number based of sent time
(1,2,3, etc., oldest being 1, incrementing the number with newer
emails)?


ok, I'll add a hack the lets you do that. Would you open a feature
request on bitbucket? It's more likely to stay on my radar.


RE: Time & Date search...  Is this something that can't be done? Or
would this better be served as a feature request on BitBucket?


yes. Anyway even the year-month-day date trio has several possible
combinations, so I tried to optimise the routine for parsing it.

Adding hours and minutes is definitely possible just makes my job
parsing the date string more difficult.

Janos




Re: Search Issues

2015-09-30 Thread Janos SUTO


Hello Stephen,

On 2015-09-30 16:37, srisin...@meintl.com wrote:

We received a request to pull emails to-or-from three emails from a
certain date forward.

We tried several different search methods, and I used the Sphinx
advanced search criteria to come up with something like the following:

(from: ema...@domain.com| ema...@domain.com | ema...@domain.com)

When we ran this search, we seemed to get the proper amount of emails,
but no text appeared in the email results. If we released an email,
the email released correctly (text and images were present).

We were finally able to get the complete search results we needed, by
using the following search:

from: ema...@domain.com|ema...@domain.com|ema...@domain.com | to:
ema...@domain.com|ema...@domain.com|ema...@domain.com, date1:
2015-09-18

Is there a specific order that search terms need to be? Why did the
initial search not include the email text? Also, is it possible to


well, check the maillog if the parsed query looks OK. Anyway it might
be easier to do two search queries:

from: ema...@domain.com OR ema...@domain.com OR ema...@domain.com, 
date1: 2015-09-18


then another:

to: ema...@domain.com OR ema...@domain.com OR ema...@domain.com, date1: 
2015-09-18


If you insist on running a single query, then start with the date:

date1: 2015-09-18, (from: ema...@domain.com OR ema...@domain.com OR 
ema...@domain.com)

OR (to: ema...@domain.com OR ema...@domain.com OR ema...@domain.com)

Also check the maillog if the sphinx query is built properly.


change the *.eml filename to better reflect the email, perhaps using
the subject, for example?


might be possible, however how to deal with a message thread having 5 
messages

with the same subject, eg. Re: what was this?



I also can't figure out how to search for times. Adding time to a date
(-MM-DD hh:mm:ss) changes the search to incorrect results or gives
no results whatsoever. What gives? (aka, I feel like I'm doing this
wrong. Am I, or is this a bug in the search?)


well, the date parser handles year, month and day only. Don't specify 
anything else.


Btw. James is right with the highlighting, it was fixed recently in the 
master branch.


Janos



Re: Pilerimport - periodical import

2015-09-30 Thread Janos SUTO


Hello,

On 2015-09-30 18:42, Federico wrote:


 if I use pilerimport from an external IMAP mailbox, does it retrieve
*every time* all the messages or it will keep track of the index
number of the already downloaded one? In other terms, the "de-duping"
is done before importing or after importing?


it's not deduping, rather keeping a history. By default pilerimport
downloads all emails, then discards already imported emails. However,
you may instruct it to start downloading (and archiving messages from
a certain position, ie. from the 1234th message.


 The behavior I want to reach is similar on what is described on the
Mailpiler Docs regarding Gmail, but my intent is with IMAP, where "the
importer script remembers the id of the last downloaded message, so it
won't download from the first message when it runs again." Does also
pilerimport behaves like that?


you may achieve something like that, see above.


 HERE is the description of my scenario:

 a) Mr.User has one IMAP account on an external server (we cannot
touch the config)

 b) Mr.User receives 500Mb of email every month in his mailbox, and
cannot delete it.

 c) Mr. User want to keep an archive for 5 years of his email, and the
mailbox is too small.

 He asked me to set-up MailPiler to put in the archive the email older
than 1 month, every month.

 Mr.User does not want Piler to delete imported mail from his mailbox.
He will manually delete periodically some old emails.

 I thought to do as follow:

 - standard deployment of Piler
 - deployment of a separate mailserver with one mailbox
 - use of imapsync to get the mail older than 1 month from Mr.User
mailbox and put to a local mailbox
 - crontab pilerimport to get mails from the local mailbox to Piler.


pilerimport may connect to the imap account directly. The only trouble
is that the current position is unreliable if Mr. User removes messages
on his own from the mailbox.

Another possible solution may be to use a 3rd party tool to fetch only
those messages from the imap account that need to be imported. Then
pilerimport may process the local folder.

I understand that you can't configure the remote server. However, is it
possible for Mr. User to setup a message rule to forward a copy of each
incoming email to piler (=arch...@piler.domain.com)?

Janos





Re: sph_index table not populated after imap import

2015-09-21 Thread Janos SUTO


Hello Cristiano,

On 2015-09-21 10:00, Cristiano Manzoni wrote:

we've a strange situation here, after import all user's email via
imap, the sph_index table is not populated and so the search doesn't
show results.


is there any error message on stdout or in mail log?


I've to stop cron jobs and manually rerun a reindex -a to see results
of search.
After the reindex command I can see results for a search but all the
email with attachments don't show attach in the preview, btw if I
download the eml the attachments are there.


check if the matching entry in the metadata table has attachment > 0


1.2.0-master build 919
Sphinx 2.2.10


was it an upgrade? If so have you run the database schema upgrade 
script?


Janos



Re: Upgrading/updating Piler

2015-09-08 Thread Janos SUTO


Hello Stephen,

On 2015-09-08 20:06, srisin...@meintl.com wrote:

So, I'm looking at updating Piler to the latest Master Branch. We have
Exchange 2007 for our mail server, so I believe I can pause (or
suspend) our SMTP queue for piler.meintl.com. Assuming I'm doing this
during a low-email period of time, that should give me time to update
our front-end, the DB, then test to ensure everything went as planned.
Then, once everything is good, I resume the SMTP queue, and life
continues on.

Does that sound right, or am I off base here?


it sounds right. In general, the upgrade consists of the following 
steps:


1. update the database schema - if necessary (look at the 
util/db-upgrade*.sql

scripts)

2. compile piler

3. stop the piler daemon (and other binaries, eg. pilerimport if 
running)

run make install, then start the piler daemon

4. overwrite the gui with webui/* contents

Usually the whole process is done in 5 mins, and the smtp queue of the
mail server should be able to survive it.

Janos



imap import issue

2015-08-31 Thread Janos SUTO


Dear piler users,

an odd bug has been discovered in the imap import feature of piler, see
https://bitbucket.org/jsuto/piler/issues/593/emails-are-removed-from-imap-server-when
for more.

Workaround:

Do NOT use -r with -D


Fix:

Apply the following patch to src/imap.c


diff --git a/src/imap.c b/src/imap.c
index b41dcf8..bff1795 100644
--- a/src/imap.c
+++ b/src/imap.c
@@ -248,7 +248,7 @@ int process_imap_folder(int sd, int *seq, char 
*folder, struct session_data *sda

   if(rc == ERR) printf("error importing '%s'\n", filename);
   else {

- if(data->import->remove_after_import == 1){
+ if(data->import->remove_after_import == 1 && dryrun == 0){
 snprintf(buf, sizeof(buf)-1, "A%d STORE %d +FLAGS.SILENT 
(\\Deleted)\r\n", *seq, i);

 n = write1(sd, buf, strlen(buf), use_ssl, data->ssl);
 read_response(sd, buf, sizeof(buf), seq, data, use_ssl);
@@ -260,7 +260,7 @@ int process_imap_folder(int sd, int *seq, char 
*folder, struct session_data *sda


}

-   if(data->import->remove_after_import == 1){
+   if(data->import->remove_after_import == 1 && dryrun == 0){
   snprintf(buf, sizeof(buf)-1, "A%d EXPUNGE\r\n", *seq);
   n = write1(sd, buf, strlen(buf), use_ssl, data->ssl);
   read_response(sd, buf, sizeof(buf), seq, data, use_ssl);


I'll push the fix to the master branch soon.


Janos



Re: Housekeeping after email purging.

2015-08-27 Thread Janos SUTO


On 2015-08-27 08:57, Ross Gailer wrote:

If you are seeing ghosts, then chances are that the actual email has
been removed. I believe that the lists shown in the GUI are built from
the Sphinx search results. If the files have been removed and the
index has not been updated, you will end up with ghosts.

Not sphinx expert by any means. Try the killist and see how you go.


rebuilding the sphinx indices is one option, but I don't think you need
to do that, because the killlist feature of sphinx should handle it, and
prevent these ghosts to be visible in the gui.

Rebuilding may be a good options if you have lots of aged messages, and
you want to reduce the size of the sphinx index. Note that the sphinx
data for the removed emails are still present, sphinx only suppresses
those hits.

I tried the following. Just set deleted=1 for a single message, however
it was still present in the same and subsequent query.

Then I executed the delta indexing script manually (only because I was
impatient to wait 20 mins until it was scheduled), and after that this
message is disappeared from the hits.

I'll add this to the FAQ, perhaps other users may find this tidbit 
useful.


Janos

PS: as you can see, Max, your email has just reached the mailing list.
Sorry, I couldn't answer sooner.




  1   2   3   >