Migration from bitbucket to github, and the new release of 1.4.5

[Janos SUTO]

Dear piler users,

I've been busy with migrating the piler repo from bitbucket to github.
You may find the new repo at https://github.com/jsuto/piler

Note that I'll keep the bitbucket repo, because the issue tracker still
contains invaluable information. Too bad, that bitbucket returned an 
error

when I tried to export the issues.

Anyway, from now on please use the issue tracker on github. I'll compile 
a
template that helps you to provide the necessary data to create a good 
issue.


On the flip side, I've created a new release 1.4.5. One of the most 
notable
changes is the refactoring of the piler-smtp daemon. Now it has a more 
efficient

logic that provides a 10x performance increase.

The other change is to feature a read-only manticore connection to 
improve security. See
https://github.com/jsuto/piler/blob/master/RELEASE_NOTES#L1-L34 for the 
details.


Also the release workflow provides a docker image that's built at every 
release, see
https://hub.docker.com/r/sutoj/piler for the images. The workflow builds 
deb packages

for Ubuntu 22.04 on both amd64 and arm64 platforms.

And there's more. I managed to integrate modsecurity WAF to piler using
traefik, see https://mailpiler.com/protecting-piler-with-a-waf/ for 
details.


Janos

Re: FATAL: no tables found in config file '/etc/piler/manticore.conf' in docker install

[Janos SUTO]

Right. The missing piece in config-site.php is

$config['SPHINX_MAIN_INDEX'] = 'piler1';

I'll update the docker stuff with it.

Janos


On 2024-03-04 09:09, Gerd Wilhelm wrote:

Thank you for your patient help!

/etc/piler/config-site.php

already contains $config['RT'] = 1;

as the last line.

Yours, Gerd

Am 04.03.24 um 08:06 schrieb Janos SUTO:


OK, you are almost there. The gui is configured to use the plain
manticore tables. Set the following in /etc/piler/config-site.php,
then try again:

$config['RT'] = 1;

Janos

On 2024-03-03 20:26, Gerd Wilhelm wrote:
Hi Janos!

i have started rsyslogd, and found

Mar  3 19:17:55 5257465bf880 piler-webui[34]: ERROR:
SQLSTATE[42000]: Syntax error or access violation: 1064 unknown
local table(s) 'main1, dailydelta1, delta1' in search request

in /var/log/syslog when hitting the search button (there is no
mail.log inside the container).

mysql -h 172.18.0.3 -u piler -p xxx

use piler;

show tables;

gives 42 tables, but in fact no main1, dailydelta1, delta1.

yours, Gerd

Am 03.03.24 um 20:06 schrieb Janos SUTO:

Verify that rsyslogd is running in the container. Then as an auditor

just hit the search button, and notice the syslogged "sphx" query
in /var/log/mail.log. What would that be?

Janos

On 2024-03-03 18:28, Gerd Wilhelm wrote:
Hi Janos!

Thanks for your help!

"pilerconf -q rtindex" gives rtindex=1

"mysql -h0 -P9306 <<< "show tables"" shows three tables (note1,
piler1, tag1) with type rt

"mysql -h0 -P9306 <<< "select * from piler1"" gives me the number of
hits corresponding to the importet mails.

It seems as am nearly arrived.

But i still do not get any hits when searching in the Web UI.

yours, Gerd

Am 03.03.24 um 18:15 schrieb Janos SUTO:

Hello Gerd,

the compose file sets RT=1 (=realtime index). It means you should
not
run the indexer scripts any longer. Try the following:

Run "pilerconf -q rtindex", and verify the output is rtindex=1

Then run mysql -h0 -P9306 <<< "show tables", and verify that you
can see rt table(s). Then run mysql -h0 -P9306 <<< "select * from
piler1",
and you should see some hits.

Janos SUTO

On 2024-03-03 17:59, Gerd Wilhelm wrote:
Hi Everyone,

I want to install a new Server, but i do not arrive in getting the
indexing to work.

I have chosen to run piler as a docker container and have used



https://bitbucket.org/jsuto/piler/src/master/docker/docker-compose.yaml.

After starting with "docker-compose up" piler seems to run, i can
connect to the Web-Interface and i can log in with the default
credentials.

If i import mails, the counter "Processed Mails" on the Health
Monitor page  increases as expected.

But if i log in as auditor i do not find any mail.

if i enter the docker container with

"docker exec -it  /bin/bash"

then

"su - piler"FATAL: no tables found in config file
'/etc/piler/manticore.conf'

and then

"/usr/libexec/piler/indexer.delta.sh" or
"/usr/libexec/piler/indexer.main.sh"

i get the following error:

"FATAL: no tables found in config file '/etc/piler/manticore.conf'"

Can anyone give me some advise how to investigate further?

Yours,

Gerd


--
Gerd Wilhelm (RHCE)
Tel.: +496414419289
it_werk.wilhelm
Neue Str. 7
35396 Gießen
https://www.gi-it.de

Re: FATAL: no tables found in config file '/etc/piler/manticore.conf' in docker install

[Janos SUTO]

OK, you are almost there. The gui is configured to use the plain
manticore tables. Set the following in /etc/piler/config-site.php,
then try again:

$config['RT'] = 1;

Janos

On 2024-03-03 20:26, Gerd Wilhelm wrote:

Hi Janos!

i have started rsyslogd, and found

Mar  3 19:17:55 5257465bf880 piler-webui[34]: ERROR: SQLSTATE[42000]: 
Syntax error or access violation: 1064 unknown local table(s) 'main1, 
dailydelta1, delta1' in search request


in /var/log/syslog when hitting the search button (there is no mail.log 
inside the container).


mysql -h 172.18.0.3 -u piler -p xxx

use piler;

show tables;

gives 42 tables, but in fact no main1, dailydelta1, delta1.

yours, Gerd





Am 03.03.24 um 20:06 schrieb Janos SUTO:



Verify that rsyslogd is running in the container. Then as an auditor
just hit the search button, and notice the syslogged "sphx" query
in /var/log/mail.log. What would that be?

Janos

On 2024-03-03 18:28, Gerd Wilhelm wrote:

Hi Janos!

Thanks for your help!

"pilerconf -q rtindex" gives rtindex=1

"mysql -h0 -P9306 <<< "show tables"" shows three tables (note1, 
piler1, tag1) with type rt


"mysql -h0 -P9306 <<< "select * from piler1"" gives me the number of 
hits corresponding to the importet mails.


It seems as am nearly arrived.

But i still do not get any hits when searching in the Web UI.

yours, Gerd



Am 03.03.24 um 18:15 schrieb Janos SUTO:



Hello Gerd,

the compose file sets RT=1 (=realtime index). It means you should 
not

run the indexer scripts any longer. Try the following:

Run "pilerconf -q rtindex", and verify the output is rtindex=1

Then run mysql -h0 -P9306 <<< "show tables", and verify that you
can see rt table(s). Then run mysql -h0 -P9306 <<< "select * from 
piler1",

and you should see some hits.

Janos SUTO

On 2024-03-03 17:59, Gerd Wilhelm wrote:

Hi Everyone,

I want to install a new Server, but i do not arrive in getting the 
indexing to work.


I have chosen to run piler as a docker container and have used

https://bitbucket.org/jsuto/piler/src/master/docker/docker-compose.yaml. 
After starting with "docker-compose up" piler seems to run, i can 
connect to the Web-Interface and i can log in with the default 
credentials.


If i import mails, the counter "Processed Mails" on the Health 
Monitor page  increases as expected.


But if i log in as auditor i do not find any mail.

if i enter the docker container with

"docker exec -it  /bin/bash"

then

"su - piler"FATAL: no tables found in config file 
'/etc/piler/manticore.conf'


and then

"/usr/libexec/piler/indexer.delta.sh" or 
"/usr/libexec/piler/indexer.main.sh"


i get the following error:

"FATAL: no tables found in config file '/etc/piler/manticore.conf'"

Can anyone give me some advise how to investigate further?

Yours,

Gerd

Re: FATAL: no tables found in config file '/etc/piler/manticore.conf' in docker install

[Janos SUTO]

Verify that rsyslogd is running in the container. Then as an auditor
just hit the search button, and notice the syslogged "sphx" query
in /var/log/mail.log. What would that be?

Janos

On 2024-03-03 18:28, Gerd Wilhelm wrote:

Hi Janos!

Thanks for your help!

"pilerconf -q rtindex" gives rtindex=1

"mysql -h0 -P9306 <<< "show tables"" shows three tables (note1, piler1, 
tag1) with type rt


"mysql -h0 -P9306 <<< "select * from piler1"" gives me the number of 
hits corresponding to the importet mails.


It seems as am nearly arrived.

But i still do not get any hits when searching in the Web UI.

yours, Gerd



Am 03.03.24 um 18:15 schrieb Janos SUTO:



Hello Gerd,

the compose file sets RT=1 (=realtime index). It means you should not
run the indexer scripts any longer. Try the following:

Run "pilerconf -q rtindex", and verify the output is rtindex=1

Then run mysql -h0 -P9306 <<< "show tables", and verify that you
can see rt table(s). Then run mysql -h0 -P9306 <<< "select * from 
piler1",

and you should see some hits.

Janos SUTO

On 2024-03-03 17:59, Gerd Wilhelm wrote:

Hi Everyone,

I want to install a new Server, but i do not arrive in getting the 
indexing to work.


I have chosen to run piler as a docker container and have used

https://bitbucket.org/jsuto/piler/src/master/docker/docker-compose.yaml.

After starting with "docker-compose up" piler seems to run, i can 
connect to the Web-Interface and i can log in with the default 
credentials.


If i import mails, the counter "Processed Mails" on the Health 
Monitor page  increases as expected.


But if i log in as auditor i do not find any mail.

if i enter the docker container with

"docker exec -it  /bin/bash"

then

"su - piler"FATAL: no tables found in config file 
'/etc/piler/manticore.conf'


and then

"/usr/libexec/piler/indexer.delta.sh" or 
"/usr/libexec/piler/indexer.main.sh"


i get the following error:

"FATAL: no tables found in config file '/etc/piler/manticore.conf'"

Can anyone give me some advise how to investigate further?

Yours,

Gerd

Re: FATAL: no tables found in config file '/etc/piler/manticore.conf' in docker install

[Janos SUTO]

Hello Gerd,

the compose file sets RT=1 (=realtime index). It means you should not
run the indexer scripts any longer. Try the following:

Run "pilerconf -q rtindex", and verify the output is rtindex=1

Then run mysql -h0 -P9306 <<< "show tables", and verify that you
can see rt table(s). Then run mysql -h0 -P9306 <<< "select * from 
piler1",

and you should see some hits.

Janos SUTO

On 2024-03-03 17:59, Gerd Wilhelm wrote:

Hi Everyone,

I want to install a new Server, but i do not arrive in getting the 
indexing to work.


I have chosen to run piler as a docker container and have used

https://bitbucket.org/jsuto/piler/src/master/docker/docker-compose.yaml.

After starting with "docker-compose up" piler seems to run, i can 
connect to the Web-Interface and i can log in with the default 
credentials.


If i import mails, the counter "Processed Mails" on the Health Monitor 
page  increases as expected.


But if i log in as auditor i do not find any mail.

if i enter the docker container with

"docker exec -it  /bin/bash"

then

"su - piler"FATAL: no tables found in config file 
'/etc/piler/manticore.conf'


and then

"/usr/libexec/piler/indexer.delta.sh" or 
"/usr/libexec/piler/indexer.main.sh"


i get the following error:

"FATAL: no tables found in config file '/etc/piler/manticore.conf'"

Can anyone give me some advise how to investigate further?

Yours,

Gerd

Re: Import to remote piler instance?

[Janos SUTO]

Hello Mark,

two options come to my mind.

a) export the mails containing directory using nfs or similar,
let the remote archive mount it, then run pilerimport on that dir

b) create tar or zip files (perhaps one per user) on the mail dirs,
copy them to the archive host, unpack, and import them. You may use
rsync or similar as well.

Janos

On 2024-02-08 15:46, Mark wrote:

Hi all,

I want to ingest historical mail. I would like to avoid logging in via
imap for each user and ingesting mail. Is there a way I can, from the
IMAP server, point at the /var/spool/mail dir and have it send the
mail for ingest to a remote piler server? Will this also work for
something like Thunderbird mail folders?

thanks

New gui testing

[Janos SUTO]

Dear piler users,

the new gui is more or less ready for a preview, however,
there might be some sharp edges left. Anyway, I'd like you
to test it, and provide feedback, eg. what doesn't work,
what needs to be changed, etc.

Janos

Re: cant connect to mysql server

[Janos SUTO]

Hello Mark,

what happens when piler receives an email? Show me the logs.

Janos

On 2023-10-15 18:01, Mark wrote:

Hi all,

I have upgraded my piler instance to 1.4.4. I tried the deb package
but it gave errors about spx_close symbol not being found so I
compiled it and upgraded that way.

I get these errors in my piler mail.log output.

Oct 15 17:54:17 mailpiler.abc.co.za [1] piler[3033]: cant connect to
mysql server
Oct 15 17:54:17 mailpiler.abc.co.za [1] piler[3033]: ERROR: cannot
open database

I can connect to the database from the command line. I have double
checked the entries in /usr/local/etc/piler/piler.conf and they are
correct and worked before. I am not sure how else to test the
connectivity from piler to the database. How can I proceed or does
anyone have some ideas to try?

Re: Version Upgrade/ Date Migration

[Janos SUTO]

On 2023-10-07 18:06, Andreas wrote:


i checked 2 messages, one with only one receipient, another one wit 13
receipient.
The data fits 100% to the downloaded mail.


If this means that the recipient in the email is in the rcpt to table,
then fine. Now grab a few words from the email, and search for them.
Again, nothing fancy, no wildcards, etc. just 2 to 3 words. Then
check the sphinx query from the mail log. You should see some hits.

Janos



- Ursprüngliche Mail -
| Von: "Janos SUTO" 
| An: "Piler User" 
| Gesendet: Samstag, 7. Oktober 2023 17:46:01
| Betreff: Re: Version Upgrade/ Date Migration

| OK, then click on a message, the hover the mouse over the download 
link

| on the lower preview pane. It reveals the message id, eg. 123. Then
| run the following sql query, and check if the from and:
|
| select id, piler_id, `from` from metadata where id=123;
| select id, `to` from rcpt where id=123;
|
| These queries should reveal the From: and To/Cc: header values
| from the email. Check if they actually match with the email.
| You may retrieve the email with the pilerget utility, or simply
| download it.
|
| Janos
|
| On 2023-10-07 17:32, Andreas wrote:
|> Hi Janos,
|>
|> no, I do not get any result.
|>
|>> Login as an auditor, hit the search button, and you should be able 
to

|>> see some results.
|> Yes; i see 1000 records.
|>
|>
|>> Then click on a message, observe the From: address,
|> | and type it to the search bar (no wildcard, nothing fancy, just 
the

|> | email address), and you should be able to see the same message in
|> | the search results.
|>
|> No results.
|> the query.log entry is:
|> /* Sat Oct 07 15:28:23.418 2023 conn 131 real 0.000 wall 0.000 found 
0

|> */ SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('@sender
|> jobXalertXhaysXde ') ORDER BY sent DESC;
|> The error message in the UI is:
|> Empty search result. Try adding the wildcard character(*) after a 
word

|> snippet (min. 5 characters), eg. duplic* to find "duplicate",
|> "duplicated", etc.
|>
|> Any idea?
|>
|> Andreas
|>
|>
|>
|> - Ursprüngliche Mail -
|> | Von: "Janos SUTO" 
|> | An: "Piler User" 
|> | CC: "Andreas" 
|> | Gesendet: Samstag, 7. Oktober 2023 17:19:50
|> | Betreff: Re: Version Upgrade/ Date Migration
|>
|> | On 2023-10-07 07:35, Andreas wrote:
|> |
|> |>
|> |> I can see in the file query.log the following queries:
|> |> /* Sat Oct 07 05:27:27.075 2023 conn 112 real 0.000 wall 0.000 
found

|> 0
|> |> */ SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('
|> |> apconsulting') ORDER BY sent DESC;
|> |>
|> |> /* Sat Oct 07 05:30:23.492 2023 conn 113 real 0.000 wall 0.000 
found

|> 0
|> |> */ SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('
|> |> *apconsulting*') ORDER BY sent DESC;
|> |>
|> |> with advanced search i find entries like
|> |> /* Sat Oct 07 05:31:27.427 2023 conn 114 real 0.000 wall 0.000 
found

|> 0
|> |> */ SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('@rcpt
|> |> apconsulting.de ') ORDER BY sent DESC;
|> |> /* Sat Oct 07 05:32:17.605 2023 conn 115 real 0.000 wall 0.000 
found

|> 0
|> |> */ SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('@subject
|> Test
|> |> ') ORDER BY sent DESC;
|> |>
|> |>
|> |> all of them above without query result.
|> |>
|> |> only these queries work:
|> |> /* Sat Oct 07 05:33:16.686 2023 conn 116 real 0.023 wall 0.024 
found

|> |> 519 */ SELECT id FROM main1,dailydelta1,delta1 WHERE sent BETWEEN
|> |> 1690840800 AND 1693519199 ORDER BY sent DESC;
|> |> /* Sat Oct 07 05:34:21.063 2023 conn 117 real 0.001 wall 0.001 
found

|> |> 537 */ SELECT id FROM main1,dailydelta1,delta1 WHERE
|> sent>=1690840800
|> |> ORDER BY sent DESC;
|> |
|> | Login as an auditor, hit the search button, and you should be able 
to
|> | see some results. Then click on a message, observe the From: 
address,
|> | and type it to the search bar (no wildcard, nothing fancy, just 
the

|> | email address), and you should be able to see the same message in
|> | the search results.
|> |
|> | Also every time you try a search query, then check the mail log 
for

|> the
|> | sphinx query and results.
|> |
|> | Janos
| > (null)
(null)

Re: Version Upgrade/ Date Migration

[Janos SUTO]

OK, then click on a message, the hover the mouse over the download link
on the lower preview pane. It reveals the message id, eg. 123. Then
run the following sql query, and check if the from and:

select id, piler_id, `from` from metadata where id=123;
select id, `to` from rcpt where id=123;

These queries should reveal the From: and To/Cc: header values
from the email. Check if they actually match with the email.
You may retrieve the email with the pilerget utility, or simply
download it.

Janos

On 2023-10-07 17:32, Andreas wrote:

Hi Janos,

no, I do not get any result.

Login as an auditor, hit the search button, and you should be able to 
see some results.

Yes; i see 1000 records.



Then click on a message, observe the From: address,

| and type it to the search bar (no wildcard, nothing fancy, just the
| email address), and you should be able to see the same message in
| the search results.

No results.
the query.log entry is:
/* Sat Oct 07 15:28:23.418 2023 conn 131 real 0.000 wall 0.000 found 0
*/ SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('@sender
jobXalertXhaysXde ') ORDER BY sent DESC;
The error message in the UI is:
Empty search result. Try adding the wildcard character(*) after a word
snippet (min. 5 characters), eg. duplic* to find "duplicate",
"duplicated", etc.

Any idea?

Andreas



- Ursprüngliche Mail -
| Von: "Janos SUTO" 
| An: "Piler User" 
| CC: "Andreas" 
| Gesendet: Samstag, 7. Oktober 2023 17:19:50
| Betreff: Re: Version Upgrade/ Date Migration

| On 2023-10-07 07:35, Andreas wrote:
|
|>
|> I can see in the file query.log the following queries:
|> /* Sat Oct 07 05:27:27.075 2023 conn 112 real 0.000 wall 0.000 found 
0

|> */ SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('
|> apconsulting') ORDER BY sent DESC;
|>
|> /* Sat Oct 07 05:30:23.492 2023 conn 113 real 0.000 wall 0.000 found 
0

|> */ SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('
|> *apconsulting*') ORDER BY sent DESC;
|>
|> with advanced search i find entries like
|> /* Sat Oct 07 05:31:27.427 2023 conn 114 real 0.000 wall 0.000 found 
0

|> */ SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('@rcpt
|> apconsulting.de ') ORDER BY sent DESC;
|> /* Sat Oct 07 05:32:17.605 2023 conn 115 real 0.000 wall 0.000 found 
0
|> */ SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('@subject 
Test

|> ') ORDER BY sent DESC;
|>
|>
|> all of them above without query result.
|>
|> only these queries work:
|> /* Sat Oct 07 05:33:16.686 2023 conn 116 real 0.023 wall 0.024 found
|> 519 */ SELECT id FROM main1,dailydelta1,delta1 WHERE sent BETWEEN
|> 1690840800 AND 1693519199 ORDER BY sent DESC;
|> /* Sat Oct 07 05:34:21.063 2023 conn 117 real 0.001 wall 0.001 found
|> 537 */ SELECT id FROM main1,dailydelta1,delta1 WHERE 
sent>=1690840800

|> ORDER BY sent DESC;
|
| Login as an auditor, hit the search button, and you should be able to
| see some results. Then click on a message, observe the From: address,
| and type it to the search bar (no wildcard, nothing fancy, just the
| email address), and you should be able to see the same message in
| the search results.
|
| Also every time you try a search query, then check the mail log for 
the

| sphinx query and results.
|
| Janos
(null)

Re: Version Upgrade/ Date Migration

[Janos SUTO]

On 2023-10-07 07:35, Andreas wrote:



I can see in the file query.log the following queries:
/* Sat Oct 07 05:27:27.075 2023 conn 112 real 0.000 wall 0.000 found 0
*/ SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('
apconsulting') ORDER BY sent DESC;

/* Sat Oct 07 05:30:23.492 2023 conn 113 real 0.000 wall 0.000 found 0
*/ SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('
*apconsulting*') ORDER BY sent DESC;

with advanced search i find entries like
/* Sat Oct 07 05:31:27.427 2023 conn 114 real 0.000 wall 0.000 found 0
*/ SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('@rcpt
apconsulting.de ') ORDER BY sent DESC;
/* Sat Oct 07 05:32:17.605 2023 conn 115 real 0.000 wall 0.000 found 0
*/ SELECT id FROM main1,dailydelta1,delta1 WHERE MATCH('@subject Test
') ORDER BY sent DESC;


all of them above without query result.

only these queries work:
/* Sat Oct 07 05:33:16.686 2023 conn 116 real 0.023 wall 0.024 found
519 */ SELECT id FROM main1,dailydelta1,delta1 WHERE sent BETWEEN
1690840800 AND 1693519199 ORDER BY sent DESC;
/* Sat Oct 07 05:34:21.063 2023 conn 117 real 0.001 wall 0.001 found
537 */ SELECT id FROM main1,dailydelta1,delta1 WHERE sent>=1690840800
ORDER BY sent DESC;


Login as an auditor, hit the search button, and you should be able to
see some results. Then click on a message, observe the From: address,
and type it to the search bar (no wildcard, nothing fancy, just the
email address), and you should be able to see the same message in
the search results.

Also every time you try a search query, then check the mail log for the
sphinx query and results.

Janos

Re: Data discrepancy in piler database?

[Janos SUTO]

On 2023-10-07 07:05, Andreas wrote:



Is there anything I can do to track down the root cause?
Or would you just ignore the difference?

As I assume that the data in table counter is wrong: is there any way
to correct the data there?


I'd say, fix it in the counter table in the mysql piler database.

Janos

Re: Data discrepancy in piler database?

[Janos SUTO]

To me 107595+4997 are close to 111k. Anyway, perhaps there
were some errors during the processing, and might have counted
as a received message. Not sure.

Janos

On 2023-10-05 08:25, Andreas wrote:

Hi,

I received a database dump, all files from /var/piler/store and some
more stuff from my provider and migrated that into my docker instance.

Now i carfeully check the data I have received and i see a discrepance
i do not understand. The discrepancy is based only on the data I
received (I did not archive any additional mails).

As administrator in the UI i can see
* received messages  111,114
* duplicate messages 4,997
this is exaclty what is stored in the table counter

In the auditors ui i can see 107595 messages.
This is also the amount of messages i can count from table rcpt (based
on id) and metadata

Where can the discrepancy come from? Sorry, i really have no idea
where to start analysis.
Thank you for your help in advance

Andreas

Re: Version Upgrade/ Date Migration

[Janos SUTO]

Hello Andreas,

On 2023-10-04 19:35, Andreas Pampel wrote:


I then restored the database I received from my old provder and
migrated this to what I thought is the newest version:
* as described here https://www.mailpiler.org/wiki/current:upgrade and
after that
* as described here https://bitbucket.org/jsuto/piler/src/master/util/
 db-upgrade.sql [1]
All statements were processed without errors

When I compare the structure there are small differences. See attached
files:
* piler_apconsulting.sql is the migrated database
* piler.sql is the database that was installed with the docker

Do I need to run additional sql statements?
If yes: Where can I find them?


I've just run a quick diff against the 2 files, and it turns out 3 
tables

are missing from the old structure, eg. deleted, `group`, and usergroup.
Also the v_messages view seems to be different, be sure to drop it, and
recreate it. See /usr/share/piler/db-util.sql for the database schema.


In the following i restored all files in /var/piler/store and the keys
and adpoted the config files.
While reindexing i got several identical error messages "Failed to
inflate invalid or incomplete deflate data".
I am not sure if the db differences are the root cause for this. If
not: what do I need to do?


Not sure. Can you retrieve any message properly? Perhaps some without
attachments. If not, then verify that piler.key and piler.conf iv
parameters match.



When i check the ui with the auditor user, i see e-mails and they are
validated (good!). I can filter them on date, but I can not filter
them using string nor using the advanced search (except the dates).


Enter some subject words, then hit the red search button, and check
the logged sphinx query. Also try to login as an auditor and do the
same search. What happens?

Janos

Re: pilerpurge.py and manticore issue

[Janos SUTO]

On 2023-09-28 12:03, d tbsky wrote:

 Thanks for the hint. I tried it at the testing environment and it 
works.
but when there are no metadata table empty records (eg pilerpuge.py not 
run),

indexer.delta.sh will return another error:

WARNING: killlist_target is specified, but kill-list is empty


it's not an error, it's simply a warning that you have no rows in the 
metadata

table where deleted=1, so there's no record to suppress in the results.

Janos

Re: pilerpurge.py and manticore issue

[Janos SUTO]

Thank you, Frank. I've updated the master branch.

Janos

On 2023-09-27 09:08, Frank Schmirler wrote:

Hi,
Am Mittwoch, 27. September 2023 05:30 CEST, schrieb d tbsky 
:


> The kill-list should work with manticore, too.

but I don't know how to get rid of the error message "kill-list not
empty but no killlist_target specified" without dropping the empty
records. I think there are some reasons that pilerpurge.py leaves
these empty records?


I had to add the following line to the "index delta1" section:
killlist_target = main1:kl, main2:kl, main3:kl, main4:kl, 
dailydelta1:kl


Regards,
Frank

Re: pilerpurge.py and manticore issue

[Janos SUTO]

Hello,

On 2023-09-23 05:29, d tbsky wrote:

Hi:
I tried to reindex everything at the testing environment. but the
error message is the same.
I browse the mariadb data, and found that the purge mails seems
only left one empty record for each mail at the metadata table. I
don't know if this is true or not. I delete the empty record and the


It's correct. The pilerpurge.py script updates the metadata table with
empty values for the deleted emails, and removes connected entries in
the rcpt, tag, note, etc. tables.


error message disappear. but I don't know if this is the correct thing
to do.


The kill-list should work with manticore, too.


I also think about the manticore new "RT" function.  I don't care
the delay of archived mails show at the search GUI. I want a reliable
system, when piler accept a mail, I hope the mail can be archived
safely. if something goes wrong like disk-full, mysq/manticore service
down, power failure..etc, piler won't accept the mail or can fix the
problem later(maybe via crontab works?). I don't know if using "RT"
would lower the archive quality. is "RT" the preferred method in the
future?


Piler requires several external services to work. It stores crucial 
metadata,
etc. in mysql. If the mysql database is not available, then piler can't 
archive
new emails, and can't retrieve anything having an attachment. Manticore 
(or
sphinx) is also crucial as it indexes the archived data. If piler can't 
process
the email that piler-smtp has written to disk, eg. no mysql, no 
manticore, disk

full or similar, then piler moves the email to /var/piler/error dir for
later investigation.

However, you are not forced to switch to RT index, feel free to use
the usual main-delta-deltadelta indexing scheme. I personally believe
that an RT index has some benefits, eg. new emails are immediately
visible on the gui, no need to run indexer periodically. No need to
watch if the index data files grow too large causing searchd a headache.
Also less i/o requirements when merging the dailydelta to the main 
index.

The merge requires the indexer utility to rewrite the main index.

Janos

New gui is coming

[Janos SUTO]

Dear piler-users,

the current gui is nice, but quite outdated by now. It's based on
bootstrap 3.4 with jquery. So recently I've been working on a new gui.

So far the search page is ready to show, which most users interact with.
I've deployed it to the demo site. Visit https://demo.mailpiler.com
(you'll find the login details there), and after logging in, visit
https://demo.mailpiler.com/x for the new gui.

Main features:
- bootstrap 5.3 + vue.js 3
- mobile friendly layout (it looks fine even on my phone)
- support for dark mode - based on your browser's preferences
  (be sure to try both the dark and light themes)
- new layout: the main area is divided to left + right
  giving more like a webmail feeling, instead of the previous
  upper + lower layout
- no more paging links: when you scroll down, the gui detects
  if you reached the bottom, and automatically loads the next
  page data as well, appending them to the current list
- no more left side checkboxes: there's a toggle icon in the
  upper left corner to select/unselect all messages
- shift click selects/unselects individual messages

A few details still may need some polishing, but it's what I
have in mind for the next major release, so feel free to comment,
provide feedback or even fixes :-)

Janos

Re: SQL ERROR

[Janos SUTO]

Hello Francesco,

it usually indicates a sphinx (or manticore) issue. Check if
the search engine is running fine.

Janos SUTO

On 2023-08-21 15:18, Francesco Barcella wrote:

Hi, everyone
I cannot reach web app anymore, I get this error.

Error: SQLSTATE[HY000] [2002] No such file or directory on database:
piler

What can I try ?
Thank you

Francesco

Updating the piler gui

[Janos SUTO]

Dear piler users,

I'd like to get your opinion and perhaps even your help with
updating the piler gui. Though I believe that it has a decent
look, it's outdated by now. It's based on an old bootstrap and
a metro-styled css, so it's high time to replace it with something
more modern solution.

The idea is to rebuild the gui using a modern and current technology.
I've just looked at vite, vue.js and some others, but my talent is
anything but frontend related stuff :-)

So I thought I'd ask you to suggest a viable solution to replace
the current gui. If you could help even with developing it, that's
even better ;-)

Some requirements:
- responsive design with mobile support
- splitter funcionality
- a reasonable security posture of the chosen solution

Janos

Re: Antwort: Re: SSO and LDAP on mailpiler docker

[Janos SUTO]

Hello Patrick,

check out the docs: 
https://www.mailpiler.org/wiki/current:ldap-authentication


Janos

On 2023-06-30 17:02, patrick.we...@rgi.net wrote:

Hello Janos,

thank you for your help.
I think i have changed my priority. The first thing I have to make
sure is, that LDAP is working with the docker.
SSO comes second. I am still not sure how I can do it with my current
configuration (nginx, docker piler 1.4.4).

- Patrick

Von:"Janos SUTO" 
An:"Piler User" 
Datum:28.06.2023 16:14
Betreff:Re: SSO and LDAP on mailpiler docker

-

Hello Patrick,

the docker image features nginx, which is an excellent web server,
however, its sso support is not that great. So if you want sso against
AD, then your best bet is to build a new docker image with apache, and
apply the hints Alex suggested.

Janos

On 2023-06-28 12:31, patrick.we...@rgi.net wrote:

Hello Janos!

At the moment i am trying to configure my mailpiler docker solution

to

use SSO.
I am using your instruction on
https://www.mailpiler.org/wiki/current:single-sign-on [1] [1], but I

am

not sure
if this will work for the docker solution.
My server contains of 3 dockers (piler, mysql, memcached). The piler
docker runs the nginx webserver, the piler (version 1.4.4) and
manticore.
Samba and winbind is not installed. My best guest is, that i have to
install samba and winbind in the piler docker instead of installing

it

on the host. Am I right?

- Patrick





Links:
--
[1] https://www.mailpiler.org/wiki/current:single-sign-on

Re: SSO and LDAP on mailpiler docker

[Janos SUTO]

Hello Patrick,

the docker image features nginx, which is an excellent web server,
however, its sso support is not that great. So if you want sso against
AD, then your best bet is to build a new docker image with apache, and
apply the hints Alex suggested.

Janos

On 2023-06-28 12:31, patrick.we...@rgi.net wrote:

Hello Janos!

At the moment i am trying to configure my mailpiler docker solution to
use SSO.
I am using your instruction on
https://www.mailpiler.org/wiki/current:single-sign-on [1], but I am
not sure
if this will work for the docker solution.
My server contains of 3 dockers (piler, mysql, memcached). The piler
docker runs the nginx webserver, the piler (version 1.4.4) and
manticore.
Samba and winbind is not installed. My best guest is, that i have to
install samba and winbind in the piler docker instead of installing it
on the host. Am I right?

- Patrick

Re: Antwort: Re: Can not migrate to a new host

[Janos SUTO]

Hello Patrick,

I'm glad that you made it. Can you describe what you changed
in the openssl configuration? I'd add it to the FAQ.

Janos

On 2023-06-19 08:49, patrick.we...@rgi.net wrote:

Hello Janos,

I finally found the solution. The problem lies in the openssl
configuration.
The Legacy Provider was not configured in the new openssl
configuration, which results in the Mailpiler not been able to decrypt
the old mails.
Thank your very much for your support!

-Patrick

Von:"Janos SUTO" 
An:"Piler User" 
Datum:07.06.2023 19:36
Betreff:Re: Can not migrate to a new host

-

Hello Patrick,

it's indeed an odd issue. Check the following:

- you can connect to the mysql piler database, can see and read from
the
tables
- /var/piler/{store,tmp} dirs and contents are owned by piler
- copy an eml file to the new host, and verify that pilertest can
process it fine

Janos

On 2023-05-31 08:39, patrick.we...@rgi.net wrote:

Hello!

I was trying to migrate my old piler 1.2.0 to a new host with piler
1.4.2

1. Specs of the old host:
- Piler Version 1.2 build 952
-  Debian 9

2. Specs of the new host:
- Piler Version 1.4.2 build 1001
- Ubuntu 22.04

3. Installed Piler 1.4.2 on new host
4. Disabled all Cronjobs on new
5. Stopped Piler Service on old: systemctl stop rc.piler.service
6. Stopped Searchd on old: systemctl stop rc.searchd.service
7. Stopped mysql on old
8. Copied the piler.key, the old host does not have an iv configured
8. Copied the mysql data from /var/lib/mysql/*
9. Created a tar archive of the sphinx files at /var/piler/sphinx:

tar

-czvf sphinx.tar.gz ./
10. Uploaded the MySQL-files and sphinx files to the new hosts
11. Started piler, searchd and mysql on new
12. Uploaded /var/piler/store/* to the new host
13. Enabled the cronjobs on new
14. Started a reindex (reindex -a) -> Segmentation fault
15. Tried to read a mail with pilerget -> Segmentation fault

I also have tried many more solutions in the past.
The first solutions was to upgrade the piler version of the old

piler

to 1.4.2 and then copy the files in /var/piler/sphinx/*,
/var/piler/store/* and /var/lib/mysql/*
to the new piler. I used the migration documentation on
https://www.mailpiler.org/wiki/current:migration-to-new-host [1]

[1].

After i have imported all files i tried to run a reindex. The result
was a segmentation fault error.
The second solution was similar to the first. The only difference is
that i made a dump of the sql-database and read it into the piler
database. But the result was the same (segmentation fault).

Another thing i have realized while upgrading the piler is, that the
documentation on the schema update for the mysql database on the
website (https://www.mailpiler.org/wiki/current:upgrade) differs

from

the
upgrades in the installation files.

-Patrick





Links:
--
[1] https://www.mailpiler.org/wiki/current:migration-to-new-host

Re: Admin password

[Janos SUTO]

Then either restore it from backup, or simply recreate the table,
and add the admin and auditor users. See util/db-mysql.sql for its 
schema.


Janos

On 2023-06-13 19:32, Marcelo Machado wrote:

Thanks for the answer.
when I use the command that is in the FAQ this error appears: ERROR
1146 (42S02): Table 'piler.user' doesn't exist

Em ter., 13 de jun. de 2023 às 14:17, Janos SUTO  escreveu:




Hello,

yes, please read the FAQ.

Janos

On 2023-06-13 19:02, Marcelo Machado wrote:
> Hello everyone.
>
> Is it possible to reset the admin@local user password to the original
> one?
>
> Thanks in advance
>
> Marcelo Gomes

Re: Admin password

[Janos SUTO]

Hello,

yes, please read the FAQ.

Janos

On 2023-06-13 19:02, Marcelo Machado wrote:

Hello everyone.

Is it possible to reset the admin@local user password to the original 
one?


Thanks in advance

Marcelo Gomes

Re: Can not migrate to a new host

[Janos SUTO]

Hello Patrick,

it's indeed an odd issue. Check the following:

- you can connect to the mysql piler database, can see and read from the 
tables

- /var/piler/{store,tmp} dirs and contents are owned by piler
- copy an eml file to the new host, and verify that pilertest can 
process it fine


Janos

On 2023-05-31 08:39, patrick.we...@rgi.net wrote:

Hello!

I was trying to migrate my old piler 1.2.0 to a new host with piler
1.4.2

1. Specs of the old host:
- Piler Version 1.2 build 952
-  Debian 9

2. Specs of the new host:
- Piler Version 1.4.2 build 1001
- Ubuntu 22.04

3. Installed Piler 1.4.2 on new host
4. Disabled all Cronjobs on new
5. Stopped Piler Service on old: systemctl stop rc.piler.service
6. Stopped Searchd on old: systemctl stop rc.searchd.service
7. Stopped mysql on old
8. Copied the piler.key, the old host does not have an iv configured
8. Copied the mysql data from /var/lib/mysql/*
9. Created a tar archive of the sphinx files at /var/piler/sphinx: tar
-czvf sphinx.tar.gz ./
10. Uploaded the MySQL-files and sphinx files to the new hosts
11. Started piler, searchd and mysql on new
12. Uploaded /var/piler/store/* to the new host
13. Enabled the cronjobs on new
14. Started a reindex (reindex -a) -> Segmentation fault
15. Tried to read a mail with pilerget -> Segmentation fault

I also have tried many more solutions in the past.
The first solutions was to upgrade the piler version of the old piler
to 1.4.2 and then copy the files in /var/piler/sphinx/*,
/var/piler/store/* and /var/lib/mysql/*
to the new piler. I used the migration documentation on
https://www.mailpiler.org/wiki/current:migration-to-new-host [1].
After i have imported all files i tried to run a reindex. The result
was a segmentation fault error.
The second solution was similar to the first. The only difference is
that i made a dump of the sql-database and read it into the piler
database. But the result was the same (segmentation fault).

Another thing i have realized while upgrading the piler is, that the
documentation on the schema update for the mysql database on the
website (https://www.mailpiler.org/wiki/current:upgrade) differs from
the
upgrades in the installation files.

-Patrick

How to switch from sphinx delta-delta - delta - main scheme to manticore realtime index

[Janos SUTO]

Dear piler users,

starting with version 1.4.x both manticore and real time index are 
supported.

Recently, I got a request to describe the procudere of such transition:
https://docs.mailpiler.com/piler-ee/migrate-sphinx-to-manticore/

Even though the docs has been created for the enterprise version, it 
also applies
to the open source version. The bad news is that it requires a full 
reindexing

of the archive.

I'd like to emphasize that sphinx still is and will be supported, so 
switching from
sphinx to manticore is just an option, not a requirement to keep using 
piler. And if
you want to switch to real time indexing and say goodbye to the periodic 
indexer

cron jobs, you may achieve it by using either sphinx or manticore.

Btw. sphinx 3.5.1 has been released in this February, almost 2 years 
after 3.4.1.


Janos

Re: pilerimport stopped saving files?

[Janos SUTO]

On 2023-05-05 18:18, Jeremiah Poling wrote:


Is there a way to reimport the failed ones?

Maybe somehow delete a date range of mails from the system and then
reimport that range?

I still have access to the source eml files from the date range in
question, but trying to import them now just results in a bunch of
“duplicate: XXX” messages in the console


that's a tricky one. As I mentioned, piler discard these emails, because
they already exist in the metadata table. You have 2 options:

a) the "cannot open()" issue indicated that the given file is not 
present
in the store directory structure. So option #1 is to get your backup, 
and

restore the missing files

b) if you don't have backups, then try removing the affected emails from
both the metadata and rcpt tables. Also you have the attachment table
which stores info to reassemble emails with attachments. Since an 
attachment
may be present in several emails, I'm not sure if you should remove any 
rows

from the attachment table.

Anyway, be sure to backup the piler mysql database.

Janos




From: Janos SUTO 
Sent: Thursday, May 4, 2023 23:03
To: Piler User 
Subject: Re: pilerimport stopped saving files?

[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you
recognize the sender and know the content is safe.

Hello Jeremiah,

if you have newer emails to import, then run pilerimport from the
command line, and look for errors both on stdin and in the mail logs.
If no errors then try to retrieve those emails with pilerget.

Janos SUTO

On May 4, 2023, at 22:59, Jeremiah Poling  wrote:


Janos,

Sorry to pester you again, but I've run into an unusual issue that
I'm struggling to troubleshoot.

I'm using pilerimport to load eml files that have been fetched using
offlineimap.
This was working fine until February 22 of this year.
I don't remember having done anything that day, but I assume I must
have run an update or something that broke it.

Since then, the imported mails are being loaded in to the mysql
database and sphinx (they show up in the web interface) but they're
not being saved to the filesystem.
so they can't be found by the web interface to preview: " Message
Failed Verification"
and they can't be loaded by pilerget: pilerget[3417]:


/var/piler/store/00/645/a4/cb/500064540b9d1979a504001a4529a4cb.m:

cannot open()

any thoughts on how I can troubleshoot this?

Thanks,
Jeremiah

Re: pilerimport stopped saving files?

[Janos SUTO]

Hello Jeremiah,

if you have newer emails to import, then run pilerimport from the command line, 
and look for errors both on stdin and in the mail logs. If no errors then try 
to retrieve those emails with pilerget.

Janos SUTO


On May 4, 2023, 22:59, at 22:59, Jeremiah Poling  wrote:
>
>
>Janos,
>
>Sorry to pester you again, but I've run into an unusual issue that I'm
>struggling to troubleshoot.
>
>I'm using pilerimport to load eml files that have been fetched using
>offlineimap.
>This was working fine until February 22 of this year.
>I don't remember having done anything that day, but I assume I must
>have run an update or something that broke it.
>
>Since then, the imported mails are being loaded in to the mysql
>database and sphinx (they show up in the web interface) but they're not
>being saved to the filesystem.
>so they can't be found by the web interface to preview: " Message
>Failed Verification"
>and they can't be loaded by pilerget: pilerget[3417]:
>/var/piler/store/00/645/a4/cb/500064540b9d1979a504001a4529a4cb.m:
>cannot open()
>
>any thoughts on how I can troubleshoot this?
>
>Thanks,
>Jeremiah

Re: failed to store index data

[Janos SUTO]

Yes, please. Reindex works just as before.

Janos

On Mar 19, 2023, 20:47, at 20:47, Jonathan Baecker  wrote:
>
>
>Thank you, that was the trick! But I still have to run: reindex -a?
>
>Am 19.03.23 um 20:44 schrieb Janos SUTO:
>>
>>
>> From your previous emails, you set real-time (rt) indexing. For such
>> setup,
>> don't run the indexer utility, no need for it any longer.
>>
>> Also be sure to update the pilersearch.service as in
>>
>https://bitbucket.org/jsuto/piler/src/master/systemd/pilersearch.service
>>
>> I think you'll be fine after that, manticore will create the required
>> index files.
>>
>> Janos
>>
>> On 2023-03-19 20:36, Jonathan Baecker wrote:
>>> This folder is empty and I also can not reindex. I recover my system
>>> to a state where it was works. Maybe I miss a step, so here are my
>>> steps:
>>>
>>> 1. piler 1.4.3 runs with sphinx under debian 11.
>>> 2. I install manticore 6.0.x from webpage
>>> 3. cp /usr/local/etc/piler/manticore.conf.dist
>>> /usr/local/etc/piler/manticore.conf
>>> 4. add db settings to /usr/local/etc/piler/manticore.conf
>>> 4. apply all changes from mailpiler doc about switching to manticore
>>> 5. when I run as piler user: indexer --all --config
>>> /usr/local/etc/piler/manticore.conf it gives me: FATAL: no tables
>>> found in config file '/usr/local/etc/piler/manticore.conf'
>>> 6. I reboot the system to have all services running
>>>
>>> systemctl status pilersearch.service show:
>>>
>>>     Process: 611 ExecStart=/usr/bin/searchd --config
>>> /usr/local/etc/piler/sphinx.conf (code=exited, status=0/SUCCESS)
>>>    Main PID: 615 (searchd)
>>>   Tasks: 10 (limit: 4675)
>>>  Memory: 52.2M
>>>     CPU: 314ms
>>>  CGroup: /system.slice/pilersearch.service
>>>  ├─614 /usr/bin/searchd --config
>>> /usr/local/etc/piler/sphinx.conf
>>>  ├─615 /usr/bin/searchd --config
>>> /usr/local/etc/piler/sphinx.conf
>>>  └─620 /usr/bin/manticore-executor -n
>>> /tmp/manticore-buddy.phar --listen=http://127.0.0.1:9312 --threads=2
>>>
>>> Mär 19 20:27:40 piler searchd[611]: Copyright (c) 2008-2016, Sphinx
>>> Technologies Inc (http://sphinxsearch.com)
>>> Mär 19 20:27:40 piler searchd[611]: Copyright (c) 2017-2023,
>Manticore
>>> Software LTD (https://manticoresearch.com)
>>> Mär 19 20:27:40 piler searchd[611]: WARNING: key 'docinfo' was
>>> permanently removed from configuration. Refer to documentation for
>>> details.
>>>
>>> So it still uses the old sphinx.conf
>>>
>>> Was my steps ok?
>>>
>>>
>>>
>>> Am 19.03.23 um 18:48 schrieb Janos SUTO:
>>>>
>>>>
>>>> Your configs look fine. The errno=1064 means that no such local
>index.
>>>> Show me "ls -la /var/piler/manticore/" output, too.
>>>>
>>>> In my test environment it looks like:
>>>>
>>>> ls -la /var/piler/manticore/
>>>> total 4796
>>>> drwx--  2 piler piler    4096 Mar 19 17:43 .
>>>> drwxr-xr-x 10 root  root 4096 Mar 19 17:01 ..
>>>> -rw---  1 piler piler   8 Mar 19 17:43 binlog.001
>>>> -rw---  1 piler piler   0 Mar 19 17:03 binlog.lock
>>>> -rw---  1 piler piler  11 Mar 19 17:43 binlog.meta
>>>> -rw---  1 piler piler    2731 Mar 19 17:43 manticore.log
>>>> -rw---  1 piler piler   0 Mar 19 17:03 note1.lock
>>>> -rw---  1 piler piler    1200 Mar 19 17:03 note1.meta
>>>> -rw---  1 piler piler  20 Mar 19 17:03 note1.ram
>>>> -rw---  1 piler piler   0 Mar 19 17:03 piler1.lock
>>>> -rw---  1 piler piler    1569 Mar 19 17:43 piler1.meta
>>>> -rw---  1 piler piler 4862221 Mar 19 17:43 piler1.ram
>>>> -rw---  1 piler piler 893 Mar 19 17:01 query.log
>>>> -rw---  1 piler piler   0 Mar 19 17:03 tag1.lock
>>>> -rw---  1 piler piler    1199 Mar 19 17:03 tag1.meta
>>>> -rw---  1 piler piler  20 Mar 19 17:03 tag1.ram
>>>>
>>>> Janos
>>>>
>>>> On 2023-03-19 17:15, Jonathan Baecker wrote:
>>>>> Ah sorry! Here is the output, but I run:
>>>>> /usr/local/etc/piler/manticore.conf
>>>>>
>>>>> #
>>>>> # minimal manticore configuration suited to pile

Re: failed to store index data

[Janos SUTO]

From your previous emails, you set real-time (rt) indexing. For such 
setup,

don't run the indexer utility, no need for it any longer.

Also be sure to update the pilersearch.service as in
https://bitbucket.org/jsuto/piler/src/master/systemd/pilersearch.service

I think you'll be fine after that, manticore will create the required
index files.

Janos

On 2023-03-19 20:36, Jonathan Baecker wrote:

This folder is empty and I also can not reindex. I recover my system
to a state where it was works. Maybe I miss a step, so here are my
steps:

1. piler 1.4.3 runs with sphinx under debian 11.
2. I install manticore 6.0.x from webpage
3. cp /usr/local/etc/piler/manticore.conf.dist
/usr/local/etc/piler/manticore.conf
4. add db settings to /usr/local/etc/piler/manticore.conf
4. apply all changes from mailpiler doc about switching to manticore
5. when I run as piler user: indexer --all --config
/usr/local/etc/piler/manticore.conf it gives me: FATAL: no tables
found in config file '/usr/local/etc/piler/manticore.conf'
6. I reboot the system to have all services running

systemctl status pilersearch.service show:

    Process: 611 ExecStart=/usr/bin/searchd --config
/usr/local/etc/piler/sphinx.conf (code=exited, status=0/SUCCESS)
   Main PID: 615 (searchd)
  Tasks: 10 (limit: 4675)
 Memory: 52.2M
    CPU: 314ms
 CGroup: /system.slice/pilersearch.service
 ├─614 /usr/bin/searchd --config 
/usr/local/etc/piler/sphinx.conf
 ├─615 /usr/bin/searchd --config 
/usr/local/etc/piler/sphinx.conf

 └─620 /usr/bin/manticore-executor -n
/tmp/manticore-buddy.phar --listen=http://127.0.0.1:9312 --threads=2

Mär 19 20:27:40 piler searchd[611]: Copyright (c) 2008-2016, Sphinx
Technologies Inc (http://sphinxsearch.com)
Mär 19 20:27:40 piler searchd[611]: Copyright (c) 2017-2023, Manticore
Software LTD (https://manticoresearch.com)
Mär 19 20:27:40 piler searchd[611]: WARNING: key 'docinfo' was
permanently removed from configuration. Refer to documentation for
details.

So it still uses the old sphinx.conf

Was my steps ok?



Am 19.03.23 um 18:48 schrieb Janos SUTO:



Your configs look fine. The errno=1064 means that no such local index.
Show me "ls -la /var/piler/manticore/" output, too.

In my test environment it looks like:

ls -la /var/piler/manticore/
total 4796
drwx--  2 piler piler    4096 Mar 19 17:43 .
drwxr-xr-x 10 root  root 4096 Mar 19 17:01 ..
-rw---  1 piler piler   8 Mar 19 17:43 binlog.001
-rw---  1 piler piler   0 Mar 19 17:03 binlog.lock
-rw---  1 piler piler  11 Mar 19 17:43 binlog.meta
-rw---  1 piler piler    2731 Mar 19 17:43 manticore.log
-rw---  1 piler piler   0 Mar 19 17:03 note1.lock
-rw---  1 piler piler    1200 Mar 19 17:03 note1.meta
-rw---  1 piler piler  20 Mar 19 17:03 note1.ram
-rw---  1 piler piler   0 Mar 19 17:03 piler1.lock
-rw---  1 piler piler    1569 Mar 19 17:43 piler1.meta
-rw---  1 piler piler 4862221 Mar 19 17:43 piler1.ram
-rw---  1 piler piler 893 Mar 19 17:01 query.log
-rw---  1 piler piler   0 Mar 19 17:03 tag1.lock
-rw---  1 piler piler    1199 Mar 19 17:03 tag1.meta
-rw---  1 piler piler  20 Mar 19 17:03 tag1.ram

Janos

On 2023-03-19 17:15, Jonathan Baecker wrote:
Ah sorry! Here is the output, but I run: 
/usr/local/etc/piler/manticore.conf


#
# minimal manticore configuration suited to piler
#


index piler1
{
    type = rt
    path = /var/piler/manticore/piler1
    rt_mem_limit = 512M
    stored_fields =
    min_word_len = 1
    min_prefix_len = 5
    charset_table  = 0..9, english, _, \
  U+C1->U+E1, U+C4->U+E4, U+C5->U+E5,
U+C6->U+E6, U+C9->U+E9, U+CD->U+ED, U+D3->U+F3, U+D6->U+F6,
U+D8->U+F8, \
  U+DA->U+FA, U+DC->U+FC,
U+0150->U+0151, U+0152->U+0153, U+0170->U+0171, U+01E2->U+E6,
U+01E3->U+E6, U+01FC->U+E6, \
  U+01FD->U+E6, U+1D01->U+E6,
U+1D02->U+E6, U+1D2D->U+E6, U+1D46->U+E6, \
  U+DF, U+E1, U+E4, U+E5, U+E6, U+E9,
U+ED, U+00F3, U+F6, U+F8, U+FA, U+FC, U+0151, U+0153, U+0171
    # See
https://manual.manticoresearch.com/Creating_an_index/Data_types#Row-wise-and-columnar-attribute-storages 
    # if you want to enable columnar storage

    # columnar_attrs = *
    rt_field = sender
    rt_field = rcpt
    rt_field = senderdomain
    rt_field = rcptdomain
    rt_field = subject
    rt_field = body
    rt_field = attachment_types
    rt_attr_bigint = arrived
    rt_attr_bigint = sent
    rt_attr_uint = size
    rt_attr_uint = direction
    rt_attr_uint = folder
    rt_attr_uint = attachments
}

index tag1
{
    type = rt
    path = /var/piler/manticore/tag1
    rt_mem_limit = 16M
    stored_fields = tag
    min_word_len = 2
    min_prefix_len = 5
    charset_table  = 0..9, english, _, \
  U+C

Re: failed to store index data

[Janos SUTO]

Your configs look fine. The errno=1064 means that no such local index.
Show me "ls -la /var/piler/manticore/" output, too.

In my test environment it looks like:

ls -la /var/piler/manticore/
total 4796
drwx--  2 piler piler4096 Mar 19 17:43 .
drwxr-xr-x 10 root  root 4096 Mar 19 17:01 ..
-rw---  1 piler piler   8 Mar 19 17:43 binlog.001
-rw---  1 piler piler   0 Mar 19 17:03 binlog.lock
-rw---  1 piler piler  11 Mar 19 17:43 binlog.meta
-rw---  1 piler piler2731 Mar 19 17:43 manticore.log
-rw---  1 piler piler   0 Mar 19 17:03 note1.lock
-rw---  1 piler piler1200 Mar 19 17:03 note1.meta
-rw---  1 piler piler  20 Mar 19 17:03 note1.ram
-rw---  1 piler piler   0 Mar 19 17:03 piler1.lock
-rw---  1 piler piler1569 Mar 19 17:43 piler1.meta
-rw---  1 piler piler 4862221 Mar 19 17:43 piler1.ram
-rw---  1 piler piler 893 Mar 19 17:01 query.log
-rw---  1 piler piler   0 Mar 19 17:03 tag1.lock
-rw---  1 piler piler1199 Mar 19 17:03 tag1.meta
-rw---  1 piler piler  20 Mar 19 17:03 tag1.ram

Janos

On 2023-03-19 17:15, Jonathan Baecker wrote:
Ah sorry! Here is the output, but I run: 
/usr/local/etc/piler/manticore.conf


#
# minimal manticore configuration suited to piler
#


index piler1
{
    type = rt
    path = /var/piler/manticore/piler1
    rt_mem_limit = 512M
    stored_fields =
    min_word_len = 1
    min_prefix_len = 5
    charset_table  = 0..9, english, _, \
  U+C1->U+E1, U+C4->U+E4, U+C5->U+E5,
U+C6->U+E6, U+C9->U+E9, U+CD->U+ED, U+D3->U+F3, U+D6->U+F6,
U+D8->U+F8, \
  U+DA->U+FA, U+DC->U+FC,
U+0150->U+0151, U+0152->U+0153, U+0170->U+0171, U+01E2->U+E6,
U+01E3->U+E6, U+01FC->U+E6, \
  U+01FD->U+E6, U+1D01->U+E6,
U+1D02->U+E6, U+1D2D->U+E6, U+1D46->U+E6, \
  U+DF, U+E1, U+E4, U+E5, U+E6, U+E9,
U+ED, U+00F3, U+F6, U+F8, U+FA, U+FC, U+0151, U+0153, U+0171
    # See
https://manual.manticoresearch.com/Creating_an_index/Data_types#Row-wise-and-columnar-attribute-storages
    # if you want to enable columnar storage
    # columnar_attrs = *
    rt_field = sender
    rt_field = rcpt
    rt_field = senderdomain
    rt_field = rcptdomain
    rt_field = subject
    rt_field = body
    rt_field = attachment_types
    rt_attr_bigint = arrived
    rt_attr_bigint = sent
    rt_attr_uint = size
    rt_attr_uint = direction
    rt_attr_uint = folder
    rt_attr_uint = attachments
}

index tag1
{
    type = rt
    path = /var/piler/manticore/tag1
    rt_mem_limit = 16M
    stored_fields = tag
    min_word_len = 2
    min_prefix_len = 5
    charset_table  = 0..9, english, _, \
  U+C1->U+E1, U+C4->U+E4, U+C5->U+E5,
U+C6->U+E6, U+C9->U+E9, U+CD->U+ED, U+D3->U+F3, U+D6->U+F6,
U+D8->U+F8, \
  U+DA->U+FA, U+DC->U+FC,
U+0150->U+0151, U+0152->U+0153, U+0170->U+0171, U+01E2->U+E6,
U+01E3->U+E6, U+01FC->U+E6, \
  U+01FD->U+E6, U+1D01->U+E6,
U+1D02->U+E6, U+1D2D->U+E6, U+1D46->U+E6, \
  U+DF, U+E1, U+E4, U+E5, U+E6, U+E9,
U+ED, U+00F3, U+F6, U+F8, U+FA, U+FC, U+0151, U+0153, U+0171
    rt_field = tag
    rt_attr_bigint = mid
    rt_attr_uint = uid
}

index note1
{
    type = rt
    path = /var/piler/manticore/note1
    rt_mem_limit = 16M
    stored_fields = note
    min_word_len = 2
    min_prefix_len = 5
    charset_table  = 0..9, english, _, \
  U+C1->U+E1, U+C4->U+E4, U+C5->U+E5,
U+C6->U+E6, U+C9->U+E9, U+CD->U+ED, U+D3->U+F3, U+D6->U+F6,
U+D8->U+F8, \
  U+DA->U+FA, U+DC->U+FC,
U+0150->U+0151, U+0152->U+0153, U+0170->U+0171, U+01E2->U+E6,
U+01E3->U+E6, U+01FC->U+E6, \
  U+01FD->U+E6, U+1D01->U+E6,
U+1D02->U+E6, U+1D2D->U+E6, U+1D46->U+E6, \
  U+DF, U+E1, U+E4, U+E5, U+E6, U+E9,
U+ED, U+00F3, U+F6, U+F8, U+FA, U+FC, U+0151, U+0153, U+0171
    rt_field = note
    rt_attr_bigint = mid
    rt_attr_uint = uid
}

searchd
{
    listen  = 127.0.0.1:9312
    listen  = 127.0.0.1:9306:mysql41
    log = /var/piler/manticore/manticore.log
    binlog_max_log_size = 256M
    binlog_path = /var/piler/manticore
    binlog_flush    = 2
    query_log   = /var/piler/manticore/query.log
    network_timeout = 5
    pid_file    = /var/run/piler/searchd.pid
    seamless_rotate = 1
    preopen_indexes = 1
    unlink_old  = 1
    thread_stack    = 512k

Re: failed to store index data

[Janos SUTO]

I meant to execute /etc/piler/manticore.conf (yes, it's executable).

Janos

On 2023-03-19 16:06, Jonathan Baecker wrote:

Am 19.03.23 um 14:27 schrieb Janos SUTO:

Hello Jonathan,


Hello Jano, thank you for you quick response!


please tell me more about your setup. Show me the output of

pilerconf |grep -E '(sph|rtindex)'


Output is:

rtindex=1
sphxdb=piler1
sphxhost=127.0.0.1
sphxport=9306



and

/etc/piler/manticore.conf


Content is:

define('RT', 1);



Be sure to redact any passwords.

Janos


On 2023-03-19 14:11, Jonathan Baecker wrote:

Hello,

I just update piler from 1.3.9 to 1.4.3 and I also switched to
manticore. But now piler can not receive new mails. when I check 
piler

service, I get this error:


Mär 19 14:06:52 piler piler[567]:
5000641708f61c491c9c0069477d77cb: hdr_len: 3130, offset: 3
Mär 19 14:06:52 piler piler[567]:
5000641708f61c491c9c0069477d77cb: touch
2e5fa096927b9a6ac6fe82ff2b81256acfa719eb32031ca68f0d86db45415d9f OK
(<608c730b-8a9d-c4aa-4ebd-7f3dec012...@mail.de>)
Mär 19 14:06:52 piler piler[567]:
5000641708f61c491c9c0069477d77cb: stored
'5000641708f61c491c9c0069477d77cb.m' 39327/11168 bytes
Mär 19 14:06:52 piler piler[567]:
5000641708f61c491c9c0069477d77cb: updated meta reference for
'', rc=0
Mär 19 14:06:52 piler piler[567]:
5000641708f61c491c9c0069477d77cb: stored 1 recipients, rc=0
Mär 19 14:06:52 piler piler[567]: ERROR:
5000641708f61c491c9c0069477d77cb failed to store index data for
id=31270, errno=1064, append ret=0
Mär 19 14:06:52 piler piler[567]: ERROR:
5000641708f61c491c9c0069477d77cb: rollback sql stmt=DELETE FROM
sph_index WHERE id=31270
Mär 19 14:06:52 piler piler[567]: ERROR:
5000641708f61c491c9c0069477d77cb: rollback sql stmt=DELETE FROM
rcpt WHERE id=31270
Mär 19 14:06:52 piler piler[567]: ERROR:
5000641708f61c491c9c0069477d77cb: rollback sql stmt=DELETE FROM
metadata WHERE id=31270
Mär 19 14:06:52 piler piler[567]: ERROR:
5000641708f61c491c9c0069477d77cb: rollback sql stmt=DELETE FROM
attachment WHERE piler_id='5000641708f61c491c9c0069477d77cb'
Mär 19 14:06:52 piler piler[567]:
5000641708f61c491c9c0069477d77cb: unlinking
/var/piler/store/00/641/77/cb/5000641708f61c491c9c0069477d77cb.m
Mär 19 14:06:52 piler piler[567]:
5000641708f61c491c9c0069477d77cb: stored metadata, rc=1
Mär 19 14:06:52 piler piler[567]: 1/E07M4E0192DW6TN0: moved to
/var/piler/error/E07M4E0192DW6TN0
Mär 19 14:06:52 piler piler[567]: 1/E07M4E0192DW6TN0:
5000641708f61c491c9c0069477d77cb, size=39327/11168, 
attachments=0,

reference=,
message-id=<608c730b-8a9d-c4aa-4ebd-7f3dec012...@mail.de>,
retention=2557, folder=0, delay=0.1183, status=error


After Update I run mysql piler < ./util/db-upgrade.sql But I think it
had done nothing.

Do you have any idea, how to fix this?


Have a good day!

jb

Re: failed to store index data

[Janos SUTO]

Hello Jonathan,

please tell me more about your setup. Show me the output of

pilerconf |grep -E '(sph|rtindex)'

and

/etc/piler/manticore.conf

Be sure to redact any passwords.

Janos


On 2023-03-19 14:11, Jonathan Baecker wrote:

Hello,

I just update piler from 1.3.9 to 1.4.3 and I also switched to
manticore. But now piler can not receive new mails. when I check piler
service, I get this error:


Mär 19 14:06:52 piler piler[567]:
5000641708f61c491c9c0069477d77cb: hdr_len: 3130, offset: 3
Mär 19 14:06:52 piler piler[567]:
5000641708f61c491c9c0069477d77cb: touch
2e5fa096927b9a6ac6fe82ff2b81256acfa719eb32031ca68f0d86db45415d9f OK
(<608c730b-8a9d-c4aa-4ebd-7f3dec012...@mail.de>)
Mär 19 14:06:52 piler piler[567]:
5000641708f61c491c9c0069477d77cb: stored
'5000641708f61c491c9c0069477d77cb.m' 39327/11168 bytes
Mär 19 14:06:52 piler piler[567]:
5000641708f61c491c9c0069477d77cb: updated meta reference for
'', rc=0
Mär 19 14:06:52 piler piler[567]:
5000641708f61c491c9c0069477d77cb: stored 1 recipients, rc=0
Mär 19 14:06:52 piler piler[567]: ERROR:
5000641708f61c491c9c0069477d77cb failed to store index data for
id=31270, errno=1064, append ret=0
Mär 19 14:06:52 piler piler[567]: ERROR:
5000641708f61c491c9c0069477d77cb: rollback sql stmt=DELETE FROM
sph_index WHERE id=31270
Mär 19 14:06:52 piler piler[567]: ERROR:
5000641708f61c491c9c0069477d77cb: rollback sql stmt=DELETE FROM
rcpt WHERE id=31270
Mär 19 14:06:52 piler piler[567]: ERROR:
5000641708f61c491c9c0069477d77cb: rollback sql stmt=DELETE FROM
metadata WHERE id=31270
Mär 19 14:06:52 piler piler[567]: ERROR:
5000641708f61c491c9c0069477d77cb: rollback sql stmt=DELETE FROM
attachment WHERE piler_id='5000641708f61c491c9c0069477d77cb'
Mär 19 14:06:52 piler piler[567]:
5000641708f61c491c9c0069477d77cb: unlinking
/var/piler/store/00/641/77/cb/5000641708f61c491c9c0069477d77cb.m
Mär 19 14:06:52 piler piler[567]:
5000641708f61c491c9c0069477d77cb: stored metadata, rc=1
Mär 19 14:06:52 piler piler[567]: 1/E07M4E0192DW6TN0: moved to
/var/piler/error/E07M4E0192DW6TN0
Mär 19 14:06:52 piler piler[567]: 1/E07M4E0192DW6TN0:
5000641708f61c491c9c0069477d77cb, size=39327/11168, attachments=0,
reference=,
message-id=<608c730b-8a9d-c4aa-4ebd-7f3dec012...@mail.de>,
retention=2557, folder=0, delay=0.1183, status=error


After Update I run mysql piler < ./util/db-upgrade.sql But I think it
had done nothing.

Do you have any idea, how to fix this?


Have a good day!

jb

piler 1.4.1 is out

[Janos SUTO]

Dear piler users,

it's been quite a while since the last announcement. I've just release
version 1.4.1 release of piler.

The tl;dr version of the changes:

- Replaced sphinx with manticore 5.0.2, see 
https://www.mailpiler.org/wiki/current:manticore

- Replaced obsoleted libchart library with chart.js
- Added real-time index support
- Added oauth2 support to imapfetch.py
- Added support for pilerexport to spread files among several 
directories. See the -D option

- Introduced new piler.conf variable: tls_min_version
- Introduced the archive_address feature, see etc/example.conf for the 
details

- Introduced the raw: search label
- Added Italian translation. Credits: Stefano Gatto
- timestamp signing sorts by 'id' column
- timestamp hash value defaults to sha256
- Added support for php 8.1
- Fixed handling long email addresses

See the download links at https://www.mailpiler.org/wiki/download

Be sure to read the RELEASE_NOTES file as well, and I suggest to execute 
the util/db-upgrade.sql

script to update the database schema.

Some more details about the recent changes.

* Sphinx search has been replaced with Manticore search

I'm not happy with sphinx 3 anymore. It seems that it's development has 
been stalled.
The latest release of 3.4.1 (~1.5 years old) seems to have some issues, 
and not much
progress for the issues mentioned on the sphinx forum. So from now on 
the preferred
search engine is manticore search. Note that sphinx 3 is still 
supported, but for any
new piler deployments I recommend manticore. Unfortunately the binary 
index data are
not compatible, so if you decided to drop sphinx and start using 
manticore, you need

to reindex all emails.

* Real time index support

https://www.mailpiler.org/wiki/current:manticore describes how to turn 
it on.


* Oauth2 support for imapgetch.py

1.4.1 ships an auxiliary utility (get-token.py) allowing you to get an 
auth token
from o365 that can be used for imapfetch.py to download emails from an 
o365 mailbox.


* php 8.1 support

Since we have ubuntu 22.04, and other distros shipping php 8.1, piler 
1.4.1 has

support for it. php 7.4+ is still supported

* archive_address parameter in piler.conf

It's a mitigation for archives that can be accessed from anywhere. It 
forces
piler to continue the smtp transaction only if the rcpt to envelope 
address

matches the email address in the archive_address parameter.

* Fixed handling long email addresses

Both sphinx and manticore has a limit for tokens (read: words) in the 
search query.
Some of you have a few pretty long email addresses, more than 40 
characters.
Piler creates an additional token for such long addresses, adds it to 
the body
to index. Then the gui replaces the long address with the transformed 
(it's a

simple md5 hash value) value in the background.

If you have any questions, let me know, and I'll do my best to answer 
them.


Janos

Re: fetching mail from office365 after Microsoft disables "basic authentication"?

[Janos SUTO]

Hello Jeremiah,

thank you for your work. I'll take a look at this repo. Anyway, you are 
right
that it makes a pain to use imap as the source of emails. I recommend 
everyone
(if possible) to use smtp journaling to acquire emails. It spares you 
deleting

already processed emails.

Janos


On 2022-11-21 18:10, Jeremiah Poling wrote:

Janos,

For now, I've setup an imperfect workflow using offlineimap.
https://github.com/UvA-FNWI/M365-IMAP

Bit of a pain to setup, and I'm sure it will be a pain when the token
expires and has to be renewed, but my mail is getting into the archive
for now at least.

Major downside is that offline imap just syncs mail with no option to
remove from the server, so I'm having to purge it manually
periodically.

--Jeremiah

-Original Message-
From: Jeremiah Poling
Sent: Friday, July 1, 2022 13:42
To: Piler User 
Subject: RE: fetching mail from office365 after Microsoft disables
"basic authentication"?

Thank you!

--Jeremiah

-Original Message-
From: s...@acts.hu 
Sent: Friday, July 1, 2022 06:03
To: Piler User 
Subject: Re: fetching mail from office365 after Microsoft disables
"basic authentication"?

[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you
recognize the sender and know the content is safe.

Hello Jeremiah,

I'll try to come up with something usable starting at here:
https://stackoverflow.com/questions/54189837/python-o365-authentication-when-basic-authentication-becomes-deprecated

Note that I'll prefer the imapfetch.py utility to get emails via imap.

Janos SUTO

On 2022-07-01 00:08, Jeremiah Poling wrote:

Hello!

We're pilerimport -i to download mail for archiving from a journaling
mailbox.

Are there any recommendations for a similar method that would use a
more "modern" authentication method for after Microsoft disables basic
authentication in September?

Thank you for any advice you can provide!

--Jeremiah

Re: I want to know if piler can do the following

[Janos SUTO]

Unfortunately, piler is not an imap server, so it cannot do what you asked.

Janos

On Jun 6, 2022, 02:06, at 02:06, Asmodean Thor  wrote:
>I have decided to offload my mail from google, and look for another
>provider in the future.
>
>This is the solution I want:
>Google send and receive —> POP3 to piler on VPS —> delete from google
>server —> pull via IMAP from piler to various clients
>
>Thank you!

Re: Reindex Question / Clarification

[Janos SUTO]

Hello Ryan,

reindexing won't cause any waste in sphinx, so go ahead.

Btw. you have several main indices, eg. main1, main2, etc. It's possible to use 
main1 for 2017 data, main2 for 2018 data, etc. In that case you may move main1 
to a backup media to free up some space. The trick is to fix both 
indexer.delta.sh and indexer.main.sh to reference the given main index.

Janos

On 22 Sep 2020, 02:37, at 02:37, Ryan Blenis  wrote:
>Hi Janos,
>
>Quick question on reindexing and its effect on the size of the sphinx
>files.
>
>If I reindex something that is already in the index, will there be
>duplicates in sphinx wasting space, or is this detected and removed?
>
>Background: We weren't searching for anything old normally, so we
>cleared
>the sphinx data and only indexed the last years worth of data. Now we
>have
>requests for data from parts of 2018, 2017, etc. that we've reindexed
>as
>needed to make searchable, but the more requests come in it seems it
>would
>be better to reindex everything and just throw resources at it if
>needed
>and cut out the "manual" process of manually indexing anything prior to
>a
>year ago as the requests come in. If I reindex everything, will I be
>wasting space and should start fresh, or will the duplicates be
>filtered
>out at some point in the process (e.g. not wasting space)?
>
>Thank you as always.

Re: Piler 1.35 on Ubuntu 18.04 - Web UI doesn't show mail headers / contents

[Janos SUTO]

Hello Patrick,

fix the piler binary paths in config-site.php. Check out the current values in 
config.php.

Janos SUTO

On 31 Jul 2020, 14:07, at 14:07, "Wagner, Patrick"  
wrote:
>
>
>Hello everyone,
>
>
>just opened an older piler archive (version 1.35) and was surprised to
>see all the emails listed in the web GUI upper pane / list, including
>icons for attachments and correct messages sizes, but I cannot display
>the actual email headers or contents.
>
>The lower pane / detail pane per email is pretty much empty:
>
>
>Subject:
>From:
>To:
>Date:
>
>
>
>Attachment names are shown, however every attachment I try to open is 1
>
>bytes on download, so obviously unusable.
>
>Thankfully it seems to be a web GUI interface only, as a test with
>pilerexport yielded the email contents, including attachments.
>
>What's wrong? Is the Piler version too old for modern browsers? What do
>
>I need to fix?
>
>Thanks,
>
>Patrick
>
>
># piler -V
>piler 1.3.5, build 997, Janos SUTO 
>
>Build Date: Sun Apr 21 16:50:30 UTC 2019
>ldd version: ldd (Ubuntu GLIBC 2.27-3ubuntu1) 2.27
>gcc version: gcc version 7.3.0 (Ubuntu 7.3.0-27ubuntu1~18.04)
>OS: Linux 188ae4f9894f 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13
>10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>Configure command: ./configure --prefix=/usr --sysconfdir=/etc
>--localstatedir=/var --with-database=mariadb
>MySQL client library version: 10.2.9
>Extractors: /usr/bin/pdftotext /usr/bin/catdoc /usr/bin/catppt
>/usr/bin/xls2csv /usr/bin/unrtf /usr/bin/tnef
>
>Ubuntu 18.04.4 LTS
>
>PHP 7.2.24-0ubuntu0.18.04.6 (as supplied with the distro)
>
>MariaDB 10.1.44 (as supplied with the distro)
>
>
>*
>*

Re: Empty message header and body

[Janos SUTO]

Sorry, piler.key is a file, not an entry in piler.conf.

Janos

On 15 Jul 2020, 12:25, at 12:25, Marcelo Machado  wrote:
>Thanks for the answer.
>
>The parameter iv is cipher_list.There is no line with piler.key.
> I was unable to read the contents of piler.key.
>
>Marcelo
>
>Em qua., 15 de jul. de 2020 às 00:46, Janos SUTO  escreveu:
>
>> Hello Marcelo,
>>
>> check if piler.key and the iv parameter in piler.conf (if there's
>any) are
>> the same.
>>
>> Janos
>> On 14 Jul 2020, at 23:42, Marcelo Machado  wrote:
>>>
>>> Hello everyone.
>>>
>>> After I migrated MailPiler to a new hardware, the header and body of
>old
>>> messages no longer appear. The new ones are OK.
>>>
>>> I followed the migration documentation on the website.
>>>
>>> Marcelo Gomes
>>>
>>

Re: Empty message header and body

[Janos SUTO]

Hello Marcelo,

check if piler.key and the iv parameter in piler.conf (if there's any) are the 
same.

Janos

On 14 Jul 2020, 23:42, at 23:42, Marcelo Machado  wrote:
>Hello everyone.
>
>After I migrated MailPiler to a new hardware, the header and body of
>old
>messages no longer appear. The new ones are OK.
>
>I followed the migration documentation on the website.
>
>Marcelo Gomes

Re: Upgrade Piler 1.3.7 -> 1.3.8 / Upgrade Sphinx

[Janos SUTO]

Hello,

well, you need that python module, please install it.

And yes, I recommend sphinx 3.2.1 for a new installation. The 2.2.x series is 
not maintained by the sphinx team apparently. It's safe to upgrade from 3.1.1 
to 3.2.1.

Janos

On 22 Apr 2020, 00:58, at 00:58, "BKH-Netz.de - Postfach" 
 wrote:
>
>
>Hello,
>
>i upgraded my piler installation 1.3.7 to 1.3.8 on a Debian 9 (Buster).
>After deleting some mail i wanted to purge them using pilerpurge.
>
>Doing this i got an error:
>
>Traceback (most recent call last):
>  File "/usr/local/libexec/piler/pilerpurge.py", line 4, in 
>import MySQLdb as dbapi
>ModuleNotFoundError: No module named ‚MySQLdb'
>
>Using piler 1.3.7 i have never seen this error …
>
>—
>
>Do you recommend an upgrade to sphinx 3.2.1 on a new and fresh
>installation?
>Can i upgrade from sphinx 3.1.1 to 3.2.1 on an existing installation?
>
>Thanks :-)

Re: Wrong dates

[Janos SUTO]

Hello,

click on view headers, and paste the date header line here.

Janos

On 4 Feb 2020, 12:15, at 12:15, Marcelo Machado  wrote:
>Hello everyone.
>
>Can you help me why the dates are different?
>
>Marcelo
>
>
>[image: tempsnip.png]

Re: Upgrade or reinstall on newer OS version ?

[Janos SUTO]

Hello Yann,

I have no experience upgrading through several LTS versions. A mysql -> mariadb 
switch should not hurt. Anyway, be sure to backup the database, or even better 
snapshot the whole vm. When the os upgrade is ready, be sure to recompile Piler.

Be sure to stop all piler related processes even the ctontabs while upgrading 
the os.

The other option could be to start a new vm with the choice of a recent distro 
with the latest master of Piler. Then export all emails from old server, and 
import to new one. This approach has the benefit of having everything new and 
up to date, however you need a lengthy import process.

Janos


On 10 Jan 2020, 09:18, at 09:18, Yann Lehmann  wrote:
>
>
>Hello Everyone
>
>Our Piler server is doing good so far (only service on a virtual
>machine), but it is running on Ubuntu 14.04, which is no more
>supported.
>
>I would like to either upgrade to Ubuntu 18.04 (actual LTS, or may be 
>wait some time and go to 20.04 the next LTS) or move to Debian 10. I 
>know that on Ubuntu, an in place upgrade is only possible from one LTS
>version to the next, so in my case, it would be from 14.04 to 16.04,
>and
>then to 18.04.
>
>
>I also know that if I upgrade from Ubuntu 14.04 to 16.04, I would have
>to upgrade Mysql (or move to MariaDB) before the upgrade.
>
>What path would you advise: in place upgrade or move/reinstall on newer
>
>os ? And what were your experiences with in place upgrades on Ubuntu ?
>
>Thank's in advance for any advice and best regards
>Yann

Re: Authentication against multiple IMAP servers

[Janos SUTO]

Hello,

then there must be a syntax error on the introduced function. Check out the 
error logs of the webserver.

If it doesn't help, then show me the function in a private email.

Janos


 Original Message 
From: BKH - Post 
Sent: Thu Oct 31 11:25:37 GMT+01:00 2019
To: piler-user@list.acts.hu
Subject: Re: Authentication against multiple IMAP servers



Hello Janos,

i tried this and set CUSTOM_PRE_AUTH_FUNCTION as described in the docs.
For testing reasons i copied your "my_imap"-script into config-site.php.

After this tried to log into piler with different users but the login 
page is shown as acompletely blank page.
Disabling the script, which is mentioned in the docs, login page is 
shown correctly.

I changes domain values to my domains but this error persists.

Can you help me with this?

BKH

Am 29.10.19 um 20:20 schrieb s...@acts.hu:
>
>
> Hello,
>
> I've just updated the custom authentication docs at
> http://www.mailpiler.org/wiki/current:custom-authentication
>
> Please check out the CUSTOM_PRE_AUTH_FUNCTION feature at the end.
>
> Janos
>
> On 2019-10-29 10:12, BKH - Post wrote:
>> Hi @all,
>>
>> is there a possibility to establish an authentication against multiple
>> IMAP servers?
>> Having here a scenario with two to three internal IMAP servers in
>> different workgroups i don't want to setup three mailpiler instances.
>> In fact that setting setting up such system is not a problem i want to
>> reduce administrative tasks.
>>
>> Thanks for your advice.
>>
>> Regards
>>
>> BKH
>

Re: Upgrade from 1.3.5 -> 1.3.7

[Janos SUTO]

Hello,

I'll clarify this in the upgrade docs, however you may overwrite all files 
(except the configs), because there hasn't been any incompatible change.

As a precaution, always backup your config files and especially the encryption 
key.

Janos


 Original Message 
From: BKH - Post 
Sent: Fri Oct 25 12:01:35 GMT+02:00 2019
To: piler-user@list.acts.hu
Subject: Upgrade from 1.3.5 -> 1.3.7



Hi @all,

version 1.3.7 of mailpiler is out.
After reading upgrade instructions i found nothing regarding version 1.3.7.
As i want to do an upgraade of piler form the first time i need some 
hints to do this.

It is clear that making a backup of all configs is recommended.
Can i copy over all files?

Thanks for your advice

Regards

BKH

Re: Forward messages from another server

[Janos SUTO]

Hello Marcelo,

assuming you want to archive internal emails as well that provably never reach 
the postfix relay, I'd rather setup journaling on exchange.

Janos


 Original Message 
From: Marcelo Machado 
Sent: Mon Oct 14 15:32:45 GMT+02:00 2019
To: Piler User 
Subject: Forward messages from another server

I have a postfix that works as relay from an Exchange server. How do I get
postfix to forward these messages to Mailpiler?

Marcelo Gomes

Re: Cronjobs - change indexing interval

[Janos SUTO]

Hello,

try this:

*/10 * * * * 

Btw. check the mail log to see if it works as required. Whenever the indexer 
script starts it syslogs it.

Janos


 Original Message 
From: BKH 
Sent: Fri Aug 23 21:35:04 GMT+02:00 2019
To: piler-user@list.acts.hu
Subject: Cronjobs - change indexing interval



Hello,

for testing reasons i tried changing indexing interval.
I played around with this line:

5,35 * * * * /usr/local/libexec/piler/indexer.delta.sh

I changed minute an hour to differnte values, but could not figure out
to get it in a working state.
Indexing should take place e.g. every 10 minutes.
How can i achive this?

Thanks :-)

BKH

Re: Indexation of Excel files newer than 2007

[Janos SUTO]

Try the following. Get the latest master branch, and run the configure command. 
It prints a summary at the end including libzip status.

Anyway it's also possible to adopt the Tika project as adminshok suggested.

Janos


 Original Message 
From: Katterl Christian 
Sent: Wed May 08 10:52:33 GMT+02:00 2019
To: Piler User 
Subject: AW: Indexation of Excel files newer than 2007

In at least my case, this does not seem to work.

BR, Christian



Von: Janos SUTO 
Gesendet: Montag, 6. Mai 2019 11:33
An: Piler User 
Betreff: Re: Indexation of Excel files newer than 2007

Newer office files, eg. xlsx, etc should be handled internally by the parser, 
provided that you have libzip package installed as well as the header files, 
libzip-dev or similar.
Janos


 Christian Katterl
Teamleader Technical IT

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel +43 50 799 - 2511

mobile  +43 664 811 54 99
c.katt...@asamer.at
www.abag.at


This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this
message by mistake, please advise the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334


From: Katterl Christian
Sent: Mon May 06 10:19:07 GMT+02:00 2019
To: Piler User
Subject: AW: Indexation of Excel files newer than 2007


Hello again,


for docx, there would be: https://github.com/ankushshah89/python-docx2txt


Unfortunately, I am not a software-developer to make the adoptions by myself.


BR Christian




Von: Martin Nadvornik 
mailto:martin.nadvor...@diakonie.at>>
Gesendet: Montag, 6. Mai 2019 09:46
An: Piler User mailto:piler-user@list.acts.hu>>
Betreff: Re: Indexation of Excel files newer than 2007


Hello Christian,

catdoc is not capable of processing new office formats. As far as I know there 
is no intention for catdoc to implement this in a foreseeable future. The same 
problem exists for xls2csv. You could theoretically try to call unoconv 
(https://github.com/unoconv/unoconv) before catdoc, but it will probably have a 
big performance impact since it launches libre office / open office for the 
conversion. But if you try this I would be interested in your results since 
being limited to index only old office formats is also something we would like 
to overcome. Alternatively if you can find an open source software which is 
capable of efficiently extracting plain text from current office formats it 
should be easily implementable into piler (basically a few lines in extract.c 
as far as I can tell). For excel there is https://github.com/xevo/xls2csv and 
https://github.com/nagirrab/xls2csv which claim to be cabable of proccessing 
xlsx files. But I haven't looked into them yet.

Kind Regards
Martin
Am 06.05.2019 um 06:45 schrieb Katterl Christian:
Hello,

Indexation of Excel files newer than Excel 2007 fails in my installation.
I am using catdoc 0.95 and it tells:

This file looks like ZIP archive or Office 2007 or later file.
Not supported by catdoc

The Excel-File has been created using Excel 2010.

BR, Christian


Christian Katterl
Teamleader Technical IT

[cid:image001.png@01D503F5.230293E0]

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel  +43 50 799 - 2511
mobile +43 664 811 54 99
email c.katt...@asamer.at<mailto:c.katt...@asamer.at>
www.abag.at<https://www.abag.at>

This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this message by mistake, please advise 
the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334

Re: Indexation of Excel files newer than 2007

[Janos SUTO]

Newer office files, eg. xlsx, etc should be handled internally by the parser, 
provided that you have libzip package installed as well as the header files, 
libzip-dev or similar.

Janos


 Original Message 
From: Katterl Christian 
Sent: Mon May 06 10:19:07 GMT+02:00 2019
To: Piler User 
Subject: AW: Indexation of Excel files newer than 2007

Hello again,

for docx, there would be: https://github.com/ankushshah89/python-docx2txt

Unfortunately, I am not a software-developer to make the adoptions by myself.

BR Christian


Von: Martin Nadvornik 
Gesendet: Montag, 6. Mai 2019 09:46
An: Piler User 
Betreff: Re: Indexation of Excel files newer than 2007

Hello Christian,

catdoc is not capable of processing new office formats. As far as I know there 
is no intention for catdoc to implement this in a foreseeable future. The same 
problem exists for xls2csv. You could theoretically try to call unoconv 
(https://github.com/unoconv/unoconv) before catdoc, but it will probably have a 
big performance impact since it launches libre office / open office for the 
conversion. But if you try this I would be interested in your results since 
being limited to index only old office formats is also something we would like 
to overcome. Alternatively if you can find an open source software which is 
capable of efficiently extracting plain text from current office formats it 
should be easily implementable into piler (basically a few lines in extract.c 
as far as I can tell). For excel there is https://github.com/xevo/xls2csv and 
https://github.com/nagirrab/xls2csv which claim to be cabable of proccessing 
xlsx files. But I haven't looked into them yet.

Kind Regards
Martin
Am 06.05.2019 um 06:45 schrieb Katterl Christian:
Hello,

Indexation of Excel files newer than Excel 2007 fails in my installation.
I am using catdoc 0.95 and it tells:

This file looks like ZIP archive or Office 2007 or later file.
Not supported by catdoc

The Excel-File has been created using Excel 2010.

BR, Christian


Christian Katterl
Teamleader Technical IT

[cid:image001.png@01D503F5.230293E0]

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel  +43 50 799 - 2511
mobile +43 664 811 54 99
email c.katt...@asamer.at
www.abag.at


This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this message by mistake, please advise 
the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334



 Christian Katterl
Teamleader Technical IT

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel +43 50 799 - 2511

mobile  +43 664 811 54 99
c.katt...@asamer.at
www.abag.at


This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this
message by mistake, please advise the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334

Re: How to configure info@ adresses?

[Janos SUTO]

Hello,

I think it can be done using the group feature. Its purpose is to give access 
to another mailbox as well.

You may setup the group settings as admin in the GUI.

Janos


 Original Message 
From: Biderius - 
Sent: Fri May 03 12:34:41 GMT+02:00 2019
To: "piler-user@list.acts.hu" 
Subject: How to configure info@ adresses?

Hello,
I have configured piler using my LDAP as login-service. When I log in, I can 
see all my messages associated with my user-account.

But now, I'm facing the problem, that our company uses an info@ addresss which 
is splitted to severel agents via "virtual-transports" tables in postfix. How 
can I see these messages when I log in to piler?
I already tried to configure the $config[LDAP_MAIL_ATTR] param, but when I use 
this, I cant log in to piler.
How can I see both e-mail addresses?

Re: Downloaded files corrupt

[Janos SUTO]

Hello,

what language file do you use?
Can reproduce the issue using English?

Janos


 Original Message 
From: Katterl Christian 
Sent: Tue Apr 30 05:45:59 GMT+02:00 2019
To: "piler-user@list.acts.hu" 
Subject: Downloaded files corrupt

Hello,

when I try to download archived files (attachments), I noticed, that the files 
are corrupted after download.
They are 1 byte larger than the original.

I found this here:
https://bitbucket.org/jsuto/piler/issues/949/corrupted-files-are-downloaded-from-web-ui

I even tried the language-files of the current master-tree, where I assume, the 
issue is fixed?

BR, Christian


 Christian Katterl
Teamleader Technical IT

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel +43 50 799 - 2511

mobile  +43 664 811 54 99
c.katt...@asamer.at
www.abag.at


This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this
message by mistake, please advise the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334

Re: Rentention-rules and pilerpurge

[Janos SUTO]

Hello Ryan,

you are right about the verification hash problem. However, since you are about 
to purge those messages, it's probably not that big issue.

Btw. the GUI compares only the message hash values, and the rest of the 
metadata table entries (eg. the retained column) are not used currently.

Janos


 Original Message 
From: Ryan Blenis 
Sent: Tue Nov 13 23:08:59 GMT+01:00 2018
To: piler-user@list.acts.hu
Subject: Re: Rentention-rules and pilerpurge

> > Do you have any idea what I should
> > do  so that more messages are being purged? Especially the retention
> > period seems suspect I think...
> For already archived spam, you should identify them in the metadata
table, eg. check the subject
> field. Then set the retained column for such messages only to the past,
eg. a yesterday
> timestamp, and pilerpurge will take care of them.

Hello Janos,

Seeing your above reply to Marina's post regarding purging data brought me
back to an issue we were trying to clear up (we have a certain email "from"
address we'd like purged, as it is not required in the archive). However,
we lacked the foresight to set up retention rules prior to the email
account spewing out millions of emails.

I figured I'd just go in and manually set the retention date back in the
past as you suggest, however I found the following note on
http://www.mailpiler.org/wiki/current:administering-piler:

--
Note that the retention period is included in the per message verification
digest, so the retention period should not be changed after the message has
been archived.
--

I've run a test on the database by selecting one email and changing the
retention period to the past and running

pilerpurge -d

to see the effect, and it says it will remove it.

If this is true, what are the ramifications of this and the message
verification digest no longer aligning?

Is this a viable method to get rid of the cruft we've accumulated and would
like to purge?

Thank you.

Re: var/piler/tmp files

[Janos SUTO]

Hello Kim,

you should look at the mail log to find out what happened. There might be some 
parsing or encoding issues.

Janos


 Original Message 
From: "Kim Mølgaard" 
Sent: Thu Oct 18 09:18:52 GMT+02:00 2018
To: piler-user@list.acts.hu
Subject: var/piler/tmp files

Hi !
Piler v. 1.3.6 on Ubuntu 16.04.
My /var/piler/tmp is filled with files, lots of files.
I clean it out 3 times a day, by stopping piler and removing all files, but 
that can't be right.


Do anyone have any suggestions to what is going on, and how to deal with it ?
Regards

Kim Mølgaard
ITandMore
Tlf.: +45 93 89 83 79

CVR nr. 38147110

Re: Upgrading from old piler to new

[Janos SUTO]

Hello,

well, the official site has a section for the upgrade. It should work.

It's worth to backup the MySQL database, your configs and the piler.key.

Janos


 Original Message 
From: Michael Mansour 
Sent: Fri Oct 05 01:47:18 GMT+02:00 2018
To: Piler User 
Subject: Upgrading from old piler to new



Hi,

I have an older Mail Piler version and would like to update to the latest.

Are there any upgrade docs on-line or similar that I may look at so the 
upgrade is smooth?

Thanks.

Michael.


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: Moved piler VM and then rsync'ed piler store

[Janos SUTO]

Hello Michael,

check the ownership on all the directories down to the .m file.

Janos


 Original Message 
From: Michael Mansour 
Sent: Thu Oct 04 14:13:49 GMT+02:00 2018
To: piler-user@list.acts.hu
Subject: Moved piler VM and then rsync'ed piler store



Hi Piler users,

I had to shutdown a data centre and move to another data centre.

The original piler VM was an easy copy from one to the other, all worked.

The store here:

/var/piler/store

was on an NFS share in the old data centre. I create the same NFS in the 
new data centre, then rsync'ed using:

rsync -avz etc etc

from one to the other, for everything in /var/piler/store.

Everything moved across, then I change permissions on the new 
/var/piler/store using:

cd /var/piler/store/
chown -R piler:piler 00
find . -type d -exec chmod 750 {} \;
find . -type f -exec chmod 640 {} \;

This ran for a while but eventually completed.

When I got to the Piler interface with Auditor account, I can see the 
messages there but when clicking on any of the messages I can't view 
them and instead, get this in the logs:

Oct  4 21:52:40 server pilerget[19390]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()
Oct  4 21:52:40 server pilerget[19393]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()
Oct  4 21:52:49 server pilerget[19400]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()
Oct  4 21:52:49 server pilerget[19403]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()
Oct  4 21:52:51 server pilerget[19408]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()
Oct  4 21:52:51 server pilerget[19411]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()
Oct  4 21:52:51 server pilerget[19416]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()
Oct  4 21:52:51 server pilerget[19419]: 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m: 
cannot open()

and we do see:

# ll /var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m
-rw-r- 1 piler piler 1072 Sep 28 23:01 
/var/piler/store/00/5ba/b3/f0/40005bae25ba19a0303400b43bd1b3f0.m

so permissions are correct.

Any ideas what I did wrong ?

Thanks.

Michael.


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: CentOS 7 support only for free version

[Janos SUTO]

Hello Steve,

give me some time, and I will check if I can make it on centos 7.

Janos


 Original Message 
From: Stephen Ingram 
Sent: Sun Jul 08 01:55:10 GMT+02:00 2018
To: piler-user@list.acts.hu
Subject: CentOS 7 support only for free version

Is there any way to get the commercial version on CentOS 7 or do we have to
use the free version if we want CentOS 7 support?

Steve

Re: Limit on searchs?

[Janos SUTO]

Hello Thiago,

surely not correct. In fact it's pretty bad. For starters check if the Sphinx 
database is fine.

Janos


 Original Message 
From: Thiago Lima 
Sent: Wed Jun 20 00:38:54 GMT+02:00 2018
To: Piler User 
Subject: Limit on searchs?

After a crash and rebuild of the databases, my mail piler just shows 50
results per page and doesn't allow to change pages.

All the searchs returns "total: 0" and I can only see the first page of
results.

Is that correct or there's something wrong?

RE: only sent mails are stored or also received?

[Janos SUTO]

What were your issues? Perhaps I can help with them.

Anyway, using pilerimport is possible, though it's not efficient as you have an 
ever growing mailbox, and pilerimport needs more and more time to process.

Janos


 Original Message 
From: Jackson Craig 
Sent: Thu Apr 12 15:52:18 GMT+02:00 2018
To: 'Piler User' 
Subject: RE: only sent mails are stored or also received?

Hi,

 

For me, all I added to postfix  - in the main.cf configuration file, was :

 

#Keep a copy of every email by sending bcc

always_bcc = b...@yourdomain.com  

 

Of course, replace the domain part :)  At the end of each day, I have all the 
email that has passed through the server sitting ready to be collected. I did 
initially experiment with realtime forwarding to piler, but I had various 
issues with that so just went with this instead.

 

Then on the piler server, I have a cron entry (important to run this as the 
piler user and not for example root) to collect the mail over pop3:

 

01 20 * * * /usr/local/bin/pilerimport –K your.mailserver -u bcc -p bcc

 

Hope this helps,

 

Regards

Craig

 

  -

This email and any attachments are confidential to the intended recipient 

and may also be privileged.  If you are not the intended recipient, please 

delete it from your system and notify the sender.  You should not copy it 

or use it for any purpose nor disclose or distribute its contents to any 

other person.

 

From: Lucian Lazar [mailto:lucian.la...@soteha.it] 
Sent: 12 April 2018 13:44
To: Piler User 
Subject: Re: only sent mails are stored or also received?

 

Thank you, we use Postfix as well for our mailing servers. Is there any 
reference in the documentation regarding how to setup this BCC behavior? In 
this case, all original headers will be preserved? Like mailto:, mailfrom: and 
the people that were in the CC and BCC fields?

Thank you again

-- 

Lucian Lazar - Linux System Engineer
Soteha S.r.l. a Socio Unico
Via Giuseppe Zucchi, 39C 20095 Cusano Milanino (MI)
Ingresso principale da 39/G interno 3
T. +39 02 66043166 | F. +39 02 66048942
  lucian.la...@soteha.it

From: Jackson Craig  >
Reply-To: Piler User  >
Date: Thursday, 12 April 2018 at 15:40
To: 'Piler User'  >
Subject: RE: only sent mails are stored or also received?

 

Hi there,

 

One way is to have your mailserver send a bcc of every mail to a separate 
mailbox on the server,  which is then collected by piler. For example here, 
using postfix, we configure a bcc mail for every mail that passes through 
postfix. This then puts a copy of every mail into a mailbox on the server 
(incoming and outgoing). Mailpiler then collects this mail by logging into the 
mailserver using POP3. We do this at the end of each day. Of course the exact 
configuration will depend on your mailserver.

 

Regards

Craig Jackson

 

  -

This email and any attachments are confidential to the intended recipient 

and may also be privileged.  If you are not the intended recipient, please 

delete it from your system and notify the sender.  You should not copy it 

or use it for any purpose nor disclose or distribute its contents to any 

other person.

 

From: Lucian Lazar [  
mailto:lucian.la...@soteha.it] 
Sent: 12 April 2018 13:21
To:   piler-user@list.acts.hu
Subject: only sent mails are stored or also received?

 

Hello,

We are actually evaluating a viable solution for our mail legal hold needs. We 
currently need to archive all sent and received emails for all users in a given 
mailserver.

Please correct me if i’m wrong, but i have red that only sendemail from the 
users are being saved into Piler database? Is there any way we can als store 
emails received by the users?

Thank you

 

-- 

Lucian Lazar - Linux System Engineer
Soteha S.r.l. a Socio Unico
Via Giuseppe Zucchi, 39C 20095 Cusano Milanino (MI)
Ingresso principale da 39/G interno 3
T. +39 02 66043166 | F. +39 02 66048942
  lucian.la...@soteha.it


Missiles & Space Batteries Limited is registered in Scotland under company 
number 162626. The company's registered office is at Hagmill Road, East 
Shawhead, Coatbridge, ML5 4UZ. 


Missiles & Space Batteries Limited is registered in Scotland under company
number 162626.  The company's registered office is at Hagmill Road, East
Shawhead, Coatbridge, ML5 4UZ.

Re: only sent mails are stored or also received?

[Janos SUTO]

Hello Lucian,

piler archives every email you send to it. Just configure your mail server to 
copy both sent and received emails to piler, and you'll be fine.

Janos


 Original Message 
From: Lucian Lazar 
Sent: Thu Apr 12 14:20:36 GMT+02:00 2018
To: "piler-user@list.acts.hu" 
Subject: only sent mails are stored or also received?

Hello,
We are actually evaluating a viable solution for our mail legal hold needs. We 
currently need to archive all sent and received emails for all users in a given 
mailserver.
Please correct me if i’m wrong, but i have red that only sendemail from the 
users are being saved into Piler database? Is there any way we can als store 
emails received by the users?
Thank you

--
Lucian Lazar - Linux System Engineer
Soteha S.r.l. a Socio Unico
Via Giuseppe Zucchi, 39C 20095 Cusano Milanino (MI)
Ingresso principale da 39/G interno 3
T. +39 02 66043166 | F. +39 02 66048942
lucian.la...@soteha.it

Re: BCC not shown

[Janos SUTO]

Hello Christian,

check the Sphinx query in the mail log. I suspect that your bcc address is not 
in the query.

The fix depends on what authentication you are using.


Janos


 Original Message 
From: "Christian Röser - PELMA" 
Sent: Mon Feb 12 17:37:09 GMT+01:00 2018
To: Piler User 
Subject: BCC not shown



Hello,

I need some help. Today I made some tests with my piler installation and I 
found a problem that drives me crazy - but I assume there could be simple 
configuration problem. Messages I receive as bcc are not shown in the search 
results.

I checked if the exchange server behaves correctly, here seems everything fine. 
The system generates two e-mails, one for the normal recipient and a separate 
one for the bcc recipient. Piler receive both e-mails, stores the first one and 
recognizes for the second e-mail that it have the same message-id. In the 
metadata table exists an entry and in the rcpt table two entries for this id. 
Nevertheless only in the search results of the "normal" recipient the message 
is shown.

Is the some parameter missing?

Best regards,
Christian

Re: Feature Request: Disable Restore to Mailbox

[Janos SUTO]

Hello,

you can disable the feature: set the smarthost parameter to an empty string.

Janos


 Original Message 
From: Katterl Christian 
Sent: Wed Nov 29 10:43:42 GMT+01:00 2017
To: Piler User 
Subject: Feature Request: Disable Restore to Mailbox

Hello,

as the „restore-to-mailbox“ feature sends an item using smtp, the original date 
and time is screwed up and the item get’s restored with the date and time of 
restoring.

Exporting to  EML would be the better option, in my opinion.
Therefore, it would be nice to have a config-switch to disable this export to 
mailbox completey.

BR, Christian


Christian Katterl
Teamleader Technical IT

[cid:01a2-0001@01d368f6.8b9afdbc]

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel  +43 50 799 - 2511
mobile +43 664 811 54 99
email c.katt...@asamer.at
www.abag.at



This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this message by mistake, please advise 
the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334

Re: AW: LDAP Users can see other's mails

[Janos SUTO]

Thanks, it's even better. I'll add it to the FAQ.

Janos


 Original Message 
From: Katterl Christian <c.katt...@asamer.at>
Sent: Tue Nov 28 08:47:20 GMT+01:00 2017
To: Piler User <piler-user@list.acts.hu>
Subject: AW: AW: LDAP Users can see other's mails

Update:
By modifying the parameter from $config['LDAP_DISTRIBUTIONLIST_OBJECTCLASS'] = 
'group'; to $config['LDAP_DISTRIBUTIONLIST_OBJECTCLASS'] = 'user'; it seems to 
accomplish the same without changing code in /var/www/piler/model/user/auth.php



Christian Katterl
Teamleader Technical IT

[cid:0139-0001@01d3681d.1f8ac597]

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel  +43 50 799 - 2511
mobile +43 664 811 54 99
email c.katt...@asamer.at<mailto:c.katt...@asamer.at>
www.abag.at<https://www.abag.at>



This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this message by mistake, please advise 
the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334

Von: Janos SUTO [mailto:s...@acts.hu]
Gesendet: Dienstag, 28. November 2017 08:11
An: Piler User
Betreff: Re: AW: LDAP Users can see other's mails

OK, if it gives you a proper result, then case is solved. Be sure to save your 
fix in case of a future upgrade. Or I may introduce a configure option to apply 
your fix.
Janos

From: Katterl Christian
Sent: Tue Nov 28 06:35:25 GMT+01:00 2017
To: Piler User
Subject: AW: AW: LDAP Users can see other's mails



Hello,

maybe I have found a solution for this issue.

In /var/www/piler/model/user/auth.php

I changed the line 217, which originally looked like this:

$query = $ldap->query($ldap_base_dn, 
"(|(&(objectClass=$ldap_account_objectclass)($ldap_mail_attr=$username_prefix$username))(&(objectClass=$ldap_distributionlist_objectclass)($ldap_distributionlist_attr=$username_prefix$username)"
 . 
")(&(objectClass=$ldap_distributionlist_objectclass)($ldap_distributionlist_attr="
 . $a['dn'] . ")))", array());

To only:
  $query = $ldap->query($ldap_base_dn, 
"(|(&(objectClass=$ldap_account_objectclass)($ldap_mail_attr=$username_prefix$username)))",
 array());

I mean - I removed all the group- and distribution-list things.
I am not sure, what this else will/could cause (I am not a programmer)?

But from what I saw in a very quick test, now only my personal emails are shown.

BR, Christian


 Christian Katterl
Teamleader Technical IT

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel +43 50 799 - 2511

mobile  +43 664 811 54 99
c.katt...@asamer.at<mailto:c.katt...@asamer.at>
www.abag.at<http://www.abag.at>


This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this
message by mistake, please advise the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334


-Ursprüngliche Nachricht-
Von: s...@acts.hu<mailto:s...@acts.hu> [mailto:s...@acts.hu]
Gesendet: Sonntag, 26. November 2017 17:56
An: Piler User
Betreff: Re: AW: LDAP Users can see other's mails



Hmm, it's odd. Even if a user is member of a group with other users which is 
totally normal a user still shouldn't see others' emails.

Some of the addresses look like some distribution lists. Can you show me such a 
message you can see and meant for someone else?
I'm interested the headers only. (You may send it privately to my address). The 
selected messages should not belong to any distribution list you are on.

Janos

On 2017-11-23 12:09, Katterl Christian wrote:

 It seems that i can see all messages of members of the same ad-groups.


 In my case, piler would not need to take care of groups…..

 VON: Janos SUTO [mailto:s...@acts.hu]
 GESENDET: Donnerstag, 23. November 2017 09:45
 AN: Piler User
 BETREFF: Re: LDAP Users can see other's mails

 Show me the sphinx query from the mail log related to the given user.

 Janos

 -

 FROM: Katterl Christian
 SENT: Thu Nov 23 07:35:19 GMT+01:00 2017
 TO: "piler-user@list.acts.hu<mailto:piler-user@list.acts.hu>"
 SUBJECT: LDAP Users can see other's mails

 Dear all,

 i configured piler (1.3.1) on Debian (9) using LDAP authentication
 against Active Directory.

 Basically, authentication works.

 BUT:

 Successfully logged in users cannot only see their own mails, but also
 mails of other users?

 My ldap-config from config-site.php looks like this:

 $config['ENABLE_LDAP_AUTH'] = 1;

 $config['LDAP_HOST'] = 'mydomaincontroller.mydomain.myforest.tld';

 $config['LDAP_HELPER_DN'] =
 'CN=pilerldap,OU=ServicesAccounts,DC=mydomain,DC=myforest,DC=tld';

 $config['LDAP_HELPER_PASSWORD'] = 'highpressurecompressor';

 $config['LDAP_MAIL_ATTR'] = 'mail';

 $config['LDAP_ACCOUNT_OBJECTCLASS'] = 'user';

 $config['LDAP_DISTRIBUTIONLIST_OBJECTCLASS'] = 'group';

Re: AW: LDAP Users can see other's mails

[Janos SUTO]

OK, if it gives you a proper result, then case is solved. Be sure to save your 
fix in case of a future upgrade. Or I may introduce a configure option to apply 
your fix.

Janos


 Original Message 
From: Katterl Christian <c.katt...@asamer.at>
Sent: Tue Nov 28 06:35:25 GMT+01:00 2017
To: Piler User <piler-user@list.acts.hu>
Subject: AW: AW: LDAP Users can see other's mails

Hello,

maybe I have found a solution for this issue.

In /var/www/piler/model/user/auth.php

I changed the line 217, which originally looked like this:

$query = $ldap->query($ldap_base_dn, 
"(|(&(objectClass=$ldap_account_objectclass)($ldap_mail_attr=$username_prefix$username))(&(objectClass=$ldap_distributionlist_objectclass)($ldap_distributionlist_attr=$username_prefix$username)"
 . 
")(&(objectClass=$ldap_distributionlist_objectclass)($ldap_distributionlist_attr="
 . $a['dn'] . ")))", array());

To only:
  $query = $ldap->query($ldap_base_dn, 
"(|(&(objectClass=$ldap_account_objectclass)($ldap_mail_attr=$username_prefix$username)))",
 array());

I mean - I removed all the group- and distribution-list things.
I am not sure, what this else will/could cause (I am not a programmer)?

But from what I saw in a very quick test, now only my personal emails are shown.

BR, Christian


 Christian Katterl
Teamleader Technical IT

Asamer Baustoffe AG
Unterthalham Straße 2
4694 Ohlsdorf
Austria
tel +43 50 799 - 2511

mobile  +43 664 811 54 99
c.katt...@asamer.at
www.abag.at


This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this
message by mistake, please advise the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334


-Ursprüngliche Nachricht-
Von: s...@acts.hu [mailto:s...@acts.hu]
Gesendet: Sonntag, 26. November 2017 17:56
An: Piler User
Betreff: Re: AW: LDAP Users can see other's mails



Hmm, it's odd. Even if a user is member of a group with other users which is 
totally normal a user still shouldn't see others' emails.

Some of the addresses look like some distribution lists. Can you show me such a 
message you can see and meant for someone else?
I'm interested the headers only. (You may send it privately to my address). The 
selected messages should not belong to any distribution list you are on.

Janos

On 2017-11-23 12:09, Katterl Christian wrote:
> It seems that i can see all messages of members of the same ad-groups.
>
>
> In my case, piler would not need to take care of groups…..
>
> VON: Janos SUTO [mailto:s...@acts.hu]
> GESENDET: Donnerstag, 23. November 2017 09:45
> AN: Piler User
> BETREFF: Re: LDAP Users can see other's mails
>
> Show me the sphinx query from the mail log related to the given user.
>
> Janos
>
> -
>
> FROM: Katterl Christian
> SENT: Thu Nov 23 07:35:19 GMT+01:00 2017
> TO: "piler-user@list.acts.hu"
> SUBJECT: LDAP Users can see other's mails
>
> Dear all,
>
> i configured piler (1.3.1) on Debian (9) using LDAP authentication
> against Active Directory.
>
> Basically, authentication works.
>
> BUT:
>
> Successfully logged in users cannot only see their own mails, but also
> mails of other users?
>
> My ldap-config from config-site.php looks like this:
>
> $config['ENABLE_LDAP_AUTH'] = 1;
>
> $config['LDAP_HOST'] = 'mydomaincontroller.mydomain.myforest.tld';
>
> $config['LDAP_HELPER_DN'] =
> 'CN=pilerldap,OU=ServicesAccounts,DC=mydomain,DC=myforest,DC=tld';
>
> $config['LDAP_HELPER_PASSWORD'] = 'highpressurecompressor';
>
> $config['LDAP_MAIL_ATTR'] = 'mail';
>
> $config['LDAP_ACCOUNT_OBJECTCLASS'] = 'user';
>
> $config['LDAP_DISTRIBUTIONLIST_OBJECTCLASS'] = 'group';
>
> $config['LDAP_DISTRIBUTIONLIST_ATTR'] = 'member';
>
> $config['LDAP_BASE_DN'] = DC=mydomain,DC=myforest,DC=tld;
>
> $config['LDAP_AUDITOR_MEMBER_DN'] = '';
>
> $config['LDAP_ADMIN_MEMBER_DN'] = '';
>
> Any ideas?
>
> BR, Christian
>
> CHRISTIAN KATTERL
> Teamleader Technical IT
>
> ASAMER BAUSTOFFE AG
> Unterthalham Straße 2
> 4694 Ohlsdorf
> Austria
> TEL  +43 50 799 - 2511
> MOBILE +43 664 811 54 99
> EMAIL c.katt...@asamer.at
> WWW.ABAG.AT [1]
>
> This message is confidential. It may not be disclosed to, or used by,
> anyone other than the addressee. If you receive this message by
> mistake, please advise the sender.
> Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334
>
>
>
> Links:
> --
> [1] https://www.abag.at

Re: LDAP Users can see other's mails

[Janos SUTO]

Show me the sphinx query from the mail log related to the given user.

Janos


 Original Message 
From: Katterl Christian 
Sent: Thu Nov 23 07:35:19 GMT+01:00 2017
To: "piler-user@list.acts.hu" 
Subject: LDAP Users can see other's mails

Dear all,

i configured piler (1.3.1) on Debian (9) using LDAP authentication against 
Active Directory.
Basically, authentication works.

BUT:
Successfully logged in users cannot only see their own mails, but also mails of 
other users?

My ldap-config from config-site.php looks like this:

$config['ENABLE_LDAP_AUTH'] = 1;
$config['LDAP_HOST'] = 'mydomaincontroller.mydomain.myforest.tld';
$config['LDAP_HELPER_DN'] = 
'CN=pilerldap,OU=ServicesAccounts,DC=mydomain,DC=myforest,DC=tld';
$config['LDAP_HELPER_PASSWORD'] = 'highpressurecompressor';
$config['LDAP_MAIL_ATTR'] = 'mail';
$config['LDAP_ACCOUNT_OBJECTCLASS'] = 'user';
$config['LDAP_DISTRIBUTIONLIST_OBJECTCLASS'] = 'group';
$config['LDAP_DISTRIBUTIONLIST_ATTR'] = 'member';
$config['LDAP_BASE_DN'] = DC=mydomain,DC=myforest,DC=tld;
$config['LDAP_AUDITOR_MEMBER_DN'] = '';
$config['LDAP_ADMIN_MEMBER_DN'] = '';

Any ideas?

BR, Christian


Christian Katterl
Teamleader Technical IT

[cid:00d9-0001@01d36425.3b18d2bf]

Asamer Baustoffe AG
Unterthalham Stra?e 2
4694 Ohlsdorf
Austria
tel  +43 50 799 - 2511
mobile +43 664 811 54 99
email c.katt...@asamer.at
www.abag.at



This message is confidential. It may not be disclosed to, or used by, anyone 
other than the addressee. If you receive this message by mistake, please advise 
the sender.
Firmenbuch: Landesgericht Wels, FN: 407726y, ATU 68646334

Re: View all emails from a domain

[Janos SUTO]

Hello Marcelo,

well, something like that is possible. Set ENABLE_SAAS to 1, and see if it 
suits your needs.

Note that I was working on adding multitenant capabilities to piler, however at 
some point I decided to stop. I won't remove what's already done, but I don't 
intend to improve it.

If you had many domains with many customers, then piler may lack a few 
features, like separating customers' data, supporting more nodes, etc.

Janos


 Original Message 
From: Marcelo Machado <mmgom...@gmail.com>
Sent: Wed Jul 05 12:56:49 GMT+02:00 2017
To: Piler User <piler-user@list.acts.hu>
Subject: Re: View all emails from a domain



Hi Janos.

I would like an audit user to view the messages only from one domain
and another audit user only from another domain. It is possible?

Marcelo Gomes

2017-07-05 1:22 GMT-03:00 Janos SUTO <s...@acts.hu>:
> Hello,
>
> a regular user can't see all emails, however an auditor user can.
>
> Janos
> 
> From: Marcelo Machado
> Sent: Tue Jul 04 21:54:58 GMT+02:00 2017
> To: piler-user@list.acts.hu
> Subject: View all emails from a domain
>
>
>
> Hello everyone.
>
> I have a piler server receiving messages from two different domains.
>
> Is possible one regular user or auditor user see messages from all
> users only of a one
> domain?
>
> Marcelo Gomes
>

Re: View all emails from a domain

[Janos SUTO]

Hello,

a regular user can't see all emails, however an auditor user can.

Janos


 Original Message 
From: Marcelo Machado 
Sent: Tue Jul 04 21:54:58 GMT+02:00 2017
To: piler-user@list.acts.hu
Subject: View all emails from a domain



Hello everyone.

I have a piler server receiving messages from two different domains.

Is possible one regular user or auditor user see messages from all
users only of a one
domain?

Marcelo Gomes

Re: Metadata Cleanup?

[Janos SUTO]

Hello Uwe,

besides deleting from metadata table, you may delete the corresponding entries 
from rcpt table as well.

However, my personal point of view is that the metadata table serves as a 
history of emails worth to keep. Anyway it's your archive.

And one more thing. Sphinx uses the deleted=1 rows as the killlist, see 
sphinx.conf.

Janos



 Original Message 
From: "Uwe Kiewel [ML]" 
Sent: Mon Jul 03 10:52:23 GMT+02:00 2017
To: "piler-user@list.acts.hu" 
Subject: Metadata Cleanup?



Hi

hou to cleanup old metadata of deleted mails?

My current status:

mysql> select count(deleted) from metadata where deleted = 0;
++
| count(deleted) |
++
|   2918 |
++
1 row in set (0.00 sec)

mysql> select count(deleted) from metadata where deleted = 1;
++
| count(deleted) |
++
|   8517 |
++
1 row in set (0.01 sec)


Will it be save just to execute "delete from metadata where deleted = 1" 
periodically?  


Thanks
Uwe

1.3.0 is soon to be released

[Janos SUTO]

Dear piler users,

I'm about to release 1.3.0. It has a few new features,
it's mostly a maintenance release with bugfixes, minor
improvements.

tl;dr #1: 1.3.0 will be released on this week

tl;dr #2: if you use piler on FreeBSD or other non-Linux platform,
please drop me a line. There will be changes affecting you, see
the end of the email.



A short list of changes:


* removed the pdf download icon from gui

tcpdf seems to have some issues if the email has images. Until
I find a better alternative this feature is hidden from the
users.


* added HTMLpurifier to sanitize the html mime parts

* rewritten pilerpurge in python

The purge utility uses lots of SQL as well, and it's much
easier to do the job using a script language. Also, it's
half the size in lines of code.


* improved search feature

the entered text is searched in all fields (subject, body, from, ...)
unless you specify an exact keyword (eg. subject:) to narrow the search.

Thanks for Epicenter to sponsor the feature!



* improving the smtp daemon

You may or may not noticed that I've created a new branch called 
'epoll'.


The current piler daemon uses the preforking model: it spawns 10 or so
child processes that actually wait for an incoming smtp connection, and
handle the received email.

The 'epoll' branch features a separate smtp daemon for piler called
piler-smtp. It's a single process daemon using the epoll() mechanism
to process multiple connections effectively. It stores the received
emails in its queue directory which is read by piler processes. The
number of piler processes should be your number of of cpu cores + 1,
ie. 2 for a single cpu system.

As you can see the current approach uses 10+1=11 processes, however,
with epoll model uses 1+2+1=4 processes, while it can serve up to
64 parallel connections (can be set in piler.conf).

One example when the latter might be handy is receiving emails from 
o365.

Usually 10 child processes are enough, however it seems that o365 has
no flow control, ie. in case of a spike in the email volume it tries
to overwhelm piler, and if all 10 children are busy, then return an
error to the sender, which is bad news.

In such case you have two options: either to increase the number of
the piler processes (that particular user had to increase the instance
size of the virtual machine in amazon aws to run 32 piler processes).

The other option is the epoll model which is able to handle such a spike
effectively with low memory footprint.

The bottomline is that 1.3.0 will be the last release to support the
preforking model. After that I'll switch to the epoll model. One 
drawback

however, it's Linux only. I'm aware of that some of you use piler on
FreeBSD. I'll contact you to figure out something for you as well.


Best regards,
Janos

Re: Starting in Piller

[Janos SUTO]

Look at the mail log. The piler daemon syslogs every
complaint or issue it encounters. You didn't expect
this, right? :-)

Anyway my best bet is that you have a mta running
(exim, postfix, whatever) thus blocking piler's
ability to bind to port 25.

Janos


On 2017-03-03 14:52, Marcelo Soares wrote:

Dears,

Is there a way to start piler in verbose mode?

I've installed on dependencies

Apache and php7 is running:

But piler service don't come up:

Att

Marcelo Soares
2017-02-23 18:11 GMT-03:00 admin-at-extremeshok-dot-com
<ad...@extremeshok.com>:


Usually mariadb is configured to use TCP 127.0.0.1:3206 [1]

__.https://eXtremeSHOK.com
.__

On 02/23/2017 11:09 PM, Janos SUTO wrote:


Hello Marcelo,

On 2017-02-23 21:05, Marcelo Soares wrote:

Dear,
Do I have to change  mysqlsocket=/var/run/mysqld/mysqld.sock to
mariadb? (Ubuntu 16.04 MariaDB10)


unlikely. Mariadb uses just the same socket file.
Anyway start mariadb, then check if that socket file
exists.

Janos





Links:
--
[1] http://127.0.0.1:3206

Re: Starting in Piller

[Janos SUTO]

Hello Marcelo,

On 2017-02-23 21:05, Marcelo Soares wrote:

Dear,
Do I have to change  mysqlsocket=/var/run/mysqld/mysqld.sock to
mariadb? (Ubuntu 16.04 MariaDB10)


unlikely. Mariadb uses just the same socket file.
Anyway start mariadb, then check if that socket file
exists.

Janos

Re: Starting in Piller

[Janos SUTO]

Hello Marcelo,

I recommend ubuntu xenial x64 with mariadb 10.1 (and not
the stock shipped mysql 5.7 which has some utf8 and other
problems with piler).

Janos

On 2017-02-21 18:58, Marcelo Soares wrote:

Hello Everybody,

My name is Marcelo Soares, from Brazil.

I am starting on Piler, i have a Zimbra Server with 600 mailboxes and
4tb.

Which is the best linux distro to use Piller? Starting from zero.

Att

Marcelo Soares

Re: Verification Failure

[Janos SUTO]

Hello Stephan,

you exported the message, and checked the current sha256 values.

You also queried the stored sha256 sums from the metadata table.

Comparing these values we can assume that either the metadata entries have 
changed or the stored message.

Janos


 Original Message 
From: "Stephan Schöffel" <s.schoef...@aktivoptik.de>
Sent: Mon Feb 20 15:52:01 GMT+01:00 2017
To: Piler User <piler-user@list.acts.hu>
Subject: Re: Verification Failure


Hello Janos,

still don't know exactly what to compare.

Form pilertest I get:

root@mailarchiv:/tmp/mailarchive# pilertest message.txt | grep 'body digest'
body digest: 
ae927d2d6228f75a9bf8be6d96b6264287f615bf0273a170aaa50c2f30b721da

 From sha256sum I get:

root@mailarchiv:/tmp/mailarchive# sha256sum message.txt
acaf159d68e628bbd5397dcc163d700a76e306168680b8beb1bc09761cb60c44 message.txt

 From the metadata table I get:

MariaDB [piler]> select digest, bodydigest  from metadata where piler_id 
= '4000586a3e193199bb2400c9daf66a7e' ;
+--+--+
| digest   | 
bodydigest   |
+--+--+
| 3758f9e4c13b590810786c0ced2f4bbb6901fee894120da436b09c25e3e21522 | 
bbfef40739b9e1447479ef066f1d0d98bda82eba8fcdff9328df266386072ab6 |
+--+--+
1 row in set (0.00 sec)


Cheers
Stephan

Am 09.02.2017 um 20:01 schrieb Janos SUTO:
> the sha256 values you can see in pilertest output (also run sha256 
> )
> with the stored results.
>
> Janos
>
> On 2017-02-02 11:08, Stephan Schöffel wrote:
>> Which values should I compare?
>>
>> Am 01.02.2017 um 10:27 schrieb Janos SUTO:
>>
>>> Hello Stephan,
>>>
>>> no, it shouldn't. I need some troubleshooting to figure it out. 3℅
>>> verification problem is way too much.
>>>
>>> So please do the following:
>>>
>>> Get a problematic message with pilerget, save to a file, run
>>> pilertest against it, and get the values from the metadata table,
>>> and compare the results.
>>>
>>> Janos
>>>
>>> Janos
>>>
>>> -
>>> FROM: "Stephan Schöffel"
>>> SENT: Wed Feb 01 09:16:22 GMT+01:00 2017
>>> TO: Piler User
>>>
>>> SUBJECT: Re: Verification Failure
>>>
>>> Janos, actually nothing of the both has occured. Can the parser bug
>>> occur without an update? Cheers Stephan Am 31.01.2017 um 19:26
>>> schrieb Janos SUTO: Hello, On 2017-01-30 12:10, Stephan Schöffel
>>> wrote: upon trying to export emails from the archive I get about 3%
>>> "verification FAILED" messages. Unfortunately, I cannot find any log
>>> file to check what's going on. Maybe you can give me a hint. the
>>> 'verification failed' message means that the computed checksums (two
>>> sha256 hash values) are not the same that were stored at archiving
>>> time. This might be the result of a parser bug (assuming there was
>>> an upgrade in the meantime), or the message itself is actually
>>> changed. I can't tell which. Janos

Re: Verification Failure

[Janos SUTO]

the sha256 values you can see in pilertest output (also run sha256 
)

with the stored results.

Janos

On 2017-02-02 11:08, Stephan Schöffel wrote:

Which values should I compare?

Am 01.02.2017 um 10:27 schrieb Janos SUTO:


Hello Stephan,

no, it shouldn't. I need some troubleshooting to figure it out. 3℅
verification problem is way too much.

So please do the following:

Get a problematic message with pilerget, save to a file, run
pilertest against it, and get the values from the metadata table,
and compare the results.

Janos

Janos

-
FROM: "Stephan Schöffel"
SENT: Wed Feb 01 09:16:22 GMT+01:00 2017
TO: Piler User

SUBJECT: Re: Verification Failure

Janos, actually nothing of the both has occured. Can the parser bug
occur without an update? Cheers Stephan Am 31.01.2017 um 19:26
schrieb Janos SUTO: Hello, On 2017-01-30 12:10, Stephan Schöffel
wrote: upon trying to export emails from the archive I get about 3%
"verification FAILED" messages. Unfortunately, I cannot find any log
file to check what's going on. Maybe you can give me a hint. the
'verification failed' message means that the computed checksums (two
sha256 hash values) are not the same that were stored at archiving
time. This might be the result of a parser bug (assuming there was
an upgrade in the meantime), or the message itself is actually
changed. I can't tell which. Janos


--

 [1]

 STEPHAN SCHÖFFEL

BITTE BEACHTEN SIE UNSERE NEUE ADRESSE:

AKTIVOPTIK SERVICE AG
John-F.-Kennedy-Straße 26
D-55543 Bad Kreuznach

Tel.: +49 671.796467 150
Mobil: +49 151.28448 602
Fax: +49 671.796467 101
E-Mail: s.schoef...@aktivoptik.de
Web: www.aktivoptik.de [1]

Sitz der Gesellschaft: Bad Kreuznach
HRB 20838 • Amtsgericht Bad Kreuznach
Vorstand: Rolf Schneider (Vorsitzender), Ruth Weißmann, Jan Schneider
Aufsichtsratsvorsitzender: Hans-Walter Liese
USt-IdNr.: DE 269 834 610

Bankverbindung:
Sparkasse Rhein-Nahe
IBAN: DE13 5605 0180 0010 0621 80
BIC: MALADE51KRE

 [2]

 www.aktivoptik.de [1]

 [3]

Links:
--
[1] http://www.aktivoptik.de
[2]
http://media.opti.de/onlinekatalog/2017/Ausstellerdetails/aktivoptik_Service_AG/?elb=208.1100.2593.1.
[3] https://www.facebook.com/aktivoptik.de

Re: Exclude mails with a specific header value

[Janos SUTO]

Hello,

this feature is not supported. Currently your best bet is to put an MTA, eg. 
postfix in front of piler, and let it do the filtering. Or you may do the 
equivalent of this on the sending side.

Janos


 Original Message 
From: Yann Lehmann 
Sent: Wed Feb 08 14:42:21 GMT+01:00 2017
To: piler-user@list.acts.hu
Subject: Exclude mails with a specific header value


Hi

Our groupware-system sends notification mails for tasks and appointments.

All this notification mails have a specific header like 'X--Type: 
notification-mail'. As the use of the groupware-system is growing, we 
are seeing more and more of this messages.

Is there a way to avoid them to be archived by Piler ?

Thanks and best regards
Yann Lehmann

Re: Verification Failure

[Janos SUTO]

Hello Stephan,

no, it shouldn't. I need some troubleshooting to figure it out. 3℅ verification 
problem is way too much.

So please do the following:

Get a problematic message with pilerget, save to a file, run pilertest against 
it, and get the values from the metadata table, and compare the results.

Janos



Janos⁣​


 Original Message 
From: "Stephan Schöffel" <s.schoef...@aktivoptik.de>
Sent: Wed Feb 01 09:16:22 GMT+01:00 2017
To: Piler User <piler-user@list.acts.hu>
Subject: Re: Verification Failure


Janos,

actually nothing of the both has occured. Can the parser bug occur 
without an update?

Cheers
Stephan


Am 31.01.2017 um 19:26 schrieb Janos SUTO:
> Hello,
>
> On 2017-01-30 12:10, Stephan Schöffel wrote:
>>
>> upon trying to export emails from the archive I get about 3%
>> "verification FAILED" messages. Unfortunately, I cannot find any log
>> file to check what's going on. Maybe you can give me a hint.
>
> the 'verification failed' message means that the computed checksums
> (two sha256 hash values) are not the same that were stored at
> archiving time.
>
> This might be the result of a parser bug (assuming there was an upgrade
> in the meantime), or the message itself is actually changed. I can't
> tell which.
>
> Janos
>

Re: Verification Failure

[Janos SUTO]

Hello,

On 2017-01-30 12:10, Stephan Schöffel wrote:


upon trying to export emails from the archive I get about 3%
"verification FAILED" messages. Unfortunately, I cannot find any log
file to check what's going on. Maybe you can give me a hint.


the 'verification failed' message means that the computed checksums
(two sha256 hash values) are not the same that were stored at
archiving time.

This might be the result of a parser bug (assuming there was an upgrade
in the meantime), or the message itself is actually changed. I can't
tell which.

Janos

Re: Mysql 5.7

[Janos SUTO]

Hello Michael,

On 2016-10-23 22:48, Prof. Dr. Michael Schefczyk wrote:


Thank you very much for taking piler even further. While I only run a
small personal installation which would qualify for the free offer of
Mailarchiva, I still very much prefer truly open source software!


cool :-)


Currently, I run piler 1.1 on Debian Jessie with an internal standard
mysql database on the same virtual machine. In general, I am also
running a Percona XtraDB Cluster for my database needs. The cluster is
now upgraded to mysql 5.7. In the faq for piler, you do state an
incompatibility with mysql 5.7 (plus a possible fix, if I interpret it
correctly). The release notes on piler 1.2 refer to substantial sql
changes. Therefore I would like to ask if piler 1.2 is still
incompatible with mysql 5.7. If so, is the fix quoted to on the faq
page necessary and sufficient to solve this?


yes, the incompatibility is present with piler 1.2 and mysql 5.7, and 
the

fix is meant as a workaround for the utf8 issue.

Janos

Re: End Users are not seeing their email

[Janos SUTO]

You didn't provide any description what you are using,
your setup, piler version, anything.

Janos

On Tue, 11 Oct 2016, Karl Rossing wrote:



We have our email servers forwarding to mailpiler.

Using the auditor account, we were able to determine that the mail is 
arriving in mail piler however the sender is always postmaster and the 
recipient is always "undisclosed recipient"


How do we get the mail to be assigned to each user?

Thanks
Karl








CONFIDENTIALITY NOTICE:  This communication (including all attachments) is
confidential and is intended for the use of the named addressee(s) only and
may contain information that is private, confidential, privileged, and
exempt from disclosure under law.  All rights to privilege are expressly
claimed and reserved and are not waived.  Any use, dissemination,
distribution, copying or disclosure of this message and any attachments, in
whole or in part, by anyone other than the intended recipient(s) is strictly
prohibited.  If you have received this communication in error, please notify
the sender immediately, delete this communication from all data storage
devices and destroy all hard copies.

Re: AW: Access to mails with delimiter "+"

[Janos SUTO]

Hello,

On 2016-08-21 14:52, André Peters wrote:


A delimited address like me+t...@domain.tld is now shown as
"t...@domain.tld". I need to add "t...@domain.tld" to show delimited
mails in the search panel.
Anyways it shows the address correctly in the bottom panel.

Screenshots always help, I guess: https://i.imgur.com/Td8obp4.png

I gladly pay your hourly rate to implement this into the parser. I had
a look into it but my C isn't good at all. :-)


try the following: edit src/parser_utils.c, and locate the translateLine
function. It has the following nice condition around the 593th line:

if( (state->message_state == MSG_RECEIVED || state->message_state == 
MSG_FROM || state->message_state == MSG_TO
 || state->message_state == MSG_CC || state->message_state == 
MSG_RECIPIENT) && *p == '@'){ continue; }


and replace it with the following:

if( (state->message_state == MSG_RECEIVED || state->message_state == 
MSG_FROM || state->message_state == MSG_TO
 || state->message_state == MSG_CC || state->message_state == 
MSG_RECIPIENT) && (*p == '@' || *p == '+') ){ continue; }


(note that I wrapped the lines to fit my screen. You may keep it in one 
line).


Then recompile, and run pilertest against it, and check if the To 
address

is displayed correctly. Let me know if it works for you.

Janos



André

-Ursprüngliche Nachricht-
Von: Janos SUTO [mailto:s...@acts.hu]
Gesendet: Sonntag, 21. August 2016 14:32
An: Piler User <piler-user@list.acts.hu>
Betreff: Re: Access to mails with delimiter "+"

Hello,

try adding the following to config-site.php:

$config['CUSTOM_EMAIL_QUERY_FUNCTION'] = '';

function ($username = '') {
$session = Registry::get('session');
$data = $session->get("auth_data");

if($username == 'myn...@example.org') {
   array_push($data['emails'], "myname+appen...@example.org");
}

$session->set("auth_data", $data);
}

then check on the settings page if the appended email is listed.

Note that I didn't check if the plus sign  is handled correctly by the
parser and the indexer.

Janos


On 2016-08-21 12:18, André Peters wrote:

Hi,

Any idea how I can give access to mails with a delimiter? Like in
myname+appen...@example.org.

I wrote a function to include these aliases in the $data['emails']
array when logging in, but I cannot see them anyway. Any ideas?

André

Re: Archive size

[Janos SUTO]

Hello Konstantin,

it's possible to main merge once a week, but if it's the disk space then I'm 
not sure how it would help you.

Hopefully pilerpurge removes the attachments as well provided that they are not 
referenced anymore. Unless there's a bug somewhere. Anyway your best bet is to 
test it.

Janos




 Original Message 
From: Konstantin 
Sent: Fri Aug 12 13:35:33 GMT+02:00 2016
To: Piler User 
Subject: Archive size

Hello Janos,

We have an archive with following version: piler 0.1.25-master-branch,
build 869
A server has 2T hdd and 1.5T used currently.
Usually 420G is available but when indexer.main.sh running free disk space
reduced to 90G.
It is too low so i started looking for a reason what's taking up the space
on my drive.

Here are the questions i have.
1) I read a documentation for sphinx and understand why disk used so hard
during merge from dailydelta1 to main1.
Currently we have indexer.main.sh scheduled at 2am each day.
I would like to run it on Saturday only. For a week it will be ~180k emails.
Can it cause any issues during merge ?

2) We have pilerpurge sheduled. Old emails removed from search and i
thought that they removed from disk as well.
I looked for an attachments example and found this file
/var/piler/store/00/526/00/00/4000526faa5a1602877400bf63cb.a3

In piler.metadata message with
piler_id=4000526faa5a1602877400bf63cb has deleted=1
In piler.attachment i also see results for that piler_id:
*** 1. row ***
 id: 79578
 piler_id: 4000526faa5a1602877400bf63cb
 attachment_id: 3
 name: medshoppe
 type: application/octet-stream
 sig: 37cfd3b8d394d791780d2c478f98239ad0fd5b656d4e7d52c64035dea76e86de
 size: 3944
 ptr: 0
 deleted: 0

A wiki said 
what pilerpurge remove an old message.
I suppose that piler also remove attachments.
Is that correct?
Can you please confirm that piler build 869 remove old messages from disk ?

Thank you.
--
*This message was delivered using 100% recycled electrons*.

Re: Web UI Display encoded Mails

[Janos SUTO]

Hello Joe,

there shouldn't be such a flag. Can you send me an example email that looks 
wrong? And perhaps a screenshot how it looks for you.

Janos




 Original Message 
From: Joe Rady 
Sent: Mon Jul 25 11:33:15 GMT+02:00 2016
To: piler-user@list.acts.hu
Subject: Web UI Display encoded Mails

Hi Janos,

After importing a fair amount of mails, it appears that Mails being encoded by 
Lotus Notes (X-Mailer: Lotus Notes Release 8.5.3FP3 November 16, 2012) are not 
decoded (displayed) properly in the WebUI.
when i download the eml file, my mail.app decodes the Mail correctly.
are there specific flags that need to be activated for Lotus Notes?

thanks a million,
Joe

Re: existing users Not found

[Janos SUTO]

Hello,

On Wed, 13 Jul 2016, Carrie Knox wrote:


Here is my attempt to manually execute the mysql query

mysql> SELECT user.uid, isadmin, username, realname, domain, email
    -> FROM user,email
    -> WHERE user.uid='email.uid  group by user.uid'
    -> ORDER BY username
    -> ASC  LIMIT 0,20
    -> ;
+-+-+--+--+++
| uid | isadmin | username | realname | domain | email  
|
+-+-+--+--+++
|   0 |   1 | admin    | built-in piler admin | local  | admin@local    
|
|   0 |   1 | admin    | built-in piler admin | local  | auditor@local  
|
|   0 |   1 | admin    | built-in piler admin | local  | bb...@sgi.com  
|
|   0 |   1 | admin    | built-in piler admin | local  | jeff...@sgi.com    
|
|   0 |   1 | admin    | built-in piler admin | local  | k...@sgi.com   
|
|   0 |   1 | admin    | built-in piler admin | local  | lista...@sgi.com   
|
|   0 |   1 | admin    | built-in piler admin | local  | listarchive2@local 
|
|   0 |   1 | admin    | built-in piler admin | local  | listarchive@local  
|
|   0 |   1 | admin    | built-in piler admin | local  | localus...@sgi.com 
|
+-+-+--+--+++
9 rows in set, 1 warning (0.00 sec)


this doesn't look right. Drop both user and email tables, and recreate 
them, and add the default admin and auditor users (see util/db-mysql.sql 
script).


It should look like the this:

| uid | isadmin | username  | realname  
| domain| email |
+-+-+---+---+---+---+
|   0 |   1 | admin | built-in piler admin  
| local | admin@local   |
|   1 |   2 | auditor@local | builtin auditor   
| local | auditor@local |



Janos

Re: no mails can be found after imap import

[Janos SUTO]

Hello,

this is the problem:

SELECT id FROM main1,dailydelta1,delta1 WHERE       folder IN () AND
^^^

To fix it you have to login as administrator and assign some folders to 
the given user. He should relogin, and verify on the settings page that he 
has his folders assigned.


Janos


On Wed, 29 Jun 2016, Joe Rady wrote:


Hi,

a Newbie here:

I downloaded the ovf, startet it up, made some settings, most importantly the 
support for folders.
(my users are VERY fond of their imap folders)
went into the web-gui and set up some users plus domains etc.
everything looked nice.

then i executed

pilerimport -i imap.fleuchaus.com -u xx -p XxXxXx -R -P 993

inside the folder /home/piler/

it was counting the mails, and threw some error messages that some mails could 
not be imported.

i did look at the web Gui (health monitor) and found the correct number of 
received mails and also a few hundred of duplicate mails (which does make 
sense)...
also i see space projection and all that

however, when searching for mails i get none.
my search in the mails list brought some ideas, but no solution.

it appears that there are mails occupying space but they cannot be found.

mail.log has some infos that i cannot make a good clue of:

a whole bunch of lines like:

          "Jun 27 15:10:23 piler pilerimport[3815]: error: helper: execl"

which with my very limited knowledge of C tells me that something could not be 
executed?

and a large number of

"Jun 27 15:13:51 piler pilerimport[3207]: 4000577126992ee13a74008e6e435bad: 
cannot open: 4000577126992ee13a74008e6e435bad.m
Jun 27 15:13:51 piler pilerimport[3207]: 4000577126992ee13a74008e6e435bad: error 
storing message: 4000577126992ee13a74008e6e435bad.m"


later on in the day i tried importing from a different imap account
and got a lot of

"Jun 27 17:17:15 piler reindex[6042]: 4000577124f62c8a32e400d171e161bc: 
mysql_stmt_execute error: *Duplicate entry '780' for key 'PRIMARY'* (errno: 1062)"



the indexer in the crontab seems to do its job, but the log reports the same 
over and over:

"Jun 29 11:05:15 piler piler-webui[1921]: sphinx query: 'SELECT id FROM 
main1,dailydelta1,delta1 WHERE       folder IN () AND  MATCH('') ORDER BY `sent` 
DESC LIMIT 0,20 OPTION max_matches=1000' in 0.00 s, 0 hits, 0 total found

So although the health monitor indicates existing mails, the web gui cannot 
find them.

my apologies if i´ve overlooked some clues on the mailing list or on the 
website, i´ve tried to find some?


Best regards

Joe Rady
Head of IT
Fleuchaus & Gallo Partnerschaft mbB
__

Re: Problem: SSO Debian 8 Piler 1.1.1 not working (smbd 2:4.2.10+dfsg-0+deb8u3)

[Janos SUTO]

Enable debug log level in apache, then how me all the apache entries related to 
the SSO login.

Janos




 Original Message 
From: Herman Harperink 
Sent: Wed Jun 29 08:26:34 GMT+02:00 2016
To: "piler-user@list.acts.hu" 
Subject: Problem: SSO Debian 8 Piler 1.1.1 not working (smbd 
2:4.2.10+dfsg-0+deb8u3)

Hi,

After all was working fine I implemented SSO, but got stuck.
Apache give a error 500 when I call sso.php, errorlog gives me:

[auth_ntlm_winbind:error] [pid 10416] (20)Not a directory: [client 
x.x.x.x:25962] early EOF from helper

I did all checks I could fine and detect no problems (wbinfo –t/u/g works fine)

I followed http://www.mailpiler.org/wiki/current:single-sign-on


All I can find related to this is a samba bug from may this year but I seems 
unrelated since I use a later version.

Any hints on this one?

Thanks in advance.




With kind regards / Met vriendelijke groeten,

Herman Harperink
ICT Administrator





[Electromach 
B.V.]




Mail:   harper...@electromach.nl
Web:www.electromach.nl
Phone:  +31-74-2472492
Fax:+31-74-2435925


[cid:image8fc88e.JPG@750f3b96.47b12bc4]


[cid:imageefe799.JPG@35ed1877.44bfbc0f]


[cid:image66d4ba.JPG@e51a336c.47bdeb9c]


[cid:image12674c.JPG@93033b57.4c81f435]



Address
Electromach B.V.
Jan Tinbergenstraat 193
7559 SP Hengelo

Re: Piler new manual installation does not show archived mail in the search console

[Janos SUTO]

See the download section on the official site. You should really read the site.

Janos 



On Jun 22, 2016, 14:25, at 14:25, Mahmoud Ramadan 
 wrote:
>Thanks Janos , yes the newer Sphinx version solved my issue , but i
>noticed
>that the URL of Piler source code on the web site is version 1.1.1
>which is
>not the latest version , i guess the last version is piler -v
>1.2.0-master
>build 935 which can be found in the pre-installed ISO , i did a search
>to
>get the source code but i did not find it so if you can update me with
>the
>link of the latest Piler source code version i will be thankful to you
>,
>thanks in advance.
>
>
>
>Best Regards,
>
>Mahmoud Ramadan Ali
>
>Network and VOIP Specialist.
>
>Mobil: (+2) 01276877112
>
>Blog | Website
>| LinkedIn
>
>
>On Mon, Jun 20, 2016 at 9:19 AM, James Macdonell 
>wrote:
>
>>
>>
>> On 06/20/2016 04:24 AM, Mahmoud Ramadan wrote:
>>
>> msql>  select * main1,daildelta1,delta1;
>> ERROR 1064 (42000): You have an error in your SQL syntax;
>>
>>
>> I think the spirit here is to sanity check to ensure the
>indexer.delta.sh
>> and indexer.main.sh crons are doing their thing... filling sphinx
>with
>> stuff.  If the counts are zero, the crons need help.
>>
>> MySQL [piler]> select count(*) from main1; select count(*) from
>> dailydelta1; select count(*) from delta1;
>> +--+
>> | count(*) |
>> +--+
>> | 32971970 |
>> +--+
>> 1 row in set (0.80 sec)
>>
>> +--+
>> | count(*) |
>> +--+
>> |24832 |
>> +--+
>> 1 row in set (0.00 sec)
>>
>> +--+
>> | count(*) |
>> +--+
>> | 3256 |
>> +--+
>> 1 row in set (0.00 sec)
>>
>> 
>>
>> --
>> James Macdonell
>> Information Security & Emerging Technologies
>> California State University, San Bernardino
>> 5500 University Parkway
>> San Bernardino, CA 92407-2393
>> Main Line: 909/537-7262http://its.csusb.edu/
>> PGP: 0xBE849E4B
>>
>>

Re: MTA / Mail resending[Avira checked]

[Janos SUTO]

On Tue, 21 Jun 2016, Jan Novak wrote:


Dear piler user,

i have installed the piler vm and it works great.
2 questions:

1.) how can a user resend a mail in archive back to him out oft he WebGui as 
normal user?


click on the restore link in the preview pane (make sure you have set the 
smarthost properly)



2.) the log tells me, that "no mta is installed" when as example cron wants to 
send a mail with the output of pilerimport (I like to get this info mail at this first 
start to see whats going on, maybe i will disable it later).


because no mta is installed in the VM. If you need one, then install one 
for you, just make sure that it won't interfere with piler.


Janos

Re: Piler new manual installation does not show archived mail in the search console

[Janos SUTO]

You shouldn't get any error or warning when running delta indexer. What sphinx 
version do you have?

Janos



On Jun 21, 2016, 09:07, at 09:07, Mahmoud Ramadan 
<mahmoud.ramadan@gmail.com> wrote:
>I used this command to give Piler all the permissions og the sphinx
>directory root@Piler-VM:~# chmod -R 777 /var/piler/sphinx
>
>and all the file in the directory have the permissions like that:
>
>root@Piler-VM:~# ls -l /var/piler/sphinx/
>total 168
>-rwxrwxrwx 1 piler piler  32 Jun 21 02:26 dailydelta1.spa
>-rwxrwxrwx 1 piler piler   1 Jun 21 02:26 dailydelta1.spd
>-rwxrwxrwx 1 piler piler 582 Jun 21 02:26 dailydelta1.sph
>
>Also i used the command indexer.delta.sh as below but sill no search
>results in the search console.
>
>root@Piler-VM:~# su piler
>$ /usr/local/libexec/piler/indexer.delta.sh
>WARNING: IsEmpty != attribute size (32)
>
>Thanks in advance.
>
>
>Best Regards,
>
>Mahmoud Ramadan Ali
>
>Network and VOIP Specialist.
>
>Mobil: (+2) 01276877112
>
>Blog <http://hotciscolabs.blogspot.com/>| Website
><http://telecomandsecurity.com/>| LinkedIn
><http://ch.linkedin.com/pub/mahmoud-ali/99/923/421/en>
>
>On Mon, Jun 20, 2016 at 9:35 AM, Janos SUTO <s...@acts.hu> wrote:
>
>> Your cron jobs are either not running or you have a permission
>problem on
>> /var/piler/sphinx.
>>
>> So switch to user piler, run indexer.delta.sh, and verify that the
>sphinx
>> files are update and increased in size.
>>
>> Janos
>> On Jun 20, 2016, at 13:25, Mahmoud Ramadan
><mahmoud.ramadan@gmail.com>
>> wrote:
>>>
>>> Any updates ? need help to solve this issue please , thanks in
>advance.
>>>
>>> Best Regards,
>>>
>>> Mahmoud Ramadan Ali
>>>
>>> Network and VOIP Specialist.
>>>
>>> Mobil: (+2) 01276877112
>>>
>>> Blog <http://hotciscolabs.blogspot.com/>| Website
>>> <http://telecomandsecurity.com/>| LinkedIn
>>> <http://ch.linkedin.com/pub/mahmoud-ali/99/923/421/en>
>>>
>>> On Sun, Jun 19, 2016 at 1:36 AM, Mahmoud Ramadan <
>>> mahmoud.ramadan@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I did the following troubleshooting steps;
>>>>
>>>> *Checking the crontab jobs:*
>>>>
>>>> root@Piler-VM:~# crontab -l -u piler
>>>> ### PILERSTART
>>>> 5,35 * * * * /usr/local/libexec/piler/ indexer.delta.sh
>>>> 30   2 * * * /usr/local/libexec/piler/ indexer.main.sh
>>>> */15 * * * * /usr/bin/indexer --quiet tag1 --rotate
>>>> */15 * * * * /usr/bin/indexer --quiet note1 --rotate
>>>> 30   6 * * * /usr/bin/php
>/usr/local/libexec/piler/generate_stats.php
>>>> --webui /var/www/piler
>>>> */5 * * * * /usr/bin/find /var/www/piler/tmp -type f -name i.\*
>-exec rm
>>>> -f {} \;
>>>> ### PILEREND
>>>>
>>>> *Reading log files:*
>>>>
>>>> root@Piler-VM:~# tail -f /var/log/sphinxsearch/query.log
>>>> [Sat Jun 18 14:20:18.899 2016] 0.000 sec [ext2/0/ext 0 (0,1000)]
>>>> [main1,dailydelta1,delta1]
>>>> [Sat Jun 18 14:21:48.727 2016] 0.000 sec [ext2/0/ext 0 (0,1000)]
>>>> [main1,dailydelta1,delta1]
>>>>
>>>>
>>>>
>>>> *Checking permissions:*root@Piler-VM:~# ls -l /var/piler/sphinx
>>>> total 168
>>>> -rw-r--r-- 1 piler piler  32 Jun 19 03:05 dailydelta1.spa
>>>> -rw-r--r-- 1 piler piler   1 Jun 19 03:05 dailydelta1.spd
>>>> -rw-r--r-- 1 piler piler 582 Jun 19 03:05 dailydelta1.sph
>>>> -rw-r--r-- 1 piler piler   1 Jun 19 03:05 dailydelta1.spi
>>>> -rw-r--r-- 1 piler piler   0 Jun 19 03:05 dailydelta1.spk
>>>> -rw--- 1 piler piler   0 Jun 19 03:05 dailydelta1.spl
>>>>
>>>>
>>>>
>>>> *Reading log files:*root@Piler-VM:~# tail -f /var/log/mail.log
>>>> Jun 19 03:06:17 Piler-VM piler-webui[4484]: sphinx query: 'SELECT
>id
>>>> FROM main1,dailydelta1,delta1 WHEREMATCH('') ORDER BY
>`sent` DESC
>>>> LIMIT 0,1000 OPTION max_matches=1000' in 0.00 s, 0 hits, 0 total
>found
>>>>
>>>> Notice: i executed the "  select * main1,daildelta1,delta1; "
>command
>>>> from the web GUI because i tried to know how to execute it from CLI
>and
>>>> whenever i use the command from the Mysql CLI i get this error.
>>>>
>>>> root@Piler-VM:~# mysql -h localhost -u piler -p piler -P 

Re: Piler new manual installation does not show archived mail in the search console

[Janos SUTO]

Your cron jobs are either not running or you have a permission problem on 
/var/piler/sphinx.

So switch to user piler, run indexer.delta.sh, and verify that the sphinx files 
are update and increased in size.

Janos



On Jun 20, 2016, 13:25, at 13:25, Mahmoud Ramadan 
<mahmoud.ramadan@gmail.com> wrote:
>Any updates ? need help to solve this issue please , thanks in advance.
>
>Best Regards,
>
>Mahmoud Ramadan Ali
>
>Network and VOIP Specialist.
>
>Mobil: (+2) 01276877112
>
>Blog <http://hotciscolabs.blogspot.com/>| Website
><http://telecomandsecurity.com/>| LinkedIn
><http://ch.linkedin.com/pub/mahmoud-ali/99/923/421/en>
>
>On Sun, Jun 19, 2016 at 1:36 AM, Mahmoud Ramadan <
>mahmoud.ramadan@gmail.com> wrote:
>
>> Hi,
>> I did the following troubleshooting steps;
>>
>> *Checking the crontab jobs:*
>>
>> root@Piler-VM:~# crontab -l -u piler
>> ### PILERSTART
>> 5,35 * * * * /usr/local/libexec/piler/indexer.delta.sh
>> 30   2 * * * /usr/local/libexec/piler/indexer.main.sh
>> */15 * * * * /usr/bin/indexer --quiet tag1 --rotate
>> */15 * * * * /usr/bin/indexer --quiet note1 --rotate
>> 30   6 * * * /usr/bin/php /usr/local/libexec/piler/generate_stats.php
>> --webui /var/www/piler
>> */5 * * * * /usr/bin/find /var/www/piler/tmp -type f -name i.\* -exec
>rm
>> -f {} \;
>> ### PILEREND
>>
>> *Reading log files:*
>>
>> root@Piler-VM:~# tail -f /var/log/sphinxsearch/query.log
>> [Sat Jun 18 14:20:18.899 2016] 0.000 sec [ext2/0/ext 0 (0,1000)]
>> [main1,dailydelta1,delta1]
>> [Sat Jun 18 14:21:48.727 2016] 0.000 sec [ext2/0/ext 0 (0,1000)]
>> [main1,dailydelta1,delta1]
>>
>>
>>
>> *Checking permissions:*root@Piler-VM:~# ls -l /var/piler/sphinx
>> total 168
>> -rw-r--r-- 1 piler piler  32 Jun 19 03:05 dailydelta1.spa
>> -rw-r--r-- 1 piler piler   1 Jun 19 03:05 dailydelta1.spd
>> -rw-r--r-- 1 piler piler 582 Jun 19 03:05 dailydelta1.sph
>> -rw-r--r-- 1 piler piler   1 Jun 19 03:05 dailydelta1.spi
>> -rw-r--r-- 1 piler piler   0 Jun 19 03:05 dailydelta1.spk
>> -rw--- 1 piler piler   0 Jun 19 03:05 dailydelta1.spl
>>
>>
>>
>> *Reading log files:*root@Piler-VM:~# tail -f /var/log/mail.log
>> Jun 19 03:06:17 Piler-VM piler-webui[4484]: sphinx query: 'SELECT id
>FROM
>> main1,dailydelta1,delta1 WHEREMATCH('') ORDER BY `sent` DESC
>LIMIT
>> 0,1000 OPTION max_matches=1000' in 0.00 s, 0 hits, 0 total found
>>
>> Notice: i executed the "  select * main1,daildelta1,delta1; " command
>from
>> the web GUI because i tried to know how to execute it from CLI and
>whenever
>> i use the command from the Mysql CLI i get this error.
>>
>> root@Piler-VM:~# mysql -h localhost -u piler -p piler -P 9306
>> mysql> show databases;
>> ++
>> | Database   |
>> ++
>> | information_schema |
>> | piler  |
>> ++
>> 2 rows in set (0.00 sec)
>>
>> mysql> use piler;
>> Database changed
>> mysql>  select * main1,daildelta1,delta1;
>> ERROR 1064 (42000): You have an error in your SQL syntax; check the
>manual
>> that corresponds to your MySQL server version for the right syntax to
>use
>> near 'main1,daildelta1,delta1' at line 1
>>
>> Thanks in advance.
>>
>> Best Regards,
>>
>> Mahmoud Ramadan Ali
>>
>> Network and VOIP Specialist.
>>
>> Mobil: (+2) 01276877112
>>
>> Blog <http://hotciscolabs.blogspot.com/>| Website
>> <http://telecomandsecurity.com/>| LinkedIn
>> <http://ch.linkedin.com/pub/mahmoud-ali/99/923/421/en>
>>
>> On Sat, Jun 18, 2016 at 8:08 AM, Janos SUTO <s...@acts.hu> wrote:
>>
>>>
>>> as with any software if you encountered a problem, you should check
>the
>>> logs. Piler logs everything to the mail log, so be sure to read it.
>>>
>>> To address your problem, if you get no search results, then check
>the
>>> sphinx database. Don't try to select from v_messages, because it's
>in
>>> the mysql database (not in sphinx).
>>>
>>> Rather, try this query:
>>>
>>> select * main1,daildelta1,delta1;
>>>
>>> If it's empty, then check why the sphinx database is not updated.
>Missing
>>> cron entries? Bad permissions on /var/piler/sphinx?
>>>
>>> So I'd expect some troubleshooting on your behalf.
>>>
>>> Janos
>>>
>>> PS: don't call me/us your d

Re: AW: Error when using " Download the selected PDF " feature in the end user web GUI.

[Janos SUTO]

On 2016-06-14 19:59, André Peters wrote:

The directory is probably just not writeable by the user who runs
piler.


almost. The directory must be writable by the webserver, eg. www-data,
apache, etc. Btw. note that the installer takes care of it.

Janos




VON: Mahmoud Ramadan [mailto:mahmoud.ramadan@gmail.com]
GESENDET: Dienstag, 14. Juni 2016 19:56
AN: ad...@extremeshok.com
CC: piler-user@list.acts.hu
BETREFF: Re: Error when using " Download the selected PDF " feature in
the end user web GUI.

what do you mean by " check your permissions ? I'm a network and VoIP
specialist with Linux experience , I'm deploying open source solutions
like Piler for companies , Still waiting for someone to answer my
question , thanks in advance.

Best Regards,

Mahmoud Ramadan Ali

Network and VOIP Specialist.

Mobil: (+2) 01276877112

Blog [1]| Website [2]| LinkedIn [3]

On Tue, Jun 14, 2016 at 2:37 PM, admin-at-extremeshok-dot-com
 wrote:


Network and VOIP Specialist. ? really .. check your permissions

__.https://eXtremeSHOK.com
.__

On 14-Jun-16 2:02 PM, Mahmoud Ramadan wrote:


Network and VOIP Specialist.




Links:
--
[1] http://hotciscolabs.blogspot.com/
[2] http://telecomandsecurity.com/
[3] http://ch.linkedin.com/pub/mahmoud-ali/99/923/421/en

Re: Single mail stuck in queue, smtp times out

[Janos SUTO]

Hello Andre,

what piler version do you have? Also try increase the verbosity level to see at 
which stage the connection is stuck.

And finally it would be useful to see an ngrep output for only this specific 
email.

Anyway you may zip this email and send it to me.

Janos 



On Jun 3, 2016, 12:39, at 12:39, "André Peters"  wrote:
>
>Hi,
>
>I have a single 161kb message stuck in my postfix queue. Any time I try
>to flush it, the smtp connection to piler times out.
>It is just this one message. I have about 20 mails per minute and none
>failed in months. :-)
>
>Maybe the message contains some weird formatting which piler does not
>like?
>
>Janos, I may send you this message. 
>
>Syslog looks fine. Restarted all services, too. I have 10 workers
>online, so this is not a critical problem. The other mails are
>processed fine. 
>
>
>André
>
>Von meinem iPhone gesendet

Re: Diskspace of archive

[Janos SUTO]

Piler compresses and deduplicates emails. 1.2GB is the raw size, 165 MB is the 
size on the disk.

Janos



On Jun 2, 2016, 16:46, at 16:46, Yann Lehmann  wrote:
>
>Hi
>
>On the stats page of my test-installation of Piler, the 'Archive size' 
>is showed as '1.2 GB (165 MB)'.
>
>A 'su -sh /var/piler' returns 219M.
>
>Where does the difference come from ?
>
>Thanks and regards
>Yann Lehmann

Re: Some questions about piler (user rights, ignored messages, etc)

[Janos SUTO]

Hello,

On 2016-05-24 16:55, Yann Lehmann wrote:

Hi,

I am new to piler and after some trouble, I have a working system that
I will test in the next days.

I have some questions about the use of piler:

1) User rights
I would like one user being able to see all messages. Do I have to
grant this user rights on all existing adresses and add new users when
these are created ? Or is there a "magic" way to grant access to all
adresses ?


just grant auditor role to this user, and you are done.



2) Ignored messages
I have seen on the "health page", that a lot of messages are ignored.
For now, I have only one user. Could it be that messages not for this
user are ignored ? If yes, is there a setting to archive all messages,


no. Piler archives each email it receives. A message can be ignored if 
it
matches an archiving rule or if it's a duplicate (based on the 
message-id)

If you have lots of messages WITHOUT a valid message-id, then they would
be discarded as well.


even if users or addresses do not exist on the piler system ?

3) If a user wants a message sent back to his mailbox, I guess piler
will send the message to the 'smarthost' defined in the
'config-site.php' file. Our mailserver only accepts mails from


that's right


authenticated users. Is it possible to send credentials with the
message and how ?


not at the moment


4) Is there a list which explains the parameters of the 'piler.conf' 
file ?


see src/examples.conf

Janos

Re: piler on arm-based device

[Janos SUTO]

Hello Yann,

I've never tried to port piler to ARM. However, I'm curious whether
it can be compiled at all. Unfortunately I don't have access to such
a device. If you can't make it, and can afford to provide some access
to the device, then I may try to do the job. No promises, though.

Janos


On 2016-04-19 18:23, Yann Lehmann wrote:

Hi

I am setting up a small arm-based computer (Cubietruck) as a "test 
lab".


I would like to test it as a mail server and archiving solution
(though I have read it should not be run on the same machine than the
mail server).

Is it possible to compile piler for arm ?

Thanks and regards
Yann Lehmann

PS: sorry if the question has already been asked and answered

Re: Folders

[Janos SUTO]

Hello Raimer,

On 2016-04-09 14:12, Raimar Sandner wrote:

Hi all,

I'm reaching out one more time, maybe someone can help me with this 
folders-

problem described below.

Cheers
Raimar

On Montag, 1. Februar 2016 13:51:51 CEST Raimar Sandner wrote:

I'm having trouble making folders work in piler. I do realize this is 
not an
'official' feature, but it might be exactly the thing I need. This is 
my
use case: I want to archive the emails of two users, Alice and Bob, 
and
both should have access only to their own emails. I won't be using 
smtp,
just pilerimport from time to time. Ideally I would not like to bother 
with

the various email addresses or domains the users have.

To me this looks just like the feature folders could offer, using

pilerimport -F alice ...

or

pilerimport -F bob ...

However I cannot make this work the way I want, I'm confused how to 
manage
the folders. I deleted everything, databases and sphinx index to have 
a
fresh start as described in the FAQ, and I edited the configuration 
files
to enable folders. Then I imported a few mails with pilerimport -F 
alice,

but I cannot see them as auditor. I tried to create a regular user
alice@local and to give this user permission to the folder alice. As 
admin,
on the 'edit user' page I see 'Search folders' and 'Folders', but when 
I

enter 'alice' to either field this does not seem to have any visible
effect, when the user settings are loaded again both fields are empty. 
Also

I'm not sure if I have to create the folders somewhere before actually
using them. In the admin account there is no folder menu.

Sorry if I got this all wrong. Could you maybe explain how it is 
supposed to

work?


the folder idea may be used to override the usual permissions based on 
the sender
or recipient addresses. Thus it's no more any use if you are an auditor, 
you need

specific access rights for the given folder(s).

So login as admin, create a user, and assign him either alice or bob 
folder.
Then login as this user, and verify that you can actually see this 
folder on

the settings page.

Finally click on the search button, and show me the sphinx query (see 
the maillog).


Janos

Re: Pilerpurge and attachments

[Janos SUTO]

Hello Konstantin,

the 0.1.25 release was an interim build between two major
releases, and I don't plan to support fixes for that, sorry.

I suggest to upgrade to either 1.1.1, and I believe that
this issue would be gone.

Janos

On 2016-04-04 17:02, Konstantin wrote:

Hello Janos

I found pilerpurge remove attachment even if it has another attachment
depends on it (table `attachment`, id match value in ptr)

Is it possible:
- patch pilerpurge and recompile it to fix such behavior
- or do not remove attachments at all?

piler 0.1.25-master-branch, build 857

Thank you.

RE: 3 Message with wrong 2036 date

[Janos SUTO]

Hello Edwin,

can you show me the headers for these emails? In Tim's case it turned 
out

there were some bogus Date: headers, and we managed to fix it with a
manual sphinx update.

Janos

On 2016-02-18 17:17, Nichols, Edwin (BeneFACT) wrote:

Strangely, I have the same issue with 8 emails. The emails themselves
show a date of 2010, but the Piler interface shows them as being from
2036.  There may be a bug?

EDWIN NICHOLS  |  _Director – IT & Operations_



' 416-360-SRED (7733) ext.161

' Vancouver: 604-628-9870

' Calgary: 403-775-7565

' Toll Free: 1-855-TAX-BACK (829-2225)

, 647-689-3127

*  edwin.nich...@benefact.ca

8 WWW.BENEFACT.CA [1]

P PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL

-

Privacy Disclaimer - Français à suivre
This e-mail message (including attachments, if any) is intended for
the use of the individual or entity to which it is addressed and may
contain information that is privileged, proprietary, confidential. If
you are not the intended recipient, you are notified that any
dissemination, distribution, or copy of this communication is strictly
prohibited. If you have received this communication in error, please
notify the sender and erase this e-mail message immediately.

Déclaration de confidentialité
Le présent courrier électronique (y compris les pièces qui y sont
annexées, le cas échéant) s'adresse au destinataire indiqué et
peut contenir des renseignements de caractère privé ou confidentiel.
Si vous n'êtes pas le destinataire de ce document, nous vous
signalons qu'il est strictement interdit de le diffuser, de le
distribuer ou de le reproduire. Si ce message vous a été transmis
par erreur, veuillez en informer l'expéditeur et le supprimer
immédiatement.

FROM: Tim Stumbo [mailto:timstu...@gmail.com]
SENT: Thursday, February 18, 2016 9:45 AM
TO: piler-user@list.acts.hu
SUBJECT: 3 Message with wrong 2036 date

I have 3 messages that always appear at the top of the search field
due to them having an incorrect date. The year on the messages is
2036. I would like to either delete these 3 e-mails or fix the index
for them with the corrected date.

Could someone help me with this issue?

Thanks

Links:
--
[1] http://www.benefact.ca/

Re: Too Many Duplicate Messages With " message-id: null "

[Janos SUTO]

On 2016-02-29 12:06, Mahmoud Ramadan wrote:

Hi dears
I'm getting too many duplicated messages on Piler and by checking my
logs i found that the messages are received by Piler without message
ID (null) so Piler discards them.

Is there is any way to force Piler to accept duplicated messages and
skip message ID checking ?!  Thanks in advance.


set archive_emails_not_having_message_id=1 in piler.conf, then reload 
piler.


Janos

Re: Some Piler Messages are displayed with " Message Failed Verification ".

[Janos SUTO]

On 2016-02-29 13:37, Mahmoud Ramadan wrote:

Update: I noticed something weird ! When i changed the Piler contact
settings in Exchange to enable " Use MAPI rich text format" then all


please don't, as it forces exchange to transform the email, and this
practice may contradict with some regulations.


the messages received in rich text format are label as " Message is
Verified " BUT off course can NOT be displayed well as Piler does NOT
support archiving E-Mail in rich text format.


actually, piler can process rtf attachments by using unrtf.


And when i switch the setting again to disable " Use MAPI rich text
format" , the archived messages can be displayed fine BUT with "
Message Failed Verification ".


I'd like to get a complete non sensitive email. Select a message,
click on the download link, and let me see it. Also please confirm
that you have upgraded the gui as well.

Janos

SECURITY ALERT: the drown attack

[Janos SUTO]

Dear piler-users,

you probably have heard about the drown attack against
(see https://drownattack.com/ for more) tls/ssl.

In a nutshell it targets services (http, smtp, imap, pop3, etc)
allowing ssl v2 connections as well.

Piler 1.1.1 uses TLSv1_client_method() and TLSv1_server_method()
respectively. According to the openssl manual they force piler
to understand only the TLSv1 protocol, so piler 1.1.1 should be
safe.

However piler 1.1.0 (and older versions) uses SSLv3_client_method()
and SSLv23_server_method(). It means that pilerimport is safe from
the drown attack, since it doesn't understand SSL v2.
However the piler daemon (which understands SSL v2 as well) is
vulnerable to the drown attack.


Solution: upgrade to 1.1.1

Workaround: none. You may mitigate the issue if you have <= 1.1.0
if you allow connections only from trusted smtp clients.


Janos