Re: S3 Object Storage

[André Peters]

Hi,

I think it is important to have an enterprise solution and I would say S3 is a 
great feature to begin with. 

We need to keep Piler alive. Open Source projects do not live from warm words 
or "it is good for your reputation when we use your software". :) Piler is 
probably the best open source archiver. Let us not lose it. 

Thank you, Janos!

Mit besten Grüßen
André Peters

> Am 05.04.2020 um 13:16 schrieb "s...@acts.hu" :
> 
> 
> 
> Hello Ryan,
> 
>> On 2020-04-05 10:55, Ryan Blenis wrote:
>> I see the enterprise version you offer it says one of the features
>> piler has is S3 object storage, which looks like it was originally
>> requested on the mailing list in 2018:
>> https://www.mail-archive.com/piler-user@list.acts.hu/msg01335.html
>> Does this mean that it is fully implemented now? And if so, will we
> 
> yes, it's fully implemented by now.
> 
> 
>> ever see this in the open source version? Given the lower prices on S3
>> compatible storage versus block storage and the unchanging nature of
>> the files, this would be great to have as an option that can reduce
>> hosting costs from what is now being used for [comparatively]
>> expensive block storage.
> 
> 
> well, I've been thinking about it for a while, and still not decided,
> it's a difficult business decision, so let me share my current position
> on the matter.
> 
> TL;DR: not in the near future.
> 
> I'm really proud that the open source edition has gained a pretty nice
> traction, and I decided to somehow monetize its popularity.
> 
> So I've forked piler and built an enterprise edition along with some 
> commercial
> services. To make the enterprise edition successful, I must offer some 
> features
> that are worth to pay for that is not available for free in the open source 
> edition.
> 
> I believe that the S3 storage support you asked for is such an attractive 
> feature,
> so I'll keep it for the enterprise edition only, not sure how long.
> 
> However, I'm well aware of that the S3 costs are significantly lower than 
> block
> storage costs at cloud hosting companies. I encourage you to try the 
> enterprise
> edition, and see if it works for you. If so, and you like it, then perhaps we
> could figure out something viable to make the transition from open source to
> enterprise. I'd like to emphasize that I don't want to force you in this 
> direction,
> and it's totally ok to decline.
> 
> Let me know what you (and other piler users as well) think about the matter.
> 
> Janos
> 

Re[2]: AW: Access to mails with delimiter "+"

[André Peters]

Hi,

Yes, this worked! Though I still need to know the tags my users are 
using and assigning them these addresses.


Can this be done by automatically? Like assigning 
user+whate...@domain.tld automatically to u...@domain.tld?


Another idea to fix this is to extend the PCRE pattern to add a 
X-Envelope-To Header:


/(.*)(\+([^+@]*)\@)(.*)/i   prepend X-Envelope-To: $1@$4
/(.*)/   prepend X-Envelope-To: $1

The first prepend will only be made on a match.

Regarding older mails which were already archived a solutioni to 
auto-assign these tagged mails would be better, I guess.


André

-- Originalnachricht --
Von: "Janos SUTO" <s...@acts.hu>
An: "Piler User" <piler-user@list.acts.hu>
Gesendet: 21.08.2016 17:39:34
Betreff: Re: AW: Access to mails with delimiter "+"


Hello,

On 2016-08-21 14:52, André Peters wrote:


A delimited address like me+t...@domain.tld is now shown as
"t...@domain.tld". I need to add "t...@domain.tld" to show delimited
mails in the search panel.
Anyways it shows the address correctly in the bottom panel.

Screenshots always help, I guess: https://i.imgur.com/Td8obp4.png

I gladly pay your hourly rate to implement this into the parser. I had
a look into it but my C isn't good at all. :-)


try the following: edit src/parser_utils.c, and locate the 
translateLine

function. It has the following nice condition around the 593th line:

if( (state->message_state == MSG_RECEIVED || state->message_state == 
MSG_FROM || state->message_state == MSG_TO
 || state->message_state == MSG_CC || state->message_state == 
MSG_RECIPIENT) && *p == '@'){ continue; }


and replace it with the following:

if( (state->message_state == MSG_RECEIVED || state->message_state == 
MSG_FROM || state->message_state == MSG_TO
 || state->message_state == MSG_CC || state->message_state == 
MSG_RECIPIENT) && (*p == '@' || *p == '+') ){ continue; }


(note that I wrapped the lines to fit my screen. You may keep it in one 
line).


Then recompile, and run pilertest against it, and check if the To 
address

is displayed correctly. Let me know if it works for you.

Janos



André

-Ursprüngliche Nachricht-
Von: Janos SUTO [mailto:s...@acts.hu]
Gesendet: Sonntag, 21. August 2016 14:32
An: Piler User <piler-user@list.acts.hu>
Betreff: Re: Access to mails with delimiter "+"

Hello,

try adding the following to config-site.php:

$config['CUSTOM_EMAIL_QUERY_FUNCTION'] = '';

function ($username = '') {
$session = Registry::get('session');
$data = $session->get("auth_data");

if($username == 'myn...@example.org') {
   array_push($data['emails'], "myname+appen...@example.org");
}

$session->set("auth_data", $data);
}

then check on the settings page if the appended email is listed.

Note that I didn't check if the plus sign  is handled correctly by the
parser and the indexer.

Janos


On 2016-08-21 12:18, André Peters wrote:

Hi,

Any idea how I can give access to mails with a delimiter? Like in
myname+appen...@example.org.

I wrote a function to include these aliases in the $data['emails']
array when logging in, but I cannot see them anyway. Any ideas?

André

AW: Access to mails with delimiter "+"

[André Peters]

Hi,

wow, that was frustrating ... I forgot to add the "$session->set("auth_data", 
$data);" part in the new function. But that did not completely fix the issue:

A delimited address like me+t...@domain.tld is now shown as "t...@domain.tld". 
I need to add "t...@domain.tld" to show delimited mails in the search panel.
Anyways it shows the address correctly in the bottom panel.

Screenshots always help, I guess: https://i.imgur.com/Td8obp4.png

I gladly pay your hourly rate to implement this into the parser. I had a look 
into it but my C isn't good at all. :-)

André

-Ursprüngliche Nachricht-
Von: Janos SUTO [mailto:s...@acts.hu] 
Gesendet: Sonntag, 21. August 2016 14:32
An: Piler User <piler-user@list.acts.hu>
Betreff: Re: Access to mails with delimiter "+"

Hello,

try adding the following to config-site.php:

$config['CUSTOM_EMAIL_QUERY_FUNCTION'] = '';

function ($username = '') {
$session = Registry::get('session');
$data = $session->get("auth_data");

if($username == 'myn...@example.org') {
   array_push($data['emails'], "myname+appen...@example.org");
}

$session->set("auth_data", $data);
}

then check on the settings page if the appended email is listed.

Note that I didn't check if the plus sign  is handled correctly by the parser 
and the indexer.

Janos


On 2016-08-21 12:18, André Peters wrote:
> Hi,
> 
> Any idea how I can give access to mails with a delimiter? Like in
> myname+appen...@example.org.
> 
> I wrote a function to include these aliases in the $data['emails'] 
> array when logging in, but I cannot see them anyway. Any ideas?
> 
> André


smime.p7s
Description: S/MIME cryptographic signature

AW: Error when using " Download the selected PDF " feature in the end user web GUI.

[André Peters]

The directory is probably just not writeable by the user who runs piler.

 

Von: Mahmoud Ramadan [mailto:mahmoud.ramadan@gmail.com] 
Gesendet: Dienstag, 14. Juni 2016 19:56
An: ad...@extremeshok.com
Cc: piler-user@list.acts.hu
Betreff: Re: Error when using " Download the selected PDF " feature in the end 
user web GUI.

 

what do you mean by " check your permissions ? I'm a network and VoIP 
specialist with Linux experience , I'm deploying open source solutions like 
Piler for companies , Still waiting for someone to answer my question , thanks 
in advance. 




Best Regards,

Mahmoud Ramadan Ali

Network and VOIP Specialist.

Mobil: (+2) 01276877112

Blog   | Website  
 | LinkedIn 
 

 

On Tue, Jun 14, 2016 at 2:37 PM, admin-at-extremeshok-dot-com 
 > wrote:

Network and VOIP Specialist. ? really .. check your permissions
 
__.https://eXtremeSHOK.com 
.__
 

On 14-Jun-16 2:02 PM, Mahmoud Ramadan wrote:

Network and VOIP Specialist.

 

 

AW: Single mail stuck in queue, smtp times out

[André Peters]

Hi Janos,

 

I have piler 1.1.1 running.

 

I increased verbosity to 99 and got this:

 

Jun  3 18:35:26 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
sent: 220 archive.example.de ESMTP

Jun  3 18:35:26 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
got: EHLO hostname.example.de

Jun  3 18:35:26 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
sent: 
250-archive.example.de#015#012250-PIPELINING#015#012250-STARTTLS#015#012250-SIZE#015#012250
 8BITMIME

Jun  3 18:35:26 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
got: STARTTLS

Jun  3 18:35:26 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
starttls request from client

Jun  3 18:35:26 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
sent: 220 Ready to start TLS

Jun  3 18:35:26 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
waiting for ssl handshake

Jun  3 18:35:26 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
SSL_accept() finished

Jun  3 18:35:27 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
got: EHLO hostname.example.de

Jun  3 18:35:27 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
sent: 250-archive.example.de#015#012250-PIPELINING#015#012250-SIZE#015#012250 
8BITMIME

Jun  3 18:35:27 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
got: MAIL FROM:<u...@other.de> SIZE=163839

Jun  3 18:35:27 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
got: RCPT TO:<arch...@archive.example.de>

Jun  3 18:35:27 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
got: DATA

Jun  3 18:35:27 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
sent: 250 Ok#015#012250 Ok#015#012354 Send mail data; end it with .

 

After some time:

 

Jun  3 18:45:27 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
sent: 421 archive.example.de Error: timed out

Jun  3 18:45:27 hostname piler[29727]: 40005751b1d83a1c88940062b5c2da71: 
removing stale files: 40005751b1d83a1c88940062b5c2da71, 
40005751b1d83a1c88940062b5c2da71.m

Jun  3 18:45:27 hostname piler[29727]: processed 0 messages

 

Well, I just disabled TLS to get some useful ngrep output… and guess what: 
piler was able to process the message.

 

I am running piler on the same machine and don’t really need TLS anyway. But 
that’s strange.

 

Bye

André 

 

 

Von: Janos SUTO [mailto:s...@acts.hu] 
Gesendet: Freitag, 3. Juni 2016 16:28
An: Piler User <piler-user@list.acts.hu>
Betreff: Re: Single mail stuck in queue, smtp times out

 

Hello Andre,

what piler version do you have? Also try increase the verbosity level to see at 
which stage the connection is stuck.

And finally it would be useful to see an ngrep output for only this specific 
email.

Anyway you may zip this email and send it to me.

Janos 

On Jun 3, 2016, at 12:39, "André Peters" <andre.pet...@debinux.de 
<mailto:andre.pet...@debinux.de> > wrote:


Hi,

I have a single 161kb message stuck in my postfix queue. Any time I try to 
flush it, the smtp connection to piler times out.
It is just this one message. I have about 20 mails per minute and none failed 
in months. :-)

Maybe the message contains some weird formatting which piler does not like?

Janos, I may send you this message. 

Syslog looks fine. Restarted all services, too. I have 10 workers online, so 
this is not a critical problem. The other mails are processed fine. 


André

Von meinem iPhone gesendet

Single mail stuck in queue, smtp times out

[André Peters]

Hi,

I have a single 161kb message stuck in my postfix queue. Any time I try to 
flush it, the smtp connection to piler times out.
It is just this one message. I have about 20 mails per minute and none failed 
in months. :-)

Maybe the message contains some weird formatting which piler does not like?

Janos, I may send you this message. 

Syslog looks fine. Restarted all services, too. I have 10 workers online, so 
this is not a critical problem. The other mails are processed fine. 


André

Von meinem iPhone gesendet

Re: piler unrtf process is hammering my CPU

[André Peters]

Hi, 
I am on my cell phone, so I will keep it short. 
I see there were bug reports for unrtf  about this issue a few years back. The 
devs answered in December 2014 that they fixed it, one is here: 
http://savannah.gnu.org/bugs/?34677
Can you check your version? You can get the sources right here:
http://ftp.gnu.org/gnu/unrtf/?C=M;O=A

Regards, 
André 

Am 30. April 2015 17:27:13 MESZ, schrieb Jeff Larsen jlar...@gmail.com:
In the last 24 hours I've been seeing the load and CPU on my Piler
system get out of control. I'm finding one or more processes owned by
the piler user with the command unrtf that is pegging the CPU.

The man page says this a converter for Rich Text Format. I would guess
that perhaps piler is decoding attachments so they can be searchable.

That's all well and good, but how to I track down these messages that
unrtf is choking on? And then what do I do with them?

The ps command shows me this:

/usr/bin/unrtf --text 40005542367a08ccec0c00f0dee541ab.a1.bin

I can kill the process, but eventually the problem returns.

Is piler returning to finish up the message from the killed process,
or do we have a series of problematic attachments coming in?

I am suspicious of a new contact of mine that is sending mail with the
dreaded winmail.dat attachments. I've given her info on how to stop
it, but she may not cooperate. It's not something I've seen much of
lately, maybe she has a really old Outlook version. In any case,
that's just my guess at the root of this issue.

Full disclosure: The piler store folder is on NFS, which sometimes
gives us trouble, but this particular issue is a new one. I've been
running piler for a year or more.

Piler version is 1.1.0.

Thanks,

Jeff

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

Workaround to hide X-Envelope-To from users

[André Peters]

Hi,

Another dirty workaround to hide X-Envelope-To from other recipients 
than the piler archive:


/etc/postfix/master.cf:
smtp  unix  -   -   -   -   -   smtp
  -o smtp_header_checks=pcre:/etc/postfix/x-remove-envelope-to
	smtp_keep_envelope_to  unix  -   -   -   -   - 
  smtp

lmtp  unix  -   -   -   -   -   lmtp
  -o lmtp_header_checks=pcre:/etc/postfix/x-remove-envelope-to

/etc/postfix/transport_maps:
piler.domain.tldsmtp_keep_envelope_to:[mypilerhost]:25

/etc/postfix/x-remove-envelope-to:
/^X-Envelope-To/IGNORE

This uses smtp_keep_envelope_to as transport to piler.
Everything else is sent via smtp (when default_transport is smtp, of 
course).
smtp then removes the X-Envelope-To header. Yes, I add it and then 
remove it. :-/


Just wanted to share this idea...

André



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Web-GUI shows piler archive mail address in To

[André Peters]

Absolutely!
I was a bit confused at first, too.
Though it took me only a few seconds on Google to learn its use... :-)

André

Am 26.03.2015 um 12:52 schrieb Janos SUTO:


Hello,

On 2015-03-26 12:22, André Peters wrote:


Did you set hostid in piler.conf to piler.local/your SMTP domain?


I'm about to improve the config file like this:

; SMTP HELO identification string
; this should be the FQDN part of the email address
; where you copy emails, eg. arch...@piler.yourdomain.com -
piler.yourdomain.com
hostid=piler.yourdomain.com

Does it make it clearer?

Btw. the fix (after fixing the hostid parameter) is to run the
following query, then the gui won't show this bogus address in the gui:

delete from rcpt where `to`='archive@piler.internal';


Janos






smime.p7s
Description: S/MIME Cryptographic Signature