Re: [exim] Rspamd-Proxy error with exim

[Felix Schwarz via Exim-users]

Hi,

Am 26.06.2018 um 21:50 schrieb Andrew Lewis via Exim-users:
>> If it is correct that rspamd (in current versions) not longer operates
>> with Exim then we should cease claiming it does (until and if
>> someone develops new support), and may as well remove the nonfunctional
>> code.
> 
> That's not correct. I still use Rspamd with Exim and have been upgrading
> through the various releases. It seems there has been some terrible
> miscommunication.
> 
> The Rspamd proxy is an optional interface to Rspamd- Exim users doesn't need
> it. That said I have tested it with the stable release of Rspamd and found no
> problems (proxying of the legacy protocol works as far as I see).

Thanks for pointing it out. Just yesterday I compiled the latest rspamd to
check if the main daemon is actually incompatible with Exim now but didn't get
to post my results. Sorry for not speaking up sooner but I was not 100% sure
and did not want to spread false information.

So while the old rspamd protocol might be eliminated in the future it still
works right now. Long term the ideal solution is of course to support HTTP+json.

Felix

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Heiko Schlittermann via Exim-users]

Andrew Lewis via Exim-users  (Di 26 Jun 2018 21:50:31 
CEST):
…
> Can we revive Rspamd support please? I expect a non-trivial amount of people
> are actively using it.

I'm more than happy if we can continue (native) rspamd support.
As long as rspamd supports the legacy SpamAssassin-derived protocol I
see capable of supporting nativly in Exim, even with my limited
ressources.

If the rspamd developer(s) decide to drop that legacy protocol support,
I can't promise anything for the near future.

Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
-- 
 SCHLITTERMANN.de  internet & unix support -
 Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
 gnupg encrypted messages are welcome --- key ID: F69376CE -
 ! key id 7CBF764A and 972EAC9F are revoked since 2015-01  -


signature.asc
Description: PGP signature
-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Andrew Lewis via Exim-users]

Hi,

I'm sorry about breaking the thread, I was not on the list.


If it is correct that rspamd (in current versions) not longer operates
with Exim then we should cease claiming it does (until and if
someone develops new support), and may as well remove the nonfunctional
code.


That's not correct. I still use Rspamd with Exim and have been  
upgrading through the various releases. It seems there has been some  
terrible miscommunication.


The Rspamd proxy is an optional interface to Rspamd- Exim users  
doesn't need it. That said I have tested it with the stable release of  
Rspamd and found no problems (proxying of the legacy protocol works as  
far as I see).


It seems there was something invalid about the user's configuration:  
https://github.com/vstakhov/rspamd/issues/2289


Can we revive Rspamd support please? I expect a non-trivial amount of  
people are actively using it.


Best,
-AL.


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Jan Ingvoldstad via Exim-users]

On Wed, Jun 27, 2018 at 10:00 AM, Heiko Schlittermann via Exim-users <
exim-users@exim.org> wrote:

>
> Personally I'd like to see rspamd support in Exim, but currently my
> ressources are limited (ressources hack the support into Exim), and I do
> not feel comfortable with bloating Exim by integrating a JSON parser.
> But maybe I'm wrong.
>

It's a nice-to-have, but there is already a "bloated" version of Exim that
can do this, by embedding Perl.

This should be an acceptable work-around.

JSON is a reasonably nice data exchange format, so it seems reasonable to
support it fully sometime in the future.
-- 
Jan
-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Heiko Schlittermann via Exim-users]

Hi,

Emanuel Gonzalez via Exim-users  (Fr 15 Jun 2018 13:51:19 
CEST):
> i talk with the rspamd develop, they say this:
> 
> 
> "In fact, it is Exim who SHOULD drop fucking legacy protocol support. But I 
> cannot convince its developers to do that. I have fixed this issue at some 
> point in the past but I have no Exim to test that."

It it not the problem of "convince its developers", it's a problem if
ressources to integrate the new protocol. 

For what I understand, the new protocol uses JSON to transfer its
information, while the old one was a simple text-line-by-line protocol.

This simple line-by-line protocol (IMHO) can be supported on almost any
platform, so I can't follow the reasons behind dropping it from rspamd.
But this we should/could discuss in some other thread.

Personally I'd like to see rspamd support in Exim, but currently my
ressources are limited (ressources hack the support into Exim), and I do
not feel comfortable with bloating Exim by integrating a JSON parser.
But maybe I'm wrong.

Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
-- 
 SCHLITTERMANN.de  internet & unix support -
 Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
 gnupg encrypted messages are welcome --- key ID: F69376CE -
 ! key id 7CBF764A and 972EAC9F are revoked since 2015-01  -


signature.asc
Description: PGP signature
-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Jeremy Harris via Exim-users]

On 06/16/2018 02:27 PM, Jeremy Harris via Exim-users wrote:
> On 06/15/2018 01:28 PM, Andrew C Aitchison via Exim-users wrote:
>> Hmm; rspamd used to use the spamassassin protocol.
>> At some point rspamd switched to a new protocol which the developer had
>> a reasonable reason for prefering (theoretically cleaner and IIRC
>> better performance under some conditions).
>>
>> Later on, rspamd stopped supporting the old protocol, but since
>> that is still the main protocol used by spamassassin I don't
>> think exim should drop it.
> 
> If it is correct that rspamd (in current versions) not longer operates
> with Exim then we should cease claiming it does (until and if
> someone develops new support), and may as well remove the nonfunctional
> code.

Now done (commit 590795f428).
The original spamassassin support remains.
-- 
Cheers,
  Jeremy

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Jeremy Harris via Exim-users]

On 06/15/2018 01:28 PM, Andrew C Aitchison via Exim-users wrote:
> Hmm; rspamd used to use the spamassassin protocol.
> At some point rspamd switched to a new protocol which the developer had
> a reasonable reason for prefering (theoretically cleaner and IIRC
> better performance under some conditions).
> 
> Later on, rspamd stopped supporting the old protocol, but since
> that is still the main protocol used by spamassassin I don't
> think exim should drop it.

If it is correct that rspamd (in current versions) not longer operates
with Exim then we should cease claiming it does (until and if
someone develops new support), and may as well remove the nonfunctional
code.
-- 
Jeremy


-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Phil Pennock via Exim-users]

On 2018-06-15 at 17:26 -0400, Phil Pennock via Exim-users wrote:
> On 2018-06-15 at 11:51 +, Emanuel Gonzalez via Exim-users wrote:
> > "In fact, it is Exim who SHOULD drop fucking legacy protocol support.
> > But I cannot convince its developers to do that. I have fixed this
> > issue at some point in the past but I have no Exim to test that."
> 
> For the record: this is the first I've ever heard of such an issue.
> 
> I've just checked  and there's no sign of a
> tracking issue.

I've found one mail pair on exim-dev, where I asked what was wanted and
some stuff was described in a follow-up.  That's it.

There's been no refusal, that I know of, to implement changes.

-Phil

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Phil Pennock via Exim-users]

On 2018-06-15 at 11:51 +, Emanuel Gonzalez via Exim-users wrote:
> "In fact, it is Exim who SHOULD drop fucking legacy protocol support.
> But I cannot convince its developers to do that. I have fixed this
> issue at some point in the past but I have no Exim to test that."

For the record: this is the first I've ever heard of such an issue.

I've just checked  and there's no sign of a
tracking issue.

We're open to patches to change protocols, that's how this code was
contributed in the first place.  We support open source and closed
source products both.  We've no objection to changing protocols here.

I do object to finding out that there's an issue by a developer of
another project bad-mouthing us and I object to spending my time
cleaning up the mess.  I've little to no incentive to go out of my way
to chase this down any further.

In the meantime, it looks like Exim just doesn't support rspam-proxy
if it's truly just ignoring configuration to speak their own protocol.

I've filed a feature request tracking issue:

  https://bugs.exim.org/show_bug.cgi?id=2282

I'm going to step away now because this entire situation is leaving a
bad taste in my mouth and making me grumpy.  "Feature request filing
against a project by bad-mouthing the developers and shaming them into
development" is not a model I wish to support or encourage for open
source development.  This is not why I give up my own time to work on
software such as Exim.  This is bullshit.

-Phil

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Emanuel Gonzalez via Exim-users]

rspamd develop reply:

No, that's not correct. Rspamd started the own protocol based on SA protocol 
long ago. Then there was a workaround to support SA clients. However, long time 
ago, Rspamd has switched from own protocol to the standard HTTP (using JSON to 
produce results). Hence, if Exim wants to be compatible with both Rspamd and 
SA, it should support only SA protocol and HTTP protocol for Rspamd. No need in 
3 protocols.
>From my side, I think I need to fix legacy modes support in Proxy only. Other 
>components should be fine to my knowledge.


-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Andrew C Aitchison via Exim-users]

On Fri, 15 Jun 2018, Emanuel Gonzalez via Exim-users wrote:


i talk with the rspamd develop, they say this:

"In fact, it is Exim who SHOULD drop fucking legacy protocol support.
 But I cannot convince its developers to do that. I have fixed this 
issue at some point in the past but I have no Exim to test that."


please give me a help.


Hmm; rspamd used to use the spamassassin protocol.
At some point rspamd switched to a new protocol which the developer had
a reasonable reason for prefering (theoretically cleaner and IIRC
better performance under some conditions).

Later on, rspamd stopped supporting the old protocol, but since
that is still the main protocol used by spamassassin I don't
think exim should drop it.

Also IIRC, the significant difference in the new protocol means that 
whilst there was/is code that works on linux, it would need further

effort to make it work on all of the platforms exim supports.

Phil's reply suggests that there are actually three protocols,
which would mean that my history is not telling the whole story,
but conflating two protocols at some points.

--
Andrew C. Aitchison Cambridge, UK
and...@aitchison.me.uk


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Emanuel Gonzalez via Exim-users]

I'm in contact with the rspamd developers and they tell me this


I've told them for years* already that HTTP is the only format that should be 
used when communicating with Rspamd. They just refuses to do that: Exim is the 
only MTA that still uses protocol that has been deprecated for 5 years already.


https://lists.exim.org/lurker/message/20170912.175804.72477257.en.html


please give a help
-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Emanuel Gonzalez via Exim-users]

i talk with the rspamd develop, they say this:


"In fact, it is Exim who SHOULD drop fucking legacy protocol support. But I 
cannot convince its developers to do that. I have fixed this issue at some 
point in the past but I have no Exim to test that."

please give me a help.
-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Phil Pennock via Exim-users]

On 2018-06-14 at 18:31 +, Emanuel Gonzalez via Exim-users wrote:
> Here the log:
> 
> https://github.com/vstakhov/rspamd/files/2102038/rspamdserver.log

The rspamd proxy is replying with an HTTP response, not an RSPAM
protocol response.

Since I saw logic in the proxy source-code to handle this and perform
legacy conversions, my best guess is that some older release of rspamd
was missing that conversion logic and you need to upgrade rspamd.  My
second-best guess is that you didn't restart rspamd-proxy after setting
the option to enable the legacy format, or are editing the wrong config
file, or some other glitch like that.

Exim speaks the rspamc output format when told variant=rspamd and the
spamc output format by default.  rspamd seems to class these both as
"legacy" now.  The proxy in your examples is not speaking that when
talking back to Exim.

Where the rspamd-proxy is responding "HTTP/1.1 200 OK" to Exim, it
_should_ be responding something like "RSPAMD/1.3 0 EX_OK" instead.
Once you get it responding like that, everything should work.

Regards,
-Phil

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Emanuel Gonzalez via Exim-users]

Here the log:


https://github.com/vstakhov/rspamd/files/2102038/rspamdserver.log


Thanks.!!
-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Emanuel Gonzalez via Exim-users]

I try to attach a .log file but my answer is not published in the thread.


-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] Rspamd-Proxy error with exim

[Phil Pennock via Exim-users]

On 2018-06-13 at 18:44 +, Emanuel Gonzalez via Exim-users wrote:
> rspamd-proxy doesn't work with Exim v4.87. Connection works etc but exim 
> can't parse the response.

Interesting.  From the rspamd log attached to your ticket against rspamd
it looks as though rspamd thinks things succeeded?

Since you have the service listening on a TCP port, could you provide a
tcpdump of the response, so we can see what text you're getting in
reply, please?

This means seeing the _actual_ reply, not just that there was a reply.

> 2018-06-13 13:45:23 1fT8te-0004kn-Pw spam acl condition: cannot parse spamd 
> [1.2.3.4]:11332, output: 0

Our code seems to match what I can figure out from the rspamd codebase,
but I've never looked at rspamd's code before so am unfamiliar with it.

The core parsing is unchanged from the logic which rspamd supplied to
us, I believe.

> More information see this report 
> https://github.com/vstakhov/rspamd/issues/2289

You include a tcpdump there showing that packets were exchanged, and
their sizes, but nothing about the content of the packets.

You'll want to tell tcpdump:  -Xs0

The "-s 0" says "read the entire packet, not just the start" and the
"-X" says to print a dump in hex and ASCII.

We only want the actual text from the reply, we don't need all the stuff
sending the mail across.  The reply probably includes the text:

   RSPAMD/1.3 0 EX_OK

That packet, to the end of the connection, is the bit we need, to
compare what the server sent with what Exim expected.

Regards,
-Phil

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[exim] Rspamd-Proxy error with exim

[Emanuel Gonzalez via Exim-users]

### Classification (Please choose *one* option):

* [] Crash/Hang/Data loss
* [ ] WebUI/Usability
* [ ] Serious bug
* [x] Other bug
* [ ] Feature
* [ ] Enhancement

### Reproducibility (Please choose *one* option):

* [X] Always
* [ ] Sometimes
* [ ] Rarely
* [ ] Unable
* [ ] I didn’t try
* [ ] Not applicable

### Rspamd version:

Rspamd daemon version 1.7.5

### Operation system, CPU, memory and environment:

CentOS Linux release 7.4.1708 (Core) x86_64

Cloud Hosting with 2 cores

2 Gb RAM

### Description (Please provide a descriptive summary of the issue):

rspamd-proxy doesn't work with Exim v4.87. Connection works etc but exim can't 
parse the response.

### Compile errors (if any):

### Steps to reproduce:

worker-proxy.inc

milter = no;
bind_socket = *:11332;
upstream "scan" {
default = yes; # Self-scan upstreams are always default
self_scan = yes; # Enable self-scan
hosts = "1.2.3.4:11333";
}

exim configure

spamd_address = 1.2.3.4 11332 variant=rspamd

### Expected results:

Should work via proxy.

### Actual results:

exim log:

2018-06-13 13:45:23 1fT8te-0004kn-Pw spam acl condition: cannot parse spamd 
[1.2.3.4]:11332, output: 0
2018-06-13 13:45:23 1fT8te-0004kn-Pw spam acl condition: cannot parse spamd 
[1.2.3.4]:11332, output: 0
2018-06-13 13:45:23 1fT8te-0004kn-Pw spam acl condition: cannot parse spamd 
[1.2.3.4]:11332, output: 0
2018-06-13 13:45:23 1fT8te-0004kn-Pw spam acl condition: cannot parse spamd 
[1.2.3.4]:11332, output: 0
2018-06-13 13:45:23 1fT8te-0004kn-Pw spam acl condition: cannot parse spamd 
[1.2.3.4]:11332, output: 0


### Debugging information (see details 
[here](https://rspamd.com/doc/faq.html#how-to-figure-out-why-rspamd-process-crashed)):

### Configuration:

### Additional information:

I try using this exim versions

Exim version 4.87_1 #2 built 09-Jan-2017 13:55:10

Exim version 4.90_1 #8 built 17-Apr-2018 09:43:34


More information see this report https://github.com/vstakhov/rspamd/issues/2289

[https://avatars2.githubusercontent.com/u/35730688?s=400=4]<https://github.com/vstakhov/rspamd/issues/2289>

Rspamd-Proxy error with exim · Issue #2289 · 
vstakhov/rspamd<https://github.com/vstakhov/rspamd/issues/2289>
github.com
Classification (Please choose one option): [] Crash/Hang/Data loss 
WebUI/Usability Serious bug Other bug Feature Enhancement Reproducibility 
(Please choose one option): Always Sometimes ...


-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/