Re: [exim] exim4 Versions above about 4.80 Don't Talk to my ISP's smarthost.

[Heiko Schlittermann via Exim-users]

Hi,
Martin McCormick via Exim-users  (Sa 02 Jun 2018 18:33:41 
CEST):
> 
>   I even wrote a little shell script that one runs
> under sudo
> 
> #!/bin/sh
> msg=`mailq |awk '{print $3}'`
> if ! test -z $msg;then
> exim4 -d -M $msg
> fi
> 
>   That's because there are two other lines besides the one
> with the message ID but field 3 is empty.

You missed your friend:

exipick -i 

which leads to

exipick -i | xargs -n1 exim -d -M

Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
-- 
 SCHLITTERMANN.de  internet & unix support -
 Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
 gnupg encrypted messages are welcome --- key ID: F69376CE -
 ! key id 7CBF764A and 972EAC9F are revoked since 2015-01  -


signature.asc
Description: PGP signature
-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] exim4 Versions above about 4.80 Don't Talk to my ISP's smarthost.

[Martin McCormick via Exim-users]

Adrian Zaugg via Exim-users  writes:
> Hi Martin
> 
> On Debian you set the outgoing email address map in the file:
> 
> /etc/email-addresses
> 
> You can add a line like:
> martin: marti...@suddenlink.net
> 
> The exim configuration for a simple use case can be made by
> 
> dpkg-reconfigure -plow exim4-config
> 
> and answer the questions (chose smarthost) or simply edit
> 
> /etc/exim4/update-exim4.conf.conf
> 
> use: dc_eximconfig_configtype='smarthost'
> 
> For a port specification like 587 you should use "::", so
> dc_smarthost='smtp.suddenlink.net::587'
> BTW: Have you tried 587 on smtp.suddenlink.net?
> 
> In Debian they tried to make it easy for you and for simple configs you
> should not touch any config file at all. If you have some more demands
> you can use
> /etc/exim4/conf.d/main/01_exim4-config_listmacrosdefs
> to define macros inside. Debian uses a macro for almost everything. Just
> watch at a config file, e.g.
> /etc/exim4/conf.d/transport/30_exim4-config_remote_smtp_smarthost and
> you see lines like
> 
> .ifdef REMOTE_SMTP_SMARTHOST_HOSTS_AVOID_TLS
>   hosts_avoid_tls = REMOTE_SMTP_SMARTHOST_HOSTS_AVOID_TLS
> .endif
> 
> Instead of editing there, go to
> /etc/exim4/conf.d/main/01_exim4-config_listmacrosdefs and set
> 
> REMOTE_SMTP_SMARTHOST_HOSTS_AVOID_TLS = 
> 
> The value is then used from the macro (=the word in CAPS) in the right
> place. Like this you edit just very few configs and the upgrade runs
> smooth next time. When upgrading Debian compares the original configs
> from the installed package with what's on a system. If there is a
> difference you get those questions "The config file X was changed by you
> or a script. [...]" That's why use the Debian split configuration and
> change as few files as needed.
> 
> This unfortunately does not help in your situation.
> 
> If you set the port to 465 /etc/exim4/update-exim4.conf.conf, you need
> to add a config file. There is no logic in the stock configuration to
> detect that you are going to send to a TLS only connection.
> 
> This is another safe way for upgrading: You could just add a file
> /etc/exim4/conf.d/transport/30_exim4-config_remote_smtp_smarthost_tls
> containing your stanza
> protocol = smtps
> as the only line (and maybe some comment before). Then
> 
> service exim4 reload
> 
> creates the actual config file for exim which resides then
> /var/lib/exim4/config.autogenerated
> The given configuration is linted before it replaces the
> config.autogenerated, so you get a warning at least for syntax problems.
> You can see in there that your protocol stanza appears in the right
> place belonging to the section "remote_smtp_smarthost:".
> 
> This is very Debian specific and does not belong on this list
> actually :-)
> 
> Regards, Adrian.

I added the line:

martin: marti...@suddenlink.net

and that fixed everything!  A thousand thanks.

In the helpful hints department, I was trying to come up
with a way to cause exim4 to freeze out-going messages in the
queue in order to use the debug flag and get all the good
information about the delivery process that is not normally
needed but valuable when things aren't right.  There are some
good how-tos for freezing individual users' messages but laziness
conquered proper form and I realized that I could simply log in
on the console and pull the Ethernet plug just before sending the
message.  The smarthost is unreachable, then, and the message
gets queued.  Put the plug back and now it can be unfrozen.

  I even wrote a little shell script that one runs
under sudo

#!/bin/sh
msg=`mailq |awk '{print $3}'`
if ! test -z $msg;then
exim4 -d -M $msg
fi

That's because there are two other lines besides the one
with the message ID but field 3 is empty.

One then gets 400 plus lines that include enough data to
watch the message either be delivered or some error event scuttle
everything.  It was really nice to see all the sender information
match which produces a lot of messages saying that the sender
information matches and things are good.  It's enough to break in
to a happy dance, especially after fighting with this beast for
over 2 weeks.

The whole process is getting easier every time so again,
thank you and all others who helped.

Martin

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] exim4 Versions above about 4.80 Don't Talk to my ISP's smarthost.

[Adrian Zaugg via Exim-users]

Hi Martin

On 01.06.18 04:41, Martin McCormick via Exim-users wrote:
>   The last part of this long message is the log of the
> delivery attempt.  As you see, I do now log in to the smarthost
> and the only reason for the failure is that the sender name gets
> changed.
> 
>   The ISP knows me as martin.m which is why the process
> still falls short.  End of message except for the long log.

On Debian you set the outgoing email address map in the file:

/etc/email-addresses

You can add a line like:
martin: marti...@suddenlink.net

The exim configuration for a simple use case can be made by

dpkg-reconfigure -plow exim4-config

and answer the questions (chose smarthost) or simply edit

/etc/exim4/update-exim4.conf.conf

use: dc_eximconfig_configtype='smarthost'

For a port specification like 587 you should use "::", so
dc_smarthost='smtp.suddenlink.net::587'
BTW: Have you tried 587 on smtp.suddenlink.net?

In Debian they tried to make it easy for you and for simple configs you
should not touch any config file at all. If you have some more demands
you can use
/etc/exim4/conf.d/main/01_exim4-config_listmacrosdefs
to define macros inside. Debian uses a macro for almost everything. Just
watch at a config file, e.g.
/etc/exim4/conf.d/transport/30_exim4-config_remote_smtp_smarthost and
you see lines like

.ifdef REMOTE_SMTP_SMARTHOST_HOSTS_AVOID_TLS
  hosts_avoid_tls = REMOTE_SMTP_SMARTHOST_HOSTS_AVOID_TLS
.endif

Instead of editing there, go to
/etc/exim4/conf.d/main/01_exim4-config_listmacrosdefs and set

REMOTE_SMTP_SMARTHOST_HOSTS_AVOID_TLS = 

The value is then used from the macro (=the word in CAPS) in the right
place. Like this you edit just very few configs and the upgrade runs
smooth next time. When upgrading Debian compares the original configs
from the installed package with what's on a system. If there is a
difference you get those questions "The config file X was changed by you
or a script. [...]" That's why use the Debian split configuration and
change as few files as needed.

This unfortunately does not help in your situation.

If you set the port to 465 /etc/exim4/update-exim4.conf.conf, you need
to add a config file. There is no logic in the stock configuration to
detect that you are going to send to a TLS only connection.

This is another safe way for upgrading: You could just add a file
/etc/exim4/conf.d/transport/30_exim4-config_remote_smtp_smarthost_tls
containing your stanza
protocol = smtps
as the only line (and maybe some comment before). Then

service exim4 reload

creates the actual config file for exim which resides then
/var/lib/exim4/config.autogenerated
The given configuration is linted before it replaces the
config.autogenerated, so you get a warning at least for syntax problems.
You can see in there that your protocol stanza appears in the right
place belonging to the section "remote_smtp_smarthost:".

This is very Debian specific and does not belong on this list
actually :-)

Regards, Adrian.

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] exim4 Versions above about 4.80 Don't Talk to my ISP's smarthost.

[Martin McCormick via Exim-users]

Phil Pennock via Exim-users  writes:
> I'm going to just discuss the Exim side of things, terminology and such,
> to hopefully provide enough context and terminology for you; I can't
> really help with the Debian config building.
> 
> There are two different identifiers:
> 
>  1. Which address is the sender of the email
>  2. What is the authentication account for SMTP AUTH
> 
> The log-file you show has the authentication account (2) as martin.m
> which all works fine.

Correct.

> If you generate mail with a particular sender (1), then that's what Exim
> normally sends.  You can use "rewrites" to change that.
> 
> I suspect that in your upgrade you lost a rewrite configuration item.
> 
> Does this provide enough terminology to let you fix it?

It certainly does.  I apologize for the long log message which
was actually the output of exim -d -M with a message ID.  For
those who thought I accidentally gave away the password, I
noticed what was in the output in time to do a global replace so
the password is safe and no, it's not "safe."  This is like
"Who's on First?"

My thanks to everybody.  I think the reason it seemed to
just quit working was I didn't realize when the file in
/etc/exim4/conf.d/transport got replaced and thought something
more complex was afoot.

Thanks for everybody's patience.  I am new to this list
but not new to unix or computing so I will make sure that the
rewrites are back and that should make it all good again.

Martin McCormick

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] exim4 Versions above about 4.80 Don't Talk to my ISP's smarthost.

[Phil Pennock via Exim-users]

On 2018-05-31 at 21:41 -0500, Martin McCormick via Exim-users wrote:
>   The last part of this long message is the log of the
> delivery attempt.  As you see, I do now log in to the smarthost
> and the only reason for the failure is that the sender name gets
> changed.
> 
>   The ISP knows me as martin.m which is why the process
> still falls short.  End of message except for the long log.

I'm going to just discuss the Exim side of things, terminology and such,
to hopefully provide enough context and terminology for you; I can't
really help with the Debian config building.

There are two different identifiers:

 1. Which address is the sender of the email
 2. What is the authentication account for SMTP AUTH

The log-file you show has the authentication account (2) as martin.m
which all works fine.

If you generate mail with a particular sender (1), then that's what Exim
normally sends.  You can use "rewrites" to change that.

I suspect that in your upgrade you lost a rewrite configuration item.

Does this provide enough terminology to let you fix it?

Exim's use of Rewrites is not the same as Sendmail's.  Sendmail uses
rewrites for deciding how to send a message, and they're a core part of
routing.  Exim does not, the recipient and sender addresses should not
be changed to make routing decisions, that's why we have Routers.  But
if you do explicitly want to mutate an email address, you do still need
rewrites.



I simplified a little above, (1) should be (1a) and (1b), where there's:

 1a. The SMTP Envelope Sender
 1b. Some "Internet Message Format" header ("From:", "Sender:", ...)

I suspect that you want the SMTP Transport used for the smarthost
connection to do two things:

  i) set the `return_path` option to the same identifier used for
 authentication; this handles (1a)
  ii) use `headers_rewrite` to rewrite the From: header in the same way,
  to handle (1b)

If you want these to apply to all messages processed, not just those
sent to that smarthost, then this can be done a little more simply in
the rewrites section of the config file.

-Phil

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] exim4 Versions above about 4.80 Don't Talk to my ISP's smarthost.

[Martin McCormick via Exim-users]

I am replying to 2 postings.

Jeremy Harris via Exim-users  writes:
> This will be the magic bit, and the syntax looks right, so
> best guess is that you placed it in the wrong place.

Thank you!
I certainly did but I am not quite good yet.

> Compare that file with:
> 
> http://exim.org/exim-html-current/doc/html/spec_html/ch-the_smtp_transport.html#SECID146

That's an excellent resource.

> You're trying to add an option on the transport definition
> for your smarthost.  What I don't know is if that's the right
> file for Debian these days; with luck comments in it may help.
> Alternatively, the Debian config-helper may know about
> SSL-on-connect configurations for smarthosts already; you may
> not need ot do it manually.

I double-checked and added it again only this time, I did
put it in the right place which is the very bottom of the file
and it actually now gets me logged in for the first time but read
on as I now uncovered another much less serious problem but still
a show-stopper.

Responding to another poster:


Heiko Schlittermann via Exim-users  writes:
> Try
> 
> $ exim -bP config
> 
> and check the output, maybe you even can send us the transport section
> from the output. Do you have multiple transports defined there?

Here it is and there is only 1 besides local.

transport = remote_smtp

  smarthost:
debug_print = "R: smarthost for $local_part@$domain"
driver = manualroute
domains = ! +local_domains
transport = remote_smtp_smarthost
route_list = * smtp.suddenlink.net::465 byname
host_find_failed = ignore
same_domain_copy_routing = yes
no_more
COND_LOCAL_SUBMITTER = "${if match_ip{$sender_host_address}{:@[]}{1}{0}}"

  real_local:
debug_print = "R: real_local for $local_part@$domain"
driver = accept
domains = +local_domains
condition = "${if match_ip{$sender_host_address}{:@[]}{1}{0}}"
local_part_prefix = real-
check_local_user
transport = mail_spool

  system_aliases:
debug_print = "R: system_aliases for $local_part@$domain"
driver = redirect
domains = +local_domains
allow_fail
allow_defer
data = ${lookup{$local_part}lsearch{/etc/aliases}}

While I am at it, I did see a note about a variable
called keep_environment and it is now set as follows:

keep_environment =

It appears to not be set.

> (Using protocol = smtps in the outbound transport should switch the
> remote port to "smtps")
> 
> What is the output from:
> 
> $ getent services smtps

urd   465/tcp ssmtp smtps
> 
> The log you sent doesn't have any indication about the remote port and
> about using TLS-on-connect.

Thanks for all the help and for the good questions in
which I am glad to report that now I am being authenticated
correctly up to a point and as I just reported, there is still
one small problem.  Here is the log that almost made good but my
user ID changes as you watch.  It is supposed to be
marti...@suddenlink.net but it morphs in to just
mar...@suddenlink.net.  My user account on this system is martin
and, somehow, martin.m got set back to martin.

The last part of this long message is the log of the
delivery attempt.  As you see, I do now log in to the smarthost
and the only reason for the failure is that the sender name gets
changed.

The ISP knows me as martin.m which is why the process
still falls short.  End of message except for the long log.

Exim version 4.89 uid=0 gid=0 pid=20965 D=fbb95cfd

Skipping down to the connection:

Considering: wb5...@arrl.net
unique = wb5...@arrl.net
dbfn_read: key=R:arrl.net
dbfn_read: key=R:wb5...@arrl.net
dbfn_read: key=R:wb5...@arrl.net:
no domain retry record
no address retry record
wb5...@arrl.net: queued for routing
closed hints database and lockfile
>>>
routing wb5...@arrl.net

> hubbed_hosts router <

local_part=wb5agz domain=arrl.net
checking domains
expansion of "${if 
exists{/etc/exim4/hubbed_hosts}{partial-lsearch;/etc/exim4/hubbed_hosts}fail}" 
forced failure: assume not in this list
hubbed_hosts router skipped: domains mismatch

> smarthost router <

local_part=wb5agz domain=arrl.net
checking domains
arrl.net in "@:localhost"? no (end of list)
arrl.net in "! +local_domains"? yes (end of list)
R: smarthost for wb5...@arrl.net
calling smarthost router
smarthost router called for wb5...@arrl.net
  domain = arrl.net
route_item = * smtp.suddenlink.net::465 byname
arrl.net in "*"? yes (matched "*")
original list of hosts = "smtp.suddenlink.net::465" options = byname
expanded list of hosts = "smtp.suddenlink.net::465" options = byname
set transport remote_smtp_smarthost
finding IP address for smtp.suddenlink.net:465
host=smtp.suddenlink.net port=465
calling host_find_byname
gethostbyname2(af=inet6) returned 4 (NO_DATA)
fully qualified name = smtp.suddenlink.net
gethostbyname2 looked up these IP addresses:
  name=smtp.suddenlink.net address=208.180.40.68
queued 

Re: [exim] exim4 Versions above about 4.80 Don't Talk to my ISP's smarthost.

[Klaus Maria Pfeiffer via Exim-users]

On 05/31/2018 05:02 AM, Martin McCormick via Exim-users wrote:
>  set passwd.client as follows:
> *.suddenlink.net:marti...@suddenlink.net:somePaSsword

have you checked
  man 5 exim4-config_files
and that the hostname in passwd.client is the rDNS of your smarthost, so
in your case could be smtp.suddenlink.net.

> in a certain place that indicated protocol=smtps and then the TLS
> magic worked for about 3 years until I upgraded to debian
> stretch using exim4 version 4.89.  Then the magic black box

in Debians exim TLS is enabled by default, please check
/usr/share/doc/exim4-config/README.Debian.gz chapter 2.3.1. Using Exim
as SMTP-AUTH client

gre3tings, Klaus

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] exim4 Versions above about 4.80 Don't Talk to my ISP's smarthost.

[Heiko Schlittermann via Exim-users]

Martin McCormick via Exim-users  (Do 31 Mai 2018 05:02:35 
CEST):
>   In the first place one can not add protocol=smtps to
> /etc/exim4/conf.d/transport/30_exim4-config_remote_smtp_smarthost
> as this throws an error now that protocol=smtps is not understood.

Try 

$ exim -bP config

and check the output, maybe you even can send us the transport section
from the output. Do you have multiple transports defined there?

(Using protocol = smtps in the outbound transport should switch the
remote port to "smtps")

What is the output from:

$ getent services smtps

The log you sent doesn't have any indication about the remote port and
about using TLS-on-connect.

Try adding +outgoing_port to the log_selector setting.

Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
-- 
 SCHLITTERMANN.de  internet & unix support -
 Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
 gnupg encrypted messages are welcome --- key ID: F69376CE -
 ! key id 7CBF764A and 972EAC9F are revoked since 2015-01  -


signature.asc
Description: PGP signature
-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: [exim] exim4 Versions above about 4.80 Don't Talk to my ISP's smarthost.

[Jeremy Harris via Exim-users]

On 05/31/2018 04:02 AM, Martin McCormick via Exim-users wrote:
>   It was also necessary to add a line to
> 
> /etc/exim4/conf.d/transport/30_exim4-config_remote_smtp_smarthost
> 
> in a certain place that indicated protocol=smtps

This will be the magic bit, and the syntax looks right, so
best guess is that you placed it in the wrong place.

Compare that file with:

http://exim.org/exim-html-current/doc/html/spec_html/ch-the_smtp_transport.html#SECID146

You're trying to add an option on the transport definition
for your smarthost.  What I don't know is if that's the right
file for Debian these days; with luck comments in it may help.
Alternatively, the Debian config-helper may know about
SSL-on-connect configurations for smarthosts already; you may
not need ot do it manually.
-- 
Cheers,
  Jeremy

-- 
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/