Re: [plug] Resuming a failed mail transfer

[ambrosio jr berdijo]

There was a proposal for SMTP Message Restart in the mid-2000s (2005~2007)

SMTP service extensions for transaction
checkpointingdraft-fanf-smtp-rfc1845bis
"This memo describes the SMTP service extension for checkpoint/resume,
which allows a client to recover from a lost connection to the server
without having to repeat all of the commands and message content sent prior
to the interruption, and with less risk of duplicated messages. It also
includes an updated specification of the predecessor SMTP service extension
for checkpoint/restart."

It was not implemented in PostFix because of the following reasons (from
Wietse Venema - Postfix author):
http://www.textndata.com/forums/rfc-1845-a-175069.html
- denial of service attack (as mentioned in previous PLUG message)
- too much complexity on SMTP server as messages become transactions that
need to be tracked:
   - IP of sender (so receiver knows if restart is valid)
   - Timers that decide when a connection was broken and when to resume
   - Resumption Byte-offset per message

Ambo
_
Philippine Linux Users' Group (PLUG) Mailing List
http://lists.linux.org.ph/mailman/listinfo/plug
Searchable Archives: http://archives.free.net.ph

Re: [plug] Resuming a failed mail transfer

[Michael Tinsay]

Obet, 
Nobody is thinking of improving SMTP anymore.  They are busy building "the next 
big thing" in their very own walled garden.  And trying to improve SMTP and 
getting all the players to support the improvement is gonna be a major pain.  
There are alternatives... they may be more tedious from your POV, but at least 
they are there.  Just go with the alternative you find acceptable.

--- mike t.

  From: Roberto Verzola 
 To: plug@lists.linux.org.ph 
 Sent: Saturday, 23 July 2016, 16:06
 Subject: Re: [plug] Resuming a failed mail transfer
   
> there was never a  notion of a partially delivered message, nor one that
> could be resumed.

I find it hard to believe this one. The Internet from the beginning has always 
been about partially delivered files. And not just between users within the 
same machine but between users of different machines using different OS even. 
The different packets that make up the entire file take different routes to be 
gradually assembled at the destination. Whether it is a big or a small file, 
retries have always been, at some level or another, a part of Internet 
protocols, because slow connections have always been part of the Internet. 
Internet protocols keep improving too, or get replaced by newer protocols. And 
the solution is so simple, and has existed for such a long time, that I truly 
wonder if it has not been incorporated in some mail transfer protocol or 
another that we just have not heard of.

Greetings,

Obet

On Sat, 23 Jul 2016 15:26:40 +0800
Mark David Dumlao  wrote:

> On Jul 21, 2016 18:35, "Roberto Verzola"  wrote:
> >
> > > Why not just upload the file somewhere that supports upload resume then
> > > just email the URL. Most FTP servers, and cloud storage, services like
> > > Dropbox or Google Drive, supports resuming of interrupted transfers.
> >
> > Maybe, but that is too much work, especially if the problem occurs
> regularly because you have a slow connection. Email with attachments is
> such a universal Internet event that I expect by this time that such
> 30-year (at least!) old technology as resumed file transfers would have
> found their way and become part (transparent and automatic) of the process.
> Or am I expecting too much?
> 
> Yeah you are.
> 
> Regardless of what theyre used for today, some network protocols were
> designed for something else entirely and enhancements to those protocols
> tended to stick within the limitations of those designs.
> 
> Email historically came from the mail systems from inside a multi-user
> machine, which worked under the assumption that mailbox files were written
> atomically and quickly with relatively short, text-based messages. Thus
> there was never a  notion of a partially delivered message, nor one that
> could be resumed.
> 
> Furthermore more reliable file transfer was always available throughout the
> history of mail protocols. In intra-host mail, the fact that you shared a
> filesystem meant that there was no point to duplicating the files. In
> host-to-host mail, other methods for serving and transferring files were
> always around. And many mailservers simply had attachment size limits and
> the like so large attachments never got a serious consideration. (it would
> be trivial to dos a server or user that arbitrarily accepted large files,
> like email does)
> 
> So tldr: the reality is that email was never intended for heavy file
> transfers and that is unlikely to change any time in the near future. Mail
> has historically always relied on other file sharing methods.


-- 
Roberto Verzola 
_
Philippine Linux Users' Group (PLUG) Mailing List
http://lists.linux.org.ph/mailman/listinfo/plug
Searchable Archives: http://archives.free.net.ph


  _
Philippine Linux Users' Group (PLUG) Mailing List
http://lists.linux.org.ph/mailman/listinfo/plug
Searchable Archives: http://archives.free.net.ph

Re: [plug] Resuming a failed mail transfer

[Mark David Dumlao]

On Jul 24, 2016 6:04 PM, "Roberto Verzola"  wrote:
>
> > there was never a  notion of a partially delivered message, nor one that
> > could be resumed.
>
> I find it hard to believe this one. The Internet from the beginning has
always been about partially delivered files.

Not only is there no "single" coherent Internet design (the network
protocols comprising different parts of what we recognize as tne Internet
often have conflicting and overlapping goals and responsiblities) but mail
- as in the intrahost messaging system - preceded the Internet. It was
barely syntax sugar over appending text to a shared file - the existence of
which meant that you were already sharing files reliably. So message resume
was just not factored into the design.

And to add a "simple" feature as resume takes more work than you might
think. it would require, for instance, indexing all the incompatible
mailbox formats with reliable, non-repeatable ids. SMTP - client-side -
would have to be overhauled to reliably send message sizes or wait for
application-layer ACKs to message chunks (messages arent chunked right now)
in a time when we can barely trust outlook not to mangle MIME. And then
theres defining what to do in the case of multiple redundent relay servers
- who stores the incomplete message? the server that happened to be
roundrobined that minute or an authoritative sender or the ultimate
receiver? Or multiple levels of smarthosts / relay servers which are all
user-transparent... what happens when transmission needs to be resumed at
an intermediary? Can (or should) users read partial messages? What happens
if a partial message is updated while reading? How do MIME attachments get
chunked? Is there a checksum mechanism? How do you deal with brute force
message injections (smtp is largely unauthenticated at the receiving end -
hence why spam works)? etc...

No doubt a decent programmer can make clean room designs to the questions
above - which is probably why we have a hundred different messaging
protocols - but thats because we have the luxury of thinking about the
problem after years of real world testing. They simply were never factored
into email protocols and its not easy to add them in now.

>
> Greetings,
>
> Obet
>
> On Sat, 23 Jul 2016 15:26:40 +0800
> Mark David Dumlao  wrote:
>
> > On Jul 21, 2016 18:35, "Roberto Verzola"  wrote:
> > >
> > > > Why not just upload the file somewhere that supports upload resume
then
> > > > just email the URL. Most FTP servers, and cloud storage, services
like
> > > > Dropbox or Google Drive, supports resuming of interrupted transfers.
> > >
> > > Maybe, but that is too much work, especially if the problem occurs
> > regularly because you have a slow connection. Email with attachments is
> > such a universal Internet event that I expect by this time that such
> > 30-year (at least!) old technology as resumed file transfers would have
> > found their way and become part (transparent and automatic) of the
process.
> > Or am I expecting too much?
> >
> > Yeah you are.
> >
> > Regardless of what theyre used for today, some network protocols were
> > designed for something else entirely and enhancements to those protocols
> > tended to stick within the limitations of those designs.
> >
> > Email historically came from the mail systems from inside a multi-user
> > machine, which worked under the assumption that mailbox files were
written
> > atomically and quickly with relatively short, text-based messages. Thus
> > there was never a  notion of a partially delivered message, nor one that
> > could be resumed.
> >
> > Furthermore more reliable file transfer was always available throughout
the
> > history of mail protocols. In intra-host mail, the fact that you shared
a
> > filesystem meant that there was no point to duplicating the files. In
> > host-to-host mail, other methods for serving and transferring files were
> > always around. And many mailservers simply had attachment size limits
and
> > the like so large attachments never got a serious consideration. (it
would
> > be trivial to dos a server or user that arbitrarily accepted large
files,
> > like email does)
> >
> > So tldr: the reality is that email was never intended for heavy file
> > transfers and that is unlikely to change any time in the near future.
Mail
> > has historically always relied on other file sharing methods.
>
>
> --
> Roberto Verzola 
> _
> Philippine Linux Users' Group (PLUG) Mailing List
> http://lists.linux.org.ph/mailman/listinfo/plug
> Searchable Archives: http://archives.free.net.ph
_
Philippine Linux Users' Group (PLUG) Mailing List
http://lists.linux.org.ph/mailman/listinfo/plug
Searchable Archives: http://archives.free.net.ph

Re: [plug] Resuming a failed mail transfer

[Roberto Verzola]

> there was never a  notion of a partially delivered message, nor one that
> could be resumed.

I find it hard to believe this one. The Internet from the beginning has always 
been about partially delivered files. And not just between users within the 
same machine but between users of different machines using different OS even. 
The different packets that make up the entire file take different routes to be 
gradually assembled at the destination. Whether it is a big or a small file, 
retries have always been, at some level or another, a part of Internet 
protocols, because slow connections have always been part of the Internet. 
Internet protocols keep improving too, or get replaced by newer protocols. And 
the solution is so simple, and has existed for such a long time, that I truly 
wonder if it has not been incorporated in some mail transfer protocol or 
another that we just have not heard of.

Greetings,

Obet

On Sat, 23 Jul 2016 15:26:40 +0800
Mark David Dumlao  wrote:

> On Jul 21, 2016 18:35, "Roberto Verzola"  wrote:
> >
> > > Why not just upload the file somewhere that supports upload resume then
> > > just email the URL. Most FTP servers, and cloud storage, services like
> > > Dropbox or Google Drive, supports resuming of interrupted transfers.
> >
> > Maybe, but that is too much work, especially if the problem occurs
> regularly because you have a slow connection. Email with attachments is
> such a universal Internet event that I expect by this time that such
> 30-year (at least!) old technology as resumed file transfers would have
> found their way and become part (transparent and automatic) of the process.
> Or am I expecting too much?
> 
> Yeah you are.
> 
> Regardless of what theyre used for today, some network protocols were
> designed for something else entirely and enhancements to those protocols
> tended to stick within the limitations of those designs.
> 
> Email historically came from the mail systems from inside a multi-user
> machine, which worked under the assumption that mailbox files were written
> atomically and quickly with relatively short, text-based messages. Thus
> there was never a  notion of a partially delivered message, nor one that
> could be resumed.
> 
> Furthermore more reliable file transfer was always available throughout the
> history of mail protocols. In intra-host mail, the fact that you shared a
> filesystem meant that there was no point to duplicating the files. In
> host-to-host mail, other methods for serving and transferring files were
> always around. And many mailservers simply had attachment size limits and
> the like so large attachments never got a serious consideration. (it would
> be trivial to dos a server or user that arbitrarily accepted large files,
> like email does)
> 
> So tldr: the reality is that email was never intended for heavy file
> transfers and that is unlikely to change any time in the near future. Mail
> has historically always relied on other file sharing methods.


-- 
Roberto Verzola 
_
Philippine Linux Users' Group (PLUG) Mailing List
http://lists.linux.org.ph/mailman/listinfo/plug
Searchable Archives: http://archives.free.net.ph

Re: [plug] Resuming a failed mail transfer

[Mark David Dumlao]

On Jul 21, 2016 18:35, "Roberto Verzola"  wrote:
>
> > Why not just upload the file somewhere that supports upload resume then
> > just email the URL. Most FTP servers, and cloud storage, services like
> > Dropbox or Google Drive, supports resuming of interrupted transfers.
>
> Maybe, but that is too much work, especially if the problem occurs
regularly because you have a slow connection. Email with attachments is
such a universal Internet event that I expect by this time that such
30-year (at least!) old technology as resumed file transfers would have
found their way and become part (transparent and automatic) of the process.
Or am I expecting too much?

Yeah you are.

Regardless of what theyre used for today, some network protocols were
designed for something else entirely and enhancements to those protocols
tended to stick within the limitations of those designs.

Email historically came from the mail systems from inside a multi-user
machine, which worked under the assumption that mailbox files were written
atomically and quickly with relatively short, text-based messages. Thus
there was never a  notion of a partially delivered message, nor one that
could be resumed.

Furthermore more reliable file transfer was always available throughout the
history of mail protocols. In intra-host mail, the fact that you shared a
filesystem meant that there was no point to duplicating the files. In
host-to-host mail, other methods for serving and transferring files were
always around. And many mailservers simply had attachment size limits and
the like so large attachments never got a serious consideration. (it would
be trivial to dos a server or user that arbitrarily accepted large files,
like email does)

So tldr: the reality is that email was never intended for heavy file
transfers and that is unlikely to change any time in the near future. Mail
has historically always relied on other file sharing methods.
_
Philippine Linux Users' Group (PLUG) Mailing List
http://lists.linux.org.ph/mailman/listinfo/plug
Searchable Archives: http://archives.free.net.ph

Re: [plug] Resuming a failed mail transfer

[Roberto Verzola]

> Why not just upload the file somewhere that supports upload resume then
> just email the URL. Most FTP servers, and cloud storage, services like
> Dropbox or Google Drive, supports resuming of interrupted transfers.

Maybe, but that is too much work, especially if the problem occurs regularly 
because you have a slow connection. Email with attachments is such a universal 
Internet event that I expect by this time that such 30-year (at least!) old 
technology as resumed file transfers would have found their way and become part 
(transparent and automatic) of the process. Or am I expecting too much?

Obet

On Thu, 21 Jul 2016 03:23:15 +
Gideon Guillen  wrote:

> As others have mentioned, there is no transfer resume for SMTP or POP3. I
> know IMAP allows you to download email attachments individually but you
> can't resume the downloas from where it gets interrupted either.
> 
> Why not just upload the file somewhere that supports upload resume then
> just email the URL. Most FTP servers, and cloud storage, services like
> Dropbox or Google Drive, supports resuming of interrupted transfers.
> 
> On Wed, Jul 20, 2016, 07:11 Roberto Verzola  wrote:
> 
> > Whenever I'm sending mail with a huge attachment using the POP3 protocol
> > and the transfer is cut midway, I notice that upon retry, the transfer
> > resumes from the beginning instead of the point where it was cut. Is there
> > a way to specify under POP3 to resume transfer at the point it was cut?
> >
> > I'm using sylpheed and the pop3 protocol with my email provider,
> > gn.apc.org.
> >
> > Greetings to all,
> >
> > Obet
> >
> > --
> > Roberto Verzola 
> > _
> > Philippine Linux Users' Group (PLUG) Mailing List
> > http://lists.linux.org.ph/mailman/listinfo/plug
> > Searchable Archives: http://archives.free.net.ph
> >


-- 
Roberto Verzola 
_
Philippine Linux Users' Group (PLUG) Mailing List
http://lists.linux.org.ph/mailman/listinfo/plug
Searchable Archives: http://archives.free.net.ph

Re: [plug] Resuming a failed mail transfer

[Gideon Guillen]

As others have mentioned, there is no transfer resume for SMTP or POP3. I
know IMAP allows you to download email attachments individually but you
can't resume the downloas from where it gets interrupted either.

Why not just upload the file somewhere that supports upload resume then
just email the URL. Most FTP servers, and cloud storage, services like
Dropbox or Google Drive, supports resuming of interrupted transfers.

On Wed, Jul 20, 2016, 07:11 Roberto Verzola  wrote:

> Whenever I'm sending mail with a huge attachment using the POP3 protocol
> and the transfer is cut midway, I notice that upon retry, the transfer
> resumes from the beginning instead of the point where it was cut. Is there
> a way to specify under POP3 to resume transfer at the point it was cut?
>
> I'm using sylpheed and the pop3 protocol with my email provider,
> gn.apc.org.
>
> Greetings to all,
>
> Obet
>
> --
> Roberto Verzola 
> _
> Philippine Linux Users' Group (PLUG) Mailing List
> http://lists.linux.org.ph/mailman/listinfo/plug
> Searchable Archives: http://archives.free.net.ph
>
_
Philippine Linux Users' Group (PLUG) Mailing List
http://lists.linux.org.ph/mailman/listinfo/plug
Searchable Archives: http://archives.free.net.ph

Re: [plug] Resuming a failed mail transfer

[Brian Baquiran]

There is no retry command specified in POP3 (RFC1939) either.

On Wed, Jul 20, 2016 at 9:35 PM, Roberto Verzola 
wrote:

> Yes, I stand corrected, it is not pop3 but smtp for sending, which is also
> reflected in my setup, when I double-checked. But this can also be a
> problem for receiving.
>
> I'm sure it is not about the max size limit, because if I try again and
> again, it eventually gets through.
>
> This is such a common problem, especially on slow lines, that someone
> somewhere must have thought of a way to resume the upload (or download)
> where it was cut.
>
> In file transfers during the old Fido days (only the old-timers probably
> remember...), we were already resuming transfers properly, regardless of
> direction (upload or download).
>
> Obet
>
>
> On Tue, 19 Jul 2016 21:47:00 -0400
> fooler mail  wrote:
>
> > pop3 is a protocol to retrieve email and not sending... smtp is used
> > for sending email..
> >
> > smtp protocol didnt specify for any resume upload.. most likely you
> > hit the max size limit your provider allowed to send an email...
> >
> > your work around solution is to split the large file attachment into
> > smaller files and send them one at a time.. there are lots of apps to
> > split and rebuild a file..
> >
> > fooler.
> >
> > On Tue, Jul 19, 2016 at 6:59 PM, Roberto Verzola 
> wrote:
> > > Whenever I'm sending mail with a huge attachment using the POP3
> protocol and the transfer is cut midway, I notice that upon retry, the
> transfer resumes from the beginning instead of the point where it was cut.
> Is there a way to specify under POP3 to resume transfer at the point it was
> cut?
> > >
> > > I'm using sylpheed and the pop3 protocol with my email provider,
> gn.apc.org.
> > >
> > > Greetings to all,
> > >
> > > Obet
> > >
> > > --
> > > Roberto Verzola 
> > > _
> > > Philippine Linux Users' Group (PLUG) Mailing List
> > > http://lists.linux.org.ph/mailman/listinfo/plug
> > > Searchable Archives: http://archives.free.net.ph
> > _
> > Philippine Linux Users' Group (PLUG) Mailing List
> > http://lists.linux.org.ph/mailman/listinfo/plug
> > Searchable Archives: http://archives.free.net.ph
> >
>
>
> --
> Roberto Verzola 
> _
> Philippine Linux Users' Group (PLUG) Mailing List
> http://lists.linux.org.ph/mailman/listinfo/plug
> Searchable Archives: http://archives.free.net.ph
>



-- 
Brian Baquiran  ~ LinkedIn
 | Twitter
 | Instagram

_
Philippine Linux Users' Group (PLUG) Mailing List
http://lists.linux.org.ph/mailman/listinfo/plug
Searchable Archives: http://archives.free.net.ph

Re: [plug] Resuming a failed mail transfer

[Roberto Verzola]

Yes, I stand corrected, it is not pop3 but smtp for sending, which is also 
reflected in my setup, when I double-checked. But this can also be a problem 
for receiving. 

I'm sure it is not about the max size limit, because if I try again and again, 
it eventually gets through. 

This is such a common problem, especially on slow lines, that someone somewhere 
must have thought of a way to resume the upload (or download) where it was cut. 

In file transfers during the old Fido days (only the old-timers probably 
remember...), we were already resuming transfers properly, regardless of 
direction (upload or download).

Obet


On Tue, 19 Jul 2016 21:47:00 -0400
fooler mail  wrote:

> pop3 is a protocol to retrieve email and not sending... smtp is used
> for sending email..
> 
> smtp protocol didnt specify for any resume upload.. most likely you
> hit the max size limit your provider allowed to send an email...
> 
> your work around solution is to split the large file attachment into
> smaller files and send them one at a time.. there are lots of apps to
> split and rebuild a file..
> 
> fooler.
> 
> On Tue, Jul 19, 2016 at 6:59 PM, Roberto Verzola  wrote:
> > Whenever I'm sending mail with a huge attachment using the POP3 protocol 
> > and the transfer is cut midway, I notice that upon retry, the transfer 
> > resumes from the beginning instead of the point where it was cut. Is there 
> > a way to specify under POP3 to resume transfer at the point it was cut?
> >
> > I'm using sylpheed and the pop3 protocol with my email provider, gn.apc.org.
> >
> > Greetings to all,
> >
> > Obet
> >
> > --
> > Roberto Verzola 
> > _
> > Philippine Linux Users' Group (PLUG) Mailing List
> > http://lists.linux.org.ph/mailman/listinfo/plug
> > Searchable Archives: http://archives.free.net.ph
> _
> Philippine Linux Users' Group (PLUG) Mailing List
> http://lists.linux.org.ph/mailman/listinfo/plug
> Searchable Archives: http://archives.free.net.ph
> 


-- 
Roberto Verzola 
_
Philippine Linux Users' Group (PLUG) Mailing List
http://lists.linux.org.ph/mailman/listinfo/plug
Searchable Archives: http://archives.free.net.ph

Re: [plug] Resuming a failed mail transfer

[fooler mail]

pop3 is a protocol to retrieve email and not sending... smtp is used
for sending email..

smtp protocol didnt specify for any resume upload.. most likely you
hit the max size limit your provider allowed to send an email...

your work around solution is to split the large file attachment into
smaller files and send them one at a time.. there are lots of apps to
split and rebuild a file..

fooler.

On Tue, Jul 19, 2016 at 6:59 PM, Roberto Verzola  wrote:
> Whenever I'm sending mail with a huge attachment using the POP3 protocol and 
> the transfer is cut midway, I notice that upon retry, the transfer resumes 
> from the beginning instead of the point where it was cut. Is there a way to 
> specify under POP3 to resume transfer at the point it was cut?
>
> I'm using sylpheed and the pop3 protocol with my email provider, gn.apc.org.
>
> Greetings to all,
>
> Obet
>
> --
> Roberto Verzola 
> _
> Philippine Linux Users' Group (PLUG) Mailing List
> http://lists.linux.org.ph/mailman/listinfo/plug
> Searchable Archives: http://archives.free.net.ph
_
Philippine Linux Users' Group (PLUG) Mailing List
http://lists.linux.org.ph/mailman/listinfo/plug
Searchable Archives: http://archives.free.net.ph