Hi Jeff,
Jeff McAffer wrote:
Thanks to Stefan we have been introducing the notion of
ProcessingSteps to munge the content as it is downloaded from an
artifact repository. This allows for things like inline MD5 digest
checking, unpack200 processing, delta merging, signature checking, ...
All great stuff. Pascal just raised a very interesting question.
How do we handle restarting? Some background. In the current
prototype (in my workspace, not yet in CVS) there is a chain of
ProcessingStep objects. Each step is actually an OutputStream that
knows about the next step (output stream) in the chain. When a byte
is written to step/stream N, it is processed (counted, transformed,
...) and then the result passed on to step N+1. This repeats until
finally the content gets to the last stream in the chain which is
usually a FileOutputStream of some sort and so the content is then
written to disk. All is well.
Now, what happens if we crash or the user somehow pauses the download?
The content is partially processed/transformed but it would likely be
too costly for each step to persist its intermediate results. It
would be more likely that somehow the raw content coming in to the
head of the chain of steps is cached and then when the download is
restarted after a crash/exit, the chain is recreated and the download
is effectively replayed through the chain from the cache. When that
is done, the further content from the source would then be pushed
through the chain.
So, two questions. Does this make sense? and if so, how should we
implement this? I wonder if ECF has some technology/support/designs
in this area since it seems they support restartable downloads. Scott?
Unfortunately not as much as we would like. We do have API support for
pausing/resuming downloads (IFileTransferPausable), and the existing
impls do naively support this interface, but we need/want to add
further/better implementation support (e.g. direct protocol support for
protocols that have pause/resume, partial file caching, etc).
Actually, I'm a little surprised that you have so far passed the
ProcessingSteps as output streams directly to the ECF OutputStream, as I
was expecting that you would have a temporary file to receive the file
contents, and then when the file reception is done *then* apply the
ProcessingSteps.
But in any event, we can add impl support for pause/resume/caching etc
to the ECF receive implementations w/o changing API to support required
use cases. I would appreciate a little better understanding of the
existing ProcessingSteps and their function...so could someone point me
at the relevant packages/classes and I'll take more of a look?
Seems like this would also be a good topic for the upcoming Equinox
Summit: what enhancements are needed for file transfer both at API and
impl: e.g. pause/resume enhancements, file caching, monitoring/transfer
statistics collection?, support more/other providers, etc.
Scott
Jeff
------------------------------------------------------------------------
_______________________________________________
equinox-dev mailing list
equinox-dev@eclipse.org
https://dev.eclipse.org/mailman/listinfo/equinox-dev
_______________________________________________
equinox-dev mailing list
equinox-dev@eclipse.org
https://dev.eclipse.org/mailman/listinfo/equinox-dev