Re: Download Manager
> Ah, well, yeah, why not? Well, it might be a tad painful, 'cos I suspect there are some barriers. ;-) For some reason I think I created a copy from one repo to another, but no 'getStream' or anything like that. Maybe due to trying to keep 'hands off' the protocols. The Avalon Repository work has some interesting code in this regard worth looking at. Not sure I followed it, but I think the had an URLConnection, or something. > Currently I am browsing through the API and add > some JavaDocs. Would be great, if you would check this, so that these > are correct. Looks good so far, I'll edit if/when I find things. Thanks for doign this, it is a good job to keep up on. regards Adam
Re: Download Manager
> Ah, well, yeah, why not? Currently I am browsing through the API and add > some JavaDocs. Would be great, if you would check this, so that these > are correct. Sure, will do. regards Adam
Re: Download Manager
Ah, well, yeah, why not? Currently I am browsing through the API and add some JavaDocs. Would be great, if you would check this, so that these are correct. Markus Adam R. B. Jack wrote: Markus: Would you be willing to take the DownloadManager development on? I think the 'sharing' we'd get discussing it, and how to implement it, would be education for this team as a whole. I think we'd work out more details with doing it this way. ? regards Adam
Re: Download Manager
Markus: > Would you be willing to take the DownloadManager development on? I think the > 'sharing' we'd get discussing it, and how to implement it, would be > education for this team as a whole. I think we'd work out more details with > doing it this way. ? regards Adam
Re: Download Manager
Adam R. B. Jack wrote: One thing I can chime in on here is that the mirror folks sure do like the idea of uploading to a staging area then copying to the desired location. It benefits their server side scripts to do this since an upload could take much longer time than a server side copy operation. I'm not sure how rsync handles partially complete uploaded files, but I know the signature scripts Henning is running via cron will generate an md5 for a partial file. Then when you upload your md5 they clearly don't match. Though this is a shortfall of his script, uploading to a staging area just seems more robust in the long run. Good to know, thanks. Right now we are focusing on downloading jars from a pre-populated repository (kindly managed by folks such as yourself. :-) regards Adam Oh, thats RIGHT, your working on the DOWNLOAD Manager :-) [sorry] begin:vcard fn:Mark Diggory n:Diggory;Mark org:Harvard University;Harvard MIT Data Center adr:Harvard University;;G-6 Littauer Center (North Yard);Cambridge;Ma;02138-2901;United States email;internet:[EMAIL PROTECTED] title:Software Engineer tel;work:617 496 7246 tel;fax:617 495 0438 tel;home:617 718 2033 tel;cell:617 285 4106 url:http://www.hmdc.harvard.edu version:2.1 end:vcard
Re: Download Manager
> >> > One thing I can chime in on here is that the mirror folks sure do like > the idea of uploading to a staging area then copying to the desired > location. It benefits their server side scripts to do this since an > upload could take much longer time than a server side copy operation. > I'm not sure how rsync handles partially complete uploaded files, but I > know the signature scripts Henning is running via cron will generate an > md5 for a partial file. Then when you upload your md5 they clearly don't > match. Though this is a shortfall of his script, uploading to a staging > area just seems more robust in the long run. > Good to know, thanks. Right now we are focusing on downloading jars from a pre-populated repository (kindly managed by folks such as yourself. :-) regards Adam
Re: Download Manager
Markus M. May wrote:
Hello, here are some ideas for the Download-Manager. What do you think?
- how to unit test this, without a download? - not possible, so the
internet connection should be up and running for the testcase
For test check the files into cvs and use a repo url of
file:///${basedir}/src/test-repo
R,
Nick
Re: Download Manager
Adam R. B. Jack wrote: Ought we simple download it using Download-Manager w/ trusted? :) Ought we not simple copy it down to the local repository location? - copy file to a tmp-Directory with tmp-Name - check tmp file to MD5 - if correct, copy tmp file to local repository with correct name One thing I can chime in on here is that the mirror folks sure do like the idea of uploading to a staging area then copying to the desired location. It benefits their server side scripts to do this since an upload could take much longer time than a server side copy operation. I'm not sure how rsync handles partially complete uploaded files, but I know the signature scripts Henning is running via cron will generate an md5 for a partial file. Then when you upload your md5 they clearly don't match. Though this is a shortfall of his script, uploading to a staging area just seems more robust in the long run. 2cents, Mark begin:vcard fn:Mark Diggory n:Diggory;Mark org:Harvard University;Harvard MIT Data Center adr:Harvard University;;G-6 Littauer Center (North Yard);Cambridge;Ma;02138-2901;United States email;internet:[EMAIL PROTECTED] title:Software Engineer tel;work:617 496 7246 tel;fax:617 495 0438 tel;home:617 718 2033 tel;cell:617 285 4106 url:http://www.hmdc.harvard.edu version:2.1 end:vcard
Re: Download Manager
This is good stuff. > Hello, here are some ideas for the Download-Manager. What do you think? > - Repository gets new attribute "trusted", then no check of MD5 > - Artifact gets the same attribute > - if attribute is set to true, then no check of MD5 and the file is >directly copied to the local repository All sounds good. Further, right now we have a 'runtime attribute' on the ArtifactInstance that tell it where the MD5 or ASC file is to be found. A virtual location. If it isn't there, we could see if the repository was 'untrusted' and not download? > - get MD5-file Ought we simple download it using Download-Manager w/ trusted? :) Ought we not simple copy it down to the local repository location? > - copy file to a tmp-Directory with tmp-Name > - check tmp file to MD5 > - if correct, copy tmp file to local repository with correct name Are there MD5-based streams we can use where the file isn't completed on disk until verified? If not, your approach seems good. BTW: The 'protocols abstraction' (that works over VFS or HttpClient or XXX) might need some work to do this. Right now it dumbly copies from one repo to another (i.e. local). I don't think it even has the concept of 'get'. I can help work on that. > - also copy MD5 file to the local repository, so that later checks could >be also performed > Question: do we use cmdLine md5hash or do we use java? - I like the java > idea, but this could be pretty bad, if the file is large or the md5 > implementation of java does not support the bsd/linux md5 I don't care about speed, we only download when we need to. > - how to unit test this, without a download? - not possible, so the > internet connection should be up and running for the testcase Hmm, that will do for starters, but I'd like to figure this out eventually using MockRepository, etc. Would you be willing to take the DownloadManager development on? I think the 'sharing' we'd get discussing it, and how to implement it, would be education for this team as a whole. I think we'd work out more details with doing it this way. regards Adam
Download Manager
Hello, here are some ideas for the Download-Manager. What do you think? - Repository gets new attribute "trusted", then no check of MD5 - Artifact gets the same attribute - if attribute is set to true, then no check of MD5 and the file is directly copied to the local repository - get MD5-file - copy file to a tmp-Directory with tmp-Name - check tmp file to MD5 - if correct, copy tmp file to local repository with correct name - also copy MD5 file to the local repository, so that later checks could be also performed Question: do we use cmdLine md5hash or do we use java? - I like the java idea, but this could be pretty bad, if the file is large or the md5 implementation of java does not support the bsd/linux md5 - how to unit test this, without a download? - not possible, so the internet connection should be up and running for the testcase
