Hey Guys, The key here is that the data transfer factory that the File Manager (Server and Client), the Crawler, CAS-PGE, etc., use could be leveraged to try and test different data transfer technologies.
You guys may also be interested in scoping out my PhD dissertation which covered this topic extensively: http://sunset.usc.edu/~mattmann/Dissertation.pdf Also the software framework for evaluating and classifying different movement technologies is now up on Github: https://github.com/chrismattmann/disco/ The real cool thing to do would be to make a "DISCOConnector", which uses the evaluation framework to pick the right connector for the scenario. BTW, iRODS can do data movement - we've used it before *with* OODT as one example of a data movement technology. OODT is happy to work with it in the context of the Data Transfer interface I mentioned above. Cheers, Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -----Original Message----- From: Etienne Koen <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Wednesday, August 20, 2014 5:35 AM To: Thomas Bennett <[email protected]> Cc: "[email protected]" <[email protected]> Subject: RE: Remote data transfer >Hi Tom > >One of the other tools which is currently being tested called iRods has >this capability/option you are referring to built into it. Does OODT have >something similar or how would it be done? > >Here is the definition of the checksum we want to perform given on the >confluence page: > >"A checksum or hash sum is a small-size datum from an arbitrary block of >digital data for the purpose of detecting errors which may have been >introduced during its transmission or storage." > >Does this help? > >Etienne >________________________________________ >From: Thomas Bennett [[email protected]] >Sent: Wednesday, August 20, 2014 1:26 PM >To: Etienne Koen >Cc: [email protected] >Subject: Re: Remote data transfer > >Hi Etienne, > >I am using TCP. My first thoughts were that a checksum using TCP is not >necessary since the protocol takes care of the data integrity but there >seems to be a requirement for this in the test plan to have a chescksum >capability. > >Not to confuse issues, but I perform an md5sum on all data files (hdf5 >files) once they have been created and store it as metadata for the file. >Does the requirement maybe not refer to this? > >I would like to test a UDP protocol with a checksum calculation as well >for some comparison. Please let me know if you has success with >implementing/using this functionality! > >I will also have a look at the UDT transport mechanism as you proposed. > >Woot! > >Cheers, >Tom > >________________________________ >Disclaimer: This E-mail message, including any attachments, is intended >only for the person or entity to which it is addressed, and may contain >confidential information. Each page attached hereto must also be read in >conjunction with this disclaimer. >If you are not the intended recipient you are hereby notified that any >disclosure, copying, distribution or reliance upon the contents of this >e-mail is strictly prohibited. E.&O.E. > >Disclaimer: This E-mail message, including any attachments, is intended >only for the person or entity to which it is addressed, and may contain >confidential information. Each page attached hereto must also be read in >conjunction with this disclaimer. >If you are not the intended recipient you are hereby notified that any >disclosure, copying, distribution or reliance upon the contents of this >e-mail is strictly prohibited. E.&O.E.
