HI chris, > Ahh gotcha. You've found a config file bug in oodt0.4. Can you please: > > 1. Create yourself an account at our JIRA issue tracker: > https://issues.apache.org/jira/browse/OODT > > 2. File an issue (create a new issue/bug) against oodt 0.4 (set the affects > version to 0.4) > I just created the isse named "OODT-474" related with config file bug in oodt0.4. It is the first time to write an issue report. Can you check the issue that I wrote ?
> Ahh, what FTP site are you trying to access? That way I can try and > test this myself in my own local checkout. I tried to test downloading a file based on WIKI site "https://cwiki.apache.org/OODT/oodt-push-pull-user-guide.html";. There is no problem to download the file as follows: $ ftp l4ftl01.larc.nasa.gov Connected to l4ftl01.larc.nasa.gov. 220- 220--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 220- 220- This US Government computer is for authorized users only. 220- By accessing this system you are consenting to complete 220- monitoring with no expectation of privacy. Unauthorized 220- access or use may subject you to disciplinary action and 220- criminal prosecution. 220- 220--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 220- 220- 220- 220 FTP server ready. Name (l4ftl01.larc.nasa.gov:yhkang): anonymous 331 Guest login ok, send your complete e-mail address as password. Password: 230 Guest login ok, access restrictions apply. Remote system type is UNIX. Using binary mode to transfer files. ftp> cd /TES/TL2CO2N.005 250 CWD command successful. ftp> cd 2004.09.20 250 CWD command successful. ftp> get TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5 local: TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5 remote: TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5 200 PORT command successful. 150 Opening BINARY mode data connection for TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5 (92150280 bytes). 226 Transfer complete. Checksum matched. 92150280 bytes received in 171.84 secs (523.7 kB/s) Thanks, Yunhee 2012/7/21 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: > Hi YunHee, > > On Jul 19, 2012, at 11:19 AM, YunHee Kang wrote: > >> I just found a problem that is in a file >> "$CAS_PP_HOME/policy/ProtocolFactoryInfo.xml" in oodt 0.4. >> >> The problem is caused by missing classes related with an element >> "protocolFactory" in the file described above. >> To fix this problem, It needs to replace a wrong class name with a >> correct class name. >> For example there is no class named >> org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpClientFactory in any jar >> file in the directory, $CAS_PP_HOME/lib. To run the script >> $CAS_PP_HOME/bin/pushpull, CogJGlobusFtpClientFactory needs to be >> replaced with CogJGlobusFtpProtocolFactory. > > Ahh gotcha. You've found a config file bug in oodt0.4. Can you please: > > 1. Create yourself an account at our JIRA issue tracker: > https://issues.apache.org/jira/browse/OODT > > 2. File an issue (create a new issue/bug) against oodt 0.4 (set the affects > version to 0.4) > > If you are willing to, we'd love to have a patch file from you that fixes > this config file. The patch process would be something like: > > 1. svn co http://svn.apache.org/repos/asf/oodt/trunk oodt0.5 > 2. edit odot0.5/pushpull/src/main/resources/policy/ProtocolFactoryInfo.xml and > fix XML policy file. > 3. svn diff > OODT-xxx.ykang.yyMMdd.patch.txt > > where xxx is the JIRA issue ID from #2 above. > 4. attach your patch to the JIRA issue from #2. > >> >> But I don't know how to replace >> org.apache.oodt.cas.protocol.local.LocalClientFactory. > > I'm not sure you have to -- this is probably OK. > >> To be honest, I am not familiar with OODT. >> >> >> <protocol type="ftp"> >> <!--protocolFactory >> class="org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpClientFactory"/--> >> <!-- cas-protocol-ftp-0.4.jar --> >> <protocolFactory >> class="org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpProtocolFactory"/> >> <!--protocolFactory >> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpClientFactory"/--> >> <protocolFactory >> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory"/> >> </protocol> >> >> <protocol type="http"> >> <!--protocolFactory >> class="org.apache.oodt.cas.protocol.http.HttpClientFactory"/--> >> <!-- cas-protocol-http-0.4.jar--> >> <protocolFactory >> class="org.apache.oodt.cas.protocol.http.HttpProtocolFactory"/> >> </protocol> >> >> <protocol type="sftp"> >> <!--protocolFactory >> class="org.apache.oodt.cas.protocol.sftp.JschSftpClientFactory"/--> >> <!-- cas-protocol-sftp-0.4.jar --> >> <protocolFactory >> class="org.apache.oodt.cas.protocol.sftp.JschSftpProtocolFactory"/> >> </protocol> >> >> <protocol type="imaps"> >> <!--protocolFactory >> class="org.apache.oodt.cas.protocol.imaps.ImapsClientFactory"/ --> >> <!-- cas-protocol-sftp-0.4.jar --> >> <protocolFactory >> class="org.apache.oodt.cas.protocol.imaps.ImapsProtocolFactory"/> >> </protocol> >> >> <!--protocol type="file"> >> <protocolFactory >> class="org.apache.oodt.cas.protocol.local.LocalClientFactory"/> >> </protocol> >> --> >> > > Brian Foster -- any idea here? > >> But up to now I am having a trouble to run the script >> $CAS_PP_HOME/bin/pushpull after I changed the file >> "$CAS_PP_HOME/policy/ProtocolFactoryInfo.xml". >> >> The following is the part of the error message when running the script. >> SEVERE: Failed to finish downloading per property files >> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >> : null > > >> >> I think that the error may occur the file RetrievalSetup.java in the >> following directory: >> ./apache-oodt-0.4-src/pushpull/src/main/java/org/apache/oodt/cas/pushpull/retrievalsystem/RetrievalSetup.java > > Yep might be -- the problem is that many remote FTP sites have their > own intricacies and some protocol libraries handle it better than others. > That's why we integrate many of these libraries and try to allow others > to plug in their own -- we may need a specific FTP library for this site > or hopefully we can find a protocol factory implementation that works. > >> >> Please let me know how to fix the problem. >> >> >> yhkang@yunhee-kisti:~/oodt/cas-pushpull/bin$ ./pushpull >> Jul 20, 2012 3:06:44 AM >> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher configure >> INFO: Configuring DaemonLauncher. . . >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config >> loadProtocolTypes >> INFO: Associating protocol types with ProtocolFactories . . . >> Jul 20, 2012 3:06:44 AM >> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType >> INFO: Assiging protocol 'ftp' with class 'class >> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory' >> Jul 20, 2012 3:06:44 AM >> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType >> INFO: Assiging protocol 'http' with class 'class >> org.apache.oodt.cas.protocol.http.HttpProtocolFactory' >> Jul 20, 2012 3:06:44 AM >> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType >> INFO: Assiging protocol 'sftp' with class 'class >> org.apache.oodt.cas.protocol.sftp.JschSftpProtocolFactory' >> Jul 20, 2012 3:06:44 AM >> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType >> INFO: Assiging protocol 'imaps' with class 'class >> org.apache.oodt.cas.protocol.imaps.ImapsProtocolFactory' >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config >> loadParserInfo >> INFO: Associating parsers with RetrievalMethods . . . >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo >> loadParserInfo >> INFO: Assiging parser >> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser' >> with retrievalmethod >> 'org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler' >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo >> loadParserInfo >> INFO: Assiging parser >> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.FileListParser' >> with retrievalmethod >> 'org.apache.oodt.cas.pushpull.retrievalmethod.ListRetriever' >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo >> loadParserInfo >> INFO: Assiging parser >> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.ClassNoaaEmailParser' >> with retrievalmethod >> 'org.apache.oodt.cas.pushpull.retrievalmethod.ListRetriever' >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config >> loadIngester >> INFO: Configuring and building ingester: >> [org.apache.oodt.cas.filemgr.ingest.StdIngester]: data transfer: >> [org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory]: >> to ingest to file manager: [http://localhost:9000] >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config >> loadIngester >> INFO: Configuring Ingester cache: [] >> TRANSFER: org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config >> loadProductTypeDetection >> INFO: Loading product type detection file: >> /home/yhkang/oodt/cas-pushpull/policy/mimetypes.xml >> Jul 20, 2012 3:06:44 AM >> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher configure >> INFO: Loading SiteInfo file >> '/home/yhkang/oodt/cas-pushpull/etc/examples/RemoteSpecsFiles/RemoteSpecs.xml' >> Jul 20, 2012 3:06:44 AM >> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher launchDaemons >> INFO: Launching Daemons . . . >> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.daemon.Daemon >> registerRMIServer >> INFO: Created Daemon ID = 91131 on RMI registry port 9113 >> Jul 20, 2012 3:06:44 AM >> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher launchDaemons >> INFO: Creating Daemon with ID = 1 >> Jul 20, 2012 3:06:44 AM >> org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning >> INFO: Daemon with ID = 91131 was given permission to run >> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser >> Jul 20, 2012 3:06:44 AM >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >> retrieveFiles >> INFO: Creating 'org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler' >> to download data files >> Jul 20, 2012 3:06:49 AM >> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol >> INFO: Testing protocol >> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this may >> take a few minutes . . . >> Jul 20, 2012 3:06:59 AM >> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >> username 'anonymous' >> Jul 20, 2012 3:06:59 AM >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >> retrieveFiles >> SEVERE: Failed to finish downloading per property files >> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >> : null >> Jul 20, 2012 3:06:59 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep >> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to >> sleep until Fri Jul 20 03:09:00 KST 2012 >> > Thanks let me know what you think above! > > Cheers, > Chris > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Chris Mattmann, Ph.D. > Senior Computer Scientist > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > Office: 171-266B, Mailstop: 171-246 > Email: chris.a.mattm...@nasa.gov > WWW: http://sunset.usc.edu/~mattmann/ > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Adjunct Assistant Professor, Computer Science Department > University of Southern California, Los Angeles, CA 90089 USA > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >
