Hi Chris, I am stuck on the problem related with "pushpull" script occurred the following error message: .... INFO: Testing protocol org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this may take a few minutes . . . Jul 24, 2012 4:05:02 PM org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous' Jul 24, 2012 4:05:02 PM org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup retrieveFiles SEVERE: Failed to finish downloading per property files /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml : null Jul 24, 2012 4:05:02 PM org.apache.oodt.cas.pushpull.daemon.Daemon sleep INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to sleep until Tue Jul 24 16:06:00 KST 2012 Jul 24, 2012 4:06:00 PM org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning INFO: Daemon with ID = 91131 was given permission to run org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
First of all, I would like test the pushpull framework to move data written in NetCDF from a remote site by a network protocols (like ftp, http, etc). I think that there is a test suite which can be used to check the pushpull framework. Can you give configuration files(a set of XML files) that I can test downloading files from a FTP site (public anonymous site) ? or Please let me know a reference implementation of pushpull framework that I can use to move NetCDF files. Thanks, Yunhee 2012/7/22 YunHee Kang <yunh.k...@gmail.com>: > HI chris, > >> Ahh gotcha. You've found a config file bug in oodt0.4. Can you please: >> >> 1. Create yourself an account at our JIRA issue tracker: >> https://issues.apache.org/jira/browse/OODT >> >> 2. File an issue (create a new issue/bug) against oodt 0.4 (set the affects >> version to 0.4) >> > I just created the isse named "OODT-474" related with config file > bug in oodt0.4. > It is the first time to write an issue report. Can you check the > issue that I wrote ? > > >> Ahh, what FTP site are you trying to access? That way I can try and >> test this myself in my own local checkout. > > I tried to test downloading a file based on WIKI site > "https://cwiki.apache.org/OODT/oodt-push-pull-user-guide.html";. There > is no problem to download the file as follows: > $ ftp l4ftl01.larc.nasa.gov > Connected to l4ftl01.larc.nasa.gov. > 220- > 220--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > 220- > 220- This US Government computer is for authorized users only. > 220- By accessing this system you are consenting to complete > 220- monitoring with no expectation of privacy. Unauthorized > 220- access or use may subject you to disciplinary action and > 220- criminal prosecution. > 220- > 220--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > 220- > 220- > 220- > 220 FTP server ready. > Name (l4ftl01.larc.nasa.gov:yhkang): anonymous > 331 Guest login ok, send your complete e-mail address as password. > Password: > 230 Guest login ok, access restrictions apply. > Remote system type is UNIX. > Using binary mode to transfer files. > ftp> cd /TES/TL2CO2N.005 > 250 CWD command successful. > ftp> cd 2004.09.20 > 250 CWD command successful. > ftp> get TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5 > local: TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5 remote: > TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5 > 200 PORT command successful. > 150 Opening BINARY mode data connection for > TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5 (92150280 bytes). > 226 Transfer complete. Checksum matched. > 92150280 bytes received in 171.84 secs (523.7 kB/s) > > > Thanks, > > Yunhee > > 2012/7/21 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >> Hi YunHee, >> >> On Jul 19, 2012, at 11:19 AM, YunHee Kang wrote: >> >>> I just found a problem that is in a file >>> "$CAS_PP_HOME/policy/ProtocolFactoryInfo.xml" in oodt 0.4. >>> >>> The problem is caused by missing classes related with an element >>> "protocolFactory" in the file described above. >>> To fix this problem, It needs to replace a wrong class name with a >>> correct class name. >>> For example there is no class named >>> org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpClientFactory in any jar >>> file in the directory, $CAS_PP_HOME/lib. To run the script >>> $CAS_PP_HOME/bin/pushpull, CogJGlobusFtpClientFactory needs to be >>> replaced with CogJGlobusFtpProtocolFactory. >> >> Ahh gotcha. You've found a config file bug in oodt0.4. Can you please: >> >> 1. Create yourself an account at our JIRA issue tracker: >> https://issues.apache.org/jira/browse/OODT >> >> 2. File an issue (create a new issue/bug) against oodt 0.4 (set the affects >> version to 0.4) >> >> If you are willing to, we'd love to have a patch file from you that fixes >> this config file. The patch process would be something like: >> >> 1. svn co http://svn.apache.org/repos/asf/oodt/trunk oodt0.5 >> 2. edit odot0.5/pushpull/src/main/resources/policy/ProtocolFactoryInfo.xml >> and >> fix XML policy file. >> 3. svn diff > OODT-xxx.ykang.yyMMdd.patch.txt >> >> where xxx is the JIRA issue ID from #2 above. >> 4. attach your patch to the JIRA issue from #2. >> >>> >>> But I don't know how to replace >>> org.apache.oodt.cas.protocol.local.LocalClientFactory. >> >> I'm not sure you have to -- this is probably OK. >> >>> To be honest, I am not familiar with OODT. >>> >>> >>> <protocol type="ftp"> >>> <!--protocolFactory >>> class="org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpClientFactory"/--> >>> <!-- cas-protocol-ftp-0.4.jar --> >>> <protocolFactory >>> class="org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpProtocolFactory"/> >>> <!--protocolFactory >>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpClientFactory"/--> >>> <protocolFactory >>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory"/> >>> </protocol> >>> >>> <protocol type="http"> >>> <!--protocolFactory >>> class="org.apache.oodt.cas.protocol.http.HttpClientFactory"/--> >>> <!-- cas-protocol-http-0.4.jar--> >>> <protocolFactory >>> class="org.apache.oodt.cas.protocol.http.HttpProtocolFactory"/> >>> </protocol> >>> >>> <protocol type="sftp"> >>> <!--protocolFactory >>> class="org.apache.oodt.cas.protocol.sftp.JschSftpClientFactory"/--> >>> <!-- cas-protocol-sftp-0.4.jar --> >>> <protocolFactory >>> class="org.apache.oodt.cas.protocol.sftp.JschSftpProtocolFactory"/> >>> </protocol> >>> >>> <protocol type="imaps"> >>> <!--protocolFactory >>> class="org.apache.oodt.cas.protocol.imaps.ImapsClientFactory"/ --> >>> <!-- cas-protocol-sftp-0.4.jar --> >>> <protocolFactory >>> class="org.apache.oodt.cas.protocol.imaps.ImapsProtocolFactory"/> >>> </protocol> >>> >>> <!--protocol type="file"> >>> <protocolFactory >>> class="org.apache.oodt.cas.protocol.local.LocalClientFactory"/> >>> </protocol> >>> --> >>> >> >> Brian Foster -- any idea here? >> >>> But up to now I am having a trouble to run the script >>> $CAS_PP_HOME/bin/pushpull after I changed the file >>> "$CAS_PP_HOME/policy/ProtocolFactoryInfo.xml". >>> >>> The following is the part of the error message when running the script. >>> SEVERE: Failed to finish downloading per property files >>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>> : null >> > >> >>> >>> I think that the error may occur the file RetrievalSetup.java in the >>> following directory: >>> ./apache-oodt-0.4-src/pushpull/src/main/java/org/apache/oodt/cas/pushpull/retrievalsystem/RetrievalSetup.java >> >> Yep might be -- the problem is that many remote FTP sites have their >> own intricacies and some protocol libraries handle it better than others. >> That's why we integrate many of these libraries and try to allow others >> to plug in their own -- we may need a specific FTP library for this site >> or hopefully we can find a protocol factory implementation that works. >> >>> >>> Please let me know how to fix the problem. >>> >>> >>> yhkang@yunhee-kisti:~/oodt/cas-pushpull/bin$ ./pushpull >>> Jul 20, 2012 3:06:44 AM >>> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher configure >>> INFO: Configuring DaemonLauncher. . . >>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config >>> loadProtocolTypes >>> INFO: Associating protocol types with ProtocolFactories . . . >>> Jul 20, 2012 3:06:44 AM >>> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType >>> INFO: Assiging protocol 'ftp' with class 'class >>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory' >>> Jul 20, 2012 3:06:44 AM >>> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType >>> INFO: Assiging protocol 'http' with class 'class >>> org.apache.oodt.cas.protocol.http.HttpProtocolFactory' >>> Jul 20, 2012 3:06:44 AM >>> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType >>> INFO: Assiging protocol 'sftp' with class 'class >>> org.apache.oodt.cas.protocol.sftp.JschSftpProtocolFactory' >>> Jul 20, 2012 3:06:44 AM >>> org.apache.oodt.cas.pushpull.config.ProtocolInfo addClassForType >>> INFO: Assiging protocol 'imaps' with class 'class >>> org.apache.oodt.cas.protocol.imaps.ImapsProtocolFactory' >>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config >>> loadParserInfo >>> INFO: Associating parsers with RetrievalMethods . . . >>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo >>> loadParserInfo >>> INFO: Assiging parser >>> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser' >>> with retrievalmethod >>> 'org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler' >>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo >>> loadParserInfo >>> INFO: Assiging parser >>> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.FileListParser' >>> with retrievalmethod >>> 'org.apache.oodt.cas.pushpull.retrievalmethod.ListRetriever' >>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.ParserInfo >>> loadParserInfo >>> INFO: Assiging parser >>> 'org.apache.oodt.cas.pushpull.filerestrictions.parsers.ClassNoaaEmailParser' >>> with retrievalmethod >>> 'org.apache.oodt.cas.pushpull.retrievalmethod.ListRetriever' >>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config >>> loadIngester >>> INFO: Configuring and building ingester: >>> [org.apache.oodt.cas.filemgr.ingest.StdIngester]: data transfer: >>> [org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory]: >>> to ingest to file manager: [http://localhost:9000] >>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config >>> loadIngester >>> INFO: Configuring Ingester cache: [] >>> TRANSFER: org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory >>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.config.Config >>> loadProductTypeDetection >>> INFO: Loading product type detection file: >>> /home/yhkang/oodt/cas-pushpull/policy/mimetypes.xml >>> Jul 20, 2012 3:06:44 AM >>> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher configure >>> INFO: Loading SiteInfo file >>> '/home/yhkang/oodt/cas-pushpull/etc/examples/RemoteSpecsFiles/RemoteSpecs.xml' >>> Jul 20, 2012 3:06:44 AM >>> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher launchDaemons >>> INFO: Launching Daemons . . . >>> Jul 20, 2012 3:06:44 AM org.apache.oodt.cas.pushpull.daemon.Daemon >>> registerRMIServer >>> INFO: Created Daemon ID = 91131 on RMI registry port 9113 >>> Jul 20, 2012 3:06:44 AM >>> org.apache.oodt.cas.pushpull.daemon.DaemonLauncher launchDaemons >>> INFO: Creating Daemon with ID = 1 >>> Jul 20, 2012 3:06:44 AM >>> org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning >>> INFO: Daemon with ID = 91131 was given permission to run >>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser >>> Jul 20, 2012 3:06:44 AM >>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >>> retrieveFiles >>> INFO: Creating 'org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler' >>> to download data files >>> Jul 20, 2012 3:06:49 AM >>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol >>> INFO: Testing protocol >>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this may >>> take a few minutes . . . >>> Jul 20, 2012 3:06:59 AM >>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >>> username 'anonymous' >>> Jul 20, 2012 3:06:59 AM >>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >>> retrieveFiles >>> SEVERE: Failed to finish downloading per property files >>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>> : null >>> Jul 20, 2012 3:06:59 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep >>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to >>> sleep until Fri Jul 20 03:09:00 KST 2012 >>> >> Thanks let me know what you think above! >> >> Cheers, >> Chris >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Chris Mattmann, Ph.D. >> Senior Computer Scientist >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> Office: 171-266B, Mailstop: 171-246 >> Email: chris.a.mattm...@nasa.gov >> WWW: http://sunset.usc.edu/~mattmann/ >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Adjunct Assistant Professor, Computer Science Department >> University of Southern California, Los Angeles, CA 90089 USA >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>
