Hi YunHee, Thanks for your mail.
Let me take a look at the configs again. Does anyone else using PushPull have any advice for YunHee here? That error message seems to indicate there is a config file error somewhere. I'm wondering if your etc/push_pull_framework.properties has all the right values in there. Did you go through Cameron's guide to getting Push Pull running on the wiki? Cheers, Chris On Jul 24, 2012, at 10:48 AM, YunHee Kang wrote: > Hi Chris, > > There are some problems in the configuration files. > I found there is a dead link which does not connect two information > in two files :RemoteSpecs.xml and ExternalSources.xml. > For example an alias named "DewFtp" in RemoteSpecs.xml has no > associated source in "ExternalSources.xml. > > After I tailored those two XML files, I just tested one of > configurations, whose alias is PolarNcepNoaaFtp. > I also faced the same problem as follows: > > INFO: Successfully connected to ftp://polar.ncep.noaa.gov with > protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and > username 'anonymous' > Jul 25, 2012 2:21:04 AM > org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup > retrieveFiles > SEVERE: Failed to finish downloading per property files > /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml > : null > Jul 25, 2012 2:21:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep > > > Thanks, > Yunhee > > 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >> Hi YunHee, >> >> Thanks! >> >> Cheers, >> Chris >> >> On Jul 24, 2012, at 9:54 AM, YunHee Kang wrote: >> >>> Hi Chris, >>> >>>> Thanks, can you try some of the other XML file configs. >>> Yes, I will try it and let you know its result. >>> >>>> Maybe it's being blocked in your country, >>> That site may be blocked. >>> >>> >>> Thanks, >>> Yunhee >>> >>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >>>> Hi YunHee, >>>> >>>> Thanks, can you try some of the other XML file configs. Maybe it's being >>>> blocked in your country, or something, but let's see if we can get some >>>> of the other XML RemoteSpec configs to work? >>>> >>>> Cheers, >>>> Chris >>>> >>>> On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote: >>>> >>>>> Hi Chris, >>>>> >>>>> >>>>>> Try out the MODIS one there, does that work? >>>>> No, it doesn't. >>>>> I think that the ftp site e4ftl01u.ecs.nasa.gov has a connection >>>>> problem. >>>>> >>>>> $ ftp e4ftl01u.ecs.nasa.gov >>>>> ftp: connect: Connection timed out >>>>> ftp> >>>>> >>>>> The message below also say the reason why the problem was occurred. >>>>> WARNING: Error occurred while connecting to RemoteSite: alias = >>>>> 'ModisFtp' url = 'ftp://e4ftl01u.ecs.nasa.gov' username = >>>>> 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed to >>>>> connect to server : Connection timed out >>>>> Jul 25, 2012 1:31:25 AM >>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>>>> INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp' url = >>>>> 'ftp://e4ftl01u.ecs.nasa.gov' username = 'anonymous' cdTestDir = >>>>> 'null' maxConnections = '-1' in 5 seconds >>>>> Waiting . . . . . >>>>> >>>>> Thanks, >>>>> Yunhee >>>>> >>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >>>>>> Hi YunHee, >>>>>> >>>>>> OK, can you try some of the other files, e.g., the ones in: >>>>>> >>>>>> http://s.apache.org/ks0 >>>>>> >>>>>> Try out the MODIS one there, does that work? >>>>>> >>>>>> Cheers, >>>>>> Chris >>>>>> >>>>>> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote: >>>>>> >>>>>>> Hi Chris, >>>>>>> >>>>>>> I added the header of XML as follows: >>>>>>> <?xml version="1.0"?> >>>>>>> >>>>>>> Next the value of elements was changed into specific one. >>>>>>> The following is the output of the XML: >>>>>>> <?xml version="1.0"?> >>>>>>> <root> >>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005"> >>>>>>> <nofiles/> >>>>>>> <!--dir name="\d{4}\.\d{2}\.\d{2}"--> >>>>>>> <dir name="2004\.09\.20"> >>>>>>> <nodirs/> >>>>>>> <!--file >>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/--> >>>>>>> <file name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/> >>>>>>> </dir> >>>>>>> </dirstruct> >>>>>>> </root> >>>>>>> >>>>>>> Unfortunately I got the same result before: >>>>>>> ... >>>>>>> INFO: Testing protocol >>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this ma >>>>>>> y take a few minutes . . . >>>>>>> Jul 25, 2012 12:30:03 AM >>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >>>>>>> protocol 'org.apache.oodt. >>>>>>> cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous' >>>>>>> Jul 25, 2012 12:30:03 AM >>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup retrie >>>>>>> veFiles >>>>>>> SEVERE: Failed to finish downloading per property files >>>>>>> /home/yhkang/oodt/cas-pushpull/etc/ >>>>>>> examples/DirStructXmlParserFiles/TESL2CO2.xml : null >>>>>>> Jul 25, 2012 12:30:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon >>>>>>> sleep >>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to >>>>>>> sleep until Wed Jul 25 0 >>>>>>> ... >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Yunhee >>>>>>> >>>>>>> >>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >>>>>>>> Hi YunHee, >>>>>>>> >>>>>>>> Can you try simply putting a specific <dir name="..."> instead of >>>>>>>> \d{4}, etc., and >>>>>>>> then can you put a specific file name in the <file name="..">. If that >>>>>>>> works and >>>>>>>> downloads, then we know there is something up with the regex somehow. >>>>>>>> Also can you try adding the <?xml version="1.0"?> start to your file? >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Chris >>>>>>>> >>>>>>>> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote: >>>>>>>> >>>>>>>>> Hi Chris, >>>>>>>>> >>>>>>>>> >>>>>>>>> The following is the content of TESL2CO2.xml. >>>>>>>>> $ cat >>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>>>> <root> >>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005"> >>>>>>>>> <nofiles/> >>>>>>>>> <dir name="\d{4}\.\d{2}\.\d{2}"> >>>>>>>>> <nodirs/> >>>>>>>>> <file >>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/> >>>>>>>>> </dir> >>>>>>>>> </dirstruct> >>>>>>>>> </root> >>>>>>>>> >>>>>>>>> Let me know if there is a problem in the configuration file. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Yunhee >>>>>>>>> >>>>>>>>> 2012/7/24 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >>>>>>>>>> Hi YunHee, >>>>>>>>>> >>>>>>>>>> On Jul 24, 2012, at 12:30 AM, YunHee Kang wrote: >>>>>>>>>> >>>>>>>>>>> Hi Chris, >>>>>>>>>>> >>>>>>>>>>> I am stuck on the problem related with "pushpull" script occurred >>>>>>>>>>> the following error message: >>>>>>>>>>> .... >>>>>>>>>>> INFO: Testing protocol >>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this >>>>>>>>>>> may >>>>>>>>>>> take a few minutes . . . >>>>>>>>>>> Jul 24, 2012 4:05:02 PM >>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >>>>>>>>>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' >>>>>>>>>>> and >>>>>>>>>>> username 'anonymous' >>>>>>>>>>> Jul 24, 2012 4:05:02 PM >>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >>>>>>>>>>> retrieveFiles >>>>>>>>>>> SEVERE: Failed to finish downloading per property files >>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>>>>>> : null >>>>>>>>>> >>>>>>>>>> Can you do me a favor and simply run this command: >>>>>>>>>> >>>>>>>>>> cat >>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>>>>> >>>>>>>>>> What do you get? >>>>>>>>>> >>>>>>>>>>> Jul 24, 2012 4:05:02 PM org.apache.oodt.cas.pushpull.daemon.Daemon >>>>>>>>>>> sleep >>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to >>>>>>>>>>> sleep until Tue Jul 24 16:06:00 KST 2012 >>>>>>>>>>> Jul 24, 2012 4:06:00 PM >>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning >>>>>>>>>>> INFO: Daemon with ID = 91131 was given permission to run >>>>>>>>>>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser >>>>>>>>>>> >>>>>>>>>>> First of all, I would like test the pushpull framework to move data >>>>>>>>>>> written in NetCDF from a remote site by a network protocols (like >>>>>>>>>>> ftp, >>>>>>>>>>> http, etc). >>>>>>>>>> >>>>>>>>>> Yep, sounds good. >>>>>>>>>> >>>>>>>>>>> I think that there is a test suite which can be used to check the >>>>>>>>>>> pushpull framework. >>>>>>>>>>> Can you give configuration files(a set of XML files) that I can >>>>>>>>>>> test >>>>>>>>>>> downloading files from a FTP site (public anonymous site) ? >>>>>>>>>> >>>>>>>>>> The examples in the resources directory that you found should work >>>>>>>>>> so we'll just need to figure out why they aren't working on your >>>>>>>>>> system. >>>>>>>>>> >>>>>>>>>>> or >>>>>>>>>>> Please let me know a reference implementation of pushpull framework >>>>>>>>>>> that I can use to move NetCDF files. >>>>>>>>>> >>>>>>>>>> Thanks we'll figure this out. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Chris >>>>>>>>>> >>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>> Chris Mattmann, Ph.D. >>>>>>>>>> Senior Computer Scientist >>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>> Chris Mattmann, Ph.D. >>>>>>>> Senior Computer Scientist >>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>> >>>>>> >>>>>> >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> Chris Mattmann, Ph.D. >>>>>> Senior Computer Scientist >>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>> Email: chris.a.mattm...@nasa.gov >>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> >>>> >>>> >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> Chris Mattmann, Ph.D. >>>> Senior Computer Scientist >>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>> Office: 171-266B, Mailstop: 171-246 >>>> Email: chris.a.mattm...@nasa.gov >>>> WWW: http://sunset.usc.edu/~mattmann/ >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> Adjunct Assistant Professor, Computer Science Department >>>> University of Southern California, Los Angeles, CA 90089 USA >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Chris Mattmann, Ph.D. >> Senior Computer Scientist >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> Office: 171-266B, Mailstop: 171-246 >> Email: chris.a.mattm...@nasa.gov >> WWW: http://sunset.usc.edu/~mattmann/ >> Phone: +1 (818) 354-8810 >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Adjunct Assistant Professor, Computer Science Department >> University of Southern California, Los Angeles, CA 90089 USA >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
