Hi Chris, > Did you go through Cameron's guide to getting Push Pull > running on the wiki? Yes, I did. My etc/push_pull_framework.properties was setup according to the wiki page(https://cwiki.apache.org/OODT/oodt-push-pull-user-guide.html).
The following ilists are a set of properties in my etc/push_pull_framework.properties: #external configuration files org.apache.oodt.cas.pushpull.config.external.properties.files=/home/yhkang/oodt/cas-pushpull/etc/default.properties # ingester filemgr url org.apache.oodt.cas.filemgr.url=http://localhost:9000 #protocolfactory specification for protocol types org.apache.oodt.cas.pushpull.config.protocolfactory.info.files=/home/yhkang/oodt/cas-pushpull/policy/ProtocolFactoryInfo.xml #parser to retrievalmethod map org.apache.oodt.cas.pushpull.config.parser.info.files=/home/yhkang/oodt/cas-pushpull/policy/ParserToRetrievalMethodMap.xml #directory below which all data file will be downloaded to org.apache.oodt.cas.pushpull.data.files.base.staging.area=/home/yhkang/oodt/cas-pushpull/staging #unique metadata element info org.apache.oodt.cas.pushpull.config.type.detection.file=/home/yhkang/oodt/cas-pushpull/policy/mimetypes.xml Let me know if there is a problem in the properties. Thanks, Yunhee 2012/7/25 Mattmann, Chris A (388J) <[email protected]>: > Hi YunHee, > > Thanks for your mail. > > Let me take a look at the configs again. Does anyone else using > PushPull have any advice for YunHee here? That error message > seems to indicate there is a config file error somewhere. I'm wondering > if your etc/push_pull_framework.properties has all the right values > in there. Did you go through Cameron's guide to getting Push Pull > running on the wiki? > > Cheers, > Chris > > On Jul 24, 2012, at 10:48 AM, YunHee Kang wrote: > >> Hi Chris, >> >> There are some problems in the configuration files. >> I found there is a dead link which does not connect two information >> in two files :RemoteSpecs.xml and ExternalSources.xml. >> For example an alias named "DewFtp" in RemoteSpecs.xml has no >> associated source in "ExternalSources.xml. >> >> After I tailored those two XML files, I just tested one of >> configurations, whose alias is PolarNcepNoaaFtp. >> I also faced the same problem as follows: >> >> INFO: Successfully connected to ftp://polar.ncep.noaa.gov with >> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >> username 'anonymous' >> Jul 25, 2012 2:21:04 AM >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >> retrieveFiles >> SEVERE: Failed to finish downloading per property files >> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml >> : null >> Jul 25, 2012 2:21:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep >> >> >> Thanks, >> Yunhee >> >> 2012/7/25 Mattmann, Chris A (388J) <[email protected]>: >>> Hi YunHee, >>> >>> Thanks! >>> >>> Cheers, >>> Chris >>> >>> On Jul 24, 2012, at 9:54 AM, YunHee Kang wrote: >>> >>>> Hi Chris, >>>> >>>>> Thanks, can you try some of the other XML file configs. >>>> Yes, I will try it and let you know its result. >>>> >>>>> Maybe it's being blocked in your country, >>>> That site may be blocked. >>>> >>>> >>>> Thanks, >>>> Yunhee >>>> >>>> 2012/7/25 Mattmann, Chris A (388J) <[email protected]>: >>>>> Hi YunHee, >>>>> >>>>> Thanks, can you try some of the other XML file configs. Maybe it's being >>>>> blocked in your country, or something, but let's see if we can get some >>>>> of the other XML RemoteSpec configs to work? >>>>> >>>>> Cheers, >>>>> Chris >>>>> >>>>> On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote: >>>>> >>>>>> Hi Chris, >>>>>> >>>>>> >>>>>>> Try out the MODIS one there, does that work? >>>>>> No, it doesn't. >>>>>> I think that the ftp site e4ftl01u.ecs.nasa.gov has a connection >>>>>> problem. >>>>>> >>>>>> $ ftp e4ftl01u.ecs.nasa.gov >>>>>> ftp: connect: Connection timed out >>>>>> ftp> >>>>>> >>>>>> The message below also say the reason why the problem was occurred. >>>>>> WARNING: Error occurred while connecting to RemoteSite: alias = >>>>>> 'ModisFtp' url = 'ftp://e4ftl01u.ecs.nasa.gov' username = >>>>>> 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed to >>>>>> connect to server : Connection timed out >>>>>> Jul 25, 2012 1:31:25 AM >>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>>>>> INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp' url = >>>>>> 'ftp://e4ftl01u.ecs.nasa.gov' username = 'anonymous' cdTestDir = >>>>>> 'null' maxConnections = '-1' in 5 seconds >>>>>> Waiting . . . . . >>>>>> >>>>>> Thanks, >>>>>> Yunhee >>>>>> >>>>>> 2012/7/25 Mattmann, Chris A (388J) <[email protected]>: >>>>>>> Hi YunHee, >>>>>>> >>>>>>> OK, can you try some of the other files, e.g., the ones in: >>>>>>> >>>>>>> http://s.apache.org/ks0 >>>>>>> >>>>>>> Try out the MODIS one there, does that work? >>>>>>> >>>>>>> Cheers, >>>>>>> Chris >>>>>>> >>>>>>> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote: >>>>>>> >>>>>>>> Hi Chris, >>>>>>>> >>>>>>>> I added the header of XML as follows: >>>>>>>> <?xml version="1.0"?> >>>>>>>> >>>>>>>> Next the value of elements was changed into specific one. >>>>>>>> The following is the output of the XML: >>>>>>>> <?xml version="1.0"?> >>>>>>>> <root> >>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005"> >>>>>>>> <nofiles/> >>>>>>>> <!--dir name="\d{4}\.\d{2}\.\d{2}"--> >>>>>>>> <dir name="2004\.09\.20"> >>>>>>>> <nodirs/> >>>>>>>> <!--file >>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/--> >>>>>>>> <file >>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/> >>>>>>>> </dir> >>>>>>>> </dirstruct> >>>>>>>> </root> >>>>>>>> >>>>>>>> Unfortunately I got the same result before: >>>>>>>> ... >>>>>>>> INFO: Testing protocol >>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this ma >>>>>>>> y take a few minutes . . . >>>>>>>> Jul 25, 2012 12:30:03 AM >>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >>>>>>>> protocol 'org.apache.oodt. >>>>>>>> cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous' >>>>>>>> Jul 25, 2012 12:30:03 AM >>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup retrie >>>>>>>> veFiles >>>>>>>> SEVERE: Failed to finish downloading per property files >>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/ >>>>>>>> examples/DirStructXmlParserFiles/TESL2CO2.xml : null >>>>>>>> Jul 25, 2012 12:30:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon >>>>>>>> sleep >>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to >>>>>>>> sleep until Wed Jul 25 0 >>>>>>>> ... >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Yunhee >>>>>>>> >>>>>>>> >>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <[email protected]>: >>>>>>>>> Hi YunHee, >>>>>>>>> >>>>>>>>> Can you try simply putting a specific <dir name="..."> instead of >>>>>>>>> \d{4}, etc., and >>>>>>>>> then can you put a specific file name in the <file name="..">. If >>>>>>>>> that works and >>>>>>>>> downloads, then we know there is something up with the regex somehow. >>>>>>>>> Also can you try adding the <?xml version="1.0"?> start to your file? >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Chris >>>>>>>>> >>>>>>>>> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote: >>>>>>>>> >>>>>>>>>> Hi Chris, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The following is the content of TESL2CO2.xml. >>>>>>>>>> $ cat >>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>>>>> <root> >>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005"> >>>>>>>>>> <nofiles/> >>>>>>>>>> <dir name="\d{4}\.\d{2}\.\d{2}"> >>>>>>>>>> <nodirs/> >>>>>>>>>> <file >>>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/> >>>>>>>>>> </dir> >>>>>>>>>> </dirstruct> >>>>>>>>>> </root> >>>>>>>>>> >>>>>>>>>> Let me know if there is a problem in the configuration file. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Yunhee >>>>>>>>>> >>>>>>>>>> 2012/7/24 Mattmann, Chris A (388J) <[email protected]>: >>>>>>>>>>> Hi YunHee, >>>>>>>>>>> >>>>>>>>>>> On Jul 24, 2012, at 12:30 AM, YunHee Kang wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Chris, >>>>>>>>>>>> >>>>>>>>>>>> I am stuck on the problem related with "pushpull" script occurred >>>>>>>>>>>> the following error message: >>>>>>>>>>>> .... >>>>>>>>>>>> INFO: Testing protocol >>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this >>>>>>>>>>>> may >>>>>>>>>>>> take a few minutes . . . >>>>>>>>>>>> Jul 24, 2012 4:05:02 PM >>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >>>>>>>>>>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' >>>>>>>>>>>> and >>>>>>>>>>>> username 'anonymous' >>>>>>>>>>>> Jul 24, 2012 4:05:02 PM >>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >>>>>>>>>>>> retrieveFiles >>>>>>>>>>>> SEVERE: Failed to finish downloading per property files >>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>>>>>>> : null >>>>>>>>>>> >>>>>>>>>>> Can you do me a favor and simply run this command: >>>>>>>>>>> >>>>>>>>>>> cat >>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>>>>>> >>>>>>>>>>> What do you get? >>>>>>>>>>> >>>>>>>>>>>> Jul 24, 2012 4:05:02 PM org.apache.oodt.cas.pushpull.daemon.Daemon >>>>>>>>>>>> sleep >>>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to >>>>>>>>>>>> sleep until Tue Jul 24 16:06:00 KST 2012 >>>>>>>>>>>> Jul 24, 2012 4:06:00 PM >>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning >>>>>>>>>>>> INFO: Daemon with ID = 91131 was given permission to run >>>>>>>>>>>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser >>>>>>>>>>>> >>>>>>>>>>>> First of all, I would like test the pushpull framework to move >>>>>>>>>>>> data >>>>>>>>>>>> written in NetCDF from a remote site by a network protocols (like >>>>>>>>>>>> ftp, >>>>>>>>>>>> http, etc). >>>>>>>>>>> >>>>>>>>>>> Yep, sounds good. >>>>>>>>>>> >>>>>>>>>>>> I think that there is a test suite which can be used to check the >>>>>>>>>>>> pushpull framework. >>>>>>>>>>>> Can you give configuration files(a set of XML files) that I can >>>>>>>>>>>> test >>>>>>>>>>>> downloading files from a FTP site (public anonymous site) ? >>>>>>>>>>> >>>>>>>>>>> The examples in the resources directory that you found should work >>>>>>>>>>> so we'll just need to figure out why they aren't working on your >>>>>>>>>>> system. >>>>>>>>>>> >>>>>>>>>>>> or >>>>>>>>>>>> Please let me know a reference implementation of pushpull >>>>>>>>>>>> framework >>>>>>>>>>>> that I can use to move NetCDF files. >>>>>>>>>>> >>>>>>>>>>> Thanks we'll figure this out. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Chris >>>>>>>>>>> >>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>> Chris Mattmann, Ph.D. >>>>>>>>>>> Senior Computer Scientist >>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>>>>> Email: [email protected] >>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>> Chris Mattmann, Ph.D. >>>>>>>>> Senior Computer Scientist >>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>>> Email: [email protected] >>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>> Chris Mattmann, Ph.D. >>>>>>> Senior Computer Scientist >>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>> Email: [email protected] >>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>> >>>>> >>>>> >>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>> Chris Mattmann, Ph.D. >>>>> Senior Computer Scientist >>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>> Office: 171-266B, Mailstop: 171-246 >>>>> Email: [email protected] >>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>> Adjunct Assistant Professor, Computer Science Department >>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>> >>> >>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> Chris Mattmann, Ph.D. >>> Senior Computer Scientist >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>> Office: 171-266B, Mailstop: 171-246 >>> Email: [email protected] >>> WWW: http://sunset.usc.edu/~mattmann/ >>> Phone: +1 (818) 354-8810 >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> Adjunct Assistant Professor, Computer Science Department >>> University of Southern California, Los Angeles, CA 90089 USA >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Chris Mattmann, Ph.D. > Senior Computer Scientist > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > Office: 171-266B, Mailstop: 171-246 > Email: [email protected] > WWW: http://sunset.usc.edu/~mattmann/ > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Adjunct Assistant Professor, Computer Science Department > University of Southern California, Los Angeles, CA 90089 USA > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >
