Hi Brain, I will try to apply the patch to my oodt source code.
Thanks, Yunhee 2012/7/26 Brian Foster <holeno...@mac.com>: > hey Yunhee, > > i just realized that your probably not on the dev list and my last email > just went to the list and didn't have you cc'ed > > try the patch attached to this issue mentioned below... > > -brian > > Begin forwarded message: > > From: Brian Foster <holeno...@mac.com> > Date: July 25, 2012 12:43:10 AM > To: dev@oodt.apache.org,dev@oodt.apache.org > Subject: Re: Problem when running pushpull > > hey Yunhee, > > Try patching your pushpull code with the fix i've attached to issue: > > https://issues.apache.org/jira/browse/OODT-476 > > -brian > > On Jul 24, 2012, at 09:24 PM, YunHee Kang <yunh.k...@gmail.com> wrote: > > Hi Brian, > > I changed the file RetrievalSetup.java to get the stacktrace. > So I got a new cas-pushpull-0.4.jar after rebuilt it. > > The following is a message including the stacktrace with information > about excpetion propation. > > SEVERE: Failed to finish downloading per property files > /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml > : null > java.lang.NullPointerException > at > org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:367) > at > org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:311) > at > org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:300) > at > org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:103) > at > org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) > at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) > at java.lang.Thread.run(Thread.java:662) > > Please let me know what java programs I check frist. > > Thanks, > > Yunhee > 2012/7/25 Brian Foster <holeno...@mac.com>: >> YunHee, >> >> There is a NullPointerException being thrown... on line 118 of >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.java change it >> >> rebuild and deploy and that should give you the stacktrace which will >> answer >> what is going on here. >> >> -brian >> >> >> On Jul 24, 2012, at 10:51 AM, "Mattmann, Chris A (388J)" > >> <chris.a.mattm...@jpl.nasa.gov> wrote: >> >> Hi YunHee, >> >> Thanks for your mail. >> >> Let me take a look at the configs again. Does anyone else using >> PushPull have any advice for YunHee here? That error message >> seems to indicate there is a config file error somewhere. I'm wondering >> if your etc/push_pull_framework.properties has all the right values >> in there. Did you go through Cameron's guide to getting Push Pull >> running on the wiki? >> >> Cheers, >> Chris > >> >> On Jul 24, 2012, at 10:48 AM, YunHee Kang wrote: >> >>> Hi Chris, >>> >>> There are some problems in the configuration files. >>> I found there is a dead link which does not connect two information >>> in two files :RemoteSpecs.xml and ExternalSources.xml. >>> For example an alias named "DewFtp" in RemoteSpecs.xml has no >>> associated source in "ExternalSources.xml. >>> >>> After I tailored those two XML files, I just tested one of >>> configurations, whose alias is PolarNcepNoaaFtp. >>> I also faced the same problem as follows: >>> >>> INFO: Successfully connected to ftp://polar.ncep.noaa.gov with >>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >>> username 'anonymous' >>> Jul 25, 2012 2:21:04 AM >>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >>> retrieveFiles >>> SEVERE: Failed to finish downloading per property files >>> >>> >>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml >>> : null >>> Jul 25, 2012 2:21:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep >>> >>> >>> Thanks, >>> Yunhee >>> >>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >>>> Hi YunHee, >>>> >>>> Thanks! >>>> >>>> Cheers, >>>> Chris > >>>> >>>> On Jul 24, 2012, at 9:54 AM, YunHee Kang wrote: >>>> >>>>> Hi Chris, >>>>> >>>>>> Thanks, can you try some of the other XML file configs. >>>>> Yes, I will try it and let you know its result. >>>>> >>>>>> Maybe it's being blocked in your country, >>>>> That site may be blocked. >>>>> >>>>> >>>>> Thanks, >>>>> Yunhee >>>>> >>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >>>>>> Hi YunHee, >>>>>> > >>>>>> Thanks, can you try some of the other XML file configs. Maybe it's >>>>>> being >>>>>> blocked in your country, or something, but let's see if we can get >>>>>> some >>>>>> of the other XML RemoteSpec configs to work? >>>>>> >>>>>> Cheers, >>>>>> Chris >>>>>> >>>>>> On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote: >>>>>> >>>>>>> Hi Chris, >>>>>>> >>>>>>> >>>>>>>> Try out the MODIS one there, does that work? >>>>>>> No, it doesn't. >>>>>>> I think that the ftp site e4ftl01u.ecs.nasa.gov has a connection >>>>>>> problem. >>>>>>> >>>>>>> $ ftp e4ftl01u.ecs.nasa.gov >>>>>>> ftp: connect: Connection timed out >>>>>>> ftp> >>>>>>> >>>>>>> The message below also say the reason why the problem was occurred. >>>>>>> WARNING: Error occurred while connecting to RemoteSite: alias = >>>>>>> 'ModisFtp' url = 'ftp://e4ftl01u.ecs.nasa.gov' username = >>>>>>> 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed to >>>>>>> connect to server : Connection timed out >>>>>>> Jul 25, 2012 1:31:25 AM >>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>>>>>> INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp' url = >>>>>>> 'ftp://e4ftl01u.ecs.nasa.gov' username = 'anonymous' cdTestDir = >>>>>>> 'null' maxConnections = '-1' in 5 seconds >>>>>>> Waiting . . . . . >>>>>>> >>>>>>> Thanks, >>>>>>> Yunhee >>>>>>> >>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >>>>>>>> Hi YunHee, >>>>>>>> > >>>>>>>> OK, can you try some of the other files, e.g., the ones in: >>>>>>>> >>>>>>>> http://s.apache.org/ks0 >>>>>>>> >>>>>>>> Try out the MODIS one there, does that work? >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Chris >>>>>>>> >>>>>>>> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote: >>>>>>>> >>>>>>>>> Hi Chris, >>>>>>>>> >>>>>>>>> I added the header of XML as follows: >>>>>>>>> <?xml version="1.0"?> >>>>>>>>> >>>>>>>>> Next the value of elements was changed into specific one. >>>>>>>>> The following is the output of the XML: >>>>>>>>> <?xml version="1.0"?> >>>>>>>>> <root> >>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005"> >>>>>>>>> <nofiles/> >>>>>>>>> <!--dir name="\d{4}\.\d{2}\.\d{2}"--> >>>>>>>>> <dir name="2004\.09\.20"> >>>>>>>>> <nodirs/> >>>>>>>>> <!--file >>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/--> >>>>>>>>> <file name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/> >>>>>>>>> </dir> >>>>>>>>> </dirstruct> >>>>>>>>> </root> >>>>>>>>> >>>>>>>>> Unfortunately I got the same result before: >>>>>>>>> ... >>>>>>>>> INFO: Testing protocol >>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this >>>>>>>>> ma >>>>>>>>> y take a few minutes . . . >>>>>>>>> Jul 25, 2012 12:30:03 AM >>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >>>>>>>>> protocol 'org.apache.oodt. >>>>>>>>> cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous' >>>>>>>>> Jul 25, 2012 12:30:03 AM >>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup retrie >>>>>>>>> veFiles >>>>>>>>> SEVERE: Failed to finish downloading per property files >>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/ >>>>>>>>> examples/DirStructXmlParserFiles/TESL2CO2.xml : null >>>>>>>>> Jul 25, 2012 12:30:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon >>>>>>>>> sleep >>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to >>>>>>>>> sleep until Wed Jul 25 0 >>>>>>>>> ... >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Yunhee >>>>>>>>> >>>>>>>>> >>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >>>>>>>>>> Hi YunHee, >>>>>>>>>> > >>>>>>>>>> Can you try simply putting a specific <dir name="..."> instead of >>>>>>>>>> \d{4}, etc., and >>>>>>>>>> then can you put a specific file name in the <file name="..">. If >>>>>>>>>> that works and >>>>>>>>>> downloads, then we know there is something up with the regex >>>>>>>>>> somehow. >>>>>>>>>> Also can you try adding the <?xml version="1.0"?> start to your >>>>>>>>>> file? >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Chris >>>>>>>>>> >>>>>>>>>> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote: >>>>>>>>>> >>>>>>>>>>> Hi Chris, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The following is the content of TESL2CO2.xml. >>>>>>>>>>> $ cat >>>>>>>>>>> >>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>>>>>> <root> >>>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005"> >>>>>>>>>>> <nofiles/> >>>>>>>>>>> <dir name="\d{4}\.\d{2}\.\d{2}"> >>>>>>>>>>> <nodirs/> >>>>>>>>>>> <file >>>>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/> >>>>>>>>>>> </dir> >>>>>>>>>>> </dirstruct> >>>>>>>>>>> </root> >>>>>>>>>>> >>>>>>>>>>> Let me know if there is a problem in the configuration file. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Yunhee >>>>>>>>>>> >>>>>>>>>>> 2012/7/24 Mattmann, Chris A (388J) >>>>>>>>>>> <chris.a.mattm...@jpl.nasa.gov>: >>>>>>>>>>>> Hi YunHee, > >>>>>>>>>>>> >>>>>>>>>>>> On Jul 24, 2012, at 12:30 AM, YunHee Kang wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Chris, >>>>>>>>>>>>> >>>>>>>>>>>>> I am stuck on the problem related with "pushpull" script >>>>>>>>>>>>> occurred >>>>>>>>>>>>> the following error message: >>>>>>>>>>>>> .... >>>>>>>>>>>>> INFO: Testing protocol >>>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . >>>>>>>>>>>>> this may >>>>>>>>>>>>> take a few minutes . . . >>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM >>>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>>>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov >>>>>>>>>>>>> with >>>>>>>>>>>>> protocol >>>>>>>>>>>>> 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >>>>>>>>>>>>> username 'anonymous' >>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM >>>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >>>>>>>>>>>>> retrieveFiles >>>>>>>>>>>>> SEVERE: Failed to finish downloading per property files >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>>>>>>>> : null >>>>>>>>>>>> >>>>>>>>>>>> Can you do me a favor and simply run this command: >>>>>>>>>>>> >>>>>>>>>>>> cat >>>>>>>>>>>> >>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>>>>>>> >>>>>>>>>>>> What do you get? >>>>>>>>>>>> >>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM >>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.Daemon sleep >>>>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going >>>>>>>>>>>>> to >>>>>>>>>>>>> sleep until Tue Jul 24 16:06:00 KST 2012 >>>>>>>>>>>>> Jul 24, 2012 4:06:00 PM >>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning >>>>>>>>>>>>> INFO: Daemon with ID = 91131 was given permission to run >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser >>>>>>>>>>>>> >>>>>>>>>>>>> First of all, I would like test the pushpull framework to move >>>>>>>>>>>>> data >>>>>>>>>>>>> written in NetCDF from a remote site by a network protocols >>>>>>>>>>>>> (like ftp, >>>>>>>>>>>>> http, etc). >>>>>>>>>>>> >>>>>>>>>>>> Yep, sounds good. >>>>>>>>>>>> >>>>>>>>>>>>> I think that there is a test suite which can be used to check >>>>>>>>>>>>> the >>>>>>>>>>>>> pushpull framework. >>>>>>>>>>>>> Can you give configuration files(a set of XML files) that I can >>>>>>>>>>>>> test >>>>>>>>>>>>> downloading files from a FTP site (public anonymous site) ? >>>>>>>>>>>> >>>>>>>>>>>> The examples in the resources directory that you found should >>>>>>>>>>>> work >>>>>>>>>>>> so we'll just need to figure out why they aren't working on your >>>>>>>>>>>> system. >>>>>>>>>>>> >>>>>>>>>>>>> or >>>>>>>>>>>>> Please let me know a reference implementation of pushpull >>>>>>>>>>>>> framework >>>>>>>>>>>>> that I can use to move NetCDF files. >>>>>>>>>>>> >>>>>>>>>>>> Thanks we'll figure this out. >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Chris >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>> Chris Mattmann, Ph.D. >>>>>>>>>>>> Senior Computer Scientist >>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>> Chris Mattmann, Ph.D. >>>>>>>>>> Senior Computer Scientist >>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>> Chris Mattmann, Ph.D. >>>>>>>> Senior Computer Scientist >>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>> >>>>>> >>>>>> >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> Chris Mattmann, Ph.D. >>>>>> Senior Computer Scientist >>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>> Email: chris.a.mattm...@nasa.gov >>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> >>>> >>>> >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> Chris Mattmann, Ph.D. >>>> Senior Computer Scientist >>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>> Office: 171-266B, Mailstop: 171-246 >>>> Email: chris.a.mattm...@nasa.gov >>>> WWW: http://sunset.usc.edu/~mattmann/ >>>> Phone: +1 (818) 354-8810 > >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> Adjunct Assistant Professor, Computer Science Department >>>> University of Southern California, Los Angeles, CA 90089 USA >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Chris Mattmann, Ph.D. >> Senior Computer Scientist >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> Office: 171-266B, Mailstop: 171-246 >> Email: chris.a.mattm...@nasa.gov >> WWW: http://sunset.usc.edu/~mattmann/ >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Adjunct Assistant Professor, Computer Science Department >> University of Southern California, Los Angeles, CA 90089 USA >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>
