Re: Problem when running pushpull

Brian Foster Tue, 24 Jul 2012 14:48:05 -0700

YunHee,

There is a NullPointerException being thrown... on line 118 of org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.java change it from:

LOG.log(Level.SEVERE,

"Failed to finish downloading per property files "

+ propFile.getAbsolutePath() + " : "

+ e.getMessage());

to:

LOG.log(Level.SEVERE,

"Failed to finish downloading per property files "

+ propFile.getAbsolutePath() + " : "

+ e.getMessage(), e);

rebuild and deploy and that should give you the stacktrace which will answer what is going on here.

-brian

On Jul 24, 2012, at 10:51 AM, "Mattmann, Chris A (388J)" <chris.a.mattm...@jpl.nasa.gov> wrote:

Hi YunHee,

Thanks for your mail.

Let me take a look at the configs again. Does anyone else using
PushPull have any advice for YunHee here? That error message
seems to indicate there is a config file error somewhere. I'm wondering
if your etc/push_pull_framework.properties has all the right values
in there. Did you go through Cameron's guide to getting Push Pull
running on the wiki?

Cheers,
Chris

On Jul 24, 2012, at 10:48 AM, YunHee Kang wrote:

> Hi Chris,
>
> There are some problems in the configuration files.
> I found there is a dead link which does not connect two information
> in two files :RemoteSpecs.xml and ExternalSources.xml.
> For example an alias named "DewFtp" in RemoteSpecs.xml has no
> associated source in "ExternalSources.xml.
>
> After I tailored those two XML files, I just tested one of
> configurations, whose alias is PolarNcepNoaaFtp.
> I also faced the same problem as follows:
>
> INFO: Successfully connected to ftp://polar.ncep.noaa.gov with
> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
> username 'anonymous'
> Jul 25, 2012 2:21:04 AM
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
> retrieveFiles
> SEVERE: Failed to finish downloading per property files
> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
> : null
> Jul 25, 2012 2:21:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>
>
> Thanks,
> Yunhee
>
> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>> Hi YunHee,
>>
>> Thanks!
>>
>> Cheers,
>> Chris
>>
>> On Jul 24, 2012, at 9:54 AM, YunHee Kang wrote:
>>
>>> Hi Chris,
>>>
>>>> Thanks, can you try some of the other XML file configs.
>>> Yes, I will try it and let you know its result.
>>>
>>>> Maybe it's being blocked in your country,
>>> That site may be blocked.
>>>
>>>
>>> Thanks,
>>> Yunhee
>>>
>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>>>> Hi YunHee,
>>>>
>>>> Thanks, can you try some of the other XML file configs. Maybe it's being
>>>> blocked in your country, or something, but let's see if we can get some
>>>> of the other XML RemoteSpec configs to work?
>>>>
>>>> Cheers,
>>>> Chris
>>>>
>>>> On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>>
>>>>>> Try out the MODIS one there, does that work?
>>>>> No, it doesn't.
>>>>> I think that the ftp site e4ftl01u.ecs.nasa.gov has a connection problem.
>>>>>
>>>>> $ ftp e4ftl01u.ecs.nasa.gov
>>>>> ftp: connect: Connection timed out
>>>>> ftp>
>>>>>
>>>>> The message below also say the reason why the problem was occurred.
>>>>> WARNING: Error occurred while connecting to RemoteSite: alias =
>>>>> 'ModisFtp' url = '' username =
>>>>> 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed to
>>>>> connect to server : Connection timed out
>>>>> Jul 25, 2012 1:31:25 AM
>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>> INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp' url =""> >>>>> 'ftp://e4ftl01u.ecs.nasa.gov' username = 'anonymous' cdTestDir =
>>>>> 'null' maxConnections = '-1' in 5 seconds
>>>>> Waiting . . . . .
>>>>>
>>>>> Thanks,
>>>>> Yunhee
>>>>>
>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>>>>>> Hi YunHee,
>>>>>>
>>>>>> OK, can you try some of the other files, e.g., the ones in:
>>>>>>
>>>>>> http://s.apache.org/ks0
>>>>>>
>>>>>> Try out the MODIS one there, does that work?
>>>>>>
>>>>>> Cheers,
>>>>>> Chris
>>>>>>
>>>>>> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote:
>>>>>>
>>>>>>> Hi Chris,
>>>>>>>
>>>>>>> I added the header of XML as follows:
>>>>>>> <?xml version="1.0"?>
>>>>>>>
>>>>>>> Next the value of elements was changed into specific one.
>>>>>>> The following is the output of the XML:
>>>>>>> <?xml version="1.0"?>
>>>>>>> <root>
>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005">
>>>>>>> <nofiles/>
>>>>>>> 
>>>>>>> <dir name="2004\.09\.20">
>>>>>>> <nodirs/>
>>>>>>> 
>>>>>>> <file name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/>
>>>>>>> </dir>
>>>>>>> </dirstruct>
>>>>>>> </root>
>>>>>>>
>>>>>>> Unfortunately I got the same result before:
>>>>>>> ...
>>>>>>> INFO: Testing protocol
>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this ma
>>>>>>> y take a few minutes . . .
>>>>>>> Jul 25, 2012 12:30:03 AM
>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>>>>>>> protocol 'org.apache.oodt.
>>>>>>> cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous'
>>>>>>> Jul 25, 2012 12:30:03 AM
>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup retrie
>>>>>>> veFiles
>>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/
>>>>>>> examples/DirStructXmlParserFiles/TESL2CO2.xml : null
>>>>>>> Jul 25, 2012 12:30:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to
>>>>>>> sleep until Wed Jul 25 0
>>>>>>> ...
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Yunhee
>>>>>>>
>>>>>>>
>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>>>>>>>> Hi YunHee,
>>>>>>>>
>>>>>>>> Can you try simply putting a specific <dir name="..."> instead of \d{4}, etc., and
>>>>>>>> then can you put a specific file name in the <file name="..">. If that works and
>>>>>>>> downloads, then we know there is something up with the regex somehow.
>>>>>>>> Also can you try adding the <?xml version="1.0"?> start to your file?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Chris
>>>>>>>>
>>>>>>>> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote:
>>>>>>>>
>>>>>>>>> Hi Chris,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The following is the content of TESL2CO2.xml.
>>>>>>>>> $ cat /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>> <root>
>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005">
>>>>>>>>> <nofiles/>
>>>>>>>>> <dir name="\d{4}\.\d{2}\.\d{2}">
>>>>>>>>> <nodirs/>
>>>>>>>>> <file name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/>
>>>>>>>>> </dir>
>>>>>>>>> </dirstruct>
>>>>>>>>> </root>
>>>>>>>>>
>>>>>>>>> Let me know if there is a problem in the configuration file.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Yunhee
>>>>>>>>>
>>>>>>>>> 2012/7/24 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>>>>>>>>>> Hi YunHee,
>>>>>>>>>>
>>>>>>>>>> On Jul 24, 2012, at 12:30 AM, YunHee Kang wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>
>>>>>>>>>>> I am stuck on the problem related with "pushpull" script occurred
>>>>>>>>>>> the following error message:
>>>>>>>>>>> ....
>>>>>>>>>>> INFO: Testing protocol
>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this may
>>>>>>>>>>> take a few minutes . . .
>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>>>>>>>>>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>>>>>>>>>> username 'anonymous'
>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>>>>>>>>>> retrieveFiles
>>>>>>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>>> : null
>>>>>>>>>>
>>>>>>>>>> Can you do me a favor and simply run this command:
>>>>>>>>>>
>>>>>>>>>> cat /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>>
>>>>>>>>>> What do you get?
>>>>>>>>>>
>>>>>>>>>>> Jul 24, 2012 4:05:02 PM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to
>>>>>>>>>>> sleep until Tue Jul 24 16:06:00 KST 2012
>>>>>>>>>>> Jul 24, 2012 4:06:00 PM
>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning
>>>>>>>>>>> INFO: Daemon with ID = 91131 was given permission to run
>>>>>>>>>>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
>>>>>>>>>>>
>>>>>>>>>>> First of all, I would like test the pushpull framework to move data
>>>>>>>>>>> written in NetCDF from a remote site by a network protocols (like ftp,
>>>>>>>>>>> http, etc).
>>>>>>>>>>
>>>>>>>>>> Yep, sounds good.
>>>>>>>>>>
>>>>>>>>>>> I think that there is a test suite which can be used to check the
>>>>>>>>>>> pushpull framework.
>>>>>>>>>>> Can you give configuration files(a set of XML files) that I can test
>>>>>>>>>>> downloading files from a FTP site (public anonymous site) ?
>>>>>>>>>>
>>>>>>>>>> The examples in the resources directory that you found should work
>>>>>>>>>> so we'll just need to figure out why they aren't working on your system.
>>>>>>>>>>
>>>>>>>>>>> or
>>>>>>>>>>> Please let me know a reference implementation of pushpull framework
>>>>>>>>>>> that I can use to move NetCDF files.
>>>>>>>>>>
>>>>>>>>>> Thanks we'll figure this out.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Chris
>>>>>>>>>>
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>>> Senior Computer Scientist
>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>> Senior Computer Scientist
>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Chris Mattmann, Ph.D.
>>>>>> Senior Computer Scientist
>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>
>>>>
>>>>
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Chris Mattmann, Ph.D.
>>>> Senior Computer Scientist
>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> Office: 171-266B, Mailstop: 171-246
>>>> Email: chris.a.mattm...@nasa.gov
>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Adjunct Assistant Professor, Computer Science Department
>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattm...@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> Phone: +1 (818) 354-8810
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Re: Problem when running pushpull

Reply via email to