Hi Brian,
I checked out the latest trunk of oodt 0.5 and installed it.
After I set up the file manger and pushpull framework, I ran the
pushpull script.
As I described below, I think that the trunk is useful to fix the
problem, which was caused by handling ftp protocol.
INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
username 'anonymous'
Jul 29, 2012 10:09:30 PM
org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem
initializeStagingArea
INFO: Preparing staging area /home/yhkang/oodt-0.5/cas-pushpull/staging/TESL2CO2
Jul 29, 2012 10:09:30 PM org.apache.commons.httpclient.HttpClient <clinit>
FINE: Java version: 1.6.0_31
But I got the difference problem that is related with filemgr.
The following is part of error message in the file cas-pushpull0.log :
Jul 29, 2012 10:09:30 PM
org.apache.oodt.cas.filemgr.ingest.StdIngester hasProduct
WARNING: Unable to check for existance of product: [null]: Message:
Failure writing request
org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
Failure writing request
at
org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(XmlRpcFileManagerClient.java:606)
at
org.apache.oodt.cas.filemgr.ingest.StdIngester.hasProduct(StdIngester.java:284)
at
org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.isAlreadyInDatabase(FileRetrievalSystem.java:254)
at
org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.addToDownloadQueue(FileRetrievalSystem.java:463)
at
org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:138)
at
org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
at java.lang.Thread.run(Thread.java:662)
Jul 29, 2012 10:09:30 PM
org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
retrieveFiles
SEVERE: Failed to finish downloading per property files
/home/yhkang/oodt-0.5/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
: [RemoteCrawler] Failed to communicate with database : Failure
writing request
org.apache.oodt.cas.pushpull.exceptions.RetrievalMethodException:
[RemoteCrawler] Failed to communicate with database : Failure writing
request
at
org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:150)
at
org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
at java.lang.Thread.run(Thread.java:662)
Jul 29, 2012 10:09:30 PM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
INFO: Disconnecting protocol
org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
Jul 29, 2012 10:09:30 PM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to
sleep until Sun Jul 29 22:12:00 KST 2012
Please let me know how to fix this problem.
Thanks,
Yunhee
2012/7/28 Brian Foster <holeno...@me.com>:
> Hey YunHee,
>
> Doesn't look like the patch took this time, but don't worry about that now, i
> have put the fixes into the trunk... if you just check out the latest trunk
> and mvn install it (make sure you check out more than just cas-pushpull,
> because if protocol-ftp doesn't get reinstalled your cas-pushpull mvn install
> will pick up the last mvn installed version of the protocol-ftp 0.5 in your
> local maven repository... another thing to note, you probably should utilize
> the environment variables in place in the pushpull config files because if
> you ever decide to move your install you are going to have to change all the
> paths in your configuration files instead of just being able to change
> environment variables (if you have any question about that let me know)...
> again if you run into any road blocks just hit up the list and sorry about
> the bumpy road of getting pushpull up and running... it should be good to go
> now though (it downloads the files on my machine at least... lol).
>
> -brian
>
> On Jul 27, 2012, at 10:56 AM, YunHee Kang wrote:
>
>> Hi Brain,
>>
>> Unfortunately your patch didn't fix the problem. I got the same
>> error message before:
>> SEVERE: Failed to finish downloading per property files
>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>> : null
>> java.lang.NullPointerException
>> at
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368)
>> at
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312)
>> at
>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301)
>> at
>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104)
>> at
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>> at java.lang.Thread.run(Thread.java:662)
>> Jul 28, 2012 2:31:32 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>>
>> First of all, I replaced CommonsNetFtpProtocol.java with new one
>> that you modified as follows:
>> ftp.retrieveFile(fromFile.getName(), os);
>> /*
>> if (ftp.retrieveFile(fromFile.getName(), os))// {
>> throw new ProtocolException("Failed to download file "
>> + fromFile.getName());
>> // }
>>
>> */
>> Next I rebuild a new jar file cas-protocol-ftp-0.4.jar and then
>> copied it to lib directory:
>> -rw-r--r-- 1 yhkang yhkang 17279 2012-07-28 02:09 cas-protocol-ftp-0.4.jar
>> ls -al cas-protocol-ftp-0.4.jar
>>
>> I am wondering that there is difference between the following codes(1 and
>> 2) :
>> 1) ftp.retrieveFile(fromFile.getName(), os);
>> 2) if (ftp.retrieveFile(fromFile.getName(), os))// {
>> throw new ProtocolException("Failed to download file "
>> + fromFile.getName());
>> // }
>>
>> Will you let me know if the method get() in the
>> CommonsNetFtpProtocol.java was called when pushpull was run ?
>>
>> Thanks,
>> Yunhee
>>
>>
>> 2012/7/27 Brian Foster <holeno...@mac.com>:
>>> Hey YunHee,
>>>
>>> Okay the latest patch attached to the issue now (the on with todays date)
>>> works as far as pushpull is concerned, however FTPClient.retrieveFile being
>>> used in CommonsNetFtpProtocol is returning false for some reason... so i
>>> would recommend trying the following... first unpatch the first patch from
>>> your cas-pushpull and repatch with the new patch i've uploaded... then check
>>> out https://svn.apache.org/repos/asf/oodt/trunk/protocol/ftp/ and modify
>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol line 160 from this:
>>>
>>>
>>> if (ftp.retrieveFile(fromFile.getName(), os))// {
>>>
>>> throw new ProtocolException("Failed to download file "
>>>
>>> + fromFile.getName());
>>>
>>> // }
>>>
>>>
>>> to this:
>>>
>>>
>>> ftp.retrieveFile(fromFile.getName(), os);
>>>
>>>
>>> then mvn compile jar:jar and replace ftp protocol jar with that one in your
>>> pushpull lib directory and give pushpull another run and see what happens
>>> (validate the files downloaded to make sure they are complete)... hope this
>>> helps and let me know if you need any further help!
>>>
>>>
>>> -brian
>>>
>>>
>>> On Jul 26, 2012, at 03:37 PM, YunHee Kang <yunh.k...@gmail.com> wrote:
>>>
>>> Hi Brian,
>>>
>>> I attach a tar file including etc and policy directories.
>>>
>>> Thanks,
>>> Yunhee
>>>
>>> 2012/7/27 Brian Foster <holeno...@mac.com>:
>>>> Hey YunHee,
>>>>
>>>> send me a tarball with your etc and policy directories i will spend some
>>>> time today figuring out what is going on here
>>>>
>>>> -brian
>>>>
>>>>
>>>> On Jul 26, 2012, at 01:26 AM, YunHee Kang <yunh.k...@gmail.com> wrote:
>>>>
>>>> Hi Brian,
>>>>
>>>> I just applied the patch you made according to the following steps:
>>>> cd apache-oodt-0.4-src/pushpull
>>>> patch -p0 < OODT-476.2012-07-25.txt
>>>> mvn clean
>>>> mvn install
>>>> cp target/cas-pushpull-0.4.jar ~/oodt/cas-pushpull/lib/
>>>>
>>>> After I got a new cas-pushpull-0.4.jar,
>>>> yhkang@yunhee-kisti:~/oodt/cas-pushpull/lib$ ls -l cas-pushpull-0.4.jar
>>>> -rw-r--r-- 1 yhkang yhkang 422602 2012-07-26 16:58 cas-pushpull-0.4.jar
>>>> I ran the script pushpull.
>>>>
>>>>
>>>> But the same error message was generated:
>>>>
>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>>> username 'anonymous'
>>>> Jul 26, 2012 5:12:56 PM
>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>>> retrieveFiles
>>>> SEVERE: Failed to finish downloading per property files
>>>>
>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>> : null
>>>> java.lang.NullPointerException
>>>> at
>>>>
>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368)
>>>> at
>>>>
>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312)
>>>> at
>>>>
>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301)
>>>> at
>>>>
>>>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104)
>>>> at
>>>>
>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>>>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> Could you check a class name for handling ftp protocol ?
>>>> I used "CommonsNetFtpProtocolFactory.class" in the jar named
>>>> "cas-protocol-ftp-0.4.jar".
>>>> The class file was described in the file ProtocolFactoryInfo.xml as
>>>> follows:
>>>> <protocol type="ftp">
>>>> ....
>>>> <!-- cas-protocol-ftp-0.4.jar -->
>>>> <!--protocolFactory
>>>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpClientFactory"/-->
>>>> <protocolFactory
>>>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory"/>
>>>> </protocol>
>>>>
>>>> Thanks,
>>>> Yunhee
>>>>
>>>>
>>>>
>>>>
>>>> 2012/7/26 Brian Foster <holeno...@mac.com>:
>>>>> hey Yunhee,
>>>>>
>>>>> i just realized that your probably not on the dev list and my last email
>>>>> just went to the list and didn't have you cc'ed
>>>>>
>>>>> try the patch attached to this issue mentioned below...
>>>>>
>>>>> -brian
>>>>>
>>>>> Begin forwarded message:
>>>>>
>>>>> From: Brian Foster <holeno...@mac.com>
>>>>> Date: July 25, 2012 12:43:10 AM
>>>>> To: dev@oodt.apache.org,dev@oodt.apache.org
>>>>> Subject: Re: Problem when running pushpull
>>>>>
>>>>> hey Yunhee,
>>>>>
>>>>> Try patching your pushpull code with the fix i've attached to issue:
>>>>>
>>>>> https://issues.apache.org/jira/browse/OODT-476
>>>>>
>>>>> -brian
>>>>>
>>>>> On Jul 24, 2012, at 09:24 PM, YunHee Kang <yunh.k...@gmail.com> wrote:
>>>>>
>>>>> Hi Brian,
>>>>>
>>>>> I changed the file RetrievalSetup.java to get the stacktrace.
>>>>> So I got a new cas-pushpull-0.4.jar after rebuilt it.
>>>>>
>>>>> The following is a message including the stacktrace with information
>>>>> about excpetion propation.
>>>>>
>>>>> SEVERE: Failed to finish downloading per property files
>>>>>
>>>>>
>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
>>>>> : null
>>>>> java.lang.NullPointerException
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:367)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:311)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:300)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:103)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>>>>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> Please let me know what java programs I check frist.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Yunhee
>>>>> 2012/7/25 Brian Foster <holeno...@mac.com>:
>>>>>> YunHee,
>>>>>>
>>>>>> There is a NullPointerException being thrown... on line 118 of
>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.java change
>>>>>> it
>>>>>>
>>>>>> rebuild and deploy and that should give you the stacktrace which will
>>>>>> answer
>>>>>> what is going on here.
>>>>>>
>>>>>> -brian
>>>>>>
>>>>>>
>>>>>> On Jul 24, 2012, at 10:51 AM, "Mattmann, Chris A (388J)"
>>>>>
>>>>>> <chris.a.mattm...@jpl.nasa.gov> wrote:
>>>>>>
>>>>>> Hi YunHee,
>>>>>>
>>>>>> Thanks for your mail.
>>>>>>
>>>>>> Let me take a look at the configs again. Does anyone else using
>>>>>> PushPull have any advice for YunHee here? That error message
>>>>>> seems to indicate there is a config file error somewhere. I'm wondering
>>>>>> if your etc/push_pull_framework.properties has all the right values
>>>>>> in there. Did you go through Cameron's guide to getting Push Pull
>>>>>> running on the wiki?
>>>>>>
>>>>>> Cheers,
>>>>>> Chris
>>>>>
>>>>>>
>>>>>> On Jul 24, 2012, at 10:48 AM, YunHee Kang wrote:
>>>>>>
>>>>>>> Hi Chris,
>>>>>>>
>>>>>>> There are some problems in the configuration files.
>>>>>>> I found there is a dead link which does not connect two information
>>>>>>> in two files :RemoteSpecs.xml and ExternalSources.xml.
>>>>>>> For example an alias named "DewFtp" in RemoteSpecs.xml has no
>>>>>>> associated source in "ExternalSources.xml.
>>>>>>>
>>>>>>> After I tailored those two XML files, I just tested one of
>>>>>>> configurations, whose alias is PolarNcepNoaaFtp.
>>>>>>> I also faced the same problem as follows:
>>>>>>>
>>>>>>> INFO: Successfully connected to ftp://polar.ncep.noaa.gov with
>>>>>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>>>>>> username 'anonymous'
>>>>>>> Jul 25, 2012 2:21:04 AM
>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>>>>>> retrieveFiles
>>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
>>>>>>> : null
>>>>>>> Jul 25, 2012 2:21:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon
>>>>>>> sleep
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Yunhee
>>>>>>>
>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>>>>>>>> Hi YunHee,
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Chris
>>>>>
>>>>>>>>
>>>>>>>> On Jul 24, 2012, at 9:54 AM, YunHee Kang wrote:
>>>>>>>>
>>>>>>>>> Hi Chris,
>>>>>>>>>
>>>>>>>>>> Thanks, can you try some of the other XML file configs.
>>>>>>>>> Yes, I will try it and let you know its result.
>>>>>>>>>
>>>>>>>>>> Maybe it's being blocked in your country,
>>>>>>>>> That site may be blocked.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Yunhee
>>>>>>>>>
>>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>>>>>>>>>> Hi YunHee,
>>>>>>>>>>
>>>>>
>>>>>>>>>> Thanks, can you try some of the other XML file configs. Maybe it's
>>>>>>>>>> being
>>>>>>>>>> blocked in your country, or something, but let's see if we can get
>>>>>>>>>> some
>>>>>>>>>> of the other XML RemoteSpec configs to work?
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Chris
>>>>>>>>>>
>>>>>>>>>> On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Try out the MODIS one there, does that work?
>>>>>>>>>>> No, it doesn't.
>>>>>>>>>>> I think that the ftp site e4ftl01u.ecs.nasa.gov has a connection
>>>>>>>>>>> problem.
>>>>>>>>>>>
>>>>>>>>>>> $ ftp e4ftl01u.ecs.nasa.gov
>>>>>>>>>>> ftp: connect: Connection timed out
>>>>>>>>>>> ftp>
>>>>>>>>>>>
>>>>>>>>>>> The message below also say the reason why the problem was occurred.
>>>>>>>>>>> WARNING: Error occurred while connecting to RemoteSite: alias =
>>>>>>>>>>> 'ModisFtp' url = 'ftp://e4ftl01u.ecs.nasa.gov' username =
>>>>>>>>>>> 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed to
>>>>>>>>>>> connect to server : Connection timed out
>>>>>>>>>>> Jul 25, 2012 1:31:25 AM
>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>>>>>>>> INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp' url =
>>>>>>>>>>> 'ftp://e4ftl01u.ecs.nasa.gov' username = 'anonymous' cdTestDir =
>>>>>>>>>>> 'null' maxConnections = '-1' in 5 seconds
>>>>>>>>>>> Waiting . . . . .
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Yunhee
>>>>>>>>>>>
>>>>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>>>>>>>>>>>> Hi YunHee,
>>>>>>>>>>>>
>>>>>
>>>>>>>>>>>> OK, can you try some of the other files, e.g., the ones in:
>>>>>>>>>>>>
>>>>>>>>>>>> http://s.apache.org/ks0
>>>>>>>>>>>>
>>>>>>>>>>>> Try out the MODIS one there, does that work?
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Chris
>>>>>>>>>>>>
>>>>>>>>>>>> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I added the header of XML as follows:
>>>>>>>>>>>>> <?xml version="1.0"?>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Next the value of elements was changed into specific one.
>>>>>>>>>>>>> The following is the output of the XML:
>>>>>>>>>>>>> <?xml version="1.0"?>
>>>>>>>>>>>>> <root>
>>>>>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005">
>>>>>>>>>>>>> <nofiles/>
>>>>>>>>>>>>> <!--dir name="\d{4}\.\d{2}\.\d{2}"-->
>>>>>>>>>>>>> <dir name="2004\.09\.20">
>>>>>>>>>>>>> <nodirs/>
>>>>>>>>>>>>> <!--file
>>>>>>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/-->
>>>>>>>>>>>>> <file name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/>
>>>>>>>>>>>>> </dir>
>>>>>>>>>>>>> </dirstruct>
>>>>>>>>>>>>> </root>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Unfortunately I got the same result before:
>>>>>>>>>>>>> ...
>>>>>>>>>>>>> INFO: Testing protocol
>>>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this
>>>>>>>>>>>>> ma
>>>>>>>>>>>>> y take a few minutes . . .
>>>>>>>>>>>>> Jul 25, 2012 12:30:03 AM
>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>>>>>>>>>>>>> protocol 'org.apache.oodt.
>>>>>>>>>>>>> cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous'
>>>>>>>>>>>>> Jul 25, 2012 12:30:03 AM
>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>>>>>>>>>>>> retrie
>>>>>>>>>>>>> veFiles
>>>>>>>>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/
>>>>>>>>>>>>> examples/DirStructXmlParserFiles/TESL2CO2.xml : null
>>>>>>>>>>>>> Jul 25, 2012 12:30:04 AM
>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.Daemon
>>>>>>>>>>>>> sleep
>>>>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going
>>>>>>>>>>>>> to
>>>>>>>>>>>>> sleep until Wed Jul 25 0
>>>>>>>>>>>>> ...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Yunhee
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2012/7/25 Mattmann, Chris A (388J)
>>>>>>>>>>>>> <chris.a.mattm...@jpl.nasa.gov>:
>>>>>>>>>>>>>> Hi YunHee,
>>>>>>>>>>>>>>
>>>>>
>>>>>>>>>>>>>> Can you try simply putting a specific <dir name="..."> instead
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> \d{4}, etc., and
>>>>>>>>>>>>>> then can you put a specific file name in the <file name="..">.
>>>>>>>>>>>>>> If
>>>>>>>>>>>>>> that works and
>>>>>>>>>>>>>> downloads, then we know there is something up with the regex
>>>>>>>>>>>>>> somehow.
>>>>>>>>>>>>>> Also can you try adding the <?xml version="1.0"?> start to your
>>>>>>>>>>>>>> file?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The following is the content of TESL2CO2.xml.
>>>>>>>>>>>>>>> $ cat
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>>>>>>> <root>
>>>>>>>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005">
>>>>>>>>>>>>>>> <nofiles/>
>>>>>>>>>>>>>>> <dir name="\d{4}\.\d{2}\.\d{2}">
>>>>>>>>>>>>>>> <nodirs/>
>>>>>>>>>>>>>>> <file
>>>>>>>>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/>
>>>>>>>>>>>>>>> </dir>
>>>>>>>>>>>>>>> </dirstruct>
>>>>>>>>>>>>>>> </root>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Let me know if there is a problem in the configuration file.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Yunhee
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2012/7/24 Mattmann, Chris A (388J)
>>>>>>>>>>>>>>> <chris.a.mattm...@jpl.nasa.gov>:
>>>>>>>>>>>>>>>> Hi YunHee,
>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Jul 24, 2012, at 12:30 AM, YunHee Kang wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I am stuck on the problem related with "pushpull" script
>>>>>>>>>>>>>>>>> occurred
>>>>>>>>>>>>>>>>> the following error message:
>>>>>>>>>>>>>>>>> ....
>>>>>>>>>>>>>>>>> INFO: Testing protocol
>>>>>>>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . .
>>>>>>>>>>>>>>>>> this may
>>>>>>>>>>>>>>>>> take a few minutes . . .
>>>>>>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>>>>>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>> 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>>>>>>>>>>>>>>>> username 'anonymous'
>>>>>>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>>>>>>>>>>>>>>>> retrieveFiles
>>>>>>>>>>>>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>>>>>>>>> : null
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you do me a favor and simply run this command:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> cat
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What do you get?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>>>>>>>>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is
>>>>>>>>>>>>>>>>> going
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> sleep until Tue Jul 24 16:06:00 KST 2012
>>>>>>>>>>>>>>>>> Jul 24, 2012 4:06:00 PM
>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.DaemonManager
>>>>>>>>>>>>>>>>> setAsRunning
>>>>>>>>>>>>>>>>> INFO: Daemon with ID = 91131 was given permission to run
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> First of all, I would like test the pushpull framework to
>>>>>>>>>>>>>>>>> move
>>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>> written in NetCDF from a remote site by a network protocols
>>>>>>>>>>>>>>>>> (like ftp,
>>>>>>>>>>>>>>>>> http, etc).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yep, sounds good.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think that there is a test suite which can be used to check
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> pushpull framework.
>>>>>>>>>>>>>>>>> Can you give configuration files(a set of XML files) that I
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>> test
>>>>>>>>>>>>>>>>> downloading files from a FTP site (public anonymous site) ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The examples in the resources directory that you found should
>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>> so we'll just need to figure out why they aren't working on
>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>> system.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> Please let me know a reference implementation of pushpull
>>>>>>>>>>>>>>>>> framework
>>>>>>>>>>>>>>>>> that I can use to move NetCDF files.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks we'll figure this out.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>>>>>>>>> Senior Computer Scientist
>>>>>>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>>>>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>>>>>>> Senior Computer Scientist
>>>>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>>>>> Senior Computer Scientist
>>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>>> Senior Computer Scientist
>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>> Senior Computer Scientist
>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>> Phone: +1 (818) 354-8810
>>>>>
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Chris Mattmann, Ph.D.
>>>>>> Senior Computer Scientist
>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>
>
