Since we are not using OODT 0.6, it did not compile straight away. However, I did manage to compile it with "0.5" version of cas-protocol-api. But when I was running it, It failed to run.
-- Chintu Mistry NASA Goddard Space Flight Center Bldg L40B, Room S776 Office: 240 684 0477 Mobile: 770 310 1047 On 3/28/13 12:04 PM, "Mattmann, Chris A" <[email protected]> wrote: >Hey Chintu, > >Thanks for identifying this! > >Have you tried the alternative plugins? They use a diff set of FTP >protocol >plugins (that unfortunately are not ALv2 licensed or Category A approved, >so we can't ship with Apache OODT). > >They may do better at handling that symlink than the default version of >commons-ftp >we're using. > >Let me know. We'll investigate. > >Cheers, >Chris > >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >Chris Mattmann, Ph.D. >Senior Computer Scientist >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >Office: 171-266B, Mailstop: 171-246 >Email: [email protected] >WWW: http://sunset.usc.edu/~mattmann/ >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >Adjunct Assistant Professor, Computer Science Department >University of Southern California, Los Angeles, CA 90089 USA >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > >-----Original Message----- >From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <[email protected]> >Date: Thursday, March 28, 2013 9:02 AM >To: jpluser <[email protected]>, "[email protected]" ><[email protected]> >Cc: "Ramirez, Paul M (388J)" <[email protected]> >Subject: Re: pushPull component problem > >>Chris, >> >>I figured out the problem. Don't know the solution. >> >>So the problem is the symbolic links on FTP server. >> >>/pub/OI-daily-v2/NetCDF/2013/AVHRR is actually >>/san1/oisst/NetCDF/2013/AVHRR >>Where pub -> san1 >> And OI-daily-v2 -> oisst >> >>So if you use "starting_path=/san1/oisst/NetCDF/2013/AVHRR" then it >>works. >>Otherwise it does not work. >> >>Another example is : ftp://aftp.cmdl.noaa.gov >> >>/data/radiation/surfrad/dra/2013 >>->/data/radiation/surfrad/Desert_Rock_NV/2013 >> >> >> >>I don't know how to fix this. Well, not exactly sure where to look. It >>appears that the filename filters are being applied based on the path >>specified in DirStructParserFiles instead of just the filename. Because I >>can see the correct path when I print RemoteFile (path = >>'/san1/oisst/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz' >>) >>on STDOUT. >> >> >>Hopefully you guys can fix this fast or let me know where to look. >> >>Regards >>-- >>Chintu Mistry >>NASA Goddard Space Flight Center >>Bldg L40B, Room S776 >>Office: 240 684 0477 >>Mobile: 770 310 1047 >> >> >> >> >> >> >>On 3/28/13 10:15 AM, "Mattmann, Chris A" <[email protected]> >>wrote: >> >>>Hey Chintu, >>> >>>This totally helped, thanks. >>> >>>Can you take a look at the plugins here? >>> >>>https://cwiki.apache.org/OODT/oodt-push-pull-plugins.html >>> >>> >>>Try installing those, and let me know if either of those FTP >>>plugins works better and actually downloads the filesŠ >>> >>>In the meanwhile I'll look at the logs and see what I can come up >>>with. >>> >>>Cheers, >>>Chris >>> >>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>Chris Mattmann, Ph.D. >>>Senior Computer Scientist >>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>Office: 171-266B, Mailstop: 171-246 >>>Email: [email protected] >>>WWW: http://sunset.usc.edu/~mattmann/ >>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>Adjunct Assistant Professor, Computer Science Department >>>University of Southern California, Los Angeles, CA 90089 USA >>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> >>> >>> >>> >>>-----Original Message----- >>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" >>><[email protected]> >>>Date: Thursday, March 28, 2013 7:12 AM >>>To: jpluser <[email protected]>, "[email protected]" >>><[email protected]> >>>Cc: "Ramirez, Paul M (388J)" <[email protected]> >>>Subject: Re: pushPull component problem >>> >>>>I have attached verbose log. I have added extra print statements and >>>>compiled to see what is returned from remote site. Hope this helps. >>>> >>>> >>>>-- >>>>Chintu Mistry >>>>NASA Goddard Space Flight Center >>>>Bldg L40B, Room S776 >>>>Office: 240 684 0477 >>>>Mobile: 770 310 1047 >>>> >>>> >>>> >>>> >>>> >>>> >>>>From: <Mattmann>, Chris A <[email protected]> >>>>Date: Thursday, March 28, 2013 10:05 AM >>>>To: Chintu Mistry <[email protected]>, "[email protected]" >>>><[email protected]> >>>>Cc: "Ramirez, Paul M" <[email protected]> >>>>Subject: Re: pushPull component problem >>>> >>>> >>>> >>>>Hey Chintu, >>>> >>>> >>>>Thanks for sending this ‹ really helped verify it's something else. >>>> >>>> >>>>Can you capture the log of push pull, with verbose logging turned on, >>>>and >>>>maybe do a pastebin? >>>> >>>> >>>>Cheers, >>>>Chris >>>> >>>> >>>> >>>> >>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" >>>><[email protected]> >>>>Date: Thursday, March 28, 2013 7:03 AM >>>>To: jpluser <[email protected]>, "[email protected]" >>>><[email protected]> >>>>Cc: "Ramirez, Paul M (388J)" <[email protected]> >>>>Subject: Re: pushPull component problem >>>> >>>> >>>> >>>>> >>>>> >>>>> >>>>> >>>>>Something else is wrong. I know that the regex is fine. I don't even >>>>>have regex, it is straight up file name. It should match. >>>>> >>>>> >>>>>Thanks for looking into this. >>>>>-- >>>>>Chintu Mistry >>>>>NASA Goddard Space Flight Center >>>>>Bldg L40B, Room S776 >>>>>Office: 240 684 0477 >>>>>Mobile: 770 310 1047 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>From: <Mattmann>, Chris A <[email protected]> >>>>>Date: Thursday, March 28, 2013 9:01 AM >>>>>To: Chintu Mistry <[email protected]>, "[email protected]" >>>>><[email protected]> >>>>>Cc: "Ramirez, Paul M" <[email protected]> >>>>>Subject: Re: pushPull component problem >>>>> >>>>> >>>>> >>>>>Hey Chintu, >>>>> >>>>> >>>>>Sorry that you have been having trouble and thanks for pasting the >>>>>relevant portions >>>>>of push pull config to diagnose this. >>>>> >>>>> >>>>>I think you have an issue in your RemoteFileSpecs file >>>>>(TEST_AVHRR_ONLY.xml) -- >>>>>one quick way to debug this is to use the tool that Brian Foster >>>>>wrote: >>>>> >>>>> >>>>>(in a push pull deployment dir, e.g., $PUSHPULL_HOME) >>>>>$ cd bin >>>>>$ java Djava.ext.dirs=../lib >>>>>org.apache.oodt.cas.pushpull.util.ExpressionValidator >>>>> >>>>> >>>>>It's a simple GUI that will allow you to put in regular expressions >>>>>from >>>>>your push pull >>>>>RmoteFileSpecs files, and then put in file names from the server to >>>>>see >>>>>if they match. >>>>> >>>>> >>>>>Can you run the above and let me know if it's matching? If not, we've >>>>>found your issue. >>>>>If it is matching, we'll move on to the next set of things to check. >>>>> >>>>> >>>>>Thanks! >>>>> >>>>> >>>>>Cheers, >>>>>Chris >>>>> >>>>> >>>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" >>>>><[email protected]> >>>>>Date: Thursday, March 28, 2013 5:47 AM >>>>>To: "[email protected]" <[email protected]> >>>>>Cc: jpluser <[email protected]>, "Ramirez, Paul M (388J)" >>>>><[email protected]> >>>>>Subject: pushPull component problem >>>>> >>>>> >>>>> >>>>>>Hi, >>>>>> >>>>>> >>>>>>I have been playing around with push pull for last two days and >>>>>>trying >>>>>>to download some data from remote location without success. >>>>>>I would appreciate if you guys can shed some light on what is going >>>>>>on >>>>>>and why isn't push pull find files to download even though its >>>>>>clearly >>>>>>there at remote location. >>>>>> >>>>>> >>>>>>Here is my configuration. I am trying to download >>>>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary >>>>>>. >>>>>>n >>>>>>c >>>>>>.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine >>>>>>and >>>>>>it lists all the files as well. However for some reason it >>>>>> does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I >>>>>>am >>>>>>doing wrong. I have tried almost all combinations and still could not >>>>>>get it to work. >>>>>> >>>>>> >>>>>>In my /etc/pushpull.properties file, I have types=false so that it >>>>>>downloads the file even if its not specified in mime-types.xml file. >>>>>>org.apache.oodt.cas.pushpull.allow.only.defined.types=false >>>>>> >>>>>> >>>>>>ExternalSources.xml >>>>>><sources> >>>>>> <source host="eclipse.ncdc.noaa.gov"> >>>>>> <login type="ftp" alias="eclipseClass"> >>>>>> <username>anonymous</username> >>>>>> <password>[email protected]</password> >>>>>> </login> >>>>>> </source> >>>>>></sources> >>>>>> >>>>>> >>>>>> >>>>>>RemoteSpecs.xml >>>>>> <daemon alias="eclipseClass" active="yes"> >>>>>> <runInfo firstRunDateTime="2013-03-25T00:00:00Z" >>>>>>period="1m" runOnReboot="yes"/> >>>>>> <propInfo >>>>>>dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles"> >>>>>> <propFiles regExp="TEST_AVHRR_ONLY\.xml" >>>>>>parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStru >>>>>>c >>>>>>t >>>>>>X >>>>>>mlParser"/> >>>>>> </propInfo> >>>>>> <dataInfo stagingArea="AVHRR-ONLY" deleteFromServer="no" >>>>>>queryElement="Filename"/> >>>>>> </daemon> >>>>>> >>>>>> >>>>>> >>>>>>TEST_AVHRR_ONLY.xml >>>>>><root> >>>>>> <dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR"> >>>>>> <nodirs/> >>>>>> <!-- <file >>>>>>name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> --> >>>>>> <file name="avhrr-only-v2.20130326_preliminary.nc.gz"/> >>>>>> </dirstruct> >>>>>></root> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>Regards >>>>>>-- >>>>>>Chintu Mistry >>>>>>NASA Goddard Space Flight Center >>>>>>Bldg L40B, Room S776 >>>>>>Office: 240 684 0477 >>>>>>Mobile: 770 310 1047 >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >> >
