Hey Chintu, Thanks for identifying this!
Have you tried the alternative plugins? They use a diff set of FTP protocol plugins (that unfortunately are not ALv2 licensed or Category A approved, so we can't ship with Apache OODT). They may do better at handling that symlink than the default version of commons-ftp we're using. Let me know. We'll investigate. Cheers, Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -----Original Message----- From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <[email protected]> Date: Thursday, March 28, 2013 9:02 AM To: jpluser <[email protected]>, "[email protected]" <[email protected]> Cc: "Ramirez, Paul M (388J)" <[email protected]> Subject: Re: pushPull component problem >Chris, > >I figured out the problem. Don't know the solution. > >So the problem is the symbolic links on FTP server. > >/pub/OI-daily-v2/NetCDF/2013/AVHRR is actually >/san1/oisst/NetCDF/2013/AVHRR >Where pub -> san1 > And OI-daily-v2 -> oisst > >So if you use "starting_path=/san1/oisst/NetCDF/2013/AVHRR" then it works. >Otherwise it does not work. > >Another example is : ftp://aftp.cmdl.noaa.gov > >/data/radiation/surfrad/dra/2013 >->/data/radiation/surfrad/Desert_Rock_NV/2013 > > > >I don't know how to fix this. Well, not exactly sure where to look. It >appears that the filename filters are being applied based on the path >specified in DirStructParserFiles instead of just the filename. Because I >can see the correct path when I print RemoteFile (path = >'/san1/oisst/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz' ) >on STDOUT. > > >Hopefully you guys can fix this fast or let me know where to look. > >Regards >-- >Chintu Mistry >NASA Goddard Space Flight Center >Bldg L40B, Room S776 >Office: 240 684 0477 >Mobile: 770 310 1047 > > > > > > >On 3/28/13 10:15 AM, "Mattmann, Chris A" <[email protected]> >wrote: > >>Hey Chintu, >> >>This totally helped, thanks. >> >>Can you take a look at the plugins here? >> >>https://cwiki.apache.org/OODT/oodt-push-pull-plugins.html >> >> >>Try installing those, and let me know if either of those FTP >>plugins works better and actually downloads the filesŠ >> >>In the meanwhile I'll look at the logs and see what I can come up >>with. >> >>Cheers, >>Chris >> >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>Chris Mattmann, Ph.D. >>Senior Computer Scientist >>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>Office: 171-266B, Mailstop: 171-246 >>Email: [email protected] >>WWW: http://sunset.usc.edu/~mattmann/ >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>Adjunct Assistant Professor, Computer Science Department >>University of Southern California, Los Angeles, CA 90089 USA >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> >> >> >>-----Original Message----- >>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <[email protected]> >>Date: Thursday, March 28, 2013 7:12 AM >>To: jpluser <[email protected]>, "[email protected]" >><[email protected]> >>Cc: "Ramirez, Paul M (388J)" <[email protected]> >>Subject: Re: pushPull component problem >> >>>I have attached verbose log. I have added extra print statements and >>>compiled to see what is returned from remote site. Hope this helps. >>> >>> >>>-- >>>Chintu Mistry >>>NASA Goddard Space Flight Center >>>Bldg L40B, Room S776 >>>Office: 240 684 0477 >>>Mobile: 770 310 1047 >>> >>> >>> >>> >>> >>> >>>From: <Mattmann>, Chris A <[email protected]> >>>Date: Thursday, March 28, 2013 10:05 AM >>>To: Chintu Mistry <[email protected]>, "[email protected]" >>><[email protected]> >>>Cc: "Ramirez, Paul M" <[email protected]> >>>Subject: Re: pushPull component problem >>> >>> >>> >>>Hey Chintu, >>> >>> >>>Thanks for sending this ‹ really helped verify it's something else. >>> >>> >>>Can you capture the log of push pull, with verbose logging turned on, >>>and >>>maybe do a pastebin? >>> >>> >>>Cheers, >>>Chris >>> >>> >>> >>> >>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" <[email protected]> >>>Date: Thursday, March 28, 2013 7:03 AM >>>To: jpluser <[email protected]>, "[email protected]" >>><[email protected]> >>>Cc: "Ramirez, Paul M (388J)" <[email protected]> >>>Subject: Re: pushPull component problem >>> >>> >>> >>>> >>>> >>>> >>>> >>>>Something else is wrong. I know that the regex is fine. I don't even >>>>have regex, it is straight up file name. It should match. >>>> >>>> >>>>Thanks for looking into this. >>>>-- >>>>Chintu Mistry >>>>NASA Goddard Space Flight Center >>>>Bldg L40B, Room S776 >>>>Office: 240 684 0477 >>>>Mobile: 770 310 1047 >>>> >>>> >>>> >>>> >>>> >>>>From: <Mattmann>, Chris A <[email protected]> >>>>Date: Thursday, March 28, 2013 9:01 AM >>>>To: Chintu Mistry <[email protected]>, "[email protected]" >>>><[email protected]> >>>>Cc: "Ramirez, Paul M" <[email protected]> >>>>Subject: Re: pushPull component problem >>>> >>>> >>>> >>>>Hey Chintu, >>>> >>>> >>>>Sorry that you have been having trouble and thanks for pasting the >>>>relevant portions >>>>of push pull config to diagnose this. >>>> >>>> >>>>I think you have an issue in your RemoteFileSpecs file >>>>(TEST_AVHRR_ONLY.xml) -- >>>>one quick way to debug this is to use the tool that Brian Foster wrote: >>>> >>>> >>>>(in a push pull deployment dir, e.g., $PUSHPULL_HOME) >>>>$ cd bin >>>>$ java Djava.ext.dirs=../lib >>>>org.apache.oodt.cas.pushpull.util.ExpressionValidator >>>> >>>> >>>>It's a simple GUI that will allow you to put in regular expressions >>>>from >>>>your push pull >>>>RmoteFileSpecs files, and then put in file names from the server to see >>>>if they match. >>>> >>>> >>>>Can you run the above and let me know if it's matching? If not, we've >>>>found your issue. >>>>If it is matching, we'll move on to the next set of things to check. >>>> >>>> >>>>Thanks! >>>> >>>> >>>>Cheers, >>>>Chris >>>> >>>> >>>>From: <Mistry>, "Chintu [SGT INC] (GSFC-586.0)" >>>><[email protected]> >>>>Date: Thursday, March 28, 2013 5:47 AM >>>>To: "[email protected]" <[email protected]> >>>>Cc: jpluser <[email protected]>, "Ramirez, Paul M (388J)" >>>><[email protected]> >>>>Subject: pushPull component problem >>>> >>>> >>>> >>>>>Hi, >>>>> >>>>> >>>>>I have been playing around with push pull for last two days and trying >>>>>to download some data from remote location without success. >>>>>I would appreciate if you guys can shed some light on what is going on >>>>>and why isn't push pull find files to download even though its clearly >>>>>there at remote location. >>>>> >>>>> >>>>>Here is my configuration. I am trying to download >>>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary. >>>>>n >>>>>c >>>>>.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine and >>>>>it lists all the files as well. However for some reason it >>>>> does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I am >>>>>doing wrong. I have tried almost all combinations and still could not >>>>>get it to work. >>>>> >>>>> >>>>>In my /etc/pushpull.properties file, I have types=false so that it >>>>>downloads the file even if its not specified in mime-types.xml file. >>>>>org.apache.oodt.cas.pushpull.allow.only.defined.types=false >>>>> >>>>> >>>>>ExternalSources.xml >>>>><sources> >>>>> <source host="eclipse.ncdc.noaa.gov"> >>>>> <login type="ftp" alias="eclipseClass"> >>>>> <username>anonymous</username> >>>>> <password>[email protected]</password> >>>>> </login> >>>>> </source> >>>>></sources> >>>>> >>>>> >>>>> >>>>>RemoteSpecs.xml >>>>> <daemon alias="eclipseClass" active="yes"> >>>>> <runInfo firstRunDateTime="2013-03-25T00:00:00Z" >>>>>period="1m" runOnReboot="yes"/> >>>>> <propInfo >>>>>dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles"> >>>>> <propFiles regExp="TEST_AVHRR_ONLY\.xml" >>>>>parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStruc >>>>>t >>>>>X >>>>>mlParser"/> >>>>> </propInfo> >>>>> <dataInfo stagingArea="AVHRR-ONLY" deleteFromServer="no" >>>>>queryElement="Filename"/> >>>>> </daemon> >>>>> >>>>> >>>>> >>>>>TEST_AVHRR_ONLY.xml >>>>><root> >>>>> <dirstruct starting_path="/pub/OI-daily-v2/NetCDF/2013/AVHRR"> >>>>> <nodirs/> >>>>> <!-- <file >>>>>name="avhrr\-only\-v2\.20130326_preliminary\.nc\.gz"/> --> >>>>> <file name="avhrr-only-v2.20130326_preliminary.nc.gz"/> >>>>> </dirstruct> >>>>></root> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>Regards >>>>>-- >>>>>Chintu Mistry >>>>>NASA Goddard Space Flight Center >>>>>Bldg L40B, Room S776 >>>>>Office: 240 684 0477 >>>>>Mobile: 770 310 1047 >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>> >>> >> >
