Hey Lewis, Last time I dug into this I was running toward the "Active vs. Passive" FTP settings, but I am not sure that is the root cause. One option or fix would be to add in a timeout parameter, where if the pushpull daemon doesn't get a reply in a set amount of time, it should stop and restart. Every time I have seen pushpull stall or hang, a simple stop and restart gets it working again.
I know Ross Laidlaw is using pushpull on a his Near Real Time snow processing pipeline, so I am sure he will be very interested in your work Lewis. Thanks, Cameron On Sat, Nov 1, 2014 at 10:58 AM, Lewis John Mcgibbney < lewis.mcgibb...@gmail.com> wrote: > Thanks Chris. OK doke. i am going to debug this today and find out what is > happening. > Thanks for heads up. > Lewis > > On Sat, Nov 1, 2014 at 10:31 AM, Chris Mattmann <chris.mattm...@gmail.com> > wrote: > > > Thanks Lewis - in my mind it boils down somewhat to > > the protocols that are implemented and the way they > > behave with FTP servers and so forth. Ultimately, we > > should be using the ones that are in PushPull plugins, > > but they are virally licensed and I haven¹t found a > > better alternative. > > > > Another @bfoster update here would be great, Brian? > > > > Cheers, > > Chris > > > > > > ------------------------ > > Chris Mattmann > > chris.mattm...@gmail.com > > > > > > > > > > -----Original Message----- > > From: Lewis John Mcgibbney <lewis.mcgibb...@gmail.com> > > Reply-To: <dev@oodt.apache.org> > > Date: Saturday, November 1, 2014 at 12:56 AM > > To: "dev@oodt.apache.org" <dev@oodt.apache.org> > > Subject: Why is pushpull not always working? > > > > >Hi Folks, > > >I really couldn't think of a better title for this thread so I apologize > > >in > > >advance. > > >I recently set up pushpull with FTP to crawl remote ftp site [0] which > is > > >publicly available. > > >I am getting the following > > > > > >INFO: Launching Daemons . . . > > >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon > > >registerRMIServer > > >INFO: Created Daemon ID = 90121 on RMI registry port 9012 > > >Nov 01, 2014 12:46:01 AM > > >org.apache.oodt.cas.pushpull.daemon.DaemonLauncher > > >launchDaemons > > >INFO: Creating Daemon with ID = 1 > > >Nov 01, 2014 12:46:01 AM > org.apache.oodt.cas.pushpull.daemon.DaemonManager > > >setAsRunning > > >INFO: Daemon with ID = 90121 was given permission to run > > >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon > sleep > > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep > > >until Sat Nov 01 00:48:00 PDT 2014 > > >Nov 01, 2014 12:48:00 AM > org.apache.oodt.cas.pushpull.daemon.DaemonManager > > >setAsRunning > > >INFO: Daemon with ID = 90121 was given permission to run > > >Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon > sleep > > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep > > >until Sat Nov 01 00:51:00 PDT 2014 > > >Nov 01, 2014 12:50:59 AM > org.apache.oodt.cas.pushpull.daemon.DaemonManager > > >setAsRunning > > >INFO: Daemon with ID = 90121 was given permission to run > > >Nov 01, 2014 12:51:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon > sleep > > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep > > >until Sat Nov 01 00:54:00 PDT 2014 > > >Nov 01, 2014 12:53:59 AM > org.apache.oodt.cas.pushpull.daemon.DaemonManager > > >setAsRunning > > >INFO: Daemon with ID = 90121 was given permission to run > > >Nov 01, 2014 12:54:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon > sleep > > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep > > >until Sat Nov 01 00:57:00 PDT 2014 > > > > > >No errors... no pulling... no nothing! > > >I also came across this FAQ (thanks Cam/Chris) [1] leading to Chris' > > >plugins page [2]. > > >My question is simple... why is pushpull known NOT to work on occasion? > I > > >am going to debug this tomorrow but I thought I would ask this > > >none-the-less. > > >Thanks > > >Lewis > > > > > >[0] ftp://n5eil01u.ecs.nsidc.org/SAN/AQUARIUS/AQ3_SSS.004/ > > >[1] > > > > > > https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide > > > >#OODTPush-PullUserGuide-Nodatafileisdownloadedtomystagingdirectoryafterrun > > >ningthe./pushpullscript.WhatshouldIdo > > >? > > >[2] > > >https://cwiki.apache.org/confluence/display/OODT/OODT+Push+Pull+Plugins > > > > > > > > >-- > > >*Lewis* > > > > > > > > > -- > *Lewis* > -- Sent from a Tin Can attached to a String
