Each app server could run it's own queues but that wouldn't support building a farm of PP processors unless we also teach them to talk to multiple JMS servers. Maybe something like DNS round-robin would suffice?
On Tue, Jul 17, 2012 at 8:25 AM, Erik Froese <erik.fro...@gmail.com> wrote: > Do we need to cluster activemq? Can't each app server service its own > queues? > Erik > > On Tue, Jul 17, 2012 at 11:23 AM, Carl Hall <c...@hallwaytech.com> wrote: > > What Erik describes has been on the dev wish list for a little while now. > > Moving to an event-driven model would allow us to build out concurrency > but > > there also comes the question of clustering ActiveMQ. > > > > > > On Thu, Jul 12, 2012 at 6:27 AM, Erik Froese <erik.fro...@gmail.com> > wrote: > >> > >> Hey David, > >> > >> The code is not clustered. > >> > >> You'd need to write an event listener that would fire when new content > >> is uploaded. It would put the content ids on a JMS queue. Then > >> implement a ContentFetcher that grabs a message off of the queue and > >> wire that into the PPI. Events and Messages are not clustered in OAE > >> (AFAIK) so this would have to be run on each app server. > >> > >> While we're in event-land it'd be nice to be able to regenerate a > >> preview when a content body is updated. I'm not sure if this is > >> possible yet. > >> > >> I'm not sure how we'd limit the CPU usage yet either. You could manage > >> the quartz schedule that runs the PPI. > >> > >> We can also disable concurrent executions of the job. > >> > >> Erik > >> > >> On Wed, Jul 11, 2012 at 8:44 PM, Roma, David <dr...@csu.edu.au> wrote: > >> > Awesome news Erik! > >> > > >> > Our Ops guys will be stoked when we can get this in.. A couple of > >> > questions from someone who hasn't looked at the code or read too > deeply.... > >> > - Does it support clustering > >> > -e.g. can we just run it side by side on each of our app > servers > >> > and they will play nice sharing out processing jobs? > >> > -will it affect performance of the app servers much? Can we > >> > limit the preview processor to say 10%cpu and 500mb ram or low > priority > >> > threads or limit the number of items to process or something? This > would > >> > make for a nice simple deployment that wouldn't threaten the app > server > >> > stability. > >> > > >> > Cheers, > >> > Dave. > >> > > >> > > >> > > >> > > >> > > >> > -----Original Message----- > >> > From: oae-dev-boun...@collab.sakaiproject.org > >> > [mailto:oae-dev-boun...@collab.sakaiproject.org] On Behalf Of Erik > Froese > >> > Sent: Thursday, 12 July 2012 2:37 AM > >> > To: Carl Hall > >> > Cc: oae-dev@collab.sakaiproject.org; Clay Fenlason > >> > Subject: Re: [oae-dev] Moving the preview processor to java > >> > > >> > Hey everyone, > >> > > >> > Its been a few months but I actually implemented the Java preview > >> > processor as an OSGi bundle. I filed a ticket for it [1] > >> > > >> > I'm not sure where to go from here. Is this something that could be > >> > included POST 1.4.0? > >> > Should I open a PR so we can review the code? If so, PR against which > >> > branch? > >> > > >> > Either way, have a look, give it a go. We'll probably wind up using it > >> > at rSmart. > >> > > >> > Erik > >> > > >> > [1] https://jira.sakaiproject.org/browse/KERN-3021 > >> > > >> > > >> > > >> > On Tue, Apr 17, 2012 at 9:09 AM, Carl Hall <c...@hallwaytech.com> > wrote: > >> >> I totally agree that we should ally ourselves with other > communities. I > >> >> see > >> >> where we get docsplit from DocumentCloud[1] and we use several other > >> >> libraries for processing that they've most likely contributed to. > >> >> The Java approach is very little custom code compared to the > libraries > >> >> we're > >> >> getting from Apache (tika, sanselan, commons, pdfbox), so we would > >> >> still > >> >> building on the shoulders of our friendly community giants. > >> >> > >> >> 1 https://github.com/documentcloud/docsplit > >> >> > >> >> > >> >> > >> >> On Sat, Apr 14, 2012 at 5:43 AM, John Norman <j...@caret.cam.ac.uk> > >> >> wrote: > >> >>> > >> >>> My recollection (perhaps wrong) is that we got this from Document > >> >>> Cloud > >> >>> and I /think/ Chris Roby found it. Document Cloud seems a very > >> >>> relevant and > >> >>> valuable project. If we were able to help them while helping > >> >>> ourselves, > >> >>> other good things could come from the relationship. My general point > >> >>> is that > >> >>> we are thin on resources and so, in principle, symbiotic > relationships > >> >>> are > >> >>> helpful. > >> >>> > >> >>> http://www.documentcloud.org/home > >> >>> > >> >>> John > >> >>> > >> >>> Sent from my iPad > >> >>> > >> >>> On 13 Apr 2012, at 17:03, Carl Hall <c...@hallwaytech.com> wrote: > >> >>> > >> >>> I agree with Daniel that our modifications to the preview processor > >> >>> have > >> >>> put its ownership square on us. Was there a community that this > script > >> >>> was > >> >>> borrowed from? I thought it was original development that uses > various > >> >>> external libraries to do the actual work. This is the approach that > >> >>> Erik is > >> >>> taking with the rewrite using things like Tika (text extraction), > >> >>> Sanselan > >> >>> (images) and a Java port of the python topia.termextract library. > >> >>> > >> >>> I certainly don't deny the speed of development that was realized in > >> >>> creating the PP but the current state of the code is a mess at best. > >> >>> Reuse > >> >>> of libraries in Java is showing a fast rewrite with very little > >> >>> managed code > >> >>> on our part. > >> >>> > >> >>> > >> >>> On Fri, Apr 13, 2012 at 12:50 AM, Daniel Parry > >> >>> <dan...@caret.cam.ac.uk> > >> >>> wrote: > >> >>>> > >> >>>> On Thu, Apr 12, 2012 at 04:21:36PM -0400, Clay Fenlason wrote: > >> >>>> > I think this response is at best orthogonal to the point John's > >> >>>> > trying > >> >>>> > to raise, though I gather this kind of reaction must come from a > >> >>>> > buildup of some real frustration around the PP, which I don't > mean > >> >>>> > to > >> >>>> > discount. I also think John was pretty clear about what he was > >> >>>> > suggesting: that there be a conversation with the community we > got > >> >>>> > the > >> >>>> > PP from, if the conversation hasn't happened already, to see if > >> >>>> > there > >> >>>> > might still be a way to work together before we decide to just > own > >> >>>> > it > >> >>>> > ourselves. > >> >>>> > >> >>>> I'd suggest the way that the preview processor was being extended > >> >>>> (initially a > >> >>>> python server add on, followed by a ruby rewrite for tag > extraction) > >> >>>> and > >> >>>> the > >> >>>> variety of ruby versions that deployers were using and the methods > >> >>>> used > >> >>>> to > >> >>>> deploy it were indicative of a) the OAE community already 'owning' > >> >>>> the PP > >> >>>> and b) > >> >>>> as has already been pointed out some standardization needed > restoring > >> >>>> and > >> >>>> additional functionality added for deployers. Hence, the list was > >> >>>> pinged[0] a > >> >>>> while back to ask about standardizing and extending in java. I'm > not > >> >>>> sure > >> >>>> of any > >> >>>> other way to contact the original PP community or if such a > community > >> >>>> separate > >> >>>> to OAE even still exists? > >> >>>> > >> >>>> Best wishes, > >> >>>> > >> >>>> Daniel > >> >>>> > >> >>>> [0] > >> >>>> > >> >>>> > http://collab.sakaiproject.org/pipermail/oae-dev/2012-April/001677.html > >> >>>> > >> >>>> -- > >> >>>> --| Daniel Parry: dan...@caret.cam.ac.uk. www.caret.cam.ac.uk/ |-- > >> >>>> "Of all the things a leader should fear, complacency should > >> >>>> head the list." [John C. Maxwell] > >> >>>> _______________________________________________ > >> >>>> oae-dev mailing list > >> >>>> oae-dev@collab.sakaiproject.org > >> >>>> http://collab.sakaiproject.org/mailman/listinfo/oae-dev > >> >>> > >> >>> > >> >>> _______________________________________________ > >> >>> oae-dev mailing list > >> >>> oae-dev@collab.sakaiproject.org > >> >>> http://collab.sakaiproject.org/mailman/listinfo/oae-dev > >> >> > >> >> > >> >> > >> >> _______________________________________________ > >> >> oae-dev mailing list > >> >> oae-dev@collab.sakaiproject.org > >> >> http://collab.sakaiproject.org/mailman/listinfo/oae-dev > >> >> > >> > _______________________________________________ > >> > oae-dev mailing list > >> > oae-dev@collab.sakaiproject.org > >> > http://collab.sakaiproject.org/mailman/listinfo/oae-dev > >> > Charles Sturt University > >> > > >> > | ALBURY-WODONGA | BATHURST | CANBERRA | DUBBO | GOULBURN | MELBOURNE > | > >> > ONTARIO | ORANGE | PORT MACQUARIE | SYDNEY | WAGGA WAGGA | > >> > > >> > LEGAL NOTICE > >> > This email (and any attachment) is confidential and is intended for > the > >> > use of the addressee(s) only. If you are not the intended recipient > of this > >> > email, you must not copy, distribute, take any action in reliance on > it or > >> > disclose it to anyone. Any confidentiality is not waived or lost by > reason > >> > of mistaken delivery. Email should be checked for viruses and defects > before > >> > opening. Charles Sturt University (CSU) does not accept liability for > >> > viruses or any consequence which arise as a result of this email > >> > transmission. Email communications with CSU may be subject to > automated > >> > email filtering, which could result in the delay or deletion of a > legitimate > >> > email before it is read at CSU. The views expressed in this email are > not > >> > necessarily those of CSU. > >> > > >> > Charles Sturt University in Australia http://www.csu.edu.au The > >> > Chancellery, Panorama Avenue, Bathurst NSW Australia 2795 ABN: 83 > 878 708 > >> > 551; CRICOS Provider Numbers: 00005F (NSW), 01947G (VIC), 02960B (ACT) > >> > > >> > Charles Sturt University in Ontario http://www.charlessturt.ca 860 > >> > Harrington Court, Burlington Ontario Canada L7N 3N4 Registration: > >> > www.peqab.ca > >> > > >> > Consider the environment before printing this email. > > > > >
_______________________________________________ oae-dev mailing list oae-dev@collab.sakaiproject.org http://collab.sakaiproject.org/mailman/listinfo/oae-dev
