Each app server could run it's own queues but that wouldn't
support building a farm of PP processors unless we also teach
them to talk to multiple JMS servers. Maybe something like DNS
round-robin would suffice?
On Tue, Jul 17, 2012 at 8:25 AM, Erik Froese
<erik.fro...@gmail.com> wrote:
Do we need to cluster activemq? Can't each app server service its
own queues?
Erik
On Tue, Jul 17, 2012 at 11:23 AM, Carl Hall
<c...@hallwaytech.com> wrote:
> What Erik describes has been on the dev wish list for a little
while now.
> Moving to an event-driven model would allow us to build out
concurrency but
> there also comes the question of clustering ActiveMQ.
>
>
> On Thu, Jul 12, 2012 at 6:27 AM, Erik Froese <erik.fro...@gmail.com
> wrote:
>>
>> Hey David,
>>
>> The code is not clustered.
>>
>> You'd need to write an event listener that would fire when new
content
>> is uploaded. It would put the content ids on a JMS queue. Then
>> implement a ContentFetcher that grabs a message off of the
queue and
>> wire that into the PPI. Events and Messages are not clustered
in OAE
>> (AFAIK) so this would have to be run on each app server.
>>
>> While we're in event-land it'd be nice to be able to
regenerate a
>> preview when a content body is updated. I'm not sure if this is
>> possible yet.
>>
>> I'm not sure how we'd limit the CPU usage yet either. You
could manage
>> the quartz schedule that runs the PPI.
>>
>> We can also disable concurrent executions of the job.
>>
>> Erik
>>
>> On Wed, Jul 11, 2012 at 8:44 PM, Roma, David
<dr...@csu.edu.au> wrote:
>> > Awesome news Erik!
>> >
>> > Our Ops guys will be stoked when we can get this in.. A
couple of
>> > questions from someone who hasn't looked at the code or read
too deeply....
>> > - Does it support clustering
>> > -e.g. can we just run it side by side on each of our
app servers
>> > and they will play nice sharing out processing jobs?
>> > -will it affect performance of the app servers much?
Can we
>> > limit the preview processor to say 10%cpu and 500mb ram or
low priority
>> > threads or limit the number of items to process or
something? This would
>> > make for a nice simple deployment that wouldn't threaten the
app server
>> > stability.
>> >
>> > Cheers,
>> > Dave.
>> >
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: oae-dev-boun...@collab.sakaiproject.org
>> > [mailto:oae-dev-boun...@collab.sakaiproject.org] On Behalf
Of Erik Froese
>> > Sent: Thursday, 12 July 2012 2:37 AM
>> > To: Carl Hall
>> > Cc: oae-dev@collab.sakaiproject.org; Clay Fenlason
>> > Subject: Re: [oae-dev] Moving the preview processor to java
>> >
>> > Hey everyone,
>> >
>> > Its been a few months but I actually implemented the Java
preview
>> > processor as an OSGi bundle. I filed a ticket for it [1]
>> >
>> > I'm not sure where to go from here. Is this something that
could be
>> > included POST 1.4.0?
>> > Should I open a PR so we can review the code? If so, PR
against which
>> > branch?
>> >
>> > Either way, have a look, give it a go. We'll probably wind
up using it
>> > at rSmart.
>> >
>> > Erik
>> >
>> > [1] https://jira.sakaiproject.org/browse/KERN-3021
>> >
>> >
>> >
>> > On Tue, Apr 17, 2012 at 9:09 AM, Carl Hall <c...@hallwaytech.com
> wrote:
>> >> I totally agree that we should ally ourselves with other
communities. I
>> >> see
>> >> where we get docsplit from DocumentCloud[1] and we use
several other
>> >> libraries for processing that they've most likely
contributed to.
>> >> The Java approach is very little custom code compared to
the libraries
>> >> we're
>> >> getting from Apache (tika, sanselan, commons, pdfbox), so
we would
>> >> still
>> >> building on the shoulders of our friendly community giants.
>> >>
>> >> 1 https://github.com/documentcloud/docsplit
>> >>
>> >>
>> >>
>> >> On Sat, Apr 14, 2012 at 5:43 AM, John Norman <j...@caret.cam.ac.uk
>
>> >> wrote:
>> >>>
>> >>> My recollection (perhaps wrong) is that we got this from
Document
>> >>> Cloud
>> >>> and I /think/ Chris Roby found it. Document Cloud seems a
very
>> >>> relevant and
>> >>> valuable project. If we were able to help them while helping
>> >>> ourselves,
>> >>> other good things could come from the relationship. My
general point
>> >>> is that
>> >>> we are thin on resources and so, in principle, symbiotic
relationships
>> >>> are
>> >>> helpful.
>> >>>
>> >>> http://www.documentcloud.org/home
>> >>>
>> >>> John
>> >>>
>> >>> Sent from my iPad
>> >>>
>> >>> On 13 Apr 2012, at 17:03, Carl Hall <c...@hallwaytech.com>
wrote:
>> >>>
>> >>> I agree with Daniel that our modifications to the preview
processor
>> >>> have
>> >>> put its ownership square on us. Was there a community that
this script
>> >>> was
>> >>> borrowed from? I thought it was original development that
uses various
>> >>> external libraries to do the actual work. This is the
approach that
>> >>> Erik is
>> >>> taking with the rewrite using things like Tika (text
extraction),
>> >>> Sanselan
>> >>> (images) and a Java port of the python topia.termextract
library.
>> >>>
>> >>> I certainly don't deny the speed of development that was
realized in
>> >>> creating the PP but the current state of the code is a
mess at best.
>> >>> Reuse
>> >>> of libraries in Java is showing a fast rewrite with very
little
>> >>> managed code
>> >>> on our part.
>> >>>
>> >>>
>> >>> On Fri, Apr 13, 2012 at 12:50 AM, Daniel Parry
>> >>> <dan...@caret.cam.ac.uk>
>> >>> wrote:
>> >>>>
>> >>>> On Thu, Apr 12, 2012 at 04:21:36PM -0400, Clay Fenlason
wrote:
>> >>>> > I think this response is at best orthogonal to the
point John's
>> >>>> > trying
>> >>>> > to raise, though I gather this kind of reaction must
come from a
>> >>>> > buildup of some real frustration around the PP, which I
don't mean
>> >>>> > to
>> >>>> > discount. I also think John was pretty clear about what
he was
>> >>>> > suggesting: that there be a conversation with the
community we got
>> >>>> > the
>> >>>> > PP from, if the conversation hasn't happened already,
to see if
>> >>>> > there
>> >>>> > might still be a way to work together before we decide
to just own
>> >>>> > it
>> >>>> > ourselves.
>> >>>>
>> >>>> I'd suggest the way that the preview processor was being
extended
>> >>>> (initially a
>> >>>> python server add on, followed by a ruby rewrite for tag
extraction)
>> >>>> and
>> >>>> the
>> >>>> variety of ruby versions that deployers were using and
the methods
>> >>>> used
>> >>>> to
>> >>>> deploy it were indicative of a) the OAE community already
'owning'
>> >>>> the PP
>> >>>> and b)
>> >>>> as has already been pointed out some standardization
needed restoring
>> >>>> and
>> >>>> additional functionality added for deployers. Hence, the
list was
>> >>>> pinged[0] a
>> >>>> while back to ask about standardizing and extending in
java. I'm not
>> >>>> sure
>> >>>> of any
>> >>>> other way to contact the original PP community or if such
a community
>> >>>> separate
>> >>>> to OAE even still exists?
>> >>>>
>> >>>> Best wishes,
>> >>>>
>> >>>> Daniel
>> >>>>
>> >>>> [0]
>> >>>>
>> >>>> http://collab.sakaiproject.org/pipermail/oae-dev/2012-April/001677.html
>> >>>>
>> >>>> --
>> >>>> --| Daniel Parry: dan...@caret.cam.ac.uk. www.caret.cam.ac.uk/
|--
>> >>>> "Of all the things a leader should fear, complacency should
>> >>>> head the list." [John C. Maxwell]
>> >>>> _______________________________________________
>> >>>> oae-dev mailing list
>> >>>> oae-dev@collab.sakaiproject.org
>> >>>> http://collab.sakaiproject.org/mailman/listinfo/oae-dev
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> oae-dev mailing list
>> >>> oae-dev@collab.sakaiproject.org
>> >>> http://collab.sakaiproject.org/mailman/listinfo/oae-dev
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> oae-dev mailing list
>> >> oae-dev@collab.sakaiproject.org
>> >> http://collab.sakaiproject.org/mailman/listinfo/oae-dev
>> >>
>> > _______________________________________________
>> > oae-dev mailing list
>> > oae-dev@collab.sakaiproject.org
>> > http://collab.sakaiproject.org/mailman/listinfo/oae-dev
>> > Charles Sturt University
>> >
>> > | ALBURY-WODONGA | BATHURST | CANBERRA | DUBBO | GOULBURN |
MELBOURNE |
>> > ONTARIO | ORANGE | PORT MACQUARIE | SYDNEY | WAGGA WAGGA |
>> >
>> > LEGAL NOTICE
>> > This email (and any attachment) is confidential and is
intended for the
>> > use of the addressee(s) only. If you are not the intended
recipient of this
>> > email, you must not copy, distribute, take any action in
reliance on it or
>> > disclose it to anyone. Any confidentiality is not waived or
lost by reason
>> > of mistaken delivery. Email should be checked for viruses
and defects before
>> > opening. Charles Sturt University (CSU) does not accept
liability for
>> > viruses or any consequence which arise as a result of this
email
>> > transmission. Email communications with CSU may be subject
to automated
>> > email filtering, which could result in the delay or deletion
of a legitimate
>> > email before it is read at CSU. The views expressed in this
email are not
>> > necessarily those of CSU.
>> >
>> > Charles Sturt University in Australia http://
www.csu.edu.au The
>> > Chancellery, Panorama Avenue, Bathurst NSW Australia 2795
ABN: 83 878 708
>> > 551; CRICOS Provider Numbers: 00005F (NSW), 01947G (VIC),
02960B (ACT)
>> >
>> > Charles Sturt University in Ontario http://www.charlessturt.ca
860
>> > Harrington Court, Burlington Ontario Canada L7N 3N4
Registration:
>> > www.peqab.ca
>> >
>> > Consider the environment before printing this email.
>
>
_______________________________________________
oae-dev mailing list
oae-dev@collab.sakaiproject.org
http://collab.sakaiproject.org/mailman/listinfo/oae-dev
