Hi Mark:
A couple of points:
1. My concerns have nothing to do with solr - I would say exactly the same
thing if the component in question were an RDF triple-store, a JMS provider,
etc - or any other powerful and useful piece of infrastructure that can help
DSpace scale. I think it is worth putting substantial effort (and pain) into
preserving the 'market advantage' that DSpace now enjoys of having both a very
low technical barrier to entry, but the ability to 'turn the complexity dial'
(lucene->solr->multicore->sharded) when needed for scale or performance - in
just the way XOAI appears to have engineered it.
2. Even if solr were as lightweight as a file system, our application coverage
of it is rather patchy (as you note below). I'm concerned that if efforts like
your #1 (which I completely endorse *as an option*) were done and we remove
lucene support, we risk disenfranchising many users. Last time I checked, about
half of even the 1.8 adopters were using JSPUI (which lacks many solr-backed
features) - should we cut them loose (i.e. let them worry about solrizing their
UI themselves), or preserve the legacy lucene support?
So by all means, let's continue to demonstrate the benefits of solr and expand
its use - just without pulling the lucene rug out from under us.
Thanks,
Richard
On Jul 19, 2012, at 1:28 PM, Mark Diggory wrote:
Richard, can you clarify what appears an aversion to formally adopting solr as
part of the application stack?
My concerns with your opposition:
1. We are really at the point of needing to consolidate search, browse and
discovery to only offer one unified solution. It's too much complexity to
continue to maintain three solutions. Using th database for browse is clearly
antiquated, we need an exit strategy so we can increase the actual features
DSpace can deliver out of the box, such as those we presented at OR12.
2. The features / benefits are fairly obvious, the impact on existing DSpace
servers to run discovery and solr is actually minimal, and we continue to
manifest performance improvements and options in this area, statistics by
atmire and options to run ElasticSearch under both discovery and statistics by
Peter Dietz, external website integrations such as those shown by Stuart and
Kim at last years conference show how the adoption of solr as a default part of
the application stack will offer a path to more, better features in DSpace.
In rebuttle to your concerns, XOAI additional load on the application server
using solr is offset by a reduction in load on the database. Solr can be farmed
out to other servers as needed, just like postgres.
Remember, 3.0 is the big opportunity we have to make these transitions, which
honestly, are not revolutionary.
Mark
On Thursday, July 19, 2012, Richard Rodgers wrote:
My vote would be for option 2. below.
First,
I would turn the observation
>> The original OAI app hasn't changed in 10 years…
around and say that's proof it needs little or no support.
Second,
If XOAI were a drop-in replacement, I'd vote for #1, but it has some different
requirements - viz SOLR
I am very leery of upping the base minimum DSpace stack unless absolutely
required.
There are a number of great features in XOAI - but not everyone might need them:
OpenAire/Driver has no use outside the EU, and the SOLR-based speed benefits,
while impressive, might not be
needed for basic M2M harvest scenarios.
If SOLR were made optional, and the stack footprint essentially the same as the
old OAI, I'd recast my vote to #1
Richard
On Jul 19, 2012, at 11:48 AM, Tim Donohue wrote:
> All,
>
> Copying in dspace-devel. This thread likely should be over on -devel if
> we want broader feedback.
>
> On 7/19/2012 9:17 AM, Mark Diggory wrote:
>>
>>
>> On Thursday, July 19, 2012, Robin Taylor wrote:
>>
>> Hi Sands et al,
>>
>> I inadvertently started a discussion on how to cater for addons to
>> DSpace yesterday but that wasn't my intention.
>>
>>
>> I think I was as much to blame for this ;)
>>
>> think we need to
>> consider what to do with the XOAI module but within the existing
>> framework. The Lyncode guys have gone to the trouble of contributing so
>> they deserve an answer of some sort. For the sake of discussion lets
>> assume its a good piece of work, the options would appear to be...
>>
>> 1. Replace the existing OAI module with the new one.
>>
>> The original OAI app hasn't changed in 10 years... No one really is here
>> to support the original code... I think it's time is ripe to throw it out.
>>
>> 2. Distribute it alongside the old one.
>> 3. Put the new module in Github somewhere but as a separate repo, as is
>> the case with dspace-rest. This gives it sort of semi-official status.
>> 4. Do nothing.
>> 5. Other options I can't think of.
>>
>>
>> All these paths lead to greater complexity with the only benefit being,
>> for some hypothetical status quo who fear change, not having to.
>>
>>
>> Any ideas ?
>>
>>
>> Here's my position, the contribution of a solution to DSpace does not
>> require a concensus vote, that would significantly slow the progress of
>> contributing. What is needed are one or two commiter "eyes" looking at
>> the code to assure its sane and works, then agreeing to the pull
>> request, accepting or rejecting it with comment. We already made
>> comments and recommendations, we need someone to quickly test the code
>> and confirm it works, then accept the pull request. The testing period
>> will rat out any bugs...
>
> +1 to what MarkD has said above.
>
> I'd be of the frame of mind to *replace* the existing OAI with OAI-2.0,
> ASSUMING that we get more eyes on OAI-2.0 and those
> committers/developers are comfortable with the new implementation &
> think it works well. Further testing can happen during Testathon. I
> also agree with MarkD's point that the existing OAI work is essentially
> "unmaintained" for many years...
>
> So, to anyone listening...if you have some time to try out OAI-2.0 (aka
> XOAI) work, please provide us with your feedback! Relevant links:
>
> Docs: https://wiki.duraspace.org/display/DSPACE/OAI+2.0
> Download & Install from: http://www.lyncode.com/dspace/addons/xoai/
> Add feedback/comments to: https://jira.duraspace.org/browse/DS-1202
>
> It should be noted, that there is also the option of doing the following:
>
> * Replace "OAI-1.0" with OAI 2.0, and *MOVE* the old OAI-1.0 code to a
> place where people can download & install it if they are unsatisfied
> with OAI 2.0.
>
> Personally, I think we should try to get out of the habit of keeping old
> modules (in this case OAI-1.0) around in the DSpace release as a "just
> in case" measure. We can always make that old OAI-1.0 module available
> for download separately, if people really need/want it.
>
> Just my personal opinions here... If I'm in the minority though, that's
> fine! :)
>
> - Tim
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Dspace-devel mailing list
> [email protected]<javascript:;>
> https://lists.sourceforge.net/lists/listinfo/dspace-devel
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Dspace-devel mailing list
[email protected]<javascript:;>
https://lists.sourceforge.net/lists/listinfo/dspace-devel
--
[@mire Inc.]
Mark Diggory (Schedule a Meeting<https://tungle.me/markdiggory>)
2888 Loker Avenue East, Suite 305, Carlsbad, CA. 92010
Esperantolaan 4, Heverlee 3001, Belgium
http://www.atmire.com<http://www.atmire.com/>
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Dspace-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dspace-devel
