Re: [Dspace-tech] DSpace suggestions?
Hi Andrea, I do actually have use for the bot prefix wildcard, so it would be useful if you could send me how you've built it. We're planning on using that for similar bot-detecting. Peter Dietz On Thu, Jul 14, 2011 at 5:52 PM, Andrea Schweer schw...@waikato.ac.nzwrote: Hi, On 15/07/11 05:13, Peter Dietz wrote: However, there could be a way for people to pool effort to work on common needs. I like what you describe in terms of sharing information / pooling efforts to work on common needs. Though I have no idea for a good platform for that, other than the mailing list or IRC. As to one of the specific points you mentioned: Another group has figured out how to search in SOLR for having a wildcard at the start of a query. I have a working but hackish solution for this that is probably not efficient enough for production use (I needed it for a one-off query only). I'm happy to share this if anyone is interested. cheers, Andrea -- Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets Revealed. This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- 5 Ways to Improve Secure Unified Communications Unified Communications promises greater efficiencies for business. UC can improve internal communications as well as offer faster, more efficient ways to interact with customers and streamline customer service. Learn more! http://www.accelacomm.com/jaw/sfnl/114/51426253/___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] DSpace suggestions?
Hi Peter,
On 22/07/11 09:16, Peter Dietz wrote:
I do actually have use for the bot prefix wildcard, so it would be
useful if you could send me how you've built it. We're planning on
using that for similar bot-detecting.
I added a custom query parser to solrconfig.xml:
queryParser name=wildcard
class=nz.ac.lconz.irr.solr.CleanupQParserPlugin /
My CleanupQParserPlugin extends org.apache.solr.search.QParserPlugin; I
copied most of the code out of
org.apache.solr.search.LuceneQParserPlugin and added a
setAllowLeadingWildcard(true). I'm attaching my plugin code but I don't
know if this will make it to the mailing list.
You can then use this plugin in queries like this:
http://127.0.0.1:8080/solr/statistics/select/?defType=wildcardq=userAgent:*bot*+OR+*crawler*+OR+*spider*rows=0
You can't seem to be able to use it in delete queries though. My
workaround for that is to do a userAgent facet query for all user agents
that match the query above, then go through the list of user agents
found and explicitly delete all hits by this particular user agent. I'm
attaching my code for this too (I really need to get this on github).
I used this to remove the user agents *crawler*, *bot* and *spider* with
a one-off query; it seems to have worked nicely on my test data
(~120,000 item/bitstream hits) but I haven't run it on my production
servers yet. To keep these user agents out of the data from this point
forward, I customised SolrLoggerUsageEventListener to not log views by
these user agents (and to omit other events, eg to non-public bitstreams).
It's all not the most elegant solution in the world but it seems to work
ok for my purpose.
cheers,
Andrea
--
Andrea Schweer
IRR Technical Specialist, ITS Information Systems
The University of Waikato, Hamilton, New Zealand
package nz.ac.lconz.irr.solr;
import org.apache.lucene.queryParser.ParseException;
import org.apache.lucene.queryParser.QueryParser;
import org.apache.lucene.search.Query;
import org.apache.solr.common.params.CommonParams;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.search.QParser;
import org.apache.solr.search.QParserPlugin;
import org.apache.solr.search.QueryParsing;
import org.apache.solr.search.SolrQueryParser;
/**
* Query parser plugin for solr that allows leading wildcards in queries.
* Pretty much copy/paste from existing solr/lucene code.
*/
public class CleanupQParserPlugin extends QParserPlugin {
@Override
public QParser createParser(String qstr, SolrParams localParams, SolrParams params, SolrQueryRequest req) {
return new WildcardLuceneQParser(qstr, localParams, params, req);
}
public void init(NamedList args) {
// nothing to do
}
class WildcardLuceneQParser extends QParser {
String sortStr;
SolrQueryParser lparser;
public WildcardLuceneQParser(String qstr, SolrParams localParams, SolrParams params, SolrQueryRequest req) {
super(qstr, localParams, params, req);
}
public Query parse() throws ParseException {
String qstr = getString();
String defaultField = getParam(CommonParams.DF);
if (defaultField==null) {
defaultField = getReq().getSchema().getDefaultSearchFieldName();
}
lparser = new SolrQueryParser(this, defaultField);
// these could either be checked set here, or in the SolrQueryParser constructor
String opParam = getParam(QueryParsing.OP);
if (opParam != null) {
lparser.setDefaultOperator(AND.equals(opParam) ? QueryParser.Operator.AND : QueryParser.Operator.OR);
} else {
// try to get default operator from schema
String operator = getReq().getSchema().getQueryParserDefaultOperator();
lparser.setDefaultOperator(AND.equals(operator) ?
QueryParser.Operator.AND : QueryParser.Operator.OR);
}
lparser.setAllowLeadingWildcard(true);
return lparser.parse(qstr);
}
}
}
package nz.ac.lconz.irr.statsimport;
import nz.ac.lconz.irr.solr.SolrUtils;
import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.SolrServer;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.response.FacetField;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.client.solrj.util.ClientUtils;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
/**
* @author Andrea Schweer schw...@waikato.ac.nz
*/
public class RemoveBots {
static public void main(String[] argv) {
if (argv.length 1) {
System.out.println(Usage: RemoveBots solrServer);
System.exit(1);
}
SolrServer solr = SolrUtils.setupSolrServer(argv[0]);
try {
ListString userAgents = findUserAgents(solr);
for (String userAgent : userAgents) {
removeAgentHits(solr, userAgent);
}
solr.commit();
} catch (SolrServerException e) {
e.printStackTrace(System.err);
} catch (IOException e) {
Re: [Dspace-tech] DSpace suggestions?
Hello Susan, the place for feature requests, bugs etc. is the DSpace Jira https://jira.duraspace.org/browse/DS Hope that helps Claudia Jürgen Hi, Is there an appropriate place to submit suggestions for future releases of DSpace software? I really would like to see a âNextâ button on the Item pages so that when browsing for records, when the user clicks on a record, it would not be necessary to scroll back to the browse screen to select the next record and instead this could be done from the Item pages? Thanks, Sue Sue Walker-Thornton Software Developer/Database Administrator NASA Langley Research Center|LITES Contract SGT, Inc.|130 Research Drive Hampton, Va. 23666 Office: (757) 224-4074 Mobile: (757) 506-9903 Fax: (757) 224-4001 susan.m.thorn...@nasa.gov -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets Revealed. This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech Mit freundlichen Gruessen Claudia Juergen Universitaetsbibliothek Dortmund Eldorado 0231/755-4043 https://eldorado.tu-dortmund.de/ Tel.: 0049-231-755-4043 -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets Revealed. This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] DSpace suggestions?
Hi Susan, I've seen a demo of the Wind Music site, which has implemented the feature you are talking about. An example entry would be: http://www.windmusic.org/dspace/handle/68502/97349?45735available I believe the person who developed these features was Christophe Dupriez. --- As far as the idea of having an easy feedback system for suggesting improvements to DSpace... I'm not sure how best to tackle that one. We have the user-voice portal on demo.dspace.org, so that people can report bugs, and other feedback of the demo site. That gets some use, but not a wide source of feedback, and I'm not sure we're wanting to use it for that. Jira tickets sometimes work, but they will often get dismissed if its just a feature request with no implementer and no patches. I'm not sure how much activity there is on the DSpace facebook page, or Google+ circle for DSpace. In any case, the mailing list more or less works fine for this. Bram had an idea for tagging mailing list posts with things like feature request, statistics, xmlui, DSpace 1.7, so there could be a knowledge base for finding what other people have submitted, but that would take work to build as well. However, there could be a way for people to pool effort to work on common needs. For instance, at Ohio State, I'm working on a usage statistics project with team members here. I could be more vocal and chatty with other developers to say, I need a better way to detect and remove robots. Another group, such as @mire, might say, we've got a good way to present statistical data and make reports. Another group has figured out how to search in SOLR for having a wildcard at the start of a query. Another group is saying we need to anonymize our statistical logs, since its against the law in their country to store visitors IP addresses. Peter Dietz On Thu, Jul 14, 2011 at 11:48 AM, Claudia Juergen claudia.juer...@ub.tu-dortmund.de wrote: Hello Susan, the place for feature requests, bugs etc. is the DSpace Jira https://jira.duraspace.org/browse/DS Hope that helps Claudia Jürgen Hi, Is there an appropriate place to submit suggestions for future releases of DSpace software? I really would like to see a “Next†button on the Item pages so that when browsing for records, when the user clicks on a record, it would not be necessary to scroll back to the browse screen to select the next record and instead this could be done from the Item pages? Thanks, Sue Sue Walker-Thornton Software Developer/Database Administrator NASA Langley Research Center|LITES Contract SGT, Inc.|130 Research Drive Hampton, Va. 23666 Office: (757) 224-4074 Mobile: (757) 506-9903 Fax: (757) 224-4001 susan.m.thorn...@nasa.gov -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets Revealed. This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech Mit freundlichen Gruessen Claudia Juergen Universitaetsbibliothek Dortmund Eldorado 0231/755-4043 https://eldorado.tu-dortmund.de/ Tel.: 0049-231-755-4043 -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets Revealed. This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets Revealed. This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] DSpace suggestions?
Thanks a bunch Peter. I took a look at the Wind Music site and that’s similar to what we have in mind. Sue Sue Walker-Thornton Software Developer/Database Administrator NASA Langley Research Center|LITES Contract (757) 224-4074 From: Peter Dietz [mailto:pdiet...@gmail.com] Sent: Thursday, July 14, 2011 1:14 PM To: Claudia Juergen Cc: Thornton, Susan M. (LARC-B702)[LITES]; dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] DSpace suggestions? Hi Susan, I've seen a demo of the Wind Music site, which has implemented the feature you are talking about. An example entry would be: http://www.windmusic.org/dspace/handle/68502/97349?45735available I believe the person who developed these features was Christophe Dupriez. --- As far as the idea of having an easy feedback system for suggesting improvements to DSpace... I'm not sure how best to tackle that one. We have the user-voice portal on demo.dspace.orghttp://demo.dspace.org, so that people can report bugs, and other feedback of the demo site. That gets some use, but not a wide source of feedback, and I'm not sure we're wanting to use it for that. Jira tickets sometimes work, but they will often get dismissed if its just a feature request with no implementer and no patches. I'm not sure how much activity there is on the DSpace facebook page, or Google+ circle for DSpace. In any case, the mailing list more or less works fine for this. Bram had an idea for tagging mailing list posts with things like feature request, statistics, xmlui, DSpace 1.7, so there could be a knowledge base for finding what other people have submitted, but that would take work to build as well. However, there could be a way for people to pool effort to work on common needs. For instance, at Ohio State, I'm working on a usage statistics project with team members here. I could be more vocal and chatty with other developers to say, I need a better way to detect and remove robots. Another group, such as @mire, might say, we've got a good way to present statistical data and make reports. Another group has figured out how to search in SOLR for having a wildcard at the start of a query. Another group is saying we need to anonymize our statistical logs, since its against the law in their country to store visitors IP addresses. Peter Dietz On Thu, Jul 14, 2011 at 11:48 AM, Claudia Juergen claudia.juer...@ub.tu-dortmund.demailto:claudia.juer...@ub.tu-dortmund.de wrote: Hello Susan, the place for feature requests, bugs etc. is the DSpace Jira https://jira.duraspace.org/browse/DS Hope that helps Claudia Jürgen Hi, Is there an appropriate place to submit suggestions for future releases of DSpace software? I really would like to see a “Next†button on the Item pages so that when browsing for records, when the user clicks on a record, it would not be necessary to scroll back to the browse screen to select the next record and instead this could be done from the Item pages? Thanks, Sue Sue Walker-Thornton Software Developer/Database Administrator NASA Langley Research Center|LITES Contract SGT, Inc.|130 Research Drive Hampton, Va. 23666 Office: (757) 224-4074tel:%28757%29%20224-4074 Mobile: (757) 506-9903tel:%28757%29%20506-9903 Fax: (757) 224-4001tel:%28757%29%20224-4001 susan.m.thorn...@nasa.govmailto:susan.m.thorn...@nasa.gov -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets Revealed. This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.netmailto:DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech Mit freundlichen Gruessen Claudia Juergen Universitaetsbibliothek Dortmund Eldorado 0231/755-4043tel:0231%2F755-4043 https://eldorado.tu-dortmund.de/ Tel.: 0049-231-755-4043 -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets Revealed. This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.netmailto:DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets
Re: [Dspace-tech] DSpace suggestions?
Thanks again Bram. I just created a DSpace “New feature” request: https://jira.duraspace.org/browse/DS-956 Best regards, Sue Sue Walker-Thornton Software Developer/Database Administrator NASA Langley Research Center|LITES Contract (757) 224-4074 From: bluy...@gmail.com [mailto:bluy...@gmail.com] On Behalf Of Bram Luyten Sent: Thursday, July 14, 2011 11:36 AM To: Thornton, Susan M. (LARC-B702)[LITES] Subject: Re: [Dspace-tech] DSpace suggestions? Hi Sue, submitting it as a feature request in JIRA would ensure that the request wouldn't get lost. I'd really recommend that compared to putting it on the wiki somewhere. A few of mine are listed there: https://jira.duraspace.org/browse/DS-919 https://jira.duraspace.org/browse/DS-929 (this one is actually submitted as an Improvement, so you can choose whether you want to label your request as a new feature or an improvement). good luck, Bram @mire Esperantolaan 4 - 3001 Heverlee - Belgium 2888 Loker Avenue East, Suite 305 - Carlsbad, CA 92010 - USA atmire.comhttp://atmire.com/ - Institutional Repository Solutions On Thu, Jul 14, 2011 at 5:18 PM, Thornton, Susan M. (LARC-B702)[LITES] susan.m.thorn...@nasa.govmailto:susan.m.thorn...@nasa.gov wrote: Hi, Is there an appropriate place to submit suggestions for future releases of DSpace software? I really would like to see a “Next” button on the Item pages so that when browsing for records, when the user clicks on a record, it would not be necessary to scroll back to the browse screen to select the next record and instead this could be done from the Item pages? Thanks, Sue Sue Walker-Thornton Software Developer/Database Administrator NASA Langley Research Center|LITES Contract SGT, Inc.|130 Research Drive Hampton, Va. 23666 Office: (757) 224-4074 Mobile: (757) 506-9903 Fax: (757) 224-4001 susan.m.thorn...@nasa.govmailto:susan.m.thorn...@nasa.gov -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets Revealed. This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.netmailto:DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets Revealed. This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] DSpace suggestions?
Hi, On 15/07/11 05:13, Peter Dietz wrote: However, there could be a way for people to pool effort to work on common needs. I like what you describe in terms of sharing information / pooling efforts to work on common needs. Though I have no idea for a good platform for that, other than the mailing list or IRC. As to one of the specific points you mentioned: Another group has figured out how to search in SOLR for having a wildcard at the start of a query. I have a working but hackish solution for this that is probably not efficient enough for production use (I needed it for a one-off query only). I'm happy to share this if anyone is interested. cheers, Andrea -- Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets Revealed. This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] DSpace suggestions?
All, On 7/14/2011 12:13 PM, Peter Dietz wrote: As far as the idea of having an easy feedback system for suggesting improvements to DSpace... I'm not sure how best to tackle that one. We have the user-voice portal on demo.dspace.org http://demo.dspace.org, so that people can report bugs, and other feedback of the demo site. That gets some use, but not a wide source of feedback, and I'm not sure we're wanting to use it for that. Jira tickets sometimes work, but they will often get dismissed if its just a feature request with no implementer and no patches. Just a quick point to add. JIRA is often the best place to add new feature requests. The reason is that our DSpace Community Advisory Team (DCAT) also reviews all Feature Requests which are entered into JIRA. So, I'd recommend that as the place to enter in new feature requests. Obviously, if they are just very early ideas it can also help to first discuss them on the lists before putting a feature request in JIRA. More information on DCAT their Feature Request review processes: https://wiki.duraspace.org/display/cmtygp/DSpace+Community+Advisory+Team - Tim -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on Lean Startup Secrets Revealed. This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
