Francis 66.249.76.34 is a googlebot IP
you should enable sitemaps, googlebots honor them have a look at Search Engine Optimization <https://wiki.duraspace.org/display/DSDOC5x/Search+Engine+Optimization> documentation Monika ________________ Monika Mevenkamp mo.me...@gmail.com http://mo-meven.tumblr.com/ http://mcmprogramming.com/mo.meven/ > On Oct 25, 2017, at 9:39 AM, Francis Brouns <francis.bro...@ou.nl> wrote: > > Hi all, > > our DSpace servers is being flooded with search request since the beginning > of October. Normally we get about 250000 search requests in a month, now we > get 2.5 million in 2 weeks. It seems that these requests are all aimed at a > particular Community and are searching for combination of authors and > subjects over and over. Most of the time these search request have no results. > > Running DSpace 5.4 on SLES Linux, tomcat 7, java 7, jspui > > In the dspace log, I find numerous requests like these: > ip_addr=66.249.76.34:search:scope=org.dspace.content.Community@287,query="null",results=(0,0,0) > > in tomcat localhost-access log > 66.249.76.34 - - [24/Oct/2017:01:05:30 +0200] "GET > /handle/1820/2145/simple-search?location=1820%2F2145&query=&filter_field_1=dateIssued&filter_type_1=equals&filter_value_1=2011&filter_field_2=author&filter_type_2=equals&filter_value_2=Van+Hooft%2C+W.+F.&filter_field_3=author&filter_type_3=equals&filter_value_3=Leirs%2C+H.&filter_field_4=author&filter_type_4=equals&filter_value_4=Bauer%2C+H.&filter_field_5=subject&filter_type_5=equals&filter_value_5=phylogeography&filter_field_6=author&filter_type_6=equals&filter_value_6=Van+Haeringen%2C+W.+A.&filter_field_7=author&filter_type_7=equals&filter_value_7=Bertola%2C+L.+D.&filter_field_8=subject&filter_type_8=equals&filter_value_8=evolutionary+history&filter_field_9=author&filter_type_9=equals&filter_value_9=Tumenta%2C+P.+N.&filter_field_10=author&filter_type_10=equals&filter_value_10=York%2C+D.+S.&filter_field_11=subject&filter_type_11=equals&filter_value_11=Panthera+leo&rpp=5&sort_by=dc.title_sort&order=DESC&etal=0 > HTTP/1.1" 200 30123 - /handle/1820/2145/simple-search > - 127.0.0.1 - - [24/Oct/2017:01:05:30 +0200] "GET > /solr/search/select?q=*%3A*&fl=dateIssued.year%2Chandle%2Csearch.resourcetype%2Csearch.resourceid&fq=NOT%28withdrawn%3Atrue%29&fq=NOT%28discoverable%3Afalse%29&fq=subject_keyword%3ASubsidiarity&fq=subject_keyword%3ACollaborative%5C+Learning&fq=subject_keyword%3AVirtual%5C+Campus&fq=dateIssued_keyword%3A2009&fq=subject_keyword%3AOrganizational%5C+Model&fq=subject_keyword%3ALearning%5C+for%5C+Sustainable%5C+Development&fq=subject_keyword%3AVirtual%5C+Mobility&fq=subject_keyword%3ANetworked%5C+Learning&fq=location%3Am18&fq=dateIssued.year%3A%5B*+TO+*%5D&fq=read%3A%28g0+OR+g0%29&start=0&rows=1&sort=dateIssued.year_sort+asc&wt=javabin&version=2 > HTTP/1.1" 200 611 - /solr/search/select > - 127.0.0.1 - - [24/Oct/2017:01:05:30 +0200] "GET > /solr/search/select?q=*%3A*&fl=dateIssued.year%2Chandle%2Csearch.resourcetype%2Csearch.resourceid&fq=NOT%28withdrawn%3Atrue%29&fq=NOT%28discoverable%3Afalse%29&fq=subject_keyword%3ASubsidiarity&fq=subject_keyword%3ACollaborative%5C+Learning&fq=subject_keyword%3AVirtual%5C+Campus&fq=dateIssued_keyword%3A2009&fq=subject_keyword%3AOrganizational%5C+Model&fq=subject_keyword%3ALearning%5C+for%5C+Sustainable%5C+Development&fq=subject_keyword%3AVirtual%5C+Mobility&fq=subject_keyword%3ANetworked%5C+Learning&fq=location%3Am18&fq=location%3Am18&fq=dateIssued.year%3A%5B*+TO+*%5D&fq=read%3A%28g0+OR+g0%29&start=0&rows=1&sort=dateIssued.year_sort+desc&wt=javabin&version=2 > HTTP/1.1" 200 625 - /solr/search/select > - 127.0.0.1 - - [24/Oct/2017:01:05:30 +0200] "GET > /solr/search/select?q=*%3A*&fl=dateIssued.year%2Chandle%2Csearch.resourcetype%2Csearch.resourceid&fq=NOT%28withdrawn%3Atrue%29&fq=NOT%28discoverable%3Afalse%29&fq=dateIssued_keyword%3A2011&fq=subject_keyword%3Alion&fq=author_keyword%3ASogbohossou%2C%5C+E.&fq=author_keyword%3AVan%5C+Haeringen%2C%5C+W.%5C+A.&fq=author_keyword%3AVan%5C+Hooft%2C%5C+W.%5C+F.&fq=subject_keyword%3AWest%5C+Africa&fq=subject_keyword%3Aphylogenetics&fq=author_keyword%3APrins%2C%5C+H.%5C+H.%5C+T.&fq=author_keyword%3AYork%2C%5C+D.%5C+S.&fq=author_keyword%3AUit%5C+de%5C+Weerd%2C%5C+D.%5C+R.&fq=author_keyword%3AFunston%2C%5C+P.%5C+J.&fq=subject_keyword%3Aevolutionary%5C+history&fq=author_keyword%3AUdo%5C+de%5C+Haes%2C%5C+H.%5C+A.&fq=location%3Am18&fq=dateIssued.year%3A%5B*+TO+*%5D&fq=read%3A%28g0+OR+g0%29&start=0&rows=1&sort=dateIssued.year_sort+asc&wt=javabin&version=2 > HTTP/1.1" 200 740 - /solr/search/select > - 127.0.0.1 - - [24/Oct/2017:01:05:30 +0200] "GET > /solr/search/select?q=*%3A*&fl=dateIssued.year%2Chandle%2Csearch.resourcetype%2Csearch.resourceid&fq=NOT%28withdrawn%3Atrue%29&fq=NOT%28discoverable%3Afalse%29&fq=dateIssued_keyword%3A2011&fq=subject_keyword%3Alion&fq=author_keyword%3ASogbohossou%2C%5C+E.&fq=author_keyword%3AVan%5C+Haeringen%2C%5C+W.%5C+A.&fq=author_keyword%3AVan%5C+Hooft%2C%5C+W.%5C+F.&fq=subject_keyword%3AWest%5C+Africa&fq=subject_keyword%3Aphylogenetics&fq=author_keyword%3APrins%2C%5C+H.%5C+H.%5C+T.&fq=author_keyword%3AYork%2C%5C+D.%5C+S.&fq=author_keyword%3AUit%5C+de%5C+Weerd%2C%5C+D.%5C+R.&fq=author_keyword%3AFunston%2C%5C+P.%5C+J.&fq=subject_keyword%3Aevolutionary%5C+history&fq=author_keyword%3AUdo%5C+de%5C+Haes%2C%5C+H.%5C+A.&fq=location%3Am18&fq=location%3Am18&fq=dateIssued.year%3A%5B*+TO+*%5D&fq=read%3A%28g0+OR+g0%29&start=0&rows=1&sort=dateIssued.year_sort+desc&wt=javabin&version=2 > HTTP/1.1" 200 758 - /solr/search/select > - 127.0.0.1 - - [24/Oct/2017:01:05:30 +0200] "GET > /solr/search/select?q=*%3A*&fl=handle%2Csearch.resourcetype%2Csearch.resourceid&spellcheck.q=*%3A*&spellcheck.collate=true&spellcheck=true&fq=NOT%28withdrawn%3Atrue%29&fq=NOT%28discoverable%3Afalse%29&fq=subject_keyword%3ASubsidiarity&fq=subject_keyword%3ACollaborative%5C+Learning&fq=subject_keyword%3AVirtual%5C+Campus&fq=dateIssued_keyword%3A2009&fq=subject_keyword%3AOrganizational%5C+Model&fq=subject_keyword%3ALearning%5C+for%5C+Sustainable%5C+Development&fq=subject_keyword%3AVirtual%5C+Mobility&fq=subject_keyword%3ANetworked%5C+Learning&fq=location%3Am18&fq=read%3A%28g0+OR+g0%29&start=0&rows=10&sort=dc.date.issued_dt+asc&facet.field=author_filter&facet.field=subject_filter&facet=true&f.author_filter.facet.limit=19&f.author_filter.facet.sort=count&f.author_filter.facet.offset=0&f.subject_filter.facet.limit=19&f.subject_filter.facet.sort=count&f.subject_filter.facet.offset=0&facet.mincount=1&facet.offset=0&wt=javabin&version=2 > HTTP/1.1" 200 1038 - /solr/search/select > > tomcat logging increased dramatically. Yesterday's localhost-access log was > 3.5Gb. This caused this particular volume to run out of space. > > DSpace is located on a separate volume, but nevertheless solr reports errors > in writing because of lack of file space, while there still is file space > left on the dspace volume. > > Are pointers in how to track down who is doing all these requests and why > these occur so often are highly appreciated. > How do I know what community @287 is? > Is it possible that the Google crawler now trips over special characters in > bitstream file names and community names? One of the communities has an & in > the title, and some filenames contained ( and , > > kind regards, > Francis Brouns > > > > > -- > You received this message because you are subscribed to the Google Groups > "DSpace Technical Support" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to dspace-tech+unsubscr...@googlegroups.com > <mailto:dspace-tech+unsubscr...@googlegroups.com>. > To post to this group, send email to dspace-tech@googlegroups.com > <mailto:dspace-tech@googlegroups.com>. > Visit this group at https://groups.google.com/group/dspace-tech > <https://groups.google.com/group/dspace-tech>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscr...@googlegroups.com. To post to this group, send email to dspace-tech@googlegroups.com. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
