[jira] [Commented] (NUTCH-1251) Deletion of duplicates fails with org.apache.solr.client.solrj.SolrServerException
[ https://issues.apache.org/jira/browse/NUTCH-1251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245877#comment-13245877 ] Arkadi Kosmynin commented on NUTCH-1251: Thanks Markus! Deletion of duplicates fails with org.apache.solr.client.solrj.SolrServerException -- Key: NUTCH-1251 URL: https://issues.apache.org/jira/browse/NUTCH-1251 Project: Nutch Issue Type: Bug Components: indexer Affects Versions: 1.4 Environment: Any crawl where the number of URLs in Solr exceeds 1024 (the default max number of clusters in Lucene boolean query). Reporter: Arkadi Kosmynin Priority: Critical Fix For: 1.6 Deletion of duplicates fails. This happens because the get all query used to get Solr index size is id:[* TO *], which is a range query. Lucene is trying to expand it to a Boolean query and gets as many clauses as there are ids in the index. This is too many in a real situation and it throws an exception. To correct this problem, change the get all query (SOLR_GET_ALL_QUERY) to \*:\*, which is the standard Solr get all query. Indexing log extract: java.io.IOException: org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRecordReader(SolrDeleteDuplicates.java:236) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:338) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) Caused by: org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118) at org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRecordReader(SolrDeleteDuplicates.java:234) ... 3 more Caused by: org.apache.solr.common.SolrException: Internal Server Error Internal Server Error request: http://localhost:8081/arch/select?q=id:[* TO *]fl=id,boost,tstamp,digeststart=0rows=82938wt=javabinversion=2 at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:430) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244) at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89) ... 5 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1251) Deletion of duplicates fails with org.apache.solr.client.solrj.SolrServerException
[ https://issues.apache.org/jira/browse/NUTCH-1251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188095#comment-13188095 ] Markus Jelsma commented on NUTCH-1251: -- Can you provide a patch for trunk? Deletion of duplicates fails with org.apache.solr.client.solrj.SolrServerException -- Key: NUTCH-1251 URL: https://issues.apache.org/jira/browse/NUTCH-1251 Project: Nutch Issue Type: Bug Components: indexer Affects Versions: 1.4 Environment: Any crawl where the number of URLs in Solr exceeds 1024 (the default max number of clusters in Lucene boolean query). Reporter: Arkadi Kosmynin Priority: Critical Fix For: 1.5 Deletion of duplicates fails. This happens because the get all query used to get Solr index size is id:[* TO *], which is a range query. Lucene is trying to expand it to a Boolean query and gets as many clauses as there are ids in the index. This is too many in a real situation and it throws an exception. To correct this problem, change the get all query (SOLR_GET_ALL_QUERY) to \*:\*, which is the standard Solr get all query. Indexing log extract: java.io.IOException: org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRecordReader(SolrDeleteDuplicates.java:236) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:338) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) Caused by: org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118) at org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRecordReader(SolrDeleteDuplicates.java:234) ... 3 more Caused by: org.apache.solr.common.SolrException: Internal Server Error Internal Server Error request: http://localhost:8081/arch/select?q=id:[* TO *]fl=id,boost,tstamp,digeststart=0rows=82938wt=javabinversion=2 at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:430) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244) at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89) ... 5 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1251) Deletion of duplicates fails with org.apache.solr.client.solrj.SolrServerException
[ https://issues.apache.org/jira/browse/NUTCH-1251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188115#comment-13188115 ] Arkadi Kosmynin commented on NUTCH-1251: It is one line change. File org.apache.nutch.indexer.solr.SolrDeleteDuplicates.java, line 90. Deletion of duplicates fails with org.apache.solr.client.solrj.SolrServerException -- Key: NUTCH-1251 URL: https://issues.apache.org/jira/browse/NUTCH-1251 Project: Nutch Issue Type: Bug Components: indexer Affects Versions: 1.4 Environment: Any crawl where the number of URLs in Solr exceeds 1024 (the default max number of clusters in Lucene boolean query). Reporter: Arkadi Kosmynin Priority: Critical Fix For: 1.5 Deletion of duplicates fails. This happens because the get all query used to get Solr index size is id:[* TO *], which is a range query. Lucene is trying to expand it to a Boolean query and gets as many clauses as there are ids in the index. This is too many in a real situation and it throws an exception. To correct this problem, change the get all query (SOLR_GET_ALL_QUERY) to \*:\*, which is the standard Solr get all query. Indexing log extract: java.io.IOException: org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRecordReader(SolrDeleteDuplicates.java:236) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:338) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) Caused by: org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118) at org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRecordReader(SolrDeleteDuplicates.java:234) ... 3 more Caused by: org.apache.solr.common.SolrException: Internal Server Error Internal Server Error request: http://localhost:8081/arch/select?q=id:[* TO *]fl=id,boost,tstamp,digeststart=0rows=82938wt=javabinversion=2 at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:430) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244) at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89) ... 5 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
