Re: Solr -The connection has timed out
Finally able to get the full log details ERROR - 2013-12-30 15:13:00.811; org.apache.solr.core.SolrCore; [collection1] Solr index directory '/ctgapps/apache-solr-4.6.0/solr-4.6.0/example/solr/collection1/data/index/' is locked. Throwing exception INFO - 2013-12-30 15:13:00.812; org.apache.solr.core.SolrCore; [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@de26e52 INFO - 2013-12-30 15:13:00.812; org.apache.solr.update.SolrCoreState; Closing SolrCoreState INFO - 2013-12-30 15:13:00.813; org.apache.solr.update.DefaultSolrCoreState; SolrCoreState ref count has reached 0 - closing IndexWriter INFO - 2013-12-30 15:13:00.813; org.apache.solr.core.SolrCore; [collection1] Closing main searcher on request. INFO - 2013-12-30 15:13:00.814; org.apache.solr.core.CachingDirectoryFactory; Closing NRTCachingDirectoryFactory - 2 directories currently being tracked INFO - 2013-12-30 15:13:00.814; org.apache.solr.core.CachingDirectoryFactory; looking to close /ctgapps/apache-solr-4.6.0/solr-4.6.0/example/solr/collection1/data/index [CachedDir<>] INFO - 2013-12-30 15:13:00.814; org.apache.solr.core.CachingDirectoryFactory; Closing directory: /ctgapps/apache-solr-4.6.0/solr-4.6.0/example/solr/collection1/data/index INFO - 2013-12-30 15:13:00.815; org.apache.solr.core.CachingDirectoryFactory; looking to close /ctgapps/apache-solr-4.6.0/solr-4.6.0/example/solr/collection1/data [CachedDir<>] INFO - 2013-12-30 15:13:00.815; org.apache.solr.core.CachingDirectoryFactory; Closing directory: /ctgapps/apache-solr-4.6.0/solr-4.6.0/example/solr/collection1/data ERROR - 2013-12-30 15:13:00.817; org.apache.solr.core.CoreContainer; Unable to create core: collection1 org.apache.solr.common.SolrException: Index locked for write for core collection1 at org.apache.solr.core.SolrCore.(SolrCore.java:834) at org.apache.solr.core.SolrCore.(SolrCore.java:625) at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:557) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:592) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:271) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:263) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.lucene.store.LockObtainFailedException: Index locked for write for core collection1 at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:491) at org.apache.solr.core.SolrCore.(SolrCore.java:755) ... 13 more ERROR - 2013-12-30 15:13:00.819; org.apache.solr.common.SolrException; null:org.apache.solr.common.SolrException: Unable to create core: collection1 at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:977) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:601) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:271) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:263) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.solr.common.SolrException: Index locked for write for core collection1 at org.apache.solr.core.SolrCore.(SolrCore.java:834) at org.apache.solr.core.SolrCore.(SolrCore.java:625) at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:557) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:592) ... 10 more Caused by: org.apache.lucene.store.LockObtainFailedException: Index locked for write for core collection1 at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:491) at org.apache.solr.core.SolrCore.(SolrCore.java:755) ... 13 more INFO - 2013-12-30 15:13:00.820; org.apache.solr.servlet.SolrDispatchFilter; user.dir=/ctgapps/apache-solr-4.6.0/solr-4.6.0/example INFO - 2013-12-30 15:13:00.820; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() done WARN - 2013-12-30 15:13:00.856; org.eclipse.jetty.util.component.AbstractLifeCycle; FAILED SocketConnector@0.0.0.0:8983: java.net.BindException: Address already in use java.net.BindException: Address already in use at java.net.PlainSocketImpl.socketBi
Re: Grouping results with group.limit return wrong numFound ?
Hi; group.limit is: the number of results (documents) to return for each group. Defaults to 1. Did you check the page here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=32604232 Thanks; Furkan KAMACI 25 Aralık 2013 Çarşamba tarihinde tasmaniski adlı kullanıcı şöyle yazdı: > Hi All, When I perform a search with grouping result in a groups and do limit > results in one group I got that *numFound* is the same as I didn't use > limit.looks like SOLR first perform search and calculate numFound and that > group and limit the results.I do not know if this is a bug or a feature > :)But I cannot use pagination and other stuff.Is there any workaround or I > missed something ?Example:I want to search book title and limit the search > to 3 results per one publisher.q=book_title: solr > php&group=true&group.field=publisher&group.limit=3&group.main=trueI have for > apress publisher 20 results but I show only 3 that works OKBut in numFound I > still have 20 for apress publisher... > > > > -- > View this message in context: http://lucene.472066.n3.nabble.com/Grouping-results-with-group-limit-return-wrong-numFound-tp4108174.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: REYPLAY_ERR: IOException reading log
Hi YouPeng; Could you share you schema fields? Thanks; Furkan KAMACI 27 Aralık 2013 Cuma tarihinde YouPeng Yang adlı kullanıcı şöyle yazdı: > Hi users > I have build a SolrCloud on tomcat.The cloud contains 22 shards with no > replica.Also the the solrcloud is integrated with HDFS. > >After imported data for oracle to the solrcloud, I restart the tomcat > ,it does not comes alive againt. >It always give an exceptions. > >I'm really have not aware about this excetion. Because My schema do not > contains a BigDecimal type field. > >Could you give any tips? > > 746635 [recoveryExecutor-44-thread-1] WARN > org.apache.solr.update.UpdateLog - REYPLAY_ERR: IOException reading log > org.apache.solr.common.SolrException: Invalid Number: > java.math.BigDecimal:238088174 > at > org.apache.solr.schema.TrieField.readableToIndexed(TrieField.java:396) > at > org.apache.solr.update.AddUpdateCommand.getIndexedId(AddUpdateCommand.java:98) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:582) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:435) > at > org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100) > at > org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1313) > at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1202) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:662) > 746681 [recoveryExecutor-44-thread-1] WARN > org.apache.solr.update.UpdateLog - REYPLAY_ERR: IOException reading log > org.apache.solr.common.SolrException: Invalid Number: > java.math.BigDecimal:238088175 >
RE: monitoring solr logs
Hi,
You should look at Logsene:
http://sematext.com/logsene (free)
It has Kibana + "native" UI, it's not limited to logs, and if you are
monitoring your Solr and/or Zookeeper with SPM you can have your
performance metrics graphs and your logs side by side for much more
efficient troubleshooting.
Otis
Solr & ElasticSearch Support
http://sematext.com/
On Dec 30, 2013 12:06 PM, "adfel70" wrote:
> And are you using any tool like kibana as a dashboard for the logs?
>
>
>
> Tim Potter wrote
> > We're (LucidWorks) are actively developing on logstash4solr so if you
> have
> > issues, let us know. So far, so good for me but I upgraded to logstash
> > 1.3.2 even though the logstash4solr version includes 1.2.2 you can use
> the
> > newer one. I'm not quite in production with my logstash4solr <- rabbit-mq
> > <- log4j <- Solr solution yet though ;-)
> >
> > Yeah, 50GB is too much logging for only 150K docs. Maybe start by
> > filtering by log level (WARN and more severe). If a server crashes,
> you're
> > likely to see some errors in the logstash side but sometimes you may have
> > to SSH to the specific box and look at the local log (so definitely
> append
> > all messages to the local Solr log too), I'm using something like the
> > following for local logging:
> >
> > log4j.rootLogger=INFO, file
> > log4j.appender.file=org.apache.log4j.RollingFileAppender
> > log4j.appender.file.MaxFileSize=50MB
> > log4j.appender.file.MaxBackupIndex=10
> > log4j.appender.file.File=logs/solr.log
> > log4j.appender.file.layout=org.apache.log4j.PatternLayout
> > log4j.appender.file.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{3}
> > %x - %m%n
> >
> >
> > Timothy Potter
> > Sr. Software Engineer, LucidWorks
> > www.lucidworks.com
> >
> >
> > From: adfel70 <
>
> > adfel70@
>
> > >
> > Sent: Monday, December 30, 2013 9:34 AM
> > To:
>
> > solr-user@.apache
>
> > Subject: RE: monitoring solr logs
> >
> > Actually I was considering using logstash4solr, but it didn't seem mature
> > enough.
> > does it work fine? any known bugs?
> >
> > are you collecting the logs in the same solr cluster you use for the
> > production systems?
> > if so, what will you do if for some reason solr is down and you would
> like
> > to analyze the logs to see what happend?
> >
> > btw, i started a new solr cluster with 7 shards, replicationfactor=3 and
> > run
> > indexing job of 400K docs,
> > it got stuck on 150K because I used Socketappender directly to write to
> > logstash and logstash disk got full.
> >
> > that's why I moved to using AsyncAppender, and I plan on moving to using
> > rabbit.
> > but this is also why I wanted to filter some of the logs. indexing 150K
> > docs
> > prodcued 50GB of logs.
> > this seemed too much.
> >
> >
> >
> >
> > Tim Potter wrote
> >> I'm using logstash4solr (http://logstash4solr.org) for something
> similar
> >> ...
> >>
> >> I setup my Solr to use Log4J by passing the following on the
> command-line
> >> when starting Solr:
> >> -Dlog4j.configuration=file:///$SCRIPT_DIR/log4j.properties
> >>
> >> Then I use a custom Log4J appender that writes to RabbitMQ:
> >>
> >> https://github.com/plant42/rabbitmq-log4j-appender
> >>
> >> You can then configure a RabbitMQ input for logstash -
> >> http://logstash.net/docs/1.3.2/inputs/rabbitmq
> >>
> >> This decouples the log writes from log indexing in logstash4solr, which
> >> scales better for active Solr installations.
> >>
> >> Btw ... I just log everything from Solr using this approach but you can
> >> use standard Log4J configuration settings to limit which classes / log
> >> levels to send to the RabbitMQ appender.
> >>
> >> Cheers,
> >>
> >> Timothy Potter
> >> Sr. Software Engineer, LucidWorks
> >> www.lucidworks.com
> >>
> >>
> >> From: adfel70 <
> >
> >> adfel70@
> >
> >> >
> >> Sent: Monday, December 30, 2013 8:15 AM
> >> To:
> >
> >> solr-user@.apache
> >
> >> Subject: monitoring solr logs
> >>
> >> hi
> >> i'm trying to figure out which solr and zookeeper logs i should monitor
> >> and
> >> collect.
> >> All the logs will be written to a file but I want to collect some of
> them
> >> with logstash in order to be able to analyze them efficiently.
> >> any inputs on logs of which classes i should collect?
> >>
> >> thanks.
> >>
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >> http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721p4108737.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721p4108744.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Solr search videos
Hi; It is possible to do video search with Solr and Nutch. You shoul start with deciding which infotmation you will use for video search. Alt information, title of page, extracting data with XPath parser are things to think. Start doing something and I can help you. Thanks; Furkan KAMACI 30 Aralık 2013 Pazartesi tarihinde Fkyz adlı kullanıcı şöyle yazdı: > Hi everyone. > I am very new at Solr.. i'm only starting, and i dont know if this is > possible to do. > So let's say that i want to do a video search engine. To do this i was > thinking to use nutch for crawl video websites and then create the index > files. > But then i thought.. How do I know that nutch will crawl enough to solr > return the info about the video i searched? > Is this possible? this might sound a little confuse, but i hope you can help > me and if this cant be done, that you can tell me what to do. > Thanks. > > > > -- > View this message in context: http://lucene.472066.n3.nabble.com/Solr-search-videos-tp4108731.html > Sent from the Solr - User mailing list archive at Nabble.com. >
Re: Solr -The connection has timed out
Hi Rakesh; Are you sure that you do not see any errors at your log? By the way what is the exact error you see at Solr home page? Thanks; Furkan KAMACI 30 Aralık 2013 Pazartesi tarihinde rakesh adlı kullanıcı şöyle yazdı: > I have Solr server running wit jetty. Some times i am getting connection > timed out error from the home page. In the logs no errors are shown also > .Please help how to resolve this problem.Attaching the log from the Solr > > INFO - 2013-12-26 02:51:37.460; org.eclipse.jetty.server.Server; > jetty-8.1.10.v20130312 > INFO - 2013-12-26 02:51:37.490; > org.eclipse.jetty.deploy.providers.ScanningAppProvider; Deployment monitor > /ctgapps/apache-solr-4.6.0/solr-4.6.0/example/contexts at interval 0 > INFO - 2013-12-26 02:51:37.498; org.eclipse.jetty.deploy.DeploymentManager; > Deployable added: > /ctgapps/apache-solr-4.6.0/solr-4.6.0/example/contexts/solr-jetty-context.xml > INFO - 2013-12-26 02:51:37.562; > org.eclipse.jetty.webapp.WebInfConfiguration; Extract > jar:file:/ctgapps/apache-solr-4.6.0/solr-4.6.0/example/webapps/solr.war!/ to > /ctgapps/apache-solr-4.6.0/solr-4.6.0/example/solr-webapp/webapp > INFO - 2013-12-26 02:51:39.547; > org.eclipse.jetty.webapp.StandardDescriptorProcessor; NO JSP Support for > /solr, did not find org.apache.jasper.servlet.JspServlet > INFO - 2013-12-26 02:51:39.583; org.apache.solr.servlet.SolrDispatchFilter; > SolrDispatchFilter.init() > INFO - 2013-12-26 02:51:39.597; org.apache.solr.core.SolrResourceLoader; > JNDI not configured for solr (NoInitialContextEx) > INFO - 2013-12-26 02:51:39.597; org.apache.solr.core.SolrResourceLoader; > solr home defaulted to 'solr/' (could not find system property or JNDI) > INFO - 2013-12-26 02:51:39.598; org.apache.solr.core.SolrResourceLoader; > new SolrResourceLoader for directory: 'solr/' > INFO - 2013-12-26 02:51:39.714; org.apache.solr.core.ConfigSolr; Loading > container configuration from > /ctgapps/apache-solr-4.6.0/solr-4.6.0/example/solr/solr.xml > INFO - 2013-12-26 02:51:40.031; org.apache.solr.core.ConfigSolrXml; > Config-defined core root directory: > INFO - 2013-12-26 02:51:40.041; org.apache.solr.core.CoreContainer; New > CoreContainer 709424757 > INFO - 2013-12-26 02:51:40.041; org.apache.solr.core.CoreContainer; Loading > cores into CoreContainer [instanceDir=solr/] > INFO - 2013-12-26 02:51:40.057; > org.apache.solr.handler.component.HttpShardHandlerFactory; Setting > socketTimeout to: 0 > INFO - 2013-12-26 02:51:40.057; > org.apache.solr.handler.component.HttpShardHandlerFactory; Setting urlScheme > to: http:// > INFO - 2013-12-26 02:51:40.058; > org.apache.solr.handler.component.HttpShardHandlerFactory; Setting > connTimeout to: 0 > INFO - 2013-12-26 02:51:40.060; > org.apache.solr.handler.component.HttpShardHandlerFactory; Setting > maxConnectionsPerHost to: 20 > INFO - 2013-12-26 02:51:40.061; > org.apache.solr.handler.component.HttpShardHandlerFactory; Setting > corePoolSize to: 0 > INFO - 2013-12-26 02:51:40.061; > org.apache.solr.handler.component.HttpShardHandlerFactory; Setting > maximumPoolSize to: 2147483647 > INFO - 2013-12-26 02:51:40.061; > org.apache.solr.handler.component.HttpShardHandlerFactory; Setting > maxThreadIdleTime to: 5 > INFO - 2013-12-26 02:51:40.062; > org.apache.solr.handler.component.HttpShardHandlerFactory; Setting > sizeOfQueue to: -1 > INFO - 2013-12-26 02:51:40.062; > org.apache.solr.handler.component.HttpShardHandlerFactory; Setting > fairnessPolicy to: false > INFO - 2013-12-26 02:51:40.247; org.apache.solr.logging.LogWatcher; SLF4J > impl is org.slf4j.impl.Log4jLoggerFactory > > > > -- > View this message in context: http://lucene.472066.n3.nabble.com/Solr-The-connection-has-timed-out-tp4108802.html > Sent from the Solr - User mailing list archive at Nabble.com. >
Re: Chaining plugins
You don't need to write your own handler. See the previpous comment about implementing a SearchComponent -- you can check for the params in your prepare() method and do whatever side effects you want, then register your custom component and hook it into the component chain of whatever handler configuration you want (either using the "components" or by specifying it as a "first-components"... https://cwiki.apache.org/confluence/display/solr/RequestHandlers+and+SearchComponents+in+SolrConfig : I want to save the query into a file when a user is changing a parameter in : the query, lets say he adds "logTofile=1" then the searchHandler will : provide the same result as without this parameter, but in the background it : will do some logic(ex. save the query to file) . : But I dont want to touch solr source code, all I want is to add code(like : plugin). if i understand it right I want to write my own search handler , do : some logic , then pass the data to solr default search handler. -Hoss http://www.lucidworks.com/
Re: config JoinQParserPlugin
(Sorry, I clicked a wrong Mac command button and the last email was sent out
before I finished editing it..)
Earlier I tried join queries using
curl
'http://myLinux:8983/solr/abc.edu_up/select?debug=true&q=*:*&fq={defType=join
from=id to=id fromIndex=abc.edu}subject:financial'
but didn't get any response. There was nothing on Solr log either. So, I
thought I need to config join. Is there another way to at least get some
response from join queries?
Thanks,
Ray
On Monday, December 30, 2013 1:28 PM, Ray Cheng wrote:
Earlier I tried join queries using
>curl
>'http://myLinux:8983/solr/abc.edu_up/select?debug=true&q=*:*&fq={defType=join
>from=id to=id fromIndex=abc.edu}subject:financial' "curl
>http://localhost:8983/solr/... with debug=true but didn't get any response.
>There was nothing on Solr log either. So, I thought I need to config join. Is
>there another way to at least get some response from join queries?
>
>
>
>
>On Monday, December 30, 2013 12:27 PM, Yonik Seeley
>wrote:
>
>> Does it mean the following line I added to solrconfig.xml is not needed and
>> caused Solr start error?
>>
>>Yes. Join is a builtin parser and you should not add any config for it.
>>Most likely that is what is causing your startup error (although it's
>>not entirely clear why it's causing it to fail).
>>
>>
>>-Yonik
>>http://heliosearch.com -- making solr shine
>>
>>
>>
>
>
Re: config JoinQParserPlugin
Earlier I tried join queries using
curl
'http://myLinux:8983/solr/abc.edu_up/select?debug=true&q=*:*&fq={defType=join
from=id to=id fromIndex=abc.edu}subject:financial' "curl
http://localhost:8983/solr/... with debug=true but didn't get any response.
There was nothing on Solr log either. So, I thought I need to config join. Is
there another way to at least get some response from join queries?
On Monday, December 30, 2013 12:27 PM, Yonik Seeley
wrote:
> Does it mean the following line I added to solrconfig.xml is not needed and
> caused Solr start error?
>
>Yes. Join is a builtin parser and you should not add any config for it.
>Most likely that is what is causing your startup error (although it's
>not entirely clear why it's causing it to fail).
>
>
>-Yonik
>http://heliosearch.com -- making solr shine
>
>
>
Solr -The connection has timed out
I have Solr server running wit jetty. Some times i am getting connection timed out error from the home page. In the logs no errors are shown also .Please help how to resolve this problem.Attaching the log from the Solr INFO - 2013-12-26 02:51:37.460; org.eclipse.jetty.server.Server; jetty-8.1.10.v20130312 INFO - 2013-12-26 02:51:37.490; org.eclipse.jetty.deploy.providers.ScanningAppProvider; Deployment monitor /ctgapps/apache-solr-4.6.0/solr-4.6.0/example/contexts at interval 0 INFO - 2013-12-26 02:51:37.498; org.eclipse.jetty.deploy.DeploymentManager; Deployable added: /ctgapps/apache-solr-4.6.0/solr-4.6.0/example/contexts/solr-jetty-context.xml INFO - 2013-12-26 02:51:37.562; org.eclipse.jetty.webapp.WebInfConfiguration; Extract jar:file:/ctgapps/apache-solr-4.6.0/solr-4.6.0/example/webapps/solr.war!/ to /ctgapps/apache-solr-4.6.0/solr-4.6.0/example/solr-webapp/webapp INFO - 2013-12-26 02:51:39.547; org.eclipse.jetty.webapp.StandardDescriptorProcessor; NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet INFO - 2013-12-26 02:51:39.583; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() INFO - 2013-12-26 02:51:39.597; org.apache.solr.core.SolrResourceLoader; JNDI not configured for solr (NoInitialContextEx) INFO - 2013-12-26 02:51:39.597; org.apache.solr.core.SolrResourceLoader; solr home defaulted to 'solr/' (could not find system property or JNDI) INFO - 2013-12-26 02:51:39.598; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: 'solr/' INFO - 2013-12-26 02:51:39.714; org.apache.solr.core.ConfigSolr; Loading container configuration from /ctgapps/apache-solr-4.6.0/solr-4.6.0/example/solr/solr.xml INFO - 2013-12-26 02:51:40.031; org.apache.solr.core.ConfigSolrXml; Config-defined core root directory: INFO - 2013-12-26 02:51:40.041; org.apache.solr.core.CoreContainer; New CoreContainer 709424757 INFO - 2013-12-26 02:51:40.041; org.apache.solr.core.CoreContainer; Loading cores into CoreContainer [instanceDir=solr/] INFO - 2013-12-26 02:51:40.057; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting socketTimeout to: 0 INFO - 2013-12-26 02:51:40.057; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting urlScheme to: http:// INFO - 2013-12-26 02:51:40.058; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting connTimeout to: 0 INFO - 2013-12-26 02:51:40.060; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxConnectionsPerHost to: 20 INFO - 2013-12-26 02:51:40.061; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting corePoolSize to: 0 INFO - 2013-12-26 02:51:40.061; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maximumPoolSize to: 2147483647 INFO - 2013-12-26 02:51:40.061; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxThreadIdleTime to: 5 INFO - 2013-12-26 02:51:40.062; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting sizeOfQueue to: -1 INFO - 2013-12-26 02:51:40.062; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting fairnessPolicy to: false INFO - 2013-12-26 02:51:40.247; org.apache.solr.logging.LogWatcher; SLF4J impl is org.slf4j.impl.Log4jLoggerFactory -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-The-connection-has-timed-out-tp4108802.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: config JoinQParserPlugin
> Does it mean the following line I added to solrconfig.xml is not needed and > caused Solr start error? Yes. Join is a builtin parser and you should not add any config for it. Most likely that is what is causing your startup error (although it's not entirely clear why it's causing it to fail). -Yonik http://heliosearch.com -- making solr shine
Re: config JoinQParserPlugin
> The "join" parser should already be included by default - no need for configuration. Does it mean the following line I added to solrconfig.xml is not needed and caused Solr start error? Thanks, Ray On Monday, December 30, 2013 12:12 PM, Yonik Seeley wrote: On Mon, Dec 30, 2013 at 2:43 PM, Ray Cheng wrote: > >> Hi, >> >> I have trouble configuring JoinQParserPlugin in Solr. I'm using solr-4.6.0. >> >> I put this line (and only this line) to solrconfig.xml following Query >> Parsers comments of solrconfig.xml (solr-4.6.0): >> >> >> >> >/> > > >The "join" parser should already be included by default - no need for >configuration. > >-Yonik >http://heliosearch.com -- making solr shine > > > >
Re: config JoinQParserPlugin
On Mon, Dec 30, 2013 at 2:43 PM, Ray Cheng wrote: > Hi, > > I have trouble configuring JoinQParserPlugin in Solr. I'm using solr-4.6.0. > > I put this line (and only this line) to solrconfig.xml following Query > Parsers comments of solrconfig.xml (solr-4.6.0): > > > > The "join" parser should already be included by default - no need for configuration. -Yonik http://heliosearch.com -- making solr shine
Re: solr OOM Crash
Hello Sébastien, Can you give some information about your environment so I can make sure we are having the same problem you had? Also, did you find out what caused the GC to go crazy or what caused the increased commit rate? Thanks, Sandra On Thu, Dec 19, 2013 at 12:34 PM, Sébastien Michel < sebastien.mic...@atos.net> wrote: > Hi Sandra, > > I'm not sure if your problem is same as ours, but we encountered the same > issue on our Solr 4.2, the major memory usage was due to > CompressingStoredFieldsReader and GC became crazy. > In our context, we have some stored fields and for some documents the > content of the text field could be huge. > > We resolved our issue with the backport of this fix : > https://issues.apache.org/jira/browse/LUCENE-4995 > > You should also upgrade to Solr 4.4 or more > > Regards, > Sébastien > > > 2013/12/12 Sandra Scott > > > Helllo, > > > > We are experiencing unexplained OOM crashes. We have already seen it a > few > > times, over our different solr instances. The crash happens only at a > > single shard of the collection. > > > > Environment details: > > 1. Solr 4.3, running on tomcat. > > 2. 24 Shards. > > 3. Indexing rate of ~800 docs per minute. > > > > Solrconfig.xml: > > 1. Merge factor 4 > > 2. Sofrcommit every 10 min > > 3. Hardcommit every 30 min > > > > Main findings: > > 1. Solr logs: No query failures prior to the OOM, but DOUBLE the amount > of > > soft and hard commits in comparison to other shards. > > 2. Analyzing the dump (VisualVM): Class byte[] takes 4gb out of 5gb > > resourced to the JVM, mainly referenced by CompressingStoredFieldsReader > GC > > root (which by looking at the code, we suspect they were created due to > > CompressingSortedFieldsWriter.merge). > > > > Sub findings: > > 1. GC logs: Showed 108 GC fails prior to the crash. > > 2. CPI: Overall usage seems fine, but the % of CPU time for the GC stays > > high 6 min before the OOM. > > 3. Memory: Half an hour before OOM the usage slowly rises, until it gets > to > > 5.4gb. > > > > Has anyone encountered higher than normal commit rate that seem to > increase > > merge rate and cause what I described? > > >
Re: side logging requests
And a custom SearchComponent for the querying side of things: http://wiki.apache.org/solr/SearchComponent Or, you could put some logging proxy in between your client and Solr? Or put something into the web container you're using that does what you need before requests hit Solr? Several ways to get between your client and Solr. Erik On Dec 30, 2013, at 12:26 PM, elmerfudd wrote: > Hi all, > currently there are 2 things I want to accomplish. > I want that on demend Every doc (xml) that is sent to be indexed in solr > will be copied to a big log file (I want to control when to activate this > feature and when to deactivate) > same as for queries. > Also, I may need to manipulate the data before its written. > > Is there any way achieving this without changing solr sourcecode? (So it > won't be affected by updates). > > I thought of a possible way, > I posted before about making "transparent" request handler , is it possible > ? If so, how? > > > thankkk you! > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/side-logging-requests-tp4108752.html > Sent from the Solr - User mailing list archive at Nabble.com.
config JoinQParserPlugin
Hi, I have trouble configuring JoinQParserPlugin in Solr. I'm using solr-4.6.0. I put this line (and only this line) to solrconfig.xml following Query Parsers comments of solrconfig.xml (solr-4.6.0): Since I saw JoinQParserPlugin.class in solr-webapp/webapp/WEB-INF/lib/solr-core-4.6.0.jar I thought I didn't need to load this plugin explicitly in solr.xml. Is this right? I got this log when I started solr: INFO - 2013-12-29 14:30:42.382; org.apache.solr.core.SolrCore; created velocity: solr.VelocityResponseWriter INFO - 2013-12-29 14:30:42.385; org.apache.solr.core.SolrCore; created xslt: solr.XSLTResponseWriter INFO - 2013-12-29 14:30:42.385; org.apache.solr.response.XSLTResponseWriter; xsltCacheLifetimeSeconds=5 INFO - 2013-12-29 14:30:42.393; org.apache.solr.core.SolrCore; [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@618daf35 INFO - 2013-12-29 14:30:42.393; org.apache.solr.update.SolrCoreState; Closing SolrCoreState INFO - 2013-12-29 14:30:42.393; org.apache.solr.update.DefaultSolrCoreState; SolrCoreState ref count has reached 0 - closing IndexWriter INFO - 2013-12-29 14:30:42.394; org.apache.solr.core.SolrCore; [collection1] Closing main searcher on request. INFO - 2013-12-29 14:30:42.394; org.apache.solr.core.CachingDirectoryFactory; Closing NRTCachingDirectoryFactory - 2 directories currently being tracked INFO - 2013-12-29 14:30:42.394; org.apache.solr.core.CachingDirectoryFactory; looking to close /usr/local/solr-4.6.0/example/solr/collection1/data/index [CachedDir<>] INFO - 2013-12-29 14:30:42.395; org.apache.solr.core.CachingDirectoryFactory; Closing directory: /usr/local/solr-4.6.0/example/solr/collection1/data/index INFO - 2013-12-29 14:30:42.395; org.apache.solr.core.CachingDirectoryFactory; looking to close /usr/local/solr-4.6.0/example/solr/collection1/data [CachedDir<>] INFO - 2013-12-29 14:30:42.395; org.apache.solr.core.CachingDirectoryFactory; Closing directory: /usr/local/solr-4.6.0/example/solr/collection1/data ERROR - 2013-12-29 14:30:42.396; org.apache.solr.core.CoreContainer; Unable to create core: collection1 org.apache.solr.common.SolrException: java.lang.NullPointerException at org.apache.solr.core.SolrCore.(SolrCore.java:834) at org.apache.solr.core.SolrCore.(SolrCore.java:625) at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:557) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:592) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:271) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:263) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.solr.common.SolrException: java.lang.NullPointerException at org.apache.solr.core.SolrCore.initQParsers(SolrCore.java:2052) at org.apache.solr.core.SolrCore.(SolrCore.java:758) ... 13 more Caused by: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333) at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1125) at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:146) at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51) at org.apache.solr.core.SolrCore.initQParsers(SolrCore.java:2049) ... 14 more ERROR - 2013-12-29 14:30:42.397; org.apache.solr.common.SolrException; null:org.apache.solr.common.SolrException: Unable to create core: collection1 at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:977) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:601) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:271) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:263) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.solr.common.SolrException: java.lang.NullPointerException at org.apache.solr.core.SolrCore.(SolrCore
Re: Possible memory leak after segment merge? (related to DocValues?)
That was it. Setting omitNorms="true" on all fields fixed my problem. I left it indexing all weekend, and heap usage still looks great. I'm still not clear why bouncing the solr instance freed up memory, unless the in-memory structure for this norms data is lazily loaded somehow. Anyway, thank you very much for the suggestion. -Greg On Fri, Dec 27, 2013 at 4:25 AM, Michael McCandless wrote: > Likely this is for field norms, which use doc values under the hood. > > Mike McCandless > > http://blog.mikemccandless.com > > > On Thu, Dec 26, 2013 at 5:03 PM, Greg Preston > wrote: >> Does anybody with knowledge of solr internals know why I'm seeing >> instances of Lucene42DocValuesProducer when I don't have any fields >> that are using DocValues? Or am I misunderstanding what this class is >> for? >> >> -Greg >> >> >> On Mon, Dec 23, 2013 at 12:07 PM, Greg Preston >> wrote: >>> Hello, >>> >>> I'm loading up our solr cloud with data (from a solrj client) and >>> running into a weird memory issue. I can reliably reproduce the >>> problem. >>> >>> - Using Solr Cloud 4.4.0 (also replicated with 4.6.0) >>> - 24 solr nodes (one shard each), spread across 3 physical hosts, each >>> host has 256G of memory >>> - index and tlogs on ssd >>> - Xmx=7G, G1GC >>> - Java 1.7.0_25 >>> - schema and solrconfig.xml attached >>> >>> I'm using composite routing to route documents with the same clientId >>> to the same shard. After several hours of indexing, I occasionally >>> see an IndexWriter go OOM. I think that's a symptom. When that >>> happens, indexing continues, and that node's tlog starts to grow. >>> When I notice this, I stop indexing, and bounce the problem node. >>> That's where it gets interesting. >>> >>> Upon bouncing, the tlog replays, and then segments merge. Once the >>> merging is complete, the heap is fairly full, and forced full GC only >>> helps a little. But if I then bounce the node again, the heap usage >>> goes way down, and stays low until the next segment merge. I believe >>> segment merges are also what causes the original OOM. >>> >>> More details: >>> >>> Index on disk for this node is ~13G, tlog is ~2.5G. >>> See attached mem1.png. This is a jconsole view of the heap during the >>> following: >>> >>> (Solr cloud node started at the left edge of this graph) >>> >>> A) One CPU core pegged at 100%. Thread dump shows: >>> "Lucene Merge Thread #0" daemon prio=10 tid=0x7f5a3c064800 >>> nid=0x7a74 runnable [0x7f5a41c5f000] >>>java.lang.Thread.State: RUNNABLE >>> at org.apache.lucene.util.fst.Builder.add(Builder.java:397) >>> at >>> org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter.finishTerm(BlockTreeTermsWriter.java:1000) >>> at >>> org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:112) >>> at >>> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72) >>> at >>> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365) >>> at >>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98) >>> at >>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3772) >>> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3376) >>> at >>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405) >>> at >>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482) >>> >>> B) One CPU core pegged at 100%. Manually triggered GC. Lots of >>> memory freed. Thread dump shows: >>> "Lucene Merge Thread #0" daemon prio=10 tid=0x7f5a3c064800 >>> nid=0x7a74 runnable [0x7f5a41c5f000] >>>java.lang.Thread.State: RUNNABLE >>> at >>> org.apache.lucene.codecs.DocValuesConsumer$1$1.hasNext(DocValuesConsumer.java:127) >>> at >>> org.apache.lucene.codecs.lucene42.Lucene42DocValuesConsumer.addNumericField(Lucene42DocValuesConsumer.java:144) >>> at >>> org.apache.lucene.codecs.lucene42.Lucene42DocValuesConsumer.addNumericField(Lucene42DocValuesConsumer.java:92) >>> at >>> org.apache.lucene.codecs.DocValuesConsumer.mergeNumericField(DocValuesConsumer.java:112) >>> at >>> org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:221) >>> at >>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119) >>> at >>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3772) >>> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3376) >>> at >>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405) >>> at >>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482) >>> >>> C) One CPU core pegged at 100%. Manually triggered GC. No memory >>> freed. Thread dump shows: >>> "Lucene Merge Thread #0" daemon prio=10 tid=0x7f5a3c064800 >>> nid=0x7a74 r
Re: adding wild card at the end of the text and search(like sql like search)
Ahmet, I am using solr 4.3.1. do i still need to apply this patch ? if yes please tell me the steps to follow. In the given link i see lot of patches, not sure which patch for what version of solr also i don't see the patch note how to apply. Thanks, Suren. -- View this message in context: http://lucene.472066.n3.nabble.com/adding-wild-card-at-the-end-of-the-text-and-search-like-sql-like-search-tp4108399p4108765.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: side logging requests
You can wire-in a custom UpdateRequestProcessor - http://wiki.apache.org/solr/UpdateRequestProcessor Timothy Potter Sr. Software Engineer, LucidWorks www.lucidworks.com From: elmerfudd Sent: Monday, December 30, 2013 10:26 AM To: solr-user@lucene.apache.org Subject: side logging requests Hi all, currently there are 2 things I want to accomplish. I want that on demend Every doc (xml) that is sent to be indexed in solr will be copied to a big log file (I want to control when to activate this feature and when to deactivate) same as for queries. Also, I may need to manipulate the data before its written. Is there any way achieving this without changing solr sourcecode? (So it won't be affected by updates). I thought of a possible way, I posted before about making "transparent" request handler , is it possible ? If so, how? thankkk you! -- View this message in context: http://lucene.472066.n3.nabble.com/side-logging-requests-tp4108752.html Sent from the Solr - User mailing list archive at Nabble.com.
side logging requests
Hi all, currently there are 2 things I want to accomplish. I want that on demend Every doc (xml) that is sent to be indexed in solr will be copied to a big log file (I want to control when to activate this feature and when to deactivate) same as for queries. Also, I may need to manipulate the data before its written. Is there any way achieving this without changing solr sourcecode? (So it won't be affected by updates). I thought of a possible way, I posted before about making "transparent" request handler , is it possible ? If so, how? thankkk you! -- View this message in context: http://lucene.472066.n3.nabble.com/side-logging-requests-tp4108752.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: monitoring solr logs
I've just been using the Solr query form so far :P but have plans to try out
Kibana too. Let me know how that goes for you and I'll do the same.
From: adfel70
Sent: Monday, December 30, 2013 10:06 AM
To: solr-user@lucene.apache.org
Subject: RE: monitoring solr logs
And are you using any tool like kibana as a dashboard for the logs?
Tim Potter wrote
> We're (LucidWorks) are actively developing on logstash4solr so if you have
> issues, let us know. So far, so good for me but I upgraded to logstash
> 1.3.2 even though the logstash4solr version includes 1.2.2 you can use the
> newer one. I'm not quite in production with my logstash4solr <- rabbit-mq
> <- log4j <- Solr solution yet though ;-)
>
> Yeah, 50GB is too much logging for only 150K docs. Maybe start by
> filtering by log level (WARN and more severe). If a server crashes, you're
> likely to see some errors in the logstash side but sometimes you may have
> to SSH to the specific box and look at the local log (so definitely append
> all messages to the local Solr log too), I'm using something like the
> following for local logging:
>
> log4j.rootLogger=INFO, file
> log4j.appender.file=org.apache.log4j.RollingFileAppender
> log4j.appender.file.MaxFileSize=50MB
> log4j.appender.file.MaxBackupIndex=10
> log4j.appender.file.File=logs/solr.log
> log4j.appender.file.layout=org.apache.log4j.PatternLayout
> log4j.appender.file.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{3}
> %x - %m%n
>
>
> Timothy Potter
> Sr. Software Engineer, LucidWorks
> www.lucidworks.com
>
>
> From: adfel70 <
> adfel70@
> >
> Sent: Monday, December 30, 2013 9:34 AM
> To:
> solr-user@.apache
> Subject: RE: monitoring solr logs
>
> Actually I was considering using logstash4solr, but it didn't seem mature
> enough.
> does it work fine? any known bugs?
>
> are you collecting the logs in the same solr cluster you use for the
> production systems?
> if so, what will you do if for some reason solr is down and you would like
> to analyze the logs to see what happend?
>
> btw, i started a new solr cluster with 7 shards, replicationfactor=3 and
> run
> indexing job of 400K docs,
> it got stuck on 150K because I used Socketappender directly to write to
> logstash and logstash disk got full.
>
> that's why I moved to using AsyncAppender, and I plan on moving to using
> rabbit.
> but this is also why I wanted to filter some of the logs. indexing 150K
> docs
> prodcued 50GB of logs.
> this seemed too much.
>
>
>
>
> Tim Potter wrote
>> I'm using logstash4solr (http://logstash4solr.org) for something similar
>> ...
>>
>> I setup my Solr to use Log4J by passing the following on the command-line
>> when starting Solr:
>> -Dlog4j.configuration=file:///$SCRIPT_DIR/log4j.properties
>>
>> Then I use a custom Log4J appender that writes to RabbitMQ:
>>
>> https://github.com/plant42/rabbitmq-log4j-appender
>>
>> You can then configure a RabbitMQ input for logstash -
>> http://logstash.net/docs/1.3.2/inputs/rabbitmq
>>
>> This decouples the log writes from log indexing in logstash4solr, which
>> scales better for active Solr installations.
>>
>> Btw ... I just log everything from Solr using this approach but you can
>> use standard Log4J configuration settings to limit which classes / log
>> levels to send to the RabbitMQ appender.
>>
>> Cheers,
>>
>> Timothy Potter
>> Sr. Software Engineer, LucidWorks
>> www.lucidworks.com
>>
>>
>> From: adfel70 <
>
>> adfel70@
>
>> >
>> Sent: Monday, December 30, 2013 8:15 AM
>> To:
>
>> solr-user@.apache
>
>> Subject: monitoring solr logs
>>
>> hi
>> i'm trying to figure out which solr and zookeeper logs i should monitor
>> and
>> collect.
>> All the logs will be written to a file but I want to collect some of them
>> with logstash in order to be able to analyze them efficiently.
>> any inputs on logs of which classes i should collect?
>>
>> thanks.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721p4108737.html
> Sent from the Solr - User mailing list archive at Nabble.com.
--
View this message in context:
http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721p4108744.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: monitoring solr logs
And are you using any tool like kibana as a dashboard for the logs?
Tim Potter wrote
> We're (LucidWorks) are actively developing on logstash4solr so if you have
> issues, let us know. So far, so good for me but I upgraded to logstash
> 1.3.2 even though the logstash4solr version includes 1.2.2 you can use the
> newer one. I'm not quite in production with my logstash4solr <- rabbit-mq
> <- log4j <- Solr solution yet though ;-)
>
> Yeah, 50GB is too much logging for only 150K docs. Maybe start by
> filtering by log level (WARN and more severe). If a server crashes, you're
> likely to see some errors in the logstash side but sometimes you may have
> to SSH to the specific box and look at the local log (so definitely append
> all messages to the local Solr log too), I'm using something like the
> following for local logging:
>
> log4j.rootLogger=INFO, file
> log4j.appender.file=org.apache.log4j.RollingFileAppender
> log4j.appender.file.MaxFileSize=50MB
> log4j.appender.file.MaxBackupIndex=10
> log4j.appender.file.File=logs/solr.log
> log4j.appender.file.layout=org.apache.log4j.PatternLayout
> log4j.appender.file.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{3}
> %x - %m%n
>
>
> Timothy Potter
> Sr. Software Engineer, LucidWorks
> www.lucidworks.com
>
>
> From: adfel70 <
> adfel70@
> >
> Sent: Monday, December 30, 2013 9:34 AM
> To:
> solr-user@.apache
> Subject: RE: monitoring solr logs
>
> Actually I was considering using logstash4solr, but it didn't seem mature
> enough.
> does it work fine? any known bugs?
>
> are you collecting the logs in the same solr cluster you use for the
> production systems?
> if so, what will you do if for some reason solr is down and you would like
> to analyze the logs to see what happend?
>
> btw, i started a new solr cluster with 7 shards, replicationfactor=3 and
> run
> indexing job of 400K docs,
> it got stuck on 150K because I used Socketappender directly to write to
> logstash and logstash disk got full.
>
> that's why I moved to using AsyncAppender, and I plan on moving to using
> rabbit.
> but this is also why I wanted to filter some of the logs. indexing 150K
> docs
> prodcued 50GB of logs.
> this seemed too much.
>
>
>
>
> Tim Potter wrote
>> I'm using logstash4solr (http://logstash4solr.org) for something similar
>> ...
>>
>> I setup my Solr to use Log4J by passing the following on the command-line
>> when starting Solr:
>> -Dlog4j.configuration=file:///$SCRIPT_DIR/log4j.properties
>>
>> Then I use a custom Log4J appender that writes to RabbitMQ:
>>
>> https://github.com/plant42/rabbitmq-log4j-appender
>>
>> You can then configure a RabbitMQ input for logstash -
>> http://logstash.net/docs/1.3.2/inputs/rabbitmq
>>
>> This decouples the log writes from log indexing in logstash4solr, which
>> scales better for active Solr installations.
>>
>> Btw ... I just log everything from Solr using this approach but you can
>> use standard Log4J configuration settings to limit which classes / log
>> levels to send to the RabbitMQ appender.
>>
>> Cheers,
>>
>> Timothy Potter
>> Sr. Software Engineer, LucidWorks
>> www.lucidworks.com
>>
>>
>> From: adfel70 <
>
>> adfel70@
>
>> >
>> Sent: Monday, December 30, 2013 8:15 AM
>> To:
>
>> solr-user@.apache
>
>> Subject: monitoring solr logs
>>
>> hi
>> i'm trying to figure out which solr and zookeeper logs i should monitor
>> and
>> collect.
>> All the logs will be written to a file but I want to collect some of them
>> with logstash in order to be able to analyze them efficiently.
>> any inputs on logs of which classes i should collect?
>>
>> thanks.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721p4108737.html
> Sent from the Solr - User mailing list archive at Nabble.com.
--
View this message in context:
http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721p4108744.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: monitoring solr logs
We're (LucidWorks) are actively developing on logstash4solr so if you have
issues, let us know. So far, so good for me but I upgraded to logstash 1.3.2
even though the logstash4solr version includes 1.2.2 you can use the newer one.
I'm not quite in production with my logstash4solr <- rabbit-mq <- log4j <- Solr
solution yet though ;-)
Yeah, 50GB is too much logging for only 150K docs. Maybe start by filtering by
log level (WARN and more severe). If a server crashes, you're likely to see
some errors in the logstash side but sometimes you may have to SSH to the
specific box and look at the local log (so definitely append all messages to
the local Solr log too), I'm using something like the following for local
logging:
log4j.rootLogger=INFO, file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.MaxFileSize=50MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.File=logs/solr.log
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{3} %x -
%m%n
Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com
From: adfel70
Sent: Monday, December 30, 2013 9:34 AM
To: solr-user@lucene.apache.org
Subject: RE: monitoring solr logs
Actually I was considering using logstash4solr, but it didn't seem mature
enough.
does it work fine? any known bugs?
are you collecting the logs in the same solr cluster you use for the
production systems?
if so, what will you do if for some reason solr is down and you would like
to analyze the logs to see what happend?
btw, i started a new solr cluster with 7 shards, replicationfactor=3 and run
indexing job of 400K docs,
it got stuck on 150K because I used Socketappender directly to write to
logstash and logstash disk got full.
that's why I moved to using AsyncAppender, and I plan on moving to using
rabbit.
but this is also why I wanted to filter some of the logs. indexing 150K docs
prodcued 50GB of logs.
this seemed too much.
Tim Potter wrote
> I'm using logstash4solr (http://logstash4solr.org) for something similar
> ...
>
> I setup my Solr to use Log4J by passing the following on the command-line
> when starting Solr:
> -Dlog4j.configuration=file:///$SCRIPT_DIR/log4j.properties
>
> Then I use a custom Log4J appender that writes to RabbitMQ:
>
> https://github.com/plant42/rabbitmq-log4j-appender
>
> You can then configure a RabbitMQ input for logstash -
> http://logstash.net/docs/1.3.2/inputs/rabbitmq
>
> This decouples the log writes from log indexing in logstash4solr, which
> scales better for active Solr installations.
>
> Btw ... I just log everything from Solr using this approach but you can
> use standard Log4J configuration settings to limit which classes / log
> levels to send to the RabbitMQ appender.
>
> Cheers,
>
> Timothy Potter
> Sr. Software Engineer, LucidWorks
> www.lucidworks.com
>
>
> From: adfel70 <
> adfel70@
> >
> Sent: Monday, December 30, 2013 8:15 AM
> To:
> solr-user@.apache
> Subject: monitoring solr logs
>
> hi
> i'm trying to figure out which solr and zookeeper logs i should monitor
> and
> collect.
> All the logs will be written to a file but I want to collect some of them
> with logstash in order to be able to analyze them efficiently.
> any inputs on logs of which classes i should collect?
>
> thanks.
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721.html
> Sent from the Solr - User mailing list archive at Nabble.com.
--
View this message in context:
http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721p4108737.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: monitoring solr logs
Actually I was considering using logstash4solr, but it didn't seem mature enough. does it work fine? any known bugs? are you collecting the logs in the same solr cluster you use for the production systems? if so, what will you do if for some reason solr is down and you would like to analyze the logs to see what happend? btw, i started a new solr cluster with 7 shards, replicationfactor=3 and run indexing job of 400K docs, it got stuck on 150K because I used Socketappender directly to write to logstash and logstash disk got full. that's why I moved to using AsyncAppender, and I plan on moving to using rabbit. but this is also why I wanted to filter some of the logs. indexing 150K docs prodcued 50GB of logs. this seemed too much. Tim Potter wrote > I'm using logstash4solr (http://logstash4solr.org) for something similar > ... > > I setup my Solr to use Log4J by passing the following on the command-line > when starting Solr: > -Dlog4j.configuration=file:///$SCRIPT_DIR/log4j.properties > > Then I use a custom Log4J appender that writes to RabbitMQ: > > https://github.com/plant42/rabbitmq-log4j-appender > > You can then configure a RabbitMQ input for logstash - > http://logstash.net/docs/1.3.2/inputs/rabbitmq > > This decouples the log writes from log indexing in logstash4solr, which > scales better for active Solr installations. > > Btw ... I just log everything from Solr using this approach but you can > use standard Log4J configuration settings to limit which classes / log > levels to send to the RabbitMQ appender. > > Cheers, > > Timothy Potter > Sr. Software Engineer, LucidWorks > www.lucidworks.com > > > From: adfel70 < > adfel70@ > > > Sent: Monday, December 30, 2013 8:15 AM > To: > solr-user@.apache > Subject: monitoring solr logs > > hi > i'm trying to figure out which solr and zookeeper logs i should monitor > and > collect. > All the logs will be written to a file but I want to collect some of them > with logstash in order to be able to analyze them efficiently. > any inputs on logs of which classes i should collect? > > thanks. > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721.html > Sent from the Solr - User mailing list archive at Nabble.com. -- View this message in context: http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721p4108737.html Sent from the Solr - User mailing list archive at Nabble.com.
Solr search videos
Hi everyone. I am very new at Solr.. i'm only starting, and i dont know if this is possible to do. So let's say that i want to do a video search engine. To do this i was thinking to use nutch for crawl video websites and then create the index files. But then i thought.. How do I know that nutch will crawl enough to solr return the info about the video i searched? Is this possible? this might sound a little confuse, but i hope you can help me and if this cant be done, that you can tell me what to do. Thanks. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-search-videos-tp4108731.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: monitoring solr logs
I'm using logstash4solr (http://logstash4solr.org) for something similar ... I setup my Solr to use Log4J by passing the following on the command-line when starting Solr: -Dlog4j.configuration=file:///$SCRIPT_DIR/log4j.properties Then I use a custom Log4J appender that writes to RabbitMQ: https://github.com/plant42/rabbitmq-log4j-appender You can then configure a RabbitMQ input for logstash - http://logstash.net/docs/1.3.2/inputs/rabbitmq This decouples the log writes from log indexing in logstash4solr, which scales better for active Solr installations. Btw ... I just log everything from Solr using this approach but you can use standard Log4J configuration settings to limit which classes / log levels to send to the RabbitMQ appender. Cheers, Timothy Potter Sr. Software Engineer, LucidWorks www.lucidworks.com From: adfel70 Sent: Monday, December 30, 2013 8:15 AM To: solr-user@lucene.apache.org Subject: monitoring solr logs hi i'm trying to figure out which solr and zookeeper logs i should monitor and collect. All the logs will be written to a file but I want to collect some of them with logstash in order to be able to analyze them efficiently. any inputs on logs of which classes i should collect? thanks. -- View this message in context: http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721.html Sent from the Solr - User mailing list archive at Nabble.com.
monitoring solr logs
hi i'm trying to figure out which solr and zookeeper logs i should monitor and collect. All the logs will be written to a file but I want to collect some of them with logstash in order to be able to analyze them efficiently. any inputs on logs of which classes i should collect? thanks. -- View this message in context: http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr core is asking tomcat username and password
On 30 December 2013 18:59, kumar wrote: > Hi, > > > I Created core in solr using tomcat and when i am accessing created core it > is asking tomcat authentication details. Hi, This is off-topic: If you are going to run Solr within Tomcat, please do some basic homework, or ask on a Tomcat list. Tomcat authentication is usually in tomcat-users.xml, but where that file is depends on the OS and Tomcat version. On Debian/Ubuntu, and probably other Linux distributions, it is in /etc/tomcatX where X is your Tomcat version, e.g., /etc/tomcat7 Regards, Gora
Solr core is asking tomcat username and password
Hi, I Created core in solr using tomcat and when i am accessing created core it is asking tomcat authentication details. For example i am using following url. http://localhost:8080/solr/corename/ypcity/suggest?json&&s=kumar if i pass this to ajax, it is asking tomcat username and password. How can i remove tomcat authentication details. Anybody help me how can i achieve this. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-core-is-asking-tomcat-username-and-password-tp4108695.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to boost documents ?
Hi,
Thank you for your response.
When I try the URL you sent me, I get the following error message :
org.apache.solr.search.SyntaxError: Infinite Recursion detected parsing query
'beautiful Christmas tree'
Any idea what this means ?
Best regards,
Anca
On 12/16/2013 02:00 PM, Ahmet Arslan wrote:
Hi Anca,
Can you try following URL?
q=beautiful Christmas tree&mm=2&qf=title^12
description^2&defType=dismax&bf=map(query($qq),0,0,0,100.0)&qq={!dismax qf='title
description' mm=100%}beautiful Christmas tree
Modified from Jan's solution. See his original post [1] to a similar discussion.
[1] http://search-lucene.com/m/nK6t9j1fuc2
On Monday, December 16, 2013 12:19 PM, Anca Kopetz
wrote:
Hi,
How to boost documents that contain all search terms in several of its fields ?
Below you cand find a simplified example :
The query with Min should match:
q=beautiful Christmas tree&mm=2&qf=title^12 description^2
There are two offers that match the query :
offer1 {title:"Christmas tree", description:"a joy for children"}
offer2 {title:"Christmas tree", description:"beautiful for holidays"}}
The first offer ranks before the second, despite of the fact that the second
one contains all the search terms. I tried to play with the boosts of qf, but
the results vary a lot.
Is there a way to add a boost on all search fields, the same way we do with pf
on one field : pf=title:2^3.0 ?
Thank you,
Anca
Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris
Ce message et les pièces jointes sont confidentiels et établis à l'attention
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce
message, merci de le détruire et d'en avertir l'expéditeur.
Re: org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query
This is all I see in the XML response: This XML file does not appear to have any style information associated with it. The document tree is shown below. 0 39 wordpress-data-config.xml full-import debug idle 0:0:0.57 1 0 0 0 2013-12-30 12:21:49 Indexing failed. Rolled back all changes. 2013-12-30 12:21:49 This response format is experimental. It is likely to change in the future. -- View this message in context: http://lucene.472066.n3.nabble.com/org-apache-solr-handler-dataimport-DataImportHandlerException-Unable-to-execute-query-tp4108227p4108682.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query
I'm sorry I meant that you should see the XML response returned in debug mode. The debug mode won't log anything extra but it will collect information such as the SQL queries executed per entity and per row. It will also show a stacktrace if there was an error. On Mon, Dec 30, 2013 at 3:58 PM, PeterKerk wrote: > I ran the query in debug mode: > http://localhost:8983/solr/tv-wordpress/dataimport?command=full-import&debug=true > > Here's the output, what can you tell from this? > > 22432 [qtp33142123-13] INFO > org.apache.solr.handler.dataimport.JdbcDataSource > û Creating a connection for entity article with URL: > jdbc:mysql@localhost:3306/w > ptalkman > 22435 [qtp33142123-13] INFO > org.apache.solr.handler.dataimport.JdbcDataSource > û Time taken for getConnection(): 3 > 22436 [qtp33142123-13] ERROR org.apache.solr.handler.dataimport.DocBuilder > û Ex > ception while processing: article document : > SolrInputDocument[]:org.apache.solr > .handler.dataimport.DataImportHandlerException: Unable to execute query: > SELECT > * FROM wptalkman.wp_posts WHERE post_status='publish' Processing Document # > 1 > at > org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAnd > Throw(DataImportHandlerException.java:71) > at > org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.< > init>(JdbcDataSource.java:253) > at > org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSou > rce.java:210) > at > org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSou > rce.java:38) > at > org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEn > tityProcessor.java:59) > at > org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEnti > tyProcessor.java:73) > at > org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(Ent > ityProcessorWrapper.java:243) > at > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde > r.java:465) > at > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde > r.java:404) > at > org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.j > ava:319) > at > org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java > :227) > at > org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImpo > rter.java:422) > at > org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.j > ava:487) > at > org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBod > y(DataImportHandler.java:179) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandl > erBase.java:135) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1820) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter > .java:656) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilte > r.java:359) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilte > r.java:155) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet > Handler.java:1307) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java > :453) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j > ava:137) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.jav > a:560) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandl > er.java:231) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandl > er.java:1072) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java: > 382) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandle > r.java:193) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandle > r.java:1006) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j > ava:135) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Cont > extHandlerCollection.java:255) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerColl > ection.java:154) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper > .java:116) > at org.eclipse.jetty.server.Server.handle(Server.java:365) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(Abstrac > tHttpConnection.java:485) > at > org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(Blockin > gHttpConnection.java:53) > at > org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(Abstra > ctHttpConnection.java:926) > at > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.header > Complete(AbstractHttpConnection.java:988) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) > at
Re: how to debug dataimporthandler
Tried your steps, but failed. Could you perhaps have a look at my post here: http://lucene.472066.n3.nabble.com/org-apache-solr-handler-dataimport-DataImportHandlerException-Unable-to-execute-query-td4108227.html -- View this message in context: http://lucene.472066.n3.nabble.com/how-to-debug-dataimporthandler-tp2611506p4108676.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query
I ran the query in debug mode: http://localhost:8983/solr/tv-wordpress/dataimport?command=full-import&debug=true Here's the output, what can you tell from this? 22432 [qtp33142123-13] INFO org.apache.solr.handler.dataimport.JdbcDataSource û Creating a connection for entity article with URL: jdbc:mysql@localhost:3306/w ptalkman 22435 [qtp33142123-13] INFO org.apache.solr.handler.dataimport.JdbcDataSource û Time taken for getConnection(): 3 22436 [qtp33142123-13] ERROR org.apache.solr.handler.dataimport.DocBuilder û Ex ception while processing: article document : SolrInputDocument[]:org.apache.solr .handler.dataimport.DataImportHandlerException: Unable to execute query: SELECT * FROM wptalkman.wp_posts WHERE post_status='publish' Processing Document # 1 at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAnd Throw(DataImportHandlerException.java:71) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.< init>(JdbcDataSource.java:253) at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSou rce.java:210) at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSou rce.java:38) at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEn tityProcessor.java:59) at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEnti tyProcessor.java:73) at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(Ent ityProcessorWrapper.java:243) at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde r.java:465) at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde r.java:404) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.j ava:319) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java :227) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImpo rter.java:422) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.j ava:487) at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBod y(DataImportHandler.java:179) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandl erBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1820) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter .java:656) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilte r.java:359) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilte r.java:155) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet Handler.java:1307) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java :453) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j ava:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.jav a:560) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandl er.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandl er.java:1072) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java: 382) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandle r.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandle r.java:1006) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j ava:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Cont extHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerColl ection.java:154) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper .java:116) at org.eclipse.jetty.server.Server.handle(Server.java:365) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(Abstrac tHttpConnection.java:485) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(Blockin gHttpConnection.java:53) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(Abstra ctHttpConnection.java:926) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.header Complete(AbstractHttpConnection.java:988) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpCo nnection.java:72) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(So cketConnector.java:264) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPoo l.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool .java:543) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.NullPointerException at org.apac
Re: Fetching results with a minimum from each category
Oussama Jilal, b and d have 4 products in top 20 results obtained and not overall in the database. Grouping would give me results from all categories, I only want categories corresponding to top 20 results. It can be done in 2 solr queries, but it would make the system slow. Was just wondering if there is a simpler way of doing this. Thanks -- View this message in context: http://lucene.472066.n3.nabble.com/Fetching-results-with-a-minimum-from-each-category-tp4108659p4108671.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Fetching results with a minimum from each category
I don't get it, if b and d only have 4 products, how can you fetch one more for them? But any way, if you realy know what you are doing, then I can say that this is quite specific, so you may have to do it in 2 solr queries. also see the Grouping Componant, it may help you get what you want (grouping by the category field) http://wiki.apache.org/solr/FieldCollapsing 2013/12/30 nish > I am using solr 4.4.0. The search is performed on products, each of which > has > a category field. I want to retrieve top n products. But, if some category > has less than m products among the top n, then I want to retrieve more > products only for those categories. > > Eg. I have 4 categories a, b, c, d. n=20 and m=5. Now lets say the top > 20(=n) have following category distribution (a:6, b:4, c:6, d:4). > Categories > b and d have less than m(=5) products. So I would like to fetch one more > product(with the next highest score) for both these categories. > > Is there a way I can do this using solr > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Fetching-results-with-a-minimum-from-each-category-tp4108659.html > Sent from the Solr - User mailing list archive at Nabble.com. >
RE: How to use Solr in my project
I think we may have up to 100,000 books, but I don't think the site will have a lot of traffic. Thank you for your help. I think it is a little more clear and will try to implement it now. > -Original Message- > From: Gora Mohanty [mailto:g...@mimirtech.com] > Sent: Monday, December 30, 2013 11:46 AM > To: solr-user@lucene.apache.org > Subject: Re: How to use Solr in my project > > On 30 December 2013 11:27, Fatima Issawi wrote: > > Hi again, > > > > We have another program that will be extracting the text, and it will be > extracting the top right and bottom left corners of the words. You are right, > I > do expect to have a lot of data. > > > > When would solr start experiencing issues in performance? Is it better to: > > > > INDEX: > > - document metadata > > - words > > > > STORE: > > - document metadata > > - words > > - coordinates > > > > in Solr rather than in the database? How would I set up the schema in order > to store the coordinates? > > You do not mention the number of documents, but for a few tens of > thousands of documents, your problem should be tractable in Solr. Not sure > what document metadata you have, and if you need to search through it, but > what I would do is index the words, and store the coordinates in Solr, the > assumption being that words are searched but not retrieved from Solr, while > coordinates are retrieved but never searched. > > Off the top of my head, each record can be: > > > ... > ... > ... > ... > > * and from Solr search results let you retrieve the image > from the filesystem > * The coordinates allow post-processing to highlight the word in the image > > As always, set up a prototype system with a subset of the records in order to > measure performance. > > > If storing the coordinates in solr is not recommended, what would be the > best process to get the coordinates after indexing the words and metadata? > Do I search in solr and then use the documentID to then search the database > for the words and coordinates? > > You could do that, but Solr by itself should be fine. > > Regards, > Gora
Re: How to use Solr in my project
On 30 December 2013 11:27, Fatima Issawi wrote: > Hi again, > > We have another program that will be extracting the text, and it will be > extracting the top right and bottom left corners of the words. You are right, > I do expect to have a lot of data. > > When would solr start experiencing issues in performance? Is it better to: > > INDEX: > - document metadata > - words > > STORE: > - document metadata > - words > - coordinates > > in Solr rather than in the database? How would I set up the schema in order > to store the coordinates? You do not mention the number of documents, but for a few tens of thousands of documents, your problem should be tractable in Solr. Not sure what document metadata you have, and if you need to search through it, but what I would do is index the words, and store the coordinates in Solr, the assumption being that words are searched but not retrieved from Solr, while coordinates are retrieved but never searched. Off the top of my head, each record can be: ... ... ... ... * and from Solr search results let you retrieve the image from the filesystem * The coordinates allow post-processing to highlight the word in the image As always, set up a prototype system with a subset of the records in order to measure performance. > If storing the coordinates in solr is not recommended, what would be the best > process to get the coordinates after indexing the words and metadata? Do I > search in solr and then use the documentID to then search the database for > the words and coordinates? You could do that, but Solr by itself should be fine. Regards, Gora
Fetching results with a minimum from each category
I am using solr 4.4.0. The search is performed on products, each of which has a category field. I want to retrieve top n products. But, if some category has less than m products among the top n, then I want to retrieve more products only for those categories. Eg. I have 4 categories a, b, c, d. n=20 and m=5. Now lets say the top 20(=n) have following category distribution (a:6, b:4, c:6, d:4). Categories b and d have less than m(=5) products. So I would like to fetch one more product(with the next highest score) for both these categories. Is there a way I can do this using solr -- View this message in context: http://lucene.472066.n3.nabble.com/Fetching-results-with-a-minimum-from-each-category-tp4108659.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Chaining plugins
I will make it clear. I want to save the query into a file when a user is changing a parameter in the query, lets say he adds "logTofile=1" then the searchHandler will provide the same result as without this parameter, but in the background it will do some logic(ex. save the query to file) . But I dont want to touch solr source code, all I want is to add code(like plugin). if i understand it right I want to write my own search handler , do some logic , then pass the data to solr default search handler. how do i accomplish this? -- View this message in context: http://lucene.472066.n3.nabble.com/Chaining-plugins-tp4108239p4108662.html Sent from the Solr - User mailing list archive at Nabble.com.
