Intermittent connection timeouts to Solr server using SolrNet
Hi - I have also posted this question on SO:
http://stackoverflow.com/questions/8741080/intermittent-connection-timeouts-to-solr-server-using-solrnet
I have a production webserver hosting a search webpage, which uses SolrNet
to connect to another machine which hosts the Solr search server (on a
subnet which is in the same room, so no network problems). All is fine 90%
of the time, but I consistently get a small number of The operation has
timed out errors.
I've increased the timeout in the SolrNet init to *30* seconds (!)
SolrNet.Startup.InitSolrDataObject(
new SolrNet.Impl.SolrConnection(
System.Configuration.ConfigurationManager.AppSettings[URL]
) {Timeout = 3}
);
...but all that happened is I started getting this message instead of Unable
to connect to the remote server which I was seeing before. It seems to have
made no difference to the amount of timeout errors.
I can see *nothing* in *any* log (believe me I've looked!) and clearly my
configuration is correct because it works most of the time. Anyone any
ideas how I can find more information on this problem?
Thanks!
--
Ian
i...@isfluent.com a...@endissolutions.com
+44 (0)1223 257903
Re: Unable to index documents using DataImportHandler with MSSQL
Right.
This is REALLY weird - I've now started from scratch on another
machine (this time Windows 7), and got _exactly_ the same problem !?
On Mon, Nov 28, 2011 at 7:37 AM, Husain, Yavar yhus...@firstam.com wrote:
Hi Ian
I am having exactly the same problem what you are having on Win 7 and 2008
Server http://lucene.472066.n3.nabble.com/DIH-Strange-Problem-tc3530370.html
I still have not received any replies which could solve my problem till now.
Please do let me know if you have arrived at some solution for your problem.
Thanks.
Regards,
Yavar
-Original Message-
From: Ian Grainger [mailto:i...@isfluent.com]
Sent: Friday, November 25, 2011 10:59 PM
To: solr-user@lucene.apache.org
Subject: Re: Unable to index documents using DataImportHandler with MSSQL
Update on this: I've established:
* It's not a problem in the DB (I can index from this DB into a Solr
instance on another server)
* It's not Tomcat (I get the same problem in Jetty)
* It's not the schema (I have simplified it to one field)
That leaves SolrConfig.xml and data-config.
Only thing changed in SolrConfig.xml is adding:
lib dir=D:/Software/Solr/example/solr/dist/
regex=apache-solr-cell-\d.*\.jar /
lib dir=D:/Software/Solr/example/solr/dist/
regex=apache-solr-clustering-\d.*\.jar /
lib dir=D:/Software/Solr/example/solr/dist/
regex=apache-solr-dataimporthandler-\d.*\.jar /
requestHandler name=/dataimport
class=org.apache.solr.handler.dataimport.DataImportHandler
lst name=defaults
str name=configD:/Software/Solr/example/solr/conf/data-config.xml/str
/lst
/requestHandler
And data-config.xml is pretty much as attached - except simpler.
Any help or any advice on how to diagnose would be appreciated!
On Fri, Nov 25, 2011 at 12:29 PM, Ian Grainger i...@isfluent.com wrote:
Hi I have copied my Solr config from a working Windows server to a new
one, and it can't seem to run an import.
They're both using win server 2008 and SQL 2008R2. This is the data
importer config
dataConfig
dataSource type=JdbcDataSource name=ds1
driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
url=jdbc:sqlserver://localhost;databaseName=DB
user=Solr
password=pwd/
document name=datas
entity name=data dataSource=ds1 pk=key
query=EXEC SOLR_COMPANY_SEARCH_DATA
deltaImportQuery=SELECT * FROM Company_Search_Data WHERE
[key]='${dataimporter.delta.key}'
deltaQuery=SELECT [key] FROM Company_Search_Data WHERE modify_dt
'${dataimporter.last_index_time}'
field column=WorkDesc_Comments
name=WorkDesc_Comments_Split /
field column=WorkDesc_Comments name=WorkDesc_Comments_Edge
/
/entity
/document
/dataConfig
I can use MS SQL Profiler to watch the Solr user log in successfully,
but then nothing. It doesn't seem to even try and execute the stored
procedure. Any ideas why this would be working one server and not on
another?
FTR the only thing in the tomcat catalina log is:
org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO: Creating a connection for entity data with URL:
jdbc:sqlserver://localhost;databaseName=CATLive
--
Ian
i...@isfluent.com
+44 (0)1223 257903
--
Ian
i...@isfluent.com
+44 (0)1223 257903
**
This message may contain confidential or proprietary information intended
only for the use of the
addressee(s) named above or may contain information that is legally
privileged. If you are
not the intended addressee, or the person responsible for delivering it to
the intended addressee,
you are hereby notified that reading, disseminating, distributing or copying
this message is strictly
prohibited. If you have received this message by mistake, please immediately
notify us by
replying to the message and delete the original message and any copies
immediately thereafter.
Thank you.-
**
FAFLD
--
Ian
i...@isfluent.com
+44 (0)1223 257903
Re: Unable to index documents using DataImportHandler with MSSQL
Hah, I've just come on here to suggest you do the same thing! Thanks
for getting back to me - and interesting we both came up with the same
solution!
Now I have the problem that running a delta update updates the
'dataimport.properties' file - but then just re-fetches all the data
regardless! Weird!
On Mon, Nov 28, 2011 at 11:59 AM, Husain, Yavar yhus...@firstam.com wrote:
Hi Ian
I downloaded and build latest Solr (3.4) from sources and finally hit
following line of code in Solr (where I put my debug statement) :
if(url != null){
LOG.info(Yavar: getting handle to driver manager:);
c = DriverManager.getConnection(url, initProps);
LOG.info(Yavar: got handle to driver manager:);
}
The call to Driver Manager was not returning. Here was the error!! The Driver
we were using was Microsoft Type 4 JDBC driver for SQL Server. I downloaded
another driver called jTDS jDBC driver and installed that. Problem got
fixed!!!
So please follow the following steps:
1. Download jTDS jDBC driver from http://jtds.sourceforge.net/
2. Put the driver jar file into your Solr/lib directory where you had put
Microsoft JDBC driver.
3. In the data-config.xml use this statement:
driver=net.sourceforge.jtds.jdbc.Driver
4. Also in data-config.xml mention url like this:
url=jdbc:jTDS:sqlserver://localhost:1433;databaseName=XXX
5. Now run your indexing.
It should solve the problem.
Regards,
Yavar
-Original Message-
From: Ian Grainger [mailto:i...@isfluent.com]
Sent: Monday, November 28, 2011 4:11 PM
To: Husain, Yavar
Cc: solr-user@lucene.apache.org
Subject: Re: Unable to index documents using DataImportHandler with MSSQL
Right.
This is REALLY weird - I've now started from scratch on another
machine (this time Windows 7), and got _exactly_ the same problem !?
On Mon, Nov 28, 2011 at 7:37 AM, Husain, Yavar yhus...@firstam.com wrote:
Hi Ian
I am having exactly the same problem what you are having on Win 7 and 2008
Server http://lucene.472066.n3.nabble.com/DIH-Strange-Problem-tc3530370.html
I still have not received any replies which could solve my problem till now.
Please do let me know if you have arrived at some solution for your problem.
Thanks.
Regards,
Yavar
-Original Message-
From: Ian Grainger [mailto:i...@isfluent.com]
Sent: Friday, November 25, 2011 10:59 PM
To: solr-user@lucene.apache.org
Subject: Re: Unable to index documents using DataImportHandler with MSSQL
Update on this: I've established:
* It's not a problem in the DB (I can index from this DB into a Solr
instance on another server)
* It's not Tomcat (I get the same problem in Jetty)
* It's not the schema (I have simplified it to one field)
That leaves SolrConfig.xml and data-config.
Only thing changed in SolrConfig.xml is adding:
lib dir=D:/Software/Solr/example/solr/dist/
regex=apache-solr-cell-\d.*\.jar /
lib dir=D:/Software/Solr/example/solr/dist/
regex=apache-solr-clustering-\d.*\.jar /
lib dir=D:/Software/Solr/example/solr/dist/
regex=apache-solr-dataimporthandler-\d.*\.jar /
requestHandler name=/dataimport
class=org.apache.solr.handler.dataimport.DataImportHandler
lst name=defaults
str
name=configD:/Software/Solr/example/solr/conf/data-config.xml/str
/lst
/requestHandler
And data-config.xml is pretty much as attached - except simpler.
Any help or any advice on how to diagnose would be appreciated!
On Fri, Nov 25, 2011 at 12:29 PM, Ian Grainger i...@isfluent.com wrote:
Hi I have copied my Solr config from a working Windows server to a new
one, and it can't seem to run an import.
They're both using win server 2008 and SQL 2008R2. This is the data
importer config
dataConfig
dataSource type=JdbcDataSource name=ds1
driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
url=jdbc:sqlserver://localhost;databaseName=DB
user=Solr
password=pwd/
document name=datas
entity name=data dataSource=ds1 pk=key
query=EXEC SOLR_COMPANY_SEARCH_DATA
deltaImportQuery=SELECT * FROM Company_Search_Data WHERE
[key]='${dataimporter.delta.key}'
deltaQuery=SELECT [key] FROM Company_Search_Data WHERE modify_dt
'${dataimporter.last_index_time}'
field column=WorkDesc_Comments
name=WorkDesc_Comments_Split /
field column=WorkDesc_Comments
name=WorkDesc_Comments_Edge /
/entity
/document
/dataConfig
I can use MS SQL Profiler to watch the Solr user log in successfully,
but then nothing. It doesn't seem to even try and execute the stored
procedure. Any ideas why this would be working one server and not on
another?
FTR the only thing in the tomcat catalina log is:
org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO: Creating a connection for entity data with URL:
jdbc:sqlserver://localhost;databaseName=CATLive
--
Ian
i...@isfluent.com
+44 (0)1223 257903
Re: DIH Strange Problem
Aha! That sounds like it might be it!
On Mon, Nov 28, 2011 at 4:16 PM, Husain, Yavar yhus...@firstam.com wrote:
Thanks Kai for sharing this. Ian encountered the same problem so marking him
in the mail too.
From: Kai Gülzau [kguel...@novomind.com]
Sent: Monday, November 28, 2011 6:55 PM
To: solr-user@lucene.apache.org
Subject: RE: DIH Strange Problem
Do you use Java 6 update 29? There is a known issue with the latest mssql
driver:
http://blogs.msdn.com/b/jdbcteam/archive/2011/11/07/supported-java-versions-november-2011.aspx
In addition, there are known connection failure issues with Java 6 update
29, and the developer preview (non production) versions of Java 6 update 30
and Java 6 update 30 build 12. We are in contact with Java on these issues
and we will update this blog once we have more information.
Should work with update 28.
Kai
-Original Message-
From: Husain, Yavar [mailto:yhus...@firstam.com]
Sent: Monday, November 28, 2011 1:02 PM
To: solr-user@lucene.apache.org; Shawn Heisey
Subject: RE: DIH Strange Problem
I figured out the solution and Microsoft and not Solr is the problem here :):
I downloaded and build latest Solr (3.4) from sources and finally hit
following line of code in Solr (where I put my debug statement) :
if(url != null){
LOG.info(Yavar: getting handle to driver manager:);
c = DriverManager.getConnection(url, initProps);
LOG.info(Yavar: got handle to driver manager:); }
The call to Driver Manager was not returning. Here was the error!! The Driver
we were using was Microsoft Type 4 JDBC driver for SQL Server. I downloaded
another driver called jTDS jDBC driver and installed that. Problem got
fixed!!!
So please follow the following steps:
1. Download jTDS jDBC driver from http://jtds.sourceforge.net/ 2. Put the
driver jar file into your Solr/lib directory where you had put Microsoft JDBC
driver.
3. In the data-config.xml use this statement:
driver=net.sourceforge.jtds.jdbc.Driver
4. Also in data-config.xml mention url like this:
url=jdbc:jTDS:sqlserver://localhost:1433;databaseName=XXX
5. Now run your indexing.
It should solve the problem.
-Original Message-
From: Husain, Yavar
Sent: Thursday, November 24, 2011 12:38 PM
To: solr-user@lucene.apache.org; Shawn Heisey
Subject: RE: DIH Strange Problem
Hi
Thanks for your replies.
I carried out these 2 steps (it did not solve my problem):
1. I tried setting responseBuffering to adaptive. Did not work.
2. For checking Database connection I wrote a simple java program to connect
to database and fetch some results with the same driver that I use for solr.
It worked. So it does not seem to be a problem with the connection.
Now I am stuck where Tomcat log says: Creating a connection for entity
. and does nothing, I mean after this log we usually get the
getConnection() took x millisecond however I dont get that ,I can just see
the time moving with no records getting fetched.
Original Problem listed again:
I am using Solr 1.4.1 on Windows/MS SQL Server and am using DIH for importing
data. Indexing and all was working perfectly fine. However today when I
started full indexing again, Solr halts/stucks at the line Creating a
connection for entity. There are no further messages after that. I
can see that DIH is busy and on the DIH console I can see A command is still
running, I can also see total rows fetched = 0 and total request made to
datasource = 1 and time is increasing however it is not doing anything. This
is the exact configuration that worked for me. I am not really able to
understand the problem here. Also in the index directory where I am storing
the index there are just 3 files: 2 segment files + 1 lucene*-write.lock
file.
...
data-config.xml:
dataSource type=JdbcDataSource
driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
url=jdbc:sqlserver://127.0.0.1:1433;databaseName=SampleOrders
user=testUser password=password/ document .
.
Logs:
INFO: Server startup in 2016 ms
Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.DataImporter
doFullImport
INFO: Starting Full Import
Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/dataimport params={command=full-import} status=0
QTime=11 Nov 23, 2011 4:11:27 PM
org.apache.solr.handler.dataimport.SolrWriter readIndexerProperties
INFO: Read dataimport.properties
Nov 23, 2011 4:11:27 PM org.apache.solr.update.DirectUpdateHandler2 deleteAll
INFO: [] REMOVING ALL DOCUMENTS FROM INDEX Nov 23, 2011 4:11:27 PM
org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=C:\solrindexes\index,segFN=segments_6,version=1322041133719,generation=6,filenames=[segments_6]
Nov 23, 2011 4:11:27 PM
Unable to index documents using DataImportHandler with MSSQL
Hi I have copied my Solr config from a working Windows server to a new
one, and it can't seem to run an import.
They're both using win server 2008 and SQL 2008R2. This is the data
importer config
dataConfig
dataSource type=JdbcDataSource name=ds1
driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
url=jdbc:sqlserver://localhost;databaseName=DB
user=Solr
password=pwd/
document name=datas
entity name=data dataSource=ds1 pk=key
query=EXEC SOLR_COMPANY_SEARCH_DATA
deltaImportQuery=SELECT * FROM Company_Search_Data WHERE
[key]='${dataimporter.delta.key}'
deltaQuery=SELECT [key] FROM Company_Search_Data WHERE modify_dt
'${dataimporter.last_index_time}'
field column=WorkDesc_Comments
name=WorkDesc_Comments_Split /
field column=WorkDesc_Comments name=WorkDesc_Comments_Edge /
/entity
/document
/dataConfig
I can use MS SQL Profiler to watch the Solr user log in successfully,
but then nothing. It doesn't seem to even try and execute the stored
procedure. Any ideas why this would be working one server and not on
another?
FTR the only thing in the tomcat catalina log is:
org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO: Creating a connection for entity data with URL:
jdbc:sqlserver://localhost;databaseName=CATLive
--
Ian
i...@isfluent.com
+44 (0)1223 257903
Re: Unable to index documents using DataImportHandler with MSSQL
Update on this: I've established:
* It's not a problem in the DB (I can index from this DB into a Solr
instance on another server)
* It's not Tomcat (I get the same problem in Jetty)
* It's not the schema (I have simplified it to one field)
That leaves SolrConfig.xml and data-config.
Only thing changed in SolrConfig.xml is adding:
lib dir=D:/Software/Solr/example/solr/dist/
regex=apache-solr-cell-\d.*\.jar /
lib dir=D:/Software/Solr/example/solr/dist/
regex=apache-solr-clustering-\d.*\.jar /
lib dir=D:/Software/Solr/example/solr/dist/
regex=apache-solr-dataimporthandler-\d.*\.jar /
requestHandler name=/dataimport
class=org.apache.solr.handler.dataimport.DataImportHandler
lst name=defaults
str name=configD:/Software/Solr/example/solr/conf/data-config.xml/str
/lst
/requestHandler
And data-config.xml is pretty much as attached - except simpler.
Any help or any advice on how to diagnose would be appreciated!
On Fri, Nov 25, 2011 at 12:29 PM, Ian Grainger i...@isfluent.com wrote:
Hi I have copied my Solr config from a working Windows server to a new
one, and it can't seem to run an import.
They're both using win server 2008 and SQL 2008R2. This is the data
importer config
dataConfig
dataSource type=JdbcDataSource name=ds1
driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
url=jdbc:sqlserver://localhost;databaseName=DB
user=Solr
password=pwd/
document name=datas
entity name=data dataSource=ds1 pk=key
query=EXEC SOLR_COMPANY_SEARCH_DATA
deltaImportQuery=SELECT * FROM Company_Search_Data WHERE
[key]='${dataimporter.delta.key}'
deltaQuery=SELECT [key] FROM Company_Search_Data WHERE modify_dt
'${dataimporter.last_index_time}'
field column=WorkDesc_Comments
name=WorkDesc_Comments_Split /
field column=WorkDesc_Comments name=WorkDesc_Comments_Edge
/
/entity
/document
/dataConfig
I can use MS SQL Profiler to watch the Solr user log in successfully,
but then nothing. It doesn't seem to even try and execute the stored
procedure. Any ideas why this would be working one server and not on
another?
FTR the only thing in the tomcat catalina log is:
org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO: Creating a connection for entity data with URL:
jdbc:sqlserver://localhost;databaseName=CATLive
--
Ian
i...@isfluent.com
+44 (0)1223 257903
--
Ian
i...@isfluent.com
+44 (0)1223 257903
Solr 3.4 group.truncate does not work with facet queries
Hi, I'm using Grouping with group.truncate=true, The following simple facet query: facet.query=Monitor_id:[38 TO 40] Doesn't give the same number as the nGroups result (with grouping.ngroups=true) for the equivalent filter query: fq=Monitor_id:[38 TO 40] I thought they should be the same - from the Wiki page: 'group.truncate: If true, facet counts are based on the most relevant document of each group matching the query.' What am I doing wrong? If I turn off group.truncate then the counts are the same, as I'd expect - but unfortunately I'm only interested in the grouped results. - I have also asked this question on StackOverflow, here: http://stackoverflow.com/questions/7905756/solr-3-4-group-truncate-does-not-work-with-facet-queries Thanks! -- Ian i...@isfluent.com a...@endissolutions.com +44 (0)1223 257903
Re: Solr 3.4 group.truncate does not work with facet queries
Thanks, Marijn. I have logged the bug here: https://issues.apache.org/jira/browse/SOLR-2863 Is there any chance of a workaround for this issue before the bug is fixed? If you want to answer the question on StackOverflow: http://stackoverflow.com/questions/7905756/solr-3-4-group-truncate-does-not-work-with-facet-queries I'll accept your answer. On Fri, Oct 28, 2011 at 12:14 PM, Martijn v Groningen martijn.v.gronin...@gmail.com wrote: Hi Ian, I think this is a bug. After looking into the code the facet.query feature doesn't take into account the group.truncate option. This needs to be fixed. You can open a new issue in Jira if you want to. Martijn On 28 October 2011 12:09, Ian Grainger i...@isfluent.com wrote: Hi, I'm using Grouping with group.truncate=true, The following simple facet query: facet.query=Monitor_id:[38 TO 40] Doesn't give the same number as the nGroups result (with grouping.ngroups=true) for the equivalent filter query: fq=Monitor_id:[38 TO 40] I thought they should be the same - from the Wiki page: 'group.truncate: If true, facet counts are based on the most relevant document of each group matching the query.' What am I doing wrong? If I turn off group.truncate then the counts are the same, as I'd expect - but unfortunately I'm only interested in the grouped results. - I have also asked this question on StackOverflow, here: http://stackoverflow.com/questions/7905756/solr-3-4-group-truncate-does-not-work-with-facet-queries Thanks! -- Ian i...@isfluent.com a...@endissolutions.com +44 (0)1223 257903 -- Met vriendelijke groet, Martijn van Groningen -- Ian i...@isfluent.com a...@endissolutions.com +44 (0)1223 257903
