Delta DataImport is not picking the modified value in DB

[Manupriya]

Hi,

I am using Delta DataImport feature to partially refresh the indexes. 

Note - full-import is working perfectly fine. But I need to do delta-import
as I do not want to rebuild all the indexes.

My database structure is same as mentioned in the example at
http://wiki.apache.org/solr/DataImportHandler.

http://www.nabble.com/file/p21200498/example-schema.png 
---
The dataimport.properties has the value as -

#Mon Dec 29 14:08:12 IST 2008
last_index_time=2008-12-29 14\:08\:12
---
I have changed the last_modified for the 'tem' table as current timestamp.
And when I query the DB, I get the following result - 

http://www.nabble.com/file/p21200498/data.jpg 
-

But when I call delta data-import, it returns me the following response - 

Server Response:
{responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
response format is experimental.  It is likely to change in the future.}

---
And when I query for data for '*:*', I get the following -
Note - Data returned by query doesnt reflect the correct timestamp from
database.

SolrDocument(1)
Field Name  Value: includes=12
Field Name  Value: features=[demo feature, demo feature 1]
Field Name  Value: price=12.0
Field Name  Value: timestamp=Mon Dec 29 13:57:54 IST 2008
Field Name  Value: manu=manu-12
Field Name  Value: sku=1
Field Name  Value: id=1
Field Name  Value: popularity=1
Field Name  Value: weight=12.0
SolrDocument(2)
Field Name  Value: includes=43
Field Name  Value: features=demo feature 2
Field Name  Value: price=10.0
Field Name  Value: timestamp=Mon Dec 29 13:57:54 IST 2008
Field Name  Value: manu=Demo - 12
Field Name  Value: sku=2
Field Name  Value: id=2
Field Name  Value: popularity=5
Field Name  Value: weight=12.0
SolrDocument(3)
Field Name  Value: includes=1
Field Name  Value: features=demo -3
Field Name  Value: price=1.0
Field Name  Value: timestamp=Mon Dec 29 13:57:54 IST 2008
Field Name  Value: manu=manu - 36
Field Name  Value: sku=3
Field Name  Value: id=3
Field Name  Value: popularity=1
Field Name  Value: weight=1.0
SolrDocument(4)
Field Name  Value: includes=2
Field Name  Value: features=demo - 4
Field Name  Value: price=2.0
Field Name  Value: timestamp=Mon Dec 29 13:57:55 IST 2008
Field Name  Value: manu=manu - 46
Field Name  Value: sku=4
Field Name  Value: id=4
Field Name  Value: popularity=2
Field Name  Value: weight=2.0
SolrDocument(5)
Field Name  Value: includes=3
Field Name  Value: features=demo - 5
Field Name  Value: price=3.0
Field Name  Value: timestamp=Mon Dec 29 13:57:55 IST 2008
Field Name  Value: manu=manu - 56
Field Name  Value: sku=5
Field Name  Value: id=5
Field Name  Value: popularity=3
Field Name  Value: weight=3.0
SolrDocument(6)
Field Name  Value: includes=4
Field Name  Value: features=demo - 66
Field Name  Value: price=4.0
Field Name  Value: timestamp=Mon Dec 29 13:57:55 IST 2008
Field Name  Value: manu=manu - 66
Field Name  Value: sku=6
Field Name  Value: id=6
Field Name  Value: popularity=4
Field Name  Value: weight=4.0

--
My Java code that calls delta import is as follows - 

public static void main(String[] args) throws IOException,
SolrServerException {
CommonsHttpSolrServer server = new
CommonsHttpSolrServer(http://localhost:8983/solr;);
BinaryResponseParser parser = new BinaryResponseParser();
server.setParser(parser);

ModifiableSolrParams params = new ModifiableSolrParams();
params.set(qt, /dataimport);
params.set(command, delta-import);
params.set(commit, true);
params.set(wt,json);

try{

QueryResponse response = server.query(params);
server.commit();
System.out.println(Server Response:  + response);

SolrQuery query = new SolrQuery(*:*);
QueryResponse results = server.query(query);
  
SolrDocumentList list = results.getResults();

int index = 1;
for (SolrDocument solrDocument : list) {

System.out.println(SolrDocument(+index+));
IteratorEntryString, Object iterator = solrDocument.iterator();

while(iterator.hasNext()){
System.out.println(Field Name  Value: +iterator.next());
}

index++;
}

}catch(Exception ex){

System.out.println(Exception Occured:+ex);
}

  }
---

Does SOLR do some kind of caching? I dont understand as why the updated
values are not queried correctly from DB?

There are no errors in the logs. But I can see an error on the 

Re: Delta DataImport is not picking the modified value in DB

[Shalin Shekhar Mangar]

What does your data-config look like? especially the delta query part.

On Mon, Dec 29, 2008 at 2:35 PM, Manupriya manupriya.si...@gmail.comwrote:


 Hi,

 I am using Delta DataImport feature to partially refresh the indexes.

 Note - full-import is working perfectly fine. But I need to do delta-import
 as I do not want to rebuild all the indexes.

 My database structure is same as mentioned in the example at
 http://wiki.apache.org/solr/DataImportHandler.

 http://www.nabble.com/file/p21200498/example-schema.png
 ---
 The dataimport.properties has the value as -

 #Mon Dec 29 14:08:12 IST 2008
 last_index_time=2008-12-29 14\:08\:12
 ---
 I have changed the last_modified for the 'tem' table as current timestamp.
 And when I query the DB, I get the following result -

 http://www.nabble.com/file/p21200498/data.jpg

 -

 But when I call delta data-import, it returns me the following response -

 Server Response:

 {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
 response format is experimental.  It is likely to change in the future.}


 ---
 And when I query for data for '*:*', I get the following -
 Note - Data returned by query doesnt reflect the correct timestamp from
 database.

 SolrDocument(1)
 Field Name  Value: includes=12
 Field Name  Value: features=[demo feature, demo feature 1]
 Field Name  Value: price=12.0
 Field Name  Value: timestamp=Mon Dec 29 13:57:54 IST 2008
 Field Name  Value: manu=manu-12
 Field Name  Value: sku=1
 Field Name  Value: id=1
 Field Name  Value: popularity=1
 Field Name  Value: weight=12.0
 SolrDocument(2)
 Field Name  Value: includes=43
 Field Name  Value: features=demo feature 2
 Field Name  Value: price=10.0
 Field Name  Value: timestamp=Mon Dec 29 13:57:54 IST 2008
 Field Name  Value: manu=Demo - 12
 Field Name  Value: sku=2
 Field Name  Value: id=2
 Field Name  Value: popularity=5
 Field Name  Value: weight=12.0
 SolrDocument(3)
 Field Name  Value: includes=1
 Field Name  Value: features=demo -3
 Field Name  Value: price=1.0
 Field Name  Value: timestamp=Mon Dec 29 13:57:54 IST 2008
 Field Name  Value: manu=manu - 36
 Field Name  Value: sku=3
 Field Name  Value: id=3
 Field Name  Value: popularity=1
 Field Name  Value: weight=1.0
 SolrDocument(4)
 Field Name  Value: includes=2
 Field Name  Value: features=demo - 4
 Field Name  Value: price=2.0
 Field Name  Value: timestamp=Mon Dec 29 13:57:55 IST 2008
 Field Name  Value: manu=manu - 46
 Field Name  Value: sku=4
 Field Name  Value: id=4
 Field Name  Value: popularity=2
 Field Name  Value: weight=2.0
 SolrDocument(5)
 Field Name  Value: includes=3
 Field Name  Value: features=demo - 5
 Field Name  Value: price=3.0
 Field Name  Value: timestamp=Mon Dec 29 13:57:55 IST 2008
 Field Name  Value: manu=manu - 56
 Field Name  Value: sku=5
 Field Name  Value: id=5
 Field Name  Value: popularity=3
 Field Name  Value: weight=3.0
 SolrDocument(6)
 Field Name  Value: includes=4
 Field Name  Value: features=demo - 66
 Field Name  Value: price=4.0
 Field Name  Value: timestamp=Mon Dec 29 13:57:55 IST 2008
 Field Name  Value: manu=manu - 66
 Field Name  Value: sku=6
 Field Name  Value: id=6
 Field Name  Value: popularity=4
 Field Name  Value: weight=4.0


 --
 My Java code that calls delta import is as follows -

 public static void main(String[] args) throws IOException,
 SolrServerException {
 CommonsHttpSolrServer server = new
 CommonsHttpSolrServer(http://localhost:8983/solr;);
 BinaryResponseParser parser = new BinaryResponseParser();
 server.setParser(parser);

 ModifiableSolrParams params = new ModifiableSolrParams();
 params.set(qt, /dataimport);
 params.set(command, delta-import);
 params.set(commit, true);
 params.set(wt,json);

 try{

QueryResponse response = server.query(params);
server.commit();
System.out.println(Server Response:  + response);

SolrQuery query = new SolrQuery(*:*);
QueryResponse results = server.query(query);

SolrDocumentList list = results.getResults();

int index = 1;
for (SolrDocument solrDocument : list) {

System.out.println(SolrDocument(+index+));
IteratorEntryString, Object iterator = solrDocument.iterator();

while(iterator.hasNext()){
System.out.println(Field Name  Value: +iterator.next());
}

index++;
}

}catch(Exception ex){

System.out.println(Exception Occured:+ex);
}

  }
 ---

 

Re: Delta DataImport is not picking the modified value in DB

[Noble Paul നോബിള്‍ नोब्ळ्]

This may happen if your deltaQuery has a problem

Try using the deltaImportQuery also.

--

On Mon, Dec 29, 2008 at 2:57 PM, Shalin Shekhar Mangar
shalinman...@gmail.com wrote:
 What does your data-config look like? especially the delta query part.

 On Mon, Dec 29, 2008 at 2:35 PM, Manupriya manupriya.si...@gmail.comwrote:


 Hi,

 I am using Delta DataImport feature to partially refresh the indexes.

 Note - full-import is working perfectly fine. But I need to do delta-import
 as I do not want to rebuild all the indexes.

 My database structure is same as mentioned in the example at
 http://wiki.apache.org/solr/DataImportHandler.

 http://www.nabble.com/file/p21200498/example-schema.png
 ---
 The dataimport.properties has the value as -

 #Mon Dec 29 14:08:12 IST 2008
 last_index_time=2008-12-29 14\:08\:12
 ---
 I have changed the last_modified for the 'tem' table as current timestamp.
 And when I query the DB, I get the following result -

 http://www.nabble.com/file/p21200498/data.jpg

 -

 But when I call delta data-import, it returns me the following response -

 Server Response:

 {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
 response format is experimental.  It is likely to change in the future.}


 ---
 And when I query for data for '*:*', I get the following -
 Note - Data returned by query doesnt reflect the correct timestamp from
 database.

 SolrDocument(1)
 Field Name  Value: includes=12
 Field Name  Value: features=[demo feature, demo feature 1]
 Field Name  Value: price=12.0
 Field Name  Value: timestamp=Mon Dec 29 13:57:54 IST 2008
 Field Name  Value: manu=manu-12
 Field Name  Value: sku=1
 Field Name  Value: id=1
 Field Name  Value: popularity=1
 Field Name  Value: weight=12.0
 SolrDocument(2)
 Field Name  Value: includes=43
 Field Name  Value: features=demo feature 2
 Field Name  Value: price=10.0
 Field Name  Value: timestamp=Mon Dec 29 13:57:54 IST 2008
 Field Name  Value: manu=Demo - 12
 Field Name  Value: sku=2
 Field Name  Value: id=2
 Field Name  Value: popularity=5
 Field Name  Value: weight=12.0
 SolrDocument(3)
 Field Name  Value: includes=1
 Field Name  Value: features=demo -3
 Field Name  Value: price=1.0
 Field Name  Value: timestamp=Mon Dec 29 13:57:54 IST 2008
 Field Name  Value: manu=manu - 36
 Field Name  Value: sku=3
 Field Name  Value: id=3
 Field Name  Value: popularity=1
 Field Name  Value: weight=1.0
 SolrDocument(4)
 Field Name  Value: includes=2
 Field Name  Value: features=demo - 4
 Field Name  Value: price=2.0
 Field Name  Value: timestamp=Mon Dec 29 13:57:55 IST 2008
 Field Name  Value: manu=manu - 46
 Field Name  Value: sku=4
 Field Name  Value: id=4
 Field Name  Value: popularity=2
 Field Name  Value: weight=2.0
 SolrDocument(5)
 Field Name  Value: includes=3
 Field Name  Value: features=demo - 5
 Field Name  Value: price=3.0
 Field Name  Value: timestamp=Mon Dec 29 13:57:55 IST 2008
 Field Name  Value: manu=manu - 56
 Field Name  Value: sku=5
 Field Name  Value: id=5
 Field Name  Value: popularity=3
 Field Name  Value: weight=3.0
 SolrDocument(6)
 Field Name  Value: includes=4
 Field Name  Value: features=demo - 66
 Field Name  Value: price=4.0
 Field Name  Value: timestamp=Mon Dec 29 13:57:55 IST 2008
 Field Name  Value: manu=manu - 66
 Field Name  Value: sku=6
 Field Name  Value: id=6
 Field Name  Value: popularity=4
 Field Name  Value: weight=4.0


 --
 My Java code that calls delta import is as follows -

 public static void main(String[] args) throws IOException,
 SolrServerException {
 CommonsHttpSolrServer server = new
 CommonsHttpSolrServer(http://localhost:8983/solr;);
 BinaryResponseParser parser = new BinaryResponseParser();
 server.setParser(parser);

 ModifiableSolrParams params = new ModifiableSolrParams();
 params.set(qt, /dataimport);
 params.set(command, delta-import);
 params.set(commit, true);
 params.set(wt,json);

 try{

QueryResponse response = server.query(params);
server.commit();
System.out.println(Server Response:  + response);

SolrQuery query = new SolrQuery(*:*);
QueryResponse results = server.query(query);

SolrDocumentList list = results.getResults();

int index = 1;
for (SolrDocument solrDocument : list) {

System.out.println(SolrDocument(+index+));
IteratorEntryString, Object iterator = solrDocument.iterator();

while(iterator.hasNext()){
System.out.println(Field Name  Value: +iterator.next());
}

index++;
 

Re: Delta DataImport is not picking the modified value in DB

[Manupriya]

Hi Shalin,

I am actually trying out the example provided at
http://wiki.apache.org/solr/DataImportHandler.

My db-data-config.xml is as following - 

dataConfig
dataSource driver=com.mysql.jdbc.Driver
url=jdbc:mysql://localhost/solr_demo_db user=root password=root /
document name=products
entity name=item pk=ID query=select * from item
deltaQuery=select id from item where last_modified 
'${dataimporter.last_index_time}'
field column=NAME name=name /
field column=NAME name=nameSort /
field column=NAME name=alphaNameSort /

entity name=feature pk=ITEM_ID 
query=select DESCRIPTION from FEATURE where
ITEM_ID='${item.ID}'
deltaQuery=select ITEM_ID from FEATURE where
last_modified  '${dataimporter.last_index_time}'
parentDeltaQuery=select ID from item where
ID=${feature.ITEM_ID}
field name=features column=DESCRIPTION /
/entity

entity name=item_category pk=ITEM_ID, CATEGORY_ID
query=select CATEGORY_ID from item_category where
ITEM_ID='${item.ID}'
deltaQuery=select ITEM_ID, CATEGORY_ID from
item_category where last_modified  '${dataimporter.last_index_time}'
parentDeltaQuery=select ID from item where
ID=${item_category.ITEM_ID}
entity name=category pk=ID
query=select DESCRIPTION from category where ID =
'${item_category.CATEGORY_ID}'
deltaQuery=select ID from category where
last_modified  '${dataimporter.last_index_time}'
parentDeltaQuery=select ITEM_ID, CATEGORY_ID from
item_category where CATEGORY_ID=${category.ID}
field column=description name=cat /
/entity
/entity
/entity
/document
/dataConfig
---

My DB structure is as following - 

DROP TABLE IF EXISTS solr_demo_db.item;
CREATE TABLE `item` (
  `ID` int(2) NOT NULL auto_increment,
  `name` varchar(100) default NULL,
  `manu` varchar(20) default NULL,
  `weight` varchar(20) default NULL,
  `price` varchar(20) default NULL,
  `popularity` varchar(2) default NULL,
  `includes` varchar(10) default NULL,
  `last_modified` datetime default NULL,
  PRIMARY KEY  (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;

DROP TABLE IF EXISTS solr_demo_db.feature;
CREATE TABLE `feature` (
  `DESCRIPTION` varchar(100) default NULL,
  `ITEM_ID` int(2) NOT NULL,
  `last_modified` datetime default NULL,
  PRIMARY KEY  (`ITEM_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

DROP TABLE IF EXISTS solr_demo_db.category;
CREATE TABLE `category` (
  `ID` int(2) NOT NULL auto_increment,
  `DESCRIPTION` varchar(100) default NULL,
  `last_modified` date default NULL,
  PRIMARY KEY  (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;

DROP TABLE IF EXISTS solr_demo_db.item_category;
CREATE TABLE `item_category` (
  `ITEM_ID` int(2) NOT NULL default '0',
  `CATEGORY_ID` int(2) NOT NULL default '0',
  `last_modified` date default NULL,
  PRIMARY KEY  (`ITEM_ID`,`CATEGORY_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Am I doing something wrong here?

Thanks,
Manu


Shalin Shekhar Mangar wrote:
 
 What does your data-config look like? especially the delta query part.
 
 On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
 manupriya.si...@gmail.comwrote:
 

 Hi,

 I am using Delta DataImport feature to partially refresh the indexes.

 Note - full-import is working perfectly fine. But I need to do
 delta-import
 as I do not want to rebuild all the indexes.

 My database structure is same as mentioned in the example at
 http://wiki.apache.org/solr/DataImportHandler.

 http://www.nabble.com/file/p21200498/example-schema.png
 ---
 The dataimport.properties has the value as -

 #Mon Dec 29 14:08:12 IST 2008
 last_index_time=2008-12-29 14\:08\:12
 ---
 I have changed the last_modified for the 'tem' table as current
 timestamp.
 And when I query the DB, I get the following result -

 http://www.nabble.com/file/p21200498/data.jpg

 -

 But when I call delta data-import, it returns me the following response -

 Server Response:

 {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
 response format is experimental.  It is likely to change in the future.}


 ---
 And when I query for data for '*:*', I get the following -
 Note - Data returned by query doesnt 

Re: Delta DataImport is not picking the modified value in DB

[Noble Paul നോബിള്‍ नोब्ळ्]

did you put in the data?

On Mon, Dec 29, 2008 at 3:07 PM, Manupriya manupriya.si...@gmail.com wrote:

 Hi Shalin,

 I am actually trying out the example provided at
 http://wiki.apache.org/solr/DataImportHandler.

 My db-data-config.xml is as following -

 dataConfig
dataSource driver=com.mysql.jdbc.Driver
 url=jdbc:mysql://localhost/solr_demo_db user=root password=root /
document name=products
entity name=item pk=ID query=select * from item
deltaQuery=select id from item where last_modified 
 '${dataimporter.last_index_time}'
field column=NAME name=name /
field column=NAME name=nameSort /
field column=NAME name=alphaNameSort /

entity name=feature pk=ITEM_ID
query=select DESCRIPTION from FEATURE where
 ITEM_ID='${item.ID}'
deltaQuery=select ITEM_ID from FEATURE where
 last_modified  '${dataimporter.last_index_time}'
parentDeltaQuery=select ID from item where
 ID=${feature.ITEM_ID}
field name=features column=DESCRIPTION /
/entity

entity name=item_category pk=ITEM_ID, CATEGORY_ID
query=select CATEGORY_ID from item_category where
 ITEM_ID='${item.ID}'
deltaQuery=select ITEM_ID, CATEGORY_ID from
 item_category where last_modified  '${dataimporter.last_index_time}'
parentDeltaQuery=select ID from item where
 ID=${item_category.ITEM_ID}
entity name=category pk=ID
query=select DESCRIPTION from category where ID =
 '${item_category.CATEGORY_ID}'
deltaQuery=select ID from category where
 last_modified  '${dataimporter.last_index_time}'
parentDeltaQuery=select ITEM_ID, CATEGORY_ID from
 item_category where CATEGORY_ID=${category.ID}
field column=description name=cat /
/entity
/entity
/entity
/document
 /dataConfig
 ---

 My DB structure is as following -

 DROP TABLE IF EXISTS solr_demo_db.item;
 CREATE TABLE `item` (
  `ID` int(2) NOT NULL auto_increment,
  `name` varchar(100) default NULL,
  `manu` varchar(20) default NULL,
  `weight` varchar(20) default NULL,
  `price` varchar(20) default NULL,
  `popularity` varchar(2) default NULL,
  `includes` varchar(10) default NULL,
  `last_modified` datetime default NULL,
  PRIMARY KEY  (`ID`)
 ) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;

 DROP TABLE IF EXISTS solr_demo_db.feature;
 CREATE TABLE `feature` (
  `DESCRIPTION` varchar(100) default NULL,
  `ITEM_ID` int(2) NOT NULL,
  `last_modified` datetime default NULL,
  PRIMARY KEY  (`ITEM_ID`)
 ) ENGINE=InnoDB DEFAULT CHARSET=latin1;

 DROP TABLE IF EXISTS solr_demo_db.category;
 CREATE TABLE `category` (
  `ID` int(2) NOT NULL auto_increment,
  `DESCRIPTION` varchar(100) default NULL,
  `last_modified` date default NULL,
  PRIMARY KEY  (`ID`)
 ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;

 DROP TABLE IF EXISTS solr_demo_db.item_category;
 CREATE TABLE `item_category` (
  `ITEM_ID` int(2) NOT NULL default '0',
  `CATEGORY_ID` int(2) NOT NULL default '0',
  `last_modified` date default NULL,
  PRIMARY KEY  (`ITEM_ID`,`CATEGORY_ID`)
 ) ENGINE=InnoDB DEFAULT CHARSET=latin1;

 Am I doing something wrong here?

 Thanks,
 Manu


 Shalin Shekhar Mangar wrote:

 What does your data-config look like? especially the delta query part.

 On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
 manupriya.si...@gmail.comwrote:


 Hi,

 I am using Delta DataImport feature to partially refresh the indexes.

 Note - full-import is working perfectly fine. But I need to do
 delta-import
 as I do not want to rebuild all the indexes.

 My database structure is same as mentioned in the example at
 http://wiki.apache.org/solr/DataImportHandler.

 http://www.nabble.com/file/p21200498/example-schema.png
 ---
 The dataimport.properties has the value as -

 #Mon Dec 29 14:08:12 IST 2008
 last_index_time=2008-12-29 14\:08\:12
 ---
 I have changed the last_modified for the 'tem' table as current
 timestamp.
 And when I query the DB, I get the following result -

 http://www.nabble.com/file/p21200498/data.jpg

 -

 But when I call delta data-import, it returns me the following response -

 Server Response:

 {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
 response format is experimental.  It is likely to change in the future.}


 

Re: Delta DataImport is not picking the modified value in DB

[Noble Paul നോബിള്‍ नोब्ळ्]

The same example with deltaImportQuery would look as follows
dataConfig
   dataSource driver=com.mysql.jdbc.Driver
url=jdbc:mysql://localhost/solr_demo_db user=root password=root /
   document name=products
   entity name=item pk=ID query=select * from item
   deltaQuery=select id from item where last_modified 
'${dataimporter.last_index_time}'
deltaImportQuery=select * from item where ID=${dataimporter.delta.ID}

   field column=NAME name=name /
   field column=NAME name=nameSort /
   field column=NAME name=alphaNameSort /

   entity name=feature pk=ITEM_ID
   query=select DESCRIPTION from FEATURE where
ITEM_ID='${item.ID}'
   deltaQuery=select ITEM_ID from FEATURE where
last_modified  '${dataimporter.last_index_time}'
   parentDeltaQuery=select ID from item where
ID=${feature.ITEM_ID}
   field name=features column=DESCRIPTION /
   /entity

   entity name=item_category pk=ITEM_ID, CATEGORY_ID
   query=select CATEGORY_ID from item_category where
ITEM_ID='${item.ID}'
   deltaQuery=select ITEM_ID, CATEGORY_ID from
item_category where last_modified  '${dataimporter.last_index_time}'
   parentDeltaQuery=select ID from item where
ID=${item_category.ITEM_ID}
   entity name=category pk=ID
   query=select DESCRIPTION from category where ID =
'${item_category.CATEGORY_ID}'
   deltaQuery=select ID from category where
last_modified  '${dataimporter.last_index_time}'
   parentDeltaQuery=select ITEM_ID, CATEGORY_ID from
item_category where CATEGORY_ID=${category.ID}
   field column=description name=cat /
   /entity
   /entity
   /entity
   /document
/dataConfig

On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള്‍ नोब्ळ्
noble.p...@gmail.com wrote:
 did you put in the data?

 On Mon, Dec 29, 2008 at 3:07 PM, Manupriya manupriya.si...@gmail.com wrote:

 Hi Shalin,

 I am actually trying out the example provided at
 http://wiki.apache.org/solr/DataImportHandler.

 My db-data-config.xml is as following -

 dataConfig
dataSource driver=com.mysql.jdbc.Driver
 url=jdbc:mysql://localhost/solr_demo_db user=root password=root /
document name=products
entity name=item pk=ID query=select * from item
deltaQuery=select id from item where last_modified 
 '${dataimporter.last_index_time}'
field column=NAME name=name /
field column=NAME name=nameSort /
field column=NAME name=alphaNameSort /

entity name=feature pk=ITEM_ID
query=select DESCRIPTION from FEATURE where
 ITEM_ID='${item.ID}'
deltaQuery=select ITEM_ID from FEATURE where
 last_modified  '${dataimporter.last_index_time}'
parentDeltaQuery=select ID from item where
 ID=${feature.ITEM_ID}
field name=features column=DESCRIPTION /
/entity

entity name=item_category pk=ITEM_ID, CATEGORY_ID
query=select CATEGORY_ID from item_category where
 ITEM_ID='${item.ID}'
deltaQuery=select ITEM_ID, CATEGORY_ID from
 item_category where last_modified  '${dataimporter.last_index_time}'
parentDeltaQuery=select ID from item where
 ID=${item_category.ITEM_ID}
entity name=category pk=ID
query=select DESCRIPTION from category where ID =
 '${item_category.CATEGORY_ID}'
deltaQuery=select ID from category where
 last_modified  '${dataimporter.last_index_time}'
parentDeltaQuery=select ITEM_ID, CATEGORY_ID from
 item_category where CATEGORY_ID=${category.ID}
field column=description name=cat /
/entity
/entity
/entity
/document
 /dataConfig
 ---

 My DB structure is as following -

 DROP TABLE IF EXISTS solr_demo_db.item;
 CREATE TABLE `item` (
  `ID` int(2) NOT NULL auto_increment,
  `name` varchar(100) default NULL,
  `manu` varchar(20) default NULL,
  `weight` varchar(20) default NULL,
  `price` varchar(20) default NULL,
  `popularity` varchar(2) default NULL,
  `includes` varchar(10) default NULL,
  `last_modified` datetime default NULL,
  PRIMARY KEY  (`ID`)
 ) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;

 DROP TABLE IF EXISTS solr_demo_db.feature;
 CREATE TABLE `feature` (
  `DESCRIPTION` varchar(100) default NULL,
  `ITEM_ID` int(2) NOT NULL,
  `last_modified` datetime default NULL,
  PRIMARY KEY  (`ITEM_ID`)
 ) ENGINE=InnoDB DEFAULT CHARSET=latin1;

 DROP TABLE IF EXISTS solr_demo_db.category;
 CREATE TABLE `category` (
  `ID` int(2) NOT NULL auto_increment,
  `DESCRIPTION` varchar(100) default NULL,
  `last_modified` date 

Re: Delta DataImport is not picking the modified value in DB

[Manupriya]

Hi Noble,

I tried with the deltaImportQuery as well. But still I am getting the same
exception on the server console - 

Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
doDeltaImport

SEVERE: Delta Import Failed
java.lang.NullPointerException
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
at
org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
at
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)

Another thing, I do have proper data in the DB.

Thanks,
Manu


Noble Paul നോബിള്‍ नोब्ळ् wrote:
 
 The same example with deltaImportQuery would look as follows
 dataConfig
dataSource driver=com.mysql.jdbc.Driver
 url=jdbc:mysql://localhost/solr_demo_db user=root password=root /
document name=products
entity name=item pk=ID query=select * from item
deltaQuery=select id from item where last_modified 
 '${dataimporter.last_index_time}'
 deltaImportQuery=select * from item where ID=${dataimporter.delta.ID}

field column=NAME name=name /
field column=NAME name=nameSort /
field column=NAME name=alphaNameSort /
 
entity name=feature pk=ITEM_ID
query=select DESCRIPTION from FEATURE where
 ITEM_ID='${item.ID}'
deltaQuery=select ITEM_ID from FEATURE where
 last_modified  '${dataimporter.last_index_time}'
parentDeltaQuery=select ID from item where
 ID=${feature.ITEM_ID}
field name=features column=DESCRIPTION /
/entity
 
entity name=item_category pk=ITEM_ID, CATEGORY_ID
query=select CATEGORY_ID from item_category where
 ITEM_ID='${item.ID}'
deltaQuery=select ITEM_ID, CATEGORY_ID from
 item_category where last_modified  '${dataimporter.last_index_time}'
parentDeltaQuery=select ID from item where
 ID=${item_category.ITEM_ID}
entity name=category pk=ID
query=select DESCRIPTION from category where ID =
 '${item_category.CATEGORY_ID}'
deltaQuery=select ID from category where
 last_modified  '${dataimporter.last_index_time}'
parentDeltaQuery=select ITEM_ID, CATEGORY_ID from
 item_category where CATEGORY_ID=${category.ID}
field column=description name=cat /
/entity
/entity
/entity
/document
 /dataConfig
 
 On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള്‍ नोब्ळ्
 noble.p...@gmail.com wrote:
 did you put in the data?

 On Mon, Dec 29, 2008 at 3:07 PM, Manupriya manupriya.si...@gmail.com
 wrote:

 Hi Shalin,

 I am actually trying out the example provided at
 http://wiki.apache.org/solr/DataImportHandler.

 My db-data-config.xml is as following -

 dataConfig
dataSource driver=com.mysql.jdbc.Driver
 url=jdbc:mysql://localhost/solr_demo_db user=root password=root /
document name=products
entity name=item pk=ID query=select * from item
deltaQuery=select id from item where last_modified 
 '${dataimporter.last_index_time}'
field column=NAME name=name /
field column=NAME name=nameSort /
field column=NAME name=alphaNameSort /

entity name=feature pk=ITEM_ID
query=select DESCRIPTION from FEATURE where
 ITEM_ID='${item.ID}'
deltaQuery=select ITEM_ID from FEATURE where
 last_modified  '${dataimporter.last_index_time}'
parentDeltaQuery=select ID from item where
 ID=${feature.ITEM_ID}
field name=features column=DESCRIPTION /
/entity

entity name=item_category pk=ITEM_ID, CATEGORY_ID
query=select CATEGORY_ID from item_category where
 ITEM_ID='${item.ID}'
deltaQuery=select ITEM_ID, CATEGORY_ID from
 item_category where last_modified  '${dataimporter.last_index_time}'
parentDeltaQuery=select ID from item where
 ID=${item_category.ITEM_ID}
entity name=category pk=ID
query=select DESCRIPTION from category where ID
 =
 '${item_category.CATEGORY_ID}'
deltaQuery=select ID from category where
 last_modified  '${dataimporter.last_index_time}'

Re: Delta DataImport is not picking the modified value in DB

[Shalin Shekhar Mangar]

Which version of Solr are you using? The deltaImportQuery feature was added
recently, you'd need a nightly build for it to work.

On Mon, Dec 29, 2008 at 4:57 PM, Manupriya manupriya.si...@gmail.comwrote:


 Hi Noble,

 I tried with the deltaImportQuery as well. But still I am getting the same
 exception on the server console -

 Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
 doDeltaImport

 SEVERE: Delta Import Failed
 java.lang.NullPointerException
at

 org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
at

 org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
at

 org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at

 org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
at
 org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
at
 org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
at

 org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
at

 org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
at

 org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)

 Another thing, I do have proper data in the DB.

 Thanks,
 Manu


 Noble Paul നോബിള്‍ नोब्ळ् wrote:
 
  The same example with deltaImportQuery would look as follows
  dataConfig
 dataSource driver=com.mysql.jdbc.Driver
  url=jdbc:mysql://localhost/solr_demo_db user=root password=root /
 document name=products
 entity name=item pk=ID query=select * from item
 deltaQuery=select id from item where last_modified 
  '${dataimporter.last_index_time}'
  deltaImportQuery=select * from item where ID=${dataimporter.delta.ID}
 
 field column=NAME name=name /
 field column=NAME name=nameSort /
 field column=NAME name=alphaNameSort /
 
 entity name=feature pk=ITEM_ID
 query=select DESCRIPTION from FEATURE where
  ITEM_ID='${item.ID}'
 deltaQuery=select ITEM_ID from FEATURE where
  last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ID from item where
  ID=${feature.ITEM_ID}
 field name=features column=DESCRIPTION /
 /entity
 
 entity name=item_category pk=ITEM_ID, CATEGORY_ID
 query=select CATEGORY_ID from item_category where
  ITEM_ID='${item.ID}'
 deltaQuery=select ITEM_ID, CATEGORY_ID from
  item_category where last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ID from item where
  ID=${item_category.ITEM_ID}
 entity name=category pk=ID
 query=select DESCRIPTION from category where ID =
  '${item_category.CATEGORY_ID}'
 deltaQuery=select ID from category where
  last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ITEM_ID, CATEGORY_ID from
  item_category where CATEGORY_ID=${category.ID}
 field column=description name=cat /
 /entity
 /entity
 /entity
 /document
  /dataConfig
 
  On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള്‍ नोब्ळ्
  noble.p...@gmail.com wrote:
  did you put in the data?
 
  On Mon, Dec 29, 2008 at 3:07 PM, Manupriya manupriya.si...@gmail.com
  wrote:
 
  Hi Shalin,
 
  I am actually trying out the example provided at
  http://wiki.apache.org/solr/DataImportHandler.
 
  My db-data-config.xml is as following -
 
  dataConfig
 dataSource driver=com.mysql.jdbc.Driver
  url=jdbc:mysql://localhost/solr_demo_db user=root password=root
 /
 document name=products
 entity name=item pk=ID query=select * from item
 deltaQuery=select id from item where last_modified 
  '${dataimporter.last_index_time}'
 field column=NAME name=name /
 field column=NAME name=nameSort /
 field column=NAME name=alphaNameSort /
 
 entity name=feature pk=ITEM_ID
 query=select DESCRIPTION from FEATURE where
  ITEM_ID='${item.ID}'
 deltaQuery=select ITEM_ID from FEATURE where
  last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ID from item where
  ID=${feature.ITEM_ID}
 field name=features column=DESCRIPTION /
 /entity
 
 entity name=item_category pk=ITEM_ID, CATEGORY_ID
 query=select CATEGORY_ID from item_category where
  ITEM_ID='${item.ID}'
 deltaQuery=select ITEM_ID, CATEGORY_ID from
  item_category where last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ID from 

Re: Delta DataImport is not picking the modified value in DB

[Manupriya]

Hi Shalin,

I am using apache-solr-1.3.0 in my project. I thought it to be the latest
release. Does it not support delta import?

Thanks,
Manu


Shalin Shekhar Mangar wrote:
 
 Which version of Solr are you using? The deltaImportQuery feature was
 added
 recently, you'd need a nightly build for it to work.
 
 On Mon, Dec 29, 2008 at 4:57 PM, Manupriya
 manupriya.si...@gmail.comwrote:
 

 Hi Noble,

 I tried with the deltaImportQuery as well. But still I am getting the
 same
 exception on the server console -

 Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
 doDeltaImport

 SEVERE: Delta Import Failed
 java.lang.NullPointerException
at

 org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
at

 org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
at

 org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at

 org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
at
 org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
at
 org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
at

 org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
at

 org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
at

 org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)

 Another thing, I do have proper data in the DB.

 Thanks,
 Manu


 Noble Paul നോബിള്‍ नोब्ळ् wrote:
 
  The same example with deltaImportQuery would look as follows
  dataConfig
 dataSource driver=com.mysql.jdbc.Driver
  url=jdbc:mysql://localhost/solr_demo_db user=root password=root
 /
 document name=products
 entity name=item pk=ID query=select * from item
 deltaQuery=select id from item where last_modified 
  '${dataimporter.last_index_time}'
  deltaImportQuery=select * from item where ID=${dataimporter.delta.ID}
 
 field column=NAME name=name /
 field column=NAME name=nameSort /
 field column=NAME name=alphaNameSort /
 
 entity name=feature pk=ITEM_ID
 query=select DESCRIPTION from FEATURE where
  ITEM_ID='${item.ID}'
 deltaQuery=select ITEM_ID from FEATURE where
  last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ID from item where
  ID=${feature.ITEM_ID}
 field name=features column=DESCRIPTION /
 /entity
 
 entity name=item_category pk=ITEM_ID, CATEGORY_ID
 query=select CATEGORY_ID from item_category where
  ITEM_ID='${item.ID}'
 deltaQuery=select ITEM_ID, CATEGORY_ID from
  item_category where last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ID from item where
  ID=${item_category.ITEM_ID}
 entity name=category pk=ID
 query=select DESCRIPTION from category where ID
 =
  '${item_category.CATEGORY_ID}'
 deltaQuery=select ID from category where
  last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ITEM_ID, CATEGORY_ID
 from
  item_category where CATEGORY_ID=${category.ID}
 field column=description name=cat /
 /entity
 /entity
 /entity
 /document
  /dataConfig
 
  On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള്‍ नोब्ळ्
  noble.p...@gmail.com wrote:
  did you put in the data?
 
  On Mon, Dec 29, 2008 at 3:07 PM, Manupriya manupriya.si...@gmail.com
  wrote:
 
  Hi Shalin,
 
  I am actually trying out the example provided at
  http://wiki.apache.org/solr/DataImportHandler.
 
  My db-data-config.xml is as following -
 
  dataConfig
 dataSource driver=com.mysql.jdbc.Driver
  url=jdbc:mysql://localhost/solr_demo_db user=root password=root
 /
 document name=products
 entity name=item pk=ID query=select * from item
 deltaQuery=select id from item where last_modified 
  '${dataimporter.last_index_time}'
 field column=NAME name=name /
 field column=NAME name=nameSort /
 field column=NAME name=alphaNameSort /
 
 entity name=feature pk=ITEM_ID
 query=select DESCRIPTION from FEATURE where
  ITEM_ID='${item.ID}'
 deltaQuery=select ITEM_ID from FEATURE where
  last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ID from item where
  ID=${feature.ITEM_ID}
 field name=features column=DESCRIPTION /
 /entity
 
 entity name=item_category pk=ITEM_ID, CATEGORY_ID
 query=select CATEGORY_ID from item_category where
  ITEM_ID='${item.ID}'
  

High response times after snapshot install

[Victor Hogemann]

Greetings,

Before anything I'd like to thank the Solr team for such great piece  
of software! Thank you.


Now... back to my problem. I'm experiencing slowdowns on a Solr 1.3  
multicore installation after a snapshot install, response times go up  
from 5ms to above 5sec, and don't go down unless we restart the servers.


Our index has more than 20 fields, and the slave servers got some  
50hits/sec. We're running it on Weblogic, with 6 Weblogic Nodes/Solr  
instances per machine, sharing the same index on the filesystem.


Any insight on what is happening?

Thanks.

Victor Hogemann - Ideais Tecnologia

Tel (RJ) +55 +21 3553-1301 (R217)
Tel (SP) +55 +11 4063-0443 (R217)
Fax +55 +21 3553-1302

Rua da Assembléia, nº 98, 9º andar
Centro | Rio de Janeiro | Brasil
CEP 20011-000

Re: Please help me integrate Nutch with Solr

[Andrzej Bialecki]

Tony Wang wrote:

Thanks Otis.

I've just downloaded
NUTCH-442_v8.patchhttps://issues.apache.org/jira/secure/attachment/12391810/NUTCH-442_v8.patchfrom
https://issues.apache.org/jira/browse/NUTCH-442, but the patching process
gave me lots errors, see below:


This patch will be integrated within a couple days - please monitor this 
issue, and when it's done just download the patched code.



--
Best regards,
Andrzej Bialecki 
 ___. ___ ___ ___ _ _   __
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

Re: Any new python libraries?

[Ed Summers]

Jacob,

If you are interested in contributing any of your code to the solrpy
project [1] please let us know, either on here or on the solrpy
discussion list [2].

One of the motivations for putting the code up at code.google.com was
to make it easy for people to quickly contribute enhancements/fixes
separate from the normal release cycle of Solr proper.

//Ed

[1] http://code.google.com/p/solrpy/
[2] http://groups.google.com/group/solrpy

Re: Retrieve documents that contain max value for a field

[Sushil Vegad]

This looks useful, but I am not sure how to use the component. Could you
please elaborate?

Also, this is not available in Solr 1.3. Any equivalent of it in 1.3?

Thanks,
Sushil


ryantxu wrote:
 
 not exactly what you are asking for, but check:
 http://wiki.apache.org/solr/StatsComponent
 
 this will at least tell you the max/min versionId...   right now it  
 only works with numeric values, so it won't help for timestamp.
 
 ryan
 

-- 
View this message in context: 
http://www.nabble.com/Retrieve-documents-that-contain-max-value-for-a-field-tp21175643p21203697.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Delta DataImport is not picking the modified value in DB

[Noble Paul നോബിള്‍ नोब्ळ्]

On Mon, Dec 29, 2008 at 5:11 PM, Manupriya manupriya.si...@gmail.com wrote:

 Hi Shalin,

 I am using apache-solr-1.3.0 in my project. I thought it to be the latest
 release. Does it not support delta import?
It should, 1.3 of solr was the first realease of DIH. A lot of
features got added to DIH since then. The nightly build of DIH is
compatible with Solr 1.3. BTW 1.3 supports delta import.


 Thanks,
 Manu


 Shalin Shekhar Mangar wrote:

 Which version of Solr are you using? The deltaImportQuery feature was
 added
 recently, you'd need a nightly build for it to work.

 On Mon, Dec 29, 2008 at 4:57 PM, Manupriya
 manupriya.si...@gmail.comwrote:


 Hi Noble,

 I tried with the deltaImportQuery as well. But still I am getting the
 same
 exception on the server console -

 Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
 doDeltaImport

 SEVERE: Delta Import Failed
 java.lang.NullPointerException
at

 org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
at

 org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
at

 org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at

 org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
at
 org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
at
 org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
at

 org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
at

 org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
at

 org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)

 Another thing, I do have proper data in the DB.

 Thanks,
 Manu


 Noble Paul നോബിള്‍ नोब्ळ् wrote:
 
  The same example with deltaImportQuery would look as follows
  dataConfig
 dataSource driver=com.mysql.jdbc.Driver
  url=jdbc:mysql://localhost/solr_demo_db user=root password=root
 /
 document name=products
 entity name=item pk=ID query=select * from item
 deltaQuery=select id from item where last_modified 
  '${dataimporter.last_index_time}'
  deltaImportQuery=select * from item where ID=${dataimporter.delta.ID}
 
 field column=NAME name=name /
 field column=NAME name=nameSort /
 field column=NAME name=alphaNameSort /
 
 entity name=feature pk=ITEM_ID
 query=select DESCRIPTION from FEATURE where
  ITEM_ID='${item.ID}'
 deltaQuery=select ITEM_ID from FEATURE where
  last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ID from item where
  ID=${feature.ITEM_ID}
 field name=features column=DESCRIPTION /
 /entity
 
 entity name=item_category pk=ITEM_ID, CATEGORY_ID
 query=select CATEGORY_ID from item_category where
  ITEM_ID='${item.ID}'
 deltaQuery=select ITEM_ID, CATEGORY_ID from
  item_category where last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ID from item where
  ID=${item_category.ITEM_ID}
 entity name=category pk=ID
 query=select DESCRIPTION from category where ID
 =
  '${item_category.CATEGORY_ID}'
 deltaQuery=select ID from category where
  last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ITEM_ID, CATEGORY_ID
 from
  item_category where CATEGORY_ID=${category.ID}
 field column=description name=cat /
 /entity
 /entity
 /entity
 /document
  /dataConfig
 
  On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള്‍ नोब्ळ्
  noble.p...@gmail.com wrote:
  did you put in the data?
 
  On Mon, Dec 29, 2008 at 3:07 PM, Manupriya manupriya.si...@gmail.com
  wrote:
 
  Hi Shalin,
 
  I am actually trying out the example provided at
  http://wiki.apache.org/solr/DataImportHandler.
 
  My db-data-config.xml is as following -
 
  dataConfig
 dataSource driver=com.mysql.jdbc.Driver
  url=jdbc:mysql://localhost/solr_demo_db user=root password=root
 /
 document name=products
 entity name=item pk=ID query=select * from item
 deltaQuery=select id from item where last_modified 
  '${dataimporter.last_index_time}'
 field column=NAME name=name /
 field column=NAME name=nameSort /
 field column=NAME name=alphaNameSort /
 
 entity name=feature pk=ITEM_ID
 query=select DESCRIPTION from FEATURE where
  ITEM_ID='${item.ID}'
 deltaQuery=select ITEM_ID from FEATURE where
  last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ID from item 

Units for maxFieldLength

[Alan May]

Hi,

I'd love to get more details about the maxFieldLength attribute in the
solrConfig.xml.

Is this length in characters or bytes or some other measure?  If chars, how
are double byte characters handled?  If an add request is sent with a field
that exceeds this limit what is the behavior?  (Is it truncated or does it
throw an exception?)

Kind Regards,
Alan

Re: Units for maxFieldLength

[Otis Gospodnetic]

Hi Alan,

That number represents the number of tokens.  Extra long field values get 
silently truncated.

 
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
 From: Alan May j.alan@gmail.com
 To: solr-user@lucene.apache.org
 Sent: Monday, December 29, 2008 1:09:37 PM
 Subject: Units for maxFieldLength
 
 Hi,
 
 I'd love to get more details about the maxFieldLength attribute in the
 solrConfig.xml.
 
 Is this length in characters or bytes or some other measure?  If chars, how
 are double byte characters handled?  If an add request is sent with a field
 that exceeds this limit what is the behavior?  (Is it truncated or does it
 throw an exception?)
 
 Kind Regards,
 Alan

Re: High response times after snapshot install

[Otis Gospodnetic]

Hello Victor,

The high response times could be caused by long and intense warming of the new 
searcher(s), which could be causing high IO (it sounds like all instances share 
the same index on the same file system) and high CPU usage (lots of cache items 
being copied from the old searcher).


Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
 From: Victor Hogemann vic...@ideais.com.br
 To: Solr-User solr-user@lucene.apache.org
 Sent: Monday, December 29, 2008 7:53:44 AM
 Subject: High response times after snapshot install
 
 Greetings,
 
 Before anything I'd like to thank the Solr team for such great piece of 
 software! Thank you.
 
 Now... back to my problem. I'm experiencing slowdowns on a Solr 1.3 multicore 
 installation after a snapshot install, response times go up from 5ms to above 
 5sec, and don't go down unless we restart the servers.
 
 Our index has more than 20 fields, and the slave servers got some 50hits/sec. 
 We're running it on Weblogic, with 6 Weblogic Nodes/Solr instances per 
 machine, 
 sharing the same index on the filesystem.
 
 Any insight on what is happening?
 
 Thanks.
 
 Victor Hogemann - Ideais Tecnologia
 
 Tel (RJ) +55 +21 3553-1301 (R217)
 Tel (SP) +55 +11 4063-0443 (R217)
 Fax +55 +21 3553-1302
 
 Rua da Assembléia, nº 98, 9º andar
 Centro | Rio de Janeiro | Brasil
 CEP 20011-000

Re: Multiple language support

[Otis Gospodnetic]

Hi,

The problem is that a single document (and even a field in your case) is 
multilingual.  Ideally you'd detect different languages within a document and 
apply a different tokenizer/filter to different parts of the field.  So the 
first part would be handled as EN, and the second part as Chinese.  At search 
time you would have to find the language of the query one way or the other, and 
again apply the appropriate analyzer.  If the right analyzer is applied, you 
could match even this multilingual field.  None of the existing 
Analyzers/tokenizers/filters are capable of handling a single piece of text in 
multiple languages, so you will have to create a custom analyzer that is smart 
enough to do that.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
 From: Deshpande, Mukta mudes...@ptc.com
 To: solr-user@lucene.apache.org
 Sent: Monday, December 29, 2008 4:52:19 AM
 Subject: Multiple language support 
 
 Hi All,
 
 I have a multiple language supporting schema in which there is a separate 
 field 
 for every language.
 
 I have a field product_name to store product name and its description that 
 can 
 be in any user preferred language. 
 This can be stored in fields product_name_EN if user prefers English 
 language, 
 product_name_SCH if user prefers Simplified Chinese language.
 The WhitespaceTokenizerFactory and filter EnglishPorterFilterFactory are 
 applied 
 on product_name_EN.
 The CJKAnalyzer and CJKTokenizer are applied on product_name_SCH.
 
 e.g. Value can be : ElectrolyticCapacitor - 被对立的电容器以价值220µF
 
 Now my problem is: Which field do I store the above value?
 product_name_EN OR product_name_SCH OR should it be something else?
 
 How do I find out which analyzers should get applied for this field.
 
 Did any one face a similar situation before. 
 Please help ASAP.
 
 Thanks,
 ~Mukta

Re: Custom Search Results

[Otis Gospodnetic]

Johnny,

There are lots of examples of SearchComponents in Solr itself.  Have a look:

o...@lesina:~/workspace/asf-solr$ ffjg -l extends SearchComponent
./src/java/org/apache/solr/handler/component/FacetComponent.java
./src/java/org/apache/solr/handler/component/QueryComponent.java
./src/java/org/apache/solr/handler/component/MoreLikeThisComponent.java
./src/java/org/apache/solr/handler/component/HighlightComponent.java
./src/java/org/apache/solr/handler/component/SpellCheckComponent.java
./src/java/org/apache/solr/handler/component/TermVectorComponent.java
./src/java/org/apache/solr/handler/component/DebugComponent.java
./src/java/org/apache/solr/handler/component/QueryElevationComponent.java
./src/java/org/apache/solr/handler/component/StatsComponent.java
./src/java/org/apache/solr/handler/component/TermsComponent.java
./src/java/org/apache/solr/core/SolrCore.java


Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
 From: Johnny X jonathanwel...@gmail.com
 To: solr-user@lucene.apache.org
 Sent: Saturday, December 27, 2008 2:23:32 PM
 Subject: Re: Custom Search Results
 
 
 Cheers Otis, that seems to be what I'm looking for.
 
 While the Solr pages look helpful, are there any implementation examples of
 SearchComponents available that I could base what I'm doing this on, or are
 there any tutorials about?
 
 I'm not really familiar with Java either, but I suppose if I had something
 to base what I need to do on I could pick up what I need for the
 SearchComponent from that if you see what I mean.
 
 Thanks again!
 
 
 
 Otis Gospodnetic wrote:
  
  Hi Johnny,
  
  It sounds like you wish to post-process and analyze results and categorize
  each document in the result set based on presence and positioning of
  certain key words in the document.  If that's so, I think what you are
  after is a custom SearchComponent:
  http://wiki.apache.org/solr/SearchComponent
  
  
  Please do note that post-processing large result sets can get expensive
  and will hurt performance.
   Otis
  --
  Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
  
  
  
  - Original Message 
  From: Johnny X 
  To: solr-user@lucene.apache.org
  Sent: Thursday, December 25, 2008 9:29:15 PM
  Subject: Custom Search Results
  
  
  Hi there,
  
  
  N00b here, so if you could explain in simple terms I'd appreciate it.
  
  I'm trying to create what I want to call a 'filter' to search through a
  bunch of indexed documents. The idea of this filter is search for very
  specific things in each of the indexed documents to catagorise them into
  a
  particular field if they're relevant.
  
  In this case the search is going to involve a lot of specific information
  about the occurance of certain words, their placement in the document,
  other
  words around them etc so I don't know if I need to develop this using
  some
  form of custom advanced query or produce some form of new componenet
  (analyzer/filter???).
  
  Thanks for any help!
  -- 
  View this message in context: 
  http://www.nabble.com/Custom-Search-Results-tp21171303p21171303.html
  Sent from the Solr - User mailing list archive at Nabble.com.
  
  
  
 
 -- 
 View this message in context: 
 http://www.nabble.com/Custom-Search-Results-tp21171303p21186849.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Hi w.r.t solr 1.4

[vinay kumar kaku]

Hi,  is there a stable build of solr 1.4 that i could grab from? vinay
_
Send e-mail faster without improving your typing skills.
http://windowslive.com/online/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_speed_122008

Re: Hi w.r.t solr 1.4

[Otis Gospodnetic]

Nightly build, please.


Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
 From: vinay kumar kaku vk...@hotmail.com
 To: solr-user@lucene.apache.org
 Sent: Monday, December 29, 2008 2:05:42 PM
 Subject: Hi w.r.t solr 1.4
 
 
 Hi,  is there a stable build of solr 1.4 that i could grab from? vinay
 _
 Send e-mail faster without improving your typing skills.
 http://windowslive.com/online/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_speed_122008

Re: DataImportHandler - The field :xyz present in DataConfig does not have a counterpart in Solr Schema

[Rakesh Sinha]

Oops. The fields were out of sync with those in schema.xml .

Looking at the dynamic field name configuration in schema.xml - my
dataconfig.xml file looks as follows.


dataConfig
dataSource driver=com.mysql.jdbc.Driver url=jdbc:mysql://myserver/mydb
user=username password=password /
document name=user
entity name=item
query=select id, firstname, lastname from user
field column=id  name=id /
field column=firstname name=firstname_s /
field column=lastname name=lastname_s /
/entity
/document
 /dataConfig

The naming of fields with suffix ( _s ) , as per the dynamic field
naming conventions fixed the issue.



On Mon, Dec 29, 2008 at 1:36 PM, Rakesh Sinha rakesh.use...@gmail.com wrote:
 Hi -
  I am testing around with the full - import functionality of Data
 Import Handler.  My dataconfig file looks as follows.


 dataConfig
dataSource driver=com.mysql.jdbc.Driver url=jdbc:mysql://myserver/mydb
user=username password=password /
document name=user
entity name=item
query=select id, firstname, lastname from user
field column=id /
field column=firstname /
field column=lastname /
/entity
/document
 /dataConfig

 In solrconfig.xml - I am setting the access for DIH as follows.

  requestHandler name=/dataimport
 class=org.apache.solr.handler.dataimport.DataImportHandler
lst name=defaults
  str name=configdata-config.xml/str
/lst
  /requestHandler


 When I try to access the deployed web-app ( even before hitting
 full-import functionality using command ) - I am getting the following
 sequence of errors.

 The field :lastname present in DataConfig does not have a counterpart
 in Solr Schema
 The field :firstname present in DataConfig does not have a counterpart
 in Solr Schema

 The config file is very similar to what is given in the DIH wiki.

 Curious, what gives ?

how large can the index be?

[Antonio Eggberg]

Hi,

We are running successfully a solr index of 3 million docs. I have just been 
informed that our index size will increase to 50 million. I been going through 
the doc 

http://wiki.apache.org/solr/DistributedSearch

Seems like we will loose out on the date facet and some more other stuff that 
we use. which is important to us. So far we been using 1 index and 1 machine. 

Can I still stick with my 1 index but have many query servers? We don't update 
our index so often this are rather static data. Over the past year we have 
updated the index data a total of 3 times and about 300 records :)

Can someone provide some idea how/what should I do to deal with new datasets?.

Thanks for your help.


  __
Går det långsamt? Skaffa dig en snabbare bredbandsuppkoppling. 
Sök och jämför priser hos Kelkoo.
http://www.kelkoo.se/c-100015813-bredband.html?partnerId=96914325

Re: how large can the index be?

[Otis Gospodnetic]

Hi Antonio,

Besides thinking in terms of documents, you also need to think in terms of 
index size on the file system vs. the amount of RAM your search 
application/server can use.  50M documents may be doable on a single server if 
those documents are not too large and you have sufficient RAM.  It gets even 
better if your index doesn't change very often and if you can get decent hit 
ratios on the various Solr caches.

If you are indexing largish documents, or even something as small as an average 
web page, 50M docs may be too much on a commodity box (say dual core 8 GB RAM 
box)

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
 From: Antonio Eggberg antonio_eggb...@yahoo.se
 To: solr-user@lucene.apache.org
 Sent: Monday, December 29, 2008 3:41:48 PM
 Subject: how large can the index be?
 
 Hi,
 
 We are running successfully a solr index of 3 million docs. I have just been 
 informed that our index size will increase to 50 million. I been going 
 through 
 the doc 
 
 http://wiki.apache.org/solr/DistributedSearch
 
 Seems like we will loose out on the date facet and some more other stuff that 
 we 
 use. which is important to us. So far we been using 1 index and 1 machine. 
 
 Can I still stick with my 1 index but have many query servers? We don't 
 update 
 our index so often this are rather static data. Over the past year we have 
 updated the index data a total of 3 times and about 300 records :)
 
 Can someone provide some idea how/what should I do to deal with new datasets?.
 
 Thanks for your help.
 
 
   __
 Går det långsamt? Skaffa dig en snabbare bredbandsuppkoppling. 
 Sök och jämför priser hos Kelkoo.
 http://www.kelkoo.se/c-100015813-bredband.html?partnerId=96914325

Re: Dismax Minimum Match/Stopwords Bug

[Matthew Runo]

Hmm, that makes sense to me - however I still think that even if we  
have mm set to 2 and we have the 7449078 it should still match  
7449078 in a productId field (it does not: http://zeta.zappos.com/search?department=term=the+7449078) 
. This seems like it works against the way one would reasonably expect  
it to - that stopwords shouldn't impact the counts for mm (so, the  
7449078 would count as 1 term for mm since the is a stopword).


Would there be a way around this? Could we possibly get it reworked?  
What would the downside to that be?


We have people asking for the north to return results from a brand  
called the north face - but it doesn't, and can't, because of this  
mm issue.


Thanks for your time helping us with this issue =)

Matthew Runo
Software Engineer, Zappos.com
mr...@zappos.com - 702-943-7833

On Dec 20, 2008, at 10:45 AM, Chris Hostetter wrote:



: Would this mean that, for example, if we wanted to search  
productId (long)
: we'd need to make a field type that had stopwords in it rather  
than simply

: using (long)?

not really ... that's kind of a special usecase.  if someone  
searches for
a productId that's usually *all* they search for (1 chunk of input  
fro

mthe query parser) so it's mandatory and produces a clause across all
fields.  It doesn't matter if the other fields have stopwords --  
even if

the productId happens to be a stop word, that just means it doesn't
produce a clause on those stop worded fields, but it will will on  
your

productId field.

The only case where you might get into trouble is if someone  
searches for

the 123456 ... now you have two chunks of input, so the mm param
comes into play you have no stopwords on your productId field so both
the and 123456 produce clauses, but the isn't going to be  
found in

your productId field, and because of stopwords it doens't exist in the
other fields at all ... so you don't match anything.

FWIW: if i remember right if you want to put numeric fields in the  
qf, i
think you need *all* of them to be numeric and all of your input  
needs to

be numeric, or you get exceptions from the FieldType (not the dismax
parser) when people search for normal words.   i always copyField
productId into a productId_str field for purposes like this.


-Hoss

Local Solr - Error creating document with DIH

[MapYours]

I have my data import handler set up correctly it connects and streams
results returned from the database (sqlserver 2005) using JTDS driver.

However, no documents are indexed:
Dec 29, 2008 2:03:07 PM org.apache.solr.handler.dataimport.SolrWriter upload
WARNING: Error creating document :
SolrInputDocumnt[{incident_num=incident_num(1.0)={08518194 },
incident_date=incident_date(1.0)={2008-12-29 00:00:00.0},
location=location(1.0)={4300BLK SW 173RD AVE 
}, lat=lat(1.0)={45.4882}, lng=lng(1.0)={-122.85455}}]
java.lang.ClassCastException: java.lang.Double cannot be cast to
java.lang.String
at
com.pjaol.search.solr.update.LocalUpdaterProcessor.processAdd(LocalUpdateProcessorFactory.java:136)
at
org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:69)
at
org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:288)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:319)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:178)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:136)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)

Here is my schema:
fields
   field name=incident_num type=text indexed=true stored=true/
   field name=incident_date type=date indexed=true stored=true/
   field name=location type=text indexed=true stored=true /
   
!-- local lucene field types --
   field name=lat type=sdouble indexed=true stored=true/
   field name=lng type=sdouble indexed=true stored=true/
   
dynamicField name=_local* type=sdouble indexed=true
stored=true/
 /fields

 !-- field to use to determine and enforce document uniqueness. --
 uniqueKeyincident_num/uniqueKey

 !-- field for the QueryParser to use when an explicit fieldname is absent
--
 defaultSearchFieldlocation/defaultSearchField

-- 
View this message in context: 
http://www.nabble.com/Local-Solr---Error-creating-document-with-DIH-tp21211133p21211133.html
Sent from the Solr - User mailing list archive at Nabble.com.

can i reuse the index built with 1.3 with current solr from trunk

[vinay kumar kaku]

Hi,  I have an index thats built using 1.3 (before official release). can i 
re-use same data with current solr i get from trunk? what are the things that i 
need to keep in mind.vinay
_
Send e-mail anywhere. No map, no compass.
http://windowslive.com/oneline/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_122008

Re: http internal error if i enable debugQuery=on

[Otis Gospodnetic]

Hi,

Do you have q=XXX parameter in your URL as well?  Try specifying it.


Otis --
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
 From: vinay kumar kaku vk...@hotmail.com
 To: solr-user@lucene.apache.org
 Sent: Monday, December 29, 2008 6:09:24 PM
 Subject: http internal error if i enable debugQuery=on
 
 
 Hi,  i got solr on 23rd dec and using for my application. i have a filter 
 fq=-filtername[1+TO+*] all works fine until i say debugQuery=on i get below 
 error from my tomcat catalina.out logfile. if i change that filter to say 
 fq=-filtername[1+TO+10]debugQuery=on it works fine.HTTP Status 500 - 
 java.lang.NullPointerException: value cannot be null 
 java.lang.RuntimeException: 
 java.lang.NullPointerException: value cannot be null at 
 org.apache.solr.search.QueryParsing.toString(QueryParsing.java:469) at 
 org.apache.solr.handler.component.DebugComponent.process(DebugComponent.java:75)
  
 at 
 org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:179)
  
 at 
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
  
 at 
 org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:233)
  
 at org.apache.solr.core.SolrCore.execute(SolrCore.java:1325) at 
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
  
 at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
  
 at 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
  
 at 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
  
 at 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
  
 at 
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
  
 at 
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) 
 at 
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) 
 at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568) 
 at 
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
  
 at 
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) 
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) 
 at 
 org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
  
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) at 
 java.lang.Thread.run(Thread.java:619) Caused by: 
 java.lang.NullPointerException: 
 value cannot be null at org.apache.lucene.document.Field.(Field.java:275) 
 at org.apache.lucene.document.Field.(Field.java:251) at 
 org.apache.solr.search.QueryParsing.writeFieldVal(QueryParsing.java:306) at 
 org.apache.solr.search.QueryParsing.toString(QueryParsing.java:338) at 
 org.apache.solr.search.QueryParsing.toString(QueryParsing.java:401) at 
 org.apache.solr.search.QueryParsing.toString(QueryParsing.java:466) ... 20 
 more 
 thanks,vinay
 _
 Send e-mail anywhere. No map, no compass.
 http://windowslive.com/oneline/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_122008

Re: how large can the index be?

[Otis Gospodnetic]

What you have below is not really what we call Distributed Search, but more 
of Query Load Balancing.  Yes, the diagram below will work IF a single Solr 
box (A or B) can really handle a full 50M doc index.  Of course handle can be 
fuzzy.  That is, you could have a large index on a Solr box and it will 
handle it - nothing will crash, nothing will die, it's just that it may not 
be able to handle it well enough - that is, the queries may take longer than 
you'd like.

NFS mounting an index directory is a separate story and very often a bad idea, 
again because of performance.


Otis --
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
 From: Antonio Eggberg antonio_eggb...@yahoo.se
 To: solr-user@lucene.apache.org
 Sent: Monday, December 29, 2008 4:19:23 PM
 Subject: Re: how large can the index be?
 
 Thanks you very much for your answer.
 
 I was afraid of that the each document has about 20 fields.. As you pointed 
 out 
 it will slow down. Anyway I am thinking is it not possible to do the 
 following:
 
 Load Balancer 
  |
 Solr A, Solr B, ...
  |
   one index
 
 So I send 50% query to Solr A, 50% to Solr B and so forth.. is this not good? 
 Also to add The index will be like a mounted drive to the solr boxes... On 
 the 
 above do I really need to worry about Solr Master, Solr Slave? It probably 
 solve 
 my load but I think query speed will be slow...
 
 Just curious anyone using distributed search in production?
 
 Cheers
 
 
 
 --- Den mån 2008-12-29 skrev Otis Gospodnetic :
 
  Från: Otis Gospodnetic 
  Ämne: Re: how large can the index be?
  Till: solr-user@lucene.apache.org
  Datum: måndag 29 december 2008 21.53
  Hi Antonio,
  
  Besides thinking in terms of documents, you also need to
  think in terms of index size on the file system vs. the
  amount of RAM your search application/server can use.  50M
  documents may be doable on a single server if those
  documents are not too large and you have sufficient RAM.  It
  gets even better if your index doesn't change very often
  and if you can get decent hit ratios on the various Solr
  caches.
  
  If you are indexing largish documents, or even something as
  small as an average web page, 50M docs may be too much on a
  commodity box (say dual core 8 GB RAM box)
  
  Otis
  --
  Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
  
  
  
  - Original Message 
   From: Antonio Eggberg 
   To: solr-user@lucene.apache.org
   Sent: Monday, December 29, 2008 3:41:48 PM
   Subject: how large can the index be?
   
   Hi,
   
   We are running successfully a solr index of 3 million
  docs. I have just been 
   informed that our index size will increase to 50
  million. I been going through 
   the doc 
   
   http://wiki.apache.org/solr/DistributedSearch
   
   Seems like we will loose out on the date facet and
  some more other stuff that we 
   use. which is important to us. So far we been using 1
  index and 1 machine. 
   
   Can I still stick with my 1 index but have many query
  servers? We don't update 
   our index so often this are rather static data. Over
  the past year we have 
   updated the index data a total of 3 times and about
  300 records :)
   
   Can someone provide some idea how/what should I do to
  deal with new datasets?.
   
   Thanks for your help.
   
   

  __
   Går det långsamt? Skaffa dig en snabbare
  bredbandsuppkoppling. 
   Sök och jämför priser hos Kelkoo.
  
  http://www.kelkoo.se/c-100015813-bredband.html?partnerId=96914325
 
 
   __
 Låna pengar utan säkerhet. Jämför vilkor online hos Kelkoo.
 http://www.kelkoo.se/c-100390123-lan-utan-sakerhet.html?partnerId=96915014

Re: can i reuse the index built with 1.3 with current solr from trunk

[Otis Gospodnetic]

Vinay,

Please email only one list - solr-user.
The answer is positive, but it's always best to back up your index just in case.


Otis --
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
 From: vinay kumar kaku vk...@hotmail.com
 To: solr-...@lucene.apache.org; solr-user@lucene.apache.org
 Sent: Monday, December 29, 2008 7:38:57 PM
 Subject: can i reuse the index built with 1.3 with current solr from trunk
 
 
 Hi,  I have an index thats built using 1.3 (before official release). can i 
 re-use same data with current solr i get from trunk? what are the things that 
 i 
 need to keep in mind.vinay
 _
 Send e-mail anywhere. No map, no compass.
 http://windowslive.com/oneline/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_122008

Re: DataImportHandler - The field :xyz present in DataConfig does not have a counterpart in Solr Schema

[Noble Paul നോബിള്‍ नोब्ळ्]

These are warning messages . They do not stop anything in DIH
It is very common that users make typos and that may be hard to debug.
So the messages are spit out

On Tue, Dec 30, 2008 at 12:06 AM, Rakesh Sinha rakesh.use...@gmail.com wrote:
 Hi -
  I am testing around with the full - import functionality of Data
 Import Handler.  My dataconfig file looks as follows.


 dataConfig
dataSource driver=com.mysql.jdbc.Driver url=jdbc:mysql://myserver/mydb
user=username password=password /
document name=user
entity name=item
query=select id, firstname, lastname from user
field column=id /
field column=firstname /
field column=lastname /
/entity
/document
 /dataConfig

 In solrconfig.xml - I am setting the access for DIH as follows.

  requestHandler name=/dataimport
 class=org.apache.solr.handler.dataimport.DataImportHandler
lst name=defaults
  str name=configdata-config.xml/str
/lst
  /requestHandler


 When I try to access the deployed web-app ( even before hitting
 full-import functionality using command ) - I am getting the following
 sequence of errors.

 The field :lastname present in DataConfig does not have a counterpart
 in Solr Schema
 The field :firstname present in DataConfig does not have a counterpart
 in Solr Schema

 The config file is very similar to what is given in the DIH wiki.

 Curious, what gives ?




-- 
--Noble Paul

Re: Local Solr - Error creating document with DIH

[Noble Paul നോബിള്‍ नोब्ळ्]

apparently the local solr expects everything as string. I guess we
must raise an issue with them.
meanwhile you can convert those double fields to String using
TemplateTransformer

field column=lat template=${your_entity_name.lat}/
  field column=lng template=${your_entity_name.lng}/



On Tue, Dec 30, 2008 at 5:49 AM, MapYours mapyo...@gmail.com wrote:

 I have my data import handler set up correctly it connects and streams
 results returned from the database (sqlserver 2005) using JTDS driver.

 However, no documents are indexed:
 Dec 29, 2008 2:03:07 PM org.apache.solr.handler.dataimport.SolrWriter upload
 WARNING: Error creating document :
 SolrInputDocumnt[{incident_num=incident_num(1.0)={08518194 },
 incident_date=incident_date(1.0)={2008-12-29 00:00:00.0},
 location=location(1.0)={4300BLK SW 173RD AVE
 }, lat=lat(1.0)={45.4882}, lng=lng(1.0)={-122.85455}}]
 java.lang.ClassCastException: java.lang.Double cannot be cast to
 java.lang.String
at
 com.pjaol.search.solr.update.LocalUpdaterProcessor.processAdd(LocalUpdateProcessorFactory.java:136)
at
 org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:69)
at
 org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:288)
at
 org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:319)
at
 org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:178)
at
 org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:136)
at
 org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
at
 org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386)
at
 org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)

 Here is my schema:
 fields
   field name=incident_num type=text indexed=true stored=true/
   field name=incident_date type=date indexed=true stored=true/
   field name=location type=text indexed=true stored=true /

 !-- local lucene field types --
   field name=lat type=sdouble indexed=true stored=true/
   field name=lng type=sdouble indexed=true stored=true/

dynamicField name=_local* type=sdouble indexed=true
 stored=true/
  /fields

  !-- field to use to determine and enforce document uniqueness. --
  uniqueKeyincident_num/uniqueKey

  !-- field for the QueryParser to use when an explicit fieldname is absent
 --
  defaultSearchFieldlocation/defaultSearchField

 --
 View this message in context: 
 http://www.nabble.com/Local-Solr---Error-creating-document-with-DIH-tp21211133p21211133.html
 Sent from the Solr - User mailing list archive at Nabble.com.





-- 
--Noble Paul

Problem with WT parameter when upgrading from Solr1.2 to solr1.3

[Pooja Verlani]

Hi,



I just upgraded my system from Solr 1.2 to Solr 1.3. I am using the same
plugin for the queryResponseWriter that I used in Solr1.2. Problem here is
that when I am using *wt* parameter as the plugin name with full package
then I don't get the response which I used to get in 1.2 and when I don't
give WT parameter, I get the perfect response from the default
XMLResponseWriter as expected. Also the above problem occurs only when we
use Shards. This occurs only when I am using distributed query on multiple
shards, on individual shards it working fine i.e. when we use /select clause
on individual shards.
(http://localhost:8081/solr/select?q=%22indian%20railways%22qt=modifiedfl=*,scorewt=customhl=true).

http://ec2-75-101-179-1.compute-1.amazonaws.com:8081/solr/select?q=%22indian%20railways%22qt=blogsfl=*,scorewt=blogshl=true%29.Please


On individual shards, the custom responsewriters are working absolutely fine
but not with combining shards or using /distrib/

http://ec2-75-101-179-1.compute-1.amazonaws.com:8081/solr/select?q=%22indian%20railways%22qt=blogsfl=*,scorewt=blogshl=true%29.Please
http://localhost:8081/solr/distrib?q=%22indian%20railways%22qt=modifiedfl=*,scorewt=customhl=truehttp://ec2-75-101-179-1.compute-1.amazonaws.com:8081/solr/distrib?q=%22indian%20railways%22qt=blogsfl=*,scorewt=blogspdrmhl=true

http://ec2-75-101-179-1.compute-1.amazonaws.com:8081/solr/select?q=%22indian%20railways%22qt=blogsfl=*,scorewt=blogshl=true%29.Please
http://ec2-75-101-179-1.compute-1.amazonaws.com:8081/solr/distrib?q=%22indian%20railways%22qt=blogsfl=*,scorewt=blogspdrmhl=true

Please help.





This is part of solrconfig.xml



   requestHandler name=/distrib class=solr.SearchHandler

 lst name=defaults

   str name=shardsx,y,z/str

 /lst

   /requestHandler



*queryResponseWriter name=standard default=true
class=FirstResponseWriter/*

*queryResponseWriter name=custom class=SecondResponseWriter/*

* *


http://ec2-75-101-179-1.compute-1.amazonaws.com:8081/solr/distrib?q=%22indian%20railways%22qt=blogsfl=*,scorewt=blogspdrmhl=true
**



Thanks  Regards,

Almas