Hi Noble,
I tried with the deltaImportQuery as well. But still I am getting the same
exception on the server console -
Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
doDeltaImport
SEVERE: Delta Import Failed
java.lang.NullPointerException
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
at
org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
at
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
Another thing, I do have proper data in the DB.
Thanks,
Manu
Noble Paul നോബിള് नोब्ळ् wrote:
>
> The same example with deltaImportQuery would look as follows
> <dataConfig>
> <dataSource driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root" />
> <document name="products">
> <entity name="item" pk="ID" query="select * from item"
> deltaQuery="select id from item where last_modified >
> '${dataimporter.last_index_time}'"
> deltaImportQuery="select * from item where ID=${dataimporter.delta.ID}"
>>
> <field column="NAME" name="name" />
> <field column="NAME" name="nameSort" />
> <field column="NAME" name="alphaNameSort" />
>
> <entity name="feature" pk="ITEM_ID"
> query="select DESCRIPTION from FEATURE where
> ITEM_ID='${item.ID}'"
> deltaQuery="select ITEM_ID from FEATURE where
> last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ID from item where
> ID=${feature.ITEM_ID}">
> <field name="features" column="DESCRIPTION" />
> </entity>
>
> <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
> query="select CATEGORY_ID from item_category where
> ITEM_ID='${item.ID}'"
> deltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ID from item where
> ID=${item_category.ITEM_ID}">
> <entity name="category" pk="ID"
> query="select DESCRIPTION from category where ID =
> '${item_category.CATEGORY_ID}'"
> deltaQuery="select ID from category where
> last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where CATEGORY_ID=${category.ID}">
> <field column="description" name="cat" />
> </entity>
> </entity>
> </entity>
> </document>
> </dataConfig>
>
> On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള് नोब्ळ्
> <noble.p...@gmail.com> wrote:
>> did you put in the data?
>>
>> On Mon, Dec 29, 2008 at 3:07 PM, Manupriya <manupriya.si...@gmail.com>
>> wrote:
>>>
>>> Hi Shalin,
>>>
>>> I am actually trying out the example provided at
>>> http://wiki.apache.org/solr/DataImportHandler.
>>>
>>> My db-data-config.xml is as following -
>>>
>>> <dataConfig>
>>> <dataSource driver="com.mysql.jdbc.Driver"
>>> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root" />
>>> <document name="products">
>>> <entity name="item" pk="ID" query="select * from item"
>>> deltaQuery="select id from item where last_modified >
>>> '${dataimporter.last_index_time}'">
>>> <field column="NAME" name="name" />
>>> <field column="NAME" name="nameSort" />
>>> <field column="NAME" name="alphaNameSort" />
>>>
>>> <entity name="feature" pk="ITEM_ID"
>>> query="select DESCRIPTION from FEATURE where
>>> ITEM_ID='${item.ID}'"
>>> deltaQuery="select ITEM_ID from FEATURE where
>>> last_modified > '${dataimporter.last_index_time}'"
>>> parentDeltaQuery="select ID from item where
>>> ID=${feature.ITEM_ID}">
>>> <field name="features" column="DESCRIPTION" />
>>> </entity>
>>>
>>> <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
>>> query="select CATEGORY_ID from item_category where
>>> ITEM_ID='${item.ID}'"
>>> deltaQuery="select ITEM_ID, CATEGORY_ID from
>>> item_category where last_modified > '${dataimporter.last_index_time}'"
>>> parentDeltaQuery="select ID from item where
>>> ID=${item_category.ITEM_ID}">
>>> <entity name="category" pk="ID"
>>> query="select DESCRIPTION from category where ID
>>> =
>>> '${item_category.CATEGORY_ID}'"
>>> deltaQuery="select ID from category where
>>> last_modified > '${dataimporter.last_index_time}'"
>>> parentDeltaQuery="select ITEM_ID, CATEGORY_ID
>>> from
>>> item_category where CATEGORY_ID=${category.ID}">
>>> <field column="description" name="cat" />
>>> </entity>
>>> </entity>
>>> </entity>
>>> </document>
>>> </dataConfig>
>>> ---------------------------------------------------------------------------------------------------
>>>
>>> My DB structure is as following -
>>>
>>> DROP TABLE IF EXISTS solr_demo_db.item;
>>> CREATE TABLE `item` (
>>> `ID` int(2) NOT NULL auto_increment,
>>> `name` varchar(100) default NULL,
>>> `manu` varchar(20) default NULL,
>>> `weight` varchar(20) default NULL,
>>> `price` varchar(20) default NULL,
>>> `popularity` varchar(2) default NULL,
>>> `includes` varchar(10) default NULL,
>>> `last_modified` datetime default NULL,
>>> PRIMARY KEY (`ID`)
>>> ) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
>>>
>>> DROP TABLE IF EXISTS solr_demo_db.feature;
>>> CREATE TABLE `feature` (
>>> `DESCRIPTION` varchar(100) default NULL,
>>> `ITEM_ID` int(2) NOT NULL,
>>> `last_modified` datetime default NULL,
>>> PRIMARY KEY (`ITEM_ID`)
>>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>>>
>>> DROP TABLE IF EXISTS solr_demo_db.category;
>>> CREATE TABLE `category` (
>>> `ID` int(2) NOT NULL auto_increment,
>>> `DESCRIPTION` varchar(100) default NULL,
>>> `last_modified` date default NULL,
>>> PRIMARY KEY (`ID`)
>>> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
>>>
>>> DROP TABLE IF EXISTS solr_demo_db.item_category;
>>> CREATE TABLE `item_category` (
>>> `ITEM_ID` int(2) NOT NULL default '0',
>>> `CATEGORY_ID` int(2) NOT NULL default '0',
>>> `last_modified` date default NULL,
>>> PRIMARY KEY (`ITEM_ID`,`CATEGORY_ID`)
>>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>>>
>>> Am I doing something wrong here?
>>>
>>> Thanks,
>>> Manu
>>>
>>>
>>> Shalin Shekhar Mangar wrote:
>>>>
>>>> What does your data-config look like? especially the delta query part.
>>>>
>>>> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
>>>> <manupriya.si...@gmail.com>wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am using Delta DataImport feature to partially refresh the indexes.
>>>>>
>>>>> Note - full-import is working perfectly fine. But I need to do
>>>>> delta-import
>>>>> as I do not want to rebuild all the indexes.
>>>>>
>>>>> My database structure is same as mentioned in the example at
>>>>> http://wiki.apache.org/solr/DataImportHandler.
>>>>>
>>>>> http://www.nabble.com/file/p21200498/example-schema.png
>>>>> ---------------------------------------------------------------------------
>>>>> The dataimport.properties has the value as -
>>>>>
>>>>> #Mon Dec 29 14:08:12 IST 2008
>>>>> last_index_time=2008-12-29 14\:08\:12
>>>>> ---------------------------------------------------------------------------
>>>>> I have changed the last_modified for the 'tem' table as current
>>>>> timestamp.
>>>>> And when I query the DB, I get the following result -
>>>>>
>>>>> http://www.nabble.com/file/p21200498/data.jpg
>>>>>
>>>>> -------------------------------------------------------------------------------------------------
>>>>>
>>>>> But when I call delta data-import, it returns me the following
>>>>> response -
>>>>>
>>>>> Server Response:
>>>>>
>>>>> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
>>>>> response format is experimental. It is likely to change in the
>>>>> future.}
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------------------------------------
>>>>> And when I query for data for '*:*', I get the following -
>>>>> Note - Data returned by query doesnt reflect the correct timestamp
>>>>> from
>>>>> database.
>>>>>
>>>>> SolrDocument(1)
>>>>> Field Name & Value: includes=12
>>>>> Field Name & Value: features=[demo feature, demo feature 1]
>>>>> Field Name & Value: price=12.0
>>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>>>> Field Name & Value: manu=manu-12
>>>>> Field Name & Value: sku=1
>>>>> Field Name & Value: id=1
>>>>> Field Name & Value: popularity=1
>>>>> Field Name & Value: weight=12.0
>>>>> SolrDocument(2)
>>>>> Field Name & Value: includes=43
>>>>> Field Name & Value: features=demo feature 2
>>>>> Field Name & Value: price=10.0
>>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>>>> Field Name & Value: manu=Demo - 12
>>>>> Field Name & Value: sku=2
>>>>> Field Name & Value: id=2
>>>>> Field Name & Value: popularity=5
>>>>> Field Name & Value: weight=12.0
>>>>> SolrDocument(3)
>>>>> Field Name & Value: includes=1
>>>>> Field Name & Value: features=demo -3
>>>>> Field Name & Value: price=1.0
>>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>>>> Field Name & Value: manu=manu - 36
>>>>> Field Name & Value: sku=3
>>>>> Field Name & Value: id=3
>>>>> Field Name & Value: popularity=1
>>>>> Field Name & Value: weight=1.0
>>>>> SolrDocument(4)
>>>>> Field Name & Value: includes=2
>>>>> Field Name & Value: features=demo - 4
>>>>> Field Name & Value: price=2.0
>>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>>>> Field Name & Value: manu=manu - 46
>>>>> Field Name & Value: sku=4
>>>>> Field Name & Value: id=4
>>>>> Field Name & Value: popularity=2
>>>>> Field Name & Value: weight=2.0
>>>>> SolrDocument(5)
>>>>> Field Name & Value: includes=3
>>>>> Field Name & Value: features=demo - 5
>>>>> Field Name & Value: price=3.0
>>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>>>> Field Name & Value: manu=manu - 56
>>>>> Field Name & Value: sku=5
>>>>> Field Name & Value: id=5
>>>>> Field Name & Value: popularity=3
>>>>> Field Name & Value: weight=3.0
>>>>> SolrDocument(6)
>>>>> Field Name & Value: includes=4
>>>>> Field Name & Value: features=demo - 66
>>>>> Field Name & Value: price=4.0
>>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>>>> Field Name & Value: manu=manu - 66
>>>>> Field Name & Value: sku=6
>>>>> Field Name & Value: id=6
>>>>> Field Name & Value: popularity=4
>>>>> Field Name & Value: weight=4.0
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------------------
>>>>> My Java code that calls delta import is as follows -
>>>>>
>>>>> public static void main(String[] args) throws IOException,
>>>>> SolrServerException {
>>>>> CommonsHttpSolrServer server = new
>>>>> CommonsHttpSolrServer("http://localhost:8983/solr";);
>>>>> BinaryResponseParser parser = new BinaryResponseParser();
>>>>> server.setParser(parser);
>>>>>
>>>>> ModifiableSolrParams params = new ModifiableSolrParams();
>>>>> params.set("qt", "/dataimport");
>>>>> params.set("command", "delta-import");
>>>>> params.set("commit", "true");
>>>>> params.set("wt","json");
>>>>>
>>>>> try{
>>>>>
>>>>> QueryResponse response = server.query(params);
>>>>> server.commit();
>>>>> System.out.println("Server Response: " + response);
>>>>>
>>>>> SolrQuery query = new SolrQuery("*:*");
>>>>> QueryResponse results = server.query(query);
>>>>>
>>>>> SolrDocumentList list = results.getResults();
>>>>>
>>>>> int index = 1;
>>>>> for (SolrDocument solrDocument : list) {
>>>>>
>>>>> System.out.println("SolrDocument("+index+")");
>>>>> Iterator<Entry<String, Object>> iterator =
>>>>> solrDocument.iterator();
>>>>>
>>>>> while(iterator.hasNext()){
>>>>> System.out.println("Field Name & Value:
>>>>> "+iterator.next());
>>>>> }
>>>>>
>>>>> index++;
>>>>> }
>>>>>
>>>>> }catch(Exception ex){
>>>>>
>>>>> System.out.println("Exception Occured:"+ex);
>>>>> }
>>>>>
>>>>> }
>>>>> ---------------------------------------------------------------------------
>>>>>
>>>>> Does SOLR do some kind of caching? I dont understand as why the
>>>>> updated
>>>>> values are not queried correctly from DB?
>>>>>
>>>>> There are no errors in the logs. But I can see an error on the server
>>>>> console -
>>>>>
>>>>> SEVERE: Delta Import Failed
>>>>> java.lang.NullPointerException
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>>>>> at
>>>>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>>>>> at
>>>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>>>>
>>>>>
>>>>> What can be the reason for this error?
>>>>>
>>>>> Thanks,
>>>>> Manu
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200498.html
>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200786.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>
>
>
> --
> --Noble Paul
>
>
--
View this message in context:
http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21201980.html
Sent from the Solr - User mailing list archive at Nabble.com.
