Re: dataimport.properties is not updated on delta-import
I've finally found the problem causing the delta-import to fail and thought I would post it here for future reference (if someone makes the same mistake I did). I had forgot to collect the id column in the deltaImportQuery. I should, of course, have known this from the log entires about documents not being added because they lacked id fields. My broken query: SELECT column_1, column_2, column_3 FROM table WHERE id = ${dataimporter.delta.id} My working query: SELECT id, column_1, column_2, column_3 FROM table WHERE id = ${dataimporter.delta.id} Thanks all for you help! -- View this message in context: http://lucene.472066.n3.nabble.com/dataimport-properties-is-not-updated-on-delta-import-tp916753p933342.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: dataimport.properties is not updated on delta-import
Please note that Oracle ( or Oracle jdbc driver ) converts column names to upper case eventhough you state them in lower case. If this is the case then try to rewrite your query in the following form select id as id, name as name from table On Thursday, June 24, 2010, warb w...@mail.com wrote: Hello again! Upon further investigation it seems that something is amiss with delta-import after all, the delta-import does not actually import anything (I thought it did when I ran it previously but I am not sure that was the case any longer.) It does complete successfully as seen from the front-end (dataimport?command=delta-import). Also in the logs it is stated the the import was successful (INFO: Delta Import completed successfully), but there are exception pertaining to some documents. The exception message is that the id field is missing (org.apache.solr.common.SolrException: Document [null] missing required field: id). Now, I have checked the column names in the table, the data-config.xml file and the schema.xml file and they all have the column/field names written in lowercase and are even named exactly the same. Do Solr rollback delta-imports if one or more of the documents failed? -- View this message in context: http://lucene.472066.n3.nabble.com/dataimport-properties-is-not-updated-on-delta-import-tp916753p919609.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: dataimport.properties is not updated on delta-import
Hello again! Upon further investigation it seems that something is amiss with delta-import after all, the delta-import does not actually import anything (I thought it did when I ran it previously but I am not sure that was the case any longer.) It does complete successfully as seen from the front-end (dataimport?command=delta-import). Also in the logs it is stated the the import was successful (INFO: Delta Import completed successfully), but there are exception pertaining to some documents. The exception message is that the id field is missing (org.apache.solr.common.SolrException: Document [null] missing required field: id). Now, I have checked the column names in the table, the data-config.xml file and the schema.xml file and they all have the column/field names written in lowercase and are even named exactly the same. Do Solr rollback delta-imports if one or more of the documents failed? -- View this message in context: http://lucene.472066.n3.nabble.com/dataimport-properties-is-not-updated-on-delta-import-tp916753p919609.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: dataimport.properties is not updated on delta-import
Is there any chance that the id field is, indeed, missing for those documents? Does your schema require ID? I've also seen constraints added to a DB that are not retro-active, so even if there is a constraint requiring ID it's still possible that some items in your DB don't have them. A shot in the dark. Erick On Thu, Jun 24, 2010 at 10:21 AM, warb w...@mail.com wrote: Hello again! Upon further investigation it seems that something is amiss with delta-import after all, the delta-import does not actually import anything (I thought it did when I ran it previously but I am not sure that was the case any longer.) It does complete successfully as seen from the front-end (dataimport?command=delta-import). Also in the logs it is stated the the import was successful (INFO: Delta Import completed successfully), but there are exception pertaining to some documents. The exception message is that the id field is missing (org.apache.solr.common.SolrException: Document [null] missing required field: id). Now, I have checked the column names in the table, the data-config.xml file and the schema.xml file and they all have the column/field names written in lowercase and are even named exactly the same. Do Solr rollback delta-imports if one or more of the documents failed? -- View this message in context: http://lucene.472066.n3.nabble.com/dataimport-properties-is-not-updated-on-delta-import-tp916753p919609.html Sent from the Solr - User mailing list archive at Nabble.com.
dataimport.properties is not updated on delta-import
Hello! I am having some difficulties getting dataimport (DIH) to behave correctly in Solr 1.4.0. Indexing itself works just as it is supposed to with both full-import and delta-import adding modified or newly created records to the index. The problem is however that the date and time of the last delta-import is not updated in the dataimport.properites file. The only time the file gets updated is when performing a full-import. Now, this is not a huge problem since delta-import will simply disregard records already imported (due to the primary key), but it seems wasteful to fetch records which have already been added on previous runs. Also, as the database grows the delta-imports will take longer and longer. Does anyone know of anything I might have overlooked or known bugs? Thanks in advance! Johan Andersson -- View this message in context: http://lucene.472066.n3.nabble.com/dataimport-properties-is-not-updated-on-delta-import-tp916753p916753.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: dataimport.properties is not updated on delta-import
Hi, what I have experienced is that the primary key seems to be case sensitive for the delta queries, at least for some jdcd drivers... see http://lucene.472066.n3.nabble.com/Problem-with-DIH-delta-import-on-JDBC-tp763469p765262.html ... so make sure you specify it with the correct case (e.g. ID instead of id) in your db-data-config.xml. Maybe that's the problem... Cheers, Stefan Am 23.06.2010 15:09, schrieb warb: Hello! I am having some difficulties getting dataimport (DIH) to behave correctly in Solr 1.4.0. Indexing itself works just as it is supposed to with both full-import and delta-import adding modified or newly created records to the index. The problem is however that the date and time of the last delta-import is not updated in the dataimport.properites file. The only time the file gets updated is when performing a full-import. Now, this is not a huge problem since delta-import will simply disregard records already imported (due to the primary key), but it seems wasteful to fetch records which have already been added on previous runs. Also, as the database grows the delta-imports will take longer and longer. Does anyone know of anything I might have overlooked or known bugs? Thanks in advance! Johan Andersson -- *** Stefan Moises Senior Softwareentwickler shoptimax GmbH Guntherstraße 45 a 90461 Nürnberg Amtsgericht Nürnberg HRB 21703 GF Friedrich Schreieck Tel.: 0911/25566-25 Fax: 0911/25566-29 moi...@shoptimax.de http://www.shoptimax.de ***