Re: Need help with delta import
I was having similar problem with delta import.
I am using solr 4.6 and making use of ${dih.last_index_time},
${dih.delta.xxx} shorter variable names.
I think the issue in previously discussed posts in the thread lies in
deltaQuery and deltaImportQuery.
if deltaQuery=select *rowId* from MyTable
then deltaImportQuery=select .. from MyTable where
myId='${dih.delta.*rowId*}'.
Note that *rowId* variable name should be in same case in both select clause
of /deltaQuery/ and where clause of /deltaImportQuery /.
br/
This following import configuration tested in solr 4.6 for delta import.
dataConfig
dataSource type=JdbcDataSource driver=com.mysql.jdbc.Driver
url=jdbc:mysql://localhost/product user= password=y
batchSize=-1/
document name=product
entity name=product
query=select prid, catid, srcid, name, uri, brand, year as date
from
Product transformer=DateFormatTransformer
deltaQuery=select prid from Product where year gt;
'${dih.last_index_time}';
deltaImportQuery=select prid, catid, srcid, name, uri, brand,
year as
date from Product where prid='${dih.delta.prid}'
pk=prid
field column=prid name=id/
field column=catid name=catid/
field column=srcid name=srcid/
field column=name name=name/
field column=uri name=uri/
field column=brand name=brand/
field column=date name=date dateformat=-MM-dd/
entity name=now query=select NOW() as now
transformer=DateFormatTransformer
field column=now name=imported_at dateFormat=yy-MM-dd
hh:mm:ss/
/entity
/entity
/document
sureshadapa wrote
I am using below configuration file and The problem is I do not see any
solr documents committed into Solr Core Selector 'db'
When i run full-import,Is give me message.
Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.
Requests: 1, Fetched: 8, Skipped: 0, Processed: 0
When i run delta-import,It gives me message.
Requests: 0, Fetched: 0, Skipped: 0, Processed: 0
solrconfig.xml
==
luceneMatchVersion
4.4
/luceneMatchVersion
requestHandler name=/dataimport
class=org.apache.solr.handler.dataimport.DataImportHandler
lst name=defaults
str name=config
db1-data-config.xml
/str
/lst
/requestHandler
schema.xml
schema name=db version=1.1
field name=solrp_id type=sint indexed=true stored=true
required=true /
field name=solrp_name type=string indexed=true stored=true /
field name=solrp_phone type=text indexed=true stored=true /
field name=solrp_email type=text indexed=true stored=true/
field name=solrp_smsno type=text indexed=true stored=true/
uniqueKey
solrp_id
/uniqueKey
db1-data-config.xml
=
dataConfig
dataSource autoCommit=true batchSize=-1 encoding=UTF-8
convertType=true type=JdbcDataSource driver=com.mysql.jdbc.Driver
url=jdbc:mysql://localhost:3306/suresh user=suresh
password=suresh123/
document
entity name=list pk=PROVIDERSID query=select
providersid,name,phone,email,smsno from providers
deltaImportQuery=select providersid,name,phone,email,smsno from
providers where PROVIDERSID==${dih.delta.PROVIDERSID}
deltaquery=select providersid,name,phone,email,smsno from
providers where modtime=='${dih.last_index_time}'
field column=PROVIDERSID name=solrp_id /
field column=NAME name=solrp_name /
field column=PHONE name=solrp_phone /
field column=EMAIL name=solrp_email /
field column=SMSNO name=solrp_smsno /
/entity
/document
/dataConfig
Changing ${dih.delta.PROVIDERSID} to ${dih.delta.providersid} should work.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4117167.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Need help with delta import
I am using below configuration file and The problem is I do not see any solr
documents committed into Solr Core Selector 'db'
When i run full-import,Is give me message.
Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.
Requests: 1, Fetched: 8, Skipped: 0, Processed: 0
When i run delta-import,It gives me message.
Requests: 0, Fetched: 0, Skipped: 0, Processed: 0
solrconfig.xml
==
luceneMatchVersion4.4/luceneMatchVersion
requestHandler name=/dataimport
class=org.apache.solr.handler.dataimport.DataImportHandler
lst name=defaults
str name=configdb1-data-config.xml/str
/lst
/requestHandler
schema.xml
schema name=db version=1.1
field name=solrp_id type=sint indexed=true stored=true
required=true /
field name=solrp_name type=string indexed=true stored=true /
field name=solrp_phone type=text indexed=true stored=true /
field name=solrp_email type=text indexed=true stored=true/
field name=solrp_smsno type=text indexed=true stored=true/
uniqueKeysolrp_id/uniqueKey
db1-data-config.xml
=
dataConfig
dataSource autoCommit=true batchSize=-1 encoding=UTF-8
convertType=true type=JdbcDataSource driver=com.mysql.jdbc.Driver
url=jdbc:mysql://localhost:3306/suresh user=suresh
password=suresh123/
document
entity name=list pk=PROVIDERSID query=select
providersid,name,phone,email,smsno from providers
deltaImportQuery=select providersid,name,phone,email,smsno from
providers where PROVIDERSID==${dih.delta.PROVIDERSID}
deltaquery=select providersid,name,phone,email,smsno from providers
where modtime=='${dih.last_index_time}'
field column=PROVIDERSID name=solrp_id /
field column=NAME name=solrp_name /
field column=PHONE name=solrp_phone /
field column=EMAIL name=solrp_email /
field column=SMSNO name=solrp_smsno /
/entity
/document
/dataConfig
--
View this message in context:
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4090999.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Need help with delta import
Sorry but I gave up on this issue. I could not resolve it. On Tue, Sep 10, 2013 at 8:24 PM, suren [via Lucene] ml-node+s472066n4089093...@n3.nabble.com wrote: Any update? I am also having the same issue. pls reply. This XML file does not appear to have any style information associated with it. The document tree is shown below. response lst name=responseHeader int name=status0/int int name=QTime7/int /lst lst name=initArgs lst name=defaults str name=configdb-data-config.xml/str /lst /lst str name=commanddelta-import/str str name=statusidle/str str name=importResponse/ lst name=statusMessages str name=Total Requests made to DataSource2/str str name=Total Rows Fetched1/str str name=Total Documents Skipped0/str str name=Delta Dump started2013-09-10 07:46:34/str str name=Identifying Delta2013-09-10 07:46:34/str str name=Deltas Obtained2013-09-10 07:46:35/str str name=Building documents2013-09-10 07:46:35/str str name=Total Changed Documents1/str str name=Total Documents Processed0/str str name=Time taken0:0:1.30/str /lst str name=WARNING This response format is experimental. It is likely to change in the future. /str /response -- If you reply to this email, your message will be added to the discussion below: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4089093.html To unsubscribe from Need help with delta import, click herehttp://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_codenode=4025003code=dW1hamF2YUBnbWFpbC5jb218NDAyNTAwM3wxNDc2MDQyMDE2 . NAMLhttp://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml -- View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4089714.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
Any update? I am also having the same issue. pls reply. This XML file does not appear to have any style information associated with it. The document tree is shown below. response lst name=responseHeader int name=status0/int int name=QTime7/int /lst lst name=initArgs lst name=defaults str name=configdb-data-config.xml/str /lst /lst str name=commanddelta-import/str str name=statusidle/str str name=importResponse/ lst name=statusMessages str name=Total Requests made to DataSource2/str str name=Total Rows Fetched1/str str name=Total Documents Skipped0/str str name=Delta Dump started2013-09-10 07:46:34/str str name=Identifying Delta2013-09-10 07:46:34/str str name=Deltas Obtained2013-09-10 07:46:35/str str name=Building documents2013-09-10 07:46:35/str str name=Total Changed Documents1/str str name=Total Documents Processed0/str str name=Time taken0:0:1.30/str /lst str name=WARNING This response format is experimental. It is likely to change in the future. /str /response -- View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4089093.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
This is absolutely a sintax error, I had the same problem, and with dih.delta.id it solves all my problems. Thanks to god and the special person who post the answer in this page. You have to revise your sintax in queries for delta import and watch the catalina (i use tomcat) log file for any errors. Regards,
RE: Need help with delta import
I have changed to use dih.xx but still no luck. Even with dataimport or dataimporter the query is able to fetch the delta records but they are not able to commit to solr. Would there be any other reason why this would fail? -- View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4027297.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
If I am not mistaken, it's suppose to be dataimporter.delta.ID and dataimporter.last_index_time You are using dataimport.delta.ID and dataimport.last_index_time http://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport -Original Message- From: umajava [mailto:umaj...@gmail.com] Sent: Thursday, December 13, 2012 9:35 PM To: solr-user@lucene.apache.org Subject: RE: Need help with delta import Thanks a lot for your reply. I have made the changes but it still does not work. I still get the same results. Any other suggestions please? -- View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4026910.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
Thanks, but this didn't help either. Documents are not getting commited. str name=Total Requests made to DataSource3/strstr name=Total Rows Fetched2/strstr name=Total Documents Skipped0/strstr name=Delta Dump started2012-12-14 16:57:25/strstr name=Identifying Delta2012-12-14 16:57:25/strstr name=Deltas Obtained2012-12-14 16:57:25/strstr name=Building documents2012-12-14 16:57:25/strstr name=Total Changed Documents2/strstr name=Total Documents Processed0/strstr name=Time taken0:0:0.140/str Should I do full import again as I have included email and fname in the query and start again? -- View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4027058.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
I tried full import and then delta import but still the issue is same.
entity name=person pk=ID query=select id from uma_test
deltaImportQuery=select id, email, fname from
uma_test where
ID='${dataimporter.delta.id}'
deltaQuery=select ID from uma_test where upd_ts gt;
'${dataimporter.last_index_time}'
field column=ID name=id indexed=true
stored=true /
field column=email name=email
indexed=true stored=true /
field column=fname name=fname
indexed=true stored=true /
/entity
--
View this message in context:
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4027066.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
Try ${dih.delta.ID} instead of ${dataimporter.delta.id}. Also use
${dih.last_index_time} instead of ${dataimporter.last_index_time} . I noticed
when updating the test cases that the wiki incorrectly used the longer name but
with all the versions I tested this on only the short name works. The wiki has
since been changed.
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-Original Message-
From: umajava [mailto:umaj...@gmail.com]
Sent: Friday, December 14, 2012 11:27 AM
To: solr-user@lucene.apache.org
Subject: RE: Need help with delta import
I tried full import and then delta import but still the issue is same.
entity name=person pk=ID query=select id from uma_test
deltaImportQuery=select id, email, fname from
uma_test where
ID='${dataimporter.delta.id}'
deltaQuery=select ID from uma_test where upd_ts gt;
'${dataimporter.last_index_time}'
field column=ID name=id indexed=true
stored=true /
field column=email name=email
indexed=true stored=true /
field column=fname name=fname
indexed=true stored=true /
/entity
--
View this message in context:
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4027066.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Need help with delta import
On 12/14/2012 11:39 AM, Dyer, James wrote:
Try ${dih.delta.ID} instead of ${dataimporter.delta.id}. Also use
${dih.last_index_time} instead of ${dataimporter.last_index_time} . I noticed
when updating the test cases that the wiki incorrectly used the longer name but
with all the versions I tested this on only the short name works. The wiki has
since been changed.
James,
I use DIH for full Solr reindexes. My dih config makes extensive use of
${dataimporter.request.XXX} variables for my own custom parameters. I
am using branch_4x checked out yesterday on my dev machine, and I did a
full reindex on that version, which worked. Three questions: 1) Should
I be using ${dih.request.XXX} instead? 2) Is the longer syntax going
away? 3) What issues and/or docs would be good reading material?
Thanks,
Shawn
RE: Need help with delta import
I am also confused, as I've been using dataimporter.* and not dih.* and it is
working fine.
-Original Message-
From: Shawn Heisey [mailto:s...@elyograg.org]
Sent: Friday, December 14, 2012 2:41 PM
To: solr-user@lucene.apache.org
Subject: Re: Need help with delta import
On 12/14/2012 11:39 AM, Dyer, James wrote:
Try ${dih.delta.ID} instead of ${dataimporter.delta.id}. Also use
${dih.last_index_time} instead of ${dataimporter.last_index_time} . I
noticed when updating the test cases that the wiki incorrectly used the
longer name but with all the versions I tested this on only the short name
works. The wiki has since been changed.
James,
I use DIH for full Solr reindexes. My dih config makes extensive use of
${dataimporter.request.XXX} variables for my own custom parameters. I am using
branch_4x checked out yesterday on my dev machine, and I did a full reindex on
that version, which worked. Three questions: 1) Should I be using
${dih.request.XXX} instead? 2) Is the longer syntax going away? 3) What
issues and/or docs would be good reading material?
Thanks,
Shawn
RE: Need help with delta import
Shawn,
I think it only is a problem with dih.delta.xxx ... the longer version,
dataimport.delta.xxx doesn't work. This is coded in DocBuilder#doDelta and
this line:
vri.addNamespace(ConfigNameConstants.IMPORTER_NS_SHORT + .delta, map);
There is no additional line for:
vri.addNamespace(ConfigNameConstants.IMPORTER_NS + .delta, map);
This is how it is in 3.5, 4.0 and Trunk, which are all the versions I have been
testing against.
However, at some point someone did depreciate the longer dataimporter., so it
would be safer to change to dih. everywhere you use it. I had thought about
removing the longer one for 4.0 and then figured most people wouldn't realize
it had been depreciated (there's no warning logged, etc) and I didn't want to
give people unnecessary problems. I guess there is no stopping it being
removed in 5.0 though.
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-Original Message-
From: Shawn Heisey [mailto:s...@elyograg.org]
Sent: Friday, December 14, 2012 1:41 PM
To: solr-user@lucene.apache.org
Subject: Re: Need help with delta import
On 12/14/2012 11:39 AM, Dyer, James wrote:
Try ${dih.delta.ID} instead of ${dataimporter.delta.id}. Also use
${dih.last_index_time} instead of ${dataimporter.last_index_time} . I
noticed when updating the test cases that the wiki incorrectly used the
longer name but with all the versions I tested this on only the short name
works. The wiki has since been changed.
James,
I use DIH for full Solr reindexes. My dih config makes extensive use of
${dataimporter.request.XXX} variables for my own custom parameters. I
am using branch_4x checked out yesterday on my dev machine, and I did a
full reindex on that version, which worked. Three questions: 1) Should
I be using ${dih.request.XXX} instead? 2) Is the longer syntax going
away? 3) What issues and/or docs would be good reading material?
Thanks,
Shawn
RE: Need help with delta import
Its surprising that your full import is working for you. Both your query and
your deltaImportQuery have:
SELECT ID FROM...
...So both your full-import (query attr) and your delta-import
(deltaImportQuery attr) are only getting the ID field from your db.
Shouldn't you be at least be getting email and fname to index also? So by
changing both these queries to something like:
SELECT ID, EMAIL, FNAME FROM...
...You should see these 3 fields come through after your full-import. Then,
after changing data in your rbdms and doing a delta you should see the data
update.
Besides this, your log looks right:
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Completed ModifiedRowKey for Entity: person rows obtained : 8
...so it looks like it was going to update 8 rows. But seeing that your
deltaImportQuery is only pulling back the ID, it couldn't possibly change the
values for fields like email and fname.
Make sense?
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-Original Message-
From: umajava [mailto:umaj...@gmail.com]
Sent: Thursday, December 06, 2012 8:59 PM
To: solr-user@lucene.apache.org
Subject: Need help with delta import
Hi,
I am trying to do delta import and I am not able to get it to work. How ever
full import does work. Could you please help me figure out what I am
missing?
data-config.xml file
document name=persons
entity name=person pk=ID query=select id from uma_test
deltaImportQuery=select id from uma_test where
ID='${dataimport.delta.id}'
deltaQuery=select ID from uma_test where upd_ts gt;
'${dataimport.last_index_time}'
field column=ID name=id indexed=true
stored=true /
field column=email name=email
indexed=true stored=true /
field column=fname name=fname
indexed=true stored=true /
/entity
/document
dataimport.properties file
metadataObject.last_index_time=2012-09-20 11\:12\:47
person.last_index_time=2012-11-18 13\:54\:29
interval=10
port=8080
server=localhost
params=/select?qt\=/dataimportcommand\=delta-importclean\=falsecommit\=true
webapp=solr
syncEnabled=1
last_index_time=2012-11-18 13\:54\:29
syncCores=coreHr,coreEn
log output
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DataImporter
maybeReloadConfiguration
O: Loading DIH Configuration:
C://Software//apache-solr-4.0.0//apache-solr-4.0.0//Uma//db//db-data-config.xml
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DataImporter
loadDataConfig
O: Data Configuration loaded successfully
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DataImporter
doDeltaImport
O: Starting Delta Import
Dec-2012 02:49:24 org.apache.solr.core.SolrCore execute
O: [collection1] webapp=/solr path=/dataimport
params={commit=falsecommand=delta-import} status=0 QTime=16
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.SimplePropertiesWriter
readIndexerProperties
O: Read dataimport.properties
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder doDelta
O: Starting delta collection.
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Running ModifiedRowKey() for Entity: person
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.JdbcDataSource$1 call
O: Creating a connection for entity person with URL:
jdbc:mysql://localhost/test
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.JdbcDataSource$1 call
O: Time taken for getConnection(): 125
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Completed ModifiedRowKey for Entity: person rows obtained : 8
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Completed DeletedRowKey for Entity: person rows obtained : 0
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Completed parentDeltaQuery for Entity: person
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder doDelta
O: Delta Import completed successfully
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder execute
O: Time taken = 0:0:0.156
Dec-2012 02:49:24 org.apache.solr.update.processor.LogUpdateProcessor finish
O: [collection1] webapp=/solr path=/dataimport
params={commit=falsecommand=delta-import} status=0 QTime=16 {} 0
Thanks,
--
View this message in context:
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Need help with delta import
Ouput in the browser is str name=statusidle/strstr name=importResponse/-lst name=statusMessagesstr name=Total Requests made to DataSource9/strstr name=Total Rows Fetched8/strstr name=Total Documents Skipped0/strstr name=Delta Dump started2012-12-07 03:15:36/strstr name=Identifying Delta2012-12-07 03:15:36/strstr name=Deltas Obtained2012-12-07 03:15:36/strstr name=Building documents2012-12-07 03:15:36/strstr name=Total Changed Documents8/strstr name=Total Documents Processed0/strstr name=Time taken0:0:0.32/str/lst I have tried all sorts of URL's to invoke the data import http://localhost:8080/solr/dataimport?command=delta-import http://localhost:8080/solr/dataimport?command=delta-importcommit=false http://localhost:8080/solr/dataimport?command=delta-importcommit=true http://localhost:8080/solr/dataimport?command=delta-importcommit=truedebug=true http://localhost:8080/solr/dataimport?command=delta-importcommit=truedebug=trueverbose=true -- View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4025004.html Sent from the Solr - User mailing list archive at Nabble.com.
