Re: $DeleteDocbyQuery in solr 1.4 is not working

2009-11-30 Thread cpmoser 

Ok, I think I figured out what might be happening.  It appears that the
DataImporter issues the commit command without the expungeDeletes option set
to true (default in 1.4 for a commit command is for expungeDeletes to be set
to false).  You can get around this by issuing the commit command manually:

curl
http://localhost:8983/solr/dataimport?command=delta-importcommit=falseoptimize=false;
curl http://localhost:8983/solr/update; --data-binary 'commit
expungeDeletes=true /' -H Content-type:text/xml; charset=utf-8

That fixed the issue for me, although ideally the DataImportHandler should
run with the expungeDeletes option set to true, so that this could all
happen in one command without having to wait for the DIH to finish before
issuing the commit command.  

Is there a way to set the expungeDeletes option for the commit command of
the DataImportHandler, so that delta imports with deletes can happen
automatically?


Mark.El wrote:
 
 Thanks I will look into it!
 
 

-- 
View this message in context: 
http://old.nabble.com/%24DeleteDocbyQuery-in-solr-1.4-is-not-working-tp26376265p26583260.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: $DeleteDocbyQuery in solr 1.4 is not working

2009-11-17 Thread Noble Paul നോബിള്‍ नोब्ळ् 
are you sure that the doc w/ the same id was not created after that?

On Mon, Nov 16, 2009 at 11:12 PM, Mark Ellul m...@catalystic.com wrote:
 Hi,

 I have added a deleted field in my database, and am using the
 Dataimporthandler to add rows to the index...

 I am using solr 1.4

 I have added my the deleted field to the query and the RegexTransformer...
 and the field definition below

 field column=$deleteDocByQuery
 regex=^true$
 replaceWith=id:${List.id} sourceColName=deleted/

 When I run the deltaImport command... I see the below output

 INFO: [] webapp=/solr path=/dataimport
 params={command=delta-importdebug=trueexpungeDeletes=true} status=0
 QTime=1
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DataImporter
 doDeltaImport
 INFO: Starting Delta Import
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
 readIndexerProperties
 INFO: Read dataimport.properties
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 doDelta
 INFO: Starting delta collection.
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Running ModifiedRowKey() for Entity: List
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
 call
 INFO: Creating a connection for entity List with URL:
 jdbc:postgresql://localhost:5432/tlists
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
 call
 INFO: Time taken for getConnection(): 4
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Completed parentDeltaQuery for Entity: List
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
 deleteByQuery
 INFO: Deleting documents from Solr with query: id:api__list__365522
 Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy onInit
 INFO: SolrDeletionPolicy.onInit: commits:num=1
  commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
 _bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis, _bg.frq]
 Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
 updateCommits
 INFO: newest commit = 1257863009839
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 doDelta
 INFO: Delta Import completed successfully

 It says its deleting the document... but when I do the search its still
 showing up

 Any Ideas?

 Regards

 Mark




-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: $DeleteDocbyQuery in solr 1.4 is not working

2009-11-17 Thread Mark Ellul 
The doc already existed before the delta-import has been run.

And it exists afterwards... even though it says its deleting it.

Any ideas of what I can try?

On 11/17/09, Noble Paul നോബിള്‍  नोब्ळ् noble.p...@corp.aol.com wrote:
 are you sure that the doc w/ the same id was not created after that?

 On Mon, Nov 16, 2009 at 11:12 PM, Mark Ellul m...@catalystic.com wrote:
 Hi,

 I have added a deleted field in my database, and am using the
 Dataimporthandler to add rows to the index...

 I am using solr 1.4

 I have added my the deleted field to the query and the RegexTransformer...
 and the field definition below

 field column=$deleteDocByQuery
 regex=^true$
 replaceWith=id:${List.id} sourceColName=deleted/

 When I run the deltaImport command... I see the below output

 INFO: [] webapp=/solr path=/dataimport
 params={command=delta-importdebug=trueexpungeDeletes=true} status=0
 QTime=1
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DataImporter
 doDeltaImport
 INFO: Starting Delta Import
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
 readIndexerProperties
 INFO: Read dataimport.properties
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 doDelta
 INFO: Starting delta collection.
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Running ModifiedRowKey() for Entity: List
 Nov 16, 2009 5:29:10 PM
 org.apache.solr.handler.dataimport.JdbcDataSource$1
 call
 INFO: Creating a connection for entity List with URL:
 jdbc:postgresql://localhost:5432/tlists
 Nov 16, 2009 5:29:10 PM
 org.apache.solr.handler.dataimport.JdbcDataSource$1
 call
 INFO: Time taken for getConnection(): 4
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Completed parentDeltaQuery for Entity: List
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
 deleteByQuery
 INFO: Deleting documents from Solr with query: id:api__list__365522
 Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy onInit
 INFO: SolrDeletionPolicy.onInit: commits:num=1
  commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
 _bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis, _bg.frq]
 Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
 updateCommits
 INFO: newest commit = 1257863009839
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 doDelta
 INFO: Delta Import completed successfully

 It says its deleting the document... but when I do the search its still
 showing up

 Any Ideas?

 Regards

 Mark




 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com

Re: $DeleteDocbyQuery in solr 1.4 is not working

2009-11-17 Thread Noble Paul നോബിള്‍ नोब्ळ् 
why don't you add a new timestamp  field . you can use the
TemplateTransformer with the formatDate() function

On Tue, Nov 17, 2009 at 5:49 PM, Mark Ellul m...@catalystic.com wrote:
 Hi Noble,

 Excellent Question... should the field that does the deleting be in a
 different entity to the one that does the addition and updating?

 If so that could be the issue, I have the field that does the
 DeleteByQuery command inside of the entity that does the adding.

 Is there some kind of document metadata where the create date and
 update date is show?

 How would I see this meta data if it exists?

 Regards

 Mark

 On 11/17/09, Noble Paul നോബിള്‍  नोब्ळ् noble.p...@corp.aol.com wrote:
 The question is, did your new delta-import created the doc again?

 On Tue, Nov 17, 2009 at 4:41 PM, Mark Ellul m...@catalystic.com wrote:
 The doc already existed before the delta-import has been run.

 And it exists afterwards... even though it says its deleting it.

 Any ideas of what I can try?

 On 11/17/09, Noble Paul നോബിള്‍  नोब्ळ् noble.p...@corp.aol.com wrote:
 are you sure that the doc w/ the same id was not created after that?

 On Mon, Nov 16, 2009 at 11:12 PM, Mark Ellul m...@catalystic.com wrote:
 Hi,

 I have added a deleted field in my database, and am using the
 Dataimporthandler to add rows to the index...

 I am using solr 1.4

 I have added my the deleted field to the query and the
 RegexTransformer...
 and the field definition below

 field column=$deleteDocByQuery
 regex=^true$
 replaceWith=id:${List.id} sourceColName=deleted/

 When I run the deltaImport command... I see the below output

 INFO: [] webapp=/solr path=/dataimport
 params={command=delta-importdebug=trueexpungeDeletes=true} status=0
 QTime=1
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DataImporter
 doDeltaImport
 INFO: Starting Delta Import
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
 readIndexerProperties
 INFO: Read dataimport.properties
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 doDelta
 INFO: Starting delta collection.
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Running ModifiedRowKey() for Entity: List
 Nov 16, 2009 5:29:10 PM
 org.apache.solr.handler.dataimport.JdbcDataSource$1
 call
 INFO: Creating a connection for entity List with URL:
 jdbc:postgresql://localhost:5432/tlists
 Nov 16, 2009 5:29:10 PM
 org.apache.solr.handler.dataimport.JdbcDataSource$1
 call
 INFO: Time taken for getConnection(): 4
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Completed parentDeltaQuery for Entity: List
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
 deleteByQuery
 INFO: Deleting documents from Solr with query: id:api__list__365522
 Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy onInit
 INFO: SolrDeletionPolicy.onInit: commits:num=1
  commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
 _bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis,
 _bg.frq]
 Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
 updateCommits
 INFO: newest commit = 1257863009839
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 doDelta
 INFO: Delta Import completed successfully

 It says its deleting the document... but when I do the search its still
 showing up

 Any Ideas?

 Regards

 Mark




 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com





 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com





-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: $DeleteDocbyQuery in solr 1.4 is not working

2009-11-17 Thread Mark Ellul 
Hi Noble,

I have updated my entity specs, by having a separate entity for
selecting rows which are not deleted for and ones that are deleted, so
I am sure now that the document is not getting added in the same
import.

I read in the tutorial that the deletes are not taken out until the
commit is done. Is there a way I can force a commit? I don't have the
post.jar on the machine in question.

So whats next, I can now safely say that the documents are not getting
added again, and they should be getting deleted... but they not,
unless its not committing and I have to do it manually.

Any other ideas?

Thanks for your help so far!

Regards

Mark

On 11/17/09, Noble Paul നോബിള്‍  नोब्ळ् noble.p...@corp.aol.com wrote:
 why don't you add a new timestamp  field . you can use the
 TemplateTransformer with the formatDate() function

 On Tue, Nov 17, 2009 at 5:49 PM, Mark Ellul m...@catalystic.com wrote:
 Hi Noble,

 Excellent Question... should the field that does the deleting be in a
 different entity to the one that does the addition and updating?

 If so that could be the issue, I have the field that does the
 DeleteByQuery command inside of the entity that does the adding.

 Is there some kind of document metadata where the create date and
 update date is show?

 How would I see this meta data if it exists?

 Regards

 Mark

 On 11/17/09, Noble Paul നോബിള്‍  नोब्ळ् noble.p...@corp.aol.com wrote:
 The question is, did your new delta-import created the doc again?

 On Tue, Nov 17, 2009 at 4:41 PM, Mark Ellul m...@catalystic.com wrote:
 The doc already existed before the delta-import has been run.

 And it exists afterwards... even though it says its deleting it.

 Any ideas of what I can try?

 On 11/17/09, Noble Paul നോബിള്‍  नोब्ळ् noble.p...@corp.aol.com wrote:
 are you sure that the doc w/ the same id was not created after that?

 On Mon, Nov 16, 2009 at 11:12 PM, Mark Ellul m...@catalystic.com
 wrote:
 Hi,

 I have added a deleted field in my database, and am using the
 Dataimporthandler to add rows to the index...

 I am using solr 1.4

 I have added my the deleted field to the query and the
 RegexTransformer...
 and the field definition below

 field column=$deleteDocByQuery
 regex=^true$
 replaceWith=id:${List.id} sourceColName=deleted/

 When I run the deltaImport command... I see the below output

 INFO: [] webapp=/solr path=/dataimport
 params={command=delta-importdebug=trueexpungeDeletes=true} status=0
 QTime=1
 Nov 16, 2009 5:29:10 PM
 org.apache.solr.handler.dataimport.DataImporter
 doDeltaImport
 INFO: Starting Delta Import
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
 readIndexerProperties
 INFO: Read dataimport.properties
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 doDelta
 INFO: Starting delta collection.
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Running ModifiedRowKey() for Entity: List
 Nov 16, 2009 5:29:10 PM
 org.apache.solr.handler.dataimport.JdbcDataSource$1
 call
 INFO: Creating a connection for entity List with URL:
 jdbc:postgresql://localhost:5432/tlists
 Nov 16, 2009 5:29:10 PM
 org.apache.solr.handler.dataimport.JdbcDataSource$1
 call
 INFO: Time taken for getConnection(): 4
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 collectDelta
 INFO: Completed parentDeltaQuery for Entity: List
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
 deleteByQuery
 INFO: Deleting documents from Solr with query: id:api__list__365522
 Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy onInit
 INFO: SolrDeletionPolicy.onInit: commits:num=1
  commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
 _bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis,
 _bg.frq]
 Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
 updateCommits
 INFO: newest commit = 1257863009839
 Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
 doDelta
 INFO: Delta Import completed successfully

 It says its deleting the document... but when I do the search its
 still
 showing up

 Any Ideas?

 Regards

 Mark




 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com





 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com





 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com

Re: $DeleteDocbyQuery in solr 1.4 is not working

2009-11-17 Thread Otis Gospodnetic 
Mark,

http://localhost:8983/solr/update?stream.body=%3Ccommit/%3E

Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR



- Original Message 
 From: Mark Ellul m...@catalystic.com
 To: solr-user@lucene.apache.org; noble.p...@gmail.com
 Sent: Tue, November 17, 2009 9:22:13 AM
 Subject: Re: $DeleteDocbyQuery in solr 1.4 is not working
 
 Hi Noble,
 
 I have updated my entity specs, by having a separate entity for
 selecting rows which are not deleted for and ones that are deleted, so
 I am sure now that the document is not getting added in the same
 import.
 
 I read in the tutorial that the deletes are not taken out until the
 commit is done. Is there a way I can force a commit? I don't have the
 post.jar on the machine in question.
 
 So whats next, I can now safely say that the documents are not getting
 added again, and they should be getting deleted... but they not,
 unless its not committing and I have to do it manually.
 
 Any other ideas?
 
 Thanks for your help so far!
 
 Regards
 
 Mark
 
 On 11/17/09, Noble Paul നോബിള്‍  नोब्ळ् wrote:
  why don't you add a new timestamp  field . you can use the
  TemplateTransformer with the formatDate() function
 
  On Tue, Nov 17, 2009 at 5:49 PM, Mark Ellul wrote:
  Hi Noble,
 
  Excellent Question... should the field that does the deleting be in a
  different entity to the one that does the addition and updating?
 
  If so that could be the issue, I have the field that does the
  DeleteByQuery command inside of the entity that does the adding.
 
  Is there some kind of document metadata where the create date and
  update date is show?
 
  How would I see this meta data if it exists?
 
  Regards
 
  Mark
 
  On 11/17/09, Noble Paul നോബിള്‍  नोब्ळ् wrote:
  The question is, did your new delta-import created the doc again?
 
  On Tue, Nov 17, 2009 at 4:41 PM, Mark Ellul wrote:
  The doc already existed before the delta-import has been run.
 
  And it exists afterwards... even though it says its deleting it.
 
  Any ideas of what I can try?
 
  On 11/17/09, Noble Paul നോബിള്‍  नोब्ळ् wrote:
  are you sure that the doc w/ the same id was not created after that?
 
  On Mon, Nov 16, 2009 at 11:12 PM, Mark Ellul 
  wrote:
  Hi,
 
  I have added a deleted field in my database, and am using the
  Dataimporthandler to add rows to the index...
 
  I am using solr 1.4
 
  I have added my the deleted field to the query and the
  RegexTransformer...
  and the field definition below
 
  
  regex=^true$
  replaceWith=id:${List.id} sourceColName=deleted/
 
  When I run the deltaImport command... I see the below output
 
  INFO: [] webapp=/solr path=/dataimport
  params={command=delta-importdebug=trueexpungeDeletes=true} status=0
  QTime=1
  Nov 16, 2009 5:29:10 PM
  org.apache.solr.handler.dataimport.DataImporter
  doDeltaImport
  INFO: Starting Delta Import
  Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
  readIndexerProperties
  INFO: Read dataimport.properties
  Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
  doDelta
  INFO: Starting delta collection.
  Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
  collectDelta
  INFO: Running ModifiedRowKey() for Entity: List
  Nov 16, 2009 5:29:10 PM
  org.apache.solr.handler.dataimport.JdbcDataSource$1
  call
  INFO: Creating a connection for entity List with URL:
  jdbc:postgresql://localhost:5432/tlists
  Nov 16, 2009 5:29:10 PM
  org.apache.solr.handler.dataimport.JdbcDataSource$1
  call
  INFO: Time taken for getConnection(): 4
  Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
  collectDelta
  INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
  Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
  collectDelta
  INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
  Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
  collectDelta
  INFO: Completed parentDeltaQuery for Entity: List
  Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
  deleteByQuery
  INFO: Deleting documents from Solr with query: id:api__list__365522
  Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy onInit
  INFO: SolrDeletionPolicy.onInit: commits:num=1
  
  
 commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
  _bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis,
  _bg.frq]
  Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
  updateCommits
  INFO: newest commit = 1257863009839
  Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
  doDelta
  INFO: Delta Import completed successfully
 
  It says its deleting the document... but when I do the search its
  still
  showing up
 
  Any Ideas?
 
  Regards
 
  Mark

Re: $DeleteDocbyQuery in solr 1.4 is not working

2009-11-17 Thread Mark Ellul 
Thanks Otis... I remember that one!

It still did not remove the document! So obviously its something else thats
happening.

On Tue, Nov 17, 2009 at 10:47 AM, Otis Gospodnetic 
otis_gospodne...@yahoo.com wrote:

 Mark,

 http://localhost:8983/solr/update?stream.body=%3Ccommit/%3E

 Otis
 --
 Sematext is hiring -- http://sematext.com/about/jobs.html?mls
 Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR



 - Original Message 
  From: Mark Ellul m...@catalystic.com
  To: solr-user@lucene.apache.org; noble.p...@gmail.com
  Sent: Tue, November 17, 2009 9:22:13 AM
  Subject: Re: $DeleteDocbyQuery in solr 1.4 is not working
 
  Hi Noble,
 
  I have updated my entity specs, by having a separate entity for
  selecting rows which are not deleted for and ones that are deleted, so
  I am sure now that the document is not getting added in the same
  import.
 
  I read in the tutorial that the deletes are not taken out until the
  commit is done. Is there a way I can force a commit? I don't have the
  post.jar on the machine in question.
 
  So whats next, I can now safely say that the documents are not getting
  added again, and they should be getting deleted... but they not,
  unless its not committing and I have to do it manually.
 
  Any other ideas?
 
  Thanks for your help so far!
 
  Regards
 
  Mark
 
  On 11/17/09, Noble Paul നോബിള്‍  नोब्ळ् wrote:
   why don't you add a new timestamp  field . you can use the
   TemplateTransformer with the formatDate() function
  
   On Tue, Nov 17, 2009 at 5:49 PM, Mark Ellul wrote:
   Hi Noble,
  
   Excellent Question... should the field that does the deleting be in a
   different entity to the one that does the addition and updating?
  
   If so that could be the issue, I have the field that does the
   DeleteByQuery command inside of the entity that does the adding.
  
   Is there some kind of document metadata where the create date and
   update date is show?
  
   How would I see this meta data if it exists?
  
   Regards
  
   Mark
  
   On 11/17/09, Noble Paul നോബിള്‍  नोब्ळ् wrote:
   The question is, did your new delta-import created the doc again?
  
   On Tue, Nov 17, 2009 at 4:41 PM, Mark Ellul wrote:
   The doc already existed before the delta-import has been run.
  
   And it exists afterwards... even though it says its deleting it.
  
   Any ideas of what I can try?
  
   On 11/17/09, Noble Paul നോബിള്‍  नोब्ळ् wrote:
   are you sure that the doc w/ the same id was not created after
 that?
  
   On Mon, Nov 16, 2009 at 11:12 PM, Mark Ellul
   wrote:
   Hi,
  
   I have added a deleted field in my database, and am using the
   Dataimporthandler to add rows to the index...
  
   I am using solr 1.4
  
   I have added my the deleted field to the query and the
   RegexTransformer...
   and the field definition below
  
  
   regex=^true$
   replaceWith=id:${List.id} sourceColName=deleted/
  
   When I run the deltaImport command... I see the below output
  
   INFO: [] webapp=/solr path=/dataimport
   params={command=delta-importdebug=trueexpungeDeletes=true}
 status=0
   QTime=1
   Nov 16, 2009 5:29:10 PM
   org.apache.solr.handler.dataimport.DataImporter
   doDeltaImport
   INFO: Starting Delta Import
   Nov 16, 2009 5:29:10 PM
 org.apache.solr.handler.dataimport.SolrWriter
   readIndexerProperties
   INFO: Read dataimport.properties
   Nov 16, 2009 5:29:10 PM
 org.apache.solr.handler.dataimport.DocBuilder
   doDelta
   INFO: Starting delta collection.
   Nov 16, 2009 5:29:10 PM
 org.apache.solr.handler.dataimport.DocBuilder
   collectDelta
   INFO: Running ModifiedRowKey() for Entity: List
   Nov 16, 2009 5:29:10 PM
   org.apache.solr.handler.dataimport.JdbcDataSource$1
   call
   INFO: Creating a connection for entity List with URL:
   jdbc:postgresql://localhost:5432/tlists
   Nov 16, 2009 5:29:10 PM
   org.apache.solr.handler.dataimport.JdbcDataSource$1
   call
   INFO: Time taken for getConnection(): 4
   Nov 16, 2009 5:29:10 PM
 org.apache.solr.handler.dataimport.DocBuilder
   collectDelta
   INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
   Nov 16, 2009 5:29:10 PM
 org.apache.solr.handler.dataimport.DocBuilder
   collectDelta
   INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
   Nov 16, 2009 5:29:10 PM
 org.apache.solr.handler.dataimport.DocBuilder
   collectDelta
   INFO: Completed parentDeltaQuery for Entity: List
   Nov 16, 2009 5:29:10 PM
 org.apache.solr.handler.dataimport.SolrWriter
   deleteByQuery
   INFO: Deleting documents from Solr with query:
 id:api__list__365522
   Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
 onInit
   INFO: SolrDeletionPolicy.onInit: commits:num=1
  
 
  
 commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
   _bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis,
   _bg.frq]
   Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
   updateCommits
   INFO: newest commit

$DeleteDocbyQuery in solr 1.4 is not working

2009-11-16 Thread Mark Ellul 
Hi,

I have added a deleted field in my database, and am using the
Dataimporthandler to add rows to the index...

I am using solr 1.4

I have added my the deleted field to the query and the RegexTransformer...
and the field definition below

field column=$deleteDocByQuery
regex=^true$
replaceWith=id:${List.id} sourceColName=deleted/

When I run the deltaImport command... I see the below output

INFO: [] webapp=/solr path=/dataimport
params={command=delta-importdebug=trueexpungeDeletes=true} status=0
QTime=1
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DataImporter
doDeltaImport
INFO: Starting Delta Import
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
readIndexerProperties
INFO: Read dataimport.properties
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
doDelta
INFO: Starting delta collection.
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Running ModifiedRowKey() for Entity: List
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Creating a connection for entity List with URL:
jdbc:postgresql://localhost:5432/tlists
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Time taken for getConnection(): 4
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Completed parentDeltaQuery for Entity: List
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
deleteByQuery
INFO: Deleting documents from Solr with query: id:api__list__365522
Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
 
commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
_bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis, _bg.frq]
Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1257863009839
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
doDelta
INFO: Delta Import completed successfully

It says its deleting the document... but when I do the search its still
showing up

Any Ideas?

Regards

Mark