Re: lucene vs mysql

[Otis Gospodnetic]

MySQL - not sure.  Let us know if you test both.
Locking - sequential.

Otis

--- Cédric_Grun <[EMAIL PROTECTED]> wrote:
> I'm currently using mysql for storing file which I index previously
> with 
> lucene. I've seen there is a new function in mysql which consists in 
> full text searching.
> I'd like to know which is best between mysql full text search and
> lucene 
> search.
> 
> I'm also interesting to know how lucene perform lock on index. Can I 
> index several files simultaneously using thread, or must I wait one
> file 
> is indexed for indexing the second ?
> 
> thanks
> 
>
__
> Etudiant: Wanadoo t'offre le Pack eXtense Haut Débit soit 150,92
> euros
> d'économies ! Clique ici :
> http://www.ifrance.com/_reloc/mail.etudiant 
> 
> 
> --
> To unsubscribe, e-mail:  
> 
> For additional commands, e-mail:
> 
> 


__
Do you Yahoo!?
Faith Hill - Exclusive Performances, Videos & More
http://faith.yahoo.com

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

lucene vs mysql

[Cédric Grun]

I'm currently using mysql for storing file which I index previously with 
lucene. I've seen there is a new function in mysql which consists in 
full text searching.
I'd like to know which is best between mysql full text search and lucene 
search.

I'm also interesting to know how lucene perform lock on index. Can I 
index several files simultaneously using thread, or must I wait one file 
is indexed for indexing the second ?

thanks

__
Etudiant: Wanadoo t'offre le Pack eXtense Haut Débit soit 150,92 euros
d'économies ! Clique ici : http://www.ifrance.com/_reloc/mail.etudiant 


--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: FileNotFoundException while indexing

[Otis Gospodnetic]

Regarding final - my guess is that Doug didn't want people to inherit
things that shouldn't be altered, plus I believe at one point in time
Java compiler was relying on 'final' to streamline method calls,
instance vars, classes, etcto improve performance.
This is a guess.

Otis


--- Stas Chetvertkov <[EMAIL PROTECTED]> wrote:
> Well... there is one thing with my solution. I implemented it using
> native
> code since I found no way to do it in Java :(
> 
> Here is an excerpt from 'man 2 creat':
> 
> ...O_EXCL is broken on  NFS  file systems, programs which rely on it
> for
> performing locking tasks will contain a race condition.  The solution
> for
> performing atomic file locking using a lockfile is to create a unique
> file
> on the same fs  (e.g.,  incorporating  hostname and pid), use link(2)
> to
> make a link to the lockfile. If link() returns 0, the lock is
> successful.
> Otherwise, use stat(2) on the unique file to check if its  link count
> has
> increased to 2, in which case the lock is also successful
> 
> This is exactly what I did for locking. I am attaching code that
> solved the
> problem for me (my OS is Linux).
> 
> By the way, maybe somebody knows why almost all classes / methods in
> Lucene
> are final? I would surely prefer to inherit from FSDirectory instead
> of
> writing a wrapper around it.
> 
> Cheers,
> Stas.
> 
> - Original Message -
> From: "Terry Steichen" <[EMAIL PROTECTED]>
> To: "Lucene Users List" <[EMAIL PROTECTED]>
> Sent: Friday, October 11, 2002 8:42 PM
> Subject: Re: FileNotFoundException while indexing
> 
> 
> > Stas,
> >
> > Would you be able/willing to share that improved Directory with the
> list?
> >
> > Regards,
> >
> > Terry
> >
> 
> 

> ATTACHMENT part 2 application/octet-stream name=FSLockService.java


> ATTACHMENT part 3 application/octet-stream
name=LockingFSDirectory.java


> ATTACHMENT part 4 application/octet-stream
name=FSLockService_NativeLockService.h


> ATTACHMENT part 5 application/octet-stream name=LockService.c
> --
> To unsubscribe, e-mail:  
> 
> For additional commands, e-mail:



__
Do you Yahoo!?
Faith Hill - Exclusive Performances, Videos & More
http://faith.yahoo.com

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: FileNotFoundException while indexing

[Stas Chetvertkov]

Well... there is one thing with my solution. I implemented it using native
code since I found no way to do it in Java :(

Here is an excerpt from 'man 2 creat':

...O_EXCL is broken on  NFS  file systems, programs which rely on it for
performing locking tasks will contain a race condition.  The solution for
performing atomic file locking using a lockfile is to create a unique file
on the same fs  (e.g.,  incorporating  hostname and pid), use link(2) to
make a link to the lockfile. If link() returns 0, the lock is successful.
Otherwise, use stat(2) on the unique file to check if its  link count has
increased to 2, in which case the lock is also successful

This is exactly what I did for locking. I am attaching code that solved the
problem for me (my OS is Linux).

By the way, maybe somebody knows why almost all classes / methods in Lucene
are final? I would surely prefer to inherit from FSDirectory instead of
writing a wrapper around it.

Cheers,
Stas.

- Original Message -
From: "Terry Steichen" <[EMAIL PROTECTED]>
To: "Lucene Users List" <[EMAIL PROTECTED]>
Sent: Friday, October 11, 2002 8:42 PM
Subject: Re: FileNotFoundException while indexing


> Stas,
>
> Would you be able/willing to share that improved Directory with the list?
>
> Regards,
>
> Terry
>




FSLockService.java
Description: Binary data


LockingFSDirectory.java
Description: Binary data


FSLockService_NativeLockService.h
Description: Binary data


LockService.c
Description: Binary data

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: FileNotFoundException while indexing

[Terry Steichen]

Stas,

Would you be able/willing to share that improved Directory with the list?

Regards,

Terry

- Original Message -
From: "Stas Chetvertkov" <[EMAIL PROTECTED]>
To: "Lucene Users List" <[EMAIL PROTECTED]>
Sent: Friday, October 11, 2002 12:33 PM
Subject: Re: FileNotFoundException while indexing


> I had experienced similar problem with FileNotFoundExceptions.
>
> Our system has one server writing to index (located on NFS mounted disk)
and
> several servers searching in it. In my case, those exceptions were thrown
by
> search servers.
>
> I found out that Lucene locking doesnot work on NFS - in this case 2
> different processes can call createNewFile() for the same file and get
true
> as a result, and Lucene relies on this method. Archiving server was
merging
> segments, this caused deletion of some files, and at the same time search
> server was trying to read those and did not succeed.
>
> I  solved this problem by implementing my own Directory with reliable
> makeLock() method.
>
> Cheers,
> Stas.
>
> - Original Message -
> From: "Craig Walls" <[EMAIL PROTECTED]>
> To: <[EMAIL PROTECTED]>
> Sent: Friday, October 11, 2002 8:02 PM
> Subject: FileNotFoundException while indexing
>
>
> > This is my first post to this mailing list, so I hope it works...
> >
> > We've been trying to use Lucene as our search solution, but every so
> > often we get a ton of the following in our log files:
> >
> > java.io.FileNotFoundException:
> > /usr/WebSphere/michaels/search/working/artprints/_xx.fnm (A file or
> > directory in the path name does not exist.)
> > at java.io.RandomAccessFile.open(Native Method)
> > at
> > java.io.RandomAccessFile.(RandomAccessFile.java(Compiled Code))
> > at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> > at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> > at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> > at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> > at org.apache.lucene.index.FieldInfos.(Unknown Source)
> > at org.apache.lucene.index.SegmentReader.(Unknown Source)
> > at org.apache.lucene.index.IndexWriter.mergeSegments(Unknown
> > Source)
> > at
> > org.apache.lucene.index.IndexWriter.maybeMergeSegments(Unknown Source)
> > at org.apache.lucene.index.IndexWriter.addDocument(Unknown
> > Source)
> > at
> > com.michaels.search.ProductIndexer.run(ProductIndexer.java:39)
> > at com.michaels.search.MasterIndexer.run(MasterIndexer.java:56)
> > at java.lang.Thread.run(Thread.java:512)
> >
> > At first, we thought it was because we had multiple threads trying to
> > add to the same index. That concerned me a bit because I thought that
> > the write.lock and commit.lock files would prevent bad things like this
> > from happening if multiple threads were writing to the same index.
> > Nevertheless, we now have a single thread doing our indexing and it
> > seemed to work fine for several days, but this morning we found the same
> > errors in the log file.
> >
> > What would cause this, how can we fix it, and why isn't write.lock and
> > commit.lock not seeming to help with this?
> >
> > A bit more info, if it helps: We are running this within a thread that
> > is kicked off by a servlet when a certain URL is visited. If an indexer
> > thread is already in progress, we don't kick off another thread. This is
> > all running within WebSphere 4, running on AIX.
> >
> > Any help would be greatly appreciated.
> > Thanks,
> > Craig
> >
> >
> >
> > --
> > To unsubscribe, e-mail:
> 
> > For additional commands, e-mail:
> 
> >
> >
>
>
> --
> To unsubscribe, e-mail:

> For additional commands, e-mail:

>
>


--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: FileNotFoundException while indexing

[Stas Chetvertkov]

I had experienced similar problem with FileNotFoundExceptions.

Our system has one server writing to index (located on NFS mounted disk) and
several servers searching in it. In my case, those exceptions were thrown by
search servers.

I found out that Lucene locking doesnot work on NFS - in this case 2
different processes can call createNewFile() for the same file and get true
as a result, and Lucene relies on this method. Archiving server was merging
segments, this caused deletion of some files, and at the same time search
server was trying to read those and did not succeed.

I  solved this problem by implementing my own Directory with reliable
makeLock() method.

Cheers,
Stas.

- Original Message -
From: "Craig Walls" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Friday, October 11, 2002 8:02 PM
Subject: FileNotFoundException while indexing


> This is my first post to this mailing list, so I hope it works...
>
> We've been trying to use Lucene as our search solution, but every so
> often we get a ton of the following in our log files:
>
> java.io.FileNotFoundException:
> /usr/WebSphere/michaels/search/working/artprints/_xx.fnm (A file or
> directory in the path name does not exist.)
> at java.io.RandomAccessFile.open(Native Method)
> at
> java.io.RandomAccessFile.(RandomAccessFile.java(Compiled Code))
> at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> at org.apache.lucene.index.FieldInfos.(Unknown Source)
> at org.apache.lucene.index.SegmentReader.(Unknown Source)
> at org.apache.lucene.index.IndexWriter.mergeSegments(Unknown
> Source)
> at
> org.apache.lucene.index.IndexWriter.maybeMergeSegments(Unknown Source)
> at org.apache.lucene.index.IndexWriter.addDocument(Unknown
> Source)
> at
> com.michaels.search.ProductIndexer.run(ProductIndexer.java:39)
> at com.michaels.search.MasterIndexer.run(MasterIndexer.java:56)
> at java.lang.Thread.run(Thread.java:512)
>
> At first, we thought it was because we had multiple threads trying to
> add to the same index. That concerned me a bit because I thought that
> the write.lock and commit.lock files would prevent bad things like this
> from happening if multiple threads were writing to the same index.
> Nevertheless, we now have a single thread doing our indexing and it
> seemed to work fine for several days, but this morning we found the same
> errors in the log file.
>
> What would cause this, how can we fix it, and why isn't write.lock and
> commit.lock not seeming to help with this?
>
> A bit more info, if it helps: We are running this within a thread that
> is kicked off by a servlet when a certain URL is visited. If an indexer
> thread is already in progress, we don't kick off another thread. This is
> all running within WebSphere 4, running on AIX.
>
> Any help would be greatly appreciated.
> Thanks,
> Craig
>
>
>
> --
> To unsubscribe, e-mail:

> For additional commands, e-mail:

>
>


--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

FileNotFoundException while indexing

[Craig Walls]

This is my first post to this mailing list, so I hope it works...

We've been trying to use Lucene as our search solution, but every so
often we get a ton of the following in our log files:

java.io.FileNotFoundException:
/usr/WebSphere/michaels/search/working/artprints/_xx.fnm (A file or
directory in the path name does not exist.)
at java.io.RandomAccessFile.open(Native Method)
at
java.io.RandomAccessFile.(RandomAccessFile.java(Compiled Code))
at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
at org.apache.lucene.index.FieldInfos.(Unknown Source)
at org.apache.lucene.index.SegmentReader.(Unknown Source)
at org.apache.lucene.index.IndexWriter.mergeSegments(Unknown
Source)
at
org.apache.lucene.index.IndexWriter.maybeMergeSegments(Unknown Source)
at org.apache.lucene.index.IndexWriter.addDocument(Unknown
Source)
at
com.michaels.search.ProductIndexer.run(ProductIndexer.java:39)
at com.michaels.search.MasterIndexer.run(MasterIndexer.java:56)
at java.lang.Thread.run(Thread.java:512)

At first, we thought it was because we had multiple threads trying to
add to the same index. That concerned me a bit because I thought that
the write.lock and commit.lock files would prevent bad things like this
from happening if multiple threads were writing to the same index.
Nevertheless, we now have a single thread doing our indexing and it
seemed to work fine for several days, but this morning we found the same
errors in the log file.

What would cause this, how can we fix it, and why isn't write.lock and
commit.lock not seeming to help with this?

A bit more info, if it helps: We are running this within a thread that
is kicked off by a servlet when a certain URL is visited. If an indexer
thread is already in progress, we don't kick off another thread. This is
all running within WebSphere 4, running on AIX.

Any help would be greatly appreciated.
Thanks,
Craig



--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: How to include strange characters??

[Dominator]

thx, with your help I could solve the problem

"karl ie" <[EMAIL PROTECTED]> wrote in message
[EMAIL PROTECTED]">news:[EMAIL PROTECTED]...
i had such problems with norwegian characters and it resolved into
making sure the querystring has the same encoding as the index has.

since this is again a java.lang.String encoding question i had these
problems with querystrings coming from java Servlets and CLI. For both
the quickfix was to re-encode the query in UTF-8/16:

String querystring = argv[0]; ' String querystring =
httprequest.getParameter("query");
querystring = new String(querystring.getBytes("UTF-8"));
...

this fixed my norwegian/samii problems...


mvh karl ie

On mandag, okt 7, 2002, at 13:04 Europe/Oslo, Dominator wrote:

>> I use czech language with more bizzare characters and there is no
>> problem at all. Are you sure, that your XML contains character set
>> information?
>
> yes, I tried  and  version="1.0" encoding="UTF-8"?> but I get the same strange characters.
>
>
>
>
>
>
> --
> To unsubscribe, e-mail:
> 
> For additional commands, e-mail:
> 
>





--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Incremental indexing with IndexHTML??

[Dominator]

Hi,

Is it possible to do incremental indexing with the IndexHTML? And how do you
do it?

Thx..





--
To unsubscribe, e-mail:   
For additional commands, e-mail: