oh it's a qt wrapper? i have no idea how that works, sorry.
just an idea, but where do you clean up the result from QSTRING_TO_TCHAR?
ben
On Fri, Jan 7, 2011 at 5:57 PM, Ahmed Saidi <ci7nu...@gmail.com> wrote:
> Hello Muhammad,
> What analyzer are you using?
> It seem to me that your code is based on that application
> "clucene_qt", well i think that you are an old version of that
> application, it has several bugs concerning this problem, and the
> ArabicAnalyzer is not well implemented nor optimized, you should use
> the 0.6 or 0.8 version of this application.
>
> Ahmed
>
> 2011/1/6, muhammad ismael <m.ismae...@gmail.com>:
>> Hello ben,
>> the files are more than 6 GB.
>> I am sorry i was remember to add this function but i forgot and here it is
>>
>> Document* IndexEngine::fileDocument(const QString &id, const QString &bookid
>> , const QString &text)
>>
>> {
>>
>> // make a new, empty document
>>
>> Document* doc = _CLNEW Document();
>>
>> ///page ID
>>
>> doc->add( *_CLNEW Field(_T("id"), QSTRING_TO_TCHAR(id) ,
>>
>> Field::STORE_YES | Field::INDEX_UNTOKENIZED) );
>>
>> doc->add( *_CLNEW Field(_T("bookid"), QSTRING_TO_TCHAR(bookid) ,
>>
>> Field::STORE_YES | Field::INDEX_UNTOKENIZED ) );
>>
>> doc->add( *_CLNEW Field(_T("text"), QSTRING_TO_TCHAR(text),
>>
>> Field::STORE_NO | Field::INDEX_TOKENIZED) );
>>
>> return doc;
>>
>> }
>>
>> and also i tried to remove adding documents and the size did not increased,
>> which means that leaks are in addDocument() I am trying to debug it but i am
>> lost.
>>
>> sounds pretty high. how big are the files? could you be leaking memory in
>>> the 'fileDocument' function?
>>>
>>> as a test, try not actually adding the document
>>>
>>> ben
>>>
>>> On Thu, Jan 6, 2011 at 7:43 AM, muhammad ismael <m.ismae...@gmail.com
>>> >wrote:
>>>
>>> > Hello,
>>> > I am trying to index large files as follows
>>> >
>>> > for(int j = 0; (j < pagesIds.count())&& !m_stop ; j++)
>>> >
>>> > {
>>> >
>>> > pagesText =
>>> m_DbManager->getBookPage(m_booksIds.at(i), pagesIds.at(j)).toUtf8();
>>> >
>>> > if(!pagesText.isEmpty())
>>> >
>>> > {
>>> >
>>> > Document* doc =
>>> fileDocument(QString::number(pagesIds.at(j)),
>>> QString::number(m_booksIds.at(i)), pagesText);
>>> >
>>> > writer->addDocument(doc);
>>> >
>>> > _CLDELETE(doc);
>>> >
>>> > }
>>> >
>>> > }
>>> >
>>> >
>>> > but when the number of files exceeds 5000 files the application usage of
>>> my
>>> > computer ram is 2 GB
>>> > I tried to debug and i found that this happens in
>>> >
>>> > IndexWriter::addDocument(Document*)
>>> >
>>> > i tried to set
>>> > IndexWriter->setMergeFactor(5);
>>> > and also
>>> > IndexWriter->setRAMBufferSizeMB(10);
>>> > I know the default ram usage should be 16 MB but i just tried.
>>> >
>>> > I am working on master branch and i merged with it memory_leaks branch.
>>> > is am i missing something?
>>>
>>
>> Mohammad Ismael
>>
>
> --
> Envoyé avec mon mobile
>
> ------------------------------------------------------------------------------
> Gaining the trust of online customers is vital for the success of any company
> that requires sensitive data to be transmitted over the Web. Learn how to
> best implement a security strategy that keeps consumers' information secure
> and instills the confidence they need to proceed with transactions.
> http://p.sf.net/sfu/oracle-sfdevnl
> _______________________________________________
> CLucene-developers mailing list
> CLucene-developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/clucene-developers
>
------------------------------------------------------------------------------
Gaining the trust of online customers is vital for the success of any company
that requires sensitive data to be transmitted over the Web. Learn how to
best implement a security strategy that keeps consumers' information secure
and instills the confidence they need to proceed with transactions.
http://p.sf.net/sfu/oracle-sfdevnl
_______________________________________________
CLucene-developers mailing list
CLucene-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clucene-developers
