Hello ben,
the files are more than 6 GB.
I am sorry i was remember to add this function but i forgot and here it is
Document* IndexEngine::fileDocument(const QString &id, const QString &bookid
, const QString &text)
{
// make a new, empty document
Document* doc = _CLNEW Document();
///page ID
doc->add( *_CLNEW Field(_T("id"), QSTRING_TO_TCHAR(id) ,
Field::STORE_YES | Field::INDEX_UNTOKENIZED) );
doc->add( *_CLNEW Field(_T("bookid"), QSTRING_TO_TCHAR(bookid) ,
Field::STORE_YES | Field::INDEX_UNTOKENIZED ) );
doc->add( *_CLNEW Field(_T("text"), QSTRING_TO_TCHAR(text),
Field::STORE_NO | Field::INDEX_TOKENIZED) );
return doc;
}
and also i tried to remove adding documents and the size did not increased,
which means that leaks are in addDocument() I am trying to debug it but i am
lost.
sounds pretty high. how big are the files? could you be leaking memory in
> the 'fileDocument' function?
>
> as a test, try not actually adding the document
>
> ben
>
> On Thu, Jan 6, 2011 at 7:43 AM, muhammad ismael <m.ismae...@gmail.com
> >wrote:
>
> > Hello,
> > I am trying to index large files as follows
> >
> > for(int j = 0; (j < pagesIds.count())&& !m_stop ; j++)
> >
> > {
> >
> > pagesText =
> m_DbManager->getBookPage(m_booksIds.at(i), pagesIds.at(j)).toUtf8();
> >
> > if(!pagesText.isEmpty())
> >
> > {
> >
> > Document* doc =
> fileDocument(QString::number(pagesIds.at(j)),
> QString::number(m_booksIds.at(i)), pagesText);
> >
> > writer->addDocument(doc);
> >
> > _CLDELETE(doc);
> >
> > }
> >
> > }
> >
> >
> > but when the number of files exceeds 5000 files the application usage of
> my
> > computer ram is 2 GB
> > I tried to debug and i found that this happens in
> >
> > IndexWriter::addDocument(Document*)
> >
> > i tried to set
> > IndexWriter->setMergeFactor(5);
> > and also
> > IndexWriter->setRAMBufferSizeMB(10);
> > I know the default ram usage should be 16 MB but i just tried.
> >
> > I am working on master branch and i merged with it memory_leaks branch.
> > is am i missing something?
>
Mohammad Ismael
------------------------------------------------------------------------------
Learn how Oracle Real Application Clusters (RAC) One Node allows customers
to consolidate database storage, standardize their database environment, and,
should the need arise, upgrade to a full multi-node Oracle RAC database
without downtime or disruption
http://p.sf.net/sfu/oracle-sfdevnl
_______________________________________________
CLucene-developers mailing list
CLucene-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clucene-developers
