Hey all,
I'm trying to create a Solr index for the 2010 Medline-baseline
(www.pubmed.gov, over 18 million XML documents). My goal is to be able
to retrieve single XML documents by their ID. Each document comes with a
unique ID, the PubMedID. So my schema (important portions) looks like this:
<field name="pmid" type="string" indexed="true" stored="true"
required="true" />
<field name="date" type="tdate" indexed="true" stored="true"/>
<field name="xml" type="text" indexed="true" stored="true"/>
<uniqueKey>pmid</uniqueKey>
<defaultSearchField>pmid</defaultSearchField>
pmid holds the ID, data hold the creation date; xml holds the whole XML
document (mostly below 5kb). I used the DataImporter to do this. I had
to write some classes (DataSource, EntityProcessor, DateFormatter)
myself, so theoretically, the error could lie there.
What happens is that indexing just looks fine at the beginning. Memory
usage is quite below the maximum (max of 20g, usage of below 5g, most of
the time around 3g). It goes several hours in this manner until it
suddenly stopps. I tried this a few times with minor tweaks, non of
which made any difference. The last time such a crash occurred, over
16.5 million documents already had been indexed (argh, so close...). It
never stops at the same document and trying to index the documents,
where the error occurred, just runs fine. Index size on disc was between
40g and 50g the last time I had a look.
This is the log from beginning to end:
(I decided to just attach the log for the sake of readability ;) ).
As you can see, Solr's error message is not quite complete. There are no
closing brackets. The document is cut in half on this message and not
even the error message itself is complete: The 'D' of
(D)ataImporter.runCmd(DataImporter.java:389) right after the document
text is missing.
I have one thought concerning this: I get the input documents as an
InputStream which I read buffer-wise (at most 1000bytes per read()
call). I need to deliver the documents in one large byte-Array to the
XML parser I use (VTD XML).
But I don't only get the individual small XML documents but always one
larger XML blob with exactly 30,000 of these documents. I use a
self-written EntityProcessor to extract the single documents from the
larger blob. These blobs have a size of about 50 to 150mb. So what I do
is to read these large blobs in 1000bytes steps and store each byte
array in an ArrayList<byte[]>. Afterwards, I create the ultimate byte[]
and do System.arraycopy from the ArrayList into the byte[].
I tested this and it looks fine to me. And how I said, indexing the
documents where the error occurred just works fine (that is, indexing
the whole blob containing the single document). I just mention this
because it kind of looks like there is this cut in the document and the
missing 'D' reminds me of char-encoding errors. But I don't know for
real, opening the error log in vi doesn't show any broken characters
(the last time I had such problems, vi could identify the characters in
question, other editors just wouldn't show them).
Further ideas from my side: Is the index too big? I think I read
something about a large index would be something around 10million
documents, I aim to approximately double this number. But would this
cause such an error? In the end: What exactly IS the error?
Sorry for the lot of text, just trying to describe the problem as
detailed as possible. Thanks a lot for reading and I appreciate any
ideas! :)
Best regards,
Erik
15.11.2010 11:08:22 org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1289465394071
15.11.2010 18:16:06 org.apache.solr.handler.dataimport.SolrWriter upload
WARNUNG: Error creating document : SolrInputDocument[{pmid=pmid(1.0)={8817856},
xml=xml(1.0)={<MedlineCitation Owner="NLM" Status="MEDLINE">
<PMID>8817856</PMID>
<DateCreated>
<Year>1996</Year>
<Month>12</Month>
<Day>04</Day>
</DateCreated>
<DateCompleted>
<Year>1996</Year>
<Month>12</Month>
<Day>04</Day>
</DateCompleted>
<DateRevised>
<Year>2004</Year>
<Month>11</Month>
<Day>17</Day>
</DateRevised>
<Article PubModel="Print">
<Journal>
<ISSN IssnType="Print">0042-4900</ISSN>
<JournalIssue CitedMedium="Print">
<Volume>138</Volume>
<Issue>26</Issue>
<PubDate>
<Year>1996</Year>
<Month>Jun</Month>
<Day>29</Day>
</PubDate>
</JournalIssue>
<Title>The Veterinary record</Title>
<ISOAbbreviation>Vet. Rec.</ISOAbbreviation>
</Journal>
<ArticleTitle>Restoring confidence in beef: towards a European
solution.</ArticleTitle>
<Pagination>
<MedlinePgn>631-2</MedlinePgn>
</Pagination>
<Language>eng</Language>
<PublicationTypeList>
<PublicationType>News</PublicationType>
</PublicationTypeList>
</Article>
<MedlineJournalInfo>
<Country>ENGLAND</Country>
<MedlineTA>Vet Rec</MedlineTA>
<NlmUniqueID>0031164</NlmUniqueID>
<ISSNLinking>0042-4900</ISSNLinking>
</MedlineJournalInfo>
<CitationSubset>IM</CitationSubset>
<MeshHeadingList>
<MeshHeading>
<DescriptorName MajorTopicYN="N">Animals</DescriptorName>
</MeshHeading>
<MeshHeading>
<DescriptorName MajorTopicYN="N">Cattle</DescriptorName>
</MeshHeading>
<MeshHeading>
<DescriptorName MajorTopicYN="N">Commerce</DescriptorName>
</MeshHeading>
<MeshHeading>
<DescriptorName MajorTopicYN="N">Encephalopathy, Bovine
Spongiform</DescriptorName>
<QualifierName MajorTopicYN="N">prevention & control</QualifierName>
<QualifierName MajorTopicYN="Y">transmission</QualifierName>
</MeshHeading>
<MeshHeading>
<DescriptorName MajorTopicYN="N">Food Contamination</DescriptorName>
<QualifierName MajorTopicYN="Y">prevention & control</QualifierName>
</MeshHeading>
<MeshHeading>
<DescriptorName MajorTopicYN="N">Great Britain</DescriptorName>
</MeshHeading>
<MeshHeading>
<DescriptorName MajorTopicYN="N">Humans</DescriptorName>
</MeshHeading>
<MeshHeading>
<DescriptorName
MajorTopicYN="Y">Meat</DescriptorNamataImporter.runCmd(DataImporter.java:389)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
370)
16.11.2010 03:28:16 org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: end_rollback
