Grant:
I think that "Parsing 70 files totally takes 80 minutes" really
means parsing 70 metadata files containing 10,000 XML
files each.....
One Metadata File is split into 10,000 XML files which looks as
below:
<root>
<record>
<header>
<identifier>oai:CiteSeerPSU:1</identifier>
<datestamp>1993-08-11</datestamp>
<setSpec>CiteSeerPSUset</setSpec>
</header>
<metadata>
<oai_citeseer:oai_citeseer
xmlns:oai_citeseer="http://copper.ist.psu.edu/oai/oai_citeseer/";
xmlns:dc
="http://purl.org/dc/elements/1.1/";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://copper.ist.psu.edu/oai/oai_citeseer/
http://copper.ist.psu.edu/oai/oai_citeseer.xsd ">
<dc:title>36 Problems for Semantic Interpretation</dc:title>
<oai_citeseer:author name="Gabriele Scheler">
<address>80290 Munchen , Germany</address>
<affiliation>Institut fur Informatik; Technische
Universitat
Munchen</affiliation>
</oai_citeseer:author>
<dc:subject>Gabriele Scheler 36 Problems for Semantic
Interpretation</dc:subject>
<dc:description>This paper presents a collection of problems for
natural
language analysisderived mainly from theoretical linguistics. Most of
these problemspresent major obstacles for computational systems of
language interpretation.The set of given sentences can easily be
scaled up
by introducing moreexamples per problem. The construction of
computational
systems couldbenefit from such a collection, either using it
directly for
training andtesting or as a set of benchmarks to qualify the
performance
of a NLPsystem.1 IntroductionThe main part of this paper consists
of a
collection of problems for semanticanalysis of natural language. The
problems are arranged in the following way:example sentencesconcise
description of the problemkeyword for the type of problemThe sources
(first appearance in print) of the sentences have been left
out,because
they are sometimes hard to track and will usually not be of much
use,as
they indicate a starting-point of discussion only. The keywords
howeve...
</dc:description>
<dc:contributor>The Pennsylvania State University CiteSeer
Archives</dc:contributor>
<dc:publisher>unknown</dc:publisher>
<dc:date>1993-08-11</dc:date>
<dc:format>ps</dc:format>
<dc:identifier>http://citeseer.ist.psu.edu/1.html</dc:identifier>
<dc:source>ftp://flop.informatik.tu-muenchen.de/pub/fki/
fki-179-93.ps.gz</dc:source>
<dc:language>en</dc:language>
<dc:rights>unrestricted</dc:rights>
</oai_citeseer:oai_citeseer>
</metadata>
</record>
</root>
From the above I will extract the Title and the Description tags
to index.
Code to do this:
1. I have 70 directories with the name like oai_citeseerXYZ/
2. Under each of above directory, I have 10,000 xml files each having
above xml data.
3. Program does the following
File dir = new File(dirName);
String[] children = dir.list();
if (children == null) {
// Either dir does not exist or
is not a directory
}
else
{
for (int ii=0;
ii<children.length; ii++)
{
// Get filename of file
or directory
String file =
children[ii];
//System.out.println("The name
of file parsed now ==> "+file);
nl =
ReadDump.getNodeList(filename+"/"+file, "metadata");
if(nl == null)
{
//System.out.println("Error shoudlnt be thrown ...");
}
//Get the metadata
element tags from xml file
ReadDump rd = new
ReadDump();
//Get the Extracted
Tags Title, Identifier and Description
ArrayList alist_Title =
rd.getElements(nl, "dc:title");
ArrayList alist_Descr =
rd.getElements(nl, "dc:description");
//Create an Index under
DIR
IndexWriter writer = new
IndexWriter("./FINAL/", new
StopStemmingAnalyzer(),false);
Document doc = new
Document();
//Get Array List
Elements and add them as fileds to doc
for(int k=0; k <
alist_Title.size(); k++)
{
doc.add(new
Field("Title",alist_Title.get(k).toString(),
Field.Store.YES, Field.Index.UN_TOKENIZED));
}
for(int k=0; k <
alist_Descr.size(); k++)
{
doc.add(new Field("Description",alist_Descr.get(k).toString
(),
Field.Store.YES, Field.Index.UN_TOKENIZED));
}
//Add the document created out of those fields to the
IndexWriter which
will create and index
writer.addDocument(doc);
writer.optimize();
writer.close();
}
This is the main file which does indexing.
Hope this will give you an idea.
Lokeya:
Can you confirm my supposition? And I'd still post the code
Grant requested if you can.....
So, you're talking about indexing 10,000 xml files in 2-3 hours,
8 minutes or so which is spent reading/parsing, right? It'll be
important to know how much data you're indexing and now, so
the code snippet is doubly important....
Erick
On 3/18/07, Grant Ingersoll <[EMAIL PROTECTED]> wrote:
Can you post the relevant indexing code? Are you doing things like
optimizing after every file? Both the parsing and the indexing
sound
really long. How big are these files?
Also, I assume you machine is at least somewhat current, right?
On Mar 18, 2007, at 1:00 AM, Lokeya wrote:
Thanks for your reply. I tried to check if the I/O and Parsing is
taking time
separately and Indexing time also. I observed that I/O and Parsing
70 files
totally takes 80 minutes where as when I combine this with Indexing
for a
single Metadata file it nearly 2 to 3 hours. So looks like
IndexWriter takes
time that too when we are appending to the Index file this happens.
So what is the best approach to handle this?
Thanks in Advance.
Erick Erickson wrote:
See below...
On 3/17/07, Lokeya <[EMAIL PROTECTED]> wrote:
Hi,
I am trying to index the content from XML files which are
basically the
metadata collected from a website which have a huge collection of
documents.
This metadata xml has control characters which causes errors
while trying
to
parse using the DOM parser. I tried to use encoding = UTF-8 but
looks
like
it doesn't cover all the unicode characters and I get error. Also
when I
tried to use UTF-16, I am getting Prolog content not allowed
here. So my
guess is there is no enoding which is going to cover almost all
unicode
characters. So I tried to split my metadata files into small
files and
processing records which doesnt throw parsing error.
But by breaking metadata file into smaller files I get, 10,000
xml files
per
metadata file. I have 70 metadata files, so altogether it becomes
7,00,000
files. Processing them individually takes really long time using
Lucene,
my
guess is I/O is time consuing, like opening every small xml file
loading
in
DOM extracting required data and processing.
So why don't you measure and find out before trying to make the
indexing
step more efficient? You simply cannot optimize without knowing
where
you're spending your time. I can't tell you how often I've been
wrong
about
"why my program was slow" <G>.
In this case, it should be really simple. Just comment out the
part where
you index the data and run, say, one of your metadata files.. I
suspect
that
Cheolgoo Kang's response is cogent, and you indeed are spending
your
time parsing the XML. I further suspect that the problem is not
disk IO,
but the time spent parsing. But until you measure, you have no
clue
whether you should mess around with the Lucene parameters, or find
another parser, or just live with it.. Assuming that you
comment out
Lucene and things are still slow, the next step would be to just
read in
each file and NOT parse it to figure out whether it's the IO or
the
parsing.
Then you can worry about how to fix it..
Best
Erick
Qn 1: Any suggestion to get this indexing time reduced? It
would be
really
great.
Qn 2 : Am I overlooking something in Lucene with respect to
indexing?
Right now 12 metadata files take 10 hrs nearly which is really a
long
time.
Help Appreciated.
Much Thanks.
--
View this message in context:
http://www.nabble.com/Issue-while-parsing-XML-files-due-to-
control-characters%2C-help-appreciated.-tf3418085.html#a9526527
Sent from the Lucene - Java Users mailing list archive at
Nabble.com.
-----------------------------------------------------------------
---
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
View this message in context: http://www.nabble.com/Issue-while-
parsing-XML-files-due-to-control-characters%2C-help-appreciated.-
tf3418085.html#a9536099
Sent from the Lucene - Java Users mailing list archive at
Nabble.com.
-------------------------------------------------------------------
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------
Grant Ingersoll
Center for Natural Language Processing
http://www.cnlp.org
Read the Lucene Java FAQ at http://wiki.apache.org/jakarta-lucene/
LuceneFAQ
--------------------------------------------------------------------
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
