Hi Jason,
Sorry for the confusion.
Please find below the snippet of the xml we have in DB. (DB is having 10
million xml documents)
<ja:item-info>
<ja:jid>YMSG</ja:jid>
<ja:aid>0103883</ja:aid>
<ce:pii>S0011-3840(01)70009-3</ce:pii>
<ce:doi>10.1016/S0011-3840(01)70009-3</ce:doi>
<ce:copyright type="other" year="2001"/>
</ja:item-info>
The file we used to upload will have the PIIs (which I had mentioned as terms
in my earlier email) as shown below: (There could be 100k PIIs in the file)
S0016-5085(68)70198-0
S0016-5085(68)70199-2
S0016-5085(68)70200-6
S0016-5085(68)70201-8
S0016-5085(68)70202-X
S0016-5085(68)70203-1
S0016-5085(68)70204-3
.....
.....
I need to identify documents that matches the PIIs (which I had mentioned as
terms in my earlier email) in the file.
Currently we are using search:search() API in our application. Hence I had
tried using the additional query option of search API as shown below:
cts:element-value-query(xs:QName("ce:pii"), $uploadedPIIs as xs:string*)
But this additional query option is taking lot of time to yield result.
So is there any other better way to perform this? Please suggest.
Regards,
Vijay
From: [email protected]
[mailto:[email protected]] On Behalf Of Jason Hunter
Sent: Wednesday, July 20, 2011 12:32 PM
To: General MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] Search using 100k terms
You say "the term" but you also say you have 300,000 terms. So I'm confused.
You want to find documents that have all 300,000 terms?
Or for each term you want to find documents having just that term? And you
want to do that basic query 300,000 times across all terms in less than 10
seconds?
-jh-
On Jul 19, 2011, at 11:13 PM, Vijayasekar Padmanaban wrote:
Hi Jason,
Thanks for your response.
My DB is having 10 million documents in it. I need to identify the documents
which have the term.
I would expect search to retrieve results less than 10 seconds.
Regards,
Vijay
From:
[email protected]<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Jason Hunter
Sent: Wednesday, July 20, 2011 11:33 AM
To: General MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] Search using 100k terms
I'm a little unclear on what you're trying to do.
You want to take a list of 300,000 terms and identify which documents have each
term? Or do you only need to identify which terms are present in one or more
documents and which terms aren't present anywhere? Something else?
How long are you willing to wait for the answer?
-jh-
On Jul 19, 2011, at 10:45 PM, Vijayasekar Padmanaban wrote:
Hi All,
We have a use case to perform search based on the contents uploaded as a file.
The file would have a max of 100,000 terms in it. We need to validate the
contents of the file with our repository contents and produce results. Our
repository contains 10 million contents. Each term in the file need to be
validated with an element in the enhanced xml.
Below are the two approached I had tried:
1. Using search constraints
a. Each search term would be concatenated with the constraint and would
be joined using 'OR' delimiter as shown below:
For e.g., "const:<term1> OR const:<term2> OR const:<term3> OR const:<term3> OR
....."
This ended in stack overflow error when the
number of search terms exceeded 1000
2. Using element value query
a. All the search terms would be passed as text to the
cts:element-value-query as shown below:
cts:element-value-query(<Qualifier-Name>, text as xs:string*)
This worked well when DB contains less number
of contents say 300,000. But when used with DB that has 10 million contents it
failed saying "Time limit exceeded"
Could you suggest me the best possible approach to resolve this issue?
Thanks,
Vijay
**************** CAUTION - Disclaimer *****************
This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
for the use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you are
not
to copy, disclose, or distribute this e-mail or its contents to any other
person and
any such actions are unlawful. This e-mail may contain viruses. Infosys has
taken
every reasonable precaution to minimize this risk, but is not liable for any
damage
you may sustain as a result of any virus in this e-mail. You should carry out
your
own virus checks before opening the e-mail or attachment. Infosys reserves the
right to monitor and review the content of all messages sent to or from this
e-mail
address. Messages sent to or from this e-mail address may be stored on the
Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
_______________________________________________
General mailing list
[email protected]<mailto:[email protected]>
http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
[email protected]<mailto:[email protected]>
http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
[email protected]
http://developer.marklogic.com/mailman/listinfo/general
