Hi,
We have a need to pull large amounts of data out of marklogic as quickly as
possible. I found that doing xquery searches like query-by-example were very
slow. Using the cts functions led to a big speed increase. However, it isn't
clear whether my current approach is the optimum, or whether there are other
better alternatives. Unfortunately, while there is a lot of documentation
describing many different ways of doing things in marklogic, there seems to be
very little documentation describing what are the best or most efficient
approaches (e.g., what if your goal is not only to run a query successfully,
but to maximize its performance?). At present, I'm using the java api to pull
documents. I'm using the theCall.xquery(query) function in Java to run custom
xquery through the rest api. The xquery is as follows:
<data>
for $x in cts:search(fn:doc(),cts:and-query((
cts:element-value-query(xs:QName('Department'), 'Sales'),
cts:element-range-query(xs:QName('Date'), '>', xs:date('2015-01-01')),
cts:element-range-query(xs:QName('Date'), '<', xs:date('2015-01-03')),
cts:not-query(cts:element-value-query(xs:QName('Date'), 'NULL'))
)), 'unfiltered' , 0.0)
)), 'unfiltered' , 0.0)
return fn:concat($x//Department,'|',$x//Total,'|',$x//Location' ')}
</data>
There are indexes on Date and Department. The xquery wraps all of the documents
in the <data> tags and sends the results to the java program. It then strips
the <data> tags and prints the results to text file.
I have found that you can run multiple threads in the java that request
different "chunks" of the data by using the criterions of [1 to 1000000],
[1000001 to 2000000], etc.
This approach is much faster than our original approach - 12 hours with 8
threads, rather than 75 hours using query-by-example. However, it is not clear
if this is the fastest way, or there are further optimizations or better
approaches. For instance, when pulling the actual elements from the documents,
I found that having them indexed made no different to performance. Is there a
way of pulling from the indexes to improve performance? Is there a way to
specify the elements you want in the cts:search that will improve performance?
Is there a more efficient way to restrict the search range? Is there
documentation describing the most efficient approaches to querying marklogic?
Thanks.
_______________________________________________
General mailing list
[email protected]
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general
