Mike,
Thanks a lot!!! We are planning to implement element-range-query first, as we
are still v4.2-7 we don't have UDF option in hand.
Thanks for your help!!!
Abhishek Srivastav
Tata Consultancy Services
Cell:- +91-9883389968
Mailto: [email protected]
Website: http://www.tcs.com
____________________________________________
Experience certainty. IT Services
Business Solutions
Outsourcing
____________________________________________
[email protected] wrote: -----
To: MarkLogic Developer Discussion <[email protected]>
From: Michael Blakeley <[email protected]>
Sent by: [email protected]
Date: 10/24/2012 06:46PM
Subject: Re: [MarkLogic Dev General] Performance Overhead - cts:sum
Trying http://docs.marklogic.com/cts:sum-aggregate is a good idea. I'll throw
out some others, roughly in order of probability and difficulty. Stop whenever
you achieve acceptable response times.
A full 20-30 sec of warmup sounds excessive to me. A large part of the problem
is likely to be disk I/O and a rather slow disk subsystem. Monitor the OS to
test this theory.
If disk I/O doesn't seem to be the culprit, come back with output from the
profiler, from xdmp:query-meters, and from xdmp:query-trace before proceeding.
The rest of my ideas assume that disk *is* the bottleneck, and likely won't
help if it isn't.
Assuming disk I/O, check to see if the OS paging or swapping. If it needs more
memory, fix that first. You might be able to tune MarkLogic to use less, but
it's usually easier to upgrade the hardware.
If you can't find any evidence of paging, it's time to start tuning. You can
try the database setting "preload mapped data". Also the evq(id=X) portion may
not be in list-cache. You might address that with a range index on id, too.
That way you can use cts:element-range-query(xs:QName("id"),'=',$id) instead of
cts:element-value-query(), avoiding most list lookups.
If that still doesn't do it, consider upgrading the disk. Worst case, my guess
is you are asking for about 3000 disk reads - but it could be significantly
more or less, depending on configuration. So a faster disk might help, or this
might be a good use-case for SSD.
Or you might paginate across the ids, as suggested. For example display the
first 100 ids at a time. If the performance is acceptable, you're done.
Looking further, it would be nice to reduce the 500 trips through cts:sum and
cts:element-values down to one trip. It seems to me that what you're after is
really the co-occurrence of id and sum(valuation). That's tricky to get
directly from the database, but here's an approach that might perform better in
the case where you have relatively few valuations and relatively many ids.
let $map := cts:element-value-co-occurrences(
xs:QName('id'),
xs:QName('valuation'),
('map'),
cts:and-query(
(cts:collection-query('/project'),
cts:element-range-query(xs:QName("id"), '=', 1 to 500))))
for $id in map:keys($map)
return element valuation-sum {
attribute id { $id },
sum(map:get($map, $id) ! (cts:frequency(.) * .)) }
That last bit uses the simple mapping operator, which only appears in ML6:
http://developer.marklogic.com/blog/simple-mapping-operator - if you can't use
ML6, substitute a nested FLWOR expression.
...
sum(for $v in map:get($map, $id) return cts:frequency($v) * $v) }
Finally, with ML6 you might be able to move the summation into an aggregate
user-defined function (UDF):
http://docs.marklogic.com/guide/app-dev/aggregateUDFs
-- Mike
On 23 Oct 2012, at 22:20 , Abhishek53 S <[email protected]> wrote:
>
> Hi All,
>
> I am having performance penalty (Average time 23-30 Sec for the first time
> and less than 1 sec for next onward hit[which is accecpted]) with cts:sum
> over one of the element stores valuation information
>
> -----------------------------------------------------------------------
> for $id in (1 to 500)
>
> return cts:sum(cts:element-values(
>
> xs:QName("valuation"),
>
> (),
>
> (),
>
> cts:and-query((
>
>
> cts:element-value-query(xs:QName("id"),$id),
>
> cts:collection-query("/project")
>
> ))
>
> )
>
> )
>
>
> Do we have any different way to over come performance overhead.
>
> Thanks
> Abhishek Srivastav
> I T Analyst
> Tata Consultancy Services
> Cell:- +91-9883389968
> Mailto: [email protected]
> Website: http://www.tcs.com
> ____________________________________________
> Experience certainty. IT Services
> Business Solutions
> Outsourcing
> ____________________________________________
> =====-----=====-----=====
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>
> _______________________________________________
> General mailing list
> [email protected]
> http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
[email protected]
http://developer.marklogic.com/mailman/listinfo/general_______________________________________________
General mailing list
[email protected]
http://developer.marklogic.com/mailman/listinfo/general
