What about combining all the urls into one big pattern, and using regex
("SIMILAR TO" or "~") ?
I'm sure the number of cases in OR is the culprit of the slowness.
Parallelizing may help in wall time, but the query is very inefficient, and
running it 1000 times (even if only 32 concurrently) will be a big resource
hog,
as each run will scan thorugh the whole table, inspecting each and every
url, for each and every pattern.
That's why I suggest to try to combine the pattern matching into one query,
and make that fast first!
It you've reached your limits, THEN you can start thinking about
parallellization!
For example, you can split the search space by running 32 queries in
parallel, each scanning only 1/32 of the whole search space - "MOD(id, 32)
= $1".
Or even better (more uniform and with bigger locality):
"MOD((ctid::text::point)[0]::bigint, 32) = $1".
2019. február 26., kedd 7:08:00 UTC+1 időpontban RZ a következőt írta:
>
> Thanks Tamas,
>
> Query itself is slow if i include all url strings. it takes about 10 mins.
> But when i hit one at a time, i see better response overall. Yes so was
> planning on running them in parallel.
>
> We only have read permissions and hence i am not allowed to add new index,
> but good idea, will try reaching out to Admin and see if they can add this.
>
> i have not tried that way of similar to, will try that as well.
>
>
> On Tuesday, 26 February 2019 00:36:26 UTC+5:30, Tamás Gulácsi wrote:
>>
>> How fast is the query? You can make it parallel, but if it is sliw, the
>> you have to target that first.
>>
>> How big is the set of one user's all urls? How fast is to get this? Maybe
>> adding some indexes may help
>>
>>
>> How fast is the query with one pattern only? Maybe combining them into a
>> "similar to '%(abc|def|ghi)%'" would be faster?
>>
>>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
