On Wednesday, 30 November 2016 03:37:55 UTC+2, Mandolyte wrote:
>
> I have a fairly large 2D slice of strings, about 115M rows. These are 
> (parent, child) pairs that, processed recursively, form a tree. I am 
> "materializing" all possible trees as well as determining for each root 
> node all child nodes for all levels for it.
>
> I am using the simple sort.Search() to locate each parent and, without 
> profiling, I suspect that a large chunk of time is being spent in that 
> sort.Search() function.
>
> Since the data is static, I am leveraging multiple go routines, but find 
> that 3 concurrent searches for Win7 and 7 concurrent on Linux is optimum. 
> Some of the larger trees are still taking 20+ minutes to process. So 
> concurrency helps, but has limits.
>
> I wondered if there were a way to significantly improve the search 
> performance. A search of godoc turned up some in-memory key-value stores; 
> but probably excessive for what I need. I was thinking of a btree index of 
> the parent column returning the index might do the trick.
>
> Any advice?
>

The optimal solution will depend on the data, data distribution, how it's 
updated, and what the searches look like. 

Can you show the code you are using? If you can't use the actual data, can 
you create a simulator for the data? Or if it's obvious what data it is, 
then a sample set, would be sufficient as well. It will make the discussion 
much easier.

+ Egon

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to