Thanks for the Twine pointer, it was interesting.
I wasn't thinking so much about CPU efficiency, but more so scaling code.
There's a potential N² problem with lazy data structures, because binary
operation should ideally be defined for all combinations of them to reap
full benefits. Another issue is when/how to convert to Array. What should
eg. `det(::VirtualArray)` look like? It should probably convert to a full
Array first for best performance. Should it cache the Array in case another
operation needs it? There seems to be an upper-bound on how many such lazy
abstractions we can stack for any particular problem without making it too
hard to reason about performance. OTOH, Haskell seems to manage with
laziness, kind-of? But they don't have mutability, and that simplifies the
problem.
Cédric
On Saturday, January 16, 2016 at 11:16:04 AM UTC-5, Erik Schnetter wrote:
>
> There's probably a threshold: Below, it's cheaper to copy, and above,
> it's cheaper to re-use the existing arrays. LLVM goes to great length
> to optimize string operations in this respect, with their own string
> class ("Twine"). Obviously what works for strings can be extended to
> work with any kind of array. I'd argue that, once you go to parallel
> processing, keeping independent chunks of data instead of having
> everything in a single array is a clear performance benefit.
>
> So: Does it scale? Yes. Is it efficient? That depends on where you
> place the threshold. My guess is that it's somewhere near the L1 cache
> size.
>
> -erik
>
>
> On Sat, Jan 16, 2016 at 10:52 AM, Cedric St-Jean
> <[email protected] <javascript:>> wrote:
> > That's cool, a lazy vcat!
> >
> > I wrote a similar class that's a lazy `fill`, and it worked really well
> for
> > my purposes. I wonder what the endgame will be for these datastructures.
> 0.5
> > will bring us "lazy slicing". Should everything be lazy by default? Does
> it
> > scale?
> >
> > On Friday, January 15, 2016 at 6:14:26 PM UTC-5, Yichao Yu wrote:
> >>
> >> On Fri, Jan 15, 2016 at 5:51 PM, Samuel Massinon
> >> <[email protected]> wrote:
> >> > A way to concatenate arrays without copying values.
> >> >
> >> > https://github.com/invenia/VirtualArrays.jl
> >>
> >> FYI, don't call eval in functions
>
>
>
> --
> Erik Schnetter <[email protected] <javascript:>>
> http://www.perimeterinstitute.ca/personal/eschnetter/
>
