On Tue, Dec 4, 2018 at 10:23 PM Daniel Harper wrote:
> Sorry I've been away at reinvent.
>
> Just tried out what's currently on master (with the chunked change that
> looks like it has merged). I'll do the break down of the different parts
> later but as a high level look at just running the
Sorry I've been away at reinvent.
Just tried out what's currently on master (with the chunked change that
looks like it has merged). I'll do the break down of the different parts
later but as a high level look at just running the same script as described
above these are the numbers
On Mon, Nov 19, 2018 at 11:29 PM Wes McKinney wrote:
> That seems buggy then. There is only 4.125 bytes of overhead per
> string value on average (a 32-bit offset, plus a valid bit)
> On Mon, Nov 19, 2018 at 5:02 PM Daniel Harper
> wrote:
> >
> > Uncompressed
> >
> > $ ls -la
That seems buggy then. There is only 4.125 bytes of overhead per
string value on average (a 32-bit offset, plus a valid bit)
On Mon, Nov 19, 2018 at 5:02 PM Daniel Harper wrote:
>
> Uncompressed
>
> $ ls -la concurrent_streams.csv
> -rw-r--r-- 1 danielharper 112M Nov 16 19:21
Uncompressed
$ ls -la concurrent_streams.csv
-rw-r--r-- 1 danielharper 112M Nov 16 19:21 concurrent_streams.csv
$ wc -l concurrent_streams.csv
1007481 concurrent_streams.csv
Daniel Harper
http://djhworld.github.io
On Mon, 19 Nov 2018 at 21:55, Wes McKinney wrote:
> I'm curious how the
I'm curious how the file is only 100MB if it's producing ~6GB of
strings in memory. Is it compressed?
On Mon, Nov 19, 2018 at 4:48 PM Daniel Harper wrote:
>
> Thanks,
>
> I've tried the new code and that seems to have shaved about 1GB of memory
> off, so the heap is about 8.84GB now, here is the
Thanks,
I've tried the new code and that seems to have shaved about 1GB of memory
off, so the heap is about 8.84GB now, here is the updated pprof output
https://i.imgur.com/itOHqBf.png
It looks like the majority of allocations are in the memory.GoAllocator
(pprof) top
Showing nodes accounting
hi Daniel,
On Sun, Nov 18, 2018 at 10:17 PM Daniel Harper wrote:
> Sorry just realised SVG doesn't work.
>
> PNG of the pprof can be found here: https://i.imgur.com/BVXv1Jm.png
>
>
> Daniel Harper
> http://djhworld.github.io
>
>
> On Sun, 18 Nov 2018 at 21:07, Daniel Harper wrote:
>
> > Wasn't
Sorry just realised SVG doesn't work.
PNG of the pprof can be found here: https://i.imgur.com/BVXv1Jm.png
Daniel Harper
http://djhworld.github.io
On Sun, 18 Nov 2018 at 21:07, Daniel Harper wrote:
> Wasn't sure where the best place to discuss this, but I've noticed that
> when running the
