Re: [Rd] object.size vs lobstr::obj_size

[Stefan Schreiber]

Thank you Tomas, Hervé and Hadley for the input. Much appreciated!

Stefan

Stefan Schreiber, PhD, P. Biol.
Director EnviroStats Solutions Inc.
Adjunct Professor, Faculty of Agricultural, Life & Environmental
Sciences, University of Alberta
Phone: 780-221-1838
https://envirostats.ca/



On Fri, 27 Mar 2020 at 16:25, Hervé Pagès  wrote:
>
> On 3/27/20 15:19, Hadley Wickham wrote:
> >
> >
> > On Fri, Mar 27, 2020 at 4:01 PM Hervé Pagès  > > wrote:
> >
> >
> >
> > On 3/27/20 12:00, Hadley Wickham wrote:
> >  >
> >  >
> >  > On Fri, Mar 27, 2020 at 10:39 AM Hervé Pagès
> > mailto:hpa...@fredhutch.org>
> >  > >> wrote:
> >  >
> >  > Hi Tomas,
> >  >
> >  > On 3/27/20 07:01, Tomas Kalibera wrote:
> >  >  > they provide an over-approximation
> >  >
> >  > They can also provide an "under-approximation" (to say the
> > least) e.g.
> >  > on reference objects where the entire substance of the object is
> >  > ignored
> >  > which makes object.size() completely meaningless in that case:
> >  >
> >  > setRefClass("A", fields=c(stuff="ANY"))
> >  > object.size(new("A", stuff=raw(0)))  # 680 bytes
> >  > object.size(new("A", stuff=runif(1e8)))  # 680 bytes
> >  >
> >  > Why wouldn't object.size() look at the content of environments?
> >  >
> >  >
> >  > As the author, I'm obviously biased, but I do like
> > lobstr::obj_sizes()
> >  > which allows you to see the additional size occupied by one
> > object given
> >  > any number of other objects. This is particularly important for
> >  > reference classes since individual objects appear quite large:
> >  >
> >  > A <- setRefClass("A", fields=c(stuff="ANY"))
> >  > lobstr::obj_size(new("A", stuff=raw(0)))
> >  > #> 567,056 B
> >  >
> >  > But the vast majority is shared across all instances of that class:
> >  >
> >  > lobstr::obj_size(A)
> >  > #> 719,232 B
> >  > lobstr::obj_sizes(A, new("A", stuff=raw(0)))
> >  > #> * 719,232 B
> >  > #> * 720 B
> >  > lobstr::obj_sizes(A, new("A", stuff=runif(1e8)))
> >  > #> * 719,232 B
> >  > #> * 800,000,720 B
> >
> > Nice. Can you clarify the situation with lobstr::obj_size vs
> > pryr::object_size? I've heard of the latter before and use it sometimes
> > but never heard of the former before seeing Stefan's post. Then I
> > checked the authors of both and thought maybe they should talk to each
> > other ;-)
> >
> >
> > pryr is basically retired :) TBH I don't know why I gave up on it,
> > except lobstr is a cooler name 🤣 That's where all active development is
> > happening. (The underlying code is substantially similar although
> > lobstr includes bug fixes not present in pryr)
>
> Good to know, thanks! Couldn't find any mention of pryr being abandoned
> and superseded by lobster (which definitely sounds more yummy) in pryr's
> README.md or DESCRIPTION file. Would be good to put this somewhere.
>
> H.
>
>
> >
> > Hadley
> > --
> > http://hadley.nz
> > 
>
> --
> Hervé Pagès
>
> Program in Computational Biology
> Division of Public Health Sciences
> Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N, M1-B514
> P.O. Box 19024
> Seattle, WA 98109-1024
>
> E-mail: hpa...@fredhutch.org
> Phone:  (206) 667-5791
> Fax:(206) 667-1319

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] object.size vs lobstr::obj_size

[Hervé Pagès]

On 3/27/20 15:19, Hadley Wickham wrote:



On Fri, Mar 27, 2020 at 4:01 PM Hervé Pagès > wrote:




On 3/27/20 12:00, Hadley Wickham wrote:
 >
 >
 > On Fri, Mar 27, 2020 at 10:39 AM Hervé Pagès
mailto:hpa...@fredhutch.org>
 > >> wrote:
 >
 >     Hi Tomas,
 >
 >     On 3/27/20 07:01, Tomas Kalibera wrote:
 >      > they provide an over-approximation
 >
 >     They can also provide an "under-approximation" (to say the
least) e.g.
 >     on reference objects where the entire substance of the object is
 >     ignored
 >     which makes object.size() completely meaningless in that case:
 >
 >         setRefClass("A", fields=c(stuff="ANY"))
 >         object.size(new("A", stuff=raw(0)))      # 680 bytes
 >         object.size(new("A", stuff=runif(1e8)))  # 680 bytes
 >
 >     Why wouldn't object.size() look at the content of environments?
 >
 >
 > As the author, I'm obviously biased, but I do like
lobstr::obj_sizes()
 > which allows you to see the additional size occupied by one
object given
 > any number of other objects. This is particularly important for
 > reference classes since individual objects appear quite large:
 >
 > A <- setRefClass("A", fields=c(stuff="ANY"))
 > lobstr::obj_size(new("A", stuff=raw(0)))
 > #> 567,056 B
 >
 > But the vast majority is shared across all instances of that class:
 >
 > lobstr::obj_size(A)
 > #> 719,232 B
 > lobstr::obj_sizes(A, new("A", stuff=raw(0)))
 > #> * 719,232 B
 > #> *     720 B
 > lobstr::obj_sizes(A, new("A", stuff=runif(1e8)))
 > #> *     719,232 B
 > #> * 800,000,720 B

Nice. Can you clarify the situation with lobstr::obj_size vs
pryr::object_size? I've heard of the latter before and use it sometimes
but never heard of the former before seeing Stefan's post. Then I
checked the authors of both and thought maybe they should talk to each
other ;-)


pryr is basically retired :) TBH I don't know why I gave up on it, 
except lobstr is a cooler name 🤣 That's where all active development is 
happening. (The underlying code is substantially similar although 
lobstr includes bug fixes not present in pryr)


Good to know, thanks! Couldn't find any mention of pryr being abandoned 
and superseded by lobster (which definitely sounds more yummy) in pryr's 
README.md or DESCRIPTION file. Would be good to put this somewhere.


H.




Hadley
--
http://hadley.nz 



--
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpa...@fredhutch.org
Phone:  (206) 667-5791
Fax:(206) 667-1319

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] object.size vs lobstr::obj_size

[Hadley Wickham]

On Fri, Mar 27, 2020 at 4:01 PM Hervé Pagès  wrote:

>
>
> On 3/27/20 12:00, Hadley Wickham wrote:
> >
> >
> > On Fri, Mar 27, 2020 at 10:39 AM Hervé Pagès  > > wrote:
> >
> > Hi Tomas,
> >
> > On 3/27/20 07:01, Tomas Kalibera wrote:
> >  > they provide an over-approximation
> >
> > They can also provide an "under-approximation" (to say the least)
> e.g.
> > on reference objects where the entire substance of the object is
> > ignored
> > which makes object.size() completely meaningless in that case:
> >
> > setRefClass("A", fields=c(stuff="ANY"))
> > object.size(new("A", stuff=raw(0)))  # 680 bytes
> > object.size(new("A", stuff=runif(1e8)))  # 680 bytes
> >
> > Why wouldn't object.size() look at the content of environments?
> >
> >
> > As the author, I'm obviously biased, but I do like lobstr::obj_sizes()
> > which allows you to see the additional size occupied by one object given
> > any number of other objects. This is particularly important for
> > reference classes since individual objects appear quite large:
> >
> > A <- setRefClass("A", fields=c(stuff="ANY"))
> > lobstr::obj_size(new("A", stuff=raw(0)))
> > #> 567,056 B
> >
> > But the vast majority is shared across all instances of that class:
> >
> > lobstr::obj_size(A)
> > #> 719,232 B
> > lobstr::obj_sizes(A, new("A", stuff=raw(0)))
> > #> * 719,232 B
> > #> * 720 B
> > lobstr::obj_sizes(A, new("A", stuff=runif(1e8)))
> > #> * 719,232 B
> > #> * 800,000,720 B
>
> Nice. Can you clarify the situation with lobstr::obj_size vs
> pryr::object_size? I've heard of the latter before and use it sometimes
> but never heard of the former before seeing Stefan's post. Then I
> checked the authors of both and thought maybe they should talk to each
> other ;-)
>

pryr is basically retired :) TBH I don't know why I gave up on it, except
lobstr is a cooler name 🤣 That's where all active development is
happening. (The underlying code is substantially similar although
lobstr includes bug fixes not present in pryr)

Hadley

-- 
http://hadley.nz

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] object.size vs lobstr::obj_size

[Hervé Pagès]

On 3/27/20 12:00, Hadley Wickham wrote:



On Fri, Mar 27, 2020 at 10:39 AM Hervé Pagès > wrote:


Hi Tomas,

On 3/27/20 07:01, Tomas Kalibera wrote:
 > they provide an over-approximation

They can also provide an "under-approximation" (to say the least) e.g.
on reference objects where the entire substance of the object is
ignored
which makes object.size() completely meaningless in that case:

    setRefClass("A", fields=c(stuff="ANY"))
    object.size(new("A", stuff=raw(0)))      # 680 bytes
    object.size(new("A", stuff=runif(1e8)))  # 680 bytes

Why wouldn't object.size() look at the content of environments?


As the author, I'm obviously biased, but I do like lobstr::obj_sizes() 
which allows you to see the additional size occupied by one object given 
any number of other objects. This is particularly important for 
reference classes since individual objects appear quite large:


A <- setRefClass("A", fields=c(stuff="ANY"))
lobstr::obj_size(new("A", stuff=raw(0)))
#> 567,056 B

But the vast majority is shared across all instances of that class:

lobstr::obj_size(A)
#> 719,232 B
lobstr::obj_sizes(A, new("A", stuff=raw(0)))
#> * 719,232 B
#> *     720 B
lobstr::obj_sizes(A, new("A", stuff=runif(1e8)))
#> *     719,232 B
#> * 800,000,720 B


Nice. Can you clarify the situation with lobstr::obj_size vs 
pryr::object_size? I've heard of the latter before and use it sometimes 
but never heard of the former before seeing Stefan's post. Then I 
checked the authors of both and thought maybe they should talk to each 
other ;-)


Thanks,
H.



Hadley
--
http://hadley.nz 



--
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpa...@fredhutch.org
Phone:  (206) 667-5791
Fax:(206) 667-1319

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] object.size vs lobstr::obj_size

[Hadley Wickham]

On Fri, Mar 27, 2020 at 11:08 AM Tomas Kalibera 
wrote:

> On 3/27/20 4:39 PM, Hervé Pagès wrote:
> > Hi Tomas,
> >
> > On 3/27/20 07:01, Tomas Kalibera wrote:
> >> they provide an over-approximation
> >
> > They can also provide an "under-approximation" (to say the least) e.g.
> > on reference objects where the entire substance of the object is
> > ignored which makes object.size() completely meaningless in that case:
> >
> >   setRefClass("A", fields=c(stuff="ANY"))
> >   object.size(new("A", stuff=raw(0)))  # 680 bytes
> >   object.size(new("A", stuff=runif(1e8)))  # 680 bytes
> >
> > Why wouldn't object.size() look at the content of environments?
>
> Yes, the treatment of environments is not "over-approximative". It has
> to be bounded somewhere, you can't traverse all captured environments,
> getting to say package namespaces, global environment, code of all
> functions, that would be too over-approximating. For environments used
> as hash maps that contain data, such as in reference classes, it would
> of course be much better to include them, but you can't differentiate
> programmatically. In principle the same environment can be used for both
> things, say a namespace environment can contain data (not clearly
> related to any user-level R object) as well as code. Not mentioning
> things like source references and parse data.
>
>
I think the heuristic used in lobstr works well in practice: don't traverse
further than the current environment (supplied as an argument so you can
override), and don't ever traverse past the global or base environments.

Hadley

-- 
http://hadley.nz

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] object.size vs lobstr::obj_size

[Hadley Wickham]

On Fri, Mar 27, 2020 at 10:39 AM Hervé Pagès  wrote:

> Hi Tomas,
>
> On 3/27/20 07:01, Tomas Kalibera wrote:
> > they provide an over-approximation
>
> They can also provide an "under-approximation" (to say the least) e.g.
> on reference objects where the entire substance of the object is ignored
> which makes object.size() completely meaningless in that case:
>
>setRefClass("A", fields=c(stuff="ANY"))
>object.size(new("A", stuff=raw(0)))  # 680 bytes
>object.size(new("A", stuff=runif(1e8)))  # 680 bytes
>
> Why wouldn't object.size() look at the content of environments?
>

As the author, I'm obviously biased, but I do like lobstr::obj_sizes()
which allows you to see the additional size occupied by one object given
any number of other objects. This is particularly important for reference
classes since individual objects appear quite large:

A <- setRefClass("A", fields=c(stuff="ANY"))
lobstr::obj_size(new("A", stuff=raw(0)))
#> 567,056 B

But the vast majority is shared across all instances of that class:

lobstr::obj_size(A)
#> 719,232 B
lobstr::obj_sizes(A, new("A", stuff=raw(0)))
#> * 719,232 B
#> * 720 B
lobstr::obj_sizes(A, new("A", stuff=runif(1e8)))
#> * 719,232 B
#> * 800,000,720 B

Hadley
-- 
http://hadley.nz

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] object.size vs lobstr::obj_size

[Tomas Kalibera]

On 3/27/20 4:39 PM, Hervé Pagès wrote:

Hi Tomas,

On 3/27/20 07:01, Tomas Kalibera wrote:

they provide an over-approximation


They can also provide an "under-approximation" (to say the least) e.g. 
on reference objects where the entire substance of the object is 
ignored which makes object.size() completely meaningless in that case:


  setRefClass("A", fields=c(stuff="ANY"))
  object.size(new("A", stuff=raw(0)))  # 680 bytes
  object.size(new("A", stuff=runif(1e8)))  # 680 bytes

Why wouldn't object.size() look at the content of environments?


Yes, the treatment of environments is not "over-approximative". It has 
to be bounded somewhere, you can't traverse all captured environments, 
getting to say package namespaces, global environment, code of all 
functions, that would be too over-approximating. For environments used 
as hash maps that contain data, such as in reference classes, it would 
of course be much better to include them, but you can't differentiate 
programmatically. In principle the same environment can be used for both 
things, say a namespace environment can contain data (not clearly 
related to any user-level R object) as well as code. Not mentioning 
things like source references and parse data.


Tomas



Thanks,
H.



__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] object.size vs lobstr::obj_size

[Hervé Pagès]

Hi Tomas,

On 3/27/20 07:01, Tomas Kalibera wrote:

they provide an over-approximation


They can also provide an "under-approximation" (to say the least) e.g. 
on reference objects where the entire substance of the object is ignored 
which makes object.size() completely meaningless in that case:


  setRefClass("A", fields=c(stuff="ANY"))
  object.size(new("A", stuff=raw(0)))  # 680 bytes
  object.size(new("A", stuff=runif(1e8)))  # 680 bytes

Why wouldn't object.size() look at the content of environments?

Thanks,
H.

--
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpa...@fredhutch.org
Phone:  (206) 667-5791
Fax:(206) 667-1319

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] object.size vs lobstr::obj_size

[Tomas Kalibera]

On 2/19/20 3:55 AM, Stefan Schreiber wrote:

I have posted this question on R-help where it was suggested to me
that I might get a better response on R-devel. So far I have gotten no
response. The post I am talking about is here:
https://stat.ethz.ch/pipermail/r-help/2020-February/465700.html

My apologies for cross-posting, which I am aware is impolite and I
should have posted on R-devel in the first place - but I wasn't sure.

Here is my question again:

I am currently working through Advanced R by H. Wickham and came
across the `lobstr::obj_size` function which appears to calculate the
size of an object by taking into account whether the same object has
been referenced multiple times, e.g.

x <- runif(1e6)
y <- list(x, x, x)
lobstr::obj_size(y)
# 8,000,128 B

# versus:
object.size(y)
# 24000224 bytes

Reading through `?object.size` in the "Details" it reads: [...] but
does not detect if elements of a list are shared [...].

My questions are:

(1) is the result of `obj_size()` the "correct" one when it comes to
actual size used in memory?

(2) And if yes, why wouldn't `object.size()` be updated to reflect the
more precise calculation of an object in question similar to
`obj_size()`?


Please keep in mind that "actual size used in memory" is an elusive 
concept, particularly in managed languages such as R. Even in native 
languages, you have on-demand paging (not all data in physical memory, 
some may be imputed (all zeros), some may be swapped out, some may be 
stored in files (code), etc). Also you have internal and external 
fragmentation caused by the "C library" memory allocator, overhead of 
object headers and allocator meta-data. On top of that you have the 
managed heap: more of internal and external fragmentation, more headers. 
Moreover, memory representation may change invisibly and sometimes in 
surprising ways (in R it is copy-on-write, so the sharing, but also 
compact objects via ALTREP, e.g. sequences). R has the symbol table, 
string cache (strings are interned, as in some other language runtimes, 
so the price is paid only once for each string). In principle, managed 
runtimes could do much more, including say compression of objects with 
adaptive decompression, some systems internally split representation of 
large objects depending on their size with additional overheads, systems 
could have some transparent de-duplication (not only for strings), some 
choices could be adaptive based on memory pressure. Then in R, packages 
often can maintain memory related to specific R objects, linked say via 
external pointers, and again there may be no meaningful way to map that 
usage to individual objects.


Not only that what is a size of an object tree is not easy to define. 
That information is in addition not very useful, either, because 
innocuous changes may change it in arbitrary ways out of control of the 
user: there is no good intuition how much that size will change from 
intended application-level modifications of the tree. Users of the 
system could hardly create a reliable mental model of the memory usage, 
because it depends on internal design of the virtual machine, which in 
addition can change over time.


As the concept is elusive, the best advice would be don't ask for the 
object size, find some other solutions to your problem. In some cases, 
it makes sense to ask for object size in some application-specific way, 
and then implement object size methods for specific application classes 
(e.g. structures holding strings would sum up number of characters in 
the strings, etc). Such application-specific way may be inspired by some 
particular (perhaps trivial) serialization format.


I've used object.size() myself only for profiling when quickly 
identifying objects that are probably very large from objects of trivial 
size, where these nuances did not matter, but for that I knew roughly 
what the objects were (e.g. that they were not hiding things in 
environments).


Intuitively, the choices made by object.size() in R are conservative, 
they provide an over-approximation that somewhat intuitively makes sense 
at user level, and they reduce surprises of significant size expansion 
due to minimal updates. The choices and their limitations are 
documented. I think this at least no worse than than say taking into 
account sharing, looking at current "size" of compact objects, etc. One 
could provide more options to object.size(), but I don't think that it 
would be useful.


Best,
Tomas




There are probably valid reasons for this and any insight would be
greatly appreciated.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel


__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] object.size vs lobstr::obj_size

[Stefan Schreiber]

I have posted this question on R-help where it was suggested to me
that I might get a better response on R-devel. So far I have gotten no
response. The post I am talking about is here:
https://stat.ethz.ch/pipermail/r-help/2020-February/465700.html

My apologies for cross-posting, which I am aware is impolite and I
should have posted on R-devel in the first place - but I wasn't sure.

Here is my question again:

I am currently working through Advanced R by H. Wickham and came
across the `lobstr::obj_size` function which appears to calculate the
size of an object by taking into account whether the same object has
been referenced multiple times, e.g.

x <- runif(1e6)
y <- list(x, x, x)
lobstr::obj_size(y)
# 8,000,128 B

# versus:
object.size(y)
# 24000224 bytes

Reading through `?object.size` in the "Details" it reads: [...] but
does not detect if elements of a list are shared [...].

My questions are:

(1) is the result of `obj_size()` the "correct" one when it comes to
actual size used in memory?

(2) And if yes, why wouldn't `object.size()` be updated to reflect the
more precise calculation of an object in question similar to
`obj_size()`?

There are probably valid reasons for this and any insight would be
greatly appreciated.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel