Re: [Rd] [External] Re: Choices to remove `srcref` (and its buddies) when serializing objects

[luke-tierney--- via R-devel]

On Thu, 18 Jan 2024, Ivan Krylov via R-devel wrote:


В Tue, 16 Jan 2024 14:16:19 -0500
Dipterix Wang  пишет:


Could you recommend any packages/functions that compute hash such
that the source references and sexpinfo_struct are ignored? Basically
a version of `serialize` that convert R objects to raw without
storing the ancillary source reference and sexpinfo.


I can show how this can be done, but it's not currently on CRAN or even
a well-defined package API. I have adapted a copy of R's serialize()
[*] with the following changes:

* Function bytecode and flags are ignored:

f <- function() invisible()
depcache:::hash(f, 2) # This is plain FNV1a-64 of serialize() output
# [1] "9b7a1af5468deba4"
.Call(depcache:::C_hash2, f) # This is the new hash
[1] 91 5f b8 a1 b0 6b cb 40
f() # called once: function gets the MAYBEJIT_MASK flag
depcache:::hash(f, 2)
# [1] "7d30e05546e7a230"
.Call(depcache:::C_hash2, f)
# [1] 91 5f b8 a1 b0 6b cb 40
f() # called twice: function now has bytecode
depcache:::hash(f, 2)
# [1] "2a2cba4150e722b8"
.Call(depcache:::C_hash2, f)
# [1] 91 5f b8 a1 b0 6b cb 40 # new hash stays the same

* Source references are ignored:

.Call(depcache:::C_hash2, \( ) invisible( ))
# [1] 91 5f b8 a1 b0 6b cb 40 # compare vs. above

# For quoted function definitions, source references have to be handled
# differently
.Call(depcache:::C_hash2, quote(function(){}))
[1] 58 0d 44 8e d4 fd 37 6f
.Call(depcache:::C_hash2, quote(\( ){  }))
[1] 58 0d 44 8e d4 fd 37 6f

* ALTREP is ignored:

identical(1:10, 1:10+0L)
# [1] TRUE
identical(serialize(1:10, NULL), serialize(1:10+0L, NULL))
# [1] FALSE
identical(
.Call(depcache:::C_hash2, 1:10),
.Call(depcache:::C_hash2, 1:10+0L)
)
# [1] TRUE

* Strings not marked as bytes are encoded into UTF-8:

identical('\uff', iconv('\uff', 'UTF-8', 'latin1'))
# [1] TRUE
identical(
serialize('\uff', NULL),
serialize(iconv('\uff', 'UTF-8', 'latin1'), NULL)
)
# [1] FALSE
identical(
.Call(depcache:::C_hash2, '\uff'),
.Call(depcache:::C_hash2, iconv('\uff', 'UTF-8', 'latin1'))
)
# [1] TRUE

* NaNs with different payloads (except NA_numeric_) are replaced by
  R_NaN.

One of the many downsides to the current approach is that we rely on
the non-API entry point getPRIMNAME() in order to hash builtins.
Looking at the source code for identical() is no help here, because it
uses the private PRIMOFFSET macro.

The bitstream being hashed is also, unfortunately, not exactly
compatible with R serialization format version 2: I had to ignore the
LEVELS of the language objects being hashed both because identical()
seems to ignore those and because I was missing multiple private
definitions (e.g. the MAYBEJIT flag) to handle them properly.

Then there's also the problem of immediate bindings [**]: I've seen bits
of vctrs, rstudio, rlang blow up when calling CAR() on SEXP objects that
are not safe to handle this way, but R_expand_binding_value() (used by
serialize()) is again a private function that is not accessible from
packages. identical() won't help here, because it compares reference
objects (which may or may not contain such immediate bindings) by their
pointer values instead of digging down into them.


What does 'blow up' mean? If it is anything other than signal a "bad
binding access" error then it would be good to have more details.

Best,

luke


Dropping the (already violated) requirement to be compatible with R
serialization bitstream will make it possible to simplify the code
further.

Finally:

a <- new.env()
b <- new.env()
a$x <- b$x <- 42
identical(a, b)
# [1] FALSE
.Call(depcache:::C_hash2, a)
# [1] 44 21 f1 36 5d 92 03 1b
.Call(depcache:::C_hash2, b)
# [1] 44 21 f1 36 5d 92 03 1b

...but that's unavoidable when looking at frozen object contents
instead of their live memory layout.

If you're interested, here's the development version of the package:
install.packages('depcache',contriburl='https://aitap.github.io/Rpackages')




--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] readChar() could read the whole file by default?

[luke-tierney--- via R-devel]

On Fri, 26 Jan 2024, Michael Chirico wrote:


I am curious why readLines() has a default (n=-1L) to read the full
file while readChar() has no default for nchars= (i.e., readChar(file)
is an error). Is there a technical reason for this?

I often[1] see code like paste(readLines(f), collapse="\n") which
would be better served by readChar(), especially given issues with the
global string cache I've come across[2]. But lacking the default, the
replacement might come across less clean.


The string cache seems like a very dark pink herring to me. The fact
that the lines are allocated on the heap might create an issue; the
cache isn't likely to add much to that. In any case I would need to
see a realistic example to convince me this is worth addressing on
performance grounds.

I don't see any reason in principle not to have readChar and readBin
read the entire file if n = -1 (others might) but someone would need
to write a patch to implement that.

Best,

luke


For my own purposes the incantation readChar(file, file.size(file)) is
ubiquitous. Taking CRAN code[3] as a sample[4], 41% of readChar()
calls use either readChar(f, file.info(f)$size) or readChar(f,
file.size(f))[5].

Thanks for the consideration and feedback,
Mike C

[1] e.g. a quick search shows O(100) usages in CRAN packages:
https://github.com/search?q=org%3Acran+%2Fpaste%5B%28%5D%5Cs*readLines%5B%28%5D.*%5B%29%5D%2C%5Cs*collapse%5Cs*%3D%5Cs*%5B%27%22%5D%5B%5C%5C%5D%2F+lang%3AR&type=code,
and O(1000) usages generally on GitHub:
https://github.com/search?q=lang%3AR+%2Fpaste%5B%28%5D%5Cs*readLines%5B%28%5D.*%5B%29%5D%2C%5Cs*collapse%5Cs*%3D%5Cs*%5B%27%22%5D%5B%5C%5C%5D%2F+lang%3AR&type=code
[2] AIUI the readLines() approach "pollutes" the global string cache
with potentially 1000s/1s of strings for each line, only to get
them gc()'d after combining everything with paste(collapse="\n")
[3] The mirror on GitHub, which includes archived packages as well as
current (well, eventually-consistent) versions.
[4] Note that usage in packages is likely not representative of usage
in scripts, e.g. I often saw readChar(f, 1), or eol-finders like
readChar(f, 500) + grep("[\n\r]"), which makes more sense to me as
something to find in package internals than in analysis scripts. FWIW
I searched an internal codebase (scripts and packages) and found 70%
of usages reading the full file.
[5] repro: 
https://gist.github.com/MichaelChirico/247ea9500460dca239f031e74bdcf76b
requires GitHub PAT in env GITHUB_PAT for API permissions.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Get list of active calling handlers?

[luke-tierney--- via R-devel]

On Tue, 6 Feb 2024, Duncan Murdoch wrote:

The SO post https://stackoverflow.com/q/77943180 tried to call 
globalCallingHandlers() from a function, and it failed with the error message 
"should not be called with handlers on the stack".  A much simpler 
illustration of the same error comes from this line:


 try(globalCallingHandlers(warning = function(e) e))

The problem here is that try() sets an error handler, and 
globalCallingHandlers() sees it and aborts.


If I call globalCallingHandlers() with no arguments, I get a list of 
currently active global handlers.  Is there also a way to get a list of 
active handlers, including non-global ones (like the one try() added in the 
line above)?


There is not. The internal stack is not safe to allow to escape to the
R level.  It would be possible to write a reflection function to
provide some information, but it would be a fair bit of work to design
and I don't think would be of enough value to justify that.

The original SO question would be better addressed to
Posit/RStudio. Someone with enough motivation might also be able to
figure out an answer by looking at the source code at
https://github.com/rstudio/rstudio.

Best,

luke




Duncan Murdoch

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Ordered comparison operators on language objects will signal errors

[luke-tierney--- via R-devel]

Comparison operators == and != can be used on language objects
(i.e. call objects and symbols). The == operator in particular often
seems to be used as a shorthand for calling identical(). The current
implementation involves comparing deparsed calls as strings. This has
a number of drawbacks and we would like to transition to a more robust
and efficient implementation. As a first step, R-devel will soon be
modified to signal an error when the ordered comparison operators <,
<=, >, >= are used on language objects. A small number of CRAN and
BIOC packages will fail after this change. If you want to check your
packages or code before the change is committed you can run the
current R-devel with the environment variable setting

_R_COMPARE_LANG_OBJECTS=eqonly

where using such a comparison now produces

> quote(x + y) > 1
Error in quote(x + y) > 1 :
  comparison (>) is not possible for language types

Best,

luke


--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Re: Bug in out-of-bounds assignment of list object to expression() vector

[luke-tierney--- via R-devel]

On Fri, 5 Apr 2024, Ivan Krylov via R-devel wrote:


On Fri, 5 Apr 2024 08:15:20 -0400
June Choe  wrote:


When assigning a list to an out of bounds index (ex: the next, n+1
index), it errors the same but now changes the values of the vector
to NULL:

```
x <- expression(a,b,c)
x[[4]] <- list() # Error
x
#> expression(NULL, NULL, NULL)
```

Curiously, this behavior disappears if a prior attempt is made at
assigning to the same index, using a different incompatible object
that does not share this bug (like a function)


Here's how the problem happens:

1. The call lands in src/main/subassign.c, do_subassign2_dflt().

2. do_subassign2_dflt() calls SubassignTypeFix() to prepare the operand
for the assignment.

3. Since the assignment is "stretching", SubassignTypeFix() calls
EnlargeVector() to provide the space for the assignment.

The bug relies on `x` not being IS_GROWABLE(), which may explain
why a plain x[[4]] <- list() sometimes doesn't fail.

The future assignment result `x` is now expression(a, b, c, NULL), and
the old `x` set to expression(NULL, NULL, NULL) by SET_VECTOR_ELT(newx,
i, VECTOR_ELT(x, i)); CLEAR_VECTOR_ELT(x, i); during EnlargeVector().

4. But then the assignment fails, raising the error back in
do_subassign2_dflt(), because the assignment kind is invalid: there is
no way to put data.frames into an expression vector. The new resized
`x` is lost, and the old overwritten `x` stays there.

Not sure what the right way to fix this is. It's desirable to avoid
shallow_duplicate(x) for the overwriting assignments, but then the
sub-assignment must either succeed or leave the operand untouched.
Is there a way to perform the type check before overwriting the operand?


Yes. There are two places where there are some checks, one early and
the other late. The early one is explicitly letting this one through
and shouldn't. So a one line change would address this particular
problem. But it would be a good idea to review why we the late checks
are needed at all and maybe change that. I'll look into it.

Best,

luke

--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Bug in out-of-bounds assignment of list object to expression() vector

[luke-tierney--- via R-devel]

Thanks for the report. Fixed in R-devel and R-patched (both
R-4-4-branch and R-4-3-branch).

On Fri, 5 Apr 2024, June Choe wrote:


[You don't often get email from jchoe...@gmail.com. Learn why this is important 
at https://aka.ms/LearnAboutSenderIdentification ]

There seems to be a bug in out-of-bounds assignment of list objects to an
expression() vector. Tested on release and devel. (Many thanks to folks
over at Mastodon for the help narrowing down this bug)

When assigning a list into an existing index, it correctly errors on
incompatible type, and the expression vector is unchanged:

```
x <- expression(a,b,c)
x[[3]] <- list() # Error
x
#> expression(a, b, c)
```

When assigning a list to an out of bounds index (ex: the next, n+1 index),
it errors the same but now changes the values of the vector to NULL:

```
x <- expression(a,b,c)
x[[4]] <- list() # Error
x
#> expression(NULL, NULL, NULL)
```

Curiously, this behavior disappears if a prior attempt is made at assigning
to the same index, using a different incompatible object that does not
share this bug (like a function):

```
x <- expression(a,b,c)
x[[4]] <- base::sum # Error
x[[4]] <- list() # Error
x
#> expression(a, b, c)
```

That "protection" persists until x[[4]] is evaluated, at which point the
bug can be produced again:

```
x[[4]] # Error
x[[4]] <- list() # Error
x
#> expression(NULL, NULL, NULL)
```

Note that `x` has remained a 3-length vector throughout.

Best,
June

   [[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu/

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Re: Repeated library() of one package with different include.only= entries

[luke-tierney--- via R-devel]

On Thu, 11 Apr 2024, Duncan Murdoch wrote:


On 11/04/2024 7:04 a.m., Martin Maechler wrote:

Michael Chirico
 on Mon, 8 Apr 2024 10:19:29 -0700 writes:


 > Right now, attaching the same package with different include.only= 
has no

 > effect:

 > library(Matrix, include.only="fac2sparse")
 > library(Matrix)
 > ls("package:Matrix")
 > # [1] "fac2sparse"

 > ?library does not cover this case -- what is covered is the 
_loading_

 > behavior of repeated calls:

 >> [library and require] check and update the list of currently 
attached

 > packages and do not reload a namespace which is already loaded

 > But here we're looking at the _attach_ behavior of repeated calls.

 > I am particularly interested in allowing the exports of a package to 
be

 > built up gradually:

 > library(Matrix, include.only="fac2sparse")
 > library(Matrix, include.only="isDiagonal") # want: 
ls("package:Matrix") -->

 > c("fac2sparse", "isDiagonal")
 > ...

 > It seems quite hard to accomplish this at the moment. Is the 
behavior to

 > ignore new inclusions intentional? Could there be an argument to get
 > different behavior?

As you did not get an answer yet, ..., some remarks by an
R-corer who has tweaked library() behavior in the past :

- The `include.only = *` argument to library() has been a
   *relatively* recent addition {given the 25+ years of R history}:

   It was part of the extensive new features by Luke Tierney for
   R 3.6.0  [r76248 | luke | 2019-03-18 17:29:35 +0100], with NEWS entry

 • library() and require() now allow more control over handling
   search path conflicts when packages are attached. The policy is
   controlled by the new conflicts.policy option.

- I haven't seen these (then) new features been used much, unfortunately,
   also not from R-core members, but I'd be happy to be told a different 
story.


For the above reasons, it could well be that the current
implementation {of these features} has not been exercised a lot
yet, and limitations as you found them haven't been noticed yet,
or at least not noticed on the public R mailing lists, nor
otherwise by R-core (?).

Your implicitly proposed new feature (or even *changed*
default behavior) seems to make sense to me -- but as alluded
to, above, I haven't been a conscious user of any
'library(.., include.only = *)' till now.


I don't think it makes sense.  I would assume that

 library(Matrix, include.only="isDiagonal")

implies that only `isDiagonal` ends up on the search path, i.e. 
"include.only" means "include only", not "include in addition to whatever 
else has already been attached".


I think a far better approach to solve Michael's problem is simply to use

 fac2sparse <- Matrix::fac2sparse
 isDiagonal <- Matrix::isDiagonal

instead of messing around with the user's search list, which may have been 
intentionally set to include only one of those.


So I'd suggest changing the docs to say

"[library and require] check and update the list of currently attached
packages and do not reload a namespace which is already loaded.  If a package 
is already attached, no change will be made."


?library could also mention using detach() followed by library() or
attachNamespace() with a new include.only specification.

Best,

luke



Duncan Murdoch

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu/
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Calling applyClosure from a package?

[luke-tierney--- via R-devel]

On Sun, 14 Apr 2024, Matthew Kay wrote:


[You don't often get email from matthew@u.northwestern.edu. Learn why this 
is important at https://aka.ms/LearnAboutSenderIdentification ]

Hi,

Short version of my question: Rf_applyClosure was marked
attribute_hidden in Oct 2023, and I am curious why and if there is an
alternative interface to it planned.


applyClosure has never been part of the API and was/is not intended
for use by packages. Keeping things like this internal is essential to
give us flexibility to make needed improvements to the basic engine.
Moving this out of the installed headers and marking it as not to be
exported merely clarifies that it is internal.


Long version:

I have been toying with building a package that makes it easier to do
non-standard evaluation directly using promises, rather than wrapping
these in a custom type (like e.g. rlang does). The advantage of this
approach is that it should be fully compatible with functions that use
the standard R functions for NSE and inspecting function context, like
substitute(), match.call(), or parent.frame(). And indeed, it works!
-- in R 4.3, that is. The prototype version of the package is here:
https://github.com/mjskay/uneval  (the relevant function to my
question is probably do_invoke, in R/invoke.R).

While testing on R-devel, I noticed that Rf_applyClosure(), which used
to be exported, is now marked with attribute_hidden. I traced the
change to this commit in Oct 2023:
https://github.com/r-devel/r-svn/commit/57dbe8ad471c8a34314ee74362ad479db03c033a

However, the commit message did not give me clarity on the reason for
the change, and I have not been able to find mention of this change in
R-devel, R-package-devel, or the R bug tracker.
So, I am curious why this function is no longer exported and if there
is an alternative function planned to take its place.

Neither Rf_eval nor do.call can do what I need to fully support
rlang-style NSE using base R. The problem is that I need to be able to
manually set up the list of promises provided as arguments to the
function.

I fully understand that the answer to my question might be "don't do
that" ;).


That would be my advice: Don't do that. The API does not provide an
interface for working with promises; in fact the existence of promises
is not guaranteed in the future. Some packages have unfortunately made
use of some internal functions related to promises. For the ones on
CRAN we will work with the maintainers to find alternate
approaches. This may mean adding some functions to the API for dealing
with some lazy-evaluation-related features at a higher level.

Best,

luke


But I will humbly suggest that it would be really nice to be
able to do NSE that can capture expressions with heterogeneous
environments and pass these to functions in a way that is compatible
with existing R functions that do NSE. The basic tools to do it are
there in R 4.3, I think...

Thanks for the help!

---Matt

--
Matthew Kay
Associate Professor
Computer Science & Communication Studies
Northwestern University
matthew@u.northwestern.edu
http://www.mjskay.com/

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu/

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Re: Is ALTREP "non-API"?

[luke-tierney--- via R-devel]

On Wed, 24 Apr 2024, Hadley Wickham wrote:






That is not true at all - the presence of header does not constitute

declaration of something as the R API. There are cases where internal
functions are in the headers for historical or other reasons since the
headers are used both for the internal implementation and packages.

That's

why this is in R-exts under "The R API: entry points for C code":


If I understand your point correctly, does this mean that

Rf_allocVector() is not part of the "official" R API? It does not

appear to

be documented in the "The R API: entry points for C code" section.




It does, obviously:
https://cran.r-project.org/doc/manuals/R-exts.html#Allocating-storage-1



I'm just trying to understand the precise definition of the official API
here. So it's any function mentioned in R-exts, regardless of which

section

it appears in?

Does this sentence imply that all functions starting with alloc* are part
of the official API?



Again, I can only quote the R-exts (few lines below the previous "The R
API" quote):


We can classify the entry points as
API
Entry points which are documented in this manual and declared in an
installed header file. These can be used in distributed packages and will
only be changed after deprecation.


It says "in this manual" - I don't see anywhere restriction on a
particular section of the manual, so I really don't see why you would think
that allocation is not part on the API.



Because you mentioned that section explicitly earlier in the thread. This
obviously seems clear to you, but it's not at all clear to me and I suspect
many of the wider community. It's frustrating because we are trying
our best to do what y'all want us to do, but it feels like we keep getting
the rug pulled out from under us with very little notice, and then have to
spend a large amount of time figuring out workarounds.


Please try to keep this discussion non-adversarial.


That is at least
feasible for my team since we have multiple talented folks who are paid
full-time to work on R, but it's a huge struggle for most people who are
generally maintaining packages in their spare time.


As you well know, almost all R-core members are also trying to
maintain and improve R in their spare time. Good for folks to keep in
mind before demanding R-core do X, Y, or Z for you.


For the purposes of this discussion could you please "documented in the
manual" means? For example, this line mentions allocXxx functions: "There
are quite a few allocXxx functions defined in Rinternals.h—you may want to
explore them.". Does that imply that they are documented and free to use?


Where we are now in terms of what package authors can use to write R
extensions has evolved organically over many years. The current state
is certainly not ideal:

There are entry points in installed headers that might be
available;

but to find out if they are in fact available requires reading
prose text in the header files and in WRE.

Trying to fine-tune wording in WRE, or add a lot of additional entries
is not really a good or realistic way forward: WRE is both
documentation and tutorial and more legalistic language/more complete
coverage would make it less readable and still not guarantee
completeness or clarity.

We would be better off (in my view, not necessarily shared by others
in R-core) if we could get to a point where:

all entry points listed in installed header files can be used in
packages, at least with some caveats;

the caveats are expressed in a standard way that is searchable,
e.g. with a standardized comment syntax at the header file or
individual declaration level.

In principle this is achievable, but getting there from where we are
now is a lot of work. There are some 500 entry points in the R shared
library that are in the installed headers but not mentioned in WRE.
These would need to be reviewed and adjusted. My guess is about a
third are fine and intended to be API-stable, another third are not
used in packages and don't need to be in public headers. The remainder
are things that may be used in current packages but really should not
be, for example because they expose internal data in ways that can
cause segfaults or they make it difficult to implement performance
improvements in the base engine. Sorting through these and working
with package authors to find alternate, safer options takes a lot of
time (see 'spare time' above) and energy (some package authors are
easier to work with than others). Several of us have taken cracks at
moving this forward from time to time, but it rarely gets to the top
of anyone's priority list.


And in general, I'd urge R Core to make an explicit list of functions that
you consider to be part of the exported API, and then grandfather in
packages that used those functions prior to learning that we weren't
supposed to.


Making a list and hoping that it will remain up to date is not
realistic.  The only way that would wor

Re: [Rd] [External] Re: Is ALTREP "non-API"?

[luke-tierney--- via R-devel]

On Wed, 24 Apr 2024, Hadley Wickham wrote:


A few more thoughts based on a simple question: how do you determine the
length of a vector?

Rf_length() is used in example code in R-exts, but I don't think it's
formally documented anywhere (although it's possible I missed it). Is using
in an example sufficient to consider a function to be part of the public
API? If so, SET_TYPEOF() is used in a number of examples, and hence used by
CRAN packages, but is no longer considered part of the public API.

Rf_xlength() doesn't appear to be mentioned anywhere in R-exts. Does this
imply that long vectors are not part of the exported API? Or is there some
other way we should be determining the length of such vectors?

Are the macro variants LENGTH and XLENGTH part of the exported API? Are we
supposed to use them or avoid them?

Relatedly, I presume that LOGICAL() is the way we're supposed to extract
logical values from a vector, but it isn't documented in R-exts, suggesting
that it's not part of the public API?


My pragmatic approach to deciding if an entry point is usable in a
package is to

grep for it in the installed headers

grep for it in WRE

if those are good, check the text in both places to make sure it
doesn't tell me not to use is

The first two can be automated; the text reading can't for now.

One place this runs into trouble is when the prose in WRE doesn't
explicitly mention the entry point, but says something like 'this one
and similar ones are OK'. A couple of years ago I worked on improving
some of those by explicitly adding some of those implicit ones, which
did sometimes make the text more cumbersome. I'm pretty sure I added
LOGICAL() and RAW() at that point (but may be mis-remebering); they
are there now. In some other cases I left the text alone but added
index entries. That makes them findable with a text search. I think I
got most that can be handled that way, but there may be some others
left. Far from ideal, but at least a step forward.



---

It's also worth pointing out where R-exts does well, with the documentation
of utility functions (
https://cran.r-project.org/doc/manuals/R-exts.html#Utility-functions). I
think this is what most people would consider documentation to imply, i.e.
a list of input arguments/types, the output type, and basic notes on their
operation.
---

Finally, it's worth noting that there's some lingering ill feelings over
how the connections API was treated. It was documented in R-exts only to be
later removed, including expunging mentions of it in the news. That's
obviously water under the bridge, but I do believe that there is
the potential for the R core team to build goodwill with the community if
they are willing to engage a bit more with the users of their APIs.


As you well know R-core is not a monolith. There are several R-core
members who also are not happy about how that played out and where
that stands now. But there was and is no viable option other than to
agree to disagree. There is really no upside to re-litigating this
now.

Best,

luke



Hadley

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu/

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] View() segfaulting ...

[luke-tierney--- via R-devel]

I saw it also on some of my Ubuntu builds, but the issue went away
after a make clean/make, so maybe give that a try.

Best,

luke

On Wed, 24 Apr 2024, Ben Bolker wrote:

 I'm using bleeding-edge R-devel, so maybe my build is weird. Can anyone 
else reproduce this?


 View() seems to crash on just about anything.

View(1:3)
*** stack smashing detected ***: terminated
Aborted (core dumped)

 If I debug(View) I get to the last line of code with nothing obviously 
looking pathological:


Browse[1]>
debug: invisible(.External2(C_dataviewer, x, title))
Browse[1]> x
$x
[1] "1" "2" "3"

Browse[1]> title
[1] "Data: 1:3"
Browse[1]>
*** stack smashing detected ***: terminated
Aborted (core dumped)




R Under development (unstable) (2024-04-24 r86483)
Platform: x86_64-pc-linux-gnu
Running under: Pop!_OS 22.04 LTS

Matrix products: default
BLAS/LAPACK: 
/usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so; LAPACK 
version 3.10.0


locale:
[1] LC_CTYPE=en_CA.UTF-8   LC_NUMERIC=C
[3] LC_TIME=en_CA.UTF-8LC_COLLATE=en_CA.UTF-8
[5] LC_MONETARY=en_CA.UTF-8LC_MESSAGES=en_CA.UTF-8
[7] LC_PAPER=en_CA.UTF-8   LC_NAME=C
[9] LC_ADDRESS=C   LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C

time zone: America/Toronto
tzcode source: system (glibc)

attached base packages:
[1] stats graphics  grDevices utils datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_4.5.0

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu/

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] R hang/bug with circular references and promises

[luke-tierney--- via R-devel]

On Sat, 11 May 2024, Travers Ching wrote:


The following code snippet causes R to hang. This example might be a
bit contrived as I was experimenting and trying to understand
promises, but uses only base R.

It looks like it is looking for "not_a_variable" recursively but since
it doesn't exist it goes on indefinitely.

x0 <- new.env()
x1 <- new.env(parent = x0)
parent.env(x0) <- x1
delayedAssign("v", not_a_variable, eval.env=x1)
delayedAssign("w", v, assign.env=x1, eval.env=x0)
x1$w


This has nothing to do with promises. You created a cycle in the
environment chain. A simpler variant:

e <- new.env()
parent.env(e) <- e
get("x", e)

This will hang and is not interruptable -- loops searching up
environment chains are too speed-critical to check for interrupts.  It
is, however, pretty easy to check whether the parent change would
create a cycle and throw an error if it would. Need to think a bit
about exactly where the check should go.

Best,

luke



__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu/

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] R hang/bug with circular references and promises

[luke-tierney--- via R-devel]

On Sat, 11 May 2024, Peter Langfelder wrote:


On Sat, May 11, 2024 at 9:34 AM luke-tierney--- via R-devel
 wrote:


On Sat, 11 May 2024, Travers Ching wrote:


The following code snippet causes R to hang. This example might be a
bit contrived as I was experimenting and trying to understand
promises, but uses only base R.


This has nothing to do with promises. You created a cycle in the
environment chain. A simpler variant:

e <- new.env()
parent.env(e) <- e
get("x", e)

This will hang and is not interruptable -- loops searching up
environment chains are too speed-critical to check for interrupts.  It
is, however, pretty easy to check whether the parent change would
create a cycle and throw an error if it would. Need to think a bit
about exactly where the check should go.


FWIW, the help for parent.env already explicitly warns against using
parent.env <-:

The replacement function ‘parent.env<-’ is extremely dangerous as
it can be used to destructively change environments in ways that
violate assumptions made by the internal C code.  It may be
removed in the near future.


Looks like I added that warning 22 years ago, so that should be enough
notice :-). I'll look into removing it now.

Best,

luke



Peter



--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] R hang/bug with circular references and promises

[luke-tierney--- via R-devel]

On Mon, 13 May 2024, Ivan Krylov wrote:


[You don't often get email from ikry...@disroot.org. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

On Mon, 13 May 2024 09:54:27 -0500 (CDT)
luke-tierney--- via R-devel  wrote:


Looks like I added that warning 22 years ago, so that should be enough
notice :-). I'll look into removing it now.




For now I have just changed the internal code to throw an error
if the change would produce a cycle (r86545). This gives

> e <- new.env()
> parent.env(e) <- e
Error in `parent.env<-`(`*tmp*`, value = ) :
  cycles in parent chains are not allowed


Dear Luke,

I've got a somewhat niche use case: as a way of protecting myself
against rogue *.rds files and vulnerabilities in the C code, I've been
manually unserializing "plain" data objects (without anything
executable), including environments, in R [1].


I would try using two passes: create the environments in the first pass
and in a second pass, either over the file or a new object with place holders, 
fill them in.


I see that SET_ENCLOS() is already commented as "not API and probably
should not be <...> used". Do you think there is a way to recreate an
environment, taking the REFSXP entries into account, without
`parent.env<-`?  Would you recommend to abandon the folly of
unserializing environments manually?


SET_ENCLOS is one of a number of SET... functions that are not in the
API and should not be since they are potentially unsafe to use. (One
that is in the API and needs to be removed is SET_TYPEOF). So we would
like to move them out of installed headers and not export them as
entry points. For this particular case most uses I see are something
like

env = allocSExp(ENVSXP);
SET_FRAME(env, R_NilValue);
SET_ENCLOS(env, parent);
SET_HASHTAB(env, R_NilValue);
SET_ATTRIB(env, R_NilValue);

which could just use

 env = R_NewEnv(parent, FALSE, 0);

Best,

luke



--
Best regards,
Ivan

[1]
https://codeberg.org/aitap/unserializeData/src/commit/33d72705c1ee265349b3e369874ce4b47f9cd358/R/unserialize.R#L289-L313



--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu/

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] clarifying and adjusting the C API for R

[luke-tierney--- via R-devel]

This is an update on some current work on the C API for use in R
extensions.

The internal R implementation makes use of tens of thousands of C
entry points. On Linux and Windows, which support visibility
restrictions, most of these are visible only within the R executble or
shared library. About 1500 are not hidden and are visible to
dynamically loaded shared libraries, such as ones in packages, and to
embedding applications.

There are two main reasons for limiting access to entry points in a
software framework:

- Some entry points are very easy to use in ways that corrupt internal
  data, leading to segfaults or, worse, incorrect computations without
  segfaults.

- Some entry point expose internal structure and other implementation
  details, which makes it hard to make improvements without breaking
  client code that has come to depend on these details.

The API of C entry points that can be used in R extensions, both for
packages and embedding, has evolved organically over many years. The
definition for the current release expressed in the Writing R
Extensions manual (WRE) is roughly:

An entry point can be used if (1) it is declared in a header file
in R.home("include"), and (2) if it is documented for use in WRE.

Ideally, (1) would be necessary and sufficient, but for a variety of
reasons that isn't achievable, at least not in the near term. (2) can
be challenging to determine; in particular, it is not amenable to a
computational answer.

An experimental effort is underway to add annotations to the WRE
Texinfo source to allow (2) to be answered unambiguously. The
annotations so far mostly reflect my reading or WRE and may be revised
as they are reviewed by others. The annotated document can be used for
programmatically identifying what is currently considered part of the C
API. The result so far is an experimental function tools:::funAPI():

> head(tools:::funAPI())
 nameloc apitype
1 Rf_AdobeSymbol2utf8 R_ext/GraphicsDevice.heapi
2alloc3DArrayWRE api
3  allocArrayWRE api
4   allocLangWRE api
5   allocListWRE api
6 allocMatrixWRE api

The 'apitype' field has three possible levels

| api  | stable (ideally) API |
| eapi | experimental API |
| emb  | embedding API|

Entry points in the embedded API would typically only be used in
applications embedding R or providing new front ends, but might be
reasonable to use in packages that support embedding.

The 'loc' field indicates how the entry point is identified as part of
an API: explicit mention in WRE, or declaration in a header file
identified as fully part of an API.

[tools:::funAPI() may not be completely accurate as it relies on
regular expressions for examining header files considered part of the
API rather than proper parsing. But it seems to be pretty close to
what can be achieved with proper parsing.  Proper parsing would add
dependencies on additional tools, which I would like to avoid for
now. One dependency already present is that a C compiler has to be on
the search path and cc -E has to run the C pre-processor.]

Two additional experimental functions are available for analyzing
package compliance: tools:::checkPkgAPI and tools:::checkAllPkgsAPI.
These examine installed packages.

[These may produce some false positives on macOS; they may or may not
work on Windows at this point.]

Using these tools initially showed around 200 non-API entry points
used across packages on CRAN and BIOC. Ideally this number should be
reduced to zero. This will require a combination of additions to the
API and changes in packages.

Some entry points can safely be added to the API. Around 40 have
already been added to WRE with API annotations; another 40 or so can
probably be added after review.

The remainder mostly fall into two groups:

- Entry points that should never be used in packages, such as
  SET_OBJECT or SETLENGTH (or any non-API SETXYZ functions for that
  matter) that can create inconsistent or corrupt internal state.

- Entry points that depend on the existence of internal structure that
  might be subject to change, such as the existence of promise objects
  or internal structure of environments.

Many, if not most, of these seem to be used in idioms that can either
be accomplished with existing higher-level functions already in the
API, or by new higher level functions that can be created and
added. Working through these will take some time and coordination
between R-core and maintainers of affected packages.

Once things have gelled a bit more I hope to turn this into a blog
post that will include some examples of moving non-API entry point
uses into compliance.

Best,

luke

--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  P

Re: [Rd] [External] Re: clarifying and adjusting the C API for R

[luke-tierney--- via R-devel]

On Fri, 7 Jun 2024, Steven Dirkse wrote:


You don't often get email from sdir...@gams.com. Learn why this is important
Thanks for sharing this overview of an interesting and much-needed project.
You mention that R exports about 1500 symbols (on platforms supporting
visibility) but this subject isn't mentioned explicitly again in your note,
so I'm wondering how things tie together.  Un-exported symbols cannot be
part of the API - how would people use them in this case?  In a perfect
world the set of exported symbols could define the API or match it exactly,
but I guess that isn't the case at present.  So I conclude that R exports
extra (i.e. non-API) symbols.  Is part of the goal to remove these extra
exports?


No. We'll hide what we can, but base packages for one need access to
some entry points that should not be in the API, so those have to stay
un-hidden.

Best,

luke



-Steve

On Thu, Jun 6, 2024 at 10:47 AM luke-tierney--- via R-devel
 wrote:
  This is an update on some current work on the C API for use in R
  extensions.

  The internal R implementation makes use of tens of thousands of
  C
  entry points. On Linux and Windows, which support visibility
  restrictions, most of these are visible only within the R
  executble or
  shared library. About 1500 are not hidden and are visible to
  dynamically loaded shared libraries, such as ones in packages,
  and to
  embedding applications.

  There are two main reasons for limiting access to entry points
  in a
  software framework:

  - Some entry points are very easy to use in ways that corrupt
  internal
     data, leading to segfaults or, worse, incorrect computations
  without
     segfaults.

  - Some entry point expose internal structure and other
  implementation
     details, which makes it hard to make improvements without
  breaking
     client code that has come to depend on these details.

  The API of C entry points that can be used in R extensions, both
  for
  packages and embedding, has evolved organically over many years.
  The
  definition for the current release expressed in the Writing R
  Extensions manual (WRE) is roughly:

       An entry point can be used if (1) it is declared in a
  header file
       in R.home("include"), and (2) if it is documented for use
  in WRE.

  Ideally, (1) would be necessary and sufficient, but for a
  variety of
  reasons that isn't achievable, at least not in the near term.
  (2) can
  be challenging to determine; in particular, it is not amenable
  to a
  computational answer.

  An experimental effort is underway to add annotations to the WRE
  Texinfo source to allow (2) to be answered unambiguously. The
  annotations so far mostly reflect my reading or WRE and may be
  revised
  as they are reviewed by others. The annotated document can be
  used for
  programmatically identifying what is currently considered part
  of the C
  API. The result so far is an experimental function
  tools:::funAPI():

       > head(tools:::funAPI())
                       name                    loc apitype
       1 Rf_AdobeSymbol2utf8 R_ext/GraphicsDevice.h    eapi
       2        alloc3DArray                    WRE     api
       3          allocArray                    WRE     api
       4           allocLang                    WRE     api
       5           allocList                    WRE     api
       6         allocMatrix                    WRE     api

  The 'apitype' field has three possible levels

       | api  | stable (ideally) API |
       | eapi | experimental API     |
       | emb  | embedding API        |

  Entry points in the embedded API would typically only be used in
  applications embedding R or providing new front ends, but might
  be
  reasonable to use in packages that support embedding.

  The 'loc' field indicates how the entry point is identified as
  part of
  an API: explicit mention in WRE, or declaration in a header file
  identified as fully part of an API.

  [tools:::funAPI() may not be completely accurate as it relies on
  regular expressions for examining header files considered part
  of the
  API rather than proper parsing. But it seems to be pretty close
  to
  what can be achieved with proper parsing.  Proper parsing would
  add
  dependencies on additional tools, which I would like to avoid
  for
  now. One dependency already present is that a C compiler has to
  be on
  the search path and cc -E has to run the C pre-processor.]

  Two additional experimental functions are available for
  analyzing
  package compliance: tools:::checkPkgAPI and
  tools:::checkAl

Re: [Rd] [External] Re: clarifying and adjusting the C API for R

[luke-tierney--- via R-devel]

On Fri, 7 Jun 2024, Hadley Wickham wrote:


Thanks for working on this Luke! We appreciate your efforts to make it
easier to tell what's in the exported API and we're very happy to work with
you on any changes needed to tidyverse/r-lib packages.
Hadley


Thanks. Glad to hear -- I may be reminding you when we hit some of the
tougher challenges down the road :-)

Best,

luke



On Thu, Jun 6, 2024 at 9:47 AM luke-tierney--- via R-devel
 wrote:
  This is an update on some current work on the C API for use in R
  extensions.

  The internal R implementation makes use of tens of thousands of
  C
  entry points. On Linux and Windows, which support visibility
  restrictions, most of these are visible only within the R
  executble or
  shared library. About 1500 are not hidden and are visible to
  dynamically loaded shared libraries, such as ones in packages,
  and to
  embedding applications.

  There are two main reasons for limiting access to entry points
  in a
  software framework:

  - Some entry points are very easy to use in ways that corrupt
  internal
     data, leading to segfaults or, worse, incorrect computations
  without
     segfaults.

  - Some entry point expose internal structure and other
  implementation
     details, which makes it hard to make improvements without
  breaking
     client code that has come to depend on these details.

  The API of C entry points that can be used in R extensions, both
  for
  packages and embedding, has evolved organically over many years.
  The
  definition for the current release expressed in the Writing R
  Extensions manual (WRE) is roughly:

       An entry point can be used if (1) it is declared in a
  header file
       in R.home("include"), and (2) if it is documented for use
  in WRE.

  Ideally, (1) would be necessary and sufficient, but for a
  variety of
  reasons that isn't achievable, at least not in the near term.
  (2) can
  be challenging to determine; in particular, it is not amenable
  to a
  computational answer.

  An experimental effort is underway to add annotations to the WRE
  Texinfo source to allow (2) to be answered unambiguously. The
  annotations so far mostly reflect my reading or WRE and may be
  revised
  as they are reviewed by others. The annotated document can be
  used for
  programmatically identifying what is currently considered part
  of the C
  API. The result so far is an experimental function
  tools:::funAPI():

       > head(tools:::funAPI())
                       name                    loc apitype
       1 Rf_AdobeSymbol2utf8 R_ext/GraphicsDevice.h    eapi
       2        alloc3DArray                    WRE     api
       3          allocArray                    WRE     api
       4           allocLang                    WRE     api
       5           allocList                    WRE     api
       6         allocMatrix                    WRE     api

  The 'apitype' field has three possible levels

       | api  | stable (ideally) API |
       | eapi | experimental API     |
       | emb  | embedding API        |

  Entry points in the embedded API would typically only be used in
  applications embedding R or providing new front ends, but might
  be
  reasonable to use in packages that support embedding.

  The 'loc' field indicates how the entry point is identified as
  part of
  an API: explicit mention in WRE, or declaration in a header file
  identified as fully part of an API.

  [tools:::funAPI() may not be completely accurate as it relies on
  regular expressions for examining header files considered part
  of the
  API rather than proper parsing. But it seems to be pretty close
  to
  what can be achieved with proper parsing.  Proper parsing would
  add
  dependencies on additional tools, which I would like to avoid
  for
  now. One dependency already present is that a C compiler has to
  be on
  the search path and cc -E has to run the C pre-processor.]

  Two additional experimental functions are available for
  analyzing
  package compliance: tools:::checkPkgAPI and
  tools:::checkAllPkgsAPI.
  These examine installed packages.

  [These may produce some false positives on macOS; they may or
  may not
  work on Windows at this point.]

  Using these tools initially showed around 200 non-API entry
  points
  used across packages on CRAN and BIOC. Ideally this number
  should be
  reduced to zero. This will require a combination of additions to
  the
  API and changes in packages.

  Some entry points can safely be added to the API. Around 40 have
  already be

Re: [Rd] [External] Re: clarifying and adjusting the C API for R

[luke-tierney--- via R-devel]

On Sat, 8 Jun 2024, Reed A. Cartwright wrote:


[You don't often get email from racartwri...@gmail.com. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

Would it be reasonable to move the non-API stuff that cannot be hidden
into header files inside a "details" directory (or some other specific
naming scheme)?

That's what I use when I need to separate a public API from an internal API.


As do I, as does everyone else. As I wrote originally: " ... for a
variety of reasons that isn't achievable, at least not in the near
term." Can we leave it at that please?

luke



On Fri, Jun 7, 2024 at 7:30 AM luke-tierney--- via R-devel
 wrote:


On Fri, 7 Jun 2024, Steven Dirkse wrote:


You don't often get email from sdir...@gams.com. Learn why this is important
Thanks for sharing this overview of an interesting and much-needed project.
You mention that R exports about 1500 symbols (on platforms supporting
visibility) but this subject isn't mentioned explicitly again in your note,
so I'm wondering how things tie together.  Un-exported symbols cannot be
part of the API - how would people use them in this case?  In a perfect
world the set of exported symbols could define the API or match it exactly,
but I guess that isn't the case at present.  So I conclude that R exports
extra (i.e. non-API) symbols.  Is part of the goal to remove these extra
exports?


No. We'll hide what we can, but base packages for one need access to
some entry points that should not be in the API, so those have to stay
un-hidden.

Best,

luke



-Steve

On Thu, Jun 6, 2024 at 10:47 AM luke-tierney--- via R-devel
 wrote:
  This is an update on some current work on the C API for use in R
  extensions.

  The internal R implementation makes use of tens of thousands of
  C
  entry points. On Linux and Windows, which support visibility
  restrictions, most of these are visible only within the R
  executble or
  shared library. About 1500 are not hidden and are visible to
  dynamically loaded shared libraries, such as ones in packages,
  and to
  embedding applications.

  There are two main reasons for limiting access to entry points
  in a
  software framework:

  - Some entry points are very easy to use in ways that corrupt
  internal
 data, leading to segfaults or, worse, incorrect computations
  without
 segfaults.

  - Some entry point expose internal structure and other
  implementation
 details, which makes it hard to make improvements without
  breaking
 client code that has come to depend on these details.

  The API of C entry points that can be used in R extensions, both
  for
  packages and embedding, has evolved organically over many years.
  The
  definition for the current release expressed in the Writing R
  Extensions manual (WRE) is roughly:

   An entry point can be used if (1) it is declared in a
  header file
   in R.home("include"), and (2) if it is documented for use
  in WRE.

  Ideally, (1) would be necessary and sufficient, but for a
  variety of
  reasons that isn't achievable, at least not in the near term.
  (2) can
  be challenging to determine; in particular, it is not amenable
  to a
  computational answer.

  An experimental effort is underway to add annotations to the WRE
  Texinfo source to allow (2) to be answered unambiguously. The
  annotations so far mostly reflect my reading or WRE and may be
  revised
  as they are reviewed by others. The annotated document can be
  used for
  programmatically identifying what is currently considered part
  of the C
  API. The result so far is an experimental function
  tools:::funAPI():

  > head(tools:::funAPI())
   nameloc apitype
   1 Rf_AdobeSymbol2utf8 R_ext/GraphicsDevice.heapi
   2alloc3DArrayWRE api
   3  allocArrayWRE api
   4   allocLangWRE api
   5   allocListWRE api
   6 allocMatrixWRE api

  The 'apitype' field has three possible levels

   | api  | stable (ideally) API |
   | eapi | experimental API |
   | emb  | embedding API|

  Entry points in the embedded API would typically only be used in
  applications embedding R or providing new front ends, but might
  be
  reasonable to use in packages that support embedding.

  The 'loc' field indicates how the entry point is identified as
  part of
  an API: explicit mention in WRE, or declaration in a header file
  identifie

Re: [Rd] [External] Re: changes in R-devel and zero-extent objects in Rcpp

[luke-tierney--- via R-devel]

On Sat, 8 Jun 2024, Ben Bolker wrote:

 The ASAN errors occur *even if the zero-length object is not actually 
accessed*/is used in a perfectly correct manner, i.e. it's perfectly legal in 
base R to define `m <- numeric(0)` or `m <- matrix(nrow = 0, ncol = 0)`, 
whereas doing the equivalent in Rcpp will (now) lead to an ASAN error.


 i.e., these are *not* previously cryptic out-of-bounds accesses that are 
now being revealed, but instead sensible and previously legal definitions of 
zero-length objects that are now causing problems.


  I'm pretty sure I'm right about this, but it's absolutely possible that 
I'm just confused at this point; I don't have a super-simple example to show 
you at the moment. The closest is this example by Mikael Jagan: 
https://github.com/lme4/lme4/issues/794#issuecomment-2155093049


 which shows that if x is a pointer to a zero-length vector (in plain C++ 
for R, no Rcpp is involved), DATAPTR(x) and REAL(x) evaluate to different 
values.


 Mikael further points out that "Rcpp seems to cast a (void *) returned by 
DATAPTR to (double *) when constructing a Vector from a SEXP, rather 
than using the (double *) returned by REAL." So perhaps R-core doesn't want 
to guarantee that these operations give identical answers, in which case Rcpp 
will have to change the way it does things ...


It looks like REAL and friends should also get this check, but it's
not high priority at this point, at least to me. DATAPTR has been
using this check for a while in a barrier build, so you might want to
test there as well. I expect we will activate more integrity checks
from the barrier build on the API client side as things are tidied up.

However: DATAPTR is not in the API and can't be at least in this form:
It allows access to a writable pointer to STRSXP and VECSXP data and
that is too dangerous for memory manager integrity. I'm not sure
exactly how this will be resolve, but be prepared for changes.

Best,

luke



 cheers
  Ben



On 2024-06-08 6:39 p.m., Kevin Ushey wrote:

IMHO, this should be changed in both Rcpp and downstream packages:

1. Rcpp could check for out-of-bounds accesses in cases like these, and 
emit an R warning / error when such an access is detected;


2. The downstream packages unintentionally making these out-of-bounds 
accesses should be fixed to avoid doing that.


That is, I think this is ultimately a bug in the affected packages, but 
Rcpp could do better in detecting and handling this for client packages 
(avoiding a segfault).


Best,
Kevin


On Sat, Jun 8, 2024, 3:06 PM Ben Bolker > wrote:



     A change to R-devel (SVN r86629 or
https://github.com/r-devel/r-svn/commit/92c1d5de23c93576f55062e26d446feface07250 


has changed the handling of pointers to zero-length objects, leading to
ASAN issues with a number of Rcpp-based packages (the commit message
reads, in part, "Also define STRICT_TYPECHECK when compiling
inlined.c.")

    I'm interested in discussion from the community.

    Details/diagnosis for the issues in the lme4 package are here:
https://github.com/lme4/lme4/issues/794
, 
with a bit more discussion

about how zero-length objects should be handled.

    The short(ish) version is that r86629 enables the
CATCH_ZERO_LENGTH_ACCESS definition. This turns on the CHKZLN macro
>,

which returns a trivial pointer (rather than the data pointer that
would
be returned in the normal control flow) if an object has length 0:

/* Attempts to read or write elements of a zero length vector will
     result in a segfault, rather than read and write random memory.
     Returning NULL would be more natural, but Matrix seems to assume
     that even zero-length vectors have non-NULL data pointers, so
     return (void *) 1 instead. Zero-length CHARSXP objects still have 
a

     trailing zero byte so they are not handled. */

    In the Rcpp context this leads to an inconsistency, where `REAL(x)`
is a 'real' external pointer and `DATAPTR(x)` is 0x1, which in turn
leads to ASAN warnings like

runtime error: reference binding to misaligned address 0x0001
for type 'const double', which requires 8 byte alignment
0x0001: note: pointer points here

     I'm in over my head and hoping for insight into whether this
problem
should be resolved by changing R, Rcpp, or downstream Rcpp packages ...

    cheers
     Ben Bolker

__
R-devel@r-project.org  mailing list
https://stat.ethz

Re: [Rd] clarifying and adjusting the C API for R

[luke-tierney--- via R-devel]

Another quick update:

Over 100 entry points used in packages for which it was safe to do so
have now been marked as part of an API (in some cases after adding
error checking of arguments). These can be used in package C code,
with caveats for ones considered experimental or intended for embedded
use.

The remaining 100 or so non-API entry points used in packages will
require changes in package C code. In some cases the API already
provides safe alternatives to unsafe internal entry points.  In most
other cases it should be possible to develop safer interfaces that
allow packages to accomplish what they need to do in a more robust
way, while giving R maintainers and developers the freedom to make
needed internal changes without disrupting package space.

It will take some time to develop these new interfaces. 'Writing R
extensions' now has a new section 'Moving into C API compliance' that
should help with adapting to these changes.

Best,

luke

On Thu, 6 Jun 2024, luke-tier...@uiowa.edu wrote:


This is an update on some current work on the C API for use in R
extensions.

The internal R implementation makes use of tens of thousands of C
entry points. On Linux and Windows, which support visibility
restrictions, most of these are visible only within the R executble or
shared library. About 1500 are not hidden and are visible to
dynamically loaded shared libraries, such as ones in packages, and to
embedding applications.

There are two main reasons for limiting access to entry points in a
software framework:

- Some entry points are very easy to use in ways that corrupt internal
 data, leading to segfaults or, worse, incorrect computations without
 segfaults.

- Some entry point expose internal structure and other implementation
 details, which makes it hard to make improvements without breaking
 client code that has come to depend on these details.

The API of C entry points that can be used in R extensions, both for
packages and embedding, has evolved organically over many years. The
definition for the current release expressed in the Writing R
Extensions manual (WRE) is roughly:

   An entry point can be used if (1) it is declared in a header file
   in R.home("include"), and (2) if it is documented for use in WRE.

Ideally, (1) would be necessary and sufficient, but for a variety of
reasons that isn't achievable, at least not in the near term. (2) can
be challenging to determine; in particular, it is not amenable to a
computational answer.

An experimental effort is underway to add annotations to the WRE
Texinfo source to allow (2) to be answered unambiguously. The
annotations so far mostly reflect my reading or WRE and may be revised
as they are reviewed by others. The annotated document can be used for
programmatically identifying what is currently considered part of the C
API. The result so far is an experimental function tools:::funAPI():

   > head(tools:::funAPI())
 nameloc apitype
   1 Rf_AdobeSymbol2utf8 R_ext/GraphicsDevice.heapi
   2alloc3DArrayWRE api
   3  allocArrayWRE api
   4   allocLangWRE api
   5   allocListWRE api
   6 allocMatrixWRE api

The 'apitype' field has three possible levels

   | api  | stable (ideally) API |
   | eapi | experimental API |
   | emb  | embedding API|

Entry points in the embedded API would typically only be used in
applications embedding R or providing new front ends, but might be
reasonable to use in packages that support embedding.

The 'loc' field indicates how the entry point is identified as part of
an API: explicit mention in WRE, or declaration in a header file
identified as fully part of an API.

[tools:::funAPI() may not be completely accurate as it relies on
regular expressions for examining header files considered part of the
API rather than proper parsing. But it seems to be pretty close to
what can be achieved with proper parsing.  Proper parsing would add
dependencies on additional tools, which I would like to avoid for
now. One dependency already present is that a C compiler has to be on
the search path and cc -E has to run the C pre-processor.]

Two additional experimental functions are available for analyzing
package compliance: tools:::checkPkgAPI and tools:::checkAllPkgsAPI.
These examine installed packages.

[These may produce some false positives on macOS; they may or may not
work on Windows at this point.]

Using these tools initially showed around 200 non-API entry points
used across packages on CRAN and BIOC. Ideally this number should be
reduced to zero. This will require a combination of additions to the
API and changes in packages.

Some entry points can safely be added to the API. Around 40 have
already been added to WRE with API annotations; another 40 or so can
probably be added after review.

The remainder mostly fal

[Rd] non-API entry point Rf_findVarInFrame3 will be removed

[luke-tierney--- via R-devel]

The non-API entry point Rf_findVarInFrame3 used by some packages will
be removed as it is not needed in one use case and not working as
intended in the other.

The most common use case, Rf_findVarInFrame3(rho, sym, TRUE), is
equivalent to the simpler Rf_findVarInFrame(rho, sym).

The less common use case is to test for existence of a binding with

findVarInFrame(rho, sym, FALSE) != R_UnboundValue

The intent is that this have no side effects, but that is not the
case: if the binding exists and is an active binding, then its
function will be called to produce a value. This usage should be
replaced with R_existsVarInFrame(rho, sym).

R_existsVarInFrame has been marked as part of the experimental API.
It is not yet clear whether Rf_findVarInFrame will become part of an
API.  If it does, then its semantics will likely have to change; if it
does not, an alternate interface will be provided.

Best,

luke


--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Non-API updates

[luke-tierney--- via R-devel]

On Tue, 25 Jun 2024, Josiah Parry wrote:


Hey folks,

I'm sure many of you all woke to the same message I did: "Please correct
before 2024-07-09 to safely retain your package on CRAN" caused by Non-API
changes to CRAN.

This is quite unexpected as Luke Tierney's June 6th email writes (emphasis
mine):

"An *experimental* *effort* is underway to add annotations to the WRE..."

"*Once things have gelled a bit more *I hope to turn this into a blog
post that will include some examples of moving non-API entry point
uses into compliance."

Since then there has not been any indication of stabilization of the
Non-API changes nor has there been a blog post outlining how to migrate. As
things have been coming and going from the Non-API changes for quite some
time now, we (the royal we, here) have been waiting for an official
announcement from CRAN on the stabilizing changes.


I posted an update to this list a few days ago. If you missed it you
can find it in the archive.


*Can we extend this very short notice to handle the Non-API changes before
removal from CRAN? *


Timing decisions are up to CRAN.


In the case of the 3 packages I have to fix within 2 weeks, these are all
using Rust. These changes require upstream changes to the extendr library.
There are other packages that are also affected here. Making these changes
is a delicate act and requires care and focus. All of the extendr
developers work full time and cannot make addressing these changes their
only priority for the next 2 weeks.


Using non-API entry points is a choice that comes with risks. The ones
leading to WARNINGs for your packages (PRSEEN and SYMVALUE)have been
receiving NOTEs in check results for several weeks. Using
tools:::checkPkgAPI you can see that your packages are referencing a
lot of non-API entry points. Some of these may be added to the API,
but most will not. This may be a good time to look into that.

To minimize disruption we have been adding entry points to the API as
long as it is safe to do so, in some cases against our better
judgment. But ones that are unsafe to use will not be
added. Eventually their declarations will be removed from public
header files and they will be hidden when that is possible. Packages
that have chosen to use these non-API entry points will have to adapt
if they want to pass R CMD check. For now, we will try to first have
use of these entry points result in NOTEs, and then WARNINGs. Once
their declarations are removed and they are hidden, packages using
them will fail to install.


Additionally, a blog post with "examples of moving non-API entry point uses
into compliance" would be very helpful in this endeavor.


WRE now contains a section 'Moving into C API compliance'; that seems
a better option for the moment given that things are still very much
in flux. We will try to add to this section as needed. For the
specific entry points generating WARNINGs for your packages the advice
is simple: stop using them.

Best,

luke



[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu/

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] API for converting LANGSXP to LISTSXP?

[luke-tierney--- via R-devel]

We have long been discouraging the use of pairlists. So no, we will
not do anything to facilitate this conversion; if anything the
opposite. SET_TYPEOF is used more than it should be in the sources.
It is something I would like us to fix sometime, but isn't high
priority.

Best,

luke

On Fri, 5 Jul 2024, Kevin Ushey wrote:


Hi,

A common idiom in the R sources is to convert objects between LANGSXP
and LISTSXP by using SET_TYPEOF. However, this is soon going to be
disallowed in packages. From what I can see, there isn't currently a
direct way to convert between these two object types using the
available API. At the R level, one can convert calls to pairlists
with:


as.call(pairlist(as.symbol("rnorm"), 42))

rnorm(42)

However, the reverse is not possible:


as.pairlist(call("rnorm", 42))

Error in as.pairlist(call("rnorm", 42)) :
 'language' object cannot be coerced to type 'pairlist'

One can do such a conversion via conversion to e.g. an intermediate R
list (VECSXP), but that seems wasteful. Would it make sense to permit
this coercion? Or, is there some other relevant API I'm missing?

For completeness, Rf_coerceVector() also emits the same error above
since it uses the same code path.

Thanks,
Kevin

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu/

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Re: Could .Primitive("[") stop forcing R_Visible = TRUE?

[luke-tierney--- via R-devel]

On Fri, 25 Oct 2024, Ivan Krylov via R-devel wrote:


В Fri, 25 Oct 2024 08:39:39 -0400
Duncan Murdoch  пишет:


Surely you or they should be the ones to run the test across all of
CRAN?


That's fair. The question is, is there a fundamental reason I
overlooked to deny such a change? Except for positioning and
whitespace, the line has been in names.c since SVN revision 2. The
one regression test touched by the change has been there since 2010.


A couple of points first:

In principle you would need a _much_ stronger case for the
benefits of a change like this relative to the cost of breaking
backward compatibility of output. Checking CRAN/BIOC for impact
would be good to do, but test coverage of output in package tests
isn't great, and this doesn't get at changes in script and report
output.

Changing just [ would create an inconsistency with the other
extractors [[, $, and @. Changing all would affect more output.

Many, maybe most, primitives have byte code instructions that are
used by compiled code. You need to check and possible change both
the interpreted and the compiled execution paths.

However: it looks like the byte code engine implementation for the
extraction instructions does not set the visible flag for a successful
dispatch. A simple example:

x <- structure(1, class = "foo")
`[.foo` <- function(x, i) invisible(2)

withVisible(x[1])
## $value
## [1] 2
## 
## $visible

## [1] TRUE

withVisible(eval(compiler::compile(quote(x[1]
## $value
## [1] 2

## $visible
## [1] FALSE

So there is a discrepancy between interpreted and compiled code which
is a bug that ideally should be resolved. I suspect changing the
compiled code behavior would be more disruptive than changing the
interpreted code behavior, but that would need some looking into.

Filing a bug report on the discrepancy would be a good next step.

Best,

luke

--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] More precise documentation of on.exit() [when leaving function vs. "context"]

[luke-tierney--- via R-devel]

On Fri, 14 Feb 2025, Michael Chirico wrote:


The current documentation of ?on.exit reads [1]


`on.exit` records [`expr`] as needing to be executed when the current function 
exits...


This is almost always how it is used, however I don't see that as
explaining this other common way to use on.exit():

local({
 old = setwd(...)
 on.exit(setwd(old))
 readLines(...)
})

There's not really a "function" here. It may be that the actual
implementation of 'local()' eventually runs this code
indistinguishably from a function, which I'm not sure -- all I see is
that local() is a wrapper of eval.parent(), so it doesn't _look_ to me
like that's the case.


local(expr) is essentially equivalent to (function() expr)().

['Essentially' because in the interpreted implementation there might
currently be slight differences, though if there are we would want to
eventually get rid of them. For compiled code the two are exactly
equivalent.]

I don't see any benefits of changing this wording in ?on.exit that
would outweigh the costs.

In principle I don't object to noting in ?local that local(expr) is
just syntactic sugar for (function() expr)(), but not until someone
spends some time making sure it is exactly true for the interpreted
case; not something I could justify giving high priority to.


I'm not sure this word has a precise meaning in R but I think of
on.exit() as running upon leaving the "context" in this case. Would
that be an improvement? Is this more an "unintentional" benefit that
local() can be used like this?

I've definitely seen usage like this [2] that I've recommended just be
re-written with local instead:

(\(x) { old = setwd(...); on.exit(setwd(old)); readLines(...) })()

I'll also note that while this approach to using on.exit() is not
uncommon, and quite a useful idiom, it does not appear in the examples
either in ?on.exit or in ?local [3].


I'm not sure I would want to encourage this. Using tryCatch with a
finally expression seems clearer to me:

local({ old <- setwd(...); tryCatch(readLines(...), finally = setwd(old)) })

Best,

luke


(I also don't see mention of these in any other R manual [4][5])

Mike C

[1] 
https://github.com/r-devel/r-svn/blob/9bd62756d3c1289c3c01faec0e2b4de9e8a88d59/src/library/base/man/on.exit.Rd#L10-L13
[2] 
https://github.com/rstudio/sass/blob/9228fcf39deecfe32b7cb90ed40690338a18acba/scripts/build_docs.R#L5-L23
[3] 
https://github.com/r-devel/r-svn/blob/9bd62756d3c1289c3c01faec0e2b4de9e8a88d59/src/library/base/man/eval.Rd#L136-L138
[4] https://github.com/r-devel/r-svn/blob/main/doc/manual/R-intro.texi
[5] 
https://github.com/r-devel/r-svn/blob/9bd62756d3c1289c3c01faec0e2b4de9e8a88d59/doc/manual/R-lang.texi#L3755-L3757

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu/

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Suppressed "graphical parameter" warnings reviving themselves magically

[luke-tierney--- via R-devel]

Warnings are not "reviving themselves magically". Try resizing the
window a few times and hitting return in the REPL. That should give
you a hint on what is going on.

Best,

luke

On Fri, 21 Mar 2025, Henrik Bengtsson wrote:


What's going on here?

$ R --vanilla --quiet

plot.new(); suppressWarnings({ points(0, 0, foo = TRUE) })
NULL

NULL
Warning messages:
1: "foo" is not a graphical parameter
2: "foo" is not a graphical parameter

Note how the warnings are revived in that second "NULL" call.  I can
reproduce this in R 4.4.3 and R-devel (2025-03-19 r88003). This might
be specific to "graphical parameter" warnings, because it won't happen
with, say, suppressWarnings({ log(-1) }).

It also doesn't appear if I call split up the first call into to
different REPL calls;

$ R --vanilla --quiet

plot.new()
suppressWarnings({ points(0, 0, foo = TRUE) })
NULL

NULL

/Henrik

PS. I thought I had sent this many months ago, but I just now found
this message in my draft folder, so now I'm not sure. Sorry, if this
is a duplicate.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu/

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel