Re: [Bioc-devel] BiocParallel load balancing and runtime
Hello Anna,
The speed of parallel computing depends on many factors. To avoid any
potential confounders, Please try to use this code for timing (assuming you
still have all the variables you used in your example)
```
parallel_param <- SnowParam(workers = ncores, type = "SOCK", tasks =
length(my_list),exportglobals = FALSE,exportvariables = FALSE)
bpstart(param)
system.time({
res2 <- bplapply(my_list, FUN, BPPARAM = parallel_param)
})
bpstop(param)
```
Also, I encourage you to submit your question along with a reproducible
example to the GitHub issue here:
https://github.com/Bioconductor/BiocParallel/issues
It can help us manage the discussion and pinpoint the problem. Thanks
Best,
Jiefei
On Tue, Aug 8, 2023 at 8:21 AM Anna Plaxienko wrote:
> My motivation for using distributed memory was that my package is also
> accessible on Windows. Is it better to use shared memory as default but
> check the user's system and then switch to socket only if necessary?
>
> Regarding the real data. I have 68 samples (rows) of methylation EPIC array
> data (850K columns), that I split by chromosomes. So I get 22 matrices,
> each from 80K to 10K columns – that's why I need load balancing. When I use
> *clusterApplyLB*, the running time of my method is 38 minutes. With
> *bplapply* it's 42 minutes. In other examples the difference is the same
> 10-15%. It's of course not dramatic, if you've already waited 38 minutes,
> you can wait an extra 4 :) But I'm just curious as to why and if it's
> something I can fix.
>
> вт, 8 авг. 2023 г. в 15:04, Waldir Leoncio Netto >:
>
> > Dear Anna,
> >
> > According to the documentation of "BiocParallelParam", SnowParam() is a
> > subclass suitable for distributed memory (e.g. cluster) computing. If
> > you're running your code on a simpler machine with shared memory (e.g.
> your
> > PC), you're probably better off using MulticoreParam() instead. Here's a
> > modified example based on yours:
> >
> > # Setup
> > library(parallel)
> > library(BiocParallel)
> > my_list <- list(1:10, 11:20, 21:30, 31:40, 41:50, 51:60, 61:70, 71:80,
> > 81:90)
> > FUN <- function(x) return(x ^ 10)
> > ncores <- min(detectCores() - 1L, 10L)
> >
> > # Parallel
> > cl <- makeCluster(ncores)
> > print(system.time(res <- clusterApplyLB(cl, my_list, FUN)))
> > stopCluster(cl)
> >
> > # BiocParallel
> > parallel_param_1 <- SnowParam(workers = ncores, tasks = length(my_list))
> > print(system.time(res2 <- bplapply(my_list, FUN, BPPARAM =
> > parallel_param_1)))
> > parallel_param_2 <- MulticoreParam(workers = ncores, tasks =
> > length(my_list))
> > print(system.time(res3 <- bplapply(my_list, FUN, BPPARAM =
> > parallel_param_2)))
> >
> > On my machine, the output is as follows (notice the last column, with the
> > total time, shows MulticoreParam() performing better than parallel):
> >
> > brukar system brukt
> > 0.000 0.004 0.088
> > brukar system brukt
> > 0.114 0.001 1.336
> > brukar system brukt
> > 0.074 0.124 0.060
> >
> > How does that work on your actual data?
> >
> > Best,
> > Waldir
> >
> > ti., 08.08.2023 kl. 13.10 +0200, skrev Anna Plaxienko:
> >
> > Hi all!
> >
> > I'm switching from the base R *parallel* package to *BiocParallel* for my
> > Bioconductor submission and I have two questions. First, I wanted advice
> on
> > whether I've implemented load balancing correctly. Second, I've noticed
> > that the running time is about 15% longer with BiocParallel. Any ideas
> why?
> >
> >
> > Parallel code
> >
> > cl <- makeCluster(ncores)
> > res <- clusterApplyLB(cl, my_list, FUN)
> > stopCluster(cl)
> >
> > BiocParallel
> >
> > parallel_param <- SnowParam(workers = ncores, type = "SOCK", tasks =
> > length(my_list))
> > res2 <- bplapply(my_list, FUN, BPPARAM = parallel_param)
> >
> > Thank you!
> >
> > Best regards,
> > Anna Plaksienko
> >
> > [[alternative HTML version deleted]]
> >
> > ___
> > Bioc-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
> >
> >
> >
>
> [[alternative HTML version deleted]]
>
> ___
> Bioc-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>
[[alternative HTML version deleted]]
___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
[Rd] The enclosed environment does not work as expected
Hi,
It seems like the enclosed environment does not work well with the
loop. Here is a simple example
```
FuncGenerator <- function(value){
function() message(value)
}
funcSets <- list()
for(i in 1:2)
funcSets[[i]] <- FuncGenerator(i)
environment(funcSets[[1]])$value
# [1] 2
environment(funcSets[[2]])$value
# [1] 2
```
The output from the last two lines is simply 2. I expect the first
should be 1 and the second is 2. However, if I ask R to execute the
message function before accessing its environment, the result is
correct again.
```
FuncGenerator <- function(value){
function() message(value)
}
funcSets <- list()
for(i in 1:2)
funcSets[[i]] <- FuncGenerator(i)
## Ask to evaluate the function
for(i in 1:2){
funcSets[[i]]()
}
# 1
# 2
environment(funcSets[[1]])$value
# [1] 1
environment(funcSets[[2]])$value
# [1] 2
```
This does not make any sense to me since the function simply prints
out its variable `value`. It should not change the value of the
variable, so I think this might be a bug in the code optimizer. This
issue can only be corrected by the loop(like the second example),
manually evaluating the function without the loop will give you the
same result as the first example.
Here is my session information
> sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19044)
Best,
Jiefei
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] Great overhead for setTimeLimit?
Hi all,
>From the document of 'setTimeLimit', it states "Setting any limit has
a small overhead – well under 1% on the systems measured.", but
something is wrong with my benchmark code, enabling the time limit
makes my benchmark 1x slower than the benchmark without the limit.
Below is an example
```
benchFunc <- function(x, data) {
value <- 0
for(i in 1:5000){
for(j in seq_along(data))
value <- value + data[j]
}
value
}
data <- sample(1:10, 10)
setTimeLimit(Inf, Inf, FALSE)
system.time(lapply(1:5000, benchFunc, data = data))
setTimeLimit(999, 999, FALSE)
system.time(lapply(1:5000, benchFunc, data = data))
```
Here are the test results
> setTimeLimit(Inf, Inf, FALSE)
> system.time(lapply(1:5000, benchFunc, data = data))
user system elapsed
10.809 0.006 10.812
> setTimeLimit(999, 999, FALSE)
> system.time(lapply(1:5000, benchFunc, data = data))
user system elapsed
13.634 6.478 20.106
As a side note, it looks like the GC consumes the most CPU time. The
GC costs 10 secs without the time limit, but 19 secs with the limit.
Any thoughts?
Best,
Jiefei
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] How is the environment variable "R_USER" defined?
Thanks for all your help!! It actually has nothing to do with the environment variables. The issue is caused by the backup feature in Onedrive as Bill suggested. I do not know what magic trick Onedrive uses here but it misadvices R to use its directory, not my local Document directory. The answer might be from Windows FAQ question 2.14 like Duncan posted After those two user-controllable settings, R tries to find system-defined home directories. It first tries to use the Windows "personal" directory This is confusing. First, this FAQ is not about how "R_USER" is defined. Instead, it talks about how we find the home directory from "R_USER". Second, it still did not answer what is "the Windows personal directory". I think this is where the issue is. My Document still has the path "C:\Users\wangj\Documents", but there is a backup path in "C:\Users\wangj\OneDrive\Documents". Unfortunately, R believes the latter one is the "personal directory". I hope this can be clarified a little bit to avoid confusion. Best, Jiefei On Mon, Nov 22, 2021 at 12:32 PM Bill Dunlap wrote: > > Is your C:\Users\yourname\Documents linked to OneDrive (either by your choice > or by some administrator setting a group policy)? If so, ou could unlink it > using OneDrive's settings dialog. Or you could set R_USER to avoid using > ...\Documents. > > -Bill > > On Mon, Nov 22, 2021 at 8:47 AM Jiefei Wang wrote: >> >> Hi, I have a new win system and try to install R as usual. Somehow, >> the environment variable "R_LIBS_USER" is incorrectly pointed to a >> Onedrive folder. Since "R_LIBS_USER" depends on "R_USER", the root >> problem then becomes why "R_USER" is the path to the Onedrive. I did >> an exhausting search in the R directory and the only related message I >> can find is from EnvVar.html, which states >> >> R_USER: The user's ‘home’ directory. Set by R. (HOME will be set to >> the same value if not already set.) >> >> I guess that's another way to say "no document is available yet". I >> also took a look at my system environment variables but there are only >> two variables related to Onedrive, they are >> >> OneDrive=C:\Users\wangj\OneDrive >> >> OneDriveConsumer=C:\Users\wangj\OneDrive >> >> so everything looks pretty normal, I know I can correct this issue by >> manually adding Renviron but I just wonder where this default behavior >> comes from... >> >> Best, >> Jiefei >> >> __ >> R-devel@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] How is the environment variable "R_USER" defined?
Hi, I have a new win system and try to install R as usual. Somehow, the environment variable "R_LIBS_USER" is incorrectly pointed to a Onedrive folder. Since "R_LIBS_USER" depends on "R_USER", the root problem then becomes why "R_USER" is the path to the Onedrive. I did an exhausting search in the R directory and the only related message I can find is from EnvVar.html, which states R_USER: The user's ‘home’ directory. Set by R. (HOME will be set to the same value if not already set.) I guess that's another way to say "no document is available yet". I also took a look at my system environment variables but there are only two variables related to Onedrive, they are OneDrive=C:\Users\wangj\OneDrive OneDriveConsumer=C:\Users\wangj\OneDrive so everything looks pretty normal, I know I can correct this issue by manually adding Renviron but I just wonder where this default behavior comes from... Best, Jiefei __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Bioc-devel] Problems with BiocParallel on Windows
There is no need to do it in MulticoreParam as it will simply fork your master process and therefore inherit your global setting. I think the warning you get is due to a bug in RStudio, I hope they can fix it but it might not be that fast... Best, Jiefei On Mon, Oct 25, 2021 at 8:16 PM Giulia Pais wrote: > > Sorry, my bad, I actually changed the code but for it work properly it is > mandatory to rebuild the package completely (not only loading it through > devtools). It seems to work for 1 function, I'll change the option export > global in all the code and rebuild it and see if it works. Do you know if it > is necessary also for MulticoreParam? It doesn't seem to be necessary and > when I previously set the option to TRUE I always got warning messages for > the stats package, that’s way we set it to FALSE. > Thank you so much for your help > > On 10/25/21, 13:58, "Jiefei Wang" wrote: > > I do not see it from the build report here: > > http://bioconductor.org/checkResults/devel/bioc-LATEST/ISAnalytics/riesling1-checksrc.html > > Can you please provide a reproducible example? It can be very helpful > to locate the issue > > On Mon, Oct 25, 2021 at 7:53 PM Giulia Pais wrote: > > > > I mean that trying to modify the code of one of the functions that > raises errors and setting the option "exportglobals = TRUE" on SnowParam, > somehow it still fails and says it doesn't find the function > "mandatory_is_vars()" which is a function exported by the package > > > > On 10/25/21, 13:49, "Jiefei Wang" wrote: > > > > What do you mean by "exported functions are invisible to SnowParam > > workers"? Please provide more details. If you call 'bplapply' with a > > function from your package, it should be visible to the worker. ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Bioc-devel] Problems with BiocParallel on Windows
I do not see it from the build report here: http://bioconductor.org/checkResults/devel/bioc-LATEST/ISAnalytics/riesling1-checksrc.html Can you please provide a reproducible example? It can be very helpful to locate the issue On Mon, Oct 25, 2021 at 7:53 PM Giulia Pais wrote: > > I mean that trying to modify the code of one of the functions that raises > errors and setting the option "exportglobals = TRUE" on SnowParam, somehow it > still fails and says it doesn't find the function "mandatory_is_vars()" which > is a function exported by the package > > On 10/25/21, 13:49, "Jiefei Wang" wrote: > > What do you mean by "exported functions are invisible to SnowParam > workers"? Please provide more details. If you call 'bplapply' with a > function from your package, it should be visible to the worker. ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Bioc-devel] Problems with BiocParallel on Windows
What do you mean by "exported functions are invisible to SnowParam workers"? Please provide more details. If you call 'bplapply' with a function from your package, it should be visible to the worker. ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Bioc-devel] Problems with BiocParallel on Windows
Hi Giulia,
One of your errors is raised from L1288 in the file "analysis-functions.R"
if (getOption("ISAnalytics.verbose") == TRUE) { ... }
The error message is "argument is of length zero". Usually, this means
something is wrong with the global option. Then I see at L4812 in
"internal-functions.R", you have
p <- BiocParallel::SnowParam(
tasks = length(common_names),
progressbar = getOption("ISAnalytics.verbose"),
exportglobals = FALSE
)
Please keep in mind that if you use SnowParam, the worker is a fresh R
process and independent with your master. Since you have
`exportglobals = FALSE`, the worker has no idea about the option
"ISAnalytics.verbose" in your master process and hence the error.
To fix that, you can set `exportglobals = TRUE` to let the worker
know the option. If you do not want to export the globals, you can
also add the global option "ISAnalytics.verbose" manually at the
beginning of the function in bplapply.
Best,
Jiefei
On Mon, Oct 25, 2021 at 5:42 PM Giulia Pais wrote:
>
> Hello,
> I’m the developer of the package ISAnalytics. Since the last build we’re
> having issues with BiocParallel on the Windows platform only. In particular,
> trying to debug what the issue might be on a local Windows machine, it seems
> BiocParallel isn’t able to find internal package functions and fails at
> several jobs that did just fine a few versions ago.
> We’re using MulticoreParam for Linux and Mac and SnowParam for Windows as
> suggested, I’m not really sure how to fix this issue and I’ll appreciate any
> help,
> Thank you
> Giulia Pais
>
> [[alternative HTML version deleted]]
>
> ___
> Bioc-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
[Bioc-devel] Looking for suggestions for BiocParallel
Dear all, We are preparing for the new devel branch for Bioc 3.15, so we look for suggestions for BiocParallel from our community. If you think there is anything we can improve, please feel free to leave your comments here: https://github.com/Bioconductor/BiocParallel/issues/160 Cheers, Jiefei ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Bioc-devel] Spurious "dims [product 864] do not match the length of object [432]" on macOS
Hi, If you are a newbie to the terminal, all you need to do is to extract the R code from your vignette using `R CMD Stangle xcms-lcms-ms.Rmd`(assume you are under the vignettes directory). Then use `R -d valgrind -f xcms-lcms-ms.R` to check for the memory issue. In the ideal case, it can directly tell you where the problem is. If you have bad luck then you should read through some tutorials and figure out how to dig deeper into this problem. Here is a nice introduction for using Valgrind in R: https://kevinushey.github.io/blog/2015/04/05/debugging-with-valgrind/ Best, Jiefei On Sat, Sep 25, 2021 at 9:02 PM Neumann, Steffen wrote: > > Hi, > > On Fri, 2021-09-24 at 07:28 -0400, Martin Morgan wrote: > > Errors that are intermittent and platform-specific are often due to > > memory corruption, and can be investigated on Linux platforms with > > valgrind or similar tools. Have you taken this approach? > > I am afraid that is beyond my skillset :-( > > Yours, > Steffen > > > -- > > Upcoming events: > * Save-the-date: 22.-24.11.2021 3rd metaRbolomics Hackathon, Wittenberg > (Germany) > > https://www.denbi.de/training/1177-3rd-de-nbi-elixir-de-metarbolomics-hackathon > > --- > IPB HalleBioinformatics and Scientific Data > Dr. Steffen Neumann http://www.IPB-Halle.DE > Weinberg 3 Tel. +49 (0) 345 5582 - 1470 > 06120 Halle +49 (0) 345 5582 - 0 > sneumann(at)IPB-Halle.DE Fax. +49 (0) 345 5582 - 1409 > ___ > Bioc-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/bioc-devel ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Rd] User interrupts parallel excution. Why it works or why not?
Thanks for your explanation. This makes a lot of sense! SIGINT
handling is a blind spot to me, this introduction looks perfect!
Best,
Jiefei
On Tue, Jul 20, 2021 at 4:31 PM Tomas Kalibera wrote:
>
> Hi Jiefei,
>
> when you run the cluster "automatically" in your terminal and pres
> Ctrl-C in Unix, both the master and the worker processes get the SIGINT
> signal, because they belong to the same foreground process group. So you
> are directly interrupting also the worker process.
>
> When you run the cluster "manually", that is the master in one terminal
> window and the worker in another, they are in different process groups
> and if you pres Ctrl-C in the terminal running the master, only the
> master will receive SIGINT signal, not the worker.
>
> If you wanted to read the sources more, look for SIGINT handling in R,
> the onintrEx() function, etc. A good source on signal handling is e.g.
> http://www.linusakesson.net/programming/tty/
>
> Best
> Tomas
>
> On 7/20/21 9:55 AM, Jiefei Wang wrote:
> > Hi all,
> >
> > I just notice this interesting problem a few days before, but I cannot
> > find an answer for it. Say if you have a long-running job in a cluster
> > made by the parallel package and you decide to stop the execution by
> > pressing ctr + c in the terminal or the stop button in Rstudio for
> > some reason. After the interrupt, is the cluster still valid or not?
> > Below is a simple example code
> >
> > library(parallel)
> > cl <- makeCluster(1)
> > ## run and interrupt it
> > parLapply(cl, 1, function(i){Sys.sleep(10);Sys.getpid()})
> > ## run another apply function to check the cluster status
> > parLapply(cl, 1, function(i)i)
> >
> > From my test result, the answer is yes. The worker is interrupted
> > immediately and the cluster is ready for the next command, but when I
> > create the worker manually, things seem different.
> >
> > library(parallel)
> > cl <- makeCluster(1, manual = TRUE)
> > ## run and interrupt it
> > parLapply(cl, 1, function(i){Sys.sleep(10);Sys.getpid()})
> > ## run another apply function to check the cluster status
> > parLapply(cl, 1, function(i)i)
> >
> > It seems like the worker does not know the manager has been
> > interrupted and still runs the current task. I have to wait for 10
> > seconds before I can get the result from the last line of the code and
> > the return value is the PID from the first apply function.
> >
> > Both cases are reasonable, but it is surprising to see them at the
> > same time. I start to wonder how the user interrupt is handled, so I
> > looked at the code in the parallel package. However, it looks like
> > there is no related code, there is no try-catch statement in the
> > manager's code to handle the user interrupt, but the worker just
> > magically knows it should stop the current execution.
> >
> > I can see this behavior in both Win and Ubuntu. It is kind of beyond
> > my knowledge, so I wonder if anyone can help me with it. Does the
> > cluster support the user interrupt? Why the above code works or not
> > works? Many thanks!
> >
> > Best,
> > Jiefei
> >
> > __
> > R-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] User interrupts parallel excution. Why it works or why not?
Hi all,
I just notice this interesting problem a few days before, but I cannot
find an answer for it. Say if you have a long-running job in a cluster
made by the parallel package and you decide to stop the execution by
pressing ctr + c in the terminal or the stop button in Rstudio for
some reason. After the interrupt, is the cluster still valid or not?
Below is a simple example code
library(parallel)
cl <- makeCluster(1)
## run and interrupt it
parLapply(cl, 1, function(i){Sys.sleep(10);Sys.getpid()})
## run another apply function to check the cluster status
parLapply(cl, 1, function(i)i)
>From my test result, the answer is yes. The worker is interrupted
immediately and the cluster is ready for the next command, but when I
create the worker manually, things seem different.
library(parallel)
cl <- makeCluster(1, manual = TRUE)
## run and interrupt it
parLapply(cl, 1, function(i){Sys.sleep(10);Sys.getpid()})
## run another apply function to check the cluster status
parLapply(cl, 1, function(i)i)
It seems like the worker does not know the manager has been
interrupted and still runs the current task. I have to wait for 10
seconds before I can get the result from the last line of the code and
the return value is the PID from the first apply function.
Both cases are reasonable, but it is surprising to see them at the
same time. I start to wonder how the user interrupt is handled, so I
looked at the code in the parallel package. However, it looks like
there is no related code, there is no try-catch statement in the
manager's code to handle the user interrupt, but the worker just
magically knows it should stop the current execution.
I can see this behavior in both Win and Ubuntu. It is kind of beyond
my knowledge, so I wonder if anyone can help me with it. Does the
cluster support the user interrupt? Why the above code works or not
works? Many thanks!
Best,
Jiefei
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Possible ALTREP bug
Hi Gabor,
Calling back to an R function from the ALTREP function is not safe.
There has been a heated discussion on why you should not do it and
that the main reason that we do not have any R level ALTREP API. If
you are interested in it you can find it from this issue:
https://github.com/Bioconductor/Contributions/issues/1222
Best,
Jiefei
On Fri, May 28, 2021 at 4:54 PM Gábor Csárdi wrote:
>
> I have found some weird SEXP corruption behavior with ALTREP, which
> could be a bug. (Or I could be doing something wrong.)
>
> I have an integer ALTREP vector that calls back to R from the Elt
> method. When this vector is indexed in a lapply(), its first element
> gets corrupted. Sometimes it's just a type change to logical, but
> sometimes the corruption causes a crash.
>
> I saw this on macOS from R 3.5.3 to 4.2.0. I created a small package
> that demonstrates this: https://github.com/gaborcsardi/redfish
>
> The R callback in this package calls `loadNamespace("Matrix")`, but
> the same crash happens for other packages as well, and sometimes it
> also happens if I don't load any packages at all. (But that example
> was much more complicated, so I went with the package loading.)
>
> It is somewhat random, and sometimes turning off the JIT avoids the
> crash, but not always.
>
> Hopefully I am just doing something wrong in the ALTREP code (see
> https://github.com/gaborcsardi/redfish/blob/main/src/test.c), and it
> is not actually a bug.
>
> Thanks,
> Gabor
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Surprising behavior when using the reference class with the dollar symbol
Thank you for the clarification! Wow, I do not know the same rule
applies to the list object. Maybe that's because the list object will
give NULL not an error. I might have incorrectly used it in my
package.
Best,
Jiefei
On Sun, Mar 28, 2021 at 3:26 AM Bill Dunlap wrote:
>
> > > It looks like when calling the dollar symbol using the function format, it
> > > treats the input argument as a character literal and does not evaluate it
> > > inside the function. I know we have the function `field` to get the slot
> > > variable, but I wonder if this is designed on purpose as the above example
> > > works for both list and S4 system.
>
> It works almost the same way with lists, the difference being that an
> unknown name with a list gives NULL and not an error.
>
> > L <- list(A=101)
> > `$`(L, A)
> [1] 101
> > `$`(L, "A")
> [1] 101
> > i <- "A"
> > `$`(L, i)
> NULL
>
> The 2nd argument is treated as a symbol and is never evaluated. If it
> is given as a string literal then it is converted to a symbol - that
> is a holdover from the old days before backticks could be used to make
> symbol literals for symbols that included odd things like spaces and
> dollar signs.
>
> -Bill
>
> On Sat, Mar 27, 2021 at 7:49 AM Jiefei Wang wrote:
> >
> > Thanks, Duncan. Below is the repost of my question in plain text mode.
> >
> >
> > I'm trying to get the field value of a reference object by the field
> > name, but the dollar symbol behaves quite unusual. See example below
> >
> > .foo <- setRefClass(
> > "foo",
> > fields = list(
> > a = "integer"
> > )
> > )
> > x <- .foo$new(a=1L)
> > ## This is OK
> > x$a
> > ## This is OK
> > `$`(x, "a")
> > ## But this is not OK
> > i <- "a"
> > `$`(x, i)
> >
> > For the last line of code I get this error
> >
> > Error in envRefInferField(x, what, getClass(class(x)), selfEnv) :
> > ‘i’ is not a valid field or method name for reference class “foo”
> >
> > It looks like when calling the dollar symbol using the function
> > format, it treats the input argument as a character literal and does
> > not evaluate it inside the function. I know we have the function
> > `field` to get the slot variable, but I wonder if this is designed on
> > purpose as the above example works for both list and S4 system.
> >
> > Best,
> > Jiefei
> >
> >
> > On Sat, Mar 27, 2021 at 10:20 PM Duncan Murdoch
> > wrote:
> > >
> > > On 27/03/2021 10:16 a.m., Jiefei Wang wrote:
> > > > Hi all,
> > > >
> > > > I'm trying to get the field value of a reference object by the field
> > > > name,
> > > > but the dollar symbol behaves quite unusual. See example below
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *.foo <- setRefClass("foo",fields = list(a = "integer"
> > > > ))x <- .foo$new(a=1L)## This is OKx$a## This is OK`$`(x, "a")## But
> > > > this is
> > > > not OKi <- "a"`$`(x, i)*
> > >
> > > This is really hard to read. Please post it again, but don't use HTML.
> > >
> > > Duncan Murdoch
> > >
> > > >
> > > > For the last line of code I get this error
> > > >
> > > >
> > > > *Error in envRefInferField(x, what, getClass(class(x)), selfEnv) :
> > > > ‘i’ is
> > > > not a valid field or method name for reference class “foo”*
> > > >
> > > > It looks like when calling the dollar symbol using the function format,
> > > > it
> > > > treats the input argument as a character literal and does not evaluate
> > > > it
> > > > inside the function. I know we have the function `field` to get the slot
> > > > variable, but I wonder if this is designed on purpose as the above
> > > > example
> > > > works for both list and S4 system.
> > > >
> > > > Best,
> > > > Jiefei
> > > >
> > > > [[alternative HTML version deleted]]
> > > >
> > > > __
> > > > R-devel@r-project.org mailing list
> > > > https://stat.ethz.ch/mailman/listinfo/r-devel
> > > >
> > >
> >
> > __
> > R-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Surprising behavior when using the reference class with the dollar symbol
Thanks, Duncan. Below is the repost of my question in plain text mode.
I'm trying to get the field value of a reference object by the field
name, but the dollar symbol behaves quite unusual. See example below
.foo <- setRefClass(
"foo",
fields = list(
a = "integer"
)
)
x <- .foo$new(a=1L)
## This is OK
x$a
## This is OK
`$`(x, "a")
## But this is not OK
i <- "a"
`$`(x, i)
For the last line of code I get this error
Error in envRefInferField(x, what, getClass(class(x)), selfEnv) :
‘i’ is not a valid field or method name for reference class “foo”
It looks like when calling the dollar symbol using the function
format, it treats the input argument as a character literal and does
not evaluate it inside the function. I know we have the function
`field` to get the slot variable, but I wonder if this is designed on
purpose as the above example works for both list and S4 system.
Best,
Jiefei
On Sat, Mar 27, 2021 at 10:20 PM Duncan Murdoch
wrote:
>
> On 27/03/2021 10:16 a.m., Jiefei Wang wrote:
> > Hi all,
> >
> > I'm trying to get the field value of a reference object by the field name,
> > but the dollar symbol behaves quite unusual. See example below
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *.foo <- setRefClass("foo",fields = list(a = "integer"
> > ))x <- .foo$new(a=1L)## This is OKx$a## This is OK`$`(x, "a")## But this is
> > not OKi <- "a"`$`(x, i)*
>
> This is really hard to read. Please post it again, but don't use HTML.
>
> Duncan Murdoch
>
> >
> > For the last line of code I get this error
> >
> >
> > *Error in envRefInferField(x, what, getClass(class(x)), selfEnv) : ‘i’ is
> > not a valid field or method name for reference class “foo”*
> >
> > It looks like when calling the dollar symbol using the function format, it
> > treats the input argument as a character literal and does not evaluate it
> > inside the function. I know we have the function `field` to get the slot
> > variable, but I wonder if this is designed on purpose as the above example
> > works for both list and S4 system.
> >
> > Best,
> > Jiefei
> >
> > [[alternative HTML version deleted]]
> >
> > __
> > R-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
> >
>
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] Surprising behavior when using the reference class with the dollar symbol
Hi all,
I'm trying to get the field value of a reference object by the field name,
but the dollar symbol behaves quite unusual. See example below
*.foo <- setRefClass("foo",fields = list(a = "integer"
))x <- .foo$new(a=1L)## This is OKx$a## This is OK`$`(x, "a")## But this is
not OKi <- "a"`$`(x, i)*
For the last line of code I get this error
*Error in envRefInferField(x, what, getClass(class(x)), selfEnv) : ‘i’ is
not a valid field or method name for reference class “foo”*
It looks like when calling the dollar symbol using the function format, it
treats the input argument as a character literal and does not evaluate it
inside the function. I know we have the function `field` to get the slot
variable, but I wonder if this is designed on purpose as the above example
works for both list and S4 system.
Best,
Jiefei
[[alternative HTML version deleted]]
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] [External] Two ALTREP questions
Thank Dirk and Luke for the answers!
(That's C code. The confusion here is partly our fault. When Romain and I
> extended the inline package with 'cxxfunction' to support the then-young
> but
> active Rcpp package, we picked C++. Strictly speaking that isn't required;
> you are only in C++ here because ... "it made sense to us 10 years ago" and
> it could generalized to C++ or C. All ALTREP is, of course, purely C as it
> is an R API.)
Sometimes I forget to distinguish C/C++ code. Yes, this should be C code
and it is just C++ compatible.
Anyway, `inline` is a great package, `cxxfunction` makes life much easier
for reporting the low-level problem to the devel team.
- Try with 1:6 replaced by as.character(1:6), and look at the REF
>values in both cases.
> - In particular, look at what this gives you:
> x <- as.character(1:6)
> attr(x, "dim") <- c(2, 3)
> - Things can be a little different with larger vectors; try variants
>of your examples for more than 64 elements.
I see why we cannot change the attribute of the compact sequence(It is
shared, or at least marked as shared)
and the use of the wrapper for the large vector. Only the matrix function
needs to be patched.
The generally recommended way is via a bug report at bugs.r-project.org
You can find the suggested process for contributing described in the
> 'Reporting Bugs' link on the R home page https://www.r-project.org/
Bugzilla sounds like a good place to start, I will send an email to acquire
an account.
Best,
Jiefei
On Sun, Nov 22, 2020 at 6:57 AM wrote:
> On Sat, 21 Nov 2020, Jiefei Wang wrote:
>
> > Hello,
> >
> > I have two related ALTREP questions. It seems like there is no way to
> > assign attributes to an ALTREP vector without using C++ code. To be more
> > specifically, I want to make an ALTREP matrix, I have tried the
> following R
> > code but none of them work.
> > ```
> > .Internal(inspect(1:6))
> > .Internal(inspect(matrix(1:6, 2,3)))
> > .Internal(inspect(as.matrix(1:6)))
> > .Internal(inspect(structure(1:6, dim = c(2L,3L
> > .Internal(inspect({x <- 1:6;attr(x, "dim") <- c(2L,3L);x}))
> > .Internal(inspect({x <- 1:6;attributes(x)<- list(dim = c(2L,3L));x}))
> > ```
>
> Some things that my help you:
>
> - Try with 1:6 replaced by as.character(1:6), and look at the REF
>values in both cases.
>
> - In particular, look at what this gives you:
>
> x <- as.character(1:6)
> attr(x, "dim") <- c(2, 3)
>
> - Things can be a little different with larger vectors; try variants
>of your examples for more than 64 elements.
>
> > This also brings
> > my second question, it seems like the ALTREP coercion function does not
> > handle attributes correctly. After the coercion, the ALTREP object will
> > lose its attributes.
> > ```
> > coerceFunc <- inline::cxxfunction( signature(x = "SEXP", attr = "SEXP" )
> , '
> > SET_ATTRIB(x,attr);
> > return(Rf_coerceVector(x, REALSXP));
> > ')
> >> coerceFunc(1:6, pairlist(dim = c(2L, 3L)))
> > [1] 1 2 3 4 5 6
> >> coerceFunc(1:6 + 0L, pairlist(dim = c(2L, 3L)))
> > [,1] [,2] [,3]
> > [1,]135
> > [2,]246
> > ```
> > The problem is that the coercion function is directly dispatched to the
> > user-defined ALTREP coercion function, so the user is responsible to
> attach
> > the attributes after the coercion. If he forgets to do so, then the
> result
> > is a plain vector. Similar to the `Duplicate` and `DuplicateEX` functions
> > where the former one will attach the attributes by default, I feel that
> the
> > `Coerce` function should only return a plain vector and there should be a
> > `CoerceEx` function to do the attribute assignment, so the logic in the
> > no-EX ALTREP functions can be consistent. I do not know how dramastic the
> > change would be, so maybe this is too hard to do.
>
> Since you raised this earlier I have been looking at it and also think
> that this needs to he handled along the lines of
> Duplicate/DuplicateEx. I need to find some time to think that through
> and implement it; hopefully I'll get to it before the end of the year.
>
> > BTW, is there any way to contribute to the R source? I know R has a
> limited
> > resouces, so if possible, I will be happy to fix the matrix issue myself
> > and make some minor contributions to the R community.
>
> You can find the suggested process for contributing described in the
> 'Reporting Bugs' link on the R home page https://www.r-project.org/
>
> Best,
>
> luke
>
> > Best,
> >
[Rd] Two ALTREP questions
Hello,
I have two related ALTREP questions. It seems like there is no way to
assign attributes to an ALTREP vector without using C++ code. To be more
specifically, I want to make an ALTREP matrix, I have tried the following R
code but none of them work.
```
.Internal(inspect(1:6))
.Internal(inspect(matrix(1:6, 2,3)))
.Internal(inspect(as.matrix(1:6)))
.Internal(inspect(structure(1:6, dim = c(2L,3L
.Internal(inspect({x <- 1:6;attr(x, "dim") <- c(2L,3L);x}))
.Internal(inspect({x <- 1:6;attributes(x)<- list(dim = c(2L,3L));x}))
```
The only way to make an ALTREP matrix is to use the C level function
```
attachAttrib <- inline::cxxfunction( signature(x = "SEXP", attr = "SEXP" )
, '
SET_ATTRIB(x,attr);
return(R_NilValue);
')
x <- 1:6
attachAttrib(x, pairlist(dim = c(2L, 3L)))
.Internal(inspect(x))
```
Since the matrix, or adding attributes, is a common need for the object
operation, I wonder if this missing feature is intended? This also brings
my second question, it seems like the ALTREP coercion function does not
handle attributes correctly. After the coercion, the ALTREP object will
lose its attributes.
```
coerceFunc <- inline::cxxfunction( signature(x = "SEXP", attr = "SEXP" ) , '
SET_ATTRIB(x,attr);
return(Rf_coerceVector(x, REALSXP));
')
> coerceFunc(1:6, pairlist(dim = c(2L, 3L)))
[1] 1 2 3 4 5 6
> coerceFunc(1:6 + 0L, pairlist(dim = c(2L, 3L)))
[,1] [,2] [,3]
[1,]135
[2,]246
```
The problem is that the coercion function is directly dispatched to the
user-defined ALTREP coercion function, so the user is responsible to attach
the attributes after the coercion. If he forgets to do so, then the result
is a plain vector. Similar to the `Duplicate` and `DuplicateEX` functions
where the former one will attach the attributes by default, I feel that the
`Coerce` function should only return a plain vector and there should be a
`CoerceEx` function to do the attribute assignment, so the logic in the
no-EX ALTREP functions can be consistent. I do not know how dramastic the
change would be, so maybe this is too hard to do.
BTW, is there any way to contribute to the R source? I know R has a limited
resouces, so if possible, I will be happy to fix the matrix issue myself
and make some minor contributions to the R community.
Best,
Jiefei
[[alternative HTML version deleted]]
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] [External] Something is wrong with the unserialize function
Thanks for the fix! I will wait for the update and try it again.
Best,
Jiefei
On Fri, Oct 30, 2020 at 1:42 AM wrote:
> I found that also; fixed in r79386 in the trunk. Will port to R-patched
> shortly.
>
> Best,
>
> luke
>
> On Thu, 29 Oct 2020, Martin Morgan wrote:
>
> > This
> >
> > Index: src/main/altrep.c
> > ===
> > --- src/main/altrep.c (revision 79385)
> > +++ src/main/altrep.c (working copy)
> > @@ -275,10 +275,11 @@
> > SEXP psym = ALTREP_SERIALIZED_CLASS_PKGSYM(info);
> > SEXP class = LookupClass(csym, psym);
> > if (class == NULL) {
> > - SEXP pname = ScalarString(PRINTNAME(psym));
> > + SEXP pname = PROTECT(ScalarString(PRINTNAME(psym)));
> > R_tryCatchError(find_namespace, pname,
> > handle_namespace_error, NULL);
> > class = LookupClass(csym, psym);
> > + UNPROTECT(1);
> > }
> > return class;
> > }
> >
> > seems to remove the warning; I'm guessing that the other SEXP already
> exist so don't need protecting?
> >
> > Martin Morgan
> >
> >
> > On 10/29/20, 12:47 PM, "R-devel on behalf of luke-tier...@uiowa.edu" <
> r-devel-boun...@r-project.org on behalf of luke-tier...@uiowa.edu> wrote:
> >
> >Thanks for the report. Will look into it when I get a chance unless
> >someone else gets there first.
> >
> >A simpler reprex:
> >
> >## create and serialize a memmory-mapped file object
> >filePath <- "x.dat"
> >con <- file(filePath, "wrb")
> >writeBin(rep(0.0,10),con)
> >close(con)
> >
> >library(simplemmap)
> >x <- mmap(filePath, "double")
> >saveRDS(x, file = "x.Rds")
> >
> >## in a separate R process:
> >gctorture()
> >readRDS("x.Rds")
> >
> >Looks like a missing PROTECT somewhere.
> >
> >Best,
> >
> >luke
> >
> >On Thu, 29 Oct 2020, Jiefei Wang wrote:
> >
> >> Hi all,
> >>
> >> I am not able to export an ALTREP object when `gctorture` is on in
> the
> >> worker. The package simplemmap can be used to reproduce the
> problem. See
> >> the example below
> >> ```
> >> ## Create a temporary file
> >> filePath <- tempfile()
> >> con <- file(filePath, "wrb")
> >> writeBin(rep(0.0,10),con)
> >> close(con)
> >>
> >> library(simplemmap)
> >> library(parallel)
> >> cl <- makeCluster(1)
> >> x <- mmap(filePath, "double")
> >> ## Turn gctorture on
> >> clusterEvalQ(cl, gctorture())
> >> clusterExport(cl, "x")
> >> ## x is an 0-length vector on the worker
> >> clusterEvalQ(cl, x)
> >> stopCluster(cl)
> >> ```
> >>
> >> you can find more info on the problem if you manually build a
> connection
> >> between two R processes and export the ALTREP object. See output
> below
> >> ```
> >>> con <- socketConnection(port = 1234,server = FALSE)
> >>> gctorture()
> >>> x <- unserialize(con)
> >> Warning message:
> >> In unserialize(con) :
> >> cannot unserialize ALTVEC object of class 'mmap_real' from package
> >> 'simplemmap'; returning length zero vector
> >> ```
> >> It seems like simplemmap did not get loaded correctly on the
> worker. If
> >> you run `library( simplemmap)` before unserializing the ALTREP,
> there will
> >> be no problem. But I suppose we should be able to unserialize
> objects
> >> without preloading the library?
> >>
> >> This issue can be reproduced on Ubuntu with R version 4.0.2
> (2020-06-22)
> >> and Windows with R Under development (unstable) (2020-09-03 r79126).
> >>
> >> Here is the link to simplemmap:
> >> https://github.com/ALTREP-examples/Rpkg-simplemmap
> >>
> >> Best,
> >> Jiefei
> >>
> >> [[alternative HTML version deleted]]
> >>
> >> __
> >> R-devel@r-project.org mailing list
> >> https://stat.e
[Rd] Something is wrong with the unserialize function
Hi all, I am not able to export an ALTREP object when `gctorture` is on in the worker. The package simplemmap can be used to reproduce the problem. See the example below ``` ## Create a temporary file filePath <- tempfile() con <- file(filePath, "wrb") writeBin(rep(0.0,10),con) close(con) library(simplemmap) library(parallel) cl <- makeCluster(1) x <- mmap(filePath, "double") ## Turn gctorture on clusterEvalQ(cl, gctorture()) clusterExport(cl, "x") ## x is an 0-length vector on the worker clusterEvalQ(cl, x) stopCluster(cl) ``` you can find more info on the problem if you manually build a connection between two R processes and export the ALTREP object. See output below ``` > con <- socketConnection(port = 1234,server = FALSE) > gctorture() > x <- unserialize(con) Warning message: In unserialize(con) : cannot unserialize ALTVEC object of class 'mmap_real' from package 'simplemmap'; returning length zero vector ``` It seems like simplemmap did not get loaded correctly on the worker. If you run `library( simplemmap)` before unserializing the ALTREP, there will be no problem. But I suppose we should be able to unserialize objects without preloading the library? This issue can be reproduced on Ubuntu with R version 4.0.2 (2020-06-22) and Windows with R Under development (unstable) (2020-09-03 r79126). Here is the link to simplemmap: https://github.com/ALTREP-examples/Rpkg-simplemmap Best, Jiefei [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Is there any way to check the class of an ALTREP?
Hi all, By looking at the implementation of R's wrapper. I found a function ` R_altrep_inherits` that can check whether an object belongs to an ALTREP class. Even though it is a little different from what I am looking for, but it should be satisfactory for my purpose. I will suspect there is no such function in R to get the class of an ALTREP object for the SEXP struct is very compact and this less-important information might not be stored when an ALTREP object is created. Best, Jiefei On Mon, Oct 19, 2020 at 8:37 PM Jiefei Wang wrote: > Thank Denes for the clarification, glad to see my package got one citation > from my own question:) Also, thank Benjamin > for sending many useful documents. > > Actually, the question is related to the SharedObject package that > Benjamin has pointed to. I wanna > avoid sharing an object that has already been shared, so I need to check > whether an object is an ALTREP > that is defined in my package. Since the ALTREP definition involves an ` > R_altrep_class_t` object and a > class name, I will expect to see a method defined in R to retrieve such > information. I hope this can clarify my question. > > Best, > Jiefei > > On Mon, Oct 19, 2020 at 5:38 PM Benjamin Christoffersen < > boenn...@gmail.com> wrote: > >> > You happened to send a link which points to the OP's own package :) I >> > think Jiefei would like to know how one can "officially" determine if an >> > arbitrary ALTERP object belongs to a class that he owns. >> Argh, I am sorry! I did not notice that. My best bet for what I >> thought the question was is section 1.1.2 of R Internals where they >> note that: `unsigned int alt : 1; /* is this an ALTREP object? */` >> >> This is also what the ALTREP macro checks. However, I also did not >> notice the "... if an ALTREP object is from my package" part. I am >> sorry. >> >> Sincerely Yours, >> Benjamin >> >> Den man. 19. okt. 2020 kl. 11.25 skrev Dénes Tóth > >: >> > >> > Benjamin, >> > >> > You happened to send a link which points to the OP's own package :) I >> > think Jiefei would like to know how one can "officially" determine if an >> > arbitrary ALTERP object belongs to a class that he owns. >> > >> > Regards, >> > Denes >> > >> > >> > On 10/19/20 10:22 AM, Benjamin Christoffersen wrote: >> > > It seems as if you can you use the ALTREP macro as done in this >> > > package: >> https://github.com/Jiefei-Wang/SharedObject/blob/804b6ac58c63a4bae95343ab43e8b1547b07ee6b/src/C_interface.cpp#L185 >> > > >> > > and in base R: >> https://github.com/wch/r-source/blob/54fbdca9d3fc63437d9e697f442d32732fb4f443/src/include/Rinlinedfuns.h#L118 >> > > >> > > The macro is defined here in Rinternals.h: >> > > >> https://github.com/wch/r-source/blob/abb550c99b3927e5fc03d12f1a8e7593fddc04d2/src/include/Rinternals.h#L325 >> > > >> > > Den man. 19. okt. 2020 kl. 10.13 skrev Jiefei Wang > >: >> > >> >> > >> Hi all, >> > >> >> > >> I would like to determine if an ALTREP object is from my package, I >> see >> > >> there is a function `ALTREP_CLASS` defined in RInternal.h but its >> return >> > >> value is neither a `R_altrep_class_t` object nor an STRSXP >> representing a >> > >> class name. I do not know how to correctly use it. Any suggestions? >> > >> >> > >> Thanks, >> > >> Jiefei >> > >> >> > >> [[alternative HTML version deleted]] >> > >> >> > >> __ >> > >> R-devel@r-project.org mailing list >> > >> https://stat.ethz.ch/mailman/listinfo/r-devel >> > > >> > > __ >> > > R-devel@r-project.org mailing list >> > > https://stat.ethz.ch/mailman/listinfo/r-devel >> > > >> > [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Is there any way to check the class of an ALTREP?
Thank Denes for the clarification, glad to see my package got one citation from my own question:) Also, thank Benjamin for sending many useful documents. Actually, the question is related to the SharedObject package that Benjamin has pointed to. I wanna avoid sharing an object that has already been shared, so I need to check whether an object is an ALTREP that is defined in my package. Since the ALTREP definition involves an ` R_altrep_class_t` object and a class name, I will expect to see a method defined in R to retrieve such information. I hope this can clarify my question. Best, Jiefei On Mon, Oct 19, 2020 at 5:38 PM Benjamin Christoffersen wrote: > > You happened to send a link which points to the OP's own package :) I > > think Jiefei would like to know how one can "officially" determine if an > > arbitrary ALTERP object belongs to a class that he owns. > Argh, I am sorry! I did not notice that. My best bet for what I > thought the question was is section 1.1.2 of R Internals where they > note that: `unsigned int alt : 1; /* is this an ALTREP object? */` > > This is also what the ALTREP macro checks. However, I also did not > notice the "... if an ALTREP object is from my package" part. I am > sorry. > > Sincerely Yours, > Benjamin > > Den man. 19. okt. 2020 kl. 11.25 skrev Dénes Tóth >: > > > > Benjamin, > > > > You happened to send a link which points to the OP's own package :) I > > think Jiefei would like to know how one can "officially" determine if an > > arbitrary ALTERP object belongs to a class that he owns. > > > > Regards, > > Denes > > > > > > On 10/19/20 10:22 AM, Benjamin Christoffersen wrote: > > > It seems as if you can you use the ALTREP macro as done in this > > > package: > https://github.com/Jiefei-Wang/SharedObject/blob/804b6ac58c63a4bae95343ab43e8b1547b07ee6b/src/C_interface.cpp#L185 > > > > > > and in base R: > https://github.com/wch/r-source/blob/54fbdca9d3fc63437d9e697f442d32732fb4f443/src/include/Rinlinedfuns.h#L118 > > > > > > The macro is defined here in Rinternals.h: > > > > https://github.com/wch/r-source/blob/abb550c99b3927e5fc03d12f1a8e7593fddc04d2/src/include/Rinternals.h#L325 > > > > > > Den man. 19. okt. 2020 kl. 10.13 skrev Jiefei Wang >: > > >> > > >> Hi all, > > >> > > >> I would like to determine if an ALTREP object is from my package, I > see > > >> there is a function `ALTREP_CLASS` defined in RInternal.h but its > return > > >> value is neither a `R_altrep_class_t` object nor an STRSXP > representing a > > >> class name. I do not know how to correctly use it. Any suggestions? > > >> > > >> Thanks, > > >> Jiefei > > >> > > >> [[alternative HTML version deleted]] > > >> > > >> __ > > >> R-devel@r-project.org mailing list > > >> https://stat.ethz.ch/mailman/listinfo/r-devel > > > > > > __ > > > R-devel@r-project.org mailing list > > > https://stat.ethz.ch/mailman/listinfo/r-devel > > > > [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] Is there any way to check the class of an ALTREP?
Hi all, I would like to determine if an ALTREP object is from my package, I see there is a function `ALTREP_CLASS` defined in RInternal.h but its return value is neither a `R_altrep_class_t` object nor an STRSXP representing a class name. I do not know how to correctly use it. Any suggestions? Thanks, Jiefei [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Coercion function does not work for the ALTREP object
Hi Gabriel, here is a simple package for reproducing the problem. https://github.com/Jiefei-Wang/testPkg Best, Jiefei On Thu, Oct 8, 2020 at 5:04 AM Gabriel Becker wrote: > Jiefei, > > Where does the code for your altrep class live? > > Thanks, > ~G > > On Wed, Oct 7, 2020 at 4:25 AM Jiefei Wang wrote: > >> Hi all, >> >> The coercion function defined for the ALTREP object will not be called by >> R >> when an assignment operation implicitly introduces coercion for a large >> ALTREP object. >> >> For example, If I create a vector of length 10, the ALTREP coercion >> function seems to work fine. >> ``` >> > x <- 1:10 >> > y <- wrap_altrep(x) >> > .Internal(inspect(y)) >> @0x1f9271c0 13 INTSXP g0c0 [REF(2)] I am altrep >> > y[1] <- 1.0 >> Duplicating object >> Coercing object >> > .Internal(inspect(y)) >> @0x1f927c08 14 REALSXP g0c0 [REF(1)] I am altrep >> ``` >> >> However, if I create a vector of length 1024, R will give me a normal >> real-type vector >> ``` >> > x <- 1:1024 >> > y <- wrap_altrep(x) >> > .Internal(inspect(y)) >> @0x1f8ddb20 13 INTSXP g0c0 [REF(2)] I am altrep >> > y[1] <- 1.0 >> > .Internal(inspect(y)) >> @0x1f0d72a0 14 REALSXP g0c7 [REF(1)] (len=1024, tl=0) >> 1,2,3,4,5,... >> ``` >> >> Note that the duplicate function is also called for the first example. It >> seems like R completely ignores my ALTREP functions in the second example. >> I feel this might be designed on purpose, but I do not understand the >> reason behind it. Is there any reason why we are not consistent here? Here >> is my session info >> >> sessionInfo() >> R Under development (unstable) (2020-09-03 r79126) >> Platform: x86_64-w64-mingw32/x64 (64-bit) >> Running under: Windows 10 x64 (build 18362) >> >> Best, >> Jiefei >> >> [[alternative HTML version deleted]] >> >> __ >> R-devel@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel >> > [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] Coercion function does not work for the ALTREP object
Hi all, The coercion function defined for the ALTREP object will not be called by R when an assignment operation implicitly introduces coercion for a large ALTREP object. For example, If I create a vector of length 10, the ALTREP coercion function seems to work fine. ``` > x <- 1:10 > y <- wrap_altrep(x) > .Internal(inspect(y)) @0x1f9271c0 13 INTSXP g0c0 [REF(2)] I am altrep > y[1] <- 1.0 Duplicating object Coercing object > .Internal(inspect(y)) @0x1f927c08 14 REALSXP g0c0 [REF(1)] I am altrep ``` However, if I create a vector of length 1024, R will give me a normal real-type vector ``` > x <- 1:1024 > y <- wrap_altrep(x) > .Internal(inspect(y)) @0x1f8ddb20 13 INTSXP g0c0 [REF(2)] I am altrep > y[1] <- 1.0 > .Internal(inspect(y)) @0x1f0d72a0 14 REALSXP g0c7 [REF(1)] (len=1024, tl=0) 1,2,3,4,5,... ``` Note that the duplicate function is also called for the first example. It seems like R completely ignores my ALTREP functions in the second example. I feel this might be designed on purpose, but I do not understand the reason behind it. Is there any reason why we are not consistent here? Here is my session info sessionInfo() R Under development (unstable) (2020-09-03 r79126) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 18362) Best, Jiefei [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Bioc-devel] MPFE Bioconductor package
Hi Conrad,
Glad to hear that you have solved the problem. Sorry for hearing your bad
experience, I guess it could be better if you can ask your question in a
git mailing list(If there is one) because it is really not a package
administrative problem which the Bioconductor core team is responsible for.
What Nitesh said does not mean you should sort it out for your self, it
just means you should ask it in a better place, and he forwarded your email
to here. Bioc mailing list is mostly for package development, so you can
get two responses for that(Even though we did not correctly understand your
question). Please do not feel anxious about it, you are welcome to ask for
help.
Best,
Jiefei
On Fri, Sep 18, 2020 at 12:46 PM Conrad Burden
wrote:
>
> Hi guys,
>
> It seems to be working now. There is one warning message, which I believe
> is just to tell me that the package had been "deprecated” (which
> apparently does not mean what my Oxford English Dictionary tells me it
> means).
>
> Could you please confirm whether I have to do anything else for it to go
> into the next release?
>
> By the way, I should have pointed out from the beginning that my problem
> was NOT with debugging the R code. I had that sorted out almost
> immediately back in April (but thanks for confirming that you came up with
> the same solution Jiefei). My problem was with working out how to get this
> trivial 1-line change into the bioconductor system. If one is not already
> familiar with unix, GitHub, ssh keys, repositories and heaven knows what
> else, and if one doesn’t know the terminology, the on-liine instructions
> are impossible to follow, assuming, that is, one can find the right web
> page to start with. I was somewhat put off by asking further by the
> sentence "Unfortunately we are not able to devote time towards individual
> packages”, which I took to mean “go away, stop bothering us, and sort it
> out for yourself”.
>
> cheers, Conrad.
>
> A/Prof. Conrad Burden
> Mathematical Sciences Institute
> Building 145
> Australian National University
>
> Phone: 61-2-61250730
> E-mail: conrad.bur...@anu.edu.au
> Web page: http://maths.anu.edu.au/people/conrad-burden
>
>
> On 17 Sep 2020, at 8:36 pm, Jiefei Wang wrote:
>
> Hi Conrad,
>
> Just a friendly reminder, there is a bug in your package. Here is the
> build result from the link you provided:
>
> ==
> --- Error message ---
> --- failure: length > 1 in coercion to logical ---
>
> --- call from argument ---
> columns < 1 || columns > nColumns
>
> --- place where the error occurs ---
> if (any(columns < 1 || columns > nColumns)) {
> stop("Column indices must be between 1 and ", nColumns,
> "\n")
> }
> ==
>
> The symbol `||` only applies the logical OR on the first element of two
> operands. Since you need the elementwise comparison, you should use the
> symbol `|` instead.
>
> Best,
> Jiefei
>
>
>
> On Thu, Sep 17, 2020 at 2:11 PM Conrad Burden
> wrote:
>
>>
>> Hi Matt, Nitesh,
>>
>> Thanks for replying quickly and for sending the link to the full check
>> report, which I had not been able to find.
>>
>> I think I have worked out what the problem might be. I have found the
>> page “Troubleshooting Build Report” which tells me "Changes pushed to
>> Bioconductor before 4:45 will be reflected in the following day’s build
>> report that is posted around 1:00 PM EST”, which I assume is US eastern
>> standard time.
>>
>> I pushed my changes (MPFE v1.5.2) to the Bioconductor repository at
>> approximately 2:30pm Canberra time yesterday, 16 September, which is 12:30
>> am 16 September on the east coast of the US. It won’t appear in the build
>> report unitl 1:00pm 17 September US time, which is 3:00am 18 September in
>> Canberra. I’ll check it again tomorrow. Sorry to put you to all this
>> trouble.
>>
>> - cheers, Conrad.
>>
>> A/Prof. Conrad Burden
>> Mathematical Sciences Institute
>> Building 145
>> Australian National University
>>
>> Phone: 61-2-61250730
>> E-mail: conrad.bur...@anu.edu.au<mailto:conrad.bur...@anu.edu.au>
>> Web page: http://maths.anu.edu.au/people/conrad-burden
>>
>>
>> On 17 Sep 2020, at 1:16 pm, Stone, Matt > mst...@fredhutch.org>> wrote:
>>
>> Hi Dr. Burden,
>>
>> The "ERROR" links on that report page link to the full check reports for
>> each platform. Here is the report from malbec1, one of the Linux machines.
>>
>>
>> http
Re: [Bioc-devel] MPFE Bioconductor package
Hi Conrad,
Just a friendly reminder, there is a bug in your package. Here is the build
result from the link you provided:
==
--- Error message ---
--- failure: length > 1 in coercion to logical ---
--- call from argument ---
columns < 1 || columns > nColumns
--- place where the error occurs ---
if (any(columns < 1 || columns > nColumns)) {
stop("Column indices must be between 1 and ", nColumns,
"\n")
}
==
The symbol `||` only applies the logical OR on the first element of two
operands. Since you need the elementwise comparison, you should use the
symbol `|` instead.
Best,
Jiefei
On Thu, Sep 17, 2020 at 2:11 PM Conrad Burden
wrote:
>
> Hi Matt, Nitesh,
>
> Thanks for replying quickly and for sending the link to the full check
> report, which I had not been able to find.
>
> I think I have worked out what the problem might be. I have found the
> page “Troubleshooting Build Report” which tells me "Changes pushed to
> Bioconductor before 4:45 will be reflected in the following day’s build
> report that is posted around 1:00 PM EST”, which I assume is US eastern
> standard time.
>
> I pushed my changes (MPFE v1.5.2) to the Bioconductor repository at
> approximately 2:30pm Canberra time yesterday, 16 September, which is 12:30
> am 16 September on the east coast of the US. It won’t appear in the build
> report unitl 1:00pm 17 September US time, which is 3:00am 18 September in
> Canberra. I’ll check it again tomorrow. Sorry to put you to all this
> trouble.
>
> - cheers, Conrad.
>
> A/Prof. Conrad Burden
> Mathematical Sciences Institute
> Building 145
> Australian National University
>
> Phone: 61-2-61250730
> E-mail: conrad.bur...@anu.edu.au
> Web page: http://maths.anu.edu.au/people/conrad-burden
>
>
> On 17 Sep 2020, at 1:16 pm, Stone, Matt mst...@fredhutch.org>> wrote:
>
> Hi Dr. Burden,
>
> The "ERROR" links on that report page link to the full check reports for
> each platform. Here is the report from malbec1, one of the Linux machines.
>
>
> http://bioconductor.org/checkResults/3.12/bioc-LATEST/MPFE/malbec1-checksrc.html
>
> If you're not familiar with the output of R CMD check, it runs a series of
> diagnostic checks and reports any detected issues. In order of decreasing
> severity, they are ERROR, WARNING, and NOTE. The Bioconductor guidelines
> [1] indicate that errors and warnings must be fixed in order for the
> package to be accepted. (You can find more details on the checks in the R
> Packages book [2].)
>
> This check report found one ERROR, one WARNING, and one NOTE. The ERROR
> was caused by an error in one of the documented examples in the file
> `MPFE-Ex.R`, which you can see in this excerpt from the report:
>
> =
> * checking examples ... ERROR
> Running examples in ‘MPFE-Ex.R’ failed
> The error most likely occurred in:
>
> base::assign(".ptime", proc.time(), pos = "CheckExEnv")
> ### Name: MPFE-package
> ### Title: MPFE
> ### Aliases: MPFE-package MPFE
> ### Keywords: amplicon, bisulphite sequencing, bisufite sequencing,
> ### methylation
>
> ### ** Examples
>
> data(patternsExample)
> estimates <- estimatePatterns(patternsExample, epsilon=0.02, eta=0.01)
> =
>
> In my experience, errors in the examples are caused by either a) buggy or
> outdated code in the example or b) a bug in one of the package functions
> that the example uses. Here, I would guess that either a bug exists in the
> estimatePatterns() function, or the data loaded from patternsExample isn't
> as expected. You can reproduce and debug the error by running the example
> yourself.
>
> For the other two checks: The WARNING was caused by the current
> deprecation status of MPFE (which I'm guessing you're in the process of
> updating). The NOTE suggests you explicitly import some functions that
> haven't yet been declared in the NAMESPACE. I'm relatively new to
> Bioconductor and so I don't know if this is mandated, but I would suggest
> fixing this as well.
>
> Hope this was helpful!
>
> Matt
>
> [1] https://www.bioconductor.org/developers/package-guidelines/
> [2] https://r-pkgs.org/r-cmd-check.html#check-checks
>
> On 9/16/20, 7:28 PM, "Bioc-devel on behalf of Turaga, Nitesh" <
> bioc-devel-boun...@r-project.org on behalf of
> nitesh.tur...@roswellpark.org> wrote:
>
>Hi Conrad,
>
>Sorry to hear you’ve had such a hard time with the Bioconductor GIT
> ecosystem. Please keep in mind that for a new build to be triggered you
> have to update the version number in the DESCRIPTION file of the package(
> The “x.y.z” version format needs the “z” to be incremented ).
>
>Also, I highly encourage you to email the bioc-devel@r-project.org
> mailing list where other people can
> help you more readily.
>
>Best,
>
>Nitesh
>
>From: Conrad
Re: [Bioc-devel] DECIPHER questions
Hi Henry, It seems like you are using a Bioconductor package. Why not directly ask the author of the package(eswri...@pitt.edu)? JFYI, here is the link for the package, it seems like the author has written a lot of documentation for the package: https://www.bioconductor.org/packages/release/bioc/html/DECIPHER.html Best, Jiefei On Wed, Sep 16, 2020 at 4:03 PM Henry Luo wrote: > Hi, my name is Henry Luo. I am NCU master student from Taiwan. I need some > help. Recently I do my project about probe design to detect environmental > bacteria, and I have used DECIPHER package on my R session. Today I do my > progress report to my Professor. Professor don't know whether DECIPHER > package is appropriate for my project because my project is about probe > design for environmental sample, and he want me to make sure, and I really > don't know how to make sure. > So I want to ask you about the appropriateness. It's very important for me. > How could I make sure and tell my Professor? Thank you. > > Best regards, > > Henry Luo > > [[alternative HTML version deleted]] > > ___ > Bioc-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/bioc-devel > [[alternative HTML version deleted]] ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Rd] [External] Thread-safe R functions
Thanks, Gabriel and Luke. That is exactly what I would like to know. I appreciate your suggestions. Best, Jiefei On Mon, Sep 14, 2020 at 6:08 AM Gabriel Becker wrote: > Jiefei, > > Beyond the general response that Luke gave, to be a bit more specific to > what you said, DATAPTR and INTEGER_GET_REGION involve ALTREP method > execution (for ALTREP objects, obviously) so even they are not as simple > and straightforward as they were a couple years ago. They should not (any > longer) be thought of as being guaranteed to be essentially bare metal data > retrieval from memory. > > Best, > ~G > > On Sun, Sep 13, 2020 at 6:49 AM wrote: > >> You should assume that NO functions or macros in the R API are >> thread-safe. If some happen to be now, on some platforms, they are >> not guaranteed to be in the future. Even if you use a global lock you >> need to keep in mind that any function in the R API can signal an >> error and execute a longjmp, so you need to make sure you have set a >> top level context in your thread. >> >> Best, >> >> luke >> >> On Sun, 13 Sep 2020, Jiefei Wang wrote: >> >> > Hi, >> > >> > I am curious about whether there exist thread-safe functions in >> > `Rinternals.h`. I know that R is single-threaded designed, but for the >> > simple and straightforward functions like `DATAPTR` and >> `INTEGER_GET_REGION`, >> > are these functions safe to call in a multi-thread environment? > > > >> > Best, >> > Jiefei >> > >> > [[alternative HTML version deleted]] >> > >> > __ >> > R-devel@r-project.org mailing list >> > https://stat.ethz.ch/mailman/listinfo/r-devel >> > >> >> -- >> Luke Tierney >> Ralph E. Wareham Professor of Mathematical Sciences >> University of Iowa Phone: 319-335-3386 >> Department of Statistics andFax: 319-335-3017 >> Actuarial Science >> 241 Schaeffer Hall email: luke-tier...@uiowa.edu >> Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu >> >> __ >> R-devel@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel >> > [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] Thread-safe R functions
Hi, I am curious about whether there exist thread-safe functions in `Rinternals.h`. I know that R is single-threaded designed, but for the simple and straightforward functions like `DATAPTR` and `INTEGER_GET_REGION`, are these functions safe to call in a multi-thread environment? Best, Jiefei [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
