Re: [R-pkg-devel] About the CRAN policy on downloading pre-compiled binary
Hi, Thank you so much Tomas for helping me to figure out the things. Let me share some updates (some of them are what you've seen already in the reply to the CRAN maintainer) in case this might be eventually useful to those who are curious about using Rust on CRAN. First of all, let me correct my misunderstanding. I thought what violates the CRAN policy was that my package downloads a binary, but in actual, it seems my fault was 1. if the compiler exists, my package downloads the **sources** of the dependency, whose authorships are not described. 2. if the compiler doesn't exist, my package downloads the pre-compiled binary **without the agreement of the CRAN team** For point 2, as suggested in the previous comment, > at least check whether the compiler is present (during package installation), and use it if it is. my package first checks if Cargo, Rust's compiler toolchain and package manager, is installed (more specifically, whether the `cargo` command can be found on PATH), and falls back to downloading the pre-compiled binary only when Cargo is not available. So, I believe this can be considered as "a last resort". Again, it was clearly my fault that I didn't ask the CRAN maintainers for permission on my first submission. However, it seems Cargo is already available on the following CRAN machines: - Linux machines - Windows Server 2022 (at least on winbuilder. Thanks David B. Dahl for the information!) - M1 macOS machine (guessed from the comment from the CRAN maintainer) If I remember correctly, the Windows machine didn't have Cargo installed when I submitted my package to CRAN. I'd really appreciate the CRAN maintainers' effort to improve the infrastructure, and I hope Cargo will be available on Intel macOS machines in the not so distant future so that I no longer need to provide the pre-compiled binary (I'm not sure if I can have the same hope on the Windows 2008, which might be a bit too old to install some latest-ish version of Cargo, though [2]). For point 1, it's Cargo that downloads the sources. It does download the fixed versions recorded in the bundled "Cargo.lock" file, and verifies the checksum. While Cargo provides the vendoring mechanism [1], I thought it was not realistic to include the sources because my dependency was so huge (132MB). But, I was wrong at two points. First, no matter whether to download or to bundle the sources, the CRAN Policy requires to describe the copyright information of "all components of the package," which should refer to all the individual Rust crate in the context of my package. > The ownership of copyright and intellectual property rights of all components of the package must be clear and unambiguous (including from the authors specification in the DESCRIPTION file). Where code is copied (or derived) from the work of others (including from R itself), care must be taken that any copyright/license statements are preserved and authorship is not misrepresented. Second, it was possible to reduce the size to the extent where I can bundle it. I could tweak some libraries to drop the heavy dependency, and compress the source code (again, thanks David B. Dahl for the idea). I'm yet to figure out what's the proper way to list up the authors (I'm currently using inst/COPYRIGHTS, but I'm not fully sure at the moment), but I'm working on tweaking my package to bundle all the sources and to clarify the authorships of these dependencies [2]. I hope this will keep my package on CRAN. Best, Yutani [1]: https://doc.rust-lang.org/cargo/commands/cargo-vendor.html [2]: Actually, it keeps failing on Windows Server 2008 with the pre-compiled binary probably due to lack of some system API: https://www.r-project.org/nosvn/R.check/r-oldrel-windows-ix86+x86_64/string2path-00check.html [3]: https://github.com/yutannihilation/string2path/pull/35 (work in progress) 2022年7月27日(水) 17:12 Tomas Kalibera : > > On 7/27/22 08:08, Tomas Kalibera wrote: > > > > On 7/27/22 00:30, Hiroaki Yutani wrote: > >> Hi, > >> > >> Recently I got the following email from the CRAN maintainer about my > >> package, string2path[1]. > >> > >> However, I do ensure the binary is the pinned version and verify if the > >> hash matches with the embedded one in the DESCRIPTION [2][3]. In case > >> of a > >> mismatch, the build fails. So, this mechanism should ensure that I (or > >> anyone) cannot change the version of the binary without actually > >> resubmitting to CRAN. > > > > Please see the policy cited. Ensuring that the download is of a fixed > > version refers to the sources (which can be downloaded under the > > conditions mentioned). > > > > Downloading binaries are only a last resort option and requires the > > agreement of the CRAN team in the first place. > > > >> I believe this complies with the CRAN policy (except for not clearing > >> the > >> authorship and copyright). Is there anything I have to address to > >> prove I > >> do "ensure that the download is of a fi
Re: [R-pkg-devel] About the CRAN policy on downloading pre-compiled binary
On 7/27/22 08:08, Tomas Kalibera wrote: On 7/27/22 00:30, Hiroaki Yutani wrote: Hi, Recently I got the following email from the CRAN maintainer about my package, string2path[1]. However, I do ensure the binary is the pinned version and verify if the hash matches with the embedded one in the DESCRIPTION [2][3]. In case of a mismatch, the build fails. So, this mechanism should ensure that I (or anyone) cannot change the version of the binary without actually resubmitting to CRAN. Please see the policy cited. Ensuring that the download is of a fixed version refers to the sources (which can be downloaded under the conditions mentioned). Downloading binaries are only a last resort option and requires the agreement of the CRAN team in the first place. I believe this complies with the CRAN policy (except for not clearing the authorship and copyright). Is there anything I have to address to prove I do "ensure that the download is of a fixed version"? Any suggestions are welcome. My understanding from your email is you are ensuring a fixed version download, and with most projects you could probably do even less (simply hardcode a URL which includes a specific version of the sources if that is stable for the project), but you are not downloading sources. I take back the comment about doing less to ensure a fixed version - it is good to use a hash always when having to download sources, as software distributions do (of course including MXE and Msys2, so Rtools on Windows, etc). It is better as a guard against that the content is in fact not stable for the project (I've been told it sometimes happens and may be surprising to package authors), but of course also for security reasons. Tomas In either case, of course if there is anything unclear in an email from CRAN, you can simply respond to that and ask. Best Tomas The CRAN policy stipulates "Where a package wishes to make use of a library not written solely for the package, the package installation should first look to see if it is already installed and if so is of a suitable version. In case not, it is desirable to include the library sources in the package and compile them as part of package installation. If the sources are too large, it is acceptable to download them as part of installation, but do ensure that the download is of a fixed version rather than the latest. Only as a last resort and with the agreement of the CRAN team should a package download pre-compiled software." and we have recently seen an instance of a rust-using package whose check output changed because what it downloaded had changed. CRAN checking is not set up for that (for example, macOS checks are done once only for each version). Whilst investigating, the Windows' maintainers found that binary libs were being downloaded. And subsequently I found that salso, string2path and ymd are downloading compiled code on Intel macOS. Also. make sure that the authorship and copyright of code you download (and hence include in the package) is clear from the DESCRIPTION file. as required by the CRAN policy. Best, Hiroaki Yutani [1]: https://cran.r-project.org/package=string2path [2]: https://github.com/cran/string2path/blob/46020296410cd78e2021bff86cb6f17c681d13a6/DESCRIPTION#L29-L40 [3]: https://github.com/cran/string2path/blob/46020296410cd78e2021bff86cb6f17c681d13a6/tools/configure.R#L177-L295 [[alternative HTML version deleted]] __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] About the CRAN policy on downloading pre-compiled binary
On 7/27/22 09:03, Hiroaki Yutani wrote: > Thank you. I'll read the policy again more carefully. > > > but you are not downloading sources. > > The case is that, I include the Rust source code, but since the CRAN's > Windows & macOS machines don't have the Rust compiler installed, > there's no choice but to download the pre-compiled binary (other than > downloading the Rust compiler). I'd say at least check whether the compiler is present (during package installation), and use it if it is. Only if you want to download binaries as a last resort when it isn't, consult that first with the CRAN team. Perhaps it could also be a hard dependency for installing the package, and perhaps Rust compiler would be installed on the CRAN machines. I think it is better not to hard-code which software is installed on the CRAN systems, because that may change and because different software may be installed on user machines. However, on Windows, it is ok to expect that say Rtools42 is available when building packages for R 4.2 (yet Rtools42 don't have a rust compiler, so it doesn't help you here). Right, downloading the source code of the Rust compiler and building it seems a bit too resource intensive and then, a bit too much if multiple R packages need it. Ideally the compiler will be installed and the R source packages will really be source packages (no binaries). > > > In either case, of course if there is anything unclear in an email from > > CRAN, you can simply respond to that and ask. > > Thanks, I already responded with what I wrote above, so I hope I can > get the reply. > But, since the CRAN team might be on vacation now, I wanted to get > some advice on this mailing list. > Thanks for your help! No problem, just please only regard it as my advice. Best Tomas > > Best, > Yutani > > 2022年7月27日(水) 15:08 Tomas Kalibera : > > > On 7/27/22 00:30, Hiroaki Yutani wrote: > > Hi, > > > > Recently I got the following email from the CRAN maintainer about my > > package, string2path[1]. > > > > However, I do ensure the binary is the pinned version and verify > if the > > hash matches with the embedded one in the DESCRIPTION [2][3]. In > case of a > > mismatch, the build fails. So, this mechanism should ensure that > I (or > > anyone) cannot change the version of the binary without actually > > resubmitting to CRAN. > > Please see the policy cited. Ensuring that the download is of a fixed > version refers to the sources (which can be downloaded under the > conditions mentioned). > > Downloading binaries are only a last resort option and requires the > agreement of the CRAN team in the first place. > > > I believe this complies with the CRAN policy (except for not > clearing the > > authorship and copyright). Is there anything I have to address > to prove I > > do "ensure that the download is of a fixed version"? Any > suggestions are > > welcome. > > My understanding from your email is you are ensuring a fixed version > download, and with most projects you could probably do even less > (simply > hardcode a URL which includes a specific version of the sources if > that > is stable for the project), but you are not downloading sources. > > In either case, of course if there is anything unclear in an email > from > CRAN, you can simply respond to that and ask. > > Best > Tomas > > > > > The CRAN policy stipulates > >> "Where a package wishes to make use of a library not written > solely for > >> the package, the package installation should first look to see > if it is > >> already installed and if so is of a suitable version. In case > not, it is > >> desirable to include the library sources in the package and > compile them > >> as part of package installation. If the sources are too large, > it is > >> acceptable to download them as part of installation, but do > ensure that > >> the download is of a fixed version rather than the latest. Only > as a > >> last resort and with the agreement of the CRAN team should a > package > >> download pre-compiled software." > >> > >> and we have recently seen an instance of a rust-using package whose > >> check output changed because what it downloaded had changed. CRAN > >> checking is not set up for that (for example, macOS checks are > done once > >> only for each version). > >> > >> Whilst investigating, the Windows' maintainers found that > binary libs > >> were being downloaded. And subsequently I found that salso, > string2path > >> and ymd are downloading compiled code on Intel macOS. > >> > >> Also. make sure that the authorship and copyright of code you > download > >> (and hence include in the package) is clear from the > DESCRIPTION file. > >> as required by the CRA
[R-pkg-devel] About the CRAN policy on downloading pre-compiled binary
Hi, Recently I got the following email from the CRAN maintainer about my package, string2path[1]. However, I do ensure the binary is the pinned version and verify if the hash matches with the embedded one in the DESCRIPTION [2][3]. In case of a mismatch, the build fails. So, this mechanism should ensure that I (or anyone) cannot change the version of the binary without actually resubmitting to CRAN. I believe this complies with the CRAN policy (except for not clearing the authorship and copyright). Is there anything I have to address to prove I do "ensure that the download is of a fixed version"? Any suggestions are welcome. The CRAN policy stipulates > > "Where a package wishes to make use of a library not written solely for > the package, the package installation should first look to see if it is > already installed and if so is of a suitable version. In case not, it is > desirable to include the library sources in the package and compile them > as part of package installation. If the sources are too large, it is > acceptable to download them as part of installation, but do ensure that > the download is of a fixed version rather than the latest. Only as a > last resort and with the agreement of the CRAN team should a package > download pre-compiled software." > > and we have recently seen an instance of a rust-using package whose > check output changed because what it downloaded had changed. CRAN > checking is not set up for that (for example, macOS checks are done once > only for each version). > > Whilst investigating, the Windows' maintainers found that binary libs > were being downloaded. And subsequently I found that salso, string2path > and ymd are downloading compiled code on Intel macOS. > > Also. make sure that the authorship and copyright of code you download > (and hence include in the package) is clear from the DESCRIPTION file. > as required by the CRAN policy. > Best, Hiroaki Yutani [1]: https://cran.r-project.org/package=string2path [2]: https://github.com/cran/string2path/blob/46020296410cd78e2021bff86cb6f17c681d13a6/DESCRIPTION#L29-L40 [3]: https://github.com/cran/string2path/blob/46020296410cd78e2021bff86cb6f17c681d13a6/tools/configure.R#L177-L295 [[alternative HTML version deleted]] __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel