Re: [R-pkg-devel] What to do when a package is archived from CRAN

[Hiroaki Yutani]

Simon,

Ok, let's take a look at a real example. The first item of inst/AUTHORS of
prqlr (GitHub version) is this:

addr2line (version 0.20.0):
  addr2line authors

You can find addr2line's owners on crates.io [1], while its manifest file
(Cargo.toml) [2] doesn't contain the names of its owners or authors. In
Rust's manifest, the "authors" field is optional [3] unlike R. You might
argue "owners" is not the same as "authors," but at least crates.io
provides the names of those who are responsible for the crate.

Let's go back to your question.

> So are you saying you have to use crates.io and do some extra step during
the (misnamed) "vendor" step?

"cargo vendor" doesn't take care of generating the list of authors, so it's
not "during the vender step." It has to be done separately anyway. I was
just saying you **can** use crates.io in that step instead of searching for
the authors manually one by one (or filling it with "foo authors" when the
manifest file doesn't contain any names).

That said, I agree with you in general that the Rust community is
relatively loose about authorship and licensing when compared with R. I
don't think it's necessarily a problem, but the impedance mismatch is a
headache. I was just trying to point out this part of your opinion

> the Rust community as there doesn't seem to be any accountability with
respect to ownership and attribution.

was not quite true. I hope the R community and the Rust community have
respect for each other.

Best,
Yutani

[1]: https://crates.io/crates/addr2line
[2]: https://github.com/gimli-rs/addr2line/blob/0.20.0/Cargo.toml
[3]:
https://doc.rust-lang.org/cargo/reference/manifest.html#the-authors-field


2023年8月27日(日) 12:07 Simon Urbanek :

> Yutani,
>
>
> On Aug 27, 2023, at 2:19 PM, Hiroaki Yutani  wrote:
>
> Simon,
>
> > it's assumed that GitHub history is the canonical source with the
> provenance, but that gets lost when pulled into the package.
>
> No, not GitHub. You can usually find the ownership on crates.io. So, if
> you want a target to blame, it's probably just a problem of the script to
> auto-generate inst/AUTHORS in this specific case. But, clearly, Rust's
> ecosystem works soundly under the existence of crates.io, so I think this
> is the same kind of pain which you would feel if you use R without CRAN.
>
>
> Can you elaborate? I have not found anything that would have a list of
> authors in the sources. I fully agree that I know nothing about it, but
> even if you use R without CRAN, each package contains that information in
> the DESCRIPTION file since it's so crucial. So are you saying you have to
> use crates.io and do some extra step during the (misnamed) "vendor" step?
> (I didn't see the submitted tar ball of plqrl and its release on GitHub is
> not the actual package so can't check - thus just trying reverse-engineer
> what happens by looking at the dependencies which leads to GitHub).
>
>
> Sorry for nitpicking.
>
>
> Sure, good to get the fact straight.
>
> Cheers,
> Simon
>
>
>
> Best,
> Yutani
>
> 2023年8月27日(日) 6:57 Simon Urbanek :
>
>> Tatsuya,
>>
>> What you do is contact CRAN. I don't think anyone here can answer your
>> question, only CRAN can, so ask there.
>>
>> Generally, packages with sufficiently many Rust dependencies have to be
>> handled manually as they break the size limit, so auto-rejections are
>> normal. Archival is unusual, but it may have fallen through the cracks -
>> but the way to find out is to ask.
>>
>> One related issue with respect to CRAN policies that I don't see a good
>> solution for is that inst/AUTHORS is patently unhelpful, because most of
>> them say "foo (version ..): foo authors" with no contact, or real names or
>> any links. That seems to be a problem stemming from the Rust community as
>> there doesn't seem to be any accountability with respect to ownership and
>> attribution. I don't know if it's because it's assumed that GitHub history
>> is the canonical source with the provenance, but that gets lost when pulled
>> into the package.
>>
>> Cheers,
>> Simon
>>
>> PS: Your README says "(Rust 1.65 or later)", but the version condition is
>> missing from SystemRequirements.
>>
>>
>> > On Aug 26, 2023, at 2:46 PM, SHIMA Tatsuya  wrote:
>> >
>> > Hi,
>> >
>> > I noticed that my submitted package `prqlr` 0.5.0 was archived from
>> CRAN on 2023-08-19.
>> > <https://CRAN.R-project.org/package=prqlr
>> <https://cran.r-project.org/package=prqlr>>
>> >
>> > I submitted prqlr 0.5.0 on 2023-

Re: [R-pkg-devel] What to do when a package is archived from CRAN

[Hiroaki Yutani]

Simon,

> it's assumed that GitHub history is the canonical source with the
provenance, but that gets lost when pulled into the package.

No, not GitHub. You can usually find the ownership on crates.io. So, if you
want a target to blame, it's probably just a problem of the script to
auto-generate inst/AUTHORS in this specific case. But, clearly, Rust's
ecosystem works soundly under the existence of crates.io, so I think this
is the same kind of pain which you would feel if you use R without CRAN.

Sorry for nitpicking.

Best,
Yutani

2023年8月27日(日) 6:57 Simon Urbanek :

> Tatsuya,
>
> What you do is contact CRAN. I don't think anyone here can answer your
> question, only CRAN can, so ask there.
>
> Generally, packages with sufficiently many Rust dependencies have to be
> handled manually as they break the size limit, so auto-rejections are
> normal. Archival is unusual, but it may have fallen through the cracks -
> but the way to find out is to ask.
>
> One related issue with respect to CRAN policies that I don't see a good
> solution for is that inst/AUTHORS is patently unhelpful, because most of
> them say "foo (version ..): foo authors" with no contact, or real names or
> any links. That seems to be a problem stemming from the Rust community as
> there doesn't seem to be any accountability with respect to ownership and
> attribution. I don't know if it's because it's assumed that GitHub history
> is the canonical source with the provenance, but that gets lost when pulled
> into the package.
>
> Cheers,
> Simon
>
> PS: Your README says "(Rust 1.65 or later)", but the version condition is
> missing from SystemRequirements.
>
>
> > On Aug 26, 2023, at 2:46 PM, SHIMA Tatsuya  wrote:
> >
> > Hi,
> >
> > I noticed that my submitted package `prqlr` 0.5.0 was archived from CRAN
> on 2023-08-19.
> > 
> >
> > I submitted prqlr 0.5.0 on 2023-08-13. I believe I have since only
> received word from CRAN that it passed the automated release process. <
> https://github.com/eitsupi/prqlr/pull/161>
> > So I was very surprised to find out after I returned from my trip that
> this was archived.
> >
> > The CRAN page says "Archived on 2023-08-19 for policy violation. " but I
> don't know what exactly was the problem.
> > I have no idea what more to fix as I believe I have solved all the
> problems when I submitted 0.5.0.
> >
> > Is there any way to know what exactly was the problem?
> > (I thought I sent an e-mail to CRAN 5 days ago but have not yet received
> an answer, so I decided to ask my question on this mailing list, thinking
> that there is a possibility that there will be no answer to my e-mail,
> although I may have to wait a few weeks for an answer. My apologies if this
> idea is incorrect.)
> >
> > Best,
> > Tatsuya
> >
> > __
> > R-package-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-package-devel
> >
>
> __
> R-package-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>

[[alternative HTML version deleted]]

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] Feedback on "Using Rust in CRAN packages"

[Hiroaki Yutani]

I just found the policy is updated and I now understand why GitHub matters
in your opinion. Thanks for the clarification, I forgot this fact.

>  CRAN does not regard github.com (which hosts the index of crates.io) as
sufficiently reliable.

The good news is that, as of Rust 1.68, Cargo supports the "sparse" index
protocol [1][2]. In this case, the index is hosted at
https://index.crates.io/, crates.io's own infrastructure. So, if I
understand correctly, when all the CRAN servers have Cargo >=1.68
installed, CRAN is ready to believe crates.io is reliable?

Note that, at the time of writing this, the version on Debian testing is
still 1.66 [3] and it's not updated very frequently (about once a year?),
so it probably takes a while before the day.

Best,
Yutani


[1]:
https://blog.rust-lang.org/2023/03/09/Rust-1.68.0.html#cargos-sparse-protocol
[2]:
https://blog.rust-lang.org/inside-rust/2023/01/30/cargo-sparse-protocol.html
[3]: https://packages.debian.org/testing/cargo (it seems 0.66 means 1.66)

2023年7月14日(金) 9:58 Hiroaki Yutani :

> Simin,
>
> Sorry that my question was not clear. Let me clarify.
>
> I think we all agree that "cargo vendor" is the primary option. Since
> downloading without explicit permission is not allowed on CRAN in general,
> it's reasonable. I'm happy that the instructions will describe it clearly.
>
> But, some R packages have too large dependencies to bundle. In this case,
> downloading can be allowed with "the explicit permission of the CRAN team,"
> if I understand correctly. For this, I think Cargo's downloading mechanism
> satisfy this requirement if (1) all the dependencies are from crates.io
> and (2) Cargo.lock exists:
>
> > download a specific version from a secure site and check that the
> download is the expected code by some sort of checksum
>
> Because Cargo downloads specific versions recorded in Cargo.lock, verifies
> the checksums, and crates.io is the "secure site" that we can rely on as
> Hadley wrote.
>
> My question is, does CRAN allow Cargo to download the dependency sources
> on CRAN? The policy says:
>
> > So downloading of Rust ‘crates’ will in future require the explicit
> permission of the CRAN team
>
> To my eyes, this implies
>
> - CRAN currently allows Cargo's downloading of dependency Rust crates even
> without the permission
> - CRAN will keep allowing Cargo's downloading if the package author asks
> the permission
>
> And, if CRAN doesn't allow it, I (and probably many Rust users) would like
> to know why. As I described above, it should satisfy the requirement.
>
> >  please don't cross-post
>
> Sorry.
>
> > I thought cargo build --offline is not needed if the dependencies are
> already vendored?
>
> Yes, you are right. --offline is not needed if vendering is properly
> configured. But, this probably means you have to review the build
> configurations in .cargo/config.toml or so, so I just thought it would be
> easier for you to check if --offline is specified to the command. This
> seems a bit off-topic, so please ignore.
>
> Best,
> Yutani
>
>
> 2023年7月14日(金) 9:06 Simon Urbanek :
>
>>
>>
>> > On Jul 14, 2023, at 11:19 AM, Hadley Wickham 
>> wrote:
>> >
>> >>> If CRAN cannot trust even the official one of Rust, why does CRAN
>> have Rust at all?
>> >>>
>> >>
>> >> I don't see the connection - if you downloaded something in the past
>> it doesn't mean you will be able to do so in the future. And CRAN has Rust
>> because it sounded like a good idea to allow packages to use it, but I can
>> see that it opened a can of worms that we trying to tame here.
>> >
>> > Can you give a bit more detail about your concerns here? Obviously
>> > crates.io isn't some random site on the internet, it's the official
>> > repository of the Rust language, supported by the corresponding
>> > foundation for the language. To me that makes it feel very much like
>> > CRAN, where we can assume if you downloaded something in the past, you
>> > can download something in the future.
>> >
>>
>> I was just responding to Yutani's question why we downloaded the Rust
>> compilers on CRAN at all. This has really nothing to do with the previous
>> discussion which is why I did say "I don't see the connection". Also I
>> wasn't talking about crates.io anywhere in my responses in this thread.
>> The only thing I wanted to discuss here was that I think the existing Rust
>> model  ("vendor" into the package sources) seems like a good one to apply
>> to Go, but that got somehow hijacked...
>>
>> Cheers,
>> Simon
>>
>>

[[alternative HTML version deleted]]

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] Feedback on "Using Rust in CRAN packages"

[Hiroaki Yutani]

Simin,

Sorry that my question was not clear. Let me clarify.

I think we all agree that "cargo vendor" is the primary option. Since
downloading without explicit permission is not allowed on CRAN in general,
it's reasonable. I'm happy that the instructions will describe it clearly.

But, some R packages have too large dependencies to bundle. In this case,
downloading can be allowed with "the explicit permission of the CRAN team,"
if I understand correctly. For this, I think Cargo's downloading mechanism
satisfy this requirement if (1) all the dependencies are from crates.io and
(2) Cargo.lock exists:

> download a specific version from a secure site and check that the
download is the expected code by some sort of checksum

Because Cargo downloads specific versions recorded in Cargo.lock, verifies
the checksums, and crates.io is the "secure site" that we can rely on as
Hadley wrote.

My question is, does CRAN allow Cargo to download the dependency sources on
CRAN? The policy says:

> So downloading of Rust ‘crates’ will in future require the explicit
permission of the CRAN team

To my eyes, this implies

- CRAN currently allows Cargo's downloading of dependency Rust crates even
without the permission
- CRAN will keep allowing Cargo's downloading if the package author asks
the permission

And, if CRAN doesn't allow it, I (and probably many Rust users) would like
to know why. As I described above, it should satisfy the requirement.

>  please don't cross-post

Sorry.

> I thought cargo build --offline is not needed if the dependencies are
already vendored?

Yes, you are right. --offline is not needed if vendering is properly
configured. But, this probably means you have to review the build
configurations in .cargo/config.toml or so, so I just thought it would be
easier for you to check if --offline is specified to the command. This
seems a bit off-topic, so please ignore.

Best,
Yutani


2023年7月14日(金) 9:06 Simon Urbanek :

>
>
> > On Jul 14, 2023, at 11:19 AM, Hadley Wickham 
> wrote:
> >
> >>> If CRAN cannot trust even the official one of Rust, why does CRAN have
> Rust at all?
> >>>
> >>
> >> I don't see the connection - if you downloaded something in the past it
> doesn't mean you will be able to do so in the future. And CRAN has Rust
> because it sounded like a good idea to allow packages to use it, but I can
> see that it opened a can of worms that we trying to tame here.
> >
> > Can you give a bit more detail about your concerns here? Obviously
> > crates.io isn't some random site on the internet, it's the official
> > repository of the Rust language, supported by the corresponding
> > foundation for the language. To me that makes it feel very much like
> > CRAN, where we can assume if you downloaded something in the past, you
> > can download something in the future.
> >
>
> I was just responding to Yutani's question why we downloaded the Rust
> compilers on CRAN at all. This has really nothing to do with the previous
> discussion which is why I did say "I don't see the connection". Also I
> wasn't talking about crates.io anywhere in my responses in this thread.
> The only thing I wanted to discuss here was that I think the existing Rust
> model  ("vendor" into the package sources) seems like a good one to apply
> to Go, but that got somehow hijacked...
>
> Cheers,
> Simon
>
>

[[alternative HTML version deleted]]

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] Feedback on "Using Rust in CRAN packages"

[Hiroaki Yutani]

Thank you for the correction. I see.

Best,
Yutani

2023年7月13日(木) 16:08 Tomas Kalibera :

>
> On 7/13/23 05:08, Hiroaki Yutani wrote:
> > I actually use cargo vendor.
> >
> >
> https://github.com/yutannihilation/string2path/blob/main/src/rust/vendor.sh
> >
> > One thing to note is that, prior to R 4.3.0, the vendored directories hit
> > the Windows' path limit so I had to put them into a TAR file. I haven't
> > tested on R 4.3.0, but probably this problem is solved by this
> improvement.
> > So, if you target only R >= 4.3, you can just cargo vendor.
> >
> >
> https://blog.r-project.org/2023/03/07/path-length-limit-on-windows/index.html
>
> I wouldn't rely on that long paths on Windows are supported even in R >=
> 4.3, because it requires at least Windows 10 1607, and it needs to be
> enabled system-wide in Windows - so, users/admins have to do that, and
> it impacts also other applications. The blog post has more details and
> recommendations.
>
> Best
> Tomas
>
> >
> > Best,
> > Yutani
> >
> > 2023年7月13日(木) 11:50 Kevin Ushey :
> >
> >> Package authors could use 'cargo vendor' to include Rust crate sources
> >> directly in their source R packages. Would that be acceptable?
> >>
> >> Presumedly, the vendored sources would be built using the versions
> >> specified in an accompanying Cargo.lock as well.
> >>
> >> https://doc.rust-lang.org/cargo/commands/cargo-vendor.html
> >>
> >>
> >> On Wed, Jul 12, 2023, 7:35 PM Simon Urbanek <
> simon.urba...@r-project.org>
> >> wrote:
> >>
> >>> Yutani,
> >>>
> >>> I'm not quite sure your reading fully matches the intent of the policy.
> >>> Cargo.lock is not sufficient, it is expected that the package will
> provide
> >>> *all* the sources, it is not expected to use cargo to resolve them from
> >>> random (possibly inaccessible) places. So the package author is
> expected to
> >>> either include the sources in the package *or* (if prohibitive due to
> >>> extreme size) have a release tar ball available at a fixed, secure,
> >>> reliable location (I was recommending Zenodo.org for that reason -
> GitHub
> >>> is neither fixed nor reliable by definition).
> >>>
> >>> Based on that, I'm not sure I fully understand the scope of your
> proposal
> >>> for improvement. Carlo.lock is certainly the first step that the
> package
> >>> author should take in creating the distribution tar ball so you can
> fix the
> >>> versions, but it is not sufficient as the next step involves
> collecting the
> >>> related sources. We don't want R users to be involved in that can of
> worms
> >>> (especially since the lock file itself provides no guarantees of
> >>> accessibility of the components and we don't want to have to manually
> >>> inspect it), the package should be ready to be used which is why it
> has to
> >>> do that step first. Does that explain the intent better? (In general,
> the
> >>> downloading at install time is actually a problem, because it's not
> >>> uncommon to use R in environments that have no Internet access, but the
> >>> download is a concession for extreme cases where the tar balls may be
> too
> >>> big to make it part of the package, but it's yet another can of
> worms...).
> >>>
> >>> Cheers,
> >>> Simon
> >>>
> >>>
> >>>
> >>>> On 13/07/2023, at 12:37 PM, Hiroaki Yutani 
> >>> wrote:
> >>>> Hi,
> >>>>
> >>>> I'm glad to see CRAN now has its official policy about Rust [1]!
> >>>> It seems it probably needs some feedback from those who are familiar
> >>> with
> >>>> the Rust workflow. I'm not an expert, but let me leave some quick
> >>> feedback.
> >>>> This email is sent to the R-package-devel mailing list as well as to
> >>> cran@~
> >>>> so that we can publicly discuss.
> >>>>
> >>>> It seems most of the concern is about how to make the build
> >>> deterministic.
> >>>> In this regard, the policy should encourage including "Cargo.lock"
> file
> >>>> [2]. Cargo.lock is created on the first compile, and the resolved
> >>> versions
> >>>> of dependencies are recorded. As long as this file exists, the
> >>> dependenc

Re: [R-pkg-devel] Feedback on "Using Rust in CRAN packages"

[Hiroaki Yutani]

Hi Simon,

Thanks for the response. I thought

> download a specific version from a secure site and check that the
download is the expected code by some sort of checksum

refers to the usual process that's done by Cargo automatically. If it's
not, I think the policy should have a clear explanation. It seems it's not
only me who wondered why this policy doesn't mention Cargo.lock at all.

> it is not expected to use cargo to resolve them from random (possibly
inaccessible) places

Yes, I agree with you. So, I suggested the possibility of forbidding the
Git dependency. Or, do you call crates.io, Rust's official repository,
"random places"? If CRAN cannot trust even the official one of Rust, why
does CRAN have Rust at all?

That said, I agree with your concern about downloading via the Internet in
general. Downloading is one of the common sources of failure. If you want
to prevent cargo from downloading any source files, you can enforce adding
--offline option to "cargo build". While the package author might feel
unhappy, I think this would make your intent a bit clearer.

Best,
Yutani


2023年7月13日(木) 11:34 Simon Urbanek :

> Yutani,
>
> I'm not quite sure your reading fully matches the intent of the policy.
> Cargo.lock is not sufficient, it is expected that the package will provide
> *all* the sources, it is not expected to use cargo to resolve them from
> random (possibly inaccessible) places. So the package author is expected to
> either include the sources in the package *or* (if prohibitive due to
> extreme size) have a release tar ball available at a fixed, secure,
> reliable location (I was recommending Zenodo.org for that reason - GitHub
> is neither fixed nor reliable by definition).
>
> Based on that, I'm not sure I fully understand the scope of your proposal
> for improvement. Carlo.lock is certainly the first step that the package
> author should take in creating the distribution tar ball so you can fix the
> versions, but it is not sufficient as the next step involves collecting the
> related sources. We don't want R users to be involved in that can of worms
> (especially since the lock file itself provides no guarantees of
> accessibility of the components and we don't want to have to manually
> inspect it), the package should be ready to be used which is why it has to
> do that step first. Does that explain the intent better? (In general, the
> downloading at install time is actually a problem, because it's not
> uncommon to use R in environments that have no Internet access, but the
> download is a concession for extreme cases where the tar balls may be too
> big to make it part of the package, but it's yet another can of worms...).
>
> Cheers,
> Simon
>
>
>
> > On 13/07/2023, at 12:37 PM, Hiroaki Yutani  wrote:
> >
> > Hi,
> >
> > I'm glad to see CRAN now has its official policy about Rust [1]!
> > It seems it probably needs some feedback from those who are familiar with
> > the Rust workflow. I'm not an expert, but let me leave some quick
> feedback.
> > This email is sent to the R-package-devel mailing list as well as to
> cran@~
> > so that we can publicly discuss.
> >
> > It seems most of the concern is about how to make the build
> deterministic.
> > In this regard, the policy should encourage including "Cargo.lock" file
> > [2]. Cargo.lock is created on the first compile, and the resolved
> versions
> > of dependencies are recorded. As long as this file exists, the dependency
> > versions are locked to the ones in this file, except when the package
> > author explicitly updates the versions.
> >
> > Cargo.lock also records the SHA256 checksums of the crates if they are
> from
> > crates.io, Rust's official crate registry. If the checksums don't match,
> > the build will fail with the following message:
> >
> >error: checksum for `foo v0.1.2` changed between lock files
> >
> >this could be indicative of a few possible errors:
> >
> >* the lock file is corrupt
> >* a replacement source in use (e.g., a mirror) returned a
> different
> > checksum
> >* the source itself may be corrupt in one way or another
> >
> >unable to verify that `foo v0.1.2` is the same as when the lockfile
> was
> > generated
> >
> > For dependencies from Git repositories, Cargo.lock records the commit
> > hashes. So, the version of the source code (not the version of the crate)
> > is uniquely determined. That said, unlike cargo.io, it's possible that
> the
> > commit or the Git repository itself has disappeared at the time of
> > building, which makes the build fail. So, it might be reason

Re: [R-pkg-devel] Feedback on "Using Rust in CRAN packages"

[Hiroaki Yutani]

I actually use cargo vendor.

https://github.com/yutannihilation/string2path/blob/main/src/rust/vendor.sh

One thing to note is that, prior to R 4.3.0, the vendored directories hit
the Windows' path limit so I had to put them into a TAR file. I haven't
tested on R 4.3.0, but probably this problem is solved by this improvement.
So, if you target only R >= 4.3, you can just cargo vendor.

https://blog.r-project.org/2023/03/07/path-length-limit-on-windows/index.html

Best,
Yutani

2023年7月13日(木) 11:50 Kevin Ushey :

> Package authors could use 'cargo vendor' to include Rust crate sources
> directly in their source R packages. Would that be acceptable?
>
> Presumedly, the vendored sources would be built using the versions
> specified in an accompanying Cargo.lock as well.
>
> https://doc.rust-lang.org/cargo/commands/cargo-vendor.html
>
>
> On Wed, Jul 12, 2023, 7:35 PM Simon Urbanek 
> wrote:
>
>> Yutani,
>>
>> I'm not quite sure your reading fully matches the intent of the policy.
>> Cargo.lock is not sufficient, it is expected that the package will provide
>> *all* the sources, it is not expected to use cargo to resolve them from
>> random (possibly inaccessible) places. So the package author is expected to
>> either include the sources in the package *or* (if prohibitive due to
>> extreme size) have a release tar ball available at a fixed, secure,
>> reliable location (I was recommending Zenodo.org for that reason - GitHub
>> is neither fixed nor reliable by definition).
>>
>> Based on that, I'm not sure I fully understand the scope of your proposal
>> for improvement. Carlo.lock is certainly the first step that the package
>> author should take in creating the distribution tar ball so you can fix the
>> versions, but it is not sufficient as the next step involves collecting the
>> related sources. We don't want R users to be involved in that can of worms
>> (especially since the lock file itself provides no guarantees of
>> accessibility of the components and we don't want to have to manually
>> inspect it), the package should be ready to be used which is why it has to
>> do that step first. Does that explain the intent better? (In general, the
>> downloading at install time is actually a problem, because it's not
>> uncommon to use R in environments that have no Internet access, but the
>> download is a concession for extreme cases where the tar balls may be too
>> big to make it part of the package, but it's yet another can of worms...).
>>
>> Cheers,
>> Simon
>>
>>
>>
>> > On 13/07/2023, at 12:37 PM, Hiroaki Yutani 
>> wrote:
>> >
>> > Hi,
>> >
>> > I'm glad to see CRAN now has its official policy about Rust [1]!
>> > It seems it probably needs some feedback from those who are familiar
>> with
>> > the Rust workflow. I'm not an expert, but let me leave some quick
>> feedback.
>> > This email is sent to the R-package-devel mailing list as well as to
>> cran@~
>> > so that we can publicly discuss.
>> >
>> > It seems most of the concern is about how to make the build
>> deterministic.
>> > In this regard, the policy should encourage including "Cargo.lock" file
>> > [2]. Cargo.lock is created on the first compile, and the resolved
>> versions
>> > of dependencies are recorded. As long as this file exists, the
>> dependency
>> > versions are locked to the ones in this file, except when the package
>> > author explicitly updates the versions.
>> >
>> > Cargo.lock also records the SHA256 checksums of the crates if they are
>> from
>> > crates.io, Rust's official crate registry. If the checksums don't
>> match,
>> > the build will fail with the following message:
>> >
>> >error: checksum for `foo v0.1.2` changed between lock files
>> >
>> >this could be indicative of a few possible errors:
>> >
>> >* the lock file is corrupt
>> >* a replacement source in use (e.g., a mirror) returned a
>> different
>> > checksum
>> >* the source itself may be corrupt in one way or another
>> >
>> >unable to verify that `foo v0.1.2` is the same as when the lockfile
>> was
>> > generated
>> >
>> > For dependencies from Git repositories, Cargo.lock records the commit
>> > hashes. So, the version of the source code (not the version of the
>> crate)
>> > is uniquely determined. That said, unlike cargo.io, it's possible that
>> the
>> > commit or the Git repository 

[R-pkg-devel] Feedback on "Using Rust in CRAN packages"

[Hiroaki Yutani]

Hi,

I'm glad to see CRAN now has its official policy about Rust [1]!
It seems it probably needs some feedback from those who are familiar with
the Rust workflow. I'm not an expert, but let me leave some quick feedback.
This email is sent to the R-package-devel mailing list as well as to cran@~
so that we can publicly discuss.

It seems most of the concern is about how to make the build deterministic.
In this regard, the policy should encourage including "Cargo.lock" file
[2]. Cargo.lock is created on the first compile, and the resolved versions
of dependencies are recorded. As long as this file exists, the dependency
versions are locked to the ones in this file, except when the package
author explicitly updates the versions.

Cargo.lock also records the SHA256 checksums of the crates if they are from
crates.io, Rust's official crate registry. If the checksums don't match,
the build will fail with the following message:

error: checksum for `foo v0.1.2` changed between lock files

this could be indicative of a few possible errors:

* the lock file is corrupt
* a replacement source in use (e.g., a mirror) returned a different
checksum
* the source itself may be corrupt in one way or another

unable to verify that `foo v0.1.2` is the same as when the lockfile was
generated

For dependencies from Git repositories, Cargo.lock records the commit
hashes. So, the version of the source code (not the version of the crate)
is uniquely determined. That said, unlike cargo.io, it's possible that the
commit or the Git repository itself has disappeared at the time of
building, which makes the build fail. So, it might be reasonable the CRAN
policy prohibits the use of Git dependency unless the source code is
bundled. I have no strong opinion here.

Accordingly, I believe this sentence

> In practice maintainers have found it nigh-impossible to meet these
conditions whilst downloading as they have too little control.

is not quite true. More specifically, these things

> The standard way to download a Rust ‘crate’ is by its version number, and
these have been changed without changing their number.
> Downloading a ‘crate’ normally entails downloading its dependencies, and
that is done without fixing their version numbers

won't happen if the R package does include Cargo.lock because

- if the crate is from crates.io, "the version can never be overwritten,
and the code cannot be deleted" there [3]
- if the crate is from a Git repository, the commit hash is unique in its
nature. The version of the crate might be the same between commits, but a
git dependency is specified by the commit hash, not the version of the
crate.

I'm keen to know what problems the CRAN maintainers have experienced that
Cargo.lock cannot solve. I hope we can help somehow to improve the policy.

Best,
Yutani

[1]: https://cran.r-project.org/web/packages/using_rust.html
[2]: https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
[3]: https://doc.rust-lang.org/cargo/reference/publishing.html

[[alternative HTML version deleted]]

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] Is it a wrong assumption that ${R_HOME}/lib always exists?

[Hiroaki Yutani]

Thanks so much for the quick response. It answered everything!
It was my mistake that I didn't consider these types of installations.
I'll fix my package as soon as I can.

Best,
Yutani

2023年7月10日(月) 17:51 Martin Maechler :

> >>>>> Hiroaki Yutani
> >>>>> on Mon, 10 Jul 2023 17:24:37 +0900 writes:
>
> [...]
>
> Short answer to your questsion (from the 'Subject') is :NO !
> For an example, see below:
>
> > libR-sys assumes the path to R's shared libraries is
> > `${R_HOME}/lib` on Unix-alike platforms.
>
> In some configurations, R does not need *any* shared libraries,
> and there,  $R_HOME/lib  does *not* exist.
>
> E.g., I have one of my R-devel versions installed (on Fedora
> Linux) with
>
> ../R/configure --with-blas=-lflexiblas
>
> using the nice, currently Fedora/Redhat-only "flexiblas"
> approach, with which I can nicely switch the versions of BLAS
> and Lapack libraries that R works with from within R.
>
> As that version of R is not "shared", i.e., no libR.so , *and*
> gets both its BLAS and Lapack libraries from "external" (not
> from R), there's no need for a ./lib/  and so none is created.
>
> Martin Maechler
> ETH Zurich  and  R Core Team
>
>
> > Is it possible
> > that this path doesn't exist on the MKL server?
>
> > Actually, it compiles fine on the other Linux platforms,
> > so I'm wondering what's different there from the other
> > servers.
>
>
> > Best, Yutani
> >   [[alternative HTML version deleted]]
>
>

[[alternative HTML version deleted]]

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

[R-pkg-devel] Is it a wrong assumption that ${R_HOME}/lib always exists?

[Hiroaki Yutani]

Hi,

My package, string2path, using Rust fails on the CRAN check of MKL [1],
with an error that seems irrelevant to MKL. The error says:

>   thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value:
Os { code: 2, kind: NotFound, message: "No such file or directory" }',
/data/gannet/ripley/R/packages/tests-MKL/string2path/src/rust/vendor/libR-sys/build.rs:637
:40

The line 637 of build.rs of libR-sys crate is here [2]:

r_paths.library.canonicalize().unwrap().display()

Compared to the previous version, which didn't experience such a failure,
.canonicalize() was added, and it seems this is the cause. This method
normalizes the path expression like R's normalizePath(), and it fails
because the path specified didn't exist.

libR-sys assumes the path to R's shared libraries is `${R_HOME}/lib` on
Unix-alike platforms. Is it possible that this path doesn't exist on the
MKL server? Actually, it compiles fine on the other Linux platforms, so I'm
wondering what's different there from the other servers.

Best,
Yutani

[1]: https://cran.r-project.org/web/checks/check_results_string2path.html
[2]: https://github.com/extendr/libR-sys/blob/v0.5.0/build.rs#L637C1-L637C58

[[alternative HTML version deleted]]

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] About the CRAN policy on downloading pre-compiled binary

[Hiroaki Yutani]

Hi,

Thank you so much Tomas for helping me to figure out the things. Let me
share some updates (some of them are what you've seen already in the reply
to the CRAN maintainer) in case this might be eventually useful to those
who are curious about using Rust on CRAN.

First of all, let me correct my misunderstanding. I thought what violates
the CRAN policy was that my package downloads a binary, but in actual, it
seems my fault was

1. if the compiler exists, my package downloads the **sources** of the
dependency, whose authorships are not described.
2. if the compiler doesn't exist, my package downloads the pre-compiled
binary **without the agreement of the CRAN team**

For point 2, as suggested in the previous comment,

> at least check whether the compiler is present (during package
installation), and use it if it is.

my package first checks if Cargo, Rust's compiler toolchain and package
manager, is installed (more specifically, whether the `cargo` command can
be found on PATH), and falls back to downloading the pre-compiled binary
only when Cargo is not available. So, I believe this can be considered as
"a last resort". Again, it was clearly my fault that I didn't ask the CRAN
maintainers for permission on my first submission.

However, it seems Cargo is already available on the following CRAN machines:

- Linux machines
- Windows Server 2022 (at least on winbuilder. Thanks David B. Dahl for
the information!)
- M1 macOS machine (guessed from the comment from the CRAN maintainer)

If I remember correctly, the Windows machine didn't have Cargo installed
when I submitted my package to CRAN. I'd really appreciate the CRAN
maintainers' effort to improve the infrastructure, and I hope Cargo will be
available on Intel macOS machines in the not so distant future so that I no
longer need to provide the pre-compiled binary (I'm not sure if I can have
the same hope on the Windows 2008, which might be a bit too old to install
some latest-ish version of Cargo, though [2]).


For point 1, it's Cargo that downloads the sources. It does download the
fixed versions recorded in the bundled "Cargo.lock" file, and verifies the
checksum. While Cargo provides the vendoring mechanism [1], I thought it
was not realistic to include the sources because my dependency was so huge
(132MB). But, I was wrong at two points.

First, no matter whether to download or to bundle the sources, the CRAN
Policy requires to describe the copyright information of "all components of
the package," which should refer to all the individual Rust crate in the
context of my package.

>  The ownership of copyright and intellectual property rights of all
components of the package must be clear and unambiguous (including from the
authors specification in the DESCRIPTION file). Where code is copied (or
derived) from the work of others (including from R itself), care must be
taken that any copyright/license statements are preserved and authorship is
not misrepresented.

Second, it was possible to reduce the size to the extent where I can bundle
it. I could tweak some libraries to drop the heavy dependency, and compress
the source code (again, thanks David B. Dahl for the idea).

I'm yet to figure out what's the proper way to list up the authors (I'm
currently using inst/COPYRIGHTS, but I'm not fully sure at the moment), but
I'm working on tweaking my package to bundle all the sources and to clarify
the authorships of these dependencies [2]. I hope this will keep my package
on CRAN.

Best,
Yutani

[1]: https://doc.rust-lang.org/cargo/commands/cargo-vendor.html
[2]: Actually, it keeps failing on Windows Server 2008 with the
pre-compiled binary probably due to lack of some system API:
https://www.r-project.org/nosvn/R.check/r-oldrel-windows-ix86+x86_64/string2path-00check.html
[3]: https://github.com/yutannihilation/string2path/pull/35 (work in
progress)

2022年7月27日(水) 17:12 Tomas Kalibera :

>
> On 7/27/22 08:08, Tomas Kalibera wrote:
> >
> > On 7/27/22 00:30, Hiroaki Yutani wrote:
> >> Hi,
> >>
> >> Recently I got the following email from the CRAN maintainer about my
> >> package, string2path[1].
> >>
> >> However, I do ensure the binary is the pinned version and verify if the
> >> hash matches with the embedded one in the DESCRIPTION [2][3]. In case
> >> of a
> >> mismatch, the build fails. So, this mechanism should ensure that I (or
> >> anyone) cannot change the version of the binary without actually
> >> resubmitting to CRAN.
> >
> > Please see the policy cited. Ensuring that the download is of a fixed
> > version refers to the sources (which can be downloaded under the
> > conditions mentioned).
> >
> > Downloading binaries are only a last resort option and requires the
> > agreement of the CRAN team in the first place.
> >

[R-pkg-devel] About the CRAN policy on downloading pre-compiled binary

[Hiroaki Yutani]

Hi,

Recently I got the following email from the CRAN maintainer about my
package, string2path[1].

However, I do ensure the binary is the pinned version and verify if the
hash matches with the embedded one in the DESCRIPTION [2][3]. In case of a
mismatch, the build fails. So, this mechanism should ensure that I (or
anyone) cannot change the version of the binary without actually
resubmitting to CRAN.

I believe this complies with the CRAN policy (except for not clearing the
authorship and copyright). Is there anything I have to address to prove I
do "ensure that the download is of a fixed version"? Any suggestions are
welcome.

The CRAN policy stipulates
>
> "Where a package wishes to make use of a library not written solely for
> the package, the package installation should first look to see if it is
> already installed and if so is of a suitable version. In case not, it is
> desirable to include the library sources in the package and compile them
> as part of package installation. If the sources are too large, it is
> acceptable to download them as part of installation, but do ensure that
> the download is of a fixed version rather than the latest. Only as a
> last resort and with the agreement of the CRAN team should a package
> download pre-compiled software."
>
> and we have recently seen an instance of a rust-using package whose
> check output changed because what it downloaded had changed.  CRAN
> checking is not set up for that (for example, macOS checks are done once
> only for each version).
>
> Whilst investigating, the Windows' maintainers found that binary libs
> were being downloaded.  And subsequently I found that salso, string2path
> and ymd are downloading compiled code on Intel macOS.
>
> Also. make sure that the authorship and copyright of code you download
> (and hence include in the package) is clear from the DESCRIPTION file.
> as required by the CRAN policy.
>

Best,
Hiroaki Yutani

[1]: https://cran.r-project.org/package=string2path
[2]:
https://github.com/cran/string2path/blob/46020296410cd78e2021bff86cb6f17c681d13a6/DESCRIPTION#L29-L40
[3]:
https://github.com/cran/string2path/blob/46020296410cd78e2021bff86cb6f17c681d13a6/tools/configure.R#L177-L295

[[alternative HTML version deleted]]

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] R session crash on closing a graphic device on Windows

[Hiroaki Yutani]

I see, thanks. I filed here:

https://bugs.r-project.org/show_bug.cgi?id=18292

Best,
Yutani

2022年1月27日(木) 1:35 Tomas Kalibera :
>
> Hi Yutani,
>
> On 1/26/22 16:42, Hiroaki Yutani wrote:
> > Hi Tomas,
> >
> > Thanks, but, if I understand correctly, there's no room to call the
> > Rust allocator's "free" function in the case of `DevDesc`. A `DevDesc`
> > is supposed to be freed in `GEdestroyDevDesc()` when the device is
> > closed. If I free it by myself, it would cause double-free.
> >
> > So, now I'm wondering if it makes sense that R provides either an API
> > that creates a `DevDesc` instance, or one that accepts a custom
> > allocator for DevDesc. But, as I expect this is a minor use case, I'm
> > not confident enough this would be meaningful.
> > I might end up filing such a feature request on Bugzilla, but let me
> > hold off for a while.
>
> I see. If you are using the public interface and it allows you to
> (indirectly) free the devices, it should allow you also to allocate
> them. So yes, please file a bug report and please provide enough context
> there so that the report is self-sufficient.
>
> Beyond the general rule than any API allowing to allocate or free needs
> to allow both, I'd leave this to the experts on graphics in R.
>
> Thanks,
> Tomas
>
> >
> > Best,
> > Yutani
> >
> > 2022年1月27日(木) 0:21 Tomas Kalibera :
> >>
> >> On 1/26/22 15:44, Hiroaki Yutani wrote:
> >>> Hi Tomas,
> >>>
> >>> Thanks for your helpful advice. This time, it seems the cause of the
> >>> error was an allocator mismatch; I mistakenly allocated the struct on
> >>> Rust's side, which means it's allocated by Rust's allocator, but a
> >>> `DevDesc` is to be freed on R's side. The problem is solved by using
> >>> libc::calloc(), which allocates using the C runtime's allocator, and
> >>> compiling it with the same toolchain that compiles R.
> >> Hi Yutani,
> >>
> >> congratulations on tracing it down.
> >>
> >> Particularly on Windows, whenever a DLL (or any API) is providing a
> >> function to allocate anything, it should provide also a function to free
> >> it, and only that function should be used to do so, even if it is just a
> >> wrapper for malloc() etc. So I would recommend following that, there
> >> should be a Rust allocator's "free" function which you could then call
> >> from R.
> >>
> >> Best
> >> Tomas
> >>
> >>> I also saw some errors when it relates to GC, so it might be some
> >>> PROTECT issue. Thanks for the hint.
> >>>
> >>> During debugging, I learned a lot about how to build R with DEBUG=T
> >>> and use gdb, and it really helped me. I'm yet to unlock the power of
> >>> WinDBG, but I will try next time...
> >>>
> >>> Best,
> >>> Yutani
> >>>
> >>> 2022年1月26日(水) 23:20 Tomas Kalibera :
> >>>
> >>>> Hi Yutani,
> >>>>
> >>>> if you haven't done that already, I would recommend building R with
> >>>> debug symbols (DEBUG=T, so that make file don't strip them) and with -O0
> >>>> (no optimizations), so that the debug symbols are more accurate. Without
> >>>> that, the stack traces can be very misleading. You might try both with
> >>>> windbg and gdb, sometimes one of them provides an extra hint. Ideally
> >>>> you would also build the involved package(s) the same way.
> >>>>
> >>>> Then, it is important to check where the free() is called from, whether
> >>>> it is directly or from the GC. In both cases (but more likely in the
> >>>> latter), it could be caused by somewhat unrelated memory corruption,
> >>>> which may be hard to find - e.g. possibly a PROTECT error. Running with
> >>>> gctorture() might help, if gctorture() changes where the crash happens,
> >>>> it is more likely a somewhat unrelated memory corruption.
> >>>>
> >>>> If it were a double-free or similar allocation error inside R itself (or
> >>>> some of the involved packages), it would be easy to find with a debugger.
> >>>>
> >>>> If debugging this way does not help, you can try narrowing down the
> >>>> example, while preserving the crash. That may make debugging easier, and
> >>>> if you eventually get to a point that you have a reproducible example
> >&g

Re: [R-pkg-devel] R session crash on closing a graphic device on Windows

[Hiroaki Yutani]

Hi Tomas,

Thanks, but, if I understand correctly, there's no room to call the
Rust allocator's "free" function in the case of `DevDesc`. A `DevDesc`
is supposed to be freed in `GEdestroyDevDesc()` when the device is
closed. If I free it by myself, it would cause double-free.

So, now I'm wondering if it makes sense that R provides either an API
that creates a `DevDesc` instance, or one that accepts a custom
allocator for DevDesc. But, as I expect this is a minor use case, I'm
not confident enough this would be meaningful.
I might end up filing such a feature request on Bugzilla, but let me
hold off for a while.

Best,
Yutani

2022年1月27日(木) 0:21 Tomas Kalibera :
>
>
> On 1/26/22 15:44, Hiroaki Yutani wrote:
> > Hi Tomas,
> >
> > Thanks for your helpful advice. This time, it seems the cause of the
> > error was an allocator mismatch; I mistakenly allocated the struct on
> > Rust's side, which means it's allocated by Rust's allocator, but a
> > `DevDesc` is to be freed on R's side. The problem is solved by using
> > libc::calloc(), which allocates using the C runtime's allocator, and
> > compiling it with the same toolchain that compiles R.
>
> Hi Yutani,
>
> congratulations on tracing it down.
>
> Particularly on Windows, whenever a DLL (or any API) is providing a
> function to allocate anything, it should provide also a function to free
> it, and only that function should be used to do so, even if it is just a
> wrapper for malloc() etc. So I would recommend following that, there
> should be a Rust allocator's "free" function which you could then call
> from R.
>
> Best
> Tomas
>
> >
> > I also saw some errors when it relates to GC, so it might be some
> > PROTECT issue. Thanks for the hint.
> >
> > During debugging, I learned a lot about how to build R with DEBUG=T
> > and use gdb, and it really helped me. I'm yet to unlock the power of
> > WinDBG, but I will try next time...
> >
> > Best,
> > Yutani
> >
> > 2022年1月26日(水) 23:20 Tomas Kalibera :
> >
> >> Hi Yutani,
> >>
> >> if you haven't done that already, I would recommend building R with
> >> debug symbols (DEBUG=T, so that make file don't strip them) and with -O0
> >> (no optimizations), so that the debug symbols are more accurate. Without
> >> that, the stack traces can be very misleading. You might try both with
> >> windbg and gdb, sometimes one of them provides an extra hint. Ideally
> >> you would also build the involved package(s) the same way.
> >>
> >> Then, it is important to check where the free() is called from, whether
> >> it is directly or from the GC. In both cases (but more likely in the
> >> latter), it could be caused by somewhat unrelated memory corruption,
> >> which may be hard to find - e.g. possibly a PROTECT error. Running with
> >> gctorture() might help, if gctorture() changes where the crash happens,
> >> it is more likely a somewhat unrelated memory corruption.
> >>
> >> If it were a double-free or similar allocation error inside R itself (or
> >> some of the involved packages), it would be easy to find with a debugger.
> >>
> >> If debugging this way does not help, you can try narrowing down the
> >> example, while preserving the crash. That may make debugging easier, and
> >> if you eventually get to a point that you have a reproducible example
> >> involving only base R and base packages, you know it is a bug in R, and
> >> can submit that in a bug report for others to debug.
> >>
> >> Best
> >> Tomas
> >>
> >> On 1/22/22 10:50, Hiroaki Yutani wrote:
> >>> Hi,
> >>>
> >>> I'm trying to create a Rust library that can implement an R graphic
> >>> device[1], but the R session crashes on `dev.off()` on Windows with R
> >>> 4.1.2. Strangely, it works without errors on Linux, on macOS, and even
> >>> on Windows with R-devel.
> >>>
> >>> Looking at the stack trace below by WinDbg, the problem is probably
> >>> that either of the two free()s in GEdestroyDevDesc() tries to free
> >>> some memory that was already freed (I'm a very beginner of this kind
> >>> of debugging, so I might be wrong).
> >>>
> >>>   # Child-SP  RetAddr   Call Site
> >>>   00 `0441cb70 7ffb`3df0be63 
> >>> ntdll!RtlReportFatalFailure+0x9
> >>>   01 `0441cbc0 7ffb`3df14c82
> >>> ntdll!RtlReportCriticalFailure+0x97
> >>>   ...snip.

[R-pkg-devel] R session crash on closing a graphic device on Windows

[Hiroaki Yutani]

Hi,

I'm trying to create a Rust library that can implement an R graphic
device[1], but the R session crashes on `dev.off()` on Windows with R
4.1.2. Strangely, it works without errors on Linux, on macOS, and even
on Windows with R-devel.

Looking at the stack trace below by WinDbg, the problem is probably
that either of the two free()s in GEdestroyDevDesc() tries to free
some memory that was already freed (I'm a very beginner of this kind
of debugging, so I might be wrong).

# Child-SP  RetAddr   Call Site
00 `0441cb70 7ffb`3df0be63 ntdll!RtlReportFatalFailure+0x9
01 `0441cbc0 7ffb`3df14c82
ntdll!RtlReportCriticalFailure+0x97
...snip...
08 `0441cfc0 7ffb`3c30c6ac ntdll!RtlFreeHeap+0x51
09 `0441d000 `6c7bcf99 msvcrt!free+0x1c
0a `0441d030 `6c79e7de R!GEdestroyDevDesc+0x59
0b `0441d080 `6fc828e9 R!GEcurrentDevice+0x37e
0c `0441d0f0 `6c7a15fa grDevices!devoff+0x59
...snip...

But, I found no difference in the related code (around devoff() and
GEdestroyDevDesc()) between R 4.1.2 and R-devel. I know there are a
lot of feature additions in R-devel, but I don't think it affects
here. Is there anyone who suffered from similar crashes? Am I missing
something?

I would appreciate any advice.

Best,
Yutani

[1]: https://github.com/extendr/extendr/pull/360

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] New CRAN checks on r-devel-windows-x86_64-new-UL and the installed fonts

[Hiroaki Yutani]

Yes, it's my fault that I didn't consider the case when no fonts are
available. I'll improve the code until the next submission to CRAN.
Thanks for your advice!

Best,
Hiroaki Yutani

2021年12月17日(金) 1:40 Tomas Kalibera :
>
>
> On 12/16/21 5:16 PM, Hiroaki Yutani wrote:
> > Thanks for the details and the suggestions. My package uses
> > systemfonts package for illustration purposes only in the examples, so
> > I'm not that desperate to find the root cause this time. I'll try
> > using winbuilder in case I need to.
>
> I see. Maybe best then making the example more robust when no fonts are
> found, on different platforms.
>
> Best
> Tomas
>
> >
> > Best,
> > Hiroaki Yutani
> >
> > 2021年12月17日(金) 0:52 Tomas Kalibera :
> >
> >>
> >> On 12/16/21 4:17 PM, Hiroaki Yutani wrote:
> >>>> This would be an empty character vector on my Alpine Linux server as
> >>>> well.
> >>> I see, thanks for the information. Sorry for my lack of consideration on 
> >>> this.
> >>>
> >>>> So there are 127 *.ttf files installed, but systemfonts::system_fonts()
> >>>> does not find any of these.
> >>> Thanks for investigating quickly! Then, it seems I should wait for the
> >>> problem to be solved on systemfonts' side. I'm curious what's the
> >>> difference between r-devel-windows-x86_64-new-TK, on which the check
> >>> doesn't fail, by the way.
> >> It is also Windows Server 2022 with GUI, a quite fresh installation. The
> >> checks run in a docker container (also WS2022,
> >> mcr.microsoft.com/windows/server:ltsc2022, without any manually
> >> installed fonts).
> >>
> >> Actually there is no manually installed software there, all that is
> >> installed is (in this order):
> >>
> >> https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt/r/setup_miktex_standalone.ps1
> >> https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/r/setup.ps1
> >> https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/r_packages/setup_checks.ps1
> >>
> >> plus R and R packages.
> >>
> >> I assume you can reproduce on Winbuilder, and so perhaps you could
> >> create a version of your package with a lot of instrumentation/print
> >> messages and submit there to find the cause? Possibly also an
> >> instrumented variant of systemfonts.
> >>
> >> Best
> >> Tomas
> >>
> >>> Best,
> >>> Hiroaki Yutani
> >>>
> >>> 2021年12月16日(木) 23:57 Uwe Ligges :
> >>>>
> >>>> On 16.12.2021 15:34, Sebastian Meyer wrote:
> >>>>> Am 16.12.21 um 15:06 schrieb Hiroaki Yutani:
> >>>>>> Hi,
> >>>>>>
> >>>>>> My package is failing on CRAN check on r-devel-windows-x86_64-new-UL.
> >>>>>>
> >>>>>> https://cran.r-project.org/web/checks/check_results_string2path.html
> >>>>>>
> >>>>>> It seems the problem is that there is no available font that meets the
> >>>>>> condition in the following code. Is it irrational to assume at least
> >>>>>> one TrueType or OpenType font is installed in the system?
> >>>>>>
> >>>>>>available_fonts <- systemfonts::system_fonts()$path
> >>>>>>
> >>>>>># string2path supports only TrueType or OpenType formats
> >>>>>>ttf_or_otf <- available_fonts[grepl("\\.(ttf|otf)$",
> >>>>>> available_fonts)]
> >>>>>>
> >>>> The machine that is running " r-devel-windows-x86_64-new-UL" is a
> >>>> freshly installed Windows Server 2022 with GUI. The standard fonts are
> >>>> available, but no extra fonts isntalled.
> >>>>
> >>>> To confirm:
> >>>>
> >>>>> table(gsub(".*\\.(.{3})$", "\\1", dir("c:/WIndows/fonts")))
> >>>>
> >>>> dat fon ini ttc ttf xml
> >>>>  1 192   1  16 127   1
> >>>>
> >>>>
> >>>> So there are 127 *.ttf files installed, but systemfonts::system_fonts()
> >>>> does not find any of these.
> >>>>
> >>>> Best,
> >>>> Uwe Ligges
> >>>>
> >>>>
> >>>>
> >>>&g

Re: [R-pkg-devel] New CRAN checks on r-devel-windows-x86_64-new-UL and the installed fonts

[Hiroaki Yutani]

Thanks for the details and the suggestions. My package uses
systemfonts package for illustration purposes only in the examples, so
I'm not that desperate to find the root cause this time. I'll try
using winbuilder in case I need to.

Best,
Hiroaki Yutani

2021年12月17日(金) 0:52 Tomas Kalibera :

>
>
> On 12/16/21 4:17 PM, Hiroaki Yutani wrote:
> >> This would be an empty character vector on my Alpine Linux server as
> >> well.
> > I see, thanks for the information. Sorry for my lack of consideration on 
> > this.
> >
> >> So there are 127 *.ttf files installed, but systemfonts::system_fonts()
> >> does not find any of these.
> > Thanks for investigating quickly! Then, it seems I should wait for the
> > problem to be solved on systemfonts' side. I'm curious what's the
> > difference between r-devel-windows-x86_64-new-TK, on which the check
> > doesn't fail, by the way.
>
> It is also Windows Server 2022 with GUI, a quite fresh installation. The
> checks run in a docker container (also WS2022,
> mcr.microsoft.com/windows/server:ltsc2022, without any manually
> installed fonts).
>
> Actually there is no manually installed software there, all that is
> installed is (in this order):
>
> https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt/r/setup_miktex_standalone.ps1
> https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/r/setup.ps1
> https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/r_packages/setup_checks.ps1
>
> plus R and R packages.
>
> I assume you can reproduce on Winbuilder, and so perhaps you could
> create a version of your package with a lot of instrumentation/print
> messages and submit there to find the cause? Possibly also an
> instrumented variant of systemfonts.
>
> Best
> Tomas
>
> >
> > Best,
> > Hiroaki Yutani
> >
> > 2021年12月16日(木) 23:57 Uwe Ligges :
> >>
> >>
> >> On 16.12.2021 15:34, Sebastian Meyer wrote:
> >>> Am 16.12.21 um 15:06 schrieb Hiroaki Yutani:
> >>>> Hi,
> >>>>
> >>>> My package is failing on CRAN check on r-devel-windows-x86_64-new-UL.
> >>>>
> >>>> https://cran.r-project.org/web/checks/check_results_string2path.html
> >>>>
> >>>> It seems the problem is that there is no available font that meets the
> >>>> condition in the following code. Is it irrational to assume at least
> >>>> one TrueType or OpenType font is installed in the system?
> >>>>
> >>>>   available_fonts <- systemfonts::system_fonts()$path
> >>>>
> >>>>   # string2path supports only TrueType or OpenType formats
> >>>>   ttf_or_otf <- available_fonts[grepl("\\.(ttf|otf)$",
> >>>> available_fonts)]
> >>>>
> >> The machine that is running " r-devel-windows-x86_64-new-UL" is a
> >> freshly installed Windows Server 2022 with GUI. The standard fonts are
> >> available, but no extra fonts isntalled.
> >>
> >> To confirm:
> >>
> >>   > table(gsub(".*\\.(.{3})$", "\\1", dir("c:/WIndows/fonts")))
> >>
> >> dat fon ini ttc ttf xml
> >> 1 192   1  16 127   1
> >>
> >>
> >> So there are 127 *.ttf files installed, but systemfonts::system_fonts()
> >> does not find any of these.
> >>
> >> Best,
> >> Uwe Ligges
> >>
> >>
> >>
> >>
> >>
> >>> This would be an empty character vector on my Alpine Linux server as
> >>> well. The system_fonts() there only contain ".pcf.gz" files from
> >>> "/usr/share/fonts/misc/".
> >>>
> >>> Note that the "systemfonts" package on which you rely currently also
> >>> fails on that CRAN check flavour for a similar reason
> >>> (https://cran.r-project.org/web/checks/check_results_systemfonts.html).
> >>> On my Alpine Linux system, from example("register_font",
> >>> package="systemfonts"):
> >>>
> >>> fonts <- system_fonts()
> >>> plain <- sample(which(!fonts$italic & fonts$weight <= 'normal'), 1)
> >>> bold <- sample(which(!fonts$italic & fonts$weight > 'normal'), 1)
> >>> italic <- sample(which(fonts$italic & fonts$weight <= 'normal'), 1)
> >>> ## Error in sample.int(length(x), size, replace, prob) :
> >>> ##   invalid first argument
> >>>

Re: [R-pkg-devel] New CRAN checks on r-devel-windows-x86_64-new-UL and the installed fonts

[Hiroaki Yutani]

> This would be an empty character vector on my Alpine Linux server as
> well.

I see, thanks for the information. Sorry for my lack of consideration on this.

> So there are 127 *.ttf files installed, but systemfonts::system_fonts()
> does not find any of these.

Thanks for investigating quickly! Then, it seems I should wait for the
problem to be solved on systemfonts' side. I'm curious what's the
difference between r-devel-windows-x86_64-new-TK, on which the check
doesn't fail, by the way.

Best,
Hiroaki Yutani

2021年12月16日(木) 23:57 Uwe Ligges :
>
>
>
> On 16.12.2021 15:34, Sebastian Meyer wrote:
> > Am 16.12.21 um 15:06 schrieb Hiroaki Yutani:
> >> Hi,
> >>
> >> My package is failing on CRAN check on r-devel-windows-x86_64-new-UL.
> >>
> >> https://cran.r-project.org/web/checks/check_results_string2path.html
> >>
> >> It seems the problem is that there is no available font that meets the
> >> condition in the following code. Is it irrational to assume at least
> >> one TrueType or OpenType font is installed in the system?
> >>
> >>  available_fonts <- systemfonts::system_fonts()$path
> >>
> >>  # string2path supports only TrueType or OpenType formats
> >>  ttf_or_otf <- available_fonts[grepl("\\.(ttf|otf)$",
> >> available_fonts)]
> >>
>
> The machine that is running " r-devel-windows-x86_64-new-UL" is a
> freshly installed Windows Server 2022 with GUI. The standard fonts are
> available, but no extra fonts isntalled.
>
> To confirm:
>
>  > table(gsub(".*\\.(.{3})$", "\\1", dir("c:/WIndows/fonts")))
>
> dat fon ini ttc ttf xml
>1 192   1  16 127   1
>
>
> So there are 127 *.ttf files installed, but systemfonts::system_fonts()
> does not find any of these.
>
> Best,
> Uwe Ligges
>
>
>
>
>
> > This would be an empty character vector on my Alpine Linux server as
> > well. The system_fonts() there only contain ".pcf.gz" files from
> > "/usr/share/fonts/misc/".
> >
> > Note that the "systemfonts" package on which you rely currently also
> > fails on that CRAN check flavour for a similar reason
> > (https://cran.r-project.org/web/checks/check_results_systemfonts.html).
> > On my Alpine Linux system, from example("register_font",
> > package="systemfonts"):
> >
> > fonts <- system_fonts()
> > plain <- sample(which(!fonts$italic & fonts$weight <= 'normal'), 1)
> > bold <- sample(which(!fonts$italic & fonts$weight > 'normal'), 1)
> > italic <- sample(which(fonts$italic & fonts$weight <= 'normal'), 1)
> > ## Error in sample.int(length(x), size, replace, prob) :
> > ##   invalid first argument
> >
> > (as there are no italic fonts).
> >
> > HTH,
> >
> >  Sebastian Meyer
> >
> >> I'm wondering if I need to release a new version to avoid this test
> >> failure. Note that, the other Windows r-devel machine
> >> (r-devel-windows-x86_64-new-TK) doesn't fail. So, it might be just
> >> that something is wrong with r-devel-windows-x86_64-new-UL.
> >>
> >> Any suggestions?
> >>
> >> Best,
> >> Hiroaki Yutani
> >>
> >> __
> >> R-package-devel@r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-package-devel
> >>
> >
> > __
> > R-package-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-package-devel

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

[R-pkg-devel] New CRAN checks on r-devel-windows-x86_64-new-UL and the installed fonts

[Hiroaki Yutani]

Hi,

My package is failing on CRAN check on r-devel-windows-x86_64-new-UL.

https://cran.r-project.org/web/checks/check_results_string2path.html

It seems the problem is that there is no available font that meets the
condition in the following code. Is it irrational to assume at least
one TrueType or OpenType font is installed in the system?

available_fonts <- systemfonts::system_fonts()$path

# string2path supports only TrueType or OpenType formats
ttf_or_otf <- available_fonts[grepl("\\.(ttf|otf)$", available_fonts)]

I'm wondering if I need to release a new version to avoid this test
failure. Note that, the other Windows r-devel machine
(r-devel-windows-x86_64-new-TK) doesn't fail. So, it might be just
that something is wrong with r-devel-windows-x86_64-new-UL.

Any suggestions?

Best,
Hiroaki Yutani

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] Internet resources and Errors

[Hiroaki Yutani]

Hi,

Regarding Solaris, while the criteria is unknown, it seems Solaris can
be excluded from the CRAN check targets when there's a good reason not
to support Solaris. My package got a failure on Solaris check and I
resubmitted with the following comment, then Solaris disappeared from
the check results [1].

> I would like to request to exclude Solaris from the build targets because 
> Solaris is not a supported platform by Rust. This should be in line with the 
> treatments of other CRAN packages that use Rust; gifski, baseflow, and salso 
> are not built on Solaris. I'm sorry that I didn't write this in the first 
> submission.

Again, I'm not sure on what criteria they decided to accept my
suggestion. It's unclear if your package is the case. Some months ago,
I asked "How can I avoid Solaris build on CRAN?" on this mailing list
[2], but I got no reply. So, probably no one knows the right answer...

Best,
Hiroaki Yutani

[1]: https://cran.r-project.org/web/checks/check_results_string2path.html
[2]: https://stat.ethz.ch/pipermail/r-package-devel/2021q3/007262.html

2021年9月24日(金) 23:50 Roy Mendelssohn - NOAA Federal via R-package-devel
:


>
> Hi All:
>
> I am getting dinged again on CRAN  (just Solaris for some reason),  and the 
> problem is how to exit if there is a failure of accessing the resource,  I 
> know it has been discussed here before,  but I just am not understanding what 
> is desired to end properly. As I have been reminded:
>
> 'Packages which use Internet resources should fail gracefully with an 
> informative message
> if the resource is not available or has changed (and not give a check warning 
> nor error).'
>
> All internet calls are wrapped in 'try()'.  If that shows an error,  I  write 
> a message to the screen about the error,  and call stop(), perhaps with a 
> further message in that call.   Somehow this does not appear to meet the 
> standard.Can someone then please tell me what I should do instead.  The 
> point is I have checked that the access to the internet resources has worked, 
>  i have seen that it hasn't,  now what should be the steps to take to exit 
> gracefully.
>
> I also want to add to what others have said about the frustrations of dealing 
> with Solaris.  I have spent a fair amount of time getting things to  work 
> with Solaris which no one uses.  In this instance the package test is only 
> failing on Solaris.   Not a good use of limited time IMO.
>
> Thanks for any advice.
>
> -Roy
>
>
>
> **
> "The contents of this message do not reflect any position of the U.S. 
> Government or NOAA."
> **
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: roy.mendelss...@noaa.gov www: https://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>
> __
> R-package-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] How can I avoid Solaris build on CRAN?

[Hiroaki Yutani]

Sorry, I pointed a wrong package. I meant,

[2]: https://cran.r-project.org/package=gifski
[3]: https://cran.r-project.org/package=baseflow

Best,
Hiroaki Yutani

2021年8月5日(木) 8:17 Hiroaki Yutani :
>
> Hi,
>
> I recently submitted my package, which needs compilation of Rust code,
> to CRAN. Now I'm seeing a CRAN check error on Solaris.
>
> https://cran.r-project.org/web/checks/check_results_string2path.html
>
> Since Solaris is not a platform supported by Rust [1], I need to avoid
> building on Solaris. I assumed specifying
>
> SystemRequirements: Cargo (rustc package manager)
>
> would effectively achieve this, considering this seems the only common
> thing among other CRAN packages [2][3] with Rust compilation. But, it
> turned out I was wrong.
>
> Are there any ways to ask CRAN to exclude Solaris?
>
> Best,
> Hiroaki Yutani
>
> [1]: https://doc.rust-lang.org/nightly/rustc/platform-support.html
> [2]: https://cran.r-project.org/package=cargo
> [3]: https://cran.r-project.org/package=baseflow

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

[R-pkg-devel] How can I avoid Solaris build on CRAN?

[Hiroaki Yutani]

Hi,

I recently submitted my package, which needs compilation of Rust code,
to CRAN. Now I'm seeing a CRAN check error on Solaris.

https://cran.r-project.org/web/checks/check_results_string2path.html

Since Solaris is not a platform supported by Rust [1], I need to avoid
building on Solaris. I assumed specifying

SystemRequirements: Cargo (rustc package manager)

would effectively achieve this, considering this seems the only common
thing among other CRAN packages [2][3] with Rust compilation. But, it
turned out I was wrong.

Are there any ways to ask CRAN to exclude Solaris?

Best,
Hiroaki Yutani

[1]: https://doc.rust-lang.org/nightly/rustc/platform-support.html
[2]: https://cran.r-project.org/package=cargo
[3]: https://cran.r-project.org/package=baseflow

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel