Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
When submitting a package to CRAN, it is required that path names be shorter than 100 bytes, with the reason that paths longer than that cannot be made into portable tar files. This error is reported by `R CMD check --as-cran`. Since this pertains only to developing packages, this seemed like the appropriate list, but if you don't think so, I can instead ask on R-devel. Best wishes, Justin From: Martin Maechler Sent: Tuesday, December 12, 2023 10:13 AM To: McGrath, Justin M Cc: r-package-devel@r-project.org Subject: Wrong mailing list: [R-pkg-devel] Could the 100 byte path length limit be lifted? > McGrath, Justin M > on Tue, 12 Dec 2023 15:03:28 + writes: > We include other software in our source code. It has some long paths so a few of the files end up with paths longer than 100 bytes, and we need to manually rename them whenever we pull in updates. > The 100 byte path limit is from tar v7, and since > POSIX1.1988, there has not been a path length limit. That > standard is 35 years old now, so given that there is > probably no one using an old version of tar that also > wants to use the latest version of R, could the 100 byte > limit be lifted? Incidentally, I am a big proponent of > wide, long-term support, but it's hard to see that this > change would negatively impact anyone. > Best wishes, > Justin Wrong mailing list: This is a topic for R-devel, not at all R-package-devel, but be more accurate in what you are talking about, only between the line I could read that it is about some variants of using 'tar'. Best regards, Martin --- Martin Maechler ETH Zurich and R Core team __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
Justin, now that you clarified what you are actually talking about, this is a question about the CRAN policies, so you should really direct it to the CRAN team as it is their decision (R-devel would be appropriate if this was a limitation in R itself, and R-package-devel would be appropriate if you wanted help with refactoring to adhere to the policy). There are still path limits on various platforms (even if they are becoming more rare), so I'd personally question the source rather than the policy, but then your email was remarkably devoid of any details. Cheers, Simon > On Dec 13, 2023, at 6:03 AM, McGrath, Justin M wrote: > > When submitting a package to CRAN, it is required that path names be shorter > than 100 bytes, with the reason that paths longer than that cannot be made > into portable tar files. This error is reported by `R CMD check --as-cran`. > Since this pertains only to developing packages, this seemed like the > appropriate list, but if you don't think so, I can instead ask on R-devel. > > Best wishes, > Justin > > > From: Martin Maechler > Sent: Tuesday, December 12, 2023 10:13 AM > To: McGrath, Justin M > Cc: r-package-devel@r-project.org > Subject: Wrong mailing list: [R-pkg-devel] Could the 100 byte path length > limit be lifted? > >> McGrath, Justin M >>on Tue, 12 Dec 2023 15:03:28 + writes: > >> We include other software in our source code. It has some long paths so a >> few of the files end up with paths longer than 100 bytes, and we need to >> manually rename them whenever we pull in updates. >> The 100 byte path limit is from tar v7, and since >> POSIX1.1988, there has not been a path length limit. That >> standard is 35 years old now, so given that there is >> probably no one using an old version of tar that also >> wants to use the latest version of R, could the 100 byte >> limit be lifted? Incidentally, I am a big proponent of >> wide, long-term support, but it's hard to see that this >> change would negatively impact anyone. > >> Best wishes, >> Justin > > Wrong mailing list: > > This is a topic for R-devel, not at all R-package-devel, > but be more accurate in what you are talking about, only between > the line I could read that it is about some variants of using > 'tar'. > > Best regards, > Martin > --- > > Martin Maechler > ETH Zurich and R Core team > > __ > R-package-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-package-devel > __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
FWIW the R-windows FAQ says: Yet another complication is a 260 character limit on the length of the entire path name imposed by Windows. The limit applies only to some system functions, and hence it is possible to create a long path using one application yet inaccessible to another. It is sometimes possible to reduce the path length by creating a drive mapping using subst and accessing files via that drive. As of Windows 10 version 1607 and R 4.3, one can remove this limit via Windows registry by setting Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled to 1. Long paths still may not always work reliably: some applications or packages may not be able to work with them and Windows cannot execute an application with long path as the current directory. I'm having trouble finding the specific check for path lengths > 100 in the R source tree. It would be helpful to have the exact wording of the NOTE/WARNING (?) that is thrown ... (I know I could make my own mini-package with a long path length in it somewhere but ...) cheers Ben Bolker On 2023-12-12 2:57 p.m., Simon Urbanek wrote: Justin, now that you clarified what you are actually talking about, this is a question about the CRAN policies, so you should really direct it to the CRAN team as it is their decision (R-devel would be appropriate if this was a limitation in R itself, and R-package-devel would be appropriate if you wanted help with refactoring to adhere to the policy). There are still path limits on various platforms (even if they are becoming more rare), so I'd personally question the source rather than the policy, but then your email was remarkably devoid of any details. Cheers, Simon On Dec 13, 2023, at 6:03 AM, McGrath, Justin M wrote: When submitting a package to CRAN, it is required that path names be shorter than 100 bytes, with the reason that paths longer than that cannot be made into portable tar files. This error is reported by `R CMD check --as-cran`. Since this pertains only to developing packages, this seemed like the appropriate list, but if you don't think so, I can instead ask on R-devel. Best wishes, Justin From: Martin Maechler Sent: Tuesday, December 12, 2023 10:13 AM To: McGrath, Justin M Cc: r-package-devel@r-project.org Subject: Wrong mailing list: [R-pkg-devel] Could the 100 byte path length limit be lifted? McGrath, Justin M on Tue, 12 Dec 2023 15:03:28 + writes: We include other software in our source code. It has some long paths so a few of the files end up with paths longer than 100 bytes, and we need to manually rename them whenever we pull in updates. The 100 byte path limit is from tar v7, and since POSIX1.1988, there has not been a path length limit. That standard is 35 years old now, so given that there is probably no one using an old version of tar that also wants to use the latest version of R, could the 100 byte limit be lifted? Incidentally, I am a big proponent of wide, long-term support, but it's hard to see that this change would negatively impact anyone. Best wishes, Justin Wrong mailing list: This is a topic for R-devel, not at all R-package-devel, but be more accurate in what you are talking about, only between the line I could read that it is about some variants of using 'tar'. Best regards, Martin --- Martin Maechler ETH Zurich and R Core team __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
I don't know what the warning looks like, but the ?tar help page discusses the issues. Duncan Murdoch On 12/12/2023 3:10 p.m., Ben Bolker wrote: FWIW the R-windows FAQ says: Yet another complication is a 260 character limit on the length of the entire path name imposed by Windows. The limit applies only to some system functions, and hence it is possible to create a long path using one application yet inaccessible to another. It is sometimes possible to reduce the path length by creating a drive mapping using subst and accessing files via that drive. As of Windows 10 version 1607 and R 4.3, one can remove this limit via Windows registry by setting Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled to 1. Long paths still may not always work reliably: some applications or packages may not be able to work with them and Windows cannot execute an application with long path as the current directory. I'm having trouble finding the specific check for path lengths > 100 in the R source tree. It would be helpful to have the exact wording of the NOTE/WARNING (?) that is thrown ... (I know I could make my own mini-package with a long path length in it somewhere but ...) cheers Ben Bolker On 2023-12-12 2:57 p.m., Simon Urbanek wrote: Justin, now that you clarified what you are actually talking about, this is a question about the CRAN policies, so you should really direct it to the CRAN team as it is their decision (R-devel would be appropriate if this was a limitation in R itself, and R-package-devel would be appropriate if you wanted help with refactoring to adhere to the policy). There are still path limits on various platforms (even if they are becoming more rare), so I'd personally question the source rather than the policy, but then your email was remarkably devoid of any details. Cheers, Simon On Dec 13, 2023, at 6:03 AM, McGrath, Justin M wrote: When submitting a package to CRAN, it is required that path names be shorter than 100 bytes, with the reason that paths longer than that cannot be made into portable tar files. This error is reported by `R CMD check --as-cran`. Since this pertains only to developing packages, this seemed like the appropriate list, but if you don't think so, I can instead ask on R-devel. Best wishes, Justin From: Martin Maechler Sent: Tuesday, December 12, 2023 10:13 AM To: McGrath, Justin M Cc: r-package-devel@r-project.org Subject: Wrong mailing list: [R-pkg-devel] Could the 100 byte path length limit be lifted? McGrath, Justin M on Tue, 12 Dec 2023 15:03:28 + writes: We include other software in our source code. It has some long paths so a few of the files end up with paths longer than 100 bytes, and we need to manually rename them whenever we pull in updates. The 100 byte path limit is from tar v7, and since POSIX1.1988, there has not been a path length limit. That standard is 35 years old now, so given that there is probably no one using an old version of tar that also wants to use the latest version of R, could the 100 byte limit be lifted? Incidentally, I am a big proponent of wide, long-term support, but it's hard to see that this change would negatively impact anyone. Best wishes, Justin Wrong mailing list: This is a topic for R-devel, not at all R-package-devel, but be more accurate in what you are talking about, only between the line I could read that it is about some variants of using 'tar'. Best regards, Martin --- Martin Maechler ETH Zurich and R Core team __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
Thanks. Pursuing this a bit further, from ?tar "Known problems": The handling of file paths of more than 100 bytes. These were unsupported in early versions of ‘tar’, and supported in one way by POSIX ‘tar’ and in another by GNU ‘tar’ and yet another by the POSIX ‘pax’ command which recent ‘tar’ programs often support. The internal implementation warns on paths of more than 100 bytes, uses the ‘ustar’ way from the 1998 POSIX standard which supports up to 256 bytes (depending on the path: in particular the final component is limited to 100 bytes) if possible, otherwise the GNU way (which is widely supported, including by ‘untar’). This issue is reminiscent of the "invalid uid value replaced ..." warning, which has happened to me a lot but which CRAN has never actually flagged when I've submitted a package: https://stackoverflow.com/questions/30599326/warning-message-during-building-an-r-package-invalid-uid-value-replaced-by-that (By the way, this thread now seems firmly on-topic for r-pkg-devel, even if the ultimate answers must come from the CRAN maintainers ...) cheers Ben On 2023-12-12 3:41 p.m., Duncan Murdoch wrote: I don't know what the warning looks like, but the ?tar help page discusses the issues. Duncan Murdoch On 12/12/2023 3:10 p.m., Ben Bolker wrote: FWIW the R-windows FAQ says: Yet another complication is a 260 character limit on the length of the entire path name imposed by Windows. The limit applies only to some system functions, and hence it is possible to create a long path using one application yet inaccessible to another. It is sometimes possible to reduce the path length by creating a drive mapping using subst and accessing files via that drive. As of Windows 10 version 1607 and R 4.3, one can remove this limit via Windows registry by setting Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled to 1. Long paths still may not always work reliably: some applications or packages may not be able to work with them and Windows cannot execute an application with long path as the current directory. I'm having trouble finding the specific check for path lengths > 100 in the R source tree. It would be helpful to have the exact wording of the NOTE/WARNING (?) that is thrown ... (I know I could make my own mini-package with a long path length in it somewhere but ...) cheers Ben Bolker On 2023-12-12 2:57 p.m., Simon Urbanek wrote: Justin, now that you clarified what you are actually talking about, this is a question about the CRAN policies, so you should really direct it to the CRAN team as it is their decision (R-devel would be appropriate if this was a limitation in R itself, and R-package-devel would be appropriate if you wanted help with refactoring to adhere to the policy). There are still path limits on various platforms (even if they are becoming more rare), so I'd personally question the source rather than the policy, but then your email was remarkably devoid of any details. Cheers, Simon On Dec 13, 2023, at 6:03 AM, McGrath, Justin M wrote: When submitting a package to CRAN, it is required that path names be shorter than 100 bytes, with the reason that paths longer than that cannot be made into portable tar files. This error is reported by `R CMD check --as-cran`. Since this pertains only to developing packages, this seemed like the appropriate list, but if you don't think so, I can instead ask on R-devel. Best wishes, Justin From: Martin Maechler Sent: Tuesday, December 12, 2023 10:13 AM To: McGrath, Justin M Cc: r-package-devel@r-project.org Subject: Wrong mailing list: [R-pkg-devel] Could the 100 byte path length limit be lifted? McGrath, Justin M on Tue, 12 Dec 2023 15:03:28 + writes: We include other software in our source code. It has some long paths so a few of the files end up with paths longer than 100 bytes, and we need to manually rename them whenever we pull in updates. The 100 byte path limit is from tar v7, and since POSIX1.1988, there has not been a path length limit. That standard is 35 years old now, so given that there is probably no one using an old version of tar that also wants to use the latest version of R, could the 100 byte limit be lifted? Incidentally, I am a big proponent of wide, long-term support, but it's hard to see that this change would negatively impact anyone. Best wishes, Justin Wrong mailing list: This is a topic for R-devel, not at all R-package-devel, but be more accurate in what you are talking about, only between the line I could read that it is about some variants of using 'tar'. Best regards, Martin --- Martin Maechler ETH Zurich and R Core team __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
> Thanks. Pursuing this a bit further, from ?tar "Known problems": > The handling of file paths of more than 100 bytes. These > were unsupported in early versions of ‘tar’, and supported in > one way by POSIX ‘tar’ and in another by GNU ‘tar’ and yet > another by the POSIX ‘pax’ command which recent ‘tar’ > programs often support. The internal implementation warns on > paths of more than 100 bytes, uses the ‘ustar’ way from the > 1998 POSIX standard which supports up to 256 bytes (depending > on the path: in particular the final component is limited to > 100 bytes) if possible, otherwise the GNU way (which is > widely supported, including by ‘untar’). R CMD check --as-cran gives a WARNING if a path is not ustar compatible, and gives the same message as a NOTE if it is not v7 compatible. The path in the example below is ustar compatible, but not v7 compatible. *** * checking for portable file names ... NOTE Found the following non-portable file path: BioCro/inc/boost/numeric/odeint/stepper/generation/generation_controlled_adams_bashforth_moulton_really_long_file_name_one_two_three.hpp Tarballs are only required to store paths of up to 100 bytes and cannot store those of more than 256 bytes, with restrictions including to 100 bytes for the final component. *** The code is in library/tools/R/check.R. An equivalent test is performed by R CMD build in library/tools/utils/tar.R. AFAIK, CRAN rejects a package regardless whether the message is given as a NOTE or a WARNING. ustar is 35 years old and is older than R itself, and thus anyone using R in the last several decades will have ustar support. Any path stored by ustar will be suitable for any common file system in use for the last several decades. Overall, if a path is supported by ustar, there is no reason to generate any message, even if that path is not v7 compatible. If the NOTE about v7 incompatibility is still desired, then could CRAN not reject packages for which the check reports a NOTE? Best wishes, Justin From: Martin Maechler Sent: Wednesday, December 13, 2023 2:22 AM To: Ben Bolker; McGrath, Justin M Cc: Simon Urbanek; Duncan Murdoch Subject: Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted? >>>>> Ben Bolker >>>>> on Tue, 12 Dec 2023 15:48:11 -0500 writes: > Thanks. Pursuing this a bit further, from ?tar "Known problems": > The handling of file paths of more than 100 bytes. These > were unsupported in early versions of ‘tar’, and supported in > one way by POSIX ‘tar’ and in another by GNU ‘tar’ and yet > another by the POSIX ‘pax’ command which recent ‘tar’ > programs often support. The internal implementation warns on > paths of more than 100 bytes, uses the ‘ustar’ way from the > 1998 POSIX standard which supports up to 256 bytes (depending > on the path: in particular the final component is limited to > 100 bytes) if possible, otherwise the GNU way (which is > widely supported, including by ‘untar’). > This issue is reminiscent of the "invalid uid value replaced ..." > warning, which has happened to me a lot but which CRAN has never > actually flagged when I've submitted a package: > https://urldefense.com/v3/__https://stackoverflow.com/questions/30599326/warning-message-during-building-an-r-package-invalid-uid-value-replaced-by-that__;!!DZ3fjg!7eG1psjusIaxJny-wr4z56esZliTkCKpBunUO0IPxzllvu3oSRehH6ZnQh8vJN6ZkaAAmNXHaAHfrqwULOipdsJh2VOFww$ > (By the way, this thread now seems firmly on-topic for r-pkg-devel, > even if the ultimate answers must come from the CRAN maintainers ...) Well, yes, in parts at least. Originally, even when it did not seem clear, it mostly seemed about the part of the help(tar) page you mention above and the implementation of tar() .. i.e. R-devel. ... but it's not really important anymore now, and hence this goes only to you. Best regards, Martin > cheers > Ben > On 2023-12-12 3:41 p.m., Duncan Murdoch wrote: >> I don't know what the warning looks like, but the ?tar help page >> discusses the issues. >> >> Duncan Murdoch >> >> On 12/12/2023 3:10 p.m., Ben Bolker wrote: >>>FWIW the R-windows FAQ says: >>> >>> Yet another complication is a 260 character limit on the length of the >>> entire path name imposed by Windows. The limit applies only to some >>> system functions, and hence it is possible to create a long path using >>> one application yet inaccessible to another. It is sometimes possible to >>> reduce the path length by creating a drive m
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
On 12/13/23 14:56, McGrath, Justin M wrote: Thanks. Pursuing this a bit further, from ?tar "Known problems": > The handling of file paths of more than 100 bytes. These > were unsupported in early versions of ‘tar’, and supported in > one way by POSIX ‘tar’ and in another by GNU ‘tar’ and yet > another by the POSIX ‘pax’ command which recent ‘tar’ > programs often support. The internal implementation warns on > paths of more than 100 bytes, uses the ‘ustar’ way from the > 1998 POSIX standard which supports up to 256 bytes (depending > on the path: in particular the final component is limited to > 100 bytes) if possible, otherwise the GNU way (which is > widely supported, including by ‘untar’). R CMD check --as-cran gives a WARNING if a path is not ustar compatible, and gives the same message as a NOTE if it is not v7 compatible. The path in the example below is ustar compatible, but not v7 compatible. *** * checking for portable file names ... NOTE Found the following non-portable file path: BioCro/inc/boost/numeric/odeint/stepper/generation/generation_controlled_adams_bashforth_moulton_really_long_file_name_one_two_three.hpp Tarballs are only required to store paths of up to 100 bytes and cannot store those of more than 256 bytes, with restrictions including to 100 bytes for the final component. *** The code is in library/tools/R/check.R. An equivalent test is performed by R CMD build in library/tools/utils/tar.R. AFAIK, CRAN rejects a package regardless whether the message is given as a NOTE or a WARNING. ustar is 35 years old and is older than R itself, and thus anyone using R in the last several decades will have ustar support. Any path stored by ustar will be suitable for any common file system in use for the last several decades. Overall, if a path is supported by ustar, there is no reason to generate any message, even if that path is not v7 compatible. If the NOTE about v7 incompatibility is still desired, then could CRAN not reject packages for which the check reports a NOTE? Please don't forget about what has been correctly mentioned on this thread already: there is essentially a 260 character limit on Windows (see https://blog.r-project.org/2023/03/07/path-length-limit-on-windows/index.html for more). Even if the relative path length limit for a CRAN package was no longer regarded important for tar compatibility, it would still make sense for compatibility with Windows. It may still be a good service to your users if you keep renaming the files to fit into that limit. Also please do not share private emails on the list. Thanks Tomas Best wishes, Justin _ __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
On 13 December 2023 at 15:32, Tomas Kalibera wrote: | Please don't forget about what has been correctly mentioned on this | thread already: there is essentially a 260 character limit on Windows | (see | https://blog.r-project.org/2023/03/07/path-length-limit-on-windows/index.html | for more). Even if the relative path length limit for a CRAN package was | no longer regarded important for tar compatibility, it would still make | sense for compatibility with Windows. It may still be a good service to | your users if you keep renaming the files to fit into that limit. So can lift the limit from 100 char to 260 char ? Dirk -- dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
On 12/13/23 15:59, Dirk Eddelbuettel wrote: On 13 December 2023 at 15:32, Tomas Kalibera wrote: | Please don't forget about what has been correctly mentioned on this | thread already: there is essentially a 260 character limit on Windows | (see | https://blog.r-project.org/2023/03/07/path-length-limit-on-windows/index.html | for more). Even if the relative path length limit for a CRAN package was | no longer regarded important for tar compatibility, it would still make | sense for compatibility with Windows. It may still be a good service to | your users if you keep renaming the files to fit into that limit. So can lift the limit from 100 char to 260 char ? The 260 char limit is for the full path. A package would be extracted in some directory, possibly also with a rather long name. Tomas Dirk __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
On 13 December 2023 at 16:02, Tomas Kalibera wrote: | | On 12/13/23 15:59, Dirk Eddelbuettel wrote: | > On 13 December 2023 at 15:32, Tomas Kalibera wrote: | > | Please don't forget about what has been correctly mentioned on this | > | thread already: there is essentially a 260 character limit on Windows | > | (see | > | https://blog.r-project.org/2023/03/07/path-length-limit-on-windows/index.html | > | for more). Even if the relative path length limit for a CRAN package was | > | no longer regarded important for tar compatibility, it would still make | > | sense for compatibility with Windows. It may still be a good service to | > | your users if you keep renaming the files to fit into that limit. | > | > So can lift the limit from 100 char to 260 char ? | | The 260 char limit is for the full path. A package would be extracted in | some directory, possibly also with a rather long name. Call a cutoff number. Any move from '100' to '100 + N' for any nonzero N is a win. Pick one, and then commit the change. N = 50 would be a great start as arbitrary as it is. Dirk -- dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
The maximum path length supported by ustar is 255 bytes, and any path compatible with ustar is compatible with path limits for all file systems used by any operating system for the last several decades. Anything that is ustar compatible is compatible with all other requirements. There is no reason today to reject packages with ustar compatible paths that are not v7 compatible. Best wishes, Justin From: Tomas Kalibera Sent: Wednesday, December 13, 2023 8:32 AM To: McGrath, Justin M; Ben Bolker; Martin Maechler; r-package-devel@r-project.org Subject: Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted? On 12/13/23 14:56, McGrath, Justin M wrote: >> Thanks. Pursuing this a bit further, from ?tar "Known problems": > > The handling of file paths of more than 100 bytes. These > > were unsupported in early versions of ‘tar’, and supported in > > one way by POSIX ‘tar’ and in another by GNU ‘tar’ and yet > > another by the POSIX ‘pax’ command which recent ‘tar’ > > programs often support. The internal implementation warns on > > paths of more than 100 bytes, uses the ‘ustar’ way from the > > 1998 POSIX standard which supports up to 256 bytes (depending > > on the path: in particular the final component is limited to > > 100 bytes) if possible, otherwise the GNU way (which is > > widely supported, including by ‘untar’). > > R CMD check --as-cran gives a WARNING if a path is not ustar compatible, and > gives the same message as a NOTE if it is not v7 compatible. > > The path in the example below is ustar compatible, but not v7 compatible. > > *** > * checking for portable file names ... NOTE > Found the following non-portable file path: > > BioCro/inc/boost/numeric/odeint/stepper/generation/generation_controlled_adams_bashforth_moulton_really_long_file_name_one_two_three.hpp > > Tarballs are only required to store paths of up to 100 bytes and cannot > store those of more than 256 bytes, with restrictions including to 100 > bytes for the final component. > *** > > The code is in library/tools/R/check.R. An equivalent test is performed by R > CMD build in library/tools/utils/tar.R. > > AFAIK, CRAN rejects a package regardless whether the message is given as a > NOTE or a WARNING. ustar is 35 years old and is older than R itself, and thus > anyone using R in the last several decades will have ustar support. Any path > stored by ustar will be suitable for any common file system in use for the > last several decades. Overall, if a path is supported by ustar, there is no > reason to generate any message, even if that path is not v7 compatible. If > the NOTE about v7 incompatibility is still desired, then could CRAN not > reject packages for which the check reports a NOTE? Please don't forget about what has been correctly mentioned on this thread already: there is essentially a 260 character limit on Windows (see https://urldefense.com/v3/__https://blog.r-project.org/2023/03/07/path-length-limit-on-windows/index.html__;!!DZ3fjg!-9O1ZyZet_W6lAzZI0JZDI27SKTG6H4M4E5zB9xFItNpCD-I-eH2xmFxJ10uU1WAy1JNb4EiygQzkDsHkO1chmsk4TKd$ for more). Even if the relative path length limit for a CRAN package was no longer regarded important for tar compatibility, it would still make sense for compatibility with Windows. It may still be a good service to your users if you keep renaming the files to fit into that limit. Also please do not share private emails on the list. Thanks Tomas > > Best wishes, > Justin > > > _ __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
I'm not even asking for that. I'm asking for the decades old ustar comparability, which is 255 characters, at most. In practice it is slightly less than that. ustar is older than R itself. There is no one using R who doesn't have ustar compatible tar. From: Dirk Eddelbuettel Sent: Wednesday, December 13, 2023 8:59 AM To: Tomas Kalibera Cc: McGrath, Justin M; Ben Bolker; Martin Maechler; r-package-devel@r-project.org Subject: Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted? On 13 December 2023 at 15:32, Tomas Kalibera wrote: | Please don't forget about what has been correctly mentioned on this | thread already: there is essentially a 260 character limit on Windows | (see | https://urldefense.com/v3/__https://blog.r-project.org/2023/03/07/path-length-limit-on-windows/index.html__;!!DZ3fjg!-YZg5PVpulgNXCVfVklP442UG_0ofB8omMLlMq5dNadF9RP_6uofnrT6IbZpG1fpvjmAtLyAm1Y9rbc$ | for more). Even if the relative path length limit for a CRAN package was | no longer regarded important for tar compatibility, it would still make | sense for compatibility with Windows. It may still be a good service to | your users if you keep renaming the files to fit into that limit. So can lift the limit from 100 char to 260 char ? Dirk -- dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
On Windows, packages will be in "C:\Users\[User Name]\Documents\R\win-library\[R version]\[Package Name]". With a 150 byte limit, that leaves 70 bytes for the user name, R version and package name. That seems more than sufficient. If people are downloading the source files, that also leaves plenty of space regardless where they choose to extract the files. From: Dirk Eddelbuettel Sent: Wednesday, December 13, 2023 9:13 AM To: Tomas Kalibera Cc: Dirk Eddelbuettel; McGrath, Justin M; Ben Bolker; Martin Maechler; r-package-devel@r-project.org Subject: Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted? On 13 December 2023 at 16:02, Tomas Kalibera wrote: | | On 12/13/23 15:59, Dirk Eddelbuettel wrote: | > On 13 December 2023 at 15:32, Tomas Kalibera wrote: | > | Please don't forget about what has been correctly mentioned on this | > | thread already: there is essentially a 260 character limit on Windows | > | (see | > | https://urldefense.com/v3/__https://blog.r-project.org/2023/03/07/path-length-limit-on-windows/index.html__;!!DZ3fjg!-NHAlEZQvY2kegkNVkuY3Lf84nKmMahrpZ4Euz2XXFDPvMWEcP28iepLlRiKzVZdAh752lyhHxd6zvk$ | > | for more). Even if the relative path length limit for a CRAN package was | > | no longer regarded important for tar compatibility, it would still make | > | sense for compatibility with Windows. It may still be a good service to | > | your users if you keep renaming the files to fit into that limit. | > | > So can lift the limit from 100 char to 260 char ? | | The 260 char limit is for the full path. A package would be extracted in | some directory, possibly also with a rather long name. Call a cutoff number. Any move from '100' to '100 + N' for any nonzero N is a win. Pick one, and then commit the change. N = 50 would be a great start as arbitrary as it is. Dirk -- dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
This is a narrow view of the world. As has been mentioned here by Tomas, the issue at this point is that a very widely-used operating system does not allow the absolute path to be longer than 254 characters unless users make possibly-breaking changes to their OS configuration. If a user is currently working in a directory with a path that is 150 characters long, and you try to un-tar a package there, it will work. If they are in a directory path that is 170 characters long, then the un-tar will fail. They then have to do some research and discover that they have to share the total absolute path length with the package creators, and they become accustomed to working in directories with path length no longer than 154 characters. Now you want to propose giving the entire path length budget plus 2 to the package developers. Not gonna happen. And Dirk wants 50 more characters. Then the user working at 150 characters tries to un-tar a package with paths 150 characters long and 150 user plus 150 package gets you to 300 characters and what used to work for them stops working suddenly. I think a package developer can figure out how to shorten their internal paths easier than thousands of users can restructure their historically useful directory structures or break their existing software. I vote no to both of these proposals. Eventually, Microsoft is going to virtualize running old Windows APIs, and once people migrate away from the old API then this stupid problem will go away. But as it is we are stuck. On December 13, 2023 7:27:42 AM PST, "McGrath, Justin M" wrote: >I'm not even asking for that. I'm asking for the decades old ustar >comparability, which is 255 characters, at most. In practice it is slightly >less than that. ustar is older than R itself. There is no one using R who >doesn't have ustar compatible tar. > > >From: Dirk Eddelbuettel >Sent: Wednesday, December 13, 2023 8:59 AM >To: Tomas Kalibera >Cc: McGrath, Justin M; Ben Bolker; Martin Maechler; >r-package-devel@r-project.org >Subject: Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length >limit be lifted? > > >On 13 December 2023 at 15:32, Tomas Kalibera wrote: >| Please don't forget about what has been correctly mentioned on this >| thread already: there is essentially a 260 character limit on Windows >| (see >| >https://urldefense.com/v3/__https://blog.r-project.org/2023/03/07/path-length-limit-on-windows/index.html__;!!DZ3fjg!-YZg5PVpulgNXCVfVklP442UG_0ofB8omMLlMq5dNadF9RP_6uofnrT6IbZpG1fpvjmAtLyAm1Y9rbc$ >| for more). Even if the relative path length limit for a CRAN package was >| no longer regarded important for tar compatibility, it would still make >| sense for compatibility with Windows. It may still be a good service to >| your users if you keep renaming the files to fit into that limit. > >So can lift the limit from 100 char to 260 char ? > >Dirk > >-- >dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org > >__ >R-package-devel@r-project.org mailing list >https://stat.ethz.ch/mailman/listinfo/r-package-devel -- Sent from my phone. Please excuse my brevity. __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
On Wed, Dec 13, 2023 at 8:06 AM McGrath, Justin M wrote: > > On Windows, packages will be in "C:\Users\[User > Name]\Documents\R\win-library\[R version]\[Package Name]". In R (>= 4.2.0), the default R_LIBS_USER path has moved to under LOCALAPPDATA, e.g. "C:\Users\[User Name]\AppData\Local\R\win-library\[R version]". See https://cran.r-project.org/bin/windows/base/rw-FAQ.html FWIW, one workaround for too long paths on MS Windows is to map a long path to a drive letter, e.g. subst Y: 'C:/VeryLongPathToo/Users/JohnDoe/AppData/Local/R/'. and then work with Y: instead of C:. We had to use that in some projects with nested data folder structures. This approach is tedious, and might require special permissions (not sure). /Henrik > > With a 150 byte limit, that leaves 70 bytes for the user name, R version and > package name. That seems more than sufficient. If people are downloading the > source files, that also leaves plenty of space regardless where they choose > to extract the files. > > > From: Dirk Eddelbuettel > Sent: Wednesday, December 13, 2023 9:13 AM > To: Tomas Kalibera > Cc: Dirk Eddelbuettel; McGrath, Justin M; Ben Bolker; Martin Maechler; > r-package-devel@r-project.org > Subject: Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length > limit be lifted? > > > On 13 December 2023 at 16:02, Tomas Kalibera wrote: > | > | On 12/13/23 15:59, Dirk Eddelbuettel wrote: > | > On 13 December 2023 at 15:32, Tomas Kalibera wrote: > | > | Please don't forget about what has been correctly mentioned on this > | > | thread already: there is essentially a 260 character limit on Windows > | > | (see > | > | > https://urldefense.com/v3/__https://blog.r-project.org/2023/03/07/path-length-limit-on-windows/index.html__;!!DZ3fjg!-NHAlEZQvY2kegkNVkuY3Lf84nKmMahrpZ4Euz2XXFDPvMWEcP28iepLlRiKzVZdAh752lyhHxd6zvk$ > | > | for more). Even if the relative path length limit for a CRAN package was > | > | no longer regarded important for tar compatibility, it would still make > | > | sense for compatibility with Windows. It may still be a good service to > | > | your users if you keep renaming the files to fit into that limit. > | > > | > So can lift the limit from 100 char to 260 char ? > | > | The 260 char limit is for the full path. A package would be extracted in > | some directory, possibly also with a rather long name. > > Call a cutoff number. > > Any move from '100' to '100 + N' for any nonzero N is a win. Pick one, and > then commit the change. N = 50 would be a great start as arbitrary as it is. > > Dirk > > -- > dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org > > __ > R-package-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
On Wed, 13 Dec 2023, McGrath, Justin M wrote: On Windows, packages will be in "C:\Users\[User Name]\Documents\R\win-library\[R version]\[Package Name]". With a 150 byte limit, that leaves 70 bytes for the user name, R version and package name. That seems more than sufficient. If people are downloading the source files, that also leaves plenty of space regardless where they choose to extract the files. 70 bytes ?? My name is 18 characters long and there are plenty of people with longer names. I also saw practice on Windows systems to append the name of organization or department. Also, this restricts the length of package name which is arguably more important that internal package path names that the user never sees. That said, that Windows limitation is only for some programs, and the pertitent question is whether R and any software used by R has this limitation. I suspect the answer is no, but as all my systems are Linux I can not check. Vladimir Dergachev __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
Compiled code is another source of long names. Some libraries are produced by companies with style restrictions that demand LongDescriptiveNames for functions and classes, and which expect file names to match the name of the class contained within. If you've got Models/SpecificModelFamily/PosteriorSamplers/SpecificTypeOfPosteriorSampler.hpp it leaves very little room (20 characters) for "C:\Users\..." etc up front. I solved my 'tar' problems with 'sed'. In human readable code I keep the long names. In the code that gets submitted to CRAN I pass specific file names through sed to rename the long files. I have an install script with lines like: hdr="MvStateSpaceRegressionPosteriorSampler.hpp" new_hdr="MVSSRPS.hpp" mv "$ibase/$dbase/$hdr" "$ibase/$dbase/$new_hdr" echo "replacing $dbase/$hdr with $dbase/$new_hdr" grep -rl "$dbase/$hdr" "$BOOM_DIR/src/Models" | xargs $SED -i "s|$dbase/$hdr|$dbase/$new_hdr|g" This renames the files, and uses sed to replace #include references to file names with the shorter file names. The SED macro is because on mac 'sed' is spelled 'gsed'. I agree the system sucks. It would be nice if one of the steps in winbuilder took non-windows compliant TAR files and replaced them with ZIP files or something similar. We'd still have the Win32 character limit, but 256 > 100 so the problem wouldn't bite so hard. On Thu, Dec 14, 2023 at 6:19 AM Vladimir Dergachev wrote: > > > On Wed, 13 Dec 2023, McGrath, Justin M wrote: > > > On Windows, packages will be in "C:\Users\[User > Name]\Documents\R\win-library\[R version]\[Package Name]". > > > > With a 150 byte limit, that leaves 70 bytes for the user name, R version > > and package name. That seems more than sufficient. If people are > > downloading the source files, that also leaves plenty of space > > regardless where they choose to extract the files. > > > > 70 bytes ?? My name is 18 characters long and there are plenty of people > with longer names. I also saw practice on Windows systems to append the > name of organization or department. > > Also, this restricts the length of package name which is arguably more > important that internal package path names that the user never sees. > > That said, that Windows limitation is only for some programs, and the > pertitent question is whether R and any software used by R has this > limitation. I suspect the answer is no, but as all my systems are Linux > I can not check. > > Vladimir Dergachev > > __ > R-package-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-package-devel > [[alternative HTML version deleted]] __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
At this point, the policy appears to have nothing to do with tar, contrary to the error message. Thus, replacing tar with zip wouldn't address CRAN's concern. The policy given in this thread (and seemingly unstated anywhere public) is that CRAN wants to allow someone on Windows to unpack an R package in a directory with a path length of 160 characters, given the 260 limit on Windows. The implication here is that developers of public libraries needs to get their act together because they have *total path lengths* of 85 characters, that one needs to rename to include in an R package, in order to accommodate someone who is unpacking R packages in directories with lengths of 160 characters (just the directory length!). I doubt such people exist, so we are accommodating a myth. I don't even have directories that long on my computer, and I certainly wouldn't be unpacking R packages in them if I did. Path lengths in packages of 130 to 150 characters would allow unpacking in directories of 130 to 110 characters. Anything in that range would likely result in no or very few problems for either the people creating or the people unpacking packages. The claim that people need to use sensible naming schemes goes both ways. One can't expect unreasonably long path in packages, and people making packages can't expect people try to unpack into unreasonably long directories. Yet, it's impossible to miss the great disparity in the current split of 100 character total path vs 160 characters for just a directory path. From: Steven Scott Sent: Thursday, December 14, 2023 12:11 PM To: Vladimir Dergachev Cc: McGrath, Justin M; r-package-devel@r-project.org Subject: Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted? Compiled code is another source of long names. Some libraries are produced by companies with style restrictions that demand LongDescriptiveNames for functions and classes, and which expect file names to match the name of the class contained within. If you've got Models/SpecificModelFamily/PosteriorSamplers/SpecificTypeOfPosteriorSampler.hpp it leaves very little room (20 characters) for "C:\Users\..." etc up front. I solved my 'tar' problems with 'sed'. In human readable code I keep the long names. In the code that gets submitted to CRAN I pass specific file names through sed to rename the long files. I have an install script with lines like: hdr="MvStateSpaceRegressionPosteriorSampler.hpp" new_hdr="MVSSRPS.hpp" mv "$ibase/$dbase/$hdr" "$ibase/$dbase/$new_hdr" echo "replacing $dbase/$hdr with $dbase/$new_hdr" grep -rl "$dbase/$hdr" "$BOOM_DIR/src/Models" | xargs $SED -i "s|$dbase/$hdr|$dbase/$new_hdr|g" This renames the files, and uses sed to replace #include references to file names with the shorter file names. The SED macro is because on mac 'sed' is spelled 'gsed'. I agree the system sucks. It would be nice if one of the steps in winbuilder took non-windows compliant TAR files and replaced them with ZIP files or something similar. We'd still have the Win32 character limit, but 256 > 100 so the problem wouldn't bite so hard. On Thu, Dec 14, 2023 at 6:19 AM Vladimir Dergachev mailto:volo...@mindspring.com>> wrote: On Wed, 13 Dec 2023, McGrath, Justin M wrote: > On Windows, packages will be in "C:\Users\[User > Name]\Documents\R\win-library\[R version]\[Package Name]". > > With a 150 byte limit, that leaves 70 bytes for the user name, R version > and package name. That seems more than sufficient. If people are > downloading the source files, that also leaves plenty of space > regardless where they choose to extract the files. > 70 bytes ?? My name is 18 characters long and there are plenty of people with longer names. I also saw practice on Windows systems to append the name of organization or department. Also, this restricts the length of package name which is arguably more important that internal package path names that the user never sees. That said, that Windows limitation is only for some programs, and the pertitent question is whether R and any software used by R has this limitation. I suspect the answer is no, but as all my systems are Linux I can not check. Vladimir Dergachev __ R-package-devel@r-project.org<mailto:R-package-devel@r-project.org> mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel<https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-package-devel__;!!DZ3fjg!93aINLPSC17ykCzT_i1DXmJ-yJAMwt2qaA6IIJjoA4XZP-dDbg_gZp8N-nN9Mcfpw4nR1wc--NRqfx6iGnr6OLa0XJUUKQ$> __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted?
On 12/14/23 19:39, McGrath, Justin M wrote: At this point, the policy appears to have nothing to do with tar, contrary to the error message. Thus, replacing tar with zip wouldn't address CRAN's concern. The policy given in this thread (and seemingly unstated anywhere public) is that CRAN wants to allow someone on Windows to unpack an R package in a directory with a path length of 160 characters, given the 260 limit on Windows. The CRAN repository policy is a publicly accessible document at [1]. If you are unsure what is acceptable or not, or if you think you a have strong case to ask for an exception for your package, or possibly for completely ignoring this warning from R, contact the CRAN team [2]. On this list you may receive help and advice from individual people, but that doesn't constitute the CRAN policy nor what CRAN "wants", even when from people who individually are members of the CRAN team. You may still find such advice useful for making your packages better. If you wanted R tar() to stop issuing warnings in certain cases, you can suggest that on R bugzilla. If that happens and instead say a new check is added to R to warn when a different limit is reached (e.g. to protect users on Windows), you can then file another bug report. I don't think you can get any more help with this on this list. 1. https://cran.r-project.org/web/packages/policies.html 2. https://cran.r-project.org Tomas The implication here is that developers of public libraries needs to get their act together because they have *total path lengths* of 85 characters, that one needs to rename to include in an R package, in order to accommodate someone who is unpacking R packages in directories with lengths of 160 characters (just the directory length!). I doubt such people exist, so we are accommodating a myth. I don't even have directories that long on my computer, and I certainly wouldn't be unpacking R packages in them if I did. Path lengths in packages of 130 to 150 characters would allow unpacking in directories of 130 to 110 characters. Anything in that range would likely result in no or very few problems for either the people creating or the people unpacking packages. The claim that people need to use sensible naming schemes goes both ways. One can't expect unreasonably long path in packages, and people making packages can't expect people try to unpack into unreasonably long directories. Yet, it's impossible to miss the great disparity in the current split of 100 character total path vs 160 characters for just a directory path. From: Steven Scott Sent: Thursday, December 14, 2023 12:11 PM To: Vladimir Dergachev Cc: McGrath, Justin M; r-package-devel@r-project.org Subject: Re: [R-pkg-devel] Wrong mailing list: Could the 100 byte path length limit be lifted? Compiled code is another source of long names. Some libraries are produced by companies with style restrictions that demand LongDescriptiveNames for functions and classes, and which expect file names to match the name of the class contained within. If you've got Models/SpecificModelFamily/PosteriorSamplers/SpecificTypeOfPosteriorSampler.hpp it leaves very little room (20 characters) for "C:\Users\..." etc up front. I solved my 'tar' problems with 'sed'. In human readable code I keep the long names. In the code that gets submitted to CRAN I pass specific file names through sed to rename the long files. I have an install script with lines like: hdr="MvStateSpaceRegressionPosteriorSampler.hpp" new_hdr="MVSSRPS.hpp" mv "$ibase/$dbase/$hdr" "$ibase/$dbase/$new_hdr" echo "replacing $dbase/$hdr with $dbase/$new_hdr" grep -rl "$dbase/$hdr" "$BOOM_DIR/src/Models" | xargs $SED -i "s|$dbase/$hdr|$dbase/$new_hdr|g" This renames the files, and uses sed to replace #include references to file names with the shorter file names. The SED macro is because on mac 'sed' is spelled 'gsed'. I agree the system sucks. It would be nice if one of the steps in winbuilder took non-windows compliant TAR files and replaced them with ZIP files or something similar. We'd still have the Win32 character limit, but 256 > 100 so the problem wouldn't bite so hard. On Thu, Dec 14, 2023 at 6:19 AM Vladimir Dergachev mailto:volo...@mindspring.com>> wrote: On Wed, 13 Dec 2023, McGrath, Justin M wrote: On Windows, packages will be in "C:\Users\[User Name]\Documents\R\win-library\[R version]\[Package Name]". With a 150 byte limit, that leaves 70 bytes for the user name, R version and package name. That seems more than sufficient. If people are downloading the source files, that also leaves plenty of space regardless where they choose to extract the files. 70 bytes ?? My name
