Re: Compiling python on windows with vs
On 6/15/23, Thomas Schweikle via Python-list wrote: > > No. This flag is not inherited. Someone has to set it for created > directories. It is easy to confirm: take a directory not under MSYS or > cygwin control (because it is mounted by MSYS or cygwin), set the flag, > then create directories. They all will have caseSensitivInfo disabled. That was how the attribute was implemented initially in Windows 10, but subsequently it was made inheritable. For example: C:\Temp\test>mkdir spam C:\Temp\test>fsutil file setCaseSensitiveInfo spam enable Error: Access is denied. Setting the case-sensitive attribute requires the right to add files and directories (i.e. "WD" = "write data" / "add file"; "AD" = "append data" / "add subdirectory") and the right to remove files and directories (i.e. "DC" = "delete child"). The owner of a directory doesn't necessarily inherit these rights from the parent directory (which is my case here), but the owner of any object usually has the implicit right to modify discretionary security. Let's simply grant the owner (i.e. "OW" = "owner rights") full control of the directory (i.e. "F"), inheritable to child directories (i.e. "CI" = "container inherit"). C:\Temp\test>icacls spam /grant *OW:(CI)(F) processed file: spam Successfully processed 1 files; Failed processing 0 files C:\Temp\test>fsutil file setCaseSensitiveInfo spam enable Case sensitive attribute on directory C:\Temp\test\spam is enabled. Now, create a child directory and confirm that it inherits the case-sensitive flag. C:\Temp\test>mkdir spam\eggs C:\Temp\test>fsutil file queryCaseSensitiveInfo spam\eggs Case sensitive attribute on directory C:\Temp\test\spam\eggs is enabled. > Python itself isn't the problem here. It is MSBuild.exe. For some reason > this tool lowercases sometimes whole paths to files included. This does > not matter if case sensitivity is disabled. It matters if case > sensitivity is enabled! There is no reason MSBUild.exe does it. But it > is done for some paths (as someone else pointed out). For the specific problem you had when building 3.10 and 3.11, it's actually a bug in Python's source code, which is no longer present in 3.12+. It can be fixed in 3.11, but 3.10 no longer gets bug fixes. Here's the link to the issue on GitHub: https://github.com/python/cpython/issues/105737 I encountered a different bug when building the main branch. Building the _decimal extension module includes an .asm file. The relative path in the project file has the correct case, but the build system resolved the fully-qualified path as all lower case. This problem only occurred for this single file, out of hundreds of relative paths in the project files, so it's not like the build system is completely broken when working in case-sensitive directories. There are probably a few such bugs that need to be fixed in msbuild, the compiler, and linker. After all, these tools have been developed and tested for decades on only case-insensitive filesystems. But you don't have to be on the bleeding edge. There's no reason to make directories case-sensitive for repositories that are intended for use on Windows, such as CPython. -- https://mail.python.org/mailman/listinfo/python-list
Re: Compiling python on windows with vs
Am Do., 15.Juni.2023 um 16:28:21 schrieb Eryk Sun: On 6/15/23, Thomas Schweikle via Python-list wrote: In this case: not sure what is going on. Possibly you have a setting configured that affects the behavior of Git via the MinGW-w64 runtime, such that calling mkdir() ends up calling NtSetInformationFile() to set the FileCaseSensitiveInformation for the directory. It is a mount option for MSYS or cygwin controlled parts of the file system. posix=1 switches case sensitivity on, posix=0 switches it off. MSYS or cygwin are then make caseSensitiveInfo "inheritated" by setting this flag for all newly created directories. This is a feature of the latest cygwin-dll. It might not be seen on older variants. Does the mkdir command in Git bash create a case-sensitive directory? It doesn't for me. I have to manually enable case sensitivity via `chattr +C`. What do you get for `which git` and `git --version`? $ which git /mingw64/bin/git $ git --version git version 2.41.0.windows.1 the same: $ which git /mingw64/bin/git $ git --version git version 2.41.0.windows.1 And in cmd.exe: +>where git C:\Program Files\Git\bin\git.exe +>git --version git version 2.41.0.windows.1 $ fsutil file queryCaseSensitiveInfo . The MSYS2 environment includes lsattr and chattr commands, with the case-sensitive flag mapped to "C". It's probably more convenient than typing `fsutil file queryCaseSensitiveInfo` or `fsutil file setCaseSensitiveInfo`. $ lsattr -d test test $ chattr +C test $ lsattr -d test ---C test True. But if you frequently change between environments fsutil is the command working in cmd.exe as in git-shell. core.ignorecase is not regarded in any way. It does not mater if it is set or not. Git tests the case-sensitivity of the target directory to configure core.ignorecase when cloning a repo. If it's case insensitive, then core.ignorecase is enabled. This overrides the global value. AFAIK, the ignorecase setting is unrelated to actually setting the case sensitivity of created directories; it just affects how Git behaves on a case-insensitive filesystem. -- Thomas OpenPGP_signature Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: Compiling python on windows with vs
Am Do., 15.Juni.2023 um 15:44:42 schrieb Inada Naoki: Then, git doesn't enable Windows NTFS case sensitivity. You enabled NTFS case sensitivity on "C:\Users\user\K". And Windows enabled case sensitivity for all new directories under the directory. No. This flag is not inherited. Someone has to set it for created directories. It is easy to confirm: take a directory not under MSYS or cygwin control (because it is mounted by MSYS or cygwin), set the flag, then create directories. They all will have caseSensitivInfo disabled. Since it is not default and minor setting, it is not a bug that current Python doesn't support building on case sensitive directory. But I think it is a nice improvement if next Python supports it. Python itself isn't the problem here. It is MSBuild.exe. For some reason this tool lowercases sometimes whole paths to files included. This does not matter if case sensitivity is disabled. It matters if case sensitivity is enabled! There is no reason MSBUild.exe does it. But it is done for some paths (as someone else pointed out). Regards, -- Inada Naoki -- Thomas OpenPGP_signature Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: Compiling python on windows with vs
On 6/15/23, Thomas Schweikle via Python-list wrote: > > In this case: not sure what is going on. Possibly you have a setting configured that affects the behavior of Git via the MinGW-w64 runtime, such that calling mkdir() ends up calling NtSetInformationFile() to set the FileCaseSensitiveInformation for the directory. Does the mkdir command in Git bash create a case-sensitive directory? It doesn't for me. I have to manually enable case sensitivity via `chattr +C`. What do you get for `which git` and `git --version`? $ which git /mingw64/bin/git $ git --version git version 2.41.0.windows.1 > $ fsutil file queryCaseSensitiveInfo . The MSYS2 environment includes lsattr and chattr commands, with the case-sensitive flag mapped to "C". It's probably more convenient than typing `fsutil file queryCaseSensitiveInfo` or `fsutil file setCaseSensitiveInfo`. $ lsattr -d test test $ chattr +C test $ lsattr -d test ---C test > core.ignorecase is not regarded in any way. It does not mater if it is > set or not. Git tests the case-sensitivity of the target directory to configure core.ignorecase when cloning a repo. If it's case insensitive, then core.ignorecase is enabled. This overrides the global value. AFAIK, the ignorecase setting is unrelated to actually setting the case sensitivity of created directories; it just affects how Git behaves on a case-insensitive filesystem. -- https://mail.python.org/mailman/listinfo/python-list
Re: Compiling python on windows with vs
Then, git doesn't enable Windows NTFS case sensitivity. You enabled NTFS case sensitivity on "C:\Users\user\K". And Windows enabled case sensitivity for all new directories under the directory. Since it is not default and minor setting, it is not a bug that current Python doesn't support building on case sensitive directory. But I think it is a nice improvement if next Python supports it. Regards, -- Inada Naoki -- https://mail.python.org/mailman/listinfo/python-list
Re: Compiling python on windows with vs
Am Mi., 14.Juni.2023 um 15:10:50 schrieb Eryk Sun:
On 6/14/23, Inada Naoki via Python-list wrote:
Since Git enables Windows NTFS case sensitivity while checking out sources
I didn't know that. Would you give us a link to this feature?
As far as I know, `git config core.ignorecase` doesn't mean NTFS case
sensitive.
If a repo is cloned into a case-insensitive directory, then
core.ignorecase should be enabled automatically. If a repo is cloned
into a case-sensitive directory, then core.ignorecase should not be
enabled automatically.
I searched through relevant issues on the Git for Windows repo on
GitHub, and I found nothing to indicate that a capability to
automatically enable NTFS case sensitivity has been added. I searched
through the source of Git and Git for Windows, and I didn't find any
references to WinAPI SetFileInformationByHandle: FileCaseSensitiveInfo
or NTAPI NtSetInformationFile: FileCaseSensitiveInformation, nor the
use of fsutil file setCaseSensitiveInfo.
In this case: not sure what is going on. In a git-shell on Windows:
If caseSensitiveInfo is disabled:
user@host MINGW64 ~/K
$ fsutil file queryCaseSensitiveInfo .
Das Attribut für Groß-/Kleinschreibung für das Verzeichnis
"C:\Users\user\K" ist deaktiviert.
$ git config --global -l
core.ignorecase=true
$ git clone https://github.com/python/cpython.git
Cloning into 'cpython'...
remote: Enumerating objects: 956870, done.
remote: Counting objects: 100% (1304/1304), done.
remote: Compressing objects: 100% (801/801), done.
Receiving objects: 100% (956870/956870), 557.02 MiB | 9.75 MiB/s, done.55566
Resolving deltas: 100% (760802/760802), done.
Updating files: 100% (4488/4488), done.
$ find . -type d -exec fsutil.exe file queryCaseSensitiveInfo {} \;
does not show any directory having caseSensitiveInfo enabled.
If caseSesitiveInfo is enabled:
user@host MINGW64 ~/K
$ fsutil file queryCaseSensitiveInfo .
Das Attribut für Groß-/Kleinschreibung für das Verzeichnis
"C:\Users\user\K" ist aktiviert.
$ git config --global -l
core.ignorecase=true
$ git clone https://github.com/python/cpython.git
Cloning into 'cpython'...
remote: Enumerating objects: 956870, done.
remote: Counting objects: 100% (1304/1304), done.
remote: Compressing objects: 100% (801/801), done.
Receiving objects: 100% (956870/956870), 557.02 MiB | 9.75 MiB/s, done.55566
Resolving deltas: 100% (760802/760802), done.
Updating files: 100% (4488/4488), done.
$ find . -type d -exec fsutil.exe file queryCaseSensitiveInfo {} \;
All directories created by git have caseSensitiveInfo enabled.
core.ignorecase is not regarded in any way. It does not mater if it is
set or not.
--
Thomas
OpenPGP_signature
Description: OpenPGP digital signature
--
https://mail.python.org/mailman/listinfo/python-list
Re: Compiling python on windows with vs
On 6/14/23, Inada Naoki via Python-list wrote: >> Since Git enables Windows NTFS case sensitivity while checking out sources > > I didn't know that. Would you give us a link to this feature? > As far as I know, `git config core.ignorecase` doesn't mean NTFS case > sensitive. If a repo is cloned into a case-insensitive directory, then core.ignorecase should be enabled automatically. If a repo is cloned into a case-sensitive directory, then core.ignorecase should not be enabled automatically. I searched through relevant issues on the Git for Windows repo on GitHub, and I found nothing to indicate that a capability to automatically enable NTFS case sensitivity has been added. I searched through the source of Git and Git for Windows, and I didn't find any references to WinAPI SetFileInformationByHandle: FileCaseSensitiveInfo or NTAPI NtSetInformationFile: FileCaseSensitiveInformation, nor the use of fsutil file setCaseSensitiveInfo. -- https://mail.python.org/mailman/listinfo/python-list
Re: Compiling python on windows with vs
> Since Git enables Windows NTFS case sensitivity while checking out sources I didn't know that. Would you give us a link to this feature? As far as I know, `git config core.ignorecase` doesn't mean NTFS case sensitive. On Wed, Jun 14, 2023 at 1:57 AM Thomas Schweikle via Python-list < python-list@python.org> wrote: > Hi! > > Trying to compile python on windows leads to following error: > >_testimportmultiple.vcxproj -> > > C:\Users\sct-muc\Documents\Projekte\cpython\PCbuild\amd64\_testimportmultiple.pyd >_testmultiphase.c > Bibliothek > "C:\Users\sct-muc\Documents\Projekte\cpython\PCbuild\amd64\_testmultiphase.lib" > > und Objekt "C:\Users\sct-muc\Docume >nts\Projekte\cpython\PCbuild\amd64\_testmultiphase.exp" werden erstellt. >Code wird generiert. >Codegenerierung ist abgeschlossen. >_testmultiphase.vcxproj -> > > C:\Users\sct-muc\Documents\Projekte\cpython\PCbuild\amd64\_testmultiphase.pyd >_testconsole.c > C:\Users\sct-muc\Documents\Projekte\cpython\PC\_testconsole.c(13,10): > fatal error C1083: Datei (Include) kann nicht geöffnet werde > n: "..\modules\_io\_iomodule.h": No such file or directory > [C:\Users\sct-muc\Documents\Projekte\cpython\PCbuild\_testconsole.vcxpro > j] > > Fehler beim Buildvorgang. > > if I rename "Modules" to "modules" it will find > "..\modules\_io\_iomodule.h" but wont find "..\Modules\..." > > Since Git enables Windows NTFS case sensitivity while checking out > sources ... is it a bug or a "feature"? And: is there a simple > workaround available besides disabling case sensitivity (which will > break others)? > -- > Thomas > > -- > https://mail.python.org/mailman/listinfo/python-list > -- Inada Naoki -- https://mail.python.org/mailman/listinfo/python-list
Re: Compiling python on windows with vs
Am Di., 13.Juni.2023 um 20:36:17 schrieb Mats Wichmann via Python-list:
On 6/13/23 12:12, Thomas Schweikle via Python-list wrote:
Am Di., 13.Juni.2023 um 19:20:38 schrieb Jim Schwartz:
What version of visual studio are you using?
Visual Studio 2022, aka 17.6.2.
What version of python?
python 3.10.11 or 3.11.4
I’ve had success with using the cython package in python and cl from
visual studio, but I haven’t tried visual studio alone.
Same problem at the same place: directory "../modules/..." not found,
Renaming it from "Modules" to "modules" it is found, but then fails to
find "Modules".
Looks like it awaits, compiling in Windows an filesystem only case
aware, not case sensitive -- I'm assuming this a bug now. Building
within cygwin (or MSYS, Ubuntu) this works as expected. But there it
does not search for "modules" once and "Modules" at an other place.
I just did this build the other day for the first time even from a git
checkout (so VS22, and not a versioned release but top of main branch),
and there was no such problem - did you follow the instructions at
https://devguide.python.org/getting-started/setup-building/index.html?
Yes.
Had git installed (2.41.0.windows.1), did "git clone
https://github.com/python/cpython.git;, then checked out the branch I
wanted: "git checkout 3.10.12" (or 3.11.4, 3.12.0b2).
Python was already installed and in path:
"python --version" gives back: "Python 3.11.4"
Changed into cpython directory, then called: ".\PCbuild\build.bat"
(".\pcbuild\build.bat" will, within cmd.exe, throw an error: File not
found -- remember directory cpython and all beyond are case sensitive,
not only case aware -- git sets case sensitiveness for checked out
directories).
It compiles until it does not find "modules", because the directory is
named "Modules" (all posix file systems behave this way and NTFS can be
forced to behave this way too. It is easy and any user can do it:
"fsutil.exe file SetCaseSensitiveInfo enable"
You may query this with:
"fsutil.exe file queryCaseSensitiveInfo "
In my case:
~/Documents/Projekte/cpython> fsutil.exe file queryCaseSensitiveInfo .
Das Attribut für Groß-/Kleinschreibung für das Verzeichnis ^
"C:\Users\user\Documents\Projekte\cpython" ist aktiviert.
Or: the directory we're in is case sensitive and distinguishes between
"Modules", "modules" and "MODULES" and you can have them all:
C:\Users\user\Documents\Projekte\cpython>dir
Datenträger in Laufwerk C: ist Windows
Volumeseriennummer: BC5E-F466
Verzeichnis von C:\Users\user\Documents\Projekte\cpython
Di, 13.Jun.2023 21:23 .
Di, 13.Jun.2023 21:23 ..
[...]
Di, 13.Jun.2023 21:23 MODULES
Mo, 12.Jun.2023 22:13 Modules
Di, 13.Jun.2023 21:23 modules
[...]
16 Datei(en), 1.320.340 Bytes
21 Verzeichnis(se), 134.699.622.400 Bytes frei
Since case sensitivity is necessary for a variety of projects normally
only used with *nix-OS I've configured git to create directories with
case sensitivity set enabled (its default meanwhile if git is installed
on windows).
--
Thomas
OpenPGP_signature
Description: OpenPGP digital signature
--
https://mail.python.org/mailman/listinfo/python-list
Re: Compiling python on windows with vs
On 6/13/23, Thomas Schweikle via Python-list wrote: > > Since Git enables Windows NTFS case sensitivity while checking out > sources ... is it a bug or a "feature"? And: is there a simple AFAIK the Windows version of Git (you're not using the Linux version of Git via WSL, right?) does not automatically enable NTFS case sensitivity. But a newly created directory does inherit the case sensitivity of its parent directory. Make sure to clone the CPython repo in a directory that has case sensitivity disabled. >_testconsole.c > C:\Users\sct-muc\Documents\Projekte\cpython\PC\_testconsole.c(13,10): > fatal error C1083: Datei (Include) kann nicht geöffnet werde > n: "..\modules\_io\_iomodule.h": No such file or directory > [C:\Users\sct-muc\Documents\Projekte\cpython\PCbuild\_testconsole.vcxpro > j] I just built the main branch in a case sensitive tree. I had no problem building "_testconsole.c". However, building the _decimal extension module raised a couple of serious warnings. In "PCbuild/_decimal.vcxproj", there's an include for "..\Modules\_decimal\libmpdec\vcdiv64.asm". However, MSBuild resolved this relative path with all lower-case names, i.e. "modules" instead of the correct name "Modules", and it incorrectly tried to output "vcdiv64.obj" in a subdirectory of "pcbuild" instead of the correct name "PCbuild". This appears to be a bug in MSBuild. A lot of Windows programs don't handle case-sensitive directories well, including Python's standard library. It's understandable when comparing paths, but the behavior in this case is inexcusably bad. -- https://mail.python.org/mailman/listinfo/python-list
RE: Compiling python on windows with vs
One expert told me to do the following when compiling via cython and cl: cython -3 --embed -o c_file_namepython_file_name Then, assuming python is installed in your apps directory and not your program files directory: set "PYTHON_DIR=%LocalAppData%\Programs\Python\Python311" or whatever directory you put python in. cl /O2 /I"%PYTHON_DIR%\Include" c_file_name /link /libpath:"%PYTHON_DIR%\libs" If that doesn't work, that's all I have. Sorry. -Original Message- From: Python-list On Behalf Of Thomas Schweikle via Python-list Sent: Tuesday, June 13, 2023 1:12 PM To: Python Cc: Thomas Schweikle Subject: Re: Compiling python on windows with vs Am Di., 13.Juni.2023 um 19:20:38 schrieb Jim Schwartz: > What version of visual studio are you using? Visual Studio 2022, aka 17.6.2. > What version of python? python 3.10.11 or 3.11.4 > I’ve had success with using the cython package in python and cl from visual > studio, but I haven’t tried visual studio alone. Same problem at the same place: directory "../modules/..." not found, Renaming it from "Modules" to "modules" it is found, but then fails to find "Modules". Looks like it awaits, compiling in Windows an filesystem only case aware, not case sensitive -- I'm assuming this a bug now. Building within cygwin (or MSYS, Ubuntu) this works as expected. But there it does not search for "modules" once and "Modules" at an other place. >> On Jun 13, 2023, at 11:59 AM, Thomas Schweikle via Python-list >> wrote: >> >> Fehler beim Buildvorgang -- Thomas -- https://mail.python.org/mailman/listinfo/python-list
Re: Compiling python on windows with vs
On 6/13/23 12:12, Thomas Schweikle via Python-list wrote: Am Di., 13.Juni.2023 um 19:20:38 schrieb Jim Schwartz: What version of visual studio are you using? Visual Studio 2022, aka 17.6.2. What version of python? python 3.10.11 or 3.11.4 I’ve had success with using the cython package in python and cl from visual studio, but I haven’t tried visual studio alone. Same problem at the same place: directory "../modules/..." not found, Renaming it from "Modules" to "modules" it is found, but then fails to find "Modules". Looks like it awaits, compiling in Windows an filesystem only case aware, not case sensitive -- I'm assuming this a bug now. Building within cygwin (or MSYS, Ubuntu) this works as expected. But there it does not search for "modules" once and "Modules" at an other place. I just did this build the other day for the first time even from a git checkout (so VS22, and not a versioned release but top of main branch), and there was no such problem - did you follow the instructions at https://devguide.python.org/getting-started/setup-building/index.html? -- https://mail.python.org/mailman/listinfo/python-list
Re: Compiling python on windows with vs
Am Di., 13.Juni.2023 um 19:20:38 schrieb Jim Schwartz: What version of visual studio are you using? Visual Studio 2022, aka 17.6.2. What version of python? python 3.10.11 or 3.11.4 I’ve had success with using the cython package in python and cl from visual studio, but I haven’t tried visual studio alone. Same problem at the same place: directory "../modules/..." not found, Renaming it from "Modules" to "modules" it is found, but then fails to find "Modules". Looks like it awaits, compiling in Windows an filesystem only case aware, not case sensitive -- I'm assuming this a bug now. Building within cygwin (or MSYS, Ubuntu) this works as expected. But there it does not search for "modules" once and "Modules" at an other place. On Jun 13, 2023, at 11:59 AM, Thomas Schweikle via Python-list wrote: Fehler beim Buildvorgang -- Thomas OpenPGP_signature Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: Compiling python on windows with vs
What version of visual studio are you using? What version of python? I’ve had success with using the cython package in python and cl from visual studio, but I haven’t tried visual studio alone. Sent from my iPhone > On Jun 13, 2023, at 11:59 AM, Thomas Schweikle via Python-list > wrote: > > Fehler beim Buildvorgang -- https://mail.python.org/mailman/listinfo/python-list
