[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/abhina-sree closed https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
perry-ca wrote: @owenca @mydeveloperday Thanks for testing on Windows and taking the time to review the PR. https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
mydeveloperday wrote: > > We don't normally commit in clang-format with a unit test > > I assume you mean "without a unit test". In this case the unit test is the > existing test cases. Some fail on z/OS because the files are not read in as > text. It's not really possible to detect this issue on other platforms. I > can't think of a new unit test that wouldn't duplicate what we already have. No its ok.. we kind of have a rule witrh clang-format that all commits need some form of test, but again I can understand this is hard to test. Go ahead. https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
@@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool ErrorOnIncompleteFormat = false) { // On Windows, overwriting a file with an open file mapping doesn't work, // so read the whole file into memory when formatting in-place. ErrorOr> CodeOrErr = - !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName) -: MemoryBuffer::getFileOrSTDIN(FileName); + !OutputXML && Inplace + ? MemoryBuffer::getFileAsStream(FileName) + : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true); mydeveloperday wrote: Yes I also was able to confirm there was no impact on windows i just wasn't sure so needed to prove it to myself, I'm ok if you want to commit this into clang-format, if you break stuff... we'll simply revert it ;-) https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
@@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool ErrorOnIncompleteFormat = false) { // On Windows, overwriting a file with an open file mapping doesn't work, // so read the whole file into memory when formatting in-place. ErrorOr> CodeOrErr = - !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName) -: MemoryBuffer::getFileOrSTDIN(FileName); + !OutputXML && Inplace + ? MemoryBuffer::getFileAsStream(FileName) + : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true); owenca wrote: I tested it on Windows and didn't have a problem. I used LF only, CRLF only, and mixed line endings for the following: ``` int i; int j; ``` Running clang-format through all values of the `LineEnding` option, with and without in-place, seemed ok. https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
perry-ca wrote: > We don't normally commit in clang-format with a unit test I assume you mean "without a unit test". In this case the unit test is the existing test cases. Some fail on z/OS because the files are not read in as text. It's not really possible to detect this issue on other platforms. I can't think of a new unit test that wouldn't duplicate what we already have. https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
@@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool ErrorOnIncompleteFormat = false) { // On Windows, overwriting a file with an open file mapping doesn't work, // so read the whole file into memory when formatting in-place. ErrorOr> CodeOrErr = - !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName) -: MemoryBuffer::getFileOrSTDIN(FileName); + !OutputXML && Inplace + ? MemoryBuffer::getFileAsStream(FileName) + : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true); perry-ca wrote: I don't believe this will be a problem for Windows. This is a common problem that we have fixed in a few places already. I don't have a Windows platform to confirm. Would you or someone else be able to verify? https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
@@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool ErrorOnIncompleteFormat = false) { // On Windows, overwriting a file with an open file mapping doesn't work, // so read the whole file into memory when formatting in-place. ErrorOr> CodeOrErr = - !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName) -: MemoryBuffer::getFileOrSTDIN(FileName); + !OutputXML && Inplace + ? MemoryBuffer::getFileAsStream(FileName) + : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true); mydeveloperday wrote: Could there be other impacts if the files that aren't utf-8 or contain internationalize characters in say comments? https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/mydeveloperday edited https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/mydeveloperday commented: We don't normally commit in clang-format with a unit test https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
@@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool ErrorOnIncompleteFormat = false) { // On Windows, overwriting a file with an open file mapping doesn't work, // so read the whole file into memory when formatting in-place. ErrorOr> CodeOrErr = - !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName) -: MemoryBuffer::getFileOrSTDIN(FileName); + !OutputXML && Inplace + ? MemoryBuffer::getFileAsStream(FileName) + : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true); mydeveloperday wrote: On windows won't this have an impact of potentially converting \n to \r\n? could a clang-format effectively become a unix2dos at the same time (for those using clang-format on cygwin is that what they want?) can we have a unit test to understand the impact? https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/perry-ca updated https://github.com/llvm/llvm-project/pull/90128 >From bf3ed6819301d9fcc7e001b9e6676d5f4ce66c4d Mon Sep 17 00:00:00 2001 From: Sean Perry Date: Thu, 25 Apr 2024 15:58:05 -0500 Subject: [PATCH] treat text files as text files so autoconversion is done --- clang/tools/clang-format/ClangFormat.cpp| 7 --- llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp | 3 ++- llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp | 9 ++--- llvm/tools/yaml2obj/yaml2obj.cpp| 2 +- llvm/utils/lit/lit/builtin_commands/cat.py | 18 -- llvm/utils/lit/lit/llvm/config.py | 7 +++ 6 files changed, 36 insertions(+), 10 deletions(-) diff --git a/clang/tools/clang-format/ClangFormat.cpp b/clang/tools/clang-format/ClangFormat.cpp index feb733fe3c9e0b..01f7c6047726e2 100644 --- a/clang/tools/clang-format/ClangFormat.cpp +++ b/clang/tools/clang-format/ClangFormat.cpp @@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool ErrorOnIncompleteFormat = false) { // On Windows, overwriting a file with an open file mapping doesn't work, // so read the whole file into memory when formatting in-place. ErrorOr> CodeOrErr = - !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName) -: MemoryBuffer::getFileOrSTDIN(FileName); + !OutputXML && Inplace + ? MemoryBuffer::getFileAsStream(FileName) + : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true); if (std::error_code EC = CodeOrErr.getError()) { errs() << EC.message() << "\n"; return true; @@ -558,7 +559,7 @@ static int dumpConfig() { // Read in the code in case the filename alone isn't enough to detect the // language. ErrorOr> CodeOrErr = -MemoryBuffer::getFileOrSTDIN(FileNames[0]); +MemoryBuffer::getFileOrSTDIN(FileNames[0], /*IsText=*/true); if (std::error_code EC = CodeOrErr.getError()) { llvm::errs() << EC.message() << "\n"; return 1; diff --git a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp index c3015d895230ea..40ee59c014b09f 100644 --- a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp +++ b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp @@ -95,7 +95,8 @@ static std::vector getSearchPaths(opt::InputArgList *Args, // Opens a file. Path has to be resolved already. (used for def file) std::unique_ptr openFile(const Twine &Path) { - ErrorOr> MB = MemoryBuffer::getFile(Path); + ErrorOr> MB = + MemoryBuffer::getFile(Path, /*IsText=*/true); if (std::error_code EC = MB.getError()) { llvm::errs() << "cannot open file " << Path << ": " << EC.message() << "\n"; diff --git a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp index 6a5646965df2cf..c5ccd64f116539 100644 --- a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp +++ b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp @@ -144,15 +144,18 @@ int main(int argc, const char *argv[]) { cl::HideUnrelatedOptions({&CXXMapCategory, &getColorCategory()}); cl::ParseCommandLineOptions(argc, argv, "LLVM C++ mangled name remapper\n"); - auto OldSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(OldSymbolFile); + auto OldSymbolBufOrError = + MemoryBuffer::getFileOrSTDIN(OldSymbolFile, /*IsText=*/true); if (!OldSymbolBufOrError) exitWithErrorCode(OldSymbolBufOrError.getError(), OldSymbolFile); - auto NewSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(NewSymbolFile); + auto NewSymbolBufOrError = + MemoryBuffer::getFileOrSTDIN(NewSymbolFile, /*IsText=*/true); if (!NewSymbolBufOrError) exitWithErrorCode(NewSymbolBufOrError.getError(), NewSymbolFile); - auto RemappingBufOrError = MemoryBuffer::getFileOrSTDIN(RemappingFile); + auto RemappingBufOrError = + MemoryBuffer::getFileOrSTDIN(RemappingFile, /*IsText=*/true); if (!RemappingBufOrError) exitWithErrorCode(RemappingBufOrError.getError(), RemappingFile); diff --git a/llvm/tools/yaml2obj/yaml2obj.cpp b/llvm/tools/yaml2obj/yaml2obj.cpp index b7f5356e22a9e6..4a060e1aad427f 100644 --- a/llvm/tools/yaml2obj/yaml2obj.cpp +++ b/llvm/tools/yaml2obj/yaml2obj.cpp @@ -130,7 +130,7 @@ int main(int argc, char **argv) { } ErrorOr> Buf = - MemoryBuffer::getFileOrSTDIN(Input); + MemoryBuffer::getFileOrSTDIN(Input, /*IsText=*/true); if (!Buf) return 1; diff --git a/llvm/utils/lit/lit/builtin_commands/cat.py b/llvm/utils/lit/lit/builtin_commands/cat.py index 37f55c0aef210b..6fb2152ef9332d 100644 --- a/llvm/utils/lit/lit/builtin_commands/cat.py +++ b/llvm/utils/lit/lit/builtin_commands/cat.py @@ -55,10 +55,24 @@ def main(argv): msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY) for filename in filenames: try: -fileToCat = open(filename, "rb") -contents = fileToCat.read() +contents = None +is_text = False +try: +if sys.platform != "win32": +
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
perry-ca wrote: Failed at `FAIL: BOLT :: RISCV/fake-label-no-entry.c` this time. I'll give it a day or so and try again. https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/abhina-sree approved this pull request. LGTM https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
perry-ca wrote: > LGTM, but check why the following test in checks failed: `FAIL: BOLT :: > RISCV/unnamed-sym-no-entry.c` Looks unrelated. I'm kicking off another build. https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/perry-ca updated https://github.com/llvm/llvm-project/pull/90128 >From 2d7034366547b176f89b076eb60bcb44424fae11 Mon Sep 17 00:00:00 2001 From: Sean Perry Date: Thu, 25 Apr 2024 15:58:05 -0500 Subject: [PATCH] treat text files as text files so autoconversion is done --- clang/tools/clang-format/ClangFormat.cpp| 7 --- llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp | 3 ++- llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp | 9 ++--- llvm/tools/yaml2obj/yaml2obj.cpp| 2 +- llvm/utils/lit/lit/builtin_commands/cat.py | 18 -- llvm/utils/lit/lit/llvm/config.py | 7 +++ 6 files changed, 36 insertions(+), 10 deletions(-) diff --git a/clang/tools/clang-format/ClangFormat.cpp b/clang/tools/clang-format/ClangFormat.cpp index feb733fe3c9e0b..01f7c6047726e2 100644 --- a/clang/tools/clang-format/ClangFormat.cpp +++ b/clang/tools/clang-format/ClangFormat.cpp @@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool ErrorOnIncompleteFormat = false) { // On Windows, overwriting a file with an open file mapping doesn't work, // so read the whole file into memory when formatting in-place. ErrorOr> CodeOrErr = - !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName) -: MemoryBuffer::getFileOrSTDIN(FileName); + !OutputXML && Inplace + ? MemoryBuffer::getFileAsStream(FileName) + : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true); if (std::error_code EC = CodeOrErr.getError()) { errs() << EC.message() << "\n"; return true; @@ -558,7 +559,7 @@ static int dumpConfig() { // Read in the code in case the filename alone isn't enough to detect the // language. ErrorOr> CodeOrErr = -MemoryBuffer::getFileOrSTDIN(FileNames[0]); +MemoryBuffer::getFileOrSTDIN(FileNames[0], /*IsText=*/true); if (std::error_code EC = CodeOrErr.getError()) { llvm::errs() << EC.message() << "\n"; return 1; diff --git a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp index c3015d895230ea..40ee59c014b09f 100644 --- a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp +++ b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp @@ -95,7 +95,8 @@ static std::vector getSearchPaths(opt::InputArgList *Args, // Opens a file. Path has to be resolved already. (used for def file) std::unique_ptr openFile(const Twine &Path) { - ErrorOr> MB = MemoryBuffer::getFile(Path); + ErrorOr> MB = + MemoryBuffer::getFile(Path, /*IsText=*/true); if (std::error_code EC = MB.getError()) { llvm::errs() << "cannot open file " << Path << ": " << EC.message() << "\n"; diff --git a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp index 6a5646965df2cf..c5ccd64f116539 100644 --- a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp +++ b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp @@ -144,15 +144,18 @@ int main(int argc, const char *argv[]) { cl::HideUnrelatedOptions({&CXXMapCategory, &getColorCategory()}); cl::ParseCommandLineOptions(argc, argv, "LLVM C++ mangled name remapper\n"); - auto OldSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(OldSymbolFile); + auto OldSymbolBufOrError = + MemoryBuffer::getFileOrSTDIN(OldSymbolFile, /*IsText=*/true); if (!OldSymbolBufOrError) exitWithErrorCode(OldSymbolBufOrError.getError(), OldSymbolFile); - auto NewSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(NewSymbolFile); + auto NewSymbolBufOrError = + MemoryBuffer::getFileOrSTDIN(NewSymbolFile, /*IsText=*/true); if (!NewSymbolBufOrError) exitWithErrorCode(NewSymbolBufOrError.getError(), NewSymbolFile); - auto RemappingBufOrError = MemoryBuffer::getFileOrSTDIN(RemappingFile); + auto RemappingBufOrError = + MemoryBuffer::getFileOrSTDIN(RemappingFile, /*IsText=*/true); if (!RemappingBufOrError) exitWithErrorCode(RemappingBufOrError.getError(), RemappingFile); diff --git a/llvm/tools/yaml2obj/yaml2obj.cpp b/llvm/tools/yaml2obj/yaml2obj.cpp index b7f5356e22a9e6..4a060e1aad427f 100644 --- a/llvm/tools/yaml2obj/yaml2obj.cpp +++ b/llvm/tools/yaml2obj/yaml2obj.cpp @@ -130,7 +130,7 @@ int main(int argc, char **argv) { } ErrorOr> Buf = - MemoryBuffer::getFileOrSTDIN(Input); + MemoryBuffer::getFileOrSTDIN(Input, /*IsText=*/true); if (!Buf) return 1; diff --git a/llvm/utils/lit/lit/builtin_commands/cat.py b/llvm/utils/lit/lit/builtin_commands/cat.py index 37f55c0aef210b..6fb2152ef9332d 100644 --- a/llvm/utils/lit/lit/builtin_commands/cat.py +++ b/llvm/utils/lit/lit/builtin_commands/cat.py @@ -55,10 +55,24 @@ def main(argv): msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY) for filename in filenames: try: -fileToCat = open(filename, "rb") -contents = fileToCat.read() +contents = None +is_text = False +try: +if sys.platform != "win32": +
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/zibi2 approved this pull request. LGTM with clean checks. https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
zibi2 wrote: LGTM, but check why the following test in checks failed: `FAIL: BOLT :: RISCV/unnamed-sym-no-entry.c` https://github.com/llvm/llvm-project/pull/90128 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
llvmbot wrote: @llvm/pr-subscribers-testing-tools Author: Sean Perry (perry-ca) Changes To support auto-conversion on z/OS text files need to be opened as text files. These changes will fix a number of LIT failures due to text files not being converted to the internal code page. - update a number of tools so they open the text files as text files - add support in the cat.py to open a text file as a text file (Windows will continue to treat all files as binary so new lines are handled correctly) - add env var definitions to enable auto-conversion in the lit config file. --- Full diff: https://github.com/llvm/llvm-project/pull/90128.diff 6 Files Affected: - (modified) clang/tools/clang-format/ClangFormat.cpp (+4-3) - (modified) llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp (+2-1) - (modified) llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp (+6-3) - (modified) llvm/tools/yaml2obj/yaml2obj.cpp (+1-1) - (modified) llvm/utils/lit/lit/builtin_commands/cat.py (+16-2) - (modified) llvm/utils/lit/lit/llvm/config.py (+7) ``diff diff --git a/clang/tools/clang-format/ClangFormat.cpp b/clang/tools/clang-format/ClangFormat.cpp index feb733fe3c9e0b..01f7c6047726e2 100644 --- a/clang/tools/clang-format/ClangFormat.cpp +++ b/clang/tools/clang-format/ClangFormat.cpp @@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool ErrorOnIncompleteFormat = false) { // On Windows, overwriting a file with an open file mapping doesn't work, // so read the whole file into memory when formatting in-place. ErrorOr> CodeOrErr = - !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName) -: MemoryBuffer::getFileOrSTDIN(FileName); + !OutputXML && Inplace + ? MemoryBuffer::getFileAsStream(FileName) + : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true); if (std::error_code EC = CodeOrErr.getError()) { errs() << EC.message() << "\n"; return true; @@ -558,7 +559,7 @@ static int dumpConfig() { // Read in the code in case the filename alone isn't enough to detect the // language. ErrorOr> CodeOrErr = -MemoryBuffer::getFileOrSTDIN(FileNames[0]); +MemoryBuffer::getFileOrSTDIN(FileNames[0], /*IsText=*/true); if (std::error_code EC = CodeOrErr.getError()) { llvm::errs() << EC.message() << "\n"; return 1; diff --git a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp index c3015d895230ea..40ee59c014b09f 100644 --- a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp +++ b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp @@ -95,7 +95,8 @@ static std::vector getSearchPaths(opt::InputArgList *Args, // Opens a file. Path has to be resolved already. (used for def file) std::unique_ptr openFile(const Twine &Path) { - ErrorOr> MB = MemoryBuffer::getFile(Path); + ErrorOr> MB = + MemoryBuffer::getFile(Path, /*IsText=*/true); if (std::error_code EC = MB.getError()) { llvm::errs() << "cannot open file " << Path << ": " << EC.message() << "\n"; diff --git a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp index 6a5646965df2cf..c5ccd64f116539 100644 --- a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp +++ b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp @@ -144,15 +144,18 @@ int main(int argc, const char *argv[]) { cl::HideUnrelatedOptions({&CXXMapCategory, &getColorCategory()}); cl::ParseCommandLineOptions(argc, argv, "LLVM C++ mangled name remapper\n"); - auto OldSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(OldSymbolFile); + auto OldSymbolBufOrError = + MemoryBuffer::getFileOrSTDIN(OldSymbolFile, /*IsText=*/true); if (!OldSymbolBufOrError) exitWithErrorCode(OldSymbolBufOrError.getError(), OldSymbolFile); - auto NewSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(NewSymbolFile); + auto NewSymbolBufOrError = + MemoryBuffer::getFileOrSTDIN(NewSymbolFile, /*IsText=*/true); if (!NewSymbolBufOrError) exitWithErrorCode(NewSymbolBufOrError.getError(), NewSymbolFile); - auto RemappingBufOrError = MemoryBuffer::getFileOrSTDIN(RemappingFile); + auto RemappingBufOrError = + MemoryBuffer::getFileOrSTDIN(RemappingFile, /*IsText=*/true); if (!RemappingBufOrError) exitWithErrorCode(RemappingBufOrError.getError(), RemappingFile); diff --git a/llvm/tools/yaml2obj/yaml2obj.cpp b/llvm/tools/yaml2obj/yaml2obj.cpp index b7f5356e22a9e6..4a060e1aad427f 100644 --- a/llvm/tools/yaml2obj/yaml2obj.cpp +++ b/llvm/tools/yaml2obj/yaml2obj.cpp @@ -130,7 +130,7 @@ int main(int argc, char **argv) { } ErrorOr> Buf = - MemoryBuffer::getFileOrSTDIN(Input); + MemoryBuffer::getFileOrSTDIN(Input, /*IsText=*/true); if (!Buf) return 1; diff --git a/llvm/utils/lit/lit/builtin_commands/cat.py b/llvm/utils/lit/lit/builtin_commands/cat.py index 37f55c0aef210b..6fb2152ef9332d 100644 --- a/llvm/utils/lit/lit/builtin_commands/cat.py +++ b/llvm/utils/lit/lit/builtin_commands/cat.py @@ -55,
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
llvmbot wrote: @llvm/pr-subscribers-clang-format Author: Sean Perry (perry-ca) Changes To support auto-conversion on z/OS text files need to be opened as text files. These changes will fix a number of LIT failures due to text files not being converted to the internal code page. - update a number of tools so they open the text files as text files - add support in the cat.py to open a text file as a text file (Windows will continue to treat all files as binary so new lines are handled correctly) - add env var definitions to enable auto-conversion in the lit config file. --- Full diff: https://github.com/llvm/llvm-project/pull/90128.diff 6 Files Affected: - (modified) clang/tools/clang-format/ClangFormat.cpp (+4-3) - (modified) llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp (+2-1) - (modified) llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp (+6-3) - (modified) llvm/tools/yaml2obj/yaml2obj.cpp (+1-1) - (modified) llvm/utils/lit/lit/builtin_commands/cat.py (+16-2) - (modified) llvm/utils/lit/lit/llvm/config.py (+7) ``diff diff --git a/clang/tools/clang-format/ClangFormat.cpp b/clang/tools/clang-format/ClangFormat.cpp index feb733fe3c9e0b..01f7c6047726e2 100644 --- a/clang/tools/clang-format/ClangFormat.cpp +++ b/clang/tools/clang-format/ClangFormat.cpp @@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool ErrorOnIncompleteFormat = false) { // On Windows, overwriting a file with an open file mapping doesn't work, // so read the whole file into memory when formatting in-place. ErrorOr> CodeOrErr = - !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName) -: MemoryBuffer::getFileOrSTDIN(FileName); + !OutputXML && Inplace + ? MemoryBuffer::getFileAsStream(FileName) + : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true); if (std::error_code EC = CodeOrErr.getError()) { errs() << EC.message() << "\n"; return true; @@ -558,7 +559,7 @@ static int dumpConfig() { // Read in the code in case the filename alone isn't enough to detect the // language. ErrorOr> CodeOrErr = -MemoryBuffer::getFileOrSTDIN(FileNames[0]); +MemoryBuffer::getFileOrSTDIN(FileNames[0], /*IsText=*/true); if (std::error_code EC = CodeOrErr.getError()) { llvm::errs() << EC.message() << "\n"; return 1; diff --git a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp index c3015d895230ea..40ee59c014b09f 100644 --- a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp +++ b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp @@ -95,7 +95,8 @@ static std::vector getSearchPaths(opt::InputArgList *Args, // Opens a file. Path has to be resolved already. (used for def file) std::unique_ptr openFile(const Twine &Path) { - ErrorOr> MB = MemoryBuffer::getFile(Path); + ErrorOr> MB = + MemoryBuffer::getFile(Path, /*IsText=*/true); if (std::error_code EC = MB.getError()) { llvm::errs() << "cannot open file " << Path << ": " << EC.message() << "\n"; diff --git a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp index 6a5646965df2cf..c5ccd64f116539 100644 --- a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp +++ b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp @@ -144,15 +144,18 @@ int main(int argc, const char *argv[]) { cl::HideUnrelatedOptions({&CXXMapCategory, &getColorCategory()}); cl::ParseCommandLineOptions(argc, argv, "LLVM C++ mangled name remapper\n"); - auto OldSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(OldSymbolFile); + auto OldSymbolBufOrError = + MemoryBuffer::getFileOrSTDIN(OldSymbolFile, /*IsText=*/true); if (!OldSymbolBufOrError) exitWithErrorCode(OldSymbolBufOrError.getError(), OldSymbolFile); - auto NewSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(NewSymbolFile); + auto NewSymbolBufOrError = + MemoryBuffer::getFileOrSTDIN(NewSymbolFile, /*IsText=*/true); if (!NewSymbolBufOrError) exitWithErrorCode(NewSymbolBufOrError.getError(), NewSymbolFile); - auto RemappingBufOrError = MemoryBuffer::getFileOrSTDIN(RemappingFile); + auto RemappingBufOrError = + MemoryBuffer::getFileOrSTDIN(RemappingFile, /*IsText=*/true); if (!RemappingBufOrError) exitWithErrorCode(RemappingBufOrError.getError(), RemappingFile); diff --git a/llvm/tools/yaml2obj/yaml2obj.cpp b/llvm/tools/yaml2obj/yaml2obj.cpp index b7f5356e22a9e6..4a060e1aad427f 100644 --- a/llvm/tools/yaml2obj/yaml2obj.cpp +++ b/llvm/tools/yaml2obj/yaml2obj.cpp @@ -130,7 +130,7 @@ int main(int argc, char **argv) { } ErrorOr> Buf = - MemoryBuffer::getFileOrSTDIN(Input); + MemoryBuffer::getFileOrSTDIN(Input, /*IsText=*/true); if (!Buf) return 1; diff --git a/llvm/utils/lit/lit/builtin_commands/cat.py b/llvm/utils/lit/lit/builtin_commands/cat.py index 37f55c0aef210b..6fb2152ef9332d 100644 --- a/llvm/utils/lit/lit/builtin_commands/cat.py +++ b/llvm/utils/lit/lit/builtin_commands/cat.py @@ -55,1
[clang] [llvm] [z/OS] treat text files as text files so auto-conversion is done (PR #90128)
https://github.com/perry-ca created https://github.com/llvm/llvm-project/pull/90128 To support auto-conversion on z/OS text files need to be opened as text files. These changes will fix a number of LIT failures due to text files not being converted to the internal code page. - update a number of tools so they open the text files as text files - add support in the cat.py to open a text file as a text file (Windows will continue to treat all files as binary so new lines are handled correctly) - add env var definitions to enable auto-conversion in the lit config file. >From 3c434710f062c011ade4030b0d79254ee5c6a1df Mon Sep 17 00:00:00 2001 From: Sean Perry Date: Thu, 25 Apr 2024 15:58:05 -0500 Subject: [PATCH] treat text files as text files so autoconversion is done --- clang/tools/clang-format/ClangFormat.cpp| 7 --- llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp | 3 ++- llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp | 9 ++--- llvm/tools/yaml2obj/yaml2obj.cpp| 2 +- llvm/utils/lit/lit/builtin_commands/cat.py | 18 -- llvm/utils/lit/lit/llvm/config.py | 7 +++ 6 files changed, 36 insertions(+), 10 deletions(-) diff --git a/clang/tools/clang-format/ClangFormat.cpp b/clang/tools/clang-format/ClangFormat.cpp index feb733fe3c9e0b..01f7c6047726e2 100644 --- a/clang/tools/clang-format/ClangFormat.cpp +++ b/clang/tools/clang-format/ClangFormat.cpp @@ -413,8 +413,9 @@ static bool format(StringRef FileName, bool ErrorOnIncompleteFormat = false) { // On Windows, overwriting a file with an open file mapping doesn't work, // so read the whole file into memory when formatting in-place. ErrorOr> CodeOrErr = - !OutputXML && Inplace ? MemoryBuffer::getFileAsStream(FileName) -: MemoryBuffer::getFileOrSTDIN(FileName); + !OutputXML && Inplace + ? MemoryBuffer::getFileAsStream(FileName) + : MemoryBuffer::getFileOrSTDIN(FileName, /*IsText=*/true); if (std::error_code EC = CodeOrErr.getError()) { errs() << EC.message() << "\n"; return true; @@ -558,7 +559,7 @@ static int dumpConfig() { // Read in the code in case the filename alone isn't enough to detect the // language. ErrorOr> CodeOrErr = -MemoryBuffer::getFileOrSTDIN(FileNames[0]); +MemoryBuffer::getFileOrSTDIN(FileNames[0], /*IsText=*/true); if (std::error_code EC = CodeOrErr.getError()) { llvm::errs() << EC.message() << "\n"; return 1; diff --git a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp index c3015d895230ea..40ee59c014b09f 100644 --- a/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp +++ b/llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp @@ -95,7 +95,8 @@ static std::vector getSearchPaths(opt::InputArgList *Args, // Opens a file. Path has to be resolved already. (used for def file) std::unique_ptr openFile(const Twine &Path) { - ErrorOr> MB = MemoryBuffer::getFile(Path); + ErrorOr> MB = + MemoryBuffer::getFile(Path, /*IsText=*/true); if (std::error_code EC = MB.getError()) { llvm::errs() << "cannot open file " << Path << ": " << EC.message() << "\n"; diff --git a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp index 6a5646965df2cf..c5ccd64f116539 100644 --- a/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp +++ b/llvm/tools/llvm-cxxmap/llvm-cxxmap.cpp @@ -144,15 +144,18 @@ int main(int argc, const char *argv[]) { cl::HideUnrelatedOptions({&CXXMapCategory, &getColorCategory()}); cl::ParseCommandLineOptions(argc, argv, "LLVM C++ mangled name remapper\n"); - auto OldSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(OldSymbolFile); + auto OldSymbolBufOrError = + MemoryBuffer::getFileOrSTDIN(OldSymbolFile, /*IsText=*/true); if (!OldSymbolBufOrError) exitWithErrorCode(OldSymbolBufOrError.getError(), OldSymbolFile); - auto NewSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(NewSymbolFile); + auto NewSymbolBufOrError = + MemoryBuffer::getFileOrSTDIN(NewSymbolFile, /*IsText=*/true); if (!NewSymbolBufOrError) exitWithErrorCode(NewSymbolBufOrError.getError(), NewSymbolFile); - auto RemappingBufOrError = MemoryBuffer::getFileOrSTDIN(RemappingFile); + auto RemappingBufOrError = + MemoryBuffer::getFileOrSTDIN(RemappingFile, /*IsText=*/true); if (!RemappingBufOrError) exitWithErrorCode(RemappingBufOrError.getError(), RemappingFile); diff --git a/llvm/tools/yaml2obj/yaml2obj.cpp b/llvm/tools/yaml2obj/yaml2obj.cpp index b7f5356e22a9e6..4a060e1aad427f 100644 --- a/llvm/tools/yaml2obj/yaml2obj.cpp +++ b/llvm/tools/yaml2obj/yaml2obj.cpp @@ -130,7 +130,7 @@ int main(int argc, char **argv) { } ErrorOr> Buf = - MemoryBuffer::getFileOrSTDIN(Input); + MemoryBuffer::getFileOrSTDIN(Input, /*IsText=*/true); if (!Buf) return 1; diff --git a/llvm/utils/lit/lit/builtin_commands/cat.py b/llvm/utils/lit/lit/builtin_commands/cat.py index 37f55c