On Mon, Jan 4, 2016, at 06:09 PM, Tom Gall wrote: > On Mon, Jan 4, 2016 at 4:44 PM, Ryan Lovelett > <[email protected]> wrote: > > I would suggest setting the environment variables and re-running but > > that is just my opinion. > > > > I went through many of the same issues on Arch as well. Arch uses Python > > 2.7.11 and Python 3.5.1; with 3.5.1 being default. > > Well point I was suggesting is that with the use of python-config the > version of python, include and libs location is easily determined.[4] > Looking at CMakeCache.txt in > build/Ninja-ReleaseAssert/swift-linux-x86_64, it's pretty obvious the > build environment isn't using it but seems to respect the PYTHON env > variable. I find this curious.
To me that sounds exactly what those bugs I talked about are trying to solve; the system selecting the wrong Python from the environment. https://bugs.swift.org/browse/SR-14?focusedCommentId=10432&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-10432 Of course, you are right I digress. > > > I can _maybe_ give you a clue on the lldb issue. Check out SR-14 [1] > > which links to the upstream LLDB Bug 25744 [2]. That report seems to > > indicate there is a known bug compiling on Gentoo. They've even created > > a specific bug Bug 25866 [3] which Bug 25744 depends on to track fixing > > it in Gentoo. > > Fixing CMakeCache.txt by hand to use the system's python2.7 default > lldb builds successfully on gentoo, no changes necessary. > > Still a good data point would be to switch my default python over to > 3.x and see what kind of trouble i can find. Easy enough to do. > > > Unfortunately, it seems that the experimental patch submitted there > > (which I use to build on Arch) seems to only work on Arch. However, > > maybe your just the person to fix that bug! Anyways food for thought. > > Happy compiling. 🍻 > > > [1] https://bugs.swift.org/browse/SR-14 > > [2] https://llvm.org/bugs/show_bug.cgi?id=25744 > > [3] https://llvm.org/bugs/show_bug.cgi?id=25866 > > [4] https://www.python.org/dev/peps/pep-0394/ > > Back to the original issue. Do you want to require unicode locales? tl;dr If you hard coded it I certainly would not object and I doubt it would break anything. If you think its necessary than I say: go for it! In my opinion you've already illustrated that your locale setup is in need of tweaking and that Python is behaving as designed. Therefore, to me, hard-coding the encoding to be UTF-8 is a work around for an improperly configured system. But I'm not in charge of anything. Like Jordan said those specific files will always be UTF-8 (because they are testing unicode encoding in Swift) so it should not break anything to force the encoding as UTF-8. > > Cheers! > > > On Mon, Jan 4, 2016, at 05:29 PM, Tom Gall wrote: > >> Hi Ryan, > >> > >> In my case I'm on Python 2.7. Your comment is interesting as I was > >> just tracking down why python-config --libs and python-config > >> --includes doesn't seem to be used to determine what is the system > >> python install. I was just starting to trace through the build tool to > >> figure out how build-script works it's initial magic. > >> > >> In my case I have both python 3.4 and 2.7 installed but python 2.7 is > >> the system default. This ends up causing some interesting brand of > >> hurt, when trying to build swift's lldb. (Test Case > >> 'TestNSTimer.test_timerTickOnce' is freezing so was looking to debug > >> that) > >> > >> To answer some of your questions: > >> > >> tgall@mars ~/swift $ locale > >> > >> LANG=en_US > >> LC_CTYPE=C > >> LC_NUMERIC="en_US" > >> LC_TIME="en_US" > >> LC_COLLATE="en_US" > >> LC_MONETARY="en_US" > >> LC_MESSAGES="en_US" > >> LC_PAPER="en_US" > >> LC_NAME="en_US" > >> LC_ADDRESS="en_US" > >> LC_TELEPHONE="en_US" > >> LC_MEASUREMENT="en_US" > >> LC_IDENTIFICATION="en_US" > >> LC_ALL= > >> > >> >>> import sys > >> >>> x=sys.getfilesystemencoding() > >> >>> print x > >> ANSI_X3.4-1968 > >> > >> That explains some things :-) > >> > >> On Mon, Jan 4, 2016 at 4:12 PM, Ryan Lovelett via swift-dev > >> <[email protected]> wrote: > >> > On Mon, Jan 4, 2016, at 03:40 PM, Tom Gall via swift-dev wrote: > >> >> Building with: ./swift/utils/build-script -R -t --foundation > >> >> > >> >> on Linux (gentoo amd64) fails with > >> >> > >> >> + /usr/bin/cmake --build > >> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64 -- -j4 > >> >> SwiftUnitTests > >> >> > >> >> [6/29] Generating UnicodeGraphemeBreakTest.cpp from > >> >> UnicodeGraphemeBreakTest.cpp.gyb with ptr size = 8 > >> >> > >> >> FAILED: cd /home/tgall/swift/swift/unittests/Basic && /usr/bin/cmake > >> >> -E make_directory > >> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8 > >> >> && /home/tgall/swift/swift/utils/gyb --test > >> >> -DunicodeGraphemeBreakPropertyFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakProperty.txt > >> >> -DunicodeGraphemeBreakTestFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakTest.txt > >> >> -DCMAKE_SIZEOF_VOID_P=8 -o > >> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp > >> >> UnicodeGraphemeBreakTest.cpp.gyb && /usr/bin/cmake -E > >> >> copy_if_different > >> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp > >> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp > >> >> && /usr/bin/cmake -E remove > >> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp > >> >> > >> >> Traceback (most recent call last): > >> >> > >> >> File "/home/tgall/swift/swift/utils/gyb", line 3, in <module> > >> >> gyb.main() > >> >> File "/home/tgall/swift/swift/utils/gyb.py", line 1071, in main > >> >> args.target.write(executeTemplate(ast, args.line_directive, > >> >> **bindings)) > >> >> File "/home/tgall/swift/swift/utils/gyb.py", line 974, in > >> >> executeTemplate > >> >> ast.execute(executionContext) > >> >> File "/home/tgall/swift/swift/utils/gyb.py", line 591, in execute > >> >> x.execute(context) > >> >> File "/home/tgall/swift/swift/utils/gyb.py", line 667, in execute > >> >> result = eval(self.code, context.localBindings) > >> >> File > >> >> > >> >> "/home/tgall/swift/swift/unittests/Basic/UnicodeGraphemeBreakTest.cpp.gyb", > >> >> line 23, in <module> > >> >> > >> >> get_grapheme_cluster_break_tests_as_UTF8(unicodeGraphemeBreakTestFile) > >> >> File "/home/tgall/swift/swift/utils/GYBUnicodeDataUtils.py", line > >> >> 553, in get_grapheme_cluster_break_tests_as_UTF8 > >> >> for line in f: > >> >> File "/usr/lib64/python2.7/codecs.py", line 687, in next > >> >> return self.reader.next() > >> >> File "/usr/lib64/python2.7/codecs.py", line 618, in next > >> >> line = self.readline() > >> >> File "/usr/lib64/python2.7/codecs.py", line 533, in readline > >> >> data = self.read(readsize, firstline=True) > >> >> File "/usr/lib64/python2.7/codecs.py", line 480, in read > >> >> newchars, decodedbytes = self.decode(data, self.errors) > >> >> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position > >> >> 0: ordinal not in range(128) > >> >> [6/29] Building CXX object > >> >> unittests/Parse/CMakeFiles/SwiftParseTests.dir/LexerTests.cpp.o > >> >> ninja: build stopped: subcommand failed. > >> >> > >> >> Ah yes ... the joys of python stack dumps... anyway, tracing this a > >> >> bit: > >> >> > >> >> in swift/utils/GYBUnicodeDataUtils.py there is: > >> >> > >> >> with codecs.open(grapheme_break_test_file_name, > >> >> encoding=sys.getfilesystemencoding(), errors='strict') as f: > >> >> > >> > > >> > I wrote that code and patch (see: > >> > https://github.com/apple/swift/commit/7dbb4127f55022bca7b191d448652b5decf8626e). > >> > The change was in service of adding Python 3 support to GYB. So first of > >> > all let me say: I'm sorry. 😏 > >> > > >> > Open up your python interpreter and figure out what your filesystem is > >> > reporting its encoding to be (e.g., `sys.getfilesystemencoding()`). On > >> > OS X and my copy of Arch linux it reports `'utf-8'` which is why it > >> > doesn't have an issue. Worst case scenario we can just force it to be > >> > `with codecs.open(grapheme_break_test_file_name, encoding='utf-8', > >> > errors='strict') as f:` but I went with the filesystem encoding because > >> > hopefully it is always UTF-8. > >> > > >> >> It appears to be our offending bit of python code. Now my unicode & > >> >> python foo isn't the strongest, but if I change what is passed as > >> >> encoding to : encoding='utf-8', the swift testcases seem to run quite > >> >> a bit better and end up reporting : > >> >> > >> >> Testing Time: 65.82s > >> >> Expected Passes : 1748 > >> >> Expected Failures : 83 > >> >> Unsupported Tests : 585 > >> >> -- check-swift-linux-x86_64 finished -- > >> >> --- Finished tests for swift --- > >> >> > >> >> Question is, is that little fix the 'right thing' (TM) ? If so happy > >> >> to submit this as my first 'lame' patch. > >> >> > >> >> Thanks > >> >> > >> >> -- > >> >> Regards, > >> >> Tom > >> >> > >> >> "Where's the kaboom!? There was supposed to be an earth-shattering > >> >> kaboom!" Marvin Martian > >> >> Director, Linaro Mobile Group > >> >> Tech Lead, GPGPU > >> >> Linaro.org │ Open source software for ARM SoCs > >> >> irc: tgall_foo | skype : tom_gall > >> >> _______________________________________________ > >> >> swift-dev mailing list > >> >> [email protected] > >> >> https://lists.swift.org/mailman/listinfo/swift-dev > >> > _______________________________________________ > >> > swift-dev mailing list > >> > [email protected] > >> > https://lists.swift.org/mailman/listinfo/swift-dev > >> > >> > >> > >> -- > >> Regards, > >> Tom > >> > >> "Where's the kaboom!? There was supposed to be an earth-shattering > >> kaboom!" Marvin Martian > >> Director, Linaro Mobile Group > >> Tech Lead, GPGPU > >> Linaro.org │ Open source software for ARM SoCs > >> irc: tgall_foo | skype : tom_gall > > > > -- > Regards, > Tom > > "Where's the kaboom!? There was supposed to be an earth-shattering > kaboom!" Marvin Martian > Director, Linaro Mobile Group > Tech Lead, GPGPU > Linaro.org │ Open source software for ARM SoCs > irc: tgall_foo | skype : tom_gall _______________________________________________ swift-dev mailing list [email protected] https://lists.swift.org/mailman/listinfo/swift-dev
