[Bug 1751051] Re: UnicodeEncodeError when creating user with non-ascii chars
This bug is believed to be fixed in cloud-init in 18.2. If this is still a problem for you, please make a comment and set the state back to New Thank you. ** Changed in: cloud-init Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1751051 Title: UnicodeEncodeError when creating user with non-ascii chars To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1751051/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1751051] Re: UnicodeEncodeError when creating user with non-ascii chars
This bug was fixed in the package livecd-rootfs - 2.515 --- livecd-rootfs (2.515) bionic; urgency=medium * Set the default locale to C.UTF-8 in all server and cloud images. (LP: #1751051, #1759003) -- Michael Hudson-Doyle Tue, 27 Mar 2018 09:59:02 +1300 ** Changed in: livecd-rootfs (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1751051 Title: UnicodeEncodeError when creating user with non-ascii chars To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1751051/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1751051] Re: UnicodeEncodeError when creating user with non-ascii chars
** Changed in: livecd-rootfs (Ubuntu) Status: New => In Progress ** Changed in: livecd-rootfs (Ubuntu) Assignee: (unassigned) => Michael Hudson-Doyle (mwhudson) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1751051 Title: UnicodeEncodeError when creating user with non-ascii chars To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1751051/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1751051] Re: UnicodeEncodeError when creating user with non-ascii chars
** Branch linked: lp:~mwhudson/livecd-rootfs/default-LANG-C.UTF-8 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1751051 Title: UnicodeEncodeError when creating user with non-ascii chars To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1751051/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1751051] Re: UnicodeEncodeError when creating user with non-ascii chars
** Changed in: cloud-init Status: Confirmed => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1751051 Title: UnicodeEncodeError when creating user with non-ascii chars To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1751051/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1751051] Re: UnicodeEncodeError when creating user with non-ascii chars
** Tags added: id-5a9fa29eda1dc1b22307ed30 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1751051 Title: UnicodeEncodeError when creating user with non-ascii chars To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1751051/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1751051] Re: UnicodeEncodeError when creating user with non-ascii chars
We need to make sure that the default locale when booting a subiquity image is C.UTF-8, not C. This probably needs fixing in livecd-rootfs and I don't think there are any code changes for subiquity. ** Project changed: subiquity => livecd-rootfs ** Project changed: livecd-rootfs => livecd-rootfs (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1751051 Title: UnicodeEncodeError when creating user with non-ascii chars To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1751051/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1751051] Re: UnicodeEncodeError when creating user with non-ascii chars
This bug was fixed in the package cloud-init - 18.1-5-g40e77380-0ubuntu1 --- cloud-init (18.1-5-g40e77380-0ubuntu1) bionic; urgency=medium * New upstream snapshot. - GCE: fix reading of user-data that is not base64 encoded. (LP: #1752711) - doc: fix chef install from apt packages example in RTD. - Implement puppet 4 support [Romanos Skiadas] (LP: #1446804) - subp: Fix subp usage with non-ascii characters when no system locale. (LP: #1751051) - salt: configure grains in grains file rather than in minion config. [Daniel Wallace] -- Chad Smith Thu, 01 Mar 2018 15:47:04 -0700 ** Changed in: cloud-init (Ubuntu) Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1751051 Title: UnicodeEncodeError when creating user with non-ascii chars To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1751051/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1751051] Re: UnicodeEncodeError when creating user with non-ascii chars
** Merge proposal linked: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/340252 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1751051 Title: UnicodeEncodeError when creating user with non-ascii chars To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1751051/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1751051] Re: UnicodeEncodeError when creating user with non-ascii chars
** Merge proposal linked: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/338586 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1751051 Title: UnicodeEncodeError when creating user with non-ascii chars To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1751051/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1751051] Re: UnicodeEncodeError when creating user with non-ascii chars
I think the issue is: a.) there is no default locale set in the subiquity installed system. b.) python3 subprocess is doing a 'decode' for each argument in the command list. python2 default encoding *is* supposed to be based on the environment [1], but python3 default encoding is not. python3 is supposed to be utf-8. In the trace above we are down in C code where it is clearly doing 'ascii' encoding. [1] https://docs.python.org/2/library/sys.html?highlight=getdefaultencoding#sys.getdefaultencoding [2] https://docs.python.org/3/library/stdtypes.html?highlight=decode#str.encode You can see the problem generally below. I only use 'json' as a convienent way to pass in utf-8 characters. You can see that either unset LANG or LANG=C causes the issue. I guess I never thought that subprocess would be converting an argument list of strings to bytes. That does make some sense. So I think there are actually two changes: a.) subiquity (via either curtin or cloud-init) should be setting a utf-8 default locale (all ubuntu generally do that). I'm not sure why the image being installed didnt have one set. b.) cloud-init's subp should probably just do the conversion to bytes of whatever it gets as an argument list for the command, and always assume that strings are to be encoded as utf-8. $ cat go.py #!/usr/bin/python3 import json, subprocess, sys cmd = json.loads(sys.argv[1]) print("cmd=%s" % [x.encode("utf-8") for x in cmd]) subprocess.check_call(cmd) # my default lang is en_US.utf-8 $ ./go.py '["echo", "Andr\u00e9 DSilva"]' cmd=[b'echo', b'Andr\xc3\xa9 DSilva'] André DSilva $ LANG=en_US.utf-8 ./go.py '["echo", "Andr\u00e9 DSilva"]' cmd=[b'echo', b'Andr\xc3\xa9 DSilva'] André DSilva $ env -u LANG ./go.py '["echo", "Andr\u00e9 DSilva"]' cmd=[b'echo', b'Andr\xc3\xa9 DSilva'] Traceback (most recent call last): File "./go.py", line 5, in subprocess.check_call(cmd) File "/usr/lib/python3.6/subprocess.py", line 286, in check_call retcode = call(*popenargs, **kwargs) File "/usr/lib/python3.6/subprocess.py", line 267, in call with Popen(*popenargs, **kwargs) as p: File "/usr/lib/python3.6/subprocess.py", line 709, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child restore_signals, start_new_session, preexec_fn) UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 4: ordinal not in range(128) $ LANG=C ./go.py '["echo", "Andr\u00e9 DSilva"]' cmd=[b'echo', b'Andr\xc3\xa9 DSilva'] Traceback (most recent call last): File "./go.py", line 5, in subprocess.check_call(cmd) File "/usr/lib/python3.6/subprocess.py", line 286, in check_call retcode = call(*popenargs, **kwargs) File "/usr/lib/python3.6/subprocess.py", line 267, in call with Popen(*popenargs, **kwargs) as p: File "/usr/lib/python3.6/subprocess.py", line 709, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child restore_signals, start_new_session, preexec_fn) UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 4: ordinal not in range(128) ** Changed in: cloud-init Status: New => Confirmed ** Changed in: cloud-init Importance: Undecided => Medium ** Also affects: cloud-init (Ubuntu) Importance: Undecided Status: New ** Changed in: cloud-init (Ubuntu) Status: New => Confirmed ** Also affects: subiquity Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1751051 Title: UnicodeEncodeError when creating user with non-ascii chars To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1751051/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs