To demonstrate how this warning is useful, I used my reference implementation.
When I try `pip install`, I found these issues soon.
https://bugs.python.org/issue43214 (Open pth file with locale-encoding)
https://github.com/pypa/pip/pull/9608 (Not a real bug, but open JSON
file with
In the documentation (not sure whether it should be the documentation
for "open" or for encoding), include at least a link to instructions
on how to (try to) verify that your codebase is using the encoding
parameter properly. Those instructions would say something like "Add
the following lines to
On Sat, Feb 13, 2021 at 4:53 AM Jim J. Jewett wrote:
>
> Offering encoding="locale" (or open.locale or ... ) instead of a long
> function call using False (locale.getpreferredencoding(False)) seems like a
> win for Explicit is Better Than Implicit. It would then be possible to say
> "yeah,
Offering encoding="locale" (or open.locale or ... ) instead of a long function
call using False (locale.getpreferredencoding(False)) seems like a win for
Explicit is Better Than Implicit. It would then be possible to say "yeah,
locale really is what I meant".
Err... unless the charset
On Fri, Feb 12, 2021 at 12:45 PM Jim J. Jewett wrote:
>
> On Thu, Feb 11, 2021 at 7:35 PM Inada Naoki wrote:
>
> > The PEP helps developers living on UTF-8 locale to find missing
> > `encoding="utf-8"` bug.
> > This type of bug is very common, and many Windows users are suffered
> > by the bug
On Fri, Feb 12, 2021 at 12:28 PM Jim J. Jewett wrote:
>
> (I apologize if my summaries distort what Inada Naoki
> explained.)
>
> He said that some people use the default None when they really want
> either UTF-8 or ASCII.
Yes. Even Python core developers.
For example:
On Thu, Feb 11, 2021 at 7:35 PM Inada Naoki wrote:
> The PEP helps developers living on UTF-8 locale to find missing
> `encoding="utf-8"` bug.
> This type of bug is very common, and many Windows users are suffered
> by the bug when reading JSON, YAML, TOML, Markdown, or any other UTF-8
> files.
(I apologize if my summaries distort what Inada Naoki
explained.)
He said that some people use the default None when they really want
either UTF-8 or ASCII.
My concern is that the warning will be a false alarm if they really do
need whatever locale returns, and that case may still be common.
On 2/11/21, Inada Naoki wrote:
>
> There is little difference between `encoding=None` and
> `encoding=locale.getpreferredencoding(False)`. The difference is:
>
> * When Python is using Windows, and
> * When when the file is console, and
> * (for open()) When PYTHONLEGACYWINDOWSSTDIO is set
> *
On Thu, Feb 11, 2021 at 4:44 PM Jim J. Jewett wrote:
>
> The PEP helps when the locale is ASCII or C, but that isn't enforced in
> actual files. I am confident that this is a frequent problem for packages
> downloaded from mostly-English sites, including many software repositories.
>
The PEP
On Fri, Feb 12, 2021 at 5:18 AM Jim J. Jewett wrote:
>
> Inada Naoki wrote:
>
> > Default encoding is used for:
>
> > a. Really need to use locale specific encoding
> > b. UTF-8 (bug. not work on Windows)
> > c. ASCII (not a bug, but slow on Windows)
>
> > I assume most usages are (b) and (c).
On Fri, Feb 12, 2021 at 6:34 AM Paul Moore wrote:
>
> On Thu, 11 Feb 2021 at 21:05, Jim J. Jewett wrote:
> >
> > Who will benefit from this new warning?
> >
> > Is this basically just changing builtins.open by adding:
> >
> > if encoding is None and sys.flags.encoding_warning: # and not
On Thu, 11 Feb 2021 at 21:05, Jim J. Jewett wrote:
>
> Who will benefit from this new warning?
>
> Is this basically just changing builtins.open by adding:
>
> if encoding is None and sys.flags.encoding_warning: # and not Android and
> not -X utf8 ?
>
Who will benefit from this new warning?
Is this basically just changing builtins.open by adding:
if encoding is None and sys.flags.encoding_warning: # and not Android and
not -X utf8 ?
warnings.warn(EncodingWarning("Are you sure you want locale instead of
utf-8?"))
Even for the
Inada Naoki wrote:
> Default encoding is used for:
> a. Really need to use locale specific encoding
> b. UTF-8 (bug. not work on Windows)
> c. ASCII (not a bug, but slow on Windows)
> I assume most usages are (b) and (c). This PEP can reduce them soon.
Is this just an assumption, based on
On Thu, Feb 11, 2021 at 4:44 PM Jim J. Jewett wrote:
>
> I just reread PEP 597, then re-reread the Rationale.
>
Do you read current PEP 597, or old PEP 597 in discuss.python.org?
--
Inada Naoki
___
Python-Dev mailing list -- python-dev@python.org
On Tue, Feb 9, 2021 at 9:51 PM Paul Moore wrote:
> * Realistically, I'd be surprised if developers actually use such a
> tool. If they were likely to do so, they could probably just as easily
> locate all the uses of open() in their code, and check that way. So
> I'm not sure this proposal is
I just reread PEP 597, then re-reread the Rationale.
The PEP helps when the locale is ASCII or C, but that isn't enforced in actual
files. I am confident that this is a frequent problem for packages downloaded
from mostly-English sites, including many software repositories.
It does not seem
On Wed, Feb 10, 2021 at 11:58 PM Anders Munch wrote:
>
> On Wed, Feb 10, 2021 at 1:46 AM Anders Munch wrote:
> >> How about swapping around "locale" and None?
> Inada Naoki wrote:
> >
> > I thought it, but it may not work. Consider about function like this:
> >
> > ```
> > def read_text(self,
On Wed, 10 Feb 2021 at 16:06, Anders Munch wrote:
>
> Paul Moore [mailto:p.f.mo...@gmail.com]: wrote:
> > On Wed, 10 Feb 2021 at 14:33, Anders Munch wrote:
> >> The idea is to make is so that working code only needs to change once,
> >> even when supporting multiple Python versions.
> >> That
On 2/10/2021 10:29 AM, Paul Moore wrote:
On Wed, 10 Feb 2021 at 14:33, Anders Munch wrote:
The idea is to make is so that working code only needs to change once, even
when supporting multiple Python versions.
That one change is to add either an explicit encoding=None (for
Paul Moore [mailto:p.f.mo...@gmail.com]: wrote:
> On Wed, 10 Feb 2021 at 14:33, Anders Munch wrote:
>> The idea is to make is so that working code only needs to change once, even
>> when supporting multiple Python versions.
>> That one change is to add either an explicit encoding=None (for
>>
Inada Naoki [mailto:songofaca...@gmail.com] wrote:
> There are several ways:
> * encoding="latin1" -- This is the best. Works perfectly.
> * Don't touch -- You don't need to enable EncodingWarning.
> [...]
I'm replying to Victor's statement that ``encoding="utf8" is backward
compatible´´.
If
On Wed, 10 Feb 2021 at 14:33, Anders Munch wrote:
>
> On Tue, 9 Feb 2021 at 16:52, Anders Munch wrote:
> >> How about swapping around "locale" and None? That is, make "locale" the
> >> new default that emits a warning, and encoding=None emits no warning.
> >> That has the advantage that old
On Wed, Feb 10, 2021 at 1:46 AM Anders Munch wrote:
>> How about swapping around "locale" and None?
Inada Naoki wrote:
>
> I thought it, but it may not work. Consider about function like this:
>
> ```
> def read_text(self, encoding=None):
> with open(self._filename, encoding=encoding) as
On Tue, 9 Feb 2021 at 16:52, Anders Munch wrote:
>> How about swapping around "locale" and None? That is, make "locale" the new
>> default that emits a warning, and encoding=None emits no warning. That has
>> the advantage that old code can be updated to say encoding=None, and then it
>>
On Wed, Feb 10, 2021 at 11:14 PM Anders Munch wrote:
>
> This program runs just fine on 3.8.7 Windows, against a file.txt that
> contains latin-1 text:
>
> with open('file.txt', 'rt') as f:
> print(f.read())
>
> But if I change it to this:
>
> with open('file.txt', 'rt', encoding='utf-8') as
Victor Stinner [mailto:vstin...@python.org] wrote:
> encoding="utf8" is backward compatible and is likely to fix encoding bugs
> when the locale encoding is not UTF-8
This program runs just fine on 3.8.7 Windows, against a file.txt that contains
latin-1 text:
with open('file.txt', 'rt') as f:
On Wed, Feb 10, 2021 at 5:00 PM Paul Moore wrote:
>
> Let's just assume until you can convince me that setting UTF-8 mode
> globally is a good idea,
Oh, you misunderstood me. My proposal is not setting UTF-8 mode globally.
What I proposed is setting UTF-8 mode per env (e.g. installation,
venv,
On Wed, 10 Feb 2021 at 01:29, Inada Naoki wrote:
> Note that many Python users don't use consoles.
I never suggested that they did. There's a GUI for setting user-level
and system-level environment variables. And whoever is introducing the
user to Python can show them how to set the necessary
On Tue, Feb 9, 2021 at 11:29 PM Terry Reedy wrote:
>
> On 2/9/2021 8:28 PM, Inada Naoki wrote:
>
> > Note that many Python users don't use consoles.
>
> Those of use who do may find it hard to imagine just how easy we have
> made computing.
>
> My daughter minored in Computer Science about 6
On 2/9/2021 8:28 PM, Inada Naoki wrote:
Note that many Python users don't use consoles.
Those of use who do may find it hard to imagine just how easy we have
made computing.
My daughter minored in Computer Science about 6 years ago. She never
saw Command Prompt until the summer after her
On Wed, Feb 10, 2021 at 5:50 AM Paul Moore wrote:
>
> On Tue, 9 Feb 2021 at 16:54, Inada Naoki wrote:
> >
> > On Tue, Feb 9, 2021 at 9:31 PM Paul Moore wrote:
> > >
> > > Personally, I'm not at all keen on the idea of making users always
> > > specify encoding in the first place, even if it's
On Tue, 9 Feb 2021 at 16:54, Inada Naoki wrote:
>
> On Tue, Feb 9, 2021 at 9:31 PM Paul Moore wrote:
> >
> > Personally, I'm not at all keen on the idea of making users always
> > specify encoding in the first place, even if it's "just for the
> > transition".
>
> I agree with you. But as I
On Tue, Feb 9, 2021 at 5:51 PM Anders Munch wrote:
> Victor Stinner [mailto:vstin...@python.org] wrote:
> > The warning can explicitly suggest to use encoding="utf8", it should work
> > in almost all cases.
>
> The warning should also explain how to get backwards-compatible behaviour,
> i.e.
On Wed, Feb 10, 2021 at 1:46 AM Anders Munch wrote:
>
>
> Inada Naoki wrote:
> > This warning is opt-in warning like BytesWarning.
>
> What use is a warning that no-one sees?
At least, I see.
We can fix stdlib and tests first, and fix some major tools too.
After that, `encoding="locale"`
On Tue, 9 Feb 2021 at 16:52, Anders Munch wrote:
> How about swapping around "locale" and None? That is, make "locale" the new
> default that emits a warning, and encoding=None emits no warning. That has
> the advantage that old code can be updated to say encoding=None, and then it
> will
On Tue, Feb 9, 2021 at 9:31 PM Paul Moore wrote:
>
> Personally, I'm not at all keen on the idea of making users always
> specify encoding in the first place, even if it's "just for the
> transition".
I agree with you. But as I wrote in the PEP, omitted encoding caused
much troubles already.
Victor Stinner [mailto:vstin...@python.org] wrote:
> The warning can explicitly suggest to use encoding="utf8", it should work in
> almost all cases.
The warning should also explain how to get backwards-compatible behaviour, i.e.
suggest encoding="locale".
Inada Naoki wrote:
> This warning
On Tue, 9 Feb 2021 at 16:28, Inada Naoki wrote:
>
> On Wed, Feb 10, 2021 at 1:19 AM Paul Moore wrote:
> >
> > But people who currently don't specify the encoding, and *don't* have
> > any issue (because the system locale is correct) will be getting told
> > to introduce a bug into their code, if
On Wed, Feb 10, 2021 at 1:19 AM Paul Moore wrote:
>
> But people who currently don't specify the encoding, and *don't* have
> any issue (because the system locale is correct) will be getting told
> to introduce a bug into their code, if they follow that advice :-(
>
This warning is opt-in
But people who currently don't specify the encoding, and *don't* have
any issue (because the system locale is correct) will be getting told
to introduce a bug into their code, if they follow that advice :-(
Paul
On Tue, 9 Feb 2021 at 16:03, Victor Stinner wrote:
>
> On Tue, Feb 9, 2021 at 1:31
On Tue, Feb 9, 2021 at 1:31 PM Paul Moore wrote:
> If we can't provide a good recommendation
> to the user on what to do, we shouldn't be warning them that what they
> are currently doing is wrong.
The warning can explicitly suggest to use encoding="utf8", it should
work in almost all cases.
On Tue, 9 Feb 2021 at 11:55, Inada Naoki wrote:
> I think only we can do is documenting the option like this:
>
> """
> EncodingWarning is warning to find missing encoding="utf-8" option. It
> is common pitfall that many Windows user
> Don't try to fix them if you need to use locale specific
On Tue, Feb 9, 2021 at 7:23 PM Victor Stinner wrote:
>
> I recall that something like 1 year ago, I basically tried to
> implement something like your PEP, to see if the stdlib calls open()
> without specifying an encoding. There were so many warnings, that the
> output was barely readable.
>
>
On Sat, Feb 6, 2021 at 3:26 PM Inada Naoki wrote:
> I changed my mind. Since there is no plan to change the default
> encoding for now,
> no need to encourage `encoding="locale"` soon.
>
> Until users can drop Python 3.9 support, they can use EncodingWarning
> only for finding missing
I send a pull request https://github.com/python/peps/pull/1799
* Add Backward/Forward Compatibility section
* Add How to teach this section
* Remove io.LOCALE_ENCODING constant
--
Inada Naoki
___
Python-Dev mailing list -- python-dev@python.org
To
On Tue, Feb 2, 2021 at 1:40 PM Inada Naoki wrote:
>
> On Tue, Feb 2, 2021 at 12:23 AM Victor Stinner wrote:
> >
> >
> > > Add ``io.LOCALE_ENCODING = "locale"`` constant too. This constant can
> > > be used to avoid confusing ``LookupError: unknown encoding: locale``
> > > error when the code is
On Tue, Feb 2, 2021 at 8:16 PM Victor Stinner wrote:
>
> > > I understand that encoding=locale.get_locale_encoding() would be
> > > different from encoding="locale":
> > > encoding=locale.get_locale_encoding() doesn't call
> > > os.device_encoding(), right?
> > >
> >
> > Yes.
>
> Would it be
On Tue, Feb 2, 2021 at 5:40 AM Inada Naoki wrote:
> > In Python 3.10, I added _locale._get_locale_encoding() function which
> > is exactly what the encoding used by open() when no encoding is
> > specified (encoding=None) and when os.device_encoding(fd) returns
> > None. See
On Tue, Feb 2, 2021 at 12:23 AM Victor Stinner wrote:
>
> Hi Inada-san,
>
> I followed the discussions on your different PEP and I like overall
> your latest PEP :-) I have some minor remarks.
>
> On Mon, Feb 1, 2021 at 6:55 AM Inada Naoki wrote:
> > The warning is disabled by default. New ``-X
Hi Inada-san,
I followed the discussions on your different PEP and I like overall
your latest PEP :-) I have some minor remarks.
On Mon, Feb 1, 2021 at 6:55 AM Inada Naoki wrote:
> The warning is disabled by default. New ``-X warn_encoding``
> command-line option and ``PYTHONWARNENCODING``
52 matches
Mail list logo
