Re: [webkit-dev] Moving to Python 3

[Philippe Normand]

Hi folks,

Now that Catalina is released, can we move on to some of the proposed
changes discussed in this thread?

Philippe

On Fri, 2019-07-12 at 12:18 -0700, Jonathan Bedard wrote:
> Hello WebKit developers,
> 
> Now that the Catalina developer seeds are available, it is official
> that the new Mac developer tools come with Python 3. As a result, we
> need to continue the ongoing discussion about migrating our Python
> 2.7 scripts to Python 3.
> 
> I propose that, over the next 9 months, we do the following:
> 
> 1. Make any no-cost Python 3 compatibility changes, in particular
> - print foo -> print(foo)
> - import .foo -> import webkitpy.foo
> 2. Convert any scripts not used in automation to Python 3 ASAP
> (scripts like bisect-builds, block-spammers, compare-results)
> 3. Make most Python 3 compatibility changes which sacrifice
> efficiency, subject to a case-by-case audit. These would be things
> like:
> - dict.iteritems() -> dict.items()
> - dict.items() -> list(dict.items())
> 4. Install Python 3 on macOS Sierra and Mojave bots
> 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts
> like clean-webkit, merge-results-json, webkit-patch)
> 6. Convert testing scripts and webkitpy to Python 3 in a single
> change
> 
> The trouble I foresee us encountering with any scheme which attempts
> a conversion which retains both Python 2.7 and Python 3 compatibility
> is code like this:
> 
> for expectation_string, expectation_enum in
> test_expectations.TestExpectations.EXPECTATIONS.iteritems():
> ...
> 
> In this code, the EXPECTATIONS dictionary is thousands of elements
> long. In Python 2.7, iteritems() gives us an iterator instead of
> creating a new list, like items() would. In Python 3, iteritems()
> doesn’t exist, but items() does, and now gives us an iterator instead
> of creating a new list. The trouble here is that, in this case,
> creating a new list will be very expensive, expensive enough that we
> might manage to impact the testing run. There isn’t really an elegant
> way around this problem if we want to support both Python 2.7 and
> Python 3, other than defining different code paths for each language.
> 
> There are other small gotchas as well. For example, ‘%’ is no longer
> a protected character, which can actually change the behavior of
> regexes. That’s why I think it’s better to just try and directly
> convert things instead of attempting to be compatible with both
> Python 2.7 and Python 3.
> 
> Jonathan
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Darin Adler]

> On Jul 16, 2019, at 12:46 PM, Alexey Proskuryakov  wrote:
> 
> - They shouldn't make it excessively difficult to do WebKit engineering on 
> older versions of macOS.
> 
> "Excessively" is not clearly defined, but it seems obvious that there is a 
> tradeoff between tools work difficulty, and difficulty for engineers who go 
> back to older macOS versions (and implicitly, difficulty for people who set 
> up bots, QA engineers, and others involved).
> 
> I don't think that anyone ever suggested that it will be up to each engineer 
> to figure out the best way to install Python 3.

Lets keep in mind our strategy to keep development of WebKit on macOS easy. We 
want to preserve this. The steps (assuming git) are:

https://webkit.org/build-tools/
• download and install Xcode from Apple
% xcode-select --install

https://webkit.org/getting-the-code/
% git clone git://git.webkit.org/WebKit.git WebKit
% Tools/Scripts/webkit-patch setup-git-clone

https://webkit.org/building-webkit/
% build-webkit --debug

https://webkit.org/running-webkit/
% run-safari

We’d really like to keep it to a small number of steps. Having to download and 
install anything else would be a significant additional step.

— Darin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Alexey Proskuryakov]

> 15 июля 2019 г., в 23:04, Fujii Hironori  
> написал(а):
> 
> 
> On Sun, Jul 14, 2019 at 7:44 AM Ryosuke Niwa  > wrote:
> 
> I don’t think anyone is arguing that we’d eventually need to move to Python3. 
> I’m arguing that it’s not okay to require random WebKit contributor to know 
> some obscure python insanity to install Python 3, or have a script that 
> installs Python 3 and breaks all other python scripts in the system.
> 
> 
>  Just out of curiosity. As far as I know, installing Python 3 breaks nothing. 
> What and why are they got broken?

As always, it will be up to the people doing the work to decide how it's done. 
The feedback is clear:

- They shouldn't make it excessively difficult to do WebKit engineering on 
older versions of macOS.

"Excessively" is not clearly defined, but it seems obvious that there is a 
tradeoff between tools work difficulty, and difficulty for engineers who go 
back to older macOS versions (and implicitly, difficulty for people who set up 
bots, QA engineers, and others involved).

I don't think that anyone ever suggested that it will be up to each engineer to 
figure out the best way to install Python 3.

- virtualenv works great for some projects.

Definitely worth looking at it as one of the primary paths forward. It's not 
just Python itself, but we don't want to pollute the system with modules 
either. Although the latter already has a pretty nice solution in WebKit with 
autoinstall.

- virtualenv shouldn't be required when building WebKit, and dynamic 
dependencies in general are not OK in this scenario.

- Alexey

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Konstantin Tokarev]

16.07.2019, 14:33, "Guillaume Emont" :
> Quoting Fujii Hironori (2019-07-16 08:04:09)
>>  On Sun, Jul 14, 2019 at 7:44 AM Ryosuke Niwa  wrote:
>>
>>  I don’t think anyone is arguing that we’d eventually need to move to
>>  Python3. I’m arguing that it’s not okay to require random WebKit
>>  contributor to know some obscure python insanity to install Python 3, or
>>  have a script that installs Python 3 and breaks all other python 
>> scripts in
>>  the system.
>>
>>   Just out of curiosity. As far as I know, installing Python 3 breaks 
>> nothing.
>>  What and why are they got broken?
>
> I suspect Ryosuke is talking about a case where python 3 has already
> been installed on the OS (but is not part of the original OS), and we
> install python 3 also, and the scripts that were using the first python
> 3 installed end up using WebKit's python 3 instead, which could lack a
> python module required by these scripts, hence breaking them.

But we have autoinstaller which handles this situation.

>
> I think this situation should be easy to avoid with a virtualenv though.
>
> Best regards,
>
> Guillaume
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

-- 
Regards,
Konstantin

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Guillaume Emont]

Quoting Fujii Hironori (2019-07-16 08:04:09)
> 
> On Sun, Jul 14, 2019 at 7:44 AM Ryosuke Niwa  wrote:
> 
> 
> I don’t think anyone is arguing that we’d eventually need to move to
> Python3. I’m arguing that it’s not okay to require random WebKit
> contributor to know some obscure python insanity to install Python 3, or
> have a script that installs Python 3 and breaks all other python scripts 
> in
> the system.
> 
> 
> 
>  Just out of curiosity. As far as I know, installing Python 3 breaks nothing.
> What and why are they got broken?

I suspect Ryosuke is talking about a case where python 3 has already
been installed on the OS (but is not part of the original OS), and we
install python 3 also, and the scripts that were using the first python
3 installed end up using WebKit's python 3 instead, which could lack a
python module required by these scripts, hence breaking them.

I think this situation should be easy to avoid with a virtualenv though.

Best regards,

Guillaume
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Fujii Hironori]

On Sun, Jul 14, 2019 at 7:44 AM Ryosuke Niwa  wrote:

>
> I don’t think anyone is arguing that we’d eventually need to move to
> Python3. I’m arguing that it’s not okay to require random WebKit
> contributor to know some obscure python insanity to install Python 3, or
> have a script that installs Python 3 and breaks all other python scripts in
> the system.
>
>
 Just out of curiosity. As far as I know, installing Python 3 breaks
nothing. What and why are they got broken?
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Michael Catanzaro]

On Sat, Jul 13, 2019 at 9:26 PM, Maciej Stachowiak  
wrote:

Can you clarify why this is needed?


Well it just wouldn't seem very kosher to use a virtualenv for the 
serious work of performing real distro builds, right? In contrast to 
developer scripts for developer convenience, where I'd say it's 
perfectly fine to do whatever we want. But official builds should 
surely be performed using system dependencies only.


Anyway, it doesn't seem like a problem. From searching for 'python' in 
my build.ninja, I find:


JSC:

ud_itab.py
generateWasmOpsHeader.py
generateWasmValidateInlinesHeader.py
generateWasmB3IRGeneratorInlinesHeader.py
create_regex_tables
generateYarrUnicodePropertyTables.py
generateIntlCanonicalizeLanguage.py
KeywordLookupGenerator.py
generate-inspector-protocol-bindings.py
generate-js-builtins.py
generateYarrCanonicalizeUnicode
generate-combined-inspector-json.py
jsmin.py
cssmin.py
make-js-file-arrays.py

WebCore:

create-html-entity-table
create-http-header-name-table
makeSelectorPseudoClassAndCompatibilityElementMap.py
makeSelectorPseudoElementsMap.py

WebKit:

generate-message-receiver.py
generate-messages-header.py

WebInspectorUI:

copy-user-interface-resources.pl (perl script that runs some python)

Tools:

generate-inspector-gresource-manifest.py

It's possible I've missed some, but that's probably most of them. 
Basically all the scripts under Source/ -- the scripts that are really 
required to build -- and that one platform-specific exception under 
Tools/, shouldn't require a virtualenv IMO. That doesn't seem too 
difficult to ensure. We could either have them use the virtualenv only 
optionally if it's somehow already available (I'm not familiar with how 
it works) or only if the scripts detect that webkitpy exists, or just 
make this subset of scripts compatible with both python2 and python3 
for the next couple years.


In contrast, the vast majority of our python scripts are not required 
to build. They live under Tools/Scripts, are only used by developers, 
and are not included in release tarballs at all. That includes all of 
webkitpy, anything used by build-webkit, anything used by 
run-webkit-tests, etc. I'd say we can go crazy here. You get a 
virtualenv, you get a virtualenv, everybody gets a virtualenv!


Michael


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Adrien Destugues]

>I don’t think anyone is arguing that we’d eventually need to move to
>Python3. I’m arguing that it’s not okay to require random WebKit
>contributor to know some obscure python insanity to install Python 3,
>or
>have a script that installs Python 3 and breaks all other python
>scripts in
>the system.
>
>

Python3 would normally install a "python3" executable that's independant from 
the "python" one from Python2? It's fairly common to have these side by side 
precisely because they are not compatible.

-- 
Adrien.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Maciej Stachowiak]

> On Jul 13, 2019, at 7:26 PM, Maciej Stachowiak  wrote:
> 
> 
> 
>> On Jul 13, 2019, at 4:14 PM, Michael Catanzaro  wrote:
>> 
>> On Sat, Jul 13, 2019 at 6:02 PM, Maciej Stachowiak  wrote:
>>> This is exactly what virtualenvs can do. And this is how we should do it 
>>> IMO, even for systems that natively have some version of Python 3.
>> 
>> I guess that's fine for everything not required by the CMake build, e.g. 
>> build-webkit and webkitpy. That's probably 99% of our python code.
>> 
>> Scripts required by the CMake build should use system python3 though, not 
>> the virtualenv. I guess that's 1% or less of our python. OK?
> 
> Can you clarify why this is needed?

(Also which specific Python scripts are needed by the CMake build?)
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Maciej Stachowiak]

> On Jul 13, 2019, at 4:14 PM, Michael Catanzaro  wrote:
> 
> On Sat, Jul 13, 2019 at 6:02 PM, Maciej Stachowiak  wrote:
>> This is exactly what virtualenvs can do. And this is how we should do it 
>> IMO, even for systems that natively have some version of Python 3.
> 
> I guess that's fine for everything not required by the CMake build, e.g. 
> build-webkit and webkitpy. That's probably 99% of our python code.
> 
> Scripts required by the CMake build should use system python3 though, not the 
> virtualenv. I guess that's 1% or less of our python. OK?

Can you clarify why this is needed?
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Ryosuke Niwa]

On Sat, Jul 13, 2019 at 4:14 PM Michael Catanzaro 
wrote:

> On Sat, Jul 13, 2019 at 6:02 PM, Maciej Stachowiak 
> wrote:
> > This is exactly what virtualenvs can do. And this is how we should do
> > it IMO, even for systems that natively have some version of Python 3.
>
> I guess that's fine for everything not required by the CMake build,
> e.g. build-webkit and webkitpy. That's probably 99% of our python code.
>
> Scripts required by the CMake build should use system python3 though,
> not the virtualenv. I guess that's 1% or less of our python. OK?


That seems okay given macOS / iOS ports don’t use CMake right now.

Virtually all ports that use CMake as the build system would have python3
in the base installation or already require installations of a bunch of
software (e.g. Windows port), right?

- R. Niwa
-- 
- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Michael Catanzaro]

On Sat, Jul 13, 2019 at 6:02 PM, Maciej Stachowiak  
wrote:
This is exactly what virtualenvs can do. And this is how we should do 
it IMO, even for systems that natively have some version of Python 3.


I guess that's fine for everything not required by the CMake build, 
e.g. build-webkit and webkitpy. That's probably 99% of our python code.


Scripts required by the CMake build should use system python3 though, 
not the virtualenv. I guess that's 1% or less of our python. OK?


Michael


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Maciej Stachowiak]

> On Jul 13, 2019, at 3:44 PM, Ryosuke Niwa  wrote:
> 
> 
> 
> 
>> On Fri, Jul 12, 2019 at 11:22 PM Jonathan Bedard  wrote:
>> I would agree that if we move to Python 3, we need a script which installs 
>> Python 3 in an impossible to mess-up way on Mojave and High Sierra.
>> 
>> I don’t think the clang comparison is fair here. Python 2 is officially 
>> deprecated in 2020, we can’t expect security updates to the language or any 
>> libraries we depend on 6 months from now. It’s not really a question of if 
>> we stop supporting Python 2, but rather, when we stop supporting Python 2.
> 
> I don’t think anyone is arguing that we’d eventually need to move to Python3. 
> I’m arguing that it’s not okay to require random WebKit contributor to know 
> some obscure python insanity to install Python 3, or have a script that 
> installs Python 3 and breaks all other python scripts in the system.
> 
>> 
>> It’s also worth noting that in Catalina, we will need some script to install 
>> XCode Command Line Tools since SVN, Python 2.7 and Python 3 are no longer 
>> included by default in the new Xcode. Given this, it doesn’t seem terrible 
>> if the script responsible for installing XCode CL tools also installs Python 
>> 3 for older Mac versions.
> 
> Yes, as long as it doesn’t replace or break existing Python2.7 and/or other 
> python scripts in the system.

This is exactly what virtualenvs can do. And this is how we should do it IMO, 
even for systems that natively have some version of Python 3.

> 
>> 
 On Jul 12, 2019, at 7:34 PM, Ryosuke Niwa  wrote:
 
>>> 
>> 
>>> 
 On Fri, Jul 12, 2019 at 7:03 PM Michael Catanzaro  
 wrote:
 On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa  wrote:
 > I frequently do WebKit development in older versions of macOS to 
 > diagnose old OS specific regressions, and having to install Python 3 
 > each time I install an old OS is too much of a trouble.
 
 I understand it would be a hassle. :/ But please consider it relative 
 to the additional difficulty of rewriting all of webkitpy to support 
 multiple versions of python at the same time, or setting up a wrapper 
 layer of bash scripts around all of our python scripts to enter a 
 virtualenv before executing the real script.
>>> 
>>> Yeah, and it sounds strictly better that the trouble is handled by people 
>>> who maintain Python code who presumably more about Python than a random 
>>> WebKit contributor who may not know how to setup virtual environment in 
>>> Python, etc... 
>>> 
>>> Again, the argument that the difficulty can be overcome and it's a minor 
>>> inconvenience isn't convincing. I can, for example, suggest that we should 
>>> just build WebKit using the latest version of clang. Anyone who uses a 
>>> system that doesn't come with the latest release of clang should just build 
>>> clang instead. There is a significant cost in having to support MSVC and 
>>> GC++ so we should just use clang everywhere and only the latest version 
>>> like Chromium does.
>>> 
>>> Each team & person has a different preference & perspective when it comes 
>>> to tooling. Please don't break someone else's working workflow based on 
>>> your preference.
>>> 
>>> - R. Niwa
>>> 
> -- 
> - R. Niwa
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Ryosuke Niwa]

On Fri, Jul 12, 2019 at 11:22 PM Jonathan Bedard  wrote:

> I would agree that if we move to Python 3, we need a script which installs
> Python 3 in an impossible to mess-up way on Mojave and High Sierra.
>
> I don’t think the clang comparison is fair here. Python 2 is officially
> deprecated in 2020, we can’t expect security updates to the language or any
> libraries we depend on 6 months from now. It’s not really a question of if
> we stop supporting Python 2, but rather, when we stop supporting Python 2.
>

I don’t think anyone is arguing that we’d eventually need to move to
Python3. I’m arguing that it’s not okay to require random WebKit
contributor to know some obscure python insanity to install Python 3, or
have a script that installs Python 3 and breaks all other python scripts in
the system.


> It’s also worth noting that in Catalina, we will need some script to
> install XCode Command Line Tools since SVN, Python 2.7 and Python 3 are no
> longer included by default in the new Xcode. Given this, it doesn’t seem
> terrible if the script responsible for installing XCode CL tools also
> installs Python 3 for older Mac versions.
>

Yes, as long as it doesn’t replace or break existing Python2.7 and/or other
python scripts in the system.


> On Jul 12, 2019, at 7:34 PM, Ryosuke Niwa  wrote:
>
> 
>
>
> On Fri, Jul 12, 2019 at 7:03 PM Michael Catanzaro 
> wrote:
>
>> On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa  wrote:
>> > I frequently do WebKit development in older versions of macOS to
>> > diagnose old OS specific regressions, and having to install Python 3
>> > each time I install an old OS is too much of a trouble.
>>
>> I understand it would be a hassle. :/ But please consider it relative
>> to the additional difficulty of rewriting all of webkitpy to support
>> multiple versions of python at the same time, or setting up a wrapper
>> layer of bash scripts around all of our python scripts to enter a
>> virtualenv before executing the real script.
>>
>
> Yeah, and it sounds strictly better that the trouble is handled by people
> who maintain Python code who presumably more about Python than a random
> WebKit contributor who may not know how to setup virtual environment in
> Python, etc...
>
> Again, the argument that the difficulty can be overcome and it's a minor
> inconvenience isn't convincing. I can, for example, suggest that we should
> just build WebKit using the latest version of clang. Anyone who uses a
> system that doesn't come with the latest release of clang should just build
> clang instead. There is a significant cost in having to support MSVC and
> GC++ so we should just use clang everywhere and only the latest version
> like Chromium does.
>
> Each team & person has a different preference & perspective when it comes
> to tooling. Please don't break someone else's working workflow based on
> your preference.
>
> - R. Niwa
>
> --
- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Jonathan Bedard]

I would agree that if we move to Python 3, we need a script which installs 
Python 3 in an impossible to mess-up way on Mojave and High Sierra.

I don’t think the clang comparison is fair here. Python 2 is officially 
deprecated in 2020, we can’t expect security updates to the language or any 
libraries we depend on 6 months from now. It’s not really a question of if we 
stop supporting Python 2, but rather, when we stop supporting Python 2.

It’s also worth noting that in Catalina, we will need some script to install 
XCode Command Line Tools since SVN, Python 2.7 and Python 3 are no longer 
included by default in the new Xcode. Given this, it doesn’t seem terrible if 
the script responsible for installing XCode CL tools also installs Python 3 for 
older Mac versions.

Jonathan

> On Jul 12, 2019, at 7:34 PM, Ryosuke Niwa  wrote:
> 
> 
> 
>> On Fri, Jul 12, 2019 at 7:03 PM Michael Catanzaro  
>> wrote:
>> On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa  wrote:
>> > I frequently do WebKit development in older versions of macOS to 
>> > diagnose old OS specific regressions, and having to install Python 3 
>> > each time I install an old OS is too much of a trouble.
>> 
>> I understand it would be a hassle. :/ But please consider it relative 
>> to the additional difficulty of rewriting all of webkitpy to support 
>> multiple versions of python at the same time, or setting up a wrapper 
>> layer of bash scripts around all of our python scripts to enter a 
>> virtualenv before executing the real script.
> 
> Yeah, and it sounds strictly better that the trouble is handled by people who 
> maintain Python code who presumably more about Python than a random WebKit 
> contributor who may not know how to setup virtual environment in Python, 
> etc... 
> 
> Again, the argument that the difficulty can be overcome and it's a minor 
> inconvenience isn't convincing. I can, for example, suggest that we should 
> just build WebKit using the latest version of clang. Anyone who uses a system 
> that doesn't come with the latest release of clang should just build clang 
> instead. There is a significant cost in having to support MSVC and GC++ so we 
> should just use clang everywhere and only the latest version like Chromium 
> does.
> 
> Each team & person has a different preference & perspective when it comes to 
> tooling. Please don't break someone else's working workflow based on your 
> preference.
> 
> - R. Niwa
> 
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Maciej Stachowiak]

> On Jul 12, 2019, at 3:23 PM, Ryosuke Niwa  wrote:
> 
> 
>> On Fri, Jul 12, 2019 at 1:04 PM Jonathan Bedard  wrote:
> 
>> 
>> > On Jul 12, 2019, at 12:49 PM, Michael Catanzaro  
>> > wrote:
>> > 
>> > On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard  wrote:
>> >> The trouble I foresee us encountering with any scheme which attempts a 
>> >> conversion which retains both Python 2.7 and Python 3 compatibility is 
>> >> code like this:
>> > 
>> > Is python2 support required for a well-motivated transitional purpose?
>> > 
>> > I had previously proposed making all our scripts work with both python2 
>> > and python3 only because I thought Apple was going to require python2 
>> > indefinitely. Now that you're interested in this transition, there's 
>> > probably no need to continue python2 support. Anyone building WebKit on 
>> > older versions of macOS can reasonably be expected to manually install 
>> > python3, right? And it's clear that you're prepared to do this for 
>> > infrastructure/bots already.
>> 
>> We still need to figure out whether (and for how long) we intend to retain 
>> Python 2 support. Over the last few months, opinions on this have changed 
>> quite a bit, so I’m trying to determine what our path forward is going to be.
>> 
>> In my opinion, a few months after Catalina GMs, we no longer need to 
>> maintain Python 2 support, assuming that we provide adequate automation for 
>> installing Python 3 on pre-Catalina macOS (ie, Mojave, High Sierra) and are 
>> explicit about shebangs.
> 
> I don't think it's acceptable to require installation of Python 3 just to 
> build & run tests on WebKit unless the installation itself is automated and 
> compartmentalized to WebKit's development given how bad Python is with 
> respect to having multiple versions of Python's and managing packages between 
> them.
> 
> I frequently do WebKit development in older versions of macOS to diagnose old 
> OS specific regressions, and having to install Python 3 each time I install 
> an old OS is too much of a trouble.

It’s possible install a python without installing it on the system, and install 
modules and any needed additions, using virtualenv: 
https://virtualenv.pypa.io/en/stable/

This is the pro way to use python without caring about what happens to be on 
the system. 

 I suggest we proceed by gradually converting our scripts to use a Python 3 
virtualenv. This will also spare us the need to install Python modules onto the 
system. The tricky part will be that webkitpy would have to work both ways 
until conversion is complete (or at the extreme I guess we could fork it).

> - R. Niwa
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Ryosuke Niwa]

On Fri, Jul 12, 2019 at 7:03 PM Michael Catanzaro 
wrote:

> On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa  wrote:
> > I frequently do WebKit development in older versions of macOS to
> > diagnose old OS specific regressions, and having to install Python 3
> > each time I install an old OS is too much of a trouble.
>
> I understand it would be a hassle. :/ But please consider it relative
> to the additional difficulty of rewriting all of webkitpy to support
> multiple versions of python at the same time, or setting up a wrapper
> layer of bash scripts around all of our python scripts to enter a
> virtualenv before executing the real script.
>

Yeah, and it sounds strictly better that the trouble is handled by people
who maintain Python code who presumably more about Python than a random
WebKit contributor who may not know how to setup virtual environment in
Python, etc...

Again, the argument that the difficulty can be overcome and it's a minor
inconvenience isn't convincing. I can, for example, suggest that we should
just build WebKit using the latest version of clang. Anyone who uses a
system that doesn't come with the latest release of clang should just build
clang instead. There is a significant cost in having to support MSVC and
GC++ so we should just use clang everywhere and only the latest version
like Chromium does.

Each team & person has a different preference & perspective when it comes
to tooling. Please don't break someone else's working workflow based on
your preference.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Michael Catanzaro]

On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa  wrote:
I frequently do WebKit development in older versions of macOS to 
diagnose old OS specific regressions, and having to install Python 3 
each time I install an old OS is too much of a trouble.


I understand it would be a hassle. :/ But please consider it relative 
to the additional difficulty of rewriting all of webkitpy to support 
multiple versions of python at the same time, or setting up a wrapper 
layer of bash scripts around all of our python scripts to enter a 
virtualenv before executing the real script.


Michael


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Ryosuke Niwa]

On Fri, Jul 12, 2019 at 1:04 PM Jonathan Bedard  wrote:

>
> > On Jul 12, 2019, at 12:49 PM, Michael Catanzaro 
> wrote:
> >
> > On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard 
> wrote:
> >> The trouble I foresee us encountering with any scheme which attempts a
> conversion which retains both Python 2.7 and Python 3 compatibility is code
> like this:
> >
> > Is python2 support required for a well-motivated transitional purpose?
> >
> > I had previously proposed making all our scripts work with both python2
> and python3 only because I thought Apple was going to require python2
> indefinitely. Now that you're interested in this transition, there's
> probably no need to continue python2 support. Anyone building WebKit on
> older versions of macOS can reasonably be expected to manually install
> python3, right? And it's clear that you're prepared to do this for
> infrastructure/bots already.
>
> We still need to figure out whether (and for how long) we intend to retain
> Python 2 support. Over the last few months, opinions on this have changed
> quite a bit, so I’m trying to determine what our path forward is going to
> be.
>
> In my opinion, a few months after Catalina GMs, we no longer need to
> maintain Python 2 support, assuming that we provide adequate automation for
> installing Python 3 on pre-Catalina macOS (ie, Mojave, High Sierra) and are
> explicit about shebangs.
>

I don't think it's acceptable to require installation of Python 3 just to
build & run tests on WebKit unless the installation itself is automated and
compartmentalized to WebKit's development given how bad Python is with
respect to having multiple versions of Python's and managing packages
between them.

I frequently do WebKit development in older versions of macOS to diagnose
old OS specific regressions, and having to install Python 3 each time I
install an old OS is too much of a trouble.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Jonathan Bedard]

> On Jul 12, 2019, at 1:07 PM, Keith Rollin  wrote:
> 
>> On Jul 12, 2019, at 13:37, Tim Horton  wrote:
>> 
>> See "Migrating to the common subset of Python 2 and 3” — you define 
>> different iteritems() helpers in the two cases. Seems pretty reasonable to 
>> me.
> 
> I did something like this in webkit-triage. Search for “viewitems” for the 
> compatibility function I used, as well as for “PY3” to see some of the other 
> thunks I put into place.
> 
> — Keith
> 

Actually, hadn’t seen those tricks yet, thanks Tim and Keith for calling them 
out!

Those would allow us to keep scripts both Python 2.7 and Python 3 compatible 
for longer.

Jonathan
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Jonathan Bedard]

> On Jul 12, 2019, at 12:49 PM, Michael Catanzaro  wrote:
> 
> On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard  wrote:
>> The trouble I foresee us encountering with any scheme which attempts a 
>> conversion which retains both Python 2.7 and Python 3 compatibility is code 
>> like this:
> 
> Is python2 support required for a well-motivated transitional purpose?
> 
> I had previously proposed making all our scripts work with both python2 and 
> python3 only because I thought Apple was going to require python2 
> indefinitely. Now that you're interested in this transition, there's probably 
> no need to continue python2 support. Anyone building WebKit on older versions 
> of macOS can reasonably be expected to manually install python3, right? And 
> it's clear that you're prepared to do this for infrastructure/bots already.

We still need to figure out whether (and for how long) we intend to retain 
Python 2 support. Over the last few months, opinions on this have changed quite 
a bit, so I’m trying to determine what our path forward is going to be.

In my opinion, a few months after Catalina GMs, we no longer need to maintain 
Python 2 support, assuming that we provide adequate automation for installing 
Python 3 on pre-Catalina macOS (ie, Mojave, High Sierra) and are explicit about 
shebangs.

> 
> Then million-dollar question is: what shebangs will we use for our scripts? 
> Will #!/usr/bin/env python3 work for Apple?

If we move straight to Python 3, we would need to use the Python 3 shebang. 
Catalina has both Python 2.7 (name ‘python’) and Python 3 (named ‘python3’). I 
think that this is what we would want to do because it allows us to pretty 
explicitly convert scripts one at a time.

> 
> Michael
> 
> 

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Adrian Perez de Castro]

Hello,

On Fri, 12 Jul 2019 12:37:43 -0700, Tim Horton  wrote:

> > On Jul 12, 2019, at 12:18 PM, Jonathan Bedard  wrote:
> > 
> > Hello WebKit developers,
> > 
> > Now that the Catalina developer seeds are available, it is official that
> > the new Mac developer tools come with Python 3. As a result, we need to
> > continue the ongoing discussion about migrating our Python 2.7 scripts to
> > Python 3.

Given that GNU/Linux distributions have started already a while ago switching
it is great that MacOS is now following suit as well \o/

We have a bug already for this: https://bugs.webkit.org/show_bug.cgi?id=184986

If one has to make code compatible with both Python 2.7 and 3.x, my advice
would be to use the Six module, which provides ready-made helpers which
behave consistently across both versions: https://six.readthedocs.io/
 
> > I propose that, over the next 9 months, we do the following:
> > 
> > 1. Make any no-cost Python 3 compatibility changes, in particular
> > - print foo -> print(foo)
> > - import .foo -> import webkitpy.foo
> > 2. Convert any scripts not used in automation to Python 3 ASAP (scripts 
> > like bisect-builds, block-spammers, compare-results)
> > 3. Make most Python 3 compatibility changes which sacrifice efficiency, 
> > subject to a case-by-case audit. These would be things like:
> > - dict.iteritems() -> dict.items()
> > - dict.items() -> list(dict.items())
> > 4. Install Python 3 on macOS Sierra and Mojave bots
> > 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like 
> > clean-webkit, merge-results-json, webkit-patch)
> > 6. Convert testing scripts and webkitpy to Python 3 in a single change
> > 
> > The trouble I foresee us encountering with any scheme which attempts a 
> > conversion which retains both Python 2.7 and Python 3 compatibility is code 
> > like this:
> > 
> > for expectation_string, expectation_enum in 
> > test_expectations.TestExpectations.EXPECTATIONS.iteritems():
> > ...
> > 
> > In this code, the EXPECTATIONS dictionary is thousands of elements long. In 
> > Python 2.7, iteritems() gives us an iterator instead of creating a new 
> > list, like items() would. In Python 3, iteritems() doesn’t exist, but 
> > items() does, and now gives us an iterator instead of creating a new list. 
> > The trouble here is that, in this case, creating a new list will be very 
> > expensive, expensive enough that we might manage to impact the testing run. 
> > There isn’t really an elegant way around this problem if we want to support 
> > both Python 2.7 and Python 3, other than defining different code paths for 
> > each language.
> 
> The official Python 3 transition documentation has a fairly elegant solution 
> to this, actually??
> 
> https://legacy.python.org/dev/peps/pep-0469/
> 
> See "Migrating to the common subset of Python 2 and 3” — you define different 
> iteritems() helpers in the two cases. Seems pretty reasonable to me.

Instead of rolling our own, I would rather use Six (mentioned above). It
covers all the differences with the different “.iter*()” methods:

  https://six.readthedocs.io/#six.iterkeys
  https://six.readthedocs.io/#six.itervalues
  https://six.readthedocs.io/#six.iteritems
  https://six.readthedocs.io/#six.iterlists
  ...

> > There are other small gotchas as well. For example, ‘%’ is no longer a
> > protected character, which can actually change the behavior of regexes.
> > That’s why I think it’s better to just try and directly convert things
> > instead of attempting to be compatible with both Python 2.7 and Python 3.

In my experience some of the major troubles making a codebase compatible
with both Pythons are the string types:

  - In Python 2, “str” is also usable for binary data (basically it's backed
by a char[], while “unicode” is actual text which may contain any code
point.
  - In Python 3, “string” is kind of the old “unicode” (textual data),
and “bytes” is kind of “string” but only holds binary data and in
general cannot be used where a string would have been used.

The Six module helps a bit with the text types (see “six.text_type” and
“six.binary_type”.

Regards,
—Adrián


pgpuX1RTgiWOr.pgp
Description: PGP signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Jonathan Bedard]

> On Jul 12, 2019, at 12:45 PM, Robert Ma  wrote:
> 
> Any thoughts on bytes and Unicode strings, especially the string literals in 
> the code base?

My experience with this has been you mostly have to pay attention to where your 
code interfaces with other processes. In webkitpy, I suspect that most of the 
our work here will probably be where we are calling other commands or reading 
files.

Because we call most commands through either the Executive class or the 
FileSystem class, I think there is a good chance we can make changes here that 
effect most of the codebase.

Jonathan

> 
> On Fri, Jul 12, 2019 at 3:38 PM Tim Horton  > wrote:
> 
> 
>> On Jul 12, 2019, at 12:18 PM, Jonathan Bedard > > wrote:
>> 
>> Hello WebKit developers,
>> 
>> Now that the Catalina developer seeds are available, it is official that the 
>> new Mac developer tools come with Python 3. As a result, we need to continue 
>> the ongoing discussion about migrating our Python 2.7 scripts to Python 3.
>> 
>> I propose that, over the next 9 months, we do the following:
>> 
>> 1. Make any no-cost Python 3 compatibility changes, in particular
>> - print foo -> print(foo)
>> - import .foo -> import webkitpy.foo
>> 2. Convert any scripts not used in automation to Python 3 ASAP (scripts like 
>> bisect-builds, block-spammers, compare-results)
>> 3. Make most Python 3 compatibility changes which sacrifice efficiency, 
>> subject to a case-by-case audit. These would be things like:
>> - dict.iteritems() -> dict.items()
>> - dict.items() -> list(dict.items())
>> 4. Install Python 3 on macOS Sierra and Mojave bots
>> 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like 
>> clean-webkit, merge-results-json, webkit-patch)
>> 6. Convert testing scripts and webkitpy to Python 3 in a single change
>> 
>> The trouble I foresee us encountering with any scheme which attempts a 
>> conversion which retains both Python 2.7 and Python 3 compatibility is code 
>> like this:
>> 
>> for expectation_string, expectation_enum in 
>> test_expectations.TestExpectations.EXPECTATIONS.iteritems():
>> ...
>> 
>> In this code, the EXPECTATIONS dictionary is thousands of elements long. In 
>> Python 2.7, iteritems() gives us an iterator instead of creating a new list, 
>> like items() would. In Python 3, iteritems() doesn’t exist, but items() 
>> does, and now gives us an iterator instead of creating a new list. The 
>> trouble here is that, in this case, creating a new list will be very 
>> expensive, expensive enough that we might manage to impact the testing run. 
>> There isn’t really an elegant way around this problem if we want to support 
>> both Python 2.7 and Python 3, other than defining different code paths for 
>> each language.
> 
> The official Python 3 transition documentation has a fairly elegant solution 
> to this, actually??
> 
> https://legacy.python.org/dev/peps/pep-0469/ 
> 
> 
> See "Migrating to the common subset of Python 2 and 3” — you define different 
> iteritems() helpers in the two cases. Seems pretty reasonable to me.
> 
>> There are other small gotchas as well. For example, ‘%’ is no longer a 
>> protected character, which can actually change the behavior of regexes. 
>> That’s why I think it’s better to just try and directly convert things 
>> instead of attempting to be compatible with both Python 2.7 and Python 3.
>> 
>> Jonathan
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org 
>> https://lists.webkit.org/mailman/listinfo/webkit-dev 
>> 
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org 
> https://lists.webkit.org/mailman/listinfo/webkit-dev 
> 

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Michael Catanzaro]

On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard  
wrote:
The trouble I foresee us encountering with any scheme which attempts 
a conversion which retains both Python 2.7 and Python 3 compatibility 
is code like this:


Is python2 support required for a well-motivated transitional purpose?

I had previously proposed making all our scripts work with both python2 
and python3 only because I thought Apple was going to require python2 
indefinitely. Now that you're interested in this transition, there's 
probably no need to continue python2 support. Anyone building WebKit on 
older versions of macOS can reasonably be expected to manually install 
python3, right? And it's clear that you're prepared to do this for 
infrastructure/bots already.


Then million-dollar question is: what shebangs will we use for our 
scripts? Will #!/usr/bin/env python3 work for Apple?


Michael


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Robert Ma]

Any thoughts on bytes and Unicode strings, especially the string literals
in the code base?

On Fri, Jul 12, 2019 at 3:38 PM Tim Horton  wrote:

>
>
> On Jul 12, 2019, at 12:18 PM, Jonathan Bedard  wrote:
>
> Hello WebKit developers,
>
> Now that the Catalina developer seeds are available, it is official that
> the new Mac developer tools come with Python 3. As a result, we need to
> continue the ongoing discussion about migrating our Python 2.7 scripts to
> Python 3.
>
> I propose that, over the next 9 months, we do the following:
>
> 1. Make any no-cost Python 3 compatibility changes, in particular
> - print foo -> print(foo)
> - import .foo -> import webkitpy.foo
> 2. Convert any scripts not used in automation to Python 3 ASAP (scripts
> like bisect-builds, block-spammers, compare-results)
> 3. Make most Python 3 compatibility changes which sacrifice efficiency,
> subject to a case-by-case audit. These would be things like:
> - dict.iteritems() -> dict.items()
> - dict.items() -> list(dict.items())
> 4. Install Python 3 on macOS Sierra and Mojave bots
> 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like
> clean-webkit, merge-results-json, webkit-patch)
> 6. Convert testing scripts and webkitpy to Python 3 in a single change
>
> The trouble I foresee us encountering with any scheme which attempts a
> conversion which retains both Python 2.7 and Python 3 compatibility is code
> like this:
>
> for expectation_string, expectation_enum in
> test_expectations.TestExpectations.EXPECTATIONS.iteritems():
> ...
>
> In this code, the EXPECTATIONS dictionary is thousands of elements long.
> In Python 2.7, iteritems() gives us an iterator instead of creating a new
> list, like items() would. In Python 3, iteritems() doesn’t exist, but
> items() does, and now gives us an iterator instead of creating a new list.
> The trouble here is that, in this case, creating a new list will be very
> expensive, expensive enough that we might manage to impact the testing run.
> There isn’t really an elegant way around this problem if we want to support
> both Python 2.7 and Python 3, other than defining different code paths for
> each language.
>
>
> The official Python 3 transition documentation has a fairly elegant
> solution to this, actually??
>
> https://legacy.python.org/dev/peps/pep-0469/
>
> See "Migrating to the common subset of Python 2 and 3” — you define
> different iteritems() helpers in the two cases. Seems pretty reasonable to
> me.
>
> There are other small gotchas as well. For example, ‘%’ is no longer a
> protected character, which can actually change the behavior of regexes.
> That’s why I think it’s better to just try and directly convert things
> instead of attempting to be compatible with both Python 2.7 and Python 3.
>
> Jonathan
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Don . Olmstead]

Mentioned this on the other thread but here’s it again.

https://python-modernize.readthedocs.io/en/latest/ and 
http://python-future.org/automatic_conversion.html might be of use considering 
the sheer amount of code. They’re both mentioned in 
https://docs.python.org/3/howto/pyporting.html for migration.

You can run individual rules one by one over the code. That might be ok rather 
than running all the rules at once. Especially for review. Modernize seemed to 
work pretty well but it wasn’t smart enough to detect shebang lines.

From: webkit-dev  
On Behalf Of Tim Horton
Sent: Friday, July 12, 2019 12:38 PM
To: Jonathan Bedard 
Cc: webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Moving to Python 3




On Jul 12, 2019, at 12:18 PM, Jonathan Bedard 
mailto:jbed...@apple.com>> wrote:

Hello WebKit developers,

Now that the Catalina developer seeds are available, it is official that the 
new Mac developer tools come with Python 3. As a result, we need to continue 
the ongoing discussion about migrating our Python 2.7 scripts to Python 3.

I propose that, over the next 9 months, we do the following:

1. Make any no-cost Python 3 compatibility changes, in particular
- print foo -> print(foo)
- import .foo -> import webkitpy.foo
2. Convert any scripts not used in automation to Python 3 ASAP (scripts like 
bisect-builds, block-spammers, compare-results)
3. Make most Python 3 compatibility changes which sacrifice efficiency, subject 
to a case-by-case audit. These would be things like:
- dict.iteritems() -> dict.items()
- dict.items() -> list(dict.items())
4. Install Python 3 on macOS Sierra and Mojave bots
5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like 
clean-webkit, merge-results-json, webkit-patch)
6. Convert testing scripts and webkitpy to Python 3 in a single change

The trouble I foresee us encountering with any scheme which attempts a 
conversion which retains both Python 2.7 and Python 3 compatibility is code 
like this:

for expectation_string, expectation_enum in 
test_expectations.TestExpectations.EXPECTATIONS.iteritems():
...

In this code, the EXPECTATIONS dictionary is thousands of elements long. In 
Python 2.7, iteritems() gives us an iterator instead of creating a new list, 
like items() would. In Python 3, iteritems() doesn’t exist, but items() does, 
and now gives us an iterator instead of creating a new list. The trouble here 
is that, in this case, creating a new list will be very expensive, expensive 
enough that we might manage to impact the testing run. There isn’t really an 
elegant way around this problem if we want to support both Python 2.7 and 
Python 3, other than defining different code paths for each language.

The official Python 3 transition documentation has a fairly elegant solution to 
this, actually??

https://legacy.python.org/dev/peps/pep-0469/

See "Migrating to the common subset of Python 2 and 3” — you define different 
iteritems() helpers in the two cases. Seems pretty reasonable to me.


There are other small gotchas as well. For example, ‘%’ is no longer a 
protected character, which can actually change the behavior of regexes. That’s 
why I think it’s better to just try and directly convert things instead of 
attempting to be compatible with both Python 2.7 and Python 3.

Jonathan
___
webkit-dev mailing list
webkit-dev@lists.webkit.org<mailto:webkit-dev@lists.webkit.org>
https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Tim Horton]

> On Jul 12, 2019, at 12:18 PM, Jonathan Bedard  wrote:
> 
> Hello WebKit developers,
> 
> Now that the Catalina developer seeds are available, it is official that the 
> new Mac developer tools come with Python 3. As a result, we need to continue 
> the ongoing discussion about migrating our Python 2.7 scripts to Python 3.
> 
> I propose that, over the next 9 months, we do the following:
> 
> 1. Make any no-cost Python 3 compatibility changes, in particular
> - print foo -> print(foo)
> - import .foo -> import webkitpy.foo
> 2. Convert any scripts not used in automation to Python 3 ASAP (scripts like 
> bisect-builds, block-spammers, compare-results)
> 3. Make most Python 3 compatibility changes which sacrifice efficiency, 
> subject to a case-by-case audit. These would be things like:
> - dict.iteritems() -> dict.items()
> - dict.items() -> list(dict.items())
> 4. Install Python 3 on macOS Sierra and Mojave bots
> 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like 
> clean-webkit, merge-results-json, webkit-patch)
> 6. Convert testing scripts and webkitpy to Python 3 in a single change
> 
> The trouble I foresee us encountering with any scheme which attempts a 
> conversion which retains both Python 2.7 and Python 3 compatibility is code 
> like this:
> 
> for expectation_string, expectation_enum in 
> test_expectations.TestExpectations.EXPECTATIONS.iteritems():
> ...
> 
> In this code, the EXPECTATIONS dictionary is thousands of elements long. In 
> Python 2.7, iteritems() gives us an iterator instead of creating a new list, 
> like items() would. In Python 3, iteritems() doesn’t exist, but items() does, 
> and now gives us an iterator instead of creating a new list. The trouble here 
> is that, in this case, creating a new list will be very expensive, expensive 
> enough that we might manage to impact the testing run. There isn’t really an 
> elegant way around this problem if we want to support both Python 2.7 and 
> Python 3, other than defining different code paths for each language.

The official Python 3 transition documentation has a fairly elegant solution to 
this, actually??

https://legacy.python.org/dev/peps/pep-0469/

See "Migrating to the common subset of Python 2 and 3” — you define different 
iteritems() helpers in the two cases. Seems pretty reasonable to me.

> There are other small gotchas as well. For example, ‘%’ is no longer a 
> protected character, which can actually change the behavior of regexes. 
> That’s why I think it’s better to just try and directly convert things 
> instead of attempting to be compatible with both Python 2.7 and Python 3.
> 
> Jonathan
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

[webkit-dev] Moving to Python 3

[Jonathan Bedard]

Hello WebKit developers,

Now that the Catalina developer seeds are available, it is official that the 
new Mac developer tools come with Python 3. As a result, we need to continue 
the ongoing discussion about migrating our Python 2.7 scripts to Python 3.

I propose that, over the next 9 months, we do the following:

1. Make any no-cost Python 3 compatibility changes, in particular
- print foo -> print(foo)
- import .foo -> import webkitpy.foo
2. Convert any scripts not used in automation to Python 3 ASAP (scripts like 
bisect-builds, block-spammers, compare-results)
3. Make most Python 3 compatibility changes which sacrifice efficiency, subject 
to a case-by-case audit. These would be things like:
- dict.iteritems() -> dict.items()
- dict.items() -> list(dict.items())
4. Install Python 3 on macOS Sierra and Mojave bots
5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like 
clean-webkit, merge-results-json, webkit-patch)
6. Convert testing scripts and webkitpy to Python 3 in a single change

The trouble I foresee us encountering with any scheme which attempts a 
conversion which retains both Python 2.7 and Python 3 compatibility is code 
like this:

for expectation_string, expectation_enum in 
test_expectations.TestExpectations.EXPECTATIONS.iteritems():
...

In this code, the EXPECTATIONS dictionary is thousands of elements long. In 
Python 2.7, iteritems() gives us an iterator instead of creating a new list, 
like items() would. In Python 3, iteritems() doesn’t exist, but items() does, 
and now gives us an iterator instead of creating a new list. The trouble here 
is that, in this case, creating a new list will be very expensive, expensive 
enough that we might manage to impact the testing run. There isn’t really an 
elegant way around this problem if we want to support both Python 2.7 and 
Python 3, other than defining different code paths for each language.

There are other small gotchas as well. For example, ‘%’ is no longer a 
protected character, which can actually change the behavior of regexes. That’s 
why I think it’s better to just try and directly convert things instead of 
attempting to be compatible with both Python 2.7 and Python 3.

Jonathan___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev