Re: [webkit-dev] Moving to Python 3

[Jonathan Bedard]

I would agree that if we move to Python 3, we need a script which installs 
Python 3 in an impossible to mess-up way on Mojave and High Sierra.

I don’t think the clang comparison is fair here. Python 2 is officially 
deprecated in 2020, we can’t expect security updates to the language or any 
libraries we depend on 6 months from now. It’s not really a question of if we 
stop supporting Python 2, but rather, when we stop supporting Python 2.

It’s also worth noting that in Catalina, we will need some script to install 
XCode Command Line Tools since SVN, Python 2.7 and Python 3 are no longer 
included by default in the new Xcode. Given this, it doesn’t seem terrible if 
the script responsible for installing XCode CL tools also installs Python 3 for 
older Mac versions.

Jonathan

> On Jul 12, 2019, at 7:34 PM, Ryosuke Niwa  wrote:
> 
> 
> 
>> On Fri, Jul 12, 2019 at 7:03 PM Michael Catanzaro  
>> wrote:
>> On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa  wrote:
>> > I frequently do WebKit development in older versions of macOS to 
>> > diagnose old OS specific regressions, and having to install Python 3 
>> > each time I install an old OS is too much of a trouble.
>> 
>> I understand it would be a hassle. :/ But please consider it relative 
>> to the additional difficulty of rewriting all of webkitpy to support 
>> multiple versions of python at the same time, or setting up a wrapper 
>> layer of bash scripts around all of our python scripts to enter a 
>> virtualenv before executing the real script.
> 
> Yeah, and it sounds strictly better that the trouble is handled by people who 
> maintain Python code who presumably more about Python than a random WebKit 
> contributor who may not know how to setup virtual environment in Python, 
> etc... 
> 
> Again, the argument that the difficulty can be overcome and it's a minor 
> inconvenience isn't convincing. I can, for example, suggest that we should 
> just build WebKit using the latest version of clang. Anyone who uses a system 
> that doesn't come with the latest release of clang should just build clang 
> instead. There is a significant cost in having to support MSVC and GC++ so we 
> should just use clang everywhere and only the latest version like Chromium 
> does.
> 
> Each team & person has a different preference & perspective when it comes to 
> tooling. Please don't break someone else's working workflow based on your 
> preference.
> 
> - R. Niwa
> 
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Maciej Stachowiak]

> On Jul 12, 2019, at 3:23 PM, Ryosuke Niwa  wrote:
> 
> 
>> On Fri, Jul 12, 2019 at 1:04 PM Jonathan Bedard  wrote:
> 
>> 
>> > On Jul 12, 2019, at 12:49 PM, Michael Catanzaro  
>> > wrote:
>> > 
>> > On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard  wrote:
>> >> The trouble I foresee us encountering with any scheme which attempts a 
>> >> conversion which retains both Python 2.7 and Python 3 compatibility is 
>> >> code like this:
>> > 
>> > Is python2 support required for a well-motivated transitional purpose?
>> > 
>> > I had previously proposed making all our scripts work with both python2 
>> > and python3 only because I thought Apple was going to require python2 
>> > indefinitely. Now that you're interested in this transition, there's 
>> > probably no need to continue python2 support. Anyone building WebKit on 
>> > older versions of macOS can reasonably be expected to manually install 
>> > python3, right? And it's clear that you're prepared to do this for 
>> > infrastructure/bots already.
>> 
>> We still need to figure out whether (and for how long) we intend to retain 
>> Python 2 support. Over the last few months, opinions on this have changed 
>> quite a bit, so I’m trying to determine what our path forward is going to be.
>> 
>> In my opinion, a few months after Catalina GMs, we no longer need to 
>> maintain Python 2 support, assuming that we provide adequate automation for 
>> installing Python 3 on pre-Catalina macOS (ie, Mojave, High Sierra) and are 
>> explicit about shebangs.
> 
> I don't think it's acceptable to require installation of Python 3 just to 
> build & run tests on WebKit unless the installation itself is automated and 
> compartmentalized to WebKit's development given how bad Python is with 
> respect to having multiple versions of Python's and managing packages between 
> them.
> 
> I frequently do WebKit development in older versions of macOS to diagnose old 
> OS specific regressions, and having to install Python 3 each time I install 
> an old OS is too much of a trouble.

It’s possible install a python without installing it on the system, and install 
modules and any needed additions, using virtualenv: 
https://virtualenv.pypa.io/en/stable/

This is the pro way to use python without caring about what happens to be on 
the system. 

 I suggest we proceed by gradually converting our scripts to use a Python 3 
virtualenv. This will also spare us the need to install Python modules onto the 
system. The tricky part will be that webkitpy would have to work both ways 
until conversion is complete (or at the extreme I guess we could fork it).

> - R. Niwa
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Ryosuke Niwa]

On Fri, Jul 12, 2019 at 7:03 PM Michael Catanzaro 
wrote:

> On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa  wrote:
> > I frequently do WebKit development in older versions of macOS to
> > diagnose old OS specific regressions, and having to install Python 3
> > each time I install an old OS is too much of a trouble.
>
> I understand it would be a hassle. :/ But please consider it relative
> to the additional difficulty of rewriting all of webkitpy to support
> multiple versions of python at the same time, or setting up a wrapper
> layer of bash scripts around all of our python scripts to enter a
> virtualenv before executing the real script.
>

Yeah, and it sounds strictly better that the trouble is handled by people
who maintain Python code who presumably more about Python than a random
WebKit contributor who may not know how to setup virtual environment in
Python, etc...

Again, the argument that the difficulty can be overcome and it's a minor
inconvenience isn't convincing. I can, for example, suggest that we should
just build WebKit using the latest version of clang. Anyone who uses a
system that doesn't come with the latest release of clang should just build
clang instead. There is a significant cost in having to support MSVC and
GC++ so we should just use clang everywhere and only the latest version
like Chromium does.

Each team & person has a different preference & perspective when it comes
to tooling. Please don't break someone else's working workflow based on
your preference.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Michael Catanzaro]

On Fri, Jul 12, 2019 at 5:22 PM, Ryosuke Niwa  wrote:
I frequently do WebKit development in older versions of macOS to 
diagnose old OS specific regressions, and having to install Python 3 
each time I install an old OS is too much of a trouble.


I understand it would be a hassle. :/ But please consider it relative 
to the additional difficulty of rewriting all of webkitpy to support 
multiple versions of python at the same time, or setting up a wrapper 
layer of bash scripts around all of our python scripts to enter a 
virtualenv before executing the real script.


Michael


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Ryosuke Niwa]

On Fri, Jul 12, 2019 at 1:04 PM Jonathan Bedard  wrote:

>
> > On Jul 12, 2019, at 12:49 PM, Michael Catanzaro 
> wrote:
> >
> > On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard 
> wrote:
> >> The trouble I foresee us encountering with any scheme which attempts a
> conversion which retains both Python 2.7 and Python 3 compatibility is code
> like this:
> >
> > Is python2 support required for a well-motivated transitional purpose?
> >
> > I had previously proposed making all our scripts work with both python2
> and python3 only because I thought Apple was going to require python2
> indefinitely. Now that you're interested in this transition, there's
> probably no need to continue python2 support. Anyone building WebKit on
> older versions of macOS can reasonably be expected to manually install
> python3, right? And it's clear that you're prepared to do this for
> infrastructure/bots already.
>
> We still need to figure out whether (and for how long) we intend to retain
> Python 2 support. Over the last few months, opinions on this have changed
> quite a bit, so I’m trying to determine what our path forward is going to
> be.
>
> In my opinion, a few months after Catalina GMs, we no longer need to
> maintain Python 2 support, assuming that we provide adequate automation for
> installing Python 3 on pre-Catalina macOS (ie, Mojave, High Sierra) and are
> explicit about shebangs.
>

I don't think it's acceptable to require installation of Python 3 just to
build & run tests on WebKit unless the installation itself is automated and
compartmentalized to WebKit's development given how bad Python is with
respect to having multiple versions of Python's and managing packages
between them.

I frequently do WebKit development in older versions of macOS to diagnose
old OS specific regressions, and having to install Python 3 each time I
install an old OS is too much of a trouble.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Jonathan Bedard]

> On Jul 12, 2019, at 1:07 PM, Keith Rollin  wrote:
> 
>> On Jul 12, 2019, at 13:37, Tim Horton  wrote:
>> 
>> See "Migrating to the common subset of Python 2 and 3” — you define 
>> different iteritems() helpers in the two cases. Seems pretty reasonable to 
>> me.
> 
> I did something like this in webkit-triage. Search for “viewitems” for the 
> compatibility function I used, as well as for “PY3” to see some of the other 
> thunks I put into place.
> 
> — Keith
> 

Actually, hadn’t seen those tricks yet, thanks Tim and Keith for calling them 
out!

Those would allow us to keep scripts both Python 2.7 and Python 3 compatible 
for longer.

Jonathan
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Jonathan Bedard]

> On Jul 12, 2019, at 12:49 PM, Michael Catanzaro  wrote:
> 
> On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard  wrote:
>> The trouble I foresee us encountering with any scheme which attempts a 
>> conversion which retains both Python 2.7 and Python 3 compatibility is code 
>> like this:
> 
> Is python2 support required for a well-motivated transitional purpose?
> 
> I had previously proposed making all our scripts work with both python2 and 
> python3 only because I thought Apple was going to require python2 
> indefinitely. Now that you're interested in this transition, there's probably 
> no need to continue python2 support. Anyone building WebKit on older versions 
> of macOS can reasonably be expected to manually install python3, right? And 
> it's clear that you're prepared to do this for infrastructure/bots already.

We still need to figure out whether (and for how long) we intend to retain 
Python 2 support. Over the last few months, opinions on this have changed quite 
a bit, so I’m trying to determine what our path forward is going to be.

In my opinion, a few months after Catalina GMs, we no longer need to maintain 
Python 2 support, assuming that we provide adequate automation for installing 
Python 3 on pre-Catalina macOS (ie, Mojave, High Sierra) and are explicit about 
shebangs.

> 
> Then million-dollar question is: what shebangs will we use for our scripts? 
> Will #!/usr/bin/env python3 work for Apple?

If we move straight to Python 3, we would need to use the Python 3 shebang. 
Catalina has both Python 2.7 (name ‘python’) and Python 3 (named ‘python3’). I 
think that this is what we would want to do because it allows us to pretty 
explicitly convert scripts one at a time.

> 
> Michael
> 
> 

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Adrian Perez de Castro]

Hello,

On Fri, 12 Jul 2019 12:37:43 -0700, Tim Horton  wrote:

> > On Jul 12, 2019, at 12:18 PM, Jonathan Bedard  wrote:
> > 
> > Hello WebKit developers,
> > 
> > Now that the Catalina developer seeds are available, it is official that
> > the new Mac developer tools come with Python 3. As a result, we need to
> > continue the ongoing discussion about migrating our Python 2.7 scripts to
> > Python 3.

Given that GNU/Linux distributions have started already a while ago switching
it is great that MacOS is now following suit as well \o/

We have a bug already for this: https://bugs.webkit.org/show_bug.cgi?id=184986

If one has to make code compatible with both Python 2.7 and 3.x, my advice
would be to use the Six module, which provides ready-made helpers which
behave consistently across both versions: https://six.readthedocs.io/
 
> > I propose that, over the next 9 months, we do the following:
> > 
> > 1. Make any no-cost Python 3 compatibility changes, in particular
> > - print foo -> print(foo)
> > - import .foo -> import webkitpy.foo
> > 2. Convert any scripts not used in automation to Python 3 ASAP (scripts 
> > like bisect-builds, block-spammers, compare-results)
> > 3. Make most Python 3 compatibility changes which sacrifice efficiency, 
> > subject to a case-by-case audit. These would be things like:
> > - dict.iteritems() -> dict.items()
> > - dict.items() -> list(dict.items())
> > 4. Install Python 3 on macOS Sierra and Mojave bots
> > 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like 
> > clean-webkit, merge-results-json, webkit-patch)
> > 6. Convert testing scripts and webkitpy to Python 3 in a single change
> > 
> > The trouble I foresee us encountering with any scheme which attempts a 
> > conversion which retains both Python 2.7 and Python 3 compatibility is code 
> > like this:
> > 
> > for expectation_string, expectation_enum in 
> > test_expectations.TestExpectations.EXPECTATIONS.iteritems():
> > ...
> > 
> > In this code, the EXPECTATIONS dictionary is thousands of elements long. In 
> > Python 2.7, iteritems() gives us an iterator instead of creating a new 
> > list, like items() would. In Python 3, iteritems() doesn’t exist, but 
> > items() does, and now gives us an iterator instead of creating a new list. 
> > The trouble here is that, in this case, creating a new list will be very 
> > expensive, expensive enough that we might manage to impact the testing run. 
> > There isn’t really an elegant way around this problem if we want to support 
> > both Python 2.7 and Python 3, other than defining different code paths for 
> > each language.
> 
> The official Python 3 transition documentation has a fairly elegant solution 
> to this, actually??
> 
> https://legacy.python.org/dev/peps/pep-0469/
> 
> See "Migrating to the common subset of Python 2 and 3” — you define different 
> iteritems() helpers in the two cases. Seems pretty reasonable to me.

Instead of rolling our own, I would rather use Six (mentioned above). It
covers all the differences with the different “.iter*()” methods:

  https://six.readthedocs.io/#six.iterkeys
  https://six.readthedocs.io/#six.itervalues
  https://six.readthedocs.io/#six.iteritems
  https://six.readthedocs.io/#six.iterlists
  ...

> > There are other small gotchas as well. For example, ‘%’ is no longer a
> > protected character, which can actually change the behavior of regexes.
> > That’s why I think it’s better to just try and directly convert things
> > instead of attempting to be compatible with both Python 2.7 and Python 3.

In my experience some of the major troubles making a codebase compatible
with both Pythons are the string types:

  - In Python 2, “str” is also usable for binary data (basically it's backed
by a char[], while “unicode” is actual text which may contain any code
point.
  - In Python 3, “string” is kind of the old “unicode” (textual data),
and “bytes” is kind of “string” but only holds binary data and in
general cannot be used where a string would have been used.

The Six module helps a bit with the text types (see “six.text_type” and
“six.binary_type”.

Regards,
—Adrián


pgpuX1RTgiWOr.pgp
Description: PGP signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Jonathan Bedard]

> On Jul 12, 2019, at 12:45 PM, Robert Ma  wrote:
> 
> Any thoughts on bytes and Unicode strings, especially the string literals in 
> the code base?

My experience with this has been you mostly have to pay attention to where your 
code interfaces with other processes. In webkitpy, I suspect that most of the 
our work here will probably be where we are calling other commands or reading 
files.

Because we call most commands through either the Executive class or the 
FileSystem class, I think there is a good chance we can make changes here that 
effect most of the codebase.

Jonathan

> 
> On Fri, Jul 12, 2019 at 3:38 PM Tim Horton  > wrote:
> 
> 
>> On Jul 12, 2019, at 12:18 PM, Jonathan Bedard > > wrote:
>> 
>> Hello WebKit developers,
>> 
>> Now that the Catalina developer seeds are available, it is official that the 
>> new Mac developer tools come with Python 3. As a result, we need to continue 
>> the ongoing discussion about migrating our Python 2.7 scripts to Python 3.
>> 
>> I propose that, over the next 9 months, we do the following:
>> 
>> 1. Make any no-cost Python 3 compatibility changes, in particular
>> - print foo -> print(foo)
>> - import .foo -> import webkitpy.foo
>> 2. Convert any scripts not used in automation to Python 3 ASAP (scripts like 
>> bisect-builds, block-spammers, compare-results)
>> 3. Make most Python 3 compatibility changes which sacrifice efficiency, 
>> subject to a case-by-case audit. These would be things like:
>> - dict.iteritems() -> dict.items()
>> - dict.items() -> list(dict.items())
>> 4. Install Python 3 on macOS Sierra and Mojave bots
>> 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like 
>> clean-webkit, merge-results-json, webkit-patch)
>> 6. Convert testing scripts and webkitpy to Python 3 in a single change
>> 
>> The trouble I foresee us encountering with any scheme which attempts a 
>> conversion which retains both Python 2.7 and Python 3 compatibility is code 
>> like this:
>> 
>> for expectation_string, expectation_enum in 
>> test_expectations.TestExpectations.EXPECTATIONS.iteritems():
>> ...
>> 
>> In this code, the EXPECTATIONS dictionary is thousands of elements long. In 
>> Python 2.7, iteritems() gives us an iterator instead of creating a new list, 
>> like items() would. In Python 3, iteritems() doesn’t exist, but items() 
>> does, and now gives us an iterator instead of creating a new list. The 
>> trouble here is that, in this case, creating a new list will be very 
>> expensive, expensive enough that we might manage to impact the testing run. 
>> There isn’t really an elegant way around this problem if we want to support 
>> both Python 2.7 and Python 3, other than defining different code paths for 
>> each language.
> 
> The official Python 3 transition documentation has a fairly elegant solution 
> to this, actually??
> 
> https://legacy.python.org/dev/peps/pep-0469/ 
> 
> 
> See "Migrating to the common subset of Python 2 and 3” — you define different 
> iteritems() helpers in the two cases. Seems pretty reasonable to me.
> 
>> There are other small gotchas as well. For example, ‘%’ is no longer a 
>> protected character, which can actually change the behavior of regexes. 
>> That’s why I think it’s better to just try and directly convert things 
>> instead of attempting to be compatible with both Python 2.7 and Python 3.
>> 
>> Jonathan
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org 
>> https://lists.webkit.org/mailman/listinfo/webkit-dev 
>> 
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org 
> https://lists.webkit.org/mailman/listinfo/webkit-dev 
> 

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Michael Catanzaro]

On Fri, Jul 12, 2019 at 2:18 PM, Jonathan Bedard  
wrote:
The trouble I foresee us encountering with any scheme which attempts 
a conversion which retains both Python 2.7 and Python 3 compatibility 
is code like this:


Is python2 support required for a well-motivated transitional purpose?

I had previously proposed making all our scripts work with both python2 
and python3 only because I thought Apple was going to require python2 
indefinitely. Now that you're interested in this transition, there's 
probably no need to continue python2 support. Anyone building WebKit on 
older versions of macOS can reasonably be expected to manually install 
python3, right? And it's clear that you're prepared to do this for 
infrastructure/bots already.


Then million-dollar question is: what shebangs will we use for our 
scripts? Will #!/usr/bin/env python3 work for Apple?


Michael


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Robert Ma]

Any thoughts on bytes and Unicode strings, especially the string literals
in the code base?

On Fri, Jul 12, 2019 at 3:38 PM Tim Horton  wrote:

>
>
> On Jul 12, 2019, at 12:18 PM, Jonathan Bedard  wrote:
>
> Hello WebKit developers,
>
> Now that the Catalina developer seeds are available, it is official that
> the new Mac developer tools come with Python 3. As a result, we need to
> continue the ongoing discussion about migrating our Python 2.7 scripts to
> Python 3.
>
> I propose that, over the next 9 months, we do the following:
>
> 1. Make any no-cost Python 3 compatibility changes, in particular
> - print foo -> print(foo)
> - import .foo -> import webkitpy.foo
> 2. Convert any scripts not used in automation to Python 3 ASAP (scripts
> like bisect-builds, block-spammers, compare-results)
> 3. Make most Python 3 compatibility changes which sacrifice efficiency,
> subject to a case-by-case audit. These would be things like:
> - dict.iteritems() -> dict.items()
> - dict.items() -> list(dict.items())
> 4. Install Python 3 on macOS Sierra and Mojave bots
> 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like
> clean-webkit, merge-results-json, webkit-patch)
> 6. Convert testing scripts and webkitpy to Python 3 in a single change
>
> The trouble I foresee us encountering with any scheme which attempts a
> conversion which retains both Python 2.7 and Python 3 compatibility is code
> like this:
>
> for expectation_string, expectation_enum in
> test_expectations.TestExpectations.EXPECTATIONS.iteritems():
> ...
>
> In this code, the EXPECTATIONS dictionary is thousands of elements long.
> In Python 2.7, iteritems() gives us an iterator instead of creating a new
> list, like items() would. In Python 3, iteritems() doesn’t exist, but
> items() does, and now gives us an iterator instead of creating a new list.
> The trouble here is that, in this case, creating a new list will be very
> expensive, expensive enough that we might manage to impact the testing run.
> There isn’t really an elegant way around this problem if we want to support
> both Python 2.7 and Python 3, other than defining different code paths for
> each language.
>
>
> The official Python 3 transition documentation has a fairly elegant
> solution to this, actually??
>
> https://legacy.python.org/dev/peps/pep-0469/
>
> See "Migrating to the common subset of Python 2 and 3” — you define
> different iteritems() helpers in the two cases. Seems pretty reasonable to
> me.
>
> There are other small gotchas as well. For example, ‘%’ is no longer a
> protected character, which can actually change the behavior of regexes.
> That’s why I think it’s better to just try and directly convert things
> instead of attempting to be compatible with both Python 2.7 and Python 3.
>
> Jonathan
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Don . Olmstead]

Mentioned this on the other thread but here’s it again.

https://python-modernize.readthedocs.io/en/latest/ and 
http://python-future.org/automatic_conversion.html might be of use considering 
the sheer amount of code. They’re both mentioned in 
https://docs.python.org/3/howto/pyporting.html for migration.

You can run individual rules one by one over the code. That might be ok rather 
than running all the rules at once. Especially for review. Modernize seemed to 
work pretty well but it wasn’t smart enough to detect shebang lines.

From: webkit-dev  
On Behalf Of Tim Horton
Sent: Friday, July 12, 2019 12:38 PM
To: Jonathan Bedard 
Cc: webkit-dev@lists.webkit.org
Subject: Re: [webkit-dev] Moving to Python 3




On Jul 12, 2019, at 12:18 PM, Jonathan Bedard 
mailto:jbed...@apple.com>> wrote:

Hello WebKit developers,

Now that the Catalina developer seeds are available, it is official that the 
new Mac developer tools come with Python 3. As a result, we need to continue 
the ongoing discussion about migrating our Python 2.7 scripts to Python 3.

I propose that, over the next 9 months, we do the following:

1. Make any no-cost Python 3 compatibility changes, in particular
- print foo -> print(foo)
- import .foo -> import webkitpy.foo
2. Convert any scripts not used in automation to Python 3 ASAP (scripts like 
bisect-builds, block-spammers, compare-results)
3. Make most Python 3 compatibility changes which sacrifice efficiency, subject 
to a case-by-case audit. These would be things like:
- dict.iteritems() -> dict.items()
- dict.items() -> list(dict.items())
4. Install Python 3 on macOS Sierra and Mojave bots
5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like 
clean-webkit, merge-results-json, webkit-patch)
6. Convert testing scripts and webkitpy to Python 3 in a single change

The trouble I foresee us encountering with any scheme which attempts a 
conversion which retains both Python 2.7 and Python 3 compatibility is code 
like this:

for expectation_string, expectation_enum in 
test_expectations.TestExpectations.EXPECTATIONS.iteritems():
...

In this code, the EXPECTATIONS dictionary is thousands of elements long. In 
Python 2.7, iteritems() gives us an iterator instead of creating a new list, 
like items() would. In Python 3, iteritems() doesn’t exist, but items() does, 
and now gives us an iterator instead of creating a new list. The trouble here 
is that, in this case, creating a new list will be very expensive, expensive 
enough that we might manage to impact the testing run. There isn’t really an 
elegant way around this problem if we want to support both Python 2.7 and 
Python 3, other than defining different code paths for each language.

The official Python 3 transition documentation has a fairly elegant solution to 
this, actually??

https://legacy.python.org/dev/peps/pep-0469/

See "Migrating to the common subset of Python 2 and 3” — you define different 
iteritems() helpers in the two cases. Seems pretty reasonable to me.


There are other small gotchas as well. For example, ‘%’ is no longer a 
protected character, which can actually change the behavior of regexes. That’s 
why I think it’s better to just try and directly convert things instead of 
attempting to be compatible with both Python 2.7 and Python 3.

Jonathan
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Moving to Python 3

[Tim Horton]

> On Jul 12, 2019, at 12:18 PM, Jonathan Bedard  wrote:
> 
> Hello WebKit developers,
> 
> Now that the Catalina developer seeds are available, it is official that the 
> new Mac developer tools come with Python 3. As a result, we need to continue 
> the ongoing discussion about migrating our Python 2.7 scripts to Python 3.
> 
> I propose that, over the next 9 months, we do the following:
> 
> 1. Make any no-cost Python 3 compatibility changes, in particular
> - print foo -> print(foo)
> - import .foo -> import webkitpy.foo
> 2. Convert any scripts not used in automation to Python 3 ASAP (scripts like 
> bisect-builds, block-spammers, compare-results)
> 3. Make most Python 3 compatibility changes which sacrifice efficiency, 
> subject to a case-by-case audit. These would be things like:
> - dict.iteritems() -> dict.items()
> - dict.items() -> list(dict.items())
> 4. Install Python 3 on macOS Sierra and Mojave bots
> 5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like 
> clean-webkit, merge-results-json, webkit-patch)
> 6. Convert testing scripts and webkitpy to Python 3 in a single change
> 
> The trouble I foresee us encountering with any scheme which attempts a 
> conversion which retains both Python 2.7 and Python 3 compatibility is code 
> like this:
> 
> for expectation_string, expectation_enum in 
> test_expectations.TestExpectations.EXPECTATIONS.iteritems():
> ...
> 
> In this code, the EXPECTATIONS dictionary is thousands of elements long. In 
> Python 2.7, iteritems() gives us an iterator instead of creating a new list, 
> like items() would. In Python 3, iteritems() doesn’t exist, but items() does, 
> and now gives us an iterator instead of creating a new list. The trouble here 
> is that, in this case, creating a new list will be very expensive, expensive 
> enough that we might manage to impact the testing run. There isn’t really an 
> elegant way around this problem if we want to support both Python 2.7 and 
> Python 3, other than defining different code paths for each language.

The official Python 3 transition documentation has a fairly elegant solution to 
this, actually??

https://legacy.python.org/dev/peps/pep-0469/

See "Migrating to the common subset of Python 2 and 3” — you define different 
iteritems() helpers in the two cases. Seems pretty reasonable to me.

> There are other small gotchas as well. For example, ‘%’ is no longer a 
> protected character, which can actually change the behavior of regexes. 
> That’s why I think it’s better to just try and directly convert things 
> instead of attempting to be compatible with both Python 2.7 and Python 3.
> 
> Jonathan
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

[webkit-dev] Moving to Python 3

[Jonathan Bedard]

Hello WebKit developers,

Now that the Catalina developer seeds are available, it is official that the 
new Mac developer tools come with Python 3. As a result, we need to continue 
the ongoing discussion about migrating our Python 2.7 scripts to Python 3.

I propose that, over the next 9 months, we do the following:

1. Make any no-cost Python 3 compatibility changes, in particular
- print foo -> print(foo)
- import .foo -> import webkitpy.foo
2. Convert any scripts not used in automation to Python 3 ASAP (scripts like 
bisect-builds, block-spammers, compare-results)
3. Make most Python 3 compatibility changes which sacrifice efficiency, subject 
to a case-by-case audit. These would be things like:
- dict.iteritems() -> dict.items()
- dict.items() -> list(dict.items())
4. Install Python 3 on macOS Sierra and Mojave bots
5. Convert peripheral automation scripts to Python 3 1-by-1 (scripts like 
clean-webkit, merge-results-json, webkit-patch)
6. Convert testing scripts and webkitpy to Python 3 in a single change

The trouble I foresee us encountering with any scheme which attempts a 
conversion which retains both Python 2.7 and Python 3 compatibility is code 
like this:

for expectation_string, expectation_enum in 
test_expectations.TestExpectations.EXPECTATIONS.iteritems():
...

In this code, the EXPECTATIONS dictionary is thousands of elements long. In 
Python 2.7, iteritems() gives us an iterator instead of creating a new list, 
like items() would. In Python 3, iteritems() doesn’t exist, but items() does, 
and now gives us an iterator instead of creating a new list. The trouble here 
is that, in this case, creating a new list will be very expensive, expensive 
enough that we might manage to impact the testing run. There isn’t really an 
elegant way around this problem if we want to support both Python 2.7 and 
Python 3, other than defining different code paths for each language.

There are other small gotchas as well. For example, ‘%’ is no longer a 
protected character, which can actually change the behavior of regexes. That’s 
why I think it’s better to just try and directly convert things instead of 
attempting to be compatible with both Python 2.7 and Python 3.

Jonathan___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] What's the current Safari UA?

[Michael Catanzaro]

I received a good answer and will upgrade our versions accordingly. 
Thanks.



___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

[webkit-dev] What's the current Safari UA?

[Michael Catanzaro]

Hi,

Relevant to [1], could someone please check what Safari's current UA 
string is, please? This is important to help me make sure WebKitGTK 
fakes the Safari user agent as well as possible. In particular, I'd 
like to know the version numbers used in the UA. I had thought we were 
frozen on:


Version/11.0 Safari/605.1.15

forever, but perhaps the Version/ component is not frozen and we're up 
to Version/12.0 or Version/12.1 now? I hope the 605.1.15, at least, is 
still frozen.


Also, for those naughty websites that require we use some serious 
fakery and pretend to be macOS, we currently use the string "Macintosh; 
Intel Mac OS X 10_13_4". I understand that Apple's previous attempt to 
freeze that value was not successful, so we still need to update this 
from time to time. What does it look like on, say, Mojave or Catalina?


Thanks,

Michael

[1] https://bugs.webkit.org/show_bug.cgi?id=199745


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev