[Bug 1804062] Re: test_hacking fails for python 3.6.7 and newer

2019-06-14 Thread OpenStack Infra
Reviewed:  https://review.opendev.org/665147
Committed: 
https://git.openstack.org/cgit/openstack/nova/commit/?id=0cb6106b83c33bded9e6cdec7737964c36be8de5
Submitter: Zuul
Branch:stable/stein

commit 0cb6106b83c33bded9e6cdec7737964c36be8de5
Author: Stephen Finucane 
Date:   Wed Jun 12 15:10:59 2019 +0100

Fix double word hacking test

At present, 'pycodestyle' feeds the following string into the 'tokenizer'
library:

["'This is the the best comment'"]

(note the added quotes because this isn't valid Python otherwise)

On previous versions of Python, this tokenizer would parse the string like 
so:

  (3, "'This is the the best comment'", (1, 0), (1, 30), "'This is the the 
best comment'")
  (0, '', (2, 0), (2, 0), '')

where (3 = 'STRING', 0 = 'ENDMARKER')

However, with the fix [1] backported to recent versions of Python, this now
resolves to:

  (3, "'This is the the best comment'", (1, 0), (1, 30), "'This is the the 
best comment'")
  (4, '', (1, 30), (1, 31), '')
  (0, '', (2, 0), (2, 0), '')

where (3 = 'STRING', 4 = 'NEWLINE', 0 = 'ENDMARKER')

Typically, 'pycodestyle' will run physical line checks on each line as it
parses the token:

  https://github.com/PyCQA/pycodestyle/blob/2.5.0/pycodestyle.py#L2036

For the former case above, the line doesn't include a newline which
means we never parse a 'NEWLINE' token with a logical line (the fifth
element of the token tuple) corresponding to our full line. This means
we don't here but that wasn't an issue previously since there's a
fallthrough case that handled tokens remaining at the end of the parse:

  https://github.com/PyCQA/pycodestyle/blob/2.5.0/pycodestyle.py#L2114-L2116

Unfortunately, because we now have an additional newline character to
parse, one that's on a separate line to our test string no less, we run
logical checks on it:

  https://github.com/PyCQA/pycodestyle/blob/2.5.0/pycodestyle.py#L2105-L2107

This is an issue since the logical check wipes stored tokens meaning
we've nothing to check when we get to the fallthrough case:

  https://github.com/PyCQA/pycodestyle/blob/2.5.0/pycodestyle.py#L2012

This fixes changes things so that a newline is included (and also adds
quotes so it's valid Python, but that's mostly unrelated). This means we
end up with the following instead:

  ["'This is the the best comment'\n"]

On both Python without the bugfix and with it, this parses as:

  (3, "'This is the the best comment'", (1, 0), (1, 30), "'This is the the 
best comment'\n")
  (4, '\n', (1, 30), (1, 31), "'This is the the best comment'\n")
  (0, '', (2, 0), (2, 0), '')

where (3 = 'STRING', 4 = 'NEWLINE', 0 = 'ENDMARKER')

Which triggers things in 'pycodestyle' correctly.

  https://github.com/PyCQA/pycodestyle/blob/2.5.0/pycodestyle.py#L2044-L2046

This isn't _really_ a fix since there's clearly still a bug in either
'pycodestyle' or Python (I think the latter, since it's adding a newline
to a file that explicitly doesn't have one), but the chances of us
hitting this bug in practice are rather low - you'd need to make a
mistake on the very last line of a file without a newline at the end
which is something Vim, for example, won't even let you do without
setting special flags - and therefore it can be reasonably ignored.

Conflicts:
nova/tests/unit/test_hacking.py

NOTE(stephenfin): Conflicts are because we don't have change
I35c654bd39f343417e0a1124263ff31dcd0b05c9 ("Bump to hacking 1.1.0") or
change I8826c3fb89690805baae6b9b7b48985abb8d62d3 ("Skip
test_check_doubled_words hacking check UT") on this branch.

[1] https://bugs.python.org/issue33899

Change-Id: Ia597594e0469c0e83d7ad22b0678390aaebaffe7
Signed-off-by: Stephen Finucane 
Closes-Bug: #1804062
(cherry picked from f545a25cc443c41dcd9bdd028064c28b53f56037)


** Changed in: cloud-archive/stein
   Status: Triaged => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1804062

Title:
  test_hacking fails for python 3.6.7 and newer

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1804062/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1804062] Re: test_hacking fails for python 3.6.7 and newer

2019-06-12 Thread OpenStack Infra
Reviewed:  https://review.opendev.org/664940
Committed: 
https://git.openstack.org/cgit/openstack/nova/commit/?id=f545a25cc443c41dcd9bdd028064c28b53f56037
Submitter: Zuul
Branch:master

commit f545a25cc443c41dcd9bdd028064c28b53f56037
Author: Stephen Finucane 
Date:   Wed Jun 12 15:10:59 2019 +0100

Fix double word hacking test

At present, 'pycodestyle' feeds the following string into the 'tokenizer'
library:

["'This is the the best comment'"]

(note the added quotes because this isn't valid Python otherwise)

On previous versions of Python, this tokenizer would parse the string like 
so:

  (3, "'This is the the best comment'", (1, 0), (1, 30), "'This is the the 
best comment'")
  (0, '', (2, 0), (2, 0), '')

where (3 = 'STRING', 0 = 'ENDMARKER')

However, with the fix [1] backported to recent versions of Python, this now
resolves to:

  (3, "'This is the the best comment'", (1, 0), (1, 30), "'This is the the 
best comment'")
  (4, '', (1, 30), (1, 31), '')
  (0, '', (2, 0), (2, 0), '')

where (3 = 'STRING', 4 = 'NEWLINE', 0 = 'ENDMARKER')

Typically, 'pycodestyle' will run physical line checks on each line as it
parses the token:

  https://github.com/PyCQA/pycodestyle/blob/2.5.0/pycodestyle.py#L2036

For the former case above, the line doesn't include a newline which
means we never parse a 'NEWLINE' token with a logical line (the fifth
element of the token tuple) corresponding to our full line. This means
we don't here but that wasn't an issue previously since there's a
fallthrough case that handled tokens remaining at the end of the parse:

  https://github.com/PyCQA/pycodestyle/blob/2.5.0/pycodestyle.py#L2114-L2116

Unfortunately, because we now have an additional newline character to
parse, one that's on a separate line to our test string no less, we run
logical checks on it:

  https://github.com/PyCQA/pycodestyle/blob/2.5.0/pycodestyle.py#L2105-L2107

This is an issue since the logical check wipes stored tokens meaning
we've nothing to check when we get to the fallthrough case:

  https://github.com/PyCQA/pycodestyle/blob/2.5.0/pycodestyle.py#L2012

This fixes changes things so that a newline is included (and also adds
quotes so it's valid Python, but that's mostly unrelated). This means we
end up with the following instead:

  ["'This is the the best comment'\n"]

On both Python without the bugfix and with it, this parses as:

  (3, "'This is the the best comment'", (1, 0), (1, 30), "'This is the the 
best comment'\n")
  (4, '\n', (1, 30), (1, 31), "'This is the the best comment'\n")
  (0, '', (2, 0), (2, 0), '')

where (3 = 'STRING', 4 = 'NEWLINE', 0 = 'ENDMARKER')

Which triggers things in 'pycodestyle' correctly.

  https://github.com/PyCQA/pycodestyle/blob/2.5.0/pycodestyle.py#L2044-L2046

This isn't _really_ a fix since there's clearly still a bug in either
'pycodestyle' or Python (I think the latter, since it's adding a newline
to a file that explicitly doesn't have one), but the chances of us
hitting this bug in practice are rather low - you'd need to make a
mistake on the very last line of a file without a newline at the end
which is something Vim, for example, won't even let you do without
setting special flags - and therefore it can be reasonably ignored.

[1] https://bugs.python.org/issue33899

Change-Id: Ia597594e0469c0e83d7ad22b0678390aaebaffe7
Signed-off-by: Stephen Finucane 
Closes-Bug: #1804062


** Changed in: nova
   Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1804062

Title:
  test_hacking fails for python 3.6.7 and newer

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1804062/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1804062] Re: test_hacking fails for python 3.6.7 and newer

2019-06-12 Thread OpenStack Infra
Fix proposed to branch: master
Review: https://review.opendev.org/664940

** Changed in: nova
   Status: Confirmed => In Progress

** Changed in: nova
 Assignee: (unassigned) => Stephen Finucane (stephenfinucane)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1804062

Title:
  test_hacking fails for python 3.6.7 and newer

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1804062/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1804062] Re: test_hacking fails for python 3.6.7 and newer

2019-05-03 Thread Corey Bryant
I'm tracking the Ubuntu SRU for the work-around in
https://bugs.launchpad.net/bugs/1782786.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1804062

Title:
  test_hacking fails for python 3.6.7 and newer

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1804062/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1804062] Re: test_hacking fails for python 3.6.7 and newer

2019-05-03 Thread Corey Bryant
** Also affects: cloud-archive/rocky
   Importance: Undecided
   Status: New

** Also affects: cloud-archive/queens
   Importance: Undecided
   Status: New

** Also affects: cloud-archive/stein
   Importance: High
   Status: Triaged

** Changed in: cloud-archive/queens
   Status: New => Triaged

** Changed in: cloud-archive/rocky
   Status: New => Triaged

** Changed in: cloud-archive/queens
   Importance: Undecided => High

** Changed in: cloud-archive/rocky
   Importance: Undecided => High

** Also affects: nova (Ubuntu Eoan)
   Importance: High
   Status: Triaged

** Also affects: nova (Ubuntu Bionic)
   Importance: Undecided
   Status: New

** Also affects: nova (Ubuntu Disco)
   Importance: Undecided
   Status: New

** Also affects: nova (Ubuntu Cosmic)
   Importance: Undecided
   Status: New

** Changed in: nova (Ubuntu Bionic)
   Importance: Undecided => High

** Changed in: nova (Ubuntu Bionic)
   Status: New => Triaged

** Changed in: nova (Ubuntu Cosmic)
   Importance: Undecided => High

** Changed in: nova (Ubuntu Cosmic)
   Status: New => Triaged

** Changed in: nova (Ubuntu Disco)
   Importance: Undecided => High

** Changed in: nova (Ubuntu Disco)
   Status: New => Triaged

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1804062

Title:
  test_hacking fails for python 3.6.7 and newer

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1804062/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1804062] Re: test_hacking fails for python 3.6.7 and newer

2019-05-03 Thread Corey Bryant
I'm also hitting this on Bionic (bionic-proposed) with the final 2.7.15
release. So this is currently seen on bionic and above with 2.7/3.6/3.7.
Ubuntu package versions can be tracked at:

https://launchpad.net/ubuntu/+source/python2.7
https://launchpad.net/ubuntu/+source/python3.6
https://launchpad.net/ubuntu/+source/python3.7

** Also affects: nova (Ubuntu)
   Importance: Undecided
   Status: New

** Also affects: cloud-archive
   Importance: Undecided
   Status: New

** Changed in: cloud-archive
   Status: New => Triaged

** Changed in: nova (Ubuntu)
   Status: New => Triaged

** Changed in: cloud-archive
   Importance: Undecided => High

** Changed in: nova (Ubuntu)
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1804062

Title:
  test_hacking fails for python 3.6.7 and newer

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1804062/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs