[
https://issues.apache.org/jira/browse/BEAM-2784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143031#comment-16143031
]
ASF GitHub Bot commented on BEAM-2784:
--------------------------------------
GitHub user holdenk opened a pull request:
https://github.com/apache/beam/pull/3772
[WIP][BEAM-2784] Run python 2 to 3 migration and fix resulting Python 2
errors
This is a WIP pull request that is the result of running the Python 2 to 3
migration and then manually fixing the errors that broke in Python 2.7 support
after the auto migration. This does not mean the code is ready for Python 3
after this -- although I am working on a follow up for that in BEAM-1373.
Most of the 2/3 compatibility is handled with the future library, although
in some places where we interact with libraries that use six to achieve 2/3
compatibility I've used six as well so that we can inter-operate.
This pull request is rather large, but because it involves changing the
types of many things it's would be challenging to isolate this into smaller
chunks could prove to be difficult.
This PR is tagged as WIP since I still need to go through it again by hand
and review the automatic changes (just because the tests pass with the fixes
doesn't mean its all good), but if some of the other folks who care about BEAM
on Py3 are around I'd appreciate some idea if this is going in the direction we
want. There are also a large number of style issues to fix from automated
tooling as well.
In addition to the automated futurization isort & limited autopep8 was also
applied (and then manually modified since we need to do some silly things to
work between 2/3 with keeping raw & compatible objects around)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/holdenk/beam
BEAM-2784-py2t3-support-python2-still-squash-pandas-r2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3772.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3772
----
commit 897fbac648a79222530a23a2a530dba08c294410
Author: Holden Karau <[email protected]>
Date: 2017-08-19T05:19:10Z
Initial first pass with futurize
commit 7527e21553d96bc39e469351dc6137b5f77dc928
Author: Holden Karau <[email protected]>
Date: 2017-08-19T05:21:18Z
Fix up the errors introduced by the automatic conversion. This is a squash
of a somewhat long process to fix all of these:
Start manually fixing up the errors.
Explicitly depend on the future package
Attempt to fix encoder issues
List was doing nothing
Use has_attr func correctly
More fixes
Set default encoding to latin-1 in some places since otherwise we run into
issues with using strings to hold codepoint above 128. Switch coder_impl to
check for basestring
Future holden says thanks for basestirng
Use Bytes IO for avroio
Try and fix assertItemsEqual py2 again
be explicit about newint
sys + b string
sys import
Explicit encode before putting into a pb2
Use BytesIO
Add missing import
Fix newint tuple magic
force positions to int
Add ignore unicode prefix annotation from Spark
Try assertItemsEqual again
Stream change
More coder fixes
Rewrite exception messages for newint to int so we can be consistent
between Py2 and Py3 with the type inferance
try converting to a string early idk
Progress around type hints being finicky now that we have future
Move _rewrite_typehint_string around, try and cleanup some builtin imports
Tentative: use basestring for str in Py2 which is a bit sketchy
Remove from builtins import str and minor fixes
Fix some more coding issues
Add an explicit test for the new coder, include the new method from the
changed inheritance, add workaround for unicode.
Second missing next
Change some ptransform tests
Fix the unicode workaround import (oops)
Fix raising the error during retry
Ok so it "works" for the one test we've been iterating on but has lots of
print debugging cause ohgod
Remove a bunch of cruft debugging added in and some things that took us
down the wrong path as well with intermediate helper transform type annotations
Keep repr as is
Make SequenceTypeConstraint an instance of IndexableTypeConstraint. Add
explicit tests in trivial_inference_test for indexing. Now at 1260 tests
passing and 256 skipped :)
Small gcp related fixes
gcsio test fixes
gcsio and pylint fixes
Import order fix
autopep8 W391,W293,W291,E306,E305,E304,E303
We don't support 3.4 yet
Fix workaround for object magic
Ok so the underlying transfer library is using six so lets use it too
Ok make the tests more like normal
Add in missing return and factor out format result since long line
----
> Fix issues from automated conversion to allow Python 2 functionality
> --------------------------------------------------------------------
>
> Key: BEAM-2784
> URL: https://issues.apache.org/jira/browse/BEAM-2784
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py
> Reporter: holdenk
> Assignee: Ahmet Altay
>
> As part of BEAM-1251 we want to move to support a Python2/3 code base. To do
> this we can use futurize but futurize will break some Python2 elements. A
> good intermediate checkpoint is contiuing to support Python 2 after
> futurization from which we can build Python 3 support on top of.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
