Hi Anand, Thanks for the feedback.
It should be no problem to run everything on DataflowRunner as well. Are there any performance tests in place to check for performance regressions? Some questions were raised in the proposal document which I want to add to this conversation: The first comment was about the targeted python 3 versions. We proposed to target 3.6 since it is the latest version available and added 3.5 because 3.6 adoption seems rather low (hard to find any relevant sources on this though). If the beam community prefers 3.4, I would propose to target 3.4 only during porting and add 3.5 and 3.6 later so we don't slow down the porting progress. 3.4 has the advantage of already being installed on the workers and allows pySpark pipelines to be moved over to beam more easily. It would be great to get some opinions on this. Another comment was made on how to avoid regression during the porting progress. After applying step 1 and step 2, no python 3 compatibility lint warnings should remain, so it would be great if we could enforce this check for every pull request on an already updated subpackage. After applying step 3, all tests should run on python 3, so again it would be great if we can enforce these per updated subpackage. Any insights on how to best accomplish this? Thanks, Robbe On Fri, 23 Mar 2018 at 19:59 Ahmet Altay <al...@google.com> wrote: > Thank you Robbe. > > I reviewed the document it looks reasonable to me. I will touch on some > points that were not mentioned: > - Runner exercise different code paths. Doing auto conversions and > focusing on DirectRunner is not enough. It is worthwhile to run things on > DataflowRunner as well. This can be triggered from Jenkins. It will > validate that we are still compatible for python 2. > - Similar to above but with an eye on perf regressions. > > For project tracking on JIRA, please feel free to create any new issues, > close stale ones, or take ownership of any open issues. All JIRAs should be > assigned to the people actively working on them. If you wan to track it in > a separate way, you can also propose that. (For example a kanban board is > used for portability effort which is fully supported in JIRA.) > > I will also call out to a few other people in addition to Holden who > helped out or showed interest in helping with Python 3. @cclaus, @luke-zhu, > @udim, @robertwb, @charlesccychen, @tvalentyn. You can include these > people (and myself) for reviews and other questions that you have. > > Welcome again, and looking forward to your contributions. > > Thank you, > Ahmet > > > > On Fri, Mar 23, 2018 at 9:27 AM, Robbe Sneyders <robbe.sneyd...@ml6.eu> > wrote: > >> Hello everyone, >> >> In the next month(s), me and my colleague Matthias will commit a lot of >> time and effort to python 3 support for beam and we would like to discuss >> the best way to go forward with this. >> >> We have drawn up a document [1] with a high level outline of the proposed >> approach and would like to get your feedback on this. >> >> The main Jira issue [2] for python 3 support has been mostly inactive for >> the past year. Other smaller issues have been opened, but it's hard to >> track the general progress. It would be great if anyone could offer some >> insights on how to best handle this project on Jira. >> >> @Holden Karau, you seem to have already put in a lot of effort to add >> python 3 support, so it would be great to get your insights and find a way >> to merge our efforts. >> >> Kind regards, >> Robbe >> >> [1] >> https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE/edit?usp=sharing >> >> [2] https://issues.apache.org/jira/browse/BEAM-1251 >> -- >> >> [image: https://ml6.eu] <https://ml6.eu/> >> >> * Robbe Sneyders* >> >> ML6 Gent >> <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl> >> >> M: +32 474 71 31 08 <+32%20474%2071%2031%2008> >> > > -- [image: https://ml6.eu] <https://ml6.eu/> * Robbe Sneyders* ML6 Gent <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl> M: +32 474 71 31 08
