Thanks! Can someone give me permission to assign issues to myself? And edit rights to the Kanban board?
Robbe On Tue, 17 Apr 2018 at 22:56 Ahmet Altay <al...@google.com> wrote: > Kanban board for python 3: > https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245 > > (Thank you Davor!) > > Ahmet > > On Fri, Apr 6, 2018 at 6:32 PM, Reuven Lax <re...@google.com> wrote: > >> I had a similar problem. >> >> On Fri, Apr 6, 2018, 6:23 PM Ahmet Altay <al...@google.com> wrote: >> >>> I tried to create a shared kanban board but I failed. I think I am >>> lacking some permission to create a shared filter. Could someone help with >>> creating this? >>> >>> The filter I planned to use was "project = BEAM AND (parent = BEAM-2784 >>> OR parent = BEAM-1251) ORDER BY Rank ASC" >>> >>> Ahmet >>> >>> On Fri, Apr 6, 2018 at 5:45 AM, Robbe Sneyders <robbe.sneyd...@ml6.eu> >>> wrote: >>> >>>> Hi all, >>>> >>>> I don't seem to have the permissions to create a Kanban board or even >>>> assign tasks to myself. Who could help me with this? >>>> >>>> I've updated the coders package pull request [1] and added the applied >>>> strategy to the proposal document [2]. >>>> It would be great to get some feedback on this, so we can start moving >>>> forward with other subpackages. >>>> >>>> Kind regards, >>>> Robbe >>>> >>>> [1] https://github.com/apache/beam/pull/4990 >>>> [2] >>>> https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE/edit?usp=sharing >>>> >>>> >>>> On Mon, 2 Apr 2018 at 21:07 Robbe Sneyders <robbe.sneyd...@ml6.eu> >>>> wrote: >>>> >>>>> Hello Robert, >>>>> >>>>> I think a Kanban board on Jira as proposed by Ahmet can be helpful for >>>>> this. I'll look into setting one up tomorrow. >>>>> >>>>> In the meantime, you can find the first pull request with the updated >>>>> coders package here: >>>>> https://github.com/apache/beam/pull/4990 >>>>> >>>>> Kind regards, >>>>> Robbe >>>>> >>>>> On Fri, 30 Mar 2018 at 18:01 Robert Bradshaw <rober...@google.com> >>>>> wrote: >>>>> >>>>>> On Fri, Mar 30, 2018 at 8:39 AM Robbe Sneyders <robbe.sneyd...@ml6.eu> >>>>>> wrote: >>>>>> >>>>>>> Thanks Ahmet and Robert, >>>>>>> >>>>>>> I think we can work on different subpackages in parallel, but it's >>>>>>> important to apply the same strategy everywhere. I'm currently working >>>>>>> on >>>>>>> applying step 1 (was mostly done already) and 2 of the proposal to the >>>>>>> coders subpackage to create a first pull request. We can then discuss >>>>>>> the >>>>>>> applied strategy in detail before merging and applying it to the other >>>>>>> subpackages. >>>>>>> >>>>>> >>>>>> Sounds good. Again, could you document (in a more permanent/easy to >>>>>> look up state than email) when packages are started/done? >>>>>> >>>>>> >>>>>>> This strategy also includes the choice of automated tools. I'm >>>>>>> focusing on writing python 3 code with python 2 compatibility, which >>>>>>> means >>>>>>> depending on the future package instead of the six package (which is >>>>>>> already used in some places in the current code base). I have already >>>>>>> noticed that this indeed requires a lot of manual work after running the >>>>>>> automated script. >>>>>>> The future package supports python 3.3+ compatibility, so I don't >>>>>>> think there is a higher cost supporting 3.4 compared to 3.5+. >>>>>>> >>>>>> >>>>>> Sure. It may incur a higher maintenance burden long-term though. >>>>>> (Basically, if we go out the door with 3.4 it's a promise to support it >>>>>> for >>>>>> some time to come.) >>>>>> >>>>>> >>>>>>> I have already added a tox environment to run pylint2 with the >>>>>>> --py3k argument per updated subpackage, which should help avoid >>>>>>> regression >>>>>>> between step 2 and step 3 of the proposal. This update will be pushed >>>>>>> with >>>>>>> the first pull request. >>>>>>> >>>>>>> Kind regards, >>>>>>> Robbe >>>>>>> >>>>>>> >>>>>>> On Fri, 30 Mar 2018 at 02:22 Robert Bradshaw <rober...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thank you, Robbie, for your offer to help with contribution here. I >>>>>>>> read over your doc and the one thing I'd like to add is that this work >>>>>>>> is >>>>>>>> very parallelizable, but if we have enough people looking at it we'll >>>>>>>> want >>>>>>>> some way to coordinate so as to not overlap work (or just waste time >>>>>>>> discovering what's been done). Tracking individual JIRAs and PRs gets >>>>>>>> unwieldy, perhaps a spreadsheet with modules/packages on one axis and >>>>>>>> the >>>>>>>> various automated/manual conversions along the other would be helpful? >>>>>>>> >>>>>>>> A note on automated tools, they're sometimes overly conservative, >>>>>>>> so we should be sure to review the changes manually. (A typical >>>>>>>> example of >>>>>>>> this is unnecessarily importing six.moves.xrange when there was no big >>>>>>>> reason to use xrange over range in Python 2, or conversely using >>>>>>>> list(range(...) in Python 3.) >>>>>>>> >>>>>>>> Also, +1 to targetting 3.4+ and upgrading tox to prevent >>>>>>>> regressions. If there's a cost to supporting 3.4 as opposed to >>>>>>>> requiring >>>>>>>> 3.5+ we should identify it and decide that before widespread >>>>>>>> announcement. >>>>>>>> >>>>>>>> On Tue, Mar 27, 2018 at 2:27 PM Ahmet Altay <al...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Mar 27, 2018 at 7:12 AM, Holden Karau < >>>>>>>>> hol...@pigscanfly.ca> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Mar 27, 2018 at 4:27 AM Robbe Sneyders < >>>>>>>>>> robbe.sneyd...@ml6.eu> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Anand, >>>>>>>>>>> >>>>>>>>>>> Thanks for the feedback. >>>>>>>>>>> >>>>>>>>>>> It should be no problem to run everything on DataflowRunner as >>>>>>>>>>> well. >>>>>>>>>>> Are there any performance tests in place to check for >>>>>>>>>>> performance regressions? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> Yes there is a suite ( >>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PerformanceTests_Python.groovy). >>>>>>>>> It may not be very comprehensive and seems to be failing for a while. >>>>>>>>> I >>>>>>>>> would not block python 3 work on performance for now. That is the >>>>>>>>> unfortuante state of things. >>>>>>>>> >>>>>>>>> If anybody in the community is interested, this would be a great >>>>>>>>> opportunity to help with benchmarks in general. >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Some questions were raised in the proposal document which I want >>>>>>>>>>> to add to this conversation: >>>>>>>>>>> >>>>>>>>>>> The first comment was about the targeted python 3 versions. We >>>>>>>>>>> proposed to target 3.6 since it is the latest version available and >>>>>>>>>>> added >>>>>>>>>>> 3.5 because 3.6 adoption seems rather low (hard to find any relevant >>>>>>>>>>> sources on this though). >>>>>>>>>>> If the beam community prefers 3.4, I would propose to target 3.4 >>>>>>>>>>> only during porting and add 3.5 and 3.6 later so we don't slow down >>>>>>>>>>> the >>>>>>>>>>> porting progress. 3.4 has the advantage of already being installed >>>>>>>>>>> on the >>>>>>>>>>> workers and allows pySpark pipelines to be moved over to beam more >>>>>>>>>>> easily. >>>>>>>>>>> It would be great to get some opinions on this. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> My preference is to support 3.4+. I searched a bit on the web to >>>>>>>>> understand the usage statistics for python 3, it seems like python >>>>>>>>> 3.4 has >>>>>>>>> ~20% usage and python 3.4+ has 99% ( >>>>>>>>> https://semaphoreci.com/blog/2017/10/18/python-versions-used-in-commercial-projects-in-2017.html). >>>>>>>>> Based on that, I think it makes sense to support it. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Another comment was made on how to avoid regression during the >>>>>>>>>>> porting progress. >>>>>>>>>>> After applying step 1 and step 2, no python 3 compatibility lint >>>>>>>>>>> warnings should remain, so it would be great if we could enforce >>>>>>>>>>> this check >>>>>>>>>>> for every pull request on an already updated subpackage. >>>>>>>>>>> After applying step 3, all tests should run on python 3, so >>>>>>>>>>> again it would be great if we can enforce these per updated >>>>>>>>>>> subpackage. >>>>>>>>>>> Any insights on how to best accomplish this? >>>>>>>>>>> >>>>>>>>>> So you can look at some of the recent changes to tox.ini in the >>>>>>>>>> git log to see what we’ve done so far around this I suspect you can >>>>>>>>>> repeat >>>>>>>>>> that same pattern. >>>>>>>>>> >>>>>>>>> >>>>>>>>> +1 updating tox.ini and adding new checks to run_mini_py3lint.sh >>>>>>>>> would help a lot to prevent regressions. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Robbe >>>>>>>>>>> >>>>>>>>>>> On Fri, 23 Mar 2018 at 19:59 Ahmet Altay <al...@google.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thank you Robbe. >>>>>>>>>>>> >>>>>>>>>>>> I reviewed the document it looks reasonable to me. I will touch >>>>>>>>>>>> on some points that were not mentioned: >>>>>>>>>>>> - Runner exercise different code paths. Doing auto conversions >>>>>>>>>>>> and focusing on DirectRunner is not enough. It is worthwhile to >>>>>>>>>>>> run things >>>>>>>>>>>> on DataflowRunner as well. This can be triggered from Jenkins. It >>>>>>>>>>>> will >>>>>>>>>>>> validate that we are still compatible for python 2. >>>>>>>>>>>> - Similar to above but with an eye on perf regressions. >>>>>>>>>>>> >>>>>>>>>>>> For project tracking on JIRA, please feel free to create any >>>>>>>>>>>> new issues, close stale ones, or take ownership of any open >>>>>>>>>>>> issues. All >>>>>>>>>>>> JIRAs should be assigned to the people actively working on them. >>>>>>>>>>>> If you wan >>>>>>>>>>>> to track it in a separate way, you can also propose that. (For >>>>>>>>>>>> example a >>>>>>>>>>>> kanban board is used for portability effort which is fully >>>>>>>>>>>> supported in >>>>>>>>>>>> JIRA.) >>>>>>>>>>>> >>>>>>>>>>>> I will also call out to a few other people in addition to >>>>>>>>>>>> Holden who helped out or showed interest in helping with Python 3. >>>>>>>>>>>> @cclaus, >>>>>>>>>>>> @luke-zhu, @udim, @robertwb, @charlesccychen, @tvalentyn. You >>>>>>>>>>>> can include these people (and myself) for reviews and other >>>>>>>>>>>> questions that >>>>>>>>>>>> you have. >>>>>>>>>>>> >>>>>>>>>>>> Welcome again, and looking forward to your contributions. >>>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> Ahmet >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Mar 23, 2018 at 9:27 AM, Robbe Sneyders < >>>>>>>>>>>> robbe.sneyd...@ml6.eu> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello everyone, >>>>>>>>>>>>> >>>>>>>>>>>>> In the next month(s), me and my colleague Matthias will commit >>>>>>>>>>>>> a lot of time and effort to python 3 support for beam and we >>>>>>>>>>>>> would like to >>>>>>>>>>>>> discuss the best way to go forward with this. >>>>>>>>>>>>> >>>>>>>>>>>>> We have drawn up a document [1] with a high level outline of >>>>>>>>>>>>> the proposed approach and would like to get your feedback on this. >>>>>>>>>>>>> >>>>>>>>>>>>> The main Jira issue [2] for python 3 support has been mostly >>>>>>>>>>>>> inactive for the past year. Other smaller issues have been >>>>>>>>>>>>> opened, but it's >>>>>>>>>>>>> hard to track the general progress. It would be great if anyone >>>>>>>>>>>>> could offer >>>>>>>>>>>>> some insights on how to best handle this project on Jira. >>>>>>>>>>>>> >>>>>>>>>>>>> @Holden Karau, you seem to have already put in a lot of effort >>>>>>>>>>>>> to add python 3 support, so it would be great to get your >>>>>>>>>>>>> insights and find >>>>>>>>>>>>> a way to merge our efforts. >>>>>>>>>>>>> >>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>> Robbe >>>>>>>>>>>>> >>>>>>>>>>>>> [1] >>>>>>>>>>>>> https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE/edit?usp=sharing >>>>>>>>>>>>> >>>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/BEAM-1251 >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> [image: https://ml6.eu] <https://ml6.eu/> >>>>>>>>>>>>> >>>>>>>>>>>>> * Robbe Sneyders* >>>>>>>>>>>>> >>>>>>>>>>>>> ML6 Gent >>>>>>>>>>>>> <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl> >>>>>>>>>>>>> >>>>>>>>>>>>> M: +32 474 71 31 08 <+32%20474%2071%2031%2008> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> [image: https://ml6.eu] <https://ml6.eu/> >>>>>>>>>>> >>>>>>>>>>> * Robbe Sneyders* >>>>>>>>>>> >>>>>>>>>>> ML6 Gent >>>>>>>>>>> <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl> >>>>>>>>>>> >>>>>>>>>>> M: +32 474 71 31 08 <+32%20474%2071%2031%2008> >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>> >>>>>>> [image: https://ml6.eu] <https://ml6.eu/> >>>>>>> >>>>>>> * Robbe Sneyders* >>>>>>> >>>>>>> ML6 Gent >>>>>>> <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl> >>>>>>> >>>>>>> M: +32 474 71 31 08 <+32%20474%2071%2031%2008> >>>>>>> >>>>>> -- >>>>> >>>>> [image: https://ml6.eu] <https://ml6.eu/> >>>>> >>>>> * Robbe Sneyders* >>>>> >>>>> ML6 Gent >>>>> <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl> >>>>> >>>>> M: +32 474 71 31 08 <+32%20474%2071%2031%2008> >>>>> >>>> -- >>>> >>>> [image: https://ml6.eu] <https://ml6.eu/> >>>> >>>> * Robbe Sneyders* >>>> >>>> ML6 Gent >>>> <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl> >>>> >>>> M: +32 474 71 31 08 <+32%20474%2071%2031%2008> >>>> >>> >>> > -- [image: https://ml6.eu] <https://ml6.eu/> * Robbe Sneyders* ML6 Gent <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl> M: +32 474 71 31 08