Fully agree Holden, would be great to include the Outreachy project. Adding annotations is a very friendly way to get familiar with the codebase.
I've also created a PR to see what's needed to get mypy in: https://github.com/apache/spark/pull/29180 From there on we can start adding annotations. Cheers, Fokko Op di 21 jul. 2020 om 21:40 schreef Holden Karau <hol...@pigscanfly.ca>: > Yeah I think this could be a great project now that we're only Python > 3.5+. One potential is making this an Outreachy project to get more folks > from different backgrounds involved in Spark. > > On Tue, Jul 21, 2020 at 12:33 PM Driesprong, Fokko <fo...@driesprong.frl> > wrote: > >> Since we've recently dropped support for Python <=3.5 >> <https://github.com/apache/spark/pull/28957>, I think it would be nice >> to add support for type annotations. Having this in the main repository >> allows us to do type checking using MyPy <http://mypy-lang.org/> in the >> CI itself. <http://mypy-lang.org/> >> >> This is now handled by the Stub file: >> https://www.python.org/dev/peps/pep-0484/#stub-files However I think it >> is nicer to integrate the types with the code itself to keep everything in >> sync, and make it easier for the people who work on the codebase itself. A >> first step would be to move the stubs into the codebase. First step would >> be to cover the public API which is the most important one. Having the >> types with the code itself makes it much easier to understand. For example, >> if you can supply a str or column here: >> https://github.com/apache/spark/pull/29122/files#diff-f5295f69bfbdbf6e161aed54057ea36dR2486 >> >> One of the implications would be that future PR's on Python should cover >> annotations on the public API's. Curious what the rest of the community >> thinks. >> >> Cheers, Fokko >> >> >> >> >> >> >> >> >> >> Op di 21 jul. 2020 om 20:04 schreef zero323 <mszymkiew...@gmail.com>: >> >>> Given a discussion related to SPARK-32320 PR >>> <https://github.com/apache/spark/pull/29122> I'd like to resurrect >>> this >>> thread. Is there any interest in migrating annotations to the main >>> repository? >>> >>> >>> >>> -- >>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >>> > > -- > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >
