[ https://issues.apache.org/jira/browse/BEAM-1677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Beam JIRA Bot updated BEAM-1677: -------------------------------- Labels: stale-P2 (was: ) > Extend automated PR check with PyCharm's code inspector for Python-SDK > ---------------------------------------------------------------------- > > Key: BEAM-1677 > URL: https://issues.apache.org/jira/browse/BEAM-1677 > Project: Beam > Issue Type: Improvement > Components: build-system > Reporter: Tibor Kiss > Priority: P2 > Labels: stale-P2 > Attachments: > inspection-results-1d8be97439051ae770f99dcb49c7e8486962b118.tar.gz > > > *Proposal:* > Leverage PyCharm's static code analysis capabilities to detect possible > errors if a Python-SDK related PR is submitted. > *Problem statement:* > The Python SDK test goal has linting phase, which utilizes {{pep8}} and > {{pylint}} to validate PEP8 compliance and do static code analysis. These > steps are helpful but {{pylint}}'s capabilities are somewhat limited compared > to modern IDE's (like PyCharm). > In particular: Unresolved attributes, unbound local variables and invalid > calls (due to argument number mismatch) type of errors are not detected by > pylint, but they do cause runtime error. See BEAM-1649 for problems not > detected by pylint but reported by PyCharm. > *Technical details:* > PyCharm provides command line interface [1] to trigger code inspection. The > results are created as entries in XML documents, containing their location, > severity, category and textual description of the problem. > Since the inspector reports back typos, warnings and errors it is required > that the result XML files are properly parsed and filtered. > A good signal-to-noise ratio is crucial for this improvement to be useful and > accepted, therefore it is required to make good selection of events to > minimize false positives. > > The filtered results shall be posted back to GitHub through Commit Status API > [2] to notify the contributor and reviewer of the quality of the change. > > The static code analysis could be done on either Apache Build Jenkins servers > or in Travis CI. > > *Alternatives considered:* > - Eclipse's PyDev plugin [3] has similar code inspection capabilities as > PyCharm. The lack of command line interface makes it difficult to integrate > it with the current build system. > - SonarQUBE's Python plugin [4] was briefly investigated. The number of > rules seems very limited compared to PyCharm's code analysis. > > *Required changes:* > - PyCharm to be installed in Jenkins Slave or Travis-CI's container > - PyCharm project & inspection profile created for Python SDK > - Result parser & poster application/script created > - Trigger to run the result parser & poster set up > > *Example Code Analysis* run on OS X (given a PyCharm project is available in > beam/sdks/python/.idea directory): > {code} > beam/sdks/python $ /Applications/PyCharm.app/Contents/MacOS/pycharm inspect > $(pwd) $(pwd)/.idea/inspectionProfiles/Project_Default.xml > $(pwd)/inspection-results -v2 -d $(pwd)/apache_beam > {code} > > *Example result:* > {code:title=PyUnresolvedReferencesInspection.xml} > <problem> > <file>file://$PROJECT_DIR$/apache_beam/io/range_trackers.py</file> > <line>342</line> > <module>python</module> > <entry_point TYPE="file" > FQNAME="file://$PROJECT_DIR$/apache_beam/io/range_trackers.py" /> > <problem_class severity="WARNING" > attribute_key="WARNING_ATTRIBUTES">Unresolved references</problem_class> > <description>Unresolved attribute reference '_end_position' for class > 'OrderedPositionRangeTracker'</description> > </problem> > {code} > > *References:* > [1] PyCharm Command Line Interface: > https://www.jetbrains.com/help/pycharm/2016.3/command-line-code-inspector.html > [2] GitHub Commit Status API: https://developer.github.com/v3/repos/statuses/ > [3] Eclipse PyDev Plugin: http://www.pydev.org/manual_adv_code_analysis.html > [4] SonarQUBE Python plugin: > https://www.sonarsource.com/why-us/products/codeanalyzers/sonarpython/rules.html > > CC: [~altay] / [~jbonofre] / [~jasonkuster] -- This message was sent by Atlassian Jira (v8.3.4#803005)