This is an automated email from the ASF dual-hosted git repository. achennaka pushed a commit to branch branch-1.17.x in repository https://gitbox.apache.org/repos/asf/kudu.git
commit b109bca3c13f09a06034f68c05096cb73d67e3bc Author: Marton Greber <[email protected]> AuthorDate: Mon Aug 26 17:57:02 2024 +0000 KUDU-3604: Pin Cython Version, Package Maintenance Currently, users need to manually install Cython before installing the Kudu Python package. By running '$ pip install Cython' on most relatively new systems, they obtain Cython 3.x. However, our setup.py and build process are not compatible with Cython 3.x. In this patch, I've added a prerequisite step to setup.py that identifies and installs the correct Cython version before all the package build/install steps. Additionally, the Kudu Python package pulled from PyPi includes all the test files. This was resolved with a simple change in MANIFEST.in to exclude the test source folder. Moreover, end-users were also getting the test dependencies from setup.py because we had a single requirements.txt file for both distribution and development. I have now split the dependencies into two files: requirements.txt and requirements_dev.txt. The dependencies in requirements.txt are automatically pulled during the execution of setup.py. For development tasks, developers need to manually install requirements_dev.txt before running setup.py. The build-and-test.sh script was also updated to use requirements_dev.txt. Furthermore, there were several details and edge cases covered in the CI script related to older OS versions (e.g., el6) and unsupported Python versions (e.g., Python 2.6), as well as unpinned package versions. Since all packages are now version-pinned and these limitations are no longer relevant, I have cleaned up the package installation part in build-and-test.sh for the Python packages. As a follow-up patch, I plan to add more developer-facing documentation on how to set up the development environment, run unit tests, debug, etc. Change-Id: I45ad45c56fc3f3ecf57a413edbd0866fccbd8f79 Reviewed-on: http://gerrit.cloudera.org:8080/21816 Reviewed-by: Abhishek Chennaka <[email protected]> Tested-by: Abhishek Chennaka <[email protected]> (cherry picked from commit 8d9b3574a12bb98b79e29375c76f98a1a48af5ad) Reviewed-on: http://gerrit.cloudera.org:8080/21872 Reviewed-by: Alexey Serbin <[email protected]> --- build-support/jenkins/build-and-test.sh | 105 +------------------------------- python/MANIFEST.in | 3 + python/README.md | 2 - python/requirements.txt | 18 +----- python/requirements_dev.txt | 30 +++++++++ python/setup.py | 39 ++++++------ python/setup_prerequisites.py | 40 ++++++++++++ 7 files changed, 94 insertions(+), 143 deletions(-) diff --git a/build-support/jenkins/build-and-test.sh b/build-support/jenkins/build-and-test.sh index 93ed40a98..2d67b14bf 100755 --- a/build-support/jenkins/build-and-test.sh +++ b/build-support/jenkins/build-and-test.sh @@ -583,59 +583,7 @@ if [ "$BUILD_PYTHON" == "1" ]; then virtualenv $KUDU_BUILD/py_env source $KUDU_BUILD/py_env/bin/activate - # Old versions of pip (such as the one found in el6) default to pypi's http:// - # endpoint which no longer exists. The -i option lets us switch to the - # https:// endpoint in such cases. - # - # Unfortunately, in these old versions of pip, -i doesn't appear to apply - # recursively to transitive dependencies installed via a direct dependency's - # "python setup.py" command. Therefore we have no choice but to upgrade to a - # new version of pip to proceed. - # - # Beginning with pip 10, Python 2.6 is no longer supported. Attempting to - # upgrade to pip 10 on Python 2.6 yields syntax errors. We don't need any new - # pip features, so let's pin to the last pip version to support Python 2.6. - # - # The absence of $PIP_FLAGS is intentional: older versions of pip may not - # support the flags that we want to use. - pip install -i https://pypi.python.org/simple $PIP_INSTALL_FLAGS --upgrade 'pip <10.0.0b1' - - # New versions of pip raise an exception when upgrading old versions of - # setuptools (such as the one found in el6). The workaround is to upgrade - # setuptools on its own, outside of requirements.txt, and with the pip version - # check disabled. - # - # Setuptools 42.0.0 changes something that causes build_ext to fail with a - # missing wheel package. Let's pin to an older version known to work. - pip $PIP_FLAGS install --disable-pip-version-check $PIP_INSTALL_FLAGS --upgrade 'setuptools >=0.8,<42.0.0' - - # One of our dependencies is pandas, installed below. It depends on numpy, and - # if we don't install numpy directly, the pandas installation will install the - # latest numpy which is incompatible with Python 2.6. - # - # To work around this, we need to install a 2.6-compatible version of numpy - # before installing pandas. Installing numpy may involve some compiler work, - # so we must pass in the current values of CC and CXX. - # - # See https://github.com/numpy/numpy/releases/tag/v1.12.0 for more details. - CC=$CLANG CXX=$CLANG++ pip $PIP_FLAGS install $PIP_INSTALL_FLAGS 'numpy <1.12.0' - - # We've got a new pip and new setuptools. We can now install the rest of the - # Python client's requirements. - # - # Installing the Cython dependency may involve some compiler work, so we must - # pass in the current values of CC and CXX. - CC=$CLANG CXX=$CLANG++ pip $PIP_FLAGS install $PIP_INSTALL_FLAGS -r requirements.txt - - # We need to install Pandas manually because although it's not a required - # package, it is needed to run all of the tests. - # - # Installing pandas may involve some compiler work, so we must pass in the - # current values of CC and CXX. - # - # pandas 0.18 dropped support for python 2.6. See https://pandas.pydata.org/pandas-docs/version/0.23.0/whatsnew.html#v0-18-0-march-13-2016 - # for more details. - CC=$CLANG CXX=$CLANG++ pip $PIP_FLAGS install $PIP_INSTALL_FLAGS 'pandas <0.18' + pip $PIP_FLAGS install $PIP_INSTALL_FLAGS -r requirements_dev.txt # Delete old Cython extensions to force them to be rebuilt. rm -Rf build kudu_python.egg-info kudu/*.so @@ -683,57 +631,8 @@ if [ "$BUILD_PYTHON3" == "1" ]; then virtualenv -p python3 $KUDU_BUILD/py_env source $KUDU_BUILD/py_env/bin/activate - # Old versions of pip (such as the one found in el6) default to pypi's http:// - # endpoint which no longer exists. The -i option lets us switch to the - # https:// endpoint in such cases. - # - # Unfortunately, in these old versions of pip, -i doesn't appear to apply - # recursively to transitive dependencies installed via a direct dependency's - # "python setup.py" command. Therefore we have no choice but to upgrade to a - # new version of pip to proceed. - # - # pip 19.1 doesn't support Python 3.4, which is the version of Python 3 - # shipped with Ubuntu 14.04. However, there appears to be a bug[1] in pip 19.0 - # preventing it from working properly with Python 3.4 as well. Therefore we - # must pin to a pip version from before 19.0. - # - # The absence of $PIP_FLAGS is intentional: older versions of pip may not - # support the flags that we want to use. - # - # 1. https://github.com/pypa/pip/issues/6175 - pip install -i https://pypi.python.org/simple $PIP_INSTALL_FLAGS --upgrade 'pip <19.0' - - # New versions of pip raise an exception when upgrading old versions of - # setuptools (such as the one found in el6). The workaround is to upgrade - # setuptools on its own, outside of requirements.txt, and with the pip version - # check disabled. - pip $PIP_FLAGS install --disable-pip-version-check $PIP_INSTALL_FLAGS --upgrade 'setuptools >=0.8' - - # One of our dependencies is pandas, installed below. It depends on numpy, and - # if we don't install numpy directly, the pandas installation will install the - # latest numpy which is incompatible with Python 3.4 (the version of Python 3 - # shipped with Ubuntu 14.04). - # - # To work around this, we need to install a 3.4-compatible version of numpy - # before installing pandas. Installing numpy may involve some compiler work, - # so we must pass in the current values of CC and CXX. - # - # See https://github.com/numpy/numpy/releases/tag/v1.16.0rc1 for more details. - CC=$CLANG CXX=$CLANG++ pip $PIP_FLAGS install $PIP_INSTALL_FLAGS 'numpy <1.16.0' - - # We've got a new pip and new setuptools. We can now install the rest of the - # Python client's requirements. - # - # Installing the Cython dependency may involve some compiler work, so we must - # pass in the current values of CC and CXX. - CC=$CLANG CXX=$CLANG++ pip $PIP_FLAGS install $PIP_INSTALL_FLAGS -r requirements.txt - # We need to install Pandas manually because although it's not a required - # package, it is needed to run all of the tests. - # - # Installing pandas may involve some compiler work, so we must pass in the - # current values of CC and CXX. - CC=$CLANG CXX=$CLANG++ pip $PIP_FLAGS install $PIP_INSTALL_FLAGS pandas + pip $PIP_FLAGS install $PIP_INSTALL_FLAGS -r requirements_dev.txt # Delete old Cython extensions to force them to be rebuilt. rm -Rf build kudu_python.egg-info kudu/*.so diff --git a/python/MANIFEST.in b/python/MANIFEST.in index fcfcff70a..70db1cabc 100644 --- a/python/MANIFEST.in +++ b/python/MANIFEST.in @@ -2,9 +2,12 @@ include LICENSE.txt include MANIFEST.in include README.md include setup.py +include setup_prerequisites.py include version.txt +include requirements.txt graft kudu +prune kudu/tests global-exclude *.so global-exclude *.pyc diff --git a/python/README.md b/python/README.md index ce00c6193..15e871d9e 100644 --- a/python/README.md +++ b/python/README.md @@ -9,5 +9,3 @@ To install from PyPI, run ``` pip install kudu-python ``` - -Installation from source requires Cython. diff --git a/python/requirements.txt b/python/requirements.txt index 6bc741b07..588b61c46 100644 --- a/python/requirements.txt +++ b/python/requirements.txt @@ -1,16 +1,2 @@ -pytest ==4.6.11; python_version == "2.7" -pytest ==6.2.5; python_version >= "3.6" - -pytest-timeout ==1.4.2; python_version == "2.7" -pytest-timeout ==2.1.0; python_version >= "3.6" - -pytest-runner <5.3.0; python_version == "2.7" -pytest-runner ==5.3.2; python_version >= "3.6" - -unittest2 ==1.1.0 - -cython ==0.29.14; python_version == "2.7" -cython ==0.29.37; python_version >= "3.6" - -six ==1.16.0 -pytz ==2024.1 +six==1.16.0 +pytz==2024.1 diff --git a/python/requirements_dev.txt b/python/requirements_dev.txt new file mode 100644 index 000000000..61d90f918 --- /dev/null +++ b/python/requirements_dev.txt @@ -0,0 +1,30 @@ +# For Python 2.7: +# - Use the highest possible version of each package that still supports Python 2.7. +# +# For Python 3.6 and above: +# - Use the latest version of each package that still supports Python 3.6, +# except for numpy and pandas, which have version restrictions to ensure compatibility with +# macOS during testing. + + +pytest==4.6.11; python_version == "2.7" +pytest==6.2.5; python_version >= "3.6" + +pytest-timeout==1.4.2; python_version == "2.7" +pytest-timeout==2.1.0; python_version >= "3.6" + +pytest-runner<5.3.0; python_version == "2.7" +pytest-runner==5.3.2; python_version >= "3.6" + +unittest2==1.1.0 + +setuptools==44.1.1; python_version == "2.7" +setuptools==58.5.3; python_version >= "3.6" + +pandas==0.23.4; python_version == "2.7" +pandas==1.1.5; python_version >= "3.6" and python_version < "3.8" +pandas==1.5.3; python_version >= "3.8" + +numpy==1.16.6; python_version == "2.7" +numpy==1.19.5; python_version >= "3.6" and python_version < "3.8" +numpy==1.23.5; python_version >= "3.8" diff --git a/python/setup.py b/python/setup.py index d46038061..f2b7d21f2 100644 --- a/python/setup.py +++ b/python/setup.py @@ -18,6 +18,9 @@ # specific language governing permissions and limitations # under the License. +# setup-prerequisites must be the very first import in this file! +import setup_prerequisites + from Cython.Distutils import build_ext from Cython.Build import cythonize import Cython @@ -30,8 +33,6 @@ import os import re import subprocess -if Cython.__version__ < '0.21.0': - raise Exception('Please upgrade to Cython 0.21.0 or newer') setup_dir = os.path.abspath(os.path.dirname(__file__)) @@ -90,8 +91,8 @@ def generate_config_pxi(include_dirs): dst = os.path.join(setup_dir, 'kudu/config.pxi') dst_tmp = dst + '.tmp' cc = os.getenv("CC","cc") - subprocess.check_call([cc, "-x", "c++", "-o", dst_tmp, - "-E", '-imacros', int128_h, src]) + compilation_cmd = [cc, "-x", "c++", "-o", dst_tmp, "-E", '-imacros', int128_h, src] + subprocess.check_call(compilation_cmd) # If our generated file is the same as the prior version, # don't replace it. This avoids rebuilding everything on every # run of setup.py. @@ -102,6 +103,14 @@ def generate_config_pxi(include_dirs): VERSION = find_version() +def get_requirements(): + """ + Gets the prerequisite requirements for the package. + """ + with open("requirements.txt") as f: + requirements = f.read().splitlines() + return requirements + # If we're in the context of the Kudu git repository, build against the # latest in-tree build artifacts if 'KUDU_HOME' in os.environ: @@ -179,6 +188,7 @@ CLASSIFIERS = [ 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: 3.9', 'Programming Language :: Python :: 3.10', + 'Programming Language :: Python :: 3.11', 'Programming Language :: Cython' ] @@ -186,30 +196,15 @@ URL = 'http://kudu.apache.org/' setup( name="kudu-python", - packages=['kudu', 'kudu.tests'], + packages=['kudu'], version=VERSION, - package_data={'kudu': ['*.pxd', '*.pyx']}, + package_data={'kudu': ['*.pxd', '*.pyx', 'requirements.txt']}, ext_modules=extensions, cmdclass={ 'clean': clean, 'build_ext': build_ext }, - setup_requires=['pytest-runner <5.3.0; python_version == "2.7"', - 'pytest-runner ==5.3.2; python_version >= "3.6"'], - - # Note: dependencies in tests_require should also be listed in - # requirements.txt so that dependencies aren't downloaded at test-time - # (when it's more difficult to override various pip installation options). - tests_require=['pytest ==4.6.11; python_version == "2.7"', - 'pytest ==6.2.5; python_version >= "3.6"', - 'pytest-timeout ==1.4.2; python_version == "2.7"', - 'pytest-timeout ==2.1.0; python_version >= "3.6"', - 'unittest2 ==1.1.0'], - - install_requires=['cython ==0.29.14; python_version == "2.7"', - 'cython ==0.29.37; python_version >= "3.6"', - 'six ==1.16.0', - 'pytz ==2024.1'], + install_requires=get_requirements(), description=DESCRIPTION, long_description=LONG_DESCRIPTION, license='Apache License, Version 2.0', diff --git a/python/setup_prerequisites.py b/python/setup_prerequisites.py new file mode 100644 index 000000000..b3e34f079 --- /dev/null +++ b/python/setup_prerequisites.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python + +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Cython is a pre-requirement to be able to run this setup.py script. +# By putting this part before the Cython imports there is no need to +# install Cython manually before the installation. + +import sys +import subprocess +import pkg_resources + +cython_version = "Cython==0.29.37" + +# Function to install a package via pip +def install(package): + subprocess.check_call([sys.executable, "-m", "pip", "install", package]) + +# Ensure the correct version of Cython is installed before proceeding +try: + pkg_resources.require(cython_version) +except (pkg_resources.DistributionNotFound, pkg_resources.VersionConflict): + sys.stdout.write("Installing Cython...") + install(cython_version)
