This is an automated email from the ASF dual-hosted git repository.
achennaka pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 8d9b3574a KUDU-3604: Pin Cython Version, Package Maintenance
8d9b3574a is described below
commit 8d9b3574a12bb98b79e29375c76f98a1a48af5ad
Author: Marton Greber <[email protected]>
AuthorDate: Mon Aug 26 17:57:02 2024 +0000
KUDU-3604: Pin Cython Version, Package Maintenance
Currently, users need to manually install Cython before installing the
Kudu Python package. By running '$ pip install Cython' on most
relatively new systems, they obtain Cython 3.x. However, our setup.py
and build process are not compatible with Cython 3.x. In this patch,
I've added a prerequisite step to setup.py that identifies and
installs the correct Cython version before all the package build/install
steps.
Additionally, the Kudu Python package pulled from PyPi includes all the
test files. This was resolved with a simple change in MANIFEST.in to
exclude the test source folder. Moreover, end-users were also getting
the test dependencies from setup.py because we had a single
requirements.txt file for both distribution and development. I have
now split the dependencies into two files: requirements.txt and
requirements_dev.txt. The dependencies in requirements.txt are
automatically pulled during the execution of setup.py. For development
tasks, developers need to manually install requirements_dev.txt before
running setup.py. The build-and-test.sh script was also updated to
use requirements_dev.txt.
Furthermore, there were several details and edge cases covered in the CI
script related to older OS versions (e.g., el6) and unsupported Python
versions (e.g., Python 2.6), as well as unpinned package versions. Since
all packages are now version-pinned and these limitations are no longer
relevant, I have cleaned up the package installation part in
build-and-test.sh for the Python packages.
As a follow-up patch, I plan to add more developer-facing documentation
on how to set up the development environment, run unit tests, debug,
etc.
Change-Id: I45ad45c56fc3f3ecf57a413edbd0866fccbd8f79
Reviewed-on: http://gerrit.cloudera.org:8080/21816
Reviewed-by: Abhishek Chennaka <[email protected]>
Tested-by: Abhishek Chennaka <[email protected]>
---
build-support/jenkins/build-and-test.sh | 105 +-------------------------------
python/MANIFEST.in | 3 +
python/README.md | 2 -
python/requirements.txt | 18 +-----
python/requirements_dev.txt | 30 +++++++++
python/setup.py | 39 ++++++------
python/setup_prerequisites.py | 40 ++++++++++++
7 files changed, 94 insertions(+), 143 deletions(-)
diff --git a/build-support/jenkins/build-and-test.sh
b/build-support/jenkins/build-and-test.sh
index 6801f9e42..d1f175349 100755
--- a/build-support/jenkins/build-and-test.sh
+++ b/build-support/jenkins/build-and-test.sh
@@ -583,59 +583,7 @@ if [ "$BUILD_PYTHON" == "1" ]; then
virtualenv $KUDU_BUILD/py_env
source $KUDU_BUILD/py_env/bin/activate
- # Old versions of pip (such as the one found in el6) default to pypi's
http://
- # endpoint which no longer exists. The -i option lets us switch to the
- # https:// endpoint in such cases.
- #
- # Unfortunately, in these old versions of pip, -i doesn't appear to apply
- # recursively to transitive dependencies installed via a direct dependency's
- # "python setup.py" command. Therefore we have no choice but to upgrade to a
- # new version of pip to proceed.
- #
- # Beginning with pip 10, Python 2.6 is no longer supported. Attempting to
- # upgrade to pip 10 on Python 2.6 yields syntax errors. We don't need any new
- # pip features, so let's pin to the last pip version to support Python 2.6.
- #
- # The absence of $PIP_FLAGS is intentional: older versions of pip may not
- # support the flags that we want to use.
- pip install -i https://pypi.python.org/simple $PIP_INSTALL_FLAGS --upgrade
'pip <10.0.0b1'
-
- # New versions of pip raise an exception when upgrading old versions of
- # setuptools (such as the one found in el6). The workaround is to upgrade
- # setuptools on its own, outside of requirements.txt, and with the pip
version
- # check disabled.
- #
- # Setuptools 42.0.0 changes something that causes build_ext to fail with a
- # missing wheel package. Let's pin to an older version known to work.
- pip $PIP_FLAGS install --disable-pip-version-check $PIP_INSTALL_FLAGS
--upgrade 'setuptools >=0.8,<42.0.0'
-
- # One of our dependencies is pandas, installed below. It depends on numpy,
and
- # if we don't install numpy directly, the pandas installation will install
the
- # latest numpy which is incompatible with Python 2.6.
- #
- # To work around this, we need to install a 2.6-compatible version of numpy
- # before installing pandas. Installing numpy may involve some compiler work,
- # so we must pass in the current values of CC and CXX.
- #
- # See https://github.com/numpy/numpy/releases/tag/v1.12.0 for more details.
- CC=$CLANG CXX=$CLANG++ pip $PIP_FLAGS install $PIP_INSTALL_FLAGS 'numpy
<1.12.0'
-
- # We've got a new pip and new setuptools. We can now install the rest of the
- # Python client's requirements.
- #
- # Installing the Cython dependency may involve some compiler work, so we must
- # pass in the current values of CC and CXX.
- CC=$CLANG CXX=$CLANG++ pip $PIP_FLAGS install $PIP_INSTALL_FLAGS -r
requirements.txt
-
- # We need to install Pandas manually because although it's not a required
- # package, it is needed to run all of the tests.
- #
- # Installing pandas may involve some compiler work, so we must pass in the
- # current values of CC and CXX.
- #
- # pandas 0.18 dropped support for python 2.6. See
https://pandas.pydata.org/pandas-docs/version/0.23.0/whatsnew.html#v0-18-0-march-13-2016
- # for more details.
- CC=$CLANG CXX=$CLANG++ pip $PIP_FLAGS install $PIP_INSTALL_FLAGS 'pandas
<0.18'
+ pip $PIP_FLAGS install $PIP_INSTALL_FLAGS -r requirements_dev.txt
# Delete old Cython extensions to force them to be rebuilt.
rm -Rf build kudu_python.egg-info kudu/*.so
@@ -683,57 +631,8 @@ if [ "$BUILD_PYTHON3" == "1" ]; then
virtualenv -p python3 $KUDU_BUILD/py_env
source $KUDU_BUILD/py_env/bin/activate
- # Old versions of pip (such as the one found in el6) default to pypi's
http://
- # endpoint which no longer exists. The -i option lets us switch to the
- # https:// endpoint in such cases.
- #
- # Unfortunately, in these old versions of pip, -i doesn't appear to apply
- # recursively to transitive dependencies installed via a direct dependency's
- # "python setup.py" command. Therefore we have no choice but to upgrade to a
- # new version of pip to proceed.
- #
- # pip 19.1 doesn't support Python 3.4, which is the version of Python 3
- # shipped with Ubuntu 14.04. However, there appears to be a bug[1] in pip
19.0
- # preventing it from working properly with Python 3.4 as well. Therefore we
- # must pin to a pip version from before 19.0.
- #
- # The absence of $PIP_FLAGS is intentional: older versions of pip may not
- # support the flags that we want to use.
- #
- # 1. https://github.com/pypa/pip/issues/6175
- pip install -i https://pypi.python.org/simple $PIP_INSTALL_FLAGS --upgrade
'pip <19.0'
-
- # New versions of pip raise an exception when upgrading old versions of
- # setuptools (such as the one found in el6). The workaround is to upgrade
- # setuptools on its own, outside of requirements.txt, and with the pip
version
- # check disabled.
- pip $PIP_FLAGS install --disable-pip-version-check $PIP_INSTALL_FLAGS
--upgrade 'setuptools >=0.8'
-
- # One of our dependencies is pandas, installed below. It depends on numpy,
and
- # if we don't install numpy directly, the pandas installation will install
the
- # latest numpy which is incompatible with Python 3.4 (the version of Python 3
- # shipped with Ubuntu 14.04).
- #
- # To work around this, we need to install a 3.4-compatible version of numpy
- # before installing pandas. Installing numpy may involve some compiler work,
- # so we must pass in the current values of CC and CXX.
- #
- # See https://github.com/numpy/numpy/releases/tag/v1.16.0rc1 for more
details.
- CC=$CLANG CXX=$CLANG++ pip $PIP_FLAGS install $PIP_INSTALL_FLAGS 'numpy
<1.16.0'
-
- # We've got a new pip and new setuptools. We can now install the rest of the
- # Python client's requirements.
- #
- # Installing the Cython dependency may involve some compiler work, so we must
- # pass in the current values of CC and CXX.
- CC=$CLANG CXX=$CLANG++ pip $PIP_FLAGS install $PIP_INSTALL_FLAGS -r
requirements.txt
- # We need to install Pandas manually because although it's not a required
- # package, it is needed to run all of the tests.
- #
- # Installing pandas may involve some compiler work, so we must pass in the
- # current values of CC and CXX.
- CC=$CLANG CXX=$CLANG++ pip $PIP_FLAGS install $PIP_INSTALL_FLAGS pandas
+ pip $PIP_FLAGS install $PIP_INSTALL_FLAGS -r requirements_dev.txt
# Delete old Cython extensions to force them to be rebuilt.
rm -Rf build kudu_python.egg-info kudu/*.so
diff --git a/python/MANIFEST.in b/python/MANIFEST.in
index fcfcff70a..70db1cabc 100644
--- a/python/MANIFEST.in
+++ b/python/MANIFEST.in
@@ -2,9 +2,12 @@ include LICENSE.txt
include MANIFEST.in
include README.md
include setup.py
+include setup_prerequisites.py
include version.txt
+include requirements.txt
graft kudu
+prune kudu/tests
global-exclude *.so
global-exclude *.pyc
diff --git a/python/README.md b/python/README.md
index ce00c6193..15e871d9e 100644
--- a/python/README.md
+++ b/python/README.md
@@ -9,5 +9,3 @@ To install from PyPI, run
```
pip install kudu-python
```
-
-Installation from source requires Cython.
diff --git a/python/requirements.txt b/python/requirements.txt
index 6bc741b07..588b61c46 100644
--- a/python/requirements.txt
+++ b/python/requirements.txt
@@ -1,16 +1,2 @@
-pytest ==4.6.11; python_version == "2.7"
-pytest ==6.2.5; python_version >= "3.6"
-
-pytest-timeout ==1.4.2; python_version == "2.7"
-pytest-timeout ==2.1.0; python_version >= "3.6"
-
-pytest-runner <5.3.0; python_version == "2.7"
-pytest-runner ==5.3.2; python_version >= "3.6"
-
-unittest2 ==1.1.0
-
-cython ==0.29.14; python_version == "2.7"
-cython ==0.29.37; python_version >= "3.6"
-
-six ==1.16.0
-pytz ==2024.1
+six==1.16.0
+pytz==2024.1
diff --git a/python/requirements_dev.txt b/python/requirements_dev.txt
new file mode 100644
index 000000000..61d90f918
--- /dev/null
+++ b/python/requirements_dev.txt
@@ -0,0 +1,30 @@
+# For Python 2.7:
+# - Use the highest possible version of each package that still supports
Python 2.7.
+#
+# For Python 3.6 and above:
+# - Use the latest version of each package that still supports Python 3.6,
+# except for numpy and pandas, which have version restrictions to ensure
compatibility with
+# macOS during testing.
+
+
+pytest==4.6.11; python_version == "2.7"
+pytest==6.2.5; python_version >= "3.6"
+
+pytest-timeout==1.4.2; python_version == "2.7"
+pytest-timeout==2.1.0; python_version >= "3.6"
+
+pytest-runner<5.3.0; python_version == "2.7"
+pytest-runner==5.3.2; python_version >= "3.6"
+
+unittest2==1.1.0
+
+setuptools==44.1.1; python_version == "2.7"
+setuptools==58.5.3; python_version >= "3.6"
+
+pandas==0.23.4; python_version == "2.7"
+pandas==1.1.5; python_version >= "3.6" and python_version < "3.8"
+pandas==1.5.3; python_version >= "3.8"
+
+numpy==1.16.6; python_version == "2.7"
+numpy==1.19.5; python_version >= "3.6" and python_version < "3.8"
+numpy==1.23.5; python_version >= "3.8"
diff --git a/python/setup.py b/python/setup.py
index d46038061..f2b7d21f2 100644
--- a/python/setup.py
+++ b/python/setup.py
@@ -18,6 +18,9 @@
# specific language governing permissions and limitations
# under the License.
+# setup-prerequisites must be the very first import in this file!
+import setup_prerequisites
+
from Cython.Distutils import build_ext
from Cython.Build import cythonize
import Cython
@@ -30,8 +33,6 @@ import os
import re
import subprocess
-if Cython.__version__ < '0.21.0':
- raise Exception('Please upgrade to Cython 0.21.0 or newer')
setup_dir = os.path.abspath(os.path.dirname(__file__))
@@ -90,8 +91,8 @@ def generate_config_pxi(include_dirs):
dst = os.path.join(setup_dir, 'kudu/config.pxi')
dst_tmp = dst + '.tmp'
cc = os.getenv("CC","cc")
- subprocess.check_call([cc, "-x", "c++", "-o", dst_tmp,
- "-E", '-imacros', int128_h, src])
+ compilation_cmd = [cc, "-x", "c++", "-o", dst_tmp, "-E", '-imacros',
int128_h, src]
+ subprocess.check_call(compilation_cmd)
# If our generated file is the same as the prior version,
# don't replace it. This avoids rebuilding everything on every
# run of setup.py.
@@ -102,6 +103,14 @@ def generate_config_pxi(include_dirs):
VERSION = find_version()
+def get_requirements():
+ """
+ Gets the prerequisite requirements for the package.
+ """
+ with open("requirements.txt") as f:
+ requirements = f.read().splitlines()
+ return requirements
+
# If we're in the context of the Kudu git repository, build against the
# latest in-tree build artifacts
if 'KUDU_HOME' in os.environ:
@@ -179,6 +188,7 @@ CLASSIFIERS = [
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
+ 'Programming Language :: Python :: 3.11',
'Programming Language :: Cython'
]
@@ -186,30 +196,15 @@ URL = 'http://kudu.apache.org/'
setup(
name="kudu-python",
- packages=['kudu', 'kudu.tests'],
+ packages=['kudu'],
version=VERSION,
- package_data={'kudu': ['*.pxd', '*.pyx']},
+ package_data={'kudu': ['*.pxd', '*.pyx', 'requirements.txt']},
ext_modules=extensions,
cmdclass={
'clean': clean,
'build_ext': build_ext
},
- setup_requires=['pytest-runner <5.3.0; python_version == "2.7"',
- 'pytest-runner ==5.3.2; python_version >= "3.6"'],
-
- # Note: dependencies in tests_require should also be listed in
- # requirements.txt so that dependencies aren't downloaded at test-time
- # (when it's more difficult to override various pip installation options).
- tests_require=['pytest ==4.6.11; python_version == "2.7"',
- 'pytest ==6.2.5; python_version >= "3.6"',
- 'pytest-timeout ==1.4.2; python_version == "2.7"',
- 'pytest-timeout ==2.1.0; python_version >= "3.6"',
- 'unittest2 ==1.1.0'],
-
- install_requires=['cython ==0.29.14; python_version == "2.7"',
- 'cython ==0.29.37; python_version >= "3.6"',
- 'six ==1.16.0',
- 'pytz ==2024.1'],
+ install_requires=get_requirements(),
description=DESCRIPTION,
long_description=LONG_DESCRIPTION,
license='Apache License, Version 2.0',
diff --git a/python/setup_prerequisites.py b/python/setup_prerequisites.py
new file mode 100644
index 000000000..b3e34f079
--- /dev/null
+++ b/python/setup_prerequisites.py
@@ -0,0 +1,40 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Cython is a pre-requirement to be able to run this setup.py script.
+# By putting this part before the Cython imports there is no need to
+# install Cython manually before the installation.
+
+import sys
+import subprocess
+import pkg_resources
+
+cython_version = "Cython==0.29.37"
+
+# Function to install a package via pip
+def install(package):
+ subprocess.check_call([sys.executable, "-m", "pip", "install", package])
+
+# Ensure the correct version of Cython is installed before proceeding
+try:
+ pkg_resources.require(cython_version)
+except (pkg_resources.DistributionNotFound, pkg_resources.VersionConflict):
+ sys.stdout.write("Installing Cython...")
+ install(cython_version)
