Re: setup.py + cython == chicken and the egg problem
On Tue, Aug 16, 2022 at 2:03 PM Dan Stromberg wrote: > Hi folks. > > I'm attempting to package up a python package that uses Cython. > > Rather than build binaries for everything under the sun, I've been > focusing on including the .pyx file and running cython on it at install > time. This requires a C compiler, but I'm OK with that. > > BTW, the pure python version works fine, and the cython version works too > as long as you preinstall cython - but I don't want users to have to know > that :) > For the actual chicken-and-egg problem, I'd needed to include my pyproject.toml in my MANIFEST.in, like: include pyx_treap.pyx pyx_treap.c pyproject.toml -- https://mail.python.org/mailman/listinfo/python-list
Re: setup.py + cython == chicken and the egg problem
On Wed, Aug 17, 2022 at 3:05 PM Dan Stromberg wrote: > I commented out those too lines, but I'm still getting errors. They seem >> to stem from: >> $ "/home/dstromberg/venv/pyx-treap-testing/bin/python3", >> ["/home/dstromberg/venv/pyx-treap-testing/bin/python3", >> "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py", >> "install", "--ignore-installed", "--no-user", "--prefix", >> "/tmp/pip-build-env-9_ivrsb6/overlay", "--no-warn-script-location", >> "--no-binary", ":none:", "--only-binary", ":none:", "-i", " >> https://test.pypi.org/simple/;, "--", "setuptools >= 44.1.1", "wheel", >> "Cython"] >> "/home/dstromberg/venv/pyx-treap-testing/bin/python3" >> "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py" >> "install" "--ignore-installed" "--no-user" "--prefix" >> "/tmp/pip-build-env-9_ivrsb6/overlay" "--no-warn-script-location" >> "--no-binary" ":none:" "--only-binary" ":none:" "-i" " >> https://test.pypi.org/simple/; "--" "setuptools >= 44.1.1" "wheel" >> "Cython" >> Looking in indexes: https://test.pypi.org/simple/ >> ERROR: Could not find a version that satisfies the requirement >> setuptools>=44.1.1 (from versions: none) >> ERROR: No matching distribution found for setuptools>=44.1.1 >> >> I copied that out of an strace. >> >> That's likely related to my pyproject.toml: >> $ cat pyproject.toml >> below cmd output started 2022 Wed Aug 17 01:57:09 PM PDT >> [build-system] >> requires = ["setuptools >= 44.1.1", "wheel", "Cython"] >> build-backend = "setuptools.build_meta" >> >> Any other suggestions folks? >> > > I don't know why, but if I delete the --ignore-installed option, I don't > get the error: > $ "/home/dstromberg/venv/pyx-treap-testing/bin/python3" > "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py" > "install" "--ignore-installed" "--prefix" > "/tmp/pip-build-env-9_ivrsb6/overlay" "-i" "https://test.pypi.org/simple/; > "--" "setuptools>=44.1.1" > below cmd output started 2022 Wed Aug 17 02:56:24 PM PDT > Looking in indexes: https://test.pypi.org/simple/ > ERROR: Could not find a version that satisfies the requirement > setuptools>=44.1.1 (from versions: none) > ERROR: No matching distribution found for setuptools>=44.1.1 > (setuptools-investigation) above cmd output done2022 Wed Aug 17 > 02:56:24 PM PDT > dstromberg@tp-mini-c:~ x86_64-pc-linux-gnu 2995 > > $ "/home/dstromberg/venv/pyx-treap-testing/bin/python3" > "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py" > "install" "--prefix" "/tmp/pip-build-env-9_ivrsb6/overlay" "-i" " > https://test.pypi.org/simple/; "--" "setuptools>=44.1.1" > below cmd output started 2022 Wed Aug 17 02:56:35 PM PDT > Looking in indexes: https://test.pypi.org/simple/ > Requirement already satisfied: setuptools>=44.1.1 in > ./venv/pyx-treap-testing/lib/python3.9/site-packages (63.4.1) > > If I search for foo on pypi and testpypi, shouldn't I get foo before > foo-bar and bar-foo? > > Because if I search for setuptools on pypi, I get setuptools as my first > hit, but on testpypi, I don't see setuptools anywhere in the first page. > That's perhaps significant if the search ordering is working as described > above. > > ? > IOW, maybe testpypi doesn't have setuptools. Indeed, it doesn't appear to. Publishing and installing from the real pypi seems to be working. Thanks folks. -- https://mail.python.org/mailman/listinfo/python-list
Re: setup.py + cython == chicken and the egg problem
On Wed, Aug 17, 2022 at 1:58 PM Dan Stromberg wrote:
> On Wed, Aug 17, 2022 at 10:20 AM Christian Gollwitzer
> wrote:
>
>> Am 16.08.22 um 23:03 schrieb Dan Stromberg:
>> > I'm attempting to package up a python package that uses Cython.
>> >
>> > Rather than build binaries for everything under the sun, I've been
>> focusing
>> > on including the .pyx file and running cython on it at install time.
>> This
>> > requires a C compiler, but I'm OK with that.
>> >
>> > However, when I try to install the package from test.pypi.org, I get:
>> > $ python3 -m pip install -i https://test.pypi.org/simple/ pyx-treap
>> > below cmd output started 2022 Tue Aug 16 01:55:16 PM PDT
>> > Looking in indexes: https://test.pypi.org/simple/
>> > Collecting pyx-treap
>> >Downloading
>> >
>> https://test-files.pythonhosted.org/packages/3a/41/af5360934adccfc086a39e1f720323895144b53454ff6dacc0f06267db55/pyx_treap-2.0.15.tar.gz
>> > (125 kB)
>> >
>> >
>>
>>
>> > 125.9/125.9 kB 1.9 MB/s eta 0:00:00
>> >Installing build dependencies ... error
>> >error: subprocess-exited-with-error
>> >
>> >×? pip subprocess to install build dependencies did not run
>> successfully.
>> >?? exit code: 1
>> >> [3 lines of output]
>> >Looking in indexes: https://test.pypi.org/simple/
>> >ERROR: Could not find a version that satisfies the requirement
>> > setuptools (from versions: none)
>> >ERROR: No matching distribution found for setuptools
>> >[end of output]
>> >
>> >note: This error originates from a subprocess, and is likely not a
>> > problem with pip.
>> > error: subprocess-exited-with-error
>>
>>
>> I looked at your code and I think you are trying too hard. As far as I
>> understand, you need Cython to be installed before the build process
>> begins. Your entry in pyproject.toml should take care of that.
>> But you also have these lines in your setup.py
>>
>> subprocess.check_call('%s -m pip install cython' % (sys.executable, ),
>> shell=True)
>> subprocess.check_call('%s -m cython pyx_treap.pyx' % (sys.executable, ),
>> shell=True)
>>
>> The first one calls out to pip while pip is already running, I'm not
>> sure that this will work, but judging from the error message it is
>> looking for the requirements also from test.pypi. Maybe this is the
>> reason that it fails (the error message says that it can't find
>> setuptools). So jut delete this line and it might already work
>>
>> The second line, which compiles the Cython code, also runs *at every
>> invocation of setup.py*, even if you'd do just
>>
>> python3 setup.py --help
>>
>> It may still work, but the correct way to do it is to create a build
>> extension for setuptools. In my project you can see this here:
>>
>> https://github.com/j-from-b/CDEF/blob/main/setup.py#L88
>>
>> OTOH, I would be surprised if Cython did not have this already, indeed
>> you imported cythonize from Cython.Build. So maybe just deleting these
>> two lines and it might work?
>>
>
> I commented out those too lines, but I'm still getting errors. They seem
> to stem from:
> $ "/home/dstromberg/venv/pyx-treap-testing/bin/python3",
> ["/home/dstromberg/venv/pyx-treap-testing/bin/python3",
> "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py",
> "install", "--ignore-installed", "--no-user", "--prefix",
> "/tmp/pip-build-env-9_ivrsb6/overlay", "--no-warn-script-location",
> "--no-binary", ":none:", "--only-binary", ":none:", "-i", "
> https://test.pypi.org/simple/;, "--", "setuptools >= 44.1.1", "wheel",
> "Cython"]
> "/home/dstromberg/venv/pyx-treap-testing/bin/python3"
> "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py"
> "install" "--ignore-installed" "--no-user" "--prefix"
> "/tmp/pip-build-env-9_ivrsb6/overlay" "--no-warn-script-location"
> "--no-binary" ":none:" "--only-binary" ":none:" "-i" "
> https://test.pypi.org/simple/; "--" "setuptools >= 44.1.1" "wheel"
> "Cython"
> Looking in indexes: https://test.pypi.org/simple/
> ERROR: Could not find a version that satisfies the requirement
> setuptools>=44.1.1 (from versions: none)
> ERROR: No matching distribution found for setuptools>=44.1.1
>
> I copied that out of an strace.
>
> That's likely related to my pyproject.toml:
> $ cat pyproject.toml
> below cmd output started 2022 Wed Aug 17 01:57:09 PM PDT
> [build-system]
> requires = ["setuptools >= 44.1.1", "wheel", "Cython"]
> build-backend = "setuptools.build_meta"
>
> Any other suggestions folks?
>
I don't know why, but if I delete the --ignore-installed option, I don't
get the error:
$ "/home/dstromberg/venv/pyx-treap-testing/bin/python3"
"/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py"
"install" "--ignore-installed" "--prefix"
"/tmp/pip-build-env-9_ivrsb6/overlay" "-i" "https://test.pypi.org/simple/;
"--"
Re: setup.py + cython == chicken and the egg problem
On Wed, Aug 17, 2022 at 10:20 AM Christian Gollwitzer
wrote:
> Am 16.08.22 um 23:03 schrieb Dan Stromberg:
> > I'm attempting to package up a python package that uses Cython.
> >
> > Rather than build binaries for everything under the sun, I've been
> focusing
> > on including the .pyx file and running cython on it at install time.
> This
> > requires a C compiler, but I'm OK with that.
> >
> > However, when I try to install the package from test.pypi.org, I get:
> > $ python3 -m pip install -i https://test.pypi.org/simple/ pyx-treap
> > below cmd output started 2022 Tue Aug 16 01:55:16 PM PDT
> > Looking in indexes: https://test.pypi.org/simple/
> > Collecting pyx-treap
> >Downloading
> >
> https://test-files.pythonhosted.org/packages/3a/41/af5360934adccfc086a39e1f720323895144b53454ff6dacc0f06267db55/pyx_treap-2.0.15.tar.gz
> > (125 kB)
> >
> >
>
>
> > 125.9/125.9 kB 1.9 MB/s eta 0:00:00
> >Installing build dependencies ... error
> >error: subprocess-exited-with-error
> >
> >×? pip subprocess to install build dependencies did not run
> successfully.
> >?? exit code: 1
> >> [3 lines of output]
> >Looking in indexes: https://test.pypi.org/simple/
> >ERROR: Could not find a version that satisfies the requirement
> > setuptools (from versions: none)
> >ERROR: No matching distribution found for setuptools
> >[end of output]
> >
> >note: This error originates from a subprocess, and is likely not a
> > problem with pip.
> > error: subprocess-exited-with-error
>
>
> I looked at your code and I think you are trying too hard. As far as I
> understand, you need Cython to be installed before the build process
> begins. Your entry in pyproject.toml should take care of that.
> But you also have these lines in your setup.py
>
> subprocess.check_call('%s -m pip install cython' % (sys.executable, ),
> shell=True)
> subprocess.check_call('%s -m cython pyx_treap.pyx' % (sys.executable, ),
> shell=True)
>
> The first one calls out to pip while pip is already running, I'm not
> sure that this will work, but judging from the error message it is
> looking for the requirements also from test.pypi. Maybe this is the
> reason that it fails (the error message says that it can't find
> setuptools). So jut delete this line and it might already work
>
> The second line, which compiles the Cython code, also runs *at every
> invocation of setup.py*, even if you'd do just
>
> python3 setup.py --help
>
> It may still work, but the correct way to do it is to create a build
> extension for setuptools. In my project you can see this here:
>
> https://github.com/j-from-b/CDEF/blob/main/setup.py#L88
>
> OTOH, I would be surprised if Cython did not have this already, indeed
> you imported cythonize from Cython.Build. So maybe just deleting these
> two lines and it might work?
>
I commented out those too lines, but I'm still getting errors. They seem
to stem from:
$ "/home/dstromberg/venv/pyx-treap-testing/bin/python3",
["/home/dstromberg/venv/pyx-treap-testing/bin/python3",
"/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py",
"install", "--ignore-installed", "--no-user", "--prefix",
"/tmp/pip-build-env-9_ivrsb6/overlay", "--no-warn-script-location",
"--no-binary", ":none:", "--only-binary", ":none:", "-i", "
https://test.pypi.org/simple/;, "--", "setuptools >= 44.1.1", "wheel",
"Cython"]
"/home/dstromberg/venv/pyx-treap-testing/bin/python3"
"/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py"
"install" "--ignore-installed" "--no-user" "--prefix"
"/tmp/pip-build-env-9_ivrsb6/overlay" "--no-warn-script-location"
"--no-binary" ":none:" "--only-binary" ":none:" "-i" "
https://test.pypi.org/simple/; "--" "setuptools >= 44.1.1" "wheel" "Cython"
Looking in indexes: https://test.pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement
setuptools>=44.1.1 (from versions: none)
ERROR: No matching distribution found for setuptools>=44.1.1
I copied that out of an strace.
That's likely related to my pyproject.toml:
$ cat pyproject.toml
below cmd output started 2022 Wed Aug 17 01:57:09 PM PDT
[build-system]
requires = ["setuptools >= 44.1.1", "wheel", "Cython"]
build-backend = "setuptools.build_meta"
Any other suggestions folks?
--
https://mail.python.org/mailman/listinfo/python-list
Re: setup.py + cython == chicken and the egg problem
Dan Stromberg wrote at 2022-8-16 14:03 -0700: > ... >I'm attempting to package up a python package that uses Cython. > ... > Installing build dependencies ... error > error: subprocess-exited-with-error > > ×? pip subprocess to install build dependencies did not run successfully. > ?? exit code: 1 > > [3 lines of output] > Looking in indexes: https://test.pypi.org/simple/ > ERROR: Could not find a version that satisfies the requirement >setuptools (from versions: none) > ERROR: No matching distribution found for setuptools The message tells you that there is a `setuptools` problem. I would start to locate all `setuptools` requirement locations. I am using `cython` for the package `dm.xmlsec.binding`. I have not observed nor heard of a problem similar to yours (but I have never tried `test.pypi.org`). -- https://mail.python.org/mailman/listinfo/python-list
Re: setup.py + cython == chicken and the egg problem
Am 16.08.22 um 23:03 schrieb Dan Stromberg:
I'm attempting to package up a python package that uses Cython.
Rather than build binaries for everything under the sun, I've been focusing
on including the .pyx file and running cython on it at install time. This
requires a C compiler, but I'm OK with that.
However, when I try to install the package from test.pypi.org, I get:
$ python3 -m pip install -i https://test.pypi.org/simple/ pyx-treap
below cmd output started 2022 Tue Aug 16 01:55:16 PM PDT
Looking in indexes: https://test.pypi.org/simple/
Collecting pyx-treap
Downloading
https://test-files.pythonhosted.org/packages/3a/41/af5360934adccfc086a39e1f720323895144b53454ff6dacc0f06267db55/pyx_treap-2.0.15.tar.gz
(125 kB)
125.9/125.9 kB 1.9 MB/s eta 0:00:00
Installing build dependencies ... error
error: subprocess-exited-with-error
×? pip subprocess to install build dependencies did not run successfully.
?? exit code: 1
> [3 lines of output]
Looking in indexes: https://test.pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement
setuptools (from versions: none)
ERROR: No matching distribution found for setuptools
[end of output]
note: This error originates from a subprocess, and is likely not a
problem with pip.
error: subprocess-exited-with-error
I looked at your code and I think you are trying too hard. As far as I
understand, you need Cython to be installed before the build process
begins. Your entry in pyproject.toml should take care of that.
But you also have these lines in your setup.py
subprocess.check_call('%s -m pip install cython' % (sys.executable, ),
shell=True)
subprocess.check_call('%s -m cython pyx_treap.pyx' % (sys.executable, ),
shell=True)
The first one calls out to pip while pip is already running, I'm not
sure that this will work, but judging from the error message it is
looking for the requirements also from test.pypi. Maybe this is the
reason that it fails (the error message says that it can't find
setuptools). So jut delete this line and it might already work
The second line, which compiles the Cython code, also runs *at every
invocation of setup.py*, even if you'd do just
python3 setup.py --help
It may still work, but the correct way to do it is to create a build
extension for setuptools. In my project you can see this here:
https://github.com/j-from-b/CDEF/blob/main/setup.py#L88
OTOH, I would be surprised if Cython did not have this already, indeed
you imported cythonize from Cython.Build. So maybe just deleting these
two lines and it might work?
Christian
--
https://mail.python.org/mailman/listinfo/python-list
Re: setup.py + cython == chicken and the egg problem
On Tue, Aug 16, 2022 at 2:08 PM Chris Angelico wrote: > On Wed, 17 Aug 2022 at 07:05, Dan Stromberg wrote: > > > > Hi folks. > > > > I'm attempting to package up a python package that uses Cython. > > > > Rather than build binaries for everything under the sun, I've been > focusing > > on including the .pyx file and running cython on it at install time. > This > > requires a C compiler, but I'm OK with that. > > > > Is keeping the cythonized file an option? That would still require a C > compiler, but wouldn't require Cython. > > ChrisA > That comes under "one of the many things I tried"; it seemed like some of the code in the .c was tailored to the specific version of python it was being installed on or something. But if someone has seen this work well, I'm more than open to trying it again. -- https://mail.python.org/mailman/listinfo/python-list
Re: setup.py + cython == chicken and the egg problem
On Wed, 17 Aug 2022 at 07:05, Dan Stromberg wrote: > > Hi folks. > > I'm attempting to package up a python package that uses Cython. > > Rather than build binaries for everything under the sun, I've been focusing > on including the .pyx file and running cython on it at install time. This > requires a C compiler, but I'm OK with that. > Is keeping the cythonized file an option? That would still require a C compiler, but wouldn't require Cython. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
