Re: Python bindings using cppyy (was: An update on Python bindings)

[Philipp A.]

Hi Shaheed!

The year is nearing its end, and I wonder if there has been any progress
and/or if you people need help with the bindings!

I’d really like to revive my IPython console in Kate :D

Best, Philipp

Shaheed Haque  schrieb am Sa., 13. Jan. 2018 um
19:06 Uhr:

> Thanks to some upstream fixes, I have the cppyy-based bindings for KF5 and
> also Qt5 (see below) showing signs of life. Notes:
>
>
>1. The packaging has advanced to the point where I think ECM-based
>framework-by-framework bindings are a real possibility, with both Py2 and
>Py3. AFAICS, this addresses the main feedback received to date.
>2. With reference to the remark about tracking dependencies between
>frameworks, apologies for the delayed response as I somehow missed the
>email. I note that the dependencies currently in CMake often seem
>incomplete. I'll bring that to the community separately.
>3. There is one issue still open upstream (
>
> https://bitbucket.org/wlav/cppyy/issues/16/pragma-link-defined_in-seems-to-select).
>However, I don't consider this to be a showstopper...we might even be able
>to live with it as is.
>4. For me, the jury is still out on PyQt versus a new set of
>cppyy-based Qt bindings. Clearly PyQt is solid and mature, but the
>limitations really concern me (if anybody wants to know more, I'm happy to
>discuss, but let's do that in another thread please). Now, given that there
>are examples in the wild of interoperating cppyy/cling/ROOT with PyQt, I'm
>going to sidestep this question but am playing with a cppyy-based approach.
>At this point, all of Qt has basic cppyy-based bindings, and the next step
>is to tackle things like finding a way to express the object
>ownership/destruction rules in a more-or-less systematic way.
>5. On the P2/P3 question, I'm presently still committed to both P2 and
>P3. I *have* had a couple of minor occasions where P3-only might have been
>nice *for my code*, but if I do find an issue that tips the balance, or I
>find some serious benefit *for the bindings*, I'll drop P2. One possible
>such benefit would be if I can see a sane way to address PEP484 type hints.
>
> To get here, I had to build a subset of the tooling I previously had
> developed for the SIP-based approach. The big difference is the absence of
> any need to support customisation of the generated bindings. I am hopeful
> that in the worst case, there might be some minimal customisation (known as
> Pythonisations in cppyy parlance) such as for #4 above, but nothing like
> the scale needed for SIP.
>
> The core tooling is not specific to KF5 or KDE or Qt5, and is developed in
> upstream cppyy over on bitbucket.org. The core tooling is built around
> CMake, notably for the generation phase and the C++ library build.
>
> The PoC extends the core tooling with Pythonic packaging and installation
> using pip/wheels, also from CMake. As before I would look for help to get
> an ECM equivalent, possibly based on the same approach but perhaps
> including CI and distribution via PyPi.
>
> Finally, now would be a good time for anybody else who wants to get
> involved to step up, especially as a new job limits my free time.
>
> Thanks, Shaheed
>
> P.S. Not to stoke the the P2/P3 wars unnecessarily, but while I know that
> upstream Clang just added P3 support in the clang 5.0 release, current
> Ubuntu only packages it for 2.7.14. So I won't be moving yet...
>
> On 5 November 2017 at 13:23, Boudewijn Rempt  wrote:
>
>> On Sat, 4 Nov 2017, Chris Burel wrote:
>>
>> > I think this is a remarkably short sighted statement. It assumes that
>> people that would use these bindings have no existing Python codebase at
>> all, and can afford to start a brand new project. The reality is much
>> different.
>> >
>> > Let's take a specific example. I have 6 years experience writing Python
>> for the visual effects industry. We have a 10 year old Python 2 codebase.
>> We also use an application from Autodesk called Maya. It has been a Qt 4
>> application with Python 2 embedded since 2012. In 2016 they jumped to qt 5
>> and pyside2. Now Autodesk knows that companies have built large codebase
>> around their product that requires Python 2. What would've happened if
>> pyside2 did not support Python 2.7? They'd be stuck either forcing all
>> their customers to move to Python 3 and risk people not wanting the new
>> version of the software, or they'd be prevented from moving to Qt 5.
>> >
>>
>> You will have to switch to Python 3 by 2019, since that's what the VFX
>> Reference Platform says. If you haven't started on the migration yet,
>> you're very late. And the VFX Refernece Platform is basically Autodesk
>> telling the rest of the industry what to use, including their weird
>> patchset for Qt...
>>
>> > So no, Python 2 is not dead. Not by a long shot.
>>
>> For VFX, it will be dead in 2019. See http://www.vfxplatform.com/
>>
>>
>> --
>> Boudewijn Rempt 

Re: Python bindings using cppyy (was: An update on Python bindings)

[Luca Beltrame]

Il giorno Mon, 5 Nov 2018 22:54:40 +0100
Dominik Haumann  ha scritto:

> ... wasn't there also some python related work by Stefan? Or is that
> unrelated?

That involves only the existing bindings, this is completely different
from an architectural viewpoint.

-- 
Luca Beltrame - KDE Forums team
GPG key ID: A29D259B


pgpLnRry67jGG.pgp
Description: Firma digitale OpenPGP

Re: Python bindings using cppyy (was: An update on Python bindings)

[Dominik Haumann]

... wasn't there also some python related work by Stefan? Or is that
unrelated?

Greetings
Dominik


Am Mo., 5. Nov. 2018, 16:20 hat Shaheed Haque 
geschrieben:

> I'm afraid that there has been no progress as I am buried in "startup"
> mode. I'm not sure when that might change.
>
> On Mon, 5 Nov 2018, 14:02 Philipp A. 
>> Hi Shaheed!
>>
>> The year is nearing its end, and I wonder if there has been any progress
>> and/or if you people need help with the bindings!
>>
>> I’d really like to revive my IPython console in Kate :D
>>
>> Best, Philipp
>>
>> Shaheed Haque  schrieb am Sa., 13. Jan. 2018 um
>> 19:06 Uhr:
>>
>>> Thanks to some upstream fixes, I have the cppyy-based bindings for KF5
>>> and also Qt5 (see below) showing signs of life. Notes:
>>>
>>>
>>>1. The packaging has advanced to the point where I think ECM-based
>>>framework-by-framework bindings are a real possibility, with both Py2 and
>>>Py3. AFAICS, this addresses the main feedback received to date.
>>>2. With reference to the remark about tracking dependencies between
>>>frameworks, apologies for the delayed response as I somehow missed
>>>the email. I note that the dependencies currently in CMake often
>>>seem incomplete. I'll bring that to the community separately.
>>>3. There is one issue still open upstream (
>>>
>>> https://bitbucket.org/wlav/cppyy/issues/16/pragma-link-defined_in-seems-to-select).
>>>However, I don't consider this to be a showstopper...we might even be 
>>> able
>>>to live with it as is.
>>>4. For me, the jury is still out on PyQt versus a new set of
>>>cppyy-based Qt bindings. Clearly PyQt is solid and mature, but the
>>>limitations really concern me (if anybody wants to know more, I'm happy 
>>> to
>>>discuss, but let's do that in another thread please). Now, given that 
>>> there
>>>are examples in the wild of interoperating cppyy/cling/ROOT with PyQt, 
>>> I'm
>>>going to sidestep this question but am playing with a cppyy-based 
>>> approach.
>>>At this point, all of Qt has basic cppyy-based bindings, and the next 
>>> step
>>>is to tackle things like finding a way to express the object
>>>ownership/destruction rules in a more-or-less systematic way.
>>>5. On the P2/P3 question, I'm presently still committed to both P2
>>>and P3. I *have* had a couple of minor occasions where P3-only might have
>>>been nice *for my code*, but if I do find an issue that tips the balance,
>>>or I find some serious benefit *for the bindings*, I'll drop P2. One
>>>possible such benefit would be if I can see a sane way to address PEP484
>>>type hints.
>>>
>>> To get here, I had to build a subset of the tooling I previously had
>>> developed for the SIP-based approach. The big difference is the absence of
>>> any need to support customisation of the generated bindings. I am hopeful
>>> that in the worst case, there might be some minimal customisation (known as
>>> Pythonisations in cppyy parlance) such as for #4 above, but nothing like
>>> the scale needed for SIP.
>>>
>>> The core tooling is not specific to KF5 or KDE or Qt5, and is developed
>>> in upstream cppyy over on bitbucket.org. The core tooling is built
>>> around CMake, notably for the generation phase and the C++ library build.
>>>
>>> The PoC extends the core tooling with Pythonic packaging and
>>> installation using pip/wheels, also from CMake. As before I would look for
>>> help to get an ECM equivalent, possibly based on the same approach but
>>> perhaps including CI and distribution via PyPi.
>>>
>>> Finally, now would be a good time for anybody else who wants to get
>>> involved to step up, especially as a new job limits my free time.
>>>
>>> Thanks, Shaheed
>>>
>>> P.S. Not to stoke the the P2/P3 wars unnecessarily, but while I know
>>> that upstream Clang just added P3 support in the clang 5.0 release, current
>>> Ubuntu only packages it for 2.7.14. So I won't be moving yet...
>>>
>>> On 5 November 2017 at 13:23, Boudewijn Rempt  wrote:
>>>
 On Sat, 4 Nov 2017, Chris Burel wrote:

 > I think this is a remarkably short sighted statement. It assumes that
 people that would use these bindings have no existing Python codebase at
 all, and can afford to start a brand new project. The reality is much
 different.
 >
 > Let's take a specific example. I have 6 years experience writing
 Python for the visual effects industry. We have a 10 year old Python 2
 codebase. We also use an application from Autodesk called Maya. It has been
 a Qt 4 application with Python 2 embedded since 2012. In 2016 they jumped
 to qt 5 and pyside2. Now Autodesk knows that companies have built large
 codebase around their product that requires Python 2. What would've
 happened if pyside2 did not support Python 2.7? They'd be stuck either
 forcing all their customers to move to Python 3 and risk people not wanting
 the new version of 

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

I'm afraid that there has been no progress as I am buried in "startup"
mode. I'm not sure when that might change.

On Mon, 5 Nov 2018, 14:02 Philipp A.  Hi Shaheed!
>
> The year is nearing its end, and I wonder if there has been any progress
> and/or if you people need help with the bindings!
>
> I’d really like to revive my IPython console in Kate :D
>
> Best, Philipp
>
> Shaheed Haque  schrieb am Sa., 13. Jan. 2018 um
> 19:06 Uhr:
>
>> Thanks to some upstream fixes, I have the cppyy-based bindings for KF5
>> and also Qt5 (see below) showing signs of life. Notes:
>>
>>
>>1. The packaging has advanced to the point where I think ECM-based
>>framework-by-framework bindings are a real possibility, with both Py2 and
>>Py3. AFAICS, this addresses the main feedback received to date.
>>2. With reference to the remark about tracking dependencies between
>>frameworks, apologies for the delayed response as I somehow missed
>>the email. I note that the dependencies currently in CMake often seem
>>incomplete. I'll bring that to the community separately.
>>3. There is one issue still open upstream (
>>
>> https://bitbucket.org/wlav/cppyy/issues/16/pragma-link-defined_in-seems-to-select).
>>However, I don't consider this to be a showstopper...we might even be able
>>to live with it as is.
>>4. For me, the jury is still out on PyQt versus a new set of
>>cppyy-based Qt bindings. Clearly PyQt is solid and mature, but the
>>limitations really concern me (if anybody wants to know more, I'm happy to
>>discuss, but let's do that in another thread please). Now, given that 
>> there
>>are examples in the wild of interoperating cppyy/cling/ROOT with PyQt, I'm
>>going to sidestep this question but am playing with a cppyy-based 
>> approach.
>>At this point, all of Qt has basic cppyy-based bindings, and the next step
>>is to tackle things like finding a way to express the object
>>ownership/destruction rules in a more-or-less systematic way.
>>5. On the P2/P3 question, I'm presently still committed to both P2
>>and P3. I *have* had a couple of minor occasions where P3-only might have
>>been nice *for my code*, but if I do find an issue that tips the balance,
>>or I find some serious benefit *for the bindings*, I'll drop P2. One
>>possible such benefit would be if I can see a sane way to address PEP484
>>type hints.
>>
>> To get here, I had to build a subset of the tooling I previously had
>> developed for the SIP-based approach. The big difference is the absence of
>> any need to support customisation of the generated bindings. I am hopeful
>> that in the worst case, there might be some minimal customisation (known as
>> Pythonisations in cppyy parlance) such as for #4 above, but nothing like
>> the scale needed for SIP.
>>
>> The core tooling is not specific to KF5 or KDE or Qt5, and is developed
>> in upstream cppyy over on bitbucket.org. The core tooling is built
>> around CMake, notably for the generation phase and the C++ library build.
>>
>> The PoC extends the core tooling with Pythonic packaging and installation
>> using pip/wheels, also from CMake. As before I would look for help to get
>> an ECM equivalent, possibly based on the same approach but perhaps
>> including CI and distribution via PyPi.
>>
>> Finally, now would be a good time for anybody else who wants to get
>> involved to step up, especially as a new job limits my free time.
>>
>> Thanks, Shaheed
>>
>> P.S. Not to stoke the the P2/P3 wars unnecessarily, but while I know that
>> upstream Clang just added P3 support in the clang 5.0 release, current
>> Ubuntu only packages it for 2.7.14. So I won't be moving yet...
>>
>> On 5 November 2017 at 13:23, Boudewijn Rempt  wrote:
>>
>>> On Sat, 4 Nov 2017, Chris Burel wrote:
>>>
>>> > I think this is a remarkably short sighted statement. It assumes that
>>> people that would use these bindings have no existing Python codebase at
>>> all, and can afford to start a brand new project. The reality is much
>>> different.
>>> >
>>> > Let's take a specific example. I have 6 years experience writing
>>> Python for the visual effects industry. We have a 10 year old Python 2
>>> codebase. We also use an application from Autodesk called Maya. It has been
>>> a Qt 4 application with Python 2 embedded since 2012. In 2016 they jumped
>>> to qt 5 and pyside2. Now Autodesk knows that companies have built large
>>> codebase around their product that requires Python 2. What would've
>>> happened if pyside2 did not support Python 2.7? They'd be stuck either
>>> forcing all their customers to move to Python 3 and risk people not wanting
>>> the new version of the software, or they'd be prevented from moving to Qt 5.
>>> >
>>>
>>> You will have to switch to Python 3 by 2019, since that's what the VFX
>>> Reference Platform says. If you haven't started on the migration yet,
>>> you're very late. And the VFX Refernece Platform is 

Re: Python bindings using cppyy (was: An update on Python bindings)

[Ben Cooksley]

On Sun, Jan 14, 2018 at 7:05 AM, Shaheed Haque  wrote:
> Thanks to some upstream fixes, I have the cppyy-based bindings for KF5 and
> also Qt5 (see below) showing signs of life. Notes:

Hi Shaheed,

>
> The packaging has advanced to the point where I think ECM-based
> framework-by-framework bindings are a real possibility, with both Py2 and
> Py3. AFAICS, this addresses the main feedback received to date.
> With reference to the remark about tracking dependencies between frameworks,
> apologies for the delayed response as I somehow missed the email. I note
> that the dependencies currently in CMake often seem incomplete. I'll bring
> that to the community separately.
> There is one issue still open upstream
> (https://bitbucket.org/wlav/cppyy/issues/16/pragma-link-defined_in-seems-to-select).
> However, I don't consider this to be a showstopper...we might even be able
> to live with it as is.
> For me, the jury is still out on PyQt versus a new set of cppyy-based Qt
> bindings. Clearly PyQt is solid and mature, but the limitations really
> concern me (if anybody wants to know more, I'm happy to discuss, but let's
> do that in another thread please). Now, given that there are examples in the
> wild of interoperating cppyy/cling/ROOT with PyQt, I'm going to sidestep
> this question but am playing with a cppyy-based approach. At this point, all
> of Qt has basic cppyy-based bindings, and the next step is to tackle things
> like finding a way to express the object ownership/destruction rules in a
> more-or-less systematic way.
> On the P2/P3 question, I'm presently still committed to both P2 and P3. I
> *have* had a couple of minor occasions where P3-only might have been nice
> *for my code*, but if I do find an issue that tips the balance, or I find
> some serious benefit *for the bindings*, I'll drop P2. One possible such
> benefit would be if I can see a sane way to address PEP484 type hints.
>
> To get here, I had to build a subset of the tooling I previously had
> developed for the SIP-based approach. The big difference is the absence of
> any need to support customisation of the generated bindings. I am hopeful
> that in the worst case, there might be some minimal customisation (known as
> Pythonisations in cppyy parlance) such as for #4 above, but nothing like the
> scale needed for SIP.
>
> The core tooling is not specific to KF5 or KDE or Qt5, and is developed in
> upstream cppyy over on bitbucket.org. The core tooling is built around
> CMake, notably for the generation phase and the C++ library build.
>
> The PoC extends the core tooling with Pythonic packaging and installation
> using pip/wheels, also from CMake. As before I would look for help to get an
> ECM equivalent, possibly based on the same approach but perhaps including CI
> and distribution via PyPi.

Once the necessary changes and tooling have been landed into the
Frameworks (including ECM) repositories we can certainly look into any
changes that are needed on the CI system. I think the Python
development packages should already be installed though, courtesy of
the needs of other KDE software (these might be Python 3 ones though).

In regards to PyPi - are you thinking of source packages here, or binaries?

If it's sources, then it's something that would need to be handled as
part of the Frameworks release process (ideally someone other than
David would look after it, making sure each one that was being
uploaded worked). For binaries, that would be best handled by the
Binary Factory.

>
> Finally, now would be a good time for anybody else who wants to get involved
> to step up, especially as a new job limits my free time.
>
> Thanks, Shaheed

Cheers,
Ben Cooksley
KDE Sysadmin

>
> P.S. Not to stoke the the P2/P3 wars unnecessarily, but while I know that
> upstream Clang just added P3 support in the clang 5.0 release, current
> Ubuntu only packages it for 2.7.14. So I won't be moving yet...
>
> On 5 November 2017 at 13:23, Boudewijn Rempt  wrote:
>>
>> On Sat, 4 Nov 2017, Chris Burel wrote:
>>
>> > I think this is a remarkably short sighted statement. It assumes that
>> > people that would use these bindings have no existing Python codebase at
>> > all, and can afford to start a brand new project. The reality is much
>> > different.
>> >
>> > Let's take a specific example. I have 6 years experience writing Python
>> > for the visual effects industry. We have a 10 year old Python 2 codebase. 
>> > We
>> > also use an application from Autodesk called Maya. It has been a Qt 4
>> > application with Python 2 embedded since 2012. In 2016 they jumped to qt 5
>> > and pyside2. Now Autodesk knows that companies have built large codebase
>> > around their product that requires Python 2. What would've happened if
>> > pyside2 did not support Python 2.7? They'd be stuck either forcing all 
>> > their
>> > customers to move to Python 3 and risk people not wanting the new version 
>> > of
>> > the software, or 

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

Hi Luca,

On 15 January 2018 at 08:24, Luca Beltrame  wrote:

> Il giorno Sat, 13 Jan 2018 18:05:45 +
> Shaheed Haque  ha scritto:
>
> Hello Shaheed,
>
> >1. The packaging has advanced to the point where I think ECM-based
> >framework-by-framework bindings are a real possibility, with both
>
> That's absolutely fantastic. Thanks for the continued effort!
>

It is a relief to get to this point after what, about 2 years? Also, a lot
of credit has to go to the support of upstream (Wim, but also not
forgettting Phil on the the SIP side of things). Still, it is not quite
time to put out the bunting...

>
> >4. For me, the jury is still out on PyQt versus a new set of
> > cppyy-based Qt bindings. Clearly PyQt is solid and mature, but the
>
> How do the bindings work? I suppose you'll need cppyy-based Qt to run
> your bindings, correct?
>

Technically, there seem to be example in the wild of people (at CERN and
possibly elsewhere?) creating applications with ROOT (the immediate
precursor of cppyy) and PyQt. And since everything is, at one level, just
Python, I don't see any bar to using PyQT. However, recall that I only went
down the cppyy route after pouring a LOT of effort into the SIP route, and
then concluding both the customisation effort and the limitations were too
great for my liking. Basically, I expect it would take ongoing large-scale
investment from somebody like me to keep the bindings going.

So, given the current cppyy Qt bindings required NO customisation effort
(but see below) and should free us of the limitations of the SIP-generated
approach, I'm putting my effort into the cppyy-based approach.

> like finding a way to express the object ownership/destruction rules
>
> How are you dealing with Python's GC? A major point (and PITA) when
> developing with PyQt and PyKDE4 was that a single slip in tracking
> objects meant the C++ object would get GC'd under your feet,
> causing crashes. This was particularly bad with the Plasma bindings
> because they would crash the whole desktop.
>

As you will understand, there is no magic bullet for this. Both SIP and
cppyy need to be told how to handle this. With SIP, part of the extensive
customisation framework I created allowed me to say things like:

-for all constructors that match a set of regexps, and which have one
argument called "parent" of type "QWidget *", declare that Python does not
own the object

There is a similar capability in cppyy, and I just have to create a way to
pass that hint.


> > benefit *for the bindings*, I'll drop P2. One possible such benefit
>
> I don't mind either approach: FYI though, most major distros are
> phasing out Py2-as-default (Ubuntu, Arch, and openSUSE are the ones I'm
> aware of).
>
> > minimal customisation (known as Pythonisations in cppyy parlance)
> > such as for #4 above, but nothing like the scale needed for SIP.
>
> That would be great. It was one of the major issues when keeping PyKDE4
> up to date.
>

Precisely. In marked contrast to SIP, I am hopeful that this ownership
thingy may in fact be the ONLY customisation needed (though I/we need to
think about Qt meta object stuff like signals/slots too).


> > look for help to get an ECM equivalent, possibly based on the same
> > approach but perhaps including CI and distribution via PyPi.
>
> Can you link the repository with the tools?
>

Not sure what you mean. It is all on PyPI (and bitbucket):
https://pypi.python.org/pypi?%3Aaction=search=cppyy=search.

Let me know if I missed your point.

Thanks, Shaheed


> --
> Luca Beltrame, Ph.D.
> Translational Genomics Unit, Department of Oncology
> IRCCS Istituto di Ricerche Farmacologiche "Mario Negri"
>

Re: Python bindings using cppyy (was: An update on Python bindings)

[Albert Astals Cid]

El dissabte, 13 de gener de 2018, a les 18:05:45 CET, Shaheed Haque va 
escriure:
> Thanks to some upstream fixes, I have the cppyy-based bindings for KF5 and
> also Qt5 (see below) showing signs of life. 
> Notes:

This is awesome, i'm really happy we're in a point that framework-by-framework 
is possible and that it all seems to be upstream so it's a "hopefully bigger 
group of people" maintaining it :)

Keep it up!

Cheers,
  Albert

> 
> 
>1. The packaging has advanced to the point where I think ECM-based
>framework-by-framework bindings are a real possibility, with both Py2 and
> Py3. AFAICS, this addresses the main feedback received to date. 2. With
> reference to the remark about tracking dependencies between frameworks,
> apologies for the delayed response as I somehow missed the email. I note
> that the dependencies currently in CMake often seem incomplete. I'll bring
> that to the community separately.
>3. There is one issue still open upstream (
>   
> https://bitbucket.org/wlav/cppyy/issues/16/pragma-link-defined_in-seems-to-> 
> select). However, I don't consider this to be a showstopper...we might even
> be able to live with it as is.
>4. For me, the jury is still out on PyQt versus a new set of cppyy-based
>Qt bindings. Clearly PyQt is solid and mature, but the limitations really
> concern me (if anybody wants to know more, I'm happy to discuss, but let's
> do that in another thread please). Now, given that there are examples in
> the wild of interoperating cppyy/cling/ROOT with PyQt, I'm going to
> sidestep this question but am playing with a cppyy-based approach. At this
> point, all of Qt has basic cppyy-based bindings, and the next step is to
> tackle things like finding a way to express the object
>ownership/destruction rules in a more-or-less systematic way.
>5. On the P2/P3 question, I'm presently still committed to both P2 and
>P3. I *have* had a couple of minor occasions where P3-only might have
> been nice *for my code*, but if I do find an issue that tips the balance,
> or I find some serious benefit *for the bindings*, I'll drop P2. One
> possible such benefit would be if I can see a sane way to address PEP484
> type hints.
> 
> To get here, I had to build a subset of the tooling I previously had
> developed for the SIP-based approach. The big difference is the absence of
> any need to support customisation of the generated bindings. I am hopeful
> that in the worst case, there might be some minimal customisation (known as
> Pythonisations in cppyy parlance) such as for #4 above, but nothing like
> the scale needed for SIP.
> 
> The core tooling is not specific to KF5 or KDE or Qt5, and is developed in
> upstream cppyy over on bitbucket.org. The core tooling is built around
> CMake, notably for the generation phase and the C++ library build.
> 
> The PoC extends the core tooling with Pythonic packaging and installation
> using pip/wheels, also from CMake. As before I would look for help to get
> an ECM equivalent, possibly based on the same approach but perhaps
> including CI and distribution via PyPi.
> 
> Finally, now would be a good time for anybody else who wants to get
> involved to step up, especially as a new job limits my free time.
> 
> Thanks, Shaheed
> 
> P.S. Not to stoke the the P2/P3 wars unnecessarily, but while I know that
> upstream Clang just added P3 support in the clang 5.0 release, current
> Ubuntu only packages it for 2.7.14. So I won't be moving yet...
> 
> On 5 November 2017 at 13:23, Boudewijn Rempt  wrote:
> > On Sat, 4 Nov 2017, Chris Burel wrote:
> > > I think this is a remarkably short sighted statement. It assumes that
> > 
> > people that would use these bindings have no existing Python codebase at
> > all, and can afford to start a brand new project. The reality is much
> > different.
> > 
> > > Let's take a specific example. I have 6 years experience writing Python
> > 
> > for the visual effects industry. We have a 10 year old Python 2 codebase.
> > We also use an application from Autodesk called Maya. It has been a Qt 4
> > application with Python 2 embedded since 2012. In 2016 they jumped to qt 5
> > and pyside2. Now Autodesk knows that companies have built large codebase
> > around their product that requires Python 2. What would've happened if
> > pyside2 did not support Python 2.7? They'd be stuck either forcing all
> > their customers to move to Python 3 and risk people not wanting the new
> > version of the software, or they'd be prevented from moving to Qt 5.
> > 
> > 
> > You will have to switch to Python 3 by 2019, since that's what the VFX
> > Reference Platform says. If you haven't started on the migration yet,
> > you're very late. And the VFX Refernece Platform is basically Autodesk
> > telling the rest of the industry what to use, including their weird
> > patchset for Qt...
> > 
> > > So no, Python 2 is not dead. Not by a long shot.
> > 
> > For VFX, it will be dead in 2019. See 

Re: Python bindings using cppyy (was: An update on Python bindings)

[Kevin Kofler]

Philipp A. wrote:
> No, because you’re missing something here: There’s no KF5 bindings. So
> every project that’ll use Shaheed’s new cool KF5 bindings will be a new
> project.

There is PyKDE4 that people will want to port their legacy programs from.

Kevin Kofler

Re: Python bindings using cppyy (was: An update on Python bindings)

[Philipp A.]

Hi Shaheed, Chris,

Shaheed Haque  schrieb am Sa., 4. Nov. 2017 um
18:35 Uhr:

> FWIW, I already tried that (types.ModuleType is itself a perfectly
> subclassable class!) […]
>
> Now, none of that may be a limiting factor in the plan you seem to be
> discussing, but it was part of what made me think "here be dragons"...


For me that sounds like a perfectly acceptable clearly-defined instruction:

   - If you can, don’t use a __init__.py for namespace packages (because
   it’s simple).
   - If you need an __init__.py, only use one or use identical ones.

Reasons for needing a __init__.py include the need for toplevel
constants/classes/functions, or supporting Python 2. The docs also say that
you can mix __init__.py omission and (identical) __init__.pys.

I created an example here: https://github.com/flying-sheep/namespace-test

You can see that by using a __init__.py in exactly one of the merged
packages, you can define toplevel constants/classes/functions like BASE in
the example.

If we need, we can also define toplevel constants and so on in multiple
distributions, like this (this specific version requires all distributions
to know about all others, but that’s automatable):

*namespace-test-1/namespace_test/namespace_test_1_init.py*:
FOO = 1
*namespace-test-2/namespace_test/**namespace_test_2_init.py*:
BAR= 1
*namespace-test-{1,2}/namespace_test/**__init__.py*:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
try:
from .namespace_test_1_init import *
except ImportError:
pass
try:
from .namespace_test_2_init import *
except ImportError:
pass

Chris Burel  schrieb am So., 5. Nov. 2017 um
02:49 Uhr:

> I think this is a remarkably short sighted statement. It assumes that
> people that would use these bindings have no existing Python codebase at
> all, and can afford to start a brand new project. The reality is much
> different. […]


No, because you’re missing something here: There’s no KF5 bindings. So
every project that’ll use Shaheed’s new cool KF5 bindings will be a new
project.

Therefore the only thing Python 2 KF5 bindings would accomplish is to make
people think that starting a *new* Python 2 project in 2018 was still a
good idea. Which it isn’t.

Re: Python bindings using cppyy (was: An update on Python bindings)

[wlavrijsen]

Hi,

On Friday 2017-11-03 12:52, Philipp A. wrote:

Am I missing something? Namespaces should be Python modules, not classes.
If we can do represent them this way, the problem is solveable:
https://packaging.python.org/guides/packaging-namespace-packages/


there are two different things that should not be mixed up together as I
believe they are in this discussion: what cppyy does for purely internal
reasons, and how some package that uses it does its organization. In fact,
the idea of cppyy has always been to provide very "C++-like" python, and
then fix that up in Python to be more pythonistic (as opposed to fixing it
up in C++ or any 3rd language).

To be concrete, in cppyy, I can (and want to be able to) do this:

  >>> import cppyy
  >>> cppyy.cppdef("namespace A { int i = 42; }")
  >>> cppyy.gbl.A.i
  42
  >>> cppyy.gbl.A.i = 17
  >>> cppyy.cppdef("void print_i() { std::cout << A::i << std::endl; }")
  >>> cppyy.gbl.print_i()
  17
  >>>

So now I have a (C++) namespace 'A' that bears no relationship to anything
to do with the file system or any type of Python packaging: it exists only
in memory for the duration of the python session.

For the above to work, I needed to make certain design decisions and the
conclusion was to make 'A' an instance of a subclass of type type, which,
yes, means that it is a class-like type.

None of this, however, should limit the choices of a pythonization layer
on top of cppyy. Yes, there is the risk of "leakage" and some of that may
be inevitable (think e.g. automatically generated __doc__ strings), but by
and large, it should be contained. There should be no clashes, however, as
the C++ namespaces are installed in sys.modules with the 'cppyy.gbl'
prefix, so a Python module called 'A' can live right next to it.

Also, the reason that I'm confident it can be done, is because it has been
done:

  http://www.rootpy.org/

albeit that the structure there is simpler than in this discussion. OTOH,
rootpy goes way farther in renaming things.

On Friday 2017-11-03 13:15, Shaheed Haque wrote:

That's likey to be a bad idea because of the potential impact on
arbitrary round trips between C++ and Python (remember that everything
is based on named-based lookups).


The renaming does not actually scare me that much from a performance point
of view: already, C++ has typedefs for classes and aliases for namespaces.
There is always the problem of "leakage" as mentioned above, where the
end-user sees both at some point, but internally, aliasing will work fine:
it's just another reference to the same object.

On Friday 2017-11-03 14:09, Philipp A. wrote:

I'd be interested in why. Usually using classes as namespaces is only done
for reasons of cuteness (callable namespaces, namespaces usable as context
managers, ...) or so.


It is to attach a meta class for a __getattr__, the use of properties, and
to allow pickling. The module type does not support meta classes.

On Friday 2017-11-03 14:09, Philipp A. wrote:

Even in this case, it's possible to replace the module's class with a
module subclass that has the necessary capabilities (modules are objects
that have a class, too)


Certainly. A module is functionally a subset of a class, so duck typing
ensures that a class can be placed anywhere a module can, so it's either
enhance module or restrict class. However, the use of a meta class is so
much easier to just reuse from type type as opposed to reinventing that
wholesale for module type.

Just to put a historic note here: Python has a lot of "meta" features to
make anything behave like anything else. Using them, however, has two
distinct disadvantages. 1) A lot of tooling relies on very specific
behavior and goes belly-up if anything is slightly different (I'm looking
at you, inspect!) 2) Way too many packages use these features and they never
work well together. (I'm looking at you, ipython!)

Thus, although in the past I've made bountiful use of all the dunderscore
features that Python offers, I've learned my lesson the hard way and these
days I much rather avoid them and use the "builtin" behavior if at all
feasible.

The meta class is one of them: although reworking its behavior is not hard,
but making sure it works _exactly_ as the builtin version across all python
versions and implementations, _is_ hard. (Same goes for reworking the use
of properties, etc.)

That doesn't mean I'm totally innocent here: I'm going through quite some
steps to satisfy inspect, including the on-the-fly generation of code
objects (p2 only for now), to make help() do things like this:

 |  SavePrimitive(self, basic_ostream& out, const 
char* option='')
 |  void TObject::SavePrimitive(basic_ostream& out, const 
char* option = "")

Note above the type info in the python (!) representation of that method
(the second line is just the doc string of course, but the first one is
build up by fooling inspect). Yes, that's clearly just "cuteness". OTOH,
this plays nicely 

Re: Python bindings using cppyy (was: An update on Python bindings)

[Philipp A.]

Hi Shaheed,

Thank you so much for all your work!

a framework-by-framework integration of the binding generation logic (as
> previously pioneered by Steve) probably cannot work in general because
> there are cases where multiple frameworks contribute to to the same C++
> namespace […]
>
> The problem is that the Python implementation of these namespaces is a
> class, and so treating these frameworks […] as separate would result in
> multiple colliding Python class definitions.


Am I missing something? Namespaces should be Python modules, not classes.
If we can do represent them this way, the problem is solveable:
https://packaging.python.org/guides/packaging-namespace-packages/

The original TODOs and bugs have been resolved, and there is the beginnings
> of support for packaging frameworks under a Python namespace as in
> "KF5.KDCRAW".


Once they’re modules, we should probably respect that Python modules are by
convention lowercase. It would be best if we named them kf5.kdcraw and so
on.

Thank you again,
Best, Philipp

Re: Python bindings using cppyy (was: An update on Python bindings)

[Philipp A.]

Hi Shaheed,

Shaheed Haque  schrieb am Fr., 3. Nov. 2017 um
14:16 Uhr:

> Philipp,
>
> - my overall understanding of this technique is that it may end up
> being fragile, especially given the difference between P2 and P3.
>

Python 2? I’m sure we shouldn’t include into our decision making an
obsolete language that has a final (yes, really this time!) expiration date
2 years in the future. 2020 is the end of the line, I certainly don’t
bother anymore to write a single line accomodating to it, and would suggest
you do the same for a new project.

- using it further down might not work as expected especially if there
> are "accidental" collisions in the directory/namespace names.
> - it is also not clear if the technique can be used multiple times.
>

We should check if this is the case. Who’s a Python guru who knows the ins
and outs of the module system?

- cppyy gives us classes. (Actually, in conversation with Wim, CC'd
> here, it turns out that the choice of using classes is not arbitrary,
> but we were pondering simple modules at that point, for other
> reasons).
>

I’d be interested in why. Usually using classes as namespaces is only done
for reasons of cuteness (callable namespaces, namespaces usable as context
managers, …) or so.

Even in this case, it’s possible to replace the module’s class with a
module subclass that has the necessary capabilities (modules are objects
that have a class, too)


> Bottom line: I'm not keen on using Python namespace modules here for
> the reasons listed.
>

I’m not entirely convinced. We should only do a final decision once we know
if either your suspicions of multiple levels not working turn out to be
true, or the reason for using classes is important.


> That's likey to be a bad idea because of the potential impact on arbitrary
> round trips between C++ and Python (remember that everything is based on
> named-based lookups). In addition, we already have problems like "gpgme++",
> and the use of capitalisation to separate legacy and forwarding headers in
> KDE: further case-based confusion is the last thing that is needed!
>

I see, makes sense. I guess the allcaps example here is not very common
anyway, and most namespaces will be UpperCamelCase like Qt’s, right?

Thanks for the review/remarks, Shaheed
>

NP!
Best, Philipp

Re: Python bindings using cppyy (was: An update on Python bindings)

[Chris Burel]

> On Nov 4, 2017, at 4:46 AM, Philipp A.  wrote:
> 
> Entirely new bindings lead to new applications being written using those 
> bindings. Writing applications in Python 2 is an immediate maintenance burden 
> and people only do it because of stubborn ideology or a complete lack of 
> awareness that Python 2 is being killed off for good.
I think this is a remarkably short sighted statement. It assumes that people 
that would use these bindings have no existing Python codebase at all, and can 
afford to start a brand new project. The reality is much different.

Let's take a specific example. I have 6 years experience writing Python for the 
visual effects industry. We have a 10 year old Python 2 codebase. We also use 
an application from Autodesk called Maya. It has been a Qt 4 application with 
Python 2 embedded since 2012. In 2016 they jumped to qt 5 and pyside2. Now 
Autodesk knows that companies have built large codebase around their product 
that requires Python 2. What would've happened if pyside2 did not support 
Python 2.7? They'd be stuck either forcing all their customers to move to 
Python 3 and risk people not wanting the new version of the software, or they'd 
be prevented from moving to Qt 5.

So no, Python 2 is not dead. Not by a long shot.

-Chris

Re: Python bindings using cppyy (was: An update on Python bindings)

[Philipp A.]

Hi Shaheed,

Thank you for the clarifications!

My observation is that *nobody* is likely to help with that problem: the
> framework owners did
> nothing obvious to either keep PyKDE4 going (out of tree) or to help
> Steve with my earlier SIP based efforts (in tree).
>

It's a bit sad, but not too surprising that the framework maintainers
didn't help. I assume it's a matter of priorities for them as well.

I do however prefer to maintain compatibility (to Python 2) until the
> burden of doing so presents an insurmountable issue.
>

At this point I'd say you do humanity a bigger favor if you ditch Python 2
even without maintenance burden.

Entirely new bindings lead to new applications being written using those
bindings. Writing applications in Python 2 is an immediate maintenance
burden and people only do it because of stubborn ideology or a complete
lack of awareness that Python 2 is being killed off for good.

There is no need for a "final decision" from me. I would suggest that
> the first question for anybody that cares is to assess the scope of
> the issue. Unfortunately, i have other more fundamental issues to fry.
>

OK, cool! So if this is possible and not over my head, I might be able to
try my hands at it.

Best regards, Shaheed
>

The same to you!

>

Re: Python bindings using cppyy (was: An update on Python bindings)

[Philipp A.]

Hi Wim!

So now I have a (C++) namespace 'A' that bears no relationship to anything
> to do with the file system or any type of Python packaging: it exists only
> in memory for the duration of the python session.
>

Yeah, cool, so we just use a path hook and are ready to go right?

https://www.python.org/dev/peps/pep-0302/#specification-part-2-registering-hooks

Every framework gets its own path hook that handles imports from that
framework.

Frameworks with shared namespaces will just handle their own specific
paths, right?

The renaming does not actually scare me that much from a performance point
> of view: already, C++ has typedefs for classes and aliases for namespaces.
> There is always the problem of "leakage" as mentioned above, where the
> end-user sees both at some point, but internally, aliasing will work fine:
> it's just another reference to the same object.
>

Cool, so this might be possible after all! Certainly not as a top priority
before getting things to work, but still!

It is to attach a meta class for a __getattr__, the use of properties, and
> to allow pickling. The module type does not support meta classes.
>

Can't we replace the framework modules’ type with a subclass of both
“module” and “type”?

Thank you for explaining,
Best regards, Philipp

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

I have made an attempt to get roughly all the bindings I was previously
attempting with SIP to be (a) generated and (b) built.

As of now, we have:

- Customisations with a diffstat that reads "21 files changed, 20
insertions(+), 48 deletions(-)".
- All but 5-6 of the formal tier 1, 2 and 3 frameworks [1] building.
- Of the wider set of 111 informal frameworks [2] I have all but 14
building.

Most of the customisations are to workaround missing dependencies in the
CMake descriptions.Most of the build issues are compile-issues (e.g.
incomplete #includes, or incorrect -I settings), but there are at least 2
that look like cppyy issues. This is pretty solid proof that building atop
technology that is based on LLVM and Cling can trivially process at least
these KDE sources (including templates, overloads, operators, and the rest).

Oh, and it has taken about a month to get here. As a baseline, the
SIP-based attempt took about 18 months, several thousand lines of custom
rules (including a lot of deleting overloads, operators and templates that
could not be handled) to get a little over half the bindings through steps
a and b. The build takes about 10 minutes on a single core, versus about
3-4 hours with the SIP-based attempt.

More anon...

Thanks, Shaheed

[1] https://api.kde.org/frameworks/
[2] More liberally defined than [1], corresponding to what Ubuntu installs
as "libkf5xxx" packages. This include [1].

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

Philipp,

On 5 November 2017 at 14:48, Philipp A.  wrote:

> Hi Shaheed, Chris,
>
> Shaheed Haque  schrieb am Sa., 4. Nov. 2017 um
> 18:35 Uhr:
>
>> FWIW, I already tried that (types.ModuleType is itself a perfectly
>> subclassable class!) […]
>>
>> Now, none of that may be a limiting factor in the plan you seem to be
>> discussing, but it was part of what made me think "here be dragons"...
>
>
> For me that sounds like a perfectly acceptable clearly-defined instruction:
>
>- If you can, don’t use a __init__.py for namespace packages (because
>it’s simple).
>- If you need an __init__.py, only use one or use identical ones.
>
> Reasons for needing a __init__.py include the need for toplevel
> constants/classes/functions, or supporting Python 2. The docs also say that
> you can mix __init__.py omission and (identical) __init__.pys.
>
> I created an example here: https://github.com/flying-sheep/namespace-test
>

As I said, I know how this works, and am using it. As I also said, I'm
happy for others to pursue ECM-based bindings because I don't plan to.

My branch in pykde5.git is now at the point where the packaging now works
with wheels. With any luck, the combination of CMake and pip/setuptools I
have
brewed there will facilitate any ECM-based bindings. Note: the packaging
code depends very much on the evolving PR at
https://bitbucket.org/wlav/cppyy-backend/pull-requests/6/an-interim-experimental-version-of-a.
Neither branch is ready to merge or formally review.

I plan to continue to push forward on the remaining functional TODOs.

Thanks, Shaheed

You can see that by using a __init__.py in exactly one of the merged
> packages, you can define toplevel constants/classes/functions like BASE in
> the example.
>
> If we need, we can also define toplevel constants and so on in multiple
> distributions, like this (this specific version requires all distributions
> to know about all others, but that’s automatable):
>
> *namespace-test-1/namespace_test/namespace_test_1_init.py*:
> FOO = 1
> *namespace-test-2/namespace_test/**namespace_test_2_init.py*:
> BAR= 1
> *namespace-test-{1,2}/namespace_test/**__init__.py*:
> from pkgutil import extend_path
> __path__ = extend_path(__path__, __name__)
> try:
> from .namespace_test_1_init import *
> except ImportError:
> pass
> try:
> from .namespace_test_2_init import *
> except ImportError:
> pass
>
> Chris Burel  schrieb am So., 5. Nov. 2017 um
> 02:49 Uhr:
>
>> I think this is a remarkably short sighted statement. It assumes that
>> people that would use these bindings have no existing Python codebase at
>> all, and can afford to start a brand new project. The reality is much
>> different. […]
>
>
> No, because you’re missing something here: There’s no KF5 bindings. So
> every project that’ll use Shaheed’s new cool KF5 bindings will be a new
> project.
>
> Therefore the only thing Python 2 KF5 bindings would accomplish is to make
> people think that starting a *new* Python 2 project in 2018 was still a
> good idea. Which it isn’t.
>

Re: Python bindings using cppyy (was: An update on Python bindings)

[Boudewijn Rempt]

On Sat, 4 Nov 2017, Chris Burel wrote:

> I think this is a remarkably short sighted statement. It assumes that people 
> that would use these bindings have no existing Python codebase at all, and 
> can afford to start a brand new project. The reality is much different.
> 
> Let's take a specific example. I have 6 years experience writing Python for 
> the visual effects industry. We have a 10 year old Python 2 codebase. We also 
> use an application from Autodesk called Maya. It has been a Qt 4 application 
> with Python 2 embedded since 2012. In 2016 they jumped to qt 5 and pyside2. 
> Now Autodesk knows that companies have built large codebase around their 
> product that requires Python 2. What would've happened if pyside2 did not 
> support Python 2.7? They'd be stuck either forcing all their customers to 
> move to Python 3 and risk people not wanting the new version of the software, 
> or they'd be prevented from moving to Qt 5.
> 

You will have to switch to Python 3 by 2019, since that's what the VFX 
Reference Platform says. If you haven't started on the migration yet, you're 
very late. And the VFX Refernece Platform is basically Autodesk telling the 
rest of the industry what to use, including their weird patchset for Qt...

> So no, Python 2 is not dead. Not by a long shot.

For VFX, it will be dead in 2019. See http://www.vfxplatform.com/


-- 
Boudewijn Rempt | http://www.krita.org, http://www.valdyas.org

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

There is a POC-quality implementation of the integration with KDE here:

https://cgit.kde.org/pykde5.git/tree/?h=srhaque-cppyy-bindings=19a94fb3ae2b40a985913ed4e49400e02df56dc2

This contains examples of bindings for Akonadi and KDcraw. My next
steps will be to do a few more, and then move on to the TODOs listed
earlier.

On 4 November 2017 at 17:35, Shaheed Haque  wrote:
> Wim, Philipp,
>
> On 4 November 2017 at 16:45, Philipp A.  wrote:
>> Hi Wim!
>>
>>> So now I have a (C++) namespace 'A' that bears no relationship to anything
>>> to do with the file system or any type of Python packaging: it exists only
>>> in memory for the duration of the python session.
>>
>>
>> Yeah, cool, so we just use a path hook and are ready to go right?
>>
>> https://www.python.org/dev/peps/pep-0302/#specification-part-2-registering-hooks
>>
>> Every framework gets its own path hook that handles imports from that
>> framework.
>>
>> Frameworks with shared namespaces will just handle their own specific paths,
>> right?
>>
>>> The renaming does not actually scare me that much from a performance point
>>> of view: already, C++ has typedefs for classes and aliases for namespaces.
>>> There is always the problem of "leakage" as mentioned above, where the
>>> end-user sees both at some point, but internally, aliasing will work fine:
>>> it's just another reference to the same object.
>>
>>
>> Cool, so this might be possible after all! Certainly not as a top priority
>> before getting things to work, but still!
>>
>>> It is to attach a meta class for a __getattr__, the use of properties, and
>>> to allow pickling. The module type does not support meta classes.
>>
>>
>> Can't we replace the framework modules’ type with a subclass of both
>> “module” and “type”?
>
> FWIW, I already tried that (types.ModuleType is itself a perfectly
> subclassable class!) in the case where the framework had the form
>  and that worked fine. I did not pursue this for the case
> where the framework name if "KF5.", i.e. the case where
> the shared namespace comes into the picture because of these
> statements on 
> https://packaging.python.org/guides/packaging-namespace-packages/
> under the native Python3-only case:
>
> - "It is extremely important that every distribution that uses the
> namespace package omits the __init__.py or uses a pkgutil-style
> __init__.py. If any distribution does not, it will cause the namespace
> logic to fail and the other sub-packages will not be importable."
>
> and under the Python2/3 compatible pkg-util case:
>
> - "The __init__.py file for the namespace package needs to contain
> only the following:
>
> __path__ = __import__('pkgutil').extend_path(__path__, __name__)
>
> Every distribution that uses the namespace package must include an
> identical __init__.py. If any distribution does not, it will cause the
> namespace logic to fail and the other sub-packages will not be
> importable. Any additional code in __init__.py will be inaccessible."
>
> Now, none of that may be a limiting factor in the plan you seem to be
> discussing, but it was part of what made me think "here be dragons"...
>
> Shaheed
>
>> Thank you for explaining,
>> Best regards, Philipp

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

Wim, Philipp,

On 4 November 2017 at 16:45, Philipp A.  wrote:
> Hi Wim!
>
>> So now I have a (C++) namespace 'A' that bears no relationship to anything
>> to do with the file system or any type of Python packaging: it exists only
>> in memory for the duration of the python session.
>
>
> Yeah, cool, so we just use a path hook and are ready to go right?
>
> https://www.python.org/dev/peps/pep-0302/#specification-part-2-registering-hooks
>
> Every framework gets its own path hook that handles imports from that
> framework.
>
> Frameworks with shared namespaces will just handle their own specific paths,
> right?
>
>> The renaming does not actually scare me that much from a performance point
>> of view: already, C++ has typedefs for classes and aliases for namespaces.
>> There is always the problem of "leakage" as mentioned above, where the
>> end-user sees both at some point, but internally, aliasing will work fine:
>> it's just another reference to the same object.
>
>
> Cool, so this might be possible after all! Certainly not as a top priority
> before getting things to work, but still!
>
>> It is to attach a meta class for a __getattr__, the use of properties, and
>> to allow pickling. The module type does not support meta classes.
>
>
> Can't we replace the framework modules’ type with a subclass of both
> “module” and “type”?

FWIW, I already tried that (types.ModuleType is itself a perfectly
subclassable class!) in the case where the framework had the form
 and that worked fine. I did not pursue this for the case
where the framework name if "KF5.", i.e. the case where
the shared namespace comes into the picture because of these
statements on https://packaging.python.org/guides/packaging-namespace-packages/
under the native Python3-only case:

- "It is extremely important that every distribution that uses the
namespace package omits the __init__.py or uses a pkgutil-style
__init__.py. If any distribution does not, it will cause the namespace
logic to fail and the other sub-packages will not be importable."

and under the Python2/3 compatible pkg-util case:

- "The __init__.py file for the namespace package needs to contain
only the following:

__path__ = __import__('pkgutil').extend_path(__path__, __name__)

Every distribution that uses the namespace package must include an
identical __init__.py. If any distribution does not, it will cause the
namespace logic to fail and the other sub-packages will not be
importable. Any additional code in __init__.py will be inaccessible."

Now, none of that may be a limiting factor in the plan you seem to be
discussing, but it was part of what made me think "here be dragons"...

Shaheed

> Thank you for explaining,
> Best regards, Philipp

Re: Python bindings using cppyy (was: An update on Python bindings)

[Luca Beltrame]

Il giorno Fri, 3 Nov 2017 16:20:19 +
Shaheed Haque  ha scritto:

> *nobody* is likely to help with that problem: the framework owners did
> nothing obvious to either keep PyKDE4 going (out of tree) or to help
> Steve with my earlier SIP based efforts (in tree).

That's because SIP maintained stuff was fragile, didn't even properly
work in non-Ubuntu distributions, and wasn't tested by the CI. You
can't help fixing things you don't know that are broken.

In addition, inter-bindings dependencies weren't tracked so builds
failed at random. And all of this was undocumented so it was even
harder for people without a solid C/C++ background like me to jump in
and try to fix things.

I've spent countless hours trying to package the existing bindings for
openSUSE and I decided it wasn't worth the effort given the 0
maintenance they had. 

This is my main argument *against* out of tree bindings. They should:

- Be part of the framework they refer to (same repo)
- Be tested by the CI

Otherwise it'll be PyKDE4 all it over again.

> cases (Akonadi). I won't be in a position to gain a wider perspective
> until these two are working solidly.

I would suggest not to work with Akonadi at this point. PIM has no ABI
or API stability guarantee, so you'd be left chasing the evolution of
the libs. 

For the reasons I outlined in the past mails, it'd be much better to
use the Frameworks themselves thanks to API and ABI stability policies.

-- 
Luca Beltrame - KDE Forums team
GPG key ID: A29D259B


pgpl7cyEaFiEF.pgp
Description: Firma digitale OpenPGP

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

Hi Philipp,

On 3 November 2017 at 14:09, Philipp A.  wrote:
> Hi Shaheed,
>
> Shaheed Haque  schrieb am Fr., 3. Nov. 2017 um 14:16
> Uhr:
>>
>> Philipp,
>>
>> - my overall understanding of this technique is that it may end up
>> being fragile, especially given the difference between P2 and P3.
>
>
> Python 2? I’m sure we shouldn’t include into our decision making an obsolete
> language that has a final (yes, really this time!) expiration date 2 years
> in the future. 2020 is the end of the line, I certainly don’t bother anymore
> to write a single line accomodating to it, and would suggest you do the same
> for a new project.

The key issue for me has only ever been to get to usable bindings for
Kate, via the necessary stepping stone of KF5. My observation is that
*nobody* is likely to help with that problem: the framework owners did
nothing obvious to either keep PyKDE4 going (out of tree) or to help
Steve with my earlier SIP based efforts (in tree).

In that context, this is a low level issue I'm not especially excited by.

>> - using it further down might not work as expected especially if there
>> are "accidental" collisions in the directory/namespace names.
>> - it is also not clear if the technique can be used multiple times.
>
>
> We should check if this is the case. Who’s a Python guru who knows the ins
> and outs of the module system?

To be clear, I'm not that guru.

Having said that, the comment about Python2 is a missing the the key
point that the separated namespace stuff relies on what seems to me to
be an intrinsically fragile name-of-directory based scheme. Arguably,
the P3 version is even more fragile in this respect.

Worse, the (limited) commentary on the web suggests that the core
Python devs not only got this "wrong" in Python2, there is an
awareness that the Python3 solution has its own issues. That's enough
to scare me away from building a deep dependency on it!

My usage of the for the KF5 namespace level is based on the low
probability of a collision on that name, and the only reason I
mentioned Python2 is that I chose the scheme that supports both P2 and
P3. If you are right, and P2 is never an issue, it should be fine. I
do however prefer to maintain compatibility until the burden of doing
so presents an insurmountable issue.

>> - cppyy gives us classes. (Actually, in conversation with Wim, CC'd
>> here, it turns out that the choice of using classes is not arbitrary,
>> but we were pondering simple modules at that point, for other
>> reasons).
>
>
> I’d be interested in why. Usually using classes as namespaces is only done
> for reasons of cuteness (callable namespaces, namespaces usable as context
> managers, …) or so.

Feel free to discuss with Wim (it has something to do with
serialisation of classes and there are some interesting grottos to do
with __module__ naming to consider too). I don't propose to follow
this up. FWIW, this came up because of some substantive issues to do
with __module__, and the approach I now have in hand, based on
discussions with Wim, will be largely decoupled from this topic, and
so the way is open for anybody who does care to take this forward.

> Even in this case, it’s possible to replace the module’s class with a module
> subclass that has the necessary capabilities (modules are objects that have
> a class, too)

Yes, I'm well aware of some of those possibilities as I've tried
several of them :-).

>> Bottom line: I'm not keen on using Python namespace modules here for
>> the reasons listed.
>
>
> I’m not entirely convinced. We should only do a final decision once we know
> if either your suspicions of multiple levels not working turn out to be
> true, or the reason for using classes is important.

There is no need for a "final decision" from me. I would suggest that
the first question for anybody that cares is to assess the scope of
the issue. Unfortunately, i have other more fundamental issues to fry.

>> That's likey to be a bad idea because of the potential impact on arbitrary
>> round trips between C++ and Python (remember that everything is based on
>> named-based lookups). In addition, we already have problems like "gpgme++",
>> and the use of capitalisation to separate legacy and forwarding headers in
>> KDE: further case-based confusion is the last thing that is needed!
>
>
> I see, makes sense. I guess the allcaps example here is not very common
> anyway, and most namespaces will be UpperCamelCase like Qt’s, right?

Ack. So far (and based on my experiences with the SIP work), I started
with something very simple (KDCRAW), and moved onto one of the worst
cases (Akonadi). I won't be in a position to gain a wider perspective
until these two are working solidly.

Best regards, Shaheed

>> Thanks for the review/remarks, Shaheed
>
>
> NP!
> Best, Philipp

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

Philipp,

On 3 November 2017 at 12:52, Philipp A.  wrote:
> Hi Shaheed,
>
> Thank you so much for all your work!
>
>> a framework-by-framework integration of the binding generation logic (as
>> previously pioneered by Steve) probably cannot work in general because there
>> are cases where multiple frameworks contribute to to the same C++ namespace
>> […]
>>
>> The problem is that the Python implementation of these namespaces is a
>> class, and so treating these frameworks […] as separate would result in
>> multiple colliding Python class definitions.
>
>
> Am I missing something? Namespaces should be Python modules, not classes. If
> we can do represent them this way, the problem is solveable:
> https://packaging.python.org/guides/packaging-namespace-packages/

First, I am using this technique at the KF5 level (the pkg-util
version). My concerns about using it to address the namespace problem
are that:

- my overall understanding of this technique is that it may end up
being fragile, especially given the difference between P2 and P3.
- using it further down might not work as expected especially if there
are "accidental" collisions in the directory/namespace names.
- it is also not clear if the technique can be used multiple times.
- cppyy gives us classes. (Actually, in conversation with Wim, CC'd
here, it turns out that the choice of using classes is not arbitrary,
but we were pondering simple modules at that point, for other
reasons).

Bottom line: I'm not keen on using Python namespace modules here for
the reasons listed.

>> The original TODOs and bugs have been resolved, and there is the
>> beginnings of support for packaging frameworks under a Python namespace as
>> in "KF5.KDCRAW".
>
>
> Once they’re modules, we should probably respect that Python modules are by
> convention lowercase. It would be best if we named them kf5.kdcraw and so
> on.

That's likey to be a bad idea because of the potential impact on
arbitrary round trips between C++ and Python (remember that everything
is based on named-based lookups). In addition, we already have
problems like "gpgme++", and the use of capitalisation to separate
legacy and forwarding headers in KDE: further case-based confusion is
the last thing that is needed!

Thanks for the review/remarks, Shaheed

> Thank you again,
> Best, Philipp

Re: Python bindings using cppyy (was: An update on Python bindings)

[Luca Beltrame]

Il giorno Thu, 2 Nov 2017 23:53:00 +
Shaheed Haque  ha scritto:

Hello Shaheed,

> Firstly, we have existence proofs in the form of PyQt and the PyKDE4

PyQt doesn't really fit the model. It's developed by one person, and
with no source control access to anyone but the maintainer. PyKDE4 had
serious problems with maintenance (stuff got broken for months, at
times), although that was in pre CI times.

> Also, I wonder how easy that will be given the need to keep source
> compatibility? I've hardly done a comprehensive survey, but if the
> Akonadi example is anything to go by, I don't think there is a short

I would suggest to avoid anything that's not part of Frameworks for
now. The policies for ABI and API compatibility make things easier.
The rest can come later, once the finer issues have been dealt with.

I'd rather have the bindings in the frameworks, to ensure that if
something breaks, it can be fixed instead of bitrotting.

-- 
Luca Beltrame - KDE Forums team
GPG key ID: A29D259B


pgpd6TWpxwWDR.pgp
Description: Firma digitale OpenPGP

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

Albert,

On 2 November 2017 at 21:43, Albert Astals Cid  wrote:
> El dijous, 2 de novembre de 2017, a les 18:22:38 CET, Shaheed Haque va
> escriure:
>> A progress update...
>>
>> On 24 October 2017 at 13:05, Shaheed Haque  wrote:
>> > Hi all,
>> >
>> > I have a preliminary version of the Cppyy bindings generator CMake
>> >
>> > support available here:
>> > https://bitbucket.org/wlav/cppyy-backend/pull-requests/6/an-interim-ex
>> > perimental-version-of-a/diff>
>> > There are some TODOs yet to be addressed,
>>
>> The original TODOs and bugs have been resolved, and there is the
>> beginnings of support for packaging frameworks under a Python
>> namespace as in "KF5.KDCRAW". Also, as a significant datapoint, I'm
>> close [1] to being able to generate a *complete* set of bindings for
>> all of Akonadi driven from CMake with just 2-3 lines of custom logic.
>> This contrasts with the 549 SLOC of customisation needed to produce a
>> substantially slash-and-burned subset of Akonadi [2] with the
>> SIP-based approach.
>>
>> > but I would appreciate
>> > feedback on how easy it would be to integrate this with KDE's
>> > buildsystem, especially for the frameworks. I'm a CMake noob, but the
>> > basic idea I have is that the packager of some_framework might do
>> > something like this:
>> >
>> > find_package(cppyy)
>> > CPPYY_ADD_BINDINGS(
>> >
>> > ...
>> > LINK_LIBRARIES some_framework_LIBRARIES
>> > H_DIR some_framework_INCLUDE_DIRS
>> > H_FILES )
>>
>> In the course of working through the "KF5" namespace implementation,
>> it has become apparent to me that a framework-by-framework integration
>> of the binding generation logic (as previously pioneered by Steve)
>> probably cannot work in general because there are cases where multiple
>> frameworks contribute to to the same C++ namespace, for example:
>>
>> $ grep -r '^namespace Akonadi' /usr/include/KF5/Akonadi*
>> /usr/include/KF5/AkonadiAgentBase/resourcesettings.h:namespace Akonadi
>> ..
>> /usr/include/KF5/AkonadiCore/agentfilterproxymodel.h:namespace Akonadi
>> ..
>> /usr/include/KF5/AkonadiSearch/Debug/akonadisearchdebugsearchpathcombobox.h:
>> namespace Akonadi
>> ..
>> /usr/include/KF5/AkonadiWidgets/agenttypedialog.h:namespace Akonadi
>> ..
>> /usr/include/KF5/AkonadiXml/xmldocument.h:namespace Akonadi
>> ..
>>
>> The problem is that the Python implementation of these namespaces is a
>> class, and so treating these frameworks (let's not quibble over
>> whether KF5Akonadi* are truly KF5 frameworks, the point is more
>> general) as separate would result in multiple colliding Python class
>> definitions. The only solution I can see would be to bundle all of
>> KF5Akonadi* into a single set of bindings, e.g. KF5.Akonadi, and
>> AFAICS, this can only be done out of tree from the individual
>> frameworks, say in kde-bindings.git [3].
>
> Sincerely if you go with out ot tree bindings they will always be broken
> because people working on the frameworks won't even see them, if the problem
> is that frameworks have to be self-contained, this seems like a good general
> rule to me and maybe what we should do is fix the libs?

Thanks for the input, but...

Firstly, we have existence proofs in the form of PyQt and the PyKDE4
bindings that (even with the SIP support tax), that out of tree
bindings are possible, and I am hopeful that things will be much
easier with cppyy.

Also, I wonder how easy that will be given the need to keep source
compatibility? I've hardly done a comprehensive survey, but if the
Akonadi example is anything to go by, I don't think there is a short
term prospect of seeing a solution to this problem. If there is
developer effort available from framework owners, I'd much rather see
that put to productive use (there are plenty of point issues like
#includes broken-from-my-POV and the like).

Finally, not to put to fine a point on it, I've been working on this
for the best part of two years, and I am very keen to get to something
workable. (Not least because I hope to get a job at some point...and
then intensive effort from me at least might be much trickier to offer
up).

Cheers, Shaheed

> Cheers,
>   Albert
>
>>
>> The work to date attempts to maintain a clean separation such that all
>> C++ builds are done from CMake, and all Python builds are done using
>> setuptools/pip.
>>
>> Apart from working through bugs [1], the remaining work items I can
>>
>> think of are, as before:
>> >> - Need to look into the exact usage of Qt-specifics: signals/slots and
>> >> interoperability with SIP-based PyQt
>> >> (https://root.cern.ch/root/htmldoc/guides/users-guide/PythonRuby.html#glu
>> >> e-ing-applications,
>> >> https://root.cern.ch/doc/v606_ORIG/guide/ROOTandQt.html)
>> >>
>> >> - Need to figure out how any customisations which *are* required
>> >> should be handled.
>>
>> plus:
>>
>> - I'm working with upstream on how to support discovery (e.g. via
>> autocompletion in Python3). There is some 

Re: Python bindings using cppyy (was: An update on Python bindings)

[Albert Astals Cid]

El dijous, 2 de novembre de 2017, a les 18:22:38 CET, Shaheed Haque va 
escriure:
> A progress update...
> 
> On 24 October 2017 at 13:05, Shaheed Haque  wrote:
> > Hi all,
> > 
> > I have a preliminary version of the Cppyy bindings generator CMake
> > 
> > support available here:
> > https://bitbucket.org/wlav/cppyy-backend/pull-requests/6/an-interim-ex
> > perimental-version-of-a/diff> 
> > There are some TODOs yet to be addressed,
> 
> The original TODOs and bugs have been resolved, and there is the
> beginnings of support for packaging frameworks under a Python
> namespace as in "KF5.KDCRAW". Also, as a significant datapoint, I'm
> close [1] to being able to generate a *complete* set of bindings for
> all of Akonadi driven from CMake with just 2-3 lines of custom logic.
> This contrasts with the 549 SLOC of customisation needed to produce a
> substantially slash-and-burned subset of Akonadi [2] with the
> SIP-based approach.
> 
> > but I would appreciate
> > feedback on how easy it would be to integrate this with KDE's
> > buildsystem, especially for the frameworks. I'm a CMake noob, but the
> > basic idea I have is that the packager of some_framework might do
> > something like this:
> > 
> > find_package(cppyy)
> > CPPYY_ADD_BINDINGS(
> > 
> > ...
> > LINK_LIBRARIES some_framework_LIBRARIES
> > H_DIR some_framework_INCLUDE_DIRS
> > H_FILES )
> 
> In the course of working through the "KF5" namespace implementation,
> it has become apparent to me that a framework-by-framework integration
> of the binding generation logic (as previously pioneered by Steve)
> probably cannot work in general because there are cases where multiple
> frameworks contribute to to the same C++ namespace, for example:
> 
> $ grep -r '^namespace Akonadi' /usr/include/KF5/Akonadi*
> /usr/include/KF5/AkonadiAgentBase/resourcesettings.h:namespace Akonadi
> ..
> /usr/include/KF5/AkonadiCore/agentfilterproxymodel.h:namespace Akonadi
> ..
> /usr/include/KF5/AkonadiSearch/Debug/akonadisearchdebugsearchpathcombobox.h:
> namespace Akonadi
> ..
> /usr/include/KF5/AkonadiWidgets/agenttypedialog.h:namespace Akonadi
> ..
> /usr/include/KF5/AkonadiXml/xmldocument.h:namespace Akonadi
> ..
> 
> The problem is that the Python implementation of these namespaces is a
> class, and so treating these frameworks (let's not quibble over
> whether KF5Akonadi* are truly KF5 frameworks, the point is more
> general) as separate would result in multiple colliding Python class
> definitions. The only solution I can see would be to bundle all of
> KF5Akonadi* into a single set of bindings, e.g. KF5.Akonadi, and
> AFAICS, this can only be done out of tree from the individual
> frameworks, say in kde-bindings.git [3].

Sincerely if you go with out ot tree bindings they will always be broken 
because people working on the frameworks won't even see them, if the problem 
is that frameworks have to be self-contained, this seems like a good general 
rule to me and maybe what we should do is fix the libs?

Cheers,
  Albert

> 
> The work to date attempts to maintain a clean separation such that all
> C++ builds are done from CMake, and all Python builds are done using
> setuptools/pip.
> 
> Apart from working through bugs [1], the remaining work items I can
> 
> think of are, as before:
> >> - Need to look into the exact usage of Qt-specifics: signals/slots and
> >> interoperability with SIP-based PyQt
> >> (https://root.cern.ch/root/htmldoc/guides/users-guide/PythonRuby.html#glu
> >> e-ing-applications,
> >> https://root.cern.ch/doc/v606_ORIG/guide/ROOTandQt.html)
> >> 
> >> - Need to figure out how any customisations which *are* required
> >> should be handled.
> 
> plus:
> 
> - I'm working with upstream on how to support discovery (e.g. via
> autocompletion in Python3). There is some POC-level hackery in git as
> above, but there is work ongoing with upstream to find a robust
> solution.
> 
> - Flesh out how to make one set of bindings depend on another (e.g.
> tier 2 framework bindings might depend on tier 1 bindings, or maybe it
> is better to avoid PyQt and just produce cppyy-based bindings for Qt
> and depend on those).
> 
> As always, comments/ideas/suggestions are welcome.
> 
> Thanks, Shaheed
> 
> [1] There is a bug with namespaced externs being worked on with upstream.
> 
> [2]
> https://github.com/ShaheedHaque/extra-cmake-modules/blob/shaheed_master/fin
> d-modules/module_generation/PyKF5/Akonadi.py
> 
> [3] I attach an example CMakeLists.txt which shows now this can be
> driven from CMake for the case of KF5Akonadi*...the implementation is
> intended to serve as the basis for a generic solution usable across
> KF5 at least.
> 
> > On 16 October 2017 at 16:16, Shaheed Haque  wrote:
> >> As promised, here is an interim update on the investigation into the
> >> use of cppyy-based bindings for KF5 (and more...) instead of SIP-based
> >> bindings.
> >> 
> >> The first thing is that the 

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

A progress update...

On 24 October 2017 at 13:05, Shaheed Haque  wrote:
> Hi all,
>
> I have a preliminary version of the Cppyy bindings generator CMake
> support available here:
>
> 
> https://bitbucket.org/wlav/cppyy-backend/pull-requests/6/an-interim-experimental-version-of-a/diff
>
> There are some TODOs yet to be addressed,

The original TODOs and bugs have been resolved, and there is the
beginnings of support for packaging frameworks under a Python
namespace as in "KF5.KDCRAW". Also, as a significant datapoint, I'm
close [1] to being able to generate a *complete* set of bindings for
all of Akonadi driven from CMake with just 2-3 lines of custom logic.
This contrasts with the 549 SLOC of customisation needed to produce a
substantially slash-and-burned subset of Akonadi [2] with the
SIP-based approach.

> but I would appreciate
> feedback on how easy it would be to integrate this with KDE's
> buildsystem, especially for the frameworks. I'm a CMake noob, but the
> basic idea I have is that the packager of some_framework might do
> something like this:
>
> find_package(cppyy)
> CPPYY_ADD_BINDINGS(
> ...
> LINK_LIBRARIES some_framework_LIBRARIES
> H_DIR some_framework_INCLUDE_DIRS
> H_FILES )

In the course of working through the "KF5" namespace implementation,
it has become apparent to me that a framework-by-framework integration
of the binding generation logic (as previously pioneered by Steve)
probably cannot work in general because there are cases where multiple
frameworks contribute to to the same C++ namespace, for example:

$ grep -r '^namespace Akonadi' /usr/include/KF5/Akonadi*
/usr/include/KF5/AkonadiAgentBase/resourcesettings.h:namespace Akonadi
...
/usr/include/KF5/AkonadiCore/agentfilterproxymodel.h:namespace Akonadi
...
/usr/include/KF5/AkonadiSearch/Debug/akonadisearchdebugsearchpathcombobox.h:namespace
Akonadi
...
/usr/include/KF5/AkonadiWidgets/agenttypedialog.h:namespace Akonadi
...
/usr/include/KF5/AkonadiXml/xmldocument.h:namespace Akonadi
...

The problem is that the Python implementation of these namespaces is a
class, and so treating these frameworks (let's not quibble over
whether KF5Akonadi* are truly KF5 frameworks, the point is more
general) as separate would result in multiple colliding Python class
definitions. The only solution I can see would be to bundle all of
KF5Akonadi* into a single set of bindings, e.g. KF5.Akonadi, and
AFAICS, this can only be done out of tree from the individual
frameworks, say in kde-bindings.git [3].

The work to date attempts to maintain a clean separation such that all
C++ builds are done from CMake, and all Python builds are done using
setuptools/pip.

Apart from working through bugs [1], the remaining work items I can
think of are, as before:

>> - Need to look into the exact usage of Qt-specifics: signals/slots and
>> interoperability with SIP-based PyQt
>> (https://root.cern.ch/root/htmldoc/guides/users-guide/PythonRuby.html#glue-ing-applications,
>> https://root.cern.ch/doc/v606_ORIG/guide/ROOTandQt.html)
>>
>> - Need to figure out how any customisations which *are* required
>> should be handled.

plus:

- I'm working with upstream on how to support discovery (e.g. via
autocompletion in Python3). There is some POC-level hackery in git as
above, but there is work ongoing with upstream to find a robust
solution.

- Flesh out how to make one set of bindings depend on another (e.g.
tier 2 framework bindings might depend on tier 1 bindings, or maybe it
is better to avoid PyQt and just produce cppyy-based bindings for Qt
and depend on those).

As always, comments/ideas/suggestions are welcome.

Thanks, Shaheed

[1] There is a bug with namespaced externs being worked on with upstream.

[2] 
https://github.com/ShaheedHaque/extra-cmake-modules/blob/shaheed_master/find-modules/module_generation/PyKF5/Akonadi.py

[3] I attach an example CMakeLists.txt which shows now this can be
driven from CMake for the case of KF5Akonadi*...the implementation is
intended to serve as the basis for a generic solution usable across
KF5 at least.

> On 16 October 2017 at 16:16, Shaheed Haque  wrote:
>> As promised, here is an interim update on the investigation into the
>> use of cppyy-based bindings for KF5 (and more...) instead of SIP-based
>> bindings.
>>
>> The first thing is that the underlying technology of cppyy,
>> cling/ROOT, has been under development at CERN for quite a while. It
>> directly reads regular C++ files (there is no intermediate format like
>> SIP).
>>
>> The bindings it generates from Python to C++ seem far more complete
>> and automatic than SIP. For example:
>>
>> - Template instantiation is done on the fly as needed.
>>
>> - Since it uses C++ directly, there is none the effort required to
>> decollide SIP's notion of forward and duplicate declarations.
>>
>> - Function overloads are cleanly handled, as are most (all?) operators.
>>
>> The net result is that so far, there is about 3 

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

Hi all,

I have a preliminary version of the Cppyy bindings generator CMake
support available here:


https://bitbucket.org/wlav/cppyy-backend/pull-requests/6/an-interim-experimental-version-of-a/diff

There are some TODOs yet to be addressed, but I would appreciate
feedback on how easy it would be to integrate this with KDE's
buildsystem, especially for the frameworks. I'm a CMake noob, but the
basic idea I have is that the packager of some_framework might do
something like this:

find_package(cppyy)
CPPYY_ADD_BINDINGS(
...
LINK_LIBRARIES some_framework_LIBRARIES
H_DIR some_framework_INCLUDE_DIRS
H_FILES )

Thanks, Shaheed

On 16 October 2017 at 16:16, Shaheed Haque  wrote:
> As promised, here is an interim update on the investigation into the
> use of cppyy-based bindings for KF5 (and more...) instead of SIP-based
> bindings.
>
> The first thing is that the underlying technology of cppyy,
> cling/ROOT, has been under development at CERN for quite a while. It
> directly reads regular C++ files (there is no intermediate format like
> SIP).
>
> The bindings it generates from Python to C++ seem far more complete
> and automatic than SIP. For example:
>
> - Template instantiation is done on the fly as needed.
>
> - Since it uses C++ directly, there is none the effort required to
> decollide SIP's notion of forward and duplicate declarations.
>
> - Function overloads are cleanly handled, as are most (all?) operators.
>
> The net result is that so far, there is about 3 days work and
> approximately [1] no "customisation" required in order to get to
> roughly where the SIP based bindings were after 18 months. Without the
> need for customisations on a mass scale, I suspect that we might get
> away without anything like the tooling I had to create to SIP, and
> just integrate with CMake
> (https://root.cern.ch/how/integrate-root-my-project-cmake).
>
> This all sounds pretty amazing, right? Well, there are a few caveats...
>
> - The packaging is pretty new, and is evolving pretty rapidly. We
> are/will be an early adopter (https://bitbucket.org/wlav/cppyy/ and
> https://bitbucket.org/wlav/cppyy-backend). Packaging is via PyPI and
> pip/pip3.
>
> - There is a lot of documentation around for the system overall, but
> frankly, it has  been/still is a struggle to understand how the
> different parts relate to each other as some parts are obsolete, and
> other parts have yet to be built out to their intended end-state.
>
> - There are bugs [1], [2]. The upstream dev has been very responsive,
> and the overall quality approach looks sound. IIUC, the vast bulk of
> the code seems to be in daily use at CERN (and is based on LLVM).
>
> - Need to look into the exact usage of Qt-specifics: signals/slots and
> interoperability with SIP-based PyQt
> (https://root.cern.ch/root/htmldoc/guides/users-guide/PythonRuby.html#glue-ing-applications,
> https://root.cern.ch/doc/v606_ORIG/guide/ROOTandQt.html)
>
> - Need to figure out how any customisations which *are* required
> should be handled.
>
> These seem like perfectly tractable issues, and so I conclude that
> using cppyy is definitely the way to go. With luck and a bit of
> effort, I am hopeful that we can get to some REALLY
> easy-to-develop-and-maintain bindings.
>
> [1] There is a bug with the binding producing stuff for private definitions.
>
> [2] There is a bug with missing globals.
>
>
[snip]

Re: Python bindings using cppyy (was: An update on Python bindings)

[Kevin Kofler]

Shaheed Haque wrote:
> As promised, here is an interim update on the investigation into the
> use of cppyy-based bindings for KF5 (and more...) instead of SIP-based
> bindings.
> 
> The first thing is that the underlying technology of cppyy,
> cling/ROOT, has been under development at CERN for quite a while. It
> directly reads regular C++ files (there is no intermediate format like
> SIP).

Unfortunately, if I understand correctly, it does that at runtime, lazily 
generating bindings only when needed the first time. They explicitly 
advertise that as a feature. But, in addition to the performance concerns I 
have with that, it turns deployment into a nightmare. It means we cannot 
just ship precompiles bindings in the distros, but our users will have to 
install the whole LLVM stack, and a special forked version at that (so it 
cannot even be shared with other LLVM users such as 
Mesa/Gallium3D/llvmpipe).

ROOT is a huge bloated framework that is not exactly reputed for its 
simplicitly to package.

Kevin Kofler