On Thursday 01 October 2009 06:40:13 pm Matti Airas wrote: > Hi Richard, > > I can give my points on the topic. > > You're very much right on the current PySide issues with Boost.Python. > The size of the bindings is quite horrible and although we did quite a > bit of optimizations to get them down, we've hit a plateau there with no > major improvement in sight along that route. Hindsight is always easy, > and you're right that we should've drawn the conclusions much earlier, > but at the time the route we took made perfect sense. However, we've > learned much from the experience and produced a lot of code which is > perfectly useful and of good quality (the API extraction and the > typesystem parts) even if we change the generator implementation. Yes, my first two language bindings projects were complete failures, but I learned a lot. I think wrapping a big C++ library like Qt is much harder than you first think it is.
> You're also right that we should've published the project at a much > earlier stage. However, we didn't deem PySide quite ready much earlier, > and in any case publishing open source projects take often surprisingly > much time in a corporate environment. The damage for Arno Rehn's > duplicate efforts has been done, and that's something I personally > regret. On the other hand, I don't think we should've delayed the > publication due to our Boost.Python generator size issues, as we much > prefer being able to work in the open and the bindings are already quite > usable on the desktop for Qt itself, despite of their size. Roberto's Raggi's parsers are being used in a lot of projects - PySide, QtScript, QtJambi, Smoke, KDevelop and Qt Creator, but I think they are all slightly different. Then the are duplicated code generators in some of these projects like PySide and Smoke too. So the parser has been a brilliant success, but has bred like rabbits it seems. > The Shiboken project was started as a private experiment by some of the > OpenBossa team members and once its potential was understood, it was > developed further to see if it could solve our size problems, and once > the initial figures were acquired, we decided to move full steam ahead > with it. The skeleton bindings generated by it are currently a bit > larger than PyQt, but it's about one-third of the size Boost.Python > PySide was at the respective point, so I believe we'll be able to match > PyQt there. (Although we're not really here to compete, even if it at > times would seem like it.) And with the existing binding generator unit > tests and stuff available, we really hope to make quite rapid progress > on getting all of Qt covered. > > Having followed the discussion after the project publication, I don't > have a slightest doubt that we couldn't achieve at least the same size, > and probably smaller bindings using Smoke. PySide's runtime performance > is already with Boost.Python very good, and there we don't have > comparative data from Smoke. However, my own intuition tells me that > wouldn't be much of an issue either. Obviously, this begs the question > why haven't we already switched to Smoke, and here the unfortunate > answer is that the reality has bitten us. We (Nokia) need to have > something "good enough" to develop on ready in time, and Shiboken will > fit the task even better than sufficiently. Changing the implementation > once again would incur an unacceptable delay for us. Hence, we're > pushing the Nokia-funded PySide development efforts in that direction > for the time being. I suppose the difference with Smoke and Shiboken would be that I would help out with the Smoke python one, and could also get the kde bindings community to help add any enhancements. I am working on the QtScript bindings, and re- writing the overloaded method resolution code in C++ (it is in Ruby in QtRuby), and that same code could be used in other projects like QtRuby or PySide. If you use Shiboken, then you would be working alone I would guess. The QtScript bindings didn't exist 4 weeks ago, and now we have 3 developers working on it with method calls and virtual method callbacks nearly there with us justing working the odd day at weekends. > However, this being said, in a slightly longer term (maybe after half a > year or so - too early to make any commitments), we definitely see Smoke > a valid future path for PySide, as stubborn duplication of effort for no > reason is just plain dumb, after all. Having tuned the generator > architecture modular to facilitate the current transition, it should > only get easier to change engines mid-flight once again in the future. Yes, but you don't need to generate any code with smoke of course, you just need to work on the dynamic runtime. > :-) So, if the technical merits support it, the long term transition > > plan could be: Boost.Python -> CPython (via Shiboken) -> Smoke. > > We really, genuinely are interested in collaborating with you guys, but > obviously the real-world obligations are limiting our latitude in the > short term. Maybe we should first begin investigating merging of Arno > Rehn's and our API extractor efforts, and then later on experiment with > Smoke bindings generation? Yes, we'll have to ask Arno as it's his baby that he is probably very proud of, but it seems a good idea. We really could use the documentation extraction and conversion for multiple languages if Nokia were happy with the copyright situation on that it would be really useful. > I'd also like to note that the text above was written more or less with > my Nokia goggles on. We're really serious on making PySide governance > model totally open, so that once the project really takes off, the > project direction won't be decided according to Nokia's whims but on > technical merits only. We're not quite there yet (as the project hasn't > really taken off yet, either), but we definitely won't want to close any > doors, especially at this early stage. If you're interested in > experimenting with PySide Smoke generation already, we'd be more than > happy to create a pyside-smoke git repo and any other required > facilities for that instead of you having to fork the project or worse > yet, start a yet another one. :-) Such work would be of great help for > PySide in the near future. Yes, if you didn't mind I could add a smoke branch to PySide and just fool around with no obligation to release, I think that would be nice. I actually don't know if smoke is a good idea, and whether it has performance problems that can't be solved. But only by having some sort of python implementation can we find out. And it is a really good opportunity for me to learn python properly which I never got round to doing in the past. > Whew, that was one long mail... Yes, ok we may have started communicating a few months late, but have made up for it now. > > ma. > > ext Richard Dale wrote: > > Thanks to the release of PySide last month I have put a lot of thought > > into Python bindings, started a QtScript bindings project, and have even > > become something of a Python fan! I thought I'd write a brain dump of my > > thoughts on using the Smoke libraries for a Python binding. > > > > Smoke was originally designed by Ashley Winters and the PerlQt team in > > 2002. Since then it has been used for QtRuby, Qyoto C#, PHP, Common Lisp > > and PerlQt4 bindings. The idea is very simple and I would call it a 'moc > > on steriods' as it works just like slots and signals are implemented in > > Qt, but for the entire library rather than only some methods, and has > > features like virtual method override callback handling, and caters for > > multiple inheritance, which the moc lacks. > > > > I am currently working on a project to implement a Smoke based binding > > for QtScript with Ian Monroe of the Amarok team. There is a an existing > > QtScript bindings project, but the Amarok guys had found the libs were > > too large, and start up time was too slow. The Smoke library to wrap for > > QtCore, QtGui, QtNetworking, QtSql, QtSvg, QtXml and QtOpenGL is only 4.3 > > Mb for the 613 classes it wraps. The existing QtScript bindings > > initialize all the classes, and all their methods at startup which is > > slow, and it is about 16.3Mb for the Qt libraries. > > > > Since the release of PySide I have studied the Python C api and looked at > > the code of several Python bindings projects, in order to get an idea of > > what would be involved. > > > > I started looking at the Boost::Python code generated for the current > > version of PySide, and it certainly is very human readable, and I > > especially how operator methods are defined. But I think it was designed > > for relatively small projects where you might have 20-30 C++ classes, and > > you write the Boost::Python code at the same time as you are writing the > > C++ classes, using Python for prototyping. However, I don't think it was > > intended to be machine generated, and used on the scale 600+ classes > > libraries like Qt. The total size of PySide for wrapping just the Qt > > libraries, let alone any extra KDE classes or whatever, is 30Mb, and that > > makes it unusable for small devices like Maemo based ones. It is also > > very much all or nothing - it looks quite hard to customize it to use > > less memory, or add more runtime dynamism. I don't think the PySide team > > should have done a first release based on Boost::Python and I have no > > idea why they have persisted for so long using something which is so > > obviously not suitable. > > > > Next I looked at how the Python C api works by playing with the 'Noddy' > > example in the docs, and reading up on how the descriptor protocol is > > used with '__getattribute__', and also how metaclasses work. Here is a > > summary in Python of how Smoke would be used: > > > > class SmokeMeta(type): > > def __new__(cls, ...): > > # Construct new Qt C++ instances here > > > > def __getattribute__(self, name): > > # Intercept class method calls here, > > # return a callable to handle calls to > > # static C++ methods in the Smoke library > > > > class QWidget(object): > > __metaclass__ = SmokeMeta > > > > def __getattribute__(self, name): > > # Intercept instance method calls here, > > # return a callable to handle calls to > > # C++ methods in the Smoke library > > > > So an actual implementation would be the same as the code above, but > > written in C. I think the Python C api would be a good fit to use with > > Smoke. > > > > After understanding the C api better, I studied the Gnome pygobject > > project, which is what I would call a 'dynamic binding' like Smoke, which > > looks up method calls and classes at runtime, instead of them being hard > > wired into the bindings library at code generation time. It uses both > > GObject itself, and gobject-introspection libraries at runtime. I think > > it is very impressive and it uses custom versions of the tp_getattro() C > > function on the Python class structs to intercept calls to > > __getattribute__ just like I thought could be done with Smoke. The code > > is LGPL'd and so it could either be used directly, or at least you could > > get ideas from it for a dynamic Python binding. > > > > I had read about an experimental branch in PySide called 'Shiboken' that > > uses pretty much the vanilla Python C api, and so I checked it out of > > gitorious and had a look. In gitorious it didn't have any Qt classes > > wrapped, and didn't have any Qt marshalling either, and so it wasn't > > possible to tell how large it might be. I spoke with 'hugpol' on IRC and > > he told me that they had a version that wrapped QtCore on an internal git > > server, and it was about 2.2 Mb. In PyQt, QtCore is about 2Mb and in the > > Boost::Python version of PySide is it 4.4Mb. I think the Smoke version > > would be less than 1Mb. So I think with enough work, it might be possible > > to produce a Shiboken version of PySide that was about the same size or > > slightly bigger than PyQt. > > > > How much work is 'enough work' though to match PyQt? I studied the PyQt > > code last and it really is very impressive indeed. It looks exactly as > > though a Python expert has worked fulltime for over 10 years on it with > > help from the community that uses it. It starts up fast, as it loads > > methods lazily only when they are needed. In fact it uses about the same > > mill to start up as they Smoke based QtScript bindings do, and about half > > the mill of QtRuby does (I think that is because Ruby is slow, rather > > than Smoke being slow). I need to study it more to see what it does, but > > there really doesn't seem to be much to fault at all. > > > > I think the bindings generators based on the QtJambi one that PySide, > > Smoke and QtScript all use are really good, and they do match the SIP > > code generation approach parsing '.sip' files, instead of parsing the C++ > > headers directly and adding XML metadata from config files. However, > > because we didn't know about the PySide project Arno Rehn developed a > > bindings generator for Smoke in a Google Summer of Code project this > > year, which is based on the QtScript one just like the PySide team did. > > Maybe their code bases can be merged. > > > > The most important advantage of dynamic language bindings is that they > > are language independent and not Python only, and are also smaller than > > conventional approaches. I think a dynamic Python binding like the > > gobject- introspection based pygobject, or a smoke based one is > > sufficiently interesting technically and different enough from PyQt to be > > worthwhile. On the other hand, although I think Shiboken can be made to > > work, at best it would be much the same as PyQt and even that would take > > a pretty heroic effort as far as I can see. > > > > To me the only justification I can see for implementing another Python > > binding (apart from the GPL vs LGPL license issue which I personally > > don't care about much), would be to implement a Maemo based development > > environment that combined Python, Ruby and QtScript using common bindings > > libs with a lower memory footprint than other approaches. Instead of just > > doing a "Let's kill PyQt on all possible platforms", it could be "Let's > > develop a great multi- language RAD environment for Maemo". For instance, > > I can't see the KDE project switching from PyQt/PyKDE to PySide anytime > > soon, no matter what approach PySide takes. And pretending that it would > > be easy in that area or anywhere PyQt is already entrenched isn't being > > realistic IMHO. > > > > -- Richard > > > > > > _______________________________________________ > > PySide mailing list > > [email protected] > > http://lists.openbossa.org/listinfo/pyside > _______________________________________________ PySide mailing list [email protected] http://lists.openbossa.org/listinfo/pyside
