Hi Jean,
maybe I'm missing the point, but I think that there is no problem to
have 1MB, 2MB or 2.5MB javascript, the problem why qooxdoo feels slow
is that it creates too much DOM elements and this is much worse for
browser than few kilobytes of javascript.
Instead of improving generator I think that decreasing number of DOM
elements needed for an application is better choice. Maybe using CSS3,
joining more DIVs into one DIV, etc. Qooxdoo 0.8 had to be faster than
0.7, but this didn't happened.
Best regards
Petr Kobalicek
On Mon, Aug 30, 2010 at 1:23 PM, Jean-Baptiste BRIAUD -- Novlog
<j-b.bri...@novlog.com> wrote:
> Hi Thomas,
> Welcome back ! Hope you had great holidays ... it look like finished ;-)
> On 30 août 2010, at 10:03, thron7 wrote:
>
> Hi,
>
> [CUT]
>
> Python subprocesses
>
> As for creating recursive python invocations: It is really
> disadvantageous to use the 'generate.py' script for testing and
> exploration, as it is just a local proxy for the *real* generator,
> tool/bin/generator.py. Any investigation should be done on this program
> instead, and this is what I will do. This gets you rid of one subprocess
> indirection. From there on, there are *no more* subprocesses being
> forked out, unless of course you explicitly say so in your config.json,
> by using the 'shell' key.
>
>
> I replaced CMD_PYTHON in all places so I have :
> CMD_PYTHON = sys.executable
> Apparently this ensure a subprocess is launched the same way the parent.
> This ensured pypy is used instead of Python, even is a child process.
> Seeing a variable CMD_PYTHON gave me the feeling Python could be called from
> Python creating subprocesses.
> I'm not sure why using project local generate.py would had cause JIT issue
> because it then call the "real" generate.py.
> Correct me if I'm wrong, this should launch pypy because of the replacement
> of CMD_PŸTHON.
> So, we have one parent pypy process would have called only one pypy process
> (the "real" generate.py) and so the JIT should have worked for the child
> process.
> Only one depth call should not have huge impact on JIT performance, only
> several call or recursive one might have.
> I don't have any clue why it is slower using pypy.
>
> Performance
>
> Now to the real topic, generator performance. For one thing, it would be
> important to know what your performance goal for the generator would be.
> What is your time line? You mention a run time of 1:58min for standard
> Python (I presume this is with a filled cache). What would you consider
> acceptable performance: 1:00, 0:30, 0:10, 0:01?!
>
>
> * On production : I want all the possible optimization done by the python
> tool chain in the less possible time even if I'm ready to wait because of
> the benefit of the optimizations.
> This is not where the pain is because we are not delivering to production
> very often, so when it happen, we can wait.
> Also, generate.py time can change depending the on application to build.
> So, I don't have acceptable performance time to give and for some our our
> app, it take longer than 1.58.
> => The idea is to improve what could be : multi-CPU, pypy, reducing time by
> an order of magnitude would be great, gaining few seconds is not really
> interesting.
> * On dev machines. Performances are criticals on Dev environment.
> That's where the pain is currently.
> We launch very often and wanted to see the result very quickly.
> Due to some reasons and maybe our way to use qooxdoo automatically,
> we are creating a new application each time (even for production).
> That's one reason why generate source doesn't save time for us.
> We're using generate source when we need to debug the js source code
> produced after generate (generate build are not readable).
> We deploy locally on a developer machine and we don't need all the
> optimizations provided by python tool chain.
> I need a deploy version here not a file:// because we are using Java as a
> backend.
> => So, the idea is to use a precompile qooxdoo version to avoid not useful
> calculation. Of course, here too multi-CPU and pypy improvements could
> apply.
>
> To improve overall performance outside of JIT compilation, your best bet
> is (as mentioned before) to use multiple processes. As discussed in
> another thread you can run multiple generator invocations in parallel,
> using the same cache directory, which will allow them to benefit from
> commonly used classes (as e.g. the framework classes).
>
> *Within* a single generator run, there is an experimental feature
> already implemented (but undocumented) that lets you distribute the
> compilation of classes to multiple processes. You just have to add a
> specific config key. But it does not work properly under Windows. And of
> course you have to have multiple cores or processors, to see any benefit
> from using it :-). If you are building on non-Windows platforms you are
> invited to try that out. Let me know and I'll provide details.
>
>
> Very useful for us because we are using Mac and Linux (Ubuntu server and
> desktop), all 64 bits in case it matters.
> Sometimes we're using Windows but it still very interesting to reduce on the
> mainly used platform.
>
> Pre-built JS file with all qooxdoo classes
>
> I presume now that in production you are using a single cache for all
> the build runs, which also stays intact (is not deleted) across those
> runs.
>
> Development environment is the main problem. That's why I though to a
> precompiled JS with all qooxdoo classes.
> Only the "merge" step between precompiled file and the project's classes
> should be done.
> We might also include all project's classes so there would be no need for
> any calculation, only IO to merge files.
>
> If the cache is fairly saturated (i.e. most classes for an
> application are already in the cache), I doubt that you will see much
> performance gain between using the cache and using a pre-built qx JS
> file. Once the framework classes are in the cache, the effort of pulling
> them together into deployable script files is marginal. But that also
> means that a well saturated cache is crucial. As already mentioned by
> others having the cache and other files on a ram disk does not provide
> significant speedup on non-Windows platforms, which conforms to my own
> measurements. The bottleneck is CPU, not disk I/O.
>
>
> I'm not sure how we benefit from the cache since we are started from
> application create each time.
> I guess it depends on what key is used to find information in the cache.
> If it is based on application name, it is OK, but if there is some unique ID
> by application created, it won't work for us.
> We never clean the cache.
> I was talking about using the generator in parallel, this is because in our
> test env (using Selenium) we are testing our software that itself build
> qooxdoo application.
> To speed up the tests, some of them are running in parallel and so the qx
> generator.
> Apparently, it is safe to use the same cache for several generator.py that
> run in parallel. We didn't experienced issue with that and it works fine to
> compile several apps in //.
>
>
> T.
>
>
>
> On 08/26/2010 05:30 PM, Jean-Baptiste BRIAUD -- Novlog wrote:
>
> OK, now I can confirm the modified version of generate.py on my project
>
> run using pypy without error.
>
> Currently (still with the probable numerous call to the python
>
> interpreter) :
>
> with python : [generate-build: 1:58.310 sec]
>
> with pypy : [generate-build: 18:28.473 sec]
>
> Resolving dependencies is the step that look like to take most of the time.
>
> Anyway, it is definitively working using pypy.
>
> On 26 août 2010, at 15:58, Jean-Baptiste BRIAUD -- Novlog wrote:
>
> Hi Andreas,
>
> On 26 août 2010, at 15:31, Andreas Ecker wrote:
>
> Hi JBB,
>
> so for pypy you had to download 2.5 GB of code and it took you more than
>
> 1 hour to compile it on a recent i7 machine!? Are you still complaining
>
> about the qooxdoo generator speed, then?
>
>
> Yes, there is worst situation than qooxdoo build, I was really shoked
>
> : more than 1 hour :-)
>
> Hopefully, I won't need recompile pypy as often as I launch generate
>
> build.
>
> Just kidding ;-). Give it another try with using the "verbose" option,
>
> i.e. call "pypy generate.py -v build". That should give you a more
>
> detailed error message. Would be great you kept trying with the help of
>
> the pypy people.
>
>
> I did, finally a pypy guy asked my to remove the reload(sys) line.
>
> It failed to remove only that line so I finally removed the following
>
> block :
>
> reload(sys)
>
> sys.setdefaultencoding('utf-8')
>
> sys.setrecursionlimit(1500)
>
> now it start working ... but it take longer than without pypy !!!
>
> Finally it had failed on this error :
>
> 'ascii' codec can't encode character u'\u2019' in position 42: ordinal
>
> not in range(128)
>
> pypy acknowledge a bug for the reload(sys)
>
> https://codespeak.net/issue/pypy-dev/issue557
>
> I'll try this "hack" suggested by arigato that was on the pypy IRC
>
> (#pypy on freenode).
>
> He is really helpful.
>
> arigato:try to preserve sys.stdout around it, i.e.
>
> arigato:a = sys.stdout; reload(sys); sys.stdout = a
>
> arigato:it may help (but of course it's still a pypy bug to solve)
>
> Since the hello world now seems to work fine with pypy (which to my
>
> knowledge previously and without the very recent fixes to pypy never
>
> worked), maybe the problems with your project are also due to the
>
> qooxdoo default encoding UTF-8 (rather than ascii), particularly if you
>
> have accented chars for French translations in your files. Just a
>
> guess...
>
>
> good guess, there is that block that failed in pypy but removing it
>
> make it failing because of some non ascii characters ...
>
> No idea what to do.
>
>
> Anyway, we already know that "as is" it will take longer than without
>
> pypy.
>
> Apparently it might be due to the numerous call to python itself.
>
> I had to replace "everywhere"
>
> CMD_PYTHON = 'python'
>
> by
>
> CMD_PYTHON = sys.executable
>
> The problem is not just to replace it but to understand why python
>
> interpreter is called apparently so often.
>
> I have to warn I don't speak py. It may not continue to be true ;-)
>
> I'm not sure if python is really called from generator.py except
>
> that CMD_PYTHON = 'python' look like it is called.
>
> The fact is that pypy rely on a JIT and relaunching it loose all the
>
> JIT benefit.
>
> Apparently, as a general purpose rule in py, it is not recommended to
>
> call python from python to execute a file.
>
> There is ways that I don't know to call python file without calling
>
> the interpreter each time.
>
> So, even without pypy context it might be a good idea to avoid
>
> launching python process between 2 py files.
>
> Hope this will help improving qx !
>
> Please note that Thomas has been on vacation for the last two weeks and
>
> is to return not before Monday.
>
> Bye,
>
> Andreas
>
>
> On Thu, 2010-08-26 at 14:45 +0200, Jean-Baptiste BRIAUD -- Novlog wrote:
>
> Finally, I didn't gave up ... :-)
>
> I manage to get a proper C dev env in my Mac (by simply downloading
>
> 2.5 Go of xcode).
>
> Yes, 2.5 Go !
>
> Anyway, I was then able to recompile pypy from the trunk.
>
> I try to run generate.py using pypy.
>
> It had failed.
>
> I have no idea what was wring in the pypy code but the pypy guys
>
> downloaded qx sdk to have the the helloword application.
>
> They correct what had failed in pypy and I recompiled it (1hour each
>
> time, yes 1 hours of gcc ...)
>
> I created a qooxdoo hello world application and manage to launch a
>
> generate build using pypy rather than python.
>
> I then try to run the same way on my project.
>
> I apply the CMD_PYTHON = sys.executable modification on all my local
>
> generate.py and use pypy rather than python.
>
> It failed on a rather cryptic C error : [Errno 9] Bad file
>
> descriptor: '<fdopen>'
>
> There must be some new python code section called in the generate.py
>
> that is called for my project that is not called for the simple
>
> helloworld.
>
> Now, I give up cause I don't speak python and I used to speak C long
>
> time ago ...
>
> I think this experiment should encourage to try the pypy way to
>
> speedup all the python tool chain.
>
> Apparently this look like doable but I don't have the knowledge to
>
> help more and reach the goal : pypy generate.py build.
>
>
> On 25 août 2010, at 13:49, Jean-Baptiste BRIAUD -- Novlog wrote:
>
> OK, pypy look like a very good thing to try.
>
> I had to replace
>
> CMD_PYTHON = 'python'
>
> by
>
> CMD_PYTHON = sys.executable
>
> in the generate.py of the application folder to ensure pypy launch
>
> the "real" generator.
>
> Unfortunately, generate.py doesn't work with pypy.
>
> It blow up in error.
>
> I had an very interesting IRC session with the pypy guys.
>
> I end up trying to build pypy from trunk.
>
> There is version for several OS prebuilt but they asked me to try
>
> the trunk because they already know that the latest prebuilt
>
> version will not work with generate.py they kindly inspected.
>
> Anyway, after hours trying to use gcc on my Mac I gave up.
>
> conclusions :
>
> 1. pypy is a real way to improve Python runtime but generate.py
>
> apparently would need some hacks. Qooxdoo can you do something here ?
>
> 2. The only thing I can think about now, is the precompiled version
>
> of qooxdoo but I still have that merge question, see the other thread.
>
>
> On 24 août 2010, at 23:35, <f...@mail.dnttm.ro
>
> <mailto:f...@mail.dnttm.ro>> <f...@mail.dnttm.ro
>
> <mailto:f...@mail.dnttm.ro>> wrote:
>
> Hi.
>
> Maybe http://codespeak.net/pypy/dist/pypy/doc/ is a more viable
>
> direction of research?
>
> As far as I could understand psyco is not developed anymore
>
> because the effort goes now into developing pypy - an optimized
>
> python interpreter, which does the optimizations psyco did on all
>
> interpreted code by default.
>
> br,
>
> flj
>
>
>
>
> ------------------------------------------------------------------------------
>
> Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
>
> Be part of this innovative community and reach millions of netbook users
>
> worldwide. Take advantage of special opportunities to increase revenue
>
> and
>
> speed time-to-market. Join now, and jumpstart your future.
>
> http://p.sf.net/sfu/intel-atom-d2d_______________________________________________
>
> qooxdoo-devel mailing list
>
> qooxdoo-devel@lists.sourceforge.net
>
> https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
>
>
>
> ------------------------------------------------------------------------------
>
> Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
>
> Be part of this innovative community and reach millions of netbook users
>
> worldwide. Take advantage of special opportunities to increase revenue and
>
> speed time-to-market. Join now, and jumpstart your future.
>
> http://p.sf.net/sfu/intel-atom-d2d
>
>
>
> _______________________________________________
>
> qooxdoo-devel mailing list
>
> qooxdoo-devel@lists.sourceforge.net
>
> https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
>
> ------------------------------------------------------------------------------
> Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
> Be part of this innovative community and reach millions of netbook users
> worldwide. Take advantage of special opportunities to increase revenue and
> speed time-to-market. Join now, and jumpstart your future.
> http://p.sf.net/sfu/intel-atom-d2d
> _______________________________________________
> qooxdoo-devel mailing list
> qooxdoo-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
>
>
> ------------------------------------------------------------------------------
> Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
> Be part of this innovative community and reach millions of netbook users
> worldwide. Take advantage of special opportunities to increase revenue and
> speed time-to-market. Join now, and jumpstart your future.
> http://p.sf.net/sfu/intel-atom-d2d
> _______________________________________________
> qooxdoo-devel mailing list
> qooxdoo-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
>
>
------------------------------------------------------------------------------
Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
Be part of this innovative community and reach millions of netbook users
worldwide. Take advantage of special opportunities to increase revenue and
speed time-to-market. Join now, and jumpstart your future.
http://p.sf.net/sfu/intel-atom-d2d
_______________________________________________
qooxdoo-devel mailing list
qooxdoo-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
