On 05/12/01 09:20 +0100, Nadim Khemir wrote:
> Hi all, since it's the first time you hear from me I'll try to give a (very)
> short intro.
Welcome!
> Nadim, 35, programs in perl, C++, C, VB, Flex, lua, ..., Interested in:
> languages, editors, IDE, tools, ...
You sound like an Inline kind of guy :)
> I had some contact with Ingy but he seems quite impossible to reach him
> these days.
True enough. Ingy has been swamped with other things. (Watch for YAML.pm
to hit the CPAN by the end of this weekend!!!)
But he's getting back onto the Inline bandwagon. No, really...
> I have an Inline ::Flex working (version 01) is someone intressted ?
> I plan an Inline::Byacc, Inline::Yacc Inline::Lemon
Wow. I think that makes almost a half dozen people you have started
something in this realm. I would really nudge you towards collaboration.
I'll grep the archives for the email addresses of others who have
attempted work in this realm. Might as well do it right!!!
BTW, this list is a fine place for discussing the project. If it
becomes overbearing we can fork the list, but I don;t see a problem in
the short term.
> Problems :
> There is a big problem with inline, SPEED. The main problem is the use of
> Parse::Recdescent.
> Here is a profiling for a small lexer (C source 90 KB, generated from 14
> regular expressions) on a 500 MHz PC
>
> I timed the inlining process to 1:40
>
> dprofpp.bat -O 50
Yow! I had no idea P::RD was such a pig. Thanks for this data. It's proof
enough.
Let me step back a second and wax philosophical...
Inline's primary usage is to *bind* languages to Perl. I've stated this
before, but I'll do it again. On a large project, one should build a
separate object out of most of the code, and only Inline the code needed
to create the Perl bindings. In other words, if a C function isn't
intended to be called by Perl directly, it probably shouldn't be Inline.
This makes sense from a code maintenance point of view as well. So in
practice the Inline C code stays rather small, and thus the compile
times are generally acceptable.
NOTE: The longest compile time I ever had was 45 seconds. That was on a
P266 running Windows on top of Vmware! I've never seen more than
3-4 seconds other than that case.
That said, Ingy is not one to set arbitrary limits upon those who use
his software. (He merely guides...) Parse::RecDescent is a *proven* hog.
Let's replace it!
I met with Neil today to work on the Inline book. We took some time to
discuss Inline Lex/Yacc, which I needed a little primer on. Neil was an
excellent help since he was one of the first to attempt writing
Inline::Yacc.
We decided that alternate parsing options are needed (thanks Nadim :) and
figured out a plan for deployment in the next Inline release. (more about
that below) Here is basically how things will work:
1) The dependency of Inline on P::RD will be removed. We will preprocess
the P::RD parser and distribute that instead. This will not improve
speed so much, but it will be one less required module, which in my
mind is a good thing. It may also allow Inline to work on Perls that
predate 5.005. (But why would you want to? :) The most important
thing is that the current parser is *stable* and works *everywhere*.
I'm not willing to abandon that at the drop of a hat.
2) We will put a hook into Inline::C so that it will look for a
subroutine called Inline_C_parser in your namespace. It will fall
back on the current one. That way when someone writes
Inline::C::FastParse (or any other name) someone else can say:
use Inline::C::FastParse;
use Inline C;
...
Inline::C::FastParse will export the Inline_C_parser function and
Inline::C will use it.
3) Neil and I will write the canonical Inline::C::FastParse in pure
Perl. This will be a highly optimized regex parser, and should be a
huge improvement over P::RD, without the complexity of an extension
module. Note: Damian already did this for me back in March. IIRC it
was under 100 lines. In contrast, Neil "pickled" the
Inline::C::grammar today and it was 350k.
4) Someone else in the community will step forward and write
Inline::C::YaccParse which will be even faster yet. And they can even
write that in Inline C. :)
5) Neil and I have always wished to implement the Inline::C parser using
cc, or whatever the native compiler happened to be. Then we'd use nm
to dump the symbols and just parse that. Although it's a "science
project" to support every platform, think of the increase in speed
and accuracy. Now we can easily offer this as an option. Or better
yet someone else could take this on :) Inline::C::NativeUnixParse...
> Total Elapsed Time = -4.26418 Seconds
> User+System Time = 75.71027 Seconds
> Exclusive Times
> %Time ExclSec CumulS #Calls sec/call Csec/c Name
> 24.3 18.40 18.006 81472 0.0002 0.0002
> Parse::RecDescent::Expectation::at
> 16.8 12.78 17.325 3349 0.0038 0.0052
> Parse::RecDescent::namespace000001::modifier
> 10.2 7.763 10.613 4760 0.0016 0.0022
> Parse::RecDescent::namespace000001::TYPE
> 9.05 6.851 9.136 2452 0.0028 0.0037
> Parse::RecDescent::namespace000001::comment
> 6.94 5.255 45.171 4735 0.0011 0.0095
> Parse::RecDescent::namespace000001::rtype
> 6.91 5.230 77.305 2452 0.0021 0.0315
> Parse::RecDescent::namespace000001::part
> 6.84 5.176 103.31 4762 0.0011 0.0217 Parse::RecDescent::_parserepeat
> 6.46 4.889 6.412 2367 0.0021 0.0027
> Parse::RecDescent::namespace000001::anything_else
> 3.40 2.575 4.851 26450 0.0001 0.0002
.... much PRD deleted
> Parse::RecDescent::Repetition::describe
> 0.19 0.141 0.859 1 0.1409 0.8595 Inline::C::get_parser
> 0.13 0.100 0.100 1 0.1000 0.0999 Inline::Flex::GenerateLexer
....more deleted
> the first non Parse:: or Text:: is Inline::C::get_parser topping at 0.19 %
> of the time
> Code generation time (exclusive Parse::Recdescent) is only a few seconds.
>
> Does someone else think the same as I do ?
Yes :)
> I plan a major project with quite a lot of inling but I can't afford 1 hour
> compiling.
So Neil and I have decided to release a new minor (well, semimajor) version
of Inline.pm (0.44) for Christmas. (And yes, we mean 2001)
This will probably include:
- Fixes for most bugs reported since 0.43
- Support for alternate parsers :)
- Better debugging output
- A rudimentary Inline shell. (insh)
- Think of it as a terminal based Inline IDE
- Neil has become a Perl shell writing whiz. Check out ActiveState's
new ppm3 installer. It's killer!
- Will eventually work with all Inline::* modules
If you have any urgent needs for the release, please speak up now. The
next planned release will be the famed 0.50 (halfway point :) and will
probably come out in mid winter.
> This is, I think, the major problem inline (C and C++) have. I still use XS
> in another project because of that.
>
> PS. Parese::Recdescent Is cool (I use too) but maybe not in Inline.
Thanks again for your research, and I look forward to seeing you push
forward the elusive Inline::Yacc effort. I'll forward you those email
addresses shortly.
Cheers, Brian
> Nadim Khemir
> Senior Software Engineer
>
> Tel : +46 (0) 46-540 10 95
> Mobile : +46 (0) 733-45 10 95
> E-mail : [EMAIL PROTECTED]
> Internet : www.cpen.com
>
> C Technologies AB
> Ideon Research Park
> Scheelev�gen 15, SE-223 70 Lund
> Sweden
>
