Re: Pegged v0.4: longest-match, left-recursion, and expandable parse trees
Thank you all for the positive comments!
Pegged v0.4: longest-match, left-recursion, and expandable parse trees
Hi,
Pegged is a parser generator based on Parsing Expression Grammars
(PEG) written in D, that aims to be both simple to use and work
at compile-time.
See: https://github.com/PhilippeSigaud/Pegged
To use Pegged, just call the grammar function with a PEG and mix
it in your module. For example:
import pegged.grammar;
mixin(grammar(`
Arithmetic:
Term < Factor (Add / Sub)*
Add < "+" Factor
Sub < "-" Factor
Factor < Primary (Mul / Div)*
Mul < "*" Primary
Div < "/" Primary
Primary < Parens / Neg / Pos / Number / Variable
Parens < "(" Term ")"
Neg < "-" Primary
Pos < "+" Primary
Number < ~([0-9]+)
Variable <- identifier
`));
This will create the `Arithmetic` parser, that can parse your
usual arithmetic expresion at runtime or compile-time.
// Parsing at compile-time:
enum parseTree1 = Arithmetic("1 + 2 - (3*x-5)*6");
// Runtime:
auto parseTree2 = Arithmetic("1 + 2 - (3*x-5)*6");
-
This minor release v0.4 (v0.4.1 right now) is on dub
(code.dlang.org) right now and introduces the following new
features:
* A longest-match alternation operator, |, which will always
choose the longest match during a parse. See
[https://github.com/PhilippeSigaud/Pegged/wiki/Extended-PEG-Syntax] for more details and the interest of this operator for grammar writers.
* The left-recursion engine, introduced in version v0.3.0 is now
fully documented in the wiki at
[https://github.com/PhilippeSigaud/Pegged/wiki/Left-Recursion].
Pegged can deal with left recursion, hidden left recursion and
indirect left recursion. Try it!
* These improvements allow Pegged to generate a parser that
fully parses Extended Pascal files, based on the official ISO
10206:1990 grammar. Have a look at it in the
[https://github.com/PhilippeSigaud/Pegged/tree/master/pegged/examples/extended_pascal] directory.
* The new toHTML function can be used to generate an HTML file
containing an expandable tree view that can be manipulated with
an HTML5-compliant browser. See
[https://github.com/PhilippeSigaud/Pegged/wiki/Parse-Result] for
more details on this new fun way to explore parse results (and
parse failures!).
Pegged documentation can be found on the wiki
[https://github.com/PhilippeSigaud/Pegged/wiki], along with a
tutorial
[https://github.com/PhilippeSigaud/Pegged/wiki/Pegged-Tutorial].
Thanks a lot for Bastiaan Veelo for these wonderful improvements
to Pegged!
Re: Regex in ctfe?
On Tuesday, 26 January 2016 at 01:34:09 UTC, Manu wrote: Possible to std.regex in ctfe? (...) I have a string import destined for a mixin, and I want to parse it with regex, but I haven't been able to make it work. The docs mention nothing about this possibility. Hi Manu, a possible solution for that would be to use my Pegged (https://github.com/PhilippeSigaud/Pegged) project. It's a parser generator that creates compile-time (and runtime) parsers. It's not directly compatible with Phobos regexes, but it's quite easy to extract the matched strings. Bastiaan Veelo recently added support for left-recursive grammars, which means we can now generate parsers for many more grammars (although if your parsing need is covered by regexes, that probably won't interest you...) Philippe
Re: Reducing Pegged ASTs
On Tue, Nov 25, 2014 at 4:12 PM, Nordlöw digitalmars-d-learn@puremagic.com wrote: Is there a way to (on the fly) reduce Pegged parse results such as C [0, 6][int, x, ;] +-C.TranslationUnit [0, 6][int, x, ;] +-C.ExternalDeclaration [0, 6][int, x, ;] +-C.Declaration [0, 6][int, x, ;] +-C.DeclarationSpecifiers [0, 4][int] | +-C.TypeSpecifier [0, 4][int] +-C.InitDeclaratorList [4, 5][x] +-C.InitDeclarator [4, 5][x] +-C.Declarator [4, 5][x] +-C.DirectDeclarator [4, 5][x] +-C.Identifier [4, 5][x] to C [0, 6][int, x, ;] +-C.TranslationUnit [0, 6][int, x, ;] +-C.ExternalDeclaration [0, 6][int, x, ;] +-C.Declaration [0, 6][int, x, ;] +-C.TypeSpecifier [0, 4][int] +-C.Identifier [4, 5][x] and still, when needed, be able query that C.Identifier is an instance of C.DirectDeclarator etc? The reducing can be done, either with operators or semantic actions. IIRC there is a free function in Pegged that does it. I did not automate it, because every time I cut down severely a parse tree, I later regret it because I lost information that way. Cutting while still retaining original info (who is this node's ancestor) is more difficult: you would have to store it somewhere anyhow. You cannot create node classes to represent the hierarchy, because of loops in the grammar: an Identifier can have many different ancestors. Note also that Pegged produces parse trees (complete parsing information), not ASTs. ASTs would indeed be much smaller, but we would have to define what are the master nodes in the D grammar. If not this seems like a cool feature to have ;) I guess it would reduce memory requirements by about a magnitude right? If you want to remember the intermediate nodes you cut down, not really, since you still need to store them somehow. I think what's consuming memory right now is that I duplicate the matched strings at each level, even concatenating them at the higher levels. I should let them only in the 'leaves' of the tree (heck, like an AST). Halas, I've no free time to code anything in D these days... but of course, feel free to push any pull request you might have!
Re: accessing numeric template parameters
On Mon, Nov 3, 2014 at 3:27 PM, Dominikus Dittes Scherkl via
Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote:
If I have a struct with numeric template parameter, how can I access it
within member functions? Like normal member variables? And how about the
constructor?
struct polynomial(uint base)
{
private:
uint[] N;
public:
this(uint x) { base = x; }
base is part of the type. polynomial is just a 'recipe' for a type,
the real struct would be Polynomial!(0), Polynomial!(1), etc. Note
that Polynomial!0, Polynomial!1, ... are all different types.
Being part of the type means it's defined only at compile-time, you
cannot use a runtime value (like 'x') to initialize it.
Note that with your current code, `base' is not visible outside
Polynomial. You can alias it to a field to make it visible:
struct Polynomial(uint base)
{
alias b = base; // b is visible outside (but set at compile-time !)
...
}
You can create one like this:
Polynomial!2 poly;
poly.N = [0,1,0,0,1,1];
assert(poly.b == 2);
Of course, you cannot change b: `poly.b = 3;' is forbidden.
Re: how to expand tuple?
Any time you want to return or store a group of values of different types.
In a way, tuples are just structs without a name (anonymous structs,
if you will).
Say you have function `foo' and want it to return an int and a string.
In D, you cannot do:
(int, string) foo() { ... return (3, abc);}
But you can do:
import std.typecons: tuple;
auto foo() { return tuple(3, abc);}
Which is more or less equivalent to:
struct FooReturn { int i; string s;}
FooReturn foo() { return FooReturn(3, abc);}
`Tuple' and its associated factory function `tuple' is just the
standard way to group values together in Phobos. You'll see some
functions returning Tuples, for example. Some D constructs also
recognize std.typecons.Tuple and deal with it elegantly. By defining
you own struct (like `FooReturn' in the previous example), you gain
some control on your code (you can define how people can interact with
FooReturn, you can test for it, etc), but you also lose the
possibility of easy interaction with some parts of Phobos.
Re: how to expand tuple?
On Sat, Nov 1, 2014 at 9:34 PM, Suliman via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: Few questions. 1. In examples tuples are created with keyword auto. Can I create them with another keyword. Or auto mean structure of data, that have not standard type like (int or string)? `tuple' is a function defined in the std.typecons module. It's a helper function to create Tuple!(T...), a templated struct acting as a tuple. Don't forget the type will depend on the types of the arguments, so: auto tup = tuple(1.0, abc, [2,3]); contains a float, a string and an array of int's, so it's type is Tuple!(float, string, int[]): Tuple!(float, string, int[]) tup = tuple(1.0, abc, [2,3]); It's a bit long to type, so most people prefer using `auto' for such declarations. 2. How ti expend tuple? I tried to do: auto imglist = tuple(aaa); imglist.expand[sss]; writeln(imglist); but got very strange error: app.d(30): Error: cannot implicitly convert expression (sss) of type string to uint Expand, you mean, as in appending to a tuple? It can be done, but it will create a new type. First, tup.expand gives you access to the 'raw' tuple ( a template parameter list, to be precise) underneath Tuple!(T...). The returned value can be indexed, sliced and its length is known at compile time. Continuing with my example: auto first = tup.expand[0]; // 1.0 auto slice = tup.expand[1..$]; // (abc, [2,3]) auto len = tup.expand.length; // 3, since tup has three elements. That explains the error you get: imglist.expand is a bit like a vector: it can be indexed, but with uint, not with a string. I thought you were expanding it, but to the compiler you were trying to index using a string. If you really want to append to a tuple, you must create a new variable, since the augmented tuple will have a different type: auto tup2 = tuple(tup.expand, 'd'); // tup2 == (1.0, abc, [2,3], 'd') The type of tup2 is then Tuple!(float, string, int[], char) Note that, in your case, as you're using only strings, an array of strings would probably be easier to use.
Re: how to expand tuple?
I thought you were expanding it Drat. *You* thought you were expanding it.
Re: D Parsing (again)/ D grammar
Thanks a lot, by the way! I've just skimmed through the code and the README... You did not use the packed forest representation, did you..? Sorry for the microscopic documentation (Pegged is more documented than that...), it was a 'for me only' project. The forest is packed, in the sense that common nodes are re-used and shared among parse trees: all intermediate parse results from any grammar part is stored and used to produce the parse nodes. The range view gives access to parse trees one after another, but the global parse forest is present in the grammar object (or rather, generated and completed during the parse process: each new parse result completes the parse forest). It has a strange effect on parsing times repartition among the parse results: If you time the different parse trees, you'll see that the first one might take maybe 40% of the entire parsing time all by itself, because it has to discover all parse results alone. The following trees are very rapidly calculated, since the intermediate parse results are already known. Of course, once the parse trees begin to deviate from the first ones, the process slows down again since they have to explore as-yet-unused rules and input slices. I'm not sure the previous paragraph is clear... Imagine you have 10 different parse trees. They could be disributed like this: # parse result % global parse time 1 40% 2 2 3 3 4 3 5 5 6 6 7 8 8 10 9 11 1012
Re: D Parsing (again)/ D grammar
Anyway, thank you very much for the code. The weekend is coming - I'll play with your implementation and see if there any improvements possible. Be sure to keep me informed of any enormous mistake I made. I tried Appender and other concatenation means, without big success. Btw, I saw on the ML that using byKeys.front() is very slow. Use keys[0] of somesuch instead.
Re: D Parsing (again)/ D grammar
Chances are that I will be able to get the original GLL parser generator from one of algorithm authors (Adrian Johnstone). He's really helpful here. From that point, I will only have to add a frontend for generating a concrete parser, starting with Python - it already has a fully working grammar. Hopefully, I will also be able to find a suitable grammar for D, it is always a pleasure to play with the language. Otherwise, I will continue my current effort - implementing the GLL parser generator in D. I did that during this summer, almost to the point it was self-sustained (that is, the GLL parser generator was used to generate a parser for grammars files). I chose a range interface: input is a string, the parse forest is output as a range of parse trees. I loved to see it generate the different possible parse trees on ambiguous grammars, and accepting left- and right-recursing grammars! GLL is a very noce algorithm. Halas, even with some shortcuts on Kleene stars it was quite slow. I tried to use threads (spawning worker threads on alternatives), but that did not change the timings much. I could make it generate a parser for JSON and compared it to the jsonx module that Sönke presented here. Bah, it was 1000 times slower (which is all relative: it still takes only a few ms to parse a big JSON file. But still, far away from the microseconds it should need). Pegged was faster that this GLL generator, but of course still far slower than jsonx. [Of course, Pegged can cheat: it can parse your file at compile-time, resulting in 0-µs parsing time at runtime :-)] Also, the way I coded it I hit CTFE limits and could not have it parse at compile-time. A shame, really. This is one of the reasons I prefer LL/RD parsers :-) They are easy to follow. I would like to try a LALR compile-time generator and compile-time parser. I'm pretty sure LALR tables could be expanded directly as D code instead of being read during parsing. Philippe
Re: D Parsing (again)/ D grammar
I did that during this summer, almost to the point it was self-sustained (that is, the GLL parser generator was used to generate a parser for grammars files). I chose a range interface: input is a string, the parse forest is output as a range of parse trees. Nice! Is it public? Github? No github repo. I could put it alongside Pegged, I suppose. I used git internally, though. I was a bit disheartened by the speed I got, so did not publish nor announced it here. Note also that I saw it as an alternative engine for my own Pegged project, so I used the same way of defining grammars (some prefixes ans suffixes for dropping nodes in the parse tree and so on). I can send you the code (well the entire repo), if you wish. I did not touch it for the past 3 months, so I already don't remember what state it was in :-(. Looking at the code now, it seems I'm still using Pegged to parse the initial grammar. Bootstrapping did not go well. Send me an email at firstname . lastname @gmail.com (philippe sigaud) Halas, even with some shortcuts on Kleene stars it was quite slow. I tried to use threads (spawning worker threads on alternatives), but that did not change the timings much. AFAIK, multithreading is a bad idea in parsing. I think, as many things in CS, that people developed parsing algorithms before multicores existed. Spawning threads or fibers for some alternatives (A | B) seems nice to me. I got interesting results with a purely multithreaded parser that spawned children for each choice. Actually, in the gll-combinators Scala project they have similar speed problems. I don't if it's a fundamental algorithm problem or an implementation details that lead to slowdowns. Well, when all resulting implementations have speed problems... I found an article by the GLL authors explaining how they coded their algorithm for more speed, but I couldn't wrap my head around it. Philippe
Re: D Parsing (again)/ D grammar
I have a huge collection of projects I never published :-) We all do, I guess. Oh, the ratio is more and 100 projects for one published... No, this is not exactly what I mean. Multithreading can be perfectly fine in some cases, very fruitful sometimes. Say, in NLP, when one has to process long or highly ambiguous strings, and the parse tree can become huge and is of importance in itself... Yes. In programming language parsing this is just a small step towards further stages within a longer pipeline. It has to be fast enough to make multithreading on this step an overkill. I don't know. Using fibers, I'm hoping to get interesting results one day. I got it by a workstorm before trying fibers. OS threads were a bit to heavy and didn't work that well. BTW, there's one an interesting related work that probably should be taken as a positive example of generalized parsing: the Elkhound parser generator. It uses a hybrid LALR/GLR approach, thus achieving better performance. There's much more to it, I didn't go too deep into it. Yes, Elkhound is interesting, their approach is nice. But It gave me the impression to be abandoned for a few years? I found an article by the GLL authors explaining how they coded their algorithm for more speed, but I couldn't wrap my head around it. By now, I have read the original Tomita's GLR paper, Antlr ALL(*) paper, a few recent GLR papers, three papers on GLL and a few related ones . It took... A while. I sort of understand the idea, but still not sure about the details :-) ALL(*) is on my todo list. I tried to implement it in Spring, but got bogged down in the details. Even the white paper has some imprecisions when you really sit down and try to code it. I could have a look at ANTLR v4 source, but wanted to have a clean start, right from the white paper. What's the name of the paper you read? Modelling GLL Parser Implementation? Yes.
Re: D Parsing (again)/ D grammar
On Thu, Oct 2, 2014 at 11:19 PM, Vladimir Kazanov via Digitalmars-d digitalmars-d@puremagic.com wrote: Check the mailbox, Thank you I sent it to you. I was asleep, sorry :-)
Re: How to pack types with variables in one message to send it to another thread? [tuple]
You can also create new types:
struct UseSprite { string s;}
struct UseAnimation { string s;}
It's not a class instance, it's a class type. Something like
`cast(Sprite) null` in parameters. It can be replaced by string
Sprite, but in this case I can't use receive() as it is. E.g.
send(tid,gameobjectId,Sprite,reload);
//must call sprite.reload();
You could use:
sent(tid, gameobjectId, UseSprite(reload));
send(tid,gameobjectId,Animation,reload);
//must call animation.reload();
sent(tid, gameobjectId, UseAnimation(reload));
Another way, if you have way to determine that gameobjectId points to
an animation or a sprite, would be to define a struct name Reload {}
and then:
sent(tid, gameobjectId, Reload());
Third way: if Animation.reload() and Sprite.reload() are static methods:
send(tid, gameobjectId, Sprite.reload);
But both messages are (int, string, string) so they can't be separate by
different receiving functions. It will be better if messages was (int,
Sprite, string) / (int, Animation, string). And it solves my problem. :)
But I don't know how to achieve this.
See my proposal: define your message as types, directly, and load them
for any data necessary for the call.
UseAnimation(reload), or whatever.
Re: literate programming in D
Ah that sounds interesting too! Immediately I start thinking in terms like
tidlywiki http://tiddlywiki.com/ or something similar, I guess the emacs way
described earlier also would support this. I personally always enjoy reading
the readthedocs stuff http://docs.readthedocs.org/en/latest/ is that the
sort of stuff you mean?
Tiddlywiki is interesting, but I'm really talking about the way LP was
used by Knuth WEB/CWEB in his explanation of TeX and METAFONT.
The Wikipedia article explains it pretty well
(http://en.wikipedia.org/wiki/Literate_programming)
You write a document combining documentation and code. Then two
different programs (weave and ? for WEB) create the resulting
documentation for one (HTML, pdf, whatever) and the code (a .d file)
for another.
I discovered LP through the incredible book Physically-based
Ray-Tracing. The book is one program, explained and documented using
literate progamming. It's an incredible achievement (1000+ pages of
explanations!).
See:
www.pbrt.org
and more precisely:
http://www.pbrt.org/chapters/pbrt_chapter7.pdf
The code begins at page 18 of the pdf.
For example, at page 22:
Sample Method Definitions ≡
Sample::Sample(SurfaceIntegrator *surf, VolumeIntegrator *vol,
const Scene *scene) {
surf-RequestSamples(this, scene);
vol-RequestSamples(this, scene);
Allocate storage for sample pointers 301
Compute total number of sample values needed 302
Allocate storage for sample values 302
}
The snippet Sample Method Definitions introduces three new snippets,
that will be explained elsewhere. Other snippets might also use the
same references, if needed.
It's not complicated to write a D tool for that, I did that some time
ago. Once you define your begin/end token for snippet definitions, you
can parse them to extract the way they are linked.
Re: String to binary conversion
On Wed, Sep 3, 2014 at 7:59 PM, Ali Çehreli digitalmars-d@puremagic.com wrote: Also, I've just noticed that Questions about learning D sounds a little off because it sounds as if the questions should be like how can I learn D. :) How about something like Questions when learning D. Or even: Learning D
Re: Daemonize v0.1 - simple way to create cross-platform daemons
Nice!
I have a few questions/remarks, mainly to simplify the API somewhat.
Please bear with me :-)
// First you need to describe your daemon via template
alias daemon = Daemon!(
DaemonizeExample1, // unique name
Does the user sees/uses this name in any way afterwards? Because I
think you could also produce a unique string at compile-time (by using
__FILE__ and __LINE__, unless someone has a better idea), if the user
does not provide one. Maybe he just wants an anonymous daemon, or
doesn't care, whatever.
// Setting associative map signal - callbacks
KeyValueList!(
If I understand correctly, the Daemon template waits for a list of (at
least one) Signal, then a delegate, then some more Signals, another
delegate, and so on?
If that's so I think you could ditch KeyValueList (or build it
invisibly to the user) and let the user write only the signals and
delegates:
alias daemon = Daemon!(
Signal.Terminate, Signal.Quit, Signal.Shutdown, Signal.Stop,
(logger, signal) {
...},
Signal.Hangup,
(logger) {
...}
...
);
Iterate the argument list, collecting Signals. When you hit a
delegate, create a Composition!( ... ) with the previous signals, jump
above the delegate and so on.
Is the idea that, if the delegate has two arguments, then the second
is the signal that will be passed to it, and if it has only one
argument, only the logger will be passed?
What if the user does not want a logger? Is a daemon always associated
to a log file in OSes?
// Main function where your code is
(logger, shouldExit) {
// will stop the daemon in 5 minutes
auto time = Clock.currSystemTick +
cast(TickDuration)5.dur!minutes;
while(!shouldExit() time Clock.currSystemTick) { }
return 0;
}
Is the main function always the last delegate?
Concerning the DaemonClient template, could you not ask for Daemon to
generate it on demand? Or is DaemonClient always used in another
module?
Because, given a Daemon, extracting the simplified DaemonClient can be
done, I think.
* Custom signals
enum Signal : string
{ ... }
@nogc Signal customSignal(string name) @safe pure nothrow
{
return cast(Signal)name;
}
I didn't know you could have an enum and extend it with a cast like this. Wow.
* Signal composition
What happens when an unhandled signal is passed to a daemon?
P.S. At the moment library doesn't support Mac and other Posix systems, the
support is going to be added at next releases.
Do you foresee any difficulty in adapting this to Mac?
Re: Daemonize v0.1 - simple way to create cross-platform daemons
Does the user sees/uses this name in any way afterwards? Because I
think you could also produce a unique string at compile-time (by using
__FILE__ and __LINE__, unless someone has a better idea), if the user
does not provide one. Maybe he just wants an anonymous daemon, or
doesn't care, whatever.
Yes, the name is used in windows service manager (you can start/stop the
daemon by control panel) and for default locations of .pid and .lock files.
OK.
Auto generated name will prevent sending signals and could be ugly displayed
in service manager. The feature is useful for simple daemons, I will play
around with that idea to find out if it worth.
Great.
I will add the approach in next release (it requires some more additional
templates to wrap all the mess) thanks to arguments grammar has no
ambiguities.
Yes, the grammar is simple, use it to simplify the life of your users.
Is the idea that, if the delegate has two arguments, then the second
is the signal that will be passed to it, and if it has only one
argument, only the logger will be passed?
Yes
OK.
IIRC, I read in your code that composed signals means the next
delegate must have the (logger, signal) {...} form.
Why?
What if the user does not want a logger? Is a daemon always associated
to a log file in OSes?
It is a general rule as the stderr and stdout files are closed. At next
version I want to use duck typing for logger (or sink approach same as
toString uses) and add a feature to reopen stderr/stdout to another file.
OK.
Concerning the DaemonClient template, could you not ask for Daemon to
generate it on demand? Or is DaemonClient always used in another
module?
DaemonClient is designed to be used in another module, you can send signals
with full Daemon template.
OK. I'd have thought that just having the name of the daemon would be
enough to send it signals.
What happens when an unhandled signal is passed to a daemon?
The event is logged down and ignored.
Is that the standard behavior for daemons in OSes?
You could have the daemon stopped, after logging the unhandled signal.
Or you could also block compilation if a daemon does not handle all signals...
Or have a 'catch-all' delegate, as std.concurrency.receive, which uses
a (Variant v) {...} delegate at the end.
See:
http://dlang.org/library/std/concurrency/receive.html
(btw, signals could also be types and you could have a handling syntax
similar to receive).
Re: Daemonize v0.1 - simple way to create cross-platform daemons
On Sun, Aug 31, 2014 at 11:36 PM, Meta via Digitalmars-d-announce digitalmars-d-announce@puremagic.com wrote: I didn't know you could have an enum and extend it with a cast like this. This is not a good thing. Enums are supposed to denote a *closed*, enumerated set of items. I agree. It's fine here (but IMO bad style) because the author expects there to be user-created values casted to Signal passed to functions/templates that expect a Signal, but this would wreak havoc on code that was expecting a valid enum value (by valid, I mean only one of the predefined enum values). I was about to suggest final switch, until I saw this extension of Signal. I wonder what happens to final switch in this case.
Re: Daemonize v0.1 - simple way to create cross-platform daemons
I can (not must) have the form, the delegate params are tested independently from signal composition. OK, good. Is that the standard behavior for daemons in OSes? Most signals are simply ignored (except termination ones). I see. Some signals could be sent without any reason: sighup or interrogate in windows. Ignoring most signals is a better strategy, the exception could be done for terminating signals - their default handlers should set `shouldExit` flag to true. OK, I didn't know that. http://dlang.org/library/std/concurrency/receive.html Good idea, it will be implemented. Great!
Re: Clojure transducers
What is D's attitude toward this concept?
Hickey's original announcement is also interesting to read. Elegant, as always.
It seems to me that ranges and range algorithms in D already permit
this composition of actions, without the creation of intermediate
structures:
import std.algorithm;
import std.functional: pipe;
alias incsum = pipe!(
map!(x = x+1),
reduce!((x,y) = x+y)
);
alias filtermap = pipe!(
filter!(x = x%2==0),
map!(x = x+1)
);
alias mapfilter = pipe!(
map!(x = x+1),
filter!(x = x%2==0)
);
void main() {
auto sequence = iota(10);
writeln(incsum(sequence)); // 55
writeln(filtermap(sequence)); // [1, 3, 5, 7, 9]
writeln(mapfilter(sequence)); // [2, 4, 6, 8, 10]
}
Re: How to build dlang.org dd files
dmd -c -o- macros.ddoc doc.ddoc -Df{ddoc_filename}.html {ddoc_filename}.dd
If someone knows of a more official syntax, please let me know.
I don't know of another syntax, but could you please put this
somewhere on the wiki?
Maybe in the cookbook section:
http://wiki.dlang.org/Cookbook
Re: alias and mixin
It is basically just an annoying grammar limitation that does not allow to use mixin / __traits as an identifier. The usual helper template: ``` alias helper(alias a) = a; ``` helps for aliasing __traits and anonymous function templates (x = x+1), but I don't see an easy solution for mixin inside the current language, apart from pushing the mixin outside.
Re: alias and mixin
I recommend slightly more generic form:
template Alias(T...)
if (T.length == 1)
{
alias Alias = T[0];
}
it is quite helpful in templated code to be able to alias _anything_
That's what I use also, but didn't want another thread on the (T...)
if (T.length ==1) trick :)
But yeah, no fun for mixins :(
Mainly, I slap them in my code, then slowly migrate them outwards (in
'head' position) until that compiles...
Re: literate programming in D
On Sat, Aug 30, 2014 at 1:37 AM, nikki wrote:
I wasn't too happy about it and I wrote my own little parse
thingie and have
a literate version nice and meta and small and sloppy ;)
http://nikkikoole.github.io/docs/dokidokDOC.html
I use it to change my d sourcefile slightly (into valid
markdown)
then I use a node module (ghmd) to make sortof sexy html from
that.
Nice. Maybe you could strip the initial whitespaces in the code
parts: due to blocks, your code lines are shifting to the right.
If you document something that's inside a method inside a class,
you're bound to have problems :)
Right now, you're documenting your code in a linear way. That's
good, but for me, most of LP is based around the idea of naming
snippets and referring to them in code, to be explained elsewhere:
```
Here is the main loop:
Main=
void main() {
Initialize Variables
Precompute State
foreach(i; 0..max) {
Do Computation
}
}
Before doing all this, we need to initialize the variables:
Initialize Variables= ...
```
Of course, snippets can refer to other snippets, recursively.
Do you have any plan to go there?
Re: literate programming in D
On Friday, 29 August 2014 at 23:58:19 UTC, Chris Cain wrote: I used https://www.npmjs.org/package/literate-programming (+ pandoc) to do this when writing https://dl.dropboxusercontent.com/u/2206555/uniformUpgrade.pdf in markdown. Do you remember if some snippets can be hidden in the final documented output (I don't find that in this package)? I know the goal of LP is to explain your code, but I remember using that with other systems: that's useful when you don't want to show some plumbing (for a tutorial for example, where you want to concentrate on the main suject).
Re: getting the adress of a function by its name in string format
instead I would like to use EAT_FOOD_DECISION and EAT_FOOD_BEHAVIOUR and get the functions? Is that possible somehow? If all your functions had the same signature, you could use an associative array FunctionType[string] and initialize it: FunctionType[string] myFuncs; myFuncs[EAT_FOOD_DECISION] = EAT_FOOD_DECISION; But it seems they have different types?
Re: Crazy code by Adam on SO
http://stackoverflow.com/questions/2329/d-finding-all-functions-with-certain-attribute That's what a good part of his book is about, also. A nice read, one of my favorite chapters of the D Cookbook. Ideally, Adam should do a PR for Phobos to add some code-walking ability like this. I'm interested by any answer on one of his points: is there currently a way to find function-local imports?
Re: Crazy code by Adam on SO
On Fri, Aug 29, 2014 at 7:12 PM, H. S. Teoh via Digitalmars-d digitalmars-d@puremagic.com wrote: The mixin(import ...) line was mind-blowingly ingenious, I have to say. I remember using the same 'module' trick a few years ago (that is, getting a name with __traits and testing whether it begins with module or not) and, like Adam, being a bit ashamed by what I just did :) While reading his book this summe, I was, like, Oh, then this is what we all do. Ideally, there should be a __traits(getModules) or __traits(getImports, ...). I know, I know, where is the PR?. Also, this helper!(__traits) just to avoid a hole in the alias syntax is... what we all do here I suppose, but honestly the parser should accept alias member = __traits(getMember, ID, memberName); as valid D code.
Re: Crazy code by Adam on SO
On Friday, 29 August 2014 at 17:29:34 UTC, H. S. Teoh via Digitalmars-d wrote: A lot of Adam's stuff are in that category. (...) I highly recommending studying how Adam's code achieves this. :-) It's also a bit hypnotic. I wanted to do a quick test of his simpledisplay.d module, after reading one of his recipes. I fetched his github repo and... found myself one hour later, still reading a totally different module :)
Re: const after initialization / pointers, references and values
On Wed, Aug 20, 2014 at 11:49 PM, Jakob Ovrum via Digitalmars-d digitalmars-d@puremagic.com wrote: Certainly the easiest, but I don't think it's always the best. If light-weightedness is desired, make the struct contain the reference, effectively making the struct a reference type Well, yes. But the OP explicitly asked not to have to deal with pointers. So...
Re: const after initialization / pointers, references and values
After initialization there is no need to modify the data anymore.
My first question is: can the data be changed to const once
initialized?
You can have const or immutable data, that can be initialized once.
I tend to use pure functions to do that:
struct Node {
Node[] block;
uint num = 0;
}
Node makeDefaultNode() pure {
return Node(null, 1);
}
void main(){
immutable Node n = makeDefaultNode();
import std.stdio;
writeln(n);
writeln(typeof(n).stringof);// immutable(Node)
}
You can also use module constructors:
module foo;
immutable myDefault;
static this() {
myDefault = ...
}
/* rest of code */
I cannot comment much more. I use functions to initialize my immutable
values. I don't know much about module constructors.
Related to that, how can the default initializer be changed to
immutable?
In your example code, your default value can be marked immutable, if
you want it.
Just use:
immutable Node[] default_nodes = [
{num:3},
{block:[{num:4},{num:6},{block:[{num:5},{num:7}]}]},
// ...
];
The second question is related to pointers, references and values
I know that structs by default use value semantics, but as data
is large I want to avoid data copying.
But I would like to avoid use of pointers, so, is there a way of
using reference semantics in the example code?
Maybe is as simple as changing from struct Node to class
Node, but seems intuitive that classes carry more overhead than
structs.
If you want reference semantics but do not want to have pointers in
your code, yes classes are your best choice.
How much overhead carries a class in comparison to a struct?
There is an overhead when calling functions, due to the virtual table.
But that's mainly when you construct hierarchies of classes, so that
the runtime can determine which method to call for each object. If you
don't need derived classes, then I guess using a final class is your
best bet. I think it discards the virtual table.
final class { ... }
I'd say: test it. Duplicate your module, change your structs to final
classes and compare the speed of the two modules.
Re: Variadic parameter of length 1
On Wed, Aug 20, 2014 at 5:34 PM, Dominikus Dittes Scherkl via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: On Wednesday, 20 August 2014 at 15:26:14 UTC, monarch_dodra wrote: AFAIK, it's a historical workaround to accept T as either alias or not alias, as varargs have auto alias. EG: foo!int //OK foo!hello //OK too Ah, ok. And why historical? Is that not necessary anymore? What better solution is there today? No better solution that I know of. alias template parameters (alias a) match symbols (names, user-defined types) whereas type parameter (T) match only pure types. So when we need to match anything, we use (T...) if (T.length == 1)
Re: No Output with shebang.
gdc just compiles the program to a.out. It doesn't run the resulting executable. You need to use something like rdmd instead of gdc. rdmd compiles to some temporary location and then runs the executable. Wow, that was fast. Thanks a lot! Can compiler switches be used with the shebang notation? If yes, there is certainly a GDC flag (-run?) that tells it to run the generated executable.
Re: Cross-Platform D
On Tuesday, 19 August 2014 at 11:14:17 UTC, Vladimir Panteleev wrote: You realize you're replying to a decade-old post, right? What about storing D files on BETAMAX tapes? Oh, wait.
Re: Why does D rely on a GC?
I won't look at it again for a different reason. They're the types that say A monad is just a monoid in the category of endofunctors, what's the problem? Sure, so one should point out that problem may be made out to be that the monoidal product [1][2] is underspecified for someone unfamiliar with the convention (in this case it should be given by composition of endofunctors, and the associator is given pointwise by identity morphisms). (But of course, the more fundamental problem is actually that this characterization is not abstract enough and hence harder to decipher than necessary. A monad can be defined in an arbitrary bicategory... :o) ) What do /you/ think is the problem? I remember finding this while exploring Haskell, some years ago: http://www.haskell.org/haskellwiki/Zygohistomorphic_prepromorphisms And thinking: Ah, I get it, it's a joke: they know they are considered a bunch of strangely mathematically-influenced developers, but they have a sense of humor and know how to be tongue-in-cheek and have gentle fun at themselves. (My strange free association was: Zygo = zygomatics muscle = smile laughter = joke). That actually got me interested in Haskell :) But, apparently, I was all wrong ;-) It got me reading entire books on category theory and type theory, though.
Re: iterate traits ?
A foreach becomes compile-time whenever the aggregate is a purely compile-
time construct such as a tuple.
Yeah, I think you transformed it into a runtime array by using [ ...
]. Tuples with compatible types can be 'downgraded' to arrays that
way. But there is no need to: tuples are iterable, indexable and
slice-able in D.
import std.stdio;
struct Vertex {float[3] position; float[3] normal;}
void main() {
foreach(e; __traits(allMembers, Vertex)) {
alias tt = typeof(__traits(getMember, Vertex, e)); // worksforme
writeln(tt.sizeof);
}
}
Re: @safe, pure and nothrow at the beginning of a module
If I understand correctly Adam Ruppe's Cookbook, by putting @safe: pure: nothrow: at the beginning of a module, I distribute it on all definitions, right? Even methods, inner classes, and so on? I read Adam's book again and I was wrong: Chapter 7, p. 173: You may add @safe: to the top of your module and aggregate definitions to apply the annotation to all function that follows, instead of writing it on each individual function. So, first, he's talking only about @safe (though I suppose it works the same for all annotations) and he says: *and aggregate definitions*. We indeed need to put annotations inside aggregates to affect their innards. If that's true, I have a lot of annotation sprinkling to do.
Re: @safe, pure and nothrow at the beginning of a module
On Sat, Aug 16, 2014 at 1:30 PM, Artur Skawina via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: On 08/16/14 13:18, Philippe Sigaud via Digitalmars-d-learn wrote: We indeed need to put annotations inside aggregates to affect their innards. If that's true, I have a lot of annotation sprinkling to do. It's not true for @safe, but true for some other attributes. http://forum.dlang.org/post/mailman.125.1397731134.2763.digitalmar...@puremagic.com Okay... So @safe includes child scopes. I suppose @trusted and @system work in the same way. *but* nothrow, @nogc and UDA's do not include child scopes. Putting them at the beginning of a module will not affect methods in aggregates... What's the situation for pure? (I don't have a D compiler handy right now, or I would test it myself).
Re: @safe, pure and nothrow at the beginning of a module
Artur: @safe, @trusted, @system, shared, immutable, const, inout and `extern (...)` affect child scopes. `synchronized` does too, but in a rather unintuitive way; hopefully nobody uses this. ;) Well, I also hope no one uses inout: at the module level? Other attributes, including 'pure' and 'nothrow' only affect symbols in the current scope. There we are. Good to know, thanks.
Re: Automatic Inference for Both Key and Value Types of an Associative Array
Kenji Hara: I implemented partial type deduction in AA keys. https://github.com/D-Programming-Language/dmd/pull/3615 For example: auto[auto[$]] aa5 = [[1,2]:1, [3,4]:2]; static assert(is(typeof(aa5) == int[int[2]])); int[int[][$]] aa15 = [[[1],[2]]:1, [[3],[4]]:2]; static assert(is(typeof(aa15) == int[int[][2]])); It's wonderful! I like it a lot. As for Walter's question about the syntax, I quite like the [$] thingie. '$' is already associated with 'length' in my mind.
Re: How to declare a parameterized recursive data structure?
On Fri, Aug 15, 2014 at 9:13 PM, artemav via Digitalmars-d
digitalmars-d@puremagic.com wrote:
:) Wow, thanks for all replies. I realized too late that should write
that's it.
Hmm, It's also a good sign that D community is active.
Can I add something? ;)
You can also use a class, as they are reference types:
class node(T) {
T data;
node next;
}
Or use a dynamic array if you want to define a tree:
struct node(T) {
T data;
node[] children;
}
Re: Appender is ... slow
Quick test...
Ah, thanks a lot Jonathan. I kept telling me I should probably test it
on a simple case.
OK, the good news is, Appender works in these cases (I mean, that's
good news for Phobos).
Now, I just have to find out why it's slower in my case :)
import std.array;
import std.datetime;
import std.stdio;
enum size = 1000;
void test1()
{
auto arr = appender!(int[])();
foreach(n; 0 .. size)
arr.put(n);
}
I used ~= to append to an Appender. The examples use .put, but ~= is
documented so I considered the operator was just syntax sugar. I
didn't have a look at its implementation.
void test2()
{
int[] arr;
foreach(n; 0 .. size)
arr ~= n;
}
void test3()
{
auto arr = new int[](size);
foreach(n; 0 .. size)
arr[n] = n;
}
This one is simple and interesting. In my case I don't know the length
in advance (I'm doing some parsing and predicting the size of the
parse forest based on the input length is somewhat tricky), but I can
pre-allocate some, based on a simple heuristics.
void test4()
{
auto arr = uninitializedArray!(int[])(size);
foreach(n; 0 .. size)
arr[n] = n;
}
Oh, I didn't know this one.
With size being 1000, I get
178 ms, 229 μs, and 4 hnsecs
321 ms, 272 μs, and 8 hnsecs
27 ms, 138 μs, and 7 hnsecs
24 ms, 970 μs, and 9 hnsecs
Same here, good.
Using ~= n instead of .put(n) gives me results consistently slightly
slower for Appender (170 ms - 190 ms, repeatedly, even going back and
forth between the two possibilities.
I created a test1prime to test that.
With size being 100,000, I get
15 secs, 731 ms, 499 μs, and 1 hnsec
29 secs, 339 ms, 553 μs, and 8 hnsecs
2 secs, 602 ms, 385 μs, and 2 hnsecs
2 secs, 409 ms, 448 μs, and 9 hnsecs
Ditto. OK, that's good. Also, it's still slower with using Appender ~=
n instead of Appender.put. (18s instead of 15, 20% slower)
So, kids: don't do that.
So, it looks like using Appender to create an array of ints is about twice
as fast as appending to the array directly, and unsurprisingly, creating the
array at the correct size up front and filling in the values is far faster
than either, whether the array's elements are default-initialized or not.
And the numbers are about the same if it's an array of char rather than an
array of int.
Perfect. That also means my go-to method will still be using standard
arrays, but with pre-allocation.
I just feel stupid writing that, because it's obvious that reserving
things should give it better speed.
Also, curiously, if I add a test which is the same as test2 (so it's just
appending to the array) except that it calls reserve on the array with size,
the result is almost the same as not reserving. It's a smidgeon faster but
probably not enough to matter. So, it looks like reserve doesn't buy you
much for some reason. Maybe the extra work for checking whether it needs to
reallocate or whatever fancy logic it has to do in ~= dwarfs the extra cost
of the reallocations? That certainly seems odd to me, but bizarrely, the
evidence does seem to say that reserving doesn't help much. That should
probably be looked into.
Yeah, I get a small boost of 5% compared to not reserving at size
1000, which completely disappears for longer arrays.
(No different for size 100_000).
In any case, from what I can see, if all you're doing is creating an array
and then throwing away the Appender, it's faster to use Appender than to
directly append to the array.
Indeed.
Maybe that changes with structs? I don't know.
I'm using classes, because what I'm pushing into the Appender are
graph nodes and I got fed up with tracing pointer to strucs
everywhere. Maybe I should change back to structs and see what it
does.
It would be interesting to know what's different about your code that's
causing Appender to be slower, but from what I can see, it's definitely
faster to use Appender unless you know the target size of the array up
front.
Good conclusion. Thanks for your help. My takeaway idea is that I'll
use arrays, but 'new T[](size)' them before. If that makes heavy
concatenation 10 times faster, it should have a positive effect (I'm
not of course waiting for anything near a 10x boost in my computation
time).
Re: Appender is ... slow
I wonder if using plain `Array` instead may be result in better performance
where immutability is not needed.
Hmm, no:
module appendertest;
import std.array;
import std.datetime;
import std.stdio;
import std.container;
enum size = 1_000;
void test1()
{
auto arr = appender!(int[])();
foreach(n; 0 .. size)
arr.put(n);
}
void test1prime()
{
auto arr = appender!(int[])();
foreach(n; 0 .. size)
arr ~= n;
}
void test2()
{
int[] arr;
foreach(n; 0 .. size)
arr ~= n;
}
void test2prime()
{
int[] arr;
arr.reserve(size);
foreach(n; 0 .. size)
arr ~= n;
}
void test3()
{
Array!int arr;
foreach(n; 0 .. size)
arr ~= n;
}
void test3prime()
{
Array!int arr;
arr.reserve(size);
foreach(n; 0 .. size)
arr ~= n;
}
void test4()
{
auto arr = new int[](size);
foreach(n; 0 .. size)
arr[n] = n;
}
void test5()
{
auto arr = uninitializedArray!(int[])(size);
foreach(n; 0 .. size)
arr[n] = n;
}
void main()
{
auto result = benchmark!(test1, test1prime,
test2, test2prime,
test3, test3prime,
test4,
test5
)(10_000);
writeln(Appender.put :, cast(Duration)result[0]);
writeln(Appender ~=:, cast(Duration)result[1]);
writeln(Std array :, cast(Duration)result[2]);
writeln(Std array.reserve :, cast(Duration)result[3]);
writeln(Array :, cast(Duration)result[4]);
writeln(Array.reserve :, cast(Duration)result[5]);
writeln(new T[]() :, cast(Duration)result[6]);
writeln(uninitializedArray :, cast(Duration)result[7]);
}
Times:
Appender.put :157 ms, 602 μs, and 3 hnsecs
Appender ~=:182 ms, 807 μs, and 1 hnsec
Std array :256 ms, 210 μs, and 7 hnsecs
Std array.reserve :244 ms, 770 μs, and 4 hnsecs
Array :336 ms, 207 μs, and 3 hnsecs
Array.reserve :321 ms, 500 μs, and 6 hnsecs
new T[]() :28 ms, 496 μs, and 6 hnsecs
uninitializedArray :26 ms and 620 μs
Re: Appender is ... slow
On Thu, Aug 14, 2014 at 11:33 PM, Joseph Rushton Wakeling via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: On 14/08/14 19:16, Philippe Sigaud via Digitalmars-d-learn wrote: Do people here get good results from Appender? And if yes, how are you using it? An example where it worked for me: http://braingam.es/2013/09/betweenness-centrality-in-dgraph/ (You will have to scroll down a bit to get to the point where appenders get introduced.) I remember reading this and loving it! Any news on this? Do you have new algorithms?
Re: Appender is ... slow
It is very different with better compiler though : $ ldmd2 -release -O a.d $ ./appendertest Appender.put :378 ms, 794 μs, and 9 hnsecs Appender ~=:378 ms, 416 μs, and 3 hnsecs Std array :2 secs, 222 ms, 256 μs, and 2 hnsecs Std array.reserve :2 secs, 199 ms, 64 μs, and 5 hnsecs Array :349 ms and 250 μs Array.reserve :347 ms, 694 μs, and 1 hnsec new T[]() :37 ms, 989 μs, and 8 hnsecs uninitializedArray :39 ms, 333 μs, and 3 hnsecs (size 10_000) OK, same here, except std array gives me only 1.4 s, while the other timings are about the same. (0.14 alpha2 : ldmd2 -O -inline). Drat, that means testing on many different compilers. Oh well, let's start small: pre-allocating, better algorithms, then I'll do real speed instrumentation.
Re: Appender is ... slow
On Fri, Aug 15, 2014 at 1:57 PM, Messenger via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: T[size] beats all of those on dmd head, though it is inarguably a bit limiting. I confirm (even with 2.065). With ldc2 it's optimized out of the way, so it gives 0 hnsecs :-) Hmm, what about a sort of linked list of static arrays, that allocates a new one when necessary?
Re: Appender is ... slow
Well, I created a wrapper around a std.array.uninitializedArray call, to manage the interface I need (queue behavior: pushing at the end, popping at the beginning). When hitting the end of the current array, it either reuse the current buffer or create a new one, depending of the remaining capacity. On the 'synthetic' benchmarks, it performs quite reasonably: half the time of Array or Appender (twice faster), 5x faster than standard array, and 3-4x slower than uninitializedArray. And... It does not change the timings in my code, it even makes things slower when pre-allocating to much. Only by pre-allocating only a few elements do I get back the original timings. So, I guess I'm suffering from a bad case of premature optimization :) I thought that, having lots of concatenation in my code, that'd be a bottleneck. But it appears than pre-allocation does not give me any speed-up. Well, at least it got me thinking, testing LDC a bit more and learning things on Array and Appender ;) Thank for your help guys, it's now time for the -profile switch again...
@safe, pure and nothrow at the beginning of a module
So I'm trying to use @safe, pure and nothrow. If I understand correctly Adam Ruppe's Cookbook, by putting @safe: pure: nothrow: at the beginning of a module, I distribute it on all definitions, right? Even methods, inner classes, and so on? Because I did just that on half a dozen of modules and the compiler did not complain. Does that mean my code is clean(?) or that what I did has no effect?
Re: Appender is ... slow
On Friday, 15 August 2014 at 16:48:10 UTC, monarch_dodra wrote: On Friday, 15 August 2014 at 14:40:36 UTC, Philippe Sigaud wrote: Well, I created a wrapper around a std.array.uninitializedArray call, to manage the interface I need Make sure you don't use that if your type has elaborate construction, or assumes a certain initial state (unless you are actually emplacing your objects of course). Hmm, what's elaborate construction? They are classes and have constructors, of course. I assumed that this produced only null's in the array. I thought that, having lots of concatenation in my code, that'd be a bottleneck. But it appears than pre-allocation does not give me any speed-up. If you are using raw GC arrays, then the raw append operation will, outweigh the relocation cost on extension. So pre-allocation wouldn't really help in this situation (though the use of Appender *should*) OK.
Re: @safe, pure and nothrow at the beginning of a module
In another module I marked as '@safe:' at the top, the compiler told me that a class opEquals could not be @safe (because Object.opEquals is @system). So it seems that indeed a module-level '@safe:' affects everything, since a class method was found lacking. (I put a @trusted attribute on it).
Re: Appender is ... slow
On Fri, Aug 15, 2014 at 10:04 PM, John Colvin via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: compiler, version, OS, architecture, flags? Compiler: DMD 2.065 and LDC 0.14 OS: Linux 64bits (8 cores, but there is no parallelism here) flags: -O -release -inline (and -noboundscheck for DMD) Have you looked at the assembly to check that all the Appender method calls are being inlined? I do not know how to look at the assembly, neither do I know how to see if Appender method calls are being inlined. I did spend some time with -profile and gained a nice 10% increase in speed with that, fighting bottlenecks in my code.
Re: Appender is ... slow
I don't know much about Phobos appender implementation details but the key thing with reusable buffer is avoid freeing them. AFAIR Appender.clear frees the allocated memory but `Appender.length = 0` does not, making it possible to just overwrite stuff again and again. I call .clear() only at the beginning of the computation, to avoid any strange effects with std.datetime.benchmark (using benchmark with memoizing functions can lead to strange results if one does not take care to flush any result between to calls.) After that initial call to clear, I just append. The thing is, it's not the first time it happens. For years, I tried to use Appender to get faster code, to no avail. btw, I saw your Dconf talk yesterday, nice content! And thanks for talking about Pegged! It might interest you to know that the code I'm trying to use Appender on is a new engine for Pegged, based on GLL parsing, that should be able to produce a parser for any grammar, even ambiguous ones.
Re: Appender is ... slow
I've never really tried to benchmark it, but it was my understanding that the idea behind Appender was to use it to create the array when you do that via a lot of appending, and then you use it as a normal array and stop using Appender. That's how I use it, yes. It sounds like you're trying to use it as a way to manage reusing the array, and I have no idea how it works for that. There is a misunderstanding there: I'm using clear only to flush the state at the beginning of the computation. The Appender is a class field, used by the class methods to calculate. If I do not clear it at the beginning of the methods, I keep appending new results to old computations, which is not what I want. But really, calling clear is a minor point: I'm interested in Appender's effect on *one* (long, concatenation-intensive) computation. I've never actually benchmarked it for just creating arrays via appending. I'd just assumed that it was faster than just using ~=, because that's what it's supposedly for. But maybe I just completely misunderstood what the point of Appender was. I don't know. People here keep telling newcomers to use it, but I'm not convinced by its results. Maybe I'm seeing worse results because my arrays are do not have millions of elements and Appender shines for long arrays?
Re: Appender is ... slow
There is a misunderstanding there: I'm using clear only to flush the state at the beginning of the computation. The Appender is a class field, used by the class methods to calculate. If I do not clear it at the beginning of the methods, I keep appending new results to old computations, which is not what I want. But really, calling clear is a minor point: I'm interested in Appender's effect on *one* (long, This is exactly what I propose to change - set `length` to 0 instead of calling `clear`. That way you will keep using same memory chunk simply abandoning its old state at the beginning of each computation. You mean by using the shrinkTo method? (Appender does not have a length, that's a bit of a bother btw). I just did, it does not change anything. Too bad. Heck, my code is simpler to read and use *and* faster by using a bog-standard D array. I'll keep my array for now :)
Re: Appender is ... slow
Thanks! Repeating what I have mentioned during DConf talk - have you ever considered proposing Pegged for Phobos inclusion? It feels like important bit of infrastructure to me. At the time, it was considered (rightfully) far too slow and memory-hogging. I think having a generic lexer and a standard D parser in Phobos would already be a great step forward.
Re: Are Delimited strings and HereDoc strings just here to suck ?
On Mon, Aug 11, 2014 at 10:09 PM, H. S. Teoh via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: On Mon, Aug 11, 2014 at 07:47:44PM +, Klaus via Digitalmars-d-learn wrote: I mean when writing a D lexer, you necessarly reach the moment when you think: Oh no! is this feature just here to suck ? The crazy variety of ways to write string literals in D, OTOH, *is* a bit over the top, as I found out myself when I also tried writing a D lexer. :-P Out of curiosity, how does a lexer deal with heredocs? It's a sort of... user-defined token, right?
Re: Are Delimited strings and HereDoc strings just here to suck ?
On Mon, Aug 11, 2014 at 10:58 PM, H. S. Teoh via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: In Flex, one way you can implement heredocs is to have a separate mode where the lexer is scanning for the ending string. So basically you have a sub-lexer that treats the heredoc as three tokens, one that defines the ending string for the heredoc (which is never returned to the caller), one that contains the content of the heredoc, and the terminating token (also never returned to the caller). Ah, a small, specialized sub-lexer. OK, I get it.
DConf 2014 talks slides
I'm catching up on some Dconf 2014 videos (great job, guys!) and wondering whether the associated slides will be put on the website somewhere. There are some where I'd like to ponder the code and ideas. It's not easy to do that on a stopped Youtube stream :) I know some links have been provided in the associated threads, but did all speakers give their slides to Andrei co? Alt question: Ali, what did you use to make your slides? I quite like the look of them and of the highlighted code. (The ability to cram 26 slides in 10 minutes is a nice bonus)
Re: How to easily construct objects with multi-param constructors from lazy ranges?
Yea, but that won't work for forward ranges. It only provides opIndex if the underlying range provides it. Since the chunk size is a runtime parameter it can't implement opIndex efficiently for non-random access ranges. But in your case, your range is random-access, no? Or else, you can always map array on the chunks... staticChunks was a bit of a misnomer. staticTake would be a better name. The range would contains a static array and pops that number of elements from the input range. Then, opIndex can easily be defined. What about takeExactly?
Re: Help with porting grammar from PEGjs to D for dustjs project!
On Wed, Aug 6, 2014 at 9:09 AM, Uranuz via Digitalmars-d-learn
digitalmars-d-learn@puremagic.com wrote:
What I was thinking about is possibility to change ParseTree struct with
user-defined version of it. And I was thinking about setting tree type as
template parameter to grammar:
grammar!(MyParseTree)(
Arithmetic:
...
);
That's already the case, look at
https://github.com/PhilippeSigaud/Pegged/blob/master/pegged/parser.d
on line 134, for example.
struct GenericPegged(TParseTree)
{
...
}
And then, the library alias a standard version (line 1531):
alias GenericPegged!(ParseTree).Pegged Pegged;
Which means the customer-facing Pegged parser is in fact a generic
parser specialized on ParseTree. But you can substitute your own parse
tree.
That's of course the same for all parsers: GenericXXX(TParseTree)
{...} and then alias XXX = GenericXXX!(ParseTree);
But you're right, it's not really documented and I don't have a
template checking whether TParseTree respect some static inferface.
This parameterization was asked by someone on github, but I don't
think they used it finally.
ParseTree is described here:
https://github.com/PhilippeSigaud/Pegged/wiki/Parse-Trees
Maybe we can continue this thread by private mail? I'm not sure people
on the D list are that interested by the internals of a library.
Re: Inner struct accessing host member
hmmm static and private... other keywords to try, but offhand it's been a while i don't know if either would change the behavior. Could just be inner scope limitations. Might be other tags/modifiers... I feel helpless :( No need to ;-) Thanks for your help, don't sweat it too much. I'm not sure if it would help, but sometimes if you reverse the logic you might get what you want by putting the data in B instead of A. I have a lot of Bs (nodes in a graph). They compute some things and when they get a result, they update A's field. Each A holds the entry point to their inner graph of Bs and waits for the results. So I don't see how I could invert it, really. What *could* do it to have the graph of Bs in thread and sending results as messages to another thread, where A is waiting for them. It's just... I'm so used to being able to mix and compose 'concepts' in D: structs in functions, classes in classes in structs, functions returning functions returning structs, etc. I'm used to begin able to organise my code as I see the problem space. But here, with a struct-in-a-struct, I hit a wall. Not fun, but not problematic too...
Re: Help with porting grammar from PEGjs to D for dustjs project!
Is there multiline comments available inside PEGGED template? As far as I understand inline comments are set via # sign. No, no multiline comment. That's based on the original PEG grammar, which allows only #-comments.
Re: How to easily construct objects with multi-param constructors from lazy ranges?
Some range which takes an at compile time known number of elements from an input range and provides opIndex seems perfect to me, but as far as I know there's no such thing in Phobos. There is chunks: http://dlang.org/phobos/std_range.html#chunks
Re: static array in templated struct/class
http://pastebin.com/34sbffSa
Your problem comes from lengthSquared:
public auto lengthSquared = function () = val.reduce!((a,b) = a + b*b);
That's an unusual way to define a method. Any reason why you are using
a pointer to a function as a member? Do you need to be able to
redefine it at runtime?
I guess that in this case, the compiler cannot determine its return
type and/or its size?
I'd use:
public auto lengthSquared () { return val.reduce!((a,b) = a + b*b);}
Inner struct accessing host member
I'd have thought that this would work:
struct A
{
int[] i;
B b;
struct B
{
void foo() { i ~= 1;}
}
}
void main()
{
A a;
a.b.foo();
}
But the compiler tells me 'need this for i of type int[]'.
Is there any way I can gain access on i inside B?
Re: Inner struct accessing host member
On Tue, Aug 5, 2014 at 11:37 PM, Martijn Pot via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: Does this help : http://www.digitalmars.com/d/archives/digitalmars/D/learn/Nested_struct_member_has_no_access_to_the_enclosing_class_data_38294.html Yes, that helps: that explains why it does not wor :). I changed my code to use classes. It's a bit less handy, but it works.
Re: Inner struct accessing host member
why it does not wor :). why it does not *work*, of course. Sigh.
Re: Inner struct accessing host member
On Tuesday, 5 August 2014 at 22:14:23 UTC, abanstadya wrote: programming Q, either youra newb or not, should rather be posted to 'http://forum.dlang.org/group/digitalmars.D.learn'. Your post appears on 'http://forum.dlang.org/group/digitalmars.D' which is more related to the lang. design rather to programming Q. Take care next time bro. This *is* D.learn, bro.
Re: Inner struct accessing host member
On Tuesday, 5 August 2014 at 23:47:00 UTC, Artur Skawina via
Digitalmars-d-learn wrote:
Is there any way I can gain access on i inside B?
Not directly, but as you ask for /any/ way -- yes:
struct B
{
void foo() { outer.i ~= 1; }
ref A outer() inout @property { return
*cast(A*)(cast(void*)this-A.b.offsetof); }
}
Note this will work only as long as you have just one B
instance in A and B is never created or copied outside of A.
OK. I have en entire graph, whose nodes are Bs inside A. So that
might not be totally appropriate for me. Thanks anyway, I always
forget about offsetof
Re: Inner struct accessing host member
Era: broken_b.foo(); //i_a is accessible invisibly because overridden or transformed assuming it would be converted or copied/moved as appropriate. return b; //if a is a local variable then b becomes invalid even though it's a struct. return i_b; //same as return b return broken_b; //same as above two cases. I see. I didn't know one could create an A.B 'outside'. I saw inner types as Voldemort types, but that is true only for inner structs in functions. Now a current way to make it safe while still leaving it structs could be passing a reference to either the outer struct or the variable in question. For simplicity it would probably be the struct. (...) Or less safe is to use a pointer and assign it when b instantiates to point back to A.. But if you pass B around without A and A goes out of scope... same problem... Maybe i'm over-thinking it. I already tried to propagate a ref through A's methods, but that made a mess: I have lots of methods, which have all to transmit this ref, only for *one* of them being able to update it. Thanks for you explanations :) I'm now using classes and inner classes. I'm not fond of classes, but that's working correctly.
Re: Threadpools, difference between DMD and LDC
Without going into much detail: Threads are heavy, and creating a thread is an expensive operation (which is partially why virtually every standard library includes a ThreadPool). I haven't looked into detail your code, but consider using the TaskPool if you just want to schedule some tasks to run amongst a few threads, or potentially using Fibers (which are fairly light-weight) instead of Threads. OK, I get it. Just to be sure, there is no ThreadPool in Phobos or in core, right? IIRC, there are fibers somewhere in core, I'll have a look. I also heard the vibe.d has them.
Re: Threadpools, difference between DMD and LDC
On Mon, Aug 4, 2014 at 2:13 PM, Chris Cain via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: OK, I get it. Just to be sure, there is no ThreadPool in Phobos or in core, right? There is. It's called taskPool, though: http://dlang.org/phobos/std_parallelism.html#.taskPool Ah, std.parallelism. I stoopidly searched in std.concurrency and core.* Thanks!
Re: Threadpools, difference between DMD and LDC
On Mon, Aug 4, 2014 at 3:36 PM, Dicebot via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: Most likely those threads either do nothing or are short living so you don't get actually 10 000 threads running simultaneously. In general you should expect your operating system to start stalling at few thousands of concurrent threads competing for context switches and system resources. Creating new thread is rather costly operation though you may not spot it in synthetic snippets, only under actual load. Modern default approach is to have amount of worker threads identical or close to amount of CPU cores and handle internal scheduling manually via fibers or some similar solution. That's what I guessed. It's juste that I have task that will generate other (linked) tasks, in a DAG. I can use a thread pool of 2-8 threads, but that means storing tasks and their relationships (which is waiting on which, etc). I rather liked the idea of spawning new threads when I needed them ;) If you are totally new to the topic of concurrent services, getting familiar with http://en.wikipedia.org/wiki/C10k_problem may be useful :) I'll have a look. I'm quite new, my only knowledge comes from reading the concurrency threads here, std.concurrency, std.parallelism and TDPL :)
Re: Threadpools, difference between DMD and LDC
On Mon, Aug 4, 2014 at 6:21 PM, Dicebot via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: vibe.d additions may help here: http://vibed.org/api/vibe.core.core/runTask http://vibed.org/api/vibe.core.core/runWorkerTask http://vibed.org/api/vibe.core.core/workerThreadCount task abstraction allows exactly that - spawning new execution context and have it scheduled automatically via underlying fiber/thread pool. However, I am not aware of any good tutorials about using those so jump in at your own risk. Has anyone used (the fiber/taks of) vibe.d for something other than powering websites?
Re: Threadpools, difference between DMD and LDC
On Mon, Aug 4, 2014 at 6:38 PM, Russel Winder via Digitalmars-d-learn
digitalmars-d-learn@puremagic.com wrote:
Are these std.concurrent threads or std.parallelism tasks?
A std.parallelism task is not a thread. Like Erlang or Java Fork/Join
framework, the program specifies units of work and then there is a
thread pool underneath that works on tasks as required. So you can have
zillions of tasks but there will only be a few actual threads working on
them.
That's it. Many tasks, a few working threads. That's what I'm
converging to. They are not particularly 'concurrent', but they can
depend on one another.
My only gripes with std.parallelism is that I cannot understand
whether it's interesting to use the module if tasks can create other
tasks and depend on them in a deeply interconnected graph. I mean, if
I have to write lots of scaffolding just to manage dependencies
between task, I might as well built it on core.thread and message
passing directly. I'm becoming quite enamored of message passing,
maybe because it's a new shiny toy for me :)
That's for parsing, btw. I'm trying to write a n-core engine for my
Pegged parser generator project.
Most likely those threads either do nothing or are short living
so you don't get actually 10 000 threads running simultaneously.
I suspect it is actually impossible to start this number of kernel
threads on any current kernel
So, what happens when I do
void doWork() { ... }
Tid[] children;
foreach(_; 0 .. 10_000)
children ~= spawn(doWork);
?
I mean, it compiles and runs happily.
In my current tests, I end the application by sending all thread a
CloseDown message and waiting for an answer from each of them. That
takes about 1s on my machine.
I have no current data, but it used to be that for a single system it
was best to have one or two more threads than the number of cores.
Processor architectures and caching changes so new data is required. I
am sure someone somewhere has it though.
I can add that, depending on the tasks I'm using, it's sometime better
to use 4, 6, 8 or 10 threads, repeatedly for a given task. I'm using a
Core i7, Linux sees it as an 8-core.
So, well, I'll try and see.
Re: Threadpools, difference between DMD and LDC
https://github.com/D-Programming-Language/phobos/pull/1910 Very interesting discussion, thanks. I'm impressed by the amount of work you guys do on github.
Threadpools, difference between DMD and LDC
I'm trying to grok message passing. That's my very first foray into this, so I'm probably making every mistake in the book :-) I wrote a small threadpool test, it's there: http://dpaste.dzfl.pl/3d3a65a00425 I'm playing with the number of threads and the number of tasks, and getting a feel about how message passing works. I must say I quite like it: it's a bit like suddenly being able to safely return different types from a function. What I don't get is the difference between DMD (I'm using 2.065) and LDC (0.14-alpha1). For DMD, I compile with -O -inline -noboundscheck For LDC, I use -03 -inline LDC gives me smaller executables than DMD (also, 3 to 5 times smaller than 0.13, good job!) but above all else incredibly, astoundingly faster. I'm used to LDC producing 20-30% faster programs, but here it's 1000 times faster! 8 threads, 1000 tasks: DMD: 4000 ms, LDC: 3 ms (!) So my current hypothesis is a) I'm doing something wrong or b) the tasks are optimized away or something. Can someone confirm the results and tell me what I'm doing wrong?
Threadpools, difference between DMD and LDC
I'm trying to grok message passing. That's my very first foray into this, so I'm probably making every mistake in the book :-) I wrote a small threadpool test, it's there: http://dpaste.dzfl.pl/3d3a65a00425 I'm playing with the number of threads and the number of tasks, and getting a feel about how message passing works. I must say I quite like it: it's a bit like suddenly being able to safely return different types from a function. What I don't get is the difference between DMD (I'm using 2.065) and LDC (0.14-alpha1). For DMD, I compile with -O -inline -noboundscheck For LDC, I use -03 -inline LDC gives me smaller executables than DMD (also, 3 to 5 times smaller than 0.13, good job!) but above all else incredibly, astoundingly faster. I'm used to LDC producing 20-30% faster programs, but here it's 1000 times faster! 8 threads, 1000 tasks: DMD: 4000 ms, LDC: 3 ms (!) So my current hypothesis is a) I'm doing something wrong or b) the tasks are optimized away or something. Can someone confirm the results and tell me what I'm doing wrong?
Re: Help with porting grammar from PEGjs to D for dustjs project!
Uranuz:
http://akdubya.github.io/dustjs/
So I need some help with rewriting grammar from PEGjs into PEGGED.
Is this the grammar?
https://github.com/akdubya/dustjs/blob/master/src/dust.pegjs
If so, then I think I can provide some help. But I don't get what
output you want (see below).
Also I don't understand in PEGGED (I have not tried to use it yet) how to
generate some logic from AST. Where should I describe it or should I walk
around all nodes for somehow and generate code for them.
You can put semantic actions in the grammar (code between curly
brackets). dust.pegjs seems
to have that in their grammar definition also (all these { return
something } blocks)
Or you can walk the parse tree afterwards.
See the Pegged tutorial here:
https://github.com/PhilippeSigaud/Pegged/wiki/Pegged-Tutorial
More particularly:
https://github.com/PhilippeSigaud/Pegged/wiki/Using-the-Parse-Tree
The example explains (I hope) how to use a wiki grammar to parse wiki
text and output LaTeX code.
Goal of this is to use dust template system as template engine at server
side.
More concretely, what's the goal? A template as input and... what
should be output? If I understand correctly, dustjs produces
Javascript code. Is that what you want? Or do you want D code?
Also, did you have a look at vide.d and its Diet templates?
Re: Threadpools, difference between DMD and LDC
This is correct – the LLVM optimizer indeed gets rid of the loop completely. OK,that's clever. But I get this even when put a writeln(some msg) inside the task. I thought a write couldn't be optimized away that way and that it's a slow operation? Anyway, I discovered Thread.wait() in core in the meantime, I'll use that. I just wanted to have tasks taking a different amount of time each time. I have another question: it seems I can spawn hundreds of threads (Heck, even 10_000 is accepted), even when I have 4-8 cores. Is there: is there a limit to the number of threads? I tried a threadpool because in my application I feared having to spawn ~100-200 threads but if that's not the case, I can drastically simplify my code. Is spawning a thread a slow operation in general?
Re: Help with porting grammar from PEGjs to D for dustjs project!
On Mon, Aug 4, 2014 at 7:13 AM, Uranuz via Digitalmars-d-learn
digitalmars-d-learn@puremagic.com wrote:
I am real noob about grammar description languages so I need some
explanation about it. As far as I understand expressions in curly bracers
are used to modify syntax tree just in process of parsing instead of
modifying it after?
Yes, that's it. Or at least that's the way Pegged does it. I'm not
such a specialist myself, I just dabbled in it to write Pegged.
As I understand it, many parsers do not produce a parse tree, they
only 'return' what their embedded action tell them to.
Personally, I have a slight preference to using the parse tree once
it's complete. a) Because that way I have access to all the
information (parent nodes, sibling nodes, even far way) whereas when
doing it with an action means you only have the local, current node
context and b) because it decouples steps that are in my mind
separated anyway: parsing, then producing a value out of the parse
tree. A bit like ranges in D, and Walter and his components
programming speech.
How I could use PEGGED to map some code to these parsed expressions to
generate code that will perform operations defined by this grammar? Should I
walk around all the syntax tree and just append code to some string and mix
it in then or are there some features for code generation?
You can insert semantic actions inside the grammar definition, as is
done by Dustjs, or you can have a function walking the tree
afterwards.
For D, since Pegged works at compile time, you can have your parse
tree at compile time. Use the walking function to generate D code (a
string) and mix it in.
enum parseTree = Grammar(input); // CT parsing
string codeMaker(ParseTree pt) { ... } // recursive walker
mixin(codeMaker(parseTree));
See https://github.com/PhilippeSigaud/Pegged/wiki/Using-the-Parse-Tree
If what you need is generating Javascript code, no need to do that at
compile-time: you can assemble the JS code as a string at runtime and
then write it to a file somewhere, I guess.
Something that I was thinking about is comparision of resulting syntax tree
to check if it was correctly implemented. It would be great if different
gramar parsers will output result in common format (JSON or XML for example)
and it will be possiple to compare them for equality. But different parsers
have different internal format of tree so maybe creating some transformation
is possible. With this feature it could be possible to say if parser is
working correctly.
Different formats and also different languages. I don't see how you
can compare a parse tree that's a D object and another tree made by
dustjs: you never see the AST produced by dust, you only see the
resulting JS code.
Re: Type deduction on templated constructor.
I expected such an answer and I do understand the decisions behind it. Yet, you gave me a really GOOD news! Having to write cast(ubyte) 1 was way too much verbose for my liking, while the new ubyte(1) is reasonable enough. Why not use `1u`?
Re: Type deduction on templated constructor.
On Wed, Jul 30, 2014 at 11:46 AM, Philippe Sigaud philippe.sig...@gmail.com wrote: I expected such an answer and I do understand the decisions behind it. Yet, you gave me a really GOOD news! Having to write cast(ubyte) 1 was way too much verbose for my liking, while the new ubyte(1) is reasonable enough. Why not use `1u`? Ow! Ignore that. 1u is an uint, not an ubyte.
Re: Compile time regex matching
I did, and I got it to work. Unfortunately, the code used to in the CTFE is left in the final executable even though it is not used at runtime. So now the question is, is there away to get rid of the excess baggage? Not that I know of. Once code is injected, it's compiled into the executable. auto result = MyRegex(import(config-file.txt)); // compile-time parsing return writeln(\~result.matches[0]~\);; mixin(get_match()); I never tried that, I'm happy that works. Another solution would be to push these actions at runtime, by using a small script instead of your compilation command. This script can be in D. - The script takes a file name as input - Open the file - Use regex to parse it - Extract the values you want and write them to a temporary file. - Invoke the compiler (with std.process) on your main file with -Jpath flag to the temporary file. Inside your real code, you can thus use mixin(import(temp file)) happily. - Delete the temporary file once the previous step is finished. Compile the script once and for all, it should execute quite rapidly. It's a unusual pre-processor, in a way.
Re: new properties for basic types
Halas, that's not what the OP wants. He needs properties on the *type* itself: int.foo instead of foo!int. So no, this is not possible.
Re: Compile time regex matching
I am trying to write some code that uses and matches to regular expressions
at compile time, but the compiler won't let me because matchFirst and
matchAll make use of malloc().
Is there an alternative that I can use that can be run at compile time?
You can try Pegged, a parser generator that works at compile-time
(both the generator and the generated parser).
https://github.com/PhilippeSigaud/Pegged
docs:
https://github.com/PhilippeSigaud/Pegged/wiki/Pegged-Tutorial
It's also on dub:
http://code.dlang.org/packages/pegged
It takes a grammar as input, not a single regular expression, but the
syntax is not too different.
import pegged.grammar;
mixin(grammar(`
MyRegex:
foo - abc* def?
`));
void main()
{
enum result = MyRegex(abcabcdefFOOBAR); // compile-time parsing
// everything can be queried and tested at compile-time, if need be.
static assert(result.matches == [abc, abc, def]);
static assert(result.begin == 0);
static assert(result.end == 9);
pragma(msg, result.toString()); // parse tree
}
It probably does not implement all those regex nifty features, but it
has all the usual Parsing Expression Grammars powers. It gives you an
entire parse result, though: matches, children, subchildren, etc. As
you can see, matches are accessible at the top level.
One thing to keep in mind, that comes from the language and not this
library: in the previous code, since 'result' is an enum, it'll be
'pasted' in place everytime it's used in code: all those static
asserts get an entire copy of the parse tree. It's a bit wasteful, but
using 'immutable' directly does not work here, but this is OK:
enum res = MyRegex(abcabcdefFOOBAR); // compile-time parsing
immutable result = res; // to avoid copying the enum value everywhere
The static asserts then works (not the toString, though). Maybe
someone more knowledgeable than me on DMD internals could certify it
indeed avoid re-allocating those parse results.
Re: Compile time regex matching
On Mon, Jul 14, 2014 at 3:19 PM, Artur Skawina via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: On 07/14/14 13:42, Philippe Sigaud via Digitalmars-d-learn wrote: asserts get an entire copy of the parse tree. It's a bit wasteful, but using 'immutable' directly does not work here, but this is OK: enum res = MyRegex(abcabcdefFOOBAR); // compile-time parsing immutable result = res; // to avoid copying the enum value everywhere static immutable result = MyRegex(abcabcdefFOOBAR); // compile-time parsing Ah, static! The static asserts then works (not the toString, though). Maybe (snip diff) I'll push that to the repo, thanks! I should sprinkle some const and pure everywhere... [completely untested; just did a git clone and fixed the two errors the compiler was whining about. Hmm, did pegged get faster? Last time i tried (years ago) it was unusably slow; right now, compiling your example, i didn't notice the extra multi-second delay that was there then.] It's still slower than some handcrafted parsers. At some time, I could get it on par with std.regex (between 1.3 and 1.8 times slower), but that meant losing some other properties. I have other parsing engines partially implemented, with either a larger specter of grammars or better speed (but not both!). I hope the coming holidays will let me go back to it.
Re: new properties for basic types
Hmm.
So how do I use stuff like this:
template defaultInit(T)
{
static if (!is(typeof({ T v = void; })))// inout(U)
@property T defaultInit(T v = T.init);
else
@property T defaultInit();
}
(this is from std.traits - ok, it's private, but anyway)
It should be invoked as `defaultInit!SomeType`
Because I have seen nowhere anything like defaultInit!T (or T.defaultInit)
and don't understand why here the attribute @property is used.
Why does it make a difference, and how?
@property allows you to call a function without the parenthesis (), to
imitate a field in a struct or class.
In this particular case, I don't know what defaultInit is used for. It
seems to compile to a forward declaration of a function, but I don't
know what for.
I cannot find it on my copy of std.traits. What DMD version are you using?
Re: Opinions: The Best and Worst of D (for a lecture/talk I intend to give)
On Tue, Jul 8, 2014 at 7:50 AM, H. S. Teoh via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote quite a wall of text Wow, what to add to that? Maybe you scared other from participating ;-) * I'll second metaprogramming: the alliance between templates, CTFE and mixins is really nice. It's *the* feature (or triplet of features) I think of when Walter says that many D parts are kind of what for? in isolation but really grow into something awesome by using one another. * I'd add static introspection to the mix: using static if, __traits(...) and is(...), clunky as the syntax is (there, one 'ugly' thing for you), is easy and very powerful: the idea to have code selecting its flow or extracting information (is that a static array? Does this aggregate have an '.empty' method?). This is the basis for std.algorithm idiom of 'static interface' which allows us compile-time duck typing, which I find very pleasing. * unittests are nice of course, H. S. Teoh already said it expressively enough :-) * I'd add arrays and slice. They are wonderfully simple to use, efficient, etc. I remember learning D by translating the Euler Project problems from C++ to D and it was a joy. Which brings me to another feature I like: ranges. The idea is nothing new, but I find it quite well-realized in D, far easier than other compiled languages alternatives and since they are pervasive in the library, you can really plug components into one another nicely. For example: http://wiki.dlang.org/Component_programming_with_ranges#Case_Study:_Formatting_a_Calendar * dub is good, and you can show code.dlang.org in your presentation. * Bonus point: different compilers. I like that. I regularly use at least two in parallel while developping (comparing results, speed, etc). Kudos to all the guys involved! As for the bad and ugly: * it's frustrating not to have a big collection module in Phobos, but then I didn't propose one, so I'll shut up. * there are still some not so nice interaction between const/shared/inout/ auto ref, though I rarely hit them these days * The compiler still has some quirks: I find it strange you can put unused qualifiers in many places. But very honestly, it was already a joy to use a few years ago, and it's becoming better and better.
Re: dependency graph
If you compile your project with the -deps flag, the compiler will output import dependencies. With -deps=filename, the output will go into filename. From there, you'll have to parse and create the graph, though. Maybe have a look into rdmd source, to see how the dependency extraction is done there?
Re: What exactly module in D means?
On Sat, Jul 5, 2014 at 6:35 PM, Andre Tampubolon via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: I've been reading the newsgroup for a while, and it seems that one of the reason folks like D is because it supports module. My question is: what does module mean? A quick google pointed my this page: http://dlang.org/module.html. Still cannot understand it, though :) How does it differ from the old C's #include? Well, module are 'more first-class' in D than in C. By this, I mean that they have an identity: a name, members, symbols. You can see them as a enormous struct/class. You can use introspection to obtain the symbols defined in a module, query their types, etc. Heck, in some cases, module names can even by used as arguments for templates. As others said, a D module is a unit of encapsulation: inside a module, functions, structs and classes can freely call one other and access their internal fields. Outside the module, only what's not private is visible. Thus, you can create complex, interdependent structures inside your module, but export only some parts for public access. This way, you control a module 'interface', its external frontier. Importing a module in D means the public symbols defined in the module become accessible in the importer. There is no need to import all visible symbols in your current scope: you can restrict your import to only one or two symbols, rename them, etc. That's much more powerful than C when it comes to name clashes. Different 'downstream' modules can import the same 'upstream' module, and choose to import different symbols. No need to do a big 'import everything in the current scope'. symbols names are also defined on a per-module basis: two different modules can have members with a similar name, without clash: just prefix the names with the module name to differentiate them (or rename one while importing). Also, `import foo;` is a statement, you can put it everywhere a statement is legit. That's a recent addition (ie, 1-2 year?), that Andrei likes to call 'Turtles all the way down' and is quickly becoming a D idiom: import a symbol near its use place: inside a function, a method, even inside an if branch: that avoid scope pollution and nicely documents where a symbol is necessary.
Re: How to test templates for equality?
Seeing his example, the OP wants a solution that works even for templates:
template Test1(T) {}
pragma(msg, instanceArgsOf!(Test1, Test1!int));
which fails because Test1!int is not a type (std.traits.isInstanceOf
fails also, for the same reason).
And is(symbol == Name!Args, Args...) does not work if Name!Args and
symbol are not types.
In this particular case, the only solution I know of is an awful hack:
using .stringof and __traits(identifier, x) and then parse the
strings:
Name!(int, double[string]) and Name(T, U[V], U, V)
and then the fun begins: in the general case, you must then equate the
arguments lists (recursively).
Philippe
Re: Trying to reproduce Node.js async waterfall pattern.. (Meta-programming)
Yes, the final callback is always called, but if an error is passed to the
callback by any of the main steps in the sequence ladder, it will
immediately jump to the final callback and not execute further steps.
OK.
What do you mean? The compiler does deduce the type of Funcs.
If you look at where I call Waterfall() in main, you'll see I had to
manually specify (Callback cb) instead of just (cb); since it didn't know
that the Funcs... were of type AsyncFunc
you can use std.typetuple.allSatisfy with a helper template:
enum isAsyncFunc(T) = is(T == AsyncFunc);
... (Funcs...)(Funcs funcs) if (allSatisfy!(isAsyncFunc, Funcs))
{ ... }
(cb) { cb(null, one);} is possible, but that means it's a function
template, not a function.
You can get this syntax by making the callbacks template arguments,
which means they must be known at compile-time. Is that OK with you or
do you need the possibility to define the callbacks at runtime?
The goal was to do as much as possible at compile time. Could you elaborate
on this a bit. I guess the answer is, yes, it's okay with me.
I mean, it's possible to get a (cb){ some code } syntax, but as that
define a function template, it has no real type: it cannot be a
runtime argument, only an alias template parameter. That means that
all callbacks must be defined in your code, none can come from user
input or runtime computation.
I don't get it: none of your callbacks have a return type per se: they
all return 'void'.
Do you want callbacks that really return something?
Yes, the callbacks at step0 should output a type in the result which is
specific to step0, and then that type should be fed in as a secondary
parameter to step1.
I didn't get that far, as I was already stuck.
OK, I get it.
thanks for your patience!
Well, we are there to explain :-)
I'll have try and code something. If I can get it to work, I'll post it there.
Re: Trying to reproduce Node.js async waterfall pattern.. (Meta-programming)
On Mon, Jun 23, 2014 at 9:39 PM, Christian Beaumont via
Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote:
Each function is given a callback, that when called, steps the waterfall
forward on to the next function to process. If an error is passed to the
callback (instead of null), then the waterfall stops processing and calls
the final callback at the end of the chain.
Just to be sure: whether or not an error is passed to a callback, the
final callback is always called?
I mean, the last callback could also be called *only when something
fails*, a bit like a default case in a switch, or an error-handling
routine.
1) I can't see any way to get the compiler to deduce the type of Funcs
What do you mean? The compiler does deduce the type of Funcs.
I had an urge to somehow specialize the variadic Funcs... but I couldn't
figure out any syntax to do that.
What do you mean by 'specialize'?
Well, because of that, I have to
repeatedly say (Callback cb) instead of (cb).
(cb) { cb(null, one);} is possible, but that means it's a function
template, not a function.
You can get this syntax by making the callbacks template arguments,
which means they must be known at compile-time. Is that OK with you or
do you need the possibility to define the callbacks at runtime?
2) I can't see a path to flow the output type of the previous callback to
the input TLastResult for the next... so it's stuck on string :(
I don't get it: none of your callbacks have a return type per se: they
all return 'void'.
Do you want callbacks that really return something?
3) I had to use a specialization to deal with the head case; (the first
function doesn't have an input from a previous result). Minor, but niggly.
You can test if func[0] takes one or two arguments:
import std.traits: isCallable, ReturnType;
static if (isCallable!(Func[0]) ReturnType!(Func[0]).length == 1)
Re: What is the correct way to forward method calls to the struct field?
On Sun, Jun 22, 2014 at 5:04 PM, monnoroch via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: There is also a problem: when i declare opDispatch to be private, i still have access to this forwarding from another package. Is it a bug or what? I don't know. I never used private in conjunction with a template. Let's hope someone more knowledgeable than me can answer this.
Re: What is the correct way to forward method calls to the struct field?
On Sun, Jun 22, 2014 at 5:02 PM, monnoroch via Digitalmars-d-learn
digitalmars-d-learn@puremagic.com wrote:
Thanks a lot!
There is a problem though: when i pass incorrect parameters to
such a method, it says, that S has no such field, which is a
misleading error message.
You can test the mixin with static if, like this:
struct SomeT {
int foo(double d) { return 0;}
void bar(double d) { return;}
}
struct S {
auto opDispatch(string name, Args...)(Args args) {
static if (is(typeof(mixin((*iface). ~ name)(args
return mixin((*iface). ~ name)(args);
else
pragma(msg, S. ~ name ~ cannot be called with arguments of
type ~ Args.stringof);
}
SomeT** iface;
}
void main()
{
SomeT st;
SomeT* st_p = st;
SomeT** st_p_p = st_p;
S s = S(st_p_p);
s.foo(3.14);
s.foo(abc);
s.bar(abc, 3.14);
}
Re: template mixins for boilerplate
Out of curiosity, why use a mixin template containing a string mixin instead of, well, directly injecting a string mixin in your struct, with a function?
Re: template mixins for boilerplate
On Sat, Jun 21, 2014 at 4:24 PM, Dicebot via Digitalmars-d-learn
digitalmars-d-learn@puremagic.com wrote:
On Saturday, 21 June 2014 at 13:45:14 UTC, Philippe Sigaud via
Digitalmars-d-learn wrote:
Out of curiosity, why use a mixin template containing a string mixin
instead of, well, directly injecting a string mixin in your struct,
with a function?
Hiding non-hygienic implementation behind a more reliable interface.
In what way is a template more reliable than the equivalent function?
That's an honest question: I'm just looking at the code a few posts
upward:
mixin template Function(string name) {
mixin(public static int ~ name ~ () { return other.module. ~
name ~; });
}
struct S {
mixin Function!fctn1;
}
And this double-mixin construction seems needlessly complicated to me,
compared to:
string Function(string name) {
return public static int ~ name ~ () { return other.module.
~ name ~; };
}
struct S {
mixin(Function(fctn1));
}
