Re: d2 file input performance
Am 04.09.2011, 19:01 Uhr, schrieb Christian Köstlin christian.koest...@gmail.com: On 9/3/11 7:53 , dennis luehring wrote: Am 26.08.2011 19:43, schrieb Christian Köstlin: Hi guys, i started the thread: http://stackoverflow.com/questions/7202710/fastest-way-of-reading-bytes-in-d2 on stackoverflow, because i ran into kind of a problem. i wanted to read data from a file (or even better from a stream, but lets stay with file), byte-by-byte. the whole thing was part of my protobuf implementation for d2, and there you have to look at each byte to read out the varints. i was very proud of my implementation until i benchmarked it first against java (ok ... i was a little slower than java) and then against c++ (ok ... this was a complete different game). after some optimizing i got better, but was still way slower than c++. so i started some small microbenchmarks regarding fileio: https://github.com/gizmomogwai/performance in c++, java and d2. could you help me improve on the d2 performance? i am sure, that i am missing something fundamental, because i thing it should be at least possible be equal or better than java. thanks in advance christian i would change the test szenario a little bit 1. use a ramdisk - so stuff like location on disk, fragmentation, driver speed will reducued down to a little bit of noise 2. make your szenario much bigger 3. would be interesting to see for example to cumulated every 1000 benchmarks-steps or something like that - to see caching coming in etc. running 10.000 times time for 1. 1000 steps xyzw time for 2. 1000 steps xyzw time for 3. 1000 steps xyzw time for 4. 1000 steps xyzw overall time ... xyz ... good point ... will see if i can adapt the tests... cK -release -O -inline -noboundscheck is the options set for D2. In D1 -release included -noboundscheck.
Re: Implicit cast to immutable
Steven Schveighoffer schvei...@yahoo.com wrote in message
news:op.v3h06olweav7ka@localhost.localdomain...
That sounds like an incorrect restriction. The implicit cast to immutable
should depend on whether the function being *called* qualifies, not if the
function you are calling *from* qualifies.
I think you've misunderstood what I'm saying. The patch I made implemented
two ways to implicly convert to immutable: the result of a pure function
returning immutable and a return statement inside a pure function.
1)
int[] fun() { return new int[]; }
immutable x = fun(); // conversion happens here
2)
immutable(int[]) fun() { return new int[]; } // conversion happens here
immutable x = fun();
Bearophile's example is of the second, where it definately matters what the
purity of the function is.
Qualifying means the return type should be mutable, and cannot be derived
from the parameters without requiring casts. The easiest way to do this
is to ensure the parameters are all const, immutable, or implicitly cast
to immutable. You could do funky things like assume for instance an int[]
cannot possibly be implicit-casted to a char[], so therefore int[]
foo(char[] data) pure can be implicitly casted to immutable, but that
might be flirting with dangerous situations.
The one exception should be allocating memory, which should always
qualify, even though it's not a pure function.
This is valid, but becomes very complicated with complex return types. I
doubt this will ever make it into the language.
I've got the beginnings of a patch to enable a sort of 'pure expression',
such as new, array.dup and array concatenation expressions. The result of a
call to a const-pure function using immutable arguments can be converted to
immutable, while calling it with mutable or const arguments cannot, without
searching the return type for anything the arguments can implicitly convert
to (or create).
Eg. I can't see a great way to detect situations like this:
struct S { const void* p; }
S[] fun(int[] arr)
{
return [ S(arr.ptr) ];
}
immutable x = fun([1, 2, 3]);
Re: Implicit cast to immutable
Daniel Murphy:
2)
immutable(int[]) fun() { return new int[]; } // conversion happens here
immutable x = fun();
Bearophile's example is of the second, where it definately matters what the
purity of the function is.
This is the enhancement request I have written days ago:
http://d.puremagic.com/issues/show_bug.cgi?id=6783
Bye,
bearophile
Re: Looking for documentation of D's lower-level aspects.
ahead = n._next; The C/C++ equivalent of this is `ahead = n-next;`, or equivalently `ahead = (*n).next;`. This is a difference in semantics from C/C++ with respect to the `.`---it seems like D turns pointer to struct property accesses into property access with indirection. Yes. It was really dumb to introduce that in C back then cause you can't easily change from a pointer to a class to a real class without editing all places where it is accessed. D chose the sane and safer way of letting the compiler figure out what to do. Nowhere that I can recall in Alexandrescu's book talked about this, but it's a really big deal! I can't recall where I read about it back then, but I did know it soon after I had started learning D. Some of the differences to C/C++ are explained there: http://www.d-programming-language.org/ctod.html Though it could use an overhaul. As for getting rid of the GC, it is theoretically possible. But nobody has put much effort into making it work yet (cause the only application platform is still x86/64). I guess it will become necessary though once D conquers ARM (we can generate code for it with LDC/GDC but druntime isn't ready).
Re: Looking for documentation of D's lower-level aspects.
On 2011-10-18 12:50, Trass3r wrote: ahead = n._next; The C/C++ equivalent of this is `ahead = n-next;`, or equivalently `ahead = (*n).next;`. This is a difference in semantics from C/C++ with respect to the `.`---it seems like D turns pointer to struct property accesses into property access with indirection. Yes. It was really dumb to introduce that in C back then cause you can't easily change from a pointer to a class to a real class without editing all places where it is accessed. D chose the sane and safer way of letting the compiler figure out what to do. Nowhere that I can recall in Alexandrescu's book talked about this, but it's a really big deal! I can't recall where I read about it back then, but I did know it soon after I had started learning D. Some of the differences to C/C++ are explained there: http://www.d-programming-language.org/ctod.html Though it could use an overhaul. As for getting rid of the GC, it is theoretically possible. But nobody has put much effort into making it work yet (cause the only application platform is still x86/64). I guess it will become necessary though once D conquers ARM (we can generate code for it with LDC/GDC but druntime isn't ready). The GC can be replaced at link time. Tango contains an example of a GC that uses malloc, should work with druntime as well. Another option is to not link a GC at all, then there will linker errors when using something that needs the GC. -- /Jacob Carlborg
Re: Implicit cast to immutable
bearophile bearophileh...@lycos.com wrote in message
news:j7jepi$prp$1...@digitalmars.com...
Daniel Murphy:
2)
immutable(int[]) fun() { return new int[]; } // conversion happens here
immutable x = fun();
Bearophile's example is of the second, where it definately matters what
the
purity of the function is.
This is the enhancement request I have written days ago:
http://d.puremagic.com/issues/show_bug.cgi?id=6783
Bye,
bearophile
Yes, and the problem in that report is that the function is const-pure, not
strong-pure.
Without checking if the return type can contain a non-immutable reference
from the arguments, it is not safe to implicitly convert the result to
immutable.
eg.
immutable(int[]) foo(in int[] x) { return x; }
auto g = [1, 2, 3];
auto a = foo(g.idup); //safe
auto b = foo(g); // unsafe
Checking at the call site is possible, but not from inside the function.
int[] foo(in int[] x) { return new int[](3); }
auto g = [1, 2, 3];
immutable a = foo(g.idup); // safe
immutable b = foo(g); // unsafe, and easily rejected
In your example, it is safe as the argument is not returned. Allowing this
in the general case requires checking (recursively) that the return type
does not contain any types that any of the arguments can implicitly convert
to that are non-immutable.
Re: FastCGI binding or implementation?
On Mon, Oct 17, 2011 at 2:11 PM, Jacob Carlborg d...@me.com wrote: On 2011-10-17 16:01, Andrea Fontana wrote: I handle request on different threads. I do some pre-processing on scgi data and I fill a struct: request.get[] request.post[] request.cookie[] request.headers[string] then I call a virtual function (to override on subclasses) like: do(request, output); where user fill output struct in a way like: output.data ~= htmlbodyh1hello world/h1/body/html; output.status = 200 output.cookies = bla bla and then if is method != head i send headers + data, else just headers. btw 99% of usage is get, post, head. Yes, but if you want to write a web site that is RESTful you need the other HTTP methods as well, at least PUT and DELETE. BTW, what about creating something like Rack but for D. Rack is a low level interface in front of the web server which web frameworks can be built on top. http://rack.github.com/ -- /Jacob Carlborg Yes, this is exactly why I was wondering whether FastCGI had been implemented (though SCGI works for me as well) - so that I could write something on top of it, in much the same way I would using (for example) WSGI in Python. I also agree with you re: supporting all of the HTTP methods. Just because the most common ones are GET, POST, and HEAD doesn't mean we should leave out the others; both PUT and DELETE are quite useful. Best regards, Jeremy Sandell
Re: FastCGI binding or implementation?
I've been having trouble with my news postings, so forgive me if I said this before. But my cgi.d module supports FastCGI via the C library, with the same D interface as normal CGI or an embedded server: https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff Just compile with -version=fastcgi to use that.
Re: FastCGI binding or implementation?
On 18.10.2011 19:24, Jeremy Sandell wrote:
On Mon, Oct 17, 2011 at 2:11 PM, Jacob Carlborg d...@me.com
mailto:d...@me.com wrote:
On 2011-10-17 16:01, Andrea Fontana wrote:
I handle request on different threads. I do some pre-processing on
scgi data and I fill a struct:
request.get[]
request.post[]
request.cookie[]
request.headers[string]
then I call a virtual function (to override on subclasses) like:
do(request, output);
where user fill output struct in a way like:
output.data ~= htmlbodyh1hello world/h1/body/html;
output.status = 200
output.cookies = bla bla
and then if is method != head i send headers + data, else just
headers.
btw 99% of usage is get, post, head.
Yes, but if you want to write a web site that is RESTful you need
the other HTTP methods as well, at least PUT and DELETE.
BTW, what about creating something like Rack but for D. Rack is a
low level interface in front of the web server which web frameworks
can be built on top.
http://rack.github.com/
--
/Jacob Carlborg
Yes, this is exactly why I was wondering whether FastCGI had been
implemented (though SCGI works for me as well) - so that I could write
something on top of it, in much the same way I would using (for example)
WSGI in Python.
I also agree with you re: supporting all of the HTTP methods. Just
because the most common ones are GET, POST, and HEAD doesn't mean we
should leave out the others; both PUT and DELETE are quite useful.
Best regards,
Jeremy Sandell
Adam D. Ruppe has a wrapper for libfcgi at github. I started
implementing fcgi, but it's basically just a very, very limited
ugly-hack prototype. Doubt you'll get much use of it, but I'll attach it
here anyway.
module fcgi;
import std.socket, std.stdio, std.socketstream;
import std.datetime;
import std.concurrency;
import core.thread;
import std.process;
import std.conv;
private import std.c.windows.windows, std.c.windows.winsock;
///
// PROTOCOL END
///
S toStruct(S, T)(T[] buf) {
static ubyte[S.sizeof] buf2;
assert(buf.length = S.sizeof);
buf2 = buf[0..S.sizeof];
return cast(S)(buf2);
}
/// Who is responsible for closing the socket?
enum SocketLifetimeOwner {
application = 1,
server = 2
}
alias void delegate(Request) RequestCallback;
alias void delegate(ushort, int, ProtocolStatus) EndRequestHandler;
class Request {
RequestId _id;
Params _params;
Role_role;
SocketLifetimeOwner _socketOwner;
EndRequestHandler _endRequestHandler;
ubyte[] _input;
ListenThread _server;
bool _ended;
this(RequestId id, ListenThread server, EndRequestHandler
endRequestHandler) {
_id = id;
_endRequestHandler = endRequestHandler;
_server = server;
}
@property ubyte[] input() pure nothrow @safe {
return _input;
}
@property RequestId id() pure const nothrow @safe {
return _id;
}
@property void id(RequestId id) pure nothrow @safe {
_id = id;
}
@property const(Params) params() pure const nothrow @safe {
return _params;
}
private @property void params(Params params) pure nothrow @safe {
_params = params;
}
@property Role role() pure const nothrow @safe {
return _role;
}
private @property void role(Role role) pure nothrow @safe {
_role = role;
}
@property SocketLifetimeOwner socketOwner() pure const nothrow @safe {
return _socketOwner;
}
@property void socketOwner(SocketLifetimeOwner owner) pure nothrow @safe {
_socketOwner = owner;
}
void write(T)(const(T[]) data, RecordType method = RecordType.stdOut) {
assert(!_ended);
// Break into chunks of 65535
auto socket = _server._active;
auto head = RecordHead(method, _id,
cast(ushort)(data.length));
socket.send(cast(ubyte[RecordHead.sizeof])head);
socket.send(cast(ubyte[])data);
head.contentLength = 0;
socket.send(cast(ubyte[RecordHead.sizeof])head);
}
void end() {
_endRequestHandler(_id, 0, ProtocolStatus.requestComplete);
_ended = true;
}
}
class Params {
private string[string] _params;
private this(string[string] params) {
_params = params;
}
private this(in ubyte[] raw) pure nothrow @trusted {
int i;
do {
auto keyLen = cast(ubyte) raw[i++];
auto valLen = cast(ubyte) raw[i++];
auto key= cast(string)raw[i .. i+keyLen];
i += keyLen;
auto value = cast(string)raw[i .. i+valLen];
i += valLen;
_params[key] = value;
} while(i raw.length);
}
string opDispatch(string key)() const pure nothrow @safe {
Re: Looking for documentation of D's lower-level aspects.
Sean Silva Wrote: I have just finished reading Alexandrescu's The D Programming Language, but it doesn't seem to talk at all about how to use D as a stand-in for C/C++ almost at all. E.g., the part of D that doesn't depend on a runtime or garbage collector. There isn't any real documentation on this. Kind of stuff. Instead you can find some links that try to teach D for C programmers. http://www.prowiki.org/wiki4d/wiki.cgi?ComingFrom/PlainC Also, in terms of programming without a GC, very little has been done on this front to make it easy. There has been talk about a compiler switch to help with it and a stubbed GC. Right now D isn't ready to be used in this fashion, imo, but someone is more than welcome to run with it and make it a reality. But doing low-level things with the GC is still possible and similar to C.
Re: FastCGI binding or implementation?
Tale time, only tangentially on topic.
Today, I was coincidentally switching one of my work apps from
standard CGI to Fast CGI.
Almost trivial. Set up Apache, then build the program with
-version=fastcgi. Done.
Well, not 100% done. I had a piece of static data in the app
that worked correctly before but now means subsequent requests
were out of it.
Changed that to an instance variable, and boom, works perfectly.
* If you are using Fast CGI, avoid static variables.
The next thing is speed. From principles, there's very little
reason for FastCGI to actually be faster than normal CGI - the
startup costs are insignificant next to the total app runtime
and network lag. (Startup is maybe 5 ms on the live server,
with runtime close to 50ms and ping to the user another 100ms.
The cost of CGI is roundoff error in the actual deployment.)
My benchmarks supported this for the kind of loads we had before.
But now, the number of concurrent users is picking up. The CGI
still performed very well, though every so often, users complained
about lag on some resources.
I ran a benchmark comparing cgi to fast cgi with a very large
number of concurrent users.
It showed better availability and about a 15% speed boost under
this load. Since Apache restarts it when it segfaults, reliability
ought not to be affected, though it's too soon to say for sure.
So, I changed the makefile to say -version=fastcgi and soon
realized I must search for static variables - found just one,
so easy fix, and we're up on fastcgi.
... but that 15% in the benchmark hasn't translated to a big change
in the live environment yet. Been several hours now, and we've
been trying to force the availability issue, and failed so far.
Looks like a win, but not a very big one. Speed on the whole -
unaffected. The difference is roundoff error once you factor in
network lag and such again.
So, how can we speed up the application? The key here is client
side caching.
Using my cgi.d, there's a function:
cgi.setCache(true);
which tells it simply to cache the response forever. It makes
an expiration date long in the future.
Set that for any content which changes infrequently - css,
javascript, images, any kind of (conceptually) pure or static data,
etc.
Now, your code doesn't run again and the user doesn't hit the
network again. What was 150ms is now 1ms. The users will feel
the difference.
You might set even data that changes often to cache for a few
minutes. Odds are the user doesn't need it to revalidate on the
server every minute. cgi.d's setResponseExpires can help here,
just set it a little bit in the future.
If the user hits a link to go back to a page then, it will
load from cache most the time, making navigating the site feel
snappy. Until the time expires, then it's wait again, but IMO
some cache is better than none.
Remember: you can cache AJAX responses too.
What if the resource actually does change? You'll want to change
the link. When compiling, there's a __TIMESTAMP__ special token.
A quick and dirty method is to use that __TIMESTAMP__ on your
resource URLs in your html so every time you recompile, it
invalidates the user's cache.
script src=/myapp/functions.js?{$timestamp}/script
A better way might be to hash the content at compile time, but
I haven't written code that can do this well enough for real
work yet.
* Caching makes a much bigger difference than just about any other
technique. You'll still want fast code for the cold cache users,
but as they browse your site, a good cache policy can shave
full seconds off the experience.
The fastest code is running no code at all.
Re: gtkD problems and general gui question.
Al 18/10/11 17:08, En/na Jordi Sayol ha escrit: Al 18/10/11 16:44, En/na %u ha escrit: Hello. I downloaded gtkD MS Windows installer, and I tried to compile one of the examples shown on the gtkD website: http://www.dsource.org/projects/gtkd There is not GtkD MS Windows installer on gtkd site, You've installed gtk+ runtime. So, you have to compile and install GtkD yourself on your system. Take a look at http://dsource.org/forums/viewtopic.php?t=5129 If you want GtkD out of the box, there are compiled packages ready for linux on gtkd site: http://www.dsource.org/projects/gtkd/wiki/DebianPackages http://www.dsource.org/projects/gtkd/wiki/FedoraPackages Regards, -- Jordi Sayol smime.p7s Description: S/MIME Cryptographic Signature
Re: Only one signal per object?
I've ran into an awful bug right now (probably not related to your
module). It basically comes down to this inside of a Widget
constructor:
this(Widget parent)
{
super(parent);
void test()
{
msgbox(this.position);
}
parent.onMouseLDown.connect( { this.position; });
parent.onMouseLDown.connect( { test(); });
}
If I comment out the first connect call I'll get an access violation
when 'test()' tries to access the position field. The order of the
two calls doesn't seem to affect the bug. I can't recreate the bug in
a simple test-case though.. :/
Oh btw, there's a couple of failing unittests in your signal module
when compiling with -g -debug -w -wi:
object.Exception@signalsnew.d(679): Handler is not connected
core.exception.AssertError@signalsnew(950): unittest failure
core.exception.AssertError@signalsnew(968): unittest failure
xml Bible for conversion for D
I've got xml text of a Bible version. I want to get the text in the
following format:
class Bible {
Book[] bs;
}
class Book {
Chapter[] cs;
}
class Chapter {
Verse[] vs;
}
class Verse {
string v;
}
Here's a part of the xml file:
?xml version=1.0 encoding=ISO-8859-1?
bible
b n=Genesis
c n=1
v n=1In the beginning, God created the heavens and the earth./v
v n=2The earth was without form and void, and darkness was over the
face of the deep. And the Spirit of God
I would like any pointers on xml with D.
- Joelcnz
Re: Frontend and backend communication
I'm trying to implement the function pointer system right now, and it
seems to work on the C side, but not D. I assume I'm missing some kind
of syntax here. I have these global variables:
struct S_FrontendFunctions {
void function() RedrawScreen;
void function(const char*, int) PrecacheCard;
}
shared S_FrontendFunctions FrontendFunctions;
And if I try to set the pointers D style, like this:
FrontendFunctions.RedrawScreen = function(){};
FrontendFunctions.PrecacheCard = function(const char*, int){};
I get errors:
Error: cannot implicitly convert expression (__funcliteral3)
of type _error_ function() to shared(void function())
Error: cannot implicitly convert expression (__funcliteral4)
of type _error_ function(const const(char*), int) to shared(void
function(const const(char*), int))
So how do I define those functions as shared?
Re: xml Bible for conversion for D
I suggest xmlp http://www.dsource.org/projects/xmlp/ Example: http://www.dsource.org/projects/xmlp/browser/trunk/test/books.d He intends to push for it to be in Phobos so it uses the std namespace. import std.xml2; On Wed, 19 Oct 2011 17:31:06 +1300, Joel Christensen wrote: I would like any pointers on xml with D. - Joelcnz
Re: xml Bible for conversion for D
I think I want to stick with the current std xml library for now. I think the books example is too different for me to work for what I want.
