Re: FastCGI binding or implementation?
On 2011-10-18 19:24, Jeremy Sandell wrote: On Mon, Oct 17, 2011 at 2:11 PM, Jacob Carlborg d...@me.com mailto:d...@me.com wrote: On 2011-10-17 16:01, Andrea Fontana wrote: I handle request on different threads. I do some pre-processing on scgi data and I fill a struct: request.get[] request.post[] request.cookie[] request.headers[string] then I call a virtual function (to override on subclasses) like: do(request, output); where user fill output struct in a way like: output.data ~= htmlbodyh1hello world/h1/body/html; output.status = 200 output.cookies = bla bla and then if is method != head i send headers + data, else just headers. btw 99% of usage is get, post, head. Yes, but if you want to write a web site that is RESTful you need the other HTTP methods as well, at least PUT and DELETE. BTW, what about creating something like Rack but for D. Rack is a low level interface in front of the web server which web frameworks can be built on top. http://rack.github.com/ -- /Jacob Carlborg Yes, this is exactly why I was wondering whether FastCGI had been implemented (though SCGI works for me as well) - so that I could write something on top of it, in much the same way I would using (for example) WSGI in Python. I also agree with you re: supporting all of the HTTP methods. Just because the most common ones are GET, POST, and HEAD doesn't mean we should leave out the others; both PUT and DELETE are quite useful. Best regards, Jeremy Sandell Although I have no idea if the rest of the 9 HTTP methods are useful, e.g. trace, options, connect and patch. -- /Jacob Carlborg
Re: FastCGI binding or implementation?
On 2011-10-18 20:21, Adam D. Ruppe wrote:
Tale time, only tangentially on topic.
Today, I was coincidentally switching one of my work apps from
standard CGI to Fast CGI.
Almost trivial. Set up Apache, then build the program with
-version=fastcgi. Done.
Well, not 100% done. I had a piece of static data in the app
that worked correctly before but now means subsequent requests
were out of it.
Changed that to an instance variable, and boom, works perfectly.
* If you are using Fast CGI, avoid static variables.
The next thing is speed. From principles, there's very little
reason for FastCGI to actually be faster than normal CGI - the
startup costs are insignificant next to the total app runtime
and network lag. (Startup is maybe 5 ms on the live server,
with runtime close to 50ms and ping to the user another 100ms.
The cost of CGI is roundoff error in the actual deployment.)
My benchmarks supported this for the kind of loads we had before.
But now, the number of concurrent users is picking up. The CGI
still performed very well, though every so often, users complained
about lag on some resources.
I ran a benchmark comparing cgi to fast cgi with a very large
number of concurrent users.
It showed better availability and about a 15% speed boost under
this load. Since Apache restarts it when it segfaults, reliability
ought not to be affected, though it's too soon to say for sure.
So, I changed the makefile to say -version=fastcgi and soon
realized I must search for static variables - found just one,
so easy fix, and we're up on fastcgi.
... but that 15% in the benchmark hasn't translated to a big change
in the live environment yet. Been several hours now, and we've
been trying to force the availability issue, and failed so far.
Looks like a win, but not a very big one. Speed on the whole -
unaffected. The difference is roundoff error once you factor in
network lag and such again.
So, how can we speed up the application? The key here is client
side caching.
Using my cgi.d, there's a function:
cgi.setCache(true);
which tells it simply to cache the response forever. It makes
an expiration date long in the future.
Set that for any content which changes infrequently - css,
javascript, images, any kind of (conceptually) pure or static data,
etc.
Now, your code doesn't run again and the user doesn't hit the
network again. What was 150ms is now 1ms. The users will feel
the difference.
You might set even data that changes often to cache for a few
minutes. Odds are the user doesn't need it to revalidate on the
server every minute. cgi.d's setResponseExpires can help here,
just set it a little bit in the future.
If the user hits a link to go back to a page then, it will
load from cache most the time, making navigating the site feel
snappy. Until the time expires, then it's wait again, but IMO
some cache is better than none.
Remember: you can cache AJAX responses too.
What if the resource actually does change? You'll want to change
the link. When compiling, there's a __TIMESTAMP__ special token.
A quick and dirty method is to use that __TIMESTAMP__ on your
resource URLs in your html so every time you recompile, it
invalidates the user's cache.
script src=/myapp/functions.js?{$timestamp}/script
A better way might be to hash the content at compile time, but
I haven't written code that can do this well enough for real
work yet.
Yeah, that's how Rails 3.1 does it now. Rails 3.1 inserts a hash of the
content in the file name instead of a time stamp after the question mark.
* Caching makes a much bigger difference than just about any other
technique. You'll still want fast code for the cold cache users,
but as they browse your site, a good cache policy can shave
full seconds off the experience.
The fastest code is running no code at all.
Why not just cache the generated HTML and let Apache handle it. Then it
doesn't even need to start the application if the cache is available.
--
/Jacob Carlborg
Re: xml Bible for conversion for D
On Wed, 19 Oct 2011 18:21:22 +1300, Joel Christensen wrote: I think I want to stick with the current std xml library for now. No, you don't. It's a terrible library, held together by lashings of hate and the power of Satan* *only half kidding
Re: xml Bible for conversion for D
On Wed, 19 Oct 2011 07:03:05 +, Bernard Helyer wrote: and the power of Satan* Err, I can see how that could be construed as in poor taste, considering the subject matter of the XML document. I meen no offense. _
Re: xml Bible for conversion for D
On Wed, 19 Oct 2011 07:04:53 +, Bernard Helyer wrote: meen Just not my day, is it?
Re: xml Bible for conversion for D
I don't care to learn xml at this stage. I just want to use it on that Bible file. On 19-Oct-11 8:03 PM, Bernard Helyer wrote: I think I want to stick with the current std xml library for now.
Re: xml Bible for conversion for D
On Wed, 19 Oct 2011 21:45:29 +1300, Joel Christensen wrote: I don't care to learn xml at this stage. What do you mean by learn xml? I'm saying std.xml is a terrible library, and furthermore will be deprecated sooner or later. Use another library. There is already a suggestion in this thread, and I can give one too if you want. Friends don't let friends use std.xml.
Re: FastCGI binding or implementation?
AFAIK other http methods have nothing special. You have just to
implement on your code:
if (request.method == PUT)
{
...
...
}
if you need them. Am i wrong?
Il giorno mer, 19/10/2011 alle 08.36 +0200, Jacob Carlborg ha scritto:
On 2011-10-18 19:24, Jeremy Sandell wrote:
On Mon, Oct 17, 2011 at 2:11 PM, Jacob Carlborg d...@me.com
mailto:d...@me.com wrote:
On 2011-10-17 16:01, Andrea Fontana wrote:
I handle request on different threads. I do some pre-processing on
scgi data and I fill a struct:
request.get[]
request.post[]
request.cookie[]
request.headers[string]
then I call a virtual function (to override on subclasses) like:
do(request, output);
where user fill output struct in a way like:
output.data ~= htmlbodyh1hello world/h1/body/html;
output.status = 200
output.cookies = bla bla
and then if is method != head i send headers + data, else just
headers.
btw 99% of usage is get, post, head.
Yes, but if you want to write a web site that is RESTful you need
the other HTTP methods as well, at least PUT and DELETE.
BTW, what about creating something like Rack but for D. Rack is a
low level interface in front of the web server which web frameworks
can be built on top.
http://rack.github.com/
--
/Jacob Carlborg
Yes, this is exactly why I was wondering whether FastCGI had been
implemented (though SCGI works for me as well) - so that I could write
something on top of it, in much the same way I would using (for example)
WSGI in Python.
I also agree with you re: supporting all of the HTTP methods. Just
because the most common ones are GET, POST, and HEAD doesn't mean we
should leave out the others; both PUT and DELETE are quite useful.
Best regards,
Jeremy Sandell
Although I have no idea if the rest of the 9 HTTP methods are useful,
e.g. trace, options, connect and patch.
use of D for ropes data structures
can someone propose the appropriate D features or libraries that would support a native D implementation of the ropes structures described in this Sgi C++ library. http://www.sgi.com/tech/stl/Rope.html
Re: FastCGI binding or implementation?
Andrea Fontana: other http methods have nothing special Indeed. The only thing that might get you is if the data's content type is different than the default. That's possible on POST too, though, so still nothing special. My cgi library has an enum to tell you what the requestMethod is, and it lists all the options in the standard. It, however, does not handle all possible content-types. It does x-www-form-urlencoded and multipart/form-data, so it can handle virtually all web forms out there - including file uploads - but if you want others, it'll take a minor modification. The best way to do it is probably to not attempt to parse it in the library at all, and just pass a range of raw data to the application.
Re: xml Bible for conversion for D
On Wed, 19 Oct 2011 18:21:22 +1300, Joel Christensen wrote: I think I want to stick with the current std xml library for now. I think the books example is too different for me to work for what I want. std.xml has had many bugs, and hasn't had most of them fixed. It might work for your desire, but it is a bad idea to rely on such. xmlp is about the same as std.xml so you're going to have to learn something about parsing with delegates.
Re: xml Bible for conversion for D
I found std.xml useless too, so I wrote my own lib.
https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff
Grab dom.d from there. First, convert your file to UTF-8. My lib
might work for you, but it assumes utf-8 so it will throw if it actually
encounters a non-ascii character from the 8859-1 charset.
Anyway the D code for dealing with that would look something like this:
import std.file;
import std.stdio;
import arsd.dom;
void main() {
auto document = new Document(readText(bible.xml));
// the document is now the bible
auto books = document.getElementsByTagName(b);
foreach(book; books) {
auto nameOfBook = b.n; // Genesis for example. All xml attributes are
available this same way
auto chapters = book.getElementsByTagName(c);
foreach(chapter; chapters) {
auto verses = chapter.getElementsByTagName(v);
foreach(verse; verses) {
auto v = verse.innerText;
// let's write out a passage
writeln(nameOfBook, , chapter.n, :, verse.n, , v); //
prints Genesis 1:1 In the beginning, God created [...]
}
}
}
}
Re: FastCGI binding or implementation?
Jacob Carlborg: Why not just cache the generated HTML and let Apache handle it. That sounds hard... configuring Apache to do anything beyond the most trivial of tasks is a huge pain to me. It is easy to call cgi.setCache(true); though. Then it doesn't even need to start the application if the cache is available. It's better with a client side cache. If it's available, you don't have to go to the server at all! Server side caching is something I haven't done yet; it's never been useful to me.
Re: Implicit cast to immutable
On Tue, 18 Oct 2011 02:40:12 -0400, Daniel Murphy
yebbl...@nospamgmail.com wrote:
Steven Schveighoffer schvei...@yahoo.com wrote in message
news:op.v3h06olweav7ka@localhost.localdomain...
That sounds like an incorrect restriction. The implicit cast to
immutable
should depend on whether the function being *called* qualifies, not if
the
function you are calling *from* qualifies.
I think you've misunderstood what I'm saying. The patch I made
implemented
two ways to implicly convert to immutable: the result of a pure function
returning immutable and a return statement inside a pure function.
1)
int[] fun() { return new int[]; }
immutable x = fun(); // conversion happens here
2)
immutable(int[]) fun() { return new int[]; } // conversion happens here
immutable x = fun();
Neither of those is relevant, since they do not contain parameters (I'm
assuming you meant them both to be pure).
Qualifying means the return type should be mutable, and cannot be
derived
from the parameters without requiring casts. The easiest way to do this
is to ensure the parameters are all const, immutable, or implicitly cast
to immutable. You could do funky things like assume for instance an
int[]
cannot possibly be implicit-casted to a char[], so therefore int[]
foo(char[] data) pure can be implicitly casted to immutable, but that
might be flirting with dangerous situations.
The one exception should be allocating memory, which should always
qualify, even though it's not a pure function.
This is valid, but becomes very complicated with complex return types. I
doubt this will ever make it into the language.
No, it's not complicated, at least in my view. The rules are:
1. if the pure function parameters are only immutable or
implicitly-convertible-to-immutable, the result is safe to cast to
immutable, regardless of what happens inside the function, or the type of
the result.
2. if the pure function parameters contain any mutable data that is
mutable and *not* implicitly convertible to mutable, and the result is
only safe to cast to immutable if it can already implicitly cast to
immutable.
3. if the pure function parameters are implicitly castible to immutable,
immutable, or are const, then:
a) if the return type is implicitly castable to immutable, obviously
it can be cast.
b) if the return type is not implicitly castable to immutable, but
contains only mutable references, then it's implicitly castable to
immutable.
c) otherwise, it cannot be implicitly cast.
You do not need the function body to determine this, and the decision
should be made by the compiler at the call site.
Your case 2 where the conversion happens at the return statement is
already covered.
I've got the beginnings of a patch to enable a sort of 'pure expression',
such as new, array.dup and array concatenation expressions. The result
of a
call to a const-pure function using immutable arguments can be converted
to
immutable, while calling it with mutable or const arguments cannot,
without
searching the return type for anything the arguments can implicitly
convert
to (or create).
Eg. I can't see a great way to detect situations like this:
struct S { const void* p; }
This struct could only be cast to immutable under my rule 1 above. It
does not implicitly cast to immutable.
S[] fun(int[] arr)
{
return [ S(arr.ptr) ];
}
immutable x = fun([1, 2, 3]);
Invalid. arr is not implicitly castable to immutable and is mutable, so
the result must already be implicitly castable (which it is not). This
falls under rule 2 above, and fails the test.
-Steve
Re: FastCGI binding or implementation?
On 2011-10-19 16:36, Adam Ruppe wrote:
Jacob Carlborg:
Why not just cache the generated HTML and let Apache handle it.
That sounds hard... configuring Apache to do anything beyond the most
trivial of tasks is a huge pain to me.
No, it's not that hard. Just add a couple of rewrite-rules that checks
if a request matches an already cached page and rewrite it to the cached
page. Something like this:
/VirtualHost *:80
...
RailsAllowModRewrite On
RewriteEngine On
RewriteCond %{THE_REQUEST} ^(GET|HEAD)
RewriteCond %{REQUEST_URI} ^/([^.]+)$
RewriteCond %{DOCUMENT_ROOT}/cache/%1.html -f
RewriteRule ^/[^.]+$ /cache/%1.html [QSA,L]
RewriteCond %{THE_REQUEST} ^(GET|HEAD)
RewriteCond %{DOCUMENT_ROOT}/cache/index.html -f
RewriteRule ^/$ /cache/index.html [QSA,L]
/VirtualHost
I found the above at:
http://www.alfajango.com/blog/make-sure-your-rails-application-is-actually-caching-and-not-just-pretending/
It is easy to call cgi.setCache(true); though.
Then it doesn't even need to start the application if the cache is
available.
It's better with a client side cache. If it's available, you don't
have to go to the server at all!
Yes of course, that is preferred.
Server side caching is something I haven't done yet; it's never
been useful to me.
--
/Jacob Carlborg
Re: use of D for ropes data structures
On 10/19/2011 06:22 AM, Jay Norwood wrote: can someone propose the appropriate D features or libraries that would support a native D implementation of the ropes structures described in this Sgi C++ library. http://www.sgi.com/tech/stl/Rope.html From the library std.range -Iteration std.container (RB tree, and interesting documentation), Language features, Templates hth,Bjoern
Re: xml Bible for conversion for D
Thanks Adam. :-) It seems to be working now, with the stuff you gave me. I was thinking of making a text file from it that my programs can load from. avoiding the xml file, so they load faster. But I think I'll have my program(s) use the xml file each time they're run, in the mean time.
Re: Looking for documentation of D's lower-level aspects.
== Quote from Jesse Phillips (jessekphillip...@gmail.com)'s article Right now D isn't ready to be used in this fashion It looks there's a more-or-less functional kernel written in D (and pretty well documented too): http://wiki.xomb.org/index.php?title=Main_Page
Re: Looking for documentation of D's lower-level aspects.
Am 20.10.2011, 00:06 Uhr, schrieb Sean Silva chisophu...@gmail.com: == Quote from Jesse Phillips (jessekphillip...@gmail.com)'s article Right now D isn't ready to be used in this fashion It looks there's a more-or-less functional kernel written in D (and pretty well documented too): http://wiki.xomb.org/index.php?title=Main_Page You do not need the Tango standard library to compile XOmB as it contains its own standard calls and runtime.
Re: Implicit cast to immutable
Steven Schveighoffer schvei...@yahoo.com wrote in message
news:op.v3lug2q2eav7ka@localhost.localdomain...
On Tue, 18 Oct 2011 02:40:12 -0400, Daniel Murphy
yebbl...@nospamgmail.com wrote:
Steven Schveighoffer schvei...@yahoo.com wrote in message
news:op.v3h06olweav7ka@localhost.localdomain...
That sounds like an incorrect restriction. The implicit cast to
immutable
should depend on whether the function being *called* qualifies, not if
the
function you are calling *from* qualifies.
I think you've misunderstood what I'm saying. The patch I made
implemented
two ways to implicly convert to immutable: the result of a pure function
returning immutable and a return statement inside a pure function.
1)
int[] fun() { return new int[]; }
immutable x = fun(); // conversion happens here
2)
immutable(int[]) fun() { return new int[]; } // conversion happens here
immutable x = fun();
Neither of those is relevant, since they do not contain parameters (I'm
assuming you meant them both to be pure).
Yeah, they were supposed to be pure. I was just demonstrating that
conversion happens at the return statement, or the call site, but only
currently for strongly pure functions.
Qualifying means the return type should be mutable, and cannot be
derived
from the parameters without requiring casts. The easiest way to do this
is to ensure the parameters are all const, immutable, or implicitly cast
to immutable. You could do funky things like assume for instance an
int[]
cannot possibly be implicit-casted to a char[], so therefore int[]
foo(char[] data) pure can be implicitly casted to immutable, but that
might be flirting with dangerous situations.
The one exception should be allocating memory, which should always
qualify, even though it's not a pure function.
This is valid, but becomes very complicated with complex return types. I
doubt this will ever make it into the language.
No, it's not complicated, at least in my view. The rules are:
1. if the pure function parameters are only immutable or
implicitly-convertible-to-immutable, the result is safe to cast to
immutable, regardless of what happens inside the function, or the type of
the result.
2. if the pure function parameters contain any mutable data that is
mutable and *not* implicitly convertible to mutable, and the result is
only safe to cast to immutable if it can already implicitly cast to
immutable.
3. if the pure function parameters are implicitly castible to immutable,
immutable, or are const, then:
a) if the return type is implicitly castable to immutable, obviously
it can be cast.
b) if the return type is not implicitly castable to immutable, but
contains only mutable references, then it's implicitly castable to
immutable.
c) otherwise, it cannot be implicitly cast.
You do not need the function body to determine this, and the decision
should be made by the compiler at the call site.
Your case 2 where the conversion happens at the return statement is
already covered.
I've got the beginnings of a patch to enable a sort of 'pure expression',
such as new, array.dup and array concatenation expressions. The result
of a
call to a const-pure function using immutable arguments can be converted
to
immutable, while calling it with mutable or const arguments cannot,
without
searching the return type for anything the arguments can implicitly
convert
to (or create).
Eg. I can't see a great way to detect situations like this:
struct S { const void* p; }
This struct could only be cast to immutable under my rule 1 above. It
does not implicitly cast to immutable.
S[] fun(int[] arr)
{
return [ S(arr.ptr) ];
}
immutable x = fun([1, 2, 3]);
Invalid. arr is not implicitly castable to immutable and is mutable, so
the result must already be implicitly castable (which it is not). This
falls under rule 2 above, and fails the test.
-Steve
Ok, I think I meant to make fun pure and fun's parameter 'in', so it would
fall under rule 3 but be rejected by 3b as it contains a const member.
It's not always simple to determine if an aggregate contains non-mutable
members. An example would be a class when you only have a base class
reference.
Another problem (which we're already seeing with the existing conversions)
is that when you screw it up the conversion fails, but doesn't give you any
hint as to why it failed. Making the rules more complicated is just going
to make this worse.
Re: Implicit cast to immutable
On Wed, 19 Oct 2011 21:39:47 -0400, Daniel Murphy
yebbl...@nospamgmail.com wrote:
Steven Schveighoffer schvei...@yahoo.com wrote in message
news:op.v3lug2q2eav7ka@localhost.localdomain...
On Tue, 18 Oct 2011 02:40:12 -0400, Daniel Murphy
yebbl...@nospamgmail.com wrote:
Steven Schveighoffer schvei...@yahoo.com wrote in message
news:op.v3h06olweav7ka@localhost.localdomain...
That sounds like an incorrect restriction. The implicit cast to
immutable
should depend on whether the function being *called* qualifies, not if
the
function you are calling *from* qualifies.
I think you've misunderstood what I'm saying. The patch I made
implemented
two ways to implicly convert to immutable: the result of a pure
function
returning immutable and a return statement inside a pure function.
1)
int[] fun() { return new int[]; }
immutable x = fun(); // conversion happens here
2)
immutable(int[]) fun() { return new int[]; } // conversion happens here
immutable x = fun();
Neither of those is relevant, since they do not contain parameters (I'm
assuming you meant them both to be pure).
Yeah, they were supposed to be pure. I was just demonstrating that
conversion happens at the return statement, or the call site, but only
currently for strongly pure functions.
Qualifying means the return type should be mutable, and cannot be
derived
from the parameters without requiring casts. The easiest way to do
this
is to ensure the parameters are all const, immutable, or implicitly
cast
to immutable. You could do funky things like assume for instance an
int[]
cannot possibly be implicit-casted to a char[], so therefore int[]
foo(char[] data) pure can be implicitly casted to immutable, but that
might be flirting with dangerous situations.
The one exception should be allocating memory, which should always
qualify, even though it's not a pure function.
This is valid, but becomes very complicated with complex return
types. I
doubt this will ever make it into the language.
No, it's not complicated, at least in my view. The rules are:
1. if the pure function parameters are only immutable or
implicitly-convertible-to-immutable, the result is safe to cast to
immutable, regardless of what happens inside the function, or the type
of
the result.
2. if the pure function parameters contain any mutable data that is
mutable and *not* implicitly convertible to mutable, and the result is
only safe to cast to immutable if it can already implicitly cast to
immutable.
3. if the pure function parameters are implicitly castible to immutable,
immutable, or are const, then:
a) if the return type is implicitly castable to immutable,
obviously
it can be cast.
b) if the return type is not implicitly castable to immutable, but
contains only mutable references, then it's implicitly castable to
immutable.
c) otherwise, it cannot be implicitly cast.
You do not need the function body to determine this, and the decision
should be made by the compiler at the call site.
Your case 2 where the conversion happens at the return statement is
already covered.
I've got the beginnings of a patch to enable a sort of 'pure
expression',
such as new, array.dup and array concatenation expressions. The result
of a
call to a const-pure function using immutable arguments can be
converted
to
immutable, while calling it with mutable or const arguments cannot,
without
searching the return type for anything the arguments can implicitly
convert
to (or create).
Eg. I can't see a great way to detect situations like this:
struct S { const void* p; }
This struct could only be cast to immutable under my rule 1 above. It
does not implicitly cast to immutable.
S[] fun(int[] arr)
{
return [ S(arr.ptr) ];
}
immutable x = fun([1, 2, 3]);
Invalid. arr is not implicitly castable to immutable and is mutable, so
the result must already be implicitly castable (which it is not). This
falls under rule 2 above, and fails the test.
-Steve
Ok, I think I meant to make fun pure and fun's parameter 'in', so it
would
fall under rule 3 but be rejected by 3b as it contains a const member.
It's not always simple to determine if an aggregate contains non-mutable
members. An example would be a class when you only have a base class
reference.
I had not thought about classes (or interfaces), I was only thinking of
concrete types. I think in the case of classes, all parameters with
classes must be marked as immutable to have an implicit cast of the result
to immutable (i.e. fall under rule 1). In reality, the rules I specify
are enough, but they simply aren't explicit about how classes make it
impossible to determine if any const members are aboard.
The same goes for void *, which could point to a type which has const
members.
All cases where classes or void * pointers are involved must fall under
rule 1, since you cannot determine whether such types are transitively
mutable.
Another problem (which
