dynamically compile and load glue logic
in how far is it or would the following be possible: dynamically compile and execute some glue logic that is also written in D under linux? and what happens if that code uses phobos or other dub libs that are available in the host binary? especially the 2nd point is important as i would want to load 100's of those snippets. For the context: the idea is to create a streaming-ETL system where you can dynamically add/remove rules/modules. it would be great if logic could be written directly in D and operate on specific (library provided) objects.
Re: Persistent key-value-store for D?
On Wednesday, 26 April 2017 at 17:06:52 UTC, krylon wrote: I looked at the DUB package registry and asked Google quite a bit now, but I did not found such a package for D. So my first question is - did I not look hard enough? I found a reimplentation of QDBM [1] (the spiritual ancestor of Tokyocabinet), but it does not seem to handle concurrency at all. Are there other options along those lines? (If there was one that also provides transactions, that would be awesome!) If I understand what I have read so far correctly, it is possible to access libraries written in C or C++ from D - in that case, I could just use Tokyocabinet directly, but I have not found any pointers on how to do this. Is this a feasible option, and if so, where can I find documentation on how to do this? i recommend leveldb http://code.dlang.org/packages/d-leveldb its easy to use and mostly faster than tokyocabinet ( only very specifically tuned tokyo btrees outperform leveldb) i used above library with great success. it also shows you how to do c bindings.
Re: new XML and JSON libs and replacement of std.net.curl
On Tuesday, 16 August 2016 at 10:06:05 UTC, ikod wrote: On Tuesday, 16 August 2016 at 09:16:40 UTC, yawniek wrote: There is common http message parser that used in nginx and nodejs. I think it can be ported from C to D. that is pico, see: https://github.com/nodejs/http-parser/pull/200 https://github.com/h2o/picohttpparser/issues/7 But async library accepted as standard need before this. i don't think there is any temporal dependency between an eventloop and a http parser making it into phobos. only later then a simple http server or client needs both (+ TLS ).
Re: new XML and JSON libs and replacement of std.net.curl
imo things should be modularized. so there should be (fast) protocol parsers first, something like https://github.com/h2o/picohttpparser or https://github.com/seanmonstar/httparse then a very simple eventloop that has abstractions and range based interfaces for reading/writing data into different types of sockets. and at this point we need to start talking about Fibers and if/how they work with the eventloop. and how to make async code nice in D. Rust has a new approach that looks very promising: http://aturon.github.io/blog/2016/08/11/futures/ only then we can think about making http calls, using TLS and having a webserver. as for TLS a dual approach might be needed anyway, because there people would probably want to be able to dynamically link to a system provided library such as OpenSSL. => in my opinion the groundwork (parsers, eventloop) etc should be DESIGNED thoroughly and we should not just take the best available library and stuff it into phobos. and there is no shame to look at Go and rust, i still think D would have a good future as Language to implement Services.
Re: JSON Serialization with runtime filtering.
On Friday, 22 July 2016 at 12:36:31 UTC, Alexander Milushev wrote: I there any json serialization library which allow to make decision about ignoring fields in runtime? I trying to write rest client but server accept either 'cmd' or 'args' field for example and I need to find solution. can you give an example? i'm not sure what you mean by "ignore at runtime" if its simply ignoring fields of a struct then thats easily doable with https://github.com/tamediadigital/asdf
Re: Singleton Pattern
On Thursday, 5 January 2012 at 22:15:32 UTC, asm wrote: how can i implementing the singleton pattern in D? https://p0nce.github.io/d-idioms/#Leveraging-TLS-for-a-fast-thread-safe-singleton
Re: How to get current time as long or ulong?
On Tuesday, 5 July 2016 at 18:16:31 UTC, Charles Hixson wrote:
What I'm looking for is the opposite of the "FromUnixTime"
function.
i often use
long toNsUnixTime(SysTime t)
{
return (t.stdTime - 621_355_968_000_000_000L)*100;
}
as a helper. any chance that something like this can be put into
phobos?
its needed to work with external libraries or network services
that expect this format.
Re: Linux and htod
On Thursday, 16 June 2016 at 19:04:38 UTC, bachmeier wrote: On Thursday, 16 June 2016 at 12:23:09 UTC, fbmac wrote: How people use it on Linux, if htod is required to import C libraries and windows only?f Just to clarify, so as to prevent confusion by someone that randomly stumbles across this post, you do not need htod, dstep, or any other tool to call C libraries from D. dstep generates bindings to C libraries for you. You can create the bindings yourself in your D source files, and if you only want to call a couple of functions from a particular C library, that's the most convenient. http://dlang.org/spec/interfaceToC.html The wording of the question implies that one of these tools is required to call into C libraries, which is not correct. https://wiki.dlang.org/D_binding_for_C is also helpful they should be put together.
Re: interfacing with C: strings and byte vectors
On Saturday, 11 June 2016 at 10:26:17 UTC, Mike Parker wrote:
On Saturday, 11 June 2016 at 09:32:54 UTC, yawniek wrote:
thanks mike for the in depth answer.
i forgot to add a few important points:
- the strings in vec_t are not c strings
- vec_t might contain other data than strings
the original ctor i pasted actually doesn't even work, temporarly
i solved it like
this(string s) {
char[] si = cast(char[]) s; //i'm scared
base = si.ptr;
len = si.length;
}
is there a better solution than to fearlessly cast away
immutability?
i guess i could just make a second vec_t that has immutable base
and length
that can be used in D to stay clean, would that be worth anything?
now what i still don't have a proper idea for is how can i create
wrappers for the
methods accepting vec_t in a clean way.
for the vec_t that are allocated in D-land we can state that the
C libs will not modify the data.
there is a lot of functions that accept vec_t.
is there no way to have strings auto cast to vec_t ?
another way i see is a UDA that generates the wrapper function(s).
other ideas?
interfacing with C: strings and byte vectors
my C library works a lot with strings defined in C as:
struct vec_t {
char *base;
size_t len;
}
is there a easy way to feed regular D strings to functions that
accept vec_t*
without creating a vec_t every time
or do i write wrappers for these functions and if so, what is the
most elegant way
to build them?
so far i defined vec_t as:
struct vec_t {
char *base;
size_t len;
this(string s) { base = s.ptr; len = s.lenght; }
nothrow @nogc inout(char)[] toString() inout @property { return
base[0 .. len]; }
nothrow @nogc @property const(char)[] toSlice()
{
return cast(string) base[0..len];
}
alias toString this;
}
but i guess there is a more elegant way?!
Re: Benchmark Dlang vs Node vs Ruby
On Friday, 27 May 2016 at 16:47:19 UTC, Daniel Kozak wrote: Why not to use distribute oprion? Dne 27. 5. 2016 17:35 napsal uživatel "yawniek via Digitalmars-d-learn" < digitalmars-d-learn@puremagic.com>: it its a flawed strategy. what you should do is let the kernel handle it and use SO_ REUSEPORT libasync supports it. see e.g. https://github.com/rejectedsoftware/vibe.d/issues/1139 and https://lwn.net/Articles/542629/
Re: Benchmark Dlang vs Node vs Ruby
On Friday, 27 May 2016 at 13:45:23 UTC, llaine wrote: Hi guys, In my journey of learning about D I tried to benchmark D with Vibe.d vs node with express and Ruby with Sinatra. And the results are pretty surprising. I have to admit that I though D was more faster than that. How is this even possible ? I am doing something wrong ? Here are the numbers with the project : https://github.com/llaine/benchmarks/blob/master/README.md you should: - use this https://github.com/etcimon/ddb Postgres client - fix your logic - NOT use option distribute - use LDC2 beta2 as compiler with release flag - use neweset vibe.d version and then your results should be easily above 1000 rps
Re: Async or event library
On Tuesday, 10 May 2016 at 13:34:36 UTC, chmike wrote: vibed uses libevent, a C library. The discussion is regarding a possible pure D equivalent of libevent. libasync is an interesting proposal but it is apparently slower than libevent. I don't know the current status because vibed improved its performance in the last months. My initial question is if there is a working group I could join to work on this pure D async library. I'm interested in working on the subject. from my experience its not really slower than libevent and it could be made even faster by taking some time to profile it. plus its battle tested in production and fully cross platform. also, it will most probably not be your bottleneck.
Re: vibe.d is blocking threads
On Wednesday, 27 April 2016 at 13:00:29 UTC, RuZzz wrote:
Code:
import std.concurrency;
import core.thread;
//import vibe.http.client; // If uncommented this
line, the thread "worker" does not start
void worker() {
foreach (i; 0 .. 5) {
Thread.sleep(500.msecs);
writeln(i, " (worker)");
}
}
[...]
you should use the built in concurrency primitives vibe.d offers.
especially runTask and runWorkerTask.
see the vibe.d examples and http://vibed.org/features
Re: how to parse a string into a phobos datatype with additional logic
On Thursday, 7 April 2016 at 18:29:19 UTC, Jonathan M Davis wrote: On Thursday, April 07, 2016 07:45:06 yawniek via Digitalmars-d-learn wrote: So, while I understand your frustration, I just don't see any other sane way to approach this problem than what you've done. Putting it all in a wrapper type encapsulates it in a way that it can actually work, whereas the other options get messy really fast if they're possible at all. - Jonathan M Davis thank you Jonathan for the extensive answer, really helpful. And the Longer i think about it the more i come to the conclusion that its actually not even that ugly as it allows you to easily add more logic to the structs if needed. i already extensively alias standard datatypes to another name where it is used in a specific context (latest example: alias ColumnName = string) and it became helpful because you can easily refactor it to something bigger.
Re: VibeCustomMain not working
On Thursday, 7 April 2016 at 13:40:17 UTC, Rene Zwanenburg wrote: On Thursday, 7 April 2016 at 13:25:29 UTC, Jerry wrote: I generated a visuald project and tried that. Now suddenly it is working as expected. So I guess it's a bug in dub. That's possible of course, but I'd expect something so fundamental breaking to be noticed sooner. Just to make sure, could you run dub with --force to rule out that it's picking up some stale object files from somewhere? i also had it multiple times that my build was broken and not even --force would fix it. then i deleted ~/.dub and .dub and rebuild everything and suddenly it worked.
Re: how to parse a string into a phobos datatype with additional logic
On Thursday, 7 April 2016 at 08:03:34 UTC, Edwin van Leeuwen wrote: You can try this library: https://code.dlang.org/packages/dateparser nope this will not work and the question is broader: i want to have a standard datatype parsed in a specific way and so that i can use other std library tools such as csvparser to then do the job. parsing unixtime with subsecond resolution was just one example, there is a lot of cases where you want to do some tranformation e.g. "33%" -> 0.33f my solution with defining a custom struct works, but it would be nice if you somehow could hook into this process in a cleaner way.
how to parse a string into a phobos datatype with additional logic
what is the way one is supposed to parse e.g. a double of unixtime (as delived by nginx logs) into a SysTime? currently i'm creating a wrapper struct around SysTime with alias this as: https://gist.github.com/yannick/6caf5a5184beea0c24f35d9d4a4c7783 really ugly imho. is there a better way to do this?
Re: BitArray: count the number of bits set
On Tuesday, 1 March 2016 at 09:32:20 UTC, Andrea Fontana wrote: Maybe you should look for "hamming weight" :) maybe. and here is a snowman for you: ☃ the question was if it exists for BitArray in phobos. https://issues.dlang.org/show_bug.cgi?id=10239 so apparently its really missing in phobos BitArray. the algo is even described in TDPL.
BitArray: count the number of bits set
i figured i can count the number of bits set for a BitArray with std.algorithm : count: BitArray([0,0,1]).bitsSet.count() but this seems not very optimal, is there a faster way directly accessible trough phobos? ideally something that is optimized by the compiler.
Re: Alternate databases
On Saturday, 20 February 2016 at 13:09:53 UTC, Andrew Edwards wrote: I'm searching for client drivers for the following databases. Are the any available? https://rethinkdb.com/docs/install-drivers/ http://docs.basho.com/riak/latest/dev/references/client-implementation/ Thanks, Andrew none for riak afaik, for rethink i found: https://github.com/search?utf8=%E2%9C%93=language%3Ad+rethinkdb=Repositories=searchresults if you try them, please report back if any of them is already usablee
Re: Getting the body of a HTTP Request
On Wednesday, 27 January 2016 at 23:42:54 UTC, brian wrote: Anyone able to shed some light on what the structure of the response is, and how I can read/output it all? Regards Brian its unlikely that vibe client misses something. for debugging i would try to go trough all requests with curl on the commandline and see if you can make it work there. in case you are looking for an oauth client implementation there seems to be this https://github.com/danielsson/dlang-oauth2 that works with vibe.d if you add the switch for it. not sure if it works but it might be a good example.
Re: Are there any D scripting engines for use with D?
On Monday, 4 January 2016 at 19:04:48 UTC, Max Klyga wrote: On 2016-01-04 18:40:03 +, Jason Jeffory said: The fastest one would probably be Lua - http://code.dlang.org/search?q=lua But there are other options: Python - http://code.dlang.org/packages/pyd Javascript - http://code.dlang.org/search?q=javascript and http://pointersgonewild.com/higgs/ Croc (previously miniD, a scripting language implemented in D) - http://jfbillingsley.com/croc/ there is also http://code.dlang.org/packages/d_mruby mruby is really nice. but i agree, a more native language that would not need to push data via a stack but instead had direct access to strings or defined objects would be something really helpful.
Re: DUB problems
On Thursday, 31 December 2015 at 08:51:42 UTC, Daniel Kozak wrote:
I try to build vibe-d Hello world app with dub, but I have many
problems with dub, nothing works at all
Error: Output file 'client.o' for module 'vibe.http.client'
collides with previous module 'vibe.db.mongo.client'. See the
-oq option
and allmost nothing works as expected. How can I do one simple
thing:
Compile vibe.d with libasync and with ldc compiler
os? versions? it kind of depends.
i just tested this on linux.
dub init -t vibe.d foo
cd foo
cat dub.json
{
"name": "foo",
"description": "A simple vibe.d server application.",
"copyright": "Copyright © 2015, yannick",
"authors": ["yannick"],
"dependencies": {
"vibe-d": "~>0.7.26"
},
"versions": ["VibeDefaultMain"],
"subConfigurations": { "vibe-d": "libasync"}
}
ws01% cat dub.selections.json
{
"fileVersion": 1,
"versions": {
"memutils": "0.4.3",
"vibe-d": "0.7.26",
"libevent": "2.0.1+2.0.16",
"libev": "5.0.0+4.04",
"openssl": "1.1.4+1.0.1g",
"libasync": "0.7.5"
}
}
dub build --compiler=ldc -b release --combined
ldc --version
LDC - the LLVM D compiler (0.16.1):
based on DMD v2.067.1 and LLVM 3.7.0
dub --version
DUB version 0.9.24-rc.2+18-g4fece3c
note the dub switches
compiling in debug mode or without --combined crashes!
failing regex
regex from https://github.com/ua-parser/uap-core/blob/master/regexes.yaml#L38 seems to work in other languages, not so in D: auto r2 = r"(?:\/[A-Za-z0-9\.]+)? *([A-Za-z0-9 _\!\[\]:]*(?:[Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]*)) (\d+)(?:\.(\d+)(?:\.(\d+))?)?".regex(); ( https://gist.github.com/4334e35e68497c0517db ) results in ``` dmd -run failing_regex.d std.regex.internal.ir.RegexException@/usr/local/Cellar/dmd/2.069.0/include/d2/std/regex/internal/parser.d(1392): invalid escape sequence Pattern with error: `(?:\/[A-Za-z0-9\.]+)? *([A-Za-z0-9 _\!` <--HERE-- `\[\]:]*(?:[Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]*)) (\d+)(?:\.(\d+)(?:\.(\d+))?)?` 4 dmd_run68HuB5 0x0001044d211d @trusted void std.regex.internal.parser.Parser!(immutable(char)[]).Parser.error(immutable(char)[]) + 297 5 dmd_run68HuB5 0x0001044da604 ref @trusted std.regex.internal.parser.Parser!(immutable(char)[]).Parser std.regex.internal.parser.Parser!(immutable(char)[]).Parser.__ctor!(const(char)[]).__ctor(immutable(char)[], const(char)[]) + 160 6 dmd_run68HuB5 0x0001044cc732 @safe std.regex.internal.ir.Regex!(char).Regex std.regex.regexImpl!(immutable(char)[]).regexImpl(immutable(char)[], const(char)[]) + 86 7 dmd_run68HuB5 0x0001044e944f std.regex.internal.ir.Regex!(char).Regex std.functional.__T7memoizeS95_D3std5regex18__T9regexImplTAyaZ9regexImplFNfAyaAxaZS3std5regex8internal2ir12__T5RegexTaZ5RegexVii8Z.memoize(immutable(char)[], const(char)[]) + 475 8 dmd_run68HuB5 0x0001044cc6bc @trusted std.regex.internal.ir.Regex!(char).Regex std.regex.regex!(immutable(char)[]).regex(immutable(char)[], const(char)[]) + 64 9 dmd_run68HuB5 0x0001044cc5de _Dmain + 46 10 dmd_run68HuB5 0x000104509ac3 D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv + 39 11 dmd_run68HuB5 0x0001045099fb void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) + 55 12 dmd_run68HuB5 0x000104509a68 void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll() + 44 13 dmd_run68HuB5 0x0001045099fb void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) + 55 14 dmd_run68HuB5 0x00010450994d _d_run_main + 497 15 dmd_run68HuB5 0x0001044cc677 main + 15 16 libdyld.dylib 0x7fff8e5185c8 start + 0 17 ??? 0x 0x0 + 0 ``` bug or did i do something wrong?
Re: regex format string problem
Hi Rikki,
On Monday, 23 November 2015 at 03:57:06 UTC, Rikki Cattermole
wrote:
I take it that browscap[0] does it not do what you want?
I have an generator at [1].
Feel free to steal.
This looks interesting, thanks for the hint. However it might be
a bit limited,
i have 15M+ different User Agents with all kind of weird cases,
sometimes not even the extensive ua-core regexs work. (if you're
interested for testing let me know)
Also once you do get yours working, you'll want to use ctRegex
and generate a file with all of them in it. That'll increase
performance significantly.
that was my plan.
Reguarding regex, if you want a named sub part use:
(?[a-z]*)
Where [a-z]* is just an example.
I would recommend you learning how input ranges work. They are
used with how to get the matches out, e.g.
auto rgx = ctRegex!`([a-z])[123]`;
foreach(match; rgx.matchAll("b3")) {
writeln(match.hit);
}
i'm aware how this works, the problem is a different one:
i do have a second string that contains $n's which can occur in
any order.
now of course i can just go and write another regex and replace
it, job done.
but from looking at std.regex this seems to be built in, i just
failed to get it to work properly, see my gist. i hoped this to
be a 1liner.
regex format string problem
hi! how can i format a string with captures from a regular expression? basically make this pass: https://gist.github.com/f17647fb2f8ff2261d42 context: i'm trying to write a implementation for https://github.com/ua-parser where the regular expression as well as the format strings are given.
Building the Docs with checked out code downoads old dmd
i'm trying to build the docs as per http://wiki.dlang.org/Building_DMD#Building_the_Docs i have a working setup to build the latest dmd/druntime/phobos but somehow the makefile tries to download an old dmd version and my bandwith currently is a bit restricted and i want an offline version. also the documentation is a bit unclear on what actually happens and is being built. would be really helpful.
Re: LuaD: creating a flexible data filter system
many thanks for the valuable insights. so far i made a simple prototype with LuaD and classes, works nicely for when my niput what so far is not 100% clear is if there is a way to have a parsed msgpack or json documents being exposed in my lua code in a way so it behaves like a lua object. Ideally in a RW fashion so that changed then again can be processed by D code.
LuaD: creating a flexible data filter system
hi, i'm reading in a stream of data that is deserialized into individual frames. a frame is either of: a) a specific D datastructure ( struct with a few ulong,string,string[string] etc members), known at compile time b) json (preferably stdx.data.json) i now want to implement something where i can dynamically add lua filters that then get data out of these frames, create a new lua object send it back to D code where its either sent to another lua filter or at last being serialized again to json und then being processed further. ideally i would not like to copy all the data into a lua object but directly access it from lua. is there an elegant approach to do this and support both a and b cases? so far i did some benchmarks, mainly with string comparisons and it turned out that luaD is about 10x faster than mruby and python D bridges.
Re: AWS API Dlang, hmac sha256 function.
On Saturday, 3 October 2015 at 03:11:06 UTC, holo wrote: Hello I'm trying to contact AWS API with D according to documentation: [...] check https://github.com/yannick/vibe-aws it has v4 implemented
Re: "if sting in string"
On Friday, 18 September 2015 at 09:42:05 UTC, smadus wrote: Ok i have rewrite :) Now: http://dpaste.dzfl.pl/cf8bb54b1390 The Problem is: http://www.directupload.net/file/d/4114/9zryku49_png.htm but i dont understand this, because, the exception should be "Something wrong" ?!? But, thanks for the answers, realy good and the code has been smaller. the question is if there is a way to handle inaccessible directories with dirEntries. the problem is that if you have a directory with insufficent access rights it throws a FileException
how does vibe's PrivateAccessProxy work
can someone explain a bit how the @before hooks works in detail, i mainly have problems understanding why ensureAuth in belows example refers to SampleService. as an instance: https://github.com/rejectedsoftware/vibe.d/blob/a1efc05c09135ca8aca21ccec72790ddfaca67c9/examples/web/source/app.d#L114 so how does PrivateAccessProxy work: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/internal/meta/funcattr.d#L225
Re: most elegant functional way to make a histogram
On Friday, 21 August 2015 at 21:08:25 UTC, Laeeth Isharc wrote: I guess this kind of thing will do: upRangeHighs.each!((ref a)=(++histogram[a][0])); int[] arr = [5,1,2,2,3,4,5,5,5]; int[int] histo; arr.each!( a = ++histo[a] ); writeln(histo); this works
Re: std.net.curl
On Monday, 17 August 2015 at 13:04:31 UTC, tired_eyes wrote: On Monday, 17 August 2015 at 12:58:54 UTC, Adam D. Ruppe wrote: On Monday, 17 August 2015 at 12:52:37 UTC, tired_eyes wrote: Hovewer, dmd app.d spits a whole bunch of strange error messages: try dmd -lcurl app.d and see if that helps. Error: unrecognized switch '-lcurl' do you use dub? if so, did you add libs: [curl] to your dub.json? if that doesn't help, please post the output of curl-config. e.g. curl-config --libs --built-shared --prefix --static-libs
dub and subpackages
i'm trying to have my own versions of my dependencies as git submodules. problem: i include a local version of vibe.d and then i add other local versions of packages that themselves include vibe.d somehow my version of vibe isn't picked up by the other dependencies and it results in an error: Sub package vibe-d: doesn't exist. when i go and simply remove the vibe-d dependency from the dub.json it compiles fine. but that can't be the solution. dub --version DUB version 0.9.24-rc.2+18-g4fece3c, built on Aug 14 2015 top level project uses dub.sdl, subpackages use dub.json. whats the correct way of having a chain of packages included from git submodules so that every packages get's only picked once?
Re: dub and subpackages
On Friday, 14 August 2015 at 08:09:18 UTC, Edwin van Leeuwen wrote: On Friday, 14 August 2015 at 08:06:15 UTC, yawniek wrote: dub add-local allows you to add local copy of a package. This will be system wide though, not only for the current package. i actually tried this, somehow did't work
format of trace.log
is there a reason the trace log is a bit weirdly formatted? a) different tables within one file b) column separation is something like (fill with space until it exceeds 7 digits could this be improved to use a standard format (e.g. tsv) or are there some legacy reasons? also it would be nice to automatically pipe it trough ddemange if that is available. speaking of ddemangle, would it be possible to support lambda functions?
zlib performance
hi,
unpacking files is kinda slow, probably i'm doing something wrong.
below code is about half the speed of gnu zcat on my os x machine.
why?
why do i need to .dup the buffer?
can i get rid of the casts?
the chunk size has only a marginal influence.
https://github.com/yannick/zcatd
import
std.zlib,
std.file,
std.stdio;
void main(string[] args)
{
auto f = File(args[1], r);
auto uncompressor = new UnCompress(HeaderFormat.gzip);
foreach (ubyte[] buffer; f.byChunk(4096))
{
auto uncompressed = cast(immutable(string))
uncompressor.uncompress(buffer.dup);
write(uncompressed);
}
}
Re: zlib performance
On Friday, 7 August 2015 at 08:50:11 UTC, Daniel Kozák wrote: ldc[2] -O -release -boundscheck=off -singleobj app.d ldc 0.15.2 beta2 2.86s user 0.55s system 77% cpu 4.392 total v2.068-devel-8f81ffc 2.86s user 0.67s system 78% cpu 4.476 total v2.067 2.88s user 0.67s system 78% cpu 4.529 total i can now reproduce the results and indeed, its faster than zcat: on a c4.xlarge aws instance running archlinux and dmd v2.067 same file as above on my macbook. best run: 2.72s user 0.39s system 99% cpu 3.134 total worst run: 3.47s user 0.46s system 99% cpu 3.970 total zcat: best: 4.45s user 0.28s system 99% cpu 4.764 total worst: 4.99s user 0.57s system 99% cpu 5.568 total so i guess on os x there is still something to be optimized
Re: zlib performance
On Friday, 7 August 2015 at 07:29:15 UTC, Daniel Kozák wrote: Which compiler and version. There has been some performance problem with IO on OSX, it should be fixed in 2.068 release i'm on master. v2.068-devel-8f81ffc also changed file read mode to rb. i don't understand why the program crashes when i do not do the .dup
Re: zlib performance
On Friday, 7 August 2015 at 07:43:25 UTC, Daniel Kozák wrote:
i don't understand why the program crashes when i do not do
the .dup
This is weird. I would say it should not crash
exactely. but try it yourself.
the fastest version i could come up so far is below.
std.conv slows it down.
going from a 4kb to a 4mb buffer helped. now i'm within 30% of
gzcat's performance.
import
std.zlib,
std.file,
std.stdio;
void main(string[] args)
{
auto f = File(args[1], rb);
auto uncompressor = new UnCompress(HeaderFormat.gzip);
foreach (ubyte[] buffer; f.byChunk(1024*1024*4))
{
auto uncompressed = cast(immutable(string))
uncompressor.uncompress(buffer.dup);
write(uncompressed);
}
}
Re: zlib performance
On Friday, 7 August 2015 at 07:48:25 UTC, yawniek wrote: On Friday, 7 August 2015 at 07:43:25 UTC, Daniel Kozák wrote: the fastest version i could come up so far is below. std.conv slows it down. going from a 4kb to a 4mb buffer helped. now i'm within 30% of gzcat's performance. ok maybe not, there is another problem, not everything seems to get flushed, i'm missing output
Re: zlib performance
On Friday, 7 August 2015 at 08:05:01 UTC, Daniel Kozák wrote:
import
std.zlib,
std.file,
std.stdio,
std.conv;
void main(string[] args)
{
auto f = File(args[1], rb);
auto uncompressor = new UnCompress(HeaderFormat.gzip);
foreach (buffer; f.byChunk(4096))
{
auto uncompressed =
cast(char[])(uncompressor.uncompress(buffer.idup));
write(uncompressed); }
write(cast(char[])uncompressor.flush);
}
this is faster for me than zcat
not here on os x:
d version: 3.06s user 1.17s system 82% cpu 5.156 total
gzcat 1.79s user 0.11s system 99% cpu 1.899 total
Re: zlib performance
On Friday, 7 August 2015 at 08:24:11 UTC, Daniel Kozák wrote: can you try it with ldc? ldc[2] -O -release -boundscheck=off -singleobj app.d ldc 0.15.2 beta2 2.86s user 0.55s system 77% cpu 4.392 total v2.068-devel-8f81ffc 2.86s user 0.67s system 78% cpu 4.476 total v2.067 2.88s user 0.67s system 78% cpu 4.529 total (different file, half the size of the one above:) archlinux, virtualbox vm, DMD64 D Compiler v2.067 real0m2.079s user0m1.193s sys 0m0.637s zcat: real0m3.023s user0m0.320s sys 0m2.440s is there a way to get rid of the flush in the end so everything happens in one loop? its a bit inconvenient when i have another subloop that does work
Re: zlib performance
On Friday, 7 August 2015 at 11:45:00 UTC, Daniel Kozak wrote:
On Friday, 7 August 2015 at 09:12:32 UTC, yawniek wrote:
[...]
Can you try it without write operation (comment out all write)?
And than try it without uncompression?
// without compression:
void main(string[] args)
{
auto f = File(args[1], r);
foreach (buffer; f.byChunk(4096))
{
write(cast(char[])buffer);
}
}
0.03s user 0.09s system 11% cpu 1.046 total
// without write:
void main(string[] args)
{
auto f = File(args[1], r);
auto uncompressor = new UnCompress(HeaderFormat.gzip);
foreach (buffer; f.byChunk(4096))
{
auto uncompressed =
cast(char[])(uncompressor.uncompress(buffer));
}
uncompressor.flush;
}
2.82s user 0.05s system 99% cpu 2.873 total
Re: extern(C) with function returning user type
On Wednesday, 29 July 2015 at 17:59:26 UTC, Kyoji Klyden wrote: How would I use a C function that's returning a struct? auto doesn't work here, and from what I can tell D can't import C headers. (If it really can't then, that would be a very welcome feature) I do have the required libs but I can't create my D obj file so I can't really get there. I know that there htod but, is there anyway I can avoid using that? I'm using GDC, and GCC on Win7 checkout dstep https://github.com/jacob-carlborg/dstep i think you can just define a struct with the same shape. see also mike's answer on http://forum.dlang.org/post/yheamworbhcaprrko...@forum.dlang.org
Re: Typed Message Passing between D Processes
On Wednesday, 29 July 2015 at 16:36:41 UTC, Atila Neves wrote: LDC: Cerealed: 970 ms, 482 μs, and 6 hnsecs MsgPack: 896 ms, 591 μs, and 2 hnsecs Not too shabby! Atila cool. what are the advantages of cereald over msgpack? can you stream in packets with cereald too? cool thing about msgpack is that there exist libraries for many language. so we use it do actually store logs and then process them with other tools too.
Re: C bindings: typedef struct conflicts with method
On Tuesday, 21 July 2015 at 06:12:53 UTC, Jacob Carlborg wrote: what the correct way to bind these? Please report an issue for this. In this case rd_kafka_metadata_t should be used for the struct name. done, https://github.com/jacob-carlborg/dstep/issues/40 i was under the impression that there is already a ticked as https://github.com/jacob-carlborg/dstep/issues/8 looks very similar (but was closed). thanks rikki, thats what i ended up doing, compiles so far.
idiom for C error strings
whats the proper way to use/wrap C functions that expect a error string buffer e.g.: somefun(T param1, char* errstr, size_t errstr_size) in D ?
C bindings: typedef struct conflicts with method
i tried to automagically create bindings for librdkafka
(https://github.com/edenhill/librdkafka)
with dstep.
now the code contains typedefs structs with the same name as
methods:
```
typedef struct rd_kafka_metadata {
int broker_cnt; /* Number of brokers in
'brokers' */
struct rd_kafka_metadata_broker *brokers; /* Brokers */
int topic_cnt; /* Number of topics in
'topics' */
struct rd_kafka_metadata_topic *topics;/* Topics */
int32_t orig_broker_id; /* Broker originating this
metadata */
char *orig_broker_name; /* Name of originating
broker */
} rd_kafka_metadata_t;
rd_kafka_metadata (rd_kafka_t *rk, int all_topics,
rd_kafka_topic_t *only_rkt,
const struct rd_kafka_metadata **metadatap,
int timeout_ms);
```
what the correct way to bind these?
Re: problem with gc?
On Thursday, 28 May 2015 at 10:11:38 UTC, zhmt wrote: On Thursday, 28 May 2015 at 02:00:57 UTC, zhmt wrote: the throughput is steady now: if buffer size is set to 1, throughput is about 20K response/second; when buffer size is big enough ,the throughput is about 60K response/second. checkout msgpack-rpc-d, i remember handling around 300k req/s
Re: vibed: how to use pure HTML instead of template engine?
On Thursday, 7 May 2015 at 18:59:13 UTC, Suliman wrote: 1. Do I need write ./public/ ? In examples often simply public/ will work too. even public it goes trough Path struct, see: https://github.com/rejectedsoftware/vibe.d/blob/11578aa956a9b3b0e305d655f9668a867fdd89bd/source/vibe/inet/path.d 2. What incoming parameters (HTTPServerRequest req, HTTPServerResponse res) mean? Why I should to specify them? HTTPServerRequest contains all data that the client sends, e.g. headers, cookies, source ip etc. see: https://github.com/rejectedsoftware/vibe.d/blob/11578aa956a9b3b0e305d655f9668a867fdd89bd/source/vibe/http/server.d#L584 HTTPServerResponse is the response you send back. https://github.com/rejectedsoftware/vibe.d/blob/11578aa956a9b3b0e305d655f9668a867fdd89bd/source/vibe/http/server.d#L788 you woud want to set at least its body, as you do below 3. Why code with: res.writeBody(Hello, World!, text/plain); and router.get(*, serveStaticFiles(./public/)); also work, but my variant (see code above) do not load say that page not found? what exactely does not work? please link code (ideally https://gist.github.com/ ) 4. How to specify page that I need to load, why in examples there is only link to folder like public? But what if I want to load public/foo.html? public servers public files, its more or less a static webserver which checks if a file exists and then serves that. if you want to dynamically modify content you send you need to use the templating or do your own string magic. generally it seems you do not fully understand the concept of how these web frameworks work. i think you should either read vibe.d's source code or more read how other such frameworks work e.g. http://www.sinatrarb.com/documentation.html has pretty good documentations and books that explain the inner workings (but mind, ruby is a dynamically typed language). then i also can recommend that you check out vibe.d's github repositories and read trough all the example projects that come with it. e.g. https://github.com/rejectedsoftware/vibe.d/tree/11578aa956a9b3b0e305d655f9668a867fdd89bd/examples/app_skeleton
Re: vibed: how to use pure HTML instead of template engine?
On Thursday, 7 May 2015 at 18:59:13 UTC, Suliman wrote:
shared static this()
{
auto router = new URLRouter;
router.get(/, root);
auto settings = new HTTPServerSettings;
settings.port = 8080;
listenHTTP(settings, router);
}
void root(HTTPServerRequest req, HTTPServerResponse res)
{
serveStaticFiles(public/);
}
i missed this in the answer sorry,
its clear that its not working you are saying that
ONLY requests to / shall be served with the staticFiles.
this makes no sense.
for static files you would want a catch all route at the end, and
before that you define a few other routes that serve specific
strings.
check this:
https://github.com/rejectedsoftware/vibe.d/blob/11578aa956a9b3b0e305d655f9668a867fdd89bd/examples/app_skeleton/source/app.d#L18
router.get(/, showHome); // GET / is handled by the showHome
function
router.get(/about, staticTemplate!about.dt); // GET /about
is handlet by the template
router.get(*, serveStaticFiles(public)); // every other get
request goes here and checks if a filename exists within the
public folder. so e.g.
GET /foo.html is either served public/foo.html if it exists or
returns an 404
Re: thrift and dub
got it to work by using the thrift code from the fbthrift repo (minus the tests).
thrift and dub
hi, i'm trying to get a thrift example working within a dub project. it seems that the thrift.d in the dub repo is not whats actually needed but i should link against libthriftd.a that comes from the official thrift distro. what i tried is add the following to dub.json: libs: [/path/to/thrift/lib/d/libthriftd.a], sourcePaths: [/path/to/thrift/lib/d/src], but that seems not to be the correct thing to do. anyone has a working example (e.g. the calculator example)? thanks y
return types of std.functional functions
i found two snippets from the functional docs that do not work (anymore?) http://dlang.org/phobos/std_functional.html assert(compose!(map!(to!(int)), split)(1 2 3) == [1, 2, 3]); and int[] a = pipe!(readText, split, map!(to!(int)))(file.txt); throwing a std.array.array into the mix works fine. did this use to work? is there any other way of doing it?
Re: Search Engine
On Monday, 6 October 2014 at 17:11:51 UTC, ANtlord wrote: Good day! I recenlty have tried create typical project on vibe.d. The web framework is not bad. And I can say, that it is better that something another web frameworks. But I have met a problem. I can't find search engine. I use xapian always. It has API for several languages, but except D. I know about another search engines, but I can't find even one for D. Does D have search engine? Or Have I find theme for starting open source project? Best Regards. Sorry for my english. you should be able to include xapian via c++ extern interface see: http://dlang.org/cpp_interface.html or you can use std.json to talk to elasticsearch.
