Re: HTTP frameworks benchmark focused on D libraries

[ikod via Digitalmars-d-announce]

On Saturday, 28 May 2022 at 05:44:11 UTC, tchaloupka wrote:

On Friday, 27 May 2022 at 20:51:14 UTC, zoujiaqing wrote:



It means two separate syscalls for header and body. This alone 
have huge impact on the performance and if it can be avoided, 
it would be much better.


sendv/writev also can help to save syscals when you have to send 
data from non-contiguous buffers.

Re: Library associative array project v0.0.1

[ikod via Digitalmars-d-announce]

On Wednesday, 11 May 2022 at 19:15:36 UTC, Steven Schveighoffer 
wrote:


* add some methods returning a copy of the value instead of a 
pointer to value


I return a reference to the value (via `opIndex`). Can you 
elaborate on why you would need it to be a copy? What is your 
API?


-Steve


Sorry, it is safe to return reference for your implementation, as 
you call `new Entry` on every insert, like standard AA. I keep 
entries in preallocated vector for performance reasons and every 
`resize` can break reference.

Re: Library associative array project v0.0.1

[ikod via Digitalmars-d-announce]

On Wednesday, 11 May 2022 at 15:31:02 UTC, Steven Schveighoffer 
wrote:
I just spent a couple hours making a library AA solution that 
is binary compatible with druntime's builtin AA.


The benefits:

For the future:

1. Implement all the things that AAs can do (which are 
possible, some are not).

2. Look at alternatives to GC for allocation/deallocation.
3. Possible use with betterC?

Anyway, enjoy!


I don't know if this compatible with your approach, but for my 
hashmap I added goals like:


* allow @nogc and/or @safe operations on hashmap if key and value 
types allow this
* add some methods returning a copy of the value instead of a 
pointer to value


Regards!

nbuff v0.1.13

[ikod via Digitalmars-d-announce]

Hello,

This is buffer management library. I created it for my networking 
daemons and recently it was used in Nats client 
(https://code.dlang.org/packages/nats), and maybe it can be 
useful for others, so I added small tutorial.


The main features are: nogc, safety, zero copy (actually 
sometimes copy required), automatic buffer management and reuse 
through thread-local buffer pool.


Here is tutorial link: 
https://github.com/ikod/nbuff/blob/master/docs/tutorial-e.md


Here is link to main repo https://github.com/ikod/nbuff

Re: requests 2.0.0 release

[ikod via Digitalmars-d-announce]

On Thursday, 29 October 2020 at 16:22:50 UTC, Anonymouse wrote:

On Thursday, 29 October 2020 at 06:42:54 UTC, ikod wrote:

Hi,

requests 2.0.0 released with single change - support for 
vibe-d moved to separate subpackage.


Thanks for all your hard work! This was an annoying point and 
I'm happy to see a solution that works for everyone.


Thanks everybody for your motivating words! I hope that next 
update will include native Windows SSL, I made some progress in 
this direction.


Regards,
Igor

requests 2.0.0 release

[ikod via Digitalmars-d-announce]

Hi,

requests 2.0.0 released with single change - support for vibe-d 
moved to separate subpackage.


The goal of this update is to prevent dub from downloading vibe-d 
packages for optional subConfiguration.


Important in case you use requests with vibe-d: starting from 
this release instead of using subConfiguration for requests in 
your dub.json you have to use package requests:vibed in 
dependencies.



dlang-requests is http/ftp client library inspired by python 
requests

https://github.com/ikod/dlang-requests

best regards,
Igor

Re: HTTP frameworks benchmark focused on D libraries

[ikod via Digitalmars-d-announce]

On Sunday, 27 September 2020 at 10:08:24 UTC, tchaloupka wrote:

Hi all, I've just pushed the updated results.



* new RAW tests in C to utilize epoll and io_uring (using 
liburing) event loops - so we have some ground base we can



I'll probably add wrk[3] load generator too to see a difference 
with a longer running tests.


[1] https://github.com/tchaloupka/during
[2] https://github.com/rakyll/hey
[3] https://github.com/wg/wrk


Thank for this job. It may be worth to add nginx as baseline for 
real C-based server.


I'll add my framework as soon as it will be ready.

Re: HTTP frameworks benchmark focused on D libraries

[ikod via Digitalmars-d-announce]

On Sunday, 20 September 2020 at 20:03:27 UTC, tchaloupka wrote:

Hi,
as it pops up now and then (last one in 
https://forum.dlang.org/thread/qttjlgxjmrzzuflrj...@forum.dlang.org) I wanted to see the various D libraries performance against each other too and ended up with https://github.com/tchaloupka/httpbench


It's just a simple plaintext response testing (nothing fancy as 
in Techempower) but this interests me the most as it gives the 
idea about the potential of the library.


More details in the README.

Hope it helps to test some ideas or improve the current 
solutions.


Tom


thanks! Very good news.

Re: khash associative array / hash map / hash set

[ikod via Digitalmars-d-announce]

On Tuesday, 25 August 2020 at 01:04:27 UTC, James Blachly wrote:

On 8/24/20 5:11 PM, ikod wrote:

Thanks, but no )


Also you may find useful safe map modification during 
iteration over map items (at the cost of creating temporary 
table copy).


In search of absolute speed I am willing to forego this, but 
certainly it could be useful in concurrency type situation


I made this copy lazy, kind of COW, as we don't have to copy kv 
storage while we iterate over it and do not change anything. I 
also added something to README.


Hope this will be usefull,

Regards!

Re: khash associative array / hash map / hash set

[ikod via Digitalmars-d-announce]

On Monday, 24 August 2020 at 20:31:42 UTC, James Blachly wrote:

On Monday, 24 August 2020 at 05:51:59 UTC, ikod wrote:

On Monday, 24 August 2020 at 01:39:26 UTC, James Blachly wrote:

OH, I almost forgot the best part. It is crazy fast.

https://attractivechaos.wordpress.com/2018/01/13/revisiting-hash-table-performance/
https://attractivechaos.wordpress.com/2019/12/28/deletion-from-hash-tables-without-tombstones/

My naive benchmark shows -- compared to 
emsi_containers.HashMap -- 30% faster inserts, 80% faster 
serial retrieval and 70% faster random retrieval. Perhaps I 
am doing something wrong?


James


Thanks, nice job!

You also may want to compare performance with 
https://github.com/ikod/ikod-containers, just add dependency 
from ikod-containers, then:


import ikod.containers;

and use ikod.containers.hashmap.HashMap as alias for 
container. I squeezed everything I was able from the 
open-addressing hash map.


Nice, thank you and great job! Performance looks very 
comparable and I would be happy to use your package as well. 
Perhaps it is time that Dlang have a faster canonical hashmap 
(phobos.next ?)




Thanks, but no )
This hashmap can't replace standard AA for next reason:
with standard AA you can safely do:

string[int] aa;
aa[0] = "null";
auto v = 0 in aa;
aa.remove(0);
assert(*v == "null");
aa[0] = "one";
assert(*v == "null");

This is because AA allocate memory in GC area for every value it 
store(and return pointer to it when "in" used), so even if you 
delete key from AA it is still safe to use pointer to value. But 
this require GC allocations.


Correct me if I'm wrong - your and mine HashMaps avoid 
allocations and store values inline, so you can't use pointer to 
values in safe code (value can be deleted, or replaced on next 
table insertion/deletion). In order to be safe my hashmap do not 
support "in" operator and always return value.


Also you may find useful safe map modification during iteration 
over map items (at the cost of creating temporary table copy).



```
hashmap benchmarks
Inserts for HashMap finished in 518 milliseconds.
Inserts for khash finished in 549 milliseconds.
Serial Lookups for HashMap finished in 21 milliseconds.
Random lookups for HashMap finished in 41 milliseconds.
Confirming stored value of last lookup: 
7353ece9-506c-467f-9cb4-7686426fa828

Serial Lookups for khash finished in 12 milliseconds.
Random lookups for khash finished in 36 milliseconds.
Confirming stored value of last lookup: 
1164a2f1-e6cb-4072-89d9-23cec5cadd95

```

Repeated tests show that ikod.containers' HashMap is 
consistently faster on insertions, while khash is consistently 
faster on retrieval.

Re: khash associative array / hash map / hash set

[ikod via Digitalmars-d-announce]

On Monday, 24 August 2020 at 01:39:26 UTC, James Blachly wrote:

OH, I almost forgot the best part. It is crazy fast.

https://attractivechaos.wordpress.com/2018/01/13/revisiting-hash-table-performance/
https://attractivechaos.wordpress.com/2019/12/28/deletion-from-hash-tables-without-tombstones/

My naive benchmark shows -- compared to emsi_containers.HashMap 
-- 30% faster inserts, 80% faster serial retrieval and 70% 
faster random retrieval. Perhaps I am doing something wrong?


James


Thanks, nice job!

You also may want to compare performance with 
https://github.com/ikod/ikod-containers, just add dependency from 
ikod-containers, then:


import ikod.containers;

and use ikod.containers.hashmap.HashMap as alias for container. I 
squeezed everything I was able from the open-addressing hash map.

Re: dlang-requests 1.1.0 released

[ikod via Digitalmars-d-announce]

On Tuesday, 12 May 2020 at 09:56:44 UTC, Pavel Shkadzko wrote:

On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:

Hello!

Just a note that dlang-requests ver 1.1.0 released with new 
'ByLine' interfaces added for get/post/put requests.


range algorithms can be applied to server responses, so that 
simple chain


getContentByLine("https://httpbin.org/anything;)
.map!"cast(string)a"
.filter!(a => a.canFind("data"))

should work.

These calls work lazily so you can apply them to large 
documents.


dlang-requests - HTTP client library, inspired by 
python-requests with goals:


small memory footprint
performance
simple, high level API
native D implementation

https://github.com/ikod/dlang-requests
https://code.dlang.org/packages/requests

Always waiting for your bugreports and proposals on project 
page.


Best regards!


Very nice to see requests for D.

It is written in README that it has a small memory footprint 
and performance but how does it compare against Python version?


I never compared performance with Python requests, but I made 
some comparisons with
curl and wget, but results were never published. I just checked 
that there are no any odd delays.


There were some performance issues with connection pooling and 
caching redirects (like 
https://github.com/ikod/dlang-requests/issues/80), but they were 
fixed quite long ago.




I am asking because we have some Python packages using requests 
and I wanted to try to rewrite them in D. To avoid ppl raising 
eyebrows it would be nice to know a bit more details if 
possible.


There are some functions of Python requests that were not 
implemented for dlang - for example, conversion to json and maybe 
a few others, but I can easily add them if someone really needs 
it.


At the same time I tried to adapt API to Dlang way - like using 
lazy ranges when sending or receiving data, and like lazy byLine 
interator over response body, so your code may look and perform 
better with dlang-requests than with python.



PS. I mentioned small memory footprint in README because, for 
some reason, I failed to keep memory usage in reasonable bounds 
with std.net.curl at the time I tried to use it.

Re: dlang-requests 1.1.0 released

[ikod via Digitalmars-d-announce]

On Sunday, 5 April 2020 at 16:25:52 UTC, Anonymouse wrote:

On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:

Hello!

Just a note that dlang-requests ver 1.1.0 released with new 
'ByLine' interfaces added for get/post/put requests.

[...]

As always, thanks for your hard work! I don't use much more 
than the normal `get` and `receiveAsRange`, but it's been 
working well.


Thanks for your support :) This work is pleasure for me, glad 
that it can be useful.

Re: dlang-requests 1.1.0 released

[ikod via Digitalmars-d-announce]

On Sunday, 5 April 2020 at 11:53:29 UTC, Petar Kirov [ZombineDev] 
wrote:

On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:

Hello!

Just a note that dlang-requests ver 1.1.0 released with new 
'ByLine' interfaces added for get/post/put requests.


range algorithms can be applied to server responses, so that 
simple chain


getContentByLine("https://httpbin.org/anything;)
.map!"cast(string)a"
.filter!(a => a.canFind("data"))

should work.

These calls work lazily so you can apply them to large 
documents.


dlang-requests - HTTP client library, inspired by 
python-requests with goals:


small memory footprint
performance
simple, high level API
native D implementation

https://github.com/ikod/dlang-requests
https://code.dlang.org/packages/requests

Always waiting for your bugreports and proposals on project 
page.


Best regards!


Nice work!

One quick suggestion: avoid direct casting from `ubyte[]` to 
`string`:


/+dub.sdl:
dependency "requests" version="~>1.1"
+/

void main()
{
import requests : getContentByLine;
import std : assumeUTF, canFind, each, filter, map, write;

getContentByLine("https://httpbin.org/anything;)
.map!assumeUTF // instead of map!"cast(string)a"
.filter!(a => a.canFind("data"))
.each!write;
}

1. From a code-style point of view, assumeUTF is better as it 
shows the intention to the reader - assume that the content is 
valid UTF8 encoded text, without performing validation. And if 
there are UTF8 errors, it is easy to go back and add validation 
there.
2. Avoid casting mutable data to immutable. The data path in 
your library is rather complex (getContentByLine -> _LineReader 
-> LineSplitter -> Buffer -> ...) and so it was hard to 
understand from a quick glance whether or not the buffer array 
is reused (but I would guess that it is). If the buffer array 
is reused, it means that the result of calling 
_LineReader.front() may be modified at a later point in time, 
which I think is obvious that it can lead to some rather nasty 
bugs in users' code.


Internally my code do not modify these data, but I understand 
your concern.




I suggest you look into Steven's iopipe[1] library, as I 
believe it can help you clean up and refactor this part of the 
codebase (and can probably yield some performance improvements 
along the way).


[1]: https://github.com/schveiguy/iopipe


Thanks! You are totally right regarding casting mutable to 
immutable. Base code was written when I did not understand this 
problem well enough. I know how to fix this but this probably 
require large enough rewrite. I hope I'll do this in some near 
future.


PS. I also have package which convert nogc unique mutable byte 
array to immutable byte array and attach this immutable data "as 
is" to buffer (which is essentally array of immutable(ubyte)[]). 
Then every operation on buffer (trim, split, search, etc) 
performed on immutable with zero copy and with warranties on data 
immutability.

dlang-requests 1.1.0 released

[ikod via Digitalmars-d-announce]

Hello!

Just a note that dlang-requests ver 1.1.0 released with new 
'ByLine' interfaces added for get/post/put requests.


range algorithms can be applied to server responses, so that 
simple chain


getContentByLine("https://httpbin.org/anything;)
.map!"cast(string)a"
.filter!(a => a.canFind("data"))

should work.

These calls work lazily so you can apply them to large documents.

dlang-requests - HTTP client library, inspired by python-requests 
with goals:


small memory footprint
performance
simple, high level API
native D implementation

https://github.com/ikod/dlang-requests
https://code.dlang.org/packages/requests

Always waiting for your bugreports and proposals on project page.

Best regards!

Re: cachetools v.0.3.1

[ikod via Digitalmars-d-announce]

On Tuesday, 13 August 2019 at 17:18:23 UTC, ag0aep6g wrote:

On 13.08.19 11:34, ikod wrote:

What is it?

cachetools - package with @safe and @nogc cache and containers 
implementations.

[...]

Project page: https://github.com/ikod/cachetools
docs: https://ikod.github.io/cachetools/


They don't seem to actually be @safe. An example:


import cachetools.containers.lists;
import std.stdio;
void main() @safe
{
DList!int dl;
dl.insert_first(42);
auto r = dl.range;
dl.clear();
writeln(r.front); /* Prints garbage, because it's accessing 
`free`d memory. */

}


As far as I can I see, that compiles because you have a bad 
@trusted here:


Probably this @trusted is not a problem (std.container.dlist also 
contain @trusted code). I mistakenly  ignored this case.


Thanks for your report, and  I'll create issue on github.


.

Other containers probably have similar problems.

Re: cachetools v.0.3.1

[ikod via Digitalmars-d-announce]

On Tuesday, 13 August 2019 at 10:12:45 UTC, Martin Tschierschke 
wrote:

On Tuesday, 13 August 2019 at 09:34:48 UTC, ikod wrote:

Hello

cachetools version 0.3.1 released


[...]
Looking at your performance numbers, I am wondering should your 
work in the end result in a better std AA implementation?


Regards mt.


I'm not sure if I can say too much here. And I can be wrong, but 
looks like std AA allocate table bucket entry on every insert. 
This can hurt performance when you have mixed insert/lookup use 
case. If this is the case then fix will require rewrite of AA 
code.


Thanks and regards,
Igor

Re: cachetools v.0.3.1

[ikod via Digitalmars-d-announce]

Hello

cachetools version 0.3.1 released

Changelog:

"Set" container added (based on hashtable)
Const-ness problems for hashtable fixed.

What is it?

cachetools - package with @safe and @nogc cache and containers  
implementations.


It inherits @nogc and @safe property from key toHash and opEquals 
methods and can store immutable keys and values (with 
restrictions).


Currently implemented:

caches: LRU, 2Q (with TTL)

containers: HashMap, OrderedHashMap, unrolled list, dlist, slist, 
set


Project page: https://github.com/ikod/cachetools
docs: https://ikod.github.io/cachetools/

Some performance test results:
https://github.com/ikod/cachetools/tree/master/testhash

Re: Redis client hunt-redis RC1 released

[ikod via Digitalmars-d-announce]

On Thursday, 25 July 2019 at 16:23:58 UTC, Dejan Lekic wrote:

On Tuesday, 23 July 2019 at 07:57:06 UTC, zoujiaqing wrote:

A Powerfull Redis client library for D Programming Language.

Porting from java Jedis, support redis 3.x / 4.x all features 
and 5.x

 some features.


Why? There is the excellent TinyRedis project. Does it not have 
some feature you need?


Correct me if I'm wrong, but TinyRedis supports only TcpSocket - 
so it can't be used with vibe (or another eventloop/async 
library) or unix socket. I'm seeing this just looking to 
https://github.com/adilbaig/Tiny-Redis/blob/master/source/tinyredis/connection.d so maybe I missed something.


Also I don't see redis cluster support.

Alas, hunt software is almost unusable because of lack of any 
documentation. So people forced reinvent redis client over and 
over again.

Re: cachetools v.0.1.1

[ikod via Digitalmars-d-announce]

Hello

cachetools version 0.2.1 released

Changelog:

OrderedHashMap added to containers,
copy constructors added to containers
addIfMissed interface added to HashMap's

What is it?

cachetools - package with @safe and @nogc cache and hashtable 
implementations.


It inherits nogc propery from key toHash and opEquals methods and 
can store immutable keys and values (with restrictions).


Currently inplemented:

caches: LRU, 2Q
containers: HashMap, OrderedHashMap, unrolled list, dlist, slist

Project page: https://github.com/ikod/cachetools
docs: https://ikod.github.io/cachetools/

Some performance test results: 
https://github.com/ikod/cachetools/tree/master/testhash

one more redis client library

[ikod via Digitalmars-d-announce]

Hello,

I didn't find D redis client with configurable transport - 
usually either std.socket or vibe-d sockets are available. So I 
wrote client library with pluggable transport provider - 
currently it supports std.socket, vibe-d sockets and hio (my own 
eventloop package) sockets, but can be easily extended for unix 
sockets, ssl or anything else - and this is in my current TODO 
list.


It also support reply decoding from asynchronous input stream, so 
it can be used in async-style code.


code: https://github.com/ikod/redisd
docs: https://ikod.github.io/redisd/

As usually your feature requests, pull requests and problem 
reports are welcome.


Best regards!

Re: dlang.ru is updated

[ikod via Digitalmars-d-announce]

On Friday, 17 May 2019 at 08:14:26 UTC, drug wrote:
I understand that there is a habit to guilt Russians and/or 
Putin in many problems but let's try to be clever?


At least he leaded Crimea annexion, war against Georgia, current 
proxy war on the east of Ukraine. He pretend to defend russians 
outside of the Russia, but suppress them in their own country.


PS. I will not follow this discussion and will not arguing as 
this is not a good place for it.

cachetools v.0.1.1

[ikod via Digitalmars-d-announce]

Hello,

v.0.1.1 released.

Significant changes since previous announce:

Caches:

1. added 2Q cache. 2Q is more advanced strategy than plain LRU. 
It is faster, scan-resistant and can give more cache hits.


2. For LRU cache implemented per-item TTL in addition to global 
TTL.


Containers:

1. Added unrolled double-linked lists. Unrolled lists are much 
faster than plain double-linked lists. Unrolled lists used for 2Q 
cache implementation.


2. HashMap  - code cleanup, use core.internal.hash: bytesHash for 
strings.


Cachetools is set of cache strategies and containers. Caches and 
containers are @safe and @nogc (inherits from key and value 
properties).


Project page: https://github.com/ikod/cachetools
docs: https://ikod.github.io/cachetools/

Some performance test results: 
https://github.com/ikod/cachetools/tree/master/testhash

cachetools v.0.0.1

[ikod via Digitalmars-d-announce]

Hello,

I released cachetools - package with @safe and @nogc cache and 
hashtable implementations.


It inherits nogc propery from key toHash and opEquals methods and 
can store immutable keys and values (with restrictions).


Currently implemented only LRU cache with TTL.

http://code.dlang.org/packages/cachetools
https://github.com/ikod/cachetools


Performance of underlying hash table seems to be better in my 
benchmarks in comparison with std associative arrays and with 
emsi containers (see 
https://github.com/ikod/cachetools/tree/master/testhash )


Test inserts and lookups int[int]
=
|std | 303 ms, 541 μs, and 3 hnsecs| GC memory Δ 41MB|
|c.t.| 181 ms, 173 μs, and 2 hnsecs| GC memory Δ 0MB|
|c.t.+GC | 184 ms, 594 μs, and 5 hnsecs| GC memory Δ 16MB|
|emsi| 642 ms and 120 μs   | GC memory Δ 0MB|

 Test insert, remove, lookup for int[int]
 ===
|std | 327 ms, 982 μs, and 1 hnsec | GC memory Δ 17MB|
|c.t.| 229 ms, 11 μs, and 7 hnsecs | GC memory Δ 0MB|
|c.t.+GC | 240 ms, 135 μs, and 4 hnsecs| GC memory Δ 16MB|
|emsi| 678 ms, 931 μs, and 9 hnsecs| GC memory Δ 0MB|

 Test inserts and lookups for struct[int]
 ===
|std | 468 ms, 411 μs, and 7 hnsecs| GC memory Δ 109MB|
|c.t.| 392 ms, 146 μs, and 1 hnsec | GC memory Δ 0MB|
|c.t.+GC | 384 ms, 771 μs, and 5 hnsecs| GC memory Δ 88MB|
|emsi| 1 sec, 328 ms, 974 μs, and 9 h  | GC memory Δ 0MB|

...

Thanks for bugreports and proposals.

Re: automem v0.3.5 - now with more vector (like std::vector, not Physics)!

[ikod via Digitalmars-d-announce]

On Thursday, 20 September 2018 at 14:57:42 UTC, Atila Neves wrote:
If you've never heard of automem before, I wrote it to have 
C++-style smart pointers in D that I could use in @nogc code:



http://code.dlang.org/packages/automem




Sorry for asking here.

Shouldn't this code work?

import automem;

struct X {
int i;
}
struct Y {
RefCounted!X x; // fails
//X x;  // ok
}
void main()
{
Y y1;
Y y2;
y2 = y1;
}

https://run.dlang.io/is/k2qWhm

Re: Optional and NotNull version 0.5.0 - swift optional like and scala option like

[ikod via Digitalmars-d-announce]

On Thursday, 16 August 2018 at 16:20:09 UTC, aliak wrote:

On Thursday, 16 August 2018 at 12:25:14 UTC, aliak wrote:

It's also @nogc and @safe


No it's not. Not dispatching at least. Dunno why though. Seems 
safey is because taking an address. Nogc will have to look in 
to.


Hello!

please, help, how can I use match in @nogc code:

import optional;

void main() @nogc {
Optional!int i = 1;
bool b = true;
i.match!(
(v) => b,
() => false
);
}

fails to compile:
source/app.d(3,6): Error: function `D main` is @nogc yet 
allocates closures with the GC
source/app.d(7,9):app.main.__lambda1 closes over variable 
b at source/app.d(5,10)


Thanks!

dlang-requests v0.9.0

[ikod via Digitalmars-d-announce]

Hello,

I just released v0.9.0 of dlang-requests. This is large 
performance improvement and code refactoring release. Nothing 
changed in API and everything should work as before (hopefully 
better and faster)


What's new:

1. Connection pool for each HTTPRequest structure added. This 
give huge performance gain if you repeatedly call rq.get/post/... 
and/or when redirects involved.


2. Cache for permanent redirects added to Request structure.

As result started from test results

--
Testing https://www.amazon.com
  Requests: 578 ms, 725 ╬╝s, and 2 hnsecs
  std.net.curl: 128 ms, 480 ╬╝s, and 6 hnsecs
--
--
Testing https://google.com
  Requests: 389 ms, 220 ╬╝s, and 7 hnsecs
  std.net.curl: 80 ms, 434 ╬╝s, and 3 hnsecs
--
--
Testing https://bing.com/
  Requests: 481 ms and 112 ╬╝s
  std.net.curl: 129 ms, 860 ╬╝s, and 9 hnsecs
--

we have

--
Testing https://www.amazon.com
  Requests: 540 ms, 527 ╬╝s, and 2 hnsecs
  std.net.curl: 135 ms, 558 ╬╝s, and 4 hnsecs
--
--
Testing https://www.google.com
  Requests: 76 ms, 981 ╬╝s, and 4 hnsecs
  std.net.curl: 67 ms, 35 ╬╝s, and 3 hnsecs
--
--
Testing https://www.bing.com/
  Requests: 108 ms, 557 ╬╝s, and 9 hnsecs
  std.net.curl: 132 ms, 256 ╬╝s, and 9 hnsecs
--

(amazon recognise curl and send very short reply, so for curl it 
work faster anyway)


See details here https://github.com/ikod/dlang-requests/issues/80

Please, post any bug reports and feature-requests on github.

dlang-request is HTTP/ftp client library, inspired by 
python-requests with goals:


small memory footprint
performance
simple, high level API
native D implementation

https://github.com/ikod/dlang-requests

Re: Blogpost about parallelizing Datacat with std.parallelism

[ikod via Digitalmars-d-announce]

On Thursday, 26 July 2018 at 12:13:01 UTC, Joakim Brännström 
wrote:

Hello,

I've written up a blog post[0] of my explorations when 
parallelizing Datacat[1].


Thanks for nice article. Just in case if you never tried this: 
https://bitbucket.org/andrewtrotman/d-profile-viewer

dlang-requests 0.7.0 released

[ikod via Digitalmars-d-announce]

Hello,

dlang-request 0.7.0 released with support for International 
Domain Names.


dlang-reuests is HTTP/FTP native client library, inspired by 
python-requests with support both for system and vibe.d sockets.


https://github.com/ikod/dlang-requests
https://code.dlang.org/packages/requests

Please report any bugs or feature requests to github.

Re: run.dlang.io can now display ASM + AST + IR

[ikod via Digitalmars-d-announce]

On Thursday, 25 January 2018 at 19:22:19 UTC, Seb wrote:

On Monday, 15 January 2018 at 22:50:55 UTC, kinke wrote:

On Sunday, 14 January 2018 at 04:18:41 UTC, Seb wrote:

3) IR (LDC only)

https://run.dlang.io/is/BOTNDf


Thanks Seb. I particularly like this feature. After a quick 
glance, the debuginfos (-g) seem to be needlessly enforced, 
bloating the IR.
A colored output would be a nice improvement; there's a plugin 
for the google-code-prettify JS library [1], which wouldn't 
increase the server load. Maybe there's one for asm too.


[1] https://gist.github.com/ndabas/2850418


Easy enough:

https://github.com/dlang-tour/core-exec/pull/18

I also added syntax coloring for LLVM IR, Assembly and D's AST 
dump:


https://github.com/dlang-tour/core/pull/643

Anything else that would help you?


Is it possible to show demangled names in asm output (or this 
option already available)?

Re: Intellij D Language v1.15.2

[ikod via Digitalmars-d-announce]

On Thursday, 23 November 2017 at 20:11:01 UTC, singingbush wrote:
Hi all. A new release intellij-dlanguage plugin has been made 
available for download from the Jetbrains repository this week.



There is also support for debugging with GDB (since v1.14 1st 
Nov). We need to completely overhaul our documentation as some 
of it is outdated now and there is no mention of the gdb 
support. If anyone with Java/Kotlin experience wants to get 
involved with helping squash bugs then we welcome pull requests 
so please feel free to browse the issues on our github 
repository and get involved.


https://github.com/intellij-dlanguage/intellij-dlanguage

If you find the plugin helpful please also rate the plugin:

https://plugins.jetbrains.com/plugin/8115-d-language


Thanks a lot for the Structure view!
Plugin became better with each release.

dlang-requetst v0.5.3 released

[ikod via Digitalmars-d-announce]

Hello,

Since last announced version 0.5.1 several bugfixes and 
improvements were added:


1. Improvement: "bind" local address enabled on outgoing 
connections (https://github.com/ikod/dlang-requests/issues/51)


2. Improvement: SNI for ssl connection implemented, as some 
servers require SNI 
(https://github.com/ikod/dlang-requests/issues/55)


3. Bugfix: Segfault fetching some URLs 
(https://github.com/ikod/dlang-requests/issues/54)


4. Improvement: you do not need openssl at all if you do not use 
https (https://github.com/ikod/dlang-requests/issues/53)


5. Minor fixes for dmd v2.076.0

6. Docs on the wiki were cleaned up and made more actual.

dlang-requests is HTTP/FTP client library, inspired by 
python-requests with goals:


small memory footprint
performance
simple, high level API
native D implementation

Re: dlang-requetst: openssl 1.1 compatible release

[ikod via Digitalmars-d-announce]

On Saturday, 5 August 2017 at 09:46:44 UTC, Jack Applegame wrote:

On Friday, 4 August 2017 at 18:28:23 UTC, ikod wrote:

On Friday, 4 August 2017 at 17:06:16 UTC, Jack Applegame wrote:
Does dlang-requests support binding interface for outgoing 
connection, like curl --interface option?


No, but this can be done. It would be nice if you post issue 
on github.



https://github.com/ikod/dlang-requests/issues/51


Hello,
Thanks for posting issue, 'bind' enabled for outgoing connections 
in v0.5.1 release

Re: dlang-requetst: openssl 1.1 compatible release

[ikod via Digitalmars-d-announce]

On Friday, 4 August 2017 at 17:06:16 UTC, Jack Applegame wrote:

On Thursday, 3 August 2017 at 06:33:38 UTC, ikod wrote:

Hello,

Since version 0.5.0 dlang-requests has become compatible with 
both 1.0.x and 1.1.x versions of openssl library.


Please try and report any issues on github.
Thanks!

dlang-requests is HTTP/FTP client library, inspired by 
python-requests with goals:


small memory footprint
performance
simple, high level API
native D implementation

https://code.dlang.org/packages/requests
https://github.com/ikod/dlang-requests


Does dlang-requests support binding interface for outgoing 
connection, like curl --interface option?


No, but this can be done. It would be nice if you post issue on 
github.


Thanks!

Re: dlang-requetst: openssl 1.1 compatible release

[ikod via Digitalmars-d-announce]

On Thursday, 3 August 2017 at 14:03:02 UTC, aberba wrote:

On Thursday, 3 August 2017 at 10:02:24 UTC, Temtaime wrote:

On Thursday, 3 August 2017 at 09:57:11 UTC, Suliman wrote:

On Thursday, 3 August 2017 at 06:33:38 UTC, ikod wrote:

Hello,

Since version 0.5.0 dlang-requests has become compatible 
with both 1.0.x and 1.1.x versions of openssl library.


Please try and report any issues on github.
Thanks!

dlang-requests is HTTP/FTP client library, inspired by 
python-requests with goals:


small memory footprint
performance
simple, high level API
native D implementation

https://code.dlang.org/packages/requests
https://github.com/ikod/dlang-requests


Vote for including it in Phobos instead curl!


Curl is well-tested and has a great number of features.


So the assumption is that curl has been there since forever, so 
its well tested. Anything new cannot be stable enough for 
everyday use?


If there's anything keeping dlang-requests from been integrated 
into Phobos (or a derivative), then lets report it as a 
bug...cus curl is not convenient to use without some 
abstraction layer.



HTTP is the most used protocol for everything. JavaScript is 
the most used language and pretty much everything done with 
JavaScript has HTTP involved.


Moving any third party code to std library have both "pro" and 
"contra" and it was discussed several times. From my point of 
view there is nothing wrong with modules outside of std library 
as long as these modules are visible to newcomers, well 
documented and have developer support.

dlang-requetst: openssl 1.1 compatible release

[ikod via Digitalmars-d-announce]

Hello,

Since version 0.5.0 dlang-requests has become compatible with 
both 1.0.x and 1.1.x versions of openssl library.


Please try and report any issues on github.
Thanks!

dlang-requests is HTTP/FTP client library, inspired by 
python-requests with goals:


small memory footprint
performance
simple, high level API
native D implementation

https://code.dlang.org/packages/requests
https://github.com/ikod/dlang-requests

dlang-requests 0.4.0 released

[ikod via Digitalmars-d-announce]

Hi,

dlang-requests v0.4.0 released.

Major new feature - threaded request pool, which works like 
InputRange. You can write code like this:



 iota(10).
 map!(n => "%d".format(n).representation).
 map!(d => 
Job("http://httpbin.org/post;).method("POST").data(d)).

 pool(5).
 filter!(r => r.code == 200).
 each!(r => writeln(cast(string)r.data));

You can supply requests to different hosts/ports, and Pool object 
will run your requests in parallel, trying to reduce response 
delay by routing each request to proper keep-alive connection.


Any InputRange with front() type Job can be supplied to pool. 
Pool produce InputRange of Result's.


Another improvement related to FTP authentication. You can use 
same method (rq.authenticator) as for HTTP requests to avoid 
problems when login/password contains reserved symbols (like "@").


If you have any ideas to implement, or issues to report - you are 
welcome to project page https://github.com/ikod/dlang-requests


Best regards!

Re: Low-level and Hight-level Kafka bindings

[ikod via Digitalmars-d-announce]

On Monday, 14 November 2016 at 16:17:33 UTC, Ilya Yaroshenko 
wrote:
Kafka is most reliable distributed log system. It is used as 
many2many messaging system.


# Links
C library: https://github.com/edenhill/librdkafka
Low-level bindings: https://github.com/DlangApache/librdkafka/
High-level bindings: 
https://github.com/tamediadigital/librdkafka-d


Best regards,
Ilya


Very nice, thanks for the info!

Re: Release D 2.072.0

[ikod via Digitalmars-d-announce]

Hello,

Here is another bug:

https://issues.dlang.org/show_bug.cgi?id=16667

app unit tests start to fail on std.conv internal unit tests.

dlang-requests 0.2.0 released

[ikod via Digitalmars-d-announce]

Hello,

dlang-requests v0.2.0 released.

Major feature improvement:
1. Nonblocking vibe.d sockets support. That means you can safely 
use dlang-requests in vibe.d applications. Thanks to Johannes 
Pfau (https://github.com/jpf91).


Minor improvements:
2. Dump network data when verbosity level>=3
3. Throw exception when maxRedirects reached instead of returning 
redirect status code.


dlang-request is D http client library, inspired by 
python-requests.


Project code: https://github.com/ikod/dlang-requests

Re: dlang-requests 0.1.7 released

[ikod via Digitalmars-d-announce]

On Wednesday, 22 June 2016 at 02:04:25 UTC, Zekereth wrote:

On Saturday, 11 June 2016 at 23:03:52 UTC, ikod wrote:

Hello,

Dlang-requests is library created under influence of 
Python-requests, with primary goal of easy to use and 
performance.


I have a couple of questions. If a url can't be found and I 
catch the ConnectError exception and handle it[2] I still get 
an additional error[1] printed. For example:


try
{
getContent(url);
}
catch(ConnectError ex)
{
debug writeln("Error getting page: ", ex.msg);
}

Will error with this:

[1]2016-06-21T20:59:15.073:streams.d:connect:750 Failed to 
connect: can't resolve www.systemcontrolpanel.com - getaddrinfo 
error: Name or service not known


[2]Error getting page: Can't connect to 
www.systemcontrolpanel.com:80: getaddrinfo error: Name or 
service not known


The additional error[1] will still print in release build. The 
only workaround I could find was to check getAddress myself and 
ensure that the page exists before calling getContent.


Thanks!


Hello,

It makes no sense to log error message and throw exception in 
next line, so I'll remove this error messages from the code.


Thanks for report(but better place it on the github project page 
so that I can link code changes with bug reports)!

Re: dlang-requests 0.1.7 released

[ikod via Digitalmars-d-announce]

On Friday, 17 June 2016 at 17:31:55 UTC, ikod wrote:
On Tuesday, 14 June 2016 at 14:59:37 UTC, Andrei Alexandrescu 
wrote:

On 6/11/16 7:03 PM, ikod wrote:

Hello,

Dlang-requests is library created under influence of 
Python-requests,

with primary goal of easy to use and performance.


...


Thanks! Does the project have a dub presence? How does it 
compare feature-wise and speed-wise with curl? -- Andrei


Hello,

Finally, I made some improvements and run minimal performance 
tests against command-line curl. I wrote simple code for file 
download using dlang-requests, run it and curl for the same 
urls(httpbin server on my notebook) and compare "total", 
"system", and "user" time for different cases. You can find 
numbers and code below.
So my conclusion is - performances are comparable for these 
cases, but there is some field for improvement in 
dlang-requests.


Case 1 - 50Mb of random data, no any encoding
Case 2 - 50Mb of random data, transfer chunked
Case 3 - 50Mb of random data, transfer chunked, content gzip

  measured times, sec
-
 |   user|   system  |   total
 Case|---|---|---
 | d-r | curl|  d-r| curl| d-r | curl
-|-|-|-|-|-|-
  1  | 0.17| 0.14| 0.20| 0.32| 51.7| 52.2
  2  | 0.19| 0.11| 0.15| 0.21| 51.8| 51.9
  3  | 0.21| 0.15| 0.11| 0.15| 51.5| 52.1


Finally after all unnecessary memory movements were cleaned up, 
results for dlang-requests and curl almost equals.


  measured times, sec
 -
  |   user|   system  |   total
  Case|---|---|---
  | d-r | curl|  d-r| curl| d-r | curl
 -|-|-|-|-|-|-
   1  | 0.14| 0.14| 0.15| 0.32| 53.3| 52.2
   2  | 0.13| 0.11| 0.11| 0.21| 51.1| 51.9
   3  | 0.18| 0.15| 0.08| 0.15| 51.2| 52.1




import std.stdio;
import requests;

pragma(lib, "ssl");
pragma(lib, "crypto");

void main()
{
auto sink = File("/dev/null", "wb");
auto rq = Request();
rq.useStreaming = true;
auto rs = 
rq.get("http://127.0.0.1:8080/stream-bytes/5120;);

auto stream = rs.receiveAsRange();
while(!stream.empty) {
sink.rawWrite(stream.front);
stream.popFront;
}
}

Re: dlang-requests 0.1.7 released

[ikod via Digitalmars-d-announce]

On Tuesday, 14 June 2016 at 14:59:37 UTC, Andrei Alexandrescu 
wrote:

On 6/11/16 7:03 PM, ikod wrote:

Hello,

Dlang-requests is library created under influence of 
Python-requests,

with primary goal of easy to use and performance.


...


Thanks! Does the project have a dub presence? How does it 
compare feature-wise and speed-wise with curl? -- Andrei


Hello,

Finally, I made some improvements and run minimal performance 
tests against command-line curl. I wrote simple code for file 
download using dlang-requests, run it and curl for the same 
urls(httpbin server on my notebook) and compare "total", 
"system", and "user" time for different cases. You can find 
numbers and code below.
So my conclusion is - performances are comparable for these 
cases, but there is some field for improvement in dlang-requests.


Case 1 - 50Mb of random data, no any encoding
Case 2 - 50Mb of random data, transfer chunked
Case 3 - 50Mb of random data, transfer chunked, content gzip

  measured times, sec
-
 |   user|   system  |   total
 Case|---|---|---
 | d-r | curl|  d-r| curl| d-r | curl
-|-|-|-|-|-|-
  1  | 0.17| 0.14| 0.20| 0.32| 51.7| 52.2
  2  | 0.19| 0.11| 0.15| 0.21| 51.8| 51.9
  3  | 0.21| 0.15| 0.11| 0.15| 51.5| 52.1


import std.stdio;
import requests;

pragma(lib, "ssl");
pragma(lib, "crypto");

void main()
{
auto sink = File("/dev/null", "wb");
auto rq = Request();
rq.useStreaming = true;
auto rs = 
rq.get("http://127.0.0.1:8080/stream-bytes/5120;);

auto stream = rs.receiveAsRange();
while(!stream.empty) {
sink.rawWrite(stream.front);
stream.popFront;
}
}

Re: dlang-requests 0.1.7 released

[ikod via Digitalmars-d-announce]

On Tuesday, 14 June 2016 at 14:59:37 UTC, Andrei Alexandrescu 
wrote:

On 6/11/16 7:03 PM, ikod wrote:

Hello,

Dlang-requests is library created under influence of

...
Code and docs available at 
https://github.com/ikod/dlang-requests or as

dub package.

Thanks Dlang authors and community for excellent language.


Thanks! Does the project have a dub presence? How does it 
compare feature-wise and speed-wise with curl? -- Andrei


Yes, it is in the dub repository under name "requests"
As for speed comparison, I'll post numbers in a day or two, when 
I'll finish some performance improvements.


Feature matrix for libcurl(per 
https://curl.haxx.se/docs/features.html) and dlang-requests: 
https://github.com/ikod/dlang-requests/blob/master/docs/matrix.txt Please note, that dlang-requests supports only http,https and ftp.

Re: dlang-requests 0.1.7 released

[ikod via Digitalmars-d-announce]

On Monday, 13 June 2016 at 08:29:28 UTC, Alexander Milushev wrote:

On Monday, 13 June 2016 at 00:09:12 UTC, ikod wrote:
On Sunday, 12 June 2016 at 22:37:34 UTC, Alexander Milushev 
wrote:

On Saturday, 11 June 2016 at 23:03:52 UTC, ikod wrote:

[...]


Hi, this is great project, but what about vibe.d 
compatibility?


This project doesn't depend on vibe.d, and I'm not sure if it 
can be directly used from inside vibe.d application, as it use 
blocked socket io.


Yeah, I understood, but is it possible to add support of vibe.d 
sockets too?


If you want to use requests from vibe.d application then this 
probably can be done using 
https://github.com/rejectedsoftware/vibe.d/wiki#integrating-non-vibed-io
But this require writing some wrapper. I can investigate this in 
the future.


Thanks!

Re: dlang-requests 0.1.7 released

[ikod via Digitalmars-d-announce]

On Sunday, 12 June 2016 at 22:37:34 UTC, Alexander Milushev wrote:

On Saturday, 11 June 2016 at 23:03:52 UTC, ikod wrote:

Hello,

Dlang-requests is library created under influence of 
Python-requests, with primary goal of easy to use and 
performance.


It provide interfaces for HTTP(S) and FTP requests. You can 
tune request details, but for most cases you will need single 
and simple API call.


Latest release added streaming content from server and some 
performance improvements (avoiding some of unnecessary buffer 
copying).


Code and docs available at 
https://github.com/ikod/dlang-requests or as dub package.


Thanks Dlang authors and community for excellent language.


Hi, this is great project, but what about vibe.d compatibility?


This project doesn't depend on vibe.d, and I'm not sure if it can 
be directly used from inside vibe.d application, as it use 
blocked socket io.

dlang-requests 0.1.7 released

[ikod via Digitalmars-d-announce]

Hello,

Dlang-requests is library created under influence of 
Python-requests, with primary goal of easy to use and performance.


It provide interfaces for HTTP(S) and FTP requests. You can tune 
request details, but for most cases you will need single and 
simple API call.


Latest release added streaming content from server and some 
performance improvements (avoiding some of unnecessary buffer 
copying).


Code and docs available at https://github.com/ikod/dlang-requests 
or as dub package.


Thanks Dlang authors and community for excellent language.

Re: New deb packages on d-apt

[ikod via Digitalmars-d-announce]

Hello,

On Wednesday, 6 April 2016 at 13:06:54 UTC, Jordi Sayol wrote:

d-apt has three new deb packages for Dfix, Dfmt and Dscanner.

$ sudo apt-get install dfix dfmt dscanner

d-apt 



JFYI - pegged is ready fo packaging: 
https://github.com/PhilippeSigaud/Pegged