Trying to reduce memory usage
I'm trying to read in a text file that has many duplicated lines
and output a file with all the duplicates removed. By the end of
this code snippet, the memory usage is ~5x the size of the infile
(which can be multiple GB each), and when this is in a loop the
memory usage becomes unmanageable and often results in an
OutOfMemory error or just a complete lock up of the system. Is
there a way to reduce the memory usage of this code without
sacrificing speed to any noticeable extent? My assumption is the
.sort.uniq needs improving, but I can't think of an easier/not
much slower way of doing it.
Windows 10 x64
LDC - the LLVM D compiler (1.21.0-beta1):
based on DMD v2.091.0 and LLVM 10.0.0
---
auto filename = "path\\to\\file.txt.temp";
auto array = appender!(string[]);
File infile = File(filename, "r");
foreach (line; infile.byLine) {
array ~= line.to!string;
}
File outfile = File(stripExtension(filename), "w");
foreach (element; (array[]).sort.uniq) {
outfile.myrawWrite(element ~ "\n"); // used to not print the \r
on windows
}
outfile.close;
array.clear;
array.shrinkTo(0);
infile.close;
---
Thanks.
Re: Is using floating point type for money/currency a good idea?
On Monday, 20 May 2019 at 11:10:32 UTC, Boqsc wrote:
https://dlang.org/spec/float.html
I'm frozen in learning basics of D lang since I want to create
a simple game and I really would like a clean and simple code,
however to me floating points are magic.
https://wiki.dlang.org/Review_Queue
Since std.decimal is still work in progress and apparently its
development is stuck for a while, should I just somehow use
floating point to store currency or wait until Decimal package
will be finally included into std Phobos of D lang?
Normally I would say no, no and no. Rounding issues will kill
you every time. However:
import std.stdio;
import std.range;
void main()
{
foreach (i; iota(1, 1000)) {
writefln("%f", cast(float) i / 100.0);
}
}
Doesn't show any rounding issues, but you still might hit them at
large values...I just remember back in the day having numbers
like 0.199, but maybe that problem has been solved.
What would be safer is to do "fixed point" math, i.e. use an
integer and when displaying the value convert it to float and
divide by 100. So if the user has 50 cents they internal value
would be 50, if you give them a dollar twenty five (125) then
they would have 175, and when you go to display it, dividing by
100 will display 1.75.
As long as you are not multiplying money by money you will be
fine. 50 cents times 5 is 2.50 (50 * 5 = 250), however 50 cents
times 50 cents is 2500, which makes no sense but so does
multiplying money by money...
Re: Linked List iterating over and inserting an element around (before/after) the current position.
Thank you, that helps big time.
This is just more curiosity, but do you happen to know why I have
to use DList.linearRemove() instead of DList.remove()?
import std.stdio;
import std.container.dlist;
import std.algorithm;
import std.range;
void main()
{
auto list = make!DList("the", "quick", "brown", "fox");
auto range = list[].find("quick").take(1);
list.remove(range);
list[].each!writeln; // the brown fox
}
This results in a compiler issue:
onlineapp.d(10): Error: function
std.container.dlist.Container!string.DList.remove(Range r) is not
callable using argument types (Take!(Range))
onlineapp.d(10):cannot pass argument range of type
Take!(Range) to parameter Range r
Changing "remove" to "linearRemove" fixes it, but both the remove
and linearRemove functions take a Range object, and linearRemove
seems to just be a pass through to remove.
Linked List iterating over and inserting an element around (before/after) the current position.
Just started looking at D this weekend, coming from a
C++/Java/Go/Rust background and it's really not going well.
Trying to write something to play with the language and need a
linked list, looking in std.container you have a single or doubly
linked list...great.
Now how to I iterate over it and look for a value and then modify
it, and maybe insert a new element before or after that
element.After spending way to long on the API realized I need
to be looking at std.range (I think, maybe not, I'm not sure).
So off to the std.range documentation, which recommended reading
http://ddili.org/ders/d.en/ranges.html, okay read that, boy if I
want to display the contents of range I'm ALL set, to bad I would
like to modify a list, so I seem to be no closer to my goaloh
and by the way if you think:
import std.stdio;
import std.range;
void main() {
int[] slice = [ 1, 2, 3 ];
int[] slice2 = slice;
put(slice2, 100);
writeln(slice2);
writeln(slice);
}
Resulting in:
[2, 3]
[100, 2, 3]← expected result
Is obvious, I have some bad news for you.
Now I'm thinking maybe std.algorithm.comparison, maybe iteration
or maybe mutation seems promising, at least that's what I want to
do, mutate a listhey there is a swap...oh it doesn't deal
with ranges...
Maybe swapAt but that sounds like I need an index, and indexes
with linked list don't really perform well...but it does take a
rangeokay I think that is what I need, so in a foreach loop,
looking at https://tour.dlang.org/tour/en/basics/ranges I would
call swapAt and pass in __rangeCopy, I think I can update, happy
days, now to figure out how to add something before or after that
element..
So in short I guess my complaint is that you show all this stuff
about displaying ranges and removing items from the front or
end...so if I want a queue we're set, but nothing about modifying
them...unless it's so blatantly obvious I'm just missing it,
which could be the case.
Trying to extract string from curl redirected URL
I have a website (say https://website.com) that I need to log in to, it will do a few 302 redirects and then I will end up with a unique session ID in the URL (such as https://website.com/welcome.html?s=636853677441448706). Is there some way of extracting this ID (I'll need it later for other things)? I'm using std.net.curl's HTTP, and when I turn verbose on and do the login POST I can see that it is in fact redirecting to the URL with the session ID. I can see that etc.c.curl has CurlInfo.effective_url and CurlInfo.redirect_url that I can use with curl_easy_getinfo, but that needs a CURL*. HTTP.handle.handle is private, so I can't get a CURL* out of the Curl struct. Is there something I'm missing? It it possible to access the private handle? Is there a better way to do what I'm trying to achieve?
Re: libcurl acting differently to curl.exe
The POST C code was:
/* Sample code generated by the curl command line tool
**
* All curl_easy_setopt() options are documented at:
* https://curl.haxx.se/libcurl/c/curl_easy_setopt.html
/
#include
int main(int argc, char *argv[])
{
CURLcode ret;
CURL *hnd;
hnd = curl_easy_init();
curl_easy_setopt(hnd, CURLOPT_BUFFERSIZE, 102400L);
curl_easy_setopt(hnd, CURLOPT_URL, "https://foo.com/logon.php;);
curl_easy_setopt(hnd, CURLOPT_POSTFIELDS,
"username=user=pass=submit");
curl_easy_setopt(hnd, CURLOPT_POSTFIELDSIZE_LARGE,
(curl_off_t)52);
curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.57.0");
curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION,
(long)CURL_HTTP_VERSION_2TLS);
curl_easy_setopt(hnd, CURLOPT_CAINFO,
"C:\\Users\\Josh\\Downloads\\curl-7.57.0-win64-mingw\\bin\\curl-ca-bundle.crt");
curl_easy_setopt(hnd, CURLOPT_SSH_KNOWNHOSTS,
"C:\\Users\\Josh\\AppData\\Roaming/_ssh/known_hosts");
curl_easy_setopt(hnd, CURLOPT_COOKIEFILE, "cookie.dat");
curl_easy_setopt(hnd, CURLOPT_COOKIEJAR, "cookie.dat");
curl_easy_setopt(hnd, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L);
/* Here is a list of options the curl code used that cannot get
generated
as source easily. You may select to either not use them or
implement
them yourself.
CURLOPT_WRITEDATA set to a objectpointer
CURLOPT_INTERLEAVEDATA set to a objectpointer
CURLOPT_WRITEFUNCTION set to a functionpointer
CURLOPT_READDATA set to a objectpointer
CURLOPT_READFUNCTION set to a functionpointer
CURLOPT_SEEKDATA set to a objectpointer
CURLOPT_SEEKFUNCTION set to a functionpointer
CURLOPT_ERRORBUFFER set to a objectpointer
CURLOPT_STDERR set to a objectpointer
CURLOPT_DEBUGFUNCTION set to a functionpointer
CURLOPT_DEBUGDATA set to a objectpointer
CURLOPT_HEADERFUNCTION set to a functionpointer
CURLOPT_HEADERDATA set to a objectpointer
*/
ret = curl_easy_perform(hnd);
curl_easy_cleanup(hnd);
hnd = NULL;
return (int)ret;
}
/ End of sample code /
libcurl acting differently to curl.exe
I'm trying to use libcurl in D to download a page that requires
logging in first. At the moment though, I can't even get the
logging in working. I tried with curl.exe, got it working, and
used the --libcurl command to export C code that I then turned
into (I think) equivalent D code.
The problem is that when I POST the login form, curl.exe gets a
HTTP 200 to the correct page, whereas libcurl gets a HTTP 302
back to the login page, which is the same behaviour I noticed
when cookies weren't being saved/reused.
I've tried with the HTTP struct functions, and also setting curl
options explicitly like the C code does. Is there something I've
missed?
import std.file;
import std.net.curl;
import std.stdio;
void main() {
getLogon();
postLogon();
getIndex();
}
void getLogon()
{
auto http = HTTP();
http.onReceive = (ubyte[] data) { /+ drop +/
std.file.write("logon1.html", data);
/+writeln(cast(char[])(data)); stdout.flush;+/ return
data.length; };
http.handle.set(CurlOption.tcp_nodelay, 1);
http.handle.set(CurlOption.buffersize, 102400);
http.handle.set(CurlOption.noprogress, 1);
//http.handle.set(CurlOption.useragent, "curl/7.57.0");
http.handle.set(CurlOption.maxredirs, 50);
//http.handle.set(CurlOption.cainfo,
"C:\\Users\\Josh\\Downloads\\curl-7.57.0-win64-mingw\\bin\\curl-ca-bundle.crt");
http.handle.set(CurlOption.cookiejar, "cookie.dat");
http.handle.set(CurlOption.cookiefile, "cookie.dat");
http.handle.set(CurlOption.verbose, 1);
http.handle.set(CurlOption.url, "https://foo.com/logon.php;);
http.method(HTTP.Method.get);
http.perform();
}
void doLogon()
{
auto http = HTTP();
http.onReceive = (ubyte[] data) { /+ drop +/
std.file.write("logon2.html", data);
/+writeln(cast(char[])(data)); stdout.flush;+/ return
data.length; };
http.handle.set(CurlOption.tcp_nodelay, 1);
http.handle.set(CurlOption.buffersize, 102400);
http.handle.set(CurlOption.noprogress, 1);
//http.handle.set(CurlOption.useragent, "curl/7.57.0");
http.handle.set(CurlOption.maxredirs, 50);
//http.handle.set(CurlOption.cainfo,
"C:\\Users\\Josh\\Downloads\\curl-7.57.0-win64-mingw\\bin\\curl-ca-bundle.crt");
http.handle.set(CurlOption.cookiejar, "cookie.dat");
http.handle.set(CurlOption.cookiefile, "cookie.dat");
http.handle.set(CurlOption.verbose, 1);
http.handle.set(CurlOption.url, "https://foo.com/logon.php;);
http.handle.set(CurlOption.postfields,
"username=user=pass=submit");
http.handle.set(CurlOption.postfieldsize_large, 52);
http.method(HTTP.Method.post);
http.perform();
}
void getIndex()
{
auto http = HTTP();
http.onReceive = (ubyte[] data) { /+ drop +/
std.file.write("index.html", data);
/+writeln(cast(char[])(data)); stdout.flush;+/ return
data.length; };
http.handle.set(CurlOption.tcp_nodelay, 1);
http.handle.set(CurlOption.buffersize, 102400);
http.handle.set(CurlOption.noprogress, 1);
//http.handle.set(CurlOption.useragent, "curl/7.57.0");
http.handle.set(CurlOption.maxredirs, 50);
//http.handle.set(CurlOption.cainfo,
"C:\\Users\\Josh\\Downloads\\curl-7.57.0-win64-mingw\\bin\\curl-ca-bundle.crt");
http.handle.set(CurlOption.cookiejar, "cookie.dat");
http.handle.set(CurlOption.cookiefile, "cookie.dat");
http.handle.set(CurlOption.verbose, 1);
http.handle.set(CurlOption.url, "https://foo.com/index.php;);
http.method(HTTP.Method.get);
http.perform();
}
libcurl verbose dump: https://pastebin.com/Sq60CLHV
curl.exe verbose dump: https://pastebin.com/KBDDNq9k
Thanks,
Josh
Re: Problems with function as parameter
On Friday, 22 September 2017 at 03:26:36 UTC, Mike Parker wrote:
On Friday, 22 September 2017 at 02:22:46 UTC, Josh wrote:
src\mixer.d(80,22): Error: function pointer
Mix_ChannelFinished (extern (C) void function(int channel)) is
not callable using argument types (extern (C) void
delegate(int channel))
Code:
void unmuteAfterPlaySound()
{
Mix_ChannelFinished();
}
extern (C) void channelDone(int channel)
{
unmuteMusic();
}
The error message indicates that `channelDone` is a member of a
class or a struct. A pointer to a member function is a delegate
(or closure), not a function pointer. Free functions, static
nested functions, and static member functions all produce
function pointer. Non-static member functions and non-static
nested functions all produce delegates/closures. See the docs:
https://dlang.org/spec/function.html#closures
If you need to manipulate instance members from a C callback,
you'll need a way to implement a mechanism to work out which
instance you need.
Perfect, that's the info I needed. As these functions were in a
class, setting channelDone and unmuteMusic to static worked.
As an aside, in that doc it says "The .funcptr property of a
delegate will return the function pointer value as a function
type". So I also tried
Mix_ChannelFinished(().funcptr); and this compiled,
but caused a segfault when
the callback ran. What would have caused this? Is it because it's
a C function?
Re: Problems with function as parameter
On Friday, 22 September 2017 at 02:18:34 UTC, Mike Parker wrote:
On Thursday, 21 September 2017 at 22:05:22 UTC, Matt Jones
wrote:
On Thursday, 21 September 2017 at 20:21:58 UTC, Josh wrote:
I'm trying to write a callback function for SDL_mixer through
Derelict, but this is the first time I've tried to use a
function as a parameter, and so I think I'm just making a
minor mistake somewhere.
[...]
Make it a C function, not a D function:
extern (C) void channelDone(int channel)
{
unmuteMusic();
}
and use & to reference the function instead of calling it:
Mix_ChannelFinished();
It should be extern(C) and, ideally, nothrow. The binding
should be enforcing nothrow, but currently doesn't. I'll change
that now.
Why should the binding force nothrow? I don't understand why you
HAVE to not throw exceptions. Is it because of the C -> D aspect?
Re: Problems with function as parameter
On Thursday, 21 September 2017 at 22:05:22 UTC, Matt Jones wrote:
On Thursday, 21 September 2017 at 20:21:58 UTC, Josh wrote:
I'm trying to write a callback function for SDL_mixer through
Derelict, but this is the first time I've tried to use a
function as a parameter, and so I think I'm just making a
minor mistake somewhere.
[...]
Make it a C function, not a D function:
extern (C) void channelDone(int channel)
{
unmuteMusic();
}
and use & to reference the function instead of calling it:
Mix_ChannelFinished();
Thanks for the help, but when I try that, I get:
src\mixer.d(80,22): Error: function pointer Mix_ChannelFinished
(extern (C) void function(int channel)) is not callable using
argument types (extern (C) void delegate(int channel))
Code:
void unmuteAfterPlaySound()
{
Mix_ChannelFinished();
}
extern (C) void channelDone(int channel)
{
unmuteMusic();
}
Problems with function as parameter
I'm trying to write a callback function for SDL_mixer through
Derelict, but this is the first time I've tried to use a function
as a parameter, and so I think I'm just making a minor mistake
somewhere.
The C SDL docs say:
// make a channelDone function
void channelDone(int channel)
{
printf("channel %d finished playing.\n", channel);
}
...
// set the callback for when a channel stops playing
Mix_ChannelFinished(channelDone);
And my D code is:
void unmuteAfterPlaySound()
{
Mix_ChannelFinished(channelDone);
}
void channelDone(int channel)
{
unmuteMusic();
}
But DMD seems to be trying to run channelDone() i.e. with no
parameters and parentheses omitted, and is giving this error:
Error: function Mixer.channelDone (int channel) is not callable
using argument types ()
Error: function pointer Mix_ChannelFinished (extern (C) void
function(int channel)) is not callable using argument types
(_error_)
Is anyone able to see what I've done wrong? Any help would be
appreciated, thanks.
Starting threads inside class
My goal with the code below is to eventually have my main
communicate with Foo and Bar classes listening for packets on a
different address/port, each in a separate thread. They would
then communicate with Foobaz and Barbaz threads respectively to
do other work. In trying to get just Foo working though, I'm
getting this error:
Error: template std.concurrency.spawn cannot deduce function from
argument types !()(void delegate(Tid ownerTid), Tid), candidates
are:
D:\D\dmd2\windows\bin\..\..\src\phobos\std\concurrency.d(466):
std.concurrency.spawn(F, T...)(F fn, T args) if
(isSpawnable!(F, T))
The code works fine if I call daemon as a normal function, but it
holds up the main thread.
Is there anything I'm doing wrong here? I'm not accessing
anything outside this class from inside, and the most I would be
doing from outside is accessing the Tid in order to send packets
from my main.
class Foo
{
private string address = "127.0.0.1";
private ushort port = 5;
private ubyte[256] buffer;
private TcpSocket mysock;
Tid listenerd;
this()
{
listenerd = spawn(, thisTid);
}
void setup()
{
mysock = new TcpSocket();
mysock.blocking = true;
try
{
mysock.connect(new InternetAddress(address, port));
}
catch (SocketOSException e)
{
}
}
void initialise()
{
// send init packet
}
void closeConnection()
{
// send close packet
}
void packetHandler()
{
// do something with buffer
}
void daemon(Tid ownerTid)
{
setup();
initialise();
int rxSize = -1;
while (true)
{
rxSize = mysock.receive(buffer);
if (rxSize == 0)
{
break;
}
packetHandler();
}
closeConnection();
}
}
