Re: std.zip expand: memory allocation failed
On Sunday, 24 October 2021 at 14:14:08 UTC, Imperatorn wrote: Create an issue and we can solve it Thanks. I opened an issue. https://issues.dlang.org/show_bug.cgi?id=22436
Re: std.zip expand: memory allocation failed
On Friday, 15 October 2021 at 20:41:36 UTC, Selim Ozel wrote: I am simply trying to unzip a compressed zip file slightly over 1GB. The de-compressed size is about 4 GB. The code is very similar to what's explained in the documentation [1] and it works for smaller files. Anyone has a solution? Memory mapping [2] previously solved some part of my issue but expand is still throwing memory allocation failure. Selim [1] https://dlang.org/phobos/std_zip.html [2] https://forum.dlang.org/thread/mfnleztnwrbgivjvz...@forum.dlang.org It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff. My work around was to call 7z from D Lang and do the compression over there. That worked like a charm. It seems that zip.d [1] calls uncompress routine from zlib.d [2]. Would calling zlib uncompress by chunks solve this memory issue? Any ideas? S [1] https://github.com/dlang/phobos/blob/master/std/zip.d [2] https://github.com/dlang/phobos/blob/master/std/zlib.d
Re: std.zip expand: memory allocation failed
Did you try the MmFile workaround? I did. I also pinpointed the problem, I use x86_mscoff to run dub and it's specific to that architecture selection. It's related to MapViewOfFileEx [1]. I still haven't found a way around it though. [1] https://stackoverflow.com/questions/12121843/mapviewoffileex-valid-lpbaseaddress
std.zip expand: memory allocation failed
I am simply trying to unzip a compressed zip file slightly over 1GB. The de-compressed size is about 4 GB. The code is very similar to what's explained in the documentation [1] and it works for smaller files. Anyone has a solution? Memory mapping [2] previously solved some part of my issue but expand is still throwing memory allocation failure. Selim [1] https://dlang.org/phobos/std_zip.html [2] https://forum.dlang.org/thread/mfnleztnwrbgivjvz...@forum.dlang.org
Re: Socket handle leak and active handle warning with Vibe-D
On Monday, 4 January 2021 at 17:17:10 UTC, Steven Schveighoffer wrote: On 1/2/21 12:52 PM, Selim Ozel wrote: After logging into to 127.0.0.1 for a single time in my browser, if I do a ctrl+c it still leaks two socket handles. With connection Windows 10: Running .\vibe_noleaks.exe [main() INF] Listening for requests on http://[::1]:8080/ [main() INF] Listening for requests on http://127.0.0.1:8080/ [main() INF] Please open http://127.0.0.1:8080/ in your browser. [() INF] Received signal 2. Shutting down. [main() INF] Stopped to listen for HTTP^ requests on C::1:8080 [main( ) INFC:\Software\vibe_noLeaks>] Stopped to listen for HTTP requests on 127.0.0.1:8080 Warning: 2 socket handles leaked at driver shutdown. Warning: 2 socket handles leaked at driver shutdown. This is normal. The server uses keepalive connections, so that in case any more requests arrive on the same connection, the initial connection setup does not need to be established. Well, at least that is what I think is happening. If you want a few seconds (I think 5 or so), then you won't get these. It would be good if vibe-d could provide a way to shut down any keepalive connections when the server is shutting down. -Steve That's interesting. I actually started to dive deeper into those and tried to pinpoint the lines of code that result in additional open sockets upon new http connections; although my understanding of vibe-d is a bit too low at this point to figure out what's exactly happening. I think from a user perspective having something a bit friendlier on warning side would be helpful. Do you have any suggestions in mind towards that? I have a bit of time this week and I could take a stab at it. Best, S
Re: Socket handle leak and active handle warning with Vibe-D
On Sunday, 3 January 2021 at 23:53:54 UTC, aberba wrote: On Friday, 1 January 2021 at 22:07:28 UTC, Selim Ozel wrote: I created the simplest possible example as explained by the Vibe-D community in [1]. The exact source code of what I run is in [2]. On Windows I get a socket handle leak warning on shutdown with crtl+c from terminal after running the executable. [...] On Ubuntu 20.04 I get leaking drivers warning with the same process. [...] I really don't know what this is all about but it is at the core of my Vibe-D development. So any pointers you might have would be very helpful to me. Thanks in advance. S [1] https://vibed.org/blog/posts/a-scalable-chat-room-service-in-d [2] https://github.com/SelimOzel/vibe_noLeaks Add this to your dub.json file to fix it "versions": [ "VibeHighEventPriority" ] This issue should be fixed in next vibe.d release Thanks. Not sure if relevant to this one but I came across that one as well before [1]. The symptom was "The simple hello world app I build with vibe-d does not seem to work on the second compile+execution." on an Ubuntu 20.04 EC2. [1] https://forum.dlang.org/post/mailman.6758.1605999004.31109.digitalmar...@puremagic.com
Re: Socket handle leak and active handle warning with Vibe-D
On Friday, 1 January 2021 at 22:07:28 UTC, Selim Ozel wrote: I created the simplest possible example as explained by the Vibe-D community in [1]. The exact source code of what I run is in [2]. On Windows I get a socket handle leak warning on shutdown with crtl+c from terminal after running the executable. [...] On Ubuntu 20.04 I get leaking drivers warning with the same process. [...] I really don't know what this is all about but it is at the core of my Vibe-D development. So any pointers you might have would be very helpful to me. Thanks in advance. S [1] https://vibed.org/blog/posts/a-scalable-chat-room-service-in-d [2] https://github.com/SelimOzel/vibe_noLeaks For further reference, I also went through this issue [1]. It seems like I am seeing the same behavior as kinexis-uk. S [1] https://github.com/vibe-d/vibe.d/issues/2245
Re: Socket handle leak and active handle warning with Vibe-D
On Saturday, 2 January 2021 at 00:28:43 UTC, Steven Schveighoffer wrote: On 1/1/21 5:07 PM, Selim Ozel wrote: I created the simplest possible example as explained by the Vibe-D community in [1]. The exact source code of what I run is in [2]. On Windows I get a socket handle leak warning on shutdown with crtl+c from terminal after running the executable. [...] On Ubuntu 20.04 I get leaking drivers warning with the same process. [...] I really don't know what this is all about but it is at the core of my Vibe-D development. So any pointers you might have would be very helpful to me. Thanks in advance. 1. the sockets are leaked for a reason that is pretty obscure -- namely, the GC might need to access those sockets as the process is shut down. Prior to this, the end result of vibe.d server was frequently a segfault. 2. The reason they are leaking is most likely because you still have a listening socket somewhere. I wish it would tell you how that socket was allocated, but it doesn't. To fix, make sure all your listening sockets are closed. In my vibe.d app, I have the following: auto listener = listenHTTP(settings, router); scope(exit) listener.stopListening(); I also clean up my session store connection (something I had to add support for in vibe.d), which was a different source of leaking handles. I also clean up database connections, which might be cached. And finally, even with all this, I still get leaking driver messages if an HTTP keepalive socket is open. I really feel like vibe.d should give you the option of not printing this message, as most of the time, it's something you can ignore. -Steve Hey Steve. Thanks a ton for all the tips. FWIW I am writing down my findings below because just maybe they might be helpful for someone else later on. The scope guard seems to have fixed some of the leak complaints. Unfortunately there is still a leak after a single connection. Without connection Windows 10: Running .\vibe_noleaks.exe [main() INF] Listening for requests on http://[::1]:8080/ [main() INF] Listening for requests on http://127.0.0.1:8080/ [main() INF] Please open http://127.0.0.1:8080/ in your browser. [() INF] Received signal 2. Shutting down. [main() INF] Stopped to listen for HTTP requests on ::1:8080 [main() INF] Stopped to listen for HTTP requests on 127.0.0.1:8080 Without connection Ubuntu 20.04: Running ./vibe_noleaks [main() INF] Listening for requests on http://[::1]:8080/ [main() INF] Listening for requests on http://127.0.0.1:8080/ [main() INF] Please open http://127.0.0.1:8080/ in your browser. ^C[main() INF] Received signal 2. Shutting down. [main() INF] Stopped to listen for HTTP requests on ::1:8080 [main() INF] Stopped to listen for HTTP requests on 127.0.0.1:8080 After logging into to 127.0.0.1 for a single time in my browser, if I do a ctrl+c it still leaks two socket handles. With connection Windows 10: Running .\vibe_noleaks.exe [main() INF] Listening for requests on http://[::1]:8080/ [main() INF] Listening for requests on http://127.0.0.1:8080/ [main() INF] Please open http://127.0.0.1:8080/ in your browser. [() INF] Received signal 2. Shutting down. [main() INF] Stopped to listen for HTTP^ requests on C::1:8080 [main( ) INFC:\Software\vibe_noLeaks>] Stopped to listen for HTTP requests on 127.0.0.1:8080 Warning: 2 socket handles leaked at driver shutdown. Warning: 2 socket handles leaked at driver shutdown. I think vibe-d is also leaking more sockets when there is a web interface attached (not included in my toy repository). These haven't stopped me from developing but they are just things I wanted to write down and learn more before building even more infrastructure with it. I might dive into vibe-d codebase at some point too. Best, S
Socket handle leak and active handle warning with Vibe-D
I created the simplest possible example as explained by the Vibe-D community in [1]. The exact source code of what I run is in [2]. On Windows I get a socket handle leak warning on shutdown with crtl+c from terminal after running the executable. [main() INF] Listening for requests on http://[::1]:8080/ [main() INF] Listening for requests on http://127.0.0.1:8080/ [main() INF] Please open http://127.0.0.1:8080/ in your browser. [() INF] Received signal 2. Shutting down. Warning: 2 socket handles leaked at driver^ Cshutdown On Ubuntu 20.04 I get leaking drivers warning with the same process. [main() INF] Listening for requests on http://[::1]:8080/ [main() INF] Listening for requests on http://127.0.0.1:8080/ [main() INF] Please open http://127.0.0.1:8080/ in your browser. ^C[main() INF] Received signal 2. Shutting down. Warning (thread: main): leaking eventcore driver because there are still active handles FD 6 (streamListen) FD 7 (streamListen) Warning (thread: main): leaking eventcore driver because there are still active handles FD 6 (streamListen) FD 7 (streamListen) I really don't know what this is all about but it is at the core of my Vibe-D development. So any pointers you might have would be very helpful to me. Thanks in advance. S [1] https://vibed.org/blog/posts/a-scalable-chat-room-service-in-d [2] https://github.com/SelimOzel/vibe_noLeaks
Re: Calling executable generated by dub with sudo
On Monday, 7 December 2020 at 20:34:36 UTC, Selim Ozel wrote:
Let's say I build my package using dub run from an Ubuntu
terminal. How can I add sudo as the executable is being run? I
tried adding preRunCommands to my dub.sdl as described in [1]
but that just runs sudo and terminal throws an error.
Best,
Selim
[1] https://dub.pm/package-format-sdl
My solution has been slightly different. I use postBuildCommands
and use dub build package:example_backend at my terminal instead
of dub run package:example_backend.
subPackage {
name "example_backend"
sourcePaths "source/example_backend" "lib"
importPaths "lib"
targetName "example_backend"
targetType "executable"
dependency "mysql-native" version="~>2.2.2"
postBuildCommands "sudo ./example_backend"
}
Best,
Selim
Calling executable generated by dub with sudo
Let's say I build my package using dub run from an Ubuntu terminal. How can I add sudo as the executable is being run? I tried adding preRunCommands to my dub.sdl as described in [1] but that just runs sudo and terminal throws an error. Best, Selim [1] https://dub.pm/package-format-sdl
Re: Switch between two structs with csvreader
On Friday, 6 November 2020 at 19:35:47 UTC, H. S. Teoh wrote:
You can use the typeof() operator to capture the type of a
long, unwieldy type in an alias. This is useful if you ever
need to store such a return type somewhere, e.g.:
alias T = typeof(csvReader(...));
struct MyStorage {
T result;
}
MyStorage s;
s.result = csvReader(...);
Let the compiler figure out the type for you. :-)
T
That's great, thanks!
S
Re: Switch between two structs with csvreader
On Thursday, 5 November 2020 at 22:36:36 UTC, Anonymouse wrote: If I'm not mistaken the `csvReader` function returns a range struct, and the full type is something long and unwieldy like `CsvReader!(struct_type1, cast(Malformed)1, string, dchar, string[])`. So just think of `records` as being that. I actually first going this route but couldn't figure out the correct name for that data type. It is quite long. You need two different variables and two different `foreach`es. For the same code to work on both types, the easy solution is templates. Perhaps make the `foreach` part after the reads a templated function that accepts any type passed to it? Embedding the foreach loop inside a template function and deciding on the data type at the higher level function solved my issue. Thanks for the pointer! Best, Selim
Switch between two structs with csvreader
Hi There,
I am trying to switch between two structs as I am using the
csvReader on a raw string. The pseudo-code below throws a "cannot
implicitly convert" error due to difference between struct_type1
and struct_type2. I must be doing something wrong or have a wrong
understanding of how this function works. Could someone give a
good suggestion on handling this?
Thanks!
Best,
Selim
auto records = rawtext.csvReader!struct_type1(';');
if(aControlCondition) {
records = rawtext.csvReader!struct_type2(';');
}
// Iterate through each data row.
foreach (record; records) {
writeln(record);
}
Re: Vibe-D File Question
On Friday, 11 September 2020 at 13:03:16 UTC, James Blachly wrote:
On 9/11/20 7:28 AM, Daniel Kozak wrote:
void fun(HTTPServerRequest req, HTTPServerResponse res) nothrow
{
try
{
res.headers["Content-Disposition"] = "filename=\"muj.csv\"";
res.writeBody("some;csv;data", "text/csv");
}
catch (Exception e)
{}
}
Selim, note the Content-Disposition header in particular.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition
Thank you Daniel and James! The example works. It's exactly what
I was asking for. I got the core idea with the Content
Disposition article.
Vibe-D File Question
It seems like rejected-software forum is flooded with spam, so I decided to ask it here. Is there a way to generate a file -csv for example- on the back-end and serve it to the front-end as a file. Serve static file [1] function does this for files saved on the disk. I want to be able to generate a file during runtime and serve it to the client. Is this possible? Thanks, S [1] https://vibed.org/api/vibe.http.fileserver/serveStaticFile
