Re: Wrapping C++ class with virtual destructor
On Wednesday, 17 February 2021 at 22:53:05 UTC, Gregor Mückl
wrote:
Hi!
How do I wrap an existing C++ class with a virtual destructor
in D? Take, for example, this C++ class:
class Base {
public:
virtual ~Base();
virtual void foo() = 0;
}
What does the equivalent extern(C++) declaration in D look
like? I don't care about calling the destructor. My only
concern is matching the vtable. Changing the C++ code is not an
option in my case.
extern(C++) class Base {
~this();
abstract void foo();
}
Re: How to get output of piped process?
On Wednesday, 17 February 2021 at 14:36:58 UTC, Steven
Schveighoffer wrote:
On 2/17/21 1:58 AM, Jedi wrote:
I an using pipeShell, I have redirected stdout, stderr, and
stdin.
I am trying to read from the output and display it in my app.
I have followed this code almost exactly except I use try wait
and flush because the app is continuously updating the output.
(it outputs a progress text on the same line and I'm trying to
poll it to report to the user)
auto pipes = pipeProcess("my_application", Redirect.stdout |
Redirect.stderr);
scope(exit) wait(pipes.pid);
// Store lines of output.
string[] output;
foreach (line; pipes.stdout.byLine) output ~= line.idup;
// Store lines of errors.
string[] errors;
foreach (line; pipes.stderr.byLine) errors ~= line.idup;
My code
auto p = pipeShell(`app.exe "`~f.name~`"`, Redirect.stdout |
Redirect.stdin | Redirect.stderr);
while(!tryWait(p.pid).terminated)
{
string[] output;
foreach (line; p.stdout.byLine)
You need to be careful here -- this will wait until stdout is
*closed*.
{
output ~= line.idup;
writeln(line);
}
string[] errors;
foreach (line; p.stderr.byLine)
{
errors ~= line.idup;
writeln("Err:"~line);
}
Same thing here.
}
wait(p.pid);
None of this works though. What is strange is that when I
close out the debugger the app starts working(no console
output but I able to see that it is doing something) but is
very slow.
auto p = executeShell(`app.exe "`~f.name~`"`);
Does work, except I have no output or input. I have another
app that I do the exact same code and I can get the output and
parse it, but this is after the app terminates. I imagine the
issue here is that I'm trying to get the output while the app
is running.
Without knowing the pattern of what your app is outputting,
it's hard to tell what will happen.
The most common problem with people dealing with piped output
is not reading data off the pipe, which then makes the child
process hang trying to write to the pipe, because the buffer is
full.
For instance, if your process outputs tons of stuff to stderr,
you will hang, because you are waiting for stdout to be closed
first before you read anything from stderr, the child process
fills up stderr pipe, and is put to sleep waiting for it to be
writable, never closing stdout.
Unfortunately, std.process wraps all the pipes in File structs,
so you have almost no good mechanisms to properly read the data.
WTF?
-Steve
Seriously, I can't simply get the output in real time? Come on,
that is lame, Surely D can do better than that? How hard is it to
get a buffer? Is there any hacks? How can one communicate with an
app using std io if one can't actually communicate until the app
is closed? It makes no sense.
But note that even executeShell doesn't display the output of the
app.exe so it is more than just pipeShell.
The app just outputs text, just like almost every other text. One
shouldn't have to know any pattern, that defeats the purpose. I
should just be able to get the output of the app.exe, and also if
the app is requesting input. This isn't rocket science but it
seems someone wants to turn it in to it?
When the app.exe is running it just prints stuff out, every once
in a while it might ask for input(e.g., to overwrite the file if
it exists, but I can get around that by checking in D)... but
ultimately I just want to consolidate the output it gives so I
need access to it BEFORE the app closes. The app.exe processes
files, takes some time to do so so if I have to wait to display
anything nothing will be displayed for a long time.
Wrapping C++ class with virtual destructor
Hi!
How do I wrap an existing C++ class with a virtual destructor in
D? Take, for example, this C++ class:
class Base {
public:
virtual ~Base();
virtual void foo() = 0;
}
What does the equivalent extern(C++) declaration in D look like?
I don't care about calling the destructor. My only concern is
matching the vtable. Changing the C++ code is not an option in my
case.
Re: null and initialized string comparisons
On Wednesday, 17 February 2021 at 20:48:22 UTC, Martin wrote: is this how it supposed to be? (https://run.dlang.io/is/7B4irm) == compares contents. Both null and "" have empty contents and are interchangable for operators that work on contents. The assert just looks at the pointer, not contents. The `is` operator also looks at pointer (and length).
null and initialized string comparisons
Hi, is this how it supposed to be? (https://run.dlang.io/is/7B4irm) --- string a = null; string t = ""; assert( ! a ); assert( t ); assert( t == a ); --- I have not expected assert(t == a) to be true - i would like to know the argument for why this is correct when at the same time assert(!a) and assert(t) is true.. feels a litle bit js-ish to me
Re: Struct delegate access corruption
On Wednesday, 17 February 2021 at 20:18:53 UTC, Paul Backus wrote: On Wednesday, 17 February 2021 at 19:42:00 UTC, tsbockman wrote: A copy constructor and opAssign can be used to update pointers that are relative to : https://dlang.org/spec/struct.html#struct-copy-constructor Unfortunately this is not enough, because the compiler is free to implicitly move struct instances in memory any time it wants to. See the bug report below for more details: https://issues.dlang.org/show_bug.cgi?id=17448 Until D gets move constructors, structs with interior pointers should be avoided. That bug is about postblits, this(this), not copy constructors: this(ref typeof(this)). Copy constructors were added to the language specifically to fix those sort of problems. From the spec: https://dlang.org/spec/struct.html#struct-postblit WARNING: The postblit is considered legacy and is not recommended for new code. Code should use copy constructors defined in the previous section. For backward compatibility reasons, a struct that explicitly defines both a copy constructor and a postblit will only use the postblit for implicit copying. However, if the postblit is disabled, the copy constructor will be used. If a struct defines a copy constructor (user-defined or generated) and has fields that define postblits, a deprecation will be issued, informing that the postblit will have priority over the copy constructor. However, since issue #17448 is still open I have posted a question to the issue report asking for clarification: https://issues.dlang.org/show_bug.cgi?id=17448#c39
Re: Struct delegate access corruption
On Wednesday, 17 February 2021 at 19:42:00 UTC, tsbockman wrote:
On Wednesday, 17 February 2021 at 17:45:01 UTC, H. S. Teoh
wrote:
I.e., the following is NOT a good idea:
struct S {
void delegate() member;
this() {
member =
}
private void impl();
}
because a pointer to S is stored inside S itself, so as soon
as S gets copied or moved, the delegate context pointer is no
longer valid (or else points to a different copy of the struct
than the one it will be called from).
A copy constructor and opAssign can be used to update pointers
that are relative to :
https://dlang.org/spec/struct.html#struct-copy-constructor
Unfortunately this is not enough, because the compiler is free to
implicitly move struct instances in memory any time it wants to.
See the bug report below for more details:
https://issues.dlang.org/show_bug.cgi?id=17448
Until D gets move constructors, structs with interior pointers
should be avoided.
Re: Struct delegate access corruption
On Wednesday, 17 February 2021 at 17:45:01 UTC, H. S. Teoh wrote:
I.e., the following is NOT a good idea:
struct S {
void delegate() member;
this() {
member =
}
private void impl();
}
because a pointer to S is stored inside S itself, so as soon as
S gets copied or moved, the delegate context pointer is no
longer valid (or else points to a different copy of the struct
than the one it will be called from).
A copy constructor and opAssign can be used to update pointers
that are relative to :
https://dlang.org/spec/struct.html#struct-copy-constructor
// The following program prints 9 with the copy constructor, or 7
if it is commented out:
module app;
import std.stdio;
struct Foo {
int[2] things;
private int* ptr;
this(const(int[2]) things...) inout pure @trusted nothrow
@nogc {
this.things = things;
ptr = &(this.things[0]);
}
// Copy constructor:
this(scope ref const(typeof(this)) that) inout pure @trusted
nothrow @nogc {
this.things = that.things;
this.ptr = this.things.ptr + (that.ptr - that.things.ptr);
}
// Defining a matching opAssign is a good idea, as well:
ref typeof(this) opAssign(scope ref const(typeof(this)) that)
return pure @trusted nothrow @nogc {
__ctor(that);
return this;
}
void choose(const(int) x) pure @trusted nothrow @nogc {
ptr = &(things[x]); }
@property ref inout(int) chosen() return inout pure @safe
nothrow @nogc {
return *ptr; }
}
void main() {
auto foo = Foo(3, 7);
foo.choose(1);
Foo bar = foo;
bar.things[1] = 9;
writeln(bar.chosen);
}
Re: Web crawler/scraping
On Wednesday, 17 February 2021 at 13:13:00 UTC, Adam D. Ruppe
wrote:
On Wednesday, 17 February 2021 at 12:12:56 UTC, Carlos Cabral
wrote:
I'm trying to collect some json data from a website/admin
panel automatically, which is behind a login form.
Does the website need javascript?
If not, my dom.d may be able to help. It can download some
HTML, parse it, fill in forms, then my http2.d submits it (I
never implemented Form.submit in dom.d but it is pretty easy to
make with other functions that are implemented, heck maybe I'll
implement it now if it sounds like it might work).
Or if it is all json you might be able to just craft some
requests with my lib or even phobos' std.net.curl that submits
the login request, saves a cookie, then fetches some json stuff.
I literally just rolled out of bed but in an hour or two I can
come back and make some example code for you if this sounds
plausible.
...and it's working :)
thank you Adam and Ferhat
leaving this here if anyone needs:
```
import std.stdio;
import std.string;
import std.net.curl;
import core.thread;
void main()
{
int waitTime = 5;
auto domain = "https://example.com;;
auto cookiesFile = "cookies.txt";
auto http = HTTP();
http.handle.set(CurlOption.use_ssl, 1);
http.handle.set(CurlOption.ssl_verifypeer, 0);
http.handle.set(CurlOption.cookiefile, cookiesFile);
http.handle.set(CurlOption.cookiejar , cookiesFile);
http.setUserAgent("...");
http.onReceive = (ubyte[] data) { (...) }
http.method = HTTP.Method.get;
http.url = domain ~ "/login";
http.perform();
Thread.sleep(waitTime.seconds);
auto data = "username=user=pass";
http.method = HTTP.Method.post;
http.url = domain ~ "/login";
http.setPostData(data, "application/x-www-form-urlencoded");
http.perform();
Thread.sleep(waitTime.seconds);
http.method = HTTP.Method.get;
http.url = domain ~ "/fetchjson";
http.perform();
}
```
Re: Struct delegate access corruption
On Wed, Feb 17, 2021 at 03:38:00PM +, z via Digitalmars-d-learn wrote:
> So i've upgraded one of my structs to use the more flexible delegates
> instead of function pointers but when the member function tries to
> access the struct's members, the contents are random and the program
> fails.
You're likely running into the struct self-reference problem. Keep in
mind that in D, structs are what Andrei calls "glorified ints", i.e.,
they are value types that get freely copied around and passed around in
registers. Meaning that the address of a struct is likely to change
(and change a lot as it gets passed around), unless you explicitly
allocated it on the heap. So if you have any pointers to the struct
(including implicit pointers like delegate context pointers) stored
inside itself, they are highly likely to become dangling pointers as the
struct gets copied to another place and the old copy goes out of scope.
I.e., the following is NOT a good idea:
struct S {
void delegate() member;
this() {
member =
}
private void impl();
}
because a pointer to S is stored inside S itself, so as soon as S gets
copied or moved, the delegate context pointer is no longer valid (or
else points to a different copy of the struct than the one it will be
called from).
If you need to store references to an aggregate inside itself, you
should be using a class instead. Or be absolutely sure you allocate the
struct on the heap with `new`, AND never pass it around except by
reference (using pointers).
T
--
Almost all proofs have bugs, but almost all theorems are true. -- Paul Pedersen
Re: Struct delegate access corruption
On 2/17/21 10:38 AM, z wrote: So i've upgraded one of my structs to use the more flexible delegates instead of function pointers but when the member function tries to access the struct's members, the contents are random and the program fails. i've isolated the problem by adding writefln calls before calling the delegate and inside the delegate(the functions are defined in the struct as member functions, the delegate itself is set in the constructor) : In the code that uses the delegate : writefln!"test %s"(a, ); T b = a.d();//the delegate While in the most used delegate : writefln!"test2 %s %s"(this, ); The contents and pointers don't match(they're random, full of 0, -nan, -inf and other invalid values), are they(delegates) supposed to be used like this? With structs and delegates, you have to be careful because structs are copied easily, and using a delegate on a struct that is no longer in scope is going to lead to memory corruption. In order to properly ensure delegates are pointing to valid data, make sure the struct is still not moved or overwritten, or it is allocated on the heap. -Steve
Struct delegate access corruption
So i've upgraded one of my structs to use the more flexible delegates instead of function pointers but when the member function tries to access the struct's members, the contents are random and the program fails. i've isolated the problem by adding writefln calls before calling the delegate and inside the delegate(the functions are defined in the struct as member functions, the delegate itself is set in the constructor) : In the code that uses the delegate : writefln!"test %s"(a, ); T b = a.d();//the delegate While in the most used delegate : writefln!"test2 %s %s"(this, ); The contents and pointers don't match(they're random, full of 0, -nan, -inf and other invalid values), are they(delegates) supposed to be used like this? Big thanks
Re: is it posible to compile individual module separately?
On Tuesday, 16 February 2021 at 18:54:08 UTC, Paul Backus wrote: I stand corrected. Shouldn't have trusted the documentation so much, I guess. It makes perfect sense to intuit that --single should be related to --build-mode=singleFile. As it is the build modes aren't explained in the documentation at all[1], so there's no way of knowing, short of running it to see what happens. Not ideal, perhaps. --build-mode= Specifies the way the compiler and linker are invoked. Valid values: separate (default), allAtOnce, singleFile (Nothing about what they do.) [1]: https://dub.pm/commandline
Re: fold on empty range
On Wednesday, 17 February 2021 at 12:58:29 UTC, Mitacha wrote:
On Wednesday, 17 February 2021 at 11:38:45 UTC, Rumbu wrote:
[...]
If you replace `fold` and `splitter` with this, then it doesn't
allocate:
```
auto fn() @nogc {
return only("k1,k2", "k3,k4")
.map!(x => x.splitter(","))
.joiner;
}
void main() {
auto range = fn();
range.writeln;
}
```
Wow, thanks a lot.
Re: How to get output of piped process?
On 2/17/21 1:58 AM, Jedi wrote:
I an using pipeShell, I have redirected stdout, stderr, and stdin.
I am trying to read from the output and display it in my app. I have
followed this code almost exactly except I use try wait and flush
because the app is continuously updating the output. (it outputs a
progress text on the same line and I'm trying to poll it to report to
the user)
auto pipes = pipeProcess("my_application", Redirect.stdout |
Redirect.stderr);
scope(exit) wait(pipes.pid);
// Store lines of output.
string[] output;
foreach (line; pipes.stdout.byLine) output ~= line.idup;
// Store lines of errors.
string[] errors;
foreach (line; pipes.stderr.byLine) errors ~= line.idup;
My code
auto p = pipeShell(`app.exe "`~f.name~`"`, Redirect.stdout |
Redirect.stdin | Redirect.stderr);
while(!tryWait(p.pid).terminated)
{
string[] output;
foreach (line; p.stdout.byLine)
You need to be careful here -- this will wait until stdout is *closed*.
{
output ~= line.idup;
writeln(line);
}
string[] errors;
foreach (line; p.stderr.byLine)
{
errors ~= line.idup;
writeln("Err:"~line);
}
Same thing here.
}
wait(p.pid);
None of this works though. What is strange is that when I close out the
debugger the app starts working(no console output but I able to see that
it is doing something) but is very slow.
auto p = executeShell(`app.exe "`~f.name~`"`);
Does work, except I have no output or input. I have another app that I
do the exact same code and I can get the output and parse it, but this
is after the app terminates. I imagine the issue here is that I'm trying
to get the output while the app is running.
Without knowing the pattern of what your app is outputting, it's hard to
tell what will happen.
The most common problem with people dealing with piped output is not
reading data off the pipe, which then makes the child process hang
trying to write to the pipe, because the buffer is full.
For instance, if your process outputs tons of stuff to stderr, you will
hang, because you are waiting for stdout to be closed first before you
read anything from stderr, the child process fills up stderr pipe, and
is put to sleep waiting for it to be writable, never closing stdout.
Unfortunately, std.process wraps all the pipes in File structs, so you
have almost no good mechanisms to properly read the data.
-Steve
Re: Web crawler/scraping
On Wednesday, 17 February 2021 at 13:13:00 UTC, Adam D. Ruppe wrote: On Wednesday, 17 February 2021 at 12:12:56 UTC, Carlos Cabral wrote: I'm trying to collect some json data from a website/admin panel automatically, which is behind a login form. Does the website need javascript? If not, my dom.d may be able to help. It can download some HTML, parse it, fill in forms, then my http2.d submits it (I never implemented Form.submit in dom.d but it is pretty easy to make with other functions that are implemented, heck maybe I'll implement it now if it sounds like it might work). Or if it is all json you might be able to just craft some requests with my lib or even phobos' std.net.curl that submits the login request, saves a cookie, then fetches some json stuff. I literally just rolled out of bed but in an hour or two I can come back and make some example code for you if this sounds plausible. No, I don't think it needs JS. I think can submit the login form and then just fetch/save the json request using the login cookie as you suggest. A crawler/scraping solution maybe overkill... I'll try with std.net.curl and come back to you in a couple of hours Thank you!!
Re: Web crawler/scraping
On Wednesday, 17 February 2021 at 12:12:56 UTC, Carlos Cabral wrote: I'm trying to collect some json data from a website/admin panel automatically, which is behind a login form. Does the website need javascript? If not, my dom.d may be able to help. It can download some HTML, parse it, fill in forms, then my http2.d submits it (I never implemented Form.submit in dom.d but it is pretty easy to make with other functions that are implemented, heck maybe I'll implement it now if it sounds like it might work). Or if it is all json you might be able to just craft some requests with my lib or even phobos' std.net.curl that submits the login request, saves a cookie, then fetches some json stuff. I literally just rolled out of bed but in an hour or two I can come back and make some example code for you if this sounds plausible.
Re: fold on empty range
On Wednesday, 17 February 2021 at 11:38:45 UTC, Rumbu wrote:
On Wednesday, 17 February 2021 at 10:15:10 UTC, Mitacha wrote:
it'll use empty string as first element in range.
BTW perheps you could use `joinner` instead of this `fold` to
join values with ",".
Thanks for that. I thought to joiner too, but it doesn't work.
I need fold to take a list of strings and concatenate them.
Basically I read comma separated keywords from various sources
and i want to iterate through all of them. If you know other
method without the involved allocation of fold...
.map!(a => a.hit.stripLeft("[").strip("]")) //"k1,k2",
"k3,k4" ...
.fold!((a, b) => a ~ "," ~ b)("")
//"k1,k2,k3,k4,..."
.splitter(',') //"k1", "k2",
"k3", "k4", ...,
.map!(a => a.stripLeft("\" '").strip("\" '"))
.filter!(a => a.length && !a.any!(b => b == ' ' || b == '\\' ||
b == '/' || b == ':'))
.array
.sort
.uniq;
If you replace `fold` and `splitter` with this, then it doesn't
allocate:
```
auto fn() @nogc {
return only("k1,k2", "k3,k4")
.map!(x => x.splitter(","))
.joiner;
}
void main() {
auto range = fn();
range.writeln;
}
```
Re: Web crawler/scraping
On Wednesday, 17 February 2021 at 12:27:16 UTC, Ferhat Kurtulmuş wrote: On Wednesday, 17 February 2021 at 12:12:56 UTC, Carlos Cabral wrote: Hi, I'm trying to collect some json data from a website/admin panel automatically, which is behind a login form. Is there a D library that can help me with this? Thank you I found this but it looks outdated: https://github.com/gedaiu/selenium.d Thanks! This seems to depend on Selenium, I was looking for something standalone, like crawler.get(...) crawler.post(...) crawler.parse(...) so that I can deploy the executable in the client's network as a single executable (the website I'm crawling is only available internally...).
Re: Web crawler/scraping
On Wednesday, 17 February 2021 at 12:12:56 UTC, Carlos Cabral wrote: Hi, I'm trying to collect some json data from a website/admin panel automatically, which is behind a login form. Is there a D library that can help me with this? Thank you I found this but it looks outdated: https://github.com/gedaiu/selenium.d
Web crawler/scraping
Hi, I'm trying to collect some json data from a website/admin panel automatically, which is behind a login form. Is there a D library that can help me with this? Thank you
Re: fold on empty range
On Wednesday, 17 February 2021 at 10:15:10 UTC, Mitacha wrote:
it'll use empty string as first element in range.
BTW perheps you could use `joinner` instead of this `fold` to
join values with ",".
Thanks for that. I thought to joiner too, but it doesn't work. I
need fold to take a list of strings and concatenate them.
Basically I read comma separated keywords from various sources
and i want to iterate through all of them. If you know other
method without the involved allocation of fold...
.map!(a => a.hit.stripLeft("[").strip("]")) //"k1,k2",
"k3,k4" ...
.fold!((a, b) => a ~ "," ~ b)("")
//"k1,k2,k3,k4,..."
.splitter(',') //"k1", "k2",
"k3", "k4", ...,
.map!(a => a.stripLeft("\" '").strip("\" '"))
.filter!(a => a.length && !a.any!(b => b == ' ' || b == '\\' || b
== '/' || b == ':'))
.array
.sort
.uniq;
Re: fold on empty range
On Wednesday, 17 February 2021 at 09:21:47 UTC, Rumbu wrote:
In the expression below:
return matchAll(content, keywordsPattern)
.map!(a => a.hit.stripLeft("[").strip("]"))
.fold!((a, b) => a ~ "," ~ b)
.splitter(',')
.map!(a => a.stripLeft("\" ").strip("\" "))
.filter!(a => !a.any!(b => b == ' ' || b == '\\' ||
b == '/' || b == ':'))
.array
.sort
.uniq;
fold is throwing an exception if the result of the previous map
is empty. Is there any way to express something to convince
fold to return the empty range received from map?
Of course, I know I can test for empty in a separate
expression, but I'd like to keep my expression flow as it is.
I think you can try using `fold` with seed value:
```
.map!(a => a.hit.stripLeft("[").strip("]"))
.fold!((a, b) => a ~ "," ~ b)("")
.splitter(',')
```
it'll use empty string as first element in range.
BTW perheps you could use `joinner` instead of this `fold` to
join values with ",".
Re: std.typecons rebindable + tuple with const class gives warning
On Thursday, 4 February 2021 at 20:40:43 UTC, tsbockman wrote: TLDR; Either make `c` mutable, or override/overload the `C` associative array support methods `toHash` and `opEquals` to support `const(C)` objects. This solved my issue. I finally understood why this was happening after digging in to the tuple code and hence understood your proposed solutions. Thank you so much! :) Saurabh
fold on empty range
In the expression below:
return matchAll(content, keywordsPattern)
.map!(a => a.hit.stripLeft("[").strip("]"))
.fold!((a, b) => a ~ "," ~ b)
.splitter(',')
.map!(a => a.stripLeft("\" ").strip("\" "))
.filter!(a => !a.any!(b => b == ' ' || b == '\\' || b
== '/' || b == ':'))
.array
.sort
.uniq;
fold is throwing an exception if the result of the previous map
is empty. Is there any way to express something to convince fold
to return the empty range received from map?
Of course, I know I can test for empty in a separate expression,
but I'd like to keep my expression flow as it is.
