Re: Program crash: GC destroys an object unexpectedly
On 9/14/21 9:56 AM, eugene wrote: > On Tuesday, 14 September 2021 at 16:43:50 UTC, jfondren wrote: >> The misaligned pointer and the >> reference-containing struct that vanishes on the return of your >> corresponding function are both problems for this. > > where did you find 'misaligned pointer'?... I think it's the align(1) for EpollEvent. I was able to reproduce the segmentation fault and was seemingly able to fix it by making the EventSource class references alive by adding a constructor: align (1) struct EpollEvent { align(1): uint event_mask; EventSource es; this(uint event_mask, EventSource es) { this.event_mask = event_mask; this.es = es; living ~= es; // <-- Introduced this constructor for this line } /* just do not want to use that union, epoll_data_t */ } // Here is the array that keeps EventSource alive: EventSource[] living; If that really is the fix, of course the references must be taken out of that container when possible. Ali
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 18:33:33 UTC, Steven Schveighoffer wrote: People are trying to help you here. Then, answer the questions. Why those sg0 and sg1 are 'collected' by this so f... antstic GC?
Re: Program crash: GC destroys an object unexpectedly
On 9/14/21 2:05 PM, eugene wrote: On Tuesday, 14 September 2021 at 17:02:32 UTC, jfondren wrote: It doesn't seem like communication between us is possible and you are wrong, as usual ,) in the "a five-pound phone won't sell" way. I am not a 'selling boy' My suggestion remains: try troubleshooting by making your program @safe. Please, take that clever bot away. People are trying to help you here. With that attitude, you are likely to stop getting help. -Steve
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 17:02:32 UTC, jfondren wrote: It doesn't seem like communication between us is possible and you are wrong, as usual ,) in the "a five-pound phone won't sell" way. I am not a 'selling boy' My suggestion remains: try troubleshooting by making your program @safe. Please, take that clever bot away.
Re: Which operators cannot be overloaded and why not?
On Mon, Sep 13, 2021 at 06:19:20PM +, NonNull via Digitalmars-d-learn wrote: > On Monday, 13 September 2021 at 16:12:34 UTC, H. S. Teoh wrote: > > On Mon, Sep 13, 2021 at 02:12:36PM +, NonNull via > > Digitalmars-d-learn wrote: > > > Which operators cannot be overloaded and why not? > > > > Others have already given the list, so I won't repeat that. > > I didn't see unary &. Maybe others are missing. [...] IIRC, unary & cannot be overloaded for the same reasons &&, ||, and ! cannot be overloaded: it leads to hard-to-understand, hard-to-maintain code. Incidentally, C++ *does* allow overloading of &, ostensibly because some wrapper types might want to use it to become transparent. But one of the consequences of this is that now you need a std::address-of pseudo-operator that *actually* takes an address of something when you *really* mean to take the address of the wrapper object instead of the wrapped object. So & loses its original meaning and has to be replaced by a std:: hack. Which, to me, is abundant proof that overloading & was a bad idea in the first place. // BTW, as an aside, as a example of why every-man-for-himself, wild-wild-west operator overloading in C++ is a bad idea, consider these two lines of C++ code: fun(a, b); gun(a, b); What do they mean? Ostensibly, these are function calls to two template functions, specifying explicit template arguments. Unfortunately, that is not the case: only *one* of these lines is a template function call, the other is something else altogether. But nobody can tell the difference unless they read the entire context, as shown below. (Incidentally, this example also shows why C++'s choice of template syntax was a poor one.) Compile and run with a C++ compiler to find out what the actual behaviour is. --- // Totally evil example of why C++ template syntax and free-for-all // operator overloading is a Bad, Bad Idea. #include struct Bad { }; struct B { }; struct A { Bad operator,(B b) { return Bad(); } }; struct D { }; struct Ugly { D operator>(Bad b) { return D(); } } U; struct Terrible { } T; struct Evil { ~Evil() { std::cout << "Hard drive reformatted." << std::endl; } }; struct Nasty { Evil operator,(D d) { return Evil(); } }; struct Idea { void operator()(A a, B b) { std::cout << "Good idea, data saved." << std::endl; } Nasty operator<(Terrible t) { return Nasty(); } } gun; template void fun(A a, B b) { std::cout << "Have fun!" << std::endl; } int main() { A a; B b; // What do these lines do? fun(a, b); gun(a, b); } --- T -- Your inconsistency is the only consistent thing about you! -- KD
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 16:56:52 UTC, eugene wrote: On Tuesday, 14 September 2021 at 16:43:50 UTC, jfondren wrote: GC needs to be able to stop your program nice fantasies... and find all of the live objects in it. The misaligned pointer and the reference-containing struct that vanishes on the return of your corresponding function are both problems for this. where did you find 'misaligned pointer'?... It doesn't seem like communication between us is possible, in the "a five-pound phone won't sell" way. You can find this answer explained with code in an earlier post. My suggestion remains: try troubleshooting by making your program @safe.
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 16:43:50 UTC, jfondren wrote: GC needs to be able to stop your program nice fantasies... and find all of the live objects in it. The misaligned pointer and the reference-containing struct that vanishes on the return of your corresponding function are both problems for this. where did you find 'misaligned pointer'?...
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 16:15:20 UTC, eugene wrote: On Tuesday, 14 September 2021 at 16:07:00 UTC, jfondren wrote: No. And when was the first one? here: On Monday, 13 September 2021 at 18:45:22 UTC, jfondren wrote: auto p = cast(EpollEvent*) pureMalloc(EpollEvent.sizeof); What? Allocate struct epoll_event on the heap? It is a feeble joke ;) It is an example of deliberately static storage that does not fix your problem, thereby proving that the broken lifetimes of the struct are not your only problem. I explained that one at the time, and I explained this one. If it comes with an explanation, it's probably not a joke. ```c static int ecap__add(int fd, void *dptr) { struct epoll_event waitfor = {0}; int flags, r; waitfor.data.ptr = dptr; r = epoll_ctl(epfd, EPOLL_CTL_ADD, fd, ); if (-1 == r) { ``` All fd's (sockets, timers etc) are added the same way and corresponding EventSources are not destroyed by GC. GC needs to be able to stop your program and find all of the live objects in it. The misaligned pointer and the reference-containing struct that vanishes on the return of your corresponding function are both problems for this.
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 16:07:00 UTC, jfondren wrote: No. And when was the first one? here: On Monday, 13 September 2021 at 18:45:22 UTC, jfondren wrote: auto p = cast(EpollEvent*) pureMalloc(EpollEvent.sizeof); What? Allocate struct epoll_event on the heap? It is a feeble joke ;) ```c static int ecap__add(int fd, void *dptr) { struct epoll_event waitfor = {0}; int flags, r; waitfor.data.ptr = dptr; r = epoll_ctl(epfd, EPOLL_CTL_ADD, fd, ); if (-1 == r) { ``` All fd's (sockets, timers etc) are added the same way and corresponding EventSources are not destroyed by GC.
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 15:37:27 UTC, eugene wrote: On Tuesday, 14 September 2021 at 14:56:00 UTC, jfondren wrote: You could fix this by having a 128-bit struct and passing C an index into it It is another "not so funny joke", isn't it? No. And when was the first one? ```d align (1) struct EpollEvent { align(1): uint event_mask; EventSource es; /* just do not want to use that union, epoll_data_t */ } static assert(EpollEvent.sizeof == 12); ``` That's 96 bits. Add 32. ```d class EventSource { } align(1) struct EpollEvent { align(1): uint event_mask; EventSource es; } struct OuterEpollEvent { int _dummy; uint event_mask; EventSource es; } EpollEvent* epollEvent(return ref OuterEpollEvent ev) @trusted { return cast(EpollEvent*) _mask; } void dumpEpollEvent(EpollEvent* ev) @trusted { import std.stdio : writeln; writeln(*ev); } unittest { // can't be @safe: // Error: field `EpollEvent.es` cannot modify misaligned pointers in `@safe` code EpollEvent ev; ev.es = new EventSource; // misaligned } @safe unittest { // this is fine OuterEpollEvent ev; ev.event_mask = 0; ev.es = new EventSource; // not misaligned ev.epollEvent.dumpEpollEvent; } ```
Re: Program crash: GC destroys an object unexpectedly
On 9/14/21 10:56 AM, jfondren wrote: On Tuesday, 14 September 2021 at 14:40:55 UTC, eugene wrote: On Tuesday, 14 September 2021 at 12:09:03 UTC, Steven Schveighoffer wrote: This project is too big and complex Really, "too big and complex"? It's as simple as a tabouret :) It's just a toy/hobby 'project'. A 5-pound phone isn't "too heavy" for an adult to carry but it won't sell well. It's not just about capabilities but what efforts people are willing to expend. I would troubleshoot your issue by gradually making it @safe and thinking about exceptions. One exception I didn't think about earlier was the 'misaligned pointer' one that I said I suppressed just to find the next @safe complaint: https://dlang.org/spec/garbage.html says: Do not misalign pointers if those pointers may point into the GC heap, So even if the lifetimes of your EventSource structs are fixed, the GC can reap the object they're pointing to. You could fix this by having a 128-bit struct and passing C an index into it, so to speak. I don't think this is the problem. The misaligned pointers are only happening within the stack frame, along with references to the objects stored also in another parameter. So they should not cause problems with the GC. The storage of the references inside other objects is not misaligned. -Steve
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 14:56:00 UTC, jfondren wrote: You could fix this by having a 128-bit struct and passing C an index into it It is another "not so funny joke", isn't it? Look ```c typedef union epoll_data { void *ptr; int fd; uint32_t u32; uint64_t u64; } epoll_data_t; struct epoll_event { uint32_t events; /* Epoll events */ epoll_data_t data;/* User data variable */ } __EPOLL_PACKED; // inside the system struct epoll_event { __u32 events; __u64 data; } EPOLL_PACKED; ``` and notice ```d align (1) struct EpollEvent { align(1): uint event_mask; EventSource es; /* just do not want to use that union, epoll_data_t */ } static assert(EpollEvent.sizeof == 12); ```
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 12:09:03 UTC, Steven Schveighoffer wrote: This project is too big and complex for me to diagnose by just reading, it would take some effort take a look at https://www.routledge.com/Modeling-Software-with-Finite-State-Machines-A-Practical-Approach/Wagner-Schmuki-Wagner-Wolstenholme/p/book/9780367390860# 'Event/Message Driven State Machines' (http://zed.karelia.ru/mmedia/bin/edsm-g2-rev-h.tar.gz) was inspired by this nice book.
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 14:40:55 UTC, eugene wrote: On Tuesday, 14 September 2021 at 12:09:03 UTC, Steven Schveighoffer wrote: This project is too big and complex Really, "too big and complex"? It's as simple as a tabouret :) It's just a toy/hobby 'project'. A 5-pound phone isn't "too heavy" for an adult to carry but it won't sell well. It's not just about capabilities but what efforts people are willing to expend. I would troubleshoot your issue by gradually making it @safe and thinking about exceptions. One exception I didn't think about earlier was the 'misaligned pointer' one that I said I suppressed just to find the next @safe complaint: https://dlang.org/spec/garbage.html says: Do not misalign pointers if those pointers may point into the GC heap, So even if the lifetimes of your EventSource structs are fixed, the GC can reap the object they're pointing to. You could fix this by having a 128-bit struct and passing C an index into it, so to speak.
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 12:09:03 UTC, Steven Schveighoffer wrote: This project is too big and complex Really, "too big and complex"? It's as simple as a tabouret :) It's just a toy/hobby 'project'.
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 12:52:44 UTC, Steven Schveighoffer wrote: But I agree that a superficial reading of your code seems like it ought to not be collected, and that problem is also worth figuring out. I have high confidence that it's probably not a design flaw in the GC, but rather some misunderstanding of GC-allocated lifetimes in your code. But that doesn't mean it's not actually a bug somewhere in D. run the server (do not run client): 'LISTENER @ INIT' got 'M0' from 'SELF' 'LISTENER' registered 104 (esrc.TCPListener) 'LISTENER' enabled 104 (esrc.TCPListener) 'LISTENER' enabled 105 (esrc.Signal) 'LISTENER' enabled 106 (esrc.Signal) wait > 6 seconds press ^C observe ___!!!___edsm.StageMachine.~this(): WORKER-95 destroyed... ___!!!___edsm.StageMachine.~this(): WORKER-96 destroyed... ___!!!___edsm.StageMachine.~this(): LISTENER destroyed... run client (do not run the server) observe 'CLIENT-9 @ CONN' got 'M2' from 'TX-1' CLIENT-9:client.EchoClient.clientConnM2() : connection to 'localhost:' failed error111) CLIENT-9:client.EchoClient.clientConnM2() : connection to 'localhost:' failed(Connection refused) press ^C observe ___!!!___edsm.StageMachine.~this(): STOPPER destroyed... run server again run client like this: ./echo-client | grep owner wait >6.seconds see !!! esrc.EventSource.~this() : esrc.Signal (owner STOPPER, fd = 24) this @ 0x7fa6cf12cf60 !!! esrc.EventSource.~this() : esrc.Signal (owner STOPPER, fd = 25) this @ 0x7fa6cf12cf90 WHY this is not happening with echo-server???
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 12:53:27 UTC, Adam D Ruppe wrote: I had a problem just like this before because I was sending objects through the pipe. And while they were in the pipe - ```rust pub fn msg(, code: u32) { let ptr: *const u32 = let n = unsafe { write(self.mxfd, ptr as *const void, 4) }; if -1 == n { panic!("write({}): {:?}", self.mxfd, Error::last_os_error()); } } ``` I failed to implement message queue as a wrapper over double list, rust borrow checker has beaten me :)
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 12:53:27 UTC, Adam D Ruppe wrote: I had a problem just like this before because I was sending objects through the pipe. This reminds my (not very successfull) attempts to implement the idea in Rust: ```rust pub struct Edsm { name: String, pub states: Vec, current: usize, //pub state : *mut State, (?) pub data: *const void, // long live void* !!! //pub buddy : &'a Edsm, // ... and a hell begins... mb: Option>, /* self-pipe write end fd, for sending internal events to this machine */ mxfd: i32, io: Option>, pub tm: Vec, sg: Vec, //pub fs : Option>, //pub ecap : &'a mut Ecap, // Welcome to <'x> HELL again!!! ecap: *mut Ecap, running: bool, /* self.run() has been invoked */ } ``` When something (a struct, for ex.) goes to a queue (DList for ex.), it is out of ANY scope and clever things like borrow checker can not analyze it's lifetime, oops...
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 12:42:51 UTC, eugene wrote: I understand your idea, but even if this will help, the question remains - why that particular object is so special for GC. I had a problem just like this before because I was sending objects through the pipe. And while they were in the pipe - after send but before receive on the other side - it was liable to be collected. idk your code though if you have a separate reference to the object you should be ok. but here's the comment from my code when i broke it https://github.com/adamdruppe/arsd/blob/master/eventloop.d#L180
Re: Program crash: GC destroys an object unexpectedly
On 9/14/21 8:42 AM, eugene wrote: On Tuesday, 14 September 2021 at 12:09:03 UTC, Steven Schveighoffer wrote: I still recommend pinning the object when adding the epoll event and seeing if that helps. I understand your idea, but even if this will help, the question remains - why that particular object is so special for GC. Philosophically, it places the responsibility of making sure the object is valid while using it on the thing that chooses to store it outside the GC's view. Looking at your examples, you are having to store these object references elsewhere, surrounding seemingly innocuous and normal D usage of objects. You have put the burden on the caller to make sure the implementation details are sound. But I agree that a superficial reading of your code seems like it ought to not be collected, and that problem is also worth figuring out. I have high confidence that it's probably not a design flaw in the GC, but rather some misunderstanding of GC-allocated lifetimes in your code. But that doesn't mean it's not actually a bug somewhere in D. -Steve
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 12:13:15 UTC, Steven Schveighoffer wrote: On 9/14/21 7:31 AM, eugene wrote: On Monday, 13 September 2021 at 17:18:30 UTC, eugene wrote: EventSource s = events[k].es; ulong ecode = s.eventCode(events[k].event_mask); // < SIGSEGV Note that s likely still points at a valid memory address. yeah, this address is obtained from OS (epoll_event struct), compiler can not zero it. However, when an object is destroyed, its vtable is nulled out (precisely to cause a segfault if you try to use an already-freed object). that's right - calling eventCode() method results in segfault.
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 12:09:03 UTC, Steven Schveighoffer wrote: Though as I have learned helping C converts before, most of the time things like this have to do with forgetting to store a GC reference somewhere. Yeah, in my first version I had ```d foreach (k; 0 .. nConnections) { auto sm = new EchoClient(rxPool, txPool); sm.run(); } ``` instead of ```d EchoClient[] wrkMachines; foreach (k; 0 .. nConnections) { auto sm = new EchoClient(rxPool, txPool); wrkMachines ~= sm; sm.run(); } ``` and even ```d { auto stopper = new Stopper(); stopper.run(); } ``` :) I still recommend pinning the object when adding the epoll event and seeing if that helps. I understand your idea, but even if this will help, the question remains - why that particular object is so special for GC.
Re: Recommendations on parsing XML via an InputRange
On 9/13/21 10:43 PM, Chris Piker wrote: Hi D I just finished a ~1K line project using `dxml` as the XML reader for my data streams. It works well in my test examples using memory mapped files, but like an impulse shopper I didn't notice that dxml requires `ForwardRange` objects. That's unfortunate, because my next enhancement was to start parsing streams as they come in from stdin. (doh!) So I've learned my lesson and will RTFM closer next time, but now I'm casting about for a solution. Two ideas, either: 1. Find a different StAX-ish parser that works with `InputRange` (and buffers internally a bit if needed), or 2. Find a way to represent standard input as a ForwardRange without saving the whole stream in memory. (iopipe?) Iopipe is no better than an input range unless you plan to read the whole stream into a buffer. A forward range is required because dxml uses saved ranges to refer to previous data. This requires the whole thing to be stored in memory. I've thought of building an xml parser on top of iopipe, and I probably will some day (maybe a port of dxml). The iopipejson library does not require the whole thing to be in memory, and has some facilities to pin parsed data to jump back. I imagine something like that is doable for xml, but probably just storing current element ancestry while parsing (probably off to the side in another stack-like thing). -Steve
Re: Program crash: GC destroys an object unexpectedly
On 9/14/21 7:31 AM, eugene wrote: On Monday, 13 September 2021 at 17:18:30 UTC, eugene wrote: Then after pressing ^C (SIGINT) the program gets SIGSEGV, since references to sg0 and sg1 are no longer valid (they are "sitting" in epoll_event structure). ... forget to mention, crashes here: ```d bool wait() { const int maxEvents = 8; EpollEvent[maxEvents] events; if (done) return false; int n = epoll_wait(id, events.ptr, maxEvents, -1); if (-1 == n) return false; foreach (k; 0 .. n) { EventSource s = events[k].es; ulong ecode = s.eventCode(events[k].event_mask); // < SIGSEGV ``` sg0/sg1 are destroyed, so s points to wrong location. Note that s likely still points at a valid memory address. However, when an object is destroyed, its vtable is nulled out (precisely to cause a segfault if you try to use an already-freed object). There is also the possibility the memory block has been reallocated to something else, and that is causing the segfault. But if the segfault is consistent, most likely it's the former problem. -Steve
Re: Program crash: GC destroys an object unexpectedly
On 9/14/21 1:49 AM, Tejas wrote: On Monday, 13 September 2021 at 18:42:47 UTC, Steven Schveighoffer wrote: On 9/13/21 1:54 PM, eugene wrote: [...] The GC only scans things that it knows about. Inside your EventQueue you have this code: [...] Umm is it okay that he declared variables `init` and `idle` of type `Stage` inside the constructor? Maybe that has something to do with this? Also, calling a variable `init` could be problematic since the compiler assigns a property of the same name to every single type? Declaring a member/field named `init` is likely a bad idea, but this is not a member, it's just a variable. That's fine. `idle` doesn't mean anything special to D. This project is too big and complex for me to diagnose by just reading, it would take some effort, and I don't have the time, sorry. Though as I have learned helping C converts before, most of the time things like this have to do with forgetting to store a GC reference somewhere. It can be subtle too... I still recommend pinning the object when adding the epoll event and seeing if that helps. -Steve
Re: Program crash: GC destroys an object unexpectedly
On Monday, 13 September 2021 at 17:18:30 UTC, eugene wrote: Then after pressing ^C (SIGINT) the program gets SIGSEGV, since references to sg0 and sg1 are no longer valid (they are "sitting" in epoll_event structure). ... forget to mention, crashes here: ```d bool wait() { const int maxEvents = 8; EpollEvent[maxEvents] events; if (done) return false; int n = epoll_wait(id, events.ptr, maxEvents, -1); if (-1 == n) return false; foreach (k; 0 .. n) { EventSource s = events[k].es; ulong ecode = s.eventCode(events[k].event_mask); // < SIGSEGV ``` sg0/sg1 are destroyed, so s points to wrong location.
Re: Program crash: GC destroys an object unexpectedly
On Monday, 13 September 2021 at 18:45:22 UTC, jfondren wrote: ```d auto p = cast(EpollEvent*) pureMalloc(EpollEvent.sizeof); ``` What? Allocate struct epoll_event on the heap? It is a feeble joke ;) ```c static int ecap__add(int fd, void *dptr) { struct epoll_event waitfor = {0}; int flags, r; waitfor.data.ptr = dptr; r = epoll_ctl(epfd, EPOLL_CTL_ADD, fd, ); if (-1 == r) { ``` All fd's (sockets, timers etc) are added the same way and corresponding EventSources are not destroyed by GC.
Re: Program crash: GC destroys an object unexpectedly
On Tuesday, 14 September 2021 at 05:49:58 UTC, Tejas wrote: Umm is it okay that he declared variables `init` and `idle` of type `Stage` inside the constructor? States of a machine are in associative array. All other machines create their states in constructor, local variables are for using addReflex() method. But this stopper machine is 'special' for GC somehow.
Re: Development of the foundation of a programming language
On Tuesday, 14 September 2021 at 05:06:01 UTC, Elronnd wrote: On Tuesday, 14 September 2021 at 03:24:45 UTC, max haughton wrote: On Tuesday, 14 September 2021 at 03:19:46 UTC, Elronnd wrote: On Monday, 13 September 2021 at 11:40:10 UTC, max haughton wrote: The dragon book barely mentions SSA for example In fairness, dmd doesn't use SSA either That's not a good thing. No, but if the OP's goal is to contribute to dmd, learning SSA wouldn't be very helpful beyond a general acclimation to compiler arcana. (Unless they wish to add SSA to dmd--a worthy goal, but perhaps not the best thing to start out with.) The backend is not where our efforts should be going. There is way too much work that needs doing above it to motivate working on the backend. The backend's design for the most part is extremely simple just buried under 40 years of code. Besides, there's more to life than dmd, everything else is SSA at least some point in compilation (i.e. GCC isn't SSA all the way down but GIMPLE is).