Re: Terminating multiple processes – SOLVED
On Fri, 2018-02-02 at 20:14 +, David Nadlinger wrote: > […] > That sounds entirely sensible. Your original question was whether it > was > possible to terminate threads blocked in a syscall, though. Signals > allow you to do that on POSIX for many "slow" syscalls, by making it > return EINTR. You would check whatever channel/atomic flag/… > mechanism > you use to signal termination before continuing to block by > reissuing > the syscall. > > I haven't tested how `read` behaves for inotify myself, but > inotify(7) > suggests that it indeed handles signals this way. […] After some discussions, Dmitry acted very quickly and added a timeout read to the blocking read in dinotify. This solves my problem "at a stroke" as I can now block waiting for a specific time on inotify and then check the input channel for a time and repeat. Not really busy wait just a sequence of blocking reads with timeouts. The important thing is that With very little latency, the thread is checking two things. DInotify 0.3.0 will likely appear soon in Dub. It would be good to get it packaged for Debian. I will take up an invitation to submit a ready made package for the Debian D packaging team. -- Russel. === Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Roadm: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk signature.asc Description: This is a digitally signed message part
Re: Terminating multiple processes
On Thu, 2018-02-01 at 20:13 +, Arek via Digitalmars-d-learn wrote:
>
[…]
> DInitify doesn't cover full capabilities of the inotify API.
> Especially it doesn't utilize newer inotify_init1 syscall and
> doesn't expose 'select' interface.
> This C++ wrapper may be interesting for you:
> https://github.com/erikzenker/inotify-cpp
> But I haven't used it.
Hummm… sounds like I need to look to create a pull request to get to
dinotify 0.2.3 or even 0.3.0.
> > >
[…]
> You may be right. But interrupting the thread is very low level
> mechanism.
> Eg in Java, it is incorporated into Thread class: thread has
> method 'interrupt()' which
> sets the flag, and there is property 'isInterrupted()'. Usually
> your thread code have something like
> while(!Thread.currentThread().isInterrupted()) { /* do
> something*/ }
> I would use similar pattern.
I do want to avoid such low-level things, they may be required for
building libraries, but they seem totally the wrong level of concept
for an application.
I am definitely headed to the (select|poll|epoll) on the inotify file
descriptor to get the timeout, and then either read or check the input
channel for termination or an atomic Boolean state variable. Channels
seem most likely since parent threads send a terminating token.
--
Russel.
===
Dr Russel Winder t: +44 20 7585 2200
41 Buckmaster Roadm: +44 7770 465 077
London SW11 1EN, UK w: www.russel.org.uk
signature.asc
Description: This is a digitally signed message part
Re: Terminating multiple processes
On Fri, 2018-02-02 at 01:09 +, David Nadlinger via Digitalmars-d- learn wrote: > On Thursday, 1 February 2018 at 11:42:32 UTC, Russel Winder wrote: > > The problem is actually a thread blocked in an inotify blocking > > read. As both Steven and yourself have pointed out I am going > > to have to use a timeout to check the state of the application. > > There are better solutions (select/...), But couldn't you in > theory just send a custom signal to the thread (which you > ignore), and then check for the exit flag after the syscall > returned EINTR? I think I am going to go the (select|poll|epoll) on the file descriptor with a timeout and then the read if that is appropriate or check the input channel for the terminate token if that is appropriate. The question whether to do this in my code or create a "read with timeout" in dinotify > The DInotify wrapper might of course have the retry loop > hardcoded; I didn't check. Dinotify adds the tree monitoring as an extra over the inotify API but U do not see any timeout related things. -- Russel. === Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Roadm: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk signature.asc Description: This is a digitally signed message part
Re: Terminating multiple processes
On Thursday, 1 February 2018 at 11:42:32 UTC, Russel Winder wrote: The problem is actually a thread blocked in an inotify blocking read. As both Steven and yourself have pointed out I am going to have to use a timeout to check the state of the application. There are better solutions (select/...), But couldn't you in theory just send a custom signal to the thread (which you ignore), and then check for the exit flag after the syscall returned EINTR? The DInotify wrapper might of course have the retry loop hardcoded; I didn't check. —David
Re: Terminating multiple processes
On Thursday, 1 February 2018 at 12:30:24 UTC, Russel Winder wrote:
On Thu, 2018-02-01 at 12:15 +, Arek via Digitalmars-d-learn
wrote:
[…]
Try to use inotify in non-blocking mode (an example here:
https://gist.github.com/pkrnjevic/6016356) with select or
epoll and timeouts.
Isn't there a C++ binding for the C API?
I am using DInotify which is a D binding. I will be checking
soon but I am assuming there is a timeout version so I can loop
to check the application state.
DInitify doesn't cover full capabilities of the inotify API.
Especially it doesn't utilize newer inotify_init1 syscall and
doesn't expose 'select' interface.
This C++ wrapper may be interesting for you:
https://github.com/erikzenker/inotify-cpp
But I haven't used it.
I would use shared memory here (eg. atomic bool) because any
communication channel introduces possibility of further
blocking problems.
A priori I am not convinced. I have used a state variable in
C++ and Python code where there is no channel system, but in
Go, Groovy/GPars, using channels is always preferable. Given
the channel has a "read if there is something to read" there
can't be a blocking problem – if the channel system is a good
one.
You may be right. But interrupting the thread is very low level
mechanism.
Eg in Java, it is incorporated into Thread class: thread has
method 'interrupt()' which
sets the flag, and there is property 'isInterrupted()'. Usually
your thread code have something like
while(!Thread.currentThread().isInterrupted()) { /* do
something*/ }
I would use similar pattern.
Arek
Re: Terminating multiple processes
On Thu, 2018-02-01 at 12:15 +, Arek via Digitalmars-d-learn wrote: > […] > Try to use inotify in non-blocking mode (an example here: > https://gist.github.com/pkrnjevic/6016356) with select or epoll > and timeouts. Isn't there a C++ binding for the C API? I am using DInotify which is a D binding. I will be checking soon but I am assuming there is a timeout version so I can loop to check the application state. > > > > I guess there is a choice here between shared memory to set the > > termination flag, or using an input channel and sending the > > termination message. I think the latter may be preferable, and > > certainly more consistent with how the other threads terminate. > > > > I would use shared memory here (eg. atomic bool) because any > communication channel introduces possibility of further blocking > problems. A priori I am not convinced. I have used a state variable in C++ and Python code where there is no channel system, but in Go, Groovy/GPars, using channels is always preferable. Given the channel has a "read if there is something to read" there can't be a blocking problem – if the channel system is a good one. -- Russel. === Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Roadm: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk signature.asc Description: This is a digitally signed message part
Re: Terminating multiple processes
On Thursday, 1 February 2018 at 11:42:32 UTC, Russel Winder wrote: On Wed, 2018-01-31 at 22:15 +, Arek via Digitalmars-d-learn wrote: […] The problem is actually a thread blocked in an inotify blocking read. As both Steven and yourself have pointed out I am going to have to use a timeout to check the state of the application. Try to use inotify in non-blocking mode (an example here: https://gist.github.com/pkrnjevic/6016356) with select or epoll and timeouts. I guess there is a choice here between shared memory to set the termination flag, or using an input channel and sending the termination message. I think the latter may be preferable, and certainly more consistent with how the other threads terminate. I would use shared memory here (eg. atomic bool) because any communication channel introduces possibility of further blocking problems. Arek
Re: Terminating multiple processes
On Wed, 2018-01-31 at 22:15 +, Arek via Digitalmars-d-learn wrote: > […] > Assuming your're talking about threads: there's no secure method > of forcing the thread to stop. Threads share the state (eg. can > hold the locks) and killing them is always risky. I am indeed talking threads not OS processes. I just like working with (processes|tasks) submitted to a thread pool, but this can lead to jargon use problems. > If your threads are blocked reading the socket, you probably can > close these sockets and exit after the read error. The problem is actually a thread blocked in an inotify blocking read. As both Steven and yourself have pointed out I am going to have to use a timeout to check the state of the application. > Another way is to use atomic flag indicating that thread needs to > be interrupted. > After any blocking operation, the thread have to check this flag > and finish the job. > It's good to use timeouts (eg socket timeout) in such scenario > (if possible). I guess there is a choice here between shared memory to set the termination flag, or using an input channel and sending the termination message. I think the latter may be preferable, and certainly more consistent with how the other threads terminate. Thanks to you and Steven for confirming what I had been fearing, but wondered if there was an easier (albeit less theoretically sound) one. -- Russel. === Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Roadm: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk signature.asc Description: This is a digitally signed message part
Re: Terminating multiple processes
On Wed, 2018-01-31 at 13:26 -0500, Steven Schveighoffer via Digitalmars-d-learn wrote: > […] > You talking about processes or threads? `receive` I believe is an > inter-thread channel, no? These are threads with no shared memory, just message passing via channels. I think of them as processes, but not OS processes. My poor choose of names, I should just have said threads. > Terminating processes is generally done via signals or in the case > of > windows, calling the right system call. Indeed. usually SIGHUP or SIGTERM, but never SIGKILL. > Threads are another story. Typically, you need to have the thread > check > periodically for a termination event. There's no "nice" way to do it > out > of band. This is what I was fearing. Of course from a theoretical standpoint it is the right way to do things. > In my experience, the best way to do it is to never block, but use > some > sort of "wait on input" for any would-be-blocking operation. You can > use > a large timeout, like 1 second, if immediate termination isn't > important. The thread in question is usually blocked in an inotify wait state having no input channel. It seems I will have to manufacture a select. I think this is what is missing in D, or at least my knowledge of it. In Go, I can set up a select between channels and block, no need for a timeout, an input on either channel causes a wakeup. The code then has to work out which channel caused the wakeup. On the other hand DInotify doesn't provide a channel style blocking wait, so select isn't an option. > If you are using Fibers, and all your i/o is done using some event- > based > system (e.g. vibe.d), then things can be easier. I am not sure I want to get into single-threaded event queues, but maybe I should. I like the dataflow approach, it is simpler for me to reason about. -- Russel. === Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Roadm: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk signature.asc Description: This is a digitally signed message part
Re: Terminating multiple processes
On Wednesday, 31 January 2018 at 17:44:37 UTC, Russel Winder wrote: So, I have an application which has a sort of nano-services architecture, basically it is a set of communicating processes. Terminating those processes blocked on an input channel is quite easy, send a terminate message on the input channel. But what about a process that has no input channel, one that is blocked on OS events? Is there a way of forcibly, but nicely, terminating a spawned process that never executes `receive()`? Assuming your're talking about threads: there's no secure method of forcing the thread to stop. Threads share the state (eg. can hold the locks) and killing them is always risky. If your threads are blocked reading the socket, you probably can close these sockets and exit after the read error. Another way is to use atomic flag indicating that thread needs to be interrupted. After any blocking operation, the thread have to check this flag and finish the job. It's good to use timeouts (eg socket timeout) in such scenario (if possible). Arek
Re: Terminating multiple processes
On 1/31/18 12:44 PM, Russel Winder wrote: So, I have an application which has a sort of nano-services architecture, basically it is a set of communicating processes. Terminating those processes blocked on an input channel is quite easy, send a terminate message on the input channel. But what about a process that has no input channel, one that is blocked on OS events? Is there a way of forcibly, but nicely, terminating a spawned process that never executes `receive()`? You talking about processes or threads? `receive` I believe is an inter-thread channel, no? Terminating processes is generally done via signals or in the case of windows, calling the right system call. Threads are another story. Typically, you need to have the thread check periodically for a termination event. There's no "nice" way to do it out of band. In my experience, the best way to do it is to never block, but use some sort of "wait on input" for any would-be-blocking operation. You can use a large timeout, like 1 second, if immediate termination isn't important. If you are using Fibers, and all your i/o is done using some event-based system (e.g. vibe.d), then things can be easier. -Steve
Terminating multiple processes
So, I have an application which has a sort of nano-services architecture, basically it is a set of communicating processes. Terminating those processes blocked on an input channel is quite easy, send a terminate message on the input channel. But what about a process that has no input channel, one that is blocked on OS events? Is there a way of forcibly, but nicely, terminating a spawned process that never executes `receive()`? -- Russel. === Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Roadm: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk signature.asc Description: This is a digitally signed message part
