Re: [systemd-devel] sd-bus: serving method call message in a separate thread
Hi Stanislav, On Mon, Mar 4, 2019 at 9:56 PM Stanislav Angelovič wrote: > > Hi sd-bus-ers! > > Quick question: How can I process a method call on service side in a > different thread, without creating race condition? > > Longer version: > > In sdbus-c++, we are working on server-side asynchronous method call support. > > In sd-bus, a service handles D-Bus method calls via sd_bus_message_handler_t > callback that is registered during object vtable registration. The callback > receives sd_bus_message* as the first parameter (among others), which is the > method call message. In a typical implementation, this method call is served > synchronously (D-Bus method parameters are deserialized from the message, an > operation is executed, and either method reply or method error, depending on > the condition, is sent back) and the callback returns execution to sd-bus. > Standard stuff so far. > > Now, I would like to handle the method call asynchronously, in a thread pool. > So a typical approach would be to increment ref count of sd_bus_message and > push the sd_bus_message* in a queue. It would then be popped by one of worker > threads and processed -- method parameters are deserialized, method logic is > executed, and either method reply or method error is created and sent back, > and sd_bus_message* ref count is decremented. Now, sending back the reply or > an error is thread-safe in my implementation, I don't have an issue with > that. The problem that I want to discuss is the method call sd_bus_message > instance. I cannot simply forward a pointer to it to a separate thread, > because of it's non-atomic ref count handling, which is the source of race > condition (hopefully this is the only data race condition here). It's > manipulated by both the D-Bus loop thread (which decrements ref count after > sd_bus_message_handler_t returns) and the worker thread concurrently (which > also decrements ref count after it has handled the method call). > > I see three potential solutions now: > > 1. In sd_bus_message_handler, create a deep copy of the method call > sd_bus_message via sd_bus_message_copy(). And pass a pointer to this copy to > a worker thread. This is very straight-forward and simple, it solves the race > condition, but introduces a performance cost of copying (I don't know how big > this cost is, perhaps it's rather negligible). > > 2. Don't pass the sd_bus_message* to the worker thread. In > sd_bus_message_handler, rather deserialize all arguments, create (empty) > method reply message, and move these to the worker thread. The worker thread > executes the logic and serializes results to that reply message, and sends it > back. The problem here is that we have to create a method reply or method > error before the fact (before executing method logic), which in case of > method error is impossible because we don't know possible error name and > message beforehand. > > 3. Solution on sd-bus side :-) which is to make sd_bus_message ref count > handling thread-safe (atomic), just like one for sd_bus. This avoids the need > for deep copy. What do you think? Anyway, we'd still have to come up with a > solution now, based on already releases sd-bus versions. > > So the only feasible solution for now seems to be #1 (while #3 could also be > a sd-bus improvement for the future). Do you agree that this is the good way > along the lines of sd-bus design principles, or are there more options I'm > unaware of? > In my use of sdbus I am going for option 3. Also because I am not sure what happens when sdbus writes on the bus in parallel two long messages. I know that sdbus is thread aware, but not thread safe. So 2 long messages could in theory be sent in collision. Maybe you can verify this opening two (or more) instances of a client that query automatically all introspectable methods of a specific server implemented with your thread pool, and see what happens in repeated trials. > Thank you for your constructive comments, and sorry for long elaboration :) I > wanted to be clear... > > Cheers, > Stanislav. > Cheers, Giaciinto > ___ > systemd-devel mailing list > systemd-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] sd-bus: serving method call message in a separate thread
Hi sd-bus-ers! Quick question: How can I process a method call on service side in a different thread, without creating race condition? Longer version: In sdbus-c++, we are working on server-side asynchronous method call support. In sd-bus, a service handles D-Bus method calls via sd_bus_message_handler_t callback that is registered during object vtable registration. The callback receives sd_bus_message* as the first parameter (among others), which is the method call message. In a typical implementation, this method call is served synchronously (D-Bus method parameters are deserialized from the message, an operation is executed, and either method reply or method error, depending on the condition, is sent back) and the callback returns execution to sd-bus. Standard stuff so far. Now, I would like to handle the method call asynchronously, in a thread pool. So a typical approach would be to increment ref count of sd_bus_message and push the sd_bus_message* in a queue. It would then be popped by one of worker threads and processed -- method parameters are deserialized, method logic is executed, and either method reply or method error is created and sent back, and sd_bus_message* ref count is decremented. Now, sending back the reply or an error is thread-safe in my implementation, I don't have an issue with that. The problem that I want to discuss is the method call sd_bus_message instance. I cannot simply forward a pointer to it to a separate thread, because of it's non-atomic ref count handling, which is the source of race condition (hopefully this is the only data race condition here). It's manipulated by both the D-Bus loop thread (which decrements ref count after sd_bus_message_handler_t returns) and the worker thread concurrently (which also decrements ref count after it has handled the method call). I see three potential solutions now: 1. In sd_bus_message_handler, create a deep copy of the method call sd_bus_message via sd_bus_message_copy(). And pass a pointer to this copy to a worker thread. This is very straight-forward and simple, it solves the race condition, but introduces a performance cost of copying (I don't know how big this cost is, perhaps it's rather negligible). 2. Don't pass the sd_bus_message* to the worker thread. In sd_bus_message_handler, rather deserialize all arguments, create (empty) method reply message, and move these to the worker thread. The worker thread executes the logic and serializes results to that reply message, and sends it back. The problem here is that we have to create a method reply or method error before the fact (before executing method logic), which in case of method error is impossible because we don't know possible error name and message beforehand. 3. Solution on sd-bus side :-) which is to make sd_bus_message ref count handling thread-safe (atomic), just like one for sd_bus. This avoids the need for deep copy. What do you think? Anyway, we'd still have to come up with a solution now, based on already releases sd-bus versions. So the only feasible solution for now seems to be #1 (while #3 could also be a sd-bus improvement for the future). Do you agree that this is the good way along the lines of sd-bus design principles, or are there more options I'm unaware of? Thank you for your constructive comments, and sorry for long elaboration :) I wanted to be clear... Cheers, Stanislav. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] OnCalendar timer starts before the scheduled date has passed
You're right, I misunderstood the purpose the the Requires directive.
Thanks for pointing this out, I am seeing the expected behavior in my
script after removing it.
Alden
On Mon, Mar 4, 2019 at 1:43 PM Andrei Borzenkov wrote:
> 04.03.2019 21:19, Alden Page пишет:
> > Hi systemd team,
> >
> > I'm having some trouble understanding the behavior of a systemd timer
> I've
> > set up in systemd version 219. I have a task that I want to run once per
> > week on Sundays at 3:00am UTC. However, as soon as I do `systemctl start
> > my_timer.timer`, the task starts no matter what time it is. This is a
>
> That is what you told it to do.
>
> > problem for me, because that means that my timer starts every single
> time I
> > recreate my server's virtual machine. After the initial erroneously
> > scheduled job has completed successfully, the timer starts behaving
> > correctly and runs on Sundays at 3:00am UTC.
> >
> > Here is how I've set up my timers. This is an excerpt from the script I
> use
> > to bootstrap my VMs:
> >
> > # Schedule weekly ingest of new data
> > cat << EOF > /etc/systemd/system/ingest_upstream.service
> > [Unit]
> > Description=Load and index image data from upstream.
> > [Service]
> > ExecStart=/usr/bin/curl -XPOST localhost:8001/task -H "Content-Type:
> > application/json" -d '{"model": "image", "action": "INGEST_UPSTREAM"}'
> > EOF
> >
> > cat << EOF > /etc/systemd/system/ingest_upstream.timer
> > [Unit]
> > Description=Ingest data from upstream every Sunday at 8:00am EST (3:00am
> > UTC).
> > Requires=ingest_upstream.service
> >
>
> Requires means - when ingest_upstream.timer is started (activated) also
> start (activate) units listed here. Which is exactly what happens.
>
> > [Timer]
> > OnCalendar=Sun *-*-* 3:00:00
> > Unit=ingest_upstream.service
> >
> > [Install]
> > WantedBy=timers.target
> > EOF
> > systemctl start ingest_upstream.timer
> >
> > After `systemctl start`, shouldn't it wait until the next OnCalendar
> > interval has passed, particularly since `Persistent=...` is not set?
> >
>
> It (presumably ingest_upstream.timer) does wait; your service is not
> activated by timer but by basic systemd unit dependency.
> ___
> systemd-devel mailing list
> systemd-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] OnCalendar timer starts before the scheduled date has passed
04.03.2019 21:19, Alden Page пишет:
> Hi systemd team,
>
> I'm having some trouble understanding the behavior of a systemd timer I've
> set up in systemd version 219. I have a task that I want to run once per
> week on Sundays at 3:00am UTC. However, as soon as I do `systemctl start
> my_timer.timer`, the task starts no matter what time it is. This is a
That is what you told it to do.
> problem for me, because that means that my timer starts every single time I
> recreate my server's virtual machine. After the initial erroneously
> scheduled job has completed successfully, the timer starts behaving
> correctly and runs on Sundays at 3:00am UTC.
>
> Here is how I've set up my timers. This is an excerpt from the script I use
> to bootstrap my VMs:
>
> # Schedule weekly ingest of new data
> cat << EOF > /etc/systemd/system/ingest_upstream.service
> [Unit]
> Description=Load and index image data from upstream.
> [Service]
> ExecStart=/usr/bin/curl -XPOST localhost:8001/task -H "Content-Type:
> application/json" -d '{"model": "image", "action": "INGEST_UPSTREAM"}'
> EOF
>
> cat << EOF > /etc/systemd/system/ingest_upstream.timer
> [Unit]
> Description=Ingest data from upstream every Sunday at 8:00am EST (3:00am
> UTC).
> Requires=ingest_upstream.service
>
Requires means - when ingest_upstream.timer is started (activated) also
start (activate) units listed here. Which is exactly what happens.
> [Timer]
> OnCalendar=Sun *-*-* 3:00:00
> Unit=ingest_upstream.service
>
> [Install]
> WantedBy=timers.target
> EOF
> systemctl start ingest_upstream.timer
>
> After `systemctl start`, shouldn't it wait until the next OnCalendar
> interval has passed, particularly since `Persistent=...` is not set?
>
It (presumably ingest_upstream.timer) does wait; your service is not
activated by timer but by basic systemd unit dependency.
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] OnCalendar timer starts before the scheduled date has passed
Hi systemd team,
I'm having some trouble understanding the behavior of a systemd timer I've
set up in systemd version 219. I have a task that I want to run once per
week on Sundays at 3:00am UTC. However, as soon as I do `systemctl start
my_timer.timer`, the task starts no matter what time it is. This is a
problem for me, because that means that my timer starts every single time I
recreate my server's virtual machine. After the initial erroneously
scheduled job has completed successfully, the timer starts behaving
correctly and runs on Sundays at 3:00am UTC.
Here is how I've set up my timers. This is an excerpt from the script I use
to bootstrap my VMs:
# Schedule weekly ingest of new data
cat << EOF > /etc/systemd/system/ingest_upstream.service
[Unit]
Description=Load and index image data from upstream.
[Service]
ExecStart=/usr/bin/curl -XPOST localhost:8001/task -H "Content-Type:
application/json" -d '{"model": "image", "action": "INGEST_UPSTREAM"}'
EOF
cat << EOF > /etc/systemd/system/ingest_upstream.timer
[Unit]
Description=Ingest data from upstream every Sunday at 8:00am EST (3:00am
UTC).
Requires=ingest_upstream.service
[Timer]
OnCalendar=Sun *-*-* 3:00:00
Unit=ingest_upstream.service
[Install]
WantedBy=timers.target
EOF
systemctl start ingest_upstream.timer
After `systemctl start`, shouldn't it wait until the next OnCalendar
interval has passed, particularly since `Persistent=...` is not set?
Thanks,
Alden
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] what does it mean to have exit-code of 251
Thanks. It seems like that call to zookeeper lib API is causing this exit with 251 and hence it is most likely in zookeeper lib code. Regards On Mon, Mar 4, 2019 at 5:26 PM Lennart Poettering wrote: > On Mo, 04.03.19 16:58, prashantkumar dhotre (prashantkumardho...@gmail.com) > wrote: > > > Hi, > > > > In my journal log, I see ; > > > > 1199473 Mar 01 15:46:03 evo-qfx-01 systemd[1]: ifmand.service: Main > process > > exited, code=exited, status=251/n/a > > > > > > I want to know what does 251 means. > > > > Can you please let me know where can I see the exit-code to meaning > mapping > > This depends on the program you are invoking as a service. i.e. ask > the folks who wrote "ifmand" about this, or consult the documentation > for that program. > > When systemd cannot invoke a program you configured it will generate a > couple of error codes on its own. These are documented here: > > > https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Process%20exit%20codes > > But as you can see there 251 is none of the exit codes systemd itself > defines, hence it must come from your own app. > > Lennart > > -- > Lennart Poettering, Red Hat > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] what does it mean to have exit-code of 251
On Mo, 04.03.19 16:58, prashantkumar dhotre (prashantkumardho...@gmail.com) wrote: > Hi, > > In my journal log, I see ; > > 1199473 Mar 01 15:46:03 evo-qfx-01 systemd[1]: ifmand.service: Main process > exited, code=exited, status=251/n/a > > > I want to know what does 251 means. > > Can you please let me know where can I see the exit-code to meaning mapping This depends on the program you are invoking as a service. i.e. ask the folks who wrote "ifmand" about this, or consult the documentation for that program. When systemd cannot invoke a program you configured it will generate a couple of error codes on its own. These are documented here: https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Process%20exit%20codes But as you can see there 251 is none of the exit codes systemd itself defines, hence it must come from your own app. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] what does it mean to have exit-code of 251
In my app, I don't explicitly call exit(251). I use zookeeper lib. Fro my logs,I see that after a call to zookeeper lib API, I see that my service exited with 251. Is 251 a std exit code or app/lib specific custom exit code ? Thanks On Mon, Mar 4, 2019 at 5:02 PM Umut Tezduyar Lindskog wrote: > You need to do some bit analysis - https://shapeshed.com/unix-exit-codes/ > > On Mon, Mar 4, 2019 at 12:29 PM prashantkumar dhotre > wrote: > > > > Hi, > > > > In my journal log, I see ; > > > > 1199473 Mar 01 15:46:03 evo-qfx-01 systemd[1]: ifmand.service: Main > process exited, code=exited, status=251/n/a > > > > > > I want to know what does 251 means. > > > > Can you please let me know where can I see the exit-code to meaning > mapping ? > > > > Thanks > > > > Prashant > > > > > > ___ > > systemd-devel mailing list > > systemd-devel@lists.freedesktop.org > > https://lists.freedesktop.org/mailman/listinfo/systemd-devel > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] what does it mean to have exit-code of 251
You need to do some bit analysis - https://shapeshed.com/unix-exit-codes/ On Mon, Mar 4, 2019 at 12:29 PM prashantkumar dhotre wrote: > > Hi, > > In my journal log, I see ; > > 1199473 Mar 01 15:46:03 evo-qfx-01 systemd[1]: ifmand.service: Main process > exited, code=exited, status=251/n/a > > > I want to know what does 251 means. > > Can you please let me know where can I see the exit-code to meaning mapping > ? > > Thanks > > Prashant > > > ___ > systemd-devel mailing list > systemd-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] what does it mean to have exit-code of 251
Hi, In my journal log, I see ; 1199473 Mar 01 15:46:03 evo-qfx-01 systemd[1]: ifmand.service: Main process exited, code=exited, status=251/n/a I want to know what does 251 means. Can you please let me know where can I see the exit-code to meaning mapping ? Thanks Prashant ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
