Re: [Dorset] Python Problem in MQTT subscriber program
On Fri, 27 Aug 2021 14:02:19 +0100, PeterMerchant wrote: > Patrick, I am most grateful for your comments. You obviously know a > lot more about MQTT than I. I wouldn't say I know a lot about MQTT, and I did not mean to imply that I do. I have a fairly basic high-level understanding of it, but I've forgotten most of the lower-level details I previously had a light grasp of! I haven't looked at paho-mqtt or Arduino MQTT before, so for my own understanding I had to go back over the basics that you are probably used to dealing with. I may well have learnt more from trying to find the problem in your code than you will have learnt from me! I didn't want to make any assumptions about what kind of data your system was sending and how it was packaging it up and unpackaging it, because there is potentially more than one MQTT client library for Python, and there is certainly more than one way to encode an MQTT payload. For example, some people might use Google's Protocol Buffers or another serialisation scheme to encode program objects into bytes and then stuff those into an MQTT payload. Others might define a scheme of their own. In your case, it is ASCII text (a perfectly valid option). Just in case it was unclear to anyone: when I said the payload is bytes, rather than a text string, what I meant was that it is not a Python text string object, and MQTT does not require it to encode a text string. However, in this particular program it does encode an ASCII text string, so in that sense it IS very much a text string, and Python makes it easy to treat it as one. Patrick -- Next meeting: Online, Jitsi, Tuesday, 2021-09-07 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
Re: [Dorset] Python Problem in MQTT subscriber program
Patrick, I am most grateful for your comments. You obviously know a lot more about MQTT than I. I think that I'll have a little play with the software that I have working, and then start cleaning it up using some of your suggestions. And perhaps read a bit more of the MQTT v5 spec. Much thanks, Peter On 27/08/2021 12:23, Patrick Wigmore wrote: Hmm. I don't think I'm able to solve this. Referring to the full MQTT subscriber/car control program you sent me off-list, this is some of the context I was missing: It is using paho-mqtt, f.k.a. Mosquitto, and message is a MQTTMessage object, with message.payload being arbitrary bytes. It looks like this payload is indeed the unadulterated MQTT payload, as sent over the network. So, the payload is bytes, rather than a text string, and there isn't any intervening code that's not shown and does additional encoding of the payload. on_message is run by paho-mqtt as an on_message callback function when an MQTT PUBLISH message is received in a subscribed topic, which I believe is what you intend. The publisher is using the Arduino MQTT Client, which wraps lwmqtt. This much is probably obvious if you've been immersed in it for a while. So, regardless of the payload contents: * Given that message.payload is a Python Bytes object, which it should be, print(message.payload) should always work. Bytes that don't encode printable characters print as hexadecimal within an escape sequence. * Testing whether the payload equals some specific bytes should at least not cause a crash, even if the test doesn't evaluate to True/ False as expected. * The if/elif/else structure looks fine. So, unless I'm missing a syntax error or typo somewhere, I don't think there's anything wrong with your non-working on_message! Just to confirm to myself how Python's print handles unprintable bytes: >>> print(b'\x00\x00hello\xff\x68\x65\x6c\x6c\x6f') b'\x00\x00hello\xffhello' I'll assume that all the functions that do the GPIO (e.g. motorleft) are correct. That could be a wrong assumption, but I'll assume it anyway because it seems likely. Given that you were interpreting the same bytes (b'Pot-left' and b'Pot-rt') successfully before, all I can think is that the code at the publishing end must have changed too, to get it to send different bytes. Since the subscriber crashes, there must be something about the way the publishing code was changed that causes it to send messages that paho-mqtt can't handle. Because paho-mqtt shouldn't care about the contents of the payload, this would seem to suggest that there is a mistake in the way the MQTT packets are being constructed at the publisher. I'm having a hard time imagining that you would have intentionally significantly altered the code at the publishing end, though, so it is probably something quite subtle; perhaps an issue that was already there but is only tripped up by something else that changed. But I feel like I'm probably missing something obvious here. Normally when I try to make these kinds of deductions on the list, someone comes along and points out a glaring error in my logic, so no doubt I am making a similar error right now! Patrick -- Next meeting: Online, Jitsi, Tuesday, 2021-09-07 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
Re: [Dorset] Python Problem in MQTT subscriber program
Hmm. I don't think I'm able to solve this. Referring to the full MQTT subscriber/car control program you sent me off-list, this is some of the context I was missing: It is using paho-mqtt, f.k.a. Mosquitto, and message is a MQTTMessage object, with message.payload being arbitrary bytes. It looks like this payload is indeed the unadulterated MQTT payload, as sent over the network. So, the payload is bytes, rather than a text string, and there isn't any intervening code that's not shown and does additional encoding of the payload. on_message is run by paho-mqtt as an on_message callback function when an MQTT PUBLISH message is received in a subscribed topic, which I believe is what you intend. The publisher is using the Arduino MQTT Client, which wraps lwmqtt. This much is probably obvious if you've been immersed in it for a while. So, regardless of the payload contents: * Given that message.payload is a Python Bytes object, which it should be, print(message.payload) should always work. Bytes that don't encode printable characters print as hexadecimal within an escape sequence. * Testing whether the payload equals some specific bytes should at least not cause a crash, even if the test doesn't evaluate to True/ False as expected. * The if/elif/else structure looks fine. So, unless I'm missing a syntax error or typo somewhere, I don't think there's anything wrong with your non-working on_message! Just to confirm to myself how Python's print handles unprintable bytes: >>> print(b'\x00\x00hello\xff\x68\x65\x6c\x6c\x6f') b'\x00\x00hello\xffhello' I'll assume that all the functions that do the GPIO (e.g. motorleft) are correct. That could be a wrong assumption, but I'll assume it anyway because it seems likely. Given that you were interpreting the same bytes (b'Pot-left' and b'Pot-rt') successfully before, all I can think is that the code at the publishing end must have changed too, to get it to send different bytes. Since the subscriber crashes, there must be something about the way the publishing code was changed that causes it to send messages that paho-mqtt can't handle. Because paho-mqtt shouldn't care about the contents of the payload, this would seem to suggest that there is a mistake in the way the MQTT packets are being constructed at the publisher. I'm having a hard time imagining that you would have intentionally significantly altered the code at the publishing end, though, so it is probably something quite subtle; perhaps an issue that was already there but is only tripped up by something else that changed. But I feel like I'm probably missing something obvious here. Normally when I try to make these kinds of deductions on the list, someone comes along and points out a glaring error in my logic, so no doubt I am making a similar error right now! Patrick -- Next meeting: Online, Jitsi, Tuesday, 2021-09-07 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
Re: [Dorset] Python Problem in MQTT subscriber program
On Thu, 26 Aug 2021 14:10:31 +0100, PeterMerchant wrote: > Problem sorted, though I don't know why. > I took out the hyphens in the Pot-Left & Right strings, and made > them Pleft and Pright. and it does not crash. > > Peter. I don't know whether you are still interested in getting to the bottom of this now that you've got it working. I've just had a look at your original message with the code excerpts and I could not see anything that stood out to me as obviously wrong. But I am missing a lot of context. Here are some questions and suppositions I might put to myself if I was trying to get to the root of why this was happening: When does it crash? i.e. what part of the program is it executing when it crashes. I'm guessing it crashes when the subscriber receives a new message, rather than as soon as you try to run the program, but does it crash for _any_ message, or only messages with particular payloads? What (if any) traceback do you get when it crashes? That's probably useful information. I don't know what kind of object 'message' is, or what calls these functions you've posted. That might be relevant. It could be that whatever code in the subscriber is creating the message object is not handling the hyphens properly, or it is not encoding the strings into exactly the format you expected. Maybe the hyphens are being converted into some other character *before* getting sent over the message bus, or perhaps the byte encoding is coming directly from the publisher and is not 100% compatible with the encoding that the subscriber is expecting. Are both the publisher and the subscriber written in Python? Does the underlying CPU architecture affect the binary format for either one, or is that all abstracted away for us? Maybe more messages than expected are being sent and received. Perhaps motorleft(), motorright(), etc, can't handle being called too frequently. No worries if you are not going to dig any further into this. Patrick -- Next meeting: Online, Jitsi, Tuesday, 2021-09-07 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
Re: [Dorset] Python Problem in MQTT subscriber program
Problem sorted, though I don't know why. I took out the hyphens in the Pot-Left & Right strings, and made them Pleft and Pright. and it does not crash. Peter. On 23/08/2021 18:29, PeterMerchant wrote: I have three versions of this program, and two work, but the third with the differences put together crashes. Can anybody advise me what is happening. Here are the relevant bits. 1. Working def on_message(client, userdata, message): if message.payload == b'forward': up_15() elif message.payload == b'slow': down_15() elif message.payload == b'stop': on() elif message.payload == b'motorleft': motorleft() elif message.payload == b'motorright': motorright() elif message.payload == b'backwards': mbackwards() else: print(message.payload) the stuff in quotes 'forward' etc is the message payload that I act on. 2 Working def on_message(client, userdata, message): if message.payload == b'forward': up_15() elif message.payload == b'slow': down_15() elif message.payload == b'stop': on() elif message.payload == b'Pot-left': motorleft() elif message.payload == b'Pot-rt': motorright() elif message.payload == b'backwards': mbackwards() else: print(message.payload) the left and right have been changed to Potentiometer values and 3, not working def on_message(client, userdata, message): if message.payload == b'forward': up_15() elif message.payload == b'slow': down_15() elif message.payload == b'stop': on() elif message.payload == b'motorleft': motorleft() elif message.payload == b'motorright': motorright() elif message.payload == b'Pot-left': motorleft() elif message.payload == b'Pot-rt': motorright() elif message.payload == b'backwards': mbackwards() else: print(message.payload) Actually the Pot-rt lines have been commented out. It did work with both the Pot-* bits commented out. Thanks for any thoughts. -- Next meeting: Online, Jitsi, Tuesday, 2021-09-07 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
[Dorset] Python Problem in MQTT subscriber program
I have three versions of this program, and two work, but the third with the differences put together crashes. Can anybody advise me what is happening. Here are the relevant bits. 1. Working def on_message(client, userdata, message): if message.payload == b'forward': up_15() elif message.payload == b'slow': down_15() elif message.payload == b'stop': on() elif message.payload == b'motorleft': motorleft() elif message.payload == b'motorright': motorright() elif message.payload == b'backwards': mbackwards() else: print(message.payload) the stuff in quotes 'forward' etc is the message payload that I act on. 2 Working def on_message(client, userdata, message): if message.payload == b'forward': up_15() elif message.payload == b'slow': down_15() elif message.payload == b'stop': on() elif message.payload == b'Pot-left': motorleft() elif message.payload == b'Pot-rt': motorright() elif message.payload == b'backwards': mbackwards() else: print(message.payload) the left and right have been changed to Potentiometer values and 3, not working def on_message(client, userdata, message): if message.payload == b'forward': up_15() elif message.payload == b'slow': down_15() elif message.payload == b'stop': on() elif message.payload == b'motorleft': motorleft() elif message.payload == b'motorright': motorright() elif message.payload == b'Pot-left': motorleft() elif message.payload == b'Pot-rt': motorright() elif message.payload == b'backwards': mbackwards() else: print(message.payload) Actually the Pot-rt lines have been commented out. It did work with both the Pot-* bits commented out. Thanks for any thoughts. -- Next meeting: Online, Jitsi, Tuesday, 2021-09-07 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
