Hi all, we are starting to use spark streaming in an automotive IOT environment with MQTT as our primary data source. During implementation we have found that the current implementation of the spark-bahir MQTT sources makes them hard to use for our purposes. As the technologies we use are often used in that sort of environment I'd like to state these points for discussion to see if they can be addressed.
The problems we face are: - No way to specify username, password and other important MQTT connection parameters (Already addressed in BAHIR-51, Thanks to Prashant Sharma for contributing the changes for MqttTextStreamSource). - The messages we receive via MQTT are mostly binary (in our case binary protocol buffer messages). Currently the MQTT streaming sources assume all MQTT messages are UTF-8 encoded strings. This destroys the binary payloads. This would mean changing the type from String to Array[Byte], but that would break the existing API. What is your stance on this? New Sources (e.g. MQTTBinaryInputDStream and MQTTBinaryStreamSource)? - In some cases the source will listen on a wildcard MQTT topic (e.g. sensordata/+/+) but during processing we need to know the exact topic the message arrived on (e.g. sensordata/deviceid1/sensorid1). So it would be great if the source not only returns the payload but also the topic. For MqttTextStreamSource the schema could be extended with a new column "topic" (I'm not sure if this would be backwards compatible). For MQTTInputDStream the change would be backwards incompatible (e.g. returning a Message object (with payload, topic and timestamp) instead of a string). I can provide pull requests for these changes, but since most are backwards incompatible I first wanted to get your opinion before doing the implementation. Greetings Sebastian Woehrl __________________________________________________ Cross-Plattform-Strategie für Ihre Mobile App https://www.maibornwolff.de/cross-plattform __________________________________________________ Sebastian Wöhrl Software Engineer T. +49 89 544 253 156 M. +49 151 544 22 156 [email protected] Theresienhöhe 13, 80339 München MaibornWolff GmbH, Theresienhöhe 13, D-80339 München http://www.maibornwolff.de, Telefon +49 89 544 253 000 USt-ID DE 129 299 525, Amtsgericht München HRB 98058, GF: Volker Maiborn, Holger Wolff, Jens Rieger, Alexander Hofmann __________________________________________________
