Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/10
**How to test these changes:**
(1) Test that the example `mqtt_wordcount.py` is running fine:
```console
export SPARK_HOME="~/Runtimes/spark/spark-2.0.0-preview-bin-hadoop2.7"
export PYTHONPATH="${BAHIR_HOME}/streaming-mqtt/python"
cd ~/Projects/bahir
mvn clean install -pl streaming-mqtt
${SPARK_HOME}/bin/spark-submit \
--packages org.apache.bahir:spark-streaming-mqtt_2.11:2.0.1-SNAPSHOT \
streaming-mqtt/examples/src/main/python/streaming/mqtt_wordcount.py \
tcp://localhost:1883 foo
```
```
...
16/07/15 19:16:47 WARN AbstractHandler: No Server set for
org.spark_project.jetty.server.handler.ErrorHandler@1e8d73e4
/Users/ckadner/Runtimes/spark/spark-2.0.0-preview-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/shuffle.py:58:
UserWarning: Please install psutil to have better support with spilling
-------------------------------------------
Time: 2016-07-15 19:16:49
-------------------------------------------
-------------------------------------------
Time: 2016-07-15 19:16:50
-------------------------------------------
-------------------------------------------
Time: 2016-07-15 19:16:51
-------------------------------------------
...
```
(2) Test the `bin/run-example` script:
```console
export SPARK_HOME="~/Runtimes/spark/spark-2.0.0-preview-bin-hadoop2.7"
cd ~/Projects/bahir
# mvn clean install -pl streaming-mqtt
bin/run-example \
streaming-mqtt/examples/src/main/python/streaming/mqtt_wordcount.py \
tcp://localhost:1883 foo
```
```
...
16/07/15 19:21:19 WARN AbstractHandler: No Server set for
org.spark_project.jetty.server.handler.ErrorHandler@64bc10e5
/Users/ckadner/Runtimes/spark/spark-2.0.0-preview-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/shuffle.py:58:
UserWarning: Please install psutil to have better support with spilling
-------------------------------------------
Time: 2016-07-15 19:21:21
-------------------------------------------
-------------------------------------------
Time: 2016-07-15 19:21:22
-------------------------------------------
-------------------------------------------
Time: 2016-07-15 19:21:23
-------------------------------------------
^Z
[1]+ Stopped bin/run-example
streaming-mqtt/examples/src/main/python/streaming/mqtt_wordcount.py
tcp://localhost:1883 foo
...
```
(3) Test the error message contains the correct Spark version and Scala
version when Bahir MQTT package is missing or was not resolved:
```console
BAHIR_HOME="~/Projects/bahir"
export SPARK_HOME="~/Runtimes/spark/spark-2.0.0-preview-bin-hadoop2.7"
export PYTHONPATH="${BAHIR_HOME}/streaming-mqtt/python"
${SPARK_HOME}/bin/spark-submit \
${BAHIR_HOME}/streaming-mqtt/examples/src/main/python/streaming/mqtt_wordcount.py
\
tcp://localhost:1883 foo
```
```
________________________________________________________________________________________________
Spark Streaming's MQTT libraries not found in class path. Try one of the
following.
1. Include the MQTT library and its dependencies with in the
spark-submit command as
${SPARK_HOME}/bin/spark-submit --packages
org.apache.bahir:spark-streaming-mqtt_2.11:2.0.0 ...
2. Download the JAR of the artifact from Maven Central
http://search.maven.org/,
Group Id = org.apache.bahir, Artifact Id = spark-streaming-mqtt,
Version = 2.0.0.
Then, include the jar in the spark-submit command as
${SPARK_HOME}/bin/spark-submit --jars <spark-streaming-mqtt.jar> ...
________________________________________________________________________________________________
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
