[
https://issues.apache.org/jira/browse/BAHIR-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380434#comment-15380434
]
ASF GitHub Bot commented on BAHIR-24:
-------------------------------------
Github user ckadner commented on the issue:
https://github.com/apache/bahir/pull/10
**How to test these changes:**
(1) Test that the example `mqtt_wordcount.py` is running fine:
```console
export SPARK_HOME="~/Runtimes/spark/spark-2.0.0-preview-bin-hadoop2.7"
export PYTHONPATH="${BAHIR_HOME}/streaming-mqtt/python"
cd ~/Projects/bahir
mvn clean install -pl streaming-mqtt
${SPARK_HOME}/bin/spark-submit \
--packages org.apache.bahir:spark-streaming-mqtt_2.11:2.0.1-SNAPSHOT \
streaming-mqtt/examples/src/main/python/streaming/mqtt_wordcount.py \
tcp://localhost:1883 foo
```
```
...
16/07/15 19:16:47 WARN AbstractHandler: No Server set for
org.spark_project.jetty.server.handler.ErrorHandler@1e8d73e4
/Users/ckadner/Runtimes/spark/spark-2.0.0-preview-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/shuffle.py:58:
UserWarning: Please install psutil to have better support with spilling
-------------------------------------------
Time: 2016-07-15 19:16:49
-------------------------------------------
-------------------------------------------
Time: 2016-07-15 19:16:50
-------------------------------------------
-------------------------------------------
Time: 2016-07-15 19:16:51
-------------------------------------------
...
```
(2) Test the `bin/run-example` script:
```console
export SPARK_HOME="~/Runtimes/spark/spark-2.0.0-preview-bin-hadoop2.7"
cd ~/Projects/bahir
# mvn clean install -pl streaming-mqtt
bin/run-example \
streaming-mqtt/examples/src/main/python/streaming/mqtt_wordcount.py \
tcp://localhost:1883 foo
```
```
...
16/07/15 19:21:19 WARN AbstractHandler: No Server set for
org.spark_project.jetty.server.handler.ErrorHandler@64bc10e5
/Users/ckadner/Runtimes/spark/spark-2.0.0-preview-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/shuffle.py:58:
UserWarning: Please install psutil to have better support with spilling
-------------------------------------------
Time: 2016-07-15 19:21:21
-------------------------------------------
-------------------------------------------
Time: 2016-07-15 19:21:22
-------------------------------------------
-------------------------------------------
Time: 2016-07-15 19:21:23
-------------------------------------------
^Z
[1]+ Stopped bin/run-example
streaming-mqtt/examples/src/main/python/streaming/mqtt_wordcount.py
tcp://localhost:1883 foo
...
```
(3) Test the error message contains the correct Spark version and Scala
version when Bahir MQTT package is missing or was not resolved:
```console
BAHIR_HOME="~/Projects/bahir"
export SPARK_HOME="~/Runtimes/spark/spark-2.0.0-preview-bin-hadoop2.7"
export PYTHONPATH="${BAHIR_HOME}/streaming-mqtt/python"
${SPARK_HOME}/bin/spark-submit \
${BAHIR_HOME}/streaming-mqtt/examples/src/main/python/streaming/mqtt_wordcount.py
\
tcp://localhost:1883 foo
```
```
________________________________________________________________________________________________
Spark Streaming's MQTT libraries not found in class path. Try one of the
following.
1. Include the MQTT library and its dependencies with in the
spark-submit command as
${SPARK_HOME}/bin/spark-submit --packages
org.apache.bahir:spark-streaming-mqtt_2.11:2.0.0 ...
2. Download the JAR of the artifact from Maven Central
http://search.maven.org/,
Group Id = org.apache.bahir, Artifact Id = spark-streaming-mqtt,
Version = 2.0.0.
Then, include the jar in the spark-submit command as
${SPARK_HOME}/bin/spark-submit --jars <spark-streaming-mqtt.jar> ...
________________________________________________________________________________________________
```
> Fix MQTT Python code
> --------------------
>
> Key: BAHIR-24
> URL: https://issues.apache.org/jira/browse/BAHIR-24
> Project: Bahir
> Issue Type: Bug
> Components: Spark Streaming Connectors
> Affects Versions: 2.0.0
> Reporter: Christian Kadner
> Assignee: Christian Kadner
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> When the Bahir project was created from Spark revision {{8301fadd8}} the
> Python code (incl. examples) were not updated with respect to the modified
> project structure and test cases were left out from the import.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
