[
https://issues.apache.org/jira/browse/BAHIR-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389047#comment-15389047
]
ASF GitHub Bot commented on BAHIR-35:
-------------------------------------
GitHub user ckadner opened a pull request:
https://github.com/apache/bahir/pull/11
[BAHIR-35] add Python sources to binary jar for use with `spark-submit
--packages …`
[BAHIR-35: Include Python code in the binary jars for use with "--packages
..."](https://issues.apache.org/jira/browse/BAHIR-35)
**Change:**
Adding `${basedir}/python/**/*.py` files as `<resources>` in the build
section of `pom.xml`:
```xml
...
<build>
<resources>
<resource>
<directory>${basedir}/python</directory>
<includes>
<include>**/*.py</include>
</includes>
</resource>
</resources>
...
```
Currently only the **MQTT** Streaming connector is affected by this change.
**Test:**
After PR #10 is merged, run the following commands:
```console
mvn clean install
rm -rf ~/.ivy2/cache/org.apache.bahir/
mosquitto -p 1883
bin/run-example \
org.apache.spark.examples.streaming.mqtt.MQTTPublisher
tcp://localhost:1883 foo
${SPARK_HOME}/bin/spark-submit \
--packages org.apache.bahir:spark-streaming-mqtt_2.11:2.0.0-SNAPSHOT \
streaming-mqtt/examples/src/main/python/streaming/mqtt_wordcount.py \
tcp://localhost:1883 foo
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ckadner/bahir
BAHIR-35_include_Python_sources_in_binary_build
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/bahir/pull/11.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11
----
commit 4c41f20b46cf782d64b1e5b45ec910806e67c71d
Author: Christian Kadner <[email protected]>
Date: 2016-07-22T06:43:49Z
[BAHIR-35] add Python sources to binary build for use with `spark-submit
--packages ...`
----
> Include Python code in the binary jars for use with "--packages ..."
> --------------------------------------------------------------------
>
> Key: BAHIR-35
> URL: https://issues.apache.org/jira/browse/BAHIR-35
> Project: Bahir
> Issue Type: Task
> Components: Build
> Affects Versions: 2.0.0
> Reporter: Christian Kadner
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> Currently, to make use the PySpark code (i.e streaming-mqtt/python) a user
> will have to download the jar from Maven central or clone the code from
> GitHub and then have to find individual *.py files, create a zip and add that
> to the {{spark-submit}} command with the {{--py-files}} option, or, add them
> to the {{PYTHONPATH}} when running locally.
> If we include the Python code in the binary build (to the jar that gets
> uploaded to Maven central), then users need not do any acrobatics besides
> using the {{--packages ...}} option.
> An example where the Python code is part of the binary jar is the
> [GraphFrames|https://spark-packages.org/package/graphframes/graphframes]
> package.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
