[
https://issues.apache.org/jira/browse/BAHIR-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390759#comment-15390759
]
ASF subversion and git services commented on BAHIR-35:
------------------------------------------------------
Commit 416252915431f01bb7a4ef4f8b7b9ed9ab02c3f5 in bahir's branch
refs/heads/master from [~ckadner]
[ https://git-wip-us.apache.org/repos/asf?p=bahir.git;h=4162529 ]
[BAHIR-35] Add Python sources to binary jar
Add python sources to jar to enable `spark-submit --packages …`
This can be verified by the following steps :
mvn clean install
rm -rf ~/.ivy2/cache/org.apache.bahir/
mosquitto -p 1883
bin/run-example \
org.apache.spark.examples.streaming.mqtt.MQTTPublisher \
tcp://localhost:1883 \
foo
${SPARK_HOME}/bin/spark-submit \
--packages org.apache.bahir:spark-streaming-mqtt_2.11:2.0.0-SNAPSHOT \
streaming-mqtt/examples/src/main/python/streaming/mqtt_wordcount.py \
tcp://localhost:1883 \
foo
Closes #11
> Include Python code in the binary jars for use with "--packages ..."
> --------------------------------------------------------------------
>
> Key: BAHIR-35
> URL: https://issues.apache.org/jira/browse/BAHIR-35
> Project: Bahir
> Issue Type: Task
> Components: Build
> Affects Versions: 2.0.0
> Reporter: Christian Kadner
> Assignee: Christian Kadner
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> Currently, to make use the PySpark code (i.e streaming-mqtt/python) a user
> will have to download the jar from Maven central or clone the code from
> GitHub and then have to find individual *.py files, create a zip and add that
> to the {{spark-submit}} command with the {{--py-files}} option, or, add them
> to the {{PYTHONPATH}} when running locally.
> If we include the Python code in the binary build (to the jar that gets
> uploaded to Maven central), then users need not do any acrobatics besides
> using the {{--packages ...}} option.
> An example where the Python code is part of the binary jar is the
> [GraphFrames|https://spark-packages.org/package/graphframes/graphframes]
> package.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
