[ https://issues.apache.org/jira/browse/BAHIR-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402362#comment-15402362 ]
ASF GitHub Bot commented on BAHIR-38: ------------------------------------- Github user asfgit closed the pull request at: https://github.com/apache/bahir/pull/14 > Spark-submit does not use latest locally installed Bahir packages > ----------------------------------------------------------------- > > Key: BAHIR-38 > URL: https://issues.apache.org/jira/browse/BAHIR-38 > Project: Bahir > Issue Type: Bug > Components: Build > Affects Versions: 2.0.0 > Environment: Maven (3.3.9) on Mac OS X > Reporter: Christian Kadner > Assignee: Christian Kadner > Fix For: 2.0.0 > > Original Estimate: 4h > Remaining Estimate: 4h > > We use {{`spark-submit --packages <maven-coordinates> ...`}} to run Spark > with any of the Bahir extensions. > In order to perform a _manual integration test_ of a Bahir code change > developers have to _build_ the respective Bahir module and then _install_ it > into their *local Maven repository*. Then, when running {{`spark-submit > --packages <maven-coordinates> ...`}} Spark will use *Ivy* to resolve the > given _maven-coordinates_ in order add the necessary jar files to the > classpath. > The first time Ivy encounters new maven coordinates, it will download them > from the local or remote Maven repository. All consecutive times Ivy will > just use the previously cached jar files based on group ID, artifact ID and > version, but irrespective of creation time stamp. > This behavior is fine when using spark-submit with released versions of Spark > packages. For continuous development and integration-testing however that Ivy > caching behavior poses a problem. > To *work around* it developers have to *clear the local Ivy cache* each time > they _install_ a new version of a Bahir package into their local Maven > repository and before the run spark-submit. > For example, to test a code change in module streaming-mqtt, we would have to > do ... > {code} > mvn clean install -pl streaming-mqtt > rm -rf ~/.ivy2/cache/org.apache.bahir/spark-streaming-mqtt_2.11/ > ${SPARK_HOME}/bin/spark-submit \ > --packages org.apache.bahir:spark-streaming-mqtt_2.11:2.0.0-SNAPSHOT \ > test.py > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)