[
https://issues.apache.org/jira/browse/BAHIR-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15396910#comment-15396910
]
ASF GitHub Bot commented on BAHIR-38:
-------------------------------------
Github user jodersky commented on the issue:
https://github.com/apache/bahir/pull/14
IIRC, @lresende is on vacation this week. Nevertheless, I think you can go
ahead and merge this quite safely, there is nothing major being changed and the
worst case scenario, messing up the ivy cache, is not that bad either.
> Spark-submit does not use latest locally installed Bahir packages
> -----------------------------------------------------------------
>
> Key: BAHIR-38
> URL: https://issues.apache.org/jira/browse/BAHIR-38
> Project: Bahir
> Issue Type: Bug
> Components: Build
> Affects Versions: 2.0.0
> Environment: Maven (3.3.9) on Mac OS X
> Reporter: Christian Kadner
> Assignee: Christian Kadner
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> We use {{`spark-submit --packages <maven-coordinates> ...`}} to run Spark
> with any of the Bahir extensions.
> In order to perform a _manual integration test_ of a Bahir code change
> developers have to _build_ the respective Bahir module and then _install_ it
> into their *local Maven repository*. Then, when running {{`spark-submit
> --packages <maven-coordinates> ...`}} Spark will use *Ivy* to resolve the
> given _maven-coordinates_ in order add the necessary jar files to the
> classpath.
> The first time Ivy encounters new maven coordinates, it will download them
> from the local or remote Maven repository. All consecutive times Ivy will
> just use the previously cached jar files based on group ID, artifact ID and
> version, but irrespective of creation time stamp.
> This behavior is fine when using spark-submit with released versions of Spark
> packages. For continuous development and integration-testing however that Ivy
> caching behavior poses a problem.
> To *work around* it developers have to *clear the local Ivy cache* each time
> they _install_ a new version of a Bahir package into their local Maven
> repository and before the run spark-submit.
> For example, to test a code change in module streaming-mqtt, we would have to
> do ...
> {code}
> mvn clean install -pl streaming-mqtt
> rm -rf ~/.ivy2/cache/org.apache.bahir/spark-streaming-mqtt_2.11/
> ${SPARK_HOME}/bin/spark-submit \
> --packages org.apache.bahir:spark-streaming-mqtt_2.11:2.0.0-SNAPSHOT \
> test.py
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
