[
https://issues.apache.org/jira/browse/BAHIR-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402361#comment-15402361
]
ASF subversion and git services commented on BAHIR-38:
------------------------------------------------------
Commit 5e07303c65e2c88cd392691bdfe9f68391f51b5c in bahir's branch
refs/heads/master from [~ckadner]
[ https://git-wip-us.apache.org/repos/asf?p=bahir.git;h=5e07303 ]
[BAHIR-38] clean Ivy cache during Maven install phase
When we install the org.apache.bahir jars into the local
Maven repository we also need to clean the previous jar
files from the Ivy cache (~/iv2/cache/org.apache.bahir/*)
so spark-submit -packages ... will pick up the new version
from the the local Maven repository.
Closes #14
> Spark-submit does not use latest locally installed Bahir packages
> -----------------------------------------------------------------
>
> Key: BAHIR-38
> URL: https://issues.apache.org/jira/browse/BAHIR-38
> Project: Bahir
> Issue Type: Bug
> Components: Build
> Affects Versions: 2.0.0
> Environment: Maven (3.3.9) on Mac OS X
> Reporter: Christian Kadner
> Assignee: Christian Kadner
> Fix For: 2.0.0
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> We use {{`spark-submit --packages <maven-coordinates> ...`}} to run Spark
> with any of the Bahir extensions.
> In order to perform a _manual integration test_ of a Bahir code change
> developers have to _build_ the respective Bahir module and then _install_ it
> into their *local Maven repository*. Then, when running {{`spark-submit
> --packages <maven-coordinates> ...`}} Spark will use *Ivy* to resolve the
> given _maven-coordinates_ in order add the necessary jar files to the
> classpath.
> The first time Ivy encounters new maven coordinates, it will download them
> from the local or remote Maven repository. All consecutive times Ivy will
> just use the previously cached jar files based on group ID, artifact ID and
> version, but irrespective of creation time stamp.
> This behavior is fine when using spark-submit with released versions of Spark
> packages. For continuous development and integration-testing however that Ivy
> caching behavior poses a problem.
> To *work around* it developers have to *clear the local Ivy cache* each time
> they _install_ a new version of a Bahir package into their local Maven
> repository and before the run spark-submit.
> For example, to test a code change in module streaming-mqtt, we would have to
> do ...
> {code}
> mvn clean install -pl streaming-mqtt
> rm -rf ~/.ivy2/cache/org.apache.bahir/spark-streaming-mqtt_2.11/
> ${SPARK_HOME}/bin/spark-submit \
> --packages org.apache.bahir:spark-streaming-mqtt_2.11:2.0.0-SNAPSHOT \
> test.py
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
