[ 
https://issues.apache.org/jira/browse/BAHIR-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402362#comment-15402362
 ] 

ASF GitHub Bot commented on BAHIR-38:
-------------------------------------

Github user asfgit closed the pull request at:

    https://github.com/apache/bahir/pull/14


> Spark-submit does not use latest locally installed Bahir packages
> -----------------------------------------------------------------
>
>                 Key: BAHIR-38
>                 URL: https://issues.apache.org/jira/browse/BAHIR-38
>             Project: Bahir
>          Issue Type: Bug
>          Components: Build
>    Affects Versions: 2.0.0
>         Environment: Maven (3.3.9) on Mac OS X
>            Reporter: Christian Kadner
>            Assignee: Christian Kadner
>             Fix For: 2.0.0
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> We use {{`spark-submit --packages <maven-coordinates> ...`}} to run Spark 
> with any of the Bahir extensions. 
> In order to perform a _manual integration test_ of a Bahir code change 
> developers have to _build_ the respective Bahir module and then _install_ it 
> into their *local Maven repository*. Then, when running {{`spark-submit 
> --packages <maven-coordinates> ...`}} Spark will use *Ivy* to resolve the 
> given _maven-coordinates_ in order add the necessary jar files to the 
> classpath.
> The first time Ivy encounters new maven coordinates, it will download them 
> from the local or remote Maven repository. All consecutive times Ivy will 
> just use the previously cached jar files based on group ID, artifact ID and 
> version, but irrespective of creation time stamp. 
> This behavior is fine when using spark-submit with released versions of Spark 
> packages. For continuous development and integration-testing however that Ivy 
> caching behavior poses a problem. 
> To *work around* it developers have to *clear the local Ivy cache* each time 
> they _install_ a new version of a Bahir package into their local Maven 
> repository and before the run spark-submit.
> For example, to test a code change in module streaming-mqtt, we would have to 
> do ...
> {code}
> mvn clean install -pl streaming-mqtt
> rm -rf ~/.ivy2/cache/org.apache.bahir/spark-streaming-mqtt_2.11/
> ${SPARK_HOME}/bin/spark-submit \
>     --packages org.apache.bahir:spark-streaming-mqtt_2.11:2.0.0-SNAPSHOT \
>     test.py
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to