[
https://issues.apache.org/jira/browse/SLIDER-330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated SLIDER-330:
----------------------------------
Sprint: Slider September #1
> Slider to upload all JARs to AM via distributed cache
> -----------------------------------------------------
>
> Key: SLIDER-330
> URL: https://issues.apache.org/jira/browse/SLIDER-330
> Project: Slider
> Issue Type: New Feature
> Components: client
> Affects Versions: Slider 0.50
> Reporter: Steve Loughran
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Currently slider only uploads the JARs we know/think aren't on the YARN
> classpath, and sets the CP from that and the {{yarn.application.classpath}}
> conf option
> This is brittle against Hadoop versions and installations
> # the option {{yarn.application.classpath}} may be missing or wrong, at
> which point the AM refuses to start
> # if YARN adds a different version of a dependency we push up, we end up in
> >1 version on the CP pain.
> # if the Hadoop installation itself has some older JARs which don't link up
> to our code (e.g. AM uses protobuf fields not in hadoop-yarn JARs), then AM
> won't start with linkage errors.
> The solution: isolate our classpath by pushing up all dependencies.
> This requires Slider to know all its dependencies, which can be done via one
> of
> * static coding (painful, brittle)
> * chained .asm dependency analysis (As Twill does). Efficient and
> near-invisible, but breaks for any introspective binding.
> * upload {{$SLIDER_HOME/lib/*.jar}}. Relies on mvn to build the files, the
> assembly code to add them to RPM and zip.
> The last option, upload {{lib/*.jar}} seems the easiest and should be
> reliable, provided {{$SLIDER_HOME/lib/*.jar}} is set.
> For slider-client-as-API, the homedir may be unset. Here we propose letting
> the client config define the homedir, having that over-ride any env var.
> For performance, especially long-haul, the distributed cache should be used
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
