Hi Martin, The AM classpath contains the application jar is because user code can implements an event handler than runs inside the AM. I have not seeing issue in my cluster. Does your application jar contains Kafka jars ?
Terence Sent from my iPhone > On Feb 10, 2017, at 12:29 PM, Martin Serrano <mar...@attivio.com> wrote: > > Devs, > > In our full deployment environment I could not get the Kafka forwarding of > logs to work. I kept getting Kafka errors on the AM trying to lookup the > topic. Seeing as how I had been able to get this same runnable working in a > unit test environment I figured it had to do with the classpath. Looking > deeper I saw that the AM runs with the application.jar contents on its > classpath. Why is that? It seems to me that the runnable classpaths should > never be part of the AM. > > I changed the TwillLauncher to not use the application.jar for the AM > classpath and got a CNFE in the AM for jsimpleopt.OptionSpec. It seems this > is an implicit dependency of Kafka that is not currently discovered by the > dependency mechanism (presumably because Kafka is written in Scala). When I > added jsimpleopt-3.2.jar to my classpath and as a dependency class for the AM > everything worked! I was not getting the CNFE when application.jar/lib/* was > on the classpath so something in my application libs must have been picked up > by Kafka initialization. > > IMO, the AppMaster is internal Twill code and its dependencies should be > fully provided by the distribution and self-contained. That may present some > build challenges, but users should not ever run into this stuff. I'll file > and ticket and submit a PR if there is agreement on this, but the > application.jar/lib/* being on the AM classpath seems pretty intentional from > looking at the code. > > Cheers, > -Martin > >
