Hi Chris, Unlike MR and its support of distributed cache, Tez does not make any inferences into the structure of the LocalResources specified ( i.e structure of tarball, jar, etc ) and therefore expects the user to modify the class path as needed.
It might be something worth considering as a new feature ( please file a jira ) but the current implementation expects the user to setup the classpath as needed to handle tar-balls, fat-jars, etc correctly. — Hitesh On Sep 11, 2014, at 12:09 PM, Chris K Wensel <[email protected]> wrote: > > I'm setting my what used to be called a hadoop job jar as a local resource, > with APPLICATION visibility, of type PATTERN with the pattern > "(?:classes/|lib/).*" (right from the JobConf) > > the good news is when a remote tez client starts, the job jar is downloaded, > and unpacked using the pattern > > proof: > > find > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/ > | grep logparser.jar > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/.tmp_logparser.jar.crc > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/jgraphx-2.0.0.1.jar > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/jgrapht-ext-0.9.0.jar > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/jgraph-5.13.0.0.jar > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/cascading-xml-3.0.0-wip-dev.jar > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/tagsoup-1.2.jar > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/riffle-0.1-dev.jar > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/jgrapht-core-0.9.0.jar > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/cascading-hadoop2-tez-3.0.0-wip-dev.jar > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/janino-2.6.1.jar > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/cascading-core-3.0.0-wip-dev.jar > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/commons-compiler-2.6.1.jar > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/logparser.jar > > the bad news is that the 'launch_container.sh' is only adding > ln -sf > "/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar" > "logparser.jar" > > so the containers only see > /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/container_1410456214816_0001_01_000004/logparser.jar > > which isn't terribly helpful as it result in > > 2014-09-11 10:57:41,001 INFO [TezChild] > org.apache.tez.runtime.task.TezTaskRunner: Encounted an error while executing > task: attempt_1410456214816_0001_1_00_000000_2 > org.apache.tez.dag.api.TezUncheckedException: Unable to load class: > cascading.flow.tez.FlowProcessor > at > org.apache.tez.common.ReflectionUtils.getClazz(ReflectionUtils.java:45) > at > org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:96) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.createProcessor(LogicalIOProcessorRuntimeTask.java:563) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.initialize(LogicalIOProcessorRuntimeTask.java:187) > > i'm obviously missing something. or is making classic job jars (with a lib > folder) isn't really supported transparently anymore (as an option) which > will cause some grief. > > this is hadoop 2.4.1 > > ckw > > -- > Chris K Wensel > [email protected] > http://concurrentinc.com >
