[jira] [Commented] (SYSTEMML-1276) Resolve jersey class not found error with Spark2 and YARN
[ https://issues.apache.org/jira/browse/SYSTEMML-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906428#comment-15906428 ] Glenn Weidner commented on SYSTEMML-1276: - A better fix has been provided in this commit: https://github.com/apache/incubator-systemml/commit/28fe4fe8ff28cb093ed345cd22e6280db6654888 Thank you Matthias! > Resolve jersey class not found error with Spark2 and YARN > - > > Key: SYSTEMML-1276 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1276 > Project: SystemML > Issue Type: Improvement > Components: Runtime >Affects Versions: SystemML 0.13 > Environment: Spark 2.x, Hadoop 2.7.3 >Reporter: Glenn Weidner >Assignee: Glenn Weidner > > This is a known issue as reported in [YARN-5271] and [SPARK-15343]. It was > observed during 0.13 performance testing and can be reproduced with following > example: > spark-submit --master yarn --deploy-mode client --class > org.apache.sysml.api.DMLScript ./systemml-0.13.0-incubating-SNAPSHOT.jar -f > ./scripts/utils/sample.dml -exec hybrid_spark -nvargs X=linRegData.csv > sv=perc.csv O=linRegDataParts ofmt=csv > Exception in thread "main" java.lang.NoClassDefFoundError: > com/sun/jersey/api/client/config/ClientConfig > at > org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:182) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:169) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:103) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:97) > at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:122) > at > org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:82) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:75) > at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475) > at org.apache.hadoop.mapred.JobClient.(JobClient.java:454) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.analyzeHadoopCluster(InfrastructureAnalyzer.java:472) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getRemoteParallelMapTasks(InfrastructureAnalyzer.java:114) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getCkMaxMR(InfrastructureAnalyzer.java:298) > at > org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:168) > at > org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:550) > at > org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:145) > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:674) > at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:354) > at org.apache.sysml.api.DMLScript.main(DMLScript.java:199) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: > com.sun.jersey.api.client.config.ClientConfig > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 32 more -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1276) Resolve jersey class not found error with Spark2 and YARN
[ https://issues.apache.org/jira/browse/SYSTEMML-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906427#comment-15906427 ] Matthias Boehm commented on SYSTEMML-1276: -- [~gweidner] there are two intermixed things here: (1) the issue with yarn.timeline-service.enabled, and (2) trying to access the yarn cluster status (which will fail if the resource manager is not running). Regarding (1), this issue is triggered by (2) and there are situations where we can't set it internally because it was already set earlier from outside (e.g., when creating a spark context outside of systemml and coming through mlcontext). I just delivered a fix for (2) as it's mostly unnecessary to probe the yarn cluster status if we're in spark execution modes. > Resolve jersey class not found error with Spark2 and YARN > - > > Key: SYSTEMML-1276 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1276 > Project: SystemML > Issue Type: Improvement > Components: Runtime >Affects Versions: SystemML 0.13 > Environment: Spark 2.x, Hadoop 2.7.3 >Reporter: Glenn Weidner >Assignee: Glenn Weidner > > This is a known issue as reported in [YARN-5271] and [SPARK-15343]. It was > observed during 0.13 performance testing and can be reproduced with following > example: > spark-submit --master yarn --deploy-mode client --class > org.apache.sysml.api.DMLScript ./systemml-0.13.0-incubating-SNAPSHOT.jar -f > ./scripts/utils/sample.dml -exec hybrid_spark -nvargs X=linRegData.csv > sv=perc.csv O=linRegDataParts ofmt=csv > Exception in thread "main" java.lang.NoClassDefFoundError: > com/sun/jersey/api/client/config/ClientConfig > at > org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:182) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:169) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:103) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:97) > at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:122) > at > org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:82) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:75) > at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475) > at org.apache.hadoop.mapred.JobClient.(JobClient.java:454) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.analyzeHadoopCluster(InfrastructureAnalyzer.java:472) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getRemoteParallelMapTasks(InfrastructureAnalyzer.java:114) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getCkMaxMR(InfrastructureAnalyzer.java:298) > at > org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:168) > at > org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:550) > at > org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:145) > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:674) > at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:354) > at org.apache.sysml.api.DMLScript.main(DMLScript.java:199) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: > com.sun.jersey.api.client.config.ClientConfig > at
[jira] [Commented] (SYSTEMML-1276) Resolve jersey class not found error with Spark2 and YARN
[ https://issues.apache.org/jira/browse/SYSTEMML-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906413#comment-15906413 ] Glenn Weidner commented on SYSTEMML-1276: - The change proposed in the PR alone should resolve the issue for parfor on Spark Standalone cluster. > Resolve jersey class not found error with Spark2 and YARN > - > > Key: SYSTEMML-1276 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1276 > Project: SystemML > Issue Type: Improvement > Components: Runtime >Affects Versions: SystemML 0.13 > Environment: Spark 2.x, Hadoop 2.7.3 >Reporter: Glenn Weidner >Assignee: Glenn Weidner > > This is a known issue as reported in [YARN-5271] and [SPARK-15343]. It was > observed during 0.13 performance testing and can be reproduced with following > example: > spark-submit --master yarn --deploy-mode client --class > org.apache.sysml.api.DMLScript ./systemml-0.13.0-incubating-SNAPSHOT.jar -f > ./scripts/utils/sample.dml -exec hybrid_spark -nvargs X=linRegData.csv > sv=perc.csv O=linRegDataParts ofmt=csv > Exception in thread "main" java.lang.NoClassDefFoundError: > com/sun/jersey/api/client/config/ClientConfig > at > org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:182) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:169) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:103) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:97) > at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:122) > at > org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:82) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:75) > at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475) > at org.apache.hadoop.mapred.JobClient.(JobClient.java:454) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.analyzeHadoopCluster(InfrastructureAnalyzer.java:472) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getRemoteParallelMapTasks(InfrastructureAnalyzer.java:114) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getCkMaxMR(InfrastructureAnalyzer.java:298) > at > org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:168) > at > org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:550) > at > org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:145) > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:674) > at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:354) > at org.apache.sysml.api.DMLScript.main(DMLScript.java:199) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: > com.sun.jersey.api.client.config.ClientConfig > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 32 more -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1276) Resolve jersey class not found error with Spark2 and YARN
[ https://issues.apache.org/jira/browse/SYSTEMML-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906386#comment-15906386 ] Matthias Boehm commented on SYSTEMML-1276: -- There is indeed room for improvement. Currently, we always analyze the yarn cluster for the number of nodes, vcores, etc and subsequently apply corrections when in spark execution mode. We should make these calls only if really needed. > Resolve jersey class not found error with Spark2 and YARN > - > > Key: SYSTEMML-1276 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1276 > Project: SystemML > Issue Type: Improvement > Components: Runtime >Affects Versions: SystemML 0.13 > Environment: Spark 2.x, Hadoop 2.7.3 >Reporter: Glenn Weidner >Assignee: Glenn Weidner > > This is a known issue as reported in [YARN-5271] and [SPARK-15343]. It was > observed during 0.13 performance testing and can be reproduced with following > example: > spark-submit --master yarn --deploy-mode client --class > org.apache.sysml.api.DMLScript ./systemml-0.13.0-incubating-SNAPSHOT.jar -f > ./scripts/utils/sample.dml -exec hybrid_spark -nvargs X=linRegData.csv > sv=perc.csv O=linRegDataParts ofmt=csv > Exception in thread "main" java.lang.NoClassDefFoundError: > com/sun/jersey/api/client/config/ClientConfig > at > org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:182) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:169) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:103) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:97) > at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:122) > at > org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:82) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:75) > at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475) > at org.apache.hadoop.mapred.JobClient.(JobClient.java:454) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.analyzeHadoopCluster(InfrastructureAnalyzer.java:472) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getRemoteParallelMapTasks(InfrastructureAnalyzer.java:114) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getCkMaxMR(InfrastructureAnalyzer.java:298) > at > org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:168) > at > org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:550) > at > org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:145) > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:674) > at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:354) > at org.apache.sysml.api.DMLScript.main(DMLScript.java:199) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: > com.sun.jersey.api.client.config.ClientConfig > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 32 more -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1276) Resolve jersey class not found error with Spark2 and YARN
[
https://issues.apache.org/jira/browse/SYSTEMML-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906362#comment-15906362
]
Mike Dusenberry commented on SYSTEMML-1276:
---
[~niketanpansare] had the following finding:
{quote}
This is what happens:
org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.analyzeHadoopCluster
(which is called only via parfor) tries to call
org.apache.hadoop.yarn.client.api.TimelineClient's createTimelineClient. This
method depends on the class com.sun.jersey.api.client.config.ClientConfig.
{quote}
> Resolve jersey class not found error with Spark2 and YARN
> -
>
> Key: SYSTEMML-1276
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1276
> Project: SystemML
> Issue Type: Improvement
> Components: Runtime
>Affects Versions: SystemML 0.13
> Environment: Spark 2.x, Hadoop 2.7.3
>Reporter: Glenn Weidner
>Assignee: Glenn Weidner
>
> This is a known issue as reported in [YARN-5271] and [SPARK-15343]. It was
> observed during 0.13 performance testing and can be reproduced with following
> example:
> spark-submit --master yarn --deploy-mode client --class
> org.apache.sysml.api.DMLScript ./systemml-0.13.0-incubating-SNAPSHOT.jar -f
> ./scripts/utils/sample.dml -exec hybrid_spark -nvargs X=linRegData.csv
> sv=perc.csv O=linRegDataParts ofmt=csv
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/sun/jersey/api/client/config/ClientConfig
> at
> org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:182)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:169)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:103)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:97)
> at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:122)
> at
> org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
> at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)
> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:82)
> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:75)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475)
> at org.apache.hadoop.mapred.JobClient.(JobClient.java:454)
> at
> org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.analyzeHadoopCluster(InfrastructureAnalyzer.java:472)
> at
> org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getRemoteParallelMapTasks(InfrastructureAnalyzer.java:114)
> at
> org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getCkMaxMR(InfrastructureAnalyzer.java:298)
> at
> org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:168)
> at
> org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:550)
> at
> org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:145)
> at org.apache.sysml.api.DMLScript.execute(DMLScript.java:674)
> at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:354)
> at org.apache.sysml.api.DMLScript.main(DMLScript.java:199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
> at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException:
> com.sun.jersey.api.client.config.ClientConfig
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at
[jira] [Commented] (SYSTEMML-1276) Resolve jersey class not found error with Spark2 and YARN
[ https://issues.apache.org/jira/browse/SYSTEMML-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906360#comment-15906360 ] Mike Dusenberry commented on SYSTEMML-1276: --- Well, am interesting thing to keep in mind here is that this only showed up on the Spark Standalone cluster when running a remote Spark parfor loop. All other SystemML jobs on this Spark Standalone cluster up to now have not encountered that error. Thus, the parfor code is uniquely invoking a code path that causes this error to surface. cc [~niketanpansare] > Resolve jersey class not found error with Spark2 and YARN > - > > Key: SYSTEMML-1276 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1276 > Project: SystemML > Issue Type: Improvement > Components: Runtime >Affects Versions: SystemML 0.13 > Environment: Spark 2.x, Hadoop 2.7.3 >Reporter: Glenn Weidner >Assignee: Glenn Weidner > > This is a known issue as reported in [YARN-5271] and [SPARK-15343]. It was > observed during 0.13 performance testing and can be reproduced with following > example: > spark-submit --master yarn --deploy-mode client --class > org.apache.sysml.api.DMLScript ./systemml-0.13.0-incubating-SNAPSHOT.jar -f > ./scripts/utils/sample.dml -exec hybrid_spark -nvargs X=linRegData.csv > sv=perc.csv O=linRegDataParts ofmt=csv > Exception in thread "main" java.lang.NoClassDefFoundError: > com/sun/jersey/api/client/config/ClientConfig > at > org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:182) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:169) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:103) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:97) > at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:122) > at > org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:82) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:75) > at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475) > at org.apache.hadoop.mapred.JobClient.(JobClient.java:454) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.analyzeHadoopCluster(InfrastructureAnalyzer.java:472) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getRemoteParallelMapTasks(InfrastructureAnalyzer.java:114) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getCkMaxMR(InfrastructureAnalyzer.java:298) > at > org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:168) > at > org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:550) > at > org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:145) > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:674) > at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:354) > at org.apache.sysml.api.DMLScript.main(DMLScript.java:199) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: > com.sun.jersey.api.client.config.ClientConfig > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at
[jira] [Commented] (SYSTEMML-1276) Resolve jersey class not found error with Spark2 and YARN
[ https://issues.apache.org/jira/browse/SYSTEMML-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906354#comment-15906354 ] Matthias Boehm commented on SYSTEMML-1276: -- I was under the impression that Spark, even in standalone mode, always has a dependency to hdfs and yarn client libs. I would recommend to locate the libs and config files which are in your classpath during runtime and change the yarn.timeline-service.enabled there. > Resolve jersey class not found error with Spark2 and YARN > - > > Key: SYSTEMML-1276 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1276 > Project: SystemML > Issue Type: Improvement > Components: Runtime >Affects Versions: SystemML 0.13 > Environment: Spark 2.x, Hadoop 2.7.3 >Reporter: Glenn Weidner >Assignee: Glenn Weidner > > This is a known issue as reported in [YARN-5271] and [SPARK-15343]. It was > observed during 0.13 performance testing and can be reproduced with following > example: > spark-submit --master yarn --deploy-mode client --class > org.apache.sysml.api.DMLScript ./systemml-0.13.0-incubating-SNAPSHOT.jar -f > ./scripts/utils/sample.dml -exec hybrid_spark -nvargs X=linRegData.csv > sv=perc.csv O=linRegDataParts ofmt=csv > Exception in thread "main" java.lang.NoClassDefFoundError: > com/sun/jersey/api/client/config/ClientConfig > at > org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:182) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:169) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:103) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:97) > at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:122) > at > org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:82) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:75) > at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475) > at org.apache.hadoop.mapred.JobClient.(JobClient.java:454) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.analyzeHadoopCluster(InfrastructureAnalyzer.java:472) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getRemoteParallelMapTasks(InfrastructureAnalyzer.java:114) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getCkMaxMR(InfrastructureAnalyzer.java:298) > at > org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:168) > at > org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:550) > at > org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:145) > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:674) > at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:354) > at org.apache.sysml.api.DMLScript.main(DMLScript.java:199) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: > com.sun.jersey.api.client.config.ClientConfig > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 32 more -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1276) Resolve jersey class not found error with Spark2 and YARN
[ https://issues.apache.org/jira/browse/SYSTEMML-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906310#comment-15906310 ] Mike Dusenberry commented on SYSTEMML-1276: --- Apparently, this is also showing up on Spark Standalone clusters as well (i.e. not Yarn). > Resolve jersey class not found error with Spark2 and YARN > - > > Key: SYSTEMML-1276 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1276 > Project: SystemML > Issue Type: Improvement > Components: Runtime >Affects Versions: SystemML 0.13 > Environment: Spark 2.x, Hadoop 2.7.3 >Reporter: Glenn Weidner >Assignee: Glenn Weidner > > This is a known issue as reported in [YARN-5271] and [SPARK-15343]. It was > observed during 0.13 performance testing and can be reproduced with following > example: > spark-submit --master yarn --deploy-mode client --class > org.apache.sysml.api.DMLScript ./systemml-0.13.0-incubating-SNAPSHOT.jar -f > ./scripts/utils/sample.dml -exec hybrid_spark -nvargs X=linRegData.csv > sv=perc.csv O=linRegDataParts ofmt=csv > Exception in thread "main" java.lang.NoClassDefFoundError: > com/sun/jersey/api/client/config/ClientConfig > at > org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:182) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:169) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:103) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:97) > at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:122) > at > org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:82) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:75) > at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475) > at org.apache.hadoop.mapred.JobClient.(JobClient.java:454) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.analyzeHadoopCluster(InfrastructureAnalyzer.java:472) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getRemoteParallelMapTasks(InfrastructureAnalyzer.java:114) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getCkMaxMR(InfrastructureAnalyzer.java:298) > at > org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:168) > at > org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:550) > at > org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:145) > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:674) > at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:354) > at org.apache.sysml.api.DMLScript.main(DMLScript.java:199) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: > com.sun.jersey.api.client.config.ClientConfig > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 32 more -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SYSTEMML-1276) Resolve jersey class not found error with Spark2 and YARN
[
https://issues.apache.org/jira/browse/SYSTEMML-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871256#comment-15871256
]
Matthias Boehm commented on SYSTEMML-1276:
--
it also fails on creating the spark context on certain hadoop distributions
(see below).
{code}
Exception in thread "main" java.lang.NoClassDefFoundError:
com/sun/jersey/api/client/config/ClientConfig
at
org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:151)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
at org.apache.spark.SparkContext.(SparkContext.scala:509)
at
org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at
org.apache.sysml.runtime.controlprogram.context.SparkExecutionContext.initSparkContext(SparkExecutionContext.java:215)
at
org.apache.sysml.runtime.controlprogram.context.SparkExecutionContext.getSparkContext(SparkExecutionContext.java:130)
at
org.apache.sysml.runtime.controlprogram.context.SparkExecutionContext.getRDDHandleForMatrixObject(SparkExecutionContext.java:359)
at
org.apache.sysml.runtime.controlprogram.context.SparkExecutionContext.getRDDHandleForVariable(SparkExecutionContext.java:304)
at
org.apache.sysml.runtime.controlprogram.context.SparkExecutionContext.getBinaryBlockRDDHandleForVariable(SparkExecutionContext.java:279)
at
org.apache.sysml.runtime.instructions.spark.MapmmSPInstruction.processInstruction(MapmmSPInstruction.java:117)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:290)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:221)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:168)
at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123)
at org.apache.sysml.api.DMLScript.execute(DMLScript.java:665)
at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:346)
at org.apache.sysml.api.DMLScript.main(DMLScript.java:207)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
com.sun.jersey.api.client.config.ClientConfig
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357
{code}
> Resolve jersey class not found error with Spark2 and YARN
> -
>
> Key: SYSTEMML-1276
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1276
> Project: SystemML
> Issue Type: Improvement
> Components: Runtime
>Affects Versions: SystemML 0.13
> Environment: Spark 2.x, Hadoop 2.7.3
>Reporter: Glenn Weidner
>Assignee: Glenn Weidner
>
> This is a known issue as reported in [YARN-5271] and [SPARK-15343]. It was
> observed during 0.13 performance testing and can be reproduced with following
> example:
> spark-submit --master yarn --deploy-mode client --class
> org.apache.sysml.api.DMLScript ./systemml-0.13.0-incubating-SNAPSHOT.jar -f
> ./scripts/utils/sample.dml -exec hybrid_spark -nvargs X=linRegData.csv
> sv=perc.csv O=linRegDataParts ofmt=csv
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/sun/jersey/api/client/config/ClientConfig
> at
> org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
> at
>
[jira] [Commented] (SYSTEMML-1276) Resolve jersey class not found error with Spark2 and YARN
[ https://issues.apache.org/jira/browse/SYSTEMML-1276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870239#comment-15870239 ] Glenn Weidner commented on SYSTEMML-1276: - [Submitted PR 393 | https://github.com/apache/incubator-systemml/pull/393]. > Resolve jersey class not found error with Spark2 and YARN > - > > Key: SYSTEMML-1276 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1276 > Project: SystemML > Issue Type: Improvement > Components: Runtime >Affects Versions: SystemML 0.13 > Environment: Spark 2.x, Hadoop 2.7.3 >Reporter: Glenn Weidner >Assignee: Glenn Weidner > > This is a known issue as reported in [YARN-5271] and [SPARK-15343]. It was > observed during 0.13 performance testing and can be reproduced with following > example: > spark-submit --master yarn --deploy-mode client --class > org.apache.sysml.api.DMLScript ./systemml-0.13.0-incubating-SNAPSHOT.jar -f > ./scripts/utils/sample.dml -exec hybrid_spark -nvargs X=linRegData.csv > sv=perc.csv O=linRegDataParts ofmt=csv > Exception in thread "main" java.lang.NoClassDefFoundError: > com/sun/jersey/api/client/config/ClientConfig > at > org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:182) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:169) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:103) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:97) > at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:122) > at > org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:82) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:75) > at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475) > at org.apache.hadoop.mapred.JobClient.(JobClient.java:454) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.analyzeHadoopCluster(InfrastructureAnalyzer.java:472) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getRemoteParallelMapTasks(InfrastructureAnalyzer.java:114) > at > org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer.getCkMaxMR(InfrastructureAnalyzer.java:298) > at > org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:168) > at > org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:550) > at > org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:145) > at org.apache.sysml.api.DMLScript.execute(DMLScript.java:674) > at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:354) > at org.apache.sysml.api.DMLScript.main(DMLScript.java:199) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: > com.sun.jersey.api.client.config.ClientConfig > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 32 more -- This message was sent by Atlassian JIRA (v6.3.15#6346)
