————— 2022-5-19 ————— Mr. Lightning 10:10 Guys, please tell me if the spark 2.4.3 plugin is also related to the hadoop version. Now I can run the spark-sql command on the ecm node, and the job can also go to yarn, but the linkis cli fails to run, and the spark context report fails. Does this mean I need to recompile the spark engine?
Mr. Lightning 10:11 linkis cli jobs can be submitted to yarn Mr. Lightning 10:14 error initializing sparkcontext peacewong@WDS 10:15 There should be no difference between the two, but the submission entry is different Mr. Lightning 10:18 i try again spark-sql -e Mr. Lightning 10:23 I can run spark-sql on both nodes, but I can't run it successfully with linkis-cli-spark-sql Mr. Lightning 10:23 Failed to initialize engine peacewong@WDS 10:37 Brother Hua Jin, you can send the log of ec to see peacewong@WDS 10:38 You can consider removing the cache and changing it to configurable. The default configuration is to use the cache. You can come to an issue and pr Mr. Lightning 10:52 I read it. I probably know that our yarn cluster doesn't know about my client [Wang Chai] Mr. Lightning 10:53 I will try again later thank you peace peacewong@WDS 10:53 [shake hands] ————— 2022-5-19 ————— 闪电先生 10:10 各位大佬 请教一下 spark 2.4.3那个插件，是不是也是和hadoop版本有关的。 我现在spark-sql命令在ecm节点上面是可以跑的，作业也可以到yarn，但是用linkis cli跑不成功，报spark上下文失败。 这是说明我要重新编译spark引擎吗？ 闪电先生 10:11 linkis cli作业可以提交到yarn 闪电先生 10:14 error initializing sparkcontext peacewong@WDS 10:15 这两个应该没有区别来着，只是提交入口不一样 闪电先生 10:18 我再试一下 spark-sql -e 闪电先生 10:23 我二个节点spark-sql都可以跑，用linkis-cli-spark-sql跑不成功 闪电先生 10:23 初始化引擎失败 peacewong@WDS 10:37 华金兄可以把ec的日志发下看看 peacewong@WDS 10:38 可以考虑把cache移除下，改成可以配置的，默认配置走缓存。可以来个issue和pr下 闪电先生 10:52 我看了。我大概知道了 我们的yarn集群不知道我的这个客户端[旺柴] 闪电先生 10:53 我等会再试试 谢谢和平 peacewong@WDS 10:53 [握手] --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@linkis.apache.org For additional commands, e-mail: dev-h...@linkis.apache.org
