This is an automated email from the ASF dual-hosted git repository.
peacewong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/linkis.git
The following commit(s) were added to refs/heads/master by this push:
new d6a86a1db Modify spark.md (#4875)
d6a86a1db is described below
commit d6a86a1dbcf25669379a4c562740c8310485f50b
Author: ChengJie1053 <[email protected]>
AuthorDate: Wed Sep 20 20:33:18 2023 +0800
Modify spark.md (#4875)
* linkis-cli add the engingeConnRuntimeModeOP
---
docs/configuration/spark.md | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/docs/configuration/spark.md b/docs/configuration/spark.md
index f40c76b43..ed070e2ac 100644
--- a/docs/configuration/spark.md
+++ b/docs/configuration/spark.md
@@ -29,7 +29,10 @@
|spark|wds.linkis.spark.engine.scala.replace_package_header.enable| true
|spark.engine.scala.replace_package_header.enable|
Use spark yarn cluster mode,need to set label "engingeConnRuntimeMode":
"yarnCluster",and need to upload the dependence of the spark to
'linkis.spark.yarn.cluster.jar'(the default value is 'hdfs:///spark/cluster')
-spark dependencies include jars and configuration files，For example:
'/appcom/Install/linkis/lib/linkis-engineconn-plugins/spark/dist/3.2.1/lib/*.jar','/appcom/Install/linkis/conf/*''
+spark dependencies include jars and configuration files，For example:
'/appcom/Install/linkis/lib/linkis-engineconn-plugins/spark/dist/3.2.1/lib/*.jar','/appcom/Install/linkis/conf/*'
+
+Precautions for using yarnCluster:
+Eureka url if 127.0.0.1 should be changed to the real host, such as
"127.0.0.1:20303/eureka/" should be changed to "wds001:20303/eureka/"
The spark-excel package may cause class conflicts,need to download
separately,put it in spark lib
wget
https://repo1.maven.org/maven2/com/crealytics/spark-excel-2.12.17-3.2.2_2.12/3.2.2_0.18.1/spark-excel-2.12.17-3.2.2_2.12-3.2.2_0.18.1.jar
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
