BenJFan opened a new issue, #1771: URL: https://github.com/apache/incubator-seatunnel/issues/1771
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened When use spark standalone cluster to submit job, and use cluster mode. Driver can't read config file. Maybe we should use `SparkFile` api to get file path. @whb-bigdata I find you fix the same issue on #1747 . Do you have any suggestion? ### SeaTunnel Version dev ### SeaTunnel Config ```conf - ``` ### Running Command ```shell ./bin/start-seatunnel-spark.sh --config config/spark.batch.conf -m spark://localhost:7077 -e cluster ``` ### Error Exception ```log Launch Command: "/usr/lib/jvm/java-17-amazon-corretto.x86_64/bin/java" "-cp" "/home/hadoop/spark/conf/:/home/hadoop/spark/jars/*:/home/hadoop/hadoop/etc/hadoop/" "-Xmx1024M" "-Dspark.executor.memory=512m" "-Dspark.driver.supervise=false" "-Dspark.submit.deployMode=cluster" "-Dspark.executor.instances=2" "-Dspark.files=file:///home/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/plugins.tar.gz,file:///home/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/config/spark.batch.conf" "-Dspark.master=spark://172.31.0.76:7077" "-Dspark.jars=file:///home/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/connectors/spark/seatunnel-connector-spark-fake-2.1.1-SNAPSHOT.jar,file:///home/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/connectors/spark/seatunnel-connector-spark-console-2.1.1-SNAPSHOT.jar,file:/home/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/lib/seatunnel-core-spark.jar" "-Dspark.app.name=SeaTunnelGhy" "-Dspark.executor.cores=1" "-Dspark.rpc.askTimeout=1 0s" "org.apache.spark.deploy.worker.DriverWrapper" "spark://[email protected]:41397" "/home/hadoop/spark/work/driver-20220428082338-0000/seatunnel-core-spark.jar" "org.apache.seatunnel.SeatunnelSpark" "--config" "config/spark.batch.conf" "-m" "spark://172.31.0.76:7077" "-e" "cluster" ======================================== 22/04/28 08:23:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 22/04/28 08:23:40 INFO spark.SecurityManager: Changing view acls to: root 22/04/28 08:23:40 INFO spark.SecurityManager: Changing modify acls to: root 22/04/28 08:23:40 INFO spark.SecurityManager: Changing view acls groups to: 22/04/28 08:23:40 INFO spark.SecurityManager: Changing modify acls groups to: 22/04/28 08:23:40 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 22/04/28 08:23:40 INFO util.Utils: Successfully started service 'Driver' on port 44727. 22/04/28 08:23:40 INFO worker.DriverWrapper: Driver address: 172.31.0.76:44727 22/04/28 08:23:40 INFO worker.WorkerWatcher: Connecting to worker spark://[email protected]:41397 22/04/28 08:23:40 INFO spark.SecurityManager: Changing view acls to: root 22/04/28 08:23:40 INFO spark.SecurityManager: Changing modify acls to: root 22/04/28 08:23:40 INFO spark.SecurityManager: Changing view acls groups to: 22/04/28 08:23:40 INFO spark.SecurityManager: Changing modify acls groups to: 22/04/28 08:23:40 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 22/04/28 08:23:40 INFO client.TransportClientFactory: Successfully created connection to /172.31.0.76:41397 after 30 ms (0 ms spent in bootstraps) 22/04/28 08:23:40 INFO worker.WorkerWatcher: Successfully connected to spark://[email protected]:41397 22/04/28 08:23:41 INFO config.ConfigBuilder: Loading config file: spark.batch.conf 22/04/28 08:23:41 INFO config.ConfigBuilder: parsed config file: {} 22/04/28 08:23:41 ERROR seatunnel.Seatunnel: =============================================================================== 22/04/28 08:23:41 ERROR seatunnel.Seatunnel: Fatal Error, 22/04/28 08:23:41 ERROR seatunnel.Seatunnel: Please submit bug report in https://github.com/apache/incubator-seatunnel/issues 22/04/28 08:23:41 ERROR seatunnel.Seatunnel: Reason:No configuration setting found for key 'env' 22/04/28 08:23:41 ERROR seatunnel.Seatunnel: Exception StackTrace:org.apache.seatunnel.shade.com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'env' at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:156) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:188) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:193) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41) at org.apache.seatunnel.config.EnvironmentFactory.getEnvironment(EnvironmentFactory.java:47) at org.apache.seatunnel.config.ExecutionContext.<init>(ExecutionContext.java:49) at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:46) at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:38) at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:48) at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:27) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65) at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) 22/04/28 08:23:41 ERROR seatunnel.Seatunnel: =============================================================================== Exception in thread "main" java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65) at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) Caused by: org.apache.seatunnel.shade.com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'env' at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:156) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:188) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:193) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274) at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41) at org.apache.seatunnel.config.EnvironmentFactory.getEnvironment(EnvironmentFactory.java:47) at org.apache.seatunnel.config.ExecutionContext.<init>(ExecutionContext.java:49) at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:46) at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:38) at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:48) at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:27) ... 6 more 22/04/28 08:23:41 INFO util.ShutdownHookManager: Shutdown hook called 22/04/28 08:23:41 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-b3013229-90c1-43d5-999b-adff6ef7652e ``` ### Flink or Spark Version spark 2.4.0 ### Java or Scala Version _No response_ ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
