yihui8776 opened a new issue, #2493: URL: https://github.com/apache/incubator-seatunnel/issues/2493
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened I start the docker cluster in docker with seatunnel docker-compose.yml file : version: '3' services: spark-master: image: yihui8776/seatunnel-sparkmaster:v1 container_name: spark-master ports: - "8082:8080" - "7077:7077" environment: - INIT_DAEMON_STEP=setup_spark spark-worker: image: yihui8776/seatunnel-sparkworker:v1 #container_name: spark-worker depends_on: - spark-master ports: - "8081:8081" environment: - "SPARK_MASTER=spark://spark-master:7077" then go in the master container bash-5.0# nc -l -p 9999 hello ,world go into the container in another terminal ring non-Spark config property: "spark.streaming.batchDuration Warning: Ignoring non-Spark config property: "spark.ui.port 22/08/22 07:06:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 22/08/22 07:06:46 INFO SecurityManager: Changing view acls to: root 22/08/22 07:06:46 INFO SecurityManager: Changing modify acls to: root 22/08/22 07:06:46 INFO SecurityManager: Changing view acls groups to: 22/08/22 07:06:46 INFO SecurityManager: Changing modify acls groups to: 22/08/22 07:06:46 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 22/08/22 07:06:47 INFO Utils: Successfully started service 'driverClient' on port 45825. 22/08/22 07:06:47 INFO TransportClientFactory: Successfully created connection to spark-master/172.27.0.2:7077 after 27 ms (0 ms spent in bootstraps) 22/08/22 07:06:47 INFO ClientEndpoint: Driver successfully submitted as driver-20220822070647-0003 22/08/22 07:06:47 INFO ClientEndpoint: ... waiting before polling master for driver state 22/08/22 07:06:52 INFO ClientEndpoint: ... polling master for driver state 22/08/22 07:06:52 INFO ClientEndpoint: State of driver-20220822070647-0003 is FAILED 22/08/22 07:06:52 INFO ShutdownHookManager: Shutdown hook called 22/08/22 07:06:52 INFO ShutdownHookManager: Deleting directory /tmp/spark-fdf61839-803b-4e5a-8ef2-655b9d61bb78 ### SeaTunnel Version 2.1.0 ### SeaTunnel Config ```conf cat config/seatunnel-env.sh #!/usr/bin/env bash # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Home directory of spark distribution. SPARK_HOME=${SPARK_HOME:-/spark} # Home directory of flink distribution. FLINK_HOME=${FLINK_HOME:-/opt/flink} ``` ### Running Command ```shell ./bin/start-seatunnel-spark.sh --master spark://spark-master:7077 --deploy-mode cluster --config ./config/application.conf ``` ### Error Exception ```log ./bin/start-seatunnel-spark.sh --master spark://spark-master:7077 --deploy-mode cluster --config ./config/application.conf [INFO] spark conf: --conf "spark.ui.port=13000" --conf "spark.streaming.batchDuration=5" --conf "spark.app.name=seatunnel" Warning: Ignoring non-Spark config property: "spark.app.name Warning: Ignoring non-Spark config property: "spark.streaming.batchDuration Warning: Ignoring non-Spark config property: "spark.ui.port 22/08/22 07:06:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 22/08/22 07:06:46 INFO SecurityManager: Changing view acls to: root 22/08/22 07:06:46 INFO SecurityManager: Changing modify acls to: root 22/08/22 07:06:46 INFO SecurityManager: Changing view acls groups to: 22/08/22 07:06:46 INFO SecurityManager: Changing modify acls groups to: 22/08/22 07:06:46 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 22/08/22 07:06:47 INFO Utils: Successfully started service 'driverClient' on port 45825. 22/08/22 07:06:47 INFO TransportClientFactory: Successfully created connection to spark-master/172.27.0.2:7077 after 27 ms (0 ms spent in bootstraps) 22/08/22 07:06:47 INFO ClientEndpoint: Driver successfully submitted as driver-20220822070647-0003 22/08/22 07:06:47 INFO ClientEndpoint: ... waiting before polling master for driver state 22/08/22 07:06:52 INFO ClientEndpoint: ... polling master for driver state 22/08/22 07:06:52 INFO ClientEndpoint: State of driver-20220822070647-0003 is FAILED 22/08/22 07:06:52 INFO ShutdownHookManager: Shutdown hook called 22/08/22 07:06:52 INFO ShutdownHookManager: Deleting directory /tmp/spark-fdf61839-803b-4e5a-8ef2-655b9d61bb78 got no message ``` ### Flink or Spark Version spark 2.4.5 ### Java or Scala Version Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_242) ### Screenshots bash-5.0# ./bin/start-seatunnel-spark.sh --master spark://spark-master:7077 --deploy-mode cluster --config ./config/application.conf [INFO] spark conf: --conf "spark.ui.port=13000" --conf "spark.streaming.batchDuration=5" --conf "spark.app.name=seatunnel" Warning: Ignoring non-Spark config property: "spark.app.name Warning: Ignoring non-Spark config property: "spark.streaming.batchDuration Warning: Ignoring non-Spark config property: "spark.ui.port 22/08/22 07:13:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 22/08/22 07:13:24 INFO SecurityManager: Changing view acls to: root 22/08/22 07:13:24 INFO SecurityManager: Changing modify acls to: root 22/08/22 07:13:24 INFO SecurityManager: Changing view acls groups to: 22/08/22 07:13:24 INFO SecurityManager: Changing modify acls groups to: 22/08/22 07:13:24 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 22/08/22 07:13:24 INFO Utils: Successfully started service 'driverClient' on port 40269. 22/08/22 07:13:24 INFO TransportClientFactory: Successfully created connection to spark-master/172.27.0.2:7077 after 30 ms (0 ms spent in bootstraps) 22/08/22 07:13:24 INFO ClientEndpoint: Driver successfully submitted as driver-20220822071324-0004 22/08/22 07:13:24 INFO ClientEndpoint: ... waiting before polling master for driver state 22/08/22 07:13:29 INFO ClientEndpoint: ... polling master for driver state 22/08/22 07:13:29 INFO ClientEndpoint: State of driver-20220822071324-0004 is FAILED 22/08/22 07:13:29 INFO ShutdownHookManager: Shutdown hook called 22/08/22 07:13:29 INFO ShutdownHookManager: Deleting directory /tmp/spark-d649efd4-3d11-4146-8404-7872e638500c ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
