somnmos opened a new issue, #2583: URL: https://github.com/apache/incubator-seatunnel/issues/2583
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened I add seatunnel as maven dependencies to start my service, when I create two kafka-source task, I found only one task works, another one failed. ### SeaTunnel Version V1.2.2 ### SeaTunnel Config ```conf env { spark.app.name = "SeaTunnel" spark.executor.instances = 2 spark.executor.cores = 1 spark.executor.memory = "1g" spark.master = "local" spark.serializer = "org.apache.spark.serializer.KryoSerializer" spark.yarn.archive = "hdfs:///app-logs/spark-2.4.4-1.0.2.zip" spark.proxy-user = "spark" spark.driver.allowMultipleContexts = true } source { KafkaStream { topics = "test1" consumer.bootstrap.servers = "xxx.xxx.xxx.xxx:9092" consumer.group.id = "consumer0001" result_table_name = "test" } } transform { Sql { sql = "SELECT * FROM test" } } sink { Console { } } ``` ### Running Command ```shell import org.apache.seatunnel.common.config.DeployMode; import org.apache.seatunnel.core.base.Seatunnel; import org.apache.seatunnel.core.base.command.Command; import org.apache.seatunnel.core.spark.args.SparkCommandArgs; import org.apache.seatunnel.core.spark.command.SparkCommandBuilder; import java.io.FileNotFoundException; import java.net.URISyntaxException; import java.net.URL; import java.nio.file.Paths; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; public class SparkEngineExample { static class SparkTask implements Runnable { private String confFile; public SparkTask(String confFile) { this.confFile = confFile; } public String getTestConfigFile(String configFile) throws URISyntaxException, FileNotFoundException { URL resource = SparkEngineExample.class.getResource(configFile); if (resource == null) { throw new FileNotFoundException("Could not find config file: " + configFile); } return Paths.get(resource.toURI()).toString(); } @Override public void run() { String configFile = null; try { configFile = getTestConfigFile(this.confFile); } catch (URISyntaxException e) { e.printStackTrace(); } catch (FileNotFoundException e) { e.printStackTrace(); } SparkCommandArgs sparkArgs = new SparkCommandArgs(); sparkArgs.setConfigFile(configFile); sparkArgs.setCheckConfig(false); sparkArgs.setVariables(null); sparkArgs.setDeployMode(DeployMode.CLIENT); Command<SparkCommandArgs> sparkCommand = new SparkCommandBuilder().buildCommand(sparkArgs); Seatunnel.run(sparkCommand); } } public static void main(String[] args) { ExecutorService service = Executors.newCachedThreadPool(); service.execute(new SparkTask("/examples/test1.conf")); service.execute(new SparkTask("/examples/test2.conf")); service.shutdown(); } } ``` ### Error Exception ```log 22/08/31 18:53:41 ERROR Seatunnel: =============================================================================== 22/08/31 18:53:41 ERROR Seatunnel: Fatal Error, 22/08/31 18:53:41 ERROR Seatunnel: Please submit bug report in https://github.com/apache/incubator-seatunnel/issues 22/08/31 18:53:41 ERROR Seatunnel: Reason:Execute Spark task error 22/08/31 18:53:41 ERROR Seatunnel: Exception StackTrace:java.lang.RuntimeException: Execute Spark task error at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:79) at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:39) at com.somnus.SparkEngineExample$SparkTask.run(SparkEngineExample.java:67) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalStateException: Only one StreamingContext may be started in this JVM. Currently running StreamingContext was started atorg.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:569) org.apache.seatunnel.spark.stream.SparkStreamingExecution.start(SparkStreamingExecution.scala:68) org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:76) org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:39) com.somnus.SparkEngineExample$SparkTask.run(SparkEngineExample.java:67) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) at org.apache.spark.streaming.StreamingContext$.org$apache$spark$streaming$StreamingContext$$assertNoOtherContextIsActive(StreamingContext.scala:738) at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:571) at org.apache.seatunnel.spark.stream.SparkStreamingExecution.start(SparkStreamingExecution.scala:68) at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:76) ... 5 more 22/08/31 18:53:41 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@16c74877{/streaming,null,AVAILABLE,@Spark} ``` ### Flink or Spark Version Spark2.4.4 ### Java or Scala Version Scala2.11.12 ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
