yihui8776 commented on issue #2496:
URL:
https://github.com/apache/incubator-seatunnel/issues/2496#issuecomment-1232361405
> you should update seatunnel command, to use `start-seatunnel-spark.sh`. I
think this method will fix it.
here is logļ¼
bash-5.0# ./bin/start-seatunnel-spark.sh --master spark://spark-master:7077
--deploy-mode cluster --config ./config/spark.streaming.conf
[INFO] spark conf: --conf "spark.app.name=SeaTunnel" --conf
"spark.streaming.batchDuration=5" --conf "spark.executor.memory=1g" --conf
"spark.executor.cores=1" --conf "spark.executor.instances=2"
Warning: Ignoring non-Spark config property: "spark.executor.memory
Warning: Ignoring non-Spark config property: "spark.app.name
Warning: Ignoring non-Spark config property: "spark.streaming.batchDuration
Warning: Ignoring non-Spark config property: "spark.executor.instances
Warning: Ignoring non-Spark config property: "spark.executor.cores
22/08/31 01:56:19 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
bash-5.0# ./bin/start-seatunnel-spark.sh --master spark://spark-master:7077
--deploy-mode local --config ./config/spark.streaming.conf
[INFO] spark conf: --conf "spark.app.name=SeaTunnel" --conf
"spark.streaming.batchDuration=5" --conf "spark.executor.memory=1g" --conf
"spark.executor.cores=1" --conf "spark.executor.instances=2"
Exception in thread "main" org.apache.spark.SparkException: --deploy-mode
must be either "client" or "cluster"
at
org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657)
at
org.apache.spark.deploy.SparkSubmitArguments.handle(SparkSubmitArguments.scala:372)
at
org.apache.spark.launcher.SparkSubmitOptionParser.parse(SparkSubmitOptionParser.java:163)
at
org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:109)
at
org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.<init>(SparkSubmit.scala:907)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:907)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
bash-5.0# ./bin/start-seatunnel-spark.sh --master spark://spark-master:7077
--deploy-mode client --config ./config/spark.streaming.conf
[INFO] spark conf: --conf "spark.app.name=SeaTunnel" --conf
"spark.streaming.batchDuration=5" --conf "spark.executor.memory=1g" --conf
"spark.executor.cores=1" --conf "spark.executor.instances=2"
Warning: Ignoring non-Spark config property: "spark.executor.memory
Warning: Ignoring non-Spark config property: "spark.app.name
Warning: Ignoring non-Spark config property: "spark.streaming.batchDuration
Warning: Ignoring non-Spark config property: "spark.executor.instances
Warning: Ignoring non-Spark config property: "spark.executor.cores
22/08/31 01:56:43 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
22/08/31 01:56:44 INFO ConfigBuilder: Loading config file:
./config/spark.streaming.conf
22/08/31 01:56:44 INFO ConfigBuilder: parsed config file: {
"env" : {
"spark.app.name" : "SeaTunnel",
"spark.executor.cores" : 1,
"spark.executor.instances" : 2,
"spark.executor.memory" : "1g",
"spark.streaming.batchDuration" : 5
},
"sink" : [
{
"plugin_name" : "Console"
}
],
"source" : [
{
"content" : [
"Hello World, SeaTunnel"
],
"plugin_name" : "fakeStream"
}
],
"transform" : [
{
"delimiter" : ",",
"fields" : [
"msg",
"name"
],
"plugin_name" : "split"
}
]
}
22/08/31 01:56:44 INFO SparkContext: Running Spark version 2.4.5
22/08/31 01:56:44 INFO SparkContext: Submitted application: SeaTunnel
22/08/31 01:56:44 INFO SecurityManager: Changing view acls to: root
22/08/31 01:56:44 INFO SecurityManager: Changing modify acls to: root
22/08/31 01:56:44 INFO SecurityManager: Changing view acls groups to:
22/08/31 01:56:44 INFO SecurityManager: Changing modify acls groups to:
22/08/31 01:56:44 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root); groups
with view permissions: Set(); users with modify permissions: Set(root); groups
with modify permissions: Set()
22/08/31 01:56:44 INFO Utils: Successfully started service 'sparkDriver' on
port 40839.
22/08/31 01:56:44 INFO SparkEnv: Registering MapOutputTracker
22/08/31 01:56:44 INFO SparkEnv: Registering BlockManagerMaster
22/08/31 01:56:44 INFO BlockManagerMasterEndpoint: Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/08/31 01:56:44 INFO BlockManagerMasterEndpoint:
BlockManagerMasterEndpoint up
22/08/31 01:56:44 INFO DiskBlockManager: Created local directory at
/tmp/blockmgr-d2b31204-f517-461c-a141-a17ae117dab8
22/08/31 01:56:44 INFO MemoryStore: MemoryStore started with capacity 366.3
MB
22/08/31 01:56:44 INFO SparkEnv: Registering OutputCommitCoordinator
22/08/31 01:56:44 INFO Utils: Successfully started service 'SparkUI' on port
4040.
22/08/31 01:56:44 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at
http://9cb7dbbe600c:4040
22/08/31 01:56:45 INFO SparkContext: Added JAR
file:///seatunnel/lib/seatunnel-core-flink.jar at
spark://9cb7dbbe600c:40839/jars/seatunnel-core-flink.jar with timestamp
1661911005002
22/08/31 01:56:45 INFO SparkContext: Added JAR
file:///seatunnel/lib/seatunnel-core-spark.jar at
spark://9cb7dbbe600c:40839/jars/seatunnel-core-spark.jar with timestamp
1661911005004
22/08/31 01:56:45 INFO SparkContext: Added JAR
file:///seatunnel/lib/seatunnel-core-sql-2.1.0-2.11.8.jar at
spark://9cb7dbbe600c:40839/jars/seatunnel-core-sql-2.1.0-2.11.8.jar with
timestamp 1661911005005
22/08/31 01:56:45 WARN SparkContext: The jar
file:/seatunnel/lib/seatunnel-core-spark.jar has been added already.
Overwriting of added jars is not supported in the current version.
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Connecting to
master spark://spark-master:7077...
22/08/31 01:56:45 INFO TransportClientFactory: Successfully created
connection to spark-master/172.24.0.3:7077 after 36 ms (0 ms spent in
bootstraps)
22/08/31 01:56:45 INFO StandaloneSchedulerBackend: Connected to Spark
cluster with app ID app-20220831015645-0002
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor added:
app-20220831015645-0002/0 on worker-20220831014546-172.24.0.2-36195
(172.24.0.2:36195) with 1 core(s)
22/08/31 01:56:45 INFO StandaloneSchedulerBackend: Granted executor ID
app-20220831015645-0002/0 on hostPort 172.24.0.2:36195 with 1 core(s), 1024.0
MB RAM
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor added:
app-20220831015645-0002/1 on worker-20220831014546-172.24.0.2-36195
(172.24.0.2:36195) with 1 core(s)
22/08/31 01:56:45 INFO StandaloneSchedulerBackend: Granted executor ID
app-20220831015645-0002/1 on hostPort 172.24.0.2:36195 with 1 core(s), 1024.0
MB RAM
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor added:
app-20220831015645-0002/2 on worker-20220831014546-172.24.0.2-36195
(172.24.0.2:36195) with 1 core(s)
22/08/31 01:56:45 INFO StandaloneSchedulerBackend: Granted executor ID
app-20220831015645-0002/2 on hostPort 172.24.0.2:36195 with 1 core(s), 1024.0
MB RAM
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor added:
app-20220831015645-0002/3 on worker-20220831014546-172.24.0.2-36195
(172.24.0.2:36195) with 1 core(s)
22/08/31 01:56:45 INFO StandaloneSchedulerBackend: Granted executor ID
app-20220831015645-0002/3 on hostPort 172.24.0.2:36195 with 1 core(s), 1024.0
MB RAM
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor added:
app-20220831015645-0002/4 on worker-20220831014546-172.24.0.2-36195
(172.24.0.2:36195) with 1 core(s)
22/08/31 01:56:45 INFO StandaloneSchedulerBackend: Granted executor ID
app-20220831015645-0002/4 on hostPort 172.24.0.2:36195 with 1 core(s), 1024.0
MB RAM
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor added:
app-20220831015645-0002/5 on worker-20220831014546-172.24.0.2-36195
(172.24.0.2:36195) with 1 core(s)
22/08/31 01:56:45 INFO StandaloneSchedulerBackend: Granted executor ID
app-20220831015645-0002/5 on hostPort 172.24.0.2:36195 with 1 core(s), 1024.0
MB RAM
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor added:
app-20220831015645-0002/6 on worker-20220831014546-172.24.0.2-36195
(172.24.0.2:36195) with 1 core(s)
22/08/31 01:56:45 INFO Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 44033.
22/08/31 01:56:45 INFO StandaloneSchedulerBackend: Granted executor ID
app-20220831015645-0002/6 on hostPort 172.24.0.2:36195 with 1 core(s), 1024.0
MB RAM
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor added:
app-20220831015645-0002/7 on worker-20220831014546-172.24.0.2-36195
(172.24.0.2:36195) with 1 core(s)
22/08/31 01:56:45 INFO NettyBlockTransferService: Server created on
9cb7dbbe600c:44033
22/08/31 01:56:45 INFO StandaloneSchedulerBackend: Granted executor ID
app-20220831015645-0002/7 on hostPort 172.24.0.2:36195 with 1 core(s), 1024.0
MB RAM
22/08/31 01:56:45 INFO BlockManager: Using
org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
policy
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated:
app-20220831015645-0002/0 is now RUNNING
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated:
app-20220831015645-0002/1 is now RUNNING
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated:
app-20220831015645-0002/2 is now RUNNING
22/08/31 01:56:45 INFO BlockManagerMaster: Registering BlockManager
BlockManagerId(driver, 9cb7dbbe600c, 44033, None)
22/08/31 01:56:45 INFO BlockManagerMasterEndpoint: Registering block manager
9cb7dbbe600c:44033 with 366.3 MB RAM, BlockManagerId(driver, 9cb7dbbe600c,
44033, None)
22/08/31 01:56:45 INFO BlockManagerMaster: Registered BlockManager
BlockManagerId(driver, 9cb7dbbe600c, 44033, None)
22/08/31 01:56:45 INFO BlockManager: Initialized BlockManager:
BlockManagerId(driver, 9cb7dbbe600c, 44033, None)
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated:
app-20220831015645-0002/3 is now RUNNING
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated:
app-20220831015645-0002/4 is now RUNNING
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated:
app-20220831015645-0002/5 is now RUNNING
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated:
app-20220831015645-0002/6 is now RUNNING
22/08/31 01:56:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated:
app-20220831015645-0002/7 is now RUNNING
22/08/31 01:56:46 INFO StandaloneSchedulerBackend: SchedulerBackend is ready
for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
22/08/31 01:56:46 ERROR Seatunnel:
===============================================================================
22/08/31 01:56:46 ERROR Seatunnel: Fatal Error,
22/08/31 01:56:46 ERROR Seatunnel: Please submit bug report in
https://github.com/apache/incubator-seatunnel/issues
22/08/31 01:56:46 ERROR Seatunnel: Reason:null
22/08/31 01:56:46 ERROR Seatunnel: Exception
StackTrace:java.lang.NullPointerException
at sun.awt.FontConfiguration.getVersion(FontConfiguration.java:1264)
at
sun.awt.FontConfiguration.readFontConfigFile(FontConfiguration.java:219)
at sun.awt.FontConfiguration.init(FontConfiguration.java:107)
at
sun.awt.X11FontManager.createFontConfiguration(X11FontManager.java:774)
at sun.font.SunFontManager$2.run(SunFontManager.java:431)
at java.security.AccessController.doPrivileged(Native Method)
at sun.font.SunFontManager.<init>(SunFontManager.java:376)
at sun.awt.FcFontManager.<init>(FcFontManager.java:35)
at sun.awt.X11FontManager.<init>(X11FontManager.java:57)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at sun.font.FontManagerFactory$1.run(FontManagerFactory.java:83)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.font.FontManagerFactory.getInstance(FontManagerFactory.java:74)
at java.awt.Font.getFont2D(Font.java:491)
at java.awt.Font.access$000(Font.java:224)
at java.awt.Font$FontAccessImpl.getFont2D(Font.java:228)
at sun.font.FontUtilities.getFont2D(FontUtilities.java:180)
at sun.java2d.SunGraphics2D.checkFontInfo(SunGraphics2D.java:670)
at sun.java2d.SunGraphics2D.getFontInfo(SunGraphics2D.java:831)
at sun.java2d.pipe.GlyphListPipe.drawString(GlyphListPipe.java:50)
at sun.java2d.pipe.ValidatePipe.drawString(ValidatePipe.java:165)
at sun.java2d.SunGraphics2D.drawString(SunGraphics2D.java:2926)
at
org.apache.seatunnel.utils.AsciiArtUtils.printAsciiArt(AsciiArtUtils.java:55)
at org.apache.seatunnel.Seatunnel.showAsciiLogo(Seatunnel.java:168)
at org.apache.seatunnel.Seatunnel.entryPoint(Seatunnel.java:105)
at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:65)
at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
22/08/31 01:56:46 ERROR Seatunnel:
===============================================================================
Exception in thread "main" java.lang.NullPointerException
at sun.awt.FontConfiguration.getVersion(FontConfiguration.java:1264)
at
sun.awt.FontConfiguration.readFontConfigFile(FontConfiguration.java:219)
at sun.awt.FontConfiguration.init(FontConfiguration.java:107)
at
sun.awt.X11FontManager.createFontConfiguration(X11FontManager.java:774)
at sun.font.SunFontManager$2.run(SunFontManager.java:431)
at java.security.AccessController.doPrivileged(Native Method)
at sun.font.SunFontManager.<init>(SunFontManager.java:376)
at sun.awt.FcFontManager.<init>(FcFontManager.java:35)
at sun.awt.X11FontManager.<init>(X11FontManager.java:57)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at sun.font.FontManagerFactory$1.run(FontManagerFactory.java:83)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.font.FontManagerFactory.getInstance(FontManagerFactory.java:74)
at java.awt.Font.getFont2D(Font.java:491)
at java.awt.Font.access$000(Font.java:224)
at java.awt.Font$FontAccessImpl.getFont2D(Font.java:228)
at sun.font.FontUtilities.getFont2D(FontUtilities.java:180)
at sun.java2d.SunGraphics2D.checkFontInfo(SunGraphics2D.java:670)
at sun.java2d.SunGraphics2D.getFontInfo(SunGraphics2D.java:831)
at sun.java2d.pipe.GlyphListPipe.drawString(GlyphListPipe.java:50)
at sun.java2d.pipe.ValidatePipe.drawString(ValidatePipe.java:165)
at sun.java2d.SunGraphics2D.drawString(SunGraphics2D.java:2926)
at
org.apache.seatunnel.utils.AsciiArtUtils.printAsciiArt(AsciiArtUtils.java:55)
at org.apache.seatunnel.Seatunnel.showAsciiLogo(Seatunnel.java:168)
at org.apache.seatunnel.Seatunnel.entryPoint(Seatunnel.java:105)
at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:65)
at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
22/08/31 01:56:46 INFO SparkContext: Invoking stop() from shutdown hook
22/08/31 01:56:46 INFO SparkUI: Stopped Spark web UI at
http://9cb7dbbe600c:4040
22/08/31 01:56:46 INFO StandaloneSchedulerBackend: Shutting down all
executors
22/08/31 01:56:46 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking
each executor to shut down
22/08/31 01:56:46 INFO MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
22/08/31 01:56:46 INFO MemoryStore: MemoryStore cleared
22/08/31 01:56:46 INFO BlockManager: BlockManager stopped
22/08/31 01:56:46 INFO BlockManagerMaster: BlockManagerMaster stopped
22/08/31 01:56:46 INFO
OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
22/08/31 01:56:46 INFO SparkContext: Successfully stopped SparkContext
22/08/31 01:56:46 INFO ShutdownHookManager: Shutdown hook called
22/08/31 01:56:46 INFO ShutdownHookManager: Deleting directory
/tmp/spark-e2255c90-a53f-4e2f-a40c-f789b3a01cf3
22/08/31 01:56:46 INFO ShutdownHookManager: Deleting directory
/tmp/spark-56149588-c099-4c5f-8484-63a2410e6cd0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
