yihui8776 commented on issue #2496:
URL:
https://github.com/apache/incubator-seatunnel/issues/2496#issuecomment-1225028418
I update the docker compose and start up but got another exception
docker-compose up
Starting spark_datasource_1 ... done
Starting spark_seatunnel_1 ... done
Attaching to spark_datasource_1, spark_seatunnel_1
seatunnel_1 | Execute SeaTunnel Spark Job: ${SPARK_HOME}/bin/spark-submit
--class "org.apache.seatunnel.core.spark.SeatunnelSpark" --name "seatunnel"
--master "local[4]" --deploy-mode "client" --conf
"spark.streaming.batchDuration=5" --conf "spark.app.name=seatunnel" --conf
"spark.ui.port=13000" /seatunnel/lib/seatunnel-core-spark.jar --master local[4]
--deploy-mode client --config /application.conf -t
spark_datasource_1 exited with code 0
seatunnel_1 | 2022-08-24 00:32:43 WARN NativeCodeLoader:62 - Unable to
load native-hadoop library for your platform... using builtin-java classes
where applicable
seatunnel_1 | 2022-08-24 00:32:43 INFO ConfigBuilder:54 - Loading config
file: /application.conf
seatunnel_1 | 2022-08-24 00:32:43 INFO ConfigBuilder:65 - parsed config
file: {
seatunnel_1 | "env" : {
seatunnel_1 | "spark.streaming.batchDuration" : 5,
seatunnel_1 | "spark.app.name" : "seatunnel",
seatunnel_1 | "spark.ui.port" : 13000
seatunnel_1 | },
seatunnel_1 | "source" : [
seatunnel_1 | {
seatunnel_1 | "path" : "/tmp/source",
seatunnel_1 | "format" : "text",
seatunnel_1 | "result_table_name" : "test",
seatunnel_1 | "plugin_name" : "file"
seatunnel_1 | }
seatunnel_1 | ],
seatunnel_1 | "transform" : [
seatunnel_1 | {
seatunnel_1 | "delimiter" : ",",
seatunnel_1 | "fields" : [
seatunnel_1 | "msg",
seatunnel_1 | "name"
seatunnel_1 | ],
seatunnel_1 | "plugin_name" : "split"
seatunnel_1 | }
seatunnel_1 | ],
seatunnel_1 | "sink" : [
seatunnel_1 | {
seatunnel_1 | "plugin_name" : "console"
seatunnel_1 | }
seatunnel_1 | ]
seatunnel_1 | }
seatunnel_1 |
seatunnel_1 | 2022-08-24 00:32:43 INFO SparkContext:54 - Running Spark
version 2.4.0
seatunnel_1 | 2022-08-24 00:32:43 INFO SparkContext:54 - Submitted
application: seatunnel
seatunnel_1 | 2022-08-24 00:32:43 INFO SecurityManager:54 - Changing view
acls to: root
seatunnel_1 | 2022-08-24 00:32:43 INFO SecurityManager:54 - Changing
modify acls to: root
seatunnel_1 | 2022-08-24 00:32:43 INFO SecurityManager:54 - Changing view
acls groups to:
seatunnel_1 | 2022-08-24 00:32:43 INFO SecurityManager:54 - Changing
modify acls groups to:
seatunnel_1 | 2022-08-24 00:32:43 INFO SecurityManager:54 -
SecurityManager: authentication disabled; ui acls disabled; users with view
permissions: Set(root); groups with view permissions: Set(); users with modify
permissions: Set(root); groups with modify permissions: Set()
seatunnel_1 | 2022-08-24 00:32:43 INFO Utils:54 - Successfully started
service 'sparkDriver' on port 39869.
seatunnel_1 | 2022-08-24 00:32:43 INFO SparkEnv:54 - Registering
MapOutputTracker
seatunnel_1 | 2022-08-24 00:32:43 INFO SparkEnv:54 - Registering
BlockManagerMaster
seatunnel_1 | 2022-08-24 00:32:43 INFO BlockManagerMasterEndpoint:54 -
Using org.apache.spark.storage.DefaultTopologyMapper for getting topology
information
seatunnel_1 | 2022-08-24 00:32:43 INFO BlockManagerMasterEndpoint:54 -
BlockManagerMasterEndpoint up
seatunnel_1 | 2022-08-24 00:32:43 INFO DiskBlockManager:54 - Created
local directory at /tmp/blockmgr-d8ef9afb-34d9-4def-9242-8062271d871b
seatunnel_1 | 2022-08-24 00:32:44 INFO MemoryStore:54 - MemoryStore
started with capacity 366.3 MB
seatunnel_1 | 2022-08-24 00:32:44 INFO SparkEnv:54 - Registering
OutputCommitCoordinator
seatunnel_1 | 2022-08-24 00:32:44 INFO log:192 - Logging initialized
@1402ms
seatunnel_1 | 2022-08-24 00:32:44 INFO Server:351 - jetty-9.3.z-SNAPSHOT,
build timestamp: unknown, git hash: unknown
seatunnel_1 | 2022-08-24 00:32:44 INFO Server:419 - Started @1461ms
seatunnel_1 | 2022-08-24 00:32:44 INFO AbstractConnector:278 - Started
ServerConnector@64d43929{HTTP/1.1,[http/1.1]}{0.0.0.0:13000}
seatunnel_1 | 2022-08-24 00:32:44 INFO Utils:54 - Successfully started
service 'SparkUI' on port 13000.
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@22d6cac2{/jobs,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@2fa3be26{/jobs/json,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@4287d447{/jobs/job,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@4e6d7365{/jobs/job/json,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@7c0da600{/stages,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@d4602a{/stages/json,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@21ae6e73{/stages/stage,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@36a7abe1{/stages/stage/json,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@64a896b0{/stages/pool,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@e044b4a{/stages/pool/json,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@11a82d0f{/storage,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@1adb7478{/storage/json,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@3ae66c85{/storage/rdd,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@16943e88{/storage/rdd/json,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@4604b900{/environment,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@73d6d0c{/environment/json,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@e36bb2a{/executors,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@3961a41a{/executors/json,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@5a4ed68f{/executors/threadDump,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@367795c7{/executors/threadDump/json,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@d2387c8{/static,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@538cd0f2{/,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@238ad8c{/api,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@654c1a54{/jobs/job/kill,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@5bdaf2ce{/stages/stage/kill,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO SparkUI:54 - Bound SparkUI to
0.0.0.0, and started at http://bb8919f4e108:13000
seatunnel_1 | 2022-08-24 00:32:44 INFO SparkContext:54 - Added JAR
file:/seatunnel/lib/seatunnel-core-spark.jar at
spark://bb8919f4e108:39869/jars/seatunnel-core-spark.jar with timestamp
1661301164210
seatunnel_1 | 2022-08-24 00:32:44 INFO Executor:54 - Starting executor ID
driver on host localhost
seatunnel_1 | 2022-08-24 00:32:44 INFO Utils:54 - Successfully started
service 'org.apache.spark.network.netty.NettyBlockTransferService' on port
40693.
seatunnel_1 | 2022-08-24 00:32:44 INFO NettyBlockTransferService:54 -
Server created on bb8919f4e108:40693
seatunnel_1 | 2022-08-24 00:32:44 INFO BlockManager:54 - Using
org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
policy
seatunnel_1 | 2022-08-24 00:32:44 INFO BlockManagerMaster:54 -
Registering BlockManager BlockManagerId(driver, bb8919f4e108, 40693, None)
seatunnel_1 | 2022-08-24 00:32:44 INFO BlockManagerMasterEndpoint:54 -
Registering block manager bb8919f4e108:40693 with 366.3 MB RAM,
BlockManagerId(driver, bb8919f4e108, 40693, None)
seatunnel_1 | 2022-08-24 00:32:44 INFO BlockManagerMaster:54 - Registered
BlockManager BlockManagerId(driver, bb8919f4e108, 40693, None)
seatunnel_1 | 2022-08-24 00:32:44 INFO BlockManager:54 - Initialized
BlockManager: BlockManagerId(driver, bb8919f4e108, 40693, None)
seatunnel_1 | 2022-08-24 00:32:44 INFO ContextHandler:781 - Started
o.s.j.s.ServletContextHandler@35ff8fc9{/metrics/json,null,AVAILABLE,@Spark}
seatunnel_1 | 2022-08-24 00:32:44 INFO AbstractPluginDiscovery:77 - Load
BaseSparkSource Plugin from /seatunnel/connectors/spark
seatunnel_1 | 2022-08-24 00:32:44 INFO AbstractPluginDiscovery:77 - Load
BaseSparkTransform Plugin from /seatunnel/connectors/spark
seatunnel_1 | 2022-08-24 00:32:44 INFO AbstractPluginDiscovery:77 - Load
BaseSparkSink Plugin from /seatunnel/connectors/spark
seatunnel_1 | 2022-08-24 00:32:44 INFO SparkEnvironment:93 - register
plugins :[]
seatunnel_1 | 2022-08-24 00:32:44 ERROR Seatunnel:61 -
seatunnel_1 |
seatunnel_1 |
===============================================================================
seatunnel_1 |
seatunnel_1 |
seatunnel_1 | 2022-08-24 00:32:44 ERROR Seatunnel:64 - Fatal Error,
seatunnel_1 |
seatunnel_1 | 2022-08-24 00:32:44 ERROR Seatunnel:66 - Please submit bug
report in https://github.com/apache/incubator-seatunnel/issues
seatunnel_1 |
seatunnel_1 | 2022-08-24 00:32:44 ERROR Seatunnel:68 - Reason:Plugin
PluginIdentifier{engineType='spark', pluginType='source', pluginName='file'}
not found.
seatunnel_1 |
seatunnel_1 | 2022-08-24 00:32:44 ERROR Seatunnel:69 - Exception
StackTrace:java.lang.RuntimeException: Plugin
PluginIdentifier{engineType='spark', pluginType='source', pluginName='file'}
not found.
seatunnel_1 | at
org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.createPluginInstance(AbstractPluginDiscovery.java:122)
seatunnel_1 | at
org.apache.seatunnel.core.spark.config.SparkExecutionContext.lambda$getSources$0(SparkExecutionContext.java:67)
seatunnel_1 | at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
seatunnel_1 | at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
seatunnel_1 | at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
seatunnel_1 | at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
seatunnel_1 | at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
seatunnel_1 | at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
seatunnel_1 | at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
seatunnel_1 | at
org.apache.seatunnel.core.spark.config.SparkExecutionContext.getSources(SparkExecutionContext.java:70)
seatunnel_1 | at
org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:55)
seatunnel_1 | at
org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:40)
seatunnel_1 | at
org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:33)
seatunnel_1 | at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
seatunnel_1 | at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
seatunnel_1 | at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
seatunnel_1 | at java.lang.reflect.Method.invoke(Method.java:498)
seatunnel_1 | at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
seatunnel_1 | at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
seatunnel_1 | at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
seatunnel_1 | at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
seatunnel_1 | at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
seatunnel_1 | at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
seatunnel_1 | at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
seatunnel_1 | at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
seatunnel_1 |
seatunnel_1 | 2022-08-24 00:32:44 ERROR Seatunnel:70 -
seatunnel_1 |
===============================================================================
seatunnel_1 |
seatunnel_1 |
seatunnel_1 |
seatunnel_1 | Exception in thread "main" java.lang.RuntimeException:
Plugin PluginIdentifier{engineType='spark', pluginType='source',
pluginName='file'} not found.
seatunnel_1 | at
org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.createPluginInstance(AbstractPluginDiscovery.java:122)
seatunnel_1 | at
org.apache.seatunnel.core.spark.config.SparkExecutionContext.lambda$getSources$0(SparkExecutionContext.java:67)
seatunnel_1 | at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
seatunnel_1 | at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
seatunnel_1 | at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
seatunnel_1 | at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
seatunnel_1 | at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
seatunnel_1 | at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
seatunnel_1 | at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
seatunnel_1 | at
org.apache.seatunnel.core.spark.config.SparkExecutionContext.getSources(SparkExecutionContext.java:70)
seatunnel_1 | at
org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:55)
seatunnel_1 | at
org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:40)
seatunnel_1 | at
org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:33)
seatunnel_1 | at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
seatunnel_1 | at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
seatunnel_1 | at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
seatunnel_1 | at java.lang.reflect.Method.invoke(Method.java:498)
seatunnel_1 | at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
seatunnel_1 | at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
seatunnel_1 | at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
seatunnel_1 | at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
seatunnel_1 | at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
seatunnel_1 | at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
seatunnel_1 | at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
seatunnel_1 | at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
seatunnel_1 | 2022-08-24 00:32:44 INFO SparkContext:54 - Invoking stop()
from shutdown hook
seatunnel_1 | 2022-08-24 00:32:44 INFO AbstractConnector:318 - Stopped
Spark@64d43929{HTTP/1.1,[http/1.1]}{0.0.0.0:13000}
seatunnel_1 | 2022-08-24 00:32:44 INFO SparkUI:54 - Stopped Spark web UI
at http://bb8919f4e108:13000
seatunnel_1 | 2022-08-24 00:32:44 INFO MapOutputTrackerMasterEndpoint:54
- MapOutputTrackerMasterEndpoint stopped!
seatunnel_1 | 2022-08-24 00:32:44 INFO MemoryStore:54 - MemoryStore
cleared
seatunnel_1 | 2022-08-24 00:32:44 INFO BlockManager:54 - BlockManager
stopped
seatunnel_1 | 2022-08-24 00:32:44 INFO BlockManagerMaster:54 -
BlockManagerMaster stopped
seatunnel_1 | 2022-08-24 00:32:44 INFO
OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 -
OutputCommitCoordinator stopped!
seatunnel_1 | 2022-08-24 00:32:44 INFO SparkContext:54 - Successfully
stopped SparkContext
seatunnel_1 | 2022-08-24 00:32:44 INFO ShutdownHookManager:54 - Shutdown
hook called
seatunnel_1 | 2022-08-24 00:32:44 INFO ShutdownHookManager:54 - Deleting
directory /tmp/spark-37613600-1da8-4fc4-9485-a8b32d082e43
seatunnel_1 | 2022-08-24 00:32:44 INFO ShutdownHookManager:54 - Deleting
directory /tmp/spark-8aa66517-49d0-4537-b32a-51a2ba5ef41a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
