[ https://issues.apache.org/jira/browse/SPARK-33199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Farhan Khan updated SPARK-33199: -------------------------------- Description: It is a bug related to shell escaping in MesosClusterScheduler. As you can see in this issue SPARK-23941, we are escaping all the options, that lead to escaping of the *$MESOS_SANDBOX* used in pyFiles in case of docker image option as well. This in turn causes *FileNotFoundErrors* during *py4j.SparkSession.getOrCreate*. In order to reproduce the bug, you need to have mesos and mesos dispatcher running. When you launch the following program: {code:bash} spark-submit --master mesos://127.0.1.1:7077 --deploy-mode cluster --class my_python_program.py --py-files /path/to/PySparkInclude.py --name my_python_program --conf spark.mesos.executor.docker.image=myrepo/myimage:mytag /path/to/my_python_program.py 100 {code} then the task failed with the following output: {code:bash} I1020 16:26:11.958860 17736 fetcher.cpp:562] Fetcher Info: {"cache_directory":"/tmp/mesos/fetch/nobody","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"/path/to/my_python_program.py"}},{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"/path/to/PySparkTestInclude.py"}}],"sandbox_directory":"/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375","stall_timeout":{"nanoseconds":60000000000},"user":"nobody"} I1020 16:26:11.962540 17736 fetcher.cpp:459] Fetching URI '/path/to/my_python_program.py' I1020 16:26:11.962558 17736 fetcher.cpp:290] Fetching '/path/to/my_python_program.py' directly into the sandbox directory I1020 16:26:11.962584 17736 fetcher.cpp:175] Downloading resource from '/path/to/my_python_program.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/my_python_program.py' W1020 16:26:12.126960 17736 fetcher.cpp:332] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: /path/to/my_python_program.py I1020 16:26:12.126986 17736 fetcher.cpp:618] Fetched '/path/to/my_python_program.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/my_python_program.py' I1020 16:26:12.126991 17736 fetcher.cpp:459] Fetching URI '/path/to/PySparkTestInclude.py' I1020 16:26:12.126996 17736 fetcher.cpp:290] Fetching '/path/to/PySparkTestInclude.py' directly into the sandbox directory I1020 16:26:12.127013 17736 fetcher.cpp:175] Downloading resource from '/path/to/PySparkTestInclude.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/PySparkTestInclude.py' W1020 16:26:12.283085 17736 fetcher.cpp:332] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: /path/to/PySparkTestInclude.py I1020 16:26:12.283108 17736 fetcher.cpp:618] Fetched '/path/to/PySparkTestInclude.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/PySparkTestInclude.py' I1020 16:26:12.283114 17736 fetcher.cpp:623] Successfully fetched all URIs into '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375' Marked '/' as rslave W1020 16:26:12.389969 2 openssl.cpp:553] Failed SSL connections will be downgraded to a non-SSL socket W1020 16:26:12.390067 2 openssl.cpp:575] TLS certificate verification was enabled by setting one of LIBPROCESS_SSL_VERIFY_CERT or LIBPROCESS_SSL_REQUIRE_CERT, but can be bypassed because TLS downgrades are enabled. I1020 16:26:12.390079 2 openssl.cpp:599] CA directory path unspecified! NOTE: Set CA directory path with LIBPROCESS_SSL_CA_DIR=<dirpath> I1020 16:26:12.390087 2 openssl.cpp:623] Will use IP address verification in subject alternative name certificate extension. I1020 16:26:12.390095 2 openssl.cpp:642] Will verify server certificates for outgoing TLS connections. I1020 16:26:12.390105 2 openssl.cpp:663] Using 'openssl' scheme for hostname validation I1020 16:26:12.390331 2 openssl.cpp:688] Using CA file: /var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/.ssl/ca-bundle.crt W1020 16:26:12.390651 2 process.cpp:1055] Failed SSL connections will be downgraded to a non-SSL socket I1020 16:26:12.392330 2 executor.cpp:210] Version: 1.10.1 I1020 16:26:12.392472 2 executor.cpp:286] Using URL http+unix://%2Fvar%2Frun%2Fmesos%2Fmesos-executors.sock/slave(1)/api/v1/executor for the executor API endpoint I1020 16:26:12.398231 8 executor.cpp:190] Received SUBSCRIBED event I1020 16:26:12.398533 8 executor.cpp:194] Subscribed executor on 172.16.72.91 I1020 16:26:12.398721 8 executor.cpp:190] Received LAUNCH event I1020 16:26:12.399922 8 executor.cpp:671] Overwriting environment variable 'LIBPROCESS_SSL_CA_FILE' I1020 16:26:12.399947 8 executor.cpp:671] Overwriting environment variable 'LIBPROCESS_SSL_CERT_FILE' I1020 16:26:12.399964 8 executor.cpp:671] Overwriting environment variable 'MESOS_NATIVE_JAVA_LIBRARY' I1020 16:26:12.399986 8 executor.cpp:671] Overwriting environment variable 'LIBPROCESS_SSL_KEY_FILE' I1020 16:26:12.400000 8 executor.cpp:671] Overwriting environment variable 'PATH' I1020 16:26:12.400053 8 executor.cpp:722] Starting task driver-20201020162611-0012 I1020 16:26:12.401970 8 executor.cpp:740] Forked command at 12 I1020 16:26:12.403370 9 executor.cpp:190] Received ACKNOWLEDGED event I1020 16:26:12.404062 9 executor.cpp:190] Received ACKNOWLEDGED event Preparing rootfs at '/var/lib/mesos/slave/provisioner/containers/f98449ad-ecba-4bc9-a979-99f21f227375/backends/overlay/rootfses/f80c3e69-73e8-4154-a8e5-8c68613f6462' Marked '/' as rslave Changing root to /var/lib/mesos/slave/provisioner/containers/f98449ad-ecba-4bc9-a979-99f21f227375/backends/overlay/rootfses/f80c3e69-73e8-4154-a8e5-8c68613f6462 spark-env: Configured SPARK_LOCAL_IP with bootstrap: 172.16.72.91 spark-env: User: nobody spark-env: No SPARK_MESOS_KRB5_CONF_BASE64 to decode spark-env: Configuring StatsD endpoint 198.51.100.1:46584 in /opt/spark/conf/metrics.properties 20/10/20 16:26:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/10/20 16:26:17 INFO SparkContext: Running Spark version 3.0.1 20/10/20 16:26:17 INFO ResourceUtils: ============================================================== 20/10/20 16:26:17 INFO ResourceUtils: Resources for spark.driver: 20/10/20 16:26:17 INFO ResourceUtils: ============================================================== 20/10/20 16:26:17 INFO SparkContext: Submitted application: PythonPi 20/10/20 16:26:17 INFO SecurityManager: Changing view acls to: nobody 20/10/20 16:26:17 INFO SecurityManager: Changing modify acls to: nobody 20/10/20 16:26:17 INFO SecurityManager: Changing view acls groups to: 20/10/20 16:26:17 INFO SecurityManager: Changing modify acls groups to: 20/10/20 16:26:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(nobody); groups with view permissions: Set(); users with modify permissions: Set(nobody); groups with modify permissions: Set() 20/10/20 16:26:17 INFO Utils: Successfully started service 'sparkDriver' on port 37759. 20/10/20 16:26:17 INFO SparkEnv: Registering MapOutputTracker 20/10/20 16:26:18 INFO SparkEnv: Registering BlockManagerMaster 20/10/20 16:26:18 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 20/10/20 16:26:18 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 20/10/20 16:26:18 INFO SparkEnv: Registering BlockManagerMasterHeartbeat 20/10/20 16:26:18 INFO DiskBlockManager: Created local directory at /mnt/mesos/sandbox/blockmgr-2be3fdd6-db33-49e8-bc0a-5b1da5ec8ded 20/10/20 16:26:18 INFO MemoryStore: MemoryStore started with capacity 912.3 MiB 20/10/20 16:26:18 INFO MetricsConfig: Loading metrics properties from resource metrics.properties 20/10/20 16:26:18 INFO MetricsConfig: Metrics properties: {*.sink.statsd.class=org.apache.spark.metrics.sink.statsd.StatsdSink, *.sink.statsd.host=198.51.100.1, *.sink.servlet.class=org.apache.spark.metrics.sink.MetricsServlet, applications.sink.servlet.path=/metrics/applications/json, *.sink.statsd.tags=, *.source.jvm.class=org.apache.spark.metrics.source.JvmSource, *.sink.statsd.port=46584, master.sink.servlet.path=/metrics/master/json, *.sink.statsd.prefix=, *.sink.servlet.path=/metrics/json} 20/10/20 16:26:18 INFO SparkEnv: Registering OutputCommitCoordinator 20/10/20 16:26:18 INFO log: Logging initialized @5603ms to org.sparkproject.jetty.util.log.Slf4jLog 20/10/20 16:26:18 INFO Server: jetty-9.4.z-SNAPSHOT; built: 2019-04-29T20:42:08.989Z; git: e1bc35120a6617ee3df052294e433f3a25ce7097; jvm 1.8.0_212-b03 20/10/20 16:26:18 INFO Server: Started @5872ms 20/10/20 16:26:18 INFO AbstractConnector: Started ServerConnector@1345a271{HTTP/1.1,[http/1.1]}{172.16.72.91:4040} 20/10/20 16:26:18 INFO Utils: Successfully started service 'SparkUI' on port 4040. 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@37925df7{/jobs,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@31660e32{/jobs/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@68c3022c{/jobs/job,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@9662252{/jobs/job/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@75e43d99{/stages,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@74c915ac{/stages/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7d15bd53{/stages/stage,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@10d10bcf{/stages/stage/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6ce43fb0{/stages/pool,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@42f60f93{/stages/pool/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@228a2392{/storage,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@50576ab{/storage/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6de665b4{/storage/rdd,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5fb35a9d{/storage/rdd/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5bb2641{/environment,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5f7aae9d{/environment/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@29b1337f{/executors,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7af3cf55{/executors/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@b196f0b{/executors/threadDump,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2383aabe{/executors/threadDump/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@545844d3{/static,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1adf4e59{/,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2ae8fb6c{/api,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@21e2f70e{/jobs/job/kill,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6199fb32{/stages/stage/kill,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO SparkUI: Bound SparkUI to 172.16.72.91, and started at http://172.16.72.91:4040 20/10/20 16:26:18 ERROR SparkContext: Error initializing SparkContext. java.io.FileNotFoundException: File file:/mnt/mesos/sandbox/$MESOS_SANDBOX/PySparkTestInclude.py does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:666) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:987) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:656) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454) at org.apache.spark.SparkContext.addFile(SparkContext.scala:1534) at org.apache.spark.SparkContext.addFile(SparkContext.scala:1498) at org.apache.spark.SparkContext.$anonfun$new$12(SparkContext.scala:494) at org.apache.spark.SparkContext.$anonfun$new$12$adapted(SparkContext.scala:494) at scala.collection.immutable.List.foreach(List.scala:392) at org.apache.spark.SparkContext.<init>(SparkContext.scala:494) at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:238) at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) 20/10/20 16:26:18 INFO AbstractConnector: Stopped Spark@1345a271{HTTP/1.1,[http/1.1]}{172.16.72.91:4040} 20/10/20 16:26:18 INFO SparkUI: Stopped Spark web UI at http://172.16.72.91:4040 20/10/20 16:26:19 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 20/10/20 16:26:19 INFO MemoryStore: MemoryStore cleared 20/10/20 16:26:19 INFO BlockManager: BlockManager stopped 20/10/20 16:26:19 INFO BlockManagerMaster: BlockManagerMaster stopped 20/10/20 16:26:19 WARN MetricsSystem: Stopping a MetricsSystem that is not running 20/10/20 16:26:19 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 20/10/20 16:26:19 INFO SparkContext: Successfully stopped SparkContext 20/10/20 16:26:19 INFO ShutdownHookManager: Shutdown hook called 20/10/20 16:26:19 INFO ShutdownHookManager: Shutdown hook called 20/10/20 16:26:19 INFO ShutdownHookManager: Deleting directory /tmp/spark-07db5724-6031-45d3-8222-c06947f80872 20/10/20 16:26:19 INFO ShutdownHookManager: Deleting directory /mnt/mesos/sandbox/spark-d4145378-f0da-4f1b-8aee-a72d427ccb8f I1020 16:26:19.363329 3 executor.cpp:1041] Command exited with status 1 (pid: 12) I1020 16:26:19.366904 4 executor.cpp:190] Received ACKNOWLEDGED event I1020 16:26:19.367244 2 process.cpp:935] Stopped the socket accept loop {code} was: It is a bug related to shell escaping in MesosClusterScheduler. As you can see in this issue SPARK-23941, we are escaping all the options, that lead to escaping of the *$MESOS_SANDBOX* used in pyFiles in case of docker image option as well. This in turn causes *FileNotFoundErrors* during *py4j.SparkSession.getOrCreate*. In order to reproduce the bug, you need to have mesos and mesos dispatcher running. When you launch the following program: {code:bash} spark-submit --master mesos://127.0.1.1:7077 --deploy-mode cluster --class my_python_program.py --py-files /path/to/PySparkInclude.py --name my_python_program --conf spark.mesos.executor.docker.image=myrepo/myimage:mytag /path/to/my_python_program.py 100 {code} then the task failed with the following output: {code:bash} I1020 16:26:11.958860 17736 fetcher.cpp:562] Fetcher Info: {"cache_directory":"/tmp/mesos/fetch/nobody","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"/path/to/my_python_program.py"}},{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"/path/to/PySparkTestInclude.py"}}],"sandbox_directory":"/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375","stall_timeout":{"nanoseconds":60000000000},"user":"nobody"} I1020 16:26:11.962540 17736 fetcher.cpp:459] Fetching URI '/path/to/my_python_program.py' I1020 16:26:11.962558 17736 fetcher.cpp:290] Fetching '/path/to/my_python_program.py' directly into the sandbox directory I1020 16:26:11.962584 17736 fetcher.cpp:175] Downloading resource from '/path/to/my_python_program.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/pi_with_include.py' W1020 16:26:12.126960 17736 fetcher.cpp:332] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: /path/to/my_python_program.py I1020 16:26:12.126986 17736 fetcher.cpp:618] Fetched '/path/to/my_python_program.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/pi_with_include.py' I1020 16:26:12.126991 17736 fetcher.cpp:459] Fetching URI '/path/to/PySparkTestInclude.py' I1020 16:26:12.126996 17736 fetcher.cpp:290] Fetching '/path/to/PySparkTestInclude.py' directly into the sandbox directory I1020 16:26:12.127013 17736 fetcher.cpp:175] Downloading resource from '/path/to/PySparkTestInclude.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/PySparkTestInclude.py' W1020 16:26:12.283085 17736 fetcher.cpp:332] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: /path/to/PySparkTestInclude.py I1020 16:26:12.283108 17736 fetcher.cpp:618] Fetched '/path/to/PySparkTestInclude.py' to '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/PySparkTestInclude.py' I1020 16:26:12.283114 17736 fetcher.cpp:623] Successfully fetched all URIs into '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375' Marked '/' as rslave W1020 16:26:12.389969 2 openssl.cpp:553] Failed SSL connections will be downgraded to a non-SSL socket W1020 16:26:12.390067 2 openssl.cpp:575] TLS certificate verification was enabled by setting one of LIBPROCESS_SSL_VERIFY_CERT or LIBPROCESS_SSL_REQUIRE_CERT, but can be bypassed because TLS downgrades are enabled. I1020 16:26:12.390079 2 openssl.cpp:599] CA directory path unspecified! NOTE: Set CA directory path with LIBPROCESS_SSL_CA_DIR=<dirpath> I1020 16:26:12.390087 2 openssl.cpp:623] Will use IP address verification in subject alternative name certificate extension. I1020 16:26:12.390095 2 openssl.cpp:642] Will verify server certificates for outgoing TLS connections. I1020 16:26:12.390105 2 openssl.cpp:663] Using 'openssl' scheme for hostname validation I1020 16:26:12.390331 2 openssl.cpp:688] Using CA file: /var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/.ssl/ca-bundle.crt W1020 16:26:12.390651 2 process.cpp:1055] Failed SSL connections will be downgraded to a non-SSL socket I1020 16:26:12.392330 2 executor.cpp:210] Version: 1.10.1 I1020 16:26:12.392472 2 executor.cpp:286] Using URL http+unix://%2Fvar%2Frun%2Fmesos%2Fmesos-executors.sock/slave(1)/api/v1/executor for the executor API endpoint I1020 16:26:12.398231 8 executor.cpp:190] Received SUBSCRIBED event I1020 16:26:12.398533 8 executor.cpp:194] Subscribed executor on 172.16.72.91 I1020 16:26:12.398721 8 executor.cpp:190] Received LAUNCH event I1020 16:26:12.399922 8 executor.cpp:671] Overwriting environment variable 'LIBPROCESS_SSL_CA_FILE' I1020 16:26:12.399947 8 executor.cpp:671] Overwriting environment variable 'LIBPROCESS_SSL_CERT_FILE' I1020 16:26:12.399964 8 executor.cpp:671] Overwriting environment variable 'MESOS_NATIVE_JAVA_LIBRARY' I1020 16:26:12.399986 8 executor.cpp:671] Overwriting environment variable 'LIBPROCESS_SSL_KEY_FILE' I1020 16:26:12.400000 8 executor.cpp:671] Overwriting environment variable 'PATH' I1020 16:26:12.400053 8 executor.cpp:722] Starting task driver-20201020162611-0012 I1020 16:26:12.401970 8 executor.cpp:740] Forked command at 12 I1020 16:26:12.403370 9 executor.cpp:190] Received ACKNOWLEDGED event I1020 16:26:12.404062 9 executor.cpp:190] Received ACKNOWLEDGED event Preparing rootfs at '/var/lib/mesos/slave/provisioner/containers/f98449ad-ecba-4bc9-a979-99f21f227375/backends/overlay/rootfses/f80c3e69-73e8-4154-a8e5-8c68613f6462' Marked '/' as rslave Changing root to /var/lib/mesos/slave/provisioner/containers/f98449ad-ecba-4bc9-a979-99f21f227375/backends/overlay/rootfses/f80c3e69-73e8-4154-a8e5-8c68613f6462 spark-env: Configured SPARK_LOCAL_IP with bootstrap: 172.16.72.91 spark-env: User: nobody spark-env: No SPARK_MESOS_KRB5_CONF_BASE64 to decode spark-env: Configuring StatsD endpoint 198.51.100.1:46584 in /opt/spark/conf/metrics.properties 20/10/20 16:26:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/10/20 16:26:17 INFO SparkContext: Running Spark version 3.0.1 20/10/20 16:26:17 INFO ResourceUtils: ============================================================== 20/10/20 16:26:17 INFO ResourceUtils: Resources for spark.driver: 20/10/20 16:26:17 INFO ResourceUtils: ============================================================== 20/10/20 16:26:17 INFO SparkContext: Submitted application: PythonPi 20/10/20 16:26:17 INFO SecurityManager: Changing view acls to: nobody 20/10/20 16:26:17 INFO SecurityManager: Changing modify acls to: nobody 20/10/20 16:26:17 INFO SecurityManager: Changing view acls groups to: 20/10/20 16:26:17 INFO SecurityManager: Changing modify acls groups to: 20/10/20 16:26:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(nobody); groups with view permissions: Set(); users with modify permissions: Set(nobody); groups with modify permissions: Set() 20/10/20 16:26:17 INFO Utils: Successfully started service 'sparkDriver' on port 37759. 20/10/20 16:26:17 INFO SparkEnv: Registering MapOutputTracker 20/10/20 16:26:18 INFO SparkEnv: Registering BlockManagerMaster 20/10/20 16:26:18 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 20/10/20 16:26:18 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 20/10/20 16:26:18 INFO SparkEnv: Registering BlockManagerMasterHeartbeat 20/10/20 16:26:18 INFO DiskBlockManager: Created local directory at /mnt/mesos/sandbox/blockmgr-2be3fdd6-db33-49e8-bc0a-5b1da5ec8ded 20/10/20 16:26:18 INFO MemoryStore: MemoryStore started with capacity 912.3 MiB 20/10/20 16:26:18 INFO MetricsConfig: Loading metrics properties from resource metrics.properties 20/10/20 16:26:18 INFO MetricsConfig: Metrics properties: {*.sink.statsd.class=org.apache.spark.metrics.sink.statsd.StatsdSink, *.sink.statsd.host=198.51.100.1, *.sink.servlet.class=org.apache.spark.metrics.sink.MetricsServlet, applications.sink.servlet.path=/metrics/applications/json, *.sink.statsd.tags=, *.source.jvm.class=org.apache.spark.metrics.source.JvmSource, *.sink.statsd.port=46584, master.sink.servlet.path=/metrics/master/json, *.sink.statsd.prefix=, *.sink.servlet.path=/metrics/json} 20/10/20 16:26:18 INFO SparkEnv: Registering OutputCommitCoordinator 20/10/20 16:26:18 INFO log: Logging initialized @5603ms to org.sparkproject.jetty.util.log.Slf4jLog 20/10/20 16:26:18 INFO Server: jetty-9.4.z-SNAPSHOT; built: 2019-04-29T20:42:08.989Z; git: e1bc35120a6617ee3df052294e433f3a25ce7097; jvm 1.8.0_212-b03 20/10/20 16:26:18 INFO Server: Started @5872ms 20/10/20 16:26:18 INFO AbstractConnector: Started ServerConnector@1345a271{HTTP/1.1,[http/1.1]}{172.16.72.91:4040} 20/10/20 16:26:18 INFO Utils: Successfully started service 'SparkUI' on port 4040. 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@37925df7{/jobs,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@31660e32{/jobs/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@68c3022c{/jobs/job,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@9662252{/jobs/job/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@75e43d99{/stages,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@74c915ac{/stages/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7d15bd53{/stages/stage,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@10d10bcf{/stages/stage/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6ce43fb0{/stages/pool,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@42f60f93{/stages/pool/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@228a2392{/storage,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@50576ab{/storage/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6de665b4{/storage/rdd,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5fb35a9d{/storage/rdd/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5bb2641{/environment,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5f7aae9d{/environment/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@29b1337f{/executors,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7af3cf55{/executors/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@b196f0b{/executors/threadDump,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2383aabe{/executors/threadDump/json,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@545844d3{/static,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1adf4e59{/,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2ae8fb6c{/api,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@21e2f70e{/jobs/job/kill,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6199fb32{/stages/stage/kill,null,AVAILABLE,@Spark} 20/10/20 16:26:18 INFO SparkUI: Bound SparkUI to 172.16.72.91, and started at http://172.16.72.91:4040 20/10/20 16:26:18 ERROR SparkContext: Error initializing SparkContext. java.io.FileNotFoundException: File file:/mnt/mesos/sandbox/$MESOS_SANDBOX/PySparkTestInclude.py does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:666) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:987) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:656) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454) at org.apache.spark.SparkContext.addFile(SparkContext.scala:1534) at org.apache.spark.SparkContext.addFile(SparkContext.scala:1498) at org.apache.spark.SparkContext.$anonfun$new$12(SparkContext.scala:494) at org.apache.spark.SparkContext.$anonfun$new$12$adapted(SparkContext.scala:494) at scala.collection.immutable.List.foreach(List.scala:392) at org.apache.spark.SparkContext.<init>(SparkContext.scala:494) at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:238) at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) 20/10/20 16:26:18 INFO AbstractConnector: Stopped Spark@1345a271{HTTP/1.1,[http/1.1]}{172.16.72.91:4040} 20/10/20 16:26:18 INFO SparkUI: Stopped Spark web UI at http://172.16.72.91:4040 20/10/20 16:26:19 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 20/10/20 16:26:19 INFO MemoryStore: MemoryStore cleared 20/10/20 16:26:19 INFO BlockManager: BlockManager stopped 20/10/20 16:26:19 INFO BlockManagerMaster: BlockManagerMaster stopped 20/10/20 16:26:19 WARN MetricsSystem: Stopping a MetricsSystem that is not running 20/10/20 16:26:19 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 20/10/20 16:26:19 INFO SparkContext: Successfully stopped SparkContext 20/10/20 16:26:19 INFO ShutdownHookManager: Shutdown hook called 20/10/20 16:26:19 INFO ShutdownHookManager: Shutdown hook called 20/10/20 16:26:19 INFO ShutdownHookManager: Deleting directory /tmp/spark-07db5724-6031-45d3-8222-c06947f80872 20/10/20 16:26:19 INFO ShutdownHookManager: Deleting directory /mnt/mesos/sandbox/spark-d4145378-f0da-4f1b-8aee-a72d427ccb8f I1020 16:26:19.363329 3 executor.cpp:1041] Command exited with status 1 (pid: 12) I1020 16:26:19.366904 4 executor.cpp:190] Received ACKNOWLEDGED event I1020 16:26:19.367244 2 process.cpp:935] Stopped the socket accept loop {code} > Mesos Task Failed when pyFiles and docker image option used together > -------------------------------------------------------------------- > > Key: SPARK-33199 > URL: https://issues.apache.org/jira/browse/SPARK-33199 > Project: Spark > Issue Type: Bug > Components: Mesos > Affects Versions: 3.0.1 > Reporter: Farhan Khan > Priority: Major > > It is a bug related to shell escaping in MesosClusterScheduler. As you can > see in this issue SPARK-23941, we are escaping all the options, that lead to > escaping of the *$MESOS_SANDBOX* used in pyFiles in case of docker image > option as well. This in turn causes *FileNotFoundErrors* during > *py4j.SparkSession.getOrCreate*. > In order to reproduce the bug, you need to have mesos and mesos dispatcher > running. When you launch the following program: > {code:bash} > spark-submit --master mesos://127.0.1.1:7077 --deploy-mode cluster --class > my_python_program.py --py-files /path/to/PySparkInclude.py --name > my_python_program --conf > spark.mesos.executor.docker.image=myrepo/myimage:mytag > /path/to/my_python_program.py 100 > {code} > then the task failed with the following output: > {code:bash} > I1020 16:26:11.958860 17736 fetcher.cpp:562] Fetcher Info: > {"cache_directory":"/tmp/mesos/fetch/nobody","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"/path/to/my_python_program.py"}},{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"/path/to/PySparkTestInclude.py"}}],"sandbox_directory":"/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375","stall_timeout":{"nanoseconds":60000000000},"user":"nobody"} > I1020 16:26:11.962540 17736 fetcher.cpp:459] Fetching URI > '/path/to/my_python_program.py' > I1020 16:26:11.962558 17736 fetcher.cpp:290] Fetching > '/path/to/my_python_program.py' directly into the sandbox directory > I1020 16:26:11.962584 17736 fetcher.cpp:175] Downloading resource from > '/path/to/my_python_program.py' to > '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/my_python_program.py' > W1020 16:26:12.126960 17736 fetcher.cpp:332] Copying instead of extracting > resource from URI with 'extract' flag, because it does not seem to be an > archive: /path/to/my_python_program.py > I1020 16:26:12.126986 17736 fetcher.cpp:618] Fetched > '/path/to/my_python_program.py' to > '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/my_python_program.py' > I1020 16:26:12.126991 17736 fetcher.cpp:459] Fetching URI > '/path/to/PySparkTestInclude.py' > I1020 16:26:12.126996 17736 fetcher.cpp:290] Fetching > '/path/to/PySparkTestInclude.py' directly into the sandbox directory > I1020 16:26:12.127013 17736 fetcher.cpp:175] Downloading resource from > '/path/to/PySparkTestInclude.py' to > '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/PySparkTestInclude.py' > W1020 16:26:12.283085 17736 fetcher.cpp:332] Copying instead of extracting > resource from URI with 'extract' flag, because it does not seem to be an > archive: /path/to/PySparkTestInclude.py > I1020 16:26:12.283108 17736 fetcher.cpp:618] Fetched > '/path/to/PySparkTestInclude.py' to > '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/PySparkTestInclude.py' > I1020 16:26:12.283114 17736 fetcher.cpp:623] Successfully fetched all URIs > into > '/var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375' > Marked '/' as rslave > W1020 16:26:12.389969 2 openssl.cpp:553] Failed SSL connections will be > downgraded to a non-SSL socket > W1020 16:26:12.390067 2 openssl.cpp:575] TLS certificate verification was > enabled by setting one of LIBPROCESS_SSL_VERIFY_CERT or > LIBPROCESS_SSL_REQUIRE_CERT, but can be bypassed because TLS downgrades are > enabled. > I1020 16:26:12.390079 2 openssl.cpp:599] CA directory path unspecified! > NOTE: Set CA directory path with LIBPROCESS_SSL_CA_DIR=<dirpath> > I1020 16:26:12.390087 2 openssl.cpp:623] Will use IP address verification > in subject alternative name certificate extension. > I1020 16:26:12.390095 2 openssl.cpp:642] Will verify server certificates > for outgoing TLS connections. > I1020 16:26:12.390105 2 openssl.cpp:663] Using 'openssl' scheme for > hostname validation > I1020 16:26:12.390331 2 openssl.cpp:688] Using CA file: > /var/lib/mesos/slave/slaves/746d6707-8689-4698-9b93-7368ad523421-S6/frameworks/746d6707-8689-4698-9b93-7368ad523421-0016/executors/driver-20201020162611-0012/runs/f98449ad-ecba-4bc9-a979-99f21f227375/.ssl/ca-bundle.crt > W1020 16:26:12.390651 2 process.cpp:1055] Failed SSL connections will be > downgraded to a non-SSL socket > I1020 16:26:12.392330 2 executor.cpp:210] Version: 1.10.1 > I1020 16:26:12.392472 2 executor.cpp:286] Using URL > http+unix://%2Fvar%2Frun%2Fmesos%2Fmesos-executors.sock/slave(1)/api/v1/executor > for the executor API endpoint > I1020 16:26:12.398231 8 executor.cpp:190] Received SUBSCRIBED event > I1020 16:26:12.398533 8 executor.cpp:194] Subscribed executor on > 172.16.72.91 > I1020 16:26:12.398721 8 executor.cpp:190] Received LAUNCH event > I1020 16:26:12.399922 8 executor.cpp:671] Overwriting environment > variable 'LIBPROCESS_SSL_CA_FILE' > I1020 16:26:12.399947 8 executor.cpp:671] Overwriting environment > variable 'LIBPROCESS_SSL_CERT_FILE' > I1020 16:26:12.399964 8 executor.cpp:671] Overwriting environment > variable 'MESOS_NATIVE_JAVA_LIBRARY' > I1020 16:26:12.399986 8 executor.cpp:671] Overwriting environment > variable 'LIBPROCESS_SSL_KEY_FILE' > I1020 16:26:12.400000 8 executor.cpp:671] Overwriting environment > variable 'PATH' > I1020 16:26:12.400053 8 executor.cpp:722] Starting task > driver-20201020162611-0012 > I1020 16:26:12.401970 8 executor.cpp:740] Forked command at 12 > I1020 16:26:12.403370 9 executor.cpp:190] Received ACKNOWLEDGED event > I1020 16:26:12.404062 9 executor.cpp:190] Received ACKNOWLEDGED event > Preparing rootfs at > '/var/lib/mesos/slave/provisioner/containers/f98449ad-ecba-4bc9-a979-99f21f227375/backends/overlay/rootfses/f80c3e69-73e8-4154-a8e5-8c68613f6462' > Marked '/' as rslave > Changing root to > /var/lib/mesos/slave/provisioner/containers/f98449ad-ecba-4bc9-a979-99f21f227375/backends/overlay/rootfses/f80c3e69-73e8-4154-a8e5-8c68613f6462 > spark-env: Configured SPARK_LOCAL_IP with bootstrap: 172.16.72.91 > spark-env: User: nobody > spark-env: No SPARK_MESOS_KRB5_CONF_BASE64 to decode > spark-env: Configuring StatsD endpoint 198.51.100.1:46584 in > /opt/spark/conf/metrics.properties > 20/10/20 16:26:15 WARN NativeCodeLoader: Unable to load native-hadoop library > for your platform... using builtin-java classes where applicable > 20/10/20 16:26:17 INFO SparkContext: Running Spark version 3.0.1 > 20/10/20 16:26:17 INFO ResourceUtils: > ============================================================== > 20/10/20 16:26:17 INFO ResourceUtils: Resources for spark.driver: > 20/10/20 16:26:17 INFO ResourceUtils: > ============================================================== > 20/10/20 16:26:17 INFO SparkContext: Submitted application: PythonPi > 20/10/20 16:26:17 INFO SecurityManager: Changing view acls to: nobody > 20/10/20 16:26:17 INFO SecurityManager: Changing modify acls to: nobody > 20/10/20 16:26:17 INFO SecurityManager: Changing view acls groups to: > 20/10/20 16:26:17 INFO SecurityManager: Changing modify acls groups to: > 20/10/20 16:26:17 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(nobody); groups > with view permissions: Set(); users with modify permissions: Set(nobody); > groups with modify permissions: Set() > 20/10/20 16:26:17 INFO Utils: Successfully started service 'sparkDriver' on > port 37759. > 20/10/20 16:26:17 INFO SparkEnv: Registering MapOutputTracker > 20/10/20 16:26:18 INFO SparkEnv: Registering BlockManagerMaster > 20/10/20 16:26:18 INFO BlockManagerMasterEndpoint: Using > org.apache.spark.storage.DefaultTopologyMapper for getting topology > information > 20/10/20 16:26:18 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint > up > 20/10/20 16:26:18 INFO SparkEnv: Registering BlockManagerMasterHeartbeat > 20/10/20 16:26:18 INFO DiskBlockManager: Created local directory at > /mnt/mesos/sandbox/blockmgr-2be3fdd6-db33-49e8-bc0a-5b1da5ec8ded > 20/10/20 16:26:18 INFO MemoryStore: MemoryStore started with capacity 912.3 > MiB > 20/10/20 16:26:18 INFO MetricsConfig: Loading metrics properties from > resource metrics.properties > 20/10/20 16:26:18 INFO MetricsConfig: Metrics properties: > {*.sink.statsd.class=org.apache.spark.metrics.sink.statsd.StatsdSink, > *.sink.statsd.host=198.51.100.1, > *.sink.servlet.class=org.apache.spark.metrics.sink.MetricsServlet, > applications.sink.servlet.path=/metrics/applications/json, > *.sink.statsd.tags=, > *.source.jvm.class=org.apache.spark.metrics.source.JvmSource, > *.sink.statsd.port=46584, master.sink.servlet.path=/metrics/master/json, > *.sink.statsd.prefix=, *.sink.servlet.path=/metrics/json} > 20/10/20 16:26:18 INFO SparkEnv: Registering OutputCommitCoordinator > 20/10/20 16:26:18 INFO log: Logging initialized @5603ms to > org.sparkproject.jetty.util.log.Slf4jLog > 20/10/20 16:26:18 INFO Server: jetty-9.4.z-SNAPSHOT; built: > 2019-04-29T20:42:08.989Z; git: e1bc35120a6617ee3df052294e433f3a25ce7097; jvm > 1.8.0_212-b03 > 20/10/20 16:26:18 INFO Server: Started @5872ms > 20/10/20 16:26:18 INFO AbstractConnector: Started > ServerConnector@1345a271{HTTP/1.1,[http/1.1]}{172.16.72.91:4040} > 20/10/20 16:26:18 INFO Utils: Successfully started service 'SparkUI' on port > 4040. > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@37925df7{/jobs,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@31660e32{/jobs/json,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@68c3022c{/jobs/job,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@9662252{/jobs/job/json,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@75e43d99{/stages,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@74c915ac{/stages/json,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@7d15bd53{/stages/stage,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@10d10bcf{/stages/stage/json,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@6ce43fb0{/stages/pool,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@42f60f93{/stages/pool/json,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@228a2392{/storage,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@50576ab{/storage/json,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@6de665b4{/storage/rdd,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@5fb35a9d{/storage/rdd/json,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@5bb2641{/environment,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@5f7aae9d{/environment/json,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@29b1337f{/executors,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@7af3cf55{/executors/json,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@b196f0b{/executors/threadDump,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@2383aabe{/executors/threadDump/json,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@545844d3{/static,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@1adf4e59{/,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@2ae8fb6c{/api,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@21e2f70e{/jobs/job/kill,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO ContextHandler: Started > o.s.j.s.ServletContextHandler@6199fb32{/stages/stage/kill,null,AVAILABLE,@Spark} > 20/10/20 16:26:18 INFO SparkUI: Bound SparkUI to 172.16.72.91, and started at > http://172.16.72.91:4040 > 20/10/20 16:26:18 ERROR SparkContext: Error initializing SparkContext. > java.io.FileNotFoundException: File > file:/mnt/mesos/sandbox/$MESOS_SANDBOX/PySparkTestInclude.py does not exist > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:666) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:987) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:656) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454) > at org.apache.spark.SparkContext.addFile(SparkContext.scala:1534) > at org.apache.spark.SparkContext.addFile(SparkContext.scala:1498) > at org.apache.spark.SparkContext.$anonfun$new$12(SparkContext.scala:494) > at > org.apache.spark.SparkContext.$anonfun$new$12$adapted(SparkContext.scala:494) > at scala.collection.immutable.List.foreach(List.scala:392) > at org.apache.spark.SparkContext.<init>(SparkContext.scala:494) > at > org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) > at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) > at py4j.Gateway.invoke(Gateway.java:238) > at > py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) > at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) > at py4j.GatewayConnection.run(GatewayConnection.java:238) > at java.lang.Thread.run(Thread.java:748) > 20/10/20 16:26:18 INFO AbstractConnector: Stopped > Spark@1345a271{HTTP/1.1,[http/1.1]}{172.16.72.91:4040} > 20/10/20 16:26:18 INFO SparkUI: Stopped Spark web UI at > http://172.16.72.91:4040 > 20/10/20 16:26:19 INFO MapOutputTrackerMasterEndpoint: > MapOutputTrackerMasterEndpoint stopped! > 20/10/20 16:26:19 INFO MemoryStore: MemoryStore cleared > 20/10/20 16:26:19 INFO BlockManager: BlockManager stopped > 20/10/20 16:26:19 INFO BlockManagerMaster: BlockManagerMaster stopped > 20/10/20 16:26:19 WARN MetricsSystem: Stopping a MetricsSystem that is not > running > 20/10/20 16:26:19 INFO > OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: > OutputCommitCoordinator stopped! > 20/10/20 16:26:19 INFO SparkContext: Successfully stopped SparkContext > 20/10/20 16:26:19 INFO ShutdownHookManager: Shutdown hook called > 20/10/20 16:26:19 INFO ShutdownHookManager: Shutdown hook called > 20/10/20 16:26:19 INFO ShutdownHookManager: Deleting directory > /tmp/spark-07db5724-6031-45d3-8222-c06947f80872 > 20/10/20 16:26:19 INFO ShutdownHookManager: Deleting directory > /mnt/mesos/sandbox/spark-d4145378-f0da-4f1b-8aee-a72d427ccb8f > I1020 16:26:19.363329 3 executor.cpp:1041] Command exited with status 1 > (pid: 12) > I1020 16:26:19.366904 4 executor.cpp:190] Received ACKNOWLEDGED event > I1020 16:26:19.367244 2 process.cpp:935] Stopped the socket accept loop > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org