LuciferYang commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2311695631
> ```
> sbin/start-connect-server.sh --conf
spark.log.structuredLogging.enabled=false
> ```
I used the command `dev/make-distribution.sh --tgz -Phive` to package the
latest code.
And without this pr, I manually run `sbin/start-connect-server.sh --conf
spark.log.structuredLogging.enabled=false`, and it did use `structuredLogging`,
even though `spark.log.structuredLogging.enabled` was specified as false:
```
tail -100f
logs/spark-yangjie01-org.apache.spark.sql.connect.service.SparkConnectServer-1-MacBook-Pro.local.out
Spark Command: /Users/yangjie01/Tools/zulu17/bin/java -cp
hive-jackson/*:/Users/yangjie01/Tools/4.0/spark-4.0.0-SNAPSHOT-bin-3.4.0/conf/:/Users/yangjie01/Tools/4.0/spark-4.0.0-SNAPSHOT-bin-3.4.0/jars/slf4j-api-2.0.16.jar:/Users/yangjie01/Tools/4.0/spark-4.0.0-SNAPSHOT-bin-3.4.0/jars/*
-Xmx1g -XX:+IgnoreUnrecognizedVMOptions --add-modules=jdk.incubator.vector
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
--add-opens=java.base/sun.security.action=ALL-UNNAMED --a
dd-opens=java.base/sun.util.calendar=ALL-UNNAMED
--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED
-Djdk.reflect.useDirectMethodHandle=false
-Dio.netty.tryReflectionSetAccessible=true
-Dderby.connection.requireAuthentication=false
org.apache.spark.deploy.SparkSubmit --conf
spark.log.structuredLogging.enabled=false --class
org.apache.spark.sql.connect.service.SparkConnectServer --name Spark Connect
server spark-internal
========================================
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile:
org/apache/spark/log4j2-defaults.properties
{"ts":"2024-08-27T06:42:25.198Z","level":"WARN","msg":"Your hostname,
MacBook-Pro.local, resolves to a loopback address: 127.0.0.1; using
172.22.200.254 instead (on interface
en0)","context":{"host":"MacBook-Pro.local","host_port":"127.0.0.1","host_port2":"172.22.200.254","network_if":"en0"},"logger":"Utils"}
{"ts":"2024-08-27T06:42:25.199Z","level":"WARN","msg":"Set SPARK_LOCAL_IP if
you need to bind to another address","logger":"Utils"}
{"ts":"2024-08-27T06:42:25.309Z","level":"INFO","msg":"Starting Spark
session.","logger":"SparkConnectServer"}
{"ts":"2024-08-27T06:42:25.321Z","level":"INFO","msg":"Running Spark version
4.0.0-SNAPSHOT","logger":"SparkContext"}
{"ts":"2024-08-27T06:42:25.322Z","level":"INFO","msg":"OS info Mac OS X,
14.6.1, aarch64","logger":"SparkContext"}
{"ts":"2024-08-27T06:42:25.322Z","level":"INFO","msg":"Java version
17.0.12","logger":"SparkContext"}
{"ts":"2024-08-27T06:42:25.359Z","level":"WARN","msg":"Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable","logger":"NativeCodeLoader"}
{"ts":"2024-08-27T06:42:25.375Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}
{"ts":"2024-08-27T06:42:25.375Z","level":"INFO","msg":"No custom resources
configured for spark.driver.","logger":"ResourceUtils"}
{"ts":"2024-08-27T06:42:25.375Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}
{"ts":"2024-08-27T06:42:25.376Z","level":"INFO","msg":"Submitted
application: Spark Connect server","logger":"SparkContext"}
{"ts":"2024-08-27T06:42:25.383Z","level":"INFO","msg":"Default
ResourceProfile created, executor resources: Map(cores -> name: cores, amount:
1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor:
, offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources:
Map(cpus -> name: cpus, amount: 1.0)","logger":"ResourceProfile"}
{"ts":"2024-08-27T06:42:25.384Z","level":"INFO","msg":"Limiting resource is
cpu","logger":"ResourceProfile"}
{"ts":"2024-08-27T06:42:25.384Z","level":"INFO","msg":"Added ResourceProfile
id: 0","logger":"ResourceProfileManager"}
{"ts":"2024-08-27T06:42:25.400Z","level":"INFO","msg":"Changing view acls
to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.401Z","level":"INFO","msg":"Changing modify acls
to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.401Z","level":"INFO","msg":"Changing view acls
groups to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.401Z","level":"INFO","msg":"Changing modify acls
groups to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.402Z","level":"INFO","msg":"SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
yangjie01 groups with view permissions: EMPTY; users with modify permissions:
yangjie01; groups with modify permissions: EMPTY; RPC SSL
disabled","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.483Z","level":"INFO","msg":"Successfully started
service 'sparkDriver' on port 51354.","logger":"Utils"}
{"ts":"2024-08-27T06:42:25.493Z","level":"INFO","msg":"Registering
MapOutputTracker","logger":"SparkEnv"}
{"ts":"2024-08-27T06:42:25.496Z","level":"INFO","msg":"Registering
BlockManagerMaster","logger":"SparkEnv"}
{"ts":"2024-08-27T06:42:25.503Z","level":"INFO","msg":"Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology
information","logger":"BlockManagerMasterEndpoint"}
{"ts":"2024-08-27T06:42:25.503Z","level":"INFO","msg":"BlockManagerMasterEndpoint
up","logger":"BlockManagerMasterEndpoint"}
{"ts":"2024-08-27T06:42:25.504Z","level":"INFO","msg":"Registering
BlockManagerMasterHeartbeat","logger":"SparkEnv"}
{"ts":"2024-08-27T06:42:25.512Z","level":"INFO","msg":"Created local
directory at
/private/var/folders/j2/cfn7w6795538n_416_27rkqm0000gn/T/blockmgr-33631b4f-63ad-49a7-8285-f286ff242854","logger":"DiskBlockManager"}
{"ts":"2024-08-27T06:42:25.519Z","level":"INFO","msg":"Registering
OutputCommitCoordinator","logger":"SparkEnv"}
{"ts":"2024-08-27T06:42:25.564Z","level":"INFO","msg":"Start Jetty
0.0.0.0:4040 for SparkUI","logger":"JettyUtils"}
{"ts":"2024-08-27T06:42:25.587Z","level":"INFO","msg":"Successfully started
service 'SparkUI' on port 4040.","logger":"Utils"}
{"ts":"2024-08-27T06:42:25.608Z","level":"INFO","msg":"Changing view acls
to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.608Z","level":"INFO","msg":"Changing modify acls
to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.608Z","level":"INFO","msg":"Changing view acls
groups to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.608Z","level":"INFO","msg":"Changing modify acls
groups to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.608Z","level":"INFO","msg":"SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
yangjie01 groups with view permissions: EMPTY; users with modify permissions:
yangjie01; groups with modify permissions: EMPTY; RPC SSL
disabled","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.631Z","level":"INFO","msg":"Starting executor ID
driver on host 172.22.200.254","logger":"Executor"}
{"ts":"2024-08-27T06:42:25.631Z","level":"INFO","msg":"OS info Mac OS X,
14.6.1, aarch64","logger":"Executor"}
{"ts":"2024-08-27T06:42:25.631Z","level":"INFO","msg":"Java version
17.0.12","logger":"Executor"}
{"ts":"2024-08-27T06:42:25.633Z","level":"INFO","msg":"Starting executor
with user classpath (userClassPathFirst = false): ''","logger":"Executor"}
{"ts":"2024-08-27T06:42:25.634Z","level":"INFO","msg":"Created or updated
repl class loader org.apache.spark.util.MutableURLClassLoader@6fff46bf for
default.","logger":"Executor"}
{"ts":"2024-08-27T06:42:25.641Z","level":"INFO","msg":"Successfully started
service 'org.apache.spark.network.netty.NettyBlockTransferService' on port
51355.","logger":"Utils"}
{"ts":"2024-08-27T06:42:25.643Z","level":"INFO","msg":"Server created on
172.22.200.254:51355","logger":"NettyBlockTransferService"}
{"ts":"2024-08-27T06:42:25.643Z","level":"INFO","msg":"Using
org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
policy","logger":"BlockManager"}
{"ts":"2024-08-27T06:42:25.648Z","level":"INFO","msg":"Registering
BlockManager BlockManagerId(driver, 172.22.200.254, 51355,
None)","logger":"BlockManagerMaster"}
{"ts":"2024-08-27T06:42:25.650Z","level":"INFO","msg":"Registering block
manager 172.22.200.254:51355 with 434.4 MiB RAM, BlockManagerId(driver,
172.22.200.254, 51355, None)","logger":"BlockManagerMasterEndpoint"}
{"ts":"2024-08-27T06:42:25.651Z","level":"INFO","msg":"Registered
BlockManager BlockManagerId(driver, 172.22.200.254, 51355,
None)","logger":"BlockManagerMaster"}
{"ts":"2024-08-27T06:42:25.651Z","level":"INFO","msg":"Initialized
BlockManager: BlockManagerId(driver, 172.22.200.254, 51355,
None)","logger":"BlockManager"}
{"ts":"2024-08-27T06:42:25.793Z","level":"INFO","msg":"Successfully started
service 'org.apache.spark.sql.connect.service.SparkConnectService$' on port
15002.","logger":"Utils"}
{"ts":"2024-08-27T06:42:25.798Z","level":"INFO","msg":"Spark Connect server
started at: 0:0:0:0:0:0:0:0:15002","logger":"SparkConnectServer"}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
