[ https://issues.apache.org/jira/browse/SPARK-45200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kent Yao resolved SPARK-45200. ------------------------------ Resolution: Not A Problem > Spark 3.4.0 always using default log4j profile > ---------------------------------------------- > > Key: SPARK-45200 > URL: https://issues.apache.org/jira/browse/SPARK-45200 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 3.4.0 > Reporter: Jitin Dominic > Priority: Major > Labels: pull-request-available > > I've been using Spark core 3.2.2 and was upgrading to 3.4.0 > On execution of my Java code with the 3.4.0, it generates some extra set of > logs but don't face this issue with 3.2.2. > > I noticed that logs says _Using Spark's default log4j profile: > org/apache/spark/log4j2-defaults.properties._ > > Is this a bug or do we have a a configuration to disable the using of > default log4j profile? > I didn't see anything in the documentation > > +*Output:*+ > {code:java} > Using Spark's default log4j profile: > org/apache/spark/log4j2-defaults.properties > 23/09/18 20:05:08 INFO SparkContext: Running Spark version 3.4.0 > 23/09/18 20:05:08 WARN NativeCodeLoader: Unable to load native-hadoop library > for your platform... using builtin-java classes where applicable > 23/09/18 20:05:08 INFO ResourceUtils: > ============================================================== > 23/09/18 20:05:08 INFO ResourceUtils: No custom resources configured for > spark.driver. > 23/09/18 20:05:08 INFO ResourceUtils: > ============================================================== > 23/09/18 20:05:08 INFO SparkContext: Submitted application: XYZ > 23/09/18 20:05:08 INFO ResourceProfile: Default ResourceProfile created, > executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , > memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: > offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: > cpus, amount: 1.0) > 23/09/18 20:05:08 INFO ResourceProfile: Limiting resource is cpu > 23/09/18 20:05:08 INFO ResourceProfileManager: Added ResourceProfile id: 0 > 23/09/18 20:05:08 INFO SecurityManager: Changing view acls to: jd > 23/09/18 20:05:08 INFO SecurityManager: Changing modify acls to: jd > 23/09/18 20:05:08 INFO SecurityManager: Changing view acls groups to: > 23/09/18 20:05:08 INFO SecurityManager: Changing modify acls groups to: > 23/09/18 20:05:08 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: jd; groups with view > permissions: EMPTY; users with modify permissions: jd; groups with modify > permissions: EMPTY > 23/09/18 20:05:08 INFO Utils: Successfully started service 'sparkDriver' on > port 39155. > 23/09/18 20:05:08 INFO SparkEnv: Registering MapOutputTracker > 23/09/18 20:05:08 INFO SparkEnv: Registering BlockManagerMaster > 23/09/18 20:05:08 INFO BlockManagerMasterEndpoint: Using > org.apache.spark.storage.DefaultTopologyMapper for getting topology > information > 23/09/18 20:05:08 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint > up > 23/09/18 20:05:08 INFO SparkEnv: Registering BlockManagerMasterHeartbeat > 23/09/18 20:05:08 INFO DiskBlockManager: Created local directory at > /tmp/blockmgr-226d599c-1511-4fae-b0e7-aae81b684012 > 23/09/18 20:05:08 INFO MemoryStore: MemoryStore started with capacity 2004.6 > MiB > 23/09/18 20:05:08 INFO SparkEnv: Registering OutputCommitCoordinator > 23/09/18 20:05:08 INFO JettyUtils: Start Jetty 0.0.0.0:4040 for SparkUI > 23/09/18 20:05:09 INFO Utils: Successfully started service 'SparkUI' on port > 4040. > 23/09/18 20:05:09 INFO Executor: Starting executor ID driver on host jd > 23/09/18 20:05:09 INFO Executor: Starting executor with user classpath > (userClassPathFirst = false): '' > 23/09/18 20:05:09 INFO Utils: Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port 32819. > 23/09/18 20:05:09 INFO NettyBlockTransferService: Server created on jd:32819 > 23/09/18 20:05:09 INFO BlockManager: Using > org.apache.spark.storage.RandomBlockReplicationPolicy for block replication > policy > 23/09/18 20:05:09 INFO BlockManagerMaster: Registering BlockManager > BlockManagerId(driver, jd, 32819, None) > 23/09/18 20:05:09 INFO BlockManagerMasterEndpoint: Registering block manager > jd:32819 with 2004.6 MiB RAM, BlockManagerId(driver, jd, 32819, None) > 23/09/18 20:05:09 INFO BlockManagerMaster: Registered BlockManager > BlockManagerId(driver, jd, 32819, None) > 23/09/18 20:05:09 INFO BlockManager: Initialized BlockManager: > BlockManagerId(driver, jd, 32819, None) > {code} > > > > I'm using Spark core dependency in one of my Jars, the jar code contains the > following: > > *+Code:+* > {code:java} > SparkSession > .builder() > .appName("XYZ") > .config("spark.master", "local") > .config("spark.io.compression.codec", "snappy") > .config("spark.io.compression.snappy.blockSize", blockSize) > .getOrCreate(); {code} > > When it calls *getOrCreate()* method, I get the above stated extra logs. > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org