[ https://issues.apache.org/jira/browse/SPARK-45200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jitin Dominic updated SPARK-45200: ---------------------------------- Attachment: (was: image-2023-09-18-18-41-46-612.png) > Ignore/Disable Spark's default log4j profile > -------------------------------------------- > > Key: SPARK-45200 > URL: https://issues.apache.org/jira/browse/SPARK-45200 > Project: Spark > Issue Type: Question > Components: Spark Core > Affects Versions: 3.4.0 > Reporter: Jitin Dominic > Priority: Major > > /I've been using Spark core 3.2.2 and was upgrading to 3.4.0 > > On execution of my code with the 3.4.0, it generates some extra set of logs > but don't face this issue with 3.2.2. > > I noticed that logs says _Using Spark's default log4j profile: > org/apache/spark/log4j2-defaults.properties._ > > Is there a configuration to disable the using of default log4j profile? > > > {code:java} > Using Spark's default log4j profile: > org/apache/spark/log4j2-defaults.properties > 23/09/18 20:05:08 INFO SparkContext: Running Spark version 3.4.0 > 23/09/18 20:05:08 WARN NativeCodeLoader: Unable to load native-hadoop library > for your platform... using builtin-java classes where applicable > 23/09/18 20:05:08 INFO ResourceUtils: > ============================================================== > 23/09/18 20:05:08 INFO ResourceUtils: No custom resources configured for > spark.driver. > 23/09/18 20:05:08 INFO ResourceUtils: > ============================================================== > 23/09/18 20:05:08 INFO SparkContext: Submitted application: XYZ > 23/09/18 20:05:08 INFO ResourceProfile: Default ResourceProfile created, > executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , > memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: > offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: > cpus, amount: 1.0) > 23/09/18 20:05:08 INFO ResourceProfile: Limiting resource is cpu > 23/09/18 20:05:08 INFO ResourceProfileManager: Added ResourceProfile id: 0 > 23/09/18 20:05:08 INFO SecurityManager: Changing view acls to: jd > 23/09/18 20:05:08 INFO SecurityManager: Changing modify acls to: jd > 23/09/18 20:05:08 INFO SecurityManager: Changing view acls groups to: > 23/09/18 20:05:08 INFO SecurityManager: Changing modify acls groups to: > 23/09/18 20:05:08 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: jd; groups with view > permissions: EMPTY; users with modify permissions: jd; groups with modify > permissions: EMPTY > 23/09/18 20:05:08 INFO Utils: Successfully started service 'sparkDriver' on > port 39155. > 23/09/18 20:05:08 INFO SparkEnv: Registering MapOutputTracker > 23/09/18 20:05:08 INFO SparkEnv: Registering BlockManagerMaster > 23/09/18 20:05:08 INFO BlockManagerMasterEndpoint: Using > org.apache.spark.storage.DefaultTopologyMapper for getting topology > information > 23/09/18 20:05:08 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint > up > 23/09/18 20:05:08 INFO SparkEnv: Registering BlockManagerMasterHeartbeat > 23/09/18 20:05:08 INFO DiskBlockManager: Created local directory at > /tmp/blockmgr-226d599c-1511-4fae-b0e7-aae81b684012 > 23/09/18 20:05:08 INFO MemoryStore: MemoryStore started with capacity 2004.6 > MiB > 23/09/18 20:05:08 INFO SparkEnv: Registering OutputCommitCoordinator > 23/09/18 20:05:08 INFO JettyUtils: Start Jetty 0.0.0.0:4040 for SparkUI > 23/09/18 20:05:09 INFO Utils: Successfully started service 'SparkUI' on port > 4040. > 23/09/18 20:05:09 INFO Executor: Starting executor ID driver on host jd > 23/09/18 20:05:09 INFO Executor: Starting executor with user classpath > (userClassPathFirst = false): '' > 23/09/18 20:05:09 INFO Utils: Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port 32819. > 23/09/18 20:05:09 INFO NettyBlockTransferService: Server created on jd:32819 > 23/09/18 20:05:09 INFO BlockManager: Using > org.apache.spark.storage.RandomBlockReplicationPolicy for block replication > policy > 23/09/18 20:05:09 INFO BlockManagerMaster: Registering BlockManager > BlockManagerId(driver, jd, 32819, None) > 23/09/18 20:05:09 INFO BlockManagerMasterEndpoint: Registering block manager > jd:32819 with 2004.6 MiB RAM, BlockManagerId(driver, jd, 32819, None) > 23/09/18 20:05:09 INFO BlockManagerMaster: Registered BlockManager > BlockManagerId(driver, jd, 32819, None) > 23/09/18 20:05:09 INFO BlockManager: Initialized BlockManager: > BlockManagerId(driver, jd, 32819, None) > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org