[
https://issues.apache.org/jira/browse/SPARK-45200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitin Dominic updated SPARK-45200:
----------------------------------
Issue Type: Bug (was: Question)
> Ignore/Disable Spark's default log4j profile
> --------------------------------------------
>
> Key: SPARK-45200
> URL: https://issues.apache.org/jira/browse/SPARK-45200
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.4.0
> Reporter: Jitin Dominic
> Priority: Major
>
> I've been using Spark core 3.2.2 and was upgrading to 3.4.0
>
> On execution of my Java code with the 3.4.0, it generates some extra set of
> logs but don't face this issue with 3.2.2.
>
> I noticed that logs says _Using Spark's default log4j profile:
> org/apache/spark/log4j2-defaults.properties._
>
> Is there a configuration to disable the using of default log4j profile?
>
>
> {code:java}
> Using Spark's default log4j profile:
> org/apache/spark/log4j2-defaults.properties
> 23/09/18 20:05:08 INFO SparkContext: Running Spark version 3.4.0
> 23/09/18 20:05:08 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 23/09/18 20:05:08 INFO ResourceUtils:
> ==============================================================
> 23/09/18 20:05:08 INFO ResourceUtils: No custom resources configured for
> spark.driver.
> 23/09/18 20:05:08 INFO ResourceUtils:
> ==============================================================
> 23/09/18 20:05:08 INFO SparkContext: Submitted application: XYZ
> 23/09/18 20:05:08 INFO ResourceProfile: Default ResourceProfile created,
> executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: ,
> memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name:
> offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name:
> cpus, amount: 1.0)
> 23/09/18 20:05:08 INFO ResourceProfile: Limiting resource is cpu
> 23/09/18 20:05:08 INFO ResourceProfileManager: Added ResourceProfile id: 0
> 23/09/18 20:05:08 INFO SecurityManager: Changing view acls to: jd
> 23/09/18 20:05:08 INFO SecurityManager: Changing modify acls to: jd
> 23/09/18 20:05:08 INFO SecurityManager: Changing view acls groups to:
> 23/09/18 20:05:08 INFO SecurityManager: Changing modify acls groups to:
> 23/09/18 20:05:08 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: jd; groups with view
> permissions: EMPTY; users with modify permissions: jd; groups with modify
> permissions: EMPTY
> 23/09/18 20:05:08 INFO Utils: Successfully started service 'sparkDriver' on
> port 39155.
> 23/09/18 20:05:08 INFO SparkEnv: Registering MapOutputTracker
> 23/09/18 20:05:08 INFO SparkEnv: Registering BlockManagerMaster
> 23/09/18 20:05:08 INFO BlockManagerMasterEndpoint: Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information
> 23/09/18 20:05:08 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint
> up
> 23/09/18 20:05:08 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
> 23/09/18 20:05:08 INFO DiskBlockManager: Created local directory at
> /tmp/blockmgr-226d599c-1511-4fae-b0e7-aae81b684012
> 23/09/18 20:05:08 INFO MemoryStore: MemoryStore started with capacity 2004.6
> MiB
> 23/09/18 20:05:08 INFO SparkEnv: Registering OutputCommitCoordinator
> 23/09/18 20:05:08 INFO JettyUtils: Start Jetty 0.0.0.0:4040 for SparkUI
> 23/09/18 20:05:09 INFO Utils: Successfully started service 'SparkUI' on port
> 4040.
> 23/09/18 20:05:09 INFO Executor: Starting executor ID driver on host jd
> 23/09/18 20:05:09 INFO Executor: Starting executor with user classpath
> (userClassPathFirst = false): ''
> 23/09/18 20:05:09 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 32819.
> 23/09/18 20:05:09 INFO NettyBlockTransferService: Server created on jd:32819
> 23/09/18 20:05:09 INFO BlockManager: Using
> org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
> policy
> 23/09/18 20:05:09 INFO BlockManagerMaster: Registering BlockManager
> BlockManagerId(driver, jd, 32819, None)
> 23/09/18 20:05:09 INFO BlockManagerMasterEndpoint: Registering block manager
> jd:32819 with 2004.6 MiB RAM, BlockManagerId(driver, jd, 32819, None)
> 23/09/18 20:05:09 INFO BlockManagerMaster: Registered BlockManager
> BlockManagerId(driver, jd, 32819, None)
> 23/09/18 20:05:09 INFO BlockManager: Initialized BlockManager:
> BlockManagerId(driver, jd, 32819, None)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
