[
https://issues.apache.org/jira/browse/SPARK-32108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxim Gekk resolved SPARK-32108.
--------------------------------
Resolution: Not A Problem
> Silent mode of spark-sql is broken
> ----------------------------------
>
> Key: SPARK-32108
> URL: https://issues.apache.org/jira/browse/SPARK-32108
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Maxim Gekk
> Priority: Major
>
> 1. I download the recent release Spark 3.0 from
> http://spark.apache.org/downloads.html
> 2. Run bin/spark-sql -S, it prints a lot of INFO
> {code}
> ➜ ~ ./spark-3.0/bin/spark-sql -S
> 20/06/26 20:43:38 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.hive.conf.HiveConf).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 20/06/26 20:43:39 INFO SharedState: spark.sql.warehouse.dir is not set, but
> hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the
> value of hive.metastore.warehouse.dir ('/user/hive/warehouse').
> 20/06/26 20:43:39 INFO SharedState: Warehouse path is '/user/hive/warehouse'.
> 20/06/26 20:43:39 INFO SessionState: Created HDFS directory:
> /tmp/hive/maximgekk/a47e882c-86a3-42b9-b43f-9dab0dd8492a
> 20/06/26 20:43:39 INFO SessionState: Created local directory:
> /var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/maximgekk/a47e882c-86a3-42b9-b43f-9dab0dd8492a
> 20/06/26 20:43:39 INFO SessionState: Created HDFS directory:
> /tmp/hive/maximgekk/a47e882c-86a3-42b9-b43f-9dab0dd8492a/_tmp_space.db
> 20/06/26 20:43:39 INFO SparkContext: Running Spark version 3.0.0
> 20/06/26 20:43:39 INFO ResourceUtils:
> ==============================================================
> 20/06/26 20:43:39 INFO ResourceUtils: Resources for spark.driver:
> 20/06/26 20:43:39 INFO ResourceUtils:
> ==============================================================
> 20/06/26 20:43:39 INFO SparkContext: Submitted application:
> SparkSQL::192.168.1.78
> 20/06/26 20:43:39 INFO SecurityManager: Changing view acls to: maximgekk
> 20/06/26 20:43:39 INFO SecurityManager: Changing modify acls to: maximgekk
> 20/06/26 20:43:39 INFO SecurityManager: Changing view acls groups to:
> 20/06/26 20:43:39 INFO SecurityManager: Changing modify acls groups to:
> 20/06/26 20:43:39 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(maximgekk);
> groups with view permissions: Set(); users with modify permissions:
> Set(maximgekk); groups with modify permissions: Set()
> 20/06/26 20:43:39 INFO Utils: Successfully started service 'sparkDriver' on
> port 59414.
> 20/06/26 20:43:39 INFO SparkEnv: Registering MapOutputTracker
> 20/06/26 20:43:39 INFO SparkEnv: Registering BlockManagerMaster
> 20/06/26 20:43:39 INFO BlockManagerMasterEndpoint: Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information
> 20/06/26 20:43:39 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint
> up
> 20/06/26 20:43:39 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
> 20/06/26 20:43:39 INFO DiskBlockManager: Created local directory at
> /private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/blockmgr-c1d041ad-dd46-4d11-bbd0-e8ba27d3bf69
> 20/06/26 20:43:39 INFO MemoryStore: MemoryStore started with capacity 408.9
> MiB
> 20/06/26 20:43:39 INFO SparkEnv: Registering OutputCommitCoordinator
> 20/06/26 20:43:40 INFO Utils: Successfully started service 'SparkUI' on port
> 4040.
> 20/06/26 20:43:40 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at
> http://192.168.1.78:4040
> 20/06/26 20:43:40 INFO Executor: Starting executor ID driver on host
> 192.168.1.78
> 20/06/26 20:43:40 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 59415.
> 20/06/26 20:43:40 INFO NettyBlockTransferService: Server created on
> 192.168.1.78:59415
> 20/06/26 20:43:40 INFO BlockManager: Using
> org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
> policy
> 20/06/26 20:43:40 INFO BlockManagerMaster: Registering BlockManager
> BlockManagerId(driver, 192.168.1.78, 59415, None)
> 20/06/26 20:43:40 INFO BlockManagerMasterEndpoint: Registering block manager
> 192.168.1.78:59415 with 408.9 MiB RAM, BlockManagerId(driver, 192.168.1.78,
> 59415, None)
> 20/06/26 20:43:40 INFO BlockManagerMaster: Registered BlockManager
> BlockManagerId(driver, 192.168.1.78, 59415, None)
> 20/06/26 20:43:40 INFO BlockManager: Initialized BlockManager:
> BlockManagerId(driver, 192.168.1.78, 59415, None)
> 20/06/26 20:43:40 INFO SharedState: Setting hive.metastore.warehouse.dir
> ('null') to the value of spark.sql.warehouse.dir
> ('file:/Users/maximgekk/spark-warehouse/').
> 20/06/26 20:43:40 INFO SharedState: Warehouse path is
> 'file:/Users/maximgekk/spark-warehouse/'.
> 20/06/26 20:43:40 INFO HiveUtils: Initializing HiveMetastoreConnection
> version 2.3.7 using Spark classes.
> 20/06/26 20:43:40 INFO HiveClientImpl: Warehouse location for Hive client
> (version 2.3.7) is file:/Users/maximgekk/spark-warehouse/
> 20/06/26 20:43:41 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout
> does not exist
> 20/06/26 20:43:41 WARN HiveConf: HiveConf of name hive.stats.retries.wait
> does not exist
> 20/06/26 20:43:41 INFO HiveMetaStore: 0: Opening raw store with
> implementation class:org.apache.hadoop.hive.metastore.ObjectStore
> 20/06/26 20:43:41 INFO ObjectStore: ObjectStore, initialize called
> 20/06/26 20:43:41 INFO Persistence: Property
> hive.metastore.integral.jdo.pushdown unknown - will be ignored
> 20/06/26 20:43:41 INFO Persistence: Property datanucleus.cache.level2 unknown
> - will be ignored
> 20/06/26 20:43:42 INFO ObjectStore: Setting MetaStore object pin classes with
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 20/06/26 20:43:42 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is
> DERBY
> 20/06/26 20:43:42 INFO ObjectStore: Initialized ObjectStore
> 20/06/26 20:43:42 ERROR ObjectStore: Version information found in metastore
> differs 1.2.0 from expected schema version 2.3.0. Schema verififcation is
> disabled hive.metastore.schema.verification
> 20/06/26 20:43:42 WARN ObjectStore: setMetaStoreSchemaVersion called but
> recording version is disabled: version = 2.3.0, comment = Set by MetaStore
> [email protected]
> 20/06/26 20:43:42 INFO HiveMetaStore: Added admin role in metastore
> 20/06/26 20:43:43 INFO HiveMetaStore: Added public role in metastore
> 20/06/26 20:43:43 INFO HiveMetaStore: No user is added in admin role, since
> config is empty
> 20/06/26 20:43:43 INFO HiveMetaStore: 0: get_all_functions
> 20/06/26 20:43:43 INFO audit: ugi=maximgekk ip=unknown-ip-addr
> cmd=get_all_functions
> 20/06/26 20:43:43 INFO HiveMetaStore: 0: get_database: default
> 20/06/26 20:43:43 INFO audit: ugi=maximgekk ip=unknown-ip-addr
> cmd=get_database: default
> {code}
> but it should output logs up to WARN:
> https://github.com/apache/spark/blob/178ca961fe2b27a821b55d2231532804f1d8b68f/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala#L323
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
