[
https://issues.apache.org/jira/browse/SPARK-21428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-21428:
------------------------------------
Assignee: Apache Spark
> CliSessionState never be recognized because of IsolatedClientLoader
> -------------------------------------------------------------------
>
> Key: SPARK-21428
> URL: https://issues.apache.org/jira/browse/SPARK-21428
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.2, 1.6.3, 2.0.2, 2.1.1, 2.2.0
> Reporter: Kent Yao
> Assignee: Apache Spark
> Priority: Minor
>
> When using bin/spark-sql with the builtin hive jars, we are expecting to
> reuse the instance of CliSessionState.
> {quote}
> // In `SparkSQLCLIDriver`, we have already started a
> `CliSessionState`,
> // which contains information like configurations from command line.
> Later
> // we call `SparkSQLEnv.init()` there, which would run into this part
> again.
> // so we should keep `conf` and reuse the existing instance of
> `CliSessionState`.
> {quote}
> Actually it never ever happened since SessionState.get() at
> https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L138
> will always be null by IsolatedClientLoader.
> The SessionState.start was called many times, which will creates
> `hive.exec.strachdir`, see the following case...
> {code:java}
> spark git:(master) bin/spark-sql --conf spark.sql.hive.metastore.jars=builtin
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 17/07/16 23:29:04 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 17/07/16 23:29:04 INFO HiveMetaStore: 0: Opening raw store with implemenation
> class:org.apache.hadoop.hive.metastore.ObjectStore
> 17/07/16 23:29:04 INFO ObjectStore: ObjectStore, initialize called
> 17/07/16 23:29:04 INFO Persistence: Property
> hive.metastore.integral.jdo.pushdown unknown - will be ignored
> 17/07/16 23:29:04 INFO Persistence: Property datanucleus.cache.level2 unknown
> - will be ignored
> 17/07/16 23:29:05 INFO ObjectStore: Setting MetaStore object pin classes with
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 17/07/16 23:29:06 INFO Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
> "embedded-only" so does not have its own datastore table.
> 17/07/16 23:29:06 INFO Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only"
> so does not have its own datastore table.
> 17/07/16 23:29:07 INFO Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
> "embedded-only" so does not have its own datastore table.
> 17/07/16 23:29:07 INFO Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only"
> so does not have its own datastore table.
> 17/07/16 23:29:07 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is
> DERBY
> 17/07/16 23:29:07 INFO ObjectStore: Initialized ObjectStore
> 17/07/16 23:29:07 WARN ObjectStore: Version information not found in
> metastore. hive.metastore.schema.verification is not enabled so recording the
> schema version 1.2.0
> 17/07/16 23:29:07 WARN ObjectStore: Failed to get database default, returning
> NoSuchObjectException
> 17/07/16 23:29:08 INFO HiveMetaStore: Added admin role in metastore
> 17/07/16 23:29:08 INFO HiveMetaStore: Added public role in metastore
> 17/07/16 23:29:08 INFO HiveMetaStore: No user is added in admin role, since
> config is empty
> 17/07/16 23:29:08 INFO HiveMetaStore: 0: get_all_databases
> 17/07/16 23:29:08 INFO audit: ugi=Kent ip=unknown-ip-addr
> cmd=get_all_databases
> 17/07/16 23:29:08 INFO HiveMetaStore: 0: get_functions: db=default pat=*
> 17/07/16 23:29:08 INFO audit: ugi=Kent ip=unknown-ip-addr
> cmd=get_functions: db=default pat=*
> 17/07/16 23:29:08 INFO Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as
> "embedded-only" so does not have its own datastore table.
> 17/07/16 23:29:08 INFO SessionState: Created local directory:
> /var/folders/k2/04p4k4ws73l6711h_mz2_tq00000gn/T/a2c40e42-08e2-4023-8464-3432ed690184_resources
> 17/07/16 23:29:08 INFO SessionState: Created HDFS directory:
> /tmp/hive/Kent/a2c40e42-08e2-4023-8464-3432ed690184
> 17/07/16 23:29:08 INFO SessionState: Created local directory:
> /var/folders/k2/04p4k4ws73l6711h_mz2_tq00000gn/T/Kent/a2c40e42-08e2-4023-8464-3432ed690184
> 17/07/16 23:29:08 INFO SessionState: Created HDFS directory:
> /tmp/hive/Kent/a2c40e42-08e2-4023-8464-3432ed690184/_tmp_space.db
> 17/07/16 23:29:08 INFO SparkContext: Running Spark version 2.3.0-SNAPSHOT
> 17/07/16 23:29:08 INFO SparkContext: Submitted application: SparkSQL::10.0.0.8
> 17/07/16 23:29:08 INFO SecurityManager: Changing view acls to: Kent
> 17/07/16 23:29:08 INFO SecurityManager: Changing modify acls to: Kent
> 17/07/16 23:29:08 INFO SecurityManager: Changing view acls groups to:
> 17/07/16 23:29:08 INFO SecurityManager: Changing modify acls groups to:
> 17/07/16 23:29:08 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(Kent); groups
> with view permissions: Set(); users with modify permissions: Set(Kent);
> groups with modify permissions: Set()
> 17/07/16 23:29:09 INFO Utils: Successfully started service 'sparkDriver' on
> port 51369.
> 17/07/16 23:29:09 INFO SparkEnv: Registering MapOutputTracker
> 17/07/16 23:29:09 INFO SparkEnv: Registering BlockManagerMaster
> 17/07/16 23:29:09 INFO BlockManagerMasterEndpoint: Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information
> 17/07/16 23:29:09 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint
> up
> 17/07/16 23:29:09 INFO DiskBlockManager: Created local directory at
> /private/var/folders/k2/04p4k4ws73l6711h_mz2_tq00000gn/T/blockmgr-8bf36015-6c27-4b4b-b90b-126f7a59b5a0
> 17/07/16 23:29:09 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
> 17/07/16 23:29:09 INFO SparkEnv: Registering OutputCommitCoordinator
> 17/07/16 23:29:09 INFO Utils: Successfully started service 'SparkUI' on port
> 4040.
> 17/07/16 23:29:09 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at
> http://10.0.0.8:4040
> 17/07/16 23:29:09 INFO Executor: Starting executor ID driver on host localhost
> 17/07/16 23:29:09 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 51371.
> 17/07/16 23:29:09 INFO NettyBlockTransferService: Server created on
> 10.0.0.8:51371
> 17/07/16 23:29:09 INFO BlockManager: Using
> org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
> policy
> 17/07/16 23:29:09 INFO BlockManagerMaster: Registering BlockManager
> BlockManagerId(driver, 10.0.0.8, 51371, None)
> 17/07/16 23:29:09 INFO BlockManagerMasterEndpoint: Registering block manager
> 10.0.0.8:51371 with 366.3 MB RAM, BlockManagerId(driver, 10.0.0.8, 51371,
> None)
> 17/07/16 23:29:09 INFO BlockManagerMaster: Registered BlockManager
> BlockManagerId(driver, 10.0.0.8, 51371, None)
> 17/07/16 23:29:09 INFO BlockManager: Initialized BlockManager:
> BlockManagerId(driver, 10.0.0.8, 51371, None)
> 17/07/16 23:29:10 INFO SharedState: Setting hive.metastore.warehouse.dir
> ('null') to the value of spark.sql.warehouse.dir
> ('file:/Users/Kent/Documents/spark/spark-warehouse').
> 17/07/16 23:29:10 INFO SharedState: Warehouse path is
> 'file:/Users/Kent/Documents/spark/spark-warehouse'.
> 17/07/16 23:29:10 INFO HiveUtils: Initializing HiveMetastoreConnection
> version 1.2.1 using Spark classes.
> 17/07/16 23:29:10 INFO HiveMetaStore: 0: Opening raw store with implemenation
> class:org.apache.hadoop.hive.metastore.ObjectStore
> 17/07/16 23:29:10 INFO ObjectStore: ObjectStore, initialize called
> 17/07/16 23:29:11 INFO Persistence: Property
> hive.metastore.integral.jdo.pushdown unknown - will be ignored
> 17/07/16 23:29:11 INFO Persistence: Property datanucleus.cache.level2 unknown
> - will be ignored
> 17/07/16 23:29:12 INFO ObjectStore: Setting MetaStore object pin classes with
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 17/07/16 23:29:13 INFO Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
> "embedded-only" so does not have its own datastore table.
> 17/07/16 23:29:13 INFO Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only"
> so does not have its own datastore table.
> 17/07/16 23:29:13 INFO Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
> "embedded-only" so does not have its own datastore table.
> 17/07/16 23:29:13 INFO Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only"
> so does not have its own datastore table.
> 17/07/16 23:29:13 INFO Query: Reading in results for query
> "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is
> closing
> 17/07/16 23:29:13 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is
> DERBY
> 17/07/16 23:29:13 INFO ObjectStore: Initialized ObjectStore
> 17/07/16 23:29:13 INFO HiveMetaStore: Added admin role in metastore
> 17/07/16 23:29:13 INFO HiveMetaStore: Added public role in metastore
> 17/07/16 23:29:13 INFO HiveMetaStore: No user is added in admin role, since
> config is empty
> 17/07/16 23:29:13 INFO HiveMetaStore: 0: get_all_databases
> 17/07/16 23:29:13 INFO audit: ugi=Kent ip=unknown-ip-addr
> cmd=get_all_databases
> 17/07/16 23:29:13 INFO HiveMetaStore: 0: get_functions: db=default pat=*
> 17/07/16 23:29:13 INFO audit: ugi=Kent ip=unknown-ip-addr
> cmd=get_functions: db=default pat=*
> 17/07/16 23:29:13 INFO Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as
> "embedded-only" so does not have its own datastore table.
> 17/07/16 23:29:13 INFO SessionState: Created local directory:
> /var/folders/k2/04p4k4ws73l6711h_mz2_tq00000gn/T/27f3c8f8-38d1-4b64-9ade-b41a07937937_resources
> 17/07/16 23:29:13 INFO SessionState: Created HDFS directory:
> /tmp/hive/Kent/27f3c8f8-38d1-4b64-9ade-b41a07937937
> 17/07/16 23:29:13 INFO SessionState: Created local directory:
> /var/folders/k2/04p4k4ws73l6711h_mz2_tq00000gn/T/Kent/27f3c8f8-38d1-4b64-9ade-b41a07937937
> 17/07/16 23:29:13 INFO SessionState: Created HDFS directory:
> /tmp/hive/Kent/27f3c8f8-38d1-4b64-9ade-b41a07937937/_tmp_space.db
> 17/07/16 23:29:13 INFO HiveClientImpl: Warehouse location for Hive client
> (version 1.2.2) is file:/Users/Kent/Documents/spark/spark-warehouse
> 17/07/16 23:29:13 INFO HiveMetaStore: 0: get_database: default
> 17/07/16 23:29:13 INFO audit: ugi=Kent ip=unknown-ip-addr
> cmd=get_database: default
> 17/07/16 23:29:14 INFO SessionState: Created local directory:
> /var/folders/k2/04p4k4ws73l6711h_mz2_tq00000gn/T/4b589938-6df6-4a9f-b044-99da6b1c76b4_resources
> 17/07/16 23:29:14 INFO SessionState: Created HDFS directory:
> /tmp/hive/Kent/4b589938-6df6-4a9f-b044-99da6b1c76b4
> 17/07/16 23:29:14 INFO SessionState: Created local directory:
> /var/folders/k2/04p4k4ws73l6711h_mz2_tq00000gn/T/Kent/4b589938-6df6-4a9f-b044-99da6b1c76b4
> 17/07/16 23:29:14 INFO SessionState: Created HDFS directory:
> /tmp/hive/Kent/4b589938-6df6-4a9f-b044-99da6b1c76b4/_tmp_space.db
> 17/07/16 23:29:14 INFO HiveClientImpl: Warehouse location for Hive client
> (version 1.2.2) is file:/Users/Kent/Documents/spark/spark-warehouse
> 17/07/16 23:29:14 INFO HiveMetaStore: 0: get_database: global_temp
> 17/07/16 23:29:14 INFO audit: ugi=Kent ip=unknown-ip-addr
> cmd=get_database: global_temp
> 17/07/16 23:29:14 WARN ObjectStore: Failed to get database global_temp,
> returning NoSuchObjectException
> 17/07/16 23:29:14 INFO SessionState: Created local directory:
> /var/folders/k2/04p4k4ws73l6711h_mz2_tq00000gn/T/1317e8bf-5a10-4758-bd56-3196ad9ae95e_resources
> 17/07/16 23:29:14 INFO SessionState: Created HDFS directory:
> /tmp/hive/Kent/1317e8bf-5a10-4758-bd56-3196ad9ae95e
> 17/07/16 23:29:14 INFO SessionState: Created local directory:
> /var/folders/k2/04p4k4ws73l6711h_mz2_tq00000gn/T/Kent/1317e8bf-5a10-4758-bd56-3196ad9ae95e
> 17/07/16 23:29:14 INFO SessionState: Created HDFS directory:
> /tmp/hive/Kent/1317e8bf-5a10-4758-bd56-3196ad9ae95e/_tmp_space.db
> 17/07/16 23:29:14 INFO HiveClientImpl: Warehouse location for Hive client
> (version 1.2.2) is file:/Users/Kent/Documents/spark/spark-warehouse
> 17/07/16 23:29:14 INFO StateStoreCoordinatorRef: Registered
> StateStoreCoordinator endpoint
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
