juliuszsompolski commented on pull request #28544: URL: https://github.com/apache/spark/pull/28544#issuecomment-630147656
@dongjoon-hyun this PR passes "SPARK-29604 external listeners should be initialized with Spark classloader" in https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122763/testReport that was failing before in https://github.com/apache/spark/pull/28155. I have no idea how this change could have been related to that failure... ``` 20/05/17 20:19:47 INFO SparkSQLEnvSuite: ===== TEST OUTPUT FOR o.a.s.sql.hive.thriftserver.SparkSQLEnvSuite: 'SPARK-29604 external listeners should be initialized with Spark classloader' ===== SparkSQLEnvSuite: 20/05/17 20:19:47 INFO SparkContext: Running Spark version 3.1.0-SNAPSHOT 20/05/17 20:19:47 INFO ResourceUtils: ============================================================== 20/05/17 20:19:47 INFO ResourceUtils: No custom resources configured for spark.driver. 20/05/17 20:19:47 INFO ResourceUtils: ============================================================== 20/05/17 20:19:47 INFO SparkContext: Submitted application: testApp 20/05/17 20:19:47 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) 20/05/17 20:19:47 INFO ResourceProfile: Limiting resource is cpu 20/05/17 20:19:47 INFO ResourceProfileManager: Added ResourceProfile id: 0 20/05/17 20:19:47 INFO SecurityManager: Changing view acls to: jenkins 20/05/17 20:19:47 INFO SecurityManager: Changing modify acls to: jenkins 20/05/17 20:19:47 INFO SecurityManager: Changing view acls groups to: 20/05/17 20:19:47 INFO SecurityManager: Changing modify acls groups to: 20/05/17 20:19:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); groups with view permissions: Set(); users with modify permissions: Set(jenkins); groups with modify permissions: Set() 20/05/17 20:19:47 INFO Utils: Successfully started service 'sparkDriver' on port 37165. 20/05/17 20:19:47 INFO SparkEnv: Registering MapOutputTracker 20/05/17 20:19:47 INFO SparkEnv: Registering BlockManagerMaster 20/05/17 20:19:47 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 20/05/17 20:19:47 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 20/05/17 20:19:47 INFO SparkEnv: Registering BlockManagerMasterHeartbeat 20/05/17 20:19:47 INFO DiskBlockManager: Created local directory at /home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/blockmgr-1d9a2431-0019-41bb-9fbe-c8ca6dd6bc1f 20/05/17 20:19:47 INFO MemoryStore: MemoryStore started with capacity 2.1 GiB 20/05/17 20:19:47 INFO SparkEnv: Registering OutputCommitCoordinator 20/05/17 20:19:47 INFO Executor: Starting executor ID driver on host amp-jenkins-worker-03.amp 20/05/17 20:19:47 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40300. 20/05/17 20:19:47 INFO NettyBlockTransferService: Server created on amp-jenkins-worker-03.amp:40300 20/05/17 20:19:47 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 20/05/17 20:19:47 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, amp-jenkins-worker-03.amp, 40300, None) 20/05/17 20:19:47 INFO BlockManagerMasterEndpoint: Registering block manager amp-jenkins-worker-03.amp:40300 with 2.1 GiB RAM, BlockManagerId(driver, amp-jenkins-worker-03.amp, 40300, None) 20/05/17 20:19:47 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, amp-jenkins-worker-03.amp, 40300, None) 20/05/17 20:19:47 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, amp-jenkins-worker-03.amp, 40300, None) 20/05/17 20:19:47 INFO SharedState: loading hive config file: file:/home/jenkins/workspace/SparkPullRequestBuilder/sql/hive/target/scala-2.12/test-classes/hive-site.xml 20/05/17 20:19:47 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('/home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/warehouse-49d39f38-dc7f-4a52-93cd-782c0d1231cd'). 20/05/17 20:19:47 INFO SharedState: Warehouse path is '/home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/warehouse-49d39f38-dc7f-4a52-93cd-782c0d1231cd'. 20/05/17 20:19:47 WARN SharedState: Not allowing to set spark.sql.warehouse.dir or hive.metastore.warehouse.dir in SparkSession's options, it should be set statically for cross-session usages 20/05/17 20:19:47 INFO HiveUtils: Initializing HiveMetastoreConnection version 2.3.7 using maven. 20/05/17 20:19:53 INFO IsolatedClientLoader: Downloaded metastore jars to /home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/hive-v2_3-a7e1202d-698a-456d-8609-2859a43dfba9 20/05/17 20:19:54 INFO HiveConf: Found configuration file null 20/05/17 20:19:54 INFO SessionState: Created HDFS directory: /tmp/hive/jenkins/eb4b5224-dc13-48e3-8b2b-e1fa57f042e0 20/05/17 20:19:54 INFO SessionState: Created local directory: /home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/jenkins/eb4b5224-dc13-48e3-8b2b-e1fa57f042e0 20/05/17 20:19:54 INFO SessionState: Created HDFS directory: /tmp/hive/jenkins/eb4b5224-dc13-48e3-8b2b-e1fa57f042e0/_tmp_space.db 20/05/17 20:19:54 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is /home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/warehouse-49d39f38-dc7f-4a52-93cd-782c0d1231cd 20/05/17 20:19:55 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist 20/05/17 20:19:55 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist 20/05/17 20:19:55 INFO HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 20/05/17 20:19:55 INFO ObjectStore: ObjectStore, initialize called 20/05/17 20:19:55 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 20/05/17 20:19:55 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored 20/05/17 20:19:57 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 20/05/17 20:19:59 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY 20/05/17 20:20:00 INFO ObjectStore: Initialized ObjectStore 20/05/17 20:20:00 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0 20/05/17 20:20:00 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore [email protected] 20/05/17 20:20:00 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException 20/05/17 20:20:00 INFO HiveMetaStore: Added admin role in metastore 20/05/17 20:20:00 INFO HiveMetaStore: Added public role in metastore 20/05/17 20:20:00 INFO HiveMetaStore: No user is added in admin role, since config is empty 20/05/17 20:20:00 INFO HiveMetaStore: 0: get_all_functions 20/05/17 20:20:00 INFO audit: ugi=jenkins ip=unknown-ip-addr cmd=get_all_functions 20/05/17 20:20:00 INFO HiveMetaStore: 0: get_database: default 20/05/17 20:20:00 INFO audit: ugi=jenkins ip=unknown-ip-addr cmd=get_database: default 20/05/17 20:20:00 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint 20/05/17 20:20:00 INFO StreamingQueryManager: Registered listener test.custom.listener.DummyStreamingQueryListener 20/05/17 20:20:00 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 20/05/17 20:20:00 INFO MemoryStore: MemoryStore cleared 20/05/17 20:20:00 INFO BlockManager: BlockManager stopped 20/05/17 20:20:00 INFO BlockManagerMaster: BlockManagerMaster stopped 20/05/17 20:20:00 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 20/05/17 20:20:00 INFO SparkContext: Successfully stopped SparkContext 20/05/17 20:20:00 INFO SparkSQLEnvSuite: ===== FINISHED o.a.s.sql.hive.thriftserver.SparkSQLEnvSuite: 'SPARK-29604 external listeners should be initialized with Spark classloader' ===== - SPARK-29604 external listeners should be initialized with Spark classloader ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
