juliuszsompolski commented on issue #25663: [SPARK-28910][SQL] Prevent schema verification when connecting to in memory derby URL: https://github.com/apache/spark/pull/25663#issuecomment-527613442 @wangyum this is similar to what you did in https://github.com/apache/spark/pull/24591/files#diff-6fd847124f8eae45ba2de1cf7d6296feR199, only that you guarded against changed defaults in Hive, while here we hit it when we enable schema verification on purpose - because we want it when interacting with the production metastore; but then setting it gets picked up by this in `HiveThriftserver2.startWithContext` -> `HiveUtils.newClientForExecution` and makes Thriftserver fail to start... `HiveUtils.newClientForExecution` seems to only be used by `HiveThriftServer2` during startup, and it's only used to take an initialized HiveConf from it, see https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala#L63. It seems overkill to create this Hive client, which will trigger initialization of a Derby metastore, just to get that HiveConf. And `newTemporaryConfiguration` is full of hacks to make it start and not interfere with metastore configs. This PR adds to these hacks, but do you think it would be possible to remove it all? Get rid of `HiveUtils.newClientForExecution` and `HiveUtils.newTemporaryConfiguration`, and have HiveThriftServer2 initialize the HiveConf it needs some other way, without initializing an actual client? That would remove a nasty hack. WDYT? Also cc @dongjoon-hyun
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
