Github user liufengdb commented on the issue:
https://github.com/apache/spark/pull/20864
I thought the directory is also created from this line:
https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java#L143.
For this one, we need to think about whether we can remove all the temp
directories creation, because the statements are executed by spark sql and it
has nothing about the Hive in the thrift server.
You are right that HiveClientImpl (the Hive inside spark sql) will also
produce such temp directories. However, it seems like the following line alone
is sufficient to add the jar to the class loader:
https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L836.
So I doubt we still need the `runSqlHive(s"ADD JAR $path")` to download the
jar to a temp directory.
Overall, I think we need an overall design to remove the Hive legacy in
both the thrift server and Spark SQL. Adding more temp fixes will make such a
design harder.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
