GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/22351
[MINOR][SQL] Add a debug log when a SQL text is used for a view
## What changes were proposed in this pull request?
This took me a while to debug and find out. Looks we better at least leave
a debug log that SQL text for a view will be used.
Here's how I got there:
**Hive:**
```
CREATE TABLE emp AS SELECT 'user' AS name, 'address' as address;
CREATE DATABASE d100;
CREATE FUNCTION d100.udf100 AS
'org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper';
CREATE VIEW testview AS SELECT d100.udf100(name) FROM default.emp;
```
**Spark:**
```
sql("SELECT * FROM testview").show()
```
```
scala> sql("SELECT * FROM testview").show()
org.apache.spark.sql.AnalysisException: Undefined function: 'd100.udf100'.
This function is neither a registered temporary function nor a permanent
function registered in the database 'default'.; line 1 pos 7
```
Under the hood, it actually makes sense since the view is defined as
`SELECT d100.udf100(name) FROM default.emp;` and Hive API:
```
org.apache.hadoop.hive.ql.metadata.Table.getViewExpandedText()
```
This returns a fully qualified SQL string for the view as below:
```
SELECT `d100.udf100`(`emp`.`name`) FROM `default`.`emp`
```
which works fine in Hive but not in Spark.
## How was this patch tested?
Manually:
```
18/09/06 19:32:48 DEBUG HiveSessionCatalog: 'SELECT
`d100.udf100`(`emp`.`name`) FROM `default`.`emp`' will be used for the
view(testview).
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark minor-debug
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22351.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22351
----
commit 207d8dfb240ce8e283716e66bb62e7330a69fa8f
Author: hyukjinkwon <gurwls223@...>
Date: 2018-09-06T11:30:10Z
Add a debug log when a SQL text is used for a view
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
