[
https://issues.apache.org/jira/browse/SPARK-50461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhukar updated SPARK-50461:
-----------------------------
Description:
Consider the changes done under HIVE-22995, for HMS4 support.
Wiki -
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ManagedandExternalTables]
Behaviour change - With HIVE4 , MANAGEDLOCATION has been introduced with
database. And LOCATION now refers to the path of external table.
So, when a database is created using sparkSQL, it dont have managed location
reference and all the tables created inside it is of external table type.
Impacts functionality like TRUNC, .....
Reference logs -
metastore.MetastoreDefaultTransformer
(MetastoreDefaultTransformer.java:transformDatabase(811)) - Starting
translation for transformDatabase for processor null with null on database
truncate_insert_overwrite
locationUri=hdfs://madzilla/warehouse/tablespace/managed/hive/truncate_insert_overwrite.db
{*}managedLocationUri=null{*}{*}{*}2024-11-27T15:42:20,320 INFO
[Metastore-Handler-Pool: Thread-3187]: ** metastore.MetastoreDefaultTransformer
(MetastoreDefaultTransformer.java:transformDatabase(827)) - Database's location
is a managed location, setting to a new default path based on external
warehouse
path:hdfs://madzilla/warehouse/tablespace/external/hive/truncate_insert_overwrite.db
Results in external table type.
!image-2024-11-29-20-12-52-808.png!
was:
Consider the changes done under HIVE-22995, for HMS4 support.
Wiki -
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ManagedandExternalTables]
Behaviour change - With HIVE4 , MANAGEDLOCATION has been introduced with
database. And LOCATION now refers to the path of external table.
So, when a database is created using sparkSQL, it dont have managed location
reference and all the tables created inside it is of external table type.
Impacts functionality like TRUNC, .....
Reference logs -
metastore.MetastoreDefaultTransformer
(MetastoreDefaultTransformer.java:transformDatabase(811)) - Starting
translation for transformDatabase for processor null with null on database
truncate_insert_overwrite
locationUri=hdfs://madzilla/warehouse/tablespace/managed/hive/truncate_insert_overwrite.db
*managedLocationUri=null*
Results in external table type.
> Add support for location for managed tables on database
> -------------------------------------------------------
>
> Key: SPARK-50461
> URL: https://issues.apache.org/jira/browse/SPARK-50461
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Madhukar
> Priority: Major
> Attachments: image-2024-11-29-20-12-52-808.png
>
>
> Consider the changes done under HIVE-22995, for HMS4 support.
> Wiki -
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ManagedandExternalTables]
>
> Behaviour change - With HIVE4 , MANAGEDLOCATION has been introduced with
> database. And LOCATION now refers to the path of external table.
>
> So, when a database is created using sparkSQL, it dont have managed location
> reference and all the tables created inside it is of external table type.
> Impacts functionality like TRUNC, .....
>
> Reference logs -
> metastore.MetastoreDefaultTransformer
> (MetastoreDefaultTransformer.java:transformDatabase(811)) - Starting
> translation for transformDatabase for processor null with null on database
> truncate_insert_overwrite
> locationUri=hdfs://madzilla/warehouse/tablespace/managed/hive/truncate_insert_overwrite.db
> {*}managedLocationUri=null{*}{*}{*}2024-11-27T15:42:20,320 INFO
> [Metastore-Handler-Pool: Thread-3187]: **
> metastore.MetastoreDefaultTransformer
> (MetastoreDefaultTransformer.java:transformDatabase(827)) - Database's
> location is a managed location, setting to a new default path based on
> external warehouse
> path:hdfs://madzilla/warehouse/tablespace/external/hive/truncate_insert_overwrite.db
>
> Results in external table type.
> !image-2024-11-29-20-12-52-808.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
