[GitHub] [spark] wangyum commented on a diff in pull request #36625: [SPARK-39203][SQL] Rewrite table location to absolute location based on database location

Fri, 27 May 2022 00:34:35 -0700 


wangyum commented on code in PR #36625:
URL: https://github.com/apache/spark/pull/36625#discussion_r883351910


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -518,7 +519,15 @@ private[hive] class HiveClientImpl(
       createTime = h.getTTable.getCreateTime.toLong * 1000,
       lastAccessTime = h.getLastAccessTime.toLong * 1000,
       storage = CatalogStorageFormat(
-        locationUri = shim.getDataLocation(h).map(CatalogUtils.stringToURI),
+        locationUri = shim.getDataLocation(h).map { loc =>
+          val tableUri = CatalogUtils.stringToURI(loc)
+          val parentUri = if (tableUri.isAbsolute) {

Review Comment:
   <img width="582" alt="image" 
src="https://user-images.githubusercontent.com/5399861/170652237-ab9d2fc9-e9ce-4b34-be76-8e0ca8814722.png";>
   and the other command:
   
   ```
   spark-sql>
            > CREATE TABLE DB_SPARK_39203.t7(id int) using parquet;
   Time taken: 0.43 seconds
   spark-sql>
            > CREATE TABLE DB_SPARK_39203.t8(id int)
            > stored as parquet
            > LOCATION '/user/hive/warehouse/db_spark_39203.db/t1';
   Time taken: 0.265 seconds
   spark-sql>
            > CREATE EXTERNAL TABLE DB_SPARK_39203.t9(id int)
            > stored as parquet
            > LOCATION '/user/hive/warehouse/db_spark_39203.db/t1';
   Time taken: 0.278 seconds
   spark-sql> desc formatted DB_SPARK_39203.t7;
   id   int     NULL
   
   # Detailed Table Information
   Database     db_spark_39203
   Table        t7
   Owner        ####
   Created Time Fri May 27 00:31:09 GMT-07:00 2022
   Last Access  UNKNOWN
   Created By   Spark 3.2.0-SNAPSHOT
   Type MANAGED
   Provider     parquet
   Location     viewfs://####/user/hive/warehouse/db_spark_39203.db/t7
   Serde Library        
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
   InputFormat  org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
   OutputFormat org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
   Time taken: 0.152 seconds, Fetched 15 row(s)
   spark-sql> desc formatted DB_SPARK_39203.t8;
   id   int     NULL
   
   # Detailed Table Information
   Database     db_spark_39203
   Table        t8
   Owner        ####
   Created Time Fri May 27 00:31:09 GMT-07:00 2022
   Last Access  UNKNOWN
   Created By   Spark 3.2.0-SNAPSHOT
   Type EXTERNAL
   Provider     hive
   Table Properties     [transient_lastDdlTime=1653636669]
   Location     viewfs://####/user/hive/warehouse/db_spark_39203.db/t1
   Serde Library        
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
   InputFormat  org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
   OutputFormat org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
   Storage Properties   [serialization.format=1]
   Partition Provider   Catalog
   Time taken: 0.224 seconds, Fetched 18 row(s)
   spark-sql> desc formatted DB_SPARK_39203.t9;
   id   int     NULL
   
   # Detailed Table Information
   Database     db_spark_39203
   Table        t9
   Owner        ####
   Created Time Fri May 27 00:31:10 GMT-07:00 2022
   Last Access  UNKNOWN
   Created By   Spark 3.2.0-SNAPSHOT
   Type EXTERNAL
   Provider     hive
   Table Properties     [transient_lastDdlTime=1653636670]
   Location     viewfs://####/user/hive/warehouse/db_spark_39203.db/t1
   Serde Library        
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
   InputFormat  org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
   OutputFormat org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
   Storage Properties   [serialization.format=1]
   Partition Provider   Catalog
   Time taken: 0.142 seconds, Fetched 18 row(s)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to