wombatu-kun opened a new issue, #11858:
URL: https://github.com/apache/hudi/issues/11858
**Describe the problem you faced**
If `hoodie.schema.on.read.enable` is set to **true** in spark-sql then
`describe table` command provides much less information about the table.
Describe's output doesn't contain a bunch of useful properties such as:
Location, Created Time, Created by, SerDe Library, InputFormat, OutputFormat.
Is there any way to fix it easier than creating our own implementation of
spark's native `desc` command?
**To Reproduce**
Here is the test case (`extends HoodieSparkSqlTestBase`):
```
test("Test describe table") {
withTempDir { tmp =>
val tbName = "wk_date"
val basePath = s"${tmp.getCanonicalPath}/$tbName"
spark.sql(
s"""
|create table $tbName (id int, driver string, precomb int, dat
string)
| using hudi
| partitioned by(dat)
| tblproperties(type='cow', primaryKey='id',
preCombineField='precomb')
| location '$basePath'
""".stripMargin)
spark.sql("set hoodie.schema.on.read.enable=false")
spark.sql(s"desc formatted $tbName").show(100)
spark.sql("set hoodie.schema.on.read.enable=true")
spark.sql(s"desc formatted $tbName").show(100)
}
}
```
And it's output:
```
/*hoodie.schema.on.read.enable=false*/
+--------------------+--------------------+-------+
| col_name| data_type|comment|
+--------------------+--------------------+-------+
| _hoodie_commit_time| string| NULL|
|_hoodie_commit_seqno| string| NULL|
| _hoodie_record_key| string| NULL|
|_hoodie_partition...| string| NULL|
| _hoodie_file_name| string| NULL|
| id| int| NULL|
| driver| string| NULL|
| precomb| int| NULL|
| dat| string| NULL|
|# Partition Infor...| | |
| # col_name| data_type|comment|
| dat| string| NULL|
| | | |
|# Detailed Table ...| | |
| Catalog| spark_catalog| |
| Database| default| |
| Table| wk_date| |
| Created Time|Thu Aug 29 06:13:...| |
| Last Access| UNKNOWN| |
| Created By| Spark 3.5.1| |
| Type| EXTERNAL| |
| Provider| hudi| |
| Table Properties|[path=file:///tmp...| |
| Location|file:///tmp/spark...| |
| Serde Library|org.apache.hadoop...| |
| InputFormat|org.apache.hudi.h...| |
| OutputFormat|org.apache.hadoop...| |
| Storage Properties|[path=file:///tmp...| |
+--------------------+--------------------+-------+
/*hoodie.schema.on.read.enable=true*/
+--------------------+--------------------+-------+
| col_name| data_type|comment|
+--------------------+--------------------+-------+
| _hoodie_commit_time| string| NULL|
|_hoodie_commit_seqno| string| NULL|
| _hoodie_record_key| string| NULL|
|_hoodie_partition...| string| NULL|
| _hoodie_file_name| string| NULL|
| id| int| NULL|
| driver| string| NULL|
| precomb| int| NULL|
| dat| string| NULL|
|# Partition Infor...| | |
| # col_name| data_type|comment|
| dat| string| NULL|
| | | |
|# Detailed Table ...| | |
| Name|spark_catalog.def...| |
| Type| MANAGED| |
| Provider| hudi| |
| Table Properties|[path=file:///tmp...| |
+--------------------+--------------------+-------+
```
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : current master
* Spark version : 3.5
* Hive version : -
* Hadoop version : -
* Storage (HDFS/S3/GCS..) : -
* Running on Docker? (yes/no) : no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
