[
https://issues.apache.org/jira/browse/SPARK-47444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szurap updated SPARK-47444:
----------------------------------
Attachment: (was: reproduction_steps_SPARK-47444.txt)
> Empty numRows table stats should not break Hive tables
> ------------------------------------------------------
>
> Key: SPARK-47444
> URL: https://issues.apache.org/jira/browse/SPARK-47444
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.8
> Reporter: Miklos Szurap
> Priority: Major
> Labels: Hive, HiveMetaStoreClient, SQL
>
> SPARK-30262 resolved/avoided the NumberFormatException in Spark when the
> "totalSize", "numRows", "rawDataSize" table properties are empty, however the
> table stats (intentionally or by mistake) can be still set to an
> invalid/empty value through SparkSQL with an ALTER TABLE statement:
> {code}
> scala> spark.sql("alter table t1p set tblproperties('numRows'='',
> 'STATS_GENERATED_VIA_STATS_TASK'='true')").show()
> {code}
>
> Spark should validate the sparkSQL "alter table" statements and not allow
> non-numeric values in the "totalSize", "numRows", "rawDataSize" table
> properties.
> Though the NumberFormatException will not occur anymore in Spark 3.x, these
> table stats should have numeric values and may cause problems in other
> applications if those are not numbers.
> Note: beeline/Hive validates alter table statements.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
