[
https://issues.apache.org/jira/browse/SPARK-40708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-40708:
------------------------------------
Assignee: Apache Spark
> Auto update table statistics based on write metrics
> ---------------------------------------------------
>
> Key: SPARK-40708
> URL: https://issues.apache.org/jira/browse/SPARK-40708
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Yuming Wang
> Assignee: Apache Spark
> Priority: Major
>
> {code:scala}
> // Get write statistics
> def getWriteStats(mode: SaveMode, metrics: Map[String, SQLMetric]):
> Option[WriteStats] = {
> val numBytes =
> metrics.get(NUM_OUTPUT_BYTES_KEY).map(_.value).map(BigInt(_))
> val numRows = metrics.get(NUM_OUTPUT_ROWS_KEY).map(_.value).map(BigInt(_))
> numBytes.map(WriteStats(mode, _, numRows))
> }
> // Update table statistics
> val stat = wroteStats.get
> stat.mode match {
> case SaveMode.Overwrite | SaveMode.ErrorIfExists =>
> catalog.alterTableStats(table.identifier,
> Some(CatalogStatistics(stat.numBytes, stat.numRows)))
> case _ if table.stats.nonEmpty => // SaveMode.Append
> catalog.alterTableStats(table.identifier, None)
> case _ => // SaveMode.Ignore Do nothing
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
