[
https://issues.apache.org/jira/browse/SPARK-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhenhua Wang updated SPARK-20881:
---------------------------------
Description:
Currently in Spark, statistics are generated by "analyze" commands.
When user updates the table and collects stats in Hive, "totalSize"/"numRows"
will be updated in metastore.
Then, in spark side, table stats becomes stale and is different from Hive's
stats.
This is expected. Currently, we have two sources of statistics, i.e. Spark's
stats and Hive's stats. In our design, once Spark's stats is available, we
respect it over Hive's stats.
If a user generated stats at Spark side, it's his responsibility to update
Spark's stats by re-running analyze commands.
But we should clearly document in related code the mechanism to choose between
these two sources of stats.
was:
Currently statistics are generated by "analyze command" in Spark.
However, when user updates the table and collects stats in Hive,
"totalSize"/"numRows" will be updated in metastore.
Now, in spark side, table stats become stale.
If cbo is enabled, this is ok because we suppose user will handle this and
re-run the command to update stats.
If cbo is disabled, then we should fallback to original way and respect hive's
stats. But in current implementation, spark's stats always override hive's
stats, no matter cbo is enabled or disabled.
The right thing to do is to use (don't override) hive's stats when cbo is
disabled.
> Clearly document the mechanism to choose between two sources of statistics
> --------------------------------------------------------------------------
>
> Key: SPARK-20881
> URL: https://issues.apache.org/jira/browse/SPARK-20881
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Zhenhua Wang
>
> Currently in Spark, statistics are generated by "analyze" commands.
> When user updates the table and collects stats in Hive, "totalSize"/"numRows"
> will be updated in metastore.
> Then, in spark side, table stats becomes stale and is different from Hive's
> stats.
> This is expected. Currently, we have two sources of statistics, i.e. Spark's
> stats and Hive's stats. In our design, once Spark's stats is available, we
> respect it over Hive's stats.
> If a user generated stats at Spark side, it's his responsibility to update
> Spark's stats by re-running analyze commands.
> But we should clearly document in related code the mechanism to choose
> between these two sources of stats.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
