Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/19831#discussion_r154069687
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
---
@@ -418,7 +418,7 @@ private[hive] class HiveClientImpl(
// Note that this statistics could be overridden by Spark's
statistics if that's available.
val totalSize =
properties.get(StatsSetupConst.TOTAL_SIZE).map(BigInt(_))
val rawDataSize =
properties.get(StatsSetupConst.RAW_DATA_SIZE).map(BigInt(_))
- val rowCount =
properties.get(StatsSetupConst.ROW_COUNT).map(BigInt(_)).filter(_ >= 0)
+ val rowCount =
properties.get(StatsSetupConst.ROW_COUNT).map(BigInt(_)).filter(_ > 0)
--- End diff --
`StatsSetupConst.COLUMN_STATS_ACCURATE` to ensure that statistics have been
updated, but can not be guaranteed to be correct:
```bash
cat <<EOF > data
1,1
2,2
3,3
4,4
5,5
EOF
hive -e "CREATE TABLE spark_22626(c1 int, c2 int) ROW FORMAT DELIMITED
FIELDS TERMINATED BY ',';"
hive -e "LOAD DATA local inpath 'data' into table spark_22626;"
hive -e "INSERT INTO table spark_22626 values(6, 6);"
hive -e "desc extended spark_22626;"
```
The result is:
```
parameters:{totalSize=24, numRows=1, rawDataSize=3,
COLUMN_STATS_ACCURATE={"BASIC_STATS":"true"}
```
`numRows` should be 6, but got 1.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
