This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 514172a [SPARK-34074][SQL][TESTS][FOLLOWUP] Fix table size parsing from statistics 514172a is described below commit 514172aae79e6e7ed542870323193d4458f9b01a Author: Max Gekk <max.g...@gmail.com> AuthorDate: Tue Jan 19 09:51:20 2021 +0900 [SPARK-34074][SQL][TESTS][FOLLOWUP] Fix table size parsing from statistics ### What changes were proposed in this pull request? Fix table size parsing from the `Statistics` field which is formed at: https://github.com/apache/spark/blob/c3d81fbe79014f693cf93c02e31af401727761d7/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala#L573 . Before the fix, `getTableSize()` returns only the last digit. This works for Hive table in the tests because its size < 10 bytes, and accidentally works for V1 In-Memory catalog table in the tests. ### Why are the changes needed? This makes tests more reliable. For example, the parsing can not work in `AlterTableDropPartitionSuite` when table size before partition dropping: ``` +---------+ |data_type| +---------+ |878 bytes| +---------+ ``` After: ``` +---------+ |data_type| +---------+ |439 bytes| +---------+ ``` at: ```scala val onePartSize = getTableSize(t) assert(0 < onePartSize && onePartSize < twoPartSize) ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By existing test suites: ``` $ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *.AlterTableAddPartitionSuite" $ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *.AlterTableDropPartitionSuite" ``` Closes #31237 from MaxGekk/optimize-updateTableStats-followup. Authored-by: Max Gekk <max.g...@gmail.com> Signed-off-by: HyukjinKwon <gurwls...@apache.org> --- .../org/apache/spark/sql/execution/command/DDLCommandTestUtils.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandTestUtils.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandTestUtils.scala index 573c450..15f9305 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandTestUtils.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandTestUtils.scala @@ -101,7 +101,7 @@ trait DDLCommandTestUtils extends SQLTestUtils { if (stats.isEmpty) { throw new IllegalArgumentException(s"The table $tableName does not have stats") } - val tableSizeInStats = ".*(\\d) bytes.*".r + val tableSizeInStats = "^(\\d+) bytes.*$".r val size = stats.first().getString(0) match { case tableSizeInStats(s) => s.toInt case _ => throw new IllegalArgumentException("Not found table size in stats") --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org