This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new e2f82fdaeba [SPARK-39019][TESTS] Use `withTempPath` to clean up
temporary data directory after `SPARK-37463: read/write Timestamp ntz to Orc
with different time zone`
e2f82fdaeba is described below
commit e2f82fdaeba8d56cf6fa2f75d6d52b5b4f9da4ea
Author: yangjie01 <[email protected]>
AuthorDate: Tue Apr 26 15:52:22 2022 -0700
[SPARK-39019][TESTS] Use `withTempPath` to clean up temporary data
directory after `SPARK-37463: read/write Timestamp ntz to Orc with different
time zone`
### What changes were proposed in this pull request?
`SPARK-37463: read/write Timestamp ntz to Orc with different time zone` use
the absolute path to save the test data, and does not clean up the test data
after the test.
This pr change to use `withTempPath` to ensure the data directory is
cleaned up after testing.
### Why are the changes needed?
Clean up the temporary data directory after test.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Pass GA
- Manual test：
Run
```
mvn clean install -pl sql/core -am -DskipTests
mvn clean test -pl sql/core -Dtest=none
-DwildcardSuites=org.apache.spark.sql.execution.datasources.orc.OrcV1QuerySuite
git status
```
**Before**
```
sql/core/ts_ntz_orc/
ls sql/core/ts_ntz_orc
_SUCCESS
part-00000-9523e257-5024-4980-8bb3-12070222b0bd-c000.snappy.orc
```
**After**
No residual `sql/core/ts_ntz_orc/`
Closes #36352 from LuciferYang/SPARK-39019.
Authored-by: yangjie01 <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 97449d23a3b2232e14e63c6645919c5d93e4491c)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../sql/execution/datasources/orc/OrcQuerySuite.scala | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala
index f093a5fc735..e44fb63c757 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala
@@ -814,16 +814,19 @@ abstract class OrcQuerySuite extends OrcQueryTest with
SharedSparkSession {
| timestamp_ntz '2021-03-14 02:15:00.0' as ts_ntz3
|""".stripMargin
- val df = sql(sqlText)
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+ val df = sql(sqlText)
- df.write.mode("overwrite").orc("ts_ntz_orc")
+ df.write.mode("overwrite").orc(path)
- val query = "select * from `orc`.`ts_ntz_orc`"
+ val query = s"select * from `orc`.`$path`"
- DateTimeTestUtils.outstandingZoneIds.foreach { zoneId =>
- DateTimeTestUtils.withDefaultTimeZone(zoneId) {
- withAllNativeOrcReaders {
- checkAnswer(sql(query), df)
+ DateTimeTestUtils.outstandingZoneIds.foreach { zoneId =>
+ DateTimeTestUtils.withDefaultTimeZone(zoneId) {
+ withAllNativeOrcReaders {
+ checkAnswer(sql(query), df)
+ }
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
