dongjoon-hyun opened a new pull request #31697:
URL: https://github.com/apache/spark/pull/31697
### What changes were proposed in this pull request?
1. This PR aims to ignore ORC encryption tests when ORC shim is loaded by
old Hadoop library by some other tests. The test coverage is preserved by
Jenkins SBT runs and GitHub Action jobs. This PR only aims to recover Maven
Jenkins jobs.
2. Also
### Why are the changes needed?
Currently, Maven test fails when it runs in a batch mode because
`HadoopShimsPre2_3$NullKeyProvider` is loaded.
**COMMAND**
```
$ mvn test -pl sql/core --am -Dtest=none
-DwildcardSuites=org.apache.spark.sql.execution.datasources.orc.OrcV1QuerySuite,org.apache.spark.sql.execution.datasources.orc.OrcEncryptionSuite
```
**BEFORE**
```
- Write and read an encrypted table *** FAILED ***
...
Cause: org.apache.spark.SparkException: Job aborted due to stage failure:
Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage
1.0 (TID 1) (localhost executor driver): java.lang.IllegalArgumentException:
Unknown key pii
at
org.apache.orc.impl.HadoopShimsPre2_3$NullKeyProvider.getCurrentKeyVersion(HadoopShimsPre2_3.java:71)
at org.apache.orc.impl.WriterImpl.getKey(WriterImpl.java:871)
```
**AFTER**
```
OrcV1QuerySuite
...
OrcEncryptionSuite:
- Write and read an encrypted file !!! CANCELED !!!
[] was empty
org.apache.orc.impl.HadoopShimsPre2_3$NullKeyProvider@1b705f65 doesn't has the
test keys. ORC shim is created with old Hadoop libraries
(OrcEncryptionSuite.scala:39)
- Write and read an encrypted table !!! CANCELED !!!
[] was empty
org.apache.orc.impl.HadoopShimsPre2_3$NullKeyProvider@22adeee1 doesn't has the
test keys. ORC shim is created with old Hadoop libraries
(OrcEncryptionSuite.scala:67)
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the Jenkins Maven tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
