Sagar Sumit created HUDI-7835:
---------------------------------
Summary: Spark context not stopped properly if the test had some
exception that was ignored
Key: HUDI-7835
URL: https://issues.apache.org/jira/browse/HUDI-7835
Project: Apache Hudi
Issue Type: Improvement
Reporter: Sagar Sumit
Fix For: 1.0.0
Found two tests that don't fail but throw an exception while running. The test
succeeds but the spark context is not stopped in time due to exception. This
causes issue for other tests. For example, `test("bucket index query")` in
`TestDataSkippingQuery` succeeds but when we check the
[logs|https://github.com/apache/hudi/actions/runs/9391927778/job/25865161535#step:6:5799]
of the test we will find an error as below
{code:java}
74954 [ScalaTest-run-running-TestDataSkippingQuery] ERROR
org.apache.hudi.HoodieFileIndex [] - Failed to lookup candidate files in File
Index
java.lang.IllegalArgumentException: Property
_hoodie.record.key.gen.partition.id not found
at
org.apache.hudi.common.config.TypedProperties.checkKey(TypedProperties.java:69)
~[classes/:?]
at
org.apache.hudi.common.config.TypedProperties.getInteger(TypedProperties.java:94)
~[classes/:?]
at
org.apache.hudi.keygen.AutoRecordGenWrapperKeyGenerator.generateSequenceId(AutoRecordGenWrapperKeyGenerator.java:115)
~[classes/:?]
at
org.apache.hudi.keygen.AutoRecordGenWrapperKeyGenerator.getRecordKey(AutoRecordGenWrapperKeyGenerator.java:67)
~[classes/:?]
at org.apache.hudi.keygen.BaseKeyGenerator.getKey(BaseKeyGenerator.java:70)
~[classes/:?]
at
org.apache.hudi.BucketIndexSupport.getBucketNumber$1(BucketIndexSupport.scala:154)
~[classes/:?]
at
org.apache.hudi.BucketIndexSupport.getBucketSetFromValue$1(BucketIndexSupport.scala:168)
~[classes/:?]
at
org.apache.hudi.BucketIndexSupport.getBucketsBySingleHashFields(BucketIndexSupport.scala:174)
~[classes/:?]
at
org.apache.hudi.BucketIndexSupport.filterQueriesWithBucketHashField(BucketIndexSupport.scala:107)
~[classes/:?]
at
org.apache.hudi.BucketIndexSupport.computeCandidateFileNames(BucketIndexSupport.scala:78)
~[classes/:?]
at
org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$3(HoodieFileIndex.scala:354)
~[classes/:?]
at
org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$3$adapted(HoodieFileIndex.scala:351)
~[classes/:?]
at
scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
~[scala-library-2.12.10.jar:?]
at scala.collection.immutable.List.foreach(List.scala:392)
~[scala-library-2.12.10.jar:?]
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)
~[scala-library-2.12.10.jar:?]
at
org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$1(HoodieFileIndex.scala:351)
~[classes/:?]
at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.10.jar:?]
at
org.apache.hudi.HoodieFileIndex.lookupCandidateFilesInMetadataTable(HoodieFileIndex.scala:338)
~[classes/:?]
at
org.apache.hudi.HoodieFileIndex.filterFileSlices(HoodieFileIndex.scala:241)
~[classes/:?]
at org.apache.hudi.HoodieFileIndex.listFiles(HoodieFileIndex.scala:160)
~[classes/:?] {code}
The test succeeds because HoodieFileIndex listFiles is under try-catch and
falls back to no pruning (except for partition pruning). So, anyway due to the
exception spark context runs into conflict with another test which throws
exception as below:
{code:java}
org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
org.scalatest.Transformer.apply(Transformer.scala:22)
org.scalatest.Transformer.apply(Transformer.scala:20)
org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:189)
org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1562)
org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:187)
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:199)
The currently active SparkContext was created at:
org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase.$anonfun$test$1(HoodieSparkSqlTestBase.scala:85)
org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
org.scalatest.Transformer.apply(Transformer.scala:22)
org.scalatest.Transformer.apply(Transformer.scala:20)
org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:189)
org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1562)
org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:187)
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:199)
org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:199)
org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:181)
org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1562)
{code}
Another example is `test("Test Functional Index With Hive Sync Non Partitioned
Table")` in `TestFunctionalIndex`.
We need to do two things:
1. First make the spark context shudown properly. Note that this happens only
with tests that extend `HoodieSparkSqlTestBase`.
2. Fix the failure in individual tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
