[
https://issues.apache.org/jira/browse/HUDI-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17853358#comment-17853358
]
Sagar Sumit commented on HUDI-7835:
-----------------------------------
I observed one more thing: Somehow the TestFunctionalIndex is attempted again
after TestSecondaryIndex. To verify, I disabled TestFunctionalIndex and
confirmed from the logs, it is attempted twice (though it does not run because
it is diabled). We need to root cause why TestFunctionalIndex is being
attempted twice. Note the test always succeeds from IDE.
To reproduce run the following maven command from hudi repo root, and direct
the logs to some file where you can grep for any TestFunctionalIndex name, e.g.
`Test Create Functional Index`
{code:java}
mvn test -Pwarn-log -Dscala-2.12 -Dspark3.2 -Dflink1.18 -Dcheckstyle.skip=true
-Drat.skip=true -Djacoco.skip=true -ntp -B -V -Pwarn-log
-Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.shade=warn
-Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.dependency=warn
-Punit-tests -Dtest=skipJavaTests -DfailIfNoTests=false
-DwildcardSuites=org.apache.hudi,org.apache.spark.hudi,org.apache.spark.sql.avro,org.apache.spark.sql.execution,org.apache.spark.sql.hudi.analysis,org.apache.spark.sql.hudi.command,org.apache.spark.sql.hudi.common,org.apache.spark.sql.hudi.dml
-pl
hudi-spark-datasource,hudi-spark-datasource/hudi-spark,hudi-spark-datasource/hudi-spark3.2.x,hudi-spark-datasource/hudi-spark3.2plus-common,hudi-spark-datasource/hudi-spark3-common,hudi-spark-datasource/hudi-spark-common
{code}
> Spark context not stopped properly if the test had some exception that was
> ignored
> ----------------------------------------------------------------------------------
>
> Key: HUDI-7835
> URL: https://issues.apache.org/jira/browse/HUDI-7835
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Sagar Sumit
> Priority: Major
> Fix For: 1.0.0
>
>
> Found two tests that don't fail but throw an exception while running. The
> test succeeds but the spark context is not stopped in time due to exception.
> This causes issue for other tests. For example, `test("bucket index query")`
> in `TestDataSkippingQuery` succeeds but when we check the
> [logs|https://github.com/apache/hudi/actions/runs/9391927778/job/25865161535#step:6:5799]
> of the test we will find an error as below
> {code:java}
> 74954 [ScalaTest-run-running-TestDataSkippingQuery] ERROR
> org.apache.hudi.HoodieFileIndex [] - Failed to lookup candidate files in File
> Index
> java.lang.IllegalArgumentException: Property
> _hoodie.record.key.gen.partition.id not found
> at
> org.apache.hudi.common.config.TypedProperties.checkKey(TypedProperties.java:69)
> ~[classes/:?]
> at
> org.apache.hudi.common.config.TypedProperties.getInteger(TypedProperties.java:94)
> ~[classes/:?]
> at
> org.apache.hudi.keygen.AutoRecordGenWrapperKeyGenerator.generateSequenceId(AutoRecordGenWrapperKeyGenerator.java:115)
> ~[classes/:?]
> at
> org.apache.hudi.keygen.AutoRecordGenWrapperKeyGenerator.getRecordKey(AutoRecordGenWrapperKeyGenerator.java:67)
> ~[classes/:?]
> at
> org.apache.hudi.keygen.BaseKeyGenerator.getKey(BaseKeyGenerator.java:70)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.getBucketNumber$1(BucketIndexSupport.scala:154)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.getBucketSetFromValue$1(BucketIndexSupport.scala:168)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.getBucketsBySingleHashFields(BucketIndexSupport.scala:174)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.filterQueriesWithBucketHashField(BucketIndexSupport.scala:107)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.computeCandidateFileNames(BucketIndexSupport.scala:78)
> ~[classes/:?]
> at
> org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$3(HoodieFileIndex.scala:354)
> ~[classes/:?]
> at
> org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$3$adapted(HoodieFileIndex.scala:351)
> ~[classes/:?]
> at
> scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
> ~[scala-library-2.12.10.jar:?]
> at scala.collection.immutable.List.foreach(List.scala:392)
> ~[scala-library-2.12.10.jar:?]
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)
> ~[scala-library-2.12.10.jar:?]
> at
> org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$1(HoodieFileIndex.scala:351)
> ~[classes/:?]
> at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.10.jar:?]
> at
> org.apache.hudi.HoodieFileIndex.lookupCandidateFilesInMetadataTable(HoodieFileIndex.scala:338)
> ~[classes/:?]
> at
> org.apache.hudi.HoodieFileIndex.filterFileSlices(HoodieFileIndex.scala:241)
> ~[classes/:?]
> at org.apache.hudi.HoodieFileIndex.listFiles(HoodieFileIndex.scala:160)
> ~[classes/:?] {code}
> The test succeeds because HoodieFileIndex listFiles is under try-catch and
> falls back to no pruning (except for partition pruning). So, anyway due to
> the exception spark context runs into conflict with another test which throws
> exception as below:
> {code:java}
> org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> org.scalatest.Transformer.apply(Transformer.scala:22)
> org.scalatest.Transformer.apply(Transformer.scala:20)
> org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:189)
> org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
> org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
> org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1562)
> org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:187)
> org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:199)
> The currently active SparkContext was created at:
> org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase.$anonfun$test$1(HoodieSparkSqlTestBase.scala:85)
> org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> org.scalatest.Transformer.apply(Transformer.scala:22)
> org.scalatest.Transformer.apply(Transformer.scala:20)
> org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:189)
> org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
> org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
> org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1562)
> org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:187)
> org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:199)
> org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
> org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:199)
> org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:181)
> org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1562)
> {code}
> Another example is `test("Test Functional Index With Hive Sync Non
> Partitioned Table")` in `TestFunctionalIndex`.
> We need to do two things:
> 1. First make the spark context shudown properly. Note that this happens only
> with tests that extend `HoodieSparkSqlTestBase`.
> 2. Fix the failure in individual tests.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
