[
https://issues.apache.org/jira/browse/IMPALA-11720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang resolved IMPALA-11720.
-------------------------------------
Fix Version/s: Impala 4.2.0
Resolution: Fixed
> FileMetadataLoaderTest is flaky due to IOException: Filesystem closed
> ---------------------------------------------------------------------
>
> Key: IMPALA-11720
> URL: https://issues.apache.org/jira/browse/IMPALA-11720
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 4.2.0
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
> Fix For: Impala 4.2.0
>
> Attachments:
> FeSupport.quanlong-OptiPlex-BJ.quanlong.log.INFO.20221111-162836.26125,
> IMPALA-11720-add-logs.patch,
> org.apache.impala.catalog.FileMetadataLoaderTest-output.txt
>
>
> IMPALA-11699 recently reverts the changes in FileMetadataLoaderTest that
> extends it to FrontendTestBase. This causes tests in it become flaky if run
> individually. E.g. the following commands all fail:
> {code:java}
> (pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testRecursiveLoading)
> (pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testHudiParquetLoading)
> (pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testIcebergLoading)
> (pushd fe && mvn test
> -Dtest=FileMetadataLoaderTest#testAcidMinorCompactionLoading)
> (pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testLoadMissingDirectory)
> (pushd fe && mvn test
> -Dtest=FileMetadataLoaderTest#testSkipHiddenDirectories){code}
> and the failures are the same, i.e. due to java.io.IOException: Filesystem
> closed
> {noformat}
> $ (pushd fe && mvn test
> -Dtest=FileMetadataLoaderTest#testLoadMissingDirectory)
> ...
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.601
> s <<< FAILURE! - in org.apache.impala.catalog.FileMetadataLoaderTest
> [ERROR]
> testLoadMissingDirectory(org.apache.impala.catalog.FileMetadataLoaderTest)
> Time elapsed: 1.565 s <<< ERROR!
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:483)
> at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1662)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1266)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1249)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1190)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1208)
> at
> org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2145)
> at
> org.apache.impala.common.FileSystemUtil.listLocatedStatusIterator(FileSystemUtil.java:804)
> at
> org.apache.impala.common.FileSystemUtil$RecursingIterator.<init>(FileSystemUtil.java:975)
> at
> org.apache.impala.common.FileSystemUtil$RecursingIterator.<init>(FileSystemUtil.java:958)
> at
> org.apache.impala.common.FileSystemUtil.listFiles(FileSystemUtil.java:768)
> at
> org.apache.impala.catalog.FileMetadataLoader.load(FileMetadataLoader.java:187)
> at
> org.apache.impala.catalog.FileMetadataLoaderTest.testLoadMissingDirectory(FileMetadataLoaderTest.java:192)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> {noformat}
> However, any test runs after the failure will succeed. If we run the test
> twice (by adding -Dsurefire.rerunFailingTestsCount=2), the second run also
> succeed:
> {noformat}
> $ (pushd fe && mvn test
> -Dtest=FileMetadataLoaderTest#testLoadMissingDirectory
> -Dsurefire.rerunFailingTestsCount=2)
> ...
> [WARNING] Flakes:
> [WARNING]
> org.apache.impala.catalog.FileMetadataLoaderTest.testLoadMissingDirectory(org.apache.impala.catalog.FileMetadataLoaderTest)
> [ERROR] Run 1: FileMetadataLoaderTest.testLoadMissingDirectory:192 » IO
> Filesystem closed
> [INFO] Run 2: PASS
> [INFO]
> [INFO]
> [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
