[
https://issues.apache.org/jira/browse/IMPALA-11720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17634101#comment-17634101
]
ASF subversion and git services commented on IMPALA-11720:
----------------------------------------------------------
Commit 86fdbfcc995a22a5836c6625fd5c86080fa7764a in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=86fdbfcc9 ]
IMPALA-11720: Deflake FileMetadataLoaderTest due to FileSystem closed
IMPALA-11699 recently reverted the changes in FileMetadataLoaderTest
that it no longer extends to FrontendTestBase. This causes tests in
FileMetadataLoaderTest to become flaky if run individually.
This is a subtle bug in FileMetadataLoader#load(), where we first call
FileSystem#getFileSystem() to get a FileSystem object, and then call
methods of FileSystemUtil which will trigger class loading of it and
running its static code.
FileSystem#getFileSystem() will create a FileSystem object in the first
get and cache it for follow-up usage. Without extending
FrontendTestBase, BackendConfig.INSTANCE is not initialized when the
test is run. So the static code in FileSystemUtil lazily initializes
BackendConfig.INSTANCE, which initializes FeSupport and finally calls
JniUtil::InitLibhdfs(). In this method, a FileSystem object is get and
closed. This is exactly the FileSystem object created in
FileMetadataLoader#load(). So the following usage on it causes an
IOException of "Filesystem closed".
The purpose of JniUtil::InitLibhdfs() is to make a simple call on
libhdfs to make it initialize the JVM. This is crucial when launching
from C++ codes for impalad and catalogd to init the embedded JVM. So we
should keep it unchanged.
The fix for this bug is to maintain the state of BackendConfig.INSTANCE
in the static code of FileSystemUtil (i.e., keep un-initialized or
initialized), which avoids the subtle side effects.
Tests:
- Verified tests in FileMetadataLoaderTest individually
Change-Id: Ib6f96950210c9a0124fe03696ef334cb00b057ab
Reviewed-on: http://gerrit.cloudera.org:8080/19233
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
Reviewed-by: Qifan Chen <[email protected]>
> FileMetadataLoaderTest is flaky due to IOException: Filesystem closed
> ---------------------------------------------------------------------
>
> Key: IMPALA-11720
> URL: https://issues.apache.org/jira/browse/IMPALA-11720
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
> Attachments:
> FeSupport.quanlong-OptiPlex-BJ.quanlong.log.INFO.20221111-162836.26125,
> IMPALA-11720-add-logs.patch,
> org.apache.impala.catalog.FileMetadataLoaderTest-output.txt
>
>
> IMPALA-11699 recently reverts the changes in FileMetadataLoaderTest that
> extends it to FrontendTestBase. This causes tests in it become flaky if run
> individually. E.g. the following commands all fail:
> {code:java}
> (pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testRecursiveLoading)
> (pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testHudiParquetLoading)
> (pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testIcebergLoading)
> (pushd fe && mvn test
> -Dtest=FileMetadataLoaderTest#testAcidMinorCompactionLoading)
> (pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testLoadMissingDirectory)
> (pushd fe && mvn test
> -Dtest=FileMetadataLoaderTest#testSkipHiddenDirectories){code}
> and the failures are the same, i.e. due to java.io.IOException: Filesystem
> closed
> {noformat}
> $ (pushd fe && mvn test
> -Dtest=FileMetadataLoaderTest#testLoadMissingDirectory)
> ...
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.601
> s <<< FAILURE! - in org.apache.impala.catalog.FileMetadataLoaderTest
> [ERROR]
> testLoadMissingDirectory(org.apache.impala.catalog.FileMetadataLoaderTest)
> Time elapsed: 1.565 s <<< ERROR!
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:483)
> at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1662)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1266)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1249)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1190)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1208)
> at
> org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2145)
> at
> org.apache.impala.common.FileSystemUtil.listLocatedStatusIterator(FileSystemUtil.java:804)
> at
> org.apache.impala.common.FileSystemUtil$RecursingIterator.<init>(FileSystemUtil.java:975)
> at
> org.apache.impala.common.FileSystemUtil$RecursingIterator.<init>(FileSystemUtil.java:958)
> at
> org.apache.impala.common.FileSystemUtil.listFiles(FileSystemUtil.java:768)
> at
> org.apache.impala.catalog.FileMetadataLoader.load(FileMetadataLoader.java:187)
> at
> org.apache.impala.catalog.FileMetadataLoaderTest.testLoadMissingDirectory(FileMetadataLoaderTest.java:192)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> {noformat}
> However, any test runs after the failure will succeed. If we run the test
> twice (by adding -Dsurefire.rerunFailingTestsCount=2), the second run also
> succeed:
> {noformat}
> $ (pushd fe && mvn test
> -Dtest=FileMetadataLoaderTest#testLoadMissingDirectory
> -Dsurefire.rerunFailingTestsCount=2)
> ...
> [WARNING] Flakes:
> [WARNING]
> org.apache.impala.catalog.FileMetadataLoaderTest.testLoadMissingDirectory(org.apache.impala.catalog.FileMetadataLoaderTest)
> [ERROR] Run 1: FileMetadataLoaderTest.testLoadMissingDirectory:192 » IO
> Filesystem closed
> [INFO] Run 2: PASS
> [INFO]
> [INFO]
> [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
