[
https://issues.apache.org/jira/browse/HADOOP-15796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631626#comment-16631626
]
Steve Loughran edited comment on HADOOP-15796 at 9/28/18 10:21 AM:
-------------------------------------------------------------------
-1 to caching of everything if possible, as it significantly slows down
Integration test runs, and doesn't address a key need: we want the mock
filesystems in the cache for the committers to pick up when they call
FileSystem.get().
* we absolutely need to do caching stuff when we want hand preconfigured
real/mock FS instances to production commit code which calls filesystem.get())
to pick up . If you look at {{AbstractCommitITest}} you can see places where
that's done, using {{FileSystemTestHelper.addFileSystemForTesting()}} to do
this.
* And I see {{StagingTestBase.createAndBindMockFSInstance()}} doing the same
thing.
you shouldn't have >1 test in the same JVM running simultaneously. If you are
getting closed problems in a specific suite, then that should be looked at. Are
you running the tests in an IDE?
was (Author: [email protected]):
-1 to caching of everything if possible, as it significantly slows down
Integration test runs, and doesn't address a key need: we want the mock
filesystems in the cache for the committers to pick up when they call
FileSystem.get().
* we absolutely need to do caching stuff when we want hand preconfigured
real/mock FS instances to production commit code which calls filesystem.get())
to pick up . If you look at {{AbstractCommitITest}} you can see places where
that's done, using {{ FileSystemTestHelper.addFileSystemForTesting()}} to do
this.
* And I see {{StagingTestBase.createAndBindMockFSInstance()}} doing the same
thing.
you shouldn't have >1 test in the same JVM running simultaneously. If you are
getting closed problems in a specific suite, then that should be looked at. Are
you running the tests in an IDE?
> ClassCastException: S3AFileSystem cannot be cast to MockS3AFileSystem when fs
> caching is disabled
> -------------------------------------------------------------------------------------------------
>
> Key: HADOOP-15796
> URL: https://issues.apache.org/jira/browse/HADOOP-15796
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.1
> Reporter: Gabor Bota
> Priority: Major
>
> Sometimes we get {{java.io.IOException: s3a://TEST-BUCKET: FileSystem is
> closed!}} when running tests, because of filesystem caching: an fs instance
> is closed from a test teardown while another test is still using it. (That
> should be fixed in another issue.)
> To avoid test errors introduced by closed fs I've disabled fs caching while
> testing. This caused the following issue in lots of test:
> {noformat}
> [ERROR]
> testPartitionsResolution(org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedFileListing)
> Time elapsed: 1.994 s <<< ERROR!
> java.lang.ClassCastException: org.apache.hadoop.fs.s3a.S3AFileSystem cannot
> be cast to org.apache.hadoop.fs.s3a.MockS3AFileSystem
> at
> org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase.lookupWrapperFS(StagingTestBase.java:145)
> at
> org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase$JobCommitterTest.setupJob(StagingTestBase.java:294)
> {noformat}
> We should fix the casting during these tests.
> To switch of fs caching during a test (mvn verify) run, add the following to
> the config:
> {code:java}
> <property>
> <name>fs.s3a.impl.disable.cache</name>
> <value>true</value>
> </property>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
