[
https://issues.apache.org/jira/browse/HADOOP-15796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-15796:
--------------------------------
Description:
Sometimes we get {{java.io.IOException: s3a://TEST-BUCKET: FileSystem is
closed!}} when running tests, because of filesystem caching: an fs instance is
closed from a test teardown while another test is still using it. (That should
be fixed in another issue.)
To avoid test errors introduced by closed fs I've disabled fs caching while
testing. This caused the following issue in lots of test:
{noformat}
[ERROR]
testPartitionsResolution(org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedFileListing)
Time elapsed: 1.994 s <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.fs.s3a.S3AFileSystem cannot be
cast to org.apache.hadoop.fs.s3a.MockS3AFileSystem
at
org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase.lookupWrapperFS(StagingTestBase.java:145)
at
org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase$JobCommitterTest.setupJob(StagingTestBase.java:294)
{noformat}
We should fix the casting during these tests.
To switch of fs caching during a test (mvn verify) run, add the following to
the config:
{code:java}
<property>
<name>fs.s3a.impl.disable.cache</name>
<value>true</value>
</property>
{code}
was:
Sometimes we get {{java.io.IOException: s3a://TEST-BUCKET: FileSystem is
closed!}} when running tests, because of filesystem caching: an fs instance is
closed from a test teardown while another test is still using it. (That should
be fixed in another issue.)
To avoid test errors introduced by closed fs I've disabled fs caching while
testing. This caused the following issue in lots of test:
{noformat}
[ERROR]
testPartitionsResolution(org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedFileListing)
Time elapsed: 1.994 s <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.fs.s3a.S3AFileSystem cannot be
cast to org.apache.hadoop.fs.s3a.MockS3AFileSystem
at
org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase.lookupWrapperFS(StagingTestBase.java:145)
at
org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase$JobCommitterTest.setupJob(StagingTestBase.java:294)
{noformat}
We should fix the casting during these tests
> ClassCastException: S3AFileSystem cannot be cast to MockS3AFileSystem when fs
> caching is disabled
> -------------------------------------------------------------------------------------------------
>
> Key: HADOOP-15796
> URL: https://issues.apache.org/jira/browse/HADOOP-15796
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.1.1
> Reporter: Gabor Bota
> Priority: Major
>
> Sometimes we get {{java.io.IOException: s3a://TEST-BUCKET: FileSystem is
> closed!}} when running tests, because of filesystem caching: an fs instance
> is closed from a test teardown while another test is still using it. (That
> should be fixed in another issue.)
> To avoid test errors introduced by closed fs I've disabled fs caching while
> testing. This caused the following issue in lots of test:
> {noformat}
> [ERROR]
> testPartitionsResolution(org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedFileListing)
> Time elapsed: 1.994 s <<< ERROR!
> java.lang.ClassCastException: org.apache.hadoop.fs.s3a.S3AFileSystem cannot
> be cast to org.apache.hadoop.fs.s3a.MockS3AFileSystem
> at
> org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase.lookupWrapperFS(StagingTestBase.java:145)
> at
> org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase$JobCommitterTest.setupJob(StagingTestBase.java:294)
> {noformat}
> We should fix the casting during these tests.
> To switch of fs caching during a test (mvn verify) run, add the following to
> the config:
> {code:java}
> <property>
> <name>fs.s3a.impl.disable.cache</name>
> <value>true</value>
> </property>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
