[jira] [Commented] (HADOOP-16027) [DOC] Effective use of FS instances during S3A integration tests
[ https://issues.apache.org/jira/browse/HADOOP-16027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739271#comment-16739271 ] Gabor Bota commented on HADOOP-16027: - Thanks [~mackrorysd]! As this is committed, could you resolve the issue? > [DOC] Effective use of FS instances during S3A integration tests > > > Key: HADOOP-16027 > URL: https://issues.apache.org/jira/browse/HADOOP-16027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Attachments: HADOOP-16027.001.patch, HADOOP-16027.002.patch > > > While fixing HADOOP-15819 we found that a closed fs got into the static fs > cache during testing, which caused other tests to fail when the tests were > running sequentially. > We should document some best practices in the testing section on the s3 docs > with the following: > {panel} > Tests using FileSystems are fastest if they can recycle the existing FS > instance from the same JVM. If you do that, you MUST NOT close or do unique > configuration on them. If you want a guarantee of 100% isolation or an > instance with unique config, create a new instance > which you MUST close in the teardown to avoid leakage of resources. > Do not add FileSystem instances (with e.g > org.apache.hadoop.fs.FileSystem#addFileSystemForTesting) to the cache that > will be modified or closed during the test runs. This can cause other tests > to fail when using the same modified or closed FS instance. For more details > see HADOOP-15819. > {panel} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16027) [DOC] Effective use of FS instances during S3A integration tests
[ https://issues.apache.org/jira/browse/HADOOP-16027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738507#comment-16738507 ] Hudson commented on HADOOP-16027: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15751 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15751/]) HADOOP-16027. [DOC] Effective use of FS instances during S3A integration (sean: rev 3420e26ae57f5946a913278a8a62ae82e930df88) * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > [DOC] Effective use of FS instances during S3A integration tests > > > Key: HADOOP-16027 > URL: https://issues.apache.org/jira/browse/HADOOP-16027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Attachments: HADOOP-16027.001.patch, HADOOP-16027.002.patch > > > While fixing HADOOP-15819 we found that a closed fs got into the static fs > cache during testing, which caused other tests to fail when the tests were > running sequentially. > We should document some best practices in the testing section on the s3 docs > with the following: > {panel} > Tests using FileSystems are fastest if they can recycle the existing FS > instance from the same JVM. If you do that, you MUST NOT close or do unique > configuration on them. If you want a guarantee of 100% isolation or an > instance with unique config, create a new instance > which you MUST close in the teardown to avoid leakage of resources. > Do not add FileSystem instances (with e.g > org.apache.hadoop.fs.FileSystem#addFileSystemForTesting) to the cache that > will be modified or closed during the test runs. This can cause other tests > to fail when using the same modified or closed FS instance. For more details > see HADOOP-15819. > {panel} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16027) [DOC] Effective use of FS instances during S3A integration tests
[ https://issues.apache.org/jira/browse/HADOOP-16027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738490#comment-16738490 ] Sean Mackrory commented on HADOOP-16027: +1, committed. Nit: added the empty lines between what appeared to be paragraphs. > [DOC] Effective use of FS instances during S3A integration tests > > > Key: HADOOP-16027 > URL: https://issues.apache.org/jira/browse/HADOOP-16027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Attachments: HADOOP-16027.001.patch, HADOOP-16027.002.patch > > > While fixing HADOOP-15819 we found that a closed fs got into the static fs > cache during testing, which caused other tests to fail when the tests were > running sequentially. > We should document some best practices in the testing section on the s3 docs > with the following: > {panel} > Tests using FileSystems are fastest if they can recycle the existing FS > instance from the same JVM. If you do that, you MUST NOT close or do unique > configuration on them. If you want a guarantee of 100% isolation or an > instance with unique config, create a new instance > which you MUST close in the teardown to avoid leakage of resources. > Do not add FileSystem instances (with e.g > org.apache.hadoop.fs.FileSystem#addFileSystemForTesting) to the cache that > will be modified or closed during the test runs. This can cause other tests > to fail when using the same modified or closed FS instance. For more details > see HADOOP-15819. > {panel} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16027) [DOC] Effective use of FS instances during S3A integration tests
[ https://issues.apache.org/jira/browse/HADOOP-16027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16735655#comment-16735655 ] Hadoop QA commented on HADOOP-16027: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 31m 26s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 44s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 28s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 46m 28s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HADOOP-16027 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12953970/HADOOP-16027.002.patch | | Optional Tests | dupname asflicense mvnsite | | uname | Linux 8fd065f3b047 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 5db7c49 | | maven | version: Apache Maven 3.3.9 | | Max. process+thread count | 410 (vs. ulimit of 1) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15739/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > [DOC] Effective use of FS instances during S3A integration tests > > > Key: HADOOP-16027 > URL: https://issues.apache.org/jira/browse/HADOOP-16027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Attachments: HADOOP-16027.001.patch, HADOOP-16027.002.patch > > > While fixing HADOOP-15819 we found that a closed fs got into the static fs > cache during testing, which caused other tests to fail when the tests were > running sequentially. > We should document some best practices in the testing section on the s3 docs > with the following: > {panel} > Tests using FileSystems are fastest if they can recycle the existing FS > instance from the same JVM. If you do that, you MUST NOT close or do unique > configuration on them. If you want a guarantee of 100% isolation or an > instance with unique config, create a new instance > which you MUST close in the teardown to avoid leakage of resources. > Do not add FileSystem instances (with e.g > org.apache.hadoop.fs.FileSystem#addFileSystemForTesting) to the cache that > will be modified or closed during the test runs. This can cause other tests > to fail when using the same modified or closed FS instance. For more details > see HADOOP-15819. > {panel} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16027) [DOC] Effective use of FS instances during S3A integration tests
[ https://issues.apache.org/jira/browse/HADOOP-16027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16735623#comment-16735623 ] Gabor Bota commented on HADOOP-16027: - Thanks [~adam.antal], uploaded v2 patch with this. > [DOC] Effective use of FS instances during S3A integration tests > > > Key: HADOOP-16027 > URL: https://issues.apache.org/jira/browse/HADOOP-16027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Attachments: HADOOP-16027.001.patch, HADOOP-16027.002.patch > > > While fixing HADOOP-15819 we found that a closed fs got into the static fs > cache during testing, which caused other tests to fail when the tests were > running sequentially. > We should document some best practices in the testing section on the s3 docs > with the following: > {panel} > Tests using FileSystems are fastest if they can recycle the existing FS > instance from the same JVM. If you do that, you MUST NOT close or do unique > configuration on them. If you want a guarantee of 100% isolation or an > instance with unique config, create a new instance > which you MUST close in the teardown to avoid leakage of resources. > Do not add FileSystem instances (with e.g > org.apache.hadoop.fs.FileSystem#addFileSystemForTesting) to the cache that > will be modified or closed during the test runs. This can cause other tests > to fail when using the same modified or closed FS instance. For more details > see HADOOP-15819. > {panel} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16027) [DOC] Effective use of FS instances during S3A integration tests
[ https://issues.apache.org/jira/browse/HADOOP-16027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734321#comment-16734321 ] Adam Antal commented on HADOOP-16027: - Thanks for the patch [~gabor.bota], it look good, as [~ste...@apache.org] said. I'll add one minor thing: I'd rather add the word manually here {code:java} Do NOT add manually `FileSystem` instances (...) to the cache {code} (instead of {code:java} Do NOT add `FileSystem` instances (...) to the cache {code} ) because many FS instances are actually added to the cache - but just as the they're closed, they got removed from it. > [DOC] Effective use of FS instances during S3A integration tests > > > Key: HADOOP-16027 > URL: https://issues.apache.org/jira/browse/HADOOP-16027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Attachments: HADOOP-16027.001.patch > > > While fixing HADOOP-15819 we found that a closed fs got into the static fs > cache during testing, which caused other tests to fail when the tests were > running sequentially. > We should document some best practices in the testing section on the s3 docs > with the following: > {panel} > Tests using FileSystems are fastest if they can recycle the existing FS > instance from the same JVM. If you do that, you MUST NOT close or do unique > configuration on them. If you want a guarantee of 100% isolation or an > instance with unique config, create a new instance > which you MUST close in the teardown to avoid leakage of resources. > Do not add FileSystem instances (with e.g > org.apache.hadoop.fs.FileSystem#addFileSystemForTesting) to the cache that > will be modified or closed during the test runs. This can cause other tests > to fail when using the same modified or closed FS instance. For more details > see HADOOP-15819. > {panel} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16027) [DOC] Effective use of FS instances during S3A integration tests
[ https://issues.apache.org/jira/browse/HADOOP-16027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734295#comment-16734295 ] Hadoop QA commented on HADOOP-16027: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 33m 28s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 41s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 25s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 49m 24s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HADOOP-16027 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12953767/HADOOP-16027.001.patch | | Optional Tests | dupname asflicense mvnsite | | uname | Linux 5603d804cd2b 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 8c6978c | | maven | version: Apache Maven 3.3.9 | | Max. process+thread count | 306 (vs. ulimit of 1) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15723/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > [DOC] Effective use of FS instances during S3A integration tests > > > Key: HADOOP-16027 > URL: https://issues.apache.org/jira/browse/HADOOP-16027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Attachments: HADOOP-16027.001.patch > > > While fixing HADOOP-15819 we found that a closed fs got into the static fs > cache during testing, which caused other tests to fail when the tests were > running sequentially. > We should document some best practices in the testing section on the s3 docs > with the following: > {panel} > Tests using FileSystems are fastest if they can recycle the existing FS > instance from the same JVM. If you do that, you MUST NOT close or do unique > configuration on them. If you want a guarantee of 100% isolation or an > instance with unique config, create a new instance > which you MUST close in the teardown to avoid leakage of resources. > Do not add FileSystem instances (with e.g > org.apache.hadoop.fs.FileSystem#addFileSystemForTesting) to the cache that > will be modified or closed during the test runs. This can cause other tests > to fail when using the same modified or closed FS instance. For more details > see HADOOP-15819. > {panel} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org