[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13171:
---
Resolution: Fixed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
I have committed this to trunk, branch-2 and branch-2.8. Steve, thank you for
the patch.
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-016.patch,
> HADOOP-13171-branch-2-001.patch, HADOOP-13171-branch-2-002.patch,
> HADOOP-13171-branch-2-003.patch, HADOOP-13171-branch-2-004.patch,
> HADOOP-13171-branch-2-005.patch, HADOOP-13171-branch-2-006.patch,
> HADOOP-13171-branch-2-007.patch, HADOOP-13171-branch-2-008.patch,
> HADOOP-13171-branch-2-009.patch, HADOOP-13171-branch-2-010.patch,
> HADOOP-13171-branch-2-011.patch, HADOOP-13171-branch-2-012.patch,
> HADOOP-13171-branch-2-013.patch, HADOOP-13171-branch-2-015.patch,
> HADOOP-13171-branch-2.8-017.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13171:
---
Attachment: HADOOP-13171-branch-2.8-017.patch
Unfortunately, this cannot go to branch-2.8 as is. First, there is a minor
merge conflict in {{TestS3ADeleteManyFiles}} due to lack of HADOOP-12753 in
branch-2.8. That's trivial to fix, and I would have committed, but the more
significant problem is lack of {{GenericTestUtils#getTestDir}} on branch-2.8,
which caused a compilation failure. This is similar to what happened with
HADOOP-13145, and we can use the same kind of workaround: inline the test
directory creation. For this part, I'd prefer to put a branch-2.8-specific
patch through pre-commit.
Steve, I took the liberty of making a branch-2.8 patch. Could I please get
your review on just the change to deal with lack of
{{GenericTestUtils#getTestDir}}? For convenience, here is the only diff
compared to the branch-2 patch, in {{TestS3AFileOperationCost}}:
{code}
-File localTestDir = getTestDir("tmp");
+String testDirProp = System.getProperty("test.build.data",
+"target" + File.separator + "test" + File.separator + "data");
+File localTestDir = new File(testDirProp, "tmp").getAbsoluteFile();
{code}
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-016.patch,
> HADOOP-13171-branch-2-001.patch, HADOOP-13171-branch-2-002.patch,
> HADOOP-13171-branch-2-003.patch, HADOOP-13171-branch-2-004.patch,
> HADOOP-13171-branch-2-005.patch, HADOOP-13171-branch-2-006.patch,
> HADOOP-13171-branch-2-007.patch, HADOOP-13171-branch-2-008.patch,
> HADOOP-13171-branch-2-009.patch, HADOOP-13171-branch-2-010.patch,
> HADOOP-13171-branch-2-011.patch, HADOOP-13171-branch-2-012.patch,
> HADOOP-13171-branch-2-013.patch, HADOOP-13171-branch-2-015.patch,
> HADOOP-13171-branch-2.8-017.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13171:
---
+1 for patch 016 on trunk. I'll commit these.
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-016.patch,
> HADOOP-13171-branch-2-001.patch, HADOOP-13171-branch-2-002.patch,
> HADOOP-13171-branch-2-003.patch, HADOOP-13171-branch-2-004.patch,
> HADOOP-13171-branch-2-005.patch, HADOOP-13171-branch-2-006.patch,
> HADOOP-13171-branch-2-007.patch, HADOOP-13171-branch-2-008.patch,
> HADOOP-13171-branch-2-009.patch, HADOOP-13171-branch-2-010.patch,
> HADOOP-13171-branch-2-011.patch, HADOOP-13171-branch-2-012.patch,
> HADOOP-13171-branch-2-013.patch, HADOOP-13171-branch-2-015.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Patch Available (was: Open)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-016.patch,
> HADOOP-13171-branch-2-001.patch, HADOOP-13171-branch-2-002.patch,
> HADOOP-13171-branch-2-003.patch, HADOOP-13171-branch-2-004.patch,
> HADOOP-13171-branch-2-005.patch, HADOOP-13171-branch-2-006.patch,
> HADOOP-13171-branch-2-007.patch, HADOOP-13171-branch-2-008.patch,
> HADOOP-13171-branch-2-009.patch, HADOOP-13171-branch-2-010.patch,
> HADOOP-13171-branch-2-011.patch, HADOOP-13171-branch-2-012.patch,
> HADOOP-13171-branch-2-013.patch, HADOOP-13171-branch-2-015.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Open (was: Patch Available)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-016.patch,
> HADOOP-13171-branch-2-001.patch, HADOOP-13171-branch-2-002.patch,
> HADOOP-13171-branch-2-003.patch, HADOOP-13171-branch-2-004.patch,
> HADOOP-13171-branch-2-005.patch, HADOOP-13171-branch-2-006.patch,
> HADOOP-13171-branch-2-007.patch, HADOOP-13171-branch-2-008.patch,
> HADOOP-13171-branch-2-009.patch, HADOOP-13171-branch-2-010.patch,
> HADOOP-13171-branch-2-011.patch, HADOOP-13171-branch-2-012.patch,
> HADOOP-13171-branch-2-013.patch, HADOOP-13171-branch-2-015.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-016.patch
Patch 016: patch 015 against trunk
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-016.patch,
> HADOOP-13171-branch-2-001.patch, HADOOP-13171-branch-2-002.patch,
> HADOOP-13171-branch-2-003.patch, HADOOP-13171-branch-2-004.patch,
> HADOOP-13171-branch-2-005.patch, HADOOP-13171-branch-2-006.patch,
> HADOOP-13171-branch-2-007.patch, HADOOP-13171-branch-2-008.patch,
> HADOOP-13171-branch-2-009.patch, HADOOP-13171-branch-2-010.patch,
> HADOOP-13171-branch-2-011.patch, HADOOP-13171-branch-2-012.patch,
> HADOOP-13171-branch-2-013.patch, HADOOP-13171-branch-2-015.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: (was: HADOOP-13171-branch-2-015.patch)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch, HADOOP-13171-branch-2-011.patch,
> HADOOP-13171-branch-2-012.patch, HADOOP-13171-branch-2-013.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-015.patch
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch, HADOOP-13171-branch-2-011.patch,
> HADOOP-13171-branch-2-012.patch, HADOOP-13171-branch-2-013.patch,
> HADOOP-13171-branch-2-015.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Patch Available (was: Open)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch, HADOOP-13171-branch-2-011.patch,
> HADOOP-13171-branch-2-012.patch, HADOOP-13171-branch-2-013.patch,
> HADOOP-13171-branch-2-015.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Open (was: Patch Available)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch, HADOOP-13171-branch-2-011.patch,
> HADOOP-13171-branch-2-012.patch, HADOOP-13171-branch-2-013.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Patch Available (was: Open)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch, HADOOP-13171-branch-2-011.patch,
> HADOOP-13171-branch-2-012.patch, HADOOP-13171-branch-2-013.patch,
> HADOOP-13171-branch-2-015.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-015.patch
patch 015: in sync with branch-2. Moved the field declaration slightly to
reduce the confict with trunk
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch, HADOOP-13171-branch-2-011.patch,
> HADOOP-13171-branch-2-012.patch, HADOOP-13171-branch-2-013.patch,
> HADOOP-13171-branch-2-015.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13171:
---
Hadoop Flags: Reviewed
+1 for patch 014 on trunk. I'll wait for a pre-commit run before committing.
bq. Tests working against S3 Ireland; couple of FNFE failures on the
{{TestS3ABlockingThreadPool}} multipart tests with parallel test runs enabled;
running non-parallel and all is well. Maybe that's one to isolate.
{{TestS3ABlockingThreadPool}} is run during the sequential phase now, so it's
odd that running in parallel mode would make it fail. Do you have a full stack
trace for that failure?
I did notice a few potential small improvements to parallel test execution, so
I filed HADOOP-13231 to track that. I wouldn't expect any of that to impact
{{TestS3ABlockingThreadPool}} though.
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch, HADOOP-13171-branch-2-011.patch,
> HADOOP-13171-branch-2-012.patch, HADOOP-13171-branch-2-013.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-014.patch
This is patch 014: the -013 patch as merged with trunk.
Tests working against S3 Ireland; couple of FNFE failures on the
{{TestS3ABlockingThreadPool}} multipart tests with parallel test runs enabled;
running non-parallel and all is well. Maybe that's one to isolate.
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch, HADOOP-13171-branch-2-011.patch,
> HADOOP-13171-branch-2-012.patch, HADOOP-13171-branch-2-013.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Open (was: Patch Available)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch, HADOOP-13171-branch-2-011.patch,
> HADOOP-13171-branch-2-012.patch, HADOOP-13171-branch-2-013.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Patch Available (was: Open)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch, HADOOP-13171-branch-2-011.patch,
> HADOOP-13171-branch-2-012.patch, HADOOP-13171-branch-2-013.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-013.patch
Patch 013: address the checkstyle complaints where applicable. The one on
function parameter count isn't covered.
I'll do a trunk build. Life would/will be easier if HADOOP-13139 got into
branch-2; the diff from the two branches would go away
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch, HADOOP-13171-branch-2-011.patch,
> HADOOP-13171-branch-2-012.patch, HADOOP-13171-branch-2-013.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-012.patch
Patch 012
this strips out the changes to the spec and test for the listFiles call.
It does retain the move of the relevant test utility classes (dir tree
creation, walking) into hadoop-common, as they will be needed there; putting
them there now means no need to revert any bits of that patch later
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch, HADOOP-13171-branch-2-011.patch,
> HADOOP-13171-branch-2-012.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Open (was: Patch Available)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch, HADOOP-13171-branch-2-011.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Patch Available (was: Open)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch, HADOOP-13171-branch-2-011.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-011.patch
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch, HADOOP-13171-branch-2-011.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-010.patch
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch,
> HADOOP-13171-branch-2-010.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-009.patch
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch, HADOOP-13171-branch-2-009.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-008.patch
Patch 008
This includes the HADOOP-13207 "specify listStatus and listFiles" work in
hadoop-common; the tests there are based on the work here, pulling them up
along with the test util classes to accompany them.
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch,
> HADOOP-13171-branch-2-008.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Open (was: Patch Available)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-007.patch
Patch 007
* lots more counting of what goes on in directory list (listFiles, listStatus,
globStatus) calls
* the full suite of getFileStatus operations
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Patch Available (was: Open)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Open (was: Patch Available)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Patch Available (was: Open)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-006.patch
HADOOP-13171 Patch 006 checkstyle, plus a bit more logging on one of the cost
tests.
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Open (was: Patch Available)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-005.patch
Patch 005: statistics on put and delete operations, more overloaded methods
(exists,() isDirectory()), test on copyFileToLocal to verify that the method
works with the metrics updated in the process.
the ProgressListener class has been factored out and cleaned up for easier
reporting; in the process the various output streams have had a lot of their
use of the s3a client moved into the S3AFilesystem; public methods are provided
which include the instrumentation counter updates.
Also fixes a possible race condition wherein the creation of empty directories
was not being awaited on, which may have been a trigger for some intermittent
race conditions
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Patch Available (was: Open)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-004.patch
Patch 004.
Synced with branch-2
Add counters for the FS API invocations, subclassing some more of the
FileSystem methods so they can be counted. The bulk list/glob calls are very
expensive; counting their use identifies calling code which could be improved.
With the precursor code in, this is ready for review
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Open (was: Patch Available)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-003.patch
Patch 003; address checkstyle and javadoc.
This patch is rebased onto HADOOP-13130 patch 008; as such it doesn't really
merit review until that patch is in...consider it a WiP for now.
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Patch Available (was: Open)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Open (was: Patch Available)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-002.patch
Patch 002; identify and fix up some checkstyle and javadoc issues before Yetus
does
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Patch Available (was: Open)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Open (was: Patch Available)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Target Version/s: 2.8.0
Status: Patch Available (was: Open)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-001.patch
Patch 001.
This patch is designed to work on branch2+ the HADOOP-13130 error handling; the
diff includes all so that it actually applies. The real diff is smaller.
# Adds a new {{S3AStorageStatistics}} class; collects storage statistics.
# Adds {{org.apache.hadoop.fs.s3a.Statistic}}: an enum of statistics supported
in both {{S3AInstrumentation}} and {{S3AStorageStatistics}}.
# Isolates the calls in {{S3AFilesystem}} to {{getObjectMetadata(}} and
{{listObjects()}} into their own methods, which increment the statistics and
instrumentation counters.
# added some tests which compare the metrics before/after some of the
filesystem operations, making assertions about their values
{{TestS3AFileStatusCost}}, and operations on directory trees
{{TestS3ADirectoryPerformance}}
I'd initially tried to bond the metrics and the statistics but backed off; it
was complex the StorageStatistics lifecycle is not per instance. Instead they
are just incremented in the same place.
There's no attempt made for any performance optimisation of {{getFileStatus()}}
or directory listing operations. I'd tried with reordering operations inside
{{getFileStatus()}} to make one fewer call when listing a directory with
children (i.e. giving that situation priority), but it didn't work, and it
wasn't clear that the optimisation would have delivered much instead.
What this patch then is the StorageStatistics and some tests.
The StorageStatistics can be picked up by any code which handles the new stats.
The tests will catch out if there's ever a regression in getFileStatus(). If
there are more checks taking place. And they will act as a starting point for
anyone looking at tuning the directory/directory tree listing operations.
Test run against AWS Ireland: in progress
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13171-branch-2-001.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Priority: Minor (was: Major)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
