[jira] [Updated] (HADOOP-13936) S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation
[ https://issues.apache.org/jira/browse/HADOOP-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13936: Target Version/s: (was: 3.2.0) > S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation > - > > Key: HADOOP-13936 > URL: https://issues.apache.org/jira/browse/HADOOP-13936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1, 3.1.0, 3.1.1 >Reporter: Rajesh Balamohan >Assignee: Steve Loughran >Priority: Blocker > > As a part of {{S3AFileSystem.delete}} operation {{innerDelete}} is invoked, > which deletes keys from S3 in batches (default is 1000). But DynamoDB is > updated only at the end of this operation. This can cause issues when > deleting large number of keys. > E.g, it is possible to get exception after deleting 1000 keys and in such > cases dynamoDB would not be updated. This can cause DynamoDB to go out of > sync. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13936) S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation
[ https://issues.apache.org/jira/browse/HADOOP-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13936: Parent Issue: HADOOP-15619 (was: HADOOP-15226) > S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation > - > > Key: HADOOP-13936 > URL: https://issues.apache.org/jira/browse/HADOOP-13936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1, 3.1.0, 3.1.1 >Reporter: Rajesh Balamohan >Assignee: Steve Loughran >Priority: Blocker > > As a part of {{S3AFileSystem.delete}} operation {{innerDelete}} is invoked, > which deletes keys from S3 in batches (default is 1000). But DynamoDB is > updated only at the end of this operation. This can cause issues when > deleting large number of keys. > E.g, it is possible to get exception after deleting 1000 keys and in such > cases dynamoDB would not be updated. This can cause DynamoDB to go out of > sync. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13936) S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation
[ https://issues.apache.org/jira/browse/HADOOP-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13936: Target Version/s: 3.2.0 > S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation > - > > Key: HADOOP-13936 > URL: https://issues.apache.org/jira/browse/HADOOP-13936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1, 3.1.0, 3.1.1 >Reporter: Rajesh Balamohan >Assignee: Steve Loughran >Priority: Blocker > > As a part of {{S3AFileSystem.delete}} operation {{innerDelete}} is invoked, > which deletes keys from S3 in batches (default is 1000). But DynamoDB is > updated only at the end of this operation. This can cause issues when > deleting large number of keys. > E.g, it is possible to get exception after deleting 1000 keys and in such > cases dynamoDB would not be updated. This can cause DynamoDB to go out of > sync. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13936) S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation
[ https://issues.apache.org/jira/browse/HADOOP-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13936: Affects Version/s: 3.1.1 3.1.0 > S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation > - > > Key: HADOOP-13936 > URL: https://issues.apache.org/jira/browse/HADOOP-13936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1, 3.1.0, 3.1.1 >Reporter: Rajesh Balamohan >Assignee: Steve Loughran >Priority: Blocker > > As a part of {{S3AFileSystem.delete}} operation {{innerDelete}} is invoked, > which deletes keys from S3 in batches (default is 1000). But DynamoDB is > updated only at the end of this operation. This can cause issues when > deleting large number of keys. > E.g, it is possible to get exception after deleting 1000 keys and in such > cases dynamoDB would not be updated. This can cause DynamoDB to go out of > sync. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13936) S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation
[ https://issues.apache.org/jira/browse/HADOOP-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13936: Priority: Blocker (was: Minor) > S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation > - > > Key: HADOOP-13936 > URL: https://issues.apache.org/jira/browse/HADOOP-13936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Priority: Blocker > > As a part of {{S3AFileSystem.delete}} operation {{innerDelete}} is invoked, > which deletes keys from S3 in batches (default is 1000). But DynamoDB is > updated only at the end of this operation. This can cause issues when > deleting large number of keys. > E.g, it is possible to get exception after deleting 1000 keys and in such > cases dynamoDB would not be updated. This can cause DynamoDB to go out of > sync. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13936) S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation
[ https://issues.apache.org/jira/browse/HADOOP-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13936: Parent Issue: HADOOP-14825 (was: HADOOP-13345) > S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation > - > > Key: HADOOP-13936 > URL: https://issues.apache.org/jira/browse/HADOOP-13936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Priority: Minor > > As a part of {{S3AFileSystem.delete}} operation {{innerDelete}} is invoked, > which deletes keys from S3 in batches (default is 1000). But DynamoDB is > updated only at the end of this operation. This can cause issues when > deleting large number of keys. > E.g, it is possible to get exception after deleting 1000 keys and in such > cases dynamoDB would not be updated. This can cause DynamoDB to go out of > sync. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13936) S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation
[ https://issues.apache.org/jira/browse/HADOOP-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13936: Affects Version/s: 3.0.0-beta1 > S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation > - > > Key: HADOOP-13936 > URL: https://issues.apache.org/jira/browse/HADOOP-13936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Rajesh Balamohan >Priority: Minor > > As a part of {{S3AFileSystem.delete}} operation {{innerDelete}} is invoked, > which deletes keys from S3 in batches (default is 1000). But DynamoDB is > updated only at the end of this operation. This can cause issues when > deleting large number of keys. > E.g, it is possible to get exception after deleting 1000 keys and in such > cases dynamoDB would not be updated. This can cause DynamoDB to go out of > sync. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org