[jira] [Commented] (HADOOP-16878) FileUtil.copy() to throw IOException if the source and destination are the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17352444#comment-17352444
]
Gabor Bota commented on HADOOP-16878:
-
I think fixing the broken functionality would be better then reverting this
commit after such a long time.
How about creating a new issue with that problem and linking it to this as
`causes`? We could fix it that way imo.
> FileUtil.copy() to throw IOException if the source and destination are the
> same
> ---
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0
>
> Attachments: hdfsTest.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16995) ITestS3AConfiguration proxy tests fail when bucket probes == 0
[
https://issues.apache.org/jira/browse/HADOOP-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16995.
-
Resolution: Fixed
got +1 from Steve on the PR, committed to trunk
> ITestS3AConfiguration proxy tests fail when bucket probes == 0
> --
>
> Key: HADOOP-16995
> URL: https://issues.apache.org/jira/browse/HADOOP-16995
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
>
> when bucket probes are disabled, proxy config tests in ITestS3AConfiguration
> fail because the probes aren't being attempted in initialize()
> {code}
>
> fs.s3a.bucket.probe
> 0
>
> {code}
> Cause: HADOOP-16711
> Fix: call unsetBaseAndBucketOverrides for bucket probe in test conf, then set
> the probe value to 2 just to be resilient to future default changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17454) [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0
[ https://issues.apache.org/jira/browse/HADOOP-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-17454. - Resolution: Fixed got +1 from Steve on PR, committed to trunk > [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 > --- > > Key: HADOOP-17454 > URL: https://issues.apache.org/jira/browse/HADOOP-17454 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Set the value of fs.s3a.bucket.probe to 0 by default. > Bucket existence checks are done in the initialization phase of the > S3AFileSystem. It's not required to run this check: the operation itself will > fail if the bucket does not exist instead of the check. > Some points on why do we want to set this to 0: > * When it's set to 0, bucket existence checks won't be done during > initialization thus making it faster. > * Avoid the additional one or two requests on the bucket root, so the user > does not need rights to read or list that folder. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17455) [s3a] Intermittent failure of ITestS3ADeleteCost.testDeleteSingleFileInDir
Gabor Bota created HADOOP-17455:
---
Summary: [s3a] Intermittent failure of
ITestS3ADeleteCost.testDeleteSingleFileInDir
Key: HADOOP-17455
URL: https://issues.apache.org/jira/browse/HADOOP-17455
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 3.3.0
Reporter: Gabor Bota
Assignee: Steve Loughran
Test failed against ireland intermittently with the following config:
{{mvn clean verify -Dparallel-tests -DtestsThreadCount=8}}
xml based config in auth-keys.xml:
{code:xml}
fs.s3a.metadatastore.impl
org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16995) ITestS3AConfiguration proxy tests fail when bucket probes == 0
[
https://issues.apache.org/jira/browse/HADOOP-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258861#comment-17258861
]
Gabor Bota commented on HADOOP-16995:
-
https://github.com/apache/hadoop/pull/2593
> ITestS3AConfiguration proxy tests fail when bucket probes == 0
> --
>
> Key: HADOOP-16995
> URL: https://issues.apache.org/jira/browse/HADOOP-16995
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
>
> when bucket probes are disabled, proxy config tests in ITestS3AConfiguration
> fail because the probes aren't being attempted in initialize()
> {code}
>
> fs.s3a.bucket.probe
> 0
>
> {code}
> Cause: HADOOP-16711
> Fix: call unsetBaseAndBucketOverrides for bucket probe in test conf, then set
> the probe value to 2 just to be resilient to future default changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17454) [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0
[ https://issues.apache.org/jira/browse/HADOOP-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258860#comment-17258860 ] Gabor Bota commented on HADOOP-17454: - https://github.com/apache/hadoop/pull/2593 > [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 > --- > > Key: HADOOP-17454 > URL: https://issues.apache.org/jira/browse/HADOOP-17454 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Set the value of fs.s3a.bucket.probe to 0 by default. > Bucket existence checks are done in the initialization phase of the > S3AFileSystem. It's not required to run this check: the operation itself will > fail if the bucket does not exist instead of the check. > Some points on why do we want to set this to 0: > * When it's set to 0, bucket existence checks won't be done during > initialization thus making it faster. > * Avoid the additional one or two requests on the bucket root, so the user > does not need rights to read or list that folder. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16995) ITestS3AConfiguration proxy tests fail when bucket probes == 0
[
https://issues.apache.org/jira/browse/HADOOP-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-16995:
Priority: Major (was: Minor)
> ITestS3AConfiguration proxy tests fail when bucket probes == 0
> --
>
> Key: HADOOP-16995
> URL: https://issues.apache.org/jira/browse/HADOOP-16995
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
>
> when bucket probes are disabled, proxy config tests in ITestS3AConfiguration
> fail because the probes aren't being attempted in initialize()
> {code}
>
> fs.s3a.bucket.probe
> 0
>
> {code}
> Cause: HADOOP-16711
> Fix: call unsetBaseAndBucketOverrides for bucket probe in test conf, then set
> the probe value to 2 just to be resilient to future default changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-17454) [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0
[ https://issues.apache.org/jira/browse/HADOOP-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-17454 started by Gabor Bota. --- > [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 > --- > > Key: HADOOP-17454 > URL: https://issues.apache.org/jira/browse/HADOOP-17454 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > Set the value of fs.s3a.bucket.probe to 0 by default. > Bucket existence checks are done in the initialization phase of the > S3AFileSystem. It's not required to run this check: the operation itself will > fail if the bucket does not exist instead of the check. > Some points on why do we want to set this to 0: > * When it's set to 0, bucket existence checks won't be done during > initialization thus making it faster. > * Avoid the additional one or two requests on the bucket root, so the user > does not need rights to read or list that folder. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17454) [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0
[ https://issues.apache.org/jira/browse/HADOOP-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-17454: Description: Set the value of fs.s3a.bucket.probe to 0 by default. Bucket existence checks are done in the initialization phase of the S3AFileSystem. It's not required to run this check: the operation itself will fail if the bucket does not exist instead of the check. Some points on why do we want to set this to 0: * When it's set to 0, bucket existence checks won't be done during initialization thus making it faster. * Avoid the additional one or two requests on the bucket root, so the user does not need rights to read or list that folder. was: Set the value of fs.s3a.bucket.probe to 0 in the code-default.xml. Bucket existence checks are done in the initialization phase of the S3AFileSystem. It's not required to run this check: the operation itself will fail if the bucket does not exist instead of the check. Some points on why do we want to set this to 0: * When it's set to 0, bucket existence checks won't be done during initialization thus making it faster. * Avoid the additional one or two requests on the bucket root, so the user does not need rights to read or list that folder. > [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 > --- > > Key: HADOOP-17454 > URL: https://issues.apache.org/jira/browse/HADOOP-17454 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > Set the value of fs.s3a.bucket.probe to 0 by default. > Bucket existence checks are done in the initialization phase of the > S3AFileSystem. It's not required to run this check: the operation itself will > fail if the bucket does not exist instead of the check. > Some points on why do we want to set this to 0: > * When it's set to 0, bucket existence checks won't be done during > initialization thus making it faster. > * Avoid the additional one or two requests on the bucket root, so the user > does not need rights to read or list that folder. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-16995) ITestS3AConfiguration proxy tests fail when bucket probes == 0
[
https://issues.apache.org/jira/browse/HADOOP-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-16995 started by Gabor Bota.
---
> ITestS3AConfiguration proxy tests fail when bucket probes == 0
> --
>
> Key: HADOOP-16995
> URL: https://issues.apache.org/jira/browse/HADOOP-16995
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
>
> when bucket probes are disabled, proxy config tests in ITestS3AConfiguration
> fail because the probes aren't being attempted in initialize()
> {code}
>
> fs.s3a.bucket.probe
> 0
>
> {code}
> Cause: HADOOP-16711
> Fix: call unsetBaseAndBucketOverrides for bucket probe in test conf, then set
> the probe value to 2 just to be resilient to future default changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16995) ITestS3AConfiguration proxy tests fail when bucket probes == 0
[
https://issues.apache.org/jira/browse/HADOOP-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota reassigned HADOOP-16995:
---
Assignee: Gabor Bota (was: Mukund Thakur)
> ITestS3AConfiguration proxy tests fail when bucket probes == 0
> --
>
> Key: HADOOP-16995
> URL: https://issues.apache.org/jira/browse/HADOOP-16995
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
>
> when bucket probes are disabled, proxy config tests in ITestS3AConfiguration
> fail because the probes aren't being attempted in initialize()
> {code}
>
> fs.s3a.bucket.probe
> 0
>
> {code}
> Cause: HADOOP-16711
> Fix: call unsetBaseAndBucketOverrides for bucket probe in test conf, then set
> the probe value to 2 just to be resilient to future default changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17454) [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0
[ https://issues.apache.org/jira/browse/HADOOP-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-17454: Release Note: S3A bucket existence check is disabled (fs.s3a.bucket.probe is 0), so there will be no existence check on the bucket during the S3AFileSystem initialization. The first operation will fail if the bucket does not exist. > [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 > --- > > Key: HADOOP-17454 > URL: https://issues.apache.org/jira/browse/HADOOP-17454 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > Set the value of fs.s3a.bucket.probe to 0 in the code-default.xml. > Bucket existence checks are done in the initialization phase of the > S3AFileSystem. It's not required to run this check: the operation itself will > fail if the bucket does not exist instead of the check. > Some points on why do we want to set this to 0: > * When it's set to 0, bucket existence checks won't be done during > initialization thus making it faster. > * Avoid the additional one or two requests on the bucket root, so the user > does not need rights to read or list that folder. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17454) [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0
[ https://issues.apache.org/jira/browse/HADOOP-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258817#comment-17258817 ] Gabor Bota commented on HADOOP-17454: - cc [~ste...@apache.org] > [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 > --- > > Key: HADOOP-17454 > URL: https://issues.apache.org/jira/browse/HADOOP-17454 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > Set the value of fs.s3a.bucket.probe to 0 in the code-default.xml. > Bucket existence checks are done in the initialization phase of the > S3AFileSystem. It's not required to run this check: the operation itself will > fail if the bucket does not exist instead of the check. > Some points on why do we want to set this to 0: > * When it's set to 0, bucket existence checks won't be done during > initialization thus making it faster. > * Avoid the additional one or two requests on the bucket root, so the user > does not need rights to read or list that folder. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17454) [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0
[ https://issues.apache.org/jira/browse/HADOOP-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-17454: Affects Version/s: 3.3.0 > [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 > --- > > Key: HADOOP-17454 > URL: https://issues.apache.org/jira/browse/HADOOP-17454 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > Set the value of fs.s3a.bucket.probe to 0 in the code-default.xml. > Bucket existence checks are done in the initialization phase of the > S3AFileSystem. It's not required to run this check: the operation itself will > fail if the bucket does not exist instead of the check. > Some points on why do we want to set this to 0: > * When it's set to 0, bucket existence checks won't be done during > initialization thus making it faster. > * Avoid the additional one or two requests on the bucket root, so the user > does not need rights to read or list that folder. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17454) [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0
Gabor Bota created HADOOP-17454: --- Summary: [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 Key: HADOOP-17454 URL: https://issues.apache.org/jira/browse/HADOOP-17454 Project: Hadoop Common Issue Type: Improvement Reporter: Gabor Bota Assignee: Gabor Bota Set the value of fs.s3a.bucket.probe to 0 in the code-default.xml. Bucket existence checks are done in the initialization phase of the S3AFileSystem. It's not required to run this check: the operation itself will fail if the bucket does not exist instead of the check. Some points on why do we want to set this to 0: * When it's set to 0, bucket existence checks won't be done during initialization thus making it faster. * Avoid the additional one or two requests on the bucket root, so the user does not need rights to read or list that folder. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16211) Update guava to 27.0-jre in hadoop-project branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-16211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16211: Labels: incompatible (was: ) > Update guava to 27.0-jre in hadoop-project branch-3.2 > - > > Key: HADOOP-16211 > URL: https://issues.apache.org/jira/browse/HADOOP-16211 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Labels: incompatible > Fix For: 3.2.1 > > Attachments: HADOOP-16211-branch-3.2.001.patch, > HADOOP-16211-branch-3.2.002.patch, HADOOP-16211-branch-3.2.003.patch, > HADOOP-16211-branch-3.2.004.patch, HADOOP-16211-branch-3.2.005.patch, > HADOOP-16211-branch-3.2.006.patch > > > com.google.guava:guava should be upgraded to 27.0-jre due to new CVE's found > CVE-2018-10237. > This is a sub-task for branch-3.2 from HADOOP-15960 to track issues on that > particular branch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16211) Update guava to 27.0-jre in hadoop-project branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-16211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195447#comment-17195447 ] Gabor Bota commented on HADOOP-16211: - Sorry [~ayushtkn], I hope that did not do any damage, and it can be marked as incompatible now > Update guava to 27.0-jre in hadoop-project branch-3.2 > - > > Key: HADOOP-16211 > URL: https://issues.apache.org/jira/browse/HADOOP-16211 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-16211-branch-3.2.001.patch, > HADOOP-16211-branch-3.2.002.patch, HADOOP-16211-branch-3.2.003.patch, > HADOOP-16211-branch-3.2.004.patch, HADOOP-16211-branch-3.2.005.patch, > HADOOP-16211-branch-3.2.006.patch > > > com.google.guava:guava should be upgraded to 27.0-jre due to new CVE's found > CVE-2018-10237. > This is a sub-task for branch-3.2 from HADOOP-15960 to track issues on that > particular branch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17050) Add support for multiple delegation tokens in S3AFilesystem
[
https://issues.apache.org/jira/browse/HADOOP-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126549#comment-17126549
]
Gabor Bota commented on HADOOP-17050:
-
so it will be the "in s3a" option from above
> Add support for multiple delegation tokens in S3AFilesystem
> ---
>
> Key: HADOOP-17050
> URL: https://issues.apache.org/jira/browse/HADOOP-17050
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Gabor Bota
>Assignee: Steve Loughran
>Priority: Major
>
> In
> {{org.apache.hadoop.fs.s3a.auth.delegation.AbstractDelegationTokenBinding}}
> the {{createDelegationToken}} should return a list of tokens.
> With this functionality, the {{AbstractDelegationTokenBinding}} can get two
> different tokens at the same time.
> {{AbstractDelegationTokenBinding.TokenSecretManager}} should be extended to
> retrieve secrets and lookup delegation tokens (use the public API for
> secretmanager in hadoop)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17050) Add support for multiple delegation tokens in S3AFilesystem
[
https://issues.apache.org/jira/browse/HADOOP-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota reassigned HADOOP-17050:
---
Assignee: Steve Loughran (was: Gabor Bota)
> Add support for multiple delegation tokens in S3AFilesystem
> ---
>
> Key: HADOOP-17050
> URL: https://issues.apache.org/jira/browse/HADOOP-17050
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Gabor Bota
>Assignee: Steve Loughran
>Priority: Major
>
> In
> {{org.apache.hadoop.fs.s3a.auth.delegation.AbstractDelegationTokenBinding}}
> the {{createDelegationToken}} should return a list of tokens.
> With this functionality, the {{AbstractDelegationTokenBinding}} can get two
> different tokens at the same time.
> {{AbstractDelegationTokenBinding.TokenSecretManager}} should be extended to
> retrieve secrets and lookup delegation tokens (use the public API for
> secretmanager in hadoop)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17050) Add support for multiple delegation tokens in S3AFilesystem
Gabor Bota created HADOOP-17050:
---
Summary: Add support for multiple delegation tokens in
S3AFilesystem
Key: HADOOP-17050
URL: https://issues.apache.org/jira/browse/HADOOP-17050
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Gabor Bota
Assignee: Gabor Bota
In {{org.apache.hadoop.fs.s3a.auth.delegation.AbstractDelegationTokenBinding}}
the {{createDelegationToken}} should return a list of tokens.
With this functionality, the {{AbstractDelegationTokenBinding}} can get two
different tokens at the same time.
{{AbstractDelegationTokenBinding.TokenSecretManager}} should be extended to
retrieve secrets and lookup delegation tokens (use the public API for
secretmanager in hadoop)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16193) add extra S3A MPU test to see what happens if a file is created during the MPU
[ https://issues.apache.org/jira/browse/HADOOP-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16193: Fix Version/s: (was: 3.1.4) 3.1.5 > add extra S3A MPU test to see what happens if a file is created during the MPU > -- > > Key: HADOOP-16193 > URL: https://issues.apache.org/jira/browse/HADOOP-16193 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.1.5 > > > Proposed extra test for the S3A MPU: if you create and then delete a file > while an MPU is in progress, when you finally complete the MPU the new data > is present. > This verifies that the other FS operations don't somehow cancel the > in-progress upload, and that eventual consistency brings the latest value out. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16917) Update dependency in branch-3.1
[ https://issues.apache.org/jira/browse/HADOOP-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16917: Target Version/s: 3.1.5 (was: 3.1.4) > Update dependency in branch-3.1 > --- > > Key: HADOOP-16917 > URL: https://issues.apache.org/jira/browse/HADOOP-16917 > Project: Hadoop Common > Issue Type: Improvement > Components: build, fs/s3 >Affects Versions: 3.1.4 >Reporter: Wei-Chiu Chuang >Priority: Blocker > Labels: release-blocker > Attachments: dependency-check-report.html > > > Jackson-databind 2.9.10.3 --> 2.10.3 > Zookeeper 3.4.13 --> 3.4.14 > hbase-client 1.2.6 --> 1.2.6.1 > aws-java-sdk-bundle 1.11.271 --> 1.11.563? (this is the version used by trunk) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16341) ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679
[ https://issues.apache.org/jira/browse/HADOOP-16341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089427#comment-17089427 ] Gabor Bota commented on HADOOP-16341: - The hadoop release 3.1.4 code freeze is today (https://cwiki.apache.org/confluence/display/HADOOP/Roadmap). Please close this issue today or move it to a different fix version. Thank you! > ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679 > -- > > Key: HADOOP-16341 > URL: https://issues.apache.org/jira/browse/HADOOP-16341 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.1.2 >Reporter: Gopal Vijayaraghavan >Assignee: Gopal Vijayaraghavan >Priority: Major > Fix For: 3.1.4 > > Attachments: HADOOP-16341.branch-3.1.002.patch, > HADOOP-16341.branch-3.1.1.patch, shutdown-hook-removal.png > > > !shutdown-hook-removal.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16917) Update dependency in branch-3.1
[ https://issues.apache.org/jira/browse/HADOOP-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089421#comment-17089421 ] Gabor Bota commented on HADOOP-16917: - The hadoop release 3.1.4 code freeze is today (https://cwiki.apache.org/confluence/display/HADOOP/Roadmap). Please close this issue today or move it to a different target branch. Thank you! > Update dependency in branch-3.1 > --- > > Key: HADOOP-16917 > URL: https://issues.apache.org/jira/browse/HADOOP-16917 > Project: Hadoop Common > Issue Type: Improvement > Components: build, fs/s3 >Affects Versions: 3.1.4 >Reporter: Wei-Chiu Chuang >Priority: Blocker > Labels: release-blocker > Attachments: dependency-check-report.html > > > Jackson-databind 2.9.10.3 --> 2.10.3 > Zookeeper 3.4.13 --> 3.4.14 > hbase-client 1.2.6 --> 1.2.6.1 > aws-java-sdk-bundle 1.11.271 --> 1.11.563? (this is the version used by trunk) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16193) add extra S3A MPU test to see what happens if a file is created during the MPU
[ https://issues.apache.org/jira/browse/HADOOP-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089424#comment-17089424 ] Gabor Bota commented on HADOOP-16193: - The hadoop release 3.1.4 code freeze is today (https://cwiki.apache.org/confluence/display/HADOOP/Roadmap). Please close this issue today or move it to a different target branch. Thank you! > add extra S3A MPU test to see what happens if a file is created during the MPU > -- > > Key: HADOOP-16193 > URL: https://issues.apache.org/jira/browse/HADOOP-16193 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.1.4 > > > Proposed extra test for the S3A MPU: if you create and then delete a file > while an MPU is in progress, when you finally complete the MPU the new data > is present. > This verifies that the other FS operations don't somehow cancel the > in-progress upload, and that eventual consistency brings the latest value out. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16473) S3Guard prune to only remove auth dir marker if files (not tombstones) are removed
[
https://issues.apache.org/jira/browse/HADOOP-16473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota reassigned HADOOP-16473:
---
Assignee: (was: Gabor Bota)
> S3Guard prune to only remove auth dir marker if files (not tombstones) are
> removed
> --
>
> Key: HADOOP-16473
> URL: https://issues.apache.org/jira/browse/HADOOP-16473
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Priority: Minor
>
> the {{s3guard prune}} command marks all dirs as non-auth if an entry was
> deleted. This makes sense from a performance perspective. But if only
> tombstones are being purged, it doesn't -all it does is hurt the performance
> of future scans
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-16961) ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics)
[ https://issues.apache.org/jira/browse/HADOOP-16961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16961 started by Gabor Bota. --- > ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) > --- > > Key: HADOOP-16961 > URL: https://issues.apache.org/jira/browse/HADOOP-16961 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) can improve the > testing and diagnostics of the connector. > Also adding some logging. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15489) S3Guard to self update on directory listings of S3
[
https://issues.apache.org/jira/browse/HADOOP-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota reassigned HADOOP-15489:
---
Assignee: (was: Gabor Bota)
> S3Guard to self update on directory listings of S3
> --
>
> Key: HADOOP-15489
> URL: https://issues.apache.org/jira/browse/HADOOP-15489
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
> Environment: s3guard
>Reporter: Steve Loughran
>Priority: Minor
>
> S3Guard updates its table on a getFileStatus call, but not on a directory
> listing.
> While this makes directory listings faster (no need to push out an update),
> it slows down subsequent queries of the files, such as a sequence of:
> {code}
> statuses = s3a.listFiles(dir)
> for (status: statuses) {
> if (status.isFile) {
> try(is = s3a.open(status.getPath())) {
> ... do something
> }
> }
> {code}
> this is because the open() is doing the getFileStatus check, even after the
> listing.
> Updating the DDB tables after a listing would give those reads a speedup,
> albeit at the expense of initiating a (bulk) update in the list call. Of
> course, we could consider making that async, though that design (essentially
> a write-buffer) would require the buffer to be checked in the reads too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16063) Docker based pseudo-cluster definitions and test scripts for Hdfs/Yarn
[ https://issues.apache.org/jira/browse/HADOOP-16063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-16063: --- Assignee: (was: Gabor Bota) > Docker based pseudo-cluster definitions and test scripts for Hdfs/Yarn > -- > > Key: HADOOP-16063 > URL: https://issues.apache.org/jira/browse/HADOOP-16063 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Marton Elek >Priority: Major > > During the recent releases of Apache Hadoop Ozone we had multiple experiments > using docker/docker-compose to support the development of ozone. > As of now the hadoop-ozone distribution contains two directories in > additional the regular hadoop directories (bin, share/lib, etc > h3. compose > The ./compose directory of the distribution contains different type of > pseudo-cluster definitions. To start an ozone cluster is as easy as "cd > compose/ozone && docker-compose up-d" > The clusters also could be scaled up and down (docker-compose scale > datanode=3) > There are multiple cluster definitions for different use cases (for example > ozone+s3 or hdfs+ozone). > The docker-compose files are based on apache/hadoop-runner image which is an > "empty" image. It doesnt' contain any hadoop distribution. Instead the > current hadoop is used (the ../.. is mapped as a volume at /opt/hadoop) > With this approach it's very easy to 1) start a cluster from the distribution > 2) test any patch from the dev tree, as after any build a new cluster can be > started easily (with multiple nodes and datanodes) > h3. smoketest > We also started to use a simple robotframework based test suite. (see > ./smoketest directory). It's a high level test definition very similar to the > smoketests which are executed manually by the contributors during a release > vote. > But it's a formal definition to start cluster from different docker-compose > definitions and execute simple shell scripts (and compare the output). > > I believe that both approaches helped a lot during the development of ozone > and I propose to do the same improvements on the main hadoop distribution. > I propose to provide docker-compose based example cluster definitions for > yarn/hdfs and for different use cases (simple hdfs, router based federation, > etc.) > It can help to understand the different configuration and try out new > features with predefined config set. > Long term we can also add robottests to help the release votes (basic > wordcount/mr tests could be scripted) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13980) S3Guard CLI: Add fsck check and fix commands
[ https://issues.apache.org/jira/browse/HADOOP-13980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-13980: --- Assignee: (was: Gabor Bota) > S3Guard CLI: Add fsck check and fix commands > > > Key: HADOOP-13980 > URL: https://issues.apache.org/jira/browse/HADOOP-13980 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Priority: Major > > As discussed in HADOOP-13650, we want to add an S3Guard CLI command which > compares S3 with MetadataStore, and returns a failure status if any > invariants are violated. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16979) S3Guard auth mode should be set to false by default in integration tests.
[ https://issues.apache.org/jira/browse/HADOOP-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16979. - Resolution: Fixed > S3Guard auth mode should be set to false by default in integration tests. > - > > Key: HADOOP-16979 > URL: https://issues.apache.org/jira/browse/HADOOP-16979 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, test >Affects Versions: 3.2.0, 3.3.0 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Minor > > As per the doc > [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md#testing-s3a-with-s3guard-enabled] > , it says if s3Guard profile is enabled all tests runs in "non-authoritative" > mode. But as per the code > [https://github.com/apache/hadoop/blob/8d49229c3764a205f21750225005a2c9a8124ab9/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java#L485] > the default value is set to true. So either we fix the doc or code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16979) S3Guard auth mode should be set to false by default in integration tests.
[ https://issues.apache.org/jira/browse/HADOOP-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084765#comment-17084765 ] Gabor Bota commented on HADOOP-16979: - LGTM +1, merging the PR #1958 > S3Guard auth mode should be set to false by default in integration tests. > - > > Key: HADOOP-16979 > URL: https://issues.apache.org/jira/browse/HADOOP-16979 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, test >Affects Versions: 3.2.0, 3.3.0 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Minor > > As per the doc > [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md#testing-s3a-with-s3guard-enabled] > , it says if s3Guard profile is enabled all tests runs in "non-authoritative" > mode. But as per the code > [https://github.com/apache/hadoop/blob/8d49229c3764a205f21750225005a2c9a8124ab9/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java#L485] > the default value is set to true. So either we fix the doc or code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16961) ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics)
[ https://issues.apache.org/jira/browse/HADOOP-16961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16961: Description: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) can improve the testing and diagnostics of the connector. Also adding some logging. was:Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) can improve the testing and diagnostics of the connector. > ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) > --- > > Key: HADOOP-16961 > URL: https://issues.apache.org/jira/browse/HADOOP-16961 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) can improve the > testing and diagnostics of the connector. > Also adding some logging. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16961) ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics)
Gabor Bota created HADOOP-16961: --- Summary: ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) Key: HADOOP-16961 URL: https://issues.apache.org/jira/browse/HADOOP-16961 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.0 Reporter: Gabor Bota Assignee: Gabor Bota Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) can improve the testing and diagnostics of the connector. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16426) S3Guard fsck: Implement fixing mechanism for inconsistencies between S3 and MetadataStore
[ https://issues.apache.org/jira/browse/HADOOP-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-16426: --- Assignee: (was: Gabor Bota) > S3Guard fsck: Implement fixing mechanism for inconsistencies between S3 and > MetadataStore > - > > Key: HADOOP-16426 > URL: https://issues.apache.org/jira/browse/HADOOP-16426 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14109) improvements to S3GuardTool destroy command
[ https://issues.apache.org/jira/browse/HADOOP-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-14109: --- Assignee: (was: Gabor Bota) > improvements to S3GuardTool destroy command > --- > > Key: HADOOP-14109 > URL: https://issues.apache.org/jira/browse/HADOOP-14109 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Priority: Minor > > The S3GuardTool destroy operation initializes dynamoDB, and in doing so has > some issues > # if the version of the table is incompatible, init fails, so table isn't > deleteable > # if the system is configured to create the table on demand, then whenever > destroy is called for a table that doesn't exist, it gets created and then > destroyed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14425) Add more s3guard metrics
[ https://issues.apache.org/jira/browse/HADOOP-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-14425: --- Assignee: (was: Gabor Bota) > Add more s3guard metrics > > > Key: HADOOP-14425 > URL: https://issues.apache.org/jira/browse/HADOOP-14425 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Ai Deng >Priority: Major > > The metrics suggested to add: > Status: > S3GUARD_METADATASTORE_ENABLED > S3GUARD_METADATASTORE_IS_AUTHORITATIVE > Operations: > S3GUARD_METADATASTORE_INITIALIZATION > S3GUARD_METADATASTORE_DELETE_PATH > S3GUARD_METADATASTORE_DELETE_PATH_LATENCY > S3GUARD_METADATASTORE_DELETE_SUBTREE_PATCH > S3GUARD_METADATASTORE_GET_PATH > S3GUARD_METADATASTORE_GET_PATH_LATENCY > S3GUARD_METADATASTORE_GET_CHILDREN_PATH > S3GUARD_METADATASTORE_GET_CHILDREN_PATH_LATENCY > S3GUARD_METADATASTORE_MOVE_PATH > S3GUARD_METADATASTORE_PUT_PATH > S3GUARD_METADATASTORE_PUT_PATH_LATENCY > S3GUARD_METADATASTORE_CLOSE > S3GUARD_METADATASTORE_DESTORY > From S3Guard: > S3GUARD_METADATASTORE_MERGE_DIRECTORY > For the failures: > S3GUARD_METADATASTORE_DELETE_FAILURE > S3GUARD_METADATASTORE_GET_FAILURE > S3GUARD_METADATASTORE_PUT_FAILURE > Etc: > S3GUARD_METADATASTORE_PUT_RETRY_TIMES -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15779) S3guard: add inconsistency detection metrics
[ https://issues.apache.org/jira/browse/HADOOP-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-15779: --- Assignee: (was: Gabor Bota) > S3guard: add inconsistency detection metrics > > > Key: HADOOP-15779 > URL: https://issues.apache.org/jira/browse/HADOOP-15779 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Aaron Fabbri >Priority: Major > > S3Guard uses a trailing log of metadata changes made to an S3 bucket to add > consistency to the eventually-consistent AWS S3 service. We should add some > metrics that are incremented when we detect inconsistency (eventual > consistency) in S3. > I'm thinking at least two counters: (1) getFileStatus() (HEAD) inconsistency > detected, and (2) listing inconsistency detected. We may want to further > separate into categories (present / not present etc.) > This is related to Auth. Mode and TTL work that is ongoing, so let me outline > how I think this should all evolve: > This should happen after HADOOP-15621 (TTL for dynamo MetadataStore), since > that will change *when* we query both S3 and the MetadataStore (e.g. Dynamo) > for metadata. There I suggest that: > # Prune time is different than TTL. Prune time: "how long until > inconsistency is no longer a problem" . TTL time "how long a MetadataStore > entry is considered authoritative before refresh" > # Prune expired: delete entries (when hadoop CLI prune command is run). TTL > Expired: entries become non-authoritative. > # Prune implemented in each MetadataStore, but TTL filtering happens in S3A. > Once we have this, S3A will be consulting both S3 and MetadataStore depending > on configuration and/or age of the entry in the MetadataStore. Today > HEAD/getFileStatus() is always short-circuit (skips S3 if MetadataStore > returns results). I think S3A should consult both when TTL is stale, and > invoke a callback on inconsistency that increments the new metrics. For > listing, we already are comparing both sources of truth (except when S3A auth > mode is on and a directory is marked authoritative in MS), so it would be > pretty simple to invoke a callback on inconsistency and bump some metrics. > Comments / suggestions / questions welcomed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16425) S3Guard fsck: Export MetadataStore and S3 bucket hierarchy
[ https://issues.apache.org/jira/browse/HADOOP-16425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-16425: --- Assignee: (was: Gabor Bota) > S3Guard fsck: Export MetadataStore and S3 bucket hierarchy > -- > > Key: HADOOP-16425 > URL: https://issues.apache.org/jira/browse/HADOOP-16425 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Priority: Major > > The export should be done in a human-readable format like csv -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16910) ABFS Streams to update FileSystem.Statistics counters on IO.
[ https://issues.apache.org/jira/browse/HADOOP-16910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071751#comment-17071751 ] Gabor Bota commented on HADOOP-16910: - compile error fixed. committed. > ABFS Streams to update FileSystem.Statistics counters on IO. > > > Key: HADOOP-16910 > URL: https://issues.apache.org/jira/browse/HADOOP-16910 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Mehakmeet Singh >Assignee: Mehakmeet Singh >Priority: Major > Fix For: 3.3.1 > > > Abfs FileSystem Counters are not populated and hence not shown on the console > side. > purpose: > * Passing Statistics in AbfsOutputStream and populating FileSystem > Counter(Write_ops) there. > * Populating Read_ops in AbfsInputStream. > * Showing Bytes_written, Bytes_read, Write_ops and Read_ops on the console > while using ABFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16910) ABFS Streams to update FileSystem.Statistics counters on IO.
[ https://issues.apache.org/jira/browse/HADOOP-16910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16910. - Resolution: Fixed > ABFS Streams to update FileSystem.Statistics counters on IO. > > > Key: HADOOP-16910 > URL: https://issues.apache.org/jira/browse/HADOOP-16910 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Mehakmeet Singh >Assignee: Mehakmeet Singh >Priority: Major > Fix For: 3.3.1 > > > Abfs FileSystem Counters are not populated and hence not shown on the console > side. > purpose: > * Passing Statistics in AbfsOutputStream and populating FileSystem > Counter(Write_ops) there. > * Populating Read_ops in AbfsInputStream. > * Showing Bytes_written, Bytes_read, Write_ops and Read_ops on the console > while using ABFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071074#comment-17071074
]
Gabor Bota commented on HADOOP-16878:
-
Yeah, I already updated the PR with that, but I cannot see this failing, which
is strange.
> Copy command in FileUtil to throw an exception if the source and destination
> is the same
>
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Attachments: hdfsTest.patch
>
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071038#comment-17071038
]
Gabor Bota commented on HADOOP-16878:
-
[~ayushtkn], I was not able to reproduce this test failure. How did you run the
test?
I added a check to be sure that {{dst}} and {{src}} are not null before
calling .toUri(), but I'd be interested in how to reproduce the issue with
{{testCopyBetweenFsEqualPath}}
.
> Copy command in FileUtil to throw an exception if the source and destination
> is the same
>
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-16878:
Status: Patch Available (was: In Progress)
> Copy command in FileUtil to throw an exception if the source and destination
> is the same
>
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066757#comment-17066757
]
Gabor Bota commented on HADOOP-16878:
-
PR is up with the small change and test.
> Copy command in FileUtil to throw an exception if the source and destination
> is the same
>
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14918) Remove the Local Dynamo DB test option
[ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-14918: Resolution: Fixed Status: Resolved (was: Patch Available) > Remove the Local Dynamo DB test option > -- > > Key: HADOOP-14918 > URL: https://issues.apache.org/jira/browse/HADOOP-14918 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-14918-001.patch, HADOOP-14918-002.patch, > HADOOP-14918-003.patch, HADOOP-14918-004.patch, > HADOOP-14918-branch-2.10.001.patch, HADOOP-14918.005.patch, > HADOOP-14918.006.patch > > > I'm going to propose cutting out the localdynamo test option for s3guard > * the local DDB JAR is unmaintained/lags the SDK We work with...eventually > there'll be differences in API. > * as the local dynamo DB is unshaded. it complicates classpath setup for the > build. Remove it and there's no need to worry about versions of anything > other than the shaded AWS > * it complicates test runs. Now we need to test for both localdynamo *and* > real dynamo > * but we can't ignore real dynamo, because that's the one which matters > While the local option promises to reduce test costs, really, it's just > adding complexity. If you are testing with s3guard, you need to have a real > table to test against., And with the exception of those people testing s3a > against non-AWS, consistent endpoints, everyone should be testing with > S3Guard. > -Straightforward to remove.- -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14918) Remove the Local Dynamo DB test option
[ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-14918: Fix Version/s: 2.10.1 > Remove the Local Dynamo DB test option > -- > > Key: HADOOP-14918 > URL: https://issues.apache.org/jira/browse/HADOOP-14918 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Fix For: 3.2.0, 2.10.1 > > Attachments: HADOOP-14918-001.patch, HADOOP-14918-002.patch, > HADOOP-14918-003.patch, HADOOP-14918-004.patch, > HADOOP-14918-branch-2.10.001.patch, HADOOP-14918.005.patch, > HADOOP-14918.006.patch > > > I'm going to propose cutting out the localdynamo test option for s3guard > * the local DDB JAR is unmaintained/lags the SDK We work with...eventually > there'll be differences in API. > * as the local dynamo DB is unshaded. it complicates classpath setup for the > build. Remove it and there's no need to worry about versions of anything > other than the shaded AWS > * it complicates test runs. Now we need to test for both localdynamo *and* > real dynamo > * but we can't ignore real dynamo, because that's the one which matters > While the local option promises to reduce test costs, really, it's just > adding complexity. If you are testing with s3guard, you need to have a real > table to test against., And with the exception of those people testing s3a > against non-AWS, consistent endpoints, everyone should be testing with > S3Guard. > -Straightforward to remove.- -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14918) Remove the Local Dynamo DB test option
[ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064787#comment-17064787 ] Gabor Bota commented on HADOOP-14918: - yes, I'm doing that. sorry for the delay. > Remove the Local Dynamo DB test option > -- > > Key: HADOOP-14918 > URL: https://issues.apache.org/jira/browse/HADOOP-14918 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-14918-001.patch, HADOOP-14918-002.patch, > HADOOP-14918-003.patch, HADOOP-14918-004.patch, > HADOOP-14918-branch-2.10.001.patch, HADOOP-14918.005.patch, > HADOOP-14918.006.patch > > > I'm going to propose cutting out the localdynamo test option for s3guard > * the local DDB JAR is unmaintained/lags the SDK We work with...eventually > there'll be differences in API. > * as the local dynamo DB is unshaded. it complicates classpath setup for the > build. Remove it and there's no need to worry about versions of anything > other than the shaded AWS > * it complicates test runs. Now we need to test for both localdynamo *and* > real dynamo > * but we can't ignore real dynamo, because that's the one which matters > While the local option promises to reduce test costs, really, it's just > adding complexity. If you are testing with s3guard, you need to have a real > table to test against., And with the exception of those people testing s3a > against non-AWS, consistent endpoints, everyone should be testing with > S3Guard. > -Straightforward to remove.- -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16858) S3Guard fsck: Add option to remove orphaned entries
[ https://issues.apache.org/jira/browse/HADOOP-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16858. - Resolution: Fixed > S3Guard fsck: Add option to remove orphaned entries > --- > > Key: HADOOP-16858 > URL: https://issues.apache.org/jira/browse/HADOOP-16858 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Fix For: 3.3.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16858) S3Guard fsck: Add option to remove orphaned entries
[ https://issues.apache.org/jira/browse/HADOOP-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061655#comment-17061655 ] Gabor Bota commented on HADOOP-16858: - Got +1 from @stevel on PR #1851, merging. > S3Guard fsck: Add option to remove orphaned entries > --- > > Key: HADOOP-16858 > URL: https://issues.apache.org/jira/browse/HADOOP-16858 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16858) S3Guard fsck: Add option to remove orphaned entries
[ https://issues.apache.org/jira/browse/HADOOP-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16858: Fix Version/s: 3.3.1 > S3Guard fsck: Add option to remove orphaned entries > --- > > Key: HADOOP-16858 > URL: https://issues.apache.org/jira/browse/HADOOP-16858 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Fix For: 3.3.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16769) LocalDirAllocator to provide diagnostics when file creation fails
[ https://issues.apache.org/jira/browse/HADOOP-16769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17055096#comment-17055096 ] Gabor Bota commented on HADOOP-16769: - Sure, but there are some comments already in the PR which need to be addressed. Please resolve those and update the PR first. > LocalDirAllocator to provide diagnostics when file creation fails > - > > Key: HADOOP-16769 > URL: https://issues.apache.org/jira/browse/HADOOP-16769 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Reporter: Ramesh Kumar Thangarajan >Priority: Minor > Attachments: HADOOP-16769.1.patch, HADOOP-16769.3.patch, > HADOOP-16769.4.patch, HADOOP-16769.5.patch, HADOOP-16769.6.patch, > HADOOP-16769.7.patch > > > Log details of requested size and available capacity when file creation is > not successuful -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16904) ABFS test failures when running with a storage account without a hierarchical namespace
[
https://issues.apache.org/jira/browse/HADOOP-16904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-16904:
Component/s: fs/azure
> ABFS test failures when running with a storage account without a hierarchical
> namespace
> ---
>
> Key: HADOOP-16904
> URL: https://issues.apache.org/jira/browse/HADOOP-16904
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> Some tests are failing when running on a storage account without hierarchical
> namespace:
> {noformat}
> [ERROR] Failures:
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testSetOwnerUnauthorized:212->runTest:274
> Expected an exception of type class
> org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testSetPermissionUnauthorized:217->runTest:274
> Expected an exception of type class
> org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException
> [ERROR]
> ITestGetNameSpaceEnabled.testXNSAccount:51->Assert.assertTrue:41->Assert.fail:88
> Expecting getIsNamespaceEnabled() return true
> [ERROR] Errors:
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testGetAclStatusAuthorized:247->runTest:278->executeOp:298
> ? UnsupportedOperation
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testGetAclStatusUnauthorized:252->runTest:274->lambda$runTest$4:275->executeOp:298
> ? UnsupportedOperation
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testModifyAclEntriesUnauthorized:222->runTest:274->lambda$runTest$4:275->executeOp:335
> ? UnsupportedOperation
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testRemoveAclEntriesUnauthorized:227->runTest:274->lambda$runTest$4:275->executeOp:331
> ? UnsupportedOperation
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testRemoveAclUnauthorized:237->runTest:274->lambda$runTest$4:275->executeOp:339
> ? UnsupportedOperation
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testRemoveDefaultAclUnauthorized:232->runTest:274->lambda$runTest$4:275->executeOp:342
> ? UnsupportedOperation
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testSetAclUnauthorized:242->runTest:274->lambda$runTest$4:275->executeOp:304
> ? UnsupportedOperation
> [ERROR]
> ITestAzureBlobFileSystemCheckAccess.testWhenCheckAccessConfigIsOff:125->setupTestDirectoryAndUserAccess:313->modifyAcl:304
> ? UnsupportedOperation
> {noformat}
> This should be fixed, so these tests will be skipped or will fail with a more
> meaningful error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16904) ABFS test failures when running with a storage account without a hierarchical namespace
[
https://issues.apache.org/jira/browse/HADOOP-16904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051359#comment-17051359
]
Gabor Bota commented on HADOOP-16904:
-
FYI [~ste...@apache.org]
> ABFS test failures when running with a storage account without a hierarchical
> namespace
> ---
>
> Key: HADOOP-16904
> URL: https://issues.apache.org/jira/browse/HADOOP-16904
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> Some tests are failing when running on a storage account without hierarchical
> namespace:
> {noformat}
> [ERROR] Failures:
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testSetOwnerUnauthorized:212->runTest:274
> Expected an exception of type class
> org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testSetPermissionUnauthorized:217->runTest:274
> Expected an exception of type class
> org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException
> [ERROR]
> ITestGetNameSpaceEnabled.testXNSAccount:51->Assert.assertTrue:41->Assert.fail:88
> Expecting getIsNamespaceEnabled() return true
> [ERROR] Errors:
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testGetAclStatusAuthorized:247->runTest:278->executeOp:298
> ? UnsupportedOperation
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testGetAclStatusUnauthorized:252->runTest:274->lambda$runTest$4:275->executeOp:298
> ? UnsupportedOperation
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testModifyAclEntriesUnauthorized:222->runTest:274->lambda$runTest$4:275->executeOp:335
> ? UnsupportedOperation
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testRemoveAclEntriesUnauthorized:227->runTest:274->lambda$runTest$4:275->executeOp:331
> ? UnsupportedOperation
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testRemoveAclUnauthorized:237->runTest:274->lambda$runTest$4:275->executeOp:339
> ? UnsupportedOperation
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testRemoveDefaultAclUnauthorized:232->runTest:274->lambda$runTest$4:275->executeOp:342
> ? UnsupportedOperation
> [ERROR]
> ITestAzureBlobFileSystemAuthorization.testSetAclUnauthorized:242->runTest:274->lambda$runTest$4:275->executeOp:304
> ? UnsupportedOperation
> [ERROR]
> ITestAzureBlobFileSystemCheckAccess.testWhenCheckAccessConfigIsOff:125->setupTestDirectoryAndUserAccess:313->modifyAcl:304
> ? UnsupportedOperation
> {noformat}
> This should be fixed, so these tests will be skipped or will fail with a more
> meaningful error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16904) ABFS test failures when running with a storage account without a hierarchical namespace
Gabor Bota created HADOOP-16904:
---
Summary: ABFS test failures when running with a storage account
without a hierarchical namespace
Key: HADOOP-16904
URL: https://issues.apache.org/jira/browse/HADOOP-16904
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.3.0
Reporter: Gabor Bota
Assignee: Gabor Bota
Some tests are failing when running on a storage account without hierarchical
namespace:
{noformat}
[ERROR] Failures:
[ERROR]
ITestAzureBlobFileSystemAuthorization.testSetOwnerUnauthorized:212->runTest:274
Expected an exception of type class
org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException
[ERROR]
ITestAzureBlobFileSystemAuthorization.testSetPermissionUnauthorized:217->runTest:274
Expected an exception of type class
org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException
[ERROR]
ITestGetNameSpaceEnabled.testXNSAccount:51->Assert.assertTrue:41->Assert.fail:88
Expecting getIsNamespaceEnabled() return true
[ERROR] Errors:
[ERROR]
ITestAzureBlobFileSystemAuthorization.testGetAclStatusAuthorized:247->runTest:278->executeOp:298
? UnsupportedOperation
[ERROR]
ITestAzureBlobFileSystemAuthorization.testGetAclStatusUnauthorized:252->runTest:274->lambda$runTest$4:275->executeOp:298
? UnsupportedOperation
[ERROR]
ITestAzureBlobFileSystemAuthorization.testModifyAclEntriesUnauthorized:222->runTest:274->lambda$runTest$4:275->executeOp:335
? UnsupportedOperation
[ERROR]
ITestAzureBlobFileSystemAuthorization.testRemoveAclEntriesUnauthorized:227->runTest:274->lambda$runTest$4:275->executeOp:331
? UnsupportedOperation
[ERROR]
ITestAzureBlobFileSystemAuthorization.testRemoveAclUnauthorized:237->runTest:274->lambda$runTest$4:275->executeOp:339
? UnsupportedOperation
[ERROR]
ITestAzureBlobFileSystemAuthorization.testRemoveDefaultAclUnauthorized:232->runTest:274->lambda$runTest$4:275->executeOp:342
? UnsupportedOperation
[ERROR]
ITestAzureBlobFileSystemAuthorization.testSetAclUnauthorized:242->runTest:274->lambda$runTest$4:275->executeOp:304
? UnsupportedOperation
[ERROR]
ITestAzureBlobFileSystemCheckAccess.testWhenCheckAccessConfigIsOff:125->setupTestDirectoryAndUserAccess:313->modifyAcl:304
? UnsupportedOperation
{noformat}
This should be fixed, so these tests will be skipped or will fail with a more
meaningful error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16507) S3Guard fsck: Add option to configure severity (level) for the scan
[
https://issues.apache.org/jira/browse/HADOOP-16507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051353#comment-17051353
]
Gabor Bota commented on HADOOP-16507:
-
Maybe worth to do this jira as well:
https://issues.apache.org/jira/browse/HADOOP-16745
> S3Guard fsck: Add option to configure severity (level) for the scan
> ---
>
> Key: HADOOP-16507
> URL: https://issues.apache.org/jira/browse/HADOOP-16507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Mukund Thakur
>Priority: Major
>
> There's the severity of Violation (inconsistency) defined in
> {{org.apache.hadoop.fs.s3a.s3guard.S3GuardFsck.Violation}}.
> This flag is only for defining the severity of the Violation, but not used to
> filter the scan for issue severity.
> The task to do: Use the severity level to define which issue should be logged
> and/or fixed during the scan.
> Note: the best way to avoid possible code duplication would be to not even
> add the consistency violation pair to the list of violations during the scan.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16745) S3Guard fsck: Use enum instead of int for severity in violations
[ https://issues.apache.org/jira/browse/HADOOP-16745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-16745: --- Assignee: (was: Gabor Bota) > S3Guard fsck: Use enum instead of int for severity in violations > > > Key: HADOOP-16745 > URL: https://issues.apache.org/jira/browse/HADOOP-16745 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Gabor Bota >Priority: Major > > Looking in the code, I can see that it's a severity level "2", so not that > serious. > # We should go away from and int value here to something comprehensible, i.e. > an enum. > # Or at least some contents used in the Violation enum setup. > # Either way, how about letting us choose which level to fail on. > # And maybe severity 2 should be logged at WARN, not ERROR, as an early user > was confused "why an error yet none reported in status" -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14936) S3Guard: remove "experimental" from documentation
[ https://issues.apache.org/jira/browse/HADOOP-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-14936. - Resolution: Fixed > S3Guard: remove "experimental" from documentation > - > > Key: HADOOP-14936 > URL: https://issues.apache.org/jira/browse/HADOOP-14936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Gabor Bota >Priority: Major > > I think it is time to remove the "experimental feature" designation in the > site docs for S3Guard. Discuss. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14936) S3Guard: remove "experimental" from documentation
[ https://issues.apache.org/jira/browse/HADOOP-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-14936: --- Assignee: Gabor Bota (was: Steve Loughran) > S3Guard: remove "experimental" from documentation > - > > Key: HADOOP-14936 > URL: https://issues.apache.org/jira/browse/HADOOP-14936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Gabor Bota >Priority: Major > > I think it is time to remove the "experimental feature" designation in the > site docs for S3Guard. Discuss. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14936) S3Guard: remove "experimental" from documentation
[ https://issues.apache.org/jira/browse/HADOOP-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046741#comment-17046741 ] Gabor Bota commented on HADOOP-14936: - [~ste...@apache.org] asked me offline to add a PR for this. > S3Guard: remove "experimental" from documentation > - > > Key: HADOOP-14936 > URL: https://issues.apache.org/jira/browse/HADOOP-14936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Steve Loughran >Priority: Major > > I think it is time to remove the "experimental feature" designation in the > site docs for S3Guard. Discuss. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-14936) S3Guard: remove "experimental" from documentation
[ https://issues.apache.org/jira/browse/HADOOP-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-14936 started by Gabor Bota. --- > S3Guard: remove "experimental" from documentation > - > > Key: HADOOP-14936 > URL: https://issues.apache.org/jira/browse/HADOOP-14936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Gabor Bota >Priority: Major > > I think it is time to remove the "experimental feature" designation in the > site docs for S3Guard. Discuss. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14918) Remove the Local Dynamo DB test option
[ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046740#comment-17046740 ] Gabor Bota edited comment on HADOOP-14918 at 2/27/20 3:34 PM: -- Thanks for [~jhung] for working on this. Could you create a PR for your patch? We prefer using PRs for merging. And reviews. was (Author: gabor.bota): Thanks for [~jhung] for working on this. Could you create a PR for your patch? We prefer using PRs for merging. > Remove the Local Dynamo DB test option > -- > > Key: HADOOP-14918 > URL: https://issues.apache.org/jira/browse/HADOOP-14918 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-14918-001.patch, HADOOP-14918-002.patch, > HADOOP-14918-003.patch, HADOOP-14918-004.patch, > HADOOP-14918-branch-2.10.001.patch, HADOOP-14918.005.patch, > HADOOP-14918.006.patch > > > I'm going to propose cutting out the localdynamo test option for s3guard > * the local DDB JAR is unmaintained/lags the SDK We work with...eventually > there'll be differences in API. > * as the local dynamo DB is unshaded. it complicates classpath setup for the > build. Remove it and there's no need to worry about versions of anything > other than the shaded AWS > * it complicates test runs. Now we need to test for both localdynamo *and* > real dynamo > * but we can't ignore real dynamo, because that's the one which matters > While the local option promises to reduce test costs, really, it's just > adding complexity. If you are testing with s3guard, you need to have a real > table to test against., And with the exception of those people testing s3a > against non-AWS, consistent endpoints, everyone should be testing with > S3Guard. > -Straightforward to remove.- -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14918) Remove the Local Dynamo DB test option
[ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046740#comment-17046740 ] Gabor Bota commented on HADOOP-14918: - Thanks for [~jhung] for working on this. Could you create a PR for your patch? We prefer using PRs for merging. > Remove the Local Dynamo DB test option > -- > > Key: HADOOP-14918 > URL: https://issues.apache.org/jira/browse/HADOOP-14918 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-14918-001.patch, HADOOP-14918-002.patch, > HADOOP-14918-003.patch, HADOOP-14918-004.patch, > HADOOP-14918-branch-2.10.001.patch, HADOOP-14918.005.patch, > HADOOP-14918.006.patch > > > I'm going to propose cutting out the localdynamo test option for s3guard > * the local DDB JAR is unmaintained/lags the SDK We work with...eventually > there'll be differences in API. > * as the local dynamo DB is unshaded. it complicates classpath setup for the > build. Remove it and there's no need to worry about versions of anything > other than the shaded AWS > * it complicates test runs. Now we need to test for both localdynamo *and* > real dynamo > * but we can't ignore real dynamo, because that's the one which matters > While the local option promises to reduce test costs, really, it's just > adding complexity. If you are testing with s3guard, you need to have a real > table to test against., And with the exception of those people testing s3a > against non-AWS, consistent endpoints, everyone should be testing with > S3Guard. > -Straightforward to remove.- -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-16878 started by Gabor Bota.
---
> Copy command in FileUtil to throw an exception if the source and destination
> is the same
>
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-16878:
Summary: Copy command in FileUtil to throw an exception if the source and
destination is the same (was: Copy command in FileUtil to LOG at warn level if
the source and destination is the same)
> Copy command in FileUtil to throw an exception if the source and destination
> is the same
>
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16878) Copy command in FileUtil to LOG at warn level if the source and destination is the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17045544#comment-17045544
]
Gabor Bota commented on HADOOP-16878:
-
That is true, we should check for the fs as well not just the path.
> Copy command in FileUtil to LOG at warn level if the source and destination
> is the same
> ---
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16878) Copy command in FileUtil to LOG at warn level if the source and destination is the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-16878:
Description:
We encountered an error during a test in our QE when the file destination and
source path were the same. This happened during an ADLS test, and there were no
meaningful error messages, so it was hard to find the root cause of the failure.
The error we saw was that file size has changed during the copy operation. The
new file creation in the destination - which is the same as the source -
creates a file and sets the file length to zero. After this, getting the source
file will fail because the sile size changed during the operation.
I propose a solution to at least log in error level in the {{FileUtil}} if the
source and destination of the copy operation is the same, so debugging issues
like this will be easier in the future.
was:
We encountered an error during a test in our QE when the file destination and
source path were the same. This happened during an ADLS test, and there were no
meaningful error messages, so it was hard to find the root cause of the failure.
The error we saw was that file size has changed during the copy operation - the
new file creation in the destination, which is the same as the source, creates
a file and sets the file length to zero. After this, getting the source file
will fail because the sile size changed during the operation.
I propose a solution to at least log in error level in the {{FileUtil}} if the
source and destination of the copy operation is the same, so debugging issues
like this will be easier in the future.
> Copy command in FileUtil to LOG at warn level if the source and destination
> is the same
> ---
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation.
> The new file creation in the destination - which is the same as the source -
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16878) Copy command in FileUtil to LOG at warn level if the source and destination is the same
[
https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-16878:
Description:
We encountered an error during a test in our QE when the file destination and
source path were the same. This happened during an ADLS test, and there were no
meaningful error messages, so it was hard to find the root cause of the failure.
The error we saw was that file size has changed during the copy operation - the
new file creation in the destination, which is the same as the source, creates
a file and sets the file length to zero. After this, getting the source file
will fail because the sile size changed during the operation.
I propose a solution to at least log in error level in the {{FileUtil}} if the
source and destination of the copy operation is the same, so debugging issues
like this will be easier in the future.
was:
We encountered an error during a test in our QE when the file destination and
source path were the same. This happened during an ADLS test, and there were no
meaningful error messages, so it was hard to find the root cause of the failure.
The error we saw was that file size has changed during the copy operation - the
new file creation in the destination, which is the same as the source, creates
a new file and sets the file length to zero. After this, getting the source
file will fail because the sile size changed during the operation.
I propose a solution to at least log in error level in the {{FileUtil}} if the
source and destination of the copy operation is the same, so debugging issues
like this will be easier in the future.
> Copy command in FileUtil to LOG at warn level if the source and destination
> is the same
> ---
>
> Key: HADOOP-16878
> URL: https://issues.apache.org/jira/browse/HADOOP-16878
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> We encountered an error during a test in our QE when the file destination and
> source path were the same. This happened during an ADLS test, and there were
> no meaningful error messages, so it was hard to find the root cause of the
> failure.
> The error we saw was that file size has changed during the copy operation -
> the new file creation in the destination, which is the same as the source,
> creates a file and sets the file length to zero. After this, getting the
> source file will fail because the sile size changed during the operation.
> I propose a solution to at least log in error level in the {{FileUtil}} if
> the source and destination of the copy operation is the same, so debugging
> issues like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16878) Copy command in FileUtil to LOG at warn level if the source and destination is the same
Gabor Bota created HADOOP-16878:
---
Summary: Copy command in FileUtil to LOG at warn level if the
source and destination is the same
Key: HADOOP-16878
URL: https://issues.apache.org/jira/browse/HADOOP-16878
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 3.3.0
Reporter: Gabor Bota
Assignee: Gabor Bota
We encountered an error during a test in our QE when the file destination and
source path were the same. This happened during an ADLS test, and there were no
meaningful error messages, so it was hard to find the root cause of the failure.
The error we saw was that file size has changed during the copy operation - the
new file creation in the destination, which is the same as the source, creates
a new file and sets the file length to zero. After this, getting the source
file will fail because the sile size changed during the operation.
I propose a solution to at least log in error level in the {{FileUtil}} if the
source and destination of the copy operation is the same, so debugging issues
like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16222) Fix new deprecations after guava 27.0 update in trunk
[
https://issues.apache.org/jira/browse/HADOOP-16222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-16222:
Fix Version/s: 3.3.0
> Fix new deprecations after guava 27.0 update in trunk
> -
>
> Key: HADOOP-16222
> URL: https://issues.apache.org/jira/browse/HADOOP-16222
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16222.001.patch, HADOOP-16222.002.patch,
> HADOOP-16222.003.patch, HADOOP-16222.004.patch
>
>
> *Note*: this can be done after the guava update.
> There are a bunch of new deprecations after the guava update. We need to fix
> those, because these will be removed after the next guava version (after 27).
> I created a separate jira for this from HADOOP-16210 because jenkins
> pre-commit test job (yetus) will time-out after 5 hours after running this
> together.
> {noformat}
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SemaphoredDelegatingExecutor.java:[110,20]
> [deprecation] immediateFailedCheckedFuture(X) in Futures has been
> deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ZKUtil.java:[175,16]
> [deprecation] toString(File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java:[44,9]
> [deprecation] write(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java:[67,9]
> [deprecation] write(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java:[131,9]
> [deprecation] write(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java:[150,9]
> [deprecation] write(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java:[169,9]
> [deprecation] write(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestZKUtil.java:[134,9]
> [deprecation] write(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java:[437,9]
> [deprecation] write(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java:[211,26]
> [deprecation] toString(File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java:[219,36]
> [deprecation] toString(File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/fpga/TestFpgaResourceHandler.java:[130,9]
> [deprecation] append(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/fpga/TestFpgaResourceHandler.java:[352,9]
> [deprecation] append(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java:[1161,18]
> [deprecation] propagate(Throwable) in Throwables has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/ServiceTestUtils.java:[413,18]
> [deprecation] propagate(Throwable) in Throwables has been deprecated
> {noformat}
> Maybe fix these by module by module instead of a single patch?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe,
[jira] [Commented] (HADOOP-16222) Fix new deprecations after guava 27.0 update in trunk
[
https://issues.apache.org/jira/browse/HADOOP-16222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043370#comment-17043370
]
Gabor Bota commented on HADOOP-16222:
-
Done.
> Fix new deprecations after guava 27.0 update in trunk
> -
>
> Key: HADOOP-16222
> URL: https://issues.apache.org/jira/browse/HADOOP-16222
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16222.001.patch, HADOOP-16222.002.patch,
> HADOOP-16222.003.patch, HADOOP-16222.004.patch
>
>
> *Note*: this can be done after the guava update.
> There are a bunch of new deprecations after the guava update. We need to fix
> those, because these will be removed after the next guava version (after 27).
> I created a separate jira for this from HADOOP-16210 because jenkins
> pre-commit test job (yetus) will time-out after 5 hours after running this
> together.
> {noformat}
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SemaphoredDelegatingExecutor.java:[110,20]
> [deprecation] immediateFailedCheckedFuture(X) in Futures has been
> deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ZKUtil.java:[175,16]
> [deprecation] toString(File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java:[44,9]
> [deprecation] write(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java:[67,9]
> [deprecation] write(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java:[131,9]
> [deprecation] write(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java:[150,9]
> [deprecation] write(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java:[169,9]
> [deprecation] write(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestZKUtil.java:[134,9]
> [deprecation] write(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java:[437,9]
> [deprecation] write(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java:[211,26]
> [deprecation] toString(File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java:[219,36]
> [deprecation] toString(File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/fpga/TestFpgaResourceHandler.java:[130,9]
> [deprecation] append(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/fpga/TestFpgaResourceHandler.java:[352,9]
> [deprecation] append(CharSequence,File,Charset) in Files has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java:[1161,18]
> [deprecation] propagate(Throwable) in Throwables has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/ServiceTestUtils.java:[413,18]
> [deprecation] propagate(Throwable) in Throwables has been deprecated
> {noformat}
> Maybe fix these by module by module instead of a single patch?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
[jira] [Updated] (HADOOP-13980) S3Guard CLI: Add fsck check and fix commands
[ https://issues.apache.org/jira/browse/HADOOP-13980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-13980: Summary: S3Guard CLI: Add fsck check and fix commands (was: S3Guard CLI: Add fsck check command) > S3Guard CLI: Add fsck check and fix commands > > > Key: HADOOP-13980 > URL: https://issues.apache.org/jira/browse/HADOOP-13980 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Gabor Bota >Priority: Major > > As discussed in HADOOP-13650, we want to add an S3Guard CLI command which > compares S3 with MetadataStore, and returns a failure status if any > invariants are violated. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16858) S3Guard fsck: Add option to remove orphaned entries
[ https://issues.apache.org/jira/browse/HADOOP-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16858: Summary: S3Guard fsck: Add option to remove orphaned entries (was: S3Guard fsck: Add option to prune orphaned entries) > S3Guard fsck: Add option to remove orphaned entries > --- > > Key: HADOOP-16858 > URL: https://issues.apache.org/jira/browse/HADOOP-16858 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16858) S3Guard fsck: Add option to prune orphaned entries
[ https://issues.apache.org/jira/browse/HADOOP-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-16858: --- Assignee: Gabor Bota > S3Guard fsck: Add option to prune orphaned entries > -- > > Key: HADOOP-16858 > URL: https://issues.apache.org/jira/browse/HADOOP-16858 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-16858) S3Guard fsck: Add option to prune orphaned entries
[ https://issues.apache.org/jira/browse/HADOOP-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16858 started by Gabor Bota. --- > S3Guard fsck: Add option to prune orphaned entries > -- > > Key: HADOOP-16858 > URL: https://issues.apache.org/jira/browse/HADOOP-16858 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16860) Prune -tombstones to remove children entries
[ https://issues.apache.org/jira/browse/HADOOP-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036371#comment-17036371 ] Gabor Bota commented on HADOOP-16860: - We don't remove directories during prune. However, there are orphan entries left somehow in ddb after some operation. This was a misunderstanding and should be solved elsehow. > Prune -tombstones to remove children entries > > > Key: HADOOP-16860 > URL: https://issues.apache.org/jira/browse/HADOOP-16860 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > When you prune a directory in S3Guard the children entries are not pruned > with the tombstoned parent (directory). > I propose the solution to remove all children entries (the whole hierarchy) > to avoid orphaned entries once the parent (directory) tombstone is removed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16860) Prune -tombstones to remove children entries
[
https://issues.apache.org/jira/browse/HADOOP-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16860.
-
Resolution: Invalid
> Prune -tombstones to remove children entries
>
>
> Key: HADOOP-16860
> URL: https://issues.apache.org/jira/browse/HADOOP-16860
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> When you prune a directory in S3Guard the children entries are not pruned
> with the tombstoned parent (directory).
> I propose the solution to remove all children entries (the whole hierarchy)
> to avoid orphaned entries once the parent (directory) tombstone is removed.
> Also the pruning itself in
> {{org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore#prune(org.apache.hadoop.fs.s3a.s3guard.MetadataStore.PruneMode,
> long, java.lang.String)}} does not make sense without removing the children
> entries - once a directory is pruned then all children will became orpans.
> That is particularly dangerous if the same directory is created again in auth
> mode - then we have entries that may not exist in S3 but got picked up from
> the old orphans.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16860) Prune -tombstones to remove children entries
[
https://issues.apache.org/jira/browse/HADOOP-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-16860:
Description:
When you prune a directory in S3Guard the children entries are not pruned with
the tombstoned parent (directory).
I propose the solution to remove all children entries (the whole hierarchy) to
avoid orphaned entries once the parent (directory) tombstone is removed.
was:
When you prune a directory in S3Guard the children entries are not pruned with
the tombstoned parent (directory).
I propose the solution to remove all children entries (the whole hierarchy) to
avoid orphaned entries once the parent (directory) tombstone is removed.
Also the pruning itself in
{{org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore#prune(org.apache.hadoop.fs.s3a.s3guard.MetadataStore.PruneMode,
long, java.lang.String)}} does not make sense without removing the children
entries - once a directory is pruned then all children will became orpans.
That is particularly dangerous if the same directory is created again in auth
mode - then we have entries that may not exist in S3 but got picked up from the
old orphans.
> Prune -tombstones to remove children entries
>
>
> Key: HADOOP-16860
> URL: https://issues.apache.org/jira/browse/HADOOP-16860
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> When you prune a directory in S3Guard the children entries are not pruned
> with the tombstoned parent (directory).
> I propose the solution to remove all children entries (the whole hierarchy)
> to avoid orphaned entries once the parent (directory) tombstone is removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16860) Prune -tombstones to remove children entries
[
https://issues.apache.org/jira/browse/HADOOP-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-16860:
Description:
When you prune a directory in S3Guard the children entries are not pruned with
the tombstoned parent (directory).
I propose the solution to remove all children entries (the whole hierarchy) to
avoid orphaned entries once the parent (directory) tombstone is removed.
Also the pruning itself in
{{org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore#prune(org.apache.hadoop.fs.s3a.s3guard.MetadataStore.PruneMode,
long, java.lang.String)}} does not make sense without removing the children
entries - once a directory is pruned then all children will became orpans.
That is particularly dangerous if the same directory is created again in auth
mode - then we have entries that may not exist in S3 but got picked up from the
old orphans.
was:
When you prune a directory in S3Guard the children entries are not pruned with
the tombstoned parent (directory).
I propose the solution to remove all children entries (the whole hierarchy) to
avoid orphaned entries once the parent (directory) tombstone is removed.
> Prune -tombstones to remove children entries
>
>
> Key: HADOOP-16860
> URL: https://issues.apache.org/jira/browse/HADOOP-16860
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> When you prune a directory in S3Guard the children entries are not pruned
> with the tombstoned parent (directory).
> I propose the solution to remove all children entries (the whole hierarchy)
> to avoid orphaned entries once the parent (directory) tombstone is removed.
> Also the pruning itself in
> {{org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore#prune(org.apache.hadoop.fs.s3a.s3guard.MetadataStore.PruneMode,
> long, java.lang.String)}} does not make sense without removing the children
> entries - once a directory is pruned then all children will became orpans.
> That is particularly dangerous if the same directory is created again in auth
> mode - then we have entries that may not exist in S3 but got picked up from
> the old orphans.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16860) Prune -tombstones to remove children entries
[ https://issues.apache.org/jira/browse/HADOOP-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16860: Parent: HADOOP-15619 Issue Type: Sub-task (was: Bug) > Prune -tombstones to remove children entries > > > Key: HADOOP-16860 > URL: https://issues.apache.org/jira/browse/HADOOP-16860 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > When you prune a directory in S3Guard the children entries are not pruned > with the tombstoned parent (directory). > I propose the solution to remove all children entries (the whole hierarchy) > to avoid orphaned entries once the parent (directory) tombstone is removed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16860) Prune -tombstones to remove children entries
Gabor Bota created HADOOP-16860: --- Summary: Prune -tombstones to remove children entries Key: HADOOP-16860 URL: https://issues.apache.org/jira/browse/HADOOP-16860 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 3.3.0 Reporter: Gabor Bota Assignee: Gabor Bota When you prune a directory in S3Guard the children entries are not pruned with the tombstoned parent (directory). I propose the solution to remove all children entries (the whole hierarchy) to avoid orphaned entries once the parent (directory) tombstone is removed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16858) S3Guard fsck: Add option to prune orphaned entries
Gabor Bota created HADOOP-16858: --- Summary: S3Guard fsck: Add option to prune orphaned entries Key: HADOOP-16858 URL: https://issues.apache.org/jira/browse/HADOOP-16858 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.3.0 Reporter: Gabor Bota -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15964) Add S3A support for Async Scatter/Gather IO
[ https://issues.apache.org/jira/browse/HADOOP-15964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-15964: --- Assignee: Gabor Bota > Add S3A support for Async Scatter/Gather IO > --- > > Key: HADOOP-15964 > URL: https://issues.apache.org/jira/browse/HADOOP-15964 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > > HADOOP-11867 is proposing adding a new scatter/gather IO API. > For an object store to take advantage of it, it should be doing things like > * coalescing reads even with a gap between them > * choosing an optimal ordering of requests > * submitting reads into the executor pool/using any async API provided by the > FS. > * detecting overlapping reads (and then what?) > * switching to HTTP 2 where supported > Do this for S3A -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16848) add initial S3A layering + async init
[ https://issues.apache.org/jira/browse/HADOOP-16848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033292#comment-17033292 ] Gabor Bota commented on HADOOP-16848: - Good idea, happy to do the review and help with the work. > add initial S3A layering + async init > - > > Key: HADOOP-16848 > URL: https://issues.apache.org/jira/browse/HADOOP-16848 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Split the S3A code into layers > * S3AFileSystem > * S3AStore + Impl > * RawS3A + Impl > S3AFS will create the others and start in order: DelegationTokens, RawS3A, > S3AStore, Metastore > this will involve wrapping all access of DTs, s3client, Metastore to block > until that layer is complete, or raise an exception if instantiation of > it/predecessor failed. > New layers will all be subclasses of Service, split into Interface and Impl, > so we can manage the init/start/stop lifecycle with existing code -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16832) S3Guard testing doc: Add required parameters for S3Guard testing in IDE
[ https://issues.apache.org/jira/browse/HADOOP-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031612#comment-17031612 ] Gabor Bota commented on HADOOP-16832: - +1 on PR #1822, committing. > S3Guard testing doc: Add required parameters for S3Guard testing in IDE > --- > > Key: HADOOP-16832 > URL: https://issues.apache.org/jira/browse/HADOOP-16832 > Project: Hadoop Common > Issue Type: Task > Components: fs/s3 >Affects Versions: 3.2.1 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > > When I tried running the S3 guard tests using my IDE, they were getting > skipped saying s3 guard is not enabled. To enable S3 guard test you have to > configure the following property in auth-keys.xml > > fs.s3a.s3guard.test.enabled > true > > Once you have done this tests will still get skipped saying "Test only > applies when DynamoDB is used for S3Guard Store" > To configure dynamo db you have to set this property in auth-keys.xml > > fs.s3a.s3guard.test.implementation > dynamo > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16832) S3Guard testing doc: Add required parameters for S3Guard testing in IDE
[ https://issues.apache.org/jira/browse/HADOOP-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16832. - Resolution: Fixed > S3Guard testing doc: Add required parameters for S3Guard testing in IDE > --- > > Key: HADOOP-16832 > URL: https://issues.apache.org/jira/browse/HADOOP-16832 > Project: Hadoop Common > Issue Type: Task > Components: fs/s3 >Affects Versions: 3.2.1 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > > When I tried running the S3 guard tests using my IDE, they were getting > skipped saying s3 guard is not enabled. To enable S3 guard test you have to > configure the following property in auth-keys.xml > > fs.s3a.s3guard.test.enabled > true > > Once you have done this tests will still get skipped saying "Test only > applies when DynamoDB is used for S3Guard Store" > To configure dynamo db you have to set this property in auth-keys.xml > > fs.s3a.s3guard.test.implementation > dynamo > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16832) S3Guard testing doc: Add required parameters for S3Guard testing in IDE
[ https://issues.apache.org/jira/browse/HADOOP-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16832: Summary: S3Guard testing doc: Add required parameters for S3Guard testing in IDE (was: [S3Guard] Testing doc: Add required parameters for S3Guard testing in IDE) > S3Guard testing doc: Add required parameters for S3Guard testing in IDE > --- > > Key: HADOOP-16832 > URL: https://issues.apache.org/jira/browse/HADOOP-16832 > Project: Hadoop Common > Issue Type: Task > Components: fs/s3 >Affects Versions: 3.2.1 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > > When I tried running the S3 guard tests using my IDE, they were getting > skipped saying s3 guard is not enabled. To enable S3 guard test you have to > configure the following property in auth-keys.xml > > fs.s3a.s3guard.test.enabled > true > > Once you have done this tests will still get skipped saying "Test only > applies when DynamoDB is used for S3Guard Store" > To configure dynamo db you have to set this property in auth-keys.xml > > fs.s3a.s3guard.test.implementation > dynamo > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16832) [S3Guard] Testing doc: Add required parameters for S3Guard testing in IDE
[ https://issues.apache.org/jira/browse/HADOOP-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16832: Summary: [S3Guard] Testing doc: Add required parameters for S3Guard testing in IDE (was: Add required parameters to be set in testing doc for S3Guard testing.) > [S3Guard] Testing doc: Add required parameters for S3Guard testing in IDE > - > > Key: HADOOP-16832 > URL: https://issues.apache.org/jira/browse/HADOOP-16832 > Project: Hadoop Common > Issue Type: Task > Components: fs/s3 >Affects Versions: 3.2.1 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > > When I tried running the S3 guard tests using my IDE, they were getting > skipped saying s3 guard is not enabled. To enable S3 guard test you have to > configure the following property in auth-keys.xml > > fs.s3a.s3guard.test.enabled > true > > Once you have done this tests will still get skipped saying "Test only > applies when DynamoDB is used for S3Guard Store" > To configure dynamo db you have to set this property in auth-keys.xml > > fs.s3a.s3guard.test.implementation > dynamo > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16801) S3Guard listFiles will not query S3 if all listings are authoritative
[ https://issues.apache.org/jira/browse/HADOOP-16801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16801: Summary: S3Guard listFiles will not query S3 if all listings are authoritative (was: S3Guard queries S3 with recursive file listings) > S3Guard listFiles will not query S3 if all listings are authoritative > - > > Key: HADOOP-16801 > URL: https://issues.apache.org/jira/browse/HADOOP-16801 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Minor > Attachments: HADOOP-aws-no-prefetch.prelim.patch > > > S3Guard does not respect authoritative metadatastore when listFiles is used > with recursive=true. It queries S3 even when given directory tree is 1-level > with no nested directories and the parent directory listing is authoritative. > S3Guard should check the listings in given directory tree for > authoritativeness and not query S3 when all listings in the tree are marked > as authoritative in metadata table (given metadatastore is configured to be > authoritative. > Below is the description of how the current code works: > S3AFileSystem#listFiles with recursive option, queries S3 even when directory > listing is authoritative. FileStatusListingIterator is created with given > entries from metadata store > [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java#L126] > . However, FileStatusListingIterator has an ObjectListingIterator that > prefetches from s3 regardless of authoritative listing. We observed this > behavior when using DynamDBMetadataStore. > I suppressed the unnecessary S3 calls by providing a dumb listing iterator to > listFiles call in the provided patch. Obviously this is not a solution. Just > demonstrating the source of the problem. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16801) S3Guard queries S3 with recursive file listings
[ https://issues.apache.org/jira/browse/HADOOP-16801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16801. - Resolution: Fixed > S3Guard queries S3 with recursive file listings > --- > > Key: HADOOP-16801 > URL: https://issues.apache.org/jira/browse/HADOOP-16801 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Minor > Attachments: HADOOP-aws-no-prefetch.prelim.patch > > > S3Guard does not respect authoritative metadatastore when listFiles is used > with recursive=true. It queries S3 even when given directory tree is 1-level > with no nested directories and the parent directory listing is authoritative. > S3Guard should check the listings in given directory tree for > authoritativeness and not query S3 when all listings in the tree are marked > as authoritative in metadata table (given metadatastore is configured to be > authoritative. > Below is the description of how the current code works: > S3AFileSystem#listFiles with recursive option, queries S3 even when directory > listing is authoritative. FileStatusListingIterator is created with given > entries from metadata store > [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java#L126] > . However, FileStatusListingIterator has an ObjectListingIterator that > prefetches from s3 regardless of authoritative listing. We observed this > behavior when using DynamDBMetadataStore. > I suppressed the unnecessary S3 calls by providing a dumb listing iterator to > listFiles call in the provided patch. Obviously this is not a solution. Just > demonstrating the source of the problem. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16801) S3Guard queries S3 with recursive file listings
[ https://issues.apache.org/jira/browse/HADOOP-16801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026559#comment-17026559 ] Gabor Bota commented on HADOOP-16801: - +1 on PR#1815, committing. > S3Guard queries S3 with recursive file listings > --- > > Key: HADOOP-16801 > URL: https://issues.apache.org/jira/browse/HADOOP-16801 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Minor > Attachments: HADOOP-aws-no-prefetch.prelim.patch > > > S3Guard does not respect authoritative metadatastore when listFiles is used > with recursive=true. It queries S3 even when given directory tree is 1-level > with no nested directories and the parent directory listing is authoritative. > S3Guard should check the listings in given directory tree for > authoritativeness and not query S3 when all listings in the tree are marked > as authoritative in metadata table (given metadatastore is configured to be > authoritative. > Below is the description of how the current code works: > S3AFileSystem#listFiles with recursive option, queries S3 even when directory > listing is authoritative. FileStatusListingIterator is created with given > entries from metadata store > [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java#L126] > . However, FileStatusListingIterator has an ObjectListingIterator that > prefetches from s3 regardless of authoritative listing. We observed this > behavior when using DynamDBMetadataStore. > I suppressed the unnecessary S3 calls by providing a dumb listing iterator to > listFiles call in the provided patch. Obviously this is not a solution. Just > demonstrating the source of the problem. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16794) S3 Encryption is always using default region-specific AWS-managed KMS key
[
https://issues.apache.org/jira/browse/HADOOP-16794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota reassigned HADOOP-16794:
---
Assignee: Mukund Thakur
> S3 Encryption is always using default region-specific AWS-managed KMS key
> -
>
> Key: HADOOP-16794
> URL: https://issues.apache.org/jira/browse/HADOOP-16794
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.1
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Major
>
> When using (bucket-level) S3 Default Encryption with SSE-KMS and a CMK, all
> files uploaded via the HDFS {{FileSystem}} {{s3a://}} scheme receive the
> wrong encryption key, always falling back to the region-specific AWS-managed
> KMS key for S3, instead of retaining the custom CMK.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16832) Add required parameters to be set in testing doc for S3Guard testing.
[jira] [Commented] (HADOOP-16832) Add required parameters to be set in testing doc for S3Guard testing.
[ https://issues.apache.org/jira/browse/HADOOP-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025762#comment-17025762 ] Gabor Bota commented on HADOOP-16832: - Please describe what are these parameters required for, and what happens if these are not passed. > Add required parameters to be set in testing doc for S3Guard testing. > - > > Key: HADOOP-16832 > URL: https://issues.apache.org/jira/browse/HADOOP-16832 > Project: Hadoop Common > Issue Type: Task > Components: fs/s3 >Reporter: Mukund Thakur >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16832) Add required parameters to be set in testing doc for S3Guard testing.
[ https://issues.apache.org/jira/browse/HADOOP-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16832: Summary: Add required parameters to be set in testing doc for S3Guard testing. (was: Add required parameters to be set in testing doc for SGuard testing.) > Add required parameters to be set in testing doc for S3Guard testing. > - > > Key: HADOOP-16832 > URL: https://issues.apache.org/jira/browse/HADOOP-16832 > Project: Hadoop Common > Issue Type: Task > Components: fs/s3 >Reporter: Mukund Thakur >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16732) S3Guard to support encrypted DynamoDB table
[ https://issues.apache.org/jira/browse/HADOOP-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022085#comment-17022085 ] Gabor Bota commented on HADOOP-16732: - +1 on PR #1752. Committing. > S3Guard to support encrypted DynamoDB table > --- > > Key: HADOOP-16732 > URL: https://issues.apache.org/jira/browse/HADOOP-16732 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Mingliang Liu >Assignee: Mingliang Liu >Priority: Major > > S3Guard is not yet supporting [encrypted DynamoDB > table|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/encryption.tutorial.html]. > We can provide an option to enable encrypted DynamoDB table so data at rest > could be encrypted. S3Guard data in DynamoDB usually is not sensitive since > it's the S3 namespace mirroring, but some times even this is a concern. By > default it's not enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16792) Let s3 clients configure request timeout
[ https://issues.apache.org/jira/browse/HADOOP-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017943#comment-17017943 ] Gabor Bota edited comment on HADOOP-16792 at 1/17/20 12:42 PM: --- Yes, sure. I added my review on the PR. was (Author: gabor.bota): Yes, sure. > Let s3 clients configure request timeout > > > Key: HADOOP-16792 > URL: https://issues.apache.org/jira/browse/HADOOP-16792 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > > S3 does not guarantee latency. Every once in a while a request may straggle > and drive latency up for the greater procedure. In these cases, simply > timing-out the individual request is beneficial so that the client > application can retry. The retry tends to complete faster than the original > straggling request most of the time. Others experienced this issue too: > [https://arxiv.org/pdf/1911.11727.pdf] . > S3 configuration already provides timeout facility via > `ClientConfiguration#setTimeout`. Exposing this configuration is beneficial > for latency sensitive applications. S3 client configuration is shared with > DynamoDB client which is also affected from unreliable worst case latency. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16792) Let s3 clients configure request timeout
[ https://issues.apache.org/jira/browse/HADOOP-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017943#comment-17017943 ] Gabor Bota commented on HADOOP-16792: - Yes, sure. > Let s3 clients configure request timeout > > > Key: HADOOP-16792 > URL: https://issues.apache.org/jira/browse/HADOOP-16792 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > > S3 does not guarantee latency. Every once in a while a request may straggle > and drive latency up for the greater procedure. In these cases, simply > timing-out the individual request is beneficial so that the client > application can retry. The retry tends to complete faster than the original > straggling request most of the time. Others experienced this issue too: > [https://arxiv.org/pdf/1911.11727.pdf] . > S3 configuration already provides timeout facility via > `ClientConfiguration#setTimeout`. Exposing this configuration is beneficial > for latency sensitive applications. S3 client configuration is shared with > DynamoDB client which is also affected from unreliable worst case latency. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16732) S3Guard to support encrypted DynamoDB table
[ https://issues.apache.org/jira/browse/HADOOP-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015045#comment-17015045 ] Gabor Bota commented on HADOOP-16732: - [~liuml07], are you still working on this? I've done the review on the github PR. If you are not working on this feature anymore I can finalize your PR. > S3Guard to support encrypted DynamoDB table > --- > > Key: HADOOP-16732 > URL: https://issues.apache.org/jira/browse/HADOOP-16732 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Mingliang Liu >Assignee: Mingliang Liu >Priority: Major > > S3Guard is not yet supporting [encrypted DynamoDB > table|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/encryption.tutorial.html]. > We can provide an option to enable encrypted DynamoDB table so data at rest > could be encrypted. S3Guard data in DynamoDB usually is not sensitive since > it's the S3 namespace mirroring, but some times even this is a concern. By > default it's not enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16772) Extract version numbers to head of pom.xml (addendum)
[ https://issues.apache.org/jira/browse/HADOOP-16772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010597#comment-17010597 ] Gabor Bota commented on HADOOP-16772: - I committed 1773 as well. Thanks for your contribution [~tamaas]! > Extract version numbers to head of pom.xml (addendum) > - > > Key: HADOOP-16772 > URL: https://issues.apache.org/jira/browse/HADOOP-16772 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Tamas Penzes >Assignee: Tamas Penzes >Priority: Major > > Frogotten to extract a few version numbers, this is a follow up ticket of > HADOOP-16729. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
