[jira] [Resolved] (HADOOP-16995) ITestS3AConfiguration proxy tests fail when bucket probes == 0
[
https://issues.apache.org/jira/browse/HADOOP-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16995.
-
Resolution: Fixed
got +1 from Steve on the PR, committed to trunk
> ITestS3AConfiguration proxy tests fail when bucket probes == 0
> --
>
> Key: HADOOP-16995
> URL: https://issues.apache.org/jira/browse/HADOOP-16995
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
>
> when bucket probes are disabled, proxy config tests in ITestS3AConfiguration
> fail because the probes aren't being attempted in initialize()
> {code}
>
> fs.s3a.bucket.probe
> 0
>
> {code}
> Cause: HADOOP-16711
> Fix: call unsetBaseAndBucketOverrides for bucket probe in test conf, then set
> the probe value to 2 just to be resilient to future default changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17454) [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0
[ https://issues.apache.org/jira/browse/HADOOP-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-17454. - Resolution: Fixed got +1 from Steve on PR, committed to trunk > [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 > --- > > Key: HADOOP-17454 > URL: https://issues.apache.org/jira/browse/HADOOP-17454 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Set the value of fs.s3a.bucket.probe to 0 by default. > Bucket existence checks are done in the initialization phase of the > S3AFileSystem. It's not required to run this check: the operation itself will > fail if the bucket does not exist instead of the check. > Some points on why do we want to set this to 0: > * When it's set to 0, bucket existence checks won't be done during > initialization thus making it faster. > * Avoid the additional one or two requests on the bucket root, so the user > does not need rights to read or list that folder. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17455) [s3a] Intermittent failure of ITestS3ADeleteCost.testDeleteSingleFileInDir
Gabor Bota created HADOOP-17455:
---
Summary: [s3a] Intermittent failure of
ITestS3ADeleteCost.testDeleteSingleFileInDir
Key: HADOOP-17455
URL: https://issues.apache.org/jira/browse/HADOOP-17455
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 3.3.0
Reporter: Gabor Bota
Assignee: Steve Loughran
Test failed against ireland intermittently with the following config:
{{mvn clean verify -Dparallel-tests -DtestsThreadCount=8}}
xml based config in auth-keys.xml:
{code:xml}
fs.s3a.metadatastore.impl
org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17454) [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0
Gabor Bota created HADOOP-17454: --- Summary: [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 Key: HADOOP-17454 URL: https://issues.apache.org/jira/browse/HADOOP-17454 Project: Hadoop Common Issue Type: Improvement Reporter: Gabor Bota Assignee: Gabor Bota Set the value of fs.s3a.bucket.probe to 0 in the code-default.xml. Bucket existence checks are done in the initialization phase of the S3AFileSystem. It's not required to run this check: the operation itself will fail if the bucket does not exist instead of the check. Some points on why do we want to set this to 0: * When it's set to 0, bucket existence checks won't be done during initialization thus making it faster. * Avoid the additional one or two requests on the bucket root, so the user does not need rights to read or list that folder. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17050) Add support for multiple delegation tokens in S3AFilesystem
Gabor Bota created HADOOP-17050:
---
Summary: Add support for multiple delegation tokens in
S3AFilesystem
Key: HADOOP-17050
URL: https://issues.apache.org/jira/browse/HADOOP-17050
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Gabor Bota
Assignee: Gabor Bota
In {{org.apache.hadoop.fs.s3a.auth.delegation.AbstractDelegationTokenBinding}}
the {{createDelegationToken}} should return a list of tokens.
With this functionality, the {{AbstractDelegationTokenBinding}} can get two
different tokens at the same time.
{{AbstractDelegationTokenBinding.TokenSecretManager}} should be extended to
retrieve secrets and lookup delegation tokens (use the public API for
secretmanager in hadoop)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16979) S3Guard auth mode should be set to false by default in integration tests.
[ https://issues.apache.org/jira/browse/HADOOP-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16979. - Resolution: Fixed > S3Guard auth mode should be set to false by default in integration tests. > - > > Key: HADOOP-16979 > URL: https://issues.apache.org/jira/browse/HADOOP-16979 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, test >Affects Versions: 3.2.0, 3.3.0 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Minor > > As per the doc > [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md#testing-s3a-with-s3guard-enabled] > , it says if s3Guard profile is enabled all tests runs in "non-authoritative" > mode. But as per the code > [https://github.com/apache/hadoop/blob/8d49229c3764a205f21750225005a2c9a8124ab9/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java#L485] > the default value is set to true. So either we fix the doc or code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16961) ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics)
Gabor Bota created HADOOP-16961: --- Summary: ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) Key: HADOOP-16961 URL: https://issues.apache.org/jira/browse/HADOOP-16961 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.0 Reporter: Gabor Bota Assignee: Gabor Bota Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) can improve the testing and diagnostics of the connector. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16910) ABFS Streams to update FileSystem.Statistics counters on IO.
[ https://issues.apache.org/jira/browse/HADOOP-16910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16910. - Resolution: Fixed > ABFS Streams to update FileSystem.Statistics counters on IO. > > > Key: HADOOP-16910 > URL: https://issues.apache.org/jira/browse/HADOOP-16910 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Mehakmeet Singh >Assignee: Mehakmeet Singh >Priority: Major > Fix For: 3.3.1 > > > Abfs FileSystem Counters are not populated and hence not shown on the console > side. > purpose: > * Passing Statistics in AbfsOutputStream and populating FileSystem > Counter(Write_ops) there. > * Populating Read_ops in AbfsInputStream. > * Showing Bytes_written, Bytes_read, Write_ops and Read_ops on the console > while using ABFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16858) S3Guard fsck: Add option to remove orphaned entries
[ https://issues.apache.org/jira/browse/HADOOP-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16858. - Resolution: Fixed > S3Guard fsck: Add option to remove orphaned entries > --- > > Key: HADOOP-16858 > URL: https://issues.apache.org/jira/browse/HADOOP-16858 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Fix For: 3.3.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16904) ABFS test failures when running with a storage account without a hierarchical namespace
Gabor Bota created HADOOP-16904:
---
Summary: ABFS test failures when running with a storage account
without a hierarchical namespace
Key: HADOOP-16904
URL: https://issues.apache.org/jira/browse/HADOOP-16904
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.3.0
Reporter: Gabor Bota
Assignee: Gabor Bota
Some tests are failing when running on a storage account without hierarchical
namespace:
{noformat}
[ERROR] Failures:
[ERROR]
ITestAzureBlobFileSystemAuthorization.testSetOwnerUnauthorized:212->runTest:274
Expected an exception of type class
org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException
[ERROR]
ITestAzureBlobFileSystemAuthorization.testSetPermissionUnauthorized:217->runTest:274
Expected an exception of type class
org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException
[ERROR]
ITestGetNameSpaceEnabled.testXNSAccount:51->Assert.assertTrue:41->Assert.fail:88
Expecting getIsNamespaceEnabled() return true
[ERROR] Errors:
[ERROR]
ITestAzureBlobFileSystemAuthorization.testGetAclStatusAuthorized:247->runTest:278->executeOp:298
? UnsupportedOperation
[ERROR]
ITestAzureBlobFileSystemAuthorization.testGetAclStatusUnauthorized:252->runTest:274->lambda$runTest$4:275->executeOp:298
? UnsupportedOperation
[ERROR]
ITestAzureBlobFileSystemAuthorization.testModifyAclEntriesUnauthorized:222->runTest:274->lambda$runTest$4:275->executeOp:335
? UnsupportedOperation
[ERROR]
ITestAzureBlobFileSystemAuthorization.testRemoveAclEntriesUnauthorized:227->runTest:274->lambda$runTest$4:275->executeOp:331
? UnsupportedOperation
[ERROR]
ITestAzureBlobFileSystemAuthorization.testRemoveAclUnauthorized:237->runTest:274->lambda$runTest$4:275->executeOp:339
? UnsupportedOperation
[ERROR]
ITestAzureBlobFileSystemAuthorization.testRemoveDefaultAclUnauthorized:232->runTest:274->lambda$runTest$4:275->executeOp:342
? UnsupportedOperation
[ERROR]
ITestAzureBlobFileSystemAuthorization.testSetAclUnauthorized:242->runTest:274->lambda$runTest$4:275->executeOp:304
? UnsupportedOperation
[ERROR]
ITestAzureBlobFileSystemCheckAccess.testWhenCheckAccessConfigIsOff:125->setupTestDirectoryAndUserAccess:313->modifyAcl:304
? UnsupportedOperation
{noformat}
This should be fixed, so these tests will be skipped or will fail with a more
meaningful error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14936) S3Guard: remove "experimental" from documentation
[ https://issues.apache.org/jira/browse/HADOOP-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-14936. - Resolution: Fixed > S3Guard: remove "experimental" from documentation > - > > Key: HADOOP-14936 > URL: https://issues.apache.org/jira/browse/HADOOP-14936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Gabor Bota >Priority: Major > > I think it is time to remove the "experimental feature" designation in the > site docs for S3Guard. Discuss. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16878) Copy command in FileUtil to LOG at warn level if the source and destination is the same
Gabor Bota created HADOOP-16878:
---
Summary: Copy command in FileUtil to LOG at warn level if the
source and destination is the same
Key: HADOOP-16878
URL: https://issues.apache.org/jira/browse/HADOOP-16878
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 3.3.0
Reporter: Gabor Bota
Assignee: Gabor Bota
We encountered an error during a test in our QE when the file destination and
source path were the same. This happened during an ADLS test, and there were no
meaningful error messages, so it was hard to find the root cause of the failure.
The error we saw was that file size has changed during the copy operation - the
new file creation in the destination, which is the same as the source, creates
a new file and sets the file length to zero. After this, getting the source
file will fail because the sile size changed during the operation.
I propose a solution to at least log in error level in the {{FileUtil}} if the
source and destination of the copy operation is the same, so debugging issues
like this will be easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16860) Prune -tombstones to remove children entries
[
https://issues.apache.org/jira/browse/HADOOP-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16860.
-
Resolution: Invalid
> Prune -tombstones to remove children entries
>
>
> Key: HADOOP-16860
> URL: https://issues.apache.org/jira/browse/HADOOP-16860
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> When you prune a directory in S3Guard the children entries are not pruned
> with the tombstoned parent (directory).
> I propose the solution to remove all children entries (the whole hierarchy)
> to avoid orphaned entries once the parent (directory) tombstone is removed.
> Also the pruning itself in
> {{org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore#prune(org.apache.hadoop.fs.s3a.s3guard.MetadataStore.PruneMode,
> long, java.lang.String)}} does not make sense without removing the children
> entries - once a directory is pruned then all children will became orpans.
> That is particularly dangerous if the same directory is created again in auth
> mode - then we have entries that may not exist in S3 but got picked up from
> the old orphans.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16860) Prune -tombstones to remove children entries
Gabor Bota created HADOOP-16860: --- Summary: Prune -tombstones to remove children entries Key: HADOOP-16860 URL: https://issues.apache.org/jira/browse/HADOOP-16860 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 3.3.0 Reporter: Gabor Bota Assignee: Gabor Bota When you prune a directory in S3Guard the children entries are not pruned with the tombstoned parent (directory). I propose the solution to remove all children entries (the whole hierarchy) to avoid orphaned entries once the parent (directory) tombstone is removed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16858) S3Guard fsck: Add option to prune orphaned entries
Gabor Bota created HADOOP-16858: --- Summary: S3Guard fsck: Add option to prune orphaned entries Key: HADOOP-16858 URL: https://issues.apache.org/jira/browse/HADOOP-16858 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.3.0 Reporter: Gabor Bota -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16832) S3Guard testing doc: Add required parameters for S3Guard testing in IDE
[ https://issues.apache.org/jira/browse/HADOOP-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16832. - Resolution: Fixed > S3Guard testing doc: Add required parameters for S3Guard testing in IDE > --- > > Key: HADOOP-16832 > URL: https://issues.apache.org/jira/browse/HADOOP-16832 > Project: Hadoop Common > Issue Type: Task > Components: fs/s3 >Affects Versions: 3.2.1 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > > When I tried running the S3 guard tests using my IDE, they were getting > skipped saying s3 guard is not enabled. To enable S3 guard test you have to > configure the following property in auth-keys.xml > > fs.s3a.s3guard.test.enabled > true > > Once you have done this tests will still get skipped saying "Test only > applies when DynamoDB is used for S3Guard Store" > To configure dynamo db you have to set this property in auth-keys.xml > > fs.s3a.s3guard.test.implementation > dynamo > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16801) S3Guard queries S3 with recursive file listings
[ https://issues.apache.org/jira/browse/HADOOP-16801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16801. - Resolution: Fixed > S3Guard queries S3 with recursive file listings > --- > > Key: HADOOP-16801 > URL: https://issues.apache.org/jira/browse/HADOOP-16801 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Minor > Attachments: HADOOP-aws-no-prefetch.prelim.patch > > > S3Guard does not respect authoritative metadatastore when listFiles is used > with recursive=true. It queries S3 even when given directory tree is 1-level > with no nested directories and the parent directory listing is authoritative. > S3Guard should check the listings in given directory tree for > authoritativeness and not query S3 when all listings in the tree are marked > as authoritative in metadata table (given metadatastore is configured to be > authoritative. > Below is the description of how the current code works: > S3AFileSystem#listFiles with recursive option, queries S3 even when directory > listing is authoritative. FileStatusListingIterator is created with given > entries from metadata store > [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java#L126] > . However, FileStatusListingIterator has an ObjectListingIterator that > prefetches from s3 regardless of authoritative listing. We observed this > behavior when using DynamDBMetadataStore. > I suppressed the unnecessary S3 calls by providing a dumb listing iterator to > listFiles call in the provided patch. Obviously this is not a solution. Just > demonstrating the source of the problem. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16772) Extract version numbers to head of pom.xml (addendum)
[ https://issues.apache.org/jira/browse/HADOOP-16772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16772. - Resolution: Fixed > Extract version numbers to head of pom.xml (addendum) > - > > Key: HADOOP-16772 > URL: https://issues.apache.org/jira/browse/HADOOP-16772 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Tamas Penzes >Assignee: Tamas Penzes >Priority: Major > > Frogotten to extract a few version numbers, this is a follow up ticket of > HADOOP-16729. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16450) ITestS3ACommitterFactory failing, S3 client is not inconsistent
[
https://issues.apache.org/jira/browse/HADOOP-16450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16450.
-
Resolution: Fixed
> ITestS3ACommitterFactory failing, S3 client is not inconsistent
> ---
>
> Key: HADOOP-16450
> URL: https://issues.apache.org/jira/browse/HADOOP-16450
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Trivial
>
> Transient failure of {{ITestS3ACommitterFactory}} during a sequential run;
> the FS created wasn't inconsistent
> That test suite doesn't override the superclass AbstractCommitITest's
> {{useInconsistentClient}} method, so declares that it expects one. If we have
> it return false, it won't care any more what kind of FS client it gets
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16424) S3Guard fsck: Check internal consistency of the MetadataStore
[ https://issues.apache.org/jira/browse/HADOOP-16424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16424. - Resolution: Fixed Got +1 on #1691 from [~ste...@apache.org]. Committing. > S3Guard fsck: Check internal consistency of the MetadataStore > - > > Key: HADOOP-16424 > URL: https://issues.apache.org/jira/browse/HADOOP-16424 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > The internal consistency should be checked e.g for orphaned entries which can > cause trouble in runtime and testing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16745) S3Guard fsck: Use enum instead of int for severity in violations
Gabor Bota created HADOOP-16745: --- Summary: S3Guard fsck: Use enum instead of int for severity in violations Key: HADOOP-16745 URL: https://issues.apache.org/jira/browse/HADOOP-16745 Project: Hadoop Common Issue Type: Sub-task Reporter: Gabor Bota Assignee: Gabor Bota Looking in the code, I can see that it's a severity level "2", so not that serious. # We should go away from and int value here to something comprehensible, i.e. an enum. # Or at least some contents used in the Violation enum setup. # Either way, how about letting us choose which level to fail on. # And maybe severity 2 should be logged at WARN, not ERROR, as I was confused "why an error yet none reported in status" -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16729) Extract version numbers to head of pom.xml
[ https://issues.apache.org/jira/browse/HADOOP-16729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16729. - Resolution: Fixed > Extract version numbers to head of pom.xml > -- > > Key: HADOOP-16729 > URL: https://issues.apache.org/jira/browse/HADOOP-16729 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Tamas Penzes >Assignee: Tamas Penzes >Priority: Minor > > To be able to easily replace third-party dependency version numbers it would > be useful to collect their version numbers at the top of the pom.xml. > For many third-parties (e.g. slf4j, jetty, etc.) this is already done, but I > would need the same for others. The change doesn't have any effect on the > code or the build, no version numbers would be changed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16709) S3Guard: Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths
[ https://issues.apache.org/jira/browse/HADOOP-16709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16709. - Resolution: Fixed > S3Guard: Make authoritative mode exclusive for metadata - don't check for > expiry for authoritative paths > > > Key: HADOOP-16709 > URL: https://issues.apache.org/jira/browse/HADOOP-16709 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > > Authoritative mode has TTL which is set to 15 minutes by default. However, > there are cases when we know for sure that the data wouldn't be > changed/updated. > In certain cases, AppMaster ends up spending good amount of time in getSplits > due to TTL expiry. It would be great to have an option to disable TTL (or > specify as -1 when TTL shouldn't be checked). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16484) S3A to warn or fail if S3Guard is disabled
[ https://issues.apache.org/jira/browse/HADOOP-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16484. - Fix Version/s: 3.3.0 Target Version/s: 3.3.0 (was: 3.2.2) Resolution: Fixed > S3A to warn or fail if S3Guard is disabled > -- > > Key: HADOOP-16484 > URL: https://issues.apache.org/jira/browse/HADOOP-16484 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Minor > Fix For: 3.3.0 > > > A seemingly recurrent problem with s3guard is "people who think S3Guard is > turned on but really it isn't" > It's not immediately obvious this is the case, and the fact S3Guard is off > tends to surface after some intermittent failure has actually been detected. > Propose: add a configuration parameter which chooses what to do when an S3A > FS is instantiated without S3Guard > * silent : today; do nothing. > * status: give s3guard on/off status > * inform: log FS is instantiated without s3guard > * warn: Warn that data may be at risk in workflows > * fail > deployments could then choose which level of reaction they want. I'd make the > default "inform" for now; any on-prem object store deployment should switch > to silent, and if you really want strictness, fail is the ultimate option -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16653) S3Guard DDB overreacts to no tag access
[ https://issues.apache.org/jira/browse/HADOOP-16653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16653. - Resolution: Fixed > S3Guard DDB overreacts to no tag access > --- > > Key: HADOOP-16653 > URL: https://issues.apache.org/jira/browse/HADOOP-16653 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Minor > > if you don't have permissions to read or write DDB tags it logs a lot every > time you bring up a guarded FS > # we shouldn't worry so much about no tag access if version is there > # if you can't read the tag, no point trying to write -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16520) Race condition in DDB table init and waiting threads
[ https://issues.apache.org/jira/browse/HADOOP-16520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16520. - Resolution: Fixed > Race condition in DDB table init and waiting threads > > > Key: HADOOP-16520 > URL: https://issues.apache.org/jira/browse/HADOOP-16520 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > > s3guard threads waiting for table creation completion can be scheduled before > the creating thread, look for the version marker and then fail. > window will be sleep times in AWS SDK Table.waitForActive(); -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16349) DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry
[
https://issues.apache.org/jira/browse/HADOOP-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16349.
-
Resolution: Fixed
> DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry
> -
>
> Key: HADOOP-16349
> URL: https://issues.apache.org/jira/browse/HADOOP-16349
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
>
> If you delete the version marker from a S3Guard table, it appears to hang for
> 5 minutes.
> Only if you restart and turn logging to debug do you see that
> {{DynamoDBMetadataStore.getVersionMarkerItem()}} is sleeping and retrying.
> # log at warn
> # add entry to troubleshooting doc on the topic
> The cause of the failure can be any of
> * table being inited elsewhere: expectation, fast recovery
> * it's not a S3Guard table: it won't recover
> * it's a S3Guard table without a version marker: it won't recover.
> + consider having a shorter retry lifespan, though if it adds a new config
> point I'm a bit reluctant. For s3guard bucket-info it would make sense to
> change the policy to be aggressively short lived
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16529) Allow AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION to be set from abfs.xml property
[
https://issues.apache.org/jira/browse/HADOOP-16529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16529.
-
Resolution: Workaround
> Allow AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION to be set from
> abfs.xml property
> ---
>
> Key: HADOOP-16529
> URL: https://issues.apache.org/jira/browse/HADOOP-16529
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> In
> org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest#AbstractAbfsIntegrationTest
> we do a
> {code:java}
>
> abfsConfig.setBoolean(AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION,
> true);
> {code}
> which is not good for some testcases (eg. HADOOP-16138) where we want to test
> against a container that is not exist.
> A property should be added to be able to override this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE
[
https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16138.
-
Target Version/s: 3.3.0
Resolution: Fixed
> hadoop fs mkdir / of nonexistent abfs container raises NPE
> --
>
> Key: HADOOP-16138
> URL: https://issues.apache.org/jira/browse/HADOOP-16138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
>
> If you try to do a mkdir on the root of a nonexistent container, you get an
> NPE
> {code}
> hadoop fs -mkdir abfs://contain...@abfswales1.dfs.core.windows.net/
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16547) s3guard prune command doesn't get AWS auth chain from FS
[ https://issues.apache.org/jira/browse/HADOOP-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16547. - Resolution: Fixed > s3guard prune command doesn't get AWS auth chain from FS > > > Key: HADOOP-16547 > URL: https://issues.apache.org/jira/browse/HADOOP-16547 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > s3guard prune command doesn't get AWS auth chain from any FS, so it just > drives the DDB store from the conf settings. If S3A is set up to use > Delegation tokens then the DTs/custom AWS auth sequence is not picked up, so > you get an auth failure. > Fix: > # instantiate the FS before calling initMetadataStore > # review other commands to make sure problem isn't replicated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-16565) Fix "com.amazonaws.SdkClientException: Unable to find a region via the region provider chain."
[
https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota reopened HADOOP-16565:
-
Reopened to add a message.
> Fix "com.amazonaws.SdkClientException: Unable to find a region via the region
> provider chain."
> --
>
> Key: HADOOP-16565
> URL: https://issues.apache.org/jira/browse/HADOOP-16565
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> The error found during testing in the following tests:
> {noformat}
> [ERROR] ITestS3ATemporaryCredentials.testInvalidSTSBinding:257 ? SdkClient
> Unable to f...
> [ERROR] ITestS3ATemporaryCredentials.testSTS:130 ? SdkClient Unable to find
> a region v...
> [ERROR]
> ITestS3ATemporaryCredentials.testSessionRequestExceptionTranslation:441->lambda$testSessionRequestExceptionTranslation$5:442
> ? SdkClient
> [ERROR] ITestS3ATemporaryCredentials.testSessionTokenExpiry:222 ? SdkClient
> Unable to ...
> [ERROR] ITestS3ATemporaryCredentials.testSessionTokenPropagation:193 ?
> SdkClient Unabl...
> [ERROR] ITestDelegatedMRJob.testJobSubmissionCollectsTokens:286 ? SdkClient
> Unable to ...
> [ERROR] ITestSessionDelegationInFileystem.testAddTokensFromFileSystem:235 ?
> SdkClient ...
> [ERROR]
> ITestSessionDelegationInFileystem.testCanRetrieveTokenFromCurrentUserCreds:260->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDTCredentialProviderFromCurrentUserCreds:278->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDelegatedFileSystem:308->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDelegationBindingMismatch1:432->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testFileSystemBoundToCreator:681
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testGetDTfromFileSystem:212 ?
> SdkClient Unab...
> [ERROR]
> ITestSessionDelegationInFileystem.testHDFSFetchDTCommand:606->lambda$testHDFSFetchDTCommand$3:607
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testYarnCredentialPickup:576 ?
> SdkClient Una...
> [ERROR] ITestSessionDelegationTokens.testCreateAndUseDT:176 ? SdkClient
> Unable to find...
> [ERROR] ITestSessionDelegationTokens.testSaveLoadTokens:121 ? SdkClient
> Unable to find...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16565) Fix "com.amazonaws.SdkClientException: Unable to find a region via the region provider chain."
[
https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16565.
-
Resolution: Workaround
> Fix "com.amazonaws.SdkClientException: Unable to find a region via the region
> provider chain."
> --
>
> Key: HADOOP-16565
> URL: https://issues.apache.org/jira/browse/HADOOP-16565
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> The error found during testing in the following tests:
> {noformat}
> [ERROR] ITestS3ATemporaryCredentials.testInvalidSTSBinding:257 ? SdkClient
> Unable to f...
> [ERROR] ITestS3ATemporaryCredentials.testSTS:130 ? SdkClient Unable to find
> a region v...
> [ERROR]
> ITestS3ATemporaryCredentials.testSessionRequestExceptionTranslation:441->lambda$testSessionRequestExceptionTranslation$5:442
> ? SdkClient
> [ERROR] ITestS3ATemporaryCredentials.testSessionTokenExpiry:222 ? SdkClient
> Unable to ...
> [ERROR] ITestS3ATemporaryCredentials.testSessionTokenPropagation:193 ?
> SdkClient Unabl...
> [ERROR] ITestDelegatedMRJob.testJobSubmissionCollectsTokens:286 ? SdkClient
> Unable to ...
> [ERROR] ITestSessionDelegationInFileystem.testAddTokensFromFileSystem:235 ?
> SdkClient ...
> [ERROR]
> ITestSessionDelegationInFileystem.testCanRetrieveTokenFromCurrentUserCreds:260->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDTCredentialProviderFromCurrentUserCreds:278->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDelegatedFileSystem:308->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR]
> ITestSessionDelegationInFileystem.testDelegationBindingMismatch1:432->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testFileSystemBoundToCreator:681
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testGetDTfromFileSystem:212 ?
> SdkClient Unab...
> [ERROR]
> ITestSessionDelegationInFileystem.testHDFSFetchDTCommand:606->lambda$testHDFSFetchDTCommand$3:607
> ? SdkClient
> [ERROR] ITestSessionDelegationInFileystem.testYarnCredentialPickup:576 ?
> SdkClient Una...
> [ERROR] ITestSessionDelegationTokens.testCreateAndUseDT:176 ? SdkClient
> Unable to find...
> [ERROR] ITestSessionDelegationTokens.testSaveLoadTokens:121 ? SdkClient
> Unable to find...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16566) S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of com.google.common.base.Stopwatch
[
https://issues.apache.org/jira/browse/HADOOP-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16566.
-
Resolution: Fixed
> S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of
> com.google.common.base.Stopwatch
> --
>
> Key: HADOOP-16566
> URL: https://issues.apache.org/jira/browse/HADOOP-16566
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> Some distributions won't have the updated guava, and
> {{org.apache.hadoop.util.StopWatch}} is only available in the newer ones.
> Fix this issue by using the hadoop util's instead.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16423) S3Guarld fsck: Check metadata consistency from S3 to metadatastore (log)
[ https://issues.apache.org/jira/browse/HADOOP-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16423. - Resolution: Fixed > S3Guarld fsck: Check metadata consistency from S3 to metadatastore (log) > > > Key: HADOOP-16423 > URL: https://issues.apache.org/jira/browse/HADOOP-16423 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > This part is only for logging the inconsistencies. > This issue only covers the part when the walk is being done in the S3 and > compares all metadata to the MS. > There will be no part where the walk is being done in the MS and compare it > to the S3. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16566) S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of com.google.common.base.Stopwatch
Gabor Bota created HADOOP-16566:
---
Summary: S3Guard fsck: Use org.apache.hadoop.util.StopWatch
instead of com.google.common.base.Stopwatch
Key: HADOOP-16566
URL: https://issues.apache.org/jira/browse/HADOOP-16566
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Gabor Bota
Assignee: Gabor Bota
Some distributions won't have the updated guava, and
{{org.apache.hadoop.util.StopWatch}} is only available in the newer ones.
Fix this issue by using the hadoop util's instead.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16565) Fix "com.amazonaws.SdkClientException: Unable to find a region via the region provider chain."
Gabor Bota created HADOOP-16565:
---
Summary: Fix "com.amazonaws.SdkClientException: Unable to find a
region via the region provider chain."
Key: HADOOP-16565
URL: https://issues.apache.org/jira/browse/HADOOP-16565
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Gabor Bota
Assignee: Gabor Bota
The error found during testing in the following tests:
{noformat}
[ERROR] ITestS3ATemporaryCredentials.testInvalidSTSBinding:257 ? SdkClient
Unable to f...
[ERROR] ITestS3ATemporaryCredentials.testSTS:130 ? SdkClient Unable to find a
region v...
[ERROR]
ITestS3ATemporaryCredentials.testSessionRequestExceptionTranslation:441->lambda$testSessionRequestExceptionTranslation$5:442
? SdkClient
[ERROR] ITestS3ATemporaryCredentials.testSessionTokenExpiry:222 ? SdkClient
Unable to ...
[ERROR] ITestS3ATemporaryCredentials.testSessionTokenPropagation:193 ?
SdkClient Unabl...
[ERROR] ITestDelegatedMRJob.testJobSubmissionCollectsTokens:286 ? SdkClient
Unable to ...
[ERROR] ITestSessionDelegationInFileystem.testAddTokensFromFileSystem:235 ?
SdkClient ...
[ERROR]
ITestSessionDelegationInFileystem.testCanRetrieveTokenFromCurrentUserCreds:260->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
? SdkClient
[ERROR]
ITestSessionDelegationInFileystem.testDTCredentialProviderFromCurrentUserCreds:278->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
? SdkClient
[ERROR]
ITestSessionDelegationInFileystem.testDelegatedFileSystem:308->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
? SdkClient
[ERROR]
ITestSessionDelegationInFileystem.testDelegationBindingMismatch1:432->createDelegationTokens:292->AbstractDelegationIT.mkTokens:88
? SdkClient
[ERROR] ITestSessionDelegationInFileystem.testFileSystemBoundToCreator:681 ?
SdkClient
[ERROR] ITestSessionDelegationInFileystem.testGetDTfromFileSystem:212 ?
SdkClient Unab...
[ERROR]
ITestSessionDelegationInFileystem.testHDFSFetchDTCommand:606->lambda$testHDFSFetchDTCommand$3:607
? SdkClient
[ERROR] ITestSessionDelegationInFileystem.testYarnCredentialPickup:576 ?
SdkClient Una...
[ERROR] ITestSessionDelegationTokens.testCreateAndUseDT:176 ? SdkClient
Unable to find...
[ERROR] ITestSessionDelegationTokens.testSaveLoadTokens:121 ? SdkClient
Unable to find...
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16564) S3Guarld fsck: Add docs to the first iteration (S3->ddbMS, -verify)
Gabor Bota created HADOOP-16564: --- Summary: S3Guarld fsck: Add docs to the first iteration (S3->ddbMS, -verify) Key: HADOOP-16564 URL: https://issues.apache.org/jira/browse/HADOOP-16564 Project: Hadoop Common Issue Type: Sub-task Reporter: Gabor Bota Followup for HADOOP-16423. Add md documentation and how to extend wtih new violations. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16563) S3Guard fsck: Detect if a directory if authoritative and highlight errors if detected in it
Gabor Bota created HADOOP-16563: --- Summary: S3Guard fsck: Detect if a directory if authoritative and highlight errors if detected in it Key: HADOOP-16563 URL: https://issues.apache.org/jira/browse/HADOOP-16563 Project: Hadoop Common Issue Type: Sub-task Reporter: Gabor Bota Followup from HADOOP-16423. One of the changes with the HADOOP-16430 PR is that we now have an S3A FS method boolean allowAuthoritative(final Path path) that takes a path and returns true iff its authoritative either by the MS being auth or the given path being marked as one of the authoritative dirs. I think the validation when an authoritative directory is consistent between the metastore and S3 should be using this when it wants to highlight an authoritative path is inconsistent. This can be a follow-on patch, because as usual it will need more tests, in the code, and someone to try out the command line. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16550) Spark config name error on the Launching Applications Using Docker Containers page
[
https://issues.apache.org/jira/browse/HADOOP-16550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16550.
-
Resolution: Fixed
> Spark config name error on the Launching Applications Using Docker Containers
> page
> --
>
> Key: HADOOP-16550
> URL: https://issues.apache.org/jira/browse/HADOOP-16550
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.9.0, 2.8.2, 2.8.3, 3.0.0, 3.1.0, 2.9.1, 3.0.1, 2.8.4,
> 3.0.2, 3.1.1, 2.9.2, 3.0.3, 2.8.5, 3.1.2
>Reporter: Attila Zsolt Piros
>Priority: Major
>
> On the "Launching Applications Using Docker Containers" page at the "Example:
> Spark" section the Spark config for configuring the environment variables for
> the application master the config prefix are wrong:
> -
> spark.yarn.{color:#DE350B}*A*{color}ppMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE
> - spark.yarn.{color:#DE350B}*A*{color}ppMasterEnv.YARN_CONTAINER_RUNTIME_TYPE
>
> The correct ones:
> - spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE
> - spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE
> See https://spark.apache.org/docs/2.4.0/running-on-yarn.html:
> {quote}
> spark.yarn.appMasterEnv.[EnvironmentVariableName]
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16529) Allow AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION to be set from abfs.xml property
Gabor Bota created HADOOP-16529:
---
Summary: Allow
AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION to be set from abfs.xml
property
Key: HADOOP-16529
URL: https://issues.apache.org/jira/browse/HADOOP-16529
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/azure
Affects Versions: 3.3.0
Reporter: Gabor Bota
Assignee: Gabor Bota
In
org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest#AbstractAbfsIntegrationTest
we do a
{code:java}
abfsConfig.setBoolean(AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION,
true);
{code}
which is not good for some testcases (eg. HADOOP-16138) where we want to test
against a container that is not exist.
A property should be added to be able to override this.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16500) S3ADelegationTokens to only log at debug on startup
[ https://issues.apache.org/jira/browse/HADOOP-16500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16500. - Resolution: Fixed Fix Version/s: 3.3.0 Committed to trunk. > S3ADelegationTokens to only log at debug on startup > --- > > Key: HADOOP-16500 > URL: https://issues.apache.org/jira/browse/HADOOP-16500 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.0 > > > downgrade the log at info to log at debug when S3A comes up with DT support. > Otherwise it's too noisy. > Things still get printed when tokens are created. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16507) S3Guard fsck: Add option to configure severity (level) for the scan
Gabor Bota created HADOOP-16507:
---
Summary: S3Guard fsck: Add option to configure severity (level)
for the scan
Key: HADOOP-16507
URL: https://issues.apache.org/jira/browse/HADOOP-16507
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.3.0
Reporter: Gabor Bota
There's the severity of Violation (inconsistency) defined in
{{org.apache.hadoop.fs.s3a.s3guard.S3GuardFsck.Violation}}.
This flag is only for defining the severity of the Violation, but not used to
filter the scan for issue severity.
The task to do: Use the severity level to define which issue should be logged
and/or fixed during the scan.
Note: the best way to avoid possible code duplication would be to not even add
the consistency violation pair to the list of violations during the scan.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16481) ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region
[
https://issues.apache.org/jira/browse/HADOOP-16481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16481.
-
Resolution: Fixed
Fix Version/s: 3.3.0
Target Version/s: 3.3.0
> ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region
> -
>
> Key: HADOOP-16481
> URL: https://issues.apache.org/jira/browse/HADOOP-16481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.3.0
>
>
> The new test {{ITestS3GuardDDBRootOperations.test_300_MetastorePrune}} fails
> if you don't explicitly set the region
> {code}
> [ERROR]
> test_300_MetastorePrune(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardDDBRootOperations)
> Time elapsed: 0.845 s <<< ERROR!
> org.apache.hadoop.util.ExitUtil$ExitException: No region found from -region
> flag, config, or S3 bucket
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardDDBRootOperations.test_300_MetastorePrune(ITestS3GuardDDBRootOperations.java:186)
> {code}
> it should be picked up from the test filesystem.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16499) S3A retry policy to be exponential
[ https://issues.apache.org/jira/browse/HADOOP-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16499. - Resolution: Fixed Fix Version/s: 3.3.0 Target Version/s: 3.3.0 > S3A retry policy to be exponential > -- > > Key: HADOOP-16499 > URL: https://issues.apache.org/jira/browse/HADOOP-16499 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > Fix For: 3.3.0 > > > the fixed s3a retry policy doesnt leave big enough gaps for cached 404s to > expire; we cant recover from this > HADOOP-16490 is a full fix for this, but one we can backport is moving from > fixed to exponential retries -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16502) Add fsck to S3A tests where additional diagnosis is needed
Gabor Bota created HADOOP-16502:
---
Summary: Add fsck to S3A tests where additional diagnosis is needed
Key: HADOOP-16502
URL: https://issues.apache.org/jira/browse/HADOOP-16502
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Gabor Bota
Extend
{{org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore#testPruneTombstoneUnderTombstone}}
{code:java}
// the child2 entry is still there, though it's now orphan (the store isn't
// meeting the rule "all entries must have a parent which exists"
getFile(child2);
+ // todo create a raw fs
+ S3GuardFsck fsck = new S3GuardFsck(rawFs, ms);
// a full prune will still find and delete it, as this
// doesn't walk the tree
getDynamoMetadataStore().prune(PruneMode.ALL_BY_MODTIME,
now + MINUTE);
{code}
Extend
{{org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore#testPutFileDeepUnderTombstone}}:
{code:java}
// now put the tombstone
putTombstone(base, now, null);
assertIsTombstone(base);
+ // todo create a raw fs for checking
+ S3GuardFsck fsck = new S3GuardFsck(rawFs, ms);
/*- */
/* Begin S3FileSystem.finishedWrite() sequence. */
/* -*/
AncestorState ancestorState = getDynamoMetadataStore()
.initiateBulkWrite(BulkOperationState.OperationType.Put,
childPath);
{code}
Add new test:
{{org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardDDBRootOperations#test_070_run_fsck_on_store}}
{code:java}
@Test
public void test_070_run_fsck_on_store() throws Throwable {
// todo create a raw fs
S3AFileSystem rawFs = ;
S3GuardFsck s3GuardFsck = new S3GuardFsck(rawFs, metastore);
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16483) S3Guarld fsck: Check metadata consistency from metadatastore to S3 (log)
Gabor Bota created HADOOP-16483: --- Summary: S3Guarld fsck: Check metadata consistency from metadatastore to S3 (log) Key: HADOOP-16483 URL: https://issues.apache.org/jira/browse/HADOOP-16483 Project: Hadoop Common Issue Type: Sub-task Reporter: Gabor Bota This issue covers the walk on the MS structure and to compare it with S3. Things to be checked: * Record is in the MS, but not in S3 * ... Only makes sense if * during testing, after a test failure it can be wired *before* the cleanup to debug the inconsistent state -> as a diagnostic tool. * the MS is running in auth mode, and there is no short term metadata expiry -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16476) Intermittent failure of ITestS3GuardConcurrentOps#testConcurrentTableCreations
Gabor Bota created HADOOP-16476:
---
Summary: Intermittent failure of
ITestS3GuardConcurrentOps#testConcurrentTableCreations
Key: HADOOP-16476
URL: https://issues.apache.org/jira/browse/HADOOP-16476
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Gabor Bota
Test is failing intermittently. One possible solution would be to wait (retry)
more because the table will be deleted eventually - it's not there after the
whole test run.
{noformat}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 142.471
s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps
[ERROR]
testConcurrentTableCreations(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps)
Time elapsed: 142.286 s <<< ERROR!
java.lang.IllegalArgumentException: Table
s3guard.test.testConcurrentTableCreations-1265635747 is not deleted.
at
com.amazonaws.services.dynamodbv2.document.Table.waitForDelete(Table.java:505)
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps.deleteTable(ITestS3GuardConcurrentOps.java:87)
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps.testConcurrentTableCreations(ITestS3GuardConcurrentOps.java:178)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.waiters.WaiterTimedOutException: Reached maximum
attempts without transitioning to the desired state
at
com.amazonaws.waiters.WaiterExecution.pollResource(WaiterExecution.java:86)
at com.amazonaws.waiters.WaiterImpl.run(WaiterImpl.java:88)
at
com.amazonaws.services.dynamodbv2.document.Table.waitForDelete(Table.java:502)
... 16 more
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16436) Fix ITestS3GuardToolLocal#testInitNegativeRead test failure
Gabor Bota created HADOOP-16436:
---
Summary: Fix ITestS3GuardToolLocal#testInitNegativeRead test
failure
Key: HADOOP-16436
URL: https://issues.apache.org/jira/browse/HADOOP-16436
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.3.0
Reporter: Gabor Bota
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal#testInitNegativeRead is
failing:
{noformat}
[ERROR] Tests run: 30, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 98.684
s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
[ERROR]
testInitNegativeRead(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal)
Time elapsed: 1.679 s <<< ERROR!
java.lang.IllegalArgumentException: bucket
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
at
org.apache.hadoop.fs.s3a.S3AUtils.propagateBucketOptions(S3AUtils.java:1134)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$Init.run(S3GuardTool.java:487)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:401)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1672)
at
org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.run(AbstractS3GuardToolTestBase.java:137)
at
org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase$1.call(AbstractS3GuardToolTestBase.java:154)
at
org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase$1.call(AbstractS3GuardToolTestBase.java:151)
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:498)
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
at
org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.runToFailure(AbstractS3GuardToolTestBase.java:150)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16383) Pass ITtlTimeProvider instance in initialize method in MetadataStore interface
[
https://issues.apache.org/jira/browse/HADOOP-16383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16383.
-
Resolution: Fixed
Fix Version/s: 3.3.0
> Pass ITtlTimeProvider instance in initialize method in MetadataStore interface
> --
>
> Key: HADOOP-16383
> URL: https://issues.apache.org/jira/browse/HADOOP-16383
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.3.0
>
>
> In HADOOP-16279 I added ITtlTimeProvider {{delete}}, {{deleteSubtree}},
> {{move}}, methods as a parameter instead of adding it to the MetadataStore
> initialization. We have to change this and pass the provider during the init
> phase.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16433) Filter expired entries and tombstones when listing with MetadataStore#listChildren
Gabor Bota created HADOOP-16433:
---
Summary: Filter expired entries and tombstones when listing with
MetadataStore#listChildren
Key: HADOOP-16433
URL: https://issues.apache.org/jira/browse/HADOOP-16433
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.3.0
Reporter: Gabor Bota
Assignee: Gabor Bota
Currently, we don't filter out entries in {{listChildren}} implementations.
This can cause bugs and inconsistencies, so this should be fixed.
It can lead to a status where we can't recover from the following:
{{guarded and raw (OOB op) clients are doing ops to S3}}
Guarded: touch /
Guarded: touch /
Guarded: rm / {{-> tombstone in MS}}
RAW: touch //file.ext {{-> file is hidden with a tombstone}}
Guarded: ls / {{-> the directory is empty}}
After we change the following code
{code:java}
final List metas = new ArrayList<>();
for (Item item : items) {
DDBPathMetadata meta = itemToPathMetadata(item, username);
metas.add(meta);
}
{code}
to
{code:java}
// handle expiry - only add not expired entries to listing.
if (meta.getLastUpdated() == 0 ||
!meta.isExpired(ttlTimeProvider.getMetadataTtl(),
ttlTimeProvider.getNow())) {
metas.add(meta);
}
{code}
we will filter out expired entries from the listing, so we can recover form
these kind of OOB ops.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16426) S3Guard fsck: Implement fixing mechanism for inconsistencies between S3 and MetadataStore
Gabor Bota created HADOOP-16426: --- Summary: S3Guard fsck: Implement fixing mechanism for inconsistencies between S3 and MetadataStore Key: HADOOP-16426 URL: https://issues.apache.org/jira/browse/HADOOP-16426 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.3.0 Reporter: Gabor Bota Assignee: Gabor Bota -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16425) S3Guard fsck: Export MetadataStore and S3 bucket hierarchy
Gabor Bota created HADOOP-16425: --- Summary: S3Guard fsck: Export MetadataStore and S3 bucket hierarchy Key: HADOOP-16425 URL: https://issues.apache.org/jira/browse/HADOOP-16425 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.3.0 Reporter: Gabor Bota Assignee: Gabor Bota The export should be done in a human-readable format like csv -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16424) S3Guard fsck: Check internal consistency of the MetadataStore
Gabor Bota created HADOOP-16424: --- Summary: S3Guard fsck: Check internal consistency of the MetadataStore Key: HADOOP-16424 URL: https://issues.apache.org/jira/browse/HADOOP-16424 Project: Hadoop Common Issue Type: Sub-task Reporter: Gabor Bota Assignee: Gabor Bota The internal consistency should be checked e.g for orphaned entries which can cause trouble in runtime and testing. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16423) S3Guarld fsck: Check metadata consistency between S3 and metadatastore (log)
Gabor Bota created HADOOP-16423: --- Summary: S3Guarld fsck: Check metadata consistency between S3 and metadatastore (log) Key: HADOOP-16423 URL: https://issues.apache.org/jira/browse/HADOOP-16423 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.3.0 Reporter: Gabor Bota Assignee: Gabor Bota This part is only for logging the inconsistencies. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16422) AbstractITCommitMRJob.testMRJob test failures
Gabor Bota created HADOOP-16422: --- Summary: AbstractITCommitMRJob.testMRJob test failures Key: HADOOP-16422 URL: https://issues.apache.org/jira/browse/HADOOP-16422 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.2.0 Reporter: Gabor Bota There are 4 recurring test failures based on AbstractITCommitMRJob.testMRJob in S3A: ``` [ERROR] ITestMagicCommitMRJob>AbstractITCommitMRJob.testMRJob:146->AbstractFSContractTestBase.assertIsDirectory:327 ? FileNotFound [ERROR] ITestDirectoryCommitMRJob>AbstractITCommitMRJob.testMRJob:146->AbstractFSContractTestBase.assertIsDirectory:327 ? FileNotFound [ERROR] ITestPartitionCommitMRJob>AbstractITCommitMRJob.testMRJob:146->AbstractFSContractTestBase.assertIsDirectory:327 ? FileNotFound [ERROR] ITestStagingCommitMRJob>AbstractITCommitMRJob.testMRJob:146->AbstractFSContractTestBase.assertIsDirectory:327 ? FileNotFound ``` The full stack traces: ``` [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 58.639 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.staging.integration.ITestPartitionCommitMRJob [ERROR] testMRJob(org.apache.hadoop.fs.s3a.commit.staging.integration.ITestPartitionCommitMRJob) Time elapsed: 34.236 s <<< ERROR! java.io.FileNotFoundException: Path s3a://gabota-versioned-bucket-ireland/test/DELAY_LISTING_ME/testMRJob is recorded as deleted by S3Guard at 2019-07-11T12:06:20.159Z at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2634) at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2601) at org.apache.hadoop.fs.contract.ContractTestUtils.assertIsDirectory(ContractTestUtils.java:561) at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertIsDirectory(AbstractFSContractTestBase.java:327) at org.apache.hadoop.fs.s3a.commit.AbstractITCommitMRJob.testMRJob(AbstractITCommitMRJob.java:146) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) ``` ``` [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 70.31 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitMRJob [ERROR] testMRJob(org.apache.hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitMRJob) Time elapsed: 29.917 s <<< ERROR! java.io.FileNotFoundException: Path s3a://gabota-versioned-bucket-ireland/test/DELAY_LISTING_ME/testMRJob is recorded as deleted by S3Guard at 2019-07-11T12:07:17.791Z at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2634) at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2601) at org.apache.hadoop.fs.contract.ContractTestUtils.assertIsDirectory(ContractTestUtils.java:561) at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertIsDirectory(AbstractFSContractTestBase.java:327) at org.apache.hadoop.fs.s3a.commit.AbstractITCommitMRJob.testMRJob(AbstractITCommitMRJob.java:146) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at
[jira] [Resolved] (HADOOP-16409) Allow authoritative mode on non-qualified paths
[ https://issues.apache.org/jira/browse/HADOOP-16409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16409. - Resolution: Fixed Fix Version/s: 3.3.0 > Allow authoritative mode on non-qualified paths > --- > > Key: HADOOP-16409 > URL: https://issues.apache.org/jira/browse/HADOOP-16409 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Fix For: 3.3.0 > > > fs.s3a.authoritative.path currently requires a qualified URI (e.g. > s3a://bucket/path) which is how I see this being used most immediately, but > it also make sense for someone to just be able to configure /path, if all of > their buckets follow that pattern, or if they're providing configuration > already in a bucket-specific context (e.g. job-level configs, etc.) Just need > to qualify whatever is passed in to allowAuthoritative to make that work. > Also, in HADOOP-16396 Gabor pointed out a few whitepace nits that I neglected > to fix before merging. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16383) Pass ITtlTimeProvider instance in initialize method in MetadataStore interface
Gabor Bota created HADOOP-16383:
---
Summary: Pass ITtlTimeProvider instance in initialize method in
MetadataStore interface
Key: HADOOP-16383
URL: https://issues.apache.org/jira/browse/HADOOP-16383
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 3.3.0
Reporter: Gabor Bota
Assignee: Gabor Bota
In HADOOP-16279 I added ITtlTimeProvider {{delete}}, {{deleteSubtree}},
{{move}}, methods as a parameter instead of adding it to the MetadataStore
initialization. We have to change this and pass the provider during the init
phase.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16184) S3Guard: Handle OOB deletions and creation of a file which has a tombstone marker
[
https://issues.apache.org/jira/browse/HADOOP-16184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16184.
-
Resolution: Fixed
> S3Guard: Handle OOB deletions and creation of a file which has a tombstone
> marker
> -
>
> Key: HADOOP-16184
> URL: https://issues.apache.org/jira/browse/HADOOP-16184
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> When a file is deleted in S3 using S3Guard a tombstone marker will be added
> for that file in the MetadataStore. If another process creates the file
> without using S3Guard (as an out of band operation - OOB) the file still not
> be visible for the client using S3Guard because of the deletion tombstone.
>
> The whole of S3Guard is potentially brittle to
> * OOB deletions: we skip it in HADOOP-15999, so no worse, but because the
> S3AInputStream retries on FNFE, so as to "debounce" cached 404s, it's
> potentially going to retry forever.
> * OOB creation of a file which has a deletion tombstone marker.
> The things this issue covers:
> * Write a test to simulate that deletion problem, to see what happens. We
> ought to have the S3AInputStream retry briefly on that initial GET failing,
> but only on that initial one. (after setting "fs.s3a.retry.limit" to
> something low & the interval down to 10ms or so to fail fast)
> * Sequences
> {noformat}
> 1. create; delete; open; read -> fail after retry
> 2. create; open, read, delete, read -> fail fast on the second read
> {noformat}
> The StoreStatistics of the filesystem's IGNORED_ERRORS stat will be increased
> on the ignored error, so on sequence 1 will have increased, whereas on
> sequence 2 it will not have. If either of these tests don't quite fail as
> expected, we can disable the tests and continue, at least now with some tests
> to simulate a condition we don't have a fix for.
> * For both, we just need to have some model of how long it takes for
> debouncing to stabilize. Then in this new check, if an FNFE is raised and the
> check is happening > (modtime+ debounce-delay) then it's a real FNFE.
> This issue is created based on [~ste...@apache.org] remarks and comments on
> HADOOP-15999.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15960) Update guava to 27.0-jre in hadoop-project
[ https://issues.apache.org/jira/browse/HADOOP-15960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-15960. - Resolution: Fixed All subtasks are resolved, guava updated on branches 3.0, 3.1, 3.2 and trunk. Resolving this as fixed. If update is needed on branch-2 I can create another issue for that. We need to update javac version to 8 to be compatible with this guava version or use the -android flavor. There's an ongoing discussion about this in HADOOP-16219 if you want to learn more. > Update guava to 27.0-jre in hadoop-project > -- > > Key: HADOOP-15960 > URL: https://issues.apache.org/jira/browse/HADOOP-15960 > Project: Hadoop Common > Issue Type: Bug > Components: common, security >Affects Versions: 3.1.0, 3.2.0, 3.0.3, 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Critical > Fix For: 3.3.0 > > > com.google.guava:guava should be upgraded to 27.0-jre due to new CVE's found > [CVE-2018-10237|https://nvd.nist.gov/vuln/detail/CVE-2018-10237]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-15888) ITestDynamoDBMetadataStore can leak (large) DDB tables in test failures/timeout
[ https://issues.apache.org/jira/browse/HADOOP-15888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reopened HADOOP-15888: - Reopened. This depends on HADOOP-15563 we will close when that is in. > ITestDynamoDBMetadataStore can leak (large) DDB tables in test > failures/timeout > --- > > Key: HADOOP-15888 > URL: https://issues.apache.org/jira/browse/HADOOP-15888 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.2 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Attachments: Screen Shot 2018-10-30 at 17.32.43.png > > > This is me doing some backporting of patches from branch-3.2, so it may be an > intermediate condition but > # I'd noticed I wasn't actually running ITestDynamoDBMetadataStore > # so I set it up to work with teh right config opts (table and region) > # but the tests were timing out > # looking at DDB tables in the AWS console showed a number of DDB tables > "testProvisionTable", "testProvisionTable", created, each with "500 read, 100 > write capacity (i.e. ~$50/month) > I haven't replicated this in trunk/branch-3.2 itself, but its clearly > dangerous. At the very least, we should have a size of 1 R/W in all > creations, so the cost of a test failure is neglible, and then we should > document the risk and best practise. > Also: use "s3guard" as the table prefix to make clear its origin -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15888) ITestDynamoDBMetadataStore can leak (large) DDB tables in test failures/timeout
[ https://issues.apache.org/jira/browse/HADOOP-15888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-15888. - Resolution: Workaround > ITestDynamoDBMetadataStore can leak (large) DDB tables in test > failures/timeout > --- > > Key: HADOOP-15888 > URL: https://issues.apache.org/jira/browse/HADOOP-15888 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.2 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Attachments: Screen Shot 2018-10-30 at 17.32.43.png > > > This is me doing some backporting of patches from branch-3.2, so it may be an > intermediate condition but > # I'd noticed I wasn't actually running ITestDynamoDBMetadataStore > # so I set it up to work with teh right config opts (table and region) > # but the tests were timing out > # looking at DDB tables in the AWS console showed a number of DDB tables > "testProvisionTable", "testProvisionTable", created, each with "500 read, 100 > write capacity (i.e. ~$50/month) > I haven't replicated this in trunk/branch-3.2 itself, but its clearly > dangerous. At the very least, we should have a size of 1 R/W in all > creations, so the cost of a test failure is neglible, and then we should > document the risk and best practise. > Also: use "s3guard" as the table prefix to make clear its origin -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16326) S3Guard: Remove LocalMetadataStore
Gabor Bota created HADOOP-16326: --- Summary: S3Guard: Remove LocalMetadataStore Key: HADOOP-16326 URL: https://issues.apache.org/jira/browse/HADOOP-16326 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 3.3.0 Reporter: Gabor Bota We use LocalMetadataStore MetadataStore implementation in S3Guard only for testing. Inside it uses Guava's cache for storing metadatas. We try to mimic how dynamo should work under the hood, but with every new feature or API modification what we do on MetadataStore interface level it gets more and more complicated to implement the same feature with different behavior. I want to start a debate on why we need to remove that, and why we want to keep it. I could rant about why is it annoying to have this implementation when we need to get an implementation right to work with dynamo, and then do a totally different set of modification in the LocalMetadata to get the same outcome, and also add more tests just for the MetadataStore we use for testing. There are also areas in our ever-growing testing matrix that would need more attention instead of fixing tests for our test implementation. But on the other hand, it is good that we have another impl for the API which we can use for drafting new ideas. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14468) S3Guard: make short-circuit getFileStatus() configurable
[
https://issues.apache.org/jira/browse/HADOOP-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-14468.
-
Resolution: Fixed
Release Note: Resolved this with HADOOP-15999 : fix for OOB operations.
> S3Guard: make short-circuit getFileStatus() configurable
>
>
> Key: HADOOP-14468
> URL: https://issues.apache.org/jira/browse/HADOOP-14468
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Gabor Bota
>Priority: Minor
>
> Currently, when S3Guard is enabled, getFileStatus() will skip S3 if it gets a
> result from the MetadataStore (e.g. dynamodb) first.
> I would like to add a new parameter
> {{fs.s3a.metadatastore.getfilestatus.authoritative}} which, when true, keeps
> the current behavior. When false, S3AFileSystem will check both S3 and the
> MetadataStore.
> I'm not sure yet if we want to have this behavior the same for all callers of
> getFileStatus(), or if we only want to check both S3 and MetadataStore for
> some internal callers such as open().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16280) S3Guard: Retry failed read with backoff in Authoritative mode when file can be opened
Gabor Bota created HADOOP-16280:
---
Summary: S3Guard: Retry failed read with backoff in Authoritative
mode when file can be opened
Key: HADOOP-16280
URL: https://issues.apache.org/jira/browse/HADOOP-16280
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Gabor Bota
When using S3Guard in authoritative mode a file can be reported from AWS S3
that's missing like it is described in the following exception:
{noformat}
java.io.FileNotFoundException: re-open
s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0
at 0 on
s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0:
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not
exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request
ID: E1FF9EA9B5DBBD7E; S3 Extended Request ID:
NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=),
S3 Extended Request ID:
NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=:NoSuchKey
{noformat}
But the metadata in S3Guard (e.g dynamo db) is there, so it can be opened. The
operation will not fail when it's opened, it will fail when we try to read it,
so the call
{noformat}FSDataInputStream is = guardedFs.open(testFilePath);{noformat}}
won't fail, but the next call
{noformat}
byte[] firstRead = new byte[text.length()];
is.read(firstRead, 0, firstRead.length);
{noformat}
will fail with the exception message like what's above.
Once Authoritative mode is on, we assume that there is no out of band
operation, so the file will appear eventually. We should re-try in this case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16279) S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)
Gabor Bota created HADOOP-16279:
---
Summary: S3Guard: Implement time-based (TTL) expiry for entries
(and tombstones)
Key: HADOOP-16279
URL: https://issues.apache.org/jira/browse/HADOOP-16279
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Gabor Bota
In HADOOP-15621 we implemented TTL for Authoritative Directory Listings and
added {{ExpirableMetadata}}. {{DDBPathMetadata}} extends {{PathMetadata}}
extends {{ExpirableMetadata}}, so all metadata entries in ddb can expire, but
the implementation is not done yet.
To complete this feature the following should be done:
* Add new tests for metadata entry and tombstone expiry to {{ITestS3GuardTtl}}
* Implement metadata entry and tombstone expiry
I would like to start a debate on whether we need to use separate expiry times
for entries and tombstones. My +1 on not using separate settings - so only one
config name and value.
Notes:
* In HADOOP-13649 the metadata TTL is implemented in LocalMetadataStore, using
an existing feature in guava's cache implementation. Expiry is set with
{{fs.s3a.s3guard.local.ttl}}.
* This is not the same, and not using the DDB's feature of ttl
[(DOCS)|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html].
We need stronger consistency guarantees than what ddb promises: [cleaning once
a day with a background
job|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/howitworks-ttl.html]
is not usable for this feature - although it can be used as a general cleanup
solution separately and independently from S3Guard.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15272) Update Guava, see what breaks
[ https://issues.apache.org/jira/browse/HADOOP-15272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-15272. - Resolution: Fixed Fixed in HADOOP-16210. > Update Guava, see what breaks > - > > Key: HADOOP-15272 > URL: https://issues.apache.org/jira/browse/HADOOP-15272 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > > We're still on Guava 11; the last attempt at an update (HADOOP-10101) failed > to take > The HBase 2 version of ATS should permit this, at least for its profile. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16237) Fix new findbugs issues after update guava to 27.0-jre in hadoop-project trunk
Gabor Bota created HADOOP-16237: --- Summary: Fix new findbugs issues after update guava to 27.0-jre in hadoop-project trunk Key: HADOOP-16237 URL: https://issues.apache.org/jira/browse/HADOOP-16237 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.3.0 Reporter: Gabor Bota Assignee: Gabor Bota Attachments: branch-findbugs-hadoop-common-project_hadoop-kms-warnings.html, branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html, branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html, branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-documentstore-warnings.html There are a bunch of new findbugs issues in the build after committing the guava update. Mostly in yarn, but we have to check and handle those. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16230) Correct findbug ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project
[
https://issues.apache.org/jira/browse/HADOOP-16230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16230.
-
Resolution: Won't Fix
I'll resolve it as won't fix. I will fix this in HADOOP-16210.
> Correct findbug ignores for unjustified issues during update to guava to
> 27.0-jre in hadoop-project
> ---
>
> Key: HADOOP-16230
> URL: https://issues.apache.org/jira/browse/HADOOP-16230
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> In HADOOP-16220 I’ve added
> {code:java}
>
>
>
>
>
> {code}
> instead of
> {code:java}
>
>
>
>
>
> {code}
> So it should be {{getPersistedPaxosData}} instead of {{persistPaxosData}}.
>
> The following description is correct, but the code was not:
> *Null passed for non-null parameter of
> com.google.common.base.Preconditions.checkState(boolean, String, Object,
> Object, Object) in
> org.apache.hadoop.hdfs.qjournal.server.Journal.getPersistedPaxosData(long)*
> In {{org/apache/hadoop/hdfs/qjournal/server/Journal.java:1064}} we call
> {code:java}
> Preconditions.checkState(ret != null &&
> ret.getSegmentState().getStartTxId() == segmentTxId,
> "Bad persisted data for segment %s: %s ; journal id: %s",
> segmentTxId, ret, journalId);
> {code}
> for this call findbug assumes that {{Argument 4 might be null but must not be
> null}}, but Guava 27.0's
> {{com.google.common.base.Preconditions#checkState(boolean, java.lang.String,
> java.lang.Object, java.lang.Object, java.lang.Object)}} is annotated like the
> following:
> {code:java}
> public static void checkState(
> boolean b,
> @Nullable String errorMessageTemplate,
> @Nullable Object p1,
> @Nullable Object p2,
> @Nullable Object p3) {
> {code}
> so we have {{@Nullable}} on each parameter for the method. I don't see this
> warning as justified, or need to be fixed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16230) Corrcet findbug ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project
Gabor Bota created HADOOP-16230:
---
Summary: Corrcet findbug ignores for unjustified issues during
update to guava to 27.0-jre in hadoop-project
Key: HADOOP-16230
URL: https://issues.apache.org/jira/browse/HADOOP-16230
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Gabor Bota
In HADOOP-16220 I’ve added
{code:java}
{code}
instead of
{code:java}
{code}
So it should be {{getPersistedPaxosData}} instead of {{persistPaxosData}}.
The following description is correct, but the code was not:
*Null passed for non-null parameter of
com.google.common.base.Preconditions.checkState(boolean, String, Object,
Object, Object) in
org.apache.hadoop.hdfs.qjournal.server.Journal.getPersistedPaxosData(long)*
In {{org/apache/hadoop/hdfs/qjournal/server/Journal.java:1064}} we call
{code:java}
Preconditions.checkState(ret != null &&
ret.getSegmentState().getStartTxId() == segmentTxId,
"Bad persisted data for segment %s: %s ; journal id: %s",
segmentTxId, ret, journalId);
{code}
for this call findbug assumes that {{Argument 4 might be null but must not be
null}}, but Guava 27.0's
{{com.google.common.base.Preconditions#checkState(boolean, java.lang.String,
java.lang.Object, java.lang.Object, java.lang.Object)}} is annotated like the
following:
{code:java}
public static void checkState(
boolean b,
@Nullable String errorMessageTemplate,
@Nullable Object p1,
@Nullable Object p2,
@Nullable Object p3) {
{code}
so we have {{@Nullable}} on each parameter for the method. I don't see this
warning as justified, or need to be fixed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15780) S3Guard: document how to deal with non-S3Guard processes writing data to S3Guarded buckets
[ https://issues.apache.org/jira/browse/HADOOP-15780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-15780. - Resolution: Fixed > S3Guard: document how to deal with non-S3Guard processes writing data to > S3Guarded buckets > -- > > Key: HADOOP-15780 > URL: https://issues.apache.org/jira/browse/HADOOP-15780 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Aaron Fabbri >Assignee: Gabor Bota >Priority: Major > > Our general policy for S3Guard is this: All modifiers of a bucket that is > configured for use with S3Guard, must use S3Guard. Otherwise, the > MetadataStore will not be properly updated as the S3 bucket changes and > problems will arise. > There are limited circumstances in which may be safe to have an external > (non-s3guard) process writing data. There are also scenarios where it > definitely breaks things. > I think we should start by documenting the cases that this works / does not > work for. After we've enumerated that, we can suggest enhancements as needed > to make this sort of configuration easier to use. > To get the ball rolling, some things that do not work: > - Deleting a path *p* with S3Guard, then writing a new file at path *p* > without S3guard (will still have delete marker in S3Guard, making the file > appear to be deleted but still visible in S3 due to false "eventual > consistency") (as [~ste...@apache.org] and I have discussed) > - When fs.s3a.metadatastore.authoritative is true, adding files to > directories without S3Guard, then listing with S3Guard may exclude > externally-written files from listings. > (Note, there are also S3A interop issues with other non-S3A clients even > without S3Guard, due to the unique way S3A interprets empty directory > markers). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16220) Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project
Gabor Bota created HADOOP-16220:
---
Summary: Add findbugs ignores for unjustified issues during update
to guava to 27.0-jre in hadoop-project
Key: HADOOP-16220
URL: https://issues.apache.org/jira/browse/HADOOP-16220
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 3.0.4, 3.3.0, 3.2.1, 3.1.3
Reporter: Gabor Bota
Assignee: Gabor Bota
There are some findbugs issues with the guava update that seemed unjustified
and should be fixed before the update:
* *Null passed for non-null parameter of
com.google.common.base.Preconditions.checkState(boolean, String, Object,
Object, Object) in
org.apache.hadoop.hdfs.qjournal.server.Journal.getPersistedPaxosData(long)*
In {{org/apache/hadoop/hdfs/qjournal/server/Journal.java:1064}} we call
{code:java}
Preconditions.checkState(ret != null &&
ret.getSegmentState().getStartTxId() == segmentTxId,
"Bad persisted data for segment %s: %s ; journal id: %s",
segmentTxId, ret, journalId);
{code}
for this call findbug assumes that {{Argument 4 might be null but must not be
null}}, but Guava 27.0's
{{com.google.common.base.Preconditions#checkState(boolean, java.lang.String,
java.lang.Object, java.lang.Object, java.lang.Object)}} is annotated like the
following:
{code:java}
public static void checkState(
boolean b,
@Nullable String errorMessageTemplate,
@Nullable Object p1,
@Nullable Object p2,
@Nullable Object p3) {
{code}
so we have {{@Nullable}} on each parameter for the method. I don't see this
warning as justified, or need to be fixed.
* *Null passed for non-null parameter of
com.google.common.base.Preconditions.checkArgument(boolean, String, Object) in
org.apache.hadoop.hdfs.qjournal.server.JournalNode.getLogDir(String, String)*
In org/apache/hadoop/hdfs/qjournal/server/JournalNode.java:325 we call
{code:java}
Preconditions.checkArgument(jid != null &&
!jid.isEmpty(),
"bad journal identifier: %s", jid);
{code}
for this call findbug assumes that {{Argument 3 might be null but must not be
null}}, but Guava 27.0's
{{com.google.common.base.Preconditions#checkArgument(boolean, java.lang.String,
java.lang.Object)}} is annotated like the following:
{code:java}
public static void checkArgument(
boolean b, @Nullable String errorMessageTemplate, @Nullable Object p1) {
{code}
so we have {{@Nullable}} on argument 3, and that renders the assumption
incorrect.
* *Nullcheck of jid at line 346 of value previously dereferenced in
org.apache.hadoop.hdfs.qjournal.server.JournalNode.getLogDir(String, String)*
This is about the {{assert jid != null;}} at JournalNode.java:346. IMHO that
check is for devs to inform that the variable can't be null at that point - so
just for visibility. I would leave it as is, it's not a redundant check, just
additional information. (I'm not a fan of using {{assert}} in production code,
but if it's there we can leave it).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16213) Update guava to 27.0-jre in hadoop-project branch-3.1
Gabor Bota created HADOOP-16213: --- Summary: Update guava to 27.0-jre in hadoop-project branch-3.1 Key: HADOOP-16213 URL: https://issues.apache.org/jira/browse/HADOOP-16213 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.1.0, 3.2.0, 3.1.1, 3.1.2 Reporter: Gabor Bota Assignee: Gabor Bota com.google.guava:guava should be upgraded to 27.0-jre due to new CVE's found CVE-2018-10237. This is a sub-task for branch-3.2 from HADOOP-15960 to track issues on that particular branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16212) Update guava to 27.0-jre in hadoop-project branch-3.0
Gabor Bota created HADOOP-16212: --- Summary: Update guava to 27.0-jre in hadoop-project branch-3.0 Key: HADOOP-16212 URL: https://issues.apache.org/jira/browse/HADOOP-16212 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.0.3, 3.0.2, 3.0.1, 3.0.0 Reporter: Gabor Bota Assignee: Gabor Bota com.google.guava:guava should be upgraded to 27.0-jre due to new CVE's found CVE-2018-10237. This is a sub-task for branch-3.0 from HADOOP-15960 to track issues on that particular branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16211) Update guava to 27.0-jre in hadoop-project branch-3.2 and branch-3.1
Gabor Bota created HADOOP-16211: --- Summary: Update guava to 27.0-jre in hadoop-project branch-3.2 and branch-3.1 Key: HADOOP-16211 URL: https://issues.apache.org/jira/browse/HADOOP-16211 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.1.2, 3.1.1, 3.2.0, 3.1.0 Reporter: Gabor Bota Assignee: Gabor Bota com.google.guava:guava should be upgraded to 27.0-jre due to new CVE's found CVE-2018-10237. This is a sub-task for branch-3.2 and branch-3.1 from HADOOP-15960 to track issues on those particular branches. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16210) Update guava to 27.0-jre in hadoop-project trunk
Gabor Bota created HADOOP-16210: --- Summary: Update guava to 27.0-jre in hadoop-project trunk Key: HADOOP-16210 URL: https://issues.apache.org/jira/browse/HADOOP-16210 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.3.0 Reporter: Gabor Bota Assignee: Gabor Bota com.google.guava:guava should be upgraded to 27.0-jre due to new CVE's found CVE-2018-10237. This is a sub-task for trunk from HADOOP-15960 to track issues with that particular branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16209) Create simple docker based pseudo-cluster for hdfs
Gabor Bota created HADOOP-16209: --- Summary: Create simple docker based pseudo-cluster for hdfs Key: HADOOP-16209 URL: https://issues.apache.org/jira/browse/HADOOP-16209 Project: Hadoop Common Issue Type: Sub-task Reporter: Gabor Bota Assignee: Gabor Bota As defined in HADOOP-16063 we can provide a simple docker composed based definition to start a local pseudo cluster. This could be useful for e.g release testing, and integration testing with other components. We could easily start a hdfs cluster with this for a sanity check. Docker compose files can be part of the standard distribution package in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16201) S3AFileSystem#innerMkdirs does not create parent directories
[ https://issues.apache.org/jira/browse/HADOOP-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16201. - Resolution: Invalid > S3AFileSystem#innerMkdirs does not create parent directories > > > Key: HADOOP-16201 > URL: https://issues.apache.org/jira/browse/HADOOP-16201 > Project: Hadoop Common > Issue Type: Bug >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > > S3AFileSystem#innerMkdirs currently does not create parent directories. The > code checks for the existence of the parent directories and if they do not > exist, it adds them to metadataStoreDirs list. But this list is not used and > therefore the parent directories are never created. Only the given directory > path is created. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16185) S3Guard: Optimize performance of handling OOB operations in non-authoritative mode
Gabor Bota created HADOOP-16185:
---
Summary: S3Guard: Optimize performance of handling OOB operations
in non-authoritative mode
Key: HADOOP-16185
URL: https://issues.apache.org/jira/browse/HADOOP-16185
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.1.0
Reporter: Gabor Bota
HADOOP-15999 modifies the S3Guard's non-authoritative mode, so when S3Guard
runs non-authoritative, every {{fs.getFileStatus}} will check S3 because we
don't handle the MetadataStore as a single source of truth. This has a negative
performance impact.
In other words HADOOP-15999 is going to reinstate the HEAD on every read, so
making non-auth S3Guard a bit slower. We could think about addressing that by
moving the checks into the input stream itself. That is: the first GET which
returns data will also act as the metadata check. That'd mean the read context
will need updating with some "metastoreProcessHeader" callback to invoke on the
first GET.
The good news is that because it's reading a file, its only one HTTP HEAD
request: no need to do any of the other two directory probes except in the case
that the file isn't there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16184) S3Guard: Handle OOB deletions and creation of a file which has a tombstone marker
Gabor Bota created HADOOP-16184:
---
Summary: S3Guard: Handle OOB deletions and creation of a file
which has a tombstone marker
Key: HADOOP-16184
URL: https://issues.apache.org/jira/browse/HADOOP-16184
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.1.0
Reporter: Gabor Bota
When a file is deleted in S3 using S3Guard a tombstone marker will be added for
that file in the MetadataStore. If another process creates the file without
using S3Guard (as an out of band operation - OOB) the file still not be visible
for the client using S3Guard because of the deletion tombstone.
The whole of S3Guard is potentially brittle to
* OOB deletions: we skip it in HADOOP-15999, so no worse, but because the
S3AInputStream retries on FNFE, so as to "debounce" cached 404s, it's
potentially going to retry forever.
* OOB creation of a file which has a deletion tombstone marker.
The things this issue will cover:
* Write a test to simulate that deletion problem, to see what happens. We
ought to have the S3AInputStream retry briefly on that initial GET failing, but
only on that initial one. (after setting "fs.s3a.retry.limit" to something low
& the interval down to 10ms or so to fail fast)
* Sequences
{noformat}
1. create; delete; open; read -> fail after retry
2. create; open, read, delete, read -> fail fast on the second read
{noformat}
The StoreStatistics of the filesystem's IGNORED_ERRORS stat will be increased
on the ignored error, so on sequence 1 will have increased, whereas on sequence
2 it will not have. If either of these tests don't quite fail as expected, we
can disable the tests and continue, at least now with some tests to simulate a
condition we don't have a fix for.
* For both, we just need to have some model of how long it takes for
debouncing to stabilize. Then in this new check, if an FNFE is raised and the
check is happening > (modtime+ debounce-delay) then it's a real FNFE.
This issue is created based on [~ste...@apache.org] remarks and comments on
HADOOP-15999.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16027) [DOC] Effective use of FS instances during S3A integration tests
Gabor Bota created HADOOP-16027:
---
Summary: [DOC] Effective use of FS instances during S3A
integration tests
Key: HADOOP-16027
URL: https://issues.apache.org/jira/browse/HADOOP-16027
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Gabor Bota
Assignee: Gabor Bota
While fixing HADOOP-15819 we found that a closed fs got into the static fs
cache during testing, which caused other tests to fail when the tests were
running sequentially.
We should document some best practices in the testing section on the s3 docs
with the following:
{panel}
Tests using FileSystems are fastest if they can recycle the existing FS
instance from the same JVM. If you do that, you MUST NOT close or do unique
configuration on them. If you want a guarantee of 100% isolation or an instance
with unique config, create a new instance
which you MUST close in the teardown to avoid leakage of resources.
Do not add FileSystem instances (with e.g
org.apache.hadoop.fs.FileSystem#addFileSystemForTesting) to the cache that will
be modified or closed during the test runs. This can cause other tests to fail
when using the same modified or closed FS instance. For more details see
HADOOP-15819.
{panel}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15796) ClassCastException: S3AFileSystem cannot be cast to MockS3AFileSystem when fs caching is disabled
[
https://issues.apache.org/jira/browse/HADOOP-15796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-15796.
-
Resolution: Cannot Reproduce
Closing this as I was not able to reproduce the issue during the testing for
HADOOP-15819
> ClassCastException: S3AFileSystem cannot be cast to MockS3AFileSystem when fs
> caching is disabled
> -
>
> Key: HADOOP-15796
> URL: https://issues.apache.org/jira/browse/HADOOP-15796
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> Sometimes we get {{java.io.IOException: s3a://TEST-BUCKET: FileSystem is
> closed!}} when running tests, because of filesystem caching: an fs instance
> is closed from a test teardown while another test is still using it. (That
> should be fixed in another issue.)
> To avoid test errors introduced by closed fs I've disabled fs caching while
> testing. This caused the following issue in lots of test:
> {noformat}
> [ERROR]
> testPartitionsResolution(org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedFileListing)
> Time elapsed: 1.994 s <<< ERROR!
> java.lang.ClassCastException: org.apache.hadoop.fs.s3a.S3AFileSystem cannot
> be cast to org.apache.hadoop.fs.s3a.MockS3AFileSystem
> at
> org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase.lookupWrapperFS(StagingTestBase.java:145)
> at
> org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase$JobCommitterTest.setupJob(StagingTestBase.java:294)
> {noformat}
> We should fix the casting during these tests.
> To switch of fs caching during a test (mvn verify) run, add the following to
> the config:
> {code:java}
>
> fs.s3a.impl.disable.cache
> true
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15988) Set empty directory flag to TRUE in DynamoDBMetadataStore#innerGet when using authoritative mode
Gabor Bota created HADOOP-15988:
---
Summary: Set empty directory flag to TRUE in
DynamoDBMetadataStore#innerGet when using authoritative mode
Key: HADOOP-15988
URL: https://issues.apache.org/jira/browse/HADOOP-15988
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.1.0
Reporter: Gabor Bota
Assignee: Gabor Bota
We have the following comment and implementation in DynamoDBMetadataStore:
{noformat}
// When this class has support for authoritative
// (fully-cached) directory listings, we may also be able to answer
// TRUE here. Until then, we don't know if we have full listing or
// not, thus the UNKNOWN here:
meta.setIsEmptyDirectory(
hasChildren ? Tristate.FALSE : Tristate.UNKNOWN);
{noformat}
We have authoritative listings now in dynamo since HADOOP-15621, so we should
resolve this comment, implement the solution and test it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15987) ITestDynamoDBMetadataStore should check if test ddb table set properly before the test
Gabor Bota created HADOOP-15987:
---
Summary: ITestDynamoDBMetadataStore should check if test ddb table
set properly before the test
Key: HADOOP-15987
URL: https://issues.apache.org/jira/browse/HADOOP-15987
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.1.0
Reporter: Gabor Bota
Assignee: Gabor Bota
The jira covers the following:
* We should assert that the table name is configured when DynamoDBMetadataStore
is used for testing, so the test should fail if it's not configured.
* We should assert that the test table is not the same as the production table,
as the test table could be modified and destroyed multiple times during the
test.
* This behavior should be added to the testing docs.
[Assume from junit
doc|http://junit.sourceforge.net/javadoc/org/junit/Assume.html]:
{noformat}
A set of methods useful for stating assumptions about the conditions in which a
test is meaningful. A failed assumption does not mean the code is broken, but
that the test provides no useful information. The default JUnit runner treats
tests with failing assumptions as ignored.
{noformat}
A failed assert will cause test failure instead of just skipping the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15960) Update guava to 27.0-jre in hadoop-common
Gabor Bota created HADOOP-15960: --- Summary: Update guava to 27.0-jre in hadoop-common Key: HADOOP-15960 URL: https://issues.apache.org/jira/browse/HADOOP-15960 Project: Hadoop Common Issue Type: Bug Components: common, security Affects Versions: 3.1.0 Reporter: Gabor Bota Assignee: Gabor Bota com.google.guava:guava should be upgraded to 27.0-jre due to new CVE's found CVE-2018-10237 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15947) Fix ITestDynamoDBMetadataStore test error issues
Gabor Bota created HADOOP-15947:
---
Summary: Fix ITestDynamoDBMetadataStore test error issues
Key: HADOOP-15947
URL: https://issues.apache.org/jira/browse/HADOOP-15947
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.1.0
Reporter: Gabor Bota
Assignee: Gabor Bota
When running regression hadoop-aws integration tests for HADOOP-15370, I got
the following errors in ITestDynamoDBMetadataStore:
{noformat}
[ERROR] Tests run: 40, Failures: 4, Errors: 2, Skipped: 0, Time elapsed:
177.303 s <<< FAILURE! - in
org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore
[ERROR]
testBatchWrite(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore)
Time elapsed: 1.262 s <<< ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore.doTestBatchWrite(ITestDynamoDBMetadataStore.java:365)
(...)
[ERROR]
testProvisionTable(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore)
Time elapsed: 11.394 s <<< FAILURE!
java.lang.AssertionError: expected:<20> but was:<10> (...)
org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore.testProvisionTable(ITestDynamoDBMetadataStore.java:594)
(...)
[ERROR]
testPruneUnsetsAuthoritative(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore)
Time elapsed: 2.323 s <<< ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase.testPruneUnsetsAuthoritative(MetadataStoreTestBase.java:737)
(...)
[ERROR]
testDeleteSubtree(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore)
Time elapsed: 2.377 s <<< FAILURE!
java.lang.AssertionError: Directory /ADirectory2 is null in cache (...)
[ERROR] testPutNew(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore)
Time elapsed: 1.466 s <<< FAILURE!
java.lang.AssertionError: Directory /da2 is null in cache (...)
[ERROR]
testDeleteSubtreeHostPath(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore)
Time elapsed: 2.378 s <<< FAILURE!
java.lang.AssertionError: Directory
s3a://cloudera-dev-gabor-ireland/ADirectory2 is null in cache (...)
{noformat}
I create this jira to handle and fix all of these issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15848) ITestS3AContractMultipartUploader#testMultipartUploadEmptyPart test failure
Gabor Bota created HADOOP-15848:
---
Summary:
ITestS3AContractMultipartUploader#testMultipartUploadEmptyPart test failure
Key: HADOOP-15848
URL: https://issues.apache.org/jira/browse/HADOOP-15848
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.1.1
Reporter: Gabor Bota
To reproduce failure: {{mvn verify -Dscale
-Dit.test=org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
-Dtest=skip}} against {{eu-west-1}}.
Test output:
{noformat}
[INFO] Running
org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 59.301
s <<< FAILURE! - in
org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[ERROR]
testMultipartUploadEmptyPart(org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader)
Time elapsed: 0.75 s <<< ERROR!
java.lang.IllegalArgumentException: partNumber must be between 1 and 1
inclusive, but is 0
{/noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15819) S3A integration test failures: FileSystem is closed! - without parallel test run
Gabor Bota created HADOOP-15819:
---
Summary: S3A integration test failures: FileSystem is closed! -
without parallel test run
Key: HADOOP-15819
URL: https://issues.apache.org/jira/browse/HADOOP-15819
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.1.1
Reporter: Gabor Bota
Running the integration tests for hadoop-aws {{mvn -Dscale verify}} against
Amazon AWS S3 (eu-west-1, us-west-1, with no s3guard) we see a lot of these
failures:
{noformat}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.408 s
<<< FAILURE! - in
org.apache.hadoop.fs.s3a.commit.staging.integration.ITDirectoryCommitMRJob
[ERROR]
testMRJob(org.apache.hadoop.fs.s3a.commit.staging.integration.ITDirectoryCommitMRJob)
Time elapsed: 0.027 s <<< ERROR!
java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed!
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 4.345 s
<<< FAILURE! - in
org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJob
[ERROR]
testStagingDirectory(org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJob)
Time elapsed: 0.021 s <<< ERROR!
java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed!
[ERROR]
testMRJob(org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJob)
Time elapsed: 0.022 s <<< ERROR!
java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed!
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.489 s
<<< FAILURE! - in
org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJobBadDest
[ERROR]
testMRJob(org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJobBadDest)
Time elapsed: 0.023 s <<< ERROR!
java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed!
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.695 s
<<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.magic.ITMagicCommitMRJob
[ERROR] testMRJob(org.apache.hadoop.fs.s3a.commit.magic.ITMagicCommitMRJob)
Time elapsed: 0.039 s <<< ERROR!
java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed!
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.015 s
<<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory
[ERROR]
testEverything(org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory) Time
elapsed: 0.014 s <<< ERROR!
java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed!
{noformat}
The big issue is that the tests are running in a serial manner - no test is
running on top of the other - so we should not see that the tests are failing
like this. The issue could be in how we handle
org.apache.hadoop.fs.FileSystem#CACHE - the tests should use the same
S3AFileSystem so if A test uses a FileSystem and closes it in teardown then B
test will get the same FileSystem object from the cache and try to use it, but
it is closed.
We see this a lot in our downstream testing too. It's not possible to tell that
the failed regression test result is an implementation issue in the runtime
code or a test implementation problem.
I've checked when and what closes the S3AFileSystem with a sightly modified
version of S3AFileSystem which logs the closers of the fs if an error should
occur. I'll attach this modified java file for reference. See the next example
of the result when it's running:
{noformat}
2018-10-04 00:52:25,596 [Thread-4201] ERROR s3a.S3ACloseEnforcedFileSystem
(S3ACloseEnforcedFileSystem.java:checkIfClosed(74)) - Use after close():
java.lang.RuntimeException: Using closed FS!.
at
org.apache.hadoop.fs.s3a.S3ACloseEnforcedFileSystem.checkIfClosed(S3ACloseEnforcedFileSystem.java:73)
at
org.apache.hadoop.fs.s3a.S3ACloseEnforcedFileSystem.mkdirs(S3ACloseEnforcedFileSystem.java:474)
at
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:338)
at
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:193)
at
org.apache.hadoop.fs.s3a.ITestS3AClosedFS.setup(ITestS3AClosedFS.java:40)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
[jira] [Created] (HADOOP-15798) ITestS3GuardListConsistency#testConsistentListAfterDelete failing with LocalMetadataStore
Gabor Bota created HADOOP-15798:
---
Summary: ITestS3GuardListConsistency#testConsistentListAfterDelete
failing with LocalMetadataStore
Key: HADOOP-15798
URL: https://issues.apache.org/jira/browse/HADOOP-15798
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.1.1
Reporter: Gabor Bota
Test fails constantly when running with LocalMetadataStore.
{noformat}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.junit.Assert.assertFalse(Assert.java:74)
at
org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListAfterDelete(ITestS3GuardListConsistency.java:205)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15796) ClassCastException: S3AFileSystem cannot be cast to MockS3AFileSystem when fs caching is disabled
Gabor Bota created HADOOP-15796:
---
Summary: ClassCastException: S3AFileSystem cannot be cast to
MockS3AFileSystem when fs caching is disabled
Key: HADOOP-15796
URL: https://issues.apache.org/jira/browse/HADOOP-15796
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.1.1
Reporter: Gabor Bota
Sometimes we get {{java.io.IOException: s3a://TEST-BUCKET: FileSystem is
closed!}} when running tests, because of filesystem caching: an fs instance is
closed from a test teardown while another test is still using it. (That should
be fixed in another issue.)
To avoid test errors introduced by closed fs I've disabled fs caching while
testing. This caused the following issue in lots of test:
{noformat}
[ERROR]
testPartitionsResolution(org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedFileListing)
Time elapsed: 1.994 s <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.fs.s3a.S3AFileSystem cannot be
cast to org.apache.hadoop.fs.s3a.MockS3AFileSystem
at
org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase.lookupWrapperFS(StagingTestBase.java:145)
at
org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase$JobCommitterTest.setupJob(StagingTestBase.java:294)
{noformat}
We should fix the casting during these tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15709) Move S3Guard LocalMetadataStore constants to org.apache.hadoop.fs.s3a.Constants
Gabor Bota created HADOOP-15709:
---
Summary: Move S3Guard LocalMetadataStore constants to
org.apache.hadoop.fs.s3a.Constants
Key: HADOOP-15709
URL: https://issues.apache.org/jira/browse/HADOOP-15709
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 3.2
Reporter: Gabor Bota
Assignee: Gabor Bota
Move the following constants from
{{org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore}} to
{{org.apache.hadoop.fs.s3a.Constants}} (where they should be):
* DEFAULT_MAX_RECORDS
* DEFAULT_CACHE_ENTRY_TTL_MSEC
* CONF_MAX_RECORDS
* CONF_CACHE_ENTRY_TTL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15519) KMS fails to read the existing key metadata after upgrading to JDK 1.8u171
[
https://issues.apache.org/jira/browse/HADOOP-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-15519.
-
Resolution: Duplicate
> KMS fails to read the existing key metadata after upgrading to JDK 1.8u171
> ---
>
> Key: HADOOP-15519
> URL: https://issues.apache.org/jira/browse/HADOOP-15519
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.7.3
>Reporter: Vipin Rathor
>Priority: Critical
>
> Steps to reproduce are:
> a. Setup a KMS with any OpenJDK 1.8 before u171 and create few KMS keys.
> b. Update KMS to run with OpenJDK 1.8u171 JDK and keys can't be read
> anymore, as can be seen below
> {code:java}
> hadoop key list -metadata
> : null
> {code}
> c. Going back to earlier JDK version fixes the issue.
>
> There are no direct error / stacktrace in kms.log when it is not able to read
> the key metadata. Only Java serialization INFO messages are printed, followed
> by this one empty line in log which just says:
> {code:java}
> ERROR RangerKeyStore -
> {code}
> In some cases, kms.log can also have these lines:
> {code:java}
> 2018-05-18 10:40:46,438 DEBUG RangerKmsAuthorizer - <==
> RangerKmsAuthorizer.assertAccess(null, rangerkms/node1.host@env.com
> (auth:KERBEROS), GET_METADATA)
> 2018-05-18 10:40:46,598 INFO serialization - ObjectInputFilter REJECTED:
> class org.apache.hadoop.crypto.key.RangerKeyStoreProvider$KeyMetadata, array
> length: -1, nRefs: 1, depth: 1, bytes: 147, ex: n/a
> 2018-05-18 10:40:46,598 ERROR RangerKeyStore -
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15674) Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite
Gabor Bota created HADOOP-15674:
---
Summary: Test failure TestSSLHttpServer.testExcludedCiphers with
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite
Key: HADOOP-15674
URL: https://issues.apache.org/jira/browse/HADOOP-15674
Project: Hadoop Common
Issue Type: Bug
Components: common
Affects Versions: 2.6.0
Reporter: Gabor Bota
Assignee: Szilard Nemeth
Running {{hadoop/hadoop-common-project/hadoop-common# mvn test
-Dtest="TestSSLHttpServer#testExcludedCiphers" -Dhttps.protocols=TLSv1.2
-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256}} fails with:
{noformat}
Error Message
No Ciphers in common, SSLHandshake must fail.
Stacktrace
java.lang.AssertionError: No Ciphers in common, SSLHandshake must fail.
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.http.TestSSLHttpServer.testExcludedCiphers(TestSSLHttpServer.java:178)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15622) UserGroupInformation TGT renewer refreshTime should be based on getNextTgtRenewalTime
Gabor Bota created HADOOP-15622:
---
Summary: UserGroupInformation TGT renewer refreshTime should be
based on getNextTgtRenewalTime
Key: HADOOP-15622
URL: https://issues.apache.org/jira/browse/HADOOP-15622
Project: Hadoop Common
Issue Type: Bug
Reporter: Gabor Bota
The calculation of nextRefresh in UserGroupInformation is currently based on:
{code:java}
nextRefresh = Math.max(getRefreshTime(tgt),
now + kerberosMinSecondsBeforeRelogin);
{code}
Most of the time nextRefresh = getRefreshTime(tgt). If it is renewing exactly
on refreshTime, and there are parallel operations using expired ticket. There
is a time gap that some operations might not perform until the next tgt is
obtained. Ideally, we want to keep service uninterrupted, therefore
getNextTgtRenewalTime supposed to calculate the time a few minutes before
Kerberos tgt expired to determine the nextRefresh time. It looks like we are
not using getNextTgtRenewalTime method to calculate nextRefresh instead opt-in
to use ticket expiration time as base line for nextRefresh. I think patch 2
approach can create time gap then strain on KDC server when ticket can not be
renewed. It would be better to calculate nextRefresh based on
getNextTgtRenewalTime.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15604) Test for for unprocessed items in DynamoDB metadata store
Gabor Bota created HADOOP-15604: --- Summary: Test for for unprocessed items in DynamoDB metadata store Key: HADOOP-15604 URL: https://issues.apache.org/jira/browse/HADOOP-15604 Project: Hadoop Common Issue Type: Sub-task Reporter: Gabor Bota Confirm that the I/O thresholds are the reason for unprocessed items in DynamoDB metadata store -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15511) ITestS3GuardListConsistency#testRollingRenames bad parameters passed to doTestRenameSequence
[
https://issues.apache.org/jira/browse/HADOOP-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-15511.
-
Resolution: Not A Problem
Sorry for the noise on this, as it turns out the test is working as expected.
The following happens:
* Tests a circular sequence of renames to verify that overwriting recently
* deleted files and reading recently created files from rename operations
* works as expected.
The first check of the directory (the first doTestRenameSequence) won't check
anything, but the followups are working as expected.
> ITestS3GuardListConsistency#testRollingRenames bad parameters passed to
> doTestRenameSequence
>
>
> Key: HADOOP-15511
> URL: https://issues.apache.org/jira/browse/HADOOP-15511
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> Bumped into this while working on HADOOP-15423.
> Currently the parameters passed to doTestRenameSequence are the following:
> {noformat}
> mkdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> {noformat}
> The problem is that we should check that the srcdir is moved to dstdirs, so
> it should be check in nodirs. Right now nodirs parameter checks for
> directories which are completely unrelated to the cases, so this part of the
> test should be fixed to check for dirs that are related to the cases.
> Basically the nodirs should be equal to srcdirs in this case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15511) ITestS3GuardListConsistency#testRollingRenames bad parameters passed to doTestRenameSequence doTestRenameSequence
Gabor Bota created HADOOP-15511:
---
Summary: ITestS3GuardListConsistency#testRollingRenames bad
parameters passed to doTestRenameSequence doTestRenameSequence
Key: HADOOP-15511
URL: https://issues.apache.org/jira/browse/HADOOP-15511
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Gabor Bota
Assignee: Gabor Bota
Bumped into this while working on HADOOP-15423.
Currently the parameters passed to doTestRenameSequence are the following:
{noformat}
mkdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
mkdirs: {}
srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
mkdirs: {}
srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
mkdirs: {}
srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
mkdirs: {}
srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
mkdirs: {}
srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
{noformat}
The problem is that we should check that the srcdir is moved to dstdirs, so it
should be check in nodirs. Right now nodirs parameter checks for directories
which are completely unrelated to the cases, so this part of the test should be
fixed to check for dirs that are related to the cases. Basically the nodirs
should be equal to srcdirs in this case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15480) AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
Gabor Bota created HADOOP-15480:
---
Summary: AbstractS3GuardToolTestBase.testDiffCommand fails when
using dynamo
Key: HADOOP-15480
URL: https://issues.apache.org/jira/browse/HADOOP-15480
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Gabor Bota
Assignee: Gabor Bota
When running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB, the
testDiffCommand test fails with the following:
{noformat}
testDiffCommand(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
Time elapsed: 8.059 s <<< FAILURE!
java.lang.AssertionError:
Mismatched metadata store outputs: MS D 0
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only
MS F 100
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0
MS F 100
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1
MS F 100
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3
MS F 100
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2
MS F 100
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4
S3 F 0
s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
MS F 0
s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
S3 F 0
s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
MS F 0
s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
S3 F 0
s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
MS F 0
s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
S3 F 0
s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
MS F 0
s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
S3 F 0
s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
MS F 0
s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
expected:<[s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4]> but
was:<[s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1,
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0,
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2,
s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3,
s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4,
s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4]>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at
org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testDiffCommand(AbstractS3GuardToolTestBase.java:382)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-15382) Log kinit output in credential renewal thread
[
https://issues.apache.org/jira/browse/HADOOP-15382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota reopened HADOOP-15382:
-
Backport to 2.7 (original issue reported to 2.7)
> Log kinit output in credential renewal thread
> -
>
> Key: HADOOP-15382
> URL: https://issues.apache.org/jira/browse/HADOOP-15382
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HADOOP-15382.001.patch, HADOOP-15382.002.patch
>
>
> We currently run kinit command in a thread to renew kerberos credentials
> periodically.
> {code:java}
> Shell.execCommand(cmd, "-R");
> if (LOG.isDebugEnabled()) {
> LOG.debug("renewed ticket");
> }
> {code}
> It seems useful to log the output of the kinit too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15423) Use single hash Path -> tuple(DirListingMetadata, PathMetadata) in LocalMetadataStore
Gabor Bota created HADOOP-15423:
---
Summary: Use single hash Path -> tuple(DirListingMetadata,
PathMetadata) in LocalMetadataStore
Key: HADOOP-15423
URL: https://issues.apache.org/jira/browse/HADOOP-15423
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Gabor Bota
Assignee: Gabor Bota
Right now the s3guard.LocalMetadataStore uses two HashMap in the implementation
one for the file and one for the dir hash.
{code:java}
/** Contains directories and files. */
private LruHashMap fileHash;
/** Contains directory listings. */
private LruHashMap dirHash;
{code}
It would be nice to have only one hash instead of these two for storing the
values. An idea for the implementation would be to have a class with nullable
fields:
{code:java}
static class LocalMetaEntry {
@Nullable
public PathMetadata pathMetadata;
@Nullable
public DirListingMetadata dirListingMetadata;
}
{code}
or a Pair (tuple):
{code:java}
Pair metaEntry;
{code}
And only one hash/cache for these elements.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15332) Fix typos in hadoop-aws markdown docs
Gabor Bota created HADOOP-15332: --- Summary: Fix typos in hadoop-aws markdown docs Key: HADOOP-15332 URL: https://issues.apache.org/jira/browse/HADOOP-15332 Project: Hadoop Common Issue Type: Improvement Reporter: Gabor Bota Assignee: Gabor Bota While reading through https://github.com/apache/hadoop/tree/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws I've found some very obvious typos, and I thought it would be a nice improvement to fix those. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
