[jira] [Created] (HADOOP-16409) Allow authoritative mode on non-qualified paths
Sean Mackrory created HADOOP-16409: -- Summary: Allow authoritative mode on non-qualified paths Key: HADOOP-16409 URL: https://issues.apache.org/jira/browse/HADOOP-16409 Project: Hadoop Common Issue Type: Improvement Reporter: Sean Mackrory Assignee: Sean Mackrory fs.s3a.authoritative.path currently requires a qualified URI (e.g. s3a://bucket/path) which is how I see this being used most immediately, but it also make sense for someone to just be able to configure /path, if all of their buckets follow that pattern, or if they're providing configuration already in a bucket-specific context (e.g. job-level configs, etc.) Just need to qualify whatever is passed in to allowAuthoritative to make that work. Also, in HADOOP-16396 Gabor pointed out a few whitepace nits that I neglected to fix before merging. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16396) Allow authoritative mode on a subdirectory
Sean Mackrory created HADOOP-16396: -- Summary: Allow authoritative mode on a subdirectory Key: HADOOP-16396 URL: https://issues.apache.org/jira/browse/HADOOP-16396 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Reporter: Sean Mackrory Assignee: Sean Mackrory Let's allow authoritative mode to be applied only to a subset of a bucket. This is coming primarily from a Hive warehousing use-case where Hive-managed tables can benefit from query planning, but can't speak for the rest of the bucket. This should be limited in scope and is not a general attempt to allow configuration on a per-path basis, as configuration is currently done on a per-process of a per-bucket basis. I propose a new property (we could overload fs.s3a.metadatastore.authoritative, but that seems likely to cause confusion somewhere). A string would be allowed that would then be qualified in the context of the FileSystem, and used to check if it is a prefix for a given path. If it is, we act as though authoritative mode is enabled. If not, we revert to the existing behavior of fs.s3a.metadatastore.authoritative (which in practice will probably be false, the default, if the new property is in use). Let's be clear about a few things: * Currently authoritative mode only short-cuts the process to avoid a round-trip to S3 if we know it is safe to do so. This means that even when authoritative mode is enabled for a bucket, if the metadata store does not have a complete (or "authoritative") current listing cached, authoritative mode still has no effect. This will still apply. * This will only apply to getFileStatus and listStatus, and internal calls to their internal counterparts. No other API is currently using authoritative mode to change behavior. * This will only apply to getFileStatus and listStatus calls INSIDE the configured prefix. If there is a recursvie listing on the parent of the configured prefix, no change in behavior will be observed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16321) ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors
[ https://issues.apache.org/jira/browse/HADOOP-16321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-16321. Resolution: Fixed > ITestS3ASSL+TestOpenSSLSocketFactory failing with > java.lang.UnsatisfiedLinkErrors > - > > Key: HADOOP-16321 > URL: https://issues.apache.org/jira/browse/HADOOP-16321 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, test >Affects Versions: 3.3.0 > Environment: macos >Reporter: Steve Loughran >Assignee: Sahil Takiar >Priority: Major > > the new test of HADOOP-16050 {{ITestS3ASSL}} is failing with > {{java.lang.UnsatisfiedLinkError}} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16101) Use lighter-weight alternatives to innerGetFileStatus where possible
Sean Mackrory created HADOOP-16101: -- Summary: Use lighter-weight alternatives to innerGetFileStatus where possible Key: HADOOP-16101 URL: https://issues.apache.org/jira/browse/HADOOP-16101 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Discussion in HADOOP-15999 highlighted the heaviness of a full innerGetFileStatus call, where many usages of it may need a lighter weight fileExists, etc. Let's investigate usage of innerGetFileStatus and slim it down where possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15819) S3A integration test failures: FileSystem is closed! - without parallel test run
[ https://issues.apache.org/jira/browse/HADOOP-15819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-15819. Resolution: Fixed > S3A integration test failures: FileSystem is closed! - without parallel test > run > > > Key: HADOOP-15819 > URL: https://issues.apache.org/jira/browse/HADOOP-15819 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 3.1.1 >Reporter: Gabor Bota >Assignee: Adam Antal >Priority: Critical > Attachments: HADOOP-15819.000.patch, HADOOP-15819.001.patch, > HADOOP-15819.002.patch, S3ACloseEnforcedFileSystem.java, > S3ACloseEnforcedFileSystem.java, closed_fs_closers_example_5klines.log.zip > > > Running the integration tests for hadoop-aws {{mvn -Dscale verify}} against > Amazon AWS S3 (eu-west-1, us-west-1, with no s3guard) we see a lot of these > failures: > {noformat} > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.408 > s <<< FAILURE! - in > org.apache.hadoop.fs.s3a.commit.staging.integration.ITDirectoryCommitMRJob > [ERROR] > testMRJob(org.apache.hadoop.fs.s3a.commit.staging.integration.ITDirectoryCommitMRJob) > Time elapsed: 0.027 s <<< ERROR! > java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed! > [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 4.345 > s <<< FAILURE! - in > org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJob > [ERROR] > testStagingDirectory(org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJob) > Time elapsed: 0.021 s <<< ERROR! > java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed! > [ERROR] > testMRJob(org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJob) > Time elapsed: 0.022 s <<< ERROR! > java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed! > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.489 > s <<< FAILURE! - in > org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJobBadDest > [ERROR] > testMRJob(org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJobBadDest) > Time elapsed: 0.023 s <<< ERROR! > java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed! > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.695 > s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.magic.ITMagicCommitMRJob > [ERROR] testMRJob(org.apache.hadoop.fs.s3a.commit.magic.ITMagicCommitMRJob) > Time elapsed: 0.039 s <<< ERROR! > java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed! > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.015 > s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory > [ERROR] > testEverything(org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory) > Time elapsed: 0.014 s <<< ERROR! > java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed! > {noformat} > The big issue is that the tests are running in a serial manner - no test is > running on top of the other - so we should not see that the tests are failing > like this. The issue could be in how we handle > org.apache.hadoop.fs.FileSystem#CACHE - the tests should use the same > S3AFileSystem so if A test uses a FileSystem and closes it in teardown then B > test will get the same FileSystem object from the cache and try to use it, > but it is closed. > We see this a lot in our downstream testing too. It's not possible to tell > that the failed regression test result is an implementation issue in the > runtime code or a test implementation problem. > I've checked when and what closes the S3AFileSystem with a sightly modified > version of S3AFileSystem which logs the closers of the fs if an error should > occur. I'll attach this modified java file for reference. See the next > example of the result when it's running: > {noformat} > 2018-10-04 00:52:25,596 [Thread-4201] ERROR s3a.S3ACloseEnforcedFileSystem > (S3ACloseEnforcedFileSystem.java:checkIfClosed(74)) - Use after close(): > java.lang.RuntimeException: Using closed FS!. > at > org.apache.hadoop.fs.s3a.S3ACloseEnforcedFileSystem.checkIfClosed(S3ACloseEnforcedFileSystem.java:73) > at > org.apache.hadoop.fs.s3a.S3ACloseEnforcedFileSystem.mkdirs(S3ACloseEnforcedFileSystem.java:474) > at > org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:338) > at > org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:193) > at > org.apache.hadoop.fs.s3a.ITestS3AClosedFS.setup(ITestS3AClosedFS.java:40) > at
[jira] [Resolved] (HADOOP-15841) ABFS: change createRemoteFileSystemDuringInitialization default to true
[ https://issues.apache.org/jira/browse/HADOOP-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-15841. Resolution: Won't Fix > ABFS: change createRemoteFileSystemDuringInitialization default to true > --- > > Key: HADOOP-15841 > URL: https://issues.apache.org/jira/browse/HADOOP-15841 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > > I haven't seen a way to create a working container (at least for the dfs > endpoint) except for setting > fs.azure.createRemoteFileSystemDuringInitialization=true. I personally don't > see that much of a downside to having it default to true, and it's a mild > inconvenience to remember to set it to true for some action to create a > container. I vaguely recall [~tmarquardt] considering changing this default > too. > I propose we do it? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15999) [s3a] Better support for out-of-band operations
Sean Mackrory created HADOOP-15999: -- Summary: [s3a] Better support for out-of-band operations Key: HADOOP-15999 URL: https://issues.apache.org/jira/browse/HADOOP-15999 Project: Hadoop Common Issue Type: New Feature Reporter: Sean Mackrory S3Guard was initially done on the premise that a new MetadataStore would be the source of truth, and that it wouldn't provide guarantees if updates were done without using S3Guard. I've been seeing increased demand for better support for scenarios where operations are done on the data that can't reasonably be done with S3Guard involved. For example: * A file is deleted using S3Guard, and replaced by some other tool. S3Guard can't tell the difference between the new file and delete / list inconsistency and continues to treat the file as deleted. * An S3Guard-ed file is overwritten by a longer file by some other tool. When reading the file, only the length of the original file is read. We could possibly have smarter behavior here by querying both S3 and the MetadataStore (even in cases where we may currently only query the MetadataStore in getFileStatus) and use whichever one has the higher modified time. This kills the performance boost we currently get in some workloads with the short-circuited getFileStatus, but we could keep it with authoritative mode which should give a larger performance boost. At least we'd get more correctness without authoritative mode and a clear declaration of when we can make the assumptions required to short-circuit the process. If we can't consider S3Guard the source of truth, we need to defer to S3 more. We'd need to be extra sure of any locality / time zone issues if we start relying on mod_time more directly, but currently we're tracking the modification time as returned by S3 anyway. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15860) Trailing period in file names gets ignored for some operations.
Sean Mackrory created HADOOP-15860: -- Summary: Trailing period in file names gets ignored for some operations. Key: HADOOP-15860 URL: https://issues.apache.org/jira/browse/HADOOP-15860 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory If you create a directory with a trailing period (e.g. '/test.') the period is silently dropped, and will be listed as simply '/test'. '/test.test' appears to work just fine. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15841) ABFS:
Sean Mackrory created HADOOP-15841: -- Summary: ABFS: Key: HADOOP-15841 URL: https://issues.apache.org/jira/browse/HADOOP-15841 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assignee: Sean Mackrory I haven't seen a way to create a working container (at least for the dfs endpoint) except for setting fs.azure.createRemoteFileSystemDuringInitialization=true. I personally don't see that much of a downside to having it default to true, and it's a mild inconvenience to remember to set it to true for some action to create a container. I vaguely recall [~tmarquardt] considering changing this default too. I propose we do it? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15823) ABDS: Stop requiring client ID and tenant ID for MSI
Sean Mackrory created HADOOP-15823: -- Summary: ABDS: Stop requiring client ID and tenant ID for MSI Key: HADOOP-15823 URL: https://issues.apache.org/jira/browse/HADOOP-15823 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.2.0 Reporter: Sean Mackrory ABFS requires the user to configure the tenant ID and client ID. From my understanding of MSI, that shouldn't be necessary and is an added requirement compared to MSI in ADLS. Can that be dropped? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15797) optional / builtin modules confused for cloud storage
[ https://issues.apache.org/jira/browse/HADOOP-15797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-15797. Resolution: Won't Fix > optional / builtin modules confused for cloud storage > - > > Key: HADOOP-15797 > URL: https://issues.apache.org/jira/browse/HADOOP-15797 > Project: Hadoop Common > Issue Type: Bug > Components: fs/adl, fs/azure, fs/s3 >Affects Versions: 3.2.0, 3.1.1 >Reporter: Sean Mackrory >Priority: Major > > Throwing this in your .hadooprc results in hadoop-aws being in the classpath > but not hadoop-azure*: > {quote} > hadoop_add_to_classpath_tools hadoop-aws > hadoop_add_to_classpath_tools hadoop-azure > hadoop_add_to_classpath_tools hadoop-azure-datalake > {quote} > It would seem that the core issue is that that requires the module to have > listed it's dependencies in MODULE_NAME.tools-builtin.txt, whereas the Azure > connectors only have them listed in MODULE_NAME.tools-optional.txt. S3 does > both, and there's a comment in it's POM about how it needs to do this because > of the "hadoop s3guard" CLI. > Maybe there's some history that I'm missing here, but I think what's wrong > here is that hadoop_add_to_classpath should get what it needs from optional > modules. builtin modules shouldn't even need hadoop_add_to_classpath to be > added anyway. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15797) optional / builtin modules confused for cloud storage
Sean Mackrory created HADOOP-15797: -- Summary: optional / builtin modules confused for cloud storage Key: HADOOP-15797 URL: https://issues.apache.org/jira/browse/HADOOP-15797 Project: Hadoop Common Issue Type: Bug Components: fs/adl, fs/azure, fs/s3 Reporter: Sean Mackrory Throwing this in your .hadooprc results in hadoop-aws being in the classpath but not hadoop-azure*: {quote} hadoop_add_to_classpath_tools hadoop-aws hadoop_add_to_classpath_tools hadoop-azure hadoop_add_to_classpath_tools hadoop-azure-datalake {quote} It would seem that the core issue is that that requires the module to have listed it's dependencies in MODULE_NAME.tools-builtin.txt, whereas the Azure connectors only have them listed in MODULE_NAME.tools-optional.txt. S3 does both, and there's a comment in it's POM about how it needs to do this because of the "hadoop s3guard" CLI. Maybe there's some history that I'm missing here, but I think what's wrong here is that hadoop_add_to_classpath should get what it needs from optional modules. builtin modules shouldn't even need hadoop_add_to_classpath to be added anyway. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15770) Merge HADOOP-15407 to trunk
[ https://issues.apache.org/jira/browse/HADOOP-15770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-15770. Resolution: Not A Problem > Merge HADOOP-15407 to trunk > --- > > Key: HADOOP-15770 > URL: https://issues.apache.org/jira/browse/HADOOP-15770 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Attachments: HADOOP-15770.001.patch, HADOOP-15770.002.patch > > > I started a [VOTE] thread last night. Will be posting a patch of the current > branch state to get pre-commit checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15773) Fix issues raised by Yetus
Sean Mackrory created HADOOP-15773: -- Summary: Fix issues raised by Yetus Key: HADOOP-15773 URL: https://issues.apache.org/jira/browse/HADOOP-15773 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Reporter: Sean Mackrory Assignee: Sean Mackrory I aggregated the HADOOP-15407 branch into a single patch and posted it on HADOOP-15770 just to get an aggregate report of all current issues raised by Yetus. There was a javac deprecation warning, a number of checkstyle issues, some whitespace issues, and there are a couple of valid javadoc errors I see locally. Let's fix them before we merge. I see a number of wildcard imports, all for the contents of large classes of configuration constants, and I think those should stay the way they are. There are a number of existing checkstyle issues in WASB, too that are irrelevant for the merge, and there are some field visibility issues in tests that are required that way for the tests to work as designed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15770) Merge HADOOP-15407 to trunk
Sean Mackrory created HADOOP-15770: -- Summary: Merge HADOOP-15407 to trunk Key: HADOOP-15770 URL: https://issues.apache.org/jira/browse/HADOOP-15770 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assignee: Sean Mackrory I started a [VOTE] thread last night. Will be posting a patch of the current branch state to get pre-commit checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15752) [s3a] testDestroyNoBucket always fails
[ https://issues.apache.org/jira/browse/HADOOP-15752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-15752. Resolution: Duplicate > [s3a] testDestroyNoBucket always fails > -- > > Key: HADOOP-15752 > URL: https://issues.apache.org/jira/browse/HADOOP-15752 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Reporter: Sean Mackrory >Priority: Major > > ITestS3GuardToolDynamoDB and ITestS3GuardToolLocal both fail the same way: > {code} > [INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal > [ERROR] Tests run: 23, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: > 73.198 s <<< FAILURE! - in > org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal > [ERROR] > testDestroyNoBucket(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal) > Time elapsed: 7.048 s <<< FAILURE! > java.lang.AssertionError: Expected a java.io.FileNotFoundException to be > thrown, but got the result: : 0 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15752) [s3a] testDestroyNoBucket always fails
Sean Mackrory created HADOOP-15752: -- Summary: [s3a] testDestroyNoBucket always fails Key: HADOOP-15752 URL: https://issues.apache.org/jira/browse/HADOOP-15752 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Reporter: Sean Mackrory ITestS3GuardToolDynamoDB and ITestS3GuardToolLocal both fail the same way: {code} [INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal [ERROR] Tests run: 23, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 73.198 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal [ERROR] testDestroyNoBucket(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal) Time elapsed: 7.048 s <<< FAILURE! java.lang.AssertionError: Expected a java.io.FileNotFoundException to be thrown, but got the result: : 0 {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15745) Add ABFS configuration to ConfigRedactor
Sean Mackrory created HADOOP-15745: -- Summary: Add ABFS configuration to ConfigRedactor Key: HADOOP-15745 URL: https://issues.apache.org/jira/browse/HADOOP-15745 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assignee: Sean Mackrory Sensitive information like credentials should be detected by ConfigRedactor so they never appear in logs or other channels. ABFS credentials are not all currently detected correctly so we should amend the default list of config patterns. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15729) [s3a] stop treat fs.s3a.max.threads as the long-term minimum
Sean Mackrory created HADOOP-15729: -- Summary: [s3a] stop treat fs.s3a.max.threads as the long-term minimum Key: HADOOP-15729 URL: https://issues.apache.org/jira/browse/HADOOP-15729 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory A while ago the s3a connector started experiencing deadlocks because the AWS SDK requires an unbounded threadpool. It places monitoring tasks on the work queue before the tasks they wait on, so it's possible (has even happened with larger-than-default threadpools) for the executor to become permanently saturated and deadlock. So we started giving an unbounded threadpool executor to the SDK, and using a bounded, blocking threadpool service for everything else S3A needs (although currently that's only in the S3ABlockOutputStream). fs.s3a.max.threads then only limits this threadpool, however we also specified fs.s3a.max.threads as the number of core threads in the unbounded threadpool, which in hindsight is pretty terrible. Currently those core threads do not timeout, so this is actually setting a sort of minimum. Once that many tasks have been submitted, the threadpool will be locked at that number until it bursts beyond that, but it will only spin down that far. If fs.s3a.max.threads is set reasonably high and someone uses a bunch of S3 buckets, they could easily have thousands of idle threads constantly. We should either not use fs.s3a.max.threads for the corepool size and introduce a new configuration, or we should simply allow core threads to timeout. I'm reading the OpenJDK source now to see what subtle differences there are between core threads and other threads if core threads can timeout. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15719) Fail-fast when using OAuth over http
Sean Mackrory created HADOOP-15719: -- Summary: Fail-fast when using OAuth over http Key: HADOOP-15719 URL: https://issues.apache.org/jira/browse/HADOOP-15719 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory If you configure OAuth and then use abfs:// instead of abfss:// it will fail, but it takes a very long time, and still isn't very clear why. Good place to put a good human-readable exception message and fail fast. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15666) ABFS: Compatibility tests can fail
[ https://issues.apache.org/jira/browse/HADOOP-15666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-15666. Resolution: Not A Problem Nope - I noticed at the end of last week it wasn't showing up any more. > ABFS: Compatibility tests can fail > -- > > Key: HADOOP-15666 > URL: https://issues.apache.org/jira/browse/HADOOP-15666 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Priority: Major > > Sounds like other folks aren't hitting this, but I'm seeing failures in 2 > compatibility test classes: > {code} > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.417 > s <<< FAILURE! - in > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat > [ERROR] > testBlobBackCompat(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat) > Time elapsed: 4.631 s <<< ERROR! > java.io.FileNotFoundException: > GET > http://ACCOUNT_NAME.dfs.core.windows.net/abfs-testcontainer-24b93630-e947-4139-8bbe-fcb96e61b7e4?resource=filesystem=5000=test/10=90=false > StatusCode=404 > StatusDescription=The specified path does not exist. > ErrorCode=PathNotFound > ErrorMessage=The specified path does not exist. > RequestId:cc2d3e74-d01f-007f-4911-30829500 > Time:2018-08-09T18:50:19.0627297Z > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:559) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:285) > at > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat.testBlobBackCompat(ITestAzureBlobFileSystemBackCompat.java:54) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) > Caused by: GET > http://ACCOUNT_NAME.dfs.core.windows.net/abfs-testcontainer-24b93630-e947-4139-8bbe-fcb96e61b7e4?resource=filesystem=5000=test/10=90=false > StatusCode=404 > StatusDescription=The specified path does not exist. > ErrorCode=PathNotFound > ErrorMessage=The specified path does not exist. > RequestId:cc2d3e74-d01f-007f-4911-30829500 > Time:2018-08-09T18:50:19.0627297Z > at > org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:126) > at > org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(AbfsClient.java:151) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:404) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:282) > ... 13 more > {code} > {code} > [ERROR] Tests run: 5, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: > 16.478 s <<< FAILURE! - in > org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility > [ERROR] > testListFileStatus(org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility) > Time elapsed: 6.334 s <<< FAILURE! > java.lang.AssertionError: expected:<2> but was:<1> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility.testListFileStatus(ITestWasbAbfsCompatibility.java:80) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) >
[jira] [Created] (HADOOP-15694) ABFS: Allow OAuth credentials to not be tied to accounts
Sean Mackrory created HADOOP-15694: -- Summary: ABFS: Allow OAuth credentials to not be tied to accounts Key: HADOOP-15694 URL: https://issues.apache.org/jira/browse/HADOOP-15694 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assignee: Sean Mackrory Now that there's OAuth support, it's possible to have a notion of identity that's distinct from the account itself. If a cluster is configured via OAuth with it's own identity, it's likely operators will want to use that identity regardless of which storage account a job uses. So OAuth configs right now (and probably others) are looked up with .. I propose that we add a function for looking up these configs that returns an account-specific value if it exists, but in the event it does not will also try to return , if that exists. I can work on a patch for this if nobody has any objections. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15688) ABFS: InputStream wrapped in FSDataInputStream
Sean Mackrory created HADOOP-15688: -- Summary: ABFS: InputStream wrapped in FSDataInputStream Key: HADOOP-15688 URL: https://issues.apache.org/jira/browse/HADOOP-15688 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assignee: Sean Mackrory I can't read Parquet files from ABFS. It has 2 different implementations to read seekable streams, and it'll use the one that uses ByteBuffer reads if it can. It currently decides to use the ByteBuffer read implementation because the FSDataInputStream it gets back wraps another FSDataInputStream, which implements ByteBufferReadable. That's not the most robust way to check that ByteBufferReads are supported by the ultimately underlying InputStream, but it's unnecessary and probably a mistake to double-wrap the InputStream, so let's not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15636) Forgot to add ITestDynamoDBMetadataStore, now times out
Sean Mackrory created HADOOP-15636: -- Summary: Forgot to add ITestDynamoDBMetadataStore, now times out Key: HADOOP-15636 URL: https://issues.apache.org/jira/browse/HADOOP-15636 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Gabor Bota I committed HADOOP-14918 but I forgot to 'git add' the renamed test file. I would just add it and commit and reference the JIRA, but testTableProvision is now timing out, so we should look into that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15632) Address recent API incompatibilities in cloud storage
Sean Mackrory created HADOOP-15632: -- Summary: Address recent API incompatibilities in cloud storage Key: HADOOP-15632 URL: https://issues.apache.org/jira/browse/HADOOP-15632 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory I think these are unlikely to cause any real issues, but I had to sift through a lot of reports of binary incompatibility between Hadoop 2 and Hadoop 3 and I'd like that report to be smaller next time :) HADOOP-14507 removes the single-parameter constructor from 2 classes that are marked as @Public and @Stable APIs. Propose we restore the original constructors and we document that they simply don't support per-bucket configs. HADOOP-14935 changes getFileStatusInternal from protected to private. This differs from the proposed patch and isn't discussed in the review, so I assume it's just an accident. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15437) AzureBlobFS - Services
[ https://issues.apache.org/jira/browse/HADOOP-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-15437. Resolution: Duplicate > AzureBlobFS - Services > -- > > Key: HADOOP-15437 > URL: https://issues.apache.org/jira/browse/HADOOP-15437 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15437-003.patch, HADOOP-15437.patch > > > AzureBlobFS services and factories in the driver. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15438) AzureBlobFS - Tests
[ https://issues.apache.org/jira/browse/HADOOP-15438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-15438. Resolution: Duplicate > AzureBlobFS - Tests > --- > > Key: HADOOP-15438 > URL: https://issues.apache.org/jira/browse/HADOOP-15438 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15438-001.patch, HADOOP-15438-003.patch > > > AzureBlobFS functional and contract tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15436) AzureBlobFS - Diagnostics and Utils
[ https://issues.apache.org/jira/browse/HADOOP-15436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-15436. Resolution: Duplicate > AzureBlobFS - Diagnostics and Utils > --- > > Key: HADOOP-15436 > URL: https://issues.apache.org/jira/browse/HADOOP-15436 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15436-001.patch, HADOOP-15436-003.patch > > > AzureBlobFS Diagnostics and Utils classes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15435) AzureBlobFS - Constants
[ https://issues.apache.org/jira/browse/HADOOP-15435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-15435. Resolution: Duplicate > AzureBlobFS - Constants > --- > > Key: HADOOP-15435 > URL: https://issues.apache.org/jira/browse/HADOOP-15435 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15435-001.patch, HADOOP-15435-003.patch > > > AzureBlobFS constants used across the driver. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15433) AzureBlobFS - Contracts
[ https://issues.apache.org/jira/browse/HADOOP-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-15433. Resolution: Duplicate > AzureBlobFS - Contracts > --- > > Key: HADOOP-15433 > URL: https://issues.apache.org/jira/browse/HADOOP-15433 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15433-001.patch, HADOOP-15433-003.patch > > > All the internal, external contracts for the AzureBlobFS driver. > Contracts include: > - Configuration annotations > - Configuration validation contract > - Custom exceptions > - Service contracts -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15432) AzureBlobFS - Base package classes and configuration files
[ https://issues.apache.org/jira/browse/HADOOP-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-15432. Resolution: Duplicate > AzureBlobFS - Base package classes and configuration files > -- > > Key: HADOOP-15432 > URL: https://issues.apache.org/jira/browse/HADOOP-15432 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15432-001.patch, HADOOP-15432-003.patch > > > Patch contains: > - AzureBlobFileSystem and SecureAzureBlobFileSystem classes which are the > main interfaces Hadoop interacts with. > - Updated Azure pom.xml with updated dependencies, updated parallel tests > configurations and maven shader plugin. > - Checkstyle suppression file. Since http layer is generated automatically by > another libraries, it will not follow hadoop coding guidelines. Therefore a > few rules for checkstyles have been disabled. > - Added test configuration file template to be used by the consumers. Similar > to wasb, all the configurations will go into this file. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14068) Add integration test version of TestMetadataStore for DynamoDB
[ https://issues.apache.org/jira/browse/HADOOP-14068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-14068. Resolution: Resolved Fix Version/s: HADOOP-14918 We ended up carrying this across the finish line in HADOOP-14918. > Add integration test version of TestMetadataStore for DynamoDB > -- > > Key: HADOOP-14068 > URL: https://issues.apache.org/jira/browse/HADOOP-14068 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Fix For: HADOOP-14918 > > Attachments: HADOOP-14068-HADOOP-13345.001.patch, > HADOOP-14068-HADOOP-13345.002.patch, HADOOP-14068-HADOOP-13345.003.patch, > HADOOP-14068-HADOOP-13345.004.patch, HADOOP-14068-HADOOP-13345.005.patch, > HADOOP-14068-HADOOP-13345.006.patch, HADOOP-14068-HADOOP-13345.007.patch, > HADOOP-14068-HADOOP-13345.008.patch > > > I tweaked TestDynamoDBMetadataStore to run against the actual Amazon DynamoDB > service (as opposed to the "local" edition). Several tests failed because of > minor variations in behavior. I think the differences that are clearly > possible are enough to warrant extending that class as an ITest (but > obviously keeping the existing test so 99% of the the coverage remains even > when not configured for actual DynamoDB usage). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15541) AWS SDK can mistake stream timeouts for EOF and throw SdkClientExceptions
Sean Mackrory created HADOOP-15541: -- Summary: AWS SDK can mistake stream timeouts for EOF and throw SdkClientExceptions Key: HADOOP-15541 URL: https://issues.apache.org/jira/browse/HADOOP-15541 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory I've gotten a few reports of read timeouts not being handled properly in some Impala workloads. What happens is the following sequence of events (credit to Sailesh Mukil for figuring this out): * S3AInputStream.read() gets a SocketTimeoutException when it calls wrappedStream.read() * This is handled by onReadFailure -> reopen -> closeStream. When we try to drain the stream, SdkFilterInputStream.read() in the AWS SDK fails because of checkLength. The underlying Apache Commons stream returns -1 in the case of a timeout, and EOF. * The SDK assumes the -1 signifies an EOF, so assumes the bytes read must equal expected bytes, and because they don't (because it's a timeout and not an EOF) it throws an SdkClientException. This is tricky to test for without a ton of mocking of AWS SDK internals, because you have to get into this conflicting state where the SDK has only read a subset of the expected bytes and gets a -1. closeStream will abort the stream in the event of an IOException when draining. We could simply also abort in the event of an SdkClientException. I'm testing that this results in correct functionality in the workloads that seem to hit these timeouts a lot, but all the s3a tests continue to work with that change. I'm going to open an issue with the AWS SDK Github as well, but I'm not sure what the ideal outcome would be unless there's a good way to distinguish between a stream that has timed out and a stream that read all the data without huge rewrites. Â Â -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15504) Upgrade Maven and Maven Wagon versions
Sean Mackrory created HADOOP-15504: -- Summary: Upgrade Maven and Maven Wagon versions Key: HADOOP-15504 URL: https://issues.apache.org/jira/browse/HADOOP-15504 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory I'm not even sure that Hadoop's combination of the relevant dependencies is vulnerable (even if they are, this is a relatively minor vulnerability), but this is at least showing up as an issue in automated vulnerability scans. Details can be found here [https://maven.apache.org/security.html] (CVE-2013-0253, CVE-2012-6153). Essentially the combination of maven 3.0.4 (we use 3.0, and I guess that maps to 3.0.4?) and older versions of wagon plugin don't use SSL properly. I know some dependencies can be especially troublesome to upgrade - I suspect that Maven's critical role in our build might make this risky - so if anyone has ideas for how to more completely test this than a full build, please chime in, -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15466) Correct units in adl.http.timeout
Sean Mackrory created HADOOP-15466: -- Summary: Correct units in adl.http.timeout Key: HADOOP-15466 URL: https://issues.apache.org/jira/browse/HADOOP-15466 Project: Hadoop Common Issue Type: Bug Components: fs/adl Reporter: Sean Mackrory Assignee: Sean Mackrory Comment in core-default.xml says seconds, but according to the SDK docs it's getting interpreted as milliseconds ([https://github.com/Azure/azure-data-lake-store-java/blob/master/src/main/java/com/microsoft/azure/datalake/store/ADLStoreOptions.java#L139-L144).] Pinging [~ASikaria] for any additional comment. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15442) ITestS3AMetrics.testMetricsRegister can't know metrics source's name
Sean Mackrory created HADOOP-15442: -- Summary: ITestS3AMetrics.testMetricsRegister can't know metrics source's name Key: HADOOP-15442 URL: https://issues.apache.org/jira/browse/HADOOP-15442 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory I've seen this test fail a bunch lately - mainly when the tests are all run (i.e. not individually) but not in parallel, it seems. If you dump out the sources when it fails, you see: * The sources are numbered in the hundreds, so it's very unlikely that this actually gets the first one. * The sources are numbered twice. There was logic to have the first one not be numbered, but that got messed up and now all sources are numbered twice, but the first one is only number once. We could just remove the bad assertion, but then we're only testing the registry and not anything else about the way metrics flow all the way through the whole system. Worth it to fix the failing test, I think - knowing the source gets registered doesn't add a whole lot of value toward end-to-end metrics testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15434) Upgrade to ADLS SDK that exposes current timeout
Sean Mackrory created HADOOP-15434: -- Summary: Upgrade to ADLS SDK that exposes current timeout Key: HADOOP-15434 URL: https://issues.apache.org/jira/browse/HADOOP-15434 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory HADOOP-15356 aims to expose the ADLS SDK base timeout as a configurable option in Hadoop, but this isn't very testable without being able to read it back after the write logic has been applied. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15316) GenericTestUtils can exceed maxSleepTime
Sean Mackrory created HADOOP-15316: -- Summary: GenericTestUtils can exceed maxSleepTime Key: HADOOP-15316 URL: https://issues.apache.org/jira/browse/HADOOP-15316 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory Probably shouldn't ever cause an issue, especially since Thread.sleep() can cause longer delays beyond your control anyway, but for larger values this could still behave unpredicatably in practice. {code:java} Thread.sleep(r.nextInt(maxSleepTime) + minSleepTime); {code} should be {code:java} Thread.sleep(r.nextInt(maxSleepTime - minSleepTime) + minSleepTime){code} Â -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15299) Bump Hadoop's Jackson 2 dependency 2.9.x
Sean Mackrory created HADOOP-15299: -- Summary: Bump Hadoop's Jackson 2 dependency 2.9.x Key: HADOOP-15299 URL: https://issues.apache.org/jira/browse/HADOOP-15299 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.1.0, 3.2.0 Reporter: Sean Mackrory Assignee: Sean Mackrory There are a few new CVEs open against Jackson 2.7.x. It doesn't (necessarily) mean Hadoop is vulnerable to the attack - I don't know that it is, but fixes were released for 2.8.x and 2.9.x but not 2.7.x (which we're on). We shouldn't be on an unmaintained line, regardless. HBase is already on 2.9.x, we have a shaded client now, the API changes are relatively minor and so far in my testing I haven't seen any problems. I think many of our usual reasons to hesitate upgrading this dependency don't apply. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15199) hadoop_verify_confdir prevents previously valid log4j config file names
Sean Mackrory created HADOOP-15199: -- Summary: hadoop_verify_confdir prevents previously valid log4j config file names Key: HADOOP-15199 URL: https://issues.apache.org/jira/browse/HADOOP-15199 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory When starting a daemon, the shell scripts check that there's a log4j.properties file and logs an error if there isn't one. But there appear to be several instances of files named with a prefix or suffix (for example - I found this starting up HttpFS with httpfs-log4j.properties in a Bigtop-style deployment). We should probably loosen the check a little, something like this {code:java} diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh index 2dc1dc8..df82bd2 100755 --- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh +++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh @@ -651,7 +651,7 @@ function hadoop_verify_confdir  {   # Check only log4j.properties by default.   # --loglevel does not work without logger settings in log4j.log4j.properties. - if [[ ! -f "${HADOOP_CONF_DIR}/log4j.properties" ]]; then + if [[ ! -f "${HADOOP_CONF_DIR}/*log4j*.properties" ]]; then hadoop_error "WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete."   fi  }{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15161) s3a: Stream and common statistics missing from metrics
Sean Mackrory created HADOOP-15161: -- Summary: s3a: Stream and common statistics missing from metrics Key: HADOOP-15161 URL: https://issues.apache.org/jira/browse/HADOOP-15161 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.1.0 Reporter: Sean Mackrory Assignee: Sean Mackrory Input stream statistics aren't being passed through to metrics once merged. Also, the following "common statistics" are not being incremented or tracked by metrics: {code} OP_APPEND OP_CREATE OP_CREATE_NON_RECURSIVE OP_DELETE OP_GET_CONTENT_SUMMARY OP_GET_FILE_CHECKSUM OP_GET_STATUS OP_MODIFY_ACL_ENTRIES OP_OPEN OP_REMOVE_ACL OP_REMOVE_ACL_ENTRIES OP_REMOVE_DEFAULT_ACL OP_SET_ACL OP_SET_OWNER OP_SET_PERMISSION OP_SET_TIMES OP_TRUNCATE {code} Most of those make sense, but we can easily add OP_CREATE (and it's non-recursive cousin), OP_DELETE, OP_OPEN. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15079) ITestS3AFileOperationCost#testFakeDirectoryDeletion failing after OutputCommitter patch
Sean Mackrory created HADOOP-15079: -- Summary: ITestS3AFileOperationCost#testFakeDirectoryDeletion failing after OutputCommitter patch Key: HADOOP-15079 URL: https://issues.apache.org/jira/browse/HADOOP-15079 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.1.0 Reporter: Sean Mackrory Priority: Critical I see this test failing with "object_delete_requests expected:<1> but was:<2>". I printed stack traces whenever this metric was incremented, and found the root cause to be that innerMkdirs is now causing two calls to delete fake directories when it previously caused only one. It is called once inside createFakeDirectory, and once directly inside innerMkdirs later: {code} at org.apache.hadoop.fs.s3a.S3AInstrumentation.incrementCounter(S3AInstrumentation.java:454) at org.apache.hadoop.fs.s3a.S3AFileSystem.incrementStatistic(S3AFileSystem.java:1108) at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObjects$8(S3AFileSystem.java:1369) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:313) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:279) at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:1366) at org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:1625) at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:2634) at org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(S3AFileSystem.java:2599) at org.apache.hadoop.fs.s3a.S3AFileSystem.putObjectDirect(S3AFileSystem.java:1498) at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$createEmptyObject$11(S3AFileSystem.java:2684) at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:108) at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:259) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:313) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:255) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:230) at org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:2682) at org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:2657) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:2021) at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1956) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2305) at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:338) at org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testFakeDirectoryDeletion(ITestS3AFileOperationCost.java:209) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74 {code} {code} at org.apache.hadoop.fs.s3a.S3AInstrumentation.incrementCounter(S3AInstrumentation.java:454) at org.apache.hadoop.fs.s3a.S3AFileSystem.incrementStatistic(S3AFileSystem.java:1108) at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObjects$8(S3AFileSystem.java:1369) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:313) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:279) at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:1366) at org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:1625) at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:2634) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:2025) at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1956) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2305) at
[jira] [Resolved] (HADOOP-14973) [s3a] Log StorageStatistics
[ https://issues.apache.org/jira/browse/HADOOP-14973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-14973. Resolution: Duplicate Resolving in favor of HADOOP-14475 since that's the right approach. > [s3a] Log StorageStatistics > --- > > Key: HADOOP-14973 > URL: https://issues.apache.org/jira/browse/HADOOP-14973 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1, 2.8.1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > > S3A is currently storing much more detailed metrics via StorageStatistics > than are logged in a MapReduce job. Eventually, it would be nice to get > Spark, MapReduce and other workloads to retrieve and store these metrics, but > it may be some time before they all do that. I'd like to consider having S3A > publish the metrics itself in some form. This is tricky, as S3A has no daemon > but lives inside various other processes. > Perhaps writing to a log file at some configurable interval and on close() > would be the best we could do. Other ideas would be welcome. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15055) Add s3 metrics from AWS SDK to s3a metrics tracking
Sean Mackrory created HADOOP-15055: -- Summary: Add s3 metrics from AWS SDK to s3a metrics tracking Key: HADOOP-15055 URL: https://issues.apache.org/jira/browse/HADOOP-15055 Project: Hadoop Common Issue Type: New Feature Components: fs/s3 Reporter: Sean Mackrory Assignee: Sean Mackrory -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14973) Log StorageStatistics
Sean Mackrory created HADOOP-14973: -- Summary: Log StorageStatistics Key: HADOOP-14973 URL: https://issues.apache.org/jira/browse/HADOOP-14973 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Reporter: Sean Mackrory Assignee: Sean Mackrory S3A is currently storing much more detailed metrics via StorageStatistics than are logged in a MapReduce job. Eventually, it would be nice to get Spark, MapReduce and other workloads to retrieve and store these metrics, but it may be some time before they all do that. I'd like to consider having S3A publish the metrics itself in some form. This is tricky, as S3A has no daemon but lives inside various other processes. Perhaps writing to a log file at some configurable interval and on close() would be the best we could do. Other ideas would be welcome. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14972) Histogram metrics types for latency, etc.
Sean Mackrory created HADOOP-14972: -- Summary: Histogram metrics types for latency, etc. Key: HADOOP-14972 URL: https://issues.apache.org/jira/browse/HADOOP-14972 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Reporter: Sean Mackrory Assignee: Sean Mackrory We'd like metrics to track latencies for various operations, such as latencies for various request types, etc. This may need to be done different from current metrics types that are just counters of type long, and it needs to be done intelligently as these measurements are very numerous, and are primarily interesting due to the outliers that are unpredictably far from normal. * An adaptive, sparse histogram type. I envision something configurable with a maximumum granularity and a maximum number of bins. Initially, datapoints are tallied in bins with the maximum granularity. As we reach the maximum number of bins, bins are merged in even / odd pairs. There's some complexity here, especially to make it perform well and allow safe concurrency, but I like the ability to configure reasonable limits and retain as much granularity as possible without knowing the exact shape of the data beforehand. * LongMetrics named "read_latency_600ms", "read_latency_800ms" to represent bins. This was suggested to me by [~fabbri]. I initially did not like the idea of having either so many hard-coded bins for however many op types, but this could also be done dynamically (we just hard-code which measurements we take, and with what granularity to group them, e.g. read_latency, 200 ms). The resulting dataset could be sparse and dynamic to allow for extreme outliers, but the granularity is still pre-determined. * We could also simply track a certain number of the highest latencies, and basic descriptive statistics like a running average, min / max, etc. Inherently more limited in what it can show us, but much simpler and might still provide some insight when analyzing performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14717) Add StreamCapabilities support to s3a
Sean Mackrory created HADOOP-14717: -- Summary: Add StreamCapabilities support to s3a Key: HADOOP-14717 URL: https://issues.apache.org/jira/browse/HADOOP-14717 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Sean Mackrory Assignee: Sean Mackrory -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14585) Ensure controls in-place to prevent clients with significant clock skews pruning aggressively
Sean Mackrory created HADOOP-14585: -- Summary: Ensure controls in-place to prevent clients with significant clock skews pruning aggressively Key: HADOOP-14585 URL: https://issues.apache.org/jira/browse/HADOOP-14585 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory >From discussion on HADOOP-14499: {quote} bear in mind that we can't guarantee that the clocks of all clients are in sync; you don't want a client whose TZ setting is wrong to aggressively prune things. Had that happen in production with files in shared filestore. This is why ant -diagnostics checks time consistency with temp files... {quote} {quote} temp files work on a shared FS. AWS is actually somewhat sensitive to clocks: if your VM is too far out of time then auth actually fails, its ~+-15 minutes. There's some stuff in the Java SDK to actually calculate and adjust clock skew, presumably parsing the timestamp of a failure, calculating the difference and retrying. Which means that the field in SDKGlobalConfiguration could help identify the difference between local time and AWS time. {quote} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14577) testGetFileStatus failing
Sean Mackrory created HADOOP-14577: -- Summary: testGetFileStatus failing Key: HADOOP-14577 URL: https://issues.apache.org/jira/browse/HADOOP-14577 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory This test is failing for me when run individually or in parallel (with -Ds3guard). Even if I revert back to the commit that introduced it. I thought I had successful test runs on that before and haven't changed anything in my test configuration. {code}Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.671 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AInconsistency testGetFileStatus(org.apache.hadoop.fs.s3a.ITestS3AInconsistency) Time elapsed: 4.475 sec <<< FAILURE! java.lang.AssertionError: S3Guard failed to list parent of inconsistent child. at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.fs.s3a.ITestS3AInconsistency.testGetFileStatus(ITestS3AInconsistency.java:83){code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14576) DynamoDB tables may leave ACTIVE state after initial connection
Sean Mackrory created HADOOP-14576: -- Summary: DynamoDB tables may leave ACTIVE state after initial connection Key: HADOOP-14576 URL: https://issues.apache.org/jira/browse/HADOOP-14576 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HADOOP-13345 Reporter: Sean Mackrory We currently only anticipate tables not being in the ACTIVE state when first connecting. It is possible for a table to be in the ACTIVE state and move to an UPDATING state during partitioning events. Attempts to read or write during that time will result in an AmazonServerException getting thrown. We should try to handle that better... -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14499) Findbufs warning in LocalMetadataStore
Sean Mackrory created HADOOP-14499: -- Summary: Findbufs warning in LocalMetadataStore Key: HADOOP-14499 URL: https://issues.apache.org/jira/browse/HADOOP-14499 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assignee: Sean Mackrory First saw this raised by Yetus on HADOOP-14433: {code} Bug type UC_USELESS_OBJECT (click for details) In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long) Value ancestors Type java.util.LinkedList At LocalMetadataStore.java:[line 300] {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14489) ITestS3GuardConcurrentOps requires explicit DynamoDB table name to be configured
[ https://issues.apache.org/jira/browse/HADOOP-14489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-14489. Resolution: Fixed Resolving as a duplicate. Thanks [~liuml07] > ITestS3GuardConcurrentOps requires explicit DynamoDB table name to be > configured > > > Key: HADOOP-14489 > URL: https://issues.apache.org/jira/browse/HADOOP-14489 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory > > testConcurrentTableCreations fails with this: > {quote}java.lang.IllegalArgumentException: No DynamoDB table name > configured!{quote} > I don't think that's necessary - should be able to shuffle stuff around and > either use the bucket name by default (like other DynamoDB tests would) or > use the table name that's configured later in the test. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14489) ITestS3GuardConcurrentOps requires explicit DynamoDB table name to be configured
Sean Mackrory created HADOOP-14489: -- Summary: ITestS3GuardConcurrentOps requires explicit DynamoDB table name to be configured Key: HADOOP-14489 URL: https://issues.apache.org/jira/browse/HADOOP-14489 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory testConcurrentTableCreations fails with this: {quote}java.lang.IllegalArgumentException: No DynamoDB table name configured!{quote} I don't think that's necessary - should be able to shuffle stuff around and either use the bucket name by default (like other DynamoDB tests would) or use the table name that's configured later in the test. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14484) Ensure deleted parent directory tombstones are overwritten when implicitly recreated
[ https://issues.apache.org/jira/browse/HADOOP-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-14484. Resolution: Duplicate Resolving this as a duplicate, since I did end up doing it as part of the first patch, and it makes sense to continue to do so. > Ensure deleted parent directory tombstones are overwritten when implicitly > recreated > > > Key: HADOOP-14484 > URL: https://issues.apache.org/jira/browse/HADOOP-14484 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > > As discussed on HADOOP-13998, there may be a test missing (and possibly > broken metadata store implementations) for the case where a directory is > deleted but is later implicitly recreated by creating a file inside it, where > the tombstone is not overwritten. In such a case, listing the parent > directory would result in an error. > This may also be happening because of HADOOP-14457, but we should add a test > for this other possibility anyway and fix it if it fails with any > implementations. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14484) Ensure deleted parent directory tombstones are overwritten when implicitly recreated
Sean Mackrory created HADOOP-14484: -- Summary: Ensure deleted parent directory tombstones are overwritten when implicitly recreated Key: HADOOP-14484 URL: https://issues.apache.org/jira/browse/HADOOP-14484 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory As discussed on HADOOP-13998, there may be a test missing (and possibly broken metadata store implementations) for the case where a directory is deleted but is later implicitly recreated by creating a file inside it, where the tombstone is not overwritten. In such a case, listing the parent directory would result in an error. This may also be happening because of HADOOP-14457, but we should add a test for this other possibility anyway and fix it if it fails with any implementations. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14457) LocalMetadataStore falsely reports empty parent directory authoritatively
Sean Mackrory created HADOOP-14457: -- Summary: LocalMetadataStore falsely reports empty parent directory authoritatively Key: HADOOP-14457 URL: https://issues.apache.org/jira/browse/HADOOP-14457 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Not a great test yet, but it at least reliably demonstrates the issue. LocalMetadataStore will sometimes erroneously report that a directory is empty with isAuthoritative = true when it *definitely* has children the metadatastore should know about. It doesn't appear to happen if the children are just directory. The fact that it's returning an empty listing is concerning, but the fact that it says it's authoritative *might* be a second bug. {code} diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java index 78b3970..1821d19 100644 --- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java +++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java @@ -965,7 +965,7 @@ public boolean hasMetadataStore() { } @VisibleForTesting - MetadataStore getMetadataStore() { + public MetadataStore getMetadataStore() { return metadataStore; } diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java index 4339649..881bdc9 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java @@ -23,6 +23,11 @@ import org.apache.hadoop.fs.contract.AbstractFSContract; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.s3a.S3AFileSystem; +import org.apache.hadoop.fs.s3a.Tristate; +import org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata; +import org.apache.hadoop.fs.s3a.s3guard.MetadataStore; +import org.junit.Test; import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset; import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset; @@ -72,4 +77,24 @@ public void testRenameDirIntoExistingDir() throws Throwable { boolean rename = fs.rename(srcDir, destDir); assertFalse("s3a doesn't support rename to non-empty directory", rename); } + + @Test + public void testMkdirPopulatesFileAncestors() throws Exception { +final FileSystem fs = getFileSystem(); +final MetadataStore ms = ((S3AFileSystem) fs).getMetadataStore(); +final Path parent = path("testMkdirPopulatesFileAncestors/source"); +try { + fs.mkdirs(parent); + final Path nestedFile = new Path(parent, "dir1/dir2/dir3/file4"); + byte[] srcDataset = dataset(256, 'a', 'z'); + writeDataset(fs, nestedFile, srcDataset, srcDataset.length, + 1024, false); + + DirListingMetadata list = ms.listChildren(parent); + assertTrue("MetadataStore falsely reports authoritative empty list", + list.isEmpty() == Tristate.FALSE || !list.isAuthoritative()); +} finally { + fs.delete(parent, true); +} + } } {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14448) Play nice with ITestS3AEncryptionSSEC
Sean Mackrory created HADOOP-14448: -- Summary: Play nice with ITestS3AEncryptionSSEC Key: HADOOP-14448 URL: https://issues.apache.org/jira/browse/HADOOP-14448 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HADOOP-13345 Reporter: Sean Mackrory HADOOP-14035 hasn't yet been merged with HADOOP-13345, but it adds tests that will break when run with S3Guard enabled. It expects that certain filesystem actions will throw exceptions when the client-provided encryption key is not configured properly, but those actions may sometimes bypass S3 entirely thanks to S3Guard (for example, getFileStatus may not actually need to invoke s3GetFileStatus). If the exception is never thrown, the test fails. At a minimum we should tweak the tests so they definitely invoke S3 directly, or just skip the offending tests when anything but the Null implementation is in use. This also opens the larger question of whether or not S3Guard should be serving up metadata that is otherwise only accessible when an encryption key is provided. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14288) TestDynamoDBMetadataStore is broken unless we can fail faster without a table version
Sean Mackrory created HADOOP-14288: -- Summary: TestDynamoDBMetadataStore is broken unless we can fail faster without a table version Key: HADOOP-14288 URL: https://issues.apache.org/jira/browse/HADOOP-14288 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Priority: Critical HADOOP-14215 causes TestDynamoDBMetadataStore to fail because not having a version meta-entry in the table no longer fails immediately, but must timeout all the retries. Let's make those configurable and have the test just turn them way down low. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14215) DynamoDB client should waitForActive on existing tables
Sean Mackrory created HADOOP-14215: -- Summary: DynamoDB client should waitForActive on existing tables Key: HADOOP-14215 URL: https://issues.apache.org/jira/browse/HADOOP-14215 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assignee: Sean Mackrory I saw a case where 2 separate applications tried to use the same non-pre-existing table with table.create = true at about the same time. One failed with a ResourceInUse exception. If a table does not exist, we attempt to create it and then wait for it to enter the active state. If another jumps in in the middle of that, the table may exist, thus bypassing our call to waitForActive(), and then try to use the table immediately. While we're at it, let's also make sure that the race condition where a table might get created between checking if it exists and attempting to create it is handled gracefully. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14181) Add validation of DynamoDB region
Sean Mackrory created HADOOP-14181: -- Summary: Add validation of DynamoDB region Key: HADOOP-14181 URL: https://issues.apache.org/jira/browse/HADOOP-14181 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assignee: Sean Mackrory Currently you can configure a nonsensical AWS region for DynamoDB and it doesn't stop you - it'll just start using another region silently. I propose that we just check it's a valid region for the current SDK and throw an exception instead of proceeding. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14172) S3Guard: import does not import empty directory
Sean Mackrory created HADOOP-14172: -- Summary: S3Guard: import does not import empty directory Key: HADOOP-14172 URL: https://issues.apache.org/jira/browse/HADOOP-14172 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory It imports everything comes up from listFiles, which includes only files (and their parent directories as a side-effect). My first thought on doing this would be to override S3AFileSystem to add an optional parameter to use AcceptAllButSelfAndS3nDirs instead of AcceptFilesOnly. But we could also manually traverse the tree to get all FileStatus objects directory by directory like we do for diff. That's far slower but doesn't add surface area to S3AFileSystem. But there's also the impact to other S3 clients to worry about - I could go either way on that. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14168) S3GuardTool tests should not run if S3Guard is not set up
Sean Mackrory created HADOOP-14168: -- Summary: S3GuardTool tests should not run if S3Guard is not set up Key: HADOOP-14168 URL: https://issues.apache.org/jira/browse/HADOOP-14168 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory I saw ITestS3GuardToolDynamoDB fail when running without any S3Guard configuration set up because it will run even with -Ds3guard. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14158) Possible for modified configuration to leak into metadatastore in S3GuardTool
Sean Mackrory created HADOOP-14158: -- Summary: Possible for modified configuration to leak into metadatastore in S3GuardTool Key: HADOOP-14158 URL: https://issues.apache.org/jira/browse/HADOOP-14158 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory It doesn't appear to do it when run from the command-line, but when running the S3GuardTool.run (i.e. the parent function of most of the functions used in the unit tests) from a unit test, you end up with a NullMetadataStore, regardless of what else was configured. We create an instance of S3AFileSystem with the metadata store implementation overridden to NullMetadataStore so that we have distinct interfaces to S3 and the metadata store. S3Guard can later be called using this filesystem, causing it to pick up the filesystem's configuration, which instructs it to use the NullMetadataStore implementation. This shouldn't be possible. It is unknown if this happens in any real-world scenario - I've been unable to reproduce the problem from the command-line. But it definitely happens in a test, it shouldn't, and fixing this will at least allow HADOOP-14145 to have an automated test. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14023) S3Guard: DynamoDBMetadataStore logs nonsense region
[ https://issues.apache.org/jira/browse/HADOOP-14023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-14023. Resolution: Resolved Yes this is fixed. The root of the problem was an imperfect way of translating from endpoint to region, and there is now no codepath where the region isn't determined by bucket or by configuration directly. > S3Guard: DynamoDBMetadataStore logs nonsense region > --- > > Key: HADOOP-14023 > URL: https://issues.apache.org/jira/browse/HADOOP-14023 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-13345 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Minor > Attachments: HADOOP-14023-HADOOP-13345.001.patch > > > When you specify a DynamoDB endpoint instead of it using the same one as the > S3 bucket, it thinks the region is "dynamodb": > {code} > > bin/hadoop s3a init -m dynamodb://region-test > 2017-01-25 11:41:28,006 INFO s3guard.S3GuardTool: create metadata store: > dynamodb://region-test scheme: dynamodb > 2017-01-25 11:41:28,116 WARN util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > 2017-01-25 11:41:29,421 INFO s3guard.DynamoDBMetadataStore: Creating > non-existent DynamoDB table region-test in region dynamodb > 2017-01-25 11:41:34,637 INFO s3guard.S3GuardTool: Metadata store > DynamoDBMetadataStore{region=dynamodb, tableName=region-test} is initialized. > {code} > My guess is it's because of DynamoDBMetadataStore.initialize will call > getEndpointPrefix, and all the published endpoints for DynamoDB actually > start with dynamodb. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14145) Ensure GenericOptionParser is used for S3Guard CLI
Sean Mackrory created HADOOP-14145: -- Summary: Ensure GenericOptionParser is used for S3Guard CLI Key: HADOOP-14145 URL: https://issues.apache.org/jira/browse/HADOOP-14145 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assignee: Sean Mackrory As discussed in HADOOP-14094. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14129) ITestS3ACredentialsInURL sometimes fails
Sean Mackrory created HADOOP-14129: -- Summary: ITestS3ACredentialsInURL sometimes fails Key: HADOOP-14129 URL: https://issues.apache.org/jira/browse/HADOOP-14129 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assignee: Sean Mackrory This test sometimes fails. I believe it's expected that DynamoDB doesn't have access to the credentials if they're embedded in the URL instead of the configuration (and IMO that's fine - since the functionality hasn't been in previous releases and since we want to discourage this practice especially now that there are better alternatives). Weirdly, I only sometimes get this failure on the HADOOP-13345 branch. But if the problem turns out to be what I think it is, a simple Assume should fix it. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14114) S3A can no longer handle unencoded + in URIs
Sean Mackrory created HADOOP-14114: -- Summary: S3A can no longer handle unencoded + in URIs Key: HADOOP-14114 URL: https://issues.apache.org/jira/browse/HADOOP-14114 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory Amazon secret access keys can include alphanumeric characters, but also / and + (I wish there was an official source that was really specific on what they can contain, but I'll have to rely on a few blog posts and my own experience). Keys containing slashes used to be impossible to embed in the URL (e.g. s3a://access_key:secret_key@bucket/) but it is now possible to do it via URL encoding. Pluses used to work, but that is now *only* possible via URL encoding. In the case of pluses, they don't appear to cause any other problems for parsing. So IMO the best all-around solution here is for people to URL-encode these keys always, but so that keys that used to work just fine can continue to work fine, all we need to do is detect that, log a warning, and we can re-encode it for the user. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14090) Allow users to specify region for DynamoDB table instead of endpoint
[ https://issues.apache.org/jira/browse/HADOOP-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-14090. Resolution: Won't Fix Assignee: Sean Mackrory > Allow users to specify region for DynamoDB table instead of endpoint > > > Key: HADOOP-14090 > URL: https://issues.apache.org/jira/browse/HADOOP-14090 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HADOOP-14090-HADOOP-13345.001.patch > > > Assuming the AWS SDK allows this, I think this would be a better way to > configure it for any usage on AWS itself (with endpoint still being an option > for AWS-compatible non-AWS use cases). Unless users actually care about a > specific endpoint, this is easier. Perhaps less important, HADOOP-14023 shows > that inferring the region from the endpoint (which granted, isn't that > necessary) doesn't work very well at all. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14094) Rethink S3GuardTool options
Sean Mackrory created HADOOP-14094: -- Summary: Rethink S3GuardTool options Key: HADOOP-14094 URL: https://issues.apache.org/jira/browse/HADOOP-14094 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory I think we need to rework the S3GuardTool options. A couple of problems I've observed in the patches I've done on top of that and seeing other developers trying it out: * We should probably wrap the current commands in an S3Guard-specific command, since 'init', 'destroy', etc. don't touch the buckets at all. * Convert to whole-word options, as the single-letter options are already getting overloaded. Some patches I've submitted have added functionality where the obvious flag is already in use (e.g. -r for region, and read throughput, -m for minutes, and metadatastore uri). I may do this early as part of HADOOP-14090. * We have some options that must be in the config in some cases, and can be in the command in other cases. But I've seen someone try to specify the table name in the config and leave out the -m option, with no luck. Also, since commands hard-code table auto-creation, you might have configured table auto-creation, try to import to a non-existent table, and it tells you table auto-creation is off. We need a more consistent policy for how things should get configured that addresses these problems and future-proofs the command a bit more. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14090) Allow users to specify region for DynamoDB table instead of endpoint
Sean Mackrory created HADOOP-14090: -- Summary: Allow users to specify region for DynamoDB table instead of endpoint Key: HADOOP-14090 URL: https://issues.apache.org/jira/browse/HADOOP-14090 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assuming the AWS SDK allows this, I think this would be a better way to configure it for any usage on AWS itself (with endpoint still being an option for AWS-compatible non-AWS use cases). Unless users actually care about a specific endpoint, this is easier. Perhaps less important, HADOOP-14023 shows that inferring the region from the endpoint (which granted, isn't that necessary) doesn't work very well at all. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14085) Drop unnecessary type assertion and cast
Sean Mackrory created HADOOP-14085: -- Summary: Drop unnecessary type assertion and cast Key: HADOOP-14085 URL: https://issues.apache.org/jira/browse/HADOOP-14085 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assignee: Sean Mackrory This is a spin-off from some discussion on HADOOP-13736. There's an assertion and type-cast that often causes a large number of test failures (this can be seen at HADOOP-14041). It appears to be unnecessary - things work really well by just removing that, and unless there's a good reason or test of something S3-specific, we should handle FileStatus in metadata anyway. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14068) Add integration test version of TestMetadataStore for DynamoDB
Sean Mackrory created HADOOP-14068: -- Summary: Add integration test version of TestMetadataStore for DynamoDB Key: HADOOP-14068 URL: https://issues.apache.org/jira/browse/HADOOP-14068 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assignee: Sean Mackrory I tweaked TestDynamoDBMetadataStore to run against the actual Amazon DynamoDB service (as opposed to the "local" edition). Several tests failed because of minor variations in behavior. I think the differences that are clearly possible are enough to warrant extending that class as an ITest (but obviously keeping the existing test so 99% of the the coverage remains even when not configured for actual DynamoDB usage). -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14046) Metastore destruction test creates table without version marker
Sean Mackrory created HADOOP-14046: -- Summary: Metastore destruction test creates table without version marker Key: HADOOP-14046 URL: https://issues.apache.org/jira/browse/HADOOP-14046 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assignee: Sean Mackrory ITestS3GuardToolDynamoDB.testDestroyDynamoDBMetadataStore fails after HADOOP-13985. The gist of that test is that it creates a table directly in DynamoDB then tries to destroy it through S3Guard CLI tools. Since there's another test that tries to create a table through S3Guard CLI tools then deletes it directly in DynamoDB, I propose just combing the two, so S3Guard CLI tools are used for init and destroy, and we only use other tools to check for all the correct side-effects. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14027) Implicitly creating DynamoDB table ignores endpoint config
Sean Mackrory created HADOOP-14027: -- Summary: Implicitly creating DynamoDB table ignores endpoint config Key: HADOOP-14027 URL: https://issues.apache.org/jira/browse/HADOOP-14027 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assignee: Sean Mackrory When you're using the 'bin/hadoop s3a init' command, it correctly uses the endpoint provided on the command-line (if provided), it will then use the endpoint in the config (if provided), and failing that it will default to the same region as the bucket. However if you just set fs.s3a.s3guard.ddb.table.create to true and create a directory for a new bucket / table, it will always use the same region as the bucket, even if another endpoint is configured. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14023) S3Guard:
Sean Mackrory created HADOOP-14023: -- Summary: S3Guard: Key: HADOOP-14023 URL: https://issues.apache.org/jira/browse/HADOOP-14023 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory When you specify a DynamoDB endpoint instead of it using the same one as the S3 bucket, it thinks the region is "dynamodb": {code} > bin/hadoop s3a init -m dynamodb://region-test 2017-01-25 11:41:28,006 INFO s3guard.S3GuardTool: create metadata store: dynamodb://region-test scheme: dynamodb 2017-01-25 11:41:28,116 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-01-25 11:41:29,421 INFO s3guard.DynamoDBMetadataStore: Creating non-existent DynamoDB table region-test in region dynamodb 2017-01-25 11:41:34,637 INFO s3guard.S3GuardTool: Metadata store DynamoDBMetadataStore{region=dynamodb, tableName=region-test} is initialized. {code} My guess is it's because of DynamoDBMetadataStore.initialize will call getEndpointPrefix, and all the published endpoints for DynamoDB actually start with dynamodb. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13868) Configure multi-part copies and uploads separately
Sean Mackrory created HADOOP-13868: -- Summary: Configure multi-part copies and uploads separately Key: HADOOP-13868 URL: https://issues.apache.org/jira/browse/HADOOP-13868 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory I've been looking at a big performance regression when writing to S3 from Spark that appears to have been introduced with HADOOP-12891. In the Amazon SDK, the default threshold for multi-part copies is 320x the threshold for multi-part uploads (and the block size is 20x bigger), so I don't think it's wise for us I did some quick tests and it seems to me the sweet spot when multi-part copies start being faster is around 512MB. It wasn't as significant, but using 104857600 (Amazon's default) for the blocksize was also slightly better. I propose we do the following, although they're independent. (1) Split the configuration. Ideally, I'd like to have fs.s3a.multipart.copy.threshold and fs.s3a.multipart.upload.threshold (and corresponding properties for the block size). But then there's the question of what to do with the existing fs.s3a.multipart.* properties. Deprecation? Leave it as a short-hand for configuring both (that's overridden by the more specific properties?). (2) Consider increasing the default values. In my tests, 256 MB seemed to be where multipart uploads came into their own, and 512 MB was where multipart copies started outperforming the alternative. Would be interested to hear what other people have seen. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13826) [s3a] Deadlock possible using Amazon S3 SDK
Sean Mackrory created HADOOP-13826: -- Summary: [s3a] Deadlock possible using Amazon S3 SDK Key: HADOOP-13826 URL: https://issues.apache.org/jira/browse/HADOOP-13826 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory In testing HIVE-15093 we have encountered deadlocks in the s3a connector. The TransferManager javadocs (http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html) explain how this is possible: {quote}It is not recommended to use a single threaded executor or a thread pool with a bounded work queue as control tasks may submit subtasks that can't complete until all sub tasks complete. Using an incorrectly configured thread pool may cause a deadlock (I.E. the work queue is filled with control tasks that can't finish until subtasks complete but subtasks can't execute because the queue is filled).{quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13662) Upgrade jackson2 version
Sean Mackrory created HADOOP-13662: -- Summary: Upgrade jackson2 version Key: HADOOP-13662 URL: https://issues.apache.org/jira/browse/HADOOP-13662 Project: Hadoop Common Issue Type: Improvement Reporter: Sean Mackrory Assignee: Sean Mackrory We're currently pulling in version 2.2.3 - I think we should upgrade to the latest 2.8.3. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13661) Upgrade HTrace version
Sean Mackrory created HADOOP-13661: -- Summary: Upgrade HTrace version Key: HADOOP-13661 URL: https://issues.apache.org/jira/browse/HADOOP-13661 Project: Hadoop Common Issue Type: Improvement Reporter: Sean Mackrory We're currently pulling in version 4.0.1-incubating - I think we should upgrade to the latest 4.1.0-incubating. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13660) Upgrade commons-configuration version
Sean Mackrory created HADOOP-13660: -- Summary: Upgrade commons-configuration version Key: HADOOP-13660 URL: https://issues.apache.org/jira/browse/HADOOP-13660 Project: Hadoop Common Issue Type: Improvement Reporter: Sean Mackrory Assignee: Sean Mackrory We're currently pulling in version 1.6 - I think we should upgrade to the latest 1.10. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13659) Upgrade jaxb-api version
Sean Mackrory created HADOOP-13659: -- Summary: Upgrade jaxb-api version Key: HADOOP-13659 URL: https://issues.apache.org/jira/browse/HADOOP-13659 Project: Hadoop Common Issue Type: Improvement Reporter: Sean Mackrory Assignee: Sean Mackrory We're currently pulling in version 2.2.2 - I think we should upgrade to the latest 2.2.12. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13629) Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers
Sean Mackrory created HADOOP-13629: -- Summary: Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers Key: HADOOP-13629 URL: https://issues.apache.org/jira/browse/HADOOP-13629 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory HDFS-10519 introduced a new field in the RemoteEditLogManifest message. It can be made optional to improve wire-compatibility with previous versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-12667) s3a: Support createNonRecursive API
Sean Mackrory created HADOOP-12667: -- Summary: s3a: Support createNonRecursive API Key: HADOOP-12667 URL: https://issues.apache.org/jira/browse/HADOOP-12667 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Reporter: Sean Mackrory Assignee: Sean Mackrory HBase and other clients rely on the createNonRecursive API, which was recently un-deprecated. S3A currently does not support it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12346) Increase some default timeouts / retries for S3a connector
Sean Mackrory created HADOOP-12346: -- Summary: Increase some default timeouts / retries for S3a connector Key: HADOOP-12346 URL: https://issues.apache.org/jira/browse/HADOOP-12346 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Reporter: Sean Mackrory I've been seeing some flakiness in jobs runnings against S3a, both first hand and with other accounts, for which increasing fs.s3a.connection.timeout and fs.s3a.attempts.maximum have been a reliable solution. I propose we increase the defaults. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-9641) hadoop-client should not exclude commons-httpclient
Sean Mackrory created HADOOP-9641: - Summary: hadoop-client should not exclude commons-httpclient Key: HADOOP-9641 URL: https://issues.apache.org/jira/browse/HADOOP-9641 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory The hadoop-client project excludes commons-httpclient from it's dependencies. I believe this is wrong, because it's necessary to run LocalJobRunner (e.g. submit jobs from Eclipse) and manually including it on the classpath from another location allows the client to succeed without any apparent problems. I've been unable to determine the original reason for the exclusion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9641) hadoop-client should not exclude commons-httpclient
[ https://issues.apache.org/jira/browse/HADOOP-9641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory resolved HADOOP-9641. --- Resolution: Duplicate hadoop-client should not exclude commons-httpclient --- Key: HADOOP-9641 URL: https://issues.apache.org/jira/browse/HADOOP-9641 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory The hadoop-client project excludes commons-httpclient from it's dependencies. I believe this is wrong, because it's necessary to run LocalJobRunner (e.g. submit jobs from Eclipse) and manually including it on the classpath from another location allows the client to succeed without any apparent problems. I've been unable to determine the original reason for the exclusion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9468) JVM path embedded in fuse binaries
Sean Mackrory created HADOOP-9468: - Summary: JVM path embedded in fuse binaries Key: HADOOP-9468 URL: https://issues.apache.org/jira/browse/HADOOP-9468 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Assignee: Sean Mackrory When the FUSE binaries are built, the paths to libraries in the JVMs is embedded in the RPATH so that they can be found at run-time. From an Apache Bigtop perspective, this is not sufficient because the software may be run on a machine configured very differently from the one on which they were built - so a wrapper sets LD_LIBRARY_PATH according to JAVA_HOME. I recently saw an issue where the original JVM path existed, causing LD_LIBRARY_PATH to be ignored in favor of the RPATH, but it was not the JVM intended for running Hadoop (not JAVA_HOME), and this caused problems. I'm told that setting LD_LIBRARY_PATH is standard practice before using the fuse program anyway, and if that's the case, I think removing the RPATH from the binaries is a good idea. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8795) BASH tab completion doesn't look in PATH, assumes path to executable is specified
Sean Mackrory created HADOOP-8795: - Summary: BASH tab completion doesn't look in PATH, assumes path to executable is specified Key: HADOOP-8795 URL: https://issues.apache.org/jira/browse/HADOOP-8795 Project: Hadoop Common Issue Type: Bug Reporter: Sean Mackrory Attachments: HADOOP-8795.patch bash-tab-completion/hadoop.sh checks that the first token in the command is an existing, executable file - which assumes that the path to the hadoop executable is specified (or that it's in the working directory). If the executable is somewhere else in PATH, tab completion will not work. I propose that the first token be passed through 'which' so that any executables in the path also get detected. I've tested that this technique will work in the event that relative and absolute paths are used as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira